SAP Generative AI Hub vs the Open Stack: Buy or Build

I keep bumping into conversations with architects at SAP-heavy enterprises keep asking me which stack is more advantageous for them: SAP's Generative AI Hub, or a do-it-yourself open stack built on LiteLLM, LangChain, a vector database and whatever is the latest framework. It is usually a conversation about lock-in, about SAP license cost or a technical tool choice. I think this is the wrong framing. Asking only these questions tend to produce expensive answers. The decision is not which set of tools is technically better (whatever that means). It is whether you commit to the SAP ecosystem for your AI platform or you do not, and that commitment carries technological and financial consequences that outlast any feature comparison.

Technically the comparison is less complicated than it looks. SAP does not replace the open-source orchestration packages with custom development, it embeds them. Popular packages like LangChain and LangGraph run inside the Hub, alongside the SAP Cloud SDK for AI. The offering is SAP's governed control plane around open source components. The alternative is assembling and operating those same (or comparable) components on your own. SAP sells the single gateway, the governance, the managed runtime, the one bill, the one throat to choke.

I want to frame this decision the way Gregor Hohpe describes it in the Architect Elevator: from the board level where the decision is taken, down to the engine room, where the components are wired together.

It is a commitment decision, not a tool choice

Both options solve the same technical use case: a governed path from an enterprise application to a large language model.

SAP sells a managed control plane. The Harmonized API gives you one model-agnostic interface, so you swap a model by changing its name from gpt-4o to anthropic--claude-... and nothing else in your code moves. The Orchestration service wraps every call with grounding, prompt templating, data masking and input/output content filtering. The Prompt Registry versions your templates. AI Core runs it. The proposition to a CIO is simple: check the governance box without assembling and maintaining five separate open-source projects. The price is lock-in, SAP's release cadence, slower access to potentially interesting features, and metering on SAP's terms.

The open stack sells optionality and velocity. LiteLLM is the model-agnostic gateway, an OpenAI-compatible proxy in front of more than a hundred providers, with cost tracking and guardrails bundled in. And you can swap it for Kong or Portkey if you like. LangChain and LangGraph orchestrate, if your team thinks they overabstract and are complicated to work with: Swap them. pgvector, Qdrant or Weaviate hold vectors; Microsoft Presidio masks PII; NeMo Guardrails and Llama Guard handle content safety; Langfuse manages prompts and observability. The stack is portable, multi-cloud, highly dynamic, and flexes to your needs. The price is that you own the integration, the governance correctness, and the on-call rotation.

The details: mapping the control plane component by component

Here is the basic mapping I share with architects to use in their own evaluation.

Read the table row by row: the boxes are the same, the question is who assembles and operates them.

The Harmonized API and LiteLLM do the same job, model-agnostic routing behind a stable interface, and both let you swap a provider without rewriting your application. SAP runs that gateway for you and bills you for it. With LiteLLM you run it yourself, a service you then own and operate forever.

Grounding is where the table stops being symmetric. SAP grounds on HANA Cloud's vector and knowledge-graph engines, with Datasphere behind them. If your enterprise data already lives in S/4HANA and HANA Cloud, that is not a feature, it is gravity. The open analog, pgvector plus a retriever plus maybe Neo4j for graph RAG, is perfectly capable, but it does not sit next to your business data.

Masking and content safety look have the same caveat on both sides. Presidio's own documentation says there is no guarantee it finds all PII. That caveat does not disappear when SAP runs the masking. Data masking is probabilistic wherever it happens, so the residual risk is yours to own regardless of who wrote the regex.

The bottom row, governance and audit, is the entire choice compressed into one line. SAP gives it to you built in. The open stack is "you assemble it" out of LiteLLM keys and spend tracking, Presidio, the guardrails and Langfuse. Everything in this row requires thoughtful engineering.

Which Lock-in to accept ?

Hohpe's most useful idea for this whole question is that you do not get to avoid lock-in, you only get to choose which kind you accept. Full SAP commitment is vendor lock-in plus Harmonized-API lock-in plus the data gravity of everything grounded in HANA. But the open stack is not lock-in-free. It is operational lock-in: you are locked into your own platform team, and the day three of them leave, you discover how locked in you were. Choosing OSS does not eliminate lock-in, it relocates it.

His second idea is that optionality has value, and that the value rises with uncertainty. Frontier AI in 2026 is about as uncertain as enterprise technology gets. The capable model this quarter is not the capable model next quarter, and the price per token keeps changing. In that environment being light on your feet is worth real money. Full SAP commitment sells you simplicity now and asks you to give up some of that option. The open stack buys the option and charges you a carrying cost. Neither is wrong. They are priced differently for different appetites.

Then there is the money. Both options hide the cost somewhere you are not looking. SAP meters the Generative AI Hub in Capacity Units, with LLM usage counted in blocks of provider tokens that convert to those units, on the Extended plan of AI Core. The open stack is the opposite: transparent in pricing, but carries cost that does not show up on any invoice. A production self-hosted gateway runs a few hundred dollars a month in infrastructure. You will not notice it. But the real line is twenty to thirty percent of a senior engineer or realistically more. SAP shows the cost in the contract. DIY shows it in payroll.

Capability as a Service ?

In a companion post I argued that enterprise AI adoption is an org-chart problem, and that the "force" AI multiplies is not a single quantity but a product of factors: the capability of your teams, times the clarity of who owns what, times the incentives, times governance throughput, times the knowledge the work stands on. Because it multiplies, the weakest factor makes a difference.

The platform choice is one factor you can plausibly buy: governance throughput (that is how fast a use case clears legal, compliance and the data board). The SAP Generative AI Hub raises governance throughput. You clear the legal and GRC path once, through a certified platform, instead of relitigating it for every use case. "Capability as a Service" is an accurate description of the Hub, I am not going to pretend otherwise.

However, the hub raises the one force-factor SAP can sell you. But it does nothing for the others. It cannot buy you capability in your teams, clarity about who owns the outcome, incentives that make someone spend Friday afternoon on it, or the institutional knowledge the work stands on.

If you cannot leverage the advantage of the "buy" option it will be money down the drain. It will be business features not delivered. This is also where I like to reach for Conway's Law: a system ends up copying the communication structure of the organization that built it. It is a constraint, not a design rule, and an undifferentiated, low-force organization will produce muddy architecture on any stack.

However you implement AI - as CoE AI, as Flash Team or directly with the engineering teams - it looks that way:

• Run on SAP when the mandate is AI on SAP business data, an order-to-cash or finance-close agent, because the Hub's HANA grounding and native connectivity are the velocity advantage there, and a DIY speedboat would burn its runway re-plumbing the basics such as Cloud Connector and OData. The choice makes sense when you are heavily regulated, when you value governance-by-default over velocity, and when "one throat to choke" is worth more to you than provider optionality.
• Run on an open stack when SAP is just one of many systems and you treat it only as a system of record, while your processes run agentic or in a microservices mesh. Choose the open stack when you need optionality, fast access to the bleeding edge, multi-cloud portability, and you actually have the platform team to own the governance you are taking on.

Avoid The Foul Compromise

The biggest trap is that enterprises land on a hybrid. An enterprise got into BTP and from there into the Generative AI Hub. But the Non-SAP teams are operating a microservice mesh on a different stack. Run parts on the Generative AI Hub for the SAP-grounded, regulated, governed-by-default use cases. Run other parts on the open stack for Non-SAP, and you are now operating two control planes and governing the whitespace between them. Where does masking happen for a flow that crosses both? Which registry is the source of truth? Who governs combined cost ? Those are complicated questions. Hybrid is a choice with its own cost, not a compromise.

The failure mode is drifting into SAP by because it is the path of least procurement resistance, SAP made it easy and gave you credits, and in parallel implementing the DIY stack because it had traction with engineering teams, it was cheap at first, and had the latest and greates features. If you go down that route you will discover the cost after the stacks have accumulated. And I can safely state: the cleanup is expensive.

This is the kind of conversations I spend my time on. If you want to compare notes I am easy to reach on LinkedIn or by email.