Companies are becoming fixated on selecting the best foundation model for AI. They are neglecting the fact that engineering artificial intelligence is more about the network of connections, security, and how data is routed than it is about the model you select.
The way most leaders think about AI is that building out an architecture is as simple as getting a couple of API keys and a database. In reality, there are multiple layers that must be constructed to develop applications that will not fail under load.
The Prototype Problem in the Generative AI Tech Stack
It seems easy to build a demo. A developer connects a Python script to an external AI system, provides a few documents for the AI to analyze and ask questions, and voila. Leaders see this prototype working and believe the generative AI tech stack is fully built. This is incorrect.
A prototype operates under ideal conditions. A production system must function correctly even under adverse conditions. As an application moves from a developer's local laptop to a live server, the underlying architecture crumbles. The basic three-layer model of infrastructure, models, and applications does not adequately represent the demands of running an actual business.
It ignores where authentication is done, how the various systems and models track costs associated with tokens, and what limits the potential for a model to provide someone with sensitive information.
The Generative AI Tech Stack Is Not One Tool
The generative AI tech stack is not something you can buy; it is a series of interconnected workflows. When a request is made by a user, that user must be authenticated and have a rate limit applied before the request is passed to an orchestration layer.
The orchestration layer classifies the request. The system conducts a search for documents that are permissible within the guidelines of collaboration. The search results will be filtered and added to the prompt. The model will then generate its response after filtering the search result. Finally, a final step must be taken of verifying the output for safety. Then, recording that all transactions took place so that latency and costs can be measured is required. If your architecture does not include this specific sequence of events, it may not support a true production environment.
To support the continuous and reliable execution of machine learning (ML), organizations must break technology down into separate layers. Mixing the various layers together will create locked data and broken systems.
Core Layers of a Machine Learning System
Infrastructure and Computing Resources
At the foundation of machine learning (ML), there is the hardware upon which it runs. Artificial intelligence (AI) relies heavily upon these physical machines to perform computing for ML. Organizations must choose between using publicly available commercial cloud platforms (AWS, Google Cloud, etc.) or host, maintain, and support their own infrastructure.
The difference between these two choices is based on the processors used to perform the computing for ML and AI. The most popular processor type for both training and executing models is the Graphics Processing Unit (GPU).
The GPU has largely dominated the market over other processor types and has been specifically designed for machine learning workloads. A second type of processor is the Tensor Processing Unit (TPU) which provides a cloud-based solution for specific ML workloads. Organizations will also look for ways to save money by running smaller models on a Central Processing Unit (CPU). This leads to a major latency issue that can damage the overall experience of users. The infrastructure layer creates your physical limits. If you do not secure enough computing resources, the rest of your generative AI tech stack will not function as intended.
Data Processing and Knowledge Storage
Models are not designed to interpret or understand your private business data. You must provide the model with the proper context to process and interpret your data.
To accomplish this, you need a dedicated data layer. Raw documents are stored in object storage solutions such as MinIO or in data lakes. Before a model can read and process this data, it must extract the text from the document, clean it, and split the text into manageable chunks. The vector database contains converted numerical representations of the textual content. This conversion into numerical representations is called an embedding.
Vector search engines such as Pinecone, Weaviate, and Milvus have become extremely popular for this type of storage. However, many engineering teams prefer to utilize the databases they’re already familiar with. Many engineering teams utilize extensions such as pgvector for PostgreSQL or the MongoDB Vector Search extension in order to store the mathematical representation of the text along with their traditional application data.
Foundation Models and APIs
The foundation model serves as the core of the software engineering stack. Many companies begin using proprietary models through application programming interfaces (APIs). Fast, capable, and requiring zero hardware management, OpenAI's GPT, Anthropic's Claude, and Google's Gemini continue to dominate the market for foundation models used via APIs.
Open-Weight Models
The trend is changing toward open-weight models. For example, Meta’s Llama, Mistral, and DeepSeek allow companies to download their models' core intelligence and run them on their own servers. The advantage to this approach is that the engineering team has complete control over the model's data. However, they must manage the deployment, inference scaling, and hardware costs associated with the use of that model.
Orchestration and Application Logic
Models do not contain any ability to exercise reasoning concerning how an application will flow. They only output predictions for text. Thus, to build software using foundation models, an additional layer of orchestration is necessary. Python remains the dominant programming language used for orchestration, comprising over 28 percent of the index of programming languages, although its growth rate is rapidly increasing. The next closest language, JavaScript, is still actively used for creating web interfaces but remains far behind Python for usage for orchestration.
Frameworks such as LangChain, LlamaIndex, Haystack, and DSPy are utilized to connect the various pieces together within the overall system architecture. These frameworks coordinate the movements of user input to the database, format that input into a prompt to the foundation model, call the foundation model, and parse the output from the foundation model back into a formatted response from the database to the end user.
Nonetheless, utilizing third-party orchestrator tools to deliver their services is fraught with risk. These types of applications tend to evolve rapidly over time, with their respective platforms releasing new versions (updates).
As a result, the core business logic of your company becomes locked away within these third-party orchestrators. This makes it nearly impossible to troubleshoot and debug your code whenever a breaking change occurs in the appropriate orchestration layer because of a new release offered by the third-party vendor.
Measuring, Observability, and Security
One way of understanding the fundamentals of measuring and managing a generative AI production ecosystem is to think about how the performance metrics associated with generative AI will allow you to measure the success of your generative AI solution. The key to ensuring that you monitor your generative AI production environment continuously is to use software tool kits (e.g., Fiddler, LangSmith, Weights & Biases) that will allow you to track system performance and user experience.
Measuring system performance is not just about how well your generative AI tech stack writes. There are many different types of performance metrics associated with generative AI solutions. For example, you will want to measure how long it takes for the generative AI solution to return the first word to a user.
You will also want to measure how long it took from the moment the user requested a response until they received it (also referred to as latency), how many tokens are used, and how much it costs to generate each response. Furthermore, there are several different types of security metrics associated with extracting and validating PII (personally identifiable information) and preventing prompt injection attacks against your generative AI solution.
Tool Selection Rules for Engineering Teams
Selecting the correct tool to use for your generative AI solution is easy. However, making the correct architectural trade-offs is difficult. By adding additional components to your generative AI tech stack, specifically the different types of API or the different ways that you will host your models (i.e., hosted vs. self-hosted), you introduce different levels of trade-offs with respect to speed, cost, and control.
Hosted APIs vs. Self-Hosted Infrastructure
For the early stages of building your generative AI solution, we believe it is best to use hosted APIs in order to relieve your team from the burden of buying hardware, maintaining the weights associated with their generative AI models, and optimizing their inference response time. With hosted APIs, you pay only for what you actually use.
You should only consider switching to self-hosted models once your specific use case requires that you operate completely offline, need strict data privacy, or will be generating significant volumes of requests where your per-token charges will exceed the overall costs of running your own servers to host the model. Hosting your own generative AI models will require hiring engineers who specialize in model serving. Don't change the way you implement APIs until it's necessary due to API pricing.
Retrieval-Augmented Generation (RAG) vs. Fine-Tuning
There is much debate amongst businesses about how to provide knowledge when their model lacks it. The most frequently cited solution is Retrieval-Augmented Generation (RAG). RAG allows a company to retrieve factual and current documents from their database and provide those documents to the model upon request. There is less expense associated with retrieval-augmented generation than with the fine-tuning of models. Data is updated instantaneously, and the data source can be located back to the document.
When fine-tuning is applied to a model, the internal weights of the model are modified. Fine-tuning should be approached only for the purpose of imparting a particular tone, format, or behaviour to the model. Fine-tuning is not an effective means of storing information in a model. Information is constantly changing; whenever a company makes a change to its policies, it is impractical to retrain a model.
Dedicated Vector Databases vs. Relational Extensions
Dedicated vector databases (e.g., Pinecone, Milvus) are designed for specific types of workloads: massive semantic search workloads. A dedicated vector database provides for extremely fast performance when scaled.
If your company currently utilizes a significant amount of PostgreSQL and adds a database that is not yet integrated, this adds to the complexity of your infrastructure. An alternative is to leverage the pgvector extension for PostgreSQL to accommodate both your vectors and traditional data in the same environment. Keeping it simple still wins the race when it comes to mid-scale applications.
Three Technology Stacks for Different Business Problems
Different technology stacks are required for different business-related problems. Companies should avoid trying to create a single stack across their whole organization as it guarantees failure.
The Internal Knowledge Repository Stack
More than 20% of a knowledge worker's time is spent looking for information. The goal of an internal knowledge repository is to resolve this issue. Since user volume is relatively small (employees of one's organization), a lean stack is adequate.
The codebase for the application is written either in Python or TypeScript and utilizes a cloud-based API which eliminates the need to manage a physical server. Data resides in a PostgreSQL database with pgvector for data storing purposes, thereby minimizing the complexity of the infrastructure. Retrieval of documents will be handled by a lightweight retrieval service only, and the outputs will be monitored using basic tracing technology. The primary consideration at this time is ensuring document security and ensuring citation accuracy, not ultra-high retrieval speed.
Enterprise Customer Support Automation
When customers are involved with one's application, there are many more variables at play. This stack has to retrieve account information, provide product troubleshooting assistance, and provide responses to questions in a manner that does not create embarrassment for the user nor present them with "hallucinated" or inaccurate material.
Given the requirements of an enterprise customer support automation, this stack requires a cloud-based object storage repository integrated to a governed document catalogue. Embedding management is provided through a separate service, and thus the vector search must include rigorous metadata filtering.
The metadata filtering ensures that one customer has no visibility or access to another customer's data. Furthermore, this particular stack has introduced an additional layer to the overall structure: the model gateway. If the primary model provider becomes unavailable, any requests sent to the gateway will automatically be routed to a backup model provider.
The High-Scale Commercial Product
When a software company embeds generative artificial intelligence into its core application, the stack complexity increases significantly. While generative AI could generate trillions in economic value globally, for that to happen, the software applications must be able to handle extreme demand from users without crashing. Therefore, it is important to build a stack architecture that can manage multiple model providers simultaneously. Additionally, a routing system must be in place to parse incoming requests.
Basic tasks are handled by a less expensive, less capable model that is likely to be much faster than the up-market, complex reasoning model. Both models will have common responses cached so the same answer is not charged twice. Batch jobs processed as queue-based jobs address the processing of work in the background. Dedicated serving infrastructure will automatically scale to accommodate unexpected peaks in traffic.
Control the System, Not Just the Model
The industry is learning a difficult lesson regarding the importance of data quality over model size. An application based on a large foundational model with poor data chunking will fail. An application using a smaller, less expensive model with a well-defined retrieval database will succeed.
Engineering teams must anticipate failure and understand the types of structural breaks that cause failures in deployed applications:
Improper retrieval of irrelevant data from the database, returning stale or completely incorrect documents, creating incorrect responses from the model.
Silent regressions in the model that affect structured coding outputs without notice.
Unintended token costs incurred by runaway orchestration loops.
Framework-breaking changes that render any custom application logic inoperable in the blink of an eye.
Stop attempting to find one universal tool that meets all generative capabilities; create a controlled pipeline that can be monitored. Remember, the stack is not the model. The generative AI tech stack is the system that prevents the model from failing.
Q&A: Surviving the AI Deployment Cycle
Why do most retrieval systems fail in production?
They generally fail because the teams treat text as raw data without processing it properly. Teams take entire manuals, dump them into vector databases, without cutting the text into logical chunks. So when a user asks a specific question, the database returns a random group of paragraphs, out of context, causing the model to hallucinate an incorrect answer based on incorrect input. Without structuring the data prior to it entering into the database, failure is likely.
How do teams deal with the rapidly declining effectiveness of AI tools?
The tool landscape is very fragmented and not very mature. Orchestration libraries have constant major version releases. They modify code in major ways that break existing code. Teams need to isolate their application logic from these external tools. Create clean boundaries for your critical business decisions with framework-based integrations, so if one fails you aren't required to rewrite your entire application.
What is the real cost of scaling these systems?
The actual costs of operating these systems have been severely underestimated. While the obvious costs are related to the token fees or charges associated with the system, the true costs of running these systems include: (1) storing millions of vectors, (2) the compute resources necessary for computing embeddings and latency impact on user experience, and (3) the human capital necessary for supporting testing and deployment pipelines. The actual cost of running a system should be determined based on overall system overhead and not just from the invoice provided by the model provider.
When is it appropriate to abandon open-source frameworks?
When they cause more complexity than they reduce. Many teams are using overly complicated orchestration libraries to perform very simple prompts and responses. If this is the case, when you're spending a lot of time debugging the internal workings of the framework rather than working on improving your application, write your own code. Simplicity is scaled. Needless obfuscation is your greatest enemy!