Enterprise AI is typically viewed as consisting of a “shop” or “data warehouse” with AI capabilities. However, value from AI is only realized once we focus on building out those orchestration and connective components, along with understanding how these components impact unit economics and governance; moving beyond model selection and choosing the “cloud tool” collection.
This article outlines the various components that make up an effective enterprise ai tech stack, and includes information on how to construct a scalable enterprise AI system. This includes the various layers of reference architecture required and the various operational and workflow processes required to support the build out of scalable AI capabilities.
Building an Enterprise AI Tech Stack
A successful AI product needs to be constructed upon a multi-layered technology stack to ensure it is functioning reliably and creating value for the business. AI stacks are not just a simple list of software tools or vendor solutions.
They represent an interrelated series of decisions made by an organization regarding the full lifecycle of development and continual improvement of a deployed AI product. This includes what data the ai tech stack has access to, where it is hosted, the infrastructure that supports it, and how we control access to data.
According to a recent McKinsey survey, approximately 88% of organizations are using some form of AI. Of these organizations, approximately one-third have been able to scale their AI programs across the entire enterprise.
The remaining two-thirds do not report any measurable impact on their financial results in the form of increased revenue or profits. This disconnect between testing and scaling relates to how AI solutions are built. Too often, organizations deploy excessive and unnecessary foundational models without first building the required data governance and supporting infrastructure.
How Leaders Define AI Layers
As one example, Intel is a leader within the industry for defining the core architecture of AI solutions within four layers: application, model, data, and infrastructure.
ITI (Information Technology Industry Council) is an industry leader for defining the architecture of AI solutions. They extend Intel's architecture by including model operations and API management, as well as a CGR layer for cross-cutting governance.
Modern production architecture has advanced significantly from earlier generations of technology stacks. In addition to supporting the full lifecycle of AI development, modern production architecture treats retrieval, orchestration, and observability as three separate categories. These are continuously interfaced with each other and provide for the creation of independent layers of technology to support the AI lifecycle.
The selection of appropriate tools for the layers in an application's architecture determines how well the application can manage the security of its private data, how effectively it can control its computing costs, and how accurately it can deliver answers to users.
AI Tech Stack Core Layers
When building an ai tech stack for a production environment, a common approach is to separate the different responsibilities of the various components of the stack into distinct layers. By separating the components of an AI stack into different layers, engineering teams can easily upgrade, replace, or even swap components without affecting the rest of the application.
Hardware and Infrastructure
The infrastructure layer consists of all the components of an AI stack that provide the raw processing power to train models, process data, and produce results. The computational needs of AI workloads involve extreme levels of parallel processing, which standard web servers are not capable of.
For the majority of computation, organizations turn to central processing units (CPUs) to execute business logic. However, in situations where heavy computation is required, engineers leverage graphics processing units (GPUs) and specialized AI accelerators, such as tensor processing units (TPUs), to perform the intense mathematical processing required of AI workloads.
Large organizations, such as Amazon Web Services (AWS), Microsoft Azure, and Google Cloud, provide this hardware on demand for organizations and enterprises to use to build and run AI applications. In the case of edge computing or local deployments, businesses build their own enterprise data centers or utilize specific hardware solutions, such as NVIDIA Jetson, to host AI workloads. The type of hardware that an organization utilizes for training and inference affects both the overall throughput and total costs associated with their AI stack.
Data Ingestion and Enterprise Storage
The data layer is responsible for managing the storage and processing of business data. It is critical that this layer works effectively to collect, cleanse, and store business data properly. To meet this need, standard tools are often SQL databases, data lakes, and large enterprise data warehouses, and teams utilize ETL (Extract, Transform, Load) tools to transfer data from internal business applications to their storage solutions.
Machine learning feature stores allow you to quickly find your data for training your model. If you have outdated data in the layer below or the data layer is slow, each layer on top of it will generate poor predictions or results.
Frameworks to Develop Models
Once developers create their final models, they need to build them, train them, and upgrade them. Frameworks provide developers with both the math and coding libraries needed to get their work done.
Most developers will use either PyTorch or TensorFlow as the framework for developing their complex models. They can also use libraries like Keras or Scikit-learn for traditional machine learning tasks.
There are some companies, such as Hugging Face Transformers, that provide libraries of ready-made models that make it easy to download and test open-weight models. Frameworks set the way that the engineering teams structure their coding and testing of early concepts.
Foundation Models and Domain Models
The model layer contains the actual artificial intelligence engines or "brains" that will run the AI algorithms. This is where the large language models (LLMs), basic classifiers, embedding models, and other specialized domain models come into play.
Companies typically have the option of either using managed API-based models such as GPT (from OpenAI), Claude (from Anthropic), or Gemini (from Google). Alternatively, they can download and host their own open-weight models using technologies such as Llama (from Meta) and Mistral on their own private servers.
As such, the model layer typically serves as an intermediary that allows the overall system to route different types of tasks to different models. Simple tasks can use smaller, faster models, while more complex reasoning tasks can use larger, heavier foundation models.
MLOps and Model Operations
Deploying a model on a laptop is easy. However, for a large number of users to be able to access it, it requires an operational infrastructure that is designed to keep it running correctly, efficiently, and safely. MLOps tools provide the management of a model's serving, versioning, and deployment.
Some MLOps tools include MLflow, Kubeflow, and Ray Serve, which enable teams to manage multiple versions of a model. If a new model has issues and needs to be reverted, it will revert back to the last known-good version automatically.
AWS SageMaker and Azure Machine Learning provide managed environments to perform these functions. By implementing proper MLOps, teams reduce operational downtime and keep system response times consistent during heavy user load.
Vector Databases and Retrieval
One of the difficulties with generative AI models is that they are unable to generate information about a topic they have not been trained on. We can fix this issue through the layer of retrieval, where we are able to provide the models with access to the internal information within a company (internally hosted documents).
The conversion of textual content into numeric representations through embedding models, and the storage of those numeric representations within a vector database, has become one of the leading solutions for content storage and retrieval. Examples of tools include Pinecone, Milvus, Weaviate, pgvector, and Elasticsearch.
When a user types a question, the vector database will search through millions of internal documents, locate the best matching paragraphs, and return that information back to the language model. This process, known as retrieval-augmented generation (RAG), is the best known method for preventing hallucination from generative AI models.
Orchestration Tools and AI Agents
Language models are only capable of generating text; they cannot perform any actions upon their own. This is where orchestration tools come into play and allow the language model to interconnect to third-party tools, control the logic of a conversation, and assist in performing multiple tasks simultaneously.
LangChain, LlamaIndex, and Semantic Kernel are some of the many orchestration tools available to developers. They provide management of prompt chain creation and resend prompts to the user when a task fails to complete or provide a response. Additionally, they enable an internal memory of the conversation and a means to store that memory for future work.
The agentic stack consists of all of the capabilities of an AI that can be employed in various ways to provide value to the user. One such function is the ability to search the internet, read a schedule, and initiate payments in the chain of command. When constructing an application for the end-user, the orchestration function must ensure that all of these actions are performed in the right sequence.
Application Interface and Integration
The business application programming interface (API) is the final software layer that provides the interface between the user of the AI and the business system. It is also where the business logic, API gateways, and visual design reside.
Typically, developers utilize React or Next.js to create front-end applications for their users. They integrate their applications into existing business applications and systems via REST APIs, webhooks, and secure connectors. The user experience in a business application should be seamless and responsive, as most of the application processing occurs within the infrastructure of various hardware layers.
Observability and Telemetry
Telemetry, observability, and evaluation provide insight to the user so that any issues related to model drift from the original baseline performance can be rapidly detected. If a user attempts to inject malicious prompts, these issues can be quickly identified and corrected through telemetry alerts sent to the engineering team.
Teams use logging tools and evaluation platforms to determine what they are spending on each token and what delay is present for each request. They also measure the degree of false responses for each request.
Security and Governance
The security and governance functions span all vertical layers of the technology stack, not just the last step in the stack. To achieve this, teams implement the NIST AI Risk Management Framework in developing their governance and security policies.
Identity and Access Management (IAM) and Data Loss Prevention (DLP) tools limit what users can see to what they are authorized to see. This added layer provides audit logging, meets compliance requirements, and preserves all private data during the model inference process.
Four Reference Architecture Patterns
Because AI projects are sometimes treated as traditional software development with no distinction, customers pay a significantly higher cost for their AI projects. The tech stack for a basic text classifier is significantly different than that of an autonomous business agent.
Traditional Predictive Machine Learning Stack
This stack is designed to process structured data with known, predictable outcomes. Examples of where this architecture could be used include credit scoring, fraud detection, and simple product recommendations.
The data layer continues to be the primary focus of this stack. Engineers spend the majority of their effort on feature extraction and structured data processing, rather than prompt engineering. Models in this stack are generally smaller in size, more highly specific to their task, and based on an internal hosting structure.
This stack is based heavily upon traditional MLOps, SQL databases, and model serving platforms instead of large language models (LLMs) and vector databases.
Modern Generative AI and RAG Stack
Organizations require the ability to train AI models that can respond to questions about their proprietary internal documents. The solution for this is a retrieval-augmented generation (RAG) architecture.
RAG architectures are designed to bypass the resource-intensive training period of traditional ML. These architectures typically take advantage of a managed foundation model API and a strong vector database. In a RAG stack, data flows from company files into a chunking tool, is converted into embeddings, and is retained in a vector database.
In response to a prompt, the orchestration layer initiates a query of the database to locate the precise factual information about the company. The foundation model is then called upon to provide an answer based on only these factual elements.
Autonomous Agentic AI Stack
The agentic stack of AI enables an agent-oriented methodology for the AI's ability to create a series of steps or actions, access other software applications or tools, and perform a workflow.
According to the McKinsey survey, 62% of the companies surveyed are testing agentic AI technologies. However, only 23% of them have successfully implemented the technology across the entire enterprise. This disparity is primarily due to the multifaceted challenges associated with the orchestration layer and the complexity of building it.
An agentic stack requires explicit permission models, processes for managing states between different agents or actors, and validation processes for all actions taken by agents. It also requires human-in-the-loop systems to review actions taken by agents prior to implementation. Even a minor error within an agentic stack can lead to significant consequences, including the transmission of incorrect email messages or modification of online business records without the approval of a higher level of authority.
Edge and Local AI Stack
In some cases, the delay between submitting data to a cloud data center and receiving a response may be deemed as unacceptable because it is extremely costly, subject to legal restrictions, or both. In these instances, edge AI employs local processing, enabling models to run directly on prepared hardware devices, production floors, or mobile devices not directly connected to the cloud.
Edge architecture is dependent upon the use of very small open-weight models that must be highly optimized. Therefore, edge designs require an increased emphasis on localized hardware acceleration and device-specific operating system solutions. Edge devices will experience short periods of lost connectivity and thus must immediately complete a task without needing to wait for data transmission latency between the edge device and the corresponding application.
Core Decision Factors for Your AI Tech Stack
When selecting any tools from the layers above, there are three core operational and functional questions to answer when examining those tools.
If these are not adequately considered, the resulting platforms may appear to be viable when evaluated during testing, but ultimately they will fail to perform under the real-world constraints of a business.
Data Sensitivity and Control Mechanisms
Data sensitivity has a direct impact on where any given model should reside. Sending customer financial records to a public API for a model is a huge security breach.
If the data is highly secured, the infrastructure should be private networking, cloud hosting for each geographic region, or completely on-prem physical servers. This makes the team use self-hosted weights as opposed to the more convenient managed services. If the information is public or of no concern, then the team can use fast, external APIs to implement immediately.
Latency Constraints and Unit Economics
A foundation model may have a high score on an intelligence benchmark. However, if the response time is ten seconds and the cost to make the request is three cents, it could destroy any high-volume application.
The teams must assess the exact expenses to complete each successful task. Most often, high-volume, predictable workloads must be supported by dedicated inference servers and smaller, highly optimized models so that expenses are kept low. If sub-second response time is required, the system should be designed to include caching, streaming data, and optimized routing to eliminate the use of large language models.
Operational Ownership and Managed Services
Managed services remove the headache of managing the physical infrastructure. They also allow small teams to build applications in weeks. However, they also create complete dependence on an external provider's price structure and the availability of the infrastructure.
In contrast, self-hosting an AI stack gives the business full ownership of the data and the servers, but it requires an immense amount of engineering expertise. The team will have to provide their own security updates, hardware updates, and handle scaling based on traffic spikes.
AI Tech Stack Implementation Steps
The implementation process of turning a concept into an operating, scalable ai tech stack is a step-by-step process. Production failures usually result from companies not following each step in this process.
Phase 1: Determining Workload and Establishing Baseline Metrics
Select a model or database only after you’ve defined what business task will be performed. Define the type of information from end users, what actions can be taken, how quickly tasks need to be completed, and whether or not the data being analyzed is sensitive.
Establish baseline performance metrics prior to implementing AI systems by measuring how efficiently the human system currently performs (measuring completion times, error rates, etc.) and calculating the total cost of processing the tasks manually. Your baseline results will provide evidence of the financial benefits of using the new technology.
Phase 2: Setting Up Enterprise Data Layers
Most AI projects fail because the data used in the system isn’t structured properly. Conduct a complete inventory of all data sources. Create data contracts and validation procedures.
Remove duplicates from your storage and create a way to automatically identify and redact PII before it reaches the model. Create a method of tracking versions of the document and setting permission controls so that only users have access to documents that they are authorized to access.
Phase 3: Selecting a Basic Viable Model
Start with the most cost-effective method of implementing a basic model. Test out basic business rules to see if they return the desired results. If business rules don’t return accurate results, test small basic ML models.
Once you identify a basic rule set, and the results are unsatisfactory or require advanced reasoning skills to implement effectively, move on to larger, more complex models or open-weight models as needed.
Phase 4: Implementing Retrieval and Orchestration
Add retrieval and orchestration of the data using a vector database and retrieval-augmented generation only when they are required to provide real-time access to current data. This includes information that must be kept private or which requires a high degree of traceability.
Orchestration agents should not be deployed across all applications, only for those systems which will interact with another external application. Complex agent workflows should not be introduced simply because an application interacts via a chatbot. Many enterprise workflows are capable of functioning using simple, predictable routing without complex workflows and limited model calls.
Phase 5: Create Your Test Set
Prior to the first use of the application by actual users, a comprehensive evaluation dataset should be created. This dataset must include non-ambiguous and difficult-to-answer requests, as well as attacks from malicious actors, and historical examples of failures for which a similar system previously failed.
Separate the accuracy of the retrieval database from the quality of the language model. Make sure that the application is aware of how to deny unsafe requests, as well as elevate ambiguous requests for human assistance.
Phase 6: Launch with Security Controls and Monitor Outcomes
Launch the application with very strong methods of authentication, as well as application rate limits. Implement input validation and mitigation of potential prompt injection vulnerabilities.
After the application is live, monitor for technical metrics such as latency and the failure rate of the retrieval function, but also focus on the business metrics, i.e., cost to complete a task, rate of human overrides, and the impact on the shrinkage of time-to-resolution. If the business metrics do not show an improvement, the technical architecture will require re-architecting.
What Causes AI Pilots to Fail
The transition from pilot to production exposes the problems with a poorly constructed technology stack. By identifying these operational problems early, the risk of termination of the project is reduced.
Absence of Baseline Business Metrics and Cost Controls
A large number of engineering teams have deployed an AI function without having an adequate understanding of the unit economics associated with the workflow. Spending a small amount of money on API tokens is acceptable during the pilot phase. In the production environment, agents that do not have optimized loops may keep calling tools in perpetuity, racking up unexpectedly high bills for companies using the cloud.
Executive leaders will not be able to provide a business model for infrastructure investment unless an appropriate baseline exists for understanding infrastructure costs, which will lead to projects being stopped very quickly.
Ignoring Model Drift and Factual Degradation
Many vendors make modifications to the models they manage on a regular basis. Therefore, what worked perfectly well for a specific prompt in January may return a false-positive in March, and new models do not always match legacy model formats and features.
With the evolving nature of business documents and the frequent updates to the vector database as time passes, inaccurate or obsolete pieces of information can linger for months in the vector database. If the organization has not set up a system of automated evaluation pipelines and processes to continuously test the accuracy of the system, this will ultimately create a breakdown of trust in the end-user experience of the product.
The Strategic Value of Enterprise AI Architecture
In the future, an enterprise AI infrastructure will be an incredibly detailed and complex factory environment, as opposed to being a simple gadget for the consumer. Therefore, the only difference between an unsuccessful AI pilot and a viable scaling solution for the business is the exact structural integrity established within the enterprise architecture. Thus, the choice of a vector database, the establishment of rigid data governance rules, and the specific way in which to measure cost per task within the case of business AI infrastructure is of far more significance than the name of the base model in the center of the system.
When an organization takes an end-to-end software engineering approach to the challenges faced by businesses adopting AI, they position the establishment of data pipelines, the capability to provide security within the system, and the setup of evaluation processes to ensure integrity throughout the AI process. The organization will have a much greater chance of navigating from an environment of experimentation with AI capabilities to one of being a fully functionally integrated solution of AI within the operational framework of the organization.
Connecting an external API of any type directly to an internal interface could result in excessive costs, extreme risk to security, and a backlog of technical debt. In addition, it is the architecture itself and not just the algorithm that will provide strategic value.
Executive Q&A: Overcoming AI Operational Bottlenecks
How should engineering teams evaluate the cost of large language models?
The most accurate way to measure the cost of implementing LLMs at scale is by looking directly at the unit cost per successfully completed business task instead of looking at the underlying cost per token. For example, using a small and inexpensive LLM may require four separate prompts and coordinating multiple pieces to accomplish a single business task, while using a more advanced (and expensive) foundation model could accomplish that task in one step. Better yet, to determine the overall cost of using an ai tech stack in production, one must take into account total compute time, the costs associated with retrieving data from the database, orchestration overhead, and how much time humans spend performing reviews of LLM-generated outputs for error corrections.
When should a company choose fine-tuning over a RAG architecture?
Companies will want to carefully evaluate when it is better to use fine-tuning as opposed to a retrieval-augmented generation (RAG) architecture. Companies should use fine-tuning when a model must learn a very specific tone of voice, should exhibit a unique writing style, or should express its output in a way that is very different from the traditional prompt that is used with LLMs to solve a problem. Fine-tuning is designed to change the basic level of operation of the model.
Conversely, a company should use a RAG architecture when it needs to provide the model with factual information that is exact, confidential, and time-sensitive. RAG lends itself to the notion that a model can use an external memory from which to draw information, thus allowing for the generation of improved and up-to-date answers without the necessity for continuously retraining the model every time an internal document is modified or updated.
What causes data retrieval failures in an enterprise AI assistant?
Within the enterprise space, the problems associated with failed data retrieval by an AI assistant arise from issues that occur within the ingestion layer of the organization. Typically, the issue of failed data retrieval is related to how large documents were broken into smaller pieces that did not contain common overlaps of the original document.
In addition, if the metadata tags associated with the document were missing or if access permissions for that document were incorrectly set up, the retrieval database will not be able to find the correct document for the search. An AI assistant's model cannot generate a correct answer when it is presented with irrelevant or broken text snippets by the retrieval database.
How do data residency requirements impact foundation model selection?
The selection of foundation models being used by an organization (from a model development perspective) is being impacted significantly by respective data residency regulations. Specifically, if an organization is prohibited from sending customer data outside of its respective geographic area or country due to local data residency regulations, the architecture team must shift its strategy to accommodate either regionally hosted or fully self-hosted infrastructure deployments.
This also increases the burden placed upon the organization's internal machine learning operations teams to develop and manage secure environments, deploy updated models, and effectively scale the respective infrastructure.