Standard DevOps practices break in specific ways when applied to AI workloads. The assumptions that underlie traditional software delivery, that the system is deterministic, that artifacts are built from code, that tests produce binary pass or fail results, do not hold for AI systems. Understanding exactly where traditional DevOps breaks is the starting point for building the practices that work.
Where Traditional DevOps Breaks for AI
Non-determinism is the most fundamental difference. Given the same inputs, a traditional software system produces the same outputs every time. An LLM-based system produces different outputs on each run. A model trained from scratch produces different results depending on random seed and hardware. This non-determinism makes traditional testing approaches insufficient: you cannot write a unit test that asserts a specific model output and expect it to pass reliably.
Data dependencies create a category of artifact that traditional DevOps does not handle. Software artifacts depend on code repositories. Model artifacts depend on training data, hyperparameters, and infrastructure configuration. Reproducing a model requires reproducing all three. A CI/CD pipeline that only versions code cannot reproduce a model build.
Model artifacts are also much larger than software artifacts. A fine-tuned language model checkpoint can be tens of gigabytes. Artifact storage, transfer, and management at this scale requires specific infrastructure choices.
Experiment tracking is a workflow that traditional software development does not have. Data scientists run dozens or hundreds of experiments with different hyperparameters, data subsets, and model architectures. Tracking which experiments produced which results, and being able to reproduce any of them, requires dedicated tooling.
MLOps: What It Adds to DevOps
MLOps is the extension of DevOps practices to cover the specific requirements of machine learning systems. It does not replace DevOps: the core principles of automation, version control, testing, and monitoring all apply. MLOps adds: experiment tracking, data versioning, model versioning, model-specific testing approaches, and model-specific monitoring.
Our cloud and DevOps engineering practice treats MLOps as a first-class capability for AI product teams. The investment in MLOps infrastructure pays back through faster iteration cycles, reproducible experiments, and production systems that can be maintained without becoming opaque black boxes that only the original author understands.
The AI Development Lifecycle
The AI development lifecycle has six phases, each with distinct infrastructure requirements: experiment, train, evaluate, deploy, monitor, retrain.
The experiment phase is interactive and exploratory. Data scientists iterate rapidly in notebook environments, trying different approaches. The infrastructure requirement is accessible compute (often GPU), convenient access to data, and experiment tracking to capture what was tried and what the results were.
The train phase scales up the best experimental approach to full training runs on complete data. Infrastructure requirements shift to scalable compute (distributed training for large models), data pipeline reliability, and checkpoint storage. Training runs can take hours or days; interruptions must be recoverable without restarting from scratch.
The evaluate phase validates model performance on held-out data and checks for specific requirements around bias, robustness, and performance on critical subsets. This is where model deployment decisions are made.
The deploy phase moves the evaluated model into a serving infrastructure. Low-latency model serving has different infrastructure requirements from training: typically CPU or smaller GPU instances, horizontally scalable, with response time SLAs.
Monitor and retrain close the loop: monitoring detects when model performance degrades, which triggers a retrain cycle that begins the lifecycle again.
Data Versioning
DVC (Data Version Control) integrates with Git to track changes to large data files alongside code changes. Data files are stored in remote storage (S3, GCS, Azure Blob) with pointers checked into Git, enabling reproducible experiments without committing large files to the code repository.
LakeFS applies Git-like branching and versioning concepts to data lakes, enabling teams to create isolated data branches for experiments, reproduce the exact dataset used for any past experiment, and merge data changes with review processes similar to code review.
Delta Lake versioning at the table level provides time-travel queries and audit history for Lakehouse architectures. Combined with the broader data and AI platforms governance model, Delta Lake versioning provides the data lineage required for reproducible model development.
Model Versioning and Registry
A model registry stores trained models alongside the metadata required to understand, reproduce, and deploy them: training data version, hyperparameters, training metrics, evaluation results, the code version used to train the model, and the deployment configuration.
MLflow Model Registry is the most widely used open-source option. Weights and Biases provides a more comprehensive experiment tracking and model management platform. Amazon SageMaker Model Registry integrates naturally with AWS infrastructure.
The metadata captured for every model should include: a unique model version identifier, links to the training data version and code version, training metrics (loss curves, final metrics), evaluation results on the validation set (including disaggregated metrics by relevant subgroups), deployment configuration, and the approval status (staging, production, deprecated).
CI/CD for ML: What Automated Testing Looks Like
A model deployment pipeline includes automated tests that run on every candidate model before deployment. These tests are different from software unit tests because they assess properties of a statistical system.
Data validation tests check that training data meets schema expectations, volume expectations, and statistical property expectations. A model trained on data with an unexpected class imbalance will underperform on the minority class; a data validation test catches this before training.
Model performance tests assert that the new model meets minimum performance thresholds on the validation set and does not perform significantly worse than the current production model on any defined subgroup. A model that improves aggregate accuracy but degrades performance on a critical demographic or input type should not deploy automatically.
Bias tests check that model outputs do not exhibit demographic disparities beyond defined thresholds. These are model performance tests disaggregated by sensitive attributes.
Infrastructure as code using Terraform manages the GPU clusters, Kubernetes namespaces, and networking configuration for the ML platform. Helm charts package Kubernetes deployments for model serving workloads. The same tools apply as for any cloud infrastructure; the difference is the specific resource types (GPU node groups, model serving controllers) and the configuration parameters.
Monitoring AI in Production
The monitoring requirements for AI systems go beyond standard application monitoring. In addition to latency, throughput, and error rates, AI systems need monitoring for:
- Data drift. The statistical distribution of incoming data shifts relative to training data. If a model was trained on data with a specific feature distribution and production data shifts, model performance degrades even without any change to the model.
- Prediction distribution shift. The distribution of model outputs changes. If a classification model that historically predicted class A 60 percent of the time starts predicting class A 40 percent of the time, something has changed.
- Model performance drift. For systems with ground truth labels available (often with a delay), track model accuracy over time. Degradation below a threshold triggers investigation.
Tools like Arize, Fiddler, and WhyLabs specialize in ML observability. They integrate with model serving infrastructure to capture prediction logs, compute statistical distance metrics between production and training distributions, and alert when drift exceeds configured thresholds. See also how this connects to the broader infrastructure decisions discussed in our guide on cloud migration strategy for enterprise teams.
Cost Governance for GPU Compute
GPU compute is expensive and usage tends to grow faster than budgets. Without explicit governance, experimental training runs that were supposed to be temporary become permanent, development environments run at full scale continuously, and the monthly GPU bill doubles without anyone making a deliberate decision.
Spot instances (AWS) or preemptible VMs (GCP) reduce training costs by 60 to 80 percent for workloads that can tolerate interruption and checkpoint. Most model training can be designed to use spot capacity with periodic checkpointing, paying significantly less than on-demand rates.
Resource quotas at the namespace level in Kubernetes prevent any single team from consuming disproportionate compute. Experiment budgets (a maximum training budget per experiment in compute hours) create healthy discipline around the exploration phase.
The Platform Team Model
The question of when to invest in a dedicated ML platform team is a function of scale. A single data science team with five people can manage their own infrastructure, tolerating the overhead of infrastructure management as part of their work. At ten or more data scientists across multiple teams, the overhead becomes significant enough that a dedicated platform team pays back through the productivity it unlocks.
The platform team's role is to build and maintain the shared infrastructure that all data science teams rely on: the training cluster, the model registry, the feature store, the serving infrastructure, and the monitoring stack. Data scientists focus on modeling problems; the platform team ensures the infrastructure they depend on is reliable, scalable, and cost-efficient.
