Deployment Patterns for Enterprise AI That Don't Require a 3am Rollback

Nobody knows until the next morning, when the approval queue is empty and the finance team is asking why their documents did not move overnight. Eight hours of silent failure. The system has no circuit breaker on token count and no monitoring on document routing. The deployment process was: test locally, deploy to production, check in the morning. That process is wrong.

AI systems fail differently than deterministic software. They fail silently. They degrade instead of crash. They fail on specific inputs that never appeared in testing. The deployment patterns that work for standard software are necessary for AI, and they are not enough. These are the ones we add.

Why AI failure modes need different patterns

Deterministic software fails loudly. Exceptions are thrown, services return 500s, monitoring catches the spike. AI systems fail quietly. The API returns 200. The output looks like output. The document gets processed. The result is wrong. No alarm fires.

Three AI-specific failure modes that standard deployment monitoring will not catch. First: silent degradation, where the model is processing inputs but accuracy has dropped because a prompt change interacted badly with a document type you did not test. Second: context-sensitive failures, where the pipeline works on 95% of documents and fails silently on a specific document type that happens to be longer or formatted differently. Third: prompt regression, where a change that improves performance on the cases you tested breaks it on cases you did not.

All three require detection at the application level, not the infrastructure level. No standard monitoring tool catches them automatically.

Blue/green deployment with behavioral monitoring

Blue/green deployment for AI means maintaining two identical pipeline versions and routing a percentage of traffic to the new version while the previous version stays live. This is standard practice for web services. For AI pipelines, the difference is what you monitor during the comparison window.

Infrastructure metrics such as latency, error rate and throughput are not enough. You need behavioral metrics: correction rate per document type on both versions. If the new prompt version is producing a 6% correction rate and the old version is at 3%, the new version has a regression even if no errors are being thrown.

The monitoring window we use: 48 hours at 10% traffic before cutover. Not 2 hours. Not 24. The 48-hour window catches document types that only appear on certain days, such as month-end invoices, which a shorter window would miss.

The shadow run pattern

For high-stakes deployments, such as a change to the core extraction prompt or a model version upgrade, we use a shadow run before any traffic is routed to the new version.

The shadow run processes real incoming documents in parallel with the production version. The new version's outputs are logged but not acted on. Nothing writes to the ERP, and no approvals or notifications are triggered. We compare outputs between the production version and the shadow version across 200–500 real documents before the new version handles any live traffic.

Divergence rate above 8% between production and shadow triggers a review before proceeding. It is not a hard block, because sometimes the new version is better in ways that look like divergence. But "someone look at these 40 diverging outputs before we flip the switch" is a 30-minute review, and it catches regressions that would otherwise reach production.

Circuit breakers specific to AI

Standard circuit breakers trip on error rates and latency. AI circuit breakers need to trip on additional conditions. Build these three into every pipeline.

Token count monitoring before each LLM call. The system prompt plus document content plus conversation history is measured before the API call executes. If it exceeds 90% of the model's context limit, the document is routed to a fallback: a model with a larger context window or a human review queue. It never fails silently. This alone would have prevented the 8-hour outage in the example above.

Confidence score distribution monitoring. In normal operation, the distribution of confidence scores across a document type has a shape: most fields extract at high confidence and a few consistently come in lower. When that distribution shifts materially, say mean confidence dropping by more than 0.05 over a 4-hour window, it signals that the model is encountering input it has not seen before. The circuit breaker increases the confidence threshold, routing more documents to human review until the shift is understood.

Error rate auto-fallback. If API error rate exceeds 5% over a 15-minute window, the system automatically falls back to the previous prompt version and pages the on-call engineer. Not the on-call engineer at 3am reading a dashboard. An automated fallback that restores service, then a page.

Canary deployment thresholds for AI

Canary deployments for AI follow the same infrastructure pattern as standard software: route a small percentage of traffic to the new version and expand if the metrics hold. The promotion criteria are different.

The thresholds we use: start at 5% traffic for 2 hours. If correction rate on the canary is not more than 1.5x the production correction rate, expand to 25% for 4 hours. Apply the same check. If it holds, move to 100%. If at any step the correction rate on the canary exceeds the threshold, roll back automatically and review.

The 1.5x threshold sounds lenient. It is not. A production system at 3% correction rate has a canary threshold of 4.5%. If the new prompt version is at 5%, it fails. That 2-point difference represents real errors that real users would need to correct, at scale, after full rollout.

The most dangerous AI deployments are the ones where nobody defined what a failure looks like. If you cannot answer "how will we know within 30 minutes that this deployment is broken?", you are not ready to deploy. Define the signal, build the alert, test that it fires. Then ship.