Data Pipeline Reliability for AI Systems: The 4 Failure Patterns We Keep Seeing

Pattern 1: silent schema drift

The most common pipeline failure we see. An upstream system — ERP, CRM, warehouse management — updates and adds a new column, renames an existing field, or changes a data type. The API continues to return 200 responses. The pipeline continues to run. The AI system continues to operate. But a field the model depends on now contains different data than it expects.

The specific incident: a supplier risk scoring agent used a field called payment_terms_days from the ERP supplier master. After an Odoo update, that field was renamed property_payment_term_id and now contained a relation ID rather than an integer day count. The agent started receiving null values (the old field no longer existed) and silently defaulted to a payment term of zero days, treating all suppliers as immediate-payment suppliers. The risk scores were wrong for three weeks before anyone noticed, because the scores looked plausible.

3 weekshow long a schema drift incident ran undetected because wrong outputs still looked plausible

The fix: schema validation at the pipeline ingestion layer. Every data source gets a schema contract — specific fields, expected types, acceptable ranges. Any deviation triggers an alert before the data reaches the model. We use pydantic models for this on Python pipelines and JSON Schema validation on Node.js pipelines. The contract is not an afterthought — it is written before the pipeline and reviewed whenever the upstream system updates.

Pattern 2: upstream API changes that are not breaking changes

API versioning disciplines exist precisely to prevent breaking changes from hitting downstream consumers without warning. The problem is that not all changes that affect AI systems are technically breaking. An API that returns the same fields in the same types but changes the semantics of a value is not a breaking change by any API versioning convention. It will break your AI system.

We hit this with a procurement agent that consumed a logistics API. The API returned a delivery_status field with values including IN_TRANSIT, DELIVERED, and DELAYED. The logistics provider updated their system and started using DELAYED to mean "expected delay of less than 24 hours — still on time for the committed window," where previously it had meant "the committed delivery window will be missed." The API version did not change. The field existed. The values existed. The meaning changed. The procurement agent started unnecessarily escalating deliveries that were on track.

Semantic drift is invisible to schema validation. The defense is behavioral monitoring: track the distribution of values your AI system receives from each upstream source. If a field that historically returned a value distribution of 85% DELIVERED / 10% IN_TRANSIT / 5% DELAYED suddenly shifts to 85% DELIVERED / 8% IN_TRANSIT / 7% DELAYED, something changed upstream. Catch these shifts before they affect downstream behavior.

Pattern 3: missing data that looks like zero

Null propagation is one of the oldest problems in data engineering, and it is still biting AI systems in production. The failure mode: a data field returns null because the data does not exist, the pipeline interprets null as zero (or empty string, or the default value for the type), and the model receives a zero where it expected a meaningful number.

For AI systems, this is particularly dangerous because the model will reason about the zero as if it is real data. A cash flow forecasting model that receives zero for a customer's historical average payment amount (because the customer is new and has no history, and the query returned null which became zero) will predict that customer's contribution to cash flow as zero and potentially flag a healthy payment as anomalous.

The distinction between "this value is zero" and "this value is unknown" must be explicit in your data model. New customers without payment history should be flagged as "insufficient data, apply population-average assumptions" rather than treated as customers who do not pay. This distinction sounds obvious but requires deliberate design in every data pipeline feeding an AI system.

Our take

The three-value model: For every input field in an AI system, there are three distinct states: the field has a value, the field is zero, and the field has no data. Model these as three distinct states in your pipeline. Use nullable types and explicit unknown/insufficient-data flags rather than defaulting to zero. Pydantic's Optional types handle this cleanly in Python. The effort to implement it upfront is small. The effort to debug null-as-zero errors in production is not.

Pattern 4: staging-production environment mismatch

AI systems are tested in staging environments that should mirror production. They often do not. The most common mismatches we have seen:

→Staging uses a sanitized copy of the production database that is 6 months old. Production has new suppliers, changed payment terms, and updated product categories that do not exist in staging. The model behaves correctly in staging for data it has seen and incorrectly in production for data it has not.
→Staging upstream APIs return mock data with cleaner formatting than production APIs return. The extraction model works perfectly on staging and fails at 12% error rate on production because real supplier invoices have the PDF quality issues that mocks did not include.
→Staging uses a different LLM API endpoint with a different rate limit than production. Load testing in staging does not expose the rate limiting behavior that production traffic triggers in the first week.

These mismatches produce the worst kind of production incident: the system worked fine in every test environment, and the failure is in production on real data, where the pressure to fix it quickly is highest and the ability to reproduce it in a controlled environment is lowest.

Our current practice: for every AI deployment, we run a shadow mode period of two weeks where the system operates on live production data but its outputs are reviewed by a human rather than acted on. This exposes the distribution shift between staging and production data before the system has taken any real actions. Two weeks of shadow mode has caught staging-production mismatch on every deployment where it existed — which is four of the last ten.

The monitoring stack these patterns require

Preventing these patterns requires monitoring that most AI system builds do not include by default. The four things worth instrumenting from day one:

1.Schema contract validation on all ingestion. Alert on any field that is missing, type-changed, or outside expected range. Run continuously, not just on deployment.
2.Input distribution monitoring. Track the statistical distribution of values for key input fields. Alert when distributions shift significantly — this catches semantic drift, source data quality changes, and gradual data model drift.
3.Null rate tracking per field. If a field that is typically null 5% of the time becomes null 40% of the time, something changed upstream. Alert on null rate changes, not just on nulls.
4.Output distribution monitoring. Track the distribution of the AI system's outputs. If a classification agent that typically routes 15% of items to "requires review" suddenly routes 40%, either the input data changed or the model behavior changed. Either way, you want to know.

This monitoring does not require expensive observability tooling. Basic statistical checks on Prometheus metrics or simple database rows tracking distribution summaries per hour are sufficient. The cost of building it is one engineer-week. The cost of not building it is measured in production incidents.