Scaling an AI document pipeline from startup to enterprise volume

Context

The platform ingested business documents, extracted structured information, and made it searchable and answerable through an AI layer. It worked well at the volume it was designed for.

Then ingestion volume grew by more than an order of magnitude, and the assumptions underneath it stopped holding. Documents queued for hours. A single malformed file could stall a batch. The AI layer returned answers slowly enough that users started avoiding it. The system was not broken exactly — it was sized for a business that no longer existed.

Client name withheld by agreement.

The constraints

Three things shaped the approach.

It could not go down. Customers were actively using the platform. A rewrite behind a flag was acceptable; a migration weekend was not.

The documents were genuinely messy. Scanned pages, inconsistent layouts, mixed quality, and a long tail of formats that appeared rarely enough to be forgotten but often enough to break things.

Answer quality was the actual product. Throughput mattered, but a fast system that returned unreliable answers would have been worse than the slow one.

What we changed

From batch to event-driven

The original design processed documents in scheduled batches. That is a reasonable choice at low volume and a poor one under bursty load, because latency is bounded by the batch window regardless of how much capacity is idle.

We restructured ingestion around a queue. Each document became an independent unit of work, processed as it arrived, retried on failure with backoff, and dead-lettered when it genuinely could not be handled. Throughput became a function of worker count rather than a schedule, and a single bad document stopped being able to hold up everything behind it.

This change alone accounted for most of the throughput improvement.

Rebuilding retrieval

The AI layer’s weakness was retrieval, not generation. It was pulling too much marginally relevant context, which made responses slower and diluted answer quality at the same time.

We rebuilt the retrieval path: better chunking that respected document structure instead of splitting at fixed lengths, embeddings regenerated against the improved chunks, and a retrieval step that filtered on metadata before ranking semantically. Less context reached the model, and more of it was relevant.

Response times dropped by roughly two thirds. Answer quality improved at the same time, which is the usual result when retrieval is the bottleneck — the model was never the problem.

Making failure visible

The original system failed quietly. Documents that could not be processed simply did not appear, and nobody found out until someone went looking for one.

We added structured logging with a trace identifier carried through every stage, so a document’s path through the system could be reconstructed. Failures surfaced as alerts with enough context to act on. Operations staff got a view showing what was stuck and why, which meant they stopped needing an engineer to answer routine questions.

Architecture

Documents land in object storage and emit an event. A queue distributes work to stateless workers that handle extraction, normalization, and embedding. Processed content and metadata go to Postgres, with vectors stored alongside for retrieval. The AI layer queries that store, assembles filtered context, and calls the model. Every stage emits structured logs against a shared trace identifier.

The important property is that each stage is independently scalable and independently retryable, so load in one place does not cascade.

Results

Throughput increased roughly 30×, absorbing enterprise ingestion volume on infrastructure that scaled with demand rather than being permanently provisioned for the peak. AI responses returned about 3× faster, and measured user satisfaction rose by 15%.

The more durable outcome is less quantifiable: the pipeline stopped being the thing limiting what the business could take on.

What we would tell you before starting something similar

The model is rarely the bottleneck. When an AI feature is slow or unreliable, retrieval and data quality are the first places to look, and they are usually where the fix is.

Batch processing is a decision with a shelf life. It is simpler and it is fine — until volume becomes bursty, at which point the cost of the migration is proportional to how long you waited.

Observability is not optional in a system with a probabilistic component. If you cannot trace why a specific document produced a specific answer, you cannot debug quality complaints, and quality complaints are the ones that matter.