Advanced RAG Data Preprocessing: From Raw Documents to Reliable Retrieval

Your RAG system returns a confident, detailed answer. It is also wrong.

The failure may begin before generation, when a PDF loses table structure, a heading is separated from the paragraph below it, multiple versions of a policy enter the index, or an OCR error changes a critical term. These are common failure paths, not the only ones: retrieval, reranking, query processing, prompting, model behavior, permissions, and evaluation can also determine whether a RAG answer is correct.

A retriever cannot return evidence that was never extracted or indexed correctly, and a language model cannot reliably reason over context that is fragmented, duplicated, stale, unauthorized, or stripped of its provenance. Preprocessing improves the evidence available to retrieval; it does not by itself prove that the final answer is grounded.

But there is an important qualification: preprocessing is not a substitute for good retrieval, reranking, prompting, or evaluation. Recent RAG research treats the system as a chain of interacting decisions, from document chunking and embeddings to retrieval and answer generation. The practical goal is therefore not to “clean everything.” It is to create evidence that is searchable, contextualized, traceable, fresh, and measurable.

This guide presents a production-oriented RAG preprocessing pipeline. It covers corpus selection, document ingestion, parsing, cleaning, metadata enrichment, filtering, deduplication, chunking, embeddings, hybrid retrieval, evaluation, and updates.

Key takeaways

What is RAG data preprocessing?

RAG data preprocessing is the process of turning selected source content into structured, traceable, and indexable evidence for retrieval-augmented generation.

Depending on the team, the boundary may include corpus selection, ingestion, extraction, normalization, metadata enrichment, filtering, deduplication, version resolution, chunking, embedding, and index preparation. Some systems treat embedding and indexing as separate services; the important requirement is an explicit handoff and versioned lineage between stages.

The preprocessing data contract

Every stage should produce a versioned artifact with an explicit input and output contract. A practical record should include the source identifier, source version or content hash, authority and effective dates, authorization reference, parser and cleaning versions, structural locations, original representation, retrieval representation, chunk ID, chunking parameters, generated-field status, embedding model and dimension, index or collection version, and processing status.

The contract makes a retrieval failure diagnosable: the team can determine whether evidence was missing at the source, damaged during parsing, removed during cleaning, split incorrectly, filtered out, embedded incompatibly, or rejected by authorization.

The exact boundary varies by team. Some engineers treat embedding and indexing as separate stages. Others include them in the broader ingestion pipeline. Either approach is reasonable as long as the handoff between stages is explicit.

A useful mental model is this:

The most important design choice is to preserve enough information at every step to diagnose a bad answer later. Store the original source reference, the parser used, the transformation version, the chunking method, and the embedding model. Without that lineage, debugging becomes guesswork.

Step 1: Select and govern the corpus

A RAG system cannot retrieve information that does not exist in its corpus. Before choosing a splitter or embedding model, decide which sources belong in the knowledge base and which should stay out.

Authorization is a separate security boundary. Store the source system’s permission reference and effective policy metadata, but do not treat a user-controlled or stale chunk field as proof that a caller may read the content. The retrieval service should evaluate the caller’s identity and current permissions, test for access-control drift, and fail closed when authorization metadata is missing or uncertain.

For a customer-support assistant, the corpus might include product manuals, troubleshooting guides, approved FAQs, and current release notes. For a legal assistant, it might include contracts, amendments, internal policies, and authoritative regulations. The right corpus depends on the questions the system is expected to answer.

Databricks recommends preserving raw source data in a durable location so the pipeline has traceability and an audit trail. That principle is easy to overlook when a prototype begins with a folder of files, but it becomes essential when documents are updated, removed, or reprocessed.

Preserving raw sources does not mean keeping every copy forever. Define retention, deletion, encryption, legal-hold, and access policies for the raw source, parsed representation, derived metadata, embeddings, logs, and backups. When a source is deleted or access is revoked, the pipeline needs an explicit tombstone and removal path so derived records do not remain searchable by accident.

At the corpus stage, record at least:

A common mistake is to ingest everything first and filter later. That increases embedding cost, storage requirements, and the chance that irrelevant or unauthorized content will enter retrieval. Corpus selection is not administrative housekeeping; it is the first relevance filter.

Decide what the system is allowed to know before deciding how to split it.

Security boundary: metadata is not authorization

RAG retrieval has at least three separate questions: Is this document relevant? Is it current and authoritative? Is the caller allowed to read it? Dense similarity, BM25, metadata filters, and reranking answer relevance questions. They do not replace identity-aware authorization.

Enforce access decisions using a trusted policy source, propagate only the minimum necessary content, test permission changes and revoked documents, and treat missing or stale authorization metadata as a failure condition. Source documents and retrieved chunks are untrusted data and must not be allowed to rewrite system instructions or trigger privileged actions.

Step 2: Parse documents without destroying their structure

Raw documents are rarely retrieval-ready. A PDF may contain text, page headers, footers, tables, diagrams, and multiple columns. A scanned contract may contain no machine-readable text at all. A spreadsheet may encode the meaning of a row through its column headings rather than through the cell values alone.

A parser that flattens all of this into one text stream can produce a technically valid output that is semantically broken. Unstructured’s preprocessing guide emphasizes preserving typed elements such as titles, narrative text, list items, tables, and images, together with page and layout metadata.

The appropriate parser depends on the source format and the information the user needs to retrieve.

Route documents by failure risk, not only by file extension

A parser can return a non-empty text string and still fail semantically. A better ingestion layer routes documents according to the kind of information they contain and the cost of getting that information wrong.

The route does not need to be complicated. The important point is to make the decision explicit, store which route was used, and send representative failures back to the parser or review queue instead of hiding them inside a generic “ingestion succeeded” status.

Databricks lists parsing, error handling, customized parsing logic, and sample-based quality review as separate best practices. That separation is useful. A parser can run without throwing an exception and still produce poor retrieval data. Review a sample of extracted output before sending the entire corpus to chunking.

For OCR-heavy collections, keep the raw image, extracted text, page or element location, and OCR confidence or error information linked together. Confidence is a triage signal, not proof that the text is correct. Route low-confidence or high-impact fields, such as names, identifiers, dates, dosage, amounts, or contractual language, to field-level validation or human review. Never silently overwrite the original representation.

Successful parsing preserves the relationships that make the document understandable, not just the characters that appear on the page.

Step 3: Clean and normalize the text

Cleaning should remove noise without erasing meaning. Useful operations include normalizing whitespace, repairing broken hyphenation, correcting common OCR errors, restoring sentence boundaries, and removing repeated headers or footers that would otherwise appear in every chunk.

The difficult part is deciding what counts as noise. A page number may be irrelevant to retrieval. A section heading may be essential. A document ID may be the most useful field for traceability. The safest approach is to create a cleaned representation while preserving the original text and source metadata.

Microsoft’s chunk-enrichment guidance uses a schema that keeps both the original Chunk and a CleanedChunk for vectorization and retrieval. That distinction avoids a common trade-off: either preserve messy source text for auditability or clean it aggressively for search. Store both.

A practical cleaning pass may include:

1. Normalize character encoding, whitespace, and line breaks.
2. Remove repeated navigation, boilerplate, headers, and footers when they do not carry meaning.
3. Repair broken hyphenation and obvious OCR substitutions.
4. Preserve headings, list structure, table labels, page numbers, and source identifiers as metadata where appropriate.
5. Detect empty, extremely short, or malformed extraction results.
6. Record every transformation as a versioned pipeline step.

Do not apply lowercasing, stop-word removal, Unicode deletion, spelling correction, or aggressive punctuation stripping as universal preprocessing rules. Microsoft documents cases where these transformations can help vector comparisons, but also warns that they can remove meaningful distinctions such as negation, proper nouns, identifiers, or domain notation.

Test each transformation against representative dense, sparse, hybrid, and citation-oriented queries, while preserving the original source text for auditability.

A security note: source content is untrusted data

A document being indexed may contain text that looks like an instruction to an AI system. This is an indirect prompt-injection risk: an attacker or an ordinary source document can place instructions in a file, web page, image, or table that later enters the model context. RAG does not fully mitigate prompt injection.

During parsing, enrichment, retrieval, and generation, treat source content as untrusted data rather than instructions to follow. Separate system instructions from retrieved content, use least-privilege service permissions, validate structured outputs, apply authorization outside the model, and require human approval for high-risk actions. If an LLM generates summaries, questions, keywords, or metadata, constrain the task and validate the output before it becomes searchable or operational.

Red-team the pipeline with direct and indirect injection cases, including hidden text, images, multilingual instructions, and poisoned documents. The goal is to reduce impact and detect failures, not to claim perfect prevention. Preserve the original and cleaned representations, and record whether generated fields were accepted, rejected, or sent for review.

Step 4: Enrich chunks with metadata

Metadata is not decoration. It gives the retriever additional signals for filtering, ranking, and context assembly.

Useful metadata can describe the document, the chunk, its position, its authority, and its likely retrieval behavior. Examples include the document title, section path, page number, author, department, language, dates, version, source type, extraction confidence, entities, keywords, and access-control labels.

A chunk can also be enriched with generated fields. Elastic’s implementation example adds keyphrases, entities, and potential questions to the representation used for retrieval. Microsoft Azure describes a similar enrichment shape using title, summary, keywords, questions, cleaned text, and the original chunk.

A practical record might look like this:

JSON: 

{

  "chunk_id": "policy-2026-0147-c12",

  "source_id": "policy-2026-0147",

  "text": "The original extracted passage...",

  "cleaned_text": "The normalized passage used for vectorization...",

  "title": "Remote Access Policy",

  "section_path": ["Security Policies", "Remote Access", "Authentication"],

  "summary": "Authentication requirements for remote access",

  "keywords": ["MFA", "remote access", "authentication"],

  "questions": ["What authentication is required for remote access?"],

  "entities": ["MFA"],

  "page_number": 12,

  "language": "en",

  "source_date": "2026-02-10",

  "version": "3.1",

  "authority": "security-team",

  "parser_version": "pdf-layout-v2",

  "chunking_strategy": "heading-aware",

  "embedding_model": "embedding-model-version-id"

}

Generated metadata can help when a user’s wording differs from the wording in the source, but it introduces a second source of error. Keep generated summaries, questions, keywords, and entities separate from authoritative source text, store the generator and prompt version, validate a sample against the source, and define what happens when enrichment fails or contradicts the source. Generated metadata may improve retrieval recall; it must not silently override the source or become an authorization decision.

Enrichment also adds model calls, latency, storage, and maintenance overhead. Estimate those costs per document and per reprocessing cycle, then keep an enriched field only when a controlled evaluation shows that its retrieval benefit justifies the operational burden.

A small Python example

The metadata should travel with the chunk rather than being re-created after retrieval. In LangChain, split_documents preserves the document metadata on the resulting chunks:

Python:

from langchain_text_splitters import RecursiveCharacterTextSplitter
from langchain_core.documents import Document

text_splitter = RecursiveCharacterTextSplitter(
    chunk_size=500,
    chunk_overlap=50,
)

doc = Document(
    page_content="Original text from legal clause...",
    metadata={
        "source": "contract_2026_v2.pdf",
        "section": "Non-Compete",
        "version": "2.1",
    },
)

This is intentionally a baseline example. A production pipeline would also validate extraction, add stable chunk IDs, preserve the original document version, and record the splitter configuration.

If an LLM call fails during metadata generation, do not drop the chunk or fail the whole batch by default. Keep the chunk, attach a deterministic fallback such as section_path + page_number, record enrichment_status=failed, and retry it through a bounded queue. Stable chunk IDs, idempotent writes, caching, and resumable ingestion make this pattern safer at scale.

Give metadata different visibility rules

Not every metadata field should be shown to every model. A filename or parser version may be useful for debugging but irrelevant to semantic similarity. A generated question may improve retrieval while adding noise to the final answer context. LlamaIndex’s documentation makes this separation explicit by allowing metadata to be excluded independently from embedding text and LLM text.

The visibility split below is an implementation pattern, not a RAG standard. Some frameworks let developers exclude selected metadata fields from embedding input or LLM context while retaining them for filtering and provenance. Verify the exact API and package version for the framework you use; do not assume that metadata is automatically excluded, filtered, or hidden from every downstream component.

Metadata also enables hybrid retrieval. A query such as “show product SKU ABC123” benefits from exact lexical matching and metadata filters. A query such as “find products similar to this one” benefits more from semantic matching. A hybrid system can combine sparse signals such as BM25 with dense embeddings, then use metadata to constrain the candidate set.

Add metadata that helps the system answer three questions: what is this, where did it come from, and when should it be trusted?

Step 5: Filter, deduplicate, and resolve versions

Filtering and deduplication are related, but they are not the same operation.

Filtering removes content that is irrelevant, unauthorized, malformed, or outside the system’s intended scope. Exact deduplication removes identical documents or chunks. Near-duplicate detection identifies passages that are almost the same but differ in formatting, dates, or small edits. Version control determines which record is canonical and whether older versions should remain searchable.

Databricks treats metadata extraction, deduplication, and filtering as distinct enrichment and quality stages. That separation makes the pipeline easier to test.

Consider a policy updated five times. Indexing all five copies may cause the retriever to return several versions of the same rule. The generator then has to infer which one is current. A better design records version metadata, identifies the canonical document, and retains historical versions only when users need historical answers.

A useful policy is:

The exact order also depends on the corpus. Document-level canonicalization before chunking can reduce work. Passage-level near-deduplication after chunking can catch repeated boilerplate across otherwise different documents. Treat both as experiments rather than universal rules.

Canonicalization is a policy decision, not merely a similarity threshold. Define which source is authoritative, how effective dates and publication status are compared, whether historical queries may retrieve superseded versions, and how conflicts are surfaced. Store the reason a document or passage was retained, merged, excluded, or marked historical. Near-deduplication can remove useful source diversity if it is applied without an evaluation set.

Reduce redundant evidence, but keep enough provenance to explain why a record was retained, merged, or excluded.

Step 6: Choose a RAG chunking strategy

Chunking determines the units that are embedded, retrieved, reranked, and passed to the language model. A chunk that is too small may lose the context needed to interpret a pronoun, table row, or clause. A chunk that is too large may dilute the matching signal, increase prompt cost, or exceed the embedding model’s input limit.

There is no universal best chunk size: choose a baseline that fits the document structure and query workload, then compare alternatives with a fixed evaluation set.

There is no universally correct chunk size. It depends on document structure, embedding limits, query patterns, retrieval method, and the way the generator uses context.

The chunking decision table

Two newer techniques to test

Two newer approaches address context loss from different directions.

Contextual Retrieval adds a short, chunk-specific explanation before the chunk is embedded and indexed. Anthropic describes using both Contextual Embeddings and Contextual BM25, with optional reranking after retrieval.

Its published experiments reported lower failed-retrieval rates when these methods were combined, but those numbers are experiment-specific rather than universal guarantees. The extra context is usually generated with an LLM, so teams should account for enrichment cost, prompt quality, caching, and validation.

Late Chunking changes the order of operations. A long-context embedding model processes the document or long passage first; the system then derives chunk-level vectors from the contextual token representations and stores those vectors against explicit chunk boundaries. An Elastic implementation with Jina Embeddings v2 demonstrates this pattern and contrasts it with ordinary short-context chunking.

Late Chunking can preserve broader document context, but it requires a compatible long-context embedding model and adds implementation complexity. It is an advanced experiment, not a replacement for a well-tested baseline.

Both methods can be useful when chunks lose meaning after splitting. Neither removes the need for parsing, metadata, version control, or evaluation. Compare them against the same fixed-size or structure-aware baseline before adopting them.

Why fixed-size chunking is still a serious baseline

A 2025 Findings of NAACL study evaluated semantic and fixed-size chunking on document retrieval, evidence retrieval, and retrieval-based answer generation. Its abstract reports that the computational cost of semantic chunking was not justified by consistent gains across the evaluated tasks. This is evidence for a baseline-and-benchmark strategy, not proof that fixed-size chunking always wins or that the result transfers unchanged to every corpus, language, model, or query distribution.

Start with a reproducible fixed-size or structure-aware baseline. Then compare semantic, hierarchical, contextual, or late-chunking alternatives while holding the embedding model, top-k, reranker, generation prompt, query set, and evaluation procedure constant.

Its abstract reports that the computational cost of semantic chunking was not justified by consistent performance gains.

That does not make semantic chunking useless. It means the choice should be tested against the actual corpus and queries. Start with a reproducible fixed-size or structure-aware baseline. Then compare alternatives using the same evaluation set, embedding model, top-k, reranker, and answer-generation configuration.

When hierarchical chunking helps

Hierarchical chunking creates multiple levels of representation. A paragraph or clause can be indexed for precise matching while retaining a parent section or chapter for context. A legal contract, for example, can be represented as contract → article → section → clause.

A 2026 ACL paper introduced a benchmark with multi-level chunking points and evidence-dense question-answer pairs, then evaluated a hierarchical structuring framework with Auto-Merge retrieval. The research reinforces a practical lesson: document hierarchy should be evaluated as part of the retrieval task, not judged only by how elegant the chunks look.

Choose the simplest chunking method that preserves the context your queries require, then prove the choice with an evaluation set.

Step 7: Generate embeddings and prepare hybrid retrieval

After cleaning, enrichment, filtering, and chunking, each retrieval unit can be converted into an embedding and stored in an index. The embedding model should be selected for the language, domain, latency, cost, and expected query types of the system.

Consistency matters. The query and indexed chunks must be comparable under the embedding model and preprocessing contract used by the retrieval system. If the embedding model, dimension, tokenizer assumptions, normalization, or vector-index configuration changes incompatibly, plan a controlled migration or re-embedding rather than silently mixing vectors.

Track the model identifier, version, dimension, preprocessing version, and index version on every record, and keep a rollback or dual-read plan for high-stakes changes.

Dense retrieval is useful for semantic similarity, but sparse lexical retrieval can be important for identifiers, product names, error codes, clause numbers, and exact terminology. Hybrid retrieval combines these relevance signals, and reranking can reorder a candidate set using a more precise model. Metadata filters can further constrain the candidate set, but relevance ranking and authorization are different concerns: a chunk that is highly relevant is still not eligible if the caller is not authorized to read it.

Index design also matters. Microsoft describes hierarchical, specialized, and hybrid index organizations for different relationships and data types. A relational filter may be better for structured fields. A graph or parent-child structure may be better for linked sections. A vector index is one component of the retrieval architecture, not the entire architecture.

When text-only retrieval is not enough

Some evidence is visual rather than textual. A chart may encode meaning in bar height, arrows, or spatial relationships, and a table may depend on row and column alignment. NVIDIA’s Nemotron RAG document-processing reference shows one vendor implementation that preserves structured text, tables, chart images, multimodal embeddings, reranking, and page or element provenance. Use this as an implementation pattern, not as evidence that every RAG system needs multimodal models, GPUs, or the same extraction stack.

Use a visual route when the user’s question depends on an image, chart, form layout, or table relationship that plain text cannot preserve safely. Keep the extracted text, visual artifact, page, and element ID linked so the system can cite the evidence it actually used.

You do not need a multimodal stack for every document. Use a separate visual route when the user’s question depends on an image, chart, form layout, or table relationship that plain text cannot represent safely. Keep the extracted text and the original visual element linked by page and element ID so the system can cite the evidence it actually used.

Step 8: Evaluate whether preprocessing actually helped

A clean-looking corpus is not proof of a better RAG system. Preprocessing should be evaluated with representative questions and known evidence.

Input and pipeline metrics

These metrics help detect upstream problems:

Retrieval and answer metrics

The outcome metrics matter more. Use a labeled set of representative questions and record which documents or chunks contain the expected evidence.

The evaluation should include retrieval precision or recall at k, ranking metrics such as MRR or nDCG where appropriate, evidence coverage, answer relevance, groundedness or faithfulness, latency, and token cost. The Fudan University study is a useful reminder that RAG quality depends on combinations of modules, not only on a single preprocessing choice.

Define each metric before running the comparison. Retrieval recall@k asks whether expected evidence appears in the first k results; precision@k asks how much of that set is relevant; MRR or nDCG captures ranking quality when graded relevance is available; evidence coverage asks whether the retrieved context contains the information needed for the answer; groundedness or faithfulness evaluates whether the answer is supported by the supplied evidence.

Keep retrieval evaluation separate from generation evaluation, and prevent test questions or answer text from leaking into the indexed corpus.

A simple experiment can compare two preprocessing variants:

1. Freeze the query set, embedding model, index type, top-k, reranker, and generation prompt.
2. Change only the preprocessing variable, such as fixed-size versus hierarchical chunking.
3. Measure retrieval and answer outcomes separately.
4. Inspect failures manually and label the cause: parsing, cleaning, chunk boundary, missing metadata, stale version, retrieval, or generation.
5. Keep the more complex method only when its improvement justifies its cost and operational burden.

This approach prevents a common mistake: declaring a chunking method successful because the final answer sounded good in a handful of examples.

Evaluate the evidence returned to the model before evaluating the prose produced by the model.

Build a goldset smoke test before changing chunking

Before comparing splitters, create a small, versioned set of representative questions. Each record should include the expected answer or an explicit “unanswerable” label, the authoritative source, and when possible, the evidence chunk or element IDs that should be retrieved. Include normal questions, ambiguous wording, exact identifiers, table lookups, cross-page references, likely typos, stale-version conflicts, permission-denied cases, and questions for which the corpus should not answer.

Run this set against the retrieval layer without generation first. Record whether the expected evidence appears in the top-k results, then inspect the misses manually. A useful failure taxonomy is: extraction failure, missing or wrong chunk boundary, missing metadata, stale or unauthorized version, ranking failure, or answer-generation failure. This prevents a fluent answer from hiding a retrieval defect.

Keep the goldset alongside the parser, chunking, metadata, and embedding versions. After each pipeline change, compare hit rate, MRR or nDCG where appropriate, evidence coverage, answer quality, and latency percentiles. The goal is not to produce a perfect benchmark; it is to make regressions visible before users find them.

A reproducible preprocessing experiment

Freeze the source snapshot, question set, expected evidence, embedding model, vector index, top-k, reranker, generation prompt, model temperature, access policy, and cache conditions. Change one preprocessing variable at a time, such as parser route, cleaning rule, chunk size, overlap, metadata field, or deduplication policy.

Measure retrieval and answer outcomes separately, inspect misses manually, record cost and latency percentiles, and retain raw results. Do not publish a percentage improvement unless the dataset, versions, metric definition, and comparison conditions are documented and reproducible.

Step 9: Keep the RAG index fresh

A RAG index is not finished when the first batch of vectors is stored. Documents change, permissions change, source systems change, and embedding models are replaced.

Microsoft’s advanced RAG guidance describes scheduled and trigger-based updates, selective reindexing, versioning, snapshotting, real-time processing, and hybrid update strategies. The right choice depends on corpus size, update frequency, freshness requirements, and available infrastructure.

At minimum, track the following fields:

Use incremental processing when only a small portion of the corpus changes. Use a full reindex when the embedding model, chunking logic, or normalization rules change in a way that affects the entire collection. Keep snapshots or rollback paths for high-stakes systems.

Freshness should also be visible to retrieval. A current policy and an archived policy may both be relevant, but they should not be indistinguishable. Add effective dates and version metadata so the retriever can apply the right temporal logic.

Reprocessing decision matrix

A successful job completion is not proof of data completeness or authorization correctness. Record counts, hashes or source versions, failed items, deletion results, and evaluation outcomes before retiring the previous index.

A worked example: preprocessing a legal contract corpus

The following is an illustrative information-retrieval design, not legal advice and not evidence that a particular legal RAG system is accurate. A production deployment should involve qualified legal and privacy reviewers, jurisdiction-specific retention rules, access controls, and an evaluation set built from authoritative clauses.

Suppose a firm wants a RAG assistant that answers questions about non-compete clauses across thousands of contracts.

A naive pipeline extracts PDF text, splits every 500 tokens, embeds the chunks, and returns the top results. It may fail when a clause is separated from its heading, a cross-reference lands in another chunk, or an amended contract is indexed alongside the original.

A stronger pipeline does the following:

• Preserves the relationship between contract, article, section, clause, and paragraph.
• Extracts tables, footnotes, and cross-references with layout-aware parsing.
• Normalizes defined terms and cross-references such as “the Parties,” clause numbers, amendments, and “hereinafter” labels so linked provisions can be retrieved together.
• Tags each clause with contract ID, execution date, parties, amendment status, jurisdiction, and source authority.
• Separates current contracts from superseded versions.
• Uses clause-level chunks for matching and parent sections for context.
• Combines lexical retrieval for phrases such as “non-compete” with semantic retrieval for related language.
• Filters by jurisdiction and execution date before reranking.
• Evaluates questions against known clauses instead of judging only whether the final answer sounds plausible.

The lesson is not that every legal corpus needs the same pipeline. The lesson is that the retrieval unit should reflect the document’s logic. A contract is not merely a long string; its hierarchy is part of its meaning.

Two documented implementation patterns

Contextual Retrieval is an Anthropic technique that prepends chunk-specific explanatory context before creating contextual embeddings and a contextual BM25 index. Anthropic’s published experiments reported lower top-20 retrieval-failure rates in its tested domains and configurations, including 35% for contextual embeddings, 49% for contextual embeddings plus BM25, and 67% when reranking was added.

These are first-party experiment results with a defined metric, dataset mix, embedding configuration, and top-k; they are not universal guarantees for every RAG corpus.

Treat the technique as an advanced experiment. Account for contextualization calls, prompt quality, caching, generated-context errors, index rebuilds, latency, and cost. Compare it with the same baseline and report raw results if you publish numbers.

Elastic: enriching a dense annual report before indexing

Elastic’s advanced RAG tutorial uses the company’s 2023 annual report as a difficult test document rather than a clean paragraph collection. The implementation combines sentence-level, token-aware chunking with generated keyphrases, potential questions, named entities, and composite multi-field embeddings.

The useful lesson is the shape of the experiment. Instead of changing every component at once, the tutorial names the preprocessing and retrieval techniques it wants to test, connects them to a concrete document, and compares the resulting system with a baseline.

The public page does not provide a universal performance number that can be generalized to every corpus, so the takeaway is methodological: use a difficult representative document and isolate the variables you change.

Databricks: treating preprocessing as a sequence of quality gates

Databricks presents an unstructured-data pipeline that converts text files and PDFs into a vector index through distinct stages: corpus selection, parsing, enrichment, metadata extraction, deduplication, filtering, chunking, embedding, and indexing. Its companion notebook makes the workflow more concrete by naming parsing and OCR tools and by recommending manual review of sampled output.

This pattern is valuable for teams that need operational clarity. A parsing failure, duplicate record, or irrelevant document can be detected before it becomes an embedding or retrieval problem. The guide is an implementation reference rather than a benchmark report, so it should be used for pipeline design and quality gates, not as evidence that one tool or splitter is universally superior.

Together, these examples suggest a practical standard: build a baseline, test it on representative documents, preserve the intermediate artifacts, and measure the effect of each major preprocessing decision.

Common RAG preprocessing mistakes

Treating extraction as a solved problem

A parser can return text and still lose tables, reading order, headings, or footnotes. Review outputs by document type.

Choosing a chunking method by reputation

A popular splitter is not automatically the right splitter. Use a reproducible baseline and measure alternatives on representative queries.

Storing metadata outside the retrieval path

Metadata that never reaches the index cannot support filtering, ranking, provenance, or freshness logic.

Mixing current and obsolete versions

Conflicting documents increase the burden on retrieval and generation. Make version status explicit.

Deleting the original text after cleaning

Keep the source representation for auditability and the cleaned representation for vectorization or lexical search.

Using generated metadata as unquestioned truth

Summaries, keywords, questions, and entities can improve recall, but they can also be wrong. Validate them and preserve their generated status.

Evaluating only the final answer

A fluent answer can hide a retrieval failure. Inspect the evidence, ranking, source version, and citation path.

Ignoring query preprocessing

Document preprocessing is central, but ambiguous, compound, or typo-filled queries may also need rewriting, decomposition, classification, or metadata filters. The full RAG workflow includes both sides of the retrieval interaction.

Failure taxonomy for debugging

When a RAG answer fails, label the first failed stage rather than blaming the language model automatically.

A useful taxonomy is: source coverage failure, authorization failure, parsing or OCR failure, cleaning failure, chunk-boundary failure, metadata or version failure, embedding/index failure, ranking failure, context-assembly failure, generation or citation failure, and freshness or deletion failure.

Store the label with the evaluation result so repeated failures reveal where engineering effort is actually needed.

Production checklist

Before deploying a RAG preprocessing pipeline, verify that:

• the corpus has a documented scope, owner, permission model, and retention policy;
• raw sources are preserved and every chunk has a traceable source ID;
• PDFs, scans, tables, images, HTML, spreadsheets, and slides have format-specific parsing rules;
• cleaning removes repeated noise without deleting meaningful headings, identifiers, negation, or provenance;
• original and cleaned text are stored separately where needed;
• metadata includes hierarchy, dates, source authority, version, language, and access controls;
• filtering, exact deduplication, near-deduplication, and version selection are tested separately;
• fixed-size or structure-aware chunking has been benchmarked before adding semantic complexity;
• embeddings and indexes carry model and pipeline version information;
• evaluation measures retrieval, evidence, answers, latency, and cost;
• update, reindexing, rollback, and monitoring paths are defined;
• source content is treated as data rather than instructions during enrichment.

Minimum release gates

Before production, require all of the following: a representative and versioned goldset; a source-to-chunk lineage check; an authorization test for allowed, revoked, and cross-tenant content; a deletion or tombstone test; an OCR and table sample review; an embedding/index compatibility check; a rollback or dual-read plan for major migrations; a prompt-injection red-team suite; and an owner for freshness, failed items, reprocessing, and incident review. A green ingestion job is not sufficient evidence that the RAG system is ready.

Conclusion

Reliable RAG does not begin when the user submits a query. It begins when a source document enters the pipeline and continues through authorization, parsing, representation, retrieval, generation, evaluation, and updates.

The strongest preprocessing strategy is not the one with the most sophisticated splitter or the longest list of tools. It is the one that preserves document meaning, carries useful context and provenance, handles duplicates and versions deliberately, enforces access decisions outside the model, keeps the index fresh, and proves its value with representative retrieval and answer tests.

Start with a simple baseline, preserve the original source, make structure and metadata visible, and add complexity only when the corpus and measurements justify it. No preprocessing pipeline can guarantee a grounded answer or eliminate prompt injection; it can make failures more observable and reduce avoidable evidence loss.

FAQ

About The Author

A Gadallh

Ahmed Gadallah is the Founder and Editor of Vertex Frontier, where he publishes research-driven articles on AI, data science, cloud computing, cybersecurity, software engineering, and emerging technologies, with a focus on technical accuracy, clarity, and practical insights.

View all articles by A Gadallh →