Loop Engineering for Cross-References: When RAG Answers ‘see Section 7.2’ Instead of the Actual Answer

gives you this answer: “the applicable sublimit is defined in Section 7.2 of the policy.” Now read it as the user. Yes… and? What does Section 7.2 say? What is the actual number? The answer is not wrong, it is unfinished: the retrieved passage points at the answer instead of containing it, and nothing in the pipeline went to get it. Contracts, standards, and papers are full of these internal pointers; the question is how to follow them systematically, rather than hoping the right page was fetched on the first try. This article is part of Part III of Enterprise Document Intelligence, a series that builds an enterprise RAG system from four bricks: document parsing, question parsing, retrieval, and generation. It builds that systematic way: the parser flags references cheaply, generation reports the unresolved ones, and the orchestrator loops back for the target. where this article sits in the series: Article 11 (cross-references), in Part III – Image by author 📓 The runnable companion walks the two-pass loop yourself: you run pass 1 on the Attention paper, print the pending_references the LLM flags on page 6, watch the resolver join “Table 3 row (E)” against the object registry, and see pass 2 come back complete. On GitHub: doc-intel/notebooks-vol1. The public companion-code repo at doc-intel/notebooks-vol1 – Image by author 1. Where cross-references live in each brick The reference happens at parse time (the document author writes it), at retrieval time (a chunk containing the reference matches the query), and at generation time (the model writes “see Section X” instead of fetching Section X). The pointer could be resolved at any of these three moments, and is silently dropped at each. The fix is a feedback field on the generation output that flags an unresolved reference, plus an orchestrator that catches the signal, resolves the reference against the parsing brick’s relational tables, re-retrieves the linked region, and re-runs generation with the linked context now in hand. The architectural point is that the orchestrator loops on signals from the structured output, not on confidence scores. The same machinery used in Article 10 for parsing quality is used here for references. Article 12 applies it to a third dimension (listing completeness); Article 13 (the RAG workflow) names the pattern explicitly and shows how the patterns compose. Parsing is where references are initiated, but cheaply. Native PDF links (the kind you click on in a PDF viewer) come for free from the parser and go into the cross_ref_df table at zero cost. Anything beyond that, regex detection of “see Section 4.2”, glossary tagging, conditional-clause flagging, is not done upfront. The vocabulary is too varied (“cf.”, “see”, “per”, “as defined in”) and the volume too high to extract everything before knowing what will be needed. Question parsing does not change. The user’s question is parsed into the usual question_df plus satellites (Article 6). References do not require a new question shape. Retrieval does not change on the first pass. It runs the standard retrieval from Article 9 (TOC pages plus keyword pages, merged). The references that exist in the candidate passages are not yet resolved. Generation is where the new contract appears. The Pydantic schema adds two fields: a list of pending_references (references the LLM detected in the candidates that look like they would change the answer if resolved) and an answer_completeness flag (complete, references_unresolved, or partial). These two fields are the feedback signal. The orchestrator (the decide.py from Article 13) reads the structured output and decides whether the answer is shippable or whether a second pass is needed. 2. The running example: “see Table 3 row (E)” The example runs on the Attention Is All You Need paper (Vaswani et al. 2017; arXiv non-exclusive distribution license, declared on the arXiv abstract page; version v7), the same document Article 1 used. Runnable code paths call OpenAI services governed by OpenAI’s Terms of Use. The question: “Do learned positional embeddings give similar results to sinusoidal ones in the Transformer paper?” The prose on page 6 (Section 3.5 Positional Encoding) gives a qualitative answer (“the two versions produced nearly identical results”) and adds “see Table 3 row (E)”. The actual numbers are on page 9. With a top-1 retrieval policy (defended in section 6), the first pass fetches page 6 and the table is missed. The rest of the article walks the steps. Pass 1 flags incompleteness; orchestrator triggers Pass 2; converged answer ships with provenance – Image by author 3. First pass: the pipeline runs once 3.1 Parsing initiates references cheaply The parser from Article 5B (the relational data model) runs on the PDF and produces the usual tables: line_df, page_df, toc_df, plus cross_ref_df. Article 5 introduces cross_ref_df (one row per body-text mention of a named object); this article extends it with a source column that records what produced each row, and fills the regex and LLM rows on demand. The cross_ref_df populated at parse time contains only the references that are free or almost free to extract: PDF native links (created by the document producer using LaTeX \hyperref or Word’s hyperlink feature), section anchors from toc_df, and named objects (figures, tables, annexes) that the parser already locates. The Transformer paper has few native links. What ends up in cross_ref_df after parsing is the structure already known from toc_df (Sections 1 through 7 plus subsections), plus the object registry (Figures 1 through 5, Tables 1 through 4). What is not in cross_ref_df at this point: the dozens of in-prose mentions like “see Figure 2”, “as described in section 3.2”, “see Table 3 row (E)”. These need a regex pass, and running it upfront is wasted work. It runs on demand, in the second pass, only on the candidates retrieval returned and generation flagged. ref_target is a polymorphic foreign key read together with ref_type. When ref_type == "section", ref_target is a section number like "5.2" that joins back to toc_df.toc_id. When ref_type is anything else (figure, table, equation, annex, appendix, bib), ref_target is an object id like "1", "A", "18" that joins back to object_registry.object_id. A single column per row, the target table is unambiguous from the type. The resolver in section 5 walks both joins. class ReferenceRow(BaseModel): origin_page: int # page where the reference appears origin_line: int # line where the reference appears anchor_text: str # "see Section 5.2", "Table 3 row (E)", "[18]" ref_type: str # "section" | "table" | "figure" | "annex" | "bib" | "external" ref_target: str # parsed target id ; polymorphic FK on ref_type # section -> toc_df.toc_id ("5.2") # else -> object_registry.object_id ("3" / "A" / "18") target_page: int | None # resolved page if cheap (native_link), else None source: str # "native_link" | "object_registry" | "toc_anchor" | "regex" | "llm" # At parse time, only rows with source in {"native_link", "object_registry", # "toc_anchor"} are populated. The regex and llm sources are filled on demand # during the second pass, when the orchestrator asks for them. Here is what cross_ref_df contains after parsing the Transformer paper. Native PDF links dominate the table; we show a representative slice of section cross-references plus a couple of bibliography citations for contrast. Every row from one page.get_links() call; section rows are what resolution joins against – Image by author 3.2 Question parsing, then top-1 retrieval Question parsing (Article 6) extracts the keywords learned positional embeddings, sinusoidal, positional encoding, marks the expected answer shape as “comparison”, and sets the scope filter empty (the question is about the whole document). Retrieval runs the standard stack from Article 9 (TOC pages plus keyword pages, merged), and we deliberately take only the top-1 page. On this question the top hit is page 6 (Section 3.5 Positional Encoding). Page 9 (Table 3) is the second-ranked candidate but is not fetched under a top-1 policy. Section 6 defends the top-1 choice; for the rest of this section, take it as the policy. The single candidate (page 6) contains a reference to “Table 3 row (E)” in its prose. On the first pass we do not resolve it. We pass page 6 to generation and let the LLM flag what it finds. The top-1 passage with the forwarding pointer the LLM will flag – Image by author 3.3 Generation flags the unresolved references The generation schema in this article extends the one from Article 8 with two fields: pending_references and answer_completeness. The LLM is prompted to populate both: if any candidate passage points to a region of the document that was not retrieved, the LLM lists that pointer in pending_references and sets answer_completeness to "references_unresolved". class ReferenceAwareAnswer(BaseModel): answer: str # the prose answer citations: list[Citation] # line-level provenance # Reference-loop feedback fields (the contribution of this article). pending_references: list[PendingReference] # what to resolve in pass 2 answer_completeness: Literal["complete", "references_unresolved", "partial"] # the routing signal class PendingReference(BaseModel): raw_text: str # "Section 5.2", "Table 3 row (E)", "SP 800-161r1" ref_type: str # "section" | "table" | "figure" | "external" | ... origin_page: int # page where the reference was mentioned origin_line: int # line where the reference was mentioned On this question, the LLM produces this: { "answer": "According to the Transformer paper, the authors experimented with learned positional embeddings as an alternative to the sinusoidal encodings used in their base model. They report that the two versions produced nearly identical results. The detailed comparison is in Table 3 row (E) of the paper.", "citations": [ {"page": 6, "line_start": 22, "line_end": 30, "retrieved_via": "primary"} ], "pending_references": [ {"raw_text": "Table 3 row (E)", "ref_type": "table", "origin_page": 6, "origin_line": 28} ], "answer_completeness": "references_unresolved" } The prose part is what the naive pipeline would have returned and stopped. It is a half-answer: qualitatively correct but missing the numbers, which are pointed to but not retrieved. The two structured fields at the bottom are what makes the second pass possible. 4. The orchestrator reads the feedback and loops back The orchestrator runs after generation. It reads answer_completeness first. If complete, the answer is returned to the user. If references_unresolved and the loop count is below the budget (1 by default, like Article 10’s adaptive re-parse), it triggers the second pass. def decide_next_pass(answer: ReferenceAwareAnswer, loop_count: int, max_loops: int = 1) -> str: \"\"\"Return one of: 'return_to_user', 'resolve_references', 'give_up_partial'.\"\"\" if answer.answer_completeness == "complete": return "return_to_user" if answer.answer_completeness == "references_unresolved": if loop_count ResolvedReference: \"\"\"Resolve one pending reference. Try cheap first, fall back to LLM.\"\"\" # 1. Deterministic : numbered section lookup. if ref.ref_type == "section": section_id = _section_id_from_text(ref.raw_text) # "Section 5.2" -> "5.2" match = toc_df[toc_df["section_id"] == section_id] if not match.empty: return ResolvedReference.from_toc(ref, match.iloc[0]) # 2. Deterministic : object registry for tables, figures, annexes. if ref.ref_type in ("table", "figure", "annex"): obj_id = _object_id_from_text(ref.raw_text) # "Table 3 row (E)" -> "3" obj = object_registry.get((ref.ref_type, obj_id)) if obj is not None: return ResolvedReference.from_object(ref, obj) # 3. External : flag as out-of-corpus, do not re-retrieve. if ref.ref_type == "external": return ResolvedReference.external(ref) # 4. Ambiguous phrasings : ask the LLM with origin context and toc_df. if llm_client is not None: origin_context = _surrounding_lines(line_df, ref.origin_page, ref.origin_line, window=3) guessed_section = _llm_resolve(llm_client, ref.raw_text, origin_context, toc_df) if guessed_section is not None: return ResolvedReference.from_llm(ref, guessed_section) return ResolvedReference.unresolved(ref) For the one pending reference in our example: “Table 3 row (E)” resolves deterministically against the object_registry to page 9, with the row tag (E) kept as a sub-selector so retrieval can focus on the right row of the table. 5.3 Re-retrieval, re-generation, complete answer The pipeline re-retrieves the lines from page 9 (the Table 3 area, with the row (E) sub-selector applied), joins them with the original primary candidate (page 6), and re-runs generation. The schema is the same ReferenceAwareAnswer, but the prompt now includes the newly-retrieved content with metadata that tells the LLM “this passage was pulled in because of the reference ‘Table 3 row (E)’ on page 6, line 28”. Second-pass fetch; row (E) carries the perplexity and BLEU numbers completing the answer – Image by author The second-pass answer: { "answer": "The Transformer paper compares learned positional embeddings against the sinusoidal positional encodings used in the base model. Both produce essentially the same performance on the English-to-German development set newstest2013 : 4.92 perplexity and 25.7 BLEU for learned embeddings (Table 3 row (E)), versus 4.92 PPL and 25.8 BLEU for the base model with sinusoids. The 0.1 BLEU difference is well within noise on this benchmark, confirming the authors' claim that the two versions produced nearly identical results.", "citations": [ {"page": 6, "line_start": 22, "line_end": 30, "retrieved_via": "primary"}, {"page": 9, "line_start": 38, "line_end": 39, "retrieved_via": "reference", "referenced_from": [6, 28], "reference_text": "Table 3 row (E)"} ], "pending_references": [], "answer_completeness": "complete" } The orchestrator reads answer_completeness = "complete" and returns the answer to the user. The loop count is 1, the budget was 1, no further iteration is allowed even if generation had asked for one. 6. Cheap defaults, signal-driven expansion Two cheap-default decisions made the example above work: the parser does not pre-extract every reference, and the retriever returns only the top-1 page. Both follow the same discipline that Article 10 applied to parsing: start cheap, let generation signal what is missing, expand only on the signal. This section makes both choices explicit and defends them. 6.1 Lazy reference extraction at parse time The instinct when implementing this is to do a regex pass at parse time and populate cross_ref_df with every detected reference upfront. Same instinct Article 10 had to resist for parsing: run the expensive parser everywhere just in case. Three arguments against. Volume: A 200-page contract may contain hundreds of in-prose references. Most will never be needed because retrieval will never return the passages they sit in. Pre-extracting all of them is waste. Vocabulary: “See”, “per”, “cf.”, “as discussed in”, “in line with”, “subject to”, “in accordance with the foregoing”. Each document family invents its own variants. A regex that covers all of them is either huge and brittle, or narrow and incomplete. Both are bad. The LLM, asked at resolution time with the origin context in hand, handles the ambiguity correctly. Maintenance: A pre-extracted cross_ref_df becomes one more artifact the team has to keep current as new document families come in. A resolver that runs on demand reads the parsed structures (toc_df, object_registry) that already exist and asks the LLM for the rest. No new artifact to maintain. The exception, mentioned in section 3.1, is the cheap cases: native PDF links and structural anchors derived from toc_df. These cost nothing at parse time and are worth populating upfront because they support deterministic resolution without a regex or an LLM call. 6.2 Top-1 retrieval, not top-k The second cheap default is the retrieval policy. We send the LLM only the top-1 page, not the top-3 or top-5. The instinct in most RAG tutorials is to default to top-3 or top-5 “to be safe”; the argument here goes the other way. Enterprise answers are usually unique. A specific question on an enterprise document has one canonical passage that answers it. The premium for water damage lives in one row of one schedule. The indemnification cap is in one clause. The Transformer’s positional-embedding comparison is in one row of one table. Pulling top-3 by default usually means pulling one good passage plus two weak ones that consume context budget and confuse generation. The loop catches what top-1 misses, when the miss matters. If the top-1 passage is a stub that points elsewhere (the “see Table 3 row (E)” shape), the cross-reference loop fetches the target deterministically. If top-1 is incomplete in a way that does not flag a reference, the orchestrator’s other signals (parsing quality, completeness) catch it on a different axis. Cost adds up across queries: The Transformer example above cost two LLM calls (first pass on page 6, second pass on page 6 plus page 9 region). A top-3 first pass that already included page 9 would have cost one LLM call but with three pages of context, two of which contributed nothing to the answer. Across thousands of queries the second pattern wastes more tokens than the first. The exception is questions where the answer is in multiple places: “every clause that mentions termination” (a listing question, Article 12) or “all cross-references to Annex A” (a sweep, also Article 12). For those, top-k is the right policy. For factual questions of the form “what is X”, top-1 plus loop is the default. 6.3 The pattern across the series The same shape recurs in every article of Part III. Article 10 keeps cheap parsing as the default and escalates on a context_structured = False signal. Article 11 keeps cheap reference initiation and top-1 retrieval as the default, and expands on a pending_references signal. Article 12 keeps a small candidate set as the default and expands on a completeness signal. Article 13 (the RAG workflow) names the pattern explicitly and composes the three: the orchestrator reads typed feedback fields from the structured output and routes the pipeline accordingly. Cheap default, signal-driven expansion, bounded iteration. These generation-triggered loops are the big loops of the pipeline, the ones that cross bricks; each brick also runs its own small bounded loops inside (the image cascade in parsing, the TOC descent in retrieval, the schema retry in generation), and the two scales follow the same three-surface discipline. 7. Variations: the four kinds of cross-reference The example walked through a table reference (“Table 3 row (E)”) resolved deterministically against the object registry. Three other kinds appear in enterprise documents and follow the same loop structure with small tweaks to the resolver. Section references: “See Section 4.2”, “As described in Section 3.2.2”. The resolver uses toc_df instead of the object registry. Re-retrieval pulls the lines from the resolved section’s page range. This is the canonical case in contracts and standards, where a clause delegates to another clause by section number. Conditional clauses: “This applies only if X”, “in the case where the policy is renewed”. The generation schema flags these too, but the loop behavior is different. The pipeline does not try to resolve the condition (no automated logical reasoning). It re-runs generation with an instruction to state the condition explicitly in the answer, so the user sees “the rule applies if the policy is renewed within 30 days” instead of an answer that silently assumes the condition is met. Definitional references: “Insured Party (as defined in Section 1.3)”, “Multi-Head Attention (defined in Section 3.2.2)”. The resolver pulls the definition into the candidate set on the second pass. This prevents the LLM from importing a generic notion when the document defines the term specifically. Definition passages are short, so the second-pass context cost is low. External references: “see SP 800-161r1”, “per the requirements of ISO 27001 Annex A”. These point to documents outside the corpus. The resolver flags them as external and the generation step names them explicitly, so the user knows there is a supplementary resource without the pipeline pretending to have fetched it. 8. Conclusion The cross-reference loop is the same architectural pattern as Article 10’s adaptive parsing: start cheap, let generation signal what is missing through typed fields, and react to the signal in a deterministic orchestrator rather than an autonomous agent. The pending references resolve against the relational tables parsing produced (deterministic when possible, LLM-assisted when ambiguous); the second pass re-retrieves and re-generates with the resolved regions; citation provenance records the loop so an audit can replay it end to end. The next article looks at a different category of question: ones whose answer is not in any single passage but distributed across the document, where top-k retrieval is structurally wrong and the pipeline needs a different shape. 9. Sources and further reading The structure-aware retrieval approach the article uses for cross-references is the same direction as Saad-Falcon et al. (PDFTriage, Adobe Research 2023). Multi-hop retrieval following references is a multi-hop QA problem, with established benchmarks from Trivedi et al. (MuSiQue, TACL 2022) and Yang et al. (HotpotQA, EMNLP 2018). GraphRAG (Edge et al., 2024) shares the same problem statement and solves it differently: LLM-extracted entity-relation graph across the corpus, traversed at query time. The agent-resolves-references-at-runtime line, with the regex-populated cross_ref_df becoming one of the audited tools the agent calls, is follow-up work. Earlier in the series: Document Intelligence: series intro. What the series builds, brick by brick, and in what order. What works, what breaks Baseline Enterprise RAG, from PDF to highlighted answer. The four-brick pipeline end to end: PDF in, highlighted answer out. Embeddings Aren’t Magic: The Predictable Failure Modes of RAG Retrieval. Where embedding similarity wins (synonyms, typos, paraphrase), where it predictably breaks (unknown terms, negation, term-vs-answer relevance), and how to use it anyway. Rerankers Aren’t Magic Either: When the Cross-Encoder Layer Is Worth the Cost. What a cross-encoder adds over bi-encoder embeddings, measured, and when it is worth the latency. RAG is not machine learning, and the ML toolkit solves the wrong problem. Why chunk-size sweeps and finetuning optimize the wrong thing; route by question type instead. From regex to vision models: which RAG technique fits which problem. Two axes, document complexity and question control, that pick the technique for each case. 10 common RAG mistakes we keep seeing in production. Ten production mistakes, organized brick by brick, with the fix for each. Document parsing Beyond extract_text: the two layers of a PDF that drive RAG quality. The first half of the parsing brick: the document’s nature, signals, and summary. Stop returning flat text from a PDF: the relational tables RAG needs. The second half of the parsing brick: the relational tables every downstream brick reads. When PyMuPDF can’t see the table: parse PDFs for RAG with Azure Layout. The same tables from Azure Layout: native table cells, OCR, paragraph roles. Parse PDFs for RAG locally with Docling: rich tables, no cloud upload. The same tables computed locally with Docling: TableFormer cells, nothing leaves the machine. Vision LLMs are PDF parsers too: reading charts and diagrams for RAG. Vision as a parser: the pictures become searchable text. Parse scanned PDFs for RAG with EasyOCR: free OCR gives you words, not a document. Where traditional OCR stops: text recovered, structure lost. Making a PDF’s images searchable for RAG, without paying to read them all. The image cascade: filter cheap, classify, describe only what is worth reading. Reconstructing the table of contents a PDF forgot to ship, so RAG can scope by section. Rebuilding toc_df when the PDF prints a contents page but ships no outline. Question parsing RAG questions need parsing too: turn the user’s string into briefs for retrieval and generation. The thesis of question parsing: why a user string needs the same parsing as a document, and how it splits into a retrieval brief and a generation brief. What the question parser extracts from a user string: keywords, scope, shape, decomposition, clarification. The five families of columns the parser reads straight from the user’s question, with the code that fills each one. Dispatching the parsed RAG question: chunk strategy, model tier, activations, audit. The decisions the parser makes on top of the user string, using the document’s profile: dispatch, activations, full schema, the audit trail (pipeline_trace.json), and a broker-corpus walkthrough. The Clarification Loop and Learned Defaults: When the Question Is Not Precise Enough. One focused clarification when the question is too vague, and the default learned from the answer. Retrieval Retrieval is filtering, not search: a mental model for enterprise RAG. Retrieval reframed as filtering on line_df and toc_df: anchors small, context large. Anchor detection for RAG: parallel detectors, then one LLM call at the end. Parallel anchor detectors: keyword always, embeddings alongside, one LLM call at the end. Letting an LLM pick the right RAG page: the arbiter pattern at the end of retrieval. The LLM arbiter: candidates ranked with reasons, one typed JSON out. Context Engineering: The Four Typed Inputs Behind Every Answer. Context engineering given a structure: the four typed pieces (fixed system prompt, retrieved lines, doc-context block, PromptContext wrapper) that fill one single-document RAG LLM call. Generation Stop returning text from RAG: the typed answer contract that prevents hallucination. The answer schema as the contract: typed values, items with evidence spans, self-assessment fields, and the completeness signal the pipeline computes itself. Assemble each RAG generation prompt from a base prompt plus the rules each question needs. The dispatcher: a fixed BASE prompt plus the rules each question needs, the schema picked from the registry, and the full trace kept on every call. Validating the RAG answer before the user sees it: spans, quotes, and the feedback loop. The post-generation validator (spans, verbatim quotes, formats), not-found as a first-class answer, and the feedback loops that close the pipeline. One-document pipelines A production RAG pipeline for PDFs: relational parsing, TOC retrieval, typed answers. Each of the four bricks upgraded one contract at a time: relational parsing, corpus-aware questions, TOC-routed retrieval, typed answers. Stop RAG hallucinations with context engineering: one pipeline, four very different PDFs (link to come). The four upgraded bricks wired into one call, run end to end on a paper, a compliance doc, and a broken-TOC document. Loop engineering with adaptive PDF parsing: start cheap, pay for a heavier parser only when the page needs it (link to come). The escalation cascade and the free deterministic checks that flag a failed parse before you pay for a deeper one. Loop engineering with adaptive parsing in action: flattened tables to Azure, figures to a vision LLM (link to come). The LLM as last line of defence, then two real escalations: a flat table to Azure, a figure to a vision model.