A specification PDF contains a timing table, an exception in a footnote, and a heading identifying low-power mode. A text extractor keeps the table value but loses the footnote and heading. Search may now retrieve a perfectly readable passage that means something different from the source. Document preparation is therefore an engineering interpretation task as well as a text-processing task. Inspect extracted passages against the approved original, particularly tables, symbols, diagrams, and qualifiers such as only, except, minimum, and maximum.
A chunk is a passage small enough to retrieve and inspect as a unit. Start with meaningful boundaries such as a requirement and its explanation, a complete table with its headers, or a short subsection. Carry the heading path into the chunk. Do not split a number from its unit or a rule from its exception merely to meet a fixed character target. Some overlap can preserve context at boundaries, but duplicated text should retain stable source identity so it is not mistaken for independent support.
Source element | What the searchable chunk must retain |
|---|---|
DemoFIFO 2.0, section 3.1 Reset | Three rising clock edges after reset release, before traffic is accepted. |
Section 4.2 Read response | Two cycles after an accepted read; backpressure condition stays attached if present. |
A table value of 0.08 | Its heading, unit such as ns, corner, mode, and any table footnote. |
Synthetic chunk manifest, not a real specification:
doc_id: demo-fifo
revision: 2.0
section: 3.1
chunk_id: reset-v2
heading: Interface / Reset
source: synthetic://demo-fifo/2.0#3.1
approved: true
text: Reset release requires three rising clock edges before accepting traffic.The source field is a fictional locator in this course, not a downloadable document. In a real system use an approved locator that resolves to the exact immutable source revision, with a page or section anchor. Keep collection access permissions alongside the manifest. A user who cannot read the original must not gain access through an index, an excerpt, or a citation. Approval for internal reading also does not imply permission to send content to an external embedding service.
No. Retain both mode-qualified rules, or make separate chunks that each repeat the applicable mode and share the source location. A question about diagnostic mode must retrieve the four-cycle rule and must not be answered from an unqualified two-cycle fragment.
Use a preparation checklist with visible outcomes: compare a reset paragraph, a multiline table, a negative requirement, and an exception with the original. Record which pages failed extraction. For scanned pages, review recognition of characters such as 0 and O or 1 and l. For RTL identifiers, preserve underscores and case where meaningful. An assistant may help reformat text, but it must not silently rewrite normative language or fill in illegible values.
When a document revision changes, create new chunk identities or a new collection snapshot and mark superseded entries. Do not overwrite the revision field while leaving old text or embeddings attached. Update deletion and revocation paths as well as the happy path for adding documents. A reproducible answer should identify the snapshot it searched, because a later re-index may legitimately return different evidence for the same wording.
Good extraction does not resolve contradictions between approved documents; preserve both locations and request an owner decision. Preparation details depend on the document format and chosen tooling. For an authoritative example of a text index's explicit tokenizer and table configuration, see SQLite FTS5 documentation, consulted 2026-10-18. FTS5 is background reading, not a dependency of this course's plain-Python lab.