This lab retrieves exact excerpts from a tiny fictional DemoFIFO specification collection. It is a complete Python 3.12 standard-library listing: no network, installations, files, embeddings, vector database, or model calls. It prints EXTRACT when it finds one eligible passage covering the requested keywords and ABSTAIN otherwise. Extraction demonstrates retrieval mechanics only. It is not an LLM answer generator, and its keyword rule is not a proof that the passage answers an arbitrary natural-language question.
Each record carries a document ID, project, revision, section, chunk ID, approval flag, and source locator. The synthetic locator identifies embedded course text; it does not point to a real specification or perform a fetch. Filtering occurs before scoring. A more recent but unapproved draft cannot displace approved revision 2.0 evidence, and revision 1.0 cannot answer a revision 2.0 request. The caller supplies a small controlled set of evidence keywords rather than relying on automatic question interpretation.
Learner run command for this listing as an independent scratch script: python3 specification_lab.py. It only reads the embedded collection and prints decisions; use a Python 3.12 interpreter.
import re
DOCS = [
{"project": "DemoFIFO", "doc_id": "demo-fifo", "revision": "1.0",
"section": "3.1", "chunk_id": "reset-v1", "approved": True,
"source": "synthetic://demo-fifo/1.0#3.1",
"text": "Reset release requires two rising clock edges before accepting traffic."},
{"project": "DemoFIFO", "doc_id": "demo-fifo", "revision": "2.0",
"section": "3.1", "chunk_id": "reset-v2", "approved": True,
"source": "synthetic://demo-fifo/2.0#3.1",
"text": "Reset release requires three rising clock edges before accepting traffic."},
{"project": "DemoFIFO", "doc_id": "demo-fifo", "revision": "2.0",
"section": "4.2", "chunk_id": "read-v2", "approved": True,
"source": "synthetic://demo-fifo/2.0#4.2",
"text": "Read latency is two cycles after an accepted read."},
{"project": "DemoFIFO", "doc_id": "demo-fifo", "revision": "2.1-draft",
"section": "3.1", "chunk_id": "reset-draft", "approved": False,
"source": "synthetic://demo-fifo/2.1-draft#3.1",
"text": "Reset release requires four rising clock edges before accepting traffic."},
]
def tokens(text):
return set(re.findall(r"[a-z0-9_]+", text.lower()))
def retrieve(query, revision, required_terms, docs=DOCS):
required = set()
for term in required_terms:
required.update(tokens(term))
if not required:
return {"status": "ABSTAIN", "reason": "No evidence terms specified"}
eligible = [d for d in docs if d["project"] == "DemoFIFO"
and d["revision"] == revision and d["approved"]]
if not eligible:
return {"status": "ABSTAIN", "reason": "No approved requested revision"}
candidates = []
query_terms = tokens(query)
for doc in eligible:
words = tokens(doc["text"])
if required <= words:
candidates.append((len(query_terms & words), doc))
if not candidates:
return {"status": "ABSTAIN", "reason": "Missing requested keyword evidence"}
candidates.sort(key=lambda item: (-item[0], item[1]["chunk_id"]))
best_score = candidates[0][0]
best = [doc for score, doc in candidates if score == best_score]
if len(best) != 1:
return {"status": "ABSTAIN", "reason": "Ambiguous evidence; review candidates"}
doc = best[0]
return {
"status": "EXTRACT", "excerpt": doc["text"],
"citation": {key: doc[key] for key in
("doc_id", "revision", "section", "chunk_id", "source")},
}
cases = [
("reset", "reset release", "2.0", ["reset", "release"]),
("read", "read latency", "2.0", ["read", "latency"]),
("wrong-version", "reset release", "3.0", ["reset", "release"]),
("missing", "reset minimum pulse width", "2.0", ["reset", "pulse", "width"]),
("unapproved", "reset release", "2.1-draft", ["reset", "release"]),
]
results = {label: retrieve(query, revision, terms)
for label, query, revision, terms in cases}
expected = {
"reset": {
"status": "EXTRACT",
"excerpt": "Reset release requires three rising clock edges before accepting traffic.",
"citation": {"doc_id": "demo-fifo", "revision": "2.0", "section": "3.1",
"chunk_id": "reset-v2", "source": "synthetic://demo-fifo/2.0#3.1"},
},
"read": {
"status": "EXTRACT", "excerpt": "Read latency is two cycles after an accepted read.",
"citation": {"doc_id": "demo-fifo", "revision": "2.0", "section": "4.2",
"chunk_id": "read-v2", "source": "synthetic://demo-fifo/2.0#4.2"},
},
"wrong-version": {"status": "ABSTAIN", "reason": "No approved requested revision"},
"missing": {"status": "ABSTAIN", "reason": "Missing requested keyword evidence"},
"unapproved": {"status": "ABSTAIN", "reason": "No approved requested revision"},
}
assert results == expected
for label, result in results.items():
print(label, result)
print("PASS: synthetic keyword excerpts, citations, and abstentions")Case | Expected result |
|---|---|
reset | EXTRACT: three rising clock edges; demo-fifo revision 2.0 section 3.1, reset-v2. |
read | EXTRACT: two cycles after an accepted read; revision 2.0 section 4.2, read-v2. |
wrong-version | ABSTAIN: No approved requested revision. Do not fall back to 1.0 or 2.0. |
missing | ABSTAIN: Missing requested keyword evidence. No reset pulse width appears. |
unapproved | ABSTAIN: No approved requested revision. The draft is excluded. |
These are expected decisions, not recorded executions or performance measurements. Inspect the reset citation as well as its number: a three-edge answer citing reset-v1 would fail review. The read result preserves 'accepted read,' which matters when a request can be stalled. In an answer-writing extension, pass the excerpt and citation together and check that every generated claim is supported. Keep the abstention path available even when a generator could produce plausible prose.
ABSTAIN with Missing requested keyword evidence: read-v2 makes revision 2.0 eligible, but it lacks reset/release. reset-v1 is filtered out before matching. Falling back would silently change the applicable rule from three edges to two.
No. Split the question into supported release-delay and unsupported pulse-width parts, preserving an explicit abstention for pulse width. Weakening the evidence requirement would conceal the missing fact. The lab cannot infer a time value from an unstated clock frequency.
For one more inspection exercise, add a second approved 2.0 reset passage with the same keywords but a contradictory value. If their scores tie, this listing abstains as ambiguous. If one scores higher, it can still select an incompatible passage: the tie rule is not contradiction detection. A production design needs source precedence and conflict review, along with access filtering, collection snapshots, and extraction checks. The lab's approval boolean illustrates a gate, not a real authorization system.
Keyword coverage cannot understand negation or prove semantic support. It will miss synonyms and can accept passages whose words match for the wrong reason. Keep data-owner approval and read-only access in any real adaptation; do not paste proprietary specifications into an unapproved service. Reference: Python 3.12 regular-expression documentation for token matching. Revisit the review set from the search lesson before adding an actual model or optional vector storage.