No. 03 — Graph RAG for PH bills
Ask about any Philippine bill and get a cited answer, plus how it connects to the laws it changes and the bills it came from.
The problem
Bills are public but buried in scattered PDFs and refiled every Congress, so tracing a bill's history takes hours of manual work.
What I built
A graph RAG system where every link between bills and laws is backed by a quote, and every answer cites its sources.
The result
Built around a 120+ question golden set that compares four retrieval setups on accuracy, faithfulness and when to say "I don't know".
01See it work
Demo video coming soonVideo coming soon: Walkthrough: ask a question, follow the citation, trace a bill's lineage.
- 120+
- hand-verified eval questions
- 4
- retrieval setups compared head to head
- 14
- unit tests on the ingest and graph code
- 20
- unanswerable questions to test abstention
02Why it matters
Compliance teams, journalists and researchers keep asking the same questions: what does this bill change, which law does it amend, and haven't we seen it before? The data is public, but it's spread across PDFs and metadata, and the same bill comes back under a new number every Congress. Plain RAG can quote a page, but it can't tell you that a bill amends a specific law and was refiled from a bill three Congresses ago.
03The idea
In legislation, the relationships matter as much as the text. So I built a knowledge graph next to the vector index, and made every edge auditable: it records how it was found, how confident that is, and the exact quote that supports it.
04How it works
- DeterministicIngestOpen Congress API metadata (8th–20th) plus bill PDFs
- DeterministicParse & embedPyMuPDF text, chunked into pgvector
- AI modelExtract edgesRegex for AMENDS / REPEALS; Claude for MANDATES
- Verification gateVerify edgesQuote must exist; REFILED_AS needs cosine ≥ 0.90 and fuzzy ≥ 80
- AI modelRetrieveVector search plus graph hops
- OutputAnswerClaude answers with citations or abstains
05Decisions that mattered
Deterministic first, LLM second
Regex handles the explicit edges with exact provisions. The LLM only adds what regex can't find, and its quotes are verified. Precision and recall are measured for both setups.
Abstaining is a feature
20 of the golden questions have no answer in the data. A legal research tool that guesses is worse than useless, so abstention gets its own metric.
Respect the source
The Senate's PDF host sits behind a bot check. The pipeline stops after 3 blocked requests and offers a manual import path instead of working around it.
Messy data, one bad record at a time
API fields change shape between records (a list in one, a string in another). Every field is parsed defensively, and one bad bill is skipped instead of failing the whole run.
06How I know it works
The golden set has 120+ hand-verified questions: 40 fact, 35 relation, 25 lineage and 20 unanswerable. Four setups are scored on recall@8, MRR, accuracy, faithfulness, abstention, latency and cost per query. An LLM judge does the grading, with a 20% human spot check.
The eval harness is built and the full-text import is in progress. Results go here once they're measured, not before.
07What's next
- Finish the full-text import (~150 bills) and run the eval
- Hand-label ~50 bills to score the graph's edges