Open AI Build Week:

IMHO, there's a rush to sell AI to government right now (note: I work in government). I kept asking a different question: has anyone checked whether government is ready to buy? I built Verity Lex over about two days of OpenAI Build Week to answer it, and one rule ended up shaping the whole build: the AI reads the court's record, but it never assigns the score. That sounds like a small design choice. It's the build, and teaching a machine to respect it was the hard part, and the one that cost me a stressful evening I could have skipped. Point Verity Lex at a California superior court (Santa Barbara is live, more courts to come) and it reads the public record, checks it against published legal standards, and returns an AI-readiness score anyone can recompute. This is the story of what I built, how I directed Codex and GPT-5.6 to build it, and the one lesson I'd like to pass on to any of my Dev.to fam shipping an AI. The thesis: you're selling to government backwards Software gets sold to public institutions the wrong way around. A vendor shows up with a solution and goes looking for a problem, without ever understanding the institution it's selling into. What is this court held to? Is it meeting that standard? Are its people ready for the thing you want to sell them? Skip those questions and you get shelfware and eroded trust, which anyone who has worked in the public sector has watched happen. AI is about to run that same play at scale. So Verity Lex starts at the bottom of a pyramid, not the top: Standards. What is this institution held to, and does its own record show it's meeting it? This is Verity Lex. Structure. Is the organization itself ready? Governance, roles, operational policy. People. Do staff know how to use AI safely? Where are the culture blockers? Solutions. Only now do you get to sell something, prescribed against a diagnosed picture instead of a pitch deck's guess. You earn the right to sell to government by understanding it first. Verity Lex is tier one, shipped. The architecture: neurosymbolic on purpose Here's the trap most AI demos fall into: they put the model in charge of the answer. Ask an LLM to score a court's compliance and you get a number shaped by whatever the sampling did that second, with citations it might have dreamed. Run it twice, get two answers. No government buyer can procure "the AI felt good about our compliance." So I split the job into three, along one boundary: perception versus judgment. Finding the documents: GPT-5.6 directs, Tavily retrieves. A model-directed ReAct loop decides where to look. GPT-5.6 reasons about the next move; Tavily runs the retrieval, searching the court's public site and pulling documents a plain fetch can't reach. Tavily was new to me going in, and giving the model a real search-and-extract layer instead of a hand-rolled fetch is a chunk of why the agent can find a document that moved. Reading the documents: GPT-5.6 extracts. It reads what comes back and pulls evidence into a strict JSON schema. Finding and reading are both perception, and perception is what a model is good at. Scoring the readiness: a deterministic rule engine. Pure TypeScript, no model imported anywhere in it. It takes the extracted signals and applies a published registry of weighted, legally-grounded artifacts. Same signals in, same score out, forever. Scoring is judgment, and judgment about a number has to be reproducible. Two of those jobs are perception, one is judgment, and the design lives on keeping them apart. Would you trust an AI to grade its own homework? Neither would a court. The model has no path to the score, by construction. Every finding cites a real document and a quoted line. Anything it can't find is marked not located, never absent, because a public record going quiet isn't proof of anything. And you can download an audit bundle and recompute the score yourself. The constraint is the product. How I built it: directing Codex I built this with Codex in VS Code using what I think of as a creative-director workflow. I own the judgment. Codex owns the implementation. The boundary between us is enforced, not trusted. In practice that meant gated, block-scoped prompts. Every one started with "propose a file plan first, do not refactor unrelated code, stop after the PR." Codex built each block, the rule engine, the agent tools, the loop, the API, the hardening, CI, the add-ons, as its own pull request. I reviewed and tested every one before it merged, and CI enforced that no red PR reached main. The commit history is the collaboration log: over twenty scoped PRs, each one a discrete piece of the argument. Codex was good. It caught a next/server import trap and fixed it by using web-standard Response.json. It diagnosed a lockfile mismatch that only showed up on the Linux CI runner. And when it was wrong, I overruled it, which is the entire point of keeping a human at the gate. At one point it proposed swapping deterministic installs for a looser command to "fix" a CI failure. That would have traded away reproducibility for a problem we hadn't even diagnosed. We diagnosed it instead. It was a stale cache. GPT-5.6 runs live inside the shipped product, directing discovery and extracting evidence, with Tavily as the retrieval layer underneath it, and it is architecturally barred from touching the score. Lessons learned: my tests were green the whole time. Then I deployed. Here's the part I'd do differently. If you're building something similar, maybe it saves you the evening it cost me. At first, my CI was green. Rule engine tested, agent loop tested, bounds tested, API contract tested. I read that green as "it works." Then I set my real API keys in production, clicked the button, and watched it fail three different ways in a row. First a 400 from the model: OpenAI's JSON mode requires the word "json" to appear in the input itself, and my instruction saying so in a separate field didn't count. Then the agent's planner returned an action with no query attached, so every search got rejected and the scan found nothing. Then the extractor produced JSON in the wrong shape and my validator threw on every document. Three bugs, one root cause: every one of them lived at the boundary between my code and the real model, and my stub had hidden all of them. Here's the precise miss. My stub model always returned correctly-shaped, pre-baked responses. So my tests proved my code could consume a good model answer. They never proved my prompts could produce one. Those are different claims. My planner test literally handed the stub an action that already had the required fields, so it passed, while the actual prompt never told a real model those fields existed. The test fixture quietly knew something the prompt didn't teach. I fixed the three shape bugs. And then a worse one showed up, because I'd finally learned to actually measure: I ran the same scan five times and got different scores. Fifty-nine one run, sixty-seven the next. Same court, same day. My first guess was budget, that the agent ran out of steps before covering everything. Wrong. I handed it a generous budget and it still wandered, using a fraction of it and stopping in different places each run. The problem wasn't resources. I was asking a non-deterministic model to do a deterministic job, march through nine known legal standards and check each one, and a model does not march. It strolls. More budget just bought a longer runway. Oops! Thereason Verity Lex is "trustworthy" is that I'd taken the score away from the model and given it to deterministic code. So my first instinct was: do the same to the search. Write the queries in code, pick the links in code, let the model only read. Cage it into repeating itself. I even confirmed the model wouldn't help me the easy way, GPT-5.6 doesn't expose a temperature setting, so I couldn't just turn its randomness down. I got most of the way to convincing myself to build the cage before I saw the trap. A court renames a policy PDF, or moves it to a new page, and a hardcoded search walks right past it. The free, uncaged agent, reading the results and reasoning about them, finds it at the new location. Caging the agent would make it repeatable and blind, blind to exactly the thing this product exists to catch: change. The improvisation I was trying to delete was the value. So the lesson is more careful than "make everything deterministic." Move a guarantee into code when a second answer is simply wrong, scoring, where two numbers for the same evidence is a bug by definition. But where the adaptivity itself is the value, like finding a document that moves, don't cage it. Make it reliable a different way. And the different way is the thing I'd cut from v1 for simplicity: memory. The scan wanders because it's stateless. It re-improvises the entire hunt every single time, starting from nothing. Ugh, my design bad. A version that remembered what it found last time wouldn't re-hunt a known policy, it would re-verify it, and when the policy moved it would notice the move instead of silently missing it. The variance I'd been treating as a defect turns into signal: convergence when the record holds still, a change alert when it doesn't. The fix for an unreliable observer was never to cage it. It was to give it a memory. That's the top of the roadmap now, and it's the sharpest thing this whole build taught me: a stateless observer cannot repeat itself, and memory, not a cage, is what makes an adaptive agent reliable. EUREKA The practical habits that would have caught all of this earlier: Smoke-test the model interface the day it's written, not the day it deploys. One real-key call doing one discover and one extract would have surfaced all three prompt bugs ealier, calmly, instead of at 11pm. Write prompt-contract tests that never call the API. Assert the extractor prompt actually contains the schema field names and the valid IDs. Cheap, static, catches underspecification. Record the stub's responses from one real call and freeze them, so the stub can't drift into being more forgiving than reality. Run anything non-deterministic several times and assert the spread is bounded. Variance is invisible to a single run, and a stub can never show it to you. Re-derive tuned parameters when their inputs change. My search budget was set when the registry was smaller and never revisited when it grew. An ode to determinism: the deterministic core had zero production bugs. The guardrails held even while the model layer was failing. Every bug degraded gracefully, structured errors and honest empty states, never a crash, because the fail-safe design was real. And once I added one line of error logging, each mystery became a one-paste diagnosis. The architecture was resilient and the bugs were findable, which is a far better place to be than a fragile system hiding quiet ones. The category I didn't see until I'd built it. Here's the last thing the build taught me, and I was shook. For most of it I thought I was making a SaaS: point it at a court, get a score. Judged as that, the variance really was fatal. Run it twice, get two numbers, look broken. Then I actually looked at what I'd built. A human gate on verification. A draft-inquiry workflow. Findings meant to be reviewed and confirmed, not consumed. Those aren't consumer-SaaS features. They're the features of an analyst's tool. I hadn't built a one-click verdict machine. I'd built readiness intelligence, the kind of thing a vendor runs (oh, the irony), reviews, and tracks over time, and I just missed the point entirely at first. That reframe dissolved the thing I'd been panicking about. A business-intelligence tool is supposed to be run repeatedly. You pull the data, you review it, you watch the trend. Variance between pulls isn't a defect, it's the raw material of a baseline. And the credibility was never in the summary number anyway. It's in the citations: every finding links to a real document and a quoted line you can open and check. The number can move. The evidence doesn't lie. You rarely see the category up front. You build into it, and one day you look up and realize what you actually made. Naming it right, mid-build, under a deadline, turned my biggest liability into the most honest thing about the product. What's next Verity Lex today is one court, scored against one registry version, on a public-record foundation, a live observer that reads the record fresh each time. It's stateless: nothing is stored yet, and it re-hunts from scratch every run. Everything past this point is roadmap, not shipped. Two things sit at the top of it, and both come straight out of the variance. First, ensemble extraction. GPT-5.6 exposes no temperature control, so a single reading of a document varies run to run. Running each extraction several times concurrently and accepting a signal only on a majority vote tightens the score without caging the agent, in parallel, so it costs time you won't feel. Second, and the one the lesson pointed to: memory. v2 would write every scan to an append-only log first, a side effect that can't touch the scan itself, then turn on the read, and that's where it stops being a one-shot scanner and becomes something worth paying for. Scan a court a few times and the baseline strengthens and converges. Store it and you can tell when the record moves, a policy posted, a plan pulled down. The agent stays free to hunt, because hunting a moving target is the point, and memory is what turns its wandering into a baseline you can trust and a change feed you can watch. Above that sit the tiers left as roadmap: structure, people, solutions, deliberately not half-built. But the wedge is real, it's live, and it holds the line it was built to hold: the AI reads the record, and it never assigns the score. *Built for OpenAI Build Week 2026 with Codex, GPT-5.6, and Tavily. AI assisted. Human approved. Powered by NLP. verity-lex.vercel.app*