We Almost Deployed a Temporal Knowledge Graph. The Eval Said No.

The eval that killed the temporal knowledge graph asserted one thing: at time T, the agent should report the state that was true at T. It failed 41% of the time. The graph had the right facts. It just handed the agent the wrong one. That number is what saved us from shipping. Every static retrieval metric looked fine. The graph answered "what is the status of Node A" with a confident, well-formed response. Trouble is, "what is the status" is a temporal question wearing a static question's clothes, and nothing in our test suite had noticed the difference until we wrote a test that actually asked about time. What I expected The pitch for a temporal knowledge graph (TKG) is genuinely good. You store facts as quadruples instead of triples: (subject, predicate, object, timestamp) or, better, (subject, predicate, object, valid_from, valid_to). Now your agent memory isn't a flat pile of embeddings, it's a structured record of what was true and when. This is the natural next step past pure vector recall, and it slots neatly into the decay-based thinking I've written about before in Eviction Without Deletion. Instead of letting old facts fade by activation weight, you make validity windows explicit. My hope was that the graph would fix the exact failure mode that plagues flat vector memory: the agent confidently recalling a stale fact because it's semantically close to the query. With valid_from and valid_to on every edge, staleness becomes a filter, not a guess. Ask for the state at time T, filter edges where T falls inside the window, done. On paper it's cleaner than a decay curve because there's no fuzziness. A fact is either valid at T or it isn't. Schema-wise, it was simple enough. In a property graph it looks like this: // A temporal fact: Node A was in maintenance for a fixed window MATCH (n:Server {name: 'node-a'}) CREATE (n)-[:HAS_STATE { status: 'maintenance', valid_from: datetime('2026-07-20T02:00:00Z'), valid_to: datetime('2026-07-20T04:30:00Z') }]->(:State {kind: 'maintenance'}) Enter fullscreen mode Exit fullscreen mode Multiple HAS_STATE edges per server, each with its own window. Query the graph, get the state for any point in time. This is the "structured shared memory" pattern I described in Multi-Agent AI Systems, except now the shared memory understands time. The whole thing felt like an upgrade in every dimension. It reads well in a design doc. It demos beautifully. That was part of the problem. What actually happened Retrieval is where it fell apart, and the failure is boring in a way that makes it dangerous. Here's roughly the query the agent's tool was generating: // The "obvious" query - find the status of a server MATCH (n:Server {name: $server})-[r:HAS_STATE]->(s:State) RETURN s.status, r.valid_from, r.valid_to ORDER BY r.valid_from DESC LIMIT 1 Enter fullscreen mode Exit fullscreen mode Read that carefully. It orders by valid_from descending and takes the most recent fact. Most of the time that's correct, because the most recently started state is usually the current one. What it never does is filter by the query's reference time. If the agent is reasoning about an incident that happened at 03:00, and a newer "online" state was recorded at 05:00, this query returns "online." The graph knew Node A was in maintenance at 03:00. The retrieval logic threw that knowledge away. This is the hallucinated-history problem, and it's insidious because the model isn't hallucinating. The fact is real. The timestamp is real. The agent is just being handed a fact from the wrong window and has no way to know it. Worse, the answer is fluent and specific, so every static evaluation gives it a pass. RAGAS-style faithfulness checks look at whether the answer is grounded in the retrieved context. It was. The retrieved context was simply the wrong slice of time. I want to be precise about where the failure lived, because it wasn't the graph. The graph was correct. The schema was correct. The data was correct. The failure was split across two places: a retrieval query that dropped the temporal filter, and an evaluation suite that had no test capable of noticing. If we'd only had the first problem, we'd have caught it in review. Having both meant the system looked healthy right up until the one test that mattered. That missing test is short: import pytest from agent_memory import query_state # our TKG retrieval tool def test_temporal_regression(): """Agent must report the state valid AT the reference time, not the most recently recorded state.""" # Maintenance window: 02:00-04:30. Online recorded at 05:00. result = query_state(server="node-a", at="2026-07-20T03:00:00Z") # A later 'online' fact exists, but at 03:00 the truth is 'maintenance' assert result["status"] == "maintenance", ( f"temporal regression: got {result['status']} " f"for a timestamp inside the maintenance window" ) Enter fullscreen mode Exit fullscreen mode Run that single assertion across a few dozen historical state transitions and you get the 41% failure rate. Not a subtle degradation. Nearly half of all time-scoped questions returned a state from the wrong window whenever a newer fact existed. Meanwhile the static suite, which only checked "does the agent know the current status," passed everything. Two evals looking at the same system, one green, one red, and only the red one described reality. The fix Correcting the retrieval was one clause. You filter edges so the reference time falls inside the validity window before you order or limit anything: // Filter by the reference time FIRST, then pick the winner MATCH (n:Server {name: $server})-[r:HAS_STATE]->(s:State) WHERE r.valid_from datetime($at)) RETURN s.status, r.valid_from, r.valid_to ORDER BY r.valid_from DESC LIMIT 1 Enter fullscreen mode Exit fullscreen mode valid_to IS NULL handles the open-ended "current" state, the fact that has started but not yet ended. Everything else is a closed window, and the WHERE clause guarantees you only ever consider edges whose window contains T. The ORDER BY ... LIMIT 1 is still there to break ties if two windows overlap, but now it's picking among facts that are all actually valid at T, not among every fact ever recorded. One clause. That's the entire retrieval fix. Which tells you the real bug was never in the query, it was in the fact that nobody wrote the eval that would have made the missing clause obvious on day one. So the second half of the fix was the more important one: the retrieval tool never gets to reason about time on its own. The agent isn't trusted to remember to pass a reference timestamp, and the LLM isn't trusted to filter windows in its head. Instead, the current time is injected as explicit context at the tool boundary, and the tool refuses to answer a state question without it: def query_state(server: str, at: str | None = None) -> dict: if at is None: raise ValueError( "query_state requires a reference time. " "Temporal facts are meaningless without one." ) # ... run the time-filtered Cypher above ... Enter fullscreen mode Exit fullscreen mode Making the reference time a required argument sounds trivial. It's the difference between a tool that silently returns plausible garbage and one that fails loudly when it's used wrong. A loud failure is a bug report. A plausible answer from the wrong window is an incident three weeks later that nobody can reproduce. We also changed how the retrieved fact is handed to the model. Rather than passing "status: maintenance" as a bare string, the prompt gets the window with it: "As of 2026-07-20T03:00:00Z, node-a status is 'maintenance' (valid 02:00-04:30). A later 'online' state exists from 05:00 and is not applicable to this query." Giving the model the window and the reference time in the same breath means that even if the retrieval ever regresses, the model has a fighting chance to notice the mismatch. Defense in depth, applied to a knowledge base. Why this matters I keep coming back to the fact that we did not ship this because of the architecture. We almost shipped it because the architecture was correct and only the evaluation was wrong. That inversion is the whole lesson. A TKG is more capable than flat vector memory, and that extra capability comes with an entirely new class of failure that your existing evals were never designed to see. Adding temporal structure to your memory adds temporal bugs, and static tests are blind to all of them by construction. Here's the trap I see constantly. Teams treat GraphRAG as a strictly-better upgrade over vector RAG, port their old eval suite unchanged, watch it stay green, and conclude the new system is at least as good as the old one. Their green suite is measuring the properties the old system could fail on. It has no assertion about sequence, no assertion about validity windows, no assertion that state at T equals the state that was actually true at T. The new failure mode is invisible not because it's rare but because nothing is looking for it. This is the same gap I described in Cognitive Memory for Agents: the retrieval method changed, so the questions your evals ask have to change too. If you're building temporal memory for an agent, write the temporal regression test before you write the graph. Seed a handful of known state transitions where a later fact contradicts an earlier one, then assert that a query scoped to the earlier window returns the earlier fact. That test is a dozen lines. It will fail the moment your retrieval forgets to filter by time, which, based on how naturally that ORDER BY valid_from DESC LIMIT 1 query wrote itself, is going to be your very first implementation. A few things I'd carry into the next attempt. Make the reference time a required parameter on every temporal query, so the tool cannot be called ambiguously. Keep the graph correct and put your paranoia in the retrieval and the eval, because that's where the wrong-window bug actually lives. Expose temporal queries to the agent through a narrow, well-typed interface, whether that's a tool boundary or an MCP server, so the agent can't hand-roll a query that drops the filter. And treat "it passed the static eval" as necessary, never sufficient, the second time itself becomes a dimension of your data. If you're standing up this kind of time-aware memory for a production agent system and want a second set of eyes on the failure modes, that's a chunk of what I do consulting. We didn't deploy the temporal knowledge graph that quarter. We deployed the eval, fixed the one-clause retrieval bug it exposed, and shipped the graph once the red test went green. The graph was never the risky part. The risky part was almost trusting a system that no test had ever asked the right question.