Flagship case
The flagship case
One public question stands for the whole claim. It needs identity held steady across documents, facts scoped to the right time, support for every step between question and answer, and a safety-relevant conflict surfaced instead of smoothed over. No single source states the answer.
In validation
Flagship question
“On the evening of the Glass Room fire, who chaired the Kestrel Board, and where was that person's niece's workshop operating that year? And is there anything on record that would make that answer unsafe to give?”
The answer has to be assembled across separate documents while preserving identity, time, relationship direction, and a safety-relevant conflict. No source states the complete answer.
We will mark this solved only when the released evaluation artifact produces the answer consistently, carries machine-checkable proof for every required step, and surfaces the record that could make the answer unsafe to give.
What makes this question difficult
A useful answer has to satisfy all four conditions. The released case will require machine-checkable proof for every step.
Identity
Keep people, places, and objects distinct when names overlap.
Time
Use the facts that apply to the date in the question.
Connection
Retain support for every step between the question and answer.
Conflict
Show disagreement instead of quietly picking a convenient record.
How today's stacks fare
Each approach below is genuinely good at something. The case does not ask whether a stack can get close to the answer; it asks whether the whole pipeline establishes every hop, respects the time scope, and surfaces the record that could make the answer unsafe to give.
Swipe to see the full table
| Approach | What it does well | What this case still requires |
|---|---|---|
| RAG (retrieval + LLM) | Retrieves relevant passages and generates a composed, fluent answer. | Requires an LLM for the generation stage. The retrieval stage selects by relatedness, not by logical role in a proof chain. The generation stage does not guarantee that every hop is individually supported or that a safety-relevant conflict was surfaced. |
| Knowledge graph or graph database | Traverses typed relationships and resolves multi-hop paths natively. | Requires someone to extract entities and relationships from the source documents first. Does not itself resolve prose into typed structure or track provenance back to the original text. |
| Dedicated AI memory layer | Maintains structured profiles and conversation history across sessions. | Typically depends on an LLM for summarization and retrieval composition. Profile summaries compress away the source evidence needed to verify each hop. Multi-hop composition and document-level provenance are not part of the retrieval contract. |
| Vector or retrieval memory | Finds passages whose embeddings are close to the question. | Embedding proximity does not establish that the chair identified is the one who held office on that evening, or that the niece relationship and workshop location are connected by identity rather than keyword overlap. |
| SQL or relational database | Answers precise questions over a schema someone has already designed. | The source material arrives as prose documents. The identities, temporal scopes, and relationship joins this case needs are not supplied as tables or keys. |
| LLM alone | Synthesizes a fluent answer from whatever context it is given. | The entire pipeline depends on the LLM. Fluency does not establish that every hop is supported, that the temporal scope is correct, or that a conflicting safety record was checked rather than overlooked. |
An open challenge
Run this question on the stack you have today: retrieval with a language model, a knowledge graph, a dedicated memory layer, or plain search. Watch where it satisfies the four demands and where it guesses. When the public acceptance evidence for CleverMemory lands on this page, you will have your own baseline to compare it against.
See how public claims are evaluated → Read the dated status record →