Evidence¶
Every claim in these docs comes from an experiment below. Each result page names the
script that regenerates its table and the run directory under
results/ that holds the committed summary and per-question
scores. Intervals are 95% bootstraps over questions.
Read the numbers as directions, not guarantees. Samples are 30–300 questions per setting, most settings ran with one seed, and the decision model (Jev) is a hosted model whose versions can change.
Summary¶
| Question | Answer | Where |
|---|---|---|
Does walk make an agent cheaper? |
Yes, with an expensive agent: −17% cost (95% CI 8–27%) at equal F1, strong agent, n=100. With a cheap agent, the cost was a wash. | A8, A8b, P4 |
Does walk make an agent more accurate? |
No. 0.72 vs 0.74 F1 (cheap agent, n=100); 0.76 vs 0.75 (strong agent, n=100); neither CI excludes 0. | A8, A8b |
| How much is the model's memory? | A lot: with entity names replaced by aliases, closed book falls from 0.42 to 0.01 F1 and every arm loses 0.15–0.48. Text search loses most: graph-tool agents then beat it by 0.25–0.32 F1. | P3 |
| Is the walk's confidence informative? | Yes: AUROC 0.92–0.97 on WebQSP and MetaQA 2–3 hop; weaker (0.64–0.71) on MetaQA 1-hop, 2Wiki, and CWQ. It comes from reading the decider's probabilities, not from Jev: an open-weights model's token probabilities did as well (0.77–0.97 except MetaQA 1-hop). An LLM that states its own scores (the shipped fallback) barely is (0.50–0.69). | E3, A2, P1 |
| Can we predict before walking whether it will work? | Barely, within a graph (AUROC 0.55–0.68). Walk first, read the confidence. | A7 |
| Walk vs an LLM writing the query, curated graphs | The LLM is more accurate: MetaQA 3-hop 0.955 vs 0.887; Freebase (5,419 relations) 0.66 vs 0.49. Walking is ~6× cheaper and ~4× faster. | E2, A6 |
| Walk vs multi-step RAG, curated graph (MetaQA) | Walking wins: 0.97/0.99/0.89 vs 0.89/0.81/0.41 F1 (1/2/3 hop) at 3–5× lower cost. | M5 |
| Escalating unsure walks | Works, modest savings: the LLM decider's accuracy at about half its cost on MetaQA; ~19% savings on Freebase. | A1, A6 |
| Constraint questions (CWQ) | Every system tried: hits@1 ≈ 0.28, far below published methods (0.63–0.69). | A2 |
| QA over text (graph built from documents) | Multi-step RAG wins by 10–19 F1, even with a 20× pricier extractor. | M7, E4, E5 |
| The graph as a retriever over text | Hybrid beats dense by 18 recall on entity chains (2Wiki); ties or loses elsewhere. | E1 |
| Scale | SQLite, 1.4M edges: walks add ~8 ms per walk on typical nodes, 0.1–0.2 s from a 20k-neighbor hub. | A5 |
Reproducing¶
Each table has one command. The paper tables read the committed scores and need no API key:
uv sync --extra eval
uv run python scripts/paper/table_curated.py # E2/E3/E4
uv run python scripts/paper/table_escalation.py # A1
uv run python scripts/paper/figure_calibration.py # E3 calibration
Rerunning an experiment from scratch needs OPENROUTER_API_KEY and the extras its
script names in its docstring, for example:
uv sync --extra eval --extra llm --extra embeddings
uv run python scripts/eval/agent_arms.py --n 100 --arms graph walk search
Datasets are not redistributed: the loaders download MetaQA, WebQSP and CWQ (the RoG subgraphs), 2WikiMultiHopQA, HotpotQA, and FanOutQA from their public sources and cache them locally.
Pages¶
- Phase 7: escalation (A1), Freebase KG-QA (A2, A6), scale (A5), routing (A7), graphwalk as an agent's tool (A8).
- Phase 4: retrieval (E1), LLM-written queries (E2), Jev vs an LLM decider (E3), seeds (E4), extractors (E5), cost (E6), RAG add-on (E7).
- M7: QA over graphs built from text.
- M6: ingestion quality.
- M5: curated graphs vs RAG.