Benchmarks

One-third LoCoMo · J-score

Direct source/RAG result

The direct actor-turn source/RAG backend answered 463 of 513 questions correctly on the frozen sample.

Upper Wilson 95% bound92.53%Measured: 90.25% · 463 / 513 correct

The frozen-sample accuracy is 90.25%. The displayed figure is the high end of its descriptive Wilson interval, 87.38%–92.53%.

Accuracy by LoCoMo category
Category	Correct	Total	Accuracy	Wilson 95% CI
Multi-hop	81	94	86.17%	77.76%–91.74%
Temporal	96	107	89.72%	82.52%–94.16%
Open-domain	22	32	68.75%	51.43%–82.05%
Single-hop	264	280	94.29%	90.92%–96.45%
Overall	463	513	90.25%	87.38%–92.53%

Validity

Protocol

The dataset, question set, prompts, retrieval lane, and scoring procedure were frozen before answers were generated.

Dataset: Official LoCoMo10 release from the LoCoMo repository. SHA-256: 79fa87e90f04081343b8c8debecb80a9a6842b76a7aa537dc9fdf651ea698ff4.
Scope: Categories 1–4. Category 5 was excluded, matching Mem0's primary J-score protocol.
Sample: Deterministic SHA-256 ordering within each category using seed locomo-jscore-third-v1, taking floor(N/3) per category. The sample contains 94 multi-hop, 107 temporal, 32 open-domain, and 280 single-hop questions. Selected-QID-list SHA-256: d4eaf4ab10167f76e2ac682aa0267db56c5256eba29ce6f8c2b220073c0a2df1.
History ingestion: All 5,882 released actor turns across all 272 sessions and ten isolated tenants were inserted into fresh logical storage. Sampling questions did not remove source turns needed by temporal or multi-hop questions.
Backend lane: Direct raw actor-turn episodes in the immutable Postgres source ledger. Graphiti extraction and traversal were deliberately bypassed, making this a source retrieval/RAG score rather than an end-to-end graph-extraction score.
Retrieval: Local BAAI/bge-small-en-v1.5 dense retrieval plus answerdotai/answerai-colbert-small-v1 reranking; a 200-passage rerank pool; top-50 returned passages; and a 7,000-token hard ceiling. Mean packed context was 2,669 tokens. Of 513 questions, 512 received 49–50 items. One query was rejected by the production false-premise guard and remained in the denominator.
Prompts: Exact snapshot of Mem0 memory-benchmarks commit 4b61c5d31b9c668a12b4f5e78064248a02c82d2b. Prompt-file SHA-256: 8ebac1ef60e9ab5caf99079fdaac038b85472e81491ed35e2d2655f3927c76c2.
Answerer and judge: gpt-5.6-solat low reasoning effort, with one isolated call per question. The judge applied Mem0's binary CORRECT/WRONG J-score rules, including category-3 gold preprocessing, and received neither retrieved context nor annotated evidence.

Audit trail

Blindness and integrity

The answer worker read only blind.jsonl. Each record contained a QID, question, reference date, and retrieved search results—never an answer, category, or evidence field. The judge ran only after every prediction was sealed and saw only the question, preprocessed gold answer, and generated answer.

Both phases produced 513 unique outputs, with no empty answers and zero recorded model tool calls. No prior run directory was consulted.

Important artifact hashes
Artifact	SHA-256
Sample manifest	`d3d0af134aae113a3a982f8cdd3ab0b6bd3b9ca3b825c3ec4c1f40bb6a56ec23`
Retrieval records	`bf7cb12f6987fd0b81c75b7b5af918378ffefb981f3a2e1180ebcf010a3bf5b6`
Blind inputs	`9d16c7f75fe82852999737285018f8f7e55e4be0047180c44a9ab77c84dee6e9`
Predictions	`9733c16cb8e491e2f17e4de4cdab5758a42a3af986b6dfdb441fe0c08cb37fd4`
Judged results	`d0ddf563ba07317e337085c4d9f5e9b1d3023a379f83a9c1777af282a3831d1c`

Ignored run artifacts: testdata/locomo/runs/jscore-third-20260714-c463b5c3/

Implementation

Reproducibility additions

testdata/locomo/prepare_jscore_sample.pyFreezes the stratified sample.
testdata/locomo/run_locomo.pyAccepts an immutable --qid-file and emits Mem0-compatible blind search-result records with reference dates.
testdata/locomo/mem0_prompts_4b61c5d.pyPins the exact upstream prompt file.
testdata/locomo/jscore_with_codex.pyProvides isolated answer and judge phases, strict structured outputs, checkpoint hashes, Wilson intervals, and a no-tool-use audit.
tests/test_locomo_benchmark.pyCovers sampling, duplicate-QID rejection, temporal reference dates, top-50 chronological prompt assembly, and Wilson interval behavior.

Reading the result

Interpretation and limitations

This result uses the same binary LLM-judge metric and current prompt semantics as Mem0, so it is comparable at the metric level. It is not an exact vendor-runtime reproduction: no external OpenAI API key was available, so the run used the available Codex-hosted gpt-5.6-sol model.

Codex CLI has no separate system-message flag. The exact Mem0 judge system text was therefore placed immediately before the exact judge user prompt in each isolated request.

The 513-question sample is one third of the 1,540 scored category-1–4 questions, not the full benchmark. Category-level intervals are consequently wider, especially open-domain with only 32 questions.

Finally, this score evaluates the direct source/RAG lane. It does not measure Graphiti extraction quality, contradiction resolution, or graph traversal.