Same harness, or it didn't happen
Every memory vendor publishes a LoCoMo score. Ours is 74.7%. Mem0 reports 66–68%. Zep reports low 70s. One vendor self-reports 92%.
None of these numbers can be compared, and the reason is not the memory systems — it's everything around them. The answer model, the judge model, the prompt, the retrieval width, the date-handling rules: each moves the score by more than most product differences do. A benchmark number without its harness is a vibe.
So we did the only comparison we'd be willing to defend to an investor or a design partner: we ran a competitor through our harness. Same dataset, same 1,540 questions, same answer model (gemini-2.5-flash), same judge (gemini-2.5-pro, a distinct model), same prompt, same rules. The only thing that differed was the memory layer.
The setup
Mem0 is the most-cited open-source memory layer, so it went first. We ran the shipped OSS library (v2.0.20) locally, configured to the sameGemini models Smara uses internally — their extraction pipeline on flash, the same 768-dim embeddings, an embedded vector store. Smara's side is not a lab build: it's the production service at api.smara.dev, the same deployment our beta users hit.
Fairness required actual work, and we're publishing all of it:
- We fixed their config for them.Out of the box, mem0's Gemini provider defaults to
max_tokens=2000— which gemini-2.5-flash's internal reasoning consumes before the extraction JSON finishes. Result: truncated JSON, failed parses, and ~80% of sessions silently producing zero memories. If we'd run that config, mem0 would have scored catastrophically and the comparison would have been garbage. We raised the budget through their own config surface and re-ingested everything from scratch. - Dates went in-band, matching Mem0's own published LoCoMo methodology (“This conversation happened on 8 May 2023” preceding each session). Their
timestampAPI parameter turns out to be platform-only — the OSS SDK rejects it. - Their hosted platform is out of scope.This measures the OSS library people actually self-host. The platform may do better; we'd happily re-run it under the same harness.
One finding survived every config fix, because it's a design property, not a bug: mem0's extraction resolves relative time against the wall clock. Told “I adopted a puppy last week” in a conversation dated May 2023, it stored the adoption as happening the week before today. Memory systems that don't anchor time to when things were said don't just lose points on temporal questions — they misremember your life politely and confidently.
The numbers
| Overall | Single-hop | Temporal | Multi-hop | Open-domain | |
|---|---|---|---|---|---|
| Smara (production, official6) | 74.7% | 85.1% | 70.7% | 58.2% | 44.8% |
| Mem0 OSS (same harness) | 63.1% | 71.6% | 59.8% | 49.3% | 40.6% |
Two things worth saying plainly:
63.1% is a respectable score.It lands inside the range Mem0 self-reports, which tells you our harness isn't rigged against them — and that their published numbers were honest under their own harness. That's the point: two honest teams, two honest harnesses, numbers that can't be compared until someone runs both systems under one of them.
The temporal gap is the architecture talking. Smara ingests conversations as dated episodic eventsand consolidates them into facts with validity windows; “last week” is resolved against the session date at extraction time, and a nightly process rewrites stale tense (“is going to Singapore” → “went in July 2026”) with every edit auditable. That design choice is worth double-digit points on temporal questions — and it's the difference between a memory and a notebook.
The confession
Honest benchmarking cuts both ways, so here's ours.
While preparing this run we discovered that our own nightly consolidation — Dreaming — had, weeks earlier, run tenant-wide across the benchmark tenant and merged near-duplicate facts between different synthetic users. Independent test users' memories, cross-pollinated. In a real multi-user deployment this class of bug is a privacy incident; in a benchmark corpus it's contamination.
Two properties turned it from a disaster into an afternoon: every dream edit is recorded as an auditable supersession, so we could enumerate exactly which 148 merged facts existed and which per-user originals they had closed — and reverse it, restoring the corpus to byte-level correctness. Then we shipped a guard so benchmark tenants are excluded from tenant-wide dreaming entirely. The audit trail we sell as a feature is the reason we could trust our own benchmark again.
What we'd love to see
We'll publish our harness invocation for this run like every other. If you're a memory vendor and think your system deserves better than what our harness shows: run it, send us the diff, or better — run usthrough yours and publish everything. Same harness, or it didn't happen.