Four LoCoMo runs, three lessons

We benchmark Smara on LoCoMo — ten very long multi-session conversations, 1,540 scored questions — against production api.smara.dev, through the same API every agent uses, with a distinct judge model. No lab rig, no special retrieval path. We have now done four full runs. The numbers went 68.4, then 74.2, then 74.1 — and the fourth number we chose not to publish as a headline at all. Each run taught us something the previous one couldn't.

Lesson one: your first benchmark run is a bug report. The 68.4 run exposed a real retrieval defect — multi-hop questions were failing because a graph traversal bug quietly starved the context of connective facts. We would not have found it by reading code. Fixing it and re-running honestly was worth more than the six points: it set the policy that the benchmark harness lives in the repo, reproducible, so any number we publish can be checked.

Lesson two: publish the latest run, even when it goes down. Run three came back at 74.1 — a tenth of a point below the 74.2 we had already published, after a change (entity-fused retrieval) we expected to help. The tempting move is to keep the better, staler number. We republished 74.1. A benchmark figure is a claim about the product as it exists, not a trophy about the product as it once was — and the flat result itself was information: the feature helps entity-centric questions in ways this particular benchmark barely exercises.

Lesson three: some results should change the product, not the website. Run four measured our community summaries — cluster-level overviews that give broad questions a place to land. Answer quality on broad questions improved. The aggregate score dropped, because summaries were crowding specific facts out of the token budget on pointed questions. That's a displacement effect, and it taught us the real requirement: a serving gate that decides when a summary belongs in context, not whether summaries are good. Until that gate ships, the number stays in our lab notes and the product keeps the previous serving behavior. Publishing it as a headline would have punished us for measuring honestly — and not measuring was never on the table.

The wider context here is that memory benchmarking is having a credibility moment. The same vendor system has scored 92.5 (self-reported) and 49.0 (an independent lab, March 2026) on this benchmark — a 43-point spread that comes from the harness, not the product. We publish ours with the full methodology, per-question records, and the harness itself, and we state plainly that comparisons across setups are unreliable. If a design partner reruns our harness and gets a different number, that is the system working.

Current published figure: 74.1% overall by LLM-judge (85.5% single-hop, 70.7% temporal, 55.7% multi-hop — multi-hop aggregation remains the stated improvement target). When it changes, in either direction, the site changes the same day.