Reproduce
Run it unchanged
Use the frozen mem0 2.0.12 track, blinded input, five fresh stores, and immutable model assets.
- No protocol changes
- Submit the complete bundle
- Best first external track
Founding 10 · applications open
Don’t endorse us. Reproduce—or break—the contract.
Up to ten independent operators, each with a 14-day window after acceptance. Reproduce unchanged, probe the protocol, or bring your own memory engine and compare A with A + Vault. Positive, negative, and blocked outcomes stay public.
10 operators maxA small founding cohort
14 days eachStarts after acceptance
Outcome-neutralEvidence quality, not praise
OpenApplications accepted now
Public by defaultPlans, deviations, and outcomes
4 result statesValidated · Rejected · Blocked · Protocol issue
No cash promiseAttribution and contributor recognition
Choose one track
Small entry point. Strict evidence.
First-time operators should start with Reproduce. Evaluation and memory-framework developers can choose Break It or Bring Your Memory.
Break It
Find what cannot be trusted
Probe installation, isolation, checksums, claim boundaries, bias, or conditions that prevent reproduction.
- Failure is a valid outcome
- Minimal reproduction or redacted log
- Original record remains after fixes
Bring Your Memory
Compare A with A + Vault
Bring mem0, Letta, AgentMemory, RAG, or your own engine and hold everything else constant.
- Same model, data, and queries
- Disclose every deviation
- No vanity leaderboard
Shared scorecard
Recall is only one cell.
Vault claims to be a memory foundation, so the evaluation must also measure whether memory that should not be used is actually blocked.
Valid Recall@kUseful eligible memory
Forbidden exposureInvalid or unauthorized memory
Expiry leakageExpired memory returned
Citation correctnessEvidence matches its source
Repeat stabilityConsistency across runs
Latency + costTime and resource overhead
Four steps
From application to public record.
Register a track
Open the GitHub form, choose a track and provider, describe the environment, and disclose affiliations.
Confirm the 14-day window
A maintainer confirms the seat and start date in the issue. Registration alone does not make an operator part of the cohort.
Run and retain raw evidence
Run preflight first. Preserve machine-readable artifacts, versions, commit SHA, deviations, and redacted logs.
Submit any honest outcome
Validated, Rejected, Environment blocked, or Protocol issue found can all close the challenge. Missing data never becomes zero.
Complete outcome
You have a bundle
Use the reproduction form so CI can validate digests, attestations, and release gates.
Submit a reproductionBlocked outcome
Report the blocker
Network, runtime, storage, memory, or platform constraints can become diagnostic evidence without being misrepresented as a pass.
Submit a blocked attemptFounding 10 applications are open.
No cash prize is promised in this pilot. The reward is public attribution, a permanent result link, and contributor recognition. Selection depends on reproducibility and evidence quality—not whether the outcome favors Vault.