AI Productivity Test
How much does context actually move the number?
A reproducible benchmark that adds Role, Context and Skill to the same model on the same tasks — and lets the official Terminal-Bench verifier grade the result. Memxus does not grade its own work.
Results
-5.0 pts
Role Lift
+5.0 pts
Context Lift
-15.0 pts
Skill Lift
-5.0 pts
Combined Lift
-11%
Productivity Gain
★ Optimal config among active conditions — best result per dollar.
View chart data as a table
| Condition | Solved (%) | Tokens | Cost (USD) | Time (s) | Tool calls |
|---|---|---|---|---|---|
| Baseline | 45% | 10889 | $0.05 | 68 | 4.3 |
| + Role | 40% | 16254 | $0.07 | 57 | 5.4 |
| + Context | 50% | 19494 | $0.08 | 58 | 5.3 |
| + Context + Skill | 35% | 18061 | $0.08 | 56 | 4.6 |
| + Role + Context + Skill | 40% | 24986 | $0.10 | 59 | 5.8 |
The clearest single example
Task aimo-airline-departures — the largest gap we observed between baseline and context in a single task, verified by the official Terminal-Bench judge.
Baseline
- Result
- Timeout
- Tool calls
- 0
- Time
- 32.3s
+ Context
- Result
- Solved
- Tool calls
- 1
- Time
- 28s
Follow-up experiment: does a cleaner prompt beat "Full"?
Hypothesis: context_clean success_rate >= 50% (original Context) and avg time_seconds lower than context_skill/full
Outcome: REJECTED — context_clean scored lower than every other condition, including baseline
Follow-up experiment, NOT part of the frozen-scaffold 5-condition comparison above: context_clean adds an explicit "do not follow any checklist" instruction, which is a scaffold change versus baseline/role/context/context_skill/full. Hypothesis tested: removing the generic Skill checklist and keeping only Context would raise the success rate toward the original Context-only result (50%) and reduce time. Result: it did NOT — context_clean scored 30%, the lowest of all six conditions, including baseline.
View exact numbers as a table
| Condition | Success rate | Avg tool calls | Avg tokens | Avg time |
|---|---|---|---|---|
| Baseline | 45% | 4.3 | 10,889 | 68s |
| + Role | 40% | 5.4 | 16,254 | 57s |
| + Context | 50% | 5.3 | 19,494 | 58s |
| + Context + Skill | 35% | 4.6 | 18,061 | 56s |
| + Role + Context + Skill (Full) | 40% | 5.8 | 24,986 | 59s |
| Context-clean (follow-up) | 30% | 4 | 13,747 | 56s |
Pipeline validation (not a result)
Pipeline validation only — not the AI Productivity Test. Models are free-tier NVIDIA NIM endpoints, not claude-sonnet-5. Role/Context/Skill are generic placeholder text, not fetched from Memxus via MCP. 5 tasks, 1 rep per model — below the 20-task minimum for a reportable benchmark. Proves the harness (real Docker + official Terminal-Bench verifier) works end-to-end across multiple models; it is not evidence for or against the context-lift hypothesis.
run: 2026-07-17 · 3 models · 5 tasks each
meta/llama-3.1-8b-instruct
| Condition | Solved | Avg tool calls | Avg tokens | Avg time |
|---|---|---|---|---|
| Baseline | 0/5 | 11 | 11,162 | 60s |
| + Role | 0/5 | 10.6 | 10,906 | 33s |
| + Context | 0/5 | 11.8 | 11,947 | 39s |
| + Context + Skill | 0/5 | 9.4 | 9,262 | 25s |
| + Role + Context + Skill | 0/5 | 8.6 | 6,917 | 24s |
mistralai/mistral-nemotron
| Condition | Solved | Avg tool calls | Avg tokens | Avg time |
|---|---|---|---|---|
| Baseline | 1/5 | 7.8 | 6,283 | 87s |
| + Role | 1/5 | 9.6 | 8,332 | 107s |
| + Context | 1/5 | 7 | 7,154 | 109s |
| + Context + Skill | 0/5 | 9 | 10,463 | 89s |
| + Role + Context + Skill | 0/5 | 11.2 | 11,816 | 116s |
nvidia/llama-3.1-nemotron-nano-vl-8b-v1
| Condition | Solved | Avg tool calls | Avg tokens | Avg time |
|---|---|---|---|---|
| Baseline | 0/5 | 7.8 | 11,959 | 113s |
| + Role | 0/5 | 9.8 | 13,046 | 165s |
| + Context | 0/5 | 7.4 | 9,717 | 111s |
| + Context + Skill | 0/5 | 6 | 11,279 | 86s |
| + Role + Context + Skill | 0/5 | 8.6 | 14,511 | 104s |
What the test measures
One question: how much does an AI improve when you add Role, Context and Skill, holding everything else constant? The model, agent, budget and environment never change. The only thing that changes is the information we add.
- Role — Who the agent is and how it should behave — priorities and judgment (e.g. “senior engineer, prioritize safety and minimal changes”).
- Context — Project-specific knowledge: architecture decisions, conventions, error history, documentation. This is what Memxus delivers.
- Skill — How to execute a concrete task: steps, checklists, strategy. Also delivered by Memxus, over MCP.
The five conditions
Each task runs under five conditions. Each one adds a layer on top of the previous, so the score difference between two conditions isolates that layer’s contribution.
| Condition | What it adds | Measures |
|---|---|---|
| Baseline | Nothing — just the model and the task. | Starting point |
| + Role | The agent’s role. | Role Lift |
| + Context | Project context (Memxus). | Context Lift |
| + Context + Skill | Context plus an execution skill (Memxus). | Skill Lift |
| + Role + Context + Skill | All three layers together. | Combined Lift |
What stays constant
Everything is held fixed except the layer under study. This is the central methodological rule of the test.
| Model | claude-sonnet-5 |
| Benchmark | Terminal-Bench (official tasks & verifier, unmodified) |
| Tasks | The same 20–30 across every condition |
| Repetitions | 1 per condition in this run (spec target: 3, averaged — see limitations) |
| Temperature | Fixed and identical |
| Timeout | 600 s per task |
| Token cap | 30,000 per task |
| Scaffold & tools | Frozen — byte-identical across all 5 conditions |
Benchmark scores are sensitive to the agent scaffold — the same model can swing more than 10 points depending on how it is run. So the scaffold is frozen and identical across the five conditions. Any difference we observe is attributable to the layer we added, not to the harness.
Who decides pass or fail
The pass/fail verdict comes from the benchmark’s official verifier, not a subjective judge. That is what makes the numbers defensible: Memxus does not evaluate its own result.
Limitations
- Single repetition. This run used 1 rep per condition (the spec target is 3, averaged) to keep cost proportionate to a first validation pass — confidence intervals are wider than a 3-rep design would give, and single-trial noise (e.g. one unlucky parse failure) can swing a condition’s score more than the underlying capability difference.
- Small sample. 20 tasks illustrate the method; they are not a definitive statistical proof.
- Single model. The effect is demonstrated on Sonnet 5. That it generalizes to other models is a hypothesis for later phases.
- Single benchmark. Terminal-Bench measures command-line tasks. SWE-bench and other domains come later.
- Scaffold-dependent. The numbers are not comparable to other leaderboards that use a different scaffold.
What this test does NOT claim: it does not say Memxus improves productivity by some universal percentage. It says, with a reproducible method, how much it improved on these tasks, with this model, under these conditions. That number is Memxus’s own and verifiable.