Context compression can introduce faithfulness issues through an information bottleneck, and QA metrics miss them. KEEP-Bench checks the compressed context itself, on 4,102 cases that require keeping plausible reference resolutions, explicit qualifiers and minority perspectives.
Context compression improves long-context efficiency but may introduce faithfulness issues through an information bottleneck. Existing evaluations rely on QA metrics, but a reader can still answer correctly from an unfaithful compression, so these metrics underestimate such loss. To bridge this gap, we introduce KEEP-Bench, a diagnostic benchmark that directly assesses compression faithfulness with 4,102 cases across three challenging settings that require keeping evidence for plausible reference resolutions, explicit qualifiers, and minority perspectives.
Experiments show that even with tailored prompting, the strongest models remain unreliable: despite context windows exceeding 100K tokens and inputs averaging only 3.1K tokens, the best open-weight model (31B) preserves all required evidence in only 66.8% of cases, and the best proprietary model in only 61.6%. Performance also varies sharply across evidence structures, so conclusions from limited structures may be biased.
We further introduce a metric, Evidence Preservation Return (EPR), to relate evidence recall to inference FLOPs, and find that it peaks at moderate scales (4B–8B), where recall remains low. Additional output budget mainly benefits already strong models and widens capability gaps, yet the stronger preservation of these models comes at a steep computational cost.
These findings highlight the importance for future work to investigate compressors with strong information preservation under a small computation budget, rather than relying on a large output budget or model scaling.
Pilot study
On 200 parallel cases, a reader answers correctly in 96.0% of cases from faithful compressions but in only 11.5% from unfaithful ones.
Those 23 cases pass QA even though required evidence is gone, so KEEP-Bench checks the compressed context itself.
A case is correct only if the reader matches both gold answers.
Benchmark
Underspecified mentions admit two valid interpretations, each with its own reasoning chain, so a faithful compression must keep both.
One step holds only under an explicit qualifier. Dropping the qualifier turns a conditional claim into an unconditional one.
Dominant perspectives fill whole paragraphs; each minority perspective gets a single sentence yet is equally required.
Main results
The best model keeps every proposition in only 66.8% of cases, and the leader changes with the structure: Claude-Haiku-4.5 is first on parallel evidence but trails GPT-5.4-mini on science.
| Parallel | Conditional | Imbalanced | ||||||
|---|---|---|---|---|---|---|---|---|
Default budget for all runs; RECOMP (abstractive) has no length control. Values are All-Rec unless marked, Overall spans all 4,102 cases, and a green underline marks the best in each column. EPR (Evidence Preservation Return) is 6·√(A/F), with A the overall All-Rec in percent and F the average TFLOPs per case; API models have no FLOP estimate.
Compute
Larger models keep more evidence, but compute grows faster than recall, so EPR peaks at moderate sizes. The cheapest method, RECOMP (extractive), keeps all evidence in under 5% of cases.
Each point is one method. Points on a dashed curve share an EPR; up and to the left is better.
Output budget
The weakest models stay low at every budget, so extra budget widens the gap instead of closing it.
All-Rec at 0.5×, 1× and 2× the default budget; imbalanced pools science and policy. Click a name to toggle its line. Dashed lines are dedicated compressors.
Evidence position
We place the two evidence blocks of the 400 two-hop parallel cases side by side, about half the context apart, or at opposite ends, and compress each version with three models.
From Adjacent to Middle, all three models lose 2.1 to 8.5 points of Para‑Rec, and Gemma-4-E4B-it recovers at Distant. Both charts average over absolute placement and block order.
Recall of each chain's first proposition when its block comes earlier (filled) or later (open). Earlier wins by 2.5 to 10.6 points; the one exception is Gemma-4-E4B-it at Distant.
Case study
Only Claude-Haiku-4.5 keeps all four propositions. Ten of the other sixteen keep both counties, so the answers survive, but lose the statements that tie each series to its state.
| Compressor | p1 | p2 | p3 | p4 | All-Rec |
|---|
Resources
@article{man2026keepbench,
title = {{KEEP-Bench}: Evaluating Evidence Preservation in Context Compression},
author = {Man, Ziqing and Wu, Di and Gu, Jia-Chen and Zhang, Junyi and Chang, Kai-Wei and Peng, Nanyun},
year = {2026}
}