01
Overview
Synthetic math data can scale model training, but a plausible-looking sample may still contain a broken premise, a missing condition, or a subtle reasoning error.
MathDebugger turns this data-quality problem into four controlled evaluation tasks, spanning both detection and fine-grained diagnosis.
- Question verification. Decide whether a mathematical problem is internally valid.
- Question diagnosis. Identify contradictions, missing conditions, unrealistic premises, or expression errors.
- Answer verification. Check the entire solution trace rather than only the final answer.
- Answer diagnosis. Distinguish logic, computation, and expression errors.
02
Four Tasks, One Diagnostic View
The benchmark separates “is it wrong?” from “what went wrong?” so model capability can be measured at two levels of granularity.
Question Error Detection
Verify whether the problem statement admits a coherent, mathematically valid solution.
Question Error Type
Classify the earliest invalidating issue across four question-side error types.
Answer Error Detection
Check a complete reasoning trace and determine whether the proposed answer is correct.
Answer Error Type
Locate whether the failure is logical, computational, or in mathematical expression.
03
Dataset at a Glance
Two difficulty tiers preserve the character of their seed domains: SIMPLE items are GSM8K-seeded and CHALLENGING items are MATH-seeded.
30 + 10STEM annotators + reviewers
Dual reviewIndependent labels and expert adjudication
0.69–0.91Per-type Fleiss’ κ agreement
Earliest errorA consistent single-label policy
04
Explore the Benchmark
Switch between a question-side failure and an answer-side reasoning trace.
Jennifer purchased 40 cans of milk…
Jennifer bought 7.5 additional cans for every 5 cans Mark bought. If Mark purchased 50 cans, how many cans did Jennifer bring home?
Why it failsThe item treats fractional cans as ordinary countable objects. The arithmetic is executable, but the premise violates the discrete nature of the quantity.
A locally plausible solution misses a global constraint.
The reasoning validates intermediate steps independently, yet the selected branch contradicts another condition in the original problem. A correct verifier must reason over the full constraint network.
What the label capturesThe earliest invalid inference—not every downstream arithmetic consequence—is annotated as the canonical error.
{
"question": "[mathematical problem]",
"answer": "[reasoning trace]",
"extent": "obvious error | not obvious error | no error",
"type": "logic error | expression error | computing error | null"
}05
Error Taxonomy
Question-side4 types
Conditions cannot hold simultaneously.
Required information is absent.
The premise violates real-world constraints.
Language or notation makes the task invalid.
Answer-side3 types
An invalid inference or mathematical method.
A numerical or symbolic calculation failure.
An unclear or malformed mathematical expression.
Each annotation includes a written justification and is independently reviewed.
06
Experimental Results
Even frontier systems remain far from saturation, especially on fine-grained error-type classification. Values below are accuracy on the SIMPLE split.
Closed-source Open-sourcex-axis: Accuracy (%)
Best answer-type Macro-F1 remains near 55, revealing substantial headroom in fine-grained diagnosis.
Qwen2.5 question-type Macro-F1 rises monotonically from 1.5B to 72B.
Math-tuned and long-reasoning models do not consistently outperform general-purpose siblings on verification.
07
From Diagnosis to Correction
Error labels are useful supervision, not merely diagnostic metadata.
Providing the ground-truth error type consistently improves correction accuracy for both DeepSeek-R1 and GPT-4o. The largest gains occur on structurally ambiguous errors such as missing or contradictory conditions.
✦
Takeaway. Knowing what kind of failure to look for helps a model revise the data more reliably.
Largest correction gainspercentage points
Missing conditions · GPT-4o+4.7
Missing conditions · DeepSeek-R1+3.5
Contradictions · GPT-4o+2.1
Computing errors · both+1.8
Largest gains confirmed by paired bootstrap, p < 0.01.
08
Access the Benchmark
09
Paper & Citation
arXiv preprint
MathDebugger: A Benchmark for Type-Aware Error Detection in Synthetic Math Data
Hao Liang, Meiyi Qiang, Yuying Li, Zefeng He, Xiaochen Ma, Ruichuan An, Yongzhen Guo, Zhengzhou Zhu, Bin Cui, and Wentao Zhang.
Read the paper ↗“ BibTeX
@article{liang2025mathdebugger,
title = {MathDebugger: A Benchmark for Type-Aware
Error Detection in Synthetic Math Data},
author = {Liang, Hao and Qiang, Meiyi and Li, Yuying
and He, Zefeng and Ma, Xiaochen and An, Ruichuan
and Guo, Yongzhen and Zhu, Zhengzhou and Cui, Bin
and Zhang, Wentao},
journal = {arXiv preprint arXiv:2502.19058},
year = {2025}
}