Benchmarks
MiniMax M3.1 Flash Benchmarks and Their Reliability
The official numbers, the third-party numbers, the known flaws in one of the headline benchmarks, and what is simply not published.
Official scores
MiniMax published these in the launch post on 19 October 2026:
| Benchmark | Score | What it measures |
|---|---|---|
| Automation-Bench (AA) | Top-tier group | Multi-step enterprise workflows across SaaS systems |
| BankerToolBench | 76.2 | Multi-step planning, document consistency, financial compliance, chained tool calls |
| DeepSWE v1.1 | 73.7% | Real software engineering tasks: understand code, add features, fix issues, validate with tests |
| OSWorld V2 | 61.7 | Computer use and cross-application execution |
| Agents’ Last Exam (ALE) | 29.6 | Real professional work with economic value |
| CyberGym | 82% | Vulnerability discovery and validation |
MiniMax also states that the model ranks first among Chinese models on OSWorld V2 and ALE.
Third-party scores
Artificial Analysis runs its own battery against the same endpoints. The figures below were read directly from the model page’s embedded data on 21 October 2026:
| Benchmark | Score |
|---|---|
| Intelligence Index | 42.71 (not estimated) |
| Humanity’s Last Exam | 47.96% |
| AA-LCR | 79.67% |
| GDPval-AA | 47.88% |
| Terminal-Bench 4.0 | 25.76% |
| SciCode | 53.36% |
| CritPt | 22.29% |
| Automation-Bench (partial score) | 72.52% |
| MMMU-Pro | 79.13% |
| frontierCode | 38.58% |
| Omniscience — accuracy | 56.2% |
| Omniscience — hallucination rate | 92.81% |
Two labels matter more than the numbers themselves:
- The Intelligence Index is a weighted composite across ten evaluations. A model can lose a single benchmark and still gain index points elsewhere, so read the per-benchmark rows, not just the headline.
- This model’s performance and speed data is flagged by Artificial Analysis as first-party (supplied by MiniMax), not independently benchmarked by Artificial Analysis. The same applies to the speed figures on the pricing page. Treat them as vendor-reported numbers presented through a third-party interface.
Known benchmark problems
The most important caveat on this page comes from the launch discussion itself. Users pointed out that a striking number of models land on almost exactly 74% on DeepSWE, and that the benchmark has documented quality problems. The cited analysis found issues in at least 23 of 113 tasks (more than 20.3%), leading to the benchmark being designated flawed.
The same comment is worth repeating for its verdict on the model itself: M3.1 Flash is “certainly capable and very cost effective”, but that commenter would not say it beats the strongest models in raw capability.
So: treat a 73.7% DeepSWE score as a rough signal that the model is competitive on real software tasks, not as evidence that it is a frontier-tier coder.
What is not published
Being explicit about gaps is more useful than filling them in:
- No independent third-party reproduction of the throughput claims. MiniMax says online inference is nearly 3× faster than the previous generation, with output speeds reaching 200+ tokens per second and a P50 above 170 tokens per second. Artificial Analysis displays a median of 104.7 tokens per second on long prompts (p05 67.6, p95 146.7) — same order of magnitude, different measurement.
- No published multilingual or Chinese-specific evaluation.
- No long-context retrieval benchmark from an independent party.
- No public comparison of tool-call reliability between M3 and M3.1 beyond the qualitative claims.
The hallucination number, properly labelled
Artificial Analysis’s Omniscience evaluation puts the hallucination rate at 92.81% with an accuracy of 56.2% — that is, on long-tail factual questions the model either answers correctly or does not know, and it often chooses to answer.
This is the figure behind Chinese-language coverage describing a “93% hallucination rate”. Two rules for using it:
- Name the benchmark. Omniscience measures accuracy on hard, long-tail factual knowledge. It is not a general “how often is this model wrong” rate.
- Draw the practical conclusion. A model that rarely says “I don’t know” needs verification in the loop. For code, that means tests. For research, that means citations you check yourself.
How to use these numbers
| If you are choosing between models | Read this |
|---|---|
| Minimising cost per unit of work | Pricing and per-task cost |
| Running long documents or large codebases | Context window (1M input, 524K output) and the fact that M3.1 does not price-step |
| Automating browser or desktop tasks | OSWorld V2 and ALE, plus the honesty caveat above |
| Building an agent loop | BankerToolBench and Automation-Bench, where this model scores well |
| Writing code you have to ship | DeepSWE (flawed benchmark) and Terminal-Bench, plus your own tests |
Next: use cases turns these capabilities into tasks you can judge, and MiniMax M3.1 Flash vs DeepSeek V4.1 Flash puts the numbers next to a direct competitor on the same run. For the informal evaluation this model got before it had a name — the community’s front-end demos and the adversarial re-test of them — see Space Bunny evals.
Sources and version notes
Checked October 21, 2026. We label official claims, third-party measurements, our own testing and community reports separately — see the sources policy for what each label means.