Android Bench 2.0
AI-assisted software engineering has seen the emergence of several benchmarks to measure the capabilities of LLMs. Android developers face specific challenges that aren't covered by existing benchmarks, so we created one that focuses on a north star of high quality Android development.
Android Bench now includes long-horizon task results using specialized agents. Click a model for detailed results, or learn more in our blog and methodology.
Long-horizon task results
| Model - Agent | Pass rate (%) Average percentage of 30 tasks successfully resolved across all runs for each model |
arrow_range
Cl range (%)
Expected performance range, reflecting the results' statistical reliability (p-value < 0.05)
|
Completion rate (%) How close each run got to a complete solution, even when the task failed | Avg latency (h) Average time taken to solve 30 tasks across all runs | Avg cost ($) Average cost per full benchmark run |
|---|---|---|---|---|---|
GPT 6 Astra
chevron_right
codex |
28.0 | 13.3 — 42.0 | 82.2 | 7.9 | $375.7 |
Claude Fable 5 1
chevron_right
claude-code |
22.7 | 10.7 — 36.0 | 82.4 | 22.2 | $492.6 |
|
GPT 5.6 Sol
chevron_right
codex |
19.3 | 7.3 — 32.0 | 74.3 | 8.6 | $235.8 |
|
Claude Opus 5
chevron_right
claude-code |
16.7 | 5.3 — 29.3 | 77.8 | 27.0 | $861.4 |
Qwen3.8 Max
chevron_right
qwen-coder |
14.0 | 4.7 — 24.0 | 74.3 | 47.2 | $260.2 |
Kimi K3
chevron_right
kimi-code |
12.0 | 3.3 — 22.0 | 72.8 | 66.2 | $418.3 |
Gemini 3.8 Flash
chevron_right
antigravity-sdk |
8.0 | 3.3 — 13.3 | 47.4 | 12.1 | $34.5 |
|
Gemini 3.7 Flash
chevron_right
antigravity-sdk |
7.3 | 1.3 — 14.7 | 50.1 | 9.9 | $26.0 |
|
Claude Sonnet 5
chevron_right
claude-code |
6.7 | 1.3 — 13.3 | 59.8 | 14.7 | $283.7 |
|
GPT 5.6 Terra
chevron_right
codex |
4.7 | 1.3 — 8.7 | 53.4 | 5.1 | $53.2 |
|
GPT 5.6 Luna
chevron_right
codex |
3.3 | 0.7 — 7.3 | 55.2 | 7.6 | $13.5 |
GPT 6 Astra
•
codex
Closed weights
model
Pass rate
28.0%
Avg completion rate
82.2%
Avg latency
7.9 h
Avg cost
$375.7
Per task results
| Task | Category | Avg completion rate | Runs passed | Failure reason(s) |
|---|---|---|---|---|
| Convert Flutter to Android: Habo app (habit tracker) | App conversion | 99% | 0/5 | All runs failed tests |
| Convert React Native to Android: BlueWallet app (Bitcoin wallet) | App conversion | 83% | 0/5 | All runs failed tests |
| New accessibility support across Droidcon (conference app) | New feature | 98% | 3/5 | 2 runs failed tests |
| New Media3 and ExoPlayer video playback in Now in Android (news app) | New feature | 88% | 0/5 | All runs failed tests |
| New picture-in-picture video mode in Now in Android (news app) | New feature | 71% | 2/5 | 2 runs failed tests, 1 run missed required test results |
| New CameraX profile picture capture in Now in Android (news app) | New feature | 100% | 5/5 | N/A |
| New user account submodule in Food Vibes (full app creation - food delivery) | App creation | 97% | 2/5 | 3 runs failed tests |
| New authentication flow with SMS verification in Food Vibes (full app creation - food delivery) | App creation | 68% | 0/5 | All runs failed validation |
| New bottom navigation and Browse search screen in Food Vibes (full app creation - food delivery) | App creation | 93% | 0/5 | 3 runs failed validation, 2 runs failed tests |
| New design system and reusable UI component library in Food Vibes (full app creation - food delivery) | App creation | 63% | 0/5 | 4 runs failed validation, 1 run failed the build |
| New All Filters screen in Food Vibes (full app creation - food delivery) | App creation | 99% |