AppSim-Bench
Bridging Real-world Apps and Reproducible Evaluation for Mobile GUI Agents
Abstract
Mobile GUI agents can execute tasks from natural-language instructions, but their evaluation remains difficult to make both realistic and reproducible. Existing benchmarks typically trade off these goals: simplified apps lack real-world mobile complexity, whereas live commercial apps introduce uncontrolled variation from recommendations, advertisements, accounts, and changing content.
We propose AppSim-Bench, which addresses this trade-off through controllable simulated apps that preserve task-relevant interaction logic while supporting deterministic evaluation. Built through a coding-agent-assisted and human-verified workflow, it contains 557 tasks across 17 high-frequency Chinese and English apps. Its controllable backend data and outcome-based verification remove major sources of environmental stochasticity, enabling reproducible cross-model comparison.
Evaluating 19 GUI agents, spanning general-purpose and GUI-specialized systems, we find that autonomous mobile execution remains far from solved. The best model completes only 50.27% of tasks, and 28.55% of tasks are not solved by any agent. Further analysis shows that failures concentrate in longer workflows, numerical reasoning tasks, and inefficient trajectories marked by high action overhead and budget exhaustion. Our project is available at https://github.com/Acrab-Lab/AppSim.
Apps in AppSim-Bench
AppSim-Bench covers 17 apps spanning video sharing, travel, food delivery, navigation, e-commerce, music streaming, social media, communication, ride hailing, and video conferencing. The table below lists each app, a brief description, and its task count (557 tasks in total).
| # | App (EN) | App (ZH) | Description | #Tasks |
|---|---|---|---|---|
| 1 | Bilibili | 哔哩哔哩 | A video-sharing app with ACG-oriented community content. | 30 |
| 2 | Booking.com | — | A travel booking app for hotels, flights, cars, taxis, and attractions. | 25 |
| 3 | Ctrip | 携程旅行 | A travel-service app for hotels, flights, trains, trips, and messages. | 35 |
| 4 | Ele.me | 饿了么 | A local-life and food-delivery app for restaurants, orders, coupons, and addresses. | 40 |
| 5 | Amap | 高德地图 | A map and navigation app with nearby places, routes, favorites, and travel tools. | 28 |
| 6 | NetEase Cloud Music | 网易云音乐 | A music-streaming app with songs, playlists, charts, lyrics, and user profiles. | 39 |
| 7 | JD | 京东 | An e-commerce app for product search, carts, orders, addresses, and store services. | 40 |
| 8 | RedNote | 小红书 | A lifestyle social-media app for notes, creators, comments, messages, and profiles. | 29 |
| 9 | Tencent Meeting | 腾讯会议 | A video-conferencing app for meetings, contacts, messages, and personal rooms. | 26 |
| 10 | Uber Eats | — | A food-delivery and local-services app with restaurants, carts, orders, and rides. | 34 |
| 11 | 微信 | A messaging and social app with chats, contacts, groups, and Moments. | 16 | |
| 12 | YouTube | — | A video platform for search, playback, comments, subscriptions, Shorts, and settings. | 35 |
| 13 | Amazon | \ | An e-commerce app for product discovery, shopping lists, carts, orders, and support. | 40 |
| 14 | — | A messaging app with chats, calls, statuses, channels, communities, and contacts. | 40 | |
| 15 | Zoom | — | A video-meeting app for instant meetings, scheduled meetings, chat, contacts, and settings. | 20 |
| 16 | Spotify | — | A music and podcast app with playback, search, playlists, lyrics, and subscriptions. | 40 |
| 17 | — | A social-media app for posts, reels, messages, profiles, search, and publishing. | 40 | |
| Total | 557 | |||
How AppSim-Bench is Built
AppSim-Bench has three components: Tasks described in natural language, Simulated Apps with controllable backend data, and Automated Evaluation via deterministic outcome verification. Simulated apps are produced through a human-led, coding-agent-assisted workflow: developers specify task requirements, data schemas, page hierarchies, and reference screenshots; coding agents generate candidate implementations; and human developers inspect, run, and correct them until the task-relevant UI flows and backend states match the specification.
Benchmark at a Glance
(a–b) Task coverage across 9 Chinese and 8 English apps. (c) 283 Chinese vs. 274 English tasks. (d) Distribution by human-annotated reference steps (≤5, 6–10, ≥11). (e) 183 numerical-reasoning tasks split into counting, arithmetic, comparison, and threshold-filtering subtypes.
The 17 Simulated Apps
AppSim-Bench spans 17 high-frequency apps across video sharing, travel, food delivery, navigation, e-commerce, music streaming, social media, communication, ride hailing, and video conferencing. Each is a controllable simulation that reproduces the task-relevant screens and flows of its real-world counterpart. Screenshots below show one representative screen per app.
Main Results
Overall accuracy of 19 agents on AppSim-Bench, spanning general-purpose and GUI-specialized systems, with task-weighted accuracy on the Chinese (ZH) and English (EN) app groups, accuracy by task length (#S = human-annotated step count), and numerical-reasoning subtypes. Even the strongest model completes only about half of the tasks, and performance drops sharply on the longest workflows (≥11 steps).
| Agent Model | Overall Benchmark | Task Length | Numerical Reasoning Subtype | |||||||
|---|---|---|---|---|---|---|---|---|---|---|
| Overall | ZH Apps | EN Apps | #S ≤ 5 | #S 6–10 | #S ≥ 11 | Cnt. | Arith. | Comp. | Thres. | |
| General-purpose Agents | ||||||||||
| Claude-Opus-4.7 | 50.27% | 60.07% | 40.15% | 58.87% | 54.29% ↓4.58 | 28.15% ↓26.14† | 52.31% | 55.56% | 52.00% | 42.86% |
| Gemini-3-Pro | 49.91% | 60.07% | 39.42% | 56.38% | 52.86% ↓3.52 | 33.33% ↓19.53 | 46.15% | 42.22% | 49.33% | 23.81% |
| GPT-5.5 | 49.73% | 58.66% | 40.51% | 59.22% | 52.14% ↓7.08 | 27.41% ↓24.73 | 44.62% | 57.78% | 52.00% | 33.33% |
| Gemini-3.1-Pro | 49.55% | 59.36% | 39.42% | 56.74% | 52.86% ↓3.88 | 31.11% ↓21.75 | 43.08% | 42.22% | 50.67% | 23.81% |
| Doubao-Seed-1.6 | 39.14% | 50.88% | 27.01% | 45.74% | 43.57% ↓2.17 | 20.74% ↓22.83 | 23.08% | 22.22% | 36.00% | 9.52% |
| GPT-5 | 33.39% | 37.81% | 28.83% | 43.62% | 31.43% ↓12.19 | 14.07% ↓17.36 | 27.69% | 35.56% | 40.00% | 4.76% |
| Doubao-Seed-1.8 | 32.14% | 34.63% | 29.56% | 43.26% | 27.86% ↓15.40 | 13.33% ↓14.53 | 23.08% | 31.11% | 28.00% | 4.76% |
| Qwen-3.6-Plus | 31.42% | 39.58% | 22.99% | 39.01% | 31.43% ↓7.58 | 15.56% ↓15.87 | 23.08% | 13.33% | 28.00% | 9.52% |
| Doubao-Seed-2.0 | 29.26% | 36.40% | 21.90% | 37.94% | 28.57% ↓9.37 | 11.85% ↓16.72 | 27.69% | 28.89% | 20.00% | 14.29% |
| GPT-5.4 | 29.08% | 34.63% | 23.36% | 38.30% | 25.71% ↓12.59 | 13.33% ↓12.38 | 24.62% | 24.44% | 30.67% | 9.52% |
| Qwen-3.6-Flash | 28.19% | 38.87% | 17.15% | 38.30% | 25.71% ↓12.59 | 9.63% ↓16.08 | 20.00% | 24.44% | 26.67% | 4.76% |
| Claude-Sonnet-4.6 | 27.29% | 31.10% | 23.36% | 36.17% | 27.86% ↓8.31 | 8.15% ↓19.71 | 29.23% | 33.33% | 26.67% | 14.29% |
| Claude-Haiku-4.5 | 22.80% | 22.61% | 22.99% | 34.04% | 15.71% ↓18.33† | 6.67% ↓9.04 | 20.00% | 22.22% | 20.00% | 9.52% |
| Claude-Sonnet-4.5 | 21.01% | 24.03% | 17.88% | 31.91% | 14.29% ↓17.62 | 5.19% ↓9.10 | 23.08% | 26.67% | 21.33% | 14.29% |
| Gemini-2.5-Pro | 17.24% | 21.20% | 13.14% | 26.24% | 14.29% ↓11.95 | 1.48% ↓12.81 | 23.08% | 17.78% | 17.33% | 9.52% |
| GUI-specialized Agents | ||||||||||
| MobileAgent-v3.5 | 31.42% | 35.34% | 27.37% | 41.49% | 26.43% ↓15.06 | 15.56% ↓10.87 | 18.46% | 24.44% | 32.00% | 4.76% |
| UI-TARS-1.5-7B | 22.62% | 31.10% | 13.87% | 30.14% | 22.86% ↓7.28 | 6.67% ↓16.19 | 20.00% | 8.89% | 16.00% | 4.76% |
| AgentCPM-GUI | 10.05% | 9.19% | 10.95% | 17.02% | 5.71% ↓11.31 | 0.00% ↓5.71 | 3.08% | 2.22% | 5.33% | 4.76% |
| V-Droid | 9.87% | 13.43% | 6.20% | 15.25% | 7.86% ↓7.39 | 0.74% ↓7.12 | 15.38% | 11.11% | 9.33% | 0.00% |
Bold marks the best value in each column and underlining marks the second-best. ZH/EN are task-weighted accuracies over the Chinese and English app groups. Down arrows indicate drops relative to the adjacent shorter task-length bucket.
BibTeX
@inproceedings{appsimbench2026,
title = {AppSim-Bench: Bridging Real-world Apps and Reproducible Evaluation for Mobile GUI Agents},
author = {Jintian Feng, Long Chen, Xiao Yu, Jiayi Dai, Chenglong Liu, Haoru Wang, Zizhen Xue, Yuxuan Shi, Ziyang Wang and Yichen Gong},
year = {2026},
booktitle = {EMNLP 2026 Main Conference},
note = {Accepted}
}