Abstract:Evaluating LLM agents in realistic service scenarios requires complex task dependencies, imperfect user behavior, and an evaluation that accommodates multiple valid solutions. We introduce CRAB-Bench (Constraint-based Realistic Agent Benchmark) and RUSE (Realistic User Simulation Engine) to address this gap. CRAB-Bench generates tasks via a constraint graph over multiple interdependent entities with structured distractors, requiring agents to reason carefully over thousands of misleading candidates where only a tiny fraction of solutions are valid. RUSE replaces cooperative, template-like simulators with realistic users grounded in human behavioral studies, instantiated across diverse personas and four behavioral dimensions. Experiments on four frontier LLM agents show that the best model achieves only 61% pass@1 on CRAB-Bench, and switching to RUSE causes further drops of up to 57%, concentrated in task-solving ability rather than conversational quality. Information Disclosure is the most damaging behavioral dimension, and agents interacting with RUSE are less likely to admit mistakes, instead masking errors through implicit corrections.
| Subjects: | Computation and Language (cs.CL) |
| Cite as: | arXiv:2606.01815 [cs.CL] |
| (or arXiv:2606.01815v1 [cs.CL] for this version) | |
| https://doi.org/10.48550/arXiv.2606.01815 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Danqing Wang [view email]
[v1]
Mon, 1 Jun 2026 07:31:04 UTC (2,808 KB)