Abstract:A surge in academic publications calls for automated deep research (DR) systems, but accurately evaluating them is still an open problem. First, existing benchmarks often focus narrowly on retrieval while neglecting high-level planning and reasoning. Second, existing benchmarks favor general domains over the academic domains that are the core application for DR agents. To address these gaps, we introduce ADRA-Bank, a modular benchmark for Academic DR Agents. Grounded in academic literature, our benchmark is a human-annotated dataset of 200 instances across 10 academic domains, including both research and review papers. Furthermore, we propose a modular Evaluation Paradigm for Academic DR Agents (ADRA-Eval), which leverages the rich structure of academic papers to assess the core capabilities of planning, retrieval, and reasoning. It employs two complementary modes: an end-to-end evaluation for \task agents and an isolated evaluation for foundational LLMs as potential backbones. Results reveal uneven capabilities: while agents show specialized strengths, they struggle with multi-source retrieval and cross-field consistency. Moreover, improving high-level planning capability is the crucial factor for unlocking the reasoning potential of foundational LLMs as backbones. By exposing these actionable failure modes, ADRA-Bank provides a diagnostic tool to guide the development of more reliable automatic academic research assistants.
| Subjects: | Computation and Language (cs.CL) |
| Cite as: | arXiv:2512.00986 [cs.CL] |
| (or arXiv:2512.00986v3 [cs.CL] for this version) | |
| https://doi.org/10.48550/arXiv.2512.00986 arXiv-issued DOI via DataCite |
Submission history
From: Zhihan Guo [view email]
[v1]
Sun, 30 Nov 2025 17:16:47 UTC (431 KB)
[v2]
Fri, 30 Jan 2026 08:08:20 UTC (370 KB)
[v3]
Mon, 1 Jun 2026 02:51:22 UTC (596 KB)