Abstract:Large language models (LLMs) have rapidly evolved from single-turn text generators into the foundation of increasingly capable agents. As these agents take on more complex reasoning, decision making, tool use, and long-horizon tasks, reinforcement learning (RL) is becoming increasingly important for shaping their behavior. This shift is especially visible in agentic RL, where models must interact with tools and environments across multiple rounds rather than produce a single standalone response. In this regime, the usual view of a trajectory as one ever-growing token sequence becomes increasingly inadequate: it makes context evolution rigid and creates representation mismatches between rollout and training. This paper presents Agent-R1, a unified and modular framework for agentic RL built around step-level trajectory representation, flexible context management, and layered interfaces for workflows, environments and optimization. The key idea is to treat each interaction step as the basic reinforcement-learning transition, while keeping the optimization layer flexible: once the interaction is modeled at the step level, the framework can support token-level credit assignment, step-level credit assignment, or other compatible designs. These design choices make the framework compatible with a range of optimization strategies rather than tying it to a single algorithm. Together, these components provide a principled, extensible, and reusable substrate for agentic RL.
| Comments: | This paper serves as the technical report of the Agent-R1 project |
| Subjects: | Computation and Language (cs.CL) |
| Cite as: | arXiv:2511.14460 [cs.CL] |
| (or arXiv:2511.14460v2 [cs.CL] for this version) | |
| https://doi.org/10.48550/arXiv.2511.14460 arXiv-issued DOI via DataCite |
Submission history
From: Shuo Yu [view email]
[v1]
Tue, 18 Nov 2025 13:03:15 UTC (440 KB)
[v2]
Sat, 30 May 2026 06:08:55 UTC (1,358 KB)