Abstract:We introduce a novel reinforcement learning (RL) framework that treats parameterized action distributions as actions, redefining the boundary between agent and environment. This reparameterization makes the new action space continuous, regardless of the original action type (discrete, continuous, hybrid, etc.). Under this new parameterization, we develop a generalized deterministic policy gradient estimator, Distributions-as-Actions Policy Gradient (DA-PG), which has lower variance than the gradient in the original action space. Although learning the critic over distribution parameters poses new challenges, we introduce Interpolated Critic Learning (ICL), a simple yet effective strategy to enhance learning, supported by insights from bandit settings. Building on TD3, a strong baseline for continuous control, we propose a practical actor-critic algorithm, Distributions-as-Actions Actor-Critic (DA-AC). Empirically, DA-AC achieves competitive performance in various settings across discrete, continuous, and hybrid control.
| Comments: | Accepted to ICLR 2026 (camera-ready) |
| Subjects: | Machine Learning (cs.LG); Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2506.16608 [cs.LG] |
| (or arXiv:2506.16608v3 [cs.LG] for this version) | |
| https://doi.org/10.48550/arXiv.2506.16608 arXiv-issued DOI via DataCite |
Submission history
From: Jiamin He [view email]
[v1]
Thu, 19 Jun 2025 21:19:19 UTC (1,451 KB)
[v2]
Mon, 2 Mar 2026 17:30:05 UTC (2,305 KB)
[v3]
Thu, 14 May 2026 06:08:12 UTC (2,311 KB)