Abstract:PAC-Bayes generalisation bounds are derived via change-of-measure inequalities that transfer concentration properties from a reference measure to all posterior measures. The specific choice of change of measure determines the assumptions required on the empirical risk; in particular, the classical Donsker--Varadhan theorem leads to bounds relying on bounded exponential moments. We study change-of-measure inequalities based on \(f\)-divergences, obtained by combining the Legendre transform of \(f\) with the Fenchel--Young inequality. Beyond their intrinsic interest in probability theory, we show how these inequalities are helpful in learning theory and yield PAC-Bayes bounds under tailored assumptions on the empirical risk, thereby extending the range of conditions under which PAC-Bayesian guarantees can be established.
| Comments: | 27 pages |
| Subjects: | Machine Learning (stat.ML); Information Theory (cs.IT); Machine Learning (cs.LG); Probability (math.PR); Statistics Theory (math.ST) |
| Cite as: | arXiv:2202.05568 [stat.ML] |
| (or arXiv:2202.05568v2 [stat.ML] for this version) | |
| https://doi.org/10.48550/arXiv.2202.05568 arXiv-issued DOI via DataCite |
Submission history
From: Benjamin Guedj [view email]
[v1]
Fri, 11 Feb 2022 11:53:28 UTC (32 KB)
[v2]
Thu, 14 May 2026 15:08:56 UTC (209 KB)