Equivalence of the Empirical Risk Minimization to Regularization on the Family of f-Divergences

Francisco Daunas (1, 2) , Iñaki Esnaola (1, 3) , Samir M Perlaza (2, 3, 4) , H. Vincent Poor (3)

1 ACSE - Department of Automatic Control and Systems Engineering [ Sheffield]
2 NEO - Network Engineering and Operations
3 ECE - Department of Electrical and Computer Engineering [Princeton]
4 GAATI - Laboratoire de Géométrie Algébrique et Applications à la Théorie de l'Information

The solution to empirical risk minimization with f-divergence regularization (ERM-fDR) is presented under mild conditions on f. Under such conditions, the optimal measure is shown to be unique. Examples of the solution for particular choices of the function f are presented. Previously known solutions to common regularization choices are obtained by leveraging the flexibility of the family of f-divergences. These include the unique solutions to empirical risk minimization with relative entropy regularization (Type-I and Type-II). The analysis of the solution unveils the following properties of f-divergences when used in the ERM-fDR problem: i) f-divergence regularization forces the support of the solution to coincide with the support of the reference measure, which introduces a strong inductive bias that dominates the evidence provided by the training data; and ii) any f-divergence regularization is equivalent to a different f-divergence regularization with an appropriate transformation of the empirical risk function.

Mots clés: Empirical risk minimization; f -divergence regularization; statistical learning

Domaines: Théorie de l'information [cs.IT]; Autres [stat.ML]

Francisco Daunas, Iñaki Esnaola, Samir M Perlaza, H. Vincent Poor. Equivalence of the Empirical Risk Minimization to Regularization on the Family of f-Divergences. Proceedings of the IEEE International Symposium on Information Theory, Jul 2024, Athens, Greece. ⟨hal-04431558v1⟩