Computer Science > Machine Learning
[Submitted on 10 Jun 2020 (v1), revised 8 Dec 2020 (this version, v2), latest version 17 Oct 2022 (v3)]
Title:On Mixup Regularization
View PDFAbstract:Mixup is a data augmentation technique that creates new examples as convex combinationsof training points and labels. This simple technique has empirically shown to improvethe accuracy of many state-of-the-art models in different settings and applications, butthe reasons behind this empirical success remain poorly understood. In this paper wetake a substantial step in explaining the theoretical foundations of Mixup, by clarifyingits regularization effects. We show that Mixup can be interpreted as standard empiricalrisk minimization estimator subject to a combination of data transformation and randomperturbation of the transformed data. We gain two core insights from this new interpretation.First, the data transformation suggests that, at test time, a model trained with Mixup shouldalso be applied to transformed data, a one-line change in code that we show empirically toimprove both accuracy and calibration of the prediction. Second, we show how the randomperturbation of the new interpretation of Mixup induces multiple known regularizationschemes, including label smoothing and reduction of the Lipschitz constant of the estimator.These schemes interact synergistically with each other, resulting in a self calibrated andeffective regularization effect that prevents overfitting and overconfident predictions. Wecorroborate our theoretical analysis with experiments that support our conclusions.
Submission history
From: Luigi Carratino [view email][v1] Wed, 10 Jun 2020 20:11:46 UTC (1,423 KB)
[v2] Tue, 8 Dec 2020 22:16:20 UTC (1,388 KB)
[v3] Mon, 17 Oct 2022 10:04:31 UTC (1,115 KB)
Bibliographic and Citation Tools
Bibliographic Explorer (What is the Explorer?)
Litmaps (What is Litmaps?)
scite Smart Citations (What are Smart Citations?)
Code, Data and Media Associated with this Article
CatalyzeX Code Finder for Papers (What is CatalyzeX?)
DagsHub (What is DagsHub?)
Gotit.pub (What is GotitPub?)
Papers with Code (What is Papers with Code?)
ScienceCast (What is ScienceCast?)
Demos
Recommenders and Search Tools
Influence Flower (What are Influence Flowers?)
Connected Papers (What is Connected Papers?)
CORE Recommender (What is CORE?)
IArxiv Recommender
(What is IArxiv?)
arXivLabs: experimental projects with community collaborators
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.