A Statistical Regularization Approach for Reducing Overfitting in Deep Neural Networks

Abstract

Deep Neural Networks (DNNs) have recorded impressive performance on vision and language tasks, but their overfitting nature is a major weakness, especially in low-data and high-dimensional scenarios. A new statistical regularization framework that unites penalized likelihood estimation with neural network training is suggested in this work. Specifically, the shrinkage approaches of elastic net, smoothly clipped absolute deviation (SCAD), and minimax concave penalty (MCP) are incorporated into the loss form to counter overfitting according to the model complexity. We present a conceptual study of the bias-variance trade-off of varying sample sizes and network depths with new information on statistical behavior of deep models. An extremely large-scale experiment on benchmark datasets (CIFAR-10, ImageNet, IMDB and AG News) demonstrates that the proposed scheme is significantly more effective in reducing the generalization error, in terms of stability, and insensible to adversarial perturbations than regularization schemes. These kinds of discoveries demonstrate that statistical regularization as one of the future possibilities of the reliability and interpretability of deep learning systems.