1 Introduction
Over the last decade, deep learning has made impressive progress on a variety of notoriously difficult tasks in computer vision
KrizhevskySH12; HeZRS16, speech recognition GravesMH13, machine translation SutskeverVL14, and gameplaying MnihKSAVBGRFO15; SilverHCGSVSAPL16. This progress hinged on a number of major advances in terms of hardware, datasets Krizhevsky09; RussakovskyDSKSMHKKBBF15, and algorithmic and architectural techniques SrivastavaHKSS14; KingmaB15; NairH10; SutskeverMDH13. One of the most prominent examples of such advances was batch normalization (BatchNorm) IoffeS15.At a high level, BatchNorm is a technique that aims to improve the training of neural networks by stabilizing the distributions of layer inputs. This is achieved by introducing additional network layers that control the first two moments (mean and variance) of these distributions.
The practical success of BatchNorm is indisputable. By now, it is used by default in most deep learning models, both in research (more than 6,000 citations) and realworld settings. Somewhat shockingly, however, despite its prominence, we still have a poor understanding of what the effectiveness of BatchNorm is stemming from. In fact, there are now a number of works that provide alternatives to BatchNorm BaKH16; ClevertUH16; KlambauerUMH17; WuHe18, but none of them seem to bring us any closer to understanding this issue. (A similar point was also raised recently in RahimiR17.)
Currently, the most widely accepted explanation of BatchNorm’s success, as well as its original motivation, relates to socalled internal covariate shift (ICS). Informally, ICS refers to the change in the distribution of layer inputs caused by updates to the preceding layers. It is conjectured that such continual change negatively impacts training. The goal of BatchNorm was to reduce ICS and thus remedy this effect.
Even though this explanation is widely accepted, we seem to have little concrete evidence supporting it. In particular, we still do not understand the link between ICS and training performance. The chief goal of this paper is to address all these shortcomings. Our exploration lead to somewhat startling discoveries.
Our Contributions.
Our point of start is demonstrating that there does not seem to be any link between the performance gain of BatchNorm and the reduction of internal covariate shift. Or that this link is tenuous, at best. In fact, we find that in a certain sense BatchNorm might not even be reducing internal covariate shift.
We then turn our attention to identifying the roots of BatchNorm’s success. Specifically, we demonstrate that BatchNorm impacts network training in a fundamental way: it makes the landscape of the corresponding optimization problem significantly more smooth. This ensures, in particular, that the gradients are more predictive and thus allows for use of larger range of learning rates and faster network convergence. We provide an empirical demonstration of these findings as well as their theoretical justification. We prove that, under natural conditions, the Lipschitzness of both the loss and the gradients (also known as smoothness Nesterov14) are improved in models with BatchNorm.
Finally, we find that this smoothening effect is not uniquely tied to BatchNorm. A number of other natural normalization techniques have a similar (and, sometime, even stronger) effect. In particular, they all offer similar improvements in the training performance.
We believe that understanding the roots of such a fundamental techniques as BatchNorm will let us have a significantly better grasp of the underlying complexities of neural network training and, in turn, will inform further algorithmic progress in this context.
Our paper is organized as follows. In Section 2, we explore the connections between BatchNorm, optimization, and internal covariate shift. Then, in Section 3, we demonstrate and analyze the exact roots of BatchNorm’s success in deep neural network training. We present our theoretical analysis in Section 4. We discuss further related work in Section 5 and conclude in Section 6.
2 Batch normalization and internal covariate shift
Batch normalization (BatchNorm) IoffeS15 has been arguably one of the most successful architectural innovations in deep learning. But even though its effectiveness is indisputable, we do not have a firm understanding of why this is the case.
Broadly speaking, BatchNorm is a mechanism that aims to stabilize the distribution (over a minibatch) of inputs to a given network layer during training. This is achieved by augmenting the network with additional layers that set the first two moments (mean and variance) of the distribution of each activation to be zero and one respectively. Then, the batch normalized inputs are also typically scaled and shifted based on trainable parameters to preserve model expressivity. This normalization is applied before the nonlinearity of the previous layer.
One of the key motivations for the development of BatchNorm was the reduction of socalled internal covariate shift (ICS). This reduction has been widely viewed as the root of BatchNorm’s success. Ioffe and Szegedy IoffeS15 describe ICS as the phenomenon wherein the distribution of inputs to a layer in the network changes due to an update of parameters of the previous layers. This change leads to a constant shift of the underlying training problem and is thus believed to have detrimental effect on the training process.


Despite its fundamental role and widespread use in deep learning, the underpinnings of BatchNorm’s success remain poorly understood RahimiR17. In this work we aim to address this gap. To this end, we start by investigating the connection between ICS and BatchNorm. Specifically, we consider first training a standard VGG SimonyanZ15 architecture on CIFAR10 Krizhevsky09 with and without BatchNorm. As expected, Figures 1(a) and (b) show a drastic improvement, both in terms of optimization and generalization performance, for networks trained with BatchNorm layers. Figure 1(c) presents, however, a surprising finding. In this figure, we visualize to what extent BatchNorm is stabilizing distributions of layer inputs by plotting the distribution (over a batch) of a random input over training. Surprisingly, the difference in distributional stability (change in the mean and variance) in networks with and without BatchNorm layers seems to be marginal. This observation raises the following questions:

Is the effectiveness of BatchNorm indeed related to internal covariate shift?

Is BatchNorm’s stabilization of layer input distributions even effective in reducing ICS?
We now explore these questions in more depth.
2.1 Does BatchNorm’s performance stem from controlling internal covariate shift?
The central claim in IoffeS15 is that controlling the mean and variance of distributions of layer inputs is directly connected to improved training performance. Can we, however, substantiate this claim?
We propose the following experiment. We train networks with random noise injected after BatchNorm layers. Specifically, we perturb each activation for each sample in the batch using i.i.d. noise sampled from a nonzero mean and nonunit variance distribution. We emphasize that this noise distribution changes at each time step (see Appendix A for implementation details).
Note that such noise injection produces a severe covariate shift that skews activations at every time step. Consequently, every unit in the layer experiences a
different distribution of inputs at each time step. We then measure the effect of this deliberately introduced distributional instability on BatchNorm’s performance. Figure 2 visualizes the training behavior of standard, BatchNorm and our “noisy” BatchNorm networks. Distributions of activations over time from layers at the same depth in each one of the three networks are shown alongside.Observe that the performance difference between models with BatchNorm layers, and “noisy” BatchNorm layers is almost nonexistent. Also, both these networks perform much better than standard networks. Moreover, the “noisy” BatchNorm network has qualitatively less stable distributions than even the standard, nonBatchNorm network, yet it still performs better in terms of training. To put the magnitude of the noise into perspective, we plot the mean and variance of random activations for select layers in Figure 7. Moreover, adding the same amount of noise to the activations of the standard (nonBatchNorm) network prevents it from training entirely.
Clearly, these findings are hard to reconcile with the claim that the performance gain due to BatchNorm stems from increased stability of layer input distributions.
2.2 Is BatchNorm reducing internal covariate shift?
Our findings in Section 2.1 make it apparent that ICS is not directly connected to the training performance, at least if we tie ICS to stability of the mean and variance of input distributions. One might wonder, however: Is there a broader notion of internal covariate shift that has such a direct link to training performance? And if so, does BatchNorm indeed reduce this notion?
Recall that each layer can be seen as solving an empirical risk minimization problem where given a set of inputs, it is optimizing some loss function (that possibly involves later layers). An update to the parameters of any previous layer will change these inputs, thus changing this empirical risk minimization problem itself. This phenomenon is at the core of the intuition that Ioffe and Szegedy
IoffeS15 provide regarding internal covariate shift. Specifically, they try to capture this phenomenon from the perspective of the resulting distributional changes in layer inputs. However, as demonstrated in Section 2.1, this perspective does not seem to properly encapsulate the roots of BatchNorm’s success.To answer this question, we consider a broader notion of internal covariate shift that is more tied to the underlying optimization task. (After all the success of BatchNorm is largely of an optimization nature.) Since the training procedure is a firstorder method, the gradient of the loss is the most natural object to study. To quantify the extent to which the parameters in a layer would have to “adjust” in reaction to a parameter update in the previous layers, we measure the difference between the gradients of each layer before and after updates to all the previous layers. This leads to the following definition.
Definition 2.1.
Let be the loss, , …, be the parameters of each of the layers and be the batch of inputlabel pairs used to train the network at time . We define internal covariate shift (ICS) of activation at time to be the difference , where
Here, corresponds to the gradient of the layer parameters that would be applied during a simultaneous update of all layers (as is typical). On the other hand, is the same gradient after all the previous layers have been updated with their new values. The difference between and thus reflects the change in the optimization landscape of caused by the changes to its input. It thus captures precisely the effect of crosslayer dependencies that could be problematic for training.
Equipped with this definition, we measure the extent of ICS with and without BatchNorm layers. To isolate the effect of nonlinearities as well as gradient stochasticity, we also perform this analysis on (25layer) deep linear networks (DLN) trained with fullbatch gradient descent (see Appendix A for details). The conventional understanding of BatchNorm suggests that the addition of BatchNorm layers in the network should increase the correlation between and , thereby reducing ICS.
Surprisingly, we observe that networks with BatchNorm often exhibit an increase in ICS (cf. Figure 3). This is particularly striking in the case of DLN. In fact, in this case, the standard network experiences almost no ICS for the entirety of training, whereas for BatchNorm it appears that and are almost uncorrelated. We emphasize that this is the case even though BatchNorm networks continue to perform drastically better in terms of attained accuracy and loss. (The stabilization of the BatchNorm VGG network later in training is an artifact of faster convergence.) This evidence suggests that, from optimization point of view BatchNorm might not even reduce the internal covariate shift.
3 Why does BatchNorm work?
Our investigation so far demonstrated that the generally asserted link between the internal covariate shift (ICS) and the optimization performance is tenuous, at best. But BatchNorm does significantly improve the training process. Can we explain why this is the case?
Aside from reducing ICS, Ioffe and Szegedy IoffeS15
identify a number of additional properties of BatchNorm. These include prevention of exploding or vanishing gradients, robustness to different settings of hyperparameters such as learning rate and initialization scheme, and keeping most of the activations away from saturation regions of nonlinearities. All these properties are clearly beneficial to the training process. But they are fairly simple consequences of the mechanics of BatchNorm and do little to uncover the underlying factors responsible for BatchNorm’s success.
Is there a more fundamental phenomenon at play here?3.1 The smoothing effect of BatchNorm
Indeed, we identify the key impact that BatchNorm has on the training process: it reparametrizes the underlying optimization problem to make its landscape significantly more smooth. The first manifestation of this impact is improvement in the Lipschitzness^{1}^{1}1Recall that is Lipschitz if , for all and . of the loss function. That is, the loss changes at a smaller rate and the magnitudes of the gradients are smaller too. There is, however, an even stronger effect at play. Namely, BatchNorm’s reparametrization makes gradients of the loss more Lipschitz too. In other words, the loss exhibits a significantly better “effective” smoothness^{2}^{2}2Recall that is smooth if its gradient is Lipschitz. It is worth noting that, due to the existence of nonlinearities, one should not expect the smoothness to be bounded in an absolute, global sense..
These smoothening effects impact the performance of the training algorithm in a major way. To understand why, recall that in a vanilla (nonBatchNorm), deep neural network, the loss function is not only nonconvex but also tends to have a large number of “kinks”, flat regions, and sharp minima LiXTG17. This makes gradient descent–based training algorithms unstable, e.g., due to exploding or vanishing gradients, and thus highly sensitive to the choice of the learning rate and initialization.
Now, the key implication of BatchNorm’s reparametrization is that it makes the gradients more reliable and predictive. After all, improved Lipschitzness of the gradients gives us confidence that when we take a larger step in a direction of a computed gradient, this gradient direction remains a fairly accurate estimate of the actual gradient direction after taking that step. It thus enables any (gradient–based) training algorithm to take larger steps without the danger of running into a sudden change of the loss landscape such as flat region (corresponding to vanishing gradient) or sharp local minimum (causing exploding gradients). This, in turn, enables us to use a broader range of (and thus larger) learning rates (see Figure
10 in Appendix B) and, in general, makes the training significantly faster and less sensitive to hyperparameter choices. (This also illustrates how the properties of BatchNorm that we discussed earlier can be viewed as a manifestation of this smoothening effect.)3.2 Exploration of the optimization landscape
To demonstrate the impact of BatchNorm on the stability of the loss itself, i.e., its Lipschitzness, for each given step in the training process, we compute the gradient of the loss at that step and measure how the loss changes as we move in that direction – see Figure 4(a). We see that, in contrast to the case when BatchNorm is in use, the loss of a vanilla, i.e., nonBatchNorm, network has a very wide range of values along the direction of the gradient, especially in the initial phases of training. (In the later stages, the network is already close to convergence.)
Similarly, to illustrate the increase in the stability and predictiveness of the gradients, we make analogous measurements for the distance between the loss gradient at a given point of the training and the gradients corresponding to different points along the original gradient direction. Figure 4(b) shows a significant difference (close to two orders of magnitude) in such gradient predictiveness between the vanilla and BatchNorm networks, especially early in training.
To further demonstrate the effect of BatchNorm on the stability/Lipschitzness of the gradients of the loss, we plot in Figure 4(c) the “effective” smoothness of the vanilla and BatchNorm networks throughout the training. (“Effective” refers here to measuring the change of gradients as we move in the direction of the gradients.). Again, we observe consistent differences between these networks. We complement the above examination by considering linear deep networks: as shown in Figures 9 and 12 in Appendix B, the BatchNorm smoothening effect is present there as well.
Finally, we emphasize that even though our explorations were focused on the behavior of the loss along the gradient directions (as they are the crucial ones from the point of view of the training process), the loss behaves in a similar way when we examine other (random) directions too.
3.3 Is BatchNorm the best (only?) way to smoothen the landscape?
Given this newly acquired understanding of BatchNorm and the roots of its effectiveness, it is natural to wonder: Is this smoothening effect a unique feature of BatchNorm? Or could a similar effect be achieved using some other normalization schemes?
To answer this question, we study a few natural data statisticsbased normalization strategies. Specifically, we study schemes that fix the first order moment of the activations, as BatchNorm does, and then normalizes them by the average of their norm (before shifting the mean), for . Note that for these normalization schemes, the distributions of layer inputs are no longer Gaussianlike (see Figure 14). Hence, normalization with such norm does not guarantee anymore any control over the distribution moments nor distributional stability.
The results are presented in Figures 13, 12 and 12 in Appendix B. We observe that all the normalization strategies offer comparable performance to BatchNorm. In fact, for deep linear networks, –normalization performs even better than BatchNorm. Note that, qualitatively, the –normalization techniques lead to larger distributional shift (as considered in IoffeS15) than the vanilla, i.e., unnormalized, networks, yet they still yield improved optimization performance. Also, all these techniques result in an improved smoothness of the landscape that is similar to the effect of BatchNorm. (See Figures 12 and 12 of Appendix B.) This suggests that the positive impact of BatchNorm on training might be somewhat serendipitous. Therefore, it might be valuable to perform a principled exploration of the design space of normalization schemes as it can lead to better performance.
4 Theoretical Analysis
Our experiments so far suggest that BatchNorm has a fundamental effect on the optimization landscape. We now explore this phenomenon from a theoretical perspective. To this end, we consider an arbitrary linear layer in a DNN (we do not necessitate that the entire network be fully linear).
4.1 Setup
We analyze the impact of adding a single BatchNorm layer after an arbitrary fullyconnected layer at a given step during the training. Specifically, we compare the optimization landscape of the original training problem to the one that results from inserting the BatchNorm layer after the fullyconnected layer – normalizing the output of this layer (see Figure 5). Our analysis therefore captures effects that stem from the reparametrization of the landscape and not merely from normalizing the inputs .
We denote the layer weights (identical for both the standard and batchnormalized networks) as . Both networks have the same arbitrary loss function that could potentially include a number of additional nonlinear layers after the current one. We refer to the loss of the normalized network as for clarity. In both networks, we have input , and let . For networks with BatchNorm, we have an additional set of activations , which are the “whitened” version of , i.e. standardized to mean 0 and variance 1. These are then multiplied by and added to to form . We assume and to be constants for our analysis. In terms of notation, we let
denote the standard deviation (computed over the minibatch) of a batch of outputs
.4.2 Theoretical Results
We begin by considering the optimization landscape with respect to the activations . We show that batch normalization causes this landscape to be more wellbehaved, inducing favourable properties in Lipschitzcontinuity, and predictability of the gradients. We then show that these improvements in the activationspace landscape translate to favorable worstcase bounds in the weightspace landscape.
We first turn our attention to the gradient magnitude , which captures the Lipschitzness of the loss. The Lipschitz constant of the loss plays a crucial role in optimization, since it controls the amount by which the loss can change when taking a step (see Nesterov14 for details). Without any assumptions on the specific weights or the loss being used, we show that the batchnormalized landscape exhibits a better Lipschitz constant. Moreover, the Lipschitz constant is significantly reduced whenever the activations correlate with the gradient or the mean of the gradient deviates from . Note that this reduction is additive, and has effect even when the scaling of BN is identical to the original layer scaling (i.e. even when ).
Theorem 4.1 (The effect of BatchNorm on the Lipschitzness of the loss).
For a BatchNorm network with loss and an identical nonBN network with (identical) loss ,
First, note that grows quadratically in the dimension, so the middle term above is significant. Furthermore, the final inner product term is expected to be bounded away from zero, as the gradient with respect to a variable is rarely uncorrelated to the variable itself. In addition to the additive reduction, tends to be large in practice (cf. Appendix Figure 8), and thus the scaling by may contribute to the relative “flatness" we see in the effective Lipschitz constant.
We now turn our attention to the secondorder properties of the landscape. We show that when a BatchNorm layer is added, the quadratic form of the loss Hessian with respect to the activations in the gradient direction, is both rescaled by the input variance (inducing resilience to minibatch variance), and decreased by an additive factor (increasing smoothness). This term captures the second order term of the Taylor expansion of the gradient around the current point. Therefore, reducing this term implies that the first order term (the gradient) is more predictive.
Theorem 4.2 (The effect of BN to smoothness).
Let and be the gradient and Hessian of the loss with respect to the layer outputs respectively. Then
If we also have that the preserves the relative norms of and ,
Note that if the quadratic forms involving the Hessian and the inner product
are nonnegative (both fairly mild assumptions), the theorem implies more predictive gradients. The Hessian is positive semidefinite (PSD) if the loss is locally convex which is true for the case of deep networks with piecewiselinear activation functions and a convex loss at the final layer (e.g. standard softmax crossentropy loss or other common losses). The condition
holds as long as the negative gradient is pointing towards the minimum of the loss (w.r.t. normalized activations). Overall, as long as these two conditions hold, the steps taken by the BatchNorm network are more predictive than those of the standard network (similarly to what we observed experimentally).Note that our results stem from the reparametrization of the problem and not a simple scaling.
Observation 4.3 (BatchNorm does more than rescaling).
For any input data and network configuration , there exists a BN configuration that results in the same activations , and where . Consequently, all of the minima of the normal landscape are preserved in the BN landscape.
Our theoretical analysis so far studied the optimization landscape of the loss w.r.t. the normalized activations. We will now translate these bounds to a favorable worstcase bound on the landscape with respect to layer weights. Note that a (near exact) analogue of this theorem for minimax gradient predictiveness appears in Theorem C.1 of Appendix C.
Theorem 4.4 (Minimax bound on weightspace Lipschitzness).
For a BatchNorm network with loss and an identical nonBN network (with identical loss ), if
Finally, in addition to a desirable landscape, we find that BN also offers an advantage in initialization:
Lemma 4.5 (BatchNorm leads to a favourable initialization).
Let and be the set of local optima for the weights in the normal and BN networks, respectively. For any initialization
if , where and are closest optima for BN and standard network, respectively.
5 Related work
A number of normalization schemes have been proposed as alternatives to BatchNorm, including normalization over layers BaKH16, subsets of the batch WuHe18, or across image dimensions UlyanovVL16. Weight Normalization SalimansK16 follows a complementary approach normalizing the weights instead of the activations. Finally, ELU ClevertUH16 and SELU KlambauerUMH17 are two proposed examples of nonlinearities that have a progressively decaying slope instead of a sharp saturation and can be used as an alternative for BatchNorm. These techniques offer an improvement over standard training that is comparable to that of BatchNorm but do not attempt to explain BatchNorm’s success.
Additionally, work on topics related to DNN optimization has uncovered a number of other BatchNorm benefits. Li et al. ImTB16 observe that networks with BatchNorm tend to have optimization trajectories that rely less on the parameter initialization. Balduzzi et al. BalduzziFLLMM17 observe that models without BatchNorm tend to suffer from small correlation between different gradient coordinates and/or unit activations. They report that this behavior is profound in deeper models and argue how it constitutes an obstacle to DNN optimization. Morcos et al. MorcosBRB18 focus on the generalization properties of DNN. They observe that the use of BatchNorm results in models that rely less on single directions in the activation space, which they find to be connected to the generalization properties of the model.
Recent work KohlerDLZNH18 identifies simple, concrete settings where a variant of training with BatchNorm provably improves over standard training algorithms. The main idea is that decoupling the length and direction of the weights (as done in BatchNorm and Weight Normalization SalimansK16) can be exploited to a large extent. By designing algorithms that optimize these parameters separately, with (different) adaptive step sizes, one can achieve significantly faster convergence rates for these problems.
6 Conclusions
In this work, we have investigated the roots of BatchNorm’s effectiveness as a technique for training deep neural networks. We find that the widely believed connection between the performance of BatchNorm and the internal covariate shift is tenuous, at best. In particular, we demonstrate that existence of internal covariate shift, at least when viewed from the – generally adopted – distributional stability perspective, is not a good predictor of training performance. Also, we show that, from an optimization viewpoint, BatchNorm might not be even reducing that shift.
Instead, we identify a key effect that BatchNorm has on the training process: it reparametrizes the underlying optimization problem to make it more stable (in the sense of loss Lipschitzness) and smooth (in the sense of “effective” smoothness of the loss). This implies that the gradients used in training are more predictive and wellbehaved, which enables faster and more effective optimization. This phenomena also explains and subsumes some of the other previously observed benefits of BatchNorm, such as robustness to hyperparameter setting and avoiding gradient explosion/vanishing. We also show that this smoothing effect is not unique to BatchNorm. In fact, several other natural normalization strategies have similar impact and result in a comparable performance gain.
We believe that these findings not only challenge the conventional wisdom about BatchNorm but also bring us closer to a better understanding of this technique. We also view these results as an opportunity to encourage the community to pursue a more systematic investigation of the algorithmic toolkit of deep learning and the underpinnings of its effectiveness.
Finally, our focus here was on the impact of BatchNorm on training but our findings might also shed some light on the BatchNorm’s tendency to improve generalization. Specifically, it could be the case that the smoothening effect of BatchNorm’s reparametrization encourages the training process to converge to more flat minima. Such minima are believed to facilitate better generalization HochreiterS96; KeskarMNST16. We hope that future work will investigate this intriguing possibility.
Acknowledgements
We thank Ali Rahimi and Ben Recht for helpful comments on a preliminary version of this paper.
Shibani Santurkar was supported by the National Science Foundation (NSF) under grants IIS1447786, IIS1607189, and CCF1563880, and the Intel Corporation. Dimitris Tsipras was supported in part by the NSF grant CCF1553428 and the NSF Frontier grant CNS1413920. Andrew Ilyas was supported in part by NSF awards CCF1617730 and IIS1741137, a Simons Investigator Award, a Google Faculty Research Award, and an MITIBM Watson AI Lab research grant. Aleksander Mądry was supported in part by an Alfred P. Sloan Research Fellowship, a Google Research Award, and the NSF grants CCF1553428 and CNS1815221.
References
Appendix A Experimental Setup
In Section A.1, we provide details regarding the architectures used in our analysis. Then in Section A.2 we discuss the specifics of the setup and measurements used in our experiments.
a.1 Models
We use two standard deep architectures – a VGGlike network, and a deep linear network (DLN). The VGG model achieves close to stateoftheart performance while being fairly simple^{3}^{3}3We choose to not experiment with ResNets [HeZRS16] since they seem to provide several similar benefits to BatchNorm [HardtM17] and would introduce conflating factors into our study.. Preliminary experiments on other architectures gave similar results. We study DLNs with fullbatch training since they allow us to isolate the effect of nonlinearities, as well as the stochasticity of the training procedure. Both these architectures show clear a performance benefits with BatchNorm.
Specific details regarding both architectures are provided below:

Convolutional VGG architecture on CIFAR10 (VGG):
We fit a VGGlike network, a standard convolutional architecture [SimonyanZ15], to a canonical image classification problem (CIFAR10 [Krizhevsky09]
). We optimize using standard stochastic gradient descent and train for
steps (training accuracy plateaus). We use a batch size of and a fixed learning rate of unless otherwise specified. Moreover, since our focus is on training, we do not use data augmentation. This architecture can fit the training dataset well and achieves close to stateofthe art test performance. Our network achieves a test accuracy of 83% with BatchNorm and 80% without (this becomes 92% and 88% respectively with data augmentation). 
Layer Deep Linear Network on Synthetic Gaussian Data (DLN):
DLN are a factorized approach to solving a simple regression problem, i.e., fitting from . Specifically, we consider a deep network with fully connected layers and an loss. Thus, we are minimizing over ^{4}^{4}4While the factorized formulation is equivalent to a single matrix in terms of expressivity, the optimization landscape is drastically different [HardtM17].. We generate inputs
from a Gaussian Distribution and a matrix
with i.i.d. Gaussian entries. We choose to be 25, and the dimensions of to be . All the matrices are square and have the same dimensions. We train DLN using fullbatch gradient descent for steps (training loss plateaus).The size of the dataset is (same as the batch size) and the learning rate is unless otherwise specified.
For both networks we use standard Glorot initialization [GlorotB10]. Further the learning rates were selected based on hyperparameter optimization to find a configuration where the training performance for the network was the best.
a.2 Details
a.2.1 “Noisy” BatchNorm Layers
Consider , the th activation of the th example in the batch. Note that batch norm will ensure that the distribution of for some will have fixed mean and variance (possibly learnable).
At every time step, our noise model consists of perturbing each activation for each sample in a batch with noise i.i.d. from a nonzero mean, nonunit variance distribution . The distribution itself is time varying and its parameters are drawn i.i.d from another distribution . The specific noise model is described in Algorithm 1. In our experiments, , and . (For convolutional layers, we follow the standard convention of treating the height and width dimensions as part of the batch.)
While plotting the distribution of activations, we sample random activations from any given layer of the network and plot its distribution over the batch dimension for fully connected layers, and over the batch, height, width dimension for convolutional layers as is standard convention in BatchNorm for convolutional networks.
a.2.2 Loss Landscape
To measure the smoothness of the loss landscape of a network during the course of training, we essentially take steps of different lengths in the direction of the gradient and measure the loss values obtained at each step. Note that this is not a training procedure, but an evaluation of the local loss landscape at every step of the training process.
For VGG we consider steps of length ranging from , whereas for DLN we choose . Here step size
denotes the hyperparameter setting with which the network is being trained. We choose these ranges to roughly reflect the range of parameters that are valid for standard training of these models. The VGG network is much more sensitive to the learning rate choices (probably due to the nonlinearities it includes), so we perform line search over a restricted range of parameters. Further, the maximum step size was chosen slightly smaller than the learning rate at which the standard (no BatchNorm) network diverges during training.
Appendix B Omitted Figures
Additional visualizations for the analysis performed in Section 3.1 are presented below.
(a) 
(b) 
(a) VGG 
(b) DLN 
(a) VGG  (b) Deep Linear Model 
Appendix C Proofs
We now prove the stated theorems regarding the landscape induced by batch normalization.
We begin with a few facts that can be derived directly from the closedform of Batch Normalization, which we use freely in proving the following theorems.
c.1 Useful facts and setup
We consider the same setup pictured in Figure 5 and described in Section 4.1. Note that in proving the theorems we use partial derivative notation instead of gradient notation, and also rely on a few simple but key facts:
Fact C.1 (Gradient through BatchNorm).
The gradient through BN and another function , where , and where are scalar elements of a batch of size and variance is
Fact C.2 (Gradients of normalized outputs).
A convenient gradient of BN is given as
(1) 
and thus
(2) 
c.2 Lipschitzness proofs
Now, we provide a proof for the Lipschitzness of the loss landscape in terms of the layer activations. In particular, we prove the following theorem from Section 4.
See 4.1
Proof.
Proving this is simply a direct application of Fact C.1. In particular, we have that
(3) 
which we can write in vectorized form as
(4) 
Now, let be the mean of the gradient vector, we can rewrite the above as the following (in the subsequent steps taking advantage of the fact that is meanzero and norm:
(5)  
(6)  
(7)  
(8)  
(9) 
Exploiting the fact that gives the desired result. ∎
Next, we can use this to prove the minimax bound on the Lipschitzness with respect to the weights.
See 4.4
Proof.
To prove this, we start with the following identity for the largest eigenvalue
of :(10) 
which in turn implies that for a matrix with , it must be that , with the choice of making this bound tight.
Now, we derive the gradient with respect to the weights via the chain rule:
(11)  
(12)  
(13)  
(14) 
where is the input matrix holding . Thus,
(15) 
and since we have , we must have , and so recalling (10),
(16) 
and applying Theorem 4.1 yields:
(17) 
Finally, by applying (10) again, note that in fact in the normal network,
(18) 
and thus
∎
See 4.2
Proof.
We use the following notation freely in the following. First, we introduce the hessian with respect to the final activations as:
where the final equality is by the assumptions of our setup. Once again for convenience, we define a function which operates on vectors and matrices and gives their elementwise mean; in particular, for and we write to be a vector with all elements equal to . Finally, we denote the gradient with respect to the batchnormalized outputs as , such that:
where again the last equality is by assumption.
Now, we begin by looking at the Hessian of the loss with respect to the preBN activations using the expanded gradient as above:
(19)  
Using the product rule and the chain rule:  
(20)  
(21)  
Distributing the derivative across subtraction:  
(22)  
(23) 
We address each of the terms in the above (22) and (23) one by one:
(24) 
(25)  
(26)  
(27)  
(28)  
(29)  
(30) 
Now, we can use the preceding to rewrite the Hessian as:
(31)  
(32) 
Now, using Fact C.2, we have that:
(33) 
and substituting this yields (letting for convenience):
(34)  
(35)  
(36)  
(37) 
Collecting the terms, and letting :
(38)  
(39)  
(40)  
(41)  
(42) 
Now, we wish to calculate the effective beta smoothness with respect to a batch of activations, which corresponds to , where is the gradient with respect to the activations (as derived in the previous proof). We expand this product noting the following identities:
(43)  
(44)  
(45)  
Comments
There are no comments yet.