NormalizationLayer
represents a trainable net layer that normalizes its input data across the second and subsequent dimensions and applies an independent scaling and bias to each component of the first dimension.
NormalizationLayer[aggregationlevels]
normalizes its input data across the specified aggregation levels and applies a learned scaling and bias on the remaining levels.
NormalizationLayer[aggregationlevels,scalinglevels]
applies a learned scaling and bias at the specified scaling levels.
Details and Options




- In NormalizationLayer[aggregationlevels], the aggregation levels determine which levels of the input are aggregated over to compute the mean and variance statistics used to normalize. The aggregation levels can be specified using the following settings:
-
a aggregate level a only a1;;a2 aggregate levels a1 through a2 {a,b,…} aggregate levels a,b,… - In NormalizationLayer[], the aggregation levels used are 2;;All.
- In NormalizationLayer[aggregationlevels,scalinglevels], the scaling levels determine which levels of the input have componentwise specific scaling values and biases applied to them. The scaling levels can be specified using the following settings:
-
"Complement" all levels except the aggregation levels (default) "Same" the same levels as the aggregation levels i level i only i1;;i2 levels i1 through i2 {i,j,…} levels i,j,… - The following optional parameters can be included:
-
"Epsilon" 0.001 stability parameter LearningRateMultipliers Automatic learning rate multipliers for the scaling and/or bias parameters - The following learnable parameters can be specified:
-
"Biases" Automatic learnable bias parameters "Scaling" Automatic learnable scaling parameters - With Automatic settings, scaling and bias parameters are initialized automatically when NetInitialize or NetTrain is used. With None settings, scaling and adding biases can be disabled.
- If scaling and bias parameters have been initialized or disabled, NormalizationLayer[…][input] explicitly computes the output from applying the layer.
- NormalizationLayer[…][input] explicitly computes the output from applying the layer to input.
- NormalizationLayer[…][{input1,input2,…}] explicitly computes outputs for each of the inputi.
- When it cannot be inferred from other layers in a larger net, the option "Input"->{d1,d2,…} can be used to fix the input dimensions of NormalizationLayer.
- NetExtract can be used to extract the scaling, biases and epsilon parameters from a NormalizationLayer object.
- NormalizationLayer is typically used inside NetChain, NetGraph, etc.
- NormalizationLayer exposes the following ports for use in NetGraph etc.:
-
"Input" an array of rank greater than 1 "Output" an array of rank greater than 1 - Consider an input array
that can be decomposed into da×db×… subarrays xA across the aggregation levels {a,b,…} and into di×dj×… subarrays xI across the scaling levels {i,j,…}. The scaling and biases parameters
and
have dimensions (di×dj×…). Then the output, with the same dimensions as the input array, is obtained via
, where the mean arrays are given by
and the variance by
.
- The default NormalizationLayer[] corresponds to instance normalization in Ulyanov et al., "Instance Normalization: The Missing Ingredient for Fast Stylization", 2016, with the channel dimension at the first level and spatial dimensions at subsequent levels.
- NormalizationLayer[2;;All,"Same","Input"{"Varying",Automatic}] corresponds to layer normalization in Ba et al., "Layer Normalization", 2016, with the channel dimension at the last level and the time dimension at the first level.
- Options[NormalizationLayer] gives the list of default options to construct the layer. Options[NormalizationLayer[…]] gives the list of default options to evaluate the layer on some data.
- Information[NormalizationLayer[…]] gives a report about the layer.
- Information[NormalizationLayer[…],prop] gives the value of the property prop of NormalizationLayer[…]. Possible properties are the same as for NetGraph.
Examples
open allclose allBasic Examples (2)
Create a NormalizationLayer:
Create an initialized NormalizationLayer that takes a rank-3 array:
The output is a list of arrays with values centered around zero:
The standard deviations of the output arrays are roughly unitary:
Scope (3)
Create an instance normalization layer that takes an RGB image and returns an RGB image:
NormalizationLayer automatically threads over batches of inputs:
Create a NormalizationLayer that takes a sequence of two-dimensional vectors and returns the sequence of normalized vectors scaled by a factor of 2 and shifted by :
Apply layer normalization to some sinusoidal input:
Visualize the values before and after normalization:
Create an initialized NormalizationLayer that takes a rank-6 array and normalize across all dimensions:
Options (5)
"Biases" (2)
Create a NormalizationLayer with the biases explicitly specified:
Create a NormalizationLayer without any bias:
"Epsilon" (1)
Create a NormalizationLayer with the "Epsilon" parameter explicitly specified:
"Scaling" (2)
Create a NormalizationLayer with the scaling parameter explicitly specified:
Extract the scaling parameters:
Create a NormalizationLayer without any scaling:
Possible Issues (1)
NormalizationLayer cannot be initialized until all its input and output dimensions are known:

Text
Wolfram Research (2019), NormalizationLayer, Wolfram Language function, https://reference.wolfram.com/language/ref/NormalizationLayer.html (updated 2020).
CMS
Wolfram Language. 2019. "NormalizationLayer." Wolfram Language & System Documentation Center. Wolfram Research. Last Modified 2020. https://reference.wolfram.com/language/ref/NormalizationLayer.html.
APA
Wolfram Language. (2019). NormalizationLayer. Wolfram Language & System Documentation Center. Retrieved from https://reference.wolfram.com/language/ref/NormalizationLayer.html