NormalizationLayer

NormalizationLayer[]

represents a trainable net layer that normalizes its input data across the second and subsequent dimensions and applies an independent scaling and bias to each component of the first dimension.

NormalizationLayer[aggregationlevels]

normalizes its input data across the specified aggregation levels and applies a learned scaling and bias on the remaining levels.

NormalizationLayer[aggregationlevels,scalinglevels]

applies a learned scaling and bias at the specified scaling levels.

Details and Options

  • In NormalizationLayer[aggregationlevels], the aggregation levels determine which levels of the input are aggregated over to compute the mean and variance statistics used to normalize. The aggregation levels can be specified using the following settings:
  • aaggregate level a only
    a1;;a2aggregate levels a1 through a2
    {a,b,…}aggregate levels a,b,…
  • In NormalizationLayer[], the aggregation levels used are 2;;All.
  • In NormalizationLayer[aggregationlevels,scalinglevels], the scaling levels determine which levels of the input have componentwise specific scaling values and biases applied to them. The scaling levels can be specified using the following settings:
  • "Complement"all levels except the aggregation levels (default)
    "Same"the same levels as the aggregation levels
    ilevel i only
    i1;;i2levels i1 through i2
    {i,j,…}levels i,j,…
  • The following optional parameters can be included:
  • "Epsilon" 0.001stability parameter
    LearningRateMultipliersAutomaticlearning rate multipliers for the scaling and/or bias parameters
  • The following learnable parameters can be specified:
  • "Biases"Automaticlearnable bias parameters
    "Scaling"Automaticlearnable scaling parameters
  • With Automatic settings, scaling and bias parameters are initialized automatically when NetInitialize or NetTrain is used. With None settings, scaling and adding biases can be disabled.
  • If scaling and bias parameters have been initialized or disabled, NormalizationLayer[…][input] explicitly computes the output from applying the layer.
  • NormalizationLayer[…][input] explicitly computes the output from applying the layer to input.
  • NormalizationLayer[…][{input1,input2,…}] explicitly computes outputs for each of the inputi.
  • When it cannot be inferred from other layers in a larger net, the option "Input"->{d1,d2,…} can be used to fix the input dimensions of NormalizationLayer.
  • NetExtract can be used to extract the scaling, biases and epsilon parameters from a NormalizationLayer object.
  • NormalizationLayer is typically used inside NetChain, NetGraph, etc.
  • NormalizationLayer exposes the following ports for use in NetGraph etc.:
  • "Input"an array of rank greater than 1
    "Output"an array of rank greater than 1
  • Consider an input array that can be decomposed into da×db×… subarrays xA across the aggregation levels {a,b,…} and into di×dj×… subarrays xI across the scaling levels {i,j,…}. The scaling and biases parameters and have dimensions (di×dj×…). Then the output, with the same dimensions as the input array, is obtained via , where the mean arrays are given by and the variance by .
  • The default NormalizationLayer[] corresponds to instance normalization in Ulyanov et al., "Instance Normalization: The Missing Ingredient for Fast Stylization", 2016, with the channel dimension at the first level and spatial dimensions at subsequent levels.
  • NormalizationLayer[2;;All,"Same","Input"{"Varying",Automatic}] corresponds to layer normalization in Ba et al., "Layer Normalization", 2016, with the channel dimension at the last level and the time dimension at the first level.
  • Options[NormalizationLayer] gives the list of default options to construct the layer. Options[NormalizationLayer[…]] gives the list of default options to evaluate the layer on some data.
  • Information[NormalizationLayer[…]] gives a report about the layer.
  • Information[NormalizationLayer[…],prop] gives the value of the property prop of NormalizationLayer[…]. Possible properties are the same as for NetGraph.

Examples

open allclose all

Basic Examples  (2)

Create a NormalizationLayer:

Create an initialized NormalizationLayer that takes a rank-3 array:

Apply the layer to an input:

The output is a list of arrays with values centered around zero:

The standard deviations of the output arrays are roughly unitary:

Scope  (3)

Create an instance normalization layer that takes an RGB image and returns an RGB image:

Apply the layer to an image:

NormalizationLayer automatically threads over batches of inputs:

Create a NormalizationLayer that takes a sequence of two-dimensional vectors and returns the sequence of normalized vectors scaled by a factor of 2 and shifted by :

Apply layer normalization to some sinusoidal input:

Visualize the values before and after normalization:

Create an initialized NormalizationLayer that takes a rank-6 array and normalize across all dimensions:

Apply the layer to an input:

Options  (5)

"Biases"  (2)

Create a NormalizationLayer with the biases explicitly specified:

Extract the biases:

Create a NormalizationLayer without any bias:

The layer does not have any learnable bias:

"Epsilon"  (1)

Create a NormalizationLayer with the "Epsilon" parameter explicitly specified:

Extract the "Epsilon" parameter:

"Scaling"  (2)

Create a NormalizationLayer with the scaling parameter explicitly specified:

Extract the scaling parameters:

Create a NormalizationLayer without any scaling:

The layer does not have any learnable scaling parameter:

Possible Issues  (1)

NormalizationLayer cannot be initialized until all its input and output dimensions are known:

Wolfram Research (2019), NormalizationLayer, Wolfram Language function, https://reference.wolfram.com/language/ref/NormalizationLayer.html (updated 2020).

Text

Wolfram Research (2019), NormalizationLayer, Wolfram Language function, https://reference.wolfram.com/language/ref/NormalizationLayer.html (updated 2020).

CMS

Wolfram Language. 2019. "NormalizationLayer." Wolfram Language & System Documentation Center. Wolfram Research. Last Modified 2020. https://reference.wolfram.com/language/ref/NormalizationLayer.html.

APA

Wolfram Language. (2019). NormalizationLayer. Wolfram Language & System Documentation Center. Retrieved from https://reference.wolfram.com/language/ref/NormalizationLayer.html

BibTeX

@misc{reference.wolfram_2022_normalizationlayer, author="Wolfram Research", title="{NormalizationLayer}", year="2020", howpublished="\url{https://reference.wolfram.com/language/ref/NormalizationLayer.html}", note=[Accessed: 06-June-2023 ]}

BibLaTeX

@online{reference.wolfram_2022_normalizationlayer, organization={Wolfram Research}, title={NormalizationLayer}, year={2020}, url={https://reference.wolfram.com/language/ref/NormalizationLayer.html}, note=[Accessed: 06-June-2023 ]}