NormalizationLayer

NormalizationLayer[]

represents a trainable net layer that normalizes its input data across the second and subsequent dimensions and applies an independent scaling and bias to each component of the first dimension.

NormalizationLayer[aggregationlevels]

normalizes its input data across the specified aggregation levels and applies a learned scaling and bias on the remaining levels.

NormalizationLayer[aggregationlevels,scalinglevels]

applies a learned scaling and bias at the specified scaling levels.

Details and Options

  • In NormalizationLayer[aggregationlevels], the aggregation levels determine which levels of the input are aggregated over to compute the mean and variance statistics used to normalize. The aggregation levels can be specified using the following settings:
  • aaggregate level a only
    a1;;a2aggregate levels a1 through a2
    {a,b,}aggregate levels a,b,
  • In NormalizationLayer[], the aggregation levels used are 2;;All.
  • In NormalizationLayer[aggregationlevels,scalinglevels], the scaling levels determine which levels of the input have component-wise specific scaling values and biases applied to them. The scaling levels can be specified using the following settings:
  • "Complement"all levels except the aggregation levels (default)
    "Same"the same levels as the aggregation levels
    ilevel i only
    i1;;i2levels i1 through i2
    {i,j,}levels i,j,.
  • The following optional parameters can be included:
  • "Epsilon"0.001`stability parameter
  • The following learnable parameters can be specified:
  • "Biases"Automaticlearnable bias parameters
    "Scaling"Automaticlearnable scaling parameters
  • With Automatic settings, scaling and bias parameters are initialized automatically when NetInitialize or NetTrain is used. With None settings, scaling and adding biases can be disabled.
  • If scaling and bias parameters have been initialized or disabled, NormalizationLayer[][input] explicitly computes the output from applying the layer.
  • NormalizationLayer[][input] explicitly computes the output from applying the layer to input.
  • NormalizationLayer[][{input1,input2,}] explicitly computes outputs for each of the inputi.
  • When given a NumericArray as input, the output will be a NumericArray.
  • When it cannot be inferred from other layers in a larger net, the option "Input"->{d1,d2,} can be used to fix the input dimensions of NormalizationLayer.
  • NetExtract can be used to extract the scaling, biases and epsilon parameters from a NormalizationLayer object.
  • NormalizationLayer is typically used inside NetChain, NetGraph, etc.
  • NormalizationLayer exposes the following ports for use in NetGraph etc.:
  • "Input"an array of rank greater than 1
    "Output"an array of rank greater than 1
  • Consider an input array that can be decomposed into da×db× subarrays xA across the aggregation levels {a,b,} and into di×dj× subarrays xI across the scaling levels {i,j,}. The scaling and biases parameters and have dimensions (di×dj×). Then the output, with the same dimensions as the input array, is obtained via , where the mean arrays are given by and the variance by .
  • The default NormalizationLayer[] corresponds to instance normalization in Ulyanov et al., "Instance Normalization: The Missing Ingredient for Fast Stylization", 2016, with the channel dimension at the first level and spatial dimensions at subsequent levels.
  • NormalizationLayer[2;;All,"Same","Input"{"Varying",Automatic}] corresponds to layer normalization in Ba et al., "Layer Normalization", 2016, with the channel dimension at the last level and the time dimension at the first level.

Examples

open all close all

Basic Examples  (2)

Create a NormalizationLayer:

In[1]:=
Click for copyable input
Out[1]=

Create an initialized NormalizationLayer that takes a rank-3 array:

In[1]:=
Click for copyable input
Out[1]=

Apply the layer to an input:

In[2]:=
Click for copyable input
In[3]:=
Click for copyable input
Out[3]=

The output is a list of arrays with values centered around zero:

In[4]:=
Click for copyable input
Out[4]=

The standard deviations of the output arrays are roughly unitary:

In[5]:=
Click for copyable input
Out[5]=

Scope  (3)

Options  (5)

Possible Issues  (1)

Introduced in 2019
(12.0)