LinearLayer

LinearLayer[n]

represents a trainable, fully connected net layer that computes with output vector of size n.

LinearLayer[{n1,n2,}]

represents a layer that outputs an array of dimensions n1×n2×.

LinearLayer[]

leaves the dimensions of the output array to be inferred from context.

LinearLayer[n,opts]

includes options for initial weights and other parameters.

Details and Options

  • The following optional parameters can be included:
  • "Biases"Automaticinitial vector of biases (b in w.x+b)
    "Weights"Automaticinitial matrix of weights (w in w.x+b)
    LearningRateMultipliersAutomaticlearning rate multiplier(s) to apply to weights and/or biases
  • When weights and biases are not explicitly specified or are given as Automatic, they are added automatically when NetInitialize or NetTrain is used.
  • The setting "Biases"->None specifies that no biases should be used.
  • If weights and biases have been added, LinearLayer[][input] explicitly computes the output from applying the layer.
  • LinearLayer[][{input1,input2,}] explicitly computes outputs for each of the inputi.
  • When given a NumericArray as input, the output will be a NumericArray.
  • NetExtract can be used to extract weights and biases from a LinearLayer object.
  • LinearLayer is typically used inside NetChain, NetGraph, etc.
  • LinearLayer exposes the following ports for use in NetGraph etc.:
  • "Input"an array
    "Output"an array of size n1×n2×
  • LinearLayer[{}] specifies that the LinearLayer should produce a single real number.
  • LinearLayer[n,"Input"->m] is the most common usage of LinearLayer and represents a LinearLayer that takes a vector of length m and produces a vector of length n.
  • When it cannot be inferred from previous layers in a larger net, the option "Input"shape can be used to fix the input of LinearLayer. Possible forms for shape include:
  • "Real"a single real number
    ma vector of length m
    {m1,m2,}an array of dimensions m1×m2×
    NetEncoder[]an encoder
  • Options[LinearLayer] gives the list of default options to construct the layer. Options[LinearLayer[]] gives the list of default options to evaluate the layer on some data.
  • Information[LinearLayer[]] gives a report about the layer.
  • Information[LinearLayer[],prop] gives the value of the property prop of LinearLayer[]. Possible properties are the same as for NetGraph.

Examples

open allclose all

Basic Examples  (2)

Create a LinearLayer whose output is a length-5 vector:

Create a randomly initialized LinearLayer:

Apply the layer to an input vector to produce an output vector:

Scope  (6)

Create a randomly initialized LinearLayer that takes length-3 vectors and produces 4×2 matrices:

Apply it to an input:

Create a randomly initialized LinearLayer that takes single real numbers and produces single real numbers:

Apply it to an input:

Create a randomly initialized LinearLayer that takes 3×2 matrices and produces single real numbers:

Apply it to an input:

Create a randomly initialized LinearLayer that takes a class label by using a NetEncoder:

Apply the layer to a member of the class:

The attached encoder produces 3-vectors in which one component is 1 and the others are 0:

Create a randomly initialized LinearLayer that takes an image by using a NetEncoder:

Use a specific weight matrix and bias vector:

Apply the layer to an input:

Options  (4)

"Biases"  (1)

Define and initialize a LinearLayer without biases:

This is equivalent to Dot:

LearningRateMultipliers  (2)

Create a LinearLayer with frozen weights and biases:

Train a net with this layer inside:

The weights and biases of the layer have been unchanged:

Create a LinearLayer with frozen weights, but free biases:

Train a net with this layer inside:

The weights have been unchanged, but the biases changed during training:

"Weights"  (1)

Specify a specific weight matrix, and use no biases:

Apply the layer to an input:

Extract the weight:

Applications  (1)

Create a two-layer perceptron by stacking linear layers in a NetChain:

Train the perceptron on the "MNIST" dataset of handwritten digits:

Classify unseen digits:

Properties & Relations  (2)

LinearLayer[n] can be specified as simply n in a NetChain:

For inputs and outputs that are vectors, LinearLayer computes:

Evaluate a LinearLayer on data:

Manually compute the same result:

Possible Issues  (3)

LinearLayer cannot be initialized until all its input and output dimensions are known:

LinearLayer cannot accept symbolic inputs:

A LinearLayer with output size n and input size m has a weight matrix of dimensions n×m:

If n and m are too large, there might not be enough system or GPU memory to initialize or train a net containing such a LinearLayer:

Introduced in 2016
 (11.0)
 |
Updated in 2020
 (12.1)