BasicRecurrentLayer

BasicRecurrentLayer[n]

represents a trainable recurrent layer that takes a sequence of vectors and produces a sequence of vectors each of size n.

BasicRecurrentLayer[n,opts]

includes options for initial weights and other parameters.

Details and Options

  • BasicRecurrentLayer[n] represents a net that takes an input matrix representing a sequence of vectors and outputs a sequence of the same length.
  • Each element of the input sequence is a vector of size k, and each element of the output sequence is a vector of size n.
  • The size k of the input vectors is usually inferred automatically within a NetGraph, NetChain, etc.
  • The input and output ports of the net represented by BasicRecurrentLayer[n] are:
  • "Input"a sequence of vectors of size k
    "Output"a sequence of vectors of size n
  • Given an input sequence {x1,x2,…,xT}, a BasicRecurrentLayer outputs a sequence of states {s1,s2,…,sT}, where st=Tanh[wi.xt+ws.st-1+b].
  • BasicRecurrentLayer[n] has a single state port, "State", which is a vector of size n.
  • Within a NetGraph, a connection of the form src->NetPort[layer,"State"] can be used to provide the initial value of the state for a BasicRecurrentLayer, corresponding to s0 in the recurrence relation. The default initial value is a zero vector.
  • Within a NetGraph, a connection of the form NetPort[layer,"State"]->dst can be used to obtain the final value of the state for a BasicRecurrentLayer, corresponding to sT in the recurrence relation.
  • NetStateObject can be used to create a net that will remember values for the state of BasicRecurrentLayer that update when the net is applied to inputs.
  • An initialized BasicRecurrentLayer[…] that operates on vectors of size k contains the following trainable arrays:
  • "InputWeights"Wxmatrix of size n×k
    "StateWeights"Wsmatrix of size n×n
    "Biases"bvector of size n
  • In BasicRecurrentLayer[n,opts], initial values can be given to the trainable arrays using a rule of the form "array"->value.
  • The following training parameters can be included:
  • "Dropout" Nonedropout regularization, in which units are probabilistically set to zero
    LearningRateMultipliersAutomaticlearning rate multipliers for trainable arrays
  • Specifying "Dropout"->None disables dropout during training.
  • Specifying "Dropout"->p uses an automatically chosen dropout method having dropout probability p.
  • Specifying "Dropout"->{"method1"->p1,"method2"->p2,…} can be used to combine specific methods of dropout with the corresponding dropout probabilities. Possible methods include:
  • "VariationalWeights"dropout applied to the recurrent connections between weight matrices (default)
    "VariationalInput"dropout applied to the gate contributions from the input, using the same pattern of units at each sequence step
    "VariationalState"dropout applied to the gate contributions from the previous state, using the same pattern of units at each sequence step
    "StateUpdate"dropout applied to the state update vector prior to it being added to the previous state, using a different pattern of units at each sequence step
  • The dropout methods "VariationalInput" and "VariationalState" are based on the Gal et al. 2016 method, while "StateUpdate" is based on the Semeniuta et al. 2016 method and "VariationalWeights" is based on the Merity et al. 2017 method.
  • BasicRecurrentLayer[n,"Input"->shape] allows the shape of the input to be specified. Possible forms for shape are:
  • NetEncoder[…]encoder producing a sequence of vectors
    {len,k}sequence of len length-k vectors
    {len,Automatic}sequence of len vectors whose length is inferred
    {"Varying",k}varying number of vectors each of length k
    {"Varying",Automatic}varying number of vectors each of inferred length
  • When given a NumericArray as input, the output will be a NumericArray.
  • Options[BasicRecurrentLayer] gives the list of default options to construct the layer. Options[BasicRecurrentLayer[…]] gives the list of default options to evaluate the layer on some data.
  • Information[BasicRecurrentLayer[…]] gives a report about the layer.
  • Information[BasicRecurrentLayer[…],prop] gives the value of the property prop of BasicRecurrentLayer[…]. Possible properties are the same as for NetGraph.

Examples

open allclose all

Basic Examples  (2)

Create a BasicRecurrentLayer that produces a sequence of length-3 vectors:

Create a randomly initialized BasicRecurrentLayer that takes a sequence of length-2 vectors and produces a sequence of length-3 vectors:

Apply the layer to an input sequence:

Scope  (4)

Create a randomly initialized BasicRecurrentLayer that takes a string and produces a sequence of length-2 vectors:

Apply the layer to an input string:

Thread the layer over a batch of inputs:

Create a randomly initialized net that takes a sequence of length-2 vectors and produces a single length-3 vector:

Apply the layer to an input:

Thread the layer across a batch of inputs:

Create a NetGraph that allows the initial state of a BasicRecurrentLayer to be set:

Create a NetGraph that allows the final state of a BasicRecurrentLayer to be obtained:

The final state is the last element of the output sequence:

Options  (2)

"Dropout"  (2)

Create a BasicRecurrentLayer with the dropout method specified:

Create a randomly initialized BasicRecurrentLayer with specified dropout probability:

Evaluate the layer on a sequence of vectors:

Dropout has no effect during evaluation:

Use NetEvaluationMode to force the training behavior of dropout:

Multiple evaluations on the same input can give different results:

Applications  (2)

Create training data consisting of strings that describe two-digit additions and the corresponding numeric result:

Create a network using stacked BasicRecurrentLayer layers that reads the input string and predicts the numeric result:

Train the network:

Apply the trained network to a list of inputs:

Create training data based on strings containing x's and y's, and either Less, Greater or Equal by comparing the number of x's and y's. The training data consists of all possible sentences up to length 8:

Create a network containing a BasicRecurrentLayer to read an input string and predict one of Less, Greater or Equal:

Train the network:

Apply the trained network to a list of inputs:

Measure the accuracy on the entire training set:

Properties & Relations  (1)

NetStateObject can be used to create a net that remembers the state of BasicRecurrentLayer:

Each evaluation modifies the state stored inside the NetStateObject:

Wolfram Research (2017), BasicRecurrentLayer, Wolfram Language function, https://reference.wolfram.com/language/ref/BasicRecurrentLayer.html (updated 2020).

Text

Wolfram Research (2017), BasicRecurrentLayer, Wolfram Language function, https://reference.wolfram.com/language/ref/BasicRecurrentLayer.html (updated 2020).

CMS

Wolfram Language. 2017. "BasicRecurrentLayer." Wolfram Language & System Documentation Center. Wolfram Research. Last Modified 2020. https://reference.wolfram.com/language/ref/BasicRecurrentLayer.html.

APA

Wolfram Language. (2017). BasicRecurrentLayer. Wolfram Language & System Documentation Center. Retrieved from https://reference.wolfram.com/language/ref/BasicRecurrentLayer.html

BibTeX

@misc{reference.wolfram_2023_basicrecurrentlayer, author="Wolfram Research", title="{BasicRecurrentLayer}", year="2020", howpublished="\url{https://reference.wolfram.com/language/ref/BasicRecurrentLayer.html}", note=[Accessed: 19-April-2024 ]}

BibLaTeX

@online{reference.wolfram_2023_basicrecurrentlayer, organization={Wolfram Research}, title={BasicRecurrentLayer}, year={2020}, url={https://reference.wolfram.com/language/ref/BasicRecurrentLayer.html}, note=[Accessed: 19-April-2024 ]}