EmbeddingLayer

EmbeddingLayer[size,n]

represents a trainable net layer that embeds integers between 1 and n into a continuous vector space of dimension size.

EmbeddingLayer[size]

leaves the n to be inferred from context.

Details and Options

  • EmbeddingLayer operates on a single integer or rank-k array of integers, producing a vector or rank-k+1 array, respectively.
  • Specifying EmbeddingLayer[size] will produce a net that infers the n when a NetEncoder[] that produces integers is connected to its input.
  • The embedding performed by EmbeddingLayer is learned during training.
  • The following optional parameters can be included:
  • LearningRateMultipliersAutomaticlearning rate multipliers for the weights
    "Weights"Automaticinitial matrix of weights of dimensions n×size
  • With Automatic settings, the weights are added automatically when NetInitialize or NetTrain is used.
  • If weights have been added, EmbeddingLayer[][input] explicitly computes the output from applying the layer.
  • EmbeddingLayer[][{input1,input2,}] explicitly computes outputs for each of the inputi.
  • When given a NumericArray as input, the output will be a NumericArray.
  • NetExtract can be used to extract weights from an EmbeddingLayer object.
  • EmbeddingLayer is typically used inside NetChain, NetGraph, etc.
  • EmbeddingLayer exposes the following ports for use in NetGraph etc.:
  • "Input"an integer or rank-k array of integers
    "Output"a vector or rank-k+1 array
  • EmbeddingLayer[size,n,"Input"->shape] allows the shape of the input to be specified. Possible forms for shape are:
  • NetEncoder[]encoder producing an integer or array of integers
    "Integer"a single integer
    da vector of integers of length d
    {d1,d2,}an array of integers of dimensions d1×d2×
    "Varying"a variable-length vector of integers
    {"Varying",d2,d3,}an array whose first dimension is variable and whose remaining dimensions are d2×d3×
  • Options[EmbeddingLayer] gives the list of default options to construct the layer. Options[EmbeddingLayer[]] gives the list of default options to evaluate the layer on some data.
  • Information[EmbeddingLayer[]] gives a report about the layer.
  • Information[EmbeddingLayer[],prop] gives the value of the property prop of EmbeddingLayer[]. Possible properties are the same as for NetGraph.

Examples

open allclose all

Basic Examples  (2)

Create an EmbeddingLayer that will take integers 1, 2 and 3 and produces vectors of size 4:

Create a randomly initialized EmbeddingLayer that will take integers 1, 2 and 3 and produces vectors of size 2:

Apply the layer to an integer:

Apply the layer to a vector of integers:

Scope  (4)

Create a randomly initialized EmbeddingLayer that embeds a variable-length sequence of integers from 1 to 10 into a sequence of vectors of length 2 represented by a matrix whose last dimension has size 2:

Apply the layer to a single sequence of integers:

Apply the layer to multiple sequences simultaneously, in which the sequence lengths are not identical:

Create an EmbeddingLayer with an encoded input, where n is inferred from the encoder:

Randomly initialize and apply the layer:

Create an EmbeddingLayer whose input is a sequence of integers representing the characters of a string:

Apply the layer to a string:

Create an EmbeddingLayer whose input is a sequence of codes representing tokens from a string:

Options  (1)

"Weights"  (1)

Create an EmbeddingLayer with weights explicitly specified:

Apply the layer to some inputs:

Applications  (2)

In text processing applications, networks that operate on entire words typically use an embedding to collapse the large vocabulary size of human languages into manageable vector sizes.

Create a chain that embeds successive words from a string into a sequence of vectors of size 10 and then processes those vectors, using a recurrent layer to produce a final vector. First create an encoder using common English words:

Create and randomly initialize the net, attaching the encoder to its input:

Randomly initialize the net and apply it to some sentences:

Obtain a pre-trained EmbeddingLayer using NetModel:

A NetEncoder has been attached to the input of the net:

Apply the embedding to a sentence containing two tokens:

Visualize the embedding of a longer sentence:

Properties & Relations  (1)

Evaluate an initialized EmbeddingLayer on a list of indices:

This is equivalent to the following:

Introduced in 2016
 (11.0)
 |
Updated in 2020
 (12.1)