"Tokens" (Net Encoder)

NetEncoder["Tokens"]

represents an encoder that converts the words in a string to a sequence of integer codes using a standard English vocabulary.

NetEncoder[{"Tokens","language"}]

represents an encoder that uses a standard vocabulary for the given language.

NetEncoder[{"Tokens",{token1,token2,}}]

represents an encoder that uses a specified list of tokens as the vocabulary.

NetEncoder[{"Tokens",,"param"value}]

represents an encoder in which additional parameters have been specified.

Details

  • NetEncoder[][input] applies the encoder to an input to produce an output.
  • NetEncoder[][{input1,input2,}] applies the encoder to a list of inputs to produce a list of outputs.
  • The input to the encoder must be a string.
  • The output of the encoder is a sequence of integers between 1 and d+1, where d is the number of tokens in the vocabulary. The integer d+1 is used to signify tokens in the input that do not occur in the dictionary.
  • An encoder can be attached to an input port of a net by specifying "port"->NetEncoder[] when constructing the net.
  • Parameters
  • The following parameters can be specified:
  • "IgnoreCase"Truewhether to ignore case when matching tokens from the string
    "SplitPattern"TemplateBox[{WordBoundary, paclet:ref/WordBoundary}, RefLink, BaseStyle -> {3ColumnTableMod}]the string pattern to use in order to split the input string into tokens
    "TargetLength"Allthe length of the final sequence to crop or pad to
  • With the parameter "IgnoreCase"->True, tokens are effectively converted to lowercase before encoding.
  • With the parameter "TargetLength"->All, all tokens found in the input string are encoded.
  • With the parameter "TargetLength"->n, the first n tokens found in the input string are encoded, with padding applied if fewer than n tokens are found. The padding value is d+1, where d is the number of tokens in the vocabulary.
  • With the parameter "SplitPattern"->None, the input to the encoder is assumed to be a pre-tokenized list of strings of the form {"token1","token2",}.

Examples

open allclose all

Basic Examples  (1)

Create a token encoder for English text:

In[5]:=
Click for copyable input
Out[5]=

Encode an English sentence:

In[2]:=
Click for copyable input
Out[2]=

Out-of-vocabulary words are encoded as the maximum code:

In[4]:=
Click for copyable input
Out[4]=

Scope  (5)

Parameters  (2)

See Also

NetEncoder  NetDecoder  NetChain  NetGraph

Tutorials

Related NetEncoders

Related NetDecoders