"Tokens" (Net Decoder)

NetDecoder["Tokens"]

represents a decoder that converts a sequence of probability vectors to a string of English vocabulary words.

NetDecoder[{"Tokens","language"}]

represents a decoder that uses a standard vocabulary for the given language.

NetDecoder[{"Tokens",{"token1","token2",}}]

represents a decoder that uses a specified list of tokens as the vocabulary.

NetDecoder[{"Tokens",,"param"->val}]

represents a decoder in which additional parameters have been specified.

Details

  • NetDecoder[][input] applies the decoder to an input to produce an output.
  • NetDecoder[][{input1,input2,}] applies the decoder to a list of inputs to produce a list of outputs.
  • The input to the decoder input is either a vector of probabilities or a sequence of probability vectors. Each probability vector sums to 1. The length of each probability vector is the size of the vocabulary.
  • NetDecoder["Tokens"] is equivalent to NetDecoder[{"Tokens","English"}].
  • A decoder can be attached to an output port of a net by specifying "port"->NetDecoder[] when constructing the net.
  • Parameters
  • With the parameter "IgnoreCase"->True, tokens are converted to lowercase after decoding. The default is True.

Examples

open allclose all

Basic Examples  (2)

Create a token decoder for English text:

In[20]:=
Click for copyable input
Out[20]=

Decode an English sentence:

In[31]:=
Click for copyable input
Out[31]=

Decode a nonsense sentence from a random matrix of probabilities:

In[22]:=
Click for copyable input
Out[22]=

Create a decoder with two tokens:

In[1]:=
Click for copyable input
Out[1]=

Evaluate the decoder on a sequence of probability vectors:

In[2]:=
Click for copyable input
Out[2]=

Scope  (1)

See Also

NetDecoder  NetEncoder  NetChain  NetGraph

Tutorials

Related NetEncoders

Related NetDecoders