"Tokens" (Net Decoder)

NetDecoder["Tokens"]

represents a decoder that converts a sequence of probability vectors to a string of English vocabulary words.

NetDecoder[{"Tokens","language"}]

represents a decoder that uses a standard vocabulary for the given language.

NetDecoder[{"Tokens",{"token1","token2",…}}]

represents a decoder that uses a specified list of tokens as the vocabulary.

NetDecoder[{"Tokens",…,"param"->val}]

represents a decoder in which additional parameters have been specified.

Details

  • NetDecoder[…][input] applies the decoder to an input to produce an output.
  • NetDecoder[…][{input1,input2,…}] applies the decoder to a list of inputs to produce a list of outputs.
  • The input to the decoder input is either a vector of probabilities or a sequence of probability vectors. Each probability vector sums to 1. The length of each probability vector is the size of the vocabulary.
  • NetDecoder["Tokens"] is equivalent to NetDecoder[{"Tokens","English"}].
  • A decoder can be attached to an output port of a net by specifying "port"->NetDecoder[…] when constructing the net.
    • Parameters
  • With the parameter "IgnoreCase"->True, tokens are converted to lowercase after decoding. The default is True.
    • Properties
  • NetDecoder[…][data,prop] can be used to calculate a specific property for the input data.
  • When a "Tokens" decoder is attached to a net, net[data,prop] or net[data,"oport"->prop] can be used to calculate a specific property of the decoded output.
  • The "Tokens" decoder only supports the bypass property. Setting prop to None bypasses decoding and returns the input to the decoder.

Examples

open allclose all

Basic Examples  (2)

Create a token decoder for English text:

Get the expected size for inputs to the decoder:

Decode an English sentence:

Decode a nonsense sentence from a random matrix of probabilities:

Create a decoder with two tokens:

Evaluate the decoder on a sequence of probability vectors:

Scope  (1)

Attach a token decoder to the output port of a net:

Evaluating the net on a sequence of probability vectors returns a string: