"Tokens" (Net Decoder)

[Experimental]

represents a decoder that converts a sequence of probability vectors to a string of English vocabulary words.

NetDecoder[{"Tokens","language"}]

represents a decoder that uses a standard vocabulary for the given language.

NetDecoder[{"Tokens",{"token₁","token₂",…}}]

represents a decoder that uses a specified list of tokens as the vocabulary.

NetDecoder[{"Tokens",…,"param"->val}]

represents a decoder in which additional parameters have been specified.

Details

NetDecoder[…][input] applies the decoder to an input to produce an output.
NetDecoder[…][{input₁,input₂,…}] applies the decoder to a list of inputs to produce a list of outputs.
The input to the decoder input is either a vector of probabilities or a sequence of probability vectors. Each probability vector sums to 1. The length of each probability vector is the size of the vocabulary.
NetDecoder["Tokens"] is equivalent to NetDecoder[{"Tokens","English"}].
A decoder can be attached to an output port of a net by specifying "port"->NetDecoder[…] when constructing the net.

With the parameter "IgnoreCase"->True, tokens are converted to lowercase after decoding. The default is True.

NetDecoder[…][data,prop] can be used to calculate a specific property for the input data.
When a "Tokens" decoder is attached to a net, net[data,prop] or net[data,"oport"->prop] can be used to calculate a specific property of the decoded output.
The "Tokens" decoder only supports the bypass property. Setting prop to None bypasses decoding and returns the input to the decoder.