TokenWords

TokenWords

is an option for Read and related functions which gives a list of token words to be used to delimit words.

Details

  • The setting for TokenWords is a list of strings which are used as delimiters for words to be read.
  • The delimiters specified by TokenWords are themselves returned as words.

Examples

Basic Examples  (1)

A text file:

Read the file:

Read the file using token words to delimit words in the text:

Wolfram Research (1991), TokenWords, Wolfram Language function, https://reference.wolfram.com/language/ref/TokenWords.html.

Text

Wolfram Research (1991), TokenWords, Wolfram Language function, https://reference.wolfram.com/language/ref/TokenWords.html.

CMS

Wolfram Language. 1991. "TokenWords." Wolfram Language & System Documentation Center. Wolfram Research. https://reference.wolfram.com/language/ref/TokenWords.html.

APA

Wolfram Language. (1991). TokenWords. Wolfram Language & System Documentation Center. Retrieved from https://reference.wolfram.com/language/ref/TokenWords.html

BibTeX

@misc{reference.wolfram_2025_tokenwords, author="Wolfram Research", title="{TokenWords}", year="1991", howpublished="\url{https://reference.wolfram.com/language/ref/TokenWords.html}", note=[Accessed: 21-February-2025 ]}

BibLaTeX

@online{reference.wolfram_2025_tokenwords, organization={Wolfram Research}, title={TokenWords}, year={1991}, url={https://reference.wolfram.com/language/ref/TokenWords.html}, note=[Accessed: 21-February-2025 ]}