CodeParser`
CodeParser`

CodeTokenize

CodeTokenize["code"]

tokenizes a string of WL code returning a list of tokens.

CodeTokenize[File[src]]

tokenizes a file of WL code returning a list of tokens.

Details and Options

  • CodeTokenize has these options:
  • SourceConvention "LineColumn"convention to use for Source metadata

Examples

open allclose all

Options  (1)

SourceConvention  (1)

Using "LineColumn" convention:

Using "SourceCharacterIndex" convention:

Possible Issues  (1)

A string representing a file path is treated as code.

File paths must use the File wrapper.