Simple stream-oriented parser for efficient basic recognition of incoming data.
class Tokenizer( [seps], [options], [tokLen], [source] )
init | Simple stream-oriented parser for efficient basic recognition of incoming data. |
next() | Advances the tokenizer up to the next token. |
nextToken() | Returns the next token from the tokenizer |
parse() | Changes or set the source data for this tokenizer. |
rewind() | Resets the status of the tokenizer. |
token() | Get the current token. |
Simple stream-oriented parser for efficient basic recognition of incoming data.
The tokenizer class is meant to provide simple and efficiente logic to parse incoming data (mainly, incoming from string).
The source can also be set at a second time with the Tokenizer.parse method. seps defaults to " " if not given.
Simple stream-oriented parser for efficient basic recognition of incoming data.
init Tokenizer( [seps], [options], [tokLen], [source] )
seps | A string representing the separators. |
options | Tokenization options. |
tokLen | Maximum length of returned tokens. |
source | The string to be tokenized, or a stream to be read for tokens. |
The tokenizer class is meant to provide simple and efficiente logic to parse incoming data (mainly, incoming from string).
The source can also be set at a second time with the Tokenizer.parse method. seps defaults to " " if not given.
Advances the tokenizer up to the next token.
Tokenizer.next( )
Returns: | True if a new token is now available, false otherwise. | ||||
Raises: |
|
Contrarily to iterators, it is necessary to call this method at least once before Tokenizer.token is available.
For example:
t = Tokenizer( source|"A string to be tokenized" ) while t.next() > "Token: ", t.token() end
Returns the next token from the tokenizer
Tokenizer.nextToken( )
Returns: | A string or nil at the end of the tokenization. | ||||
Raises: |
|
This method is actually a combination of Tokenizer.next followed by Tokenizer.token.
Sample usage:
t = Tokenizer( source|"A string to be tokenized" ) while (token = t.nextToken()) != nil > "Token: ", token end
Note: When looping, remember to check the value of the returned token against nil, as empty strings can be legally returned multiple times, and they are considered false in logic checks.
Changes or set the source data for this tokenizer.
Tokenizer.parse( source )
source | A string or a stream to be used as a source for the tokenizer. |
Resets the status of the tokenizer.
Tokenizer.rewind( )
Raises: |
|
Get the current token.
Tokenizer.token( )
Returns: | True if a new token is now available, false otherwise. | ||||
Raises: |
|
Contrarily to iterators, it is necessary to call this Tokenizer.next at least once before calling this method.