Lexer (YABR)

tonic@yabr:~$ Source Portfolio tonicbox.dev

java.lang.Object
- com.tonic.analysis.source.parser.Lexer

```
public final class Lexer
extends Object
```
Hand-written Java tokenizer with unbounded lookahead, backed by a buffer of already-scanned tokens.

Constructor Summary

Constructors
Constructor Description

Lexer(String source)
Creates a lexer positioned at line 1, column 1 of the source.

Method Summary

All Methods Instance Methods Concrete Methods
Modifier and Type	Method	Description
`Token`	`current()`
`SourcePosition`	`currentPosition()`
`Token`	`nextToken()`	Consumes the next token, taking it from the lookahead buffer if one is queued.
`Token`	`peek()`
`Token`	`peekAhead(int offset)`	Looks past the current token, scanning into the lookahead buffer as needed.

Methods inherited from class java.lang.Object
equals, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

- Constructor Detail
  - Lexer
```
public Lexer(String source)
```
    Creates a lexer positioned at line 1, column 1 of the source.
    
    Parameters:
    
    source - Java source text to tokenize
- Method Detail
  - nextToken
```
public Token nextToken()
```
    Consumes the next token, taking it from the lookahead buffer if one is queued.
    
    Returns:
    
    the consumed token, EOF once the source is exhausted
  - peek
```
public Token peek()
```
    Returns:
    
    the next token without consuming it
  - peekAhead
```
public Token peekAhead(int offset)
```
    Looks past the current token, scanning into the lookahead buffer as needed.
    
    Parameters:
    
    offset - how many tokens ahead to look, 0 being the next one
    
    Returns:
    
    the token at that distance
  - current
```
public Token current()
```
    Returns:
    
    the token last returned by nextToken, or null before the first call
  - currentPosition
```
public SourcePosition currentPosition()
```
    Returns:
    
    the line, column and offset of the scanner head, which is past any buffered lookahead