Repository: apple/pkl
File: pkl-parser/src/main/java/org/pkl/parser/Lexer.java (Java)

Code before:

import java.util.ArrayDeque;
import java.util.Deque;
import org.pkl.parser.syntax.generic.FullSpan;
import org.pkl.parser.util.ErrorMessages;

public class Lexer {

  private final char[] source;
  private final int size;
  protected int cursor = 0;
  protected int sCursor = 0;
  private int line = 1;
  private int sLine = 1;
  private int col = 1;
  private int sCol = 1;
  private char lookahead;
  private State state = State.DEFAULT;
  private final Deque<InterpolationScope> interpolationStack = new ArrayDeque<>();
  private boolean stringEnded = false;
  private boolean isEscape = false;
  // how many newlines exist between two subsequent tokens
  protected int newLinesBetween = 0;
  private boolean atEOF = false;

  private static final char EOF = Character.MAX_VALUE;

  public Lexer(String input) {
    source = input.toCharArray();
    size = source.length;
    if (size > 0) {
      lookahead = source[cursor];
    } else {
      lookahead = EOF;
      atEOF = true;
    }
  }

  // The span of the last lexed token
  public Span span() {
    return new Span(sCursor, cursor - sCursor);
  }

  // The full span of the last lexed token
  public FullSpan fullSpan() {
    return new FullSpan(sCursor, cursor - sCursor, sLine, sCol, line, col);
  }

  // The text of the last lexed token
  public String text() {
    return new String(source, sCursor, cursor - sCursor);

Review comment (suggestion):
This still has a conflict, no? `\uffff` is an "undefined character": https://www.compart.com/en/unicode/U+FFFF

I can still have Pkl code that looks like:

```pkl
// Hello there ￿ uffff
```

I think we can fix this by representing EOF as `-1`, and change `char lookahead` to `int lookahead`; since `-1` is never a valid character code, this eliminates any chance of conflict. And, we can go back to using `if (lookahead == EOF) in this case.

Code after:

import java.util.ArrayDeque;
import java.util.Deque;
import org.pkl.parser.syntax.generic.FullSpan;
import org.pkl.parser.util.ErrorMessages;

public class Lexer {

  private final char[] source;
  private final int size;
  protected int cursor = 0;
  protected int sCursor = 0;
  private int line = 1;
  private int sLine = 1;
  private int col = 1;
  private int sCol = 1;
  private int lookahead;
  private State state = State.DEFAULT;
  private final Deque<InterpolationScope> interpolationStack = new ArrayDeque<>();
  private boolean stringEnded = false;
  private boolean isEscape = false;
  // how many newlines exist between two subsequent tokens
  protected int newLinesBetween = 0;

  private static final int EOF = -1;

  public Lexer(String input) {
    source = input.toCharArray();
    size = source.length;
    if (size > 0) {
      lookahead = source[cursor];
    } else {
      lookahead = EOF;
    }
  }

  // The span of the last lexed token
  public Span span() {
    return new Span(sCursor, cursor - sCursor);
  }

  // The full span of the last lexed token
  public FullSpan fullSpan() {
    return new FullSpan(sCursor, cursor - sCursor, sLine, sCol, line, col);
  }

  // The text of the last lexed token
  public String text() {
    return new String(source, sCursor, cursor - sCursor);
  }