define token lexeme and pattern ?
Tokens, patterns and lexemes. 1  Token is a terminal symbol in the grammar for the source language. 2  Lexeme is a sequence of characters in the source program that is matched by the pattern for a token. 3  In many programming languages, the following classes cover most or all of the tokens:
A pattern is a description of the form that the lexemes of a token may take. In the case of a keyword as a token, the pattern is just the sequence of characters that form the keyword. For identifiers and some other tokens, the pattern is a more complex structure that is matched by many strings. Lexeme is a sequence of characters in the source program that is matched by the pattern for a token. A lexeme is a sequence of characters in the source program that matches the pattern for a token and is identified by the lexical analyzer as an instance of that token. In many programming languages, the following classes cover most or all of the tokens: 1. One token for each keyword. The pattern for a keyword is the same as the keyword itself. 2.