10/4 What's the difference between a token and a lexeme? I just don't
see any. A token is defined with regular expression. If "while" and
"for" are tokens for C/Java, why are they also lexemes? What about
identifiers (variable, constants)? Aren't they BOTH token and lexeme?
How come Aiken makes a distinction?
\_ If your symbol is the field "numPoints", the the token is an
Identifier and the lexeme is the string "numPoints". For "123"
the token is Integer and the lexeme is the value "123". In
short, token = type, lexeme = value. If you call "if" a token,
then the lexeme is redundant because the lexeme is just the
name of the token. But if you lump if, while, else, etc. into
the Keyword token (or token class, if you prefer), then the
lexeme becomes relevant. |