Berkeley CSUA MOTD:Entry 19414
Berkeley CSUA MOTD
 
WIKI | FAQ | Tech FAQ
http://csua.com/feed/
2025/07/10 [General] UID:1000 Activity:popular
7/10    

2000/10/5-6 [Computer/SW/Languages/Java] UID:19414 Activity:nil
10/4    What's the difference between a token and a lexeme? I just don't
        see any. A token is defined with regular expression. If "while" and
        "for" are tokens for C/Java, why are they also lexemes? What about
        identifiers (variable, constants)? Aren't they BOTH token and lexeme?
        How come Aiken makes a distinction?
        \_ If your symbol is the field "numPoints", the the token is an
           Identifier and the lexeme is the string "numPoints".  For "123"
           the token is Integer and the lexeme is the value "123". In
           short, token = type, lexeme = value.  If you call "if" a token,
           then the lexeme is redundant because the lexeme is just the
           name of the token.  But if you lump if, while, else, etc. into
           the Keyword token (or token class, if you prefer), then the
           lexeme becomes relevant.