The lexer just turns the meaningless string into a flat list of things like “number literal”, “string literal”, “identifier”, or “operator”, and can do things like recognizing reserved identifiers (“keywords”) and discarding whitespace. Formally, a lexer recognizes some set of Regular languages.

How does a lexer parser work?

A lexer and a parser work in sequence: the lexer scans the input and produces the matching tokens; the parser then scans the tokens and produces the parsing result.

Do you need a lexer?

Structure of a Parser A complete parser is usually composed of two parts: a lexer, also known as scanner or tokenizer, and the proper parser. The parser needs the lexer because it does not work directly on the text, but on the output produced by the lexer.