Chapte 4 Lexical analysis.pptVIP

CMSC 331 Fall 2000 Chapter 4 Lexical analysis Scanner Main task: identify tokens Basic building blocks of programs E.g. keywords, identifiers, numbers, punctuation marks Desk calculator language example: read A sum := A + 3.45e-3 write sum write sum / 2 Formal definition of tokens A set of tokens is a set of strings over an alphabet {read, write, +, -, *, /, :=, 1, 2, …, 10, …, 3.45e-3, …} A set of tokens is a regular set that can be defined by comprehension using a regular expression For every regular set, there is a deterministic finite automaton (DFA) that can recognize it (Aka deterministic Finite State Machine (FSM)) i.e. determine whether a string belongs to the set or not Scanners extract tokens from source code in the same way DFAs determine membership Regular Expressions A regular expression (RE) is: A single character The empty string, ? The concatenation of two regular expressions Notation: RE1 RE2 (i.e. RE1 followed by RE2) The union of two regular expressions Notation: RE1 | RE2 The closure of a regular expression Notation: RE* * is known as the Kleene star * represents the concatenation of 0 or more strings Caution: notations for regular expressions vary Learn the basic concepts and the rest is just syntactic sugar Token Definition Example Numeric literals in Pascal, e.g. 1, 123, 3.1415, 10e-3, 3.14e4 Definition of token unsignedNum DIG ? 0|1|2|3|4|5|6|7|8|9 unsignedInt ? DIG DIG* unsignedNum ? unsignedInt (( . unsignedInt) | ?) ((e ( + | – | ?) unsignedInt) | ?) Notes: Recursion is not allowed! Parentheses used to avoid ambiguity It’s always possible to rewrite removing epsilons Simple Problem Write a C program which reads in a character string, consisting of a’s and b’s, one character at a time. If the string contains a double aa, then print string accepted else print string rejected. An abstract solution to this can be expressed as a DFA Lex Lexical analyzer generator It writes a lexical analyzer Assumption each token ma