Module Expr Str ImportFrom alias Assign NameStore ListLoad Str Assign NameStore ListLoad NameLoad NameLoad NameLoad NameLoad NameLoad NameLoad NameLoad NameLoad NameLoad NameLoad NameLoad NameLoad NameLoad NameLoad NameLoad NameLoad NameLoad NameLoad NameLoad NameLoad NameLoad NameLoad NameLoad NameLoad NameLoad NameLoad NameLoad NameLoad NameLoad NameLoad NameLoad NameLoad NameLoad NameLoad NameLoad NameLoad NameLoad NameLoad NameLoad Assign NameStore ListLoad Dict Str Str Str Str Str Str Str Str Str Str Str Str Str Str Str Str Str Str Str Str Str Str Str Str Str Str Str Str Str Str Str Str Str Str Str Str Str Str Str Str Str Str Str Str Str Str Str Str Str Str Str Str Str Str Str Str Str Str Str Str Str Str Str Str Str Str Str Str Str Str Str Str Str Str Str Str Str Str Str Str Str Str Str Str Str Str Str Str Str Str Str Str Str Str Str Str Str Num Num Num Num Num Num Num Num Num Num Num Num Num Num Num Num Num Num Num Num Num Num Num Num Num Num Num Num Num Num Num Num Num Num Num Num Num Num Num Num Num Num Num Num Num Num Num Num Num Num Num Num Num Num Num Num Num Num Num Num Num Num Num Num Num Num Num Num Num Num Num Num Num Num Num Num Num Num Num Num Num Num Num Num Num Num Num Num Num Num Num Num Num Num Num Num Num Dict Str Str Str Str Str Str Str Str Str Str Str Str Str Str Str Str Str Str Str Str Str Str Str Str Str Str Str Str Str Str Str Str Str Str Str Str Str Str Str Str Str Str Str Str Str Str Str Str Str Str Str Str Str Str Str Str Str Str Str Str Str Str Str Num Num Num Num Num Num Num Num Num Num Num Num Num Num Num Num Num Num Num Num Num Num Num Num Num Num Num Num Num Num Num Num Num Num Num Num Num Num Num Num Num Num Num Num Num Num Num Num Num Num Num Num Num Num Num Num Num Num Num Num Num Num Num Dict Str Str Str Str Str Str Str Str Str Str Str Str Str Str Str Str Str Str Str Str Str Str Str Str Str Str Str Str Str Str Str Str Str Str Str Str Str Str Str Str Str Str Str Str Str Str Str Str Str Str Str Str Str Str Str Str Str Str Str Str Str Str Str Str Str Num Num Num Num Num Num Num Num Num Num Num Num Num Num Num Num Num Num Num Num Num Num Num Num Num Num Num Num Num Num Num Num Num Num Num Num Num Num Num Num Num Num Num Num Num Num Num Num Num Num Num Num Num Num Num Num Num Num Num Num Num Num Num Num Num Dict Str Str Str Str Str Str Str Str Str Str Str Str Str Str Str Str Str Str Str Str Str Str Str Str Str Str Str Str Str Str Str Str Str Str Str Str Str Str Str Str Str Str Str Str Str Str Str Str Str Str Str Str Str Str Str Str Str Str Str Str Str Str Str Str Str Num Num Num Num Num Num Num Num Num Num Num Num Num Num Num Num Num Num Num Num Num Num Num Num Num Num Num Num Num Num Num Num Num Num Num Num Num Num Num Num Num Num Num Num Num Num Num Num Num Num Num Num Num Num Num Num Num Num Num Num Num Num Num Num Num Dict Str Str Str Str Str Str Str Str Str Str Str Str Str Str Str Str Str Str Str Num Num Num Num Num Num Num Num Num Num Num Num Num Num Num Num Num Num Num Dict Str Str Str Str Str Str Str Str Str Str Str Str Str Str Str Str Str Num Num Num Num Num Num Num Num Num Num Num Num Num Num Num Num Num Dict Str Str Str Str Str Str Str Str Str Str Num Num Num Num Num Num Num Num Num Num Dict Str Str Num Num Dict Str Str Num Num Dict Str Str Str Num Num Num Dict Str Num Dict Str Str Num Num Dict Str Num Dict Dict Str Num Dict AttributeLoad NameLoad attr Str Str Str Num Num Num Num Dict AttributeLoad NameLoad attr Str Str Str Num Num Num Num Dict Str Str Num Num Dict AttributeLoad NameLoad attr Str Str Num Num Num Dict AttributeLoad NameLoad attr Str Str Str Num Num Num Num Dict AttributeLoad NameLoad attr Str Str Str Num Num Num Num Dict Str Str Str Str Str Str Str Str Str Str Str Str Str Str Str Str Str Str Str Str Str Str Str Str Num Num Num Num Num Num Num Num Num Num Num Num Num Num Num Num Num Num Num Num Num Num Num Num Dict Str Str Str Str Str Str Str Str Str Str Str Str Str Str Str Str Str Num Num Num Num Num Num Num Num Num Num Num Num Num Num Num Num Num Dict Str Str Str Str Str Str Str Str Str Str Str Str Str Str Str Num Num Num Num Num Num Num Num Num Num Num Num Num Num Num Dict Str Str Str Str Str Str Str Str Str Str Str Str Str Str Num Num Num Num Num Num Num Num Num Num Num Num Num Num Dict Str Str Str Str Str Str Str Str Str Str Str Str Num Num Num Num Num Num Num Num Num Num Num Num Dict Str Str Str Str Str Str Str Str Str Str Str Str Str Str Num Num Num Num Num Num Num Num Num Num Num Num Num Num Dict Dict Str Num Dict AttributeLoad NameLoad attr Str Str Num Num Num Dict Str Num Dict AttributeLoad NameLoad attr Str Str Num Num Num Dict Str Str Str Str Str Str Str Str Str Str Str Str Num Num Num Num Num Num Num Num Num Num Num Num Dict Str Str Str Str Str Str Str Str Str Str Num Num Num Num Num Num Num Num Num Num Dict Str Str Str Str Str Str Str Str Str Str Str Str Num Num Num Num Num Num Num Num Num Num Num Num Dict AttributeLoad NameLoad attr Str Str Str Num Num Num Num Dict AttributeLoad NameLoad attr Str Str Str Num Num Num Num Dict Str Str Str Str Str Str Str Str Str Str Num Num Num Num Num Num Num Num Num Num Dict Str Str Str Str Str Str Str Str Str Str Str Str Num Num Num Num Num Num Num Num Num Num Num Num Assign NameStore Call AttributeLoad NameLoad attr NameLoad NameLoad Assign NameStore ListLoad NameLoad NameLoad NameLoad NameLoad NameLoad NameLoad Assign NameStore ListLoad Dict AttributeLoad NameLoad attr Str Str Num Num Num Dict AttributeLoad NameLoad attr Str Str Num Num Num Dict AttributeLoad NameLoad attr Num Dict AttributeLoad NameLoad attr Str Str Num Num Num Dict AttributeLoad NameLoad attr Str Str Num Num Num Dict AttributeLoad NameLoad attr Str Str Num Num Num Assign NameStore Call AttributeLoad NameLoad attr NameLoad NameLoad Assign NameStore ListLoad NameLoad NameLoad NameLoad NameLoad NameLoad NameLoad Assign NameStore ListLoad Dict AttributeLoad NameLoad attr Str Str Num Num Num Dict AttributeLoad NameLoad attr Str Str Num Num Num Dict AttributeLoad NameLoad attr Num Dict AttributeLoad NameLoad attr Str Str Num Num Num Dict AttributeLoad NameLoad attr Str Str Num Num Num Dict AttributeLoad NameLoad attr Str Str Num Num Num Assign NameStore Call AttributeLoad NameLoad attr NameLoad NameLoad Assign NameStore ListLoad NameLoad NameLoad NameLoad Assign NameStore ListLoad Dict AttributeLoad NameLoad attr Str Str Num Num Num Dict Dict AttributeLoad NameLoad attr Num Assign NameStore Call AttributeLoad NameLoad attr NameLoad NameLoad Assign NameStore ListLoad NameLoad NameLoad NameLoad Assign NameStore ListLoad Dict AttributeLoad NameLoad attr Str Str Num Num Num Dict Dict AttributeLoad NameLoad attr Num Assign NameStore Call AttributeLoad NameLoad attr NameLoad NameLoad Assign NameStore Dict Str Str Str Str Str Str NameLoad NameLoad NameLoad NameLoad NameLoad NameLoad For NameStore TupleLoad Str Str Str body For NameStore TupleLoad Str Str Str body Assign NameStore BinOpAdd NameLoad NameLoad Assign SubscriptStore NameLoad Index BinOpAdd NameLoad Str NameLoad Assign SubscriptStore NameLoad Index BinOpAdd NameLoad Str NameLoad Assign NameStore ListLoad NameLoad Assign NameStore ListLoad Dict Str Str Str Num Num Num Assign NameStore Call AttributeLoad NameLoad attr NameLoad NameLoad Assign NameStore Dict For NameStore TupleLoad Str Str Str Str Str Str Str Str Str Str Str Str Str Str Str Str Str Str body Assign SubscriptStore NameLoad Index NameLoad NameLoad Assign NameStore Dict For NameStore TupleLoad Str Str Str Str Str Str Str Str Str Str Str Str Str Str Str Str Str Str body Assign SubscriptStore NameLoad Index NameLoad NameLoad Assign NameStore Num
N/A N/A Module pytokenize

THIS FILE WAS COPIED FROM pypy/module/parser/pytokenize.py AND ADAPTED
TO BE ANNOTABLE (Mainly made lists homogeneous)

This is a modified version of Ka-Ping Yee's tokenize module found in the
Python standard library.

The primary modification is the removal of the tokenizer's dependence on the
standard Python regular expression module, which is written in C.  The regular
expressions have been replaced with hand built DFA's using the
basil.util.automata module.

$Id: pytokenize.py,v 1.3 2003/10/03 16:31:53 jriehl Exp $
 pypy.interpreter.pyparser automata N/A __all__ N/A tokenize N/A pseudoStatesAccepts N/A True True True True True True True True True True False True True True False False False False True False False True True False True False True False True False True False False False True False False False True N/A pseudoStates N/A N/A ! " # % & ' ( ) * + , - . / 0 1 2 3 4 5 6 7 8 9 : ; < = > @ A B C D E F G H I J K L M N O P Q R S T U V W X Y Z [ \ ] ^ _ ` a b c d e f g h i j k l m n o p q r s t u v w x y z { | } ~ 0 13 0 14 0 10 16 18 12 12 15 13 13 7 12 13 12 6 11 4 5 5 5 5 5 5 5 5 5 13 13 9 12 8 13 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 2 1 1 3 1 1 1 1 1 13 17 13 12 1 13 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 2 1 1 3 1 1 1 1 1 13 12 13 13 N/A 0 1 2 3 4 5 6 7 8 9 A B C D E F G H I J K L M N O P Q R S T U V W X Y Z _ a b c d e f g h i j k l m n o p q r s t u v w x y z 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 N/A " ' 0 1 2 3 4 5 6 7 8 9 A B C D E F G H I J K L M N O P Q R S T U V W X Y Z _ a b c d e f g h i j k l m n o p q r s t u v w x y z 16 15 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 N/A " ' 0 1 2 3 4 5 6 7 8 9 A B C D E F G H I J K L M N O P Q R S T U V W X Y Z _ a b c d e f g h i j k l m n o p q r s t u v w x y z 16 15 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 2 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 2 1 1 1 1 1 1 1 1 N/A . 0 1 2 3 4 5 6 7 8 9 E J L X e j l x 24 22 22 22 22 22 22 22 22 23 23 25 13 13 21 25 13 13 21 N/A . 0 1 2 3 4 5 6 7 8 9 E J L e j l 24 5 5 5 5 5 5 5 5 5 5 25 13 13 25 13 13 N/A 0 1 2 3 4 5 6 7 8 9 26 26 26 26 26 26 26 26 26 26 N/A * = 12 13 N/A = > 13 12 N/A = < > 13 12 13 N/A = 13 N/A = / 13 12 N/A = 13 N/A N/A 13 N/A N/A automata DEFAULT \ ' 19 27 29 28 N/A N/A automata DEFAULT " \ 20 30 27 31 N/A 13 14 N/A N/A automata DEFAULT 18 27 27 N/A N/A automata DEFAULT \ ' 19 27 29 13 N/A N/A automata DEFAULT " \ 20 13 27 31 N/A 0 1 2 3 4 5 6 7 8 9 A B C D E F L a b c d e f l 21 21 21 21 21 21 21 21 21 21 21 21 21 21 21 21 13 21 21 21 21 21 21 13 N/A . 0 1 2 3 4 5 6 7 8 9 E J L e j l 24 22 22 22 22 22 22 22 22 23 23 25 13 13 25 13 13 N/A . 0 1 2 3 4 5 6 7 8 9 E J e j 24 23 23 23 23 23 23 23 23 23 23 25 13 25 13 N/A 0 1 2 3 4 5 6 7 8 9 E J e j 24 24 24 24 24 24 24 24 24 24 32 13 32 13 N/A + - 0 1 2 3 4 5 6 7 8 9 33 33 34 34 34 34 34 34 34 34 34 34 N/A 0 1 2 3 4 5 6 7 8 9 E J e j 26 26 26 26 26 26 26 26 26 26 32 13 32 13 N/A N/A ' 13 N/A N/A automata DEFAULT 35 13 14 N/A " 13 N/A N/A automata DEFAULT 36 13 14 N/A + - 0 1 2 3 4 5 6 7 8 9 37 37 38 38 38 38 38 38 38 38 38 38 N/A 0 1 2 3 4 5 6 7 8 9 34 34 34 34 34 34 34 34 34 34 N/A 0 1 2 3 4 5 6 7 8 9 J j 34 34 34 34 34 34 34 34 34 34 13 13 N/A N/A automata DEFAULT \ ' 35 27 29 13 N/A N/A automata DEFAULT " \ 36 13 27 31 N/A 0 1 2 3 4 5 6 7 8 9 38 38 38 38 38 38 38 38 38 38 N/A 0 1 2 3 4 5 6 7 8 9 J j 38 38 38 38 38 38 38 38 38 38 13 13 N/A pseudoDFA N/A N/A automata DFA pseudoStates pseudoStatesAccepts N/A double3StatesAccepts N/A False False False False False True N/A double3States N/A N/A N/A automata DEFAULT " \ 0 1 2 N/A N/A automata DEFAULT " \ 4 3 2 N/A N/A automata DEFAULT 4 N/A N/A automata DEFAULT " \ 4 5 2 N/A N/A automata DEFAULT " \ 4 1 2 N/A N/A automata DEFAULT " \ 4 5 2 N/A double3DFA N/A N/A automata NonGreedyDFA double3States double3StatesAccepts N/A single3StatesAccepts N/A False False False False False True N/A single3States N/A N/A N/A automata DEFAULT \ ' 0 2 1 N/A N/A automata DEFAULT \ ' 4 2 3 N/A N/A automata DEFAULT 4 N/A N/A automata DEFAULT \ ' 4 2 5 N/A N/A automata DEFAULT \ ' 4 2 1 N/A N/A automata DEFAULT \ ' 4 2 5 N/A single3DFA N/A N/A automata NonGreedyDFA single3States single3StatesAccepts N/A singleStatesAccepts N/A False True False N/A singleStates N/A N/A N/A automata DEFAULT \ ' 0 2 1 N/A N/A N/A automata DEFAULT 0 N/A singleDFA N/A N/A automata DFA singleStates singleStatesAccepts N/A doubleStatesAccepts N/A False True False N/A doubleStates N/A N/A N/A automata DEFAULT " \ 0 1 2 N/A N/A N/A automata DEFAULT 0 N/A doubleDFA N/A N/A automata DFA doubleStates doubleStatesAccepts N/A endDFAs N/A ' " r R u U singleDFA doubleDFA None None None None N/A uniPrefix N/A u U N/A N/A rawPrefix N/A r R N/A N/A prefix N/A uniPrefix rawPrefix N/A N/A endDFAs N/A N/A prefix ''' single3DFA N/A N/A endDFAs N/A N/A prefix """ double3DFA N/A whiteSpaceStatesAccepts N/A True N/A whiteSpaceStates N/A N/A 0 0 0 N/A whiteSpaceDFA N/A N/A automata DFA whiteSpaceStates whiteSpaceStatesAccepts N/A triple_quoted N/A N/A t N/A ''' """ r''' r""" R''' R""" u''' u""" U''' U""" ur''' ur""" Ur''' Ur""" uR''' uR""" UR''' UR""" N/A N/A N/A triple_quoted N/A t t N/A single_quoted N/A N/A t N/A ' " r' r" R' R" u' u" U' U" ur' ur" Ur' Ur" uR' uR" UR' UR" N/A N/A N/A single_quoted N/A t t N/A tabsize 8