Module Expr Str Assign NameStore Str Assign NameStore Str Import alias alias ImportFrom alias Import alias Assign NameStore ListComp NameLoad comprehension NameStore Call NameLoad NameLoad UnaryOpNot Call AttributeLoad NameLoad attr Str AugAssignAdd NameStore ListLoad Str Str Str Str Str Delete NameDel Delete NameDel Assign NameStore NameLoad Assign SubscriptStore NameLoad Index NameLoad Str Assign NameStore BinOpAdd NameLoad Num Assign SubscriptStore NameLoad Index NameLoad Str AugAssignAdd NameStore Num FunctionDef arguments args defaults vararg body Return BinOpAdd BinOpAdd Str Call AttributeLoad Str attr NameLoad Str decorator_list FunctionDef arguments args defaults vararg body Return BinOpAdd Call NameLoad NameLoad Str decorator_list FunctionDef arguments args defaults vararg body Return BinOpAdd Call NameLoad NameLoad Str decorator_list Assign NameStore Str Assign NameStore Str Assign NameStore BinOpAdd BinOpAdd NameLoad Call NameLoad BinOpAdd Str NameLoad Call NameLoad NameLoad Assign NameStore Str Assign NameStore Str Assign NameStore Str Assign NameStore Str Assign NameStore Str Assign NameStore Call NameLoad NameLoad NameLoad NameLoad NameLoad Assign NameStore Str Assign NameStore BinOpAdd Call NameLoad Str Str Call NameLoad NameLoad Assign NameStore BinOpAdd Str NameLoad Assign NameStore Call NameLoad NameLoad NameLoad Assign NameStore Call NameLoad Str BinOpAdd NameLoad Str Assign NameStore Call NameLoad NameLoad NameLoad NameLoad Assign NameStore Str Assign NameStore Str Assign NameStore Str Assign NameStore Str Assign NameStore Call NameLoad Str Str Assign NameStore Call NameLoad Str Str Assign NameStore Call NameLoad Str Str Str Str Str Str Str Str Assign NameStore Str Assign NameStore Call NameLoad Str Str Assign NameStore Call NameLoad NameLoad NameLoad NameLoad Assign NameStore Call NameLoad NameLoad NameLoad NameLoad NameLoad Assign NameStore BinOpAdd NameLoad NameLoad Assign NameStore Call NameLoad BinOpAdd Str Call NameLoad Str Str BinOpAdd Str Call NameLoad Str Str Assign NameStore Call NameLoad Str NameLoad NameLoad Assign NameStore BinOpAdd NameLoad Call NameLoad NameLoad NameLoad NameLoad NameLoad NameLoad Assign TupleStore NameStore NameStore NameStore NameStore Call NameLoad AttributeLoad NameLoad attr TupleLoad NameLoad NameLoad NameLoad NameLoad Assign NameStore Dict Str Str Str Str Str Str Str Str Str Str Str Str Str Str Str Str Str Str Str Str Str Str Str Str Str Str Str Str Str Str Str Str Str Str Str Str Str Str Call AttributeLoad NameLoad attr NameLoad Call AttributeLoad NameLoad attr NameLoad NameLoad NameLoad NameLoad NameLoad NameLoad NameLoad NameLoad NameLoad NameLoad NameLoad NameLoad NameLoad NameLoad NameLoad NameLoad NameLoad NameLoad NameLoad NameLoad NameLoad NameLoad NameLoad NameLoad NameLoad NameLoad NameLoad NameLoad NameLoad NameLoad NameLoad NameLoad NameLoad NameLoad NameLoad NameLoad NameLoad Assign NameStore Dict For NameStore TupleLoad Str Str Str Str Str Str Str Str Str Str Str Str Str Str Str Str Str Str Str Str Str Str Str Str Str Str Str Str Str Str body Assign SubscriptStore NameLoad Index NameLoad NameLoad Assign NameStore Dict For NameStore TupleLoad Str Str Str Str Str Str Str Str Str Str Str Str Str Str Str Str Str Str Str Str Str Str Str Str Str Str Str Str Str Str body Assign SubscriptStore NameLoad Index NameLoad NameLoad Assign NameStore Num ClassDef bases NameLoad body Pass decorator_list ClassDef bases NameLoad body Pass decorator_list FunctionDef arguments args NameParam NameParam NameParam NameParam NameParam defaults body Assign TupleStore NameStore NameStore NameLoad Assign TupleStore NameStore NameStore NameLoad Print BinOpMod Str TupleLoad NameLoad NameLoad NameLoad NameLoad SubscriptLoad NameLoad Index NameLoad Call NameLoad NameLoad decorator_list FunctionDef arguments args NameParam NameParam defaults NameLoad body Expr Str TryExcept body Expr Call NameLoad NameLoad NameLoad handlers ExceptHandler type NameLoad body Pass decorator_list FunctionDef arguments args NameParam NameParam defaults body For NameStore Call NameLoad NameLoad body Expr Call NameLoad NameLoad decorator_list ClassDef bases body FunctionDef arguments args NameParam defaults body Assign AttributeStore NameLoad attr ListLoad Assign AttributeStore NameLoad attr Num Assign AttributeStore NameLoad attr Num decorator_list FunctionDef arguments args NameParam NameParam defaults body Assign TupleStore NameStore NameStore NameLoad Assert CompareLtE NameLoad AttributeLoad NameLoad attr Assign NameStore BinOpSub NameLoad AttributeLoad NameLoad attr If NameLoad body Expr Call AttributeLoad AttributeLoad NameLoad attr attr BinOpMult Str NameLoad decorator_list FunctionDef arguments args NameParam NameParam defaults body For NameStore NameLoad body If CompareEq Call NameLoad NameLoad Num body Expr Call AttributeLoad NameLoad attr NameLoad NameLoad Break Assign TupleStore NameStore NameStore NameStore NameStore NameStore NameLoad Expr Call AttributeLoad NameLoad attr NameLoad Expr Call AttributeLoad AttributeLoad NameLoad attr attr NameLoad Assign TupleStore AttributeStore NameLoad attr AttributeStore NameLoad attr NameLoad If CompareIn NameLoad TupleLoad NameLoad NameLoad body AugAssignAdd AttributeStore NameLoad attr Num Assign AttributeStore NameLoad attr Num Return Call AttributeLoad Str attr AttributeLoad NameLoad attr decorator_list FunctionDef arguments args NameParam NameParam NameParam defaults body Assign NameStore NameLoad Assign NameStore ListLoad Assign NameStore AttributeLoad AttributeLoad NameLoad attr attr Assign TupleStore NameStore NameStore NameLoad If CompareIn NameLoad TupleLoad NameLoad NameLoad body AugAssignAdd NameStore Str If CompareIn NameLoad TupleLoad NameLoad NameLoad body Assign NameStore NameLoad Assign NameStore NameLoad For NameStore NameLoad body Assign TupleStore NameStore NameStore SubscriptLoad NameLoad Slice Num If CompareIn NameLoad TupleLoad NameLoad NameLoad body AugAssignAdd NameStore Str If CompareEq NameLoad NameLoad body If NameLoad body Assign NameStore BinOpAdd Str NameLoad Assign NameStore NameLoad orelse Assign NameStore NameLoad If CompareEq NameLoad NameLoad body Expr Call AttributeLoad NameLoad attr NameLoad Continue orelse If CompareEq NameLoad NameLoad body Expr Call AttributeLoad NameLoad attr Continue orelse If CompareIn NameLoad TupleLoad NameLoad NameLoad body Assign NameStore NameLoad orelse If BoolOpAnd NameLoad NameLoad body Expr Call NameLoad SubscriptLoad NameLoad Index Num Assign NameStore NameLoad Expr Call NameLoad NameLoad decorator_list decorator_list FunctionDef arguments args NameParam defaults body Expr Str Assign NameStore Call NameLoad Return Call AttributeLoad NameLoad attr NameLoad decorator_list FunctionDef arguments args NameParam defaults body Expr Str Assign NameStore NameStore NameStore Num Assign TupleStore NameStore NameStore TupleLoad BinOpAdd AttributeLoad NameLoad attr Str Str Assign TupleStore NameStore NameStore TupleLoad Str Num Assign NameStore NameLoad Assign NameStore ListLoad Num While Num body TryExcept body Assign NameStore Call NameLoad handlers ExceptHandler type NameLoad body Assign NameStore Str AugAssignAdd NameStore Num Assign TupleStore NameStore NameStore TupleLoad Num Call NameLoad NameLoad If NameLoad body If UnaryOpNot NameLoad body Raise NameLoad TupleLoad Str NameLoad Assign NameStore Call AttributeLoad NameLoad attr NameLoad If NameLoad body Assign NameStore NameStore Call AttributeLoad NameLoad attr Num Expr Yield TupleLoad NameLoad BinOpAdd NameLoad SubscriptLoad NameLoad Slice NameLoad NameLoad TupleLoad NameLoad NameLoad BinOpAdd NameLoad NameLoad Assign TupleStore NameStore NameStore TupleLoad Str Num Assign NameStore NameLoad orelse If BoolOpAnd NameLoad CompareNotEq SubscriptLoad NameLoad Slice Num Str CompareNotEq SubscriptLoad NameLoad Slice Num Str body Expr Yield TupleLoad NameLoad BinOpAdd NameLoad NameLoad NameLoad TupleLoad NameLoad Call NameLoad NameLoad NameLoad Assign NameStore Str Assign NameStore NameLoad Continue orelse Assign NameStore BinOpAdd NameLoad NameLoad Assign NameStore BinOpAdd NameLoad NameLoad Continue orelse If BoolOpAnd CompareEq NameLoad Num UnaryOpNot NameLoad body If UnaryOpNot NameLoad body Break Assign NameStore Num While CompareLt NameLoad NameLoad body If CompareEq SubscriptLoad NameLoad Index NameLoad Str body AugAssignAdd NameStore Num orelse If CompareEq SubscriptLoad NameLoad Index NameLoad Str body Assign NameStore BinOpMult BinOpAdd BinOpFloorDiv NameLoad NameLoad Num NameLoad orelse If CompareEq SubscriptLoad NameLoad Index NameLoad Str body Assign NameStore Num orelse Break AugAssignAdd NameStore Num If CompareEq NameLoad NameLoad body Break If CompareIn SubscriptLoad NameLoad Index NameLoad Str body If CompareEq SubscriptLoad NameLoad Index NameLoad Str body Assign NameStore Call AttributeLoad SubscriptLoad NameLoad Slice NameLoad attr Str Assign NameStore BinOpAdd NameLoad Call NameLoad NameLoad Expr Yield TupleLoad NameLoad NameLoad TupleLoad NameLoad NameLoad TupleLoad NameLoad BinOpAdd NameLoad Call NameLoad NameLoad NameLoad Expr Yield TupleLoad NameLoad SubscriptLoad NameLoad Slice NameLoad TupleLoad NameLoad NameLoad TupleLoad NameLoad Call NameLoad NameLoad NameLoad orelse Expr Yield TupleLoad SubscriptLoad TupleLoad NameLoad NameLoad Index CompareEq SubscriptLoad NameLoad Index NameLoad Str SubscriptLoad NameLoad Slice NameLoad TupleLoad NameLoad NameLoad TupleLoad NameLoad Call NameLoad NameLoad NameLoad Continue If CompareGt NameLoad SubscriptLoad NameLoad Index Num body Expr Call AttributeLoad NameLoad attr NameLoad Expr Yield TupleLoad NameLoad SubscriptLoad NameLoad Slice NameLoad TupleLoad NameLoad Num TupleLoad NameLoad NameLoad NameLoad While CompareLt NameLoad SubscriptLoad NameLoad Index Num body If CompareNotIn NameLoad NameLoad body Raise Call NameLoad Str TupleLoad Str NameLoad NameLoad NameLoad Assign NameStore SubscriptLoad NameLoad Slice Num Expr Yield TupleLoad NameLoad Str TupleLoad NameLoad NameLoad TupleLoad NameLoad NameLoad NameLoad orelse If UnaryOpNot NameLoad body Raise NameLoad TupleLoad Str TupleLoad NameLoad Num Assign NameStore Num While CompareLt NameLoad NameLoad body Assign NameStore Call AttributeLoad NameLoad attr NameLoad NameLoad If NameLoad body Assign TupleStore NameStore NameStore Call AttributeLoad NameLoad attr Num Assign TupleStore NameStore NameStore NameStore TupleLoad TupleLoad NameLoad NameLoad TupleLoad NameLoad NameLoad NameLoad Assign TupleStore NameStore NameStore TupleLoad SubscriptLoad NameLoad Slice NameLoad NameLoad SubscriptLoad NameLoad Index NameLoad If BoolOpOr CompareIn NameLoad NameLoad BoolOpAnd CompareEq NameLoad Str CompareNotEq NameLoad Str body Expr Yield TupleLoad NameLoad NameLoad NameLoad NameLoad NameLoad orelse If CompareIn NameLoad Str body Expr Yield TupleLoad IfExp CompareGt NameLoad Num NameLoad NameLoad NameLoad NameLoad NameLoad NameLoad orelse If CompareEq NameLoad Str body Assert UnaryOpNot Call AttributeLoad NameLoad attr Str Expr Yield TupleLoad NameLoad NameLoad NameLoad NameLoad NameLoad orelse If CompareIn NameLoad NameLoad body Assign NameStore SubscriptLoad NameLoad Index NameLoad Assign NameStore Call AttributeLoad NameLoad attr NameLoad NameLoad If NameLoad body Assign NameStore Call AttributeLoad NameLoad attr Num Assign NameStore SubscriptLoad NameLoad Slice NameLoad NameLoad Expr Yield TupleLoad NameLoad NameLoad NameLoad TupleLoad NameLoad NameLoad NameLoad orelse Assign NameStore TupleLoad NameLoad NameLoad Assign NameStore SubscriptLoad NameLoad Slice NameLoad Assign NameStore NameLoad Break orelse If BoolOpOr CompareIn NameLoad NameLoad CompareIn SubscriptLoad NameLoad Slice Num NameLoad CompareIn SubscriptLoad NameLoad Slice Num NameLoad body If CompareEq SubscriptLoad NameLoad Index Num Str body Assign NameStore TupleLoad NameLoad NameLoad Assign NameStore BoolOpOr SubscriptLoad NameLoad Index NameLoad SubscriptLoad NameLoad Index SubscriptLoad NameLoad Index Num SubscriptLoad NameLoad Index SubscriptLoad NameLoad Index Num Assign TupleStore NameStore NameStore TupleLoad SubscriptLoad NameLoad Slice NameLoad Num Assign NameStore NameLoad Break orelse Expr Yield TupleLoad NameLoad NameLoad NameLoad NameLoad NameLoad orelse If CompareIn NameLoad NameLoad body Expr Yield TupleLoad NameLoad NameLoad NameLoad NameLoad NameLoad orelse If CompareEq NameLoad Str body Assign NameStore Num orelse If CompareIn NameLoad Str body AugAssignAdd NameStore Num orelse If CompareIn NameLoad Str body AugAssignSub NameStore Num Expr Yield TupleLoad NameLoad NameLoad NameLoad NameLoad NameLoad orelse Expr Yield TupleLoad NameLoad SubscriptLoad NameLoad Index NameLoad TupleLoad NameLoad NameLoad TupleLoad NameLoad BinOpAdd NameLoad Num NameLoad AugAssignAdd NameStore Num For NameStore SubscriptLoad NameLoad Slice Num body Expr Yield TupleLoad NameLoad Str TupleLoad NameLoad Num TupleLoad NameLoad Num Str Expr Yield TupleLoad NameLoad Str TupleLoad NameLoad Num TupleLoad NameLoad Num Str decorator_list If CompareEq NameLoad Str body Import alias If CompareGt Call NameLoad AttributeLoad NameLoad attr Num body Expr Call NameLoad AttributeLoad Call NameLoad SubscriptLoad AttributeLoad NameLoad attr Index Num attr orelse Expr Call NameLoad AttributeLoad AttributeLoad NameLoad attr attr
N/A N/A Tokenization help for Python programs.

generate_tokens(readline) is a generator that breaks a stream of
text into Python tokens.  It accepts a readline-like method which is called
repeatedly to get the next line of input (or "" for EOF).  It generates
5-tuples with these members:

    the token type (see token.py)
    the token (a string)
    the starting (row, column) indices of the token (a 2-tuple of ints)
    the ending (row, column) indices of the token (a 2-tuple of ints)
    the original line (string)

It is designed to match the working of the Python tokenizer exactly, except
that it produces COMMENT tokens for comments and gives type OP for all
operators

Older entry points
    tokenize_loop(readline, tokeneater)
    tokenize(readline, tokeneater=printtoken)
are the same, except instead of generating tokens, tokeneater is a callback
function to which the 5 fields described above are passed as 5 arguments,
each time a new token is found. N/A __author__ Ka-Ping Yee <ping@lfw.org> N/A __credits__ GvR, ESR, Tim Peters, Thomas Wouters, Fred Drake, Skip Montanaro, Raymond Hettinger N/A string re token * N/A token N/A __all__ N/A x N/A x N/A dir token N/A N/A N/A x startswith _ N/A __all__ N/A COMMENT tokenize generate_tokens NL untokenize N/A x N/A token N/A COMMENT N_TOKENS N/A N/A tok_name N/A COMMENT COMMENT N/A NL N/A N_TOKENS 1 N/A N/A tok_name N/A NL NL N/A N_TOKENS 2 group N/A N/A N/A choices N/A N/A N/A N/A ( N/A N/A | join choices ) N/A any N/A N/A N/A choices N/A N/A N/A N/A group choices * N/A maybe N/A N/A N/A choices N/A N/A N/A N/A group choices ? N/A N/A Whitespace [ \f\t]* N/A Comment #[^\r\n]* N/A Ignore N/A N/A Whitespace N/A any N/A \\\r?\n Whitespace N/A maybe Comment N/A Name [a-zA-Z_]\w* N/A Hexnumber 0[xX][\da-fA-F]+[lL]? N/A Octnumber (0[oO][0-7]+)|(0[0-7]*)[lL]? N/A Binnumber 0[bB][01]+[lL]? N/A Decnumber [1-9]\d*[lL]? N/A Intnumber N/A group Hexnumber Binnumber Octnumber Decnumber N/A Exponent [eE][-+]?\d+ N/A Pointfloat N/A N/A group \d+\.\d* \.\d+ N/A maybe Exponent N/A Expfloat N/A \d+ Exponent N/A Floatnumber N/A group Pointfloat Expfloat N/A Imagnumber N/A group \d+[jJ] N/A Floatnumber [jJ] N/A Number N/A group Imagnumber Floatnumber Intnumber N/A Single [^'\\]*(?:\\.[^'\\]*)*' N/A Double [^"\\]*(?:\\.[^"\\]*)*" N/A Single3 [^'\\]*(?:(?:\\.|'(?!''))[^'\\]*)*''' N/A Double3 [^"\\]*(?:(?:\\.|"(?!""))[^"\\]*)*""" N/A Triple N/A group [uU]?[rR]?''' [uU]?[rR]?""" N/A String N/A group [uU]?[rR]?'[^\n'\\]*(?:\\.[^\n'\\]*)*' [uU]?[rR]?"[^\n"\\]*(?:\\.[^\n"\\]*)*" N/A Operator N/A group \*\*=? >>=? <<=? <> != //=? [+\-*/%&|^=<>]=? ~ N/A Bracket [][(){}] N/A Special N/A group \r?\n [:;.,`@] N/A Funny N/A group Operator Bracket Special N/A PlainToken N/A group Number Funny String Name N/A Token N/A Ignore PlainToken N/A ContStr N/A group N/A [uU]?[rR]?'[^\n'\\]*(?:\\.[^\n'\\]*)* N/A group ' \\\r?\n N/A [uU]?[rR]?"[^\n"\\]*(?:\\.[^\n"\\]*)* N/A group " \\\r?\n N/A PseudoExtras N/A group \\\r?\n Comment Triple N/A PseudoToken N/A Whitespace N/A group PseudoExtras Number Funny ContStr Name N/A N/A tokenprog pseudoprog single3prog double3prog N/A map N/A re compile N/A Token PseudoToken Single3 Double3 N/A endprogs N/A ' " ''' """ r''' r""" u''' u""" ur''' ur""" R''' R""" U''' U""" uR''' uR""" Ur''' Ur""" UR''' UR""" b''' b""" br''' br""" B''' B""" bR''' bR""" Br''' Br""" BR''' BR""" r R u U b B N/A N/A re compile Single N/A N/A re compile Double single3prog double3prog single3prog double3prog single3prog double3prog single3prog double3prog single3prog double3prog single3prog double3prog single3prog double3prog single3prog double3prog single3prog double3prog single3prog double3prog single3prog double3prog single3prog double3prog single3prog double3prog single3prog double3prog single3prog double3prog None None None None None None N/A triple_quoted N/A N/A t N/A ''' """ r''' r""" R''' R""" u''' u""" U''' U""" ur''' ur""" Ur''' Ur""" uR''' uR""" UR''' UR""" b''' b""" B''' B""" br''' br""" Br''' Br""" bR''' bR""" BR''' BR""" N/A N/A N/A triple_quoted N/A t t N/A single_quoted N/A N/A t N/A ' " r' r" R' R" u' u" U' U" ur' ur" Ur' Ur" uR' uR" UR' UR" b' b" B' B" br' br" Br' Br" bR' bR" BR' BR" N/A N/A N/A single_quoted N/A t t N/A tabsize 8 TokenError N/A Exception N/A N/A N/A StopTokenizing N/A Exception N/A N/A N/A printtoken N/A N/A type token srow_scol erow_ecol line N/A N/A N/A N/A srow scol srow_scol N/A N/A erow ecol erow_ecol N/A N/A %d,%d-%d,%d:	%s	%s N/A srow scol erow ecol N/A tok_name N/A type N/A repr token N/A tokenize N/A N/A readline tokeneater N/A printtoken N/A N/A 
    The tokenize() function accepts two parameters: one representing the
    input stream, and one providing an output mechanism for tokenize().

    The first parameter, readline, must be a callable object which provides
    the same interface as the readline() method of built-in file objects.
    Each call to the function should return one line of input as a string.

    The second parameter, tokeneater, must also be a callable object. It is
    called once for each token, with five arguments, corresponding to the
    tuples generated by generate_tokens().
     N/A N/A N/A N/A tokenize_loop readline tokeneater N/A N/A N/A StopTokenizing N/A N/A N/A tokenize_loop N/A N/A readline tokeneater N/A N/A N/A token_info N/A generate_tokens readline N/A N/A N/A tokeneater token_info N/A Untokenizer N/A N/A __init__ N/A N/A self N/A N/A N/A N/A self tokens N/A N/A N/A self prev_row 1 N/A N/A self prev_col 0 N/A add_whitespace N/A N/A self start N/A N/A N/A N/A row col start N/A N/A row N/A self prev_row N/A col_offset N/A col N/A self prev_col N/A col_offset N/A N/A N/A N/A N/A self tokens append N/A col_offset N/A untokenize N/A N/A self iterable N/A N/A N/A t iterable N/A N/A N/A N/A len t 2 N/A N/A N/A N/A self compat t iterable N/A N/A N/A tok_type token start end line t N/A N/A N/A self add_whitespace start N/A N/A N/A N/A self tokens append token N/A N/A N/A self prev_row N/A self prev_col end N/A N/A tok_type N/A NEWLINE NL N/A N/A N/A self prev_row 1 N/A N/A self prev_col 0 N/A N/A N/A join N/A self tokens N/A compat N/A N/A self token iterable N/A N/A N/A startline False N/A indents N/A N/A toks_append N/A N/A self tokens append N/A N/A toknum tokval token N/A N/A toknum N/A NAME NUMBER N/A N/A tokval N/A N/A toknum N/A NEWLINE NL N/A N/A startline True N/A prevstring False N/A tok iterable N/A N/A N/A toknum tokval N/A tok N/A 2 N/A N/A toknum N/A NAME NUMBER N/A N/A tokval N/A N/A toknum STRING N/A N/A prevstring N/A N/A tokval N/A tokval N/A prevstring True N/A N/A prevstring False N/A N/A toknum INDENT N/A N/A N/A N/A indents append tokval N/A N/A N/A N/A toknum DEDENT N/A N/A N/A N/A indents pop N/A N/A N/A N/A toknum N/A NEWLINE NL N/A N/A startline True N/A N/A N/A startline indents N/A N/A N/A toks_append N/A indents N/A -1 N/A startline False N/A N/A toks_append tokval N/A N/A untokenize N/A N/A iterable N/A N/A N/A Transform tokens back into Python source code.

    Each element returned by the iterable must be a token sequence
    with at least two elements, a token number and token value.  If
    only two tokens are passed, the resulting output is poor.

    Round-trip invariant for full input:
        Untokenized source will match input source exactly

    Round-trip invariant for limited intput:
        # Output text will tokenize the back to the input
        t1 = [tok[:2] for tok in generate_tokens(f.readline)]
        newcode = untokenize(t1)
        readline = iter(newcode.splitlines(1)).next
        t2 = [tok[:2] for tok in generate_tokens(readline)]
        assert t1 == t2
     N/A ut N/A Untokenizer N/A N/A N/A ut untokenize iterable N/A generate_tokens N/A N/A readline N/A N/A N/A 
    The generate_tokens() generator requires one argment, readline, which
    must be a callable object which provides the same interface as the
    readline() method of built-in file objects. Each call to the function
    should return one line of input as a string.  Alternately, readline
    can be a callable function terminating with StopIteration:
        readline = open(myfile).next    # Example of alternate readline

    The generator produces 5-tuples with these members: the token type; the
    token string; a 2-tuple (srow, scol) of ints specifying the row and
    column where the token begins in the source; a 2-tuple (erow, ecol) of
    ints specifying the row and column where the token ends in the source;
    and the line on which the token was found. The line passed is the
    logical line; continuation lines are included.
     N/A lnum parenlev continued 0 N/A N/A namechars numchars N/A N/A N/A string ascii_letters _ 0123456789 N/A N/A contstr needcont N/A 0 N/A contline None N/A indents N/A 0 N/A 1 N/A N/A N/A N/A line N/A readline N/A N/A N/A StopIteration N/A N/A line N/A lnum 1 N/A N/A pos max N/A 0 N/A len line N/A contstr N/A N/A N/A line N/A N/A TokenError N/A EOF in multi-line string strstart N/A endmatch N/A N/A endprog match line N/A endmatch N/A N/A pos end N/A N/A endmatch end 0 N/A N/A N/A STRING N/A contstr N/A line N/A end strstart N/A lnum end N/A contline line N/A N/A contstr needcont N/A 0 N/A contline None N/A N/A N/A needcont N/A N/A line N/A -2 \
 N/A N/A line N/A -3 \
 N/A N/A N/A N/A ERRORTOKEN N/A contstr line strstart N/A lnum N/A len line contline N/A contstr N/A contline None N/A N/A N/A contstr N/A contstr line N/A contline N/A contline line N/A N/A N/A N/A N/A parenlev 0 N/A continued N/A N/A N/A line N/A N/A N/A column 0 N/A N/A pos max N/A N/A N/A N/A line N/A pos N/A N/A column 1 N/A N/A N/A N/A line N/A pos N/A N/A column N/A N/A N/A column tabsize 1 tabsize N/A N/A N/A N/A line N/A pos N/A N/A column 0 N/A N/A N/A pos 1 N/A N/A pos max N/A N/A N/A N/A N/A line N/A pos #
 N/A N/A N/A N/A line N/A pos # N/A N/A comment_token N/A N/A N/A line N/A pos rstrip N/A nl_pos N/A pos N/A len comment_token N/A N/A N/A COMMENT comment_token N/A lnum pos N/A lnum N/A pos N/A len comment_token line N/A N/A N/A NL N/A line N/A nl_pos N/A lnum nl_pos N/A lnum N/A len line line N/A N/A N/A N/A N/A N/A NL COMMENT N/A N/A N/A line N/A pos # N/A line N/A pos N/A lnum pos N/A lnum N/A len line line N/A N/A N/A column N/A indents N/A -1 N/A N/A N/A N/A indents append column N/A N/A N/A INDENT N/A line N/A pos N/A lnum 0 N/A lnum pos line N/A N/A column N/A indents N/A -1 N/A N/A N/A column indents N/A N/A N/A IndentationError unindent does not match any outer indentation level N/A <tokenize> lnum pos line N/A indents N/A indents N/A -1 N/A N/A N/A DEDENT N/A lnum pos N/A lnum pos line N/A N/A N/A line N/A N/A TokenError N/A EOF in multi-line statement N/A lnum 0 N/A continued 0 N/A N/A pos max N/A N/A pseudomatch N/A N/A pseudoprog match line pos N/A pseudomatch N/A N/A N/A start end N/A N/A pseudomatch span 1 N/A N/A spos epos pos N/A N/A lnum start N/A lnum end end N/A N/A token initial N/A N/A line N/A start end N/A line N/A start N/A N/A N/A initial numchars N/A N/A initial . N/A token . N/A N/A N/A N/A NUMBER token spos epos line N/A N/A N/A initial N/A N/A N/A N/A N/A N/A parenlev 0 NL NEWLINE token spos epos line N/A N/A N/A initial # N/A N/A N/A N/A N/A token endswith N/A N/A N/A COMMENT token spos epos line N/A N/A N/A token triple_quoted N/A N/A endprog N/A endprogs N/A token N/A endmatch N/A N/A endprog match line pos N/A endmatch N/A N/A pos N/A N/A endmatch end 0 N/A token N/A line N/A start pos N/A N/A N/A STRING token spos N/A lnum pos line N/A N/A strstart N/A lnum start N/A contstr N/A line N/A start N/A contline line N/A N/A N/A N/A N/A initial single_quoted N/A N/A token N/A 2 single_quoted N/A N/A token N/A 3 single_quoted N/A N/A N/A N/A token N/A -1 N/A N/A strstart N/A lnum start N/A endprog N/A N/A endprogs N/A initial N/A endprogs N/A N/A token N/A 1 N/A endprogs N/A N/A token N/A 2 N/A N/A contstr needcont N/A N/A line N/A start 1 N/A contline line N/A N/A N/A N/A N/A STRING token spos epos line N/A N/A N/A initial namechars N/A N/A N/A N/A NAME token spos epos line N/A N/A N/A initial \ N/A N/A continued 1 N/A N/A N/A initial ([{ N/A N/A parenlev 1 N/A N/A N/A initial )]} N/A N/A parenlev 1 N/A N/A N/A OP token spos epos line N/A N/A N/A N/A ERRORTOKEN N/A line N/A pos N/A lnum pos N/A lnum N/A pos 1 line N/A pos 1 N/A indent N/A indents N/A 1 N/A N/A N/A N/A DEDENT N/A lnum 0 N/A lnum 0 N/A N/A N/A ENDMARKER N/A lnum 0 N/A lnum 0 N/A N/A N/A __name__ __main__ N/A N/A sys N/A N/A N/A len N/A sys argv 1 N/A N/A N/A tokenize N/A N/A open N/A N/A sys argv N/A 1 readline N/A N/A N/A tokenize N/A N/A sys stdin readline