Module ImportFrom alias Import alias Assign NameStore Str Assign NameStore Str Assign NameStore NameLoad Assign NameStore ListLoad TupleLoad Str Str TupleLoad Str Str TupleLoad Str Str TupleLoad Str Str TupleLoad Str Str TupleLoad Str Str FunctionDef arguments args NameParam defaults body For TupleStore NameStore NameStore NameLoad body Assign NameStore Call AttributeLoad NameLoad attr NameLoad NameLoad Return NameLoad decorator_list FunctionDef arguments args NameParam defaults body For TupleStore NameStore NameStore NameLoad body Assign NameStore Call AttributeLoad NameLoad attr NameLoad NameLoad Return NameLoad decorator_list Assign TupleStore NameStore NameStore NameStore TupleLoad ListLoad ListLoad ListLoad Expr Call AttributeLoad NameLoad attr TupleLoad Call AttributeLoad NameLoad attr Str Str Expr Call AttributeLoad NameLoad attr TupleLoad Call AttributeLoad NameLoad attr Str Str Expr Call AttributeLoad NameLoad attr TupleLoad Call AttributeLoad NameLoad attr Str Str Expr Call AttributeLoad NameLoad attr TupleLoad Call AttributeLoad NameLoad attr Str Str Expr Call AttributeLoad NameLoad attr TupleLoad Call AttributeLoad NameLoad attr Str Str Expr Call AttributeLoad NameLoad attr TupleLoad Call AttributeLoad NameLoad attr Str Str Expr Call AttributeLoad NameLoad attr TupleLoad Call AttributeLoad NameLoad attr Str Str Expr Call AttributeLoad NameLoad attr TupleLoad Call AttributeLoad NameLoad attr Str Str Expr Call AttributeLoad NameLoad attr TupleLoad Call AttributeLoad NameLoad attr Str Str Expr Call AttributeLoad NameLoad attr TupleLoad Call AttributeLoad NameLoad attr Str Str Expr Call AttributeLoad NameLoad attr TupleLoad Call AttributeLoad NameLoad attr Str Str Expr Call AttributeLoad NameLoad attr TupleLoad Call AttributeLoad NameLoad attr Str Str Expr Call AttributeLoad NameLoad attr TupleLoad Call AttributeLoad NameLoad attr Str Str Expr Call AttributeLoad NameLoad attr TupleLoad Call AttributeLoad NameLoad attr Str Str Expr Call AttributeLoad NameLoad attr TupleLoad Call AttributeLoad NameLoad attr Str Str Expr Call AttributeLoad NameLoad attr TupleLoad Call AttributeLoad NameLoad attr Str Str Expr Call AttributeLoad NameLoad attr TupleLoad Call AttributeLoad NameLoad attr Str Str Expr Call AttributeLoad NameLoad attr TupleLoad Call AttributeLoad NameLoad attr Str Str Expr Call AttributeLoad NameLoad attr TupleLoad Call AttributeLoad NameLoad attr Str Str Expr Call AttributeLoad NameLoad attr TupleLoad Call AttributeLoad NameLoad attr Str Str Expr Call AttributeLoad NameLoad attr TupleLoad Call AttributeLoad NameLoad attr Str Str Expr Call AttributeLoad NameLoad attr TupleLoad Call AttributeLoad NameLoad attr Str Str Expr Call AttributeLoad NameLoad attr TupleLoad Call AttributeLoad NameLoad attr Str Str Expr Call AttributeLoad NameLoad attr TupleLoad Call AttributeLoad NameLoad attr Str Str Expr Call AttributeLoad NameLoad attr TupleLoad Call AttributeLoad NameLoad attr Str Str Expr Call AttributeLoad NameLoad attr TupleLoad Call AttributeLoad NameLoad attr Str Str Expr Call AttributeLoad NameLoad attr TupleLoad Call AttributeLoad NameLoad attr Str Str Expr Call AttributeLoad NameLoad attr TupleLoad Call AttributeLoad NameLoad attr Str Str Expr Call AttributeLoad NameLoad attr TupleLoad Call AttributeLoad NameLoad attr Str Str Expr Call AttributeLoad NameLoad attr TupleLoad Call AttributeLoad NameLoad attr Str Str Expr Call AttributeLoad NameLoad attr TupleLoad Call AttributeLoad NameLoad attr Str Str Expr Call AttributeLoad NameLoad attr TupleLoad Call AttributeLoad NameLoad attr Str Str Expr Call AttributeLoad NameLoad attr TupleLoad Call AttributeLoad NameLoad attr Str Str Expr Call AttributeLoad NameLoad attr TupleLoad Call AttributeLoad NameLoad attr Str Str Expr Call AttributeLoad NameLoad attr TupleLoad Call AttributeLoad NameLoad attr Str Str Expr Call AttributeLoad NameLoad attr TupleLoad Call AttributeLoad NameLoad attr Str Str Expr Call AttributeLoad NameLoad attr TupleLoad Call AttributeLoad NameLoad attr Str Str Expr Call AttributeLoad NameLoad attr TupleLoad Call AttributeLoad NameLoad attr Str Str Expr Call AttributeLoad NameLoad attr TupleLoad Call AttributeLoad NameLoad attr Str Str Expr Call AttributeLoad NameLoad attr TupleLoad Call AttributeLoad NameLoad attr Str Str Expr Call AttributeLoad NameLoad attr TupleLoad Call AttributeLoad NameLoad attr Str Str Expr Call AttributeLoad NameLoad attr TupleLoad Call AttributeLoad NameLoad attr Str Str Expr Call AttributeLoad NameLoad attr TupleLoad Call AttributeLoad NameLoad attr Str Str Expr Call AttributeLoad NameLoad attr TupleLoad Call AttributeLoad NameLoad attr Str Str Expr Call AttributeLoad NameLoad attr TupleLoad Call AttributeLoad NameLoad attr Str Str Expr Call AttributeLoad NameLoad attr TupleLoad Call AttributeLoad NameLoad attr Str Str Expr Call AttributeLoad NameLoad attr TupleLoad Call AttributeLoad NameLoad attr Str Str Expr Call AttributeLoad NameLoad attr TupleLoad Call AttributeLoad NameLoad attr Str Str Expr Call AttributeLoad NameLoad attr TupleLoad Call AttributeLoad NameLoad attr Str Str Expr Call AttributeLoad NameLoad attr TupleLoad Call AttributeLoad NameLoad attr Str Str Expr Call AttributeLoad NameLoad attr TupleLoad Call AttributeLoad NameLoad attr Str Str Expr Call AttributeLoad NameLoad attr TupleLoad Call AttributeLoad NameLoad attr Str Str Expr Call AttributeLoad NameLoad attr TupleLoad Call AttributeLoad NameLoad attr Str Str Expr Call AttributeLoad NameLoad attr TupleLoad Call AttributeLoad NameLoad attr Str Str Expr Call AttributeLoad NameLoad attr TupleLoad Call AttributeLoad NameLoad attr Str Str Expr Call AttributeLoad NameLoad attr TupleLoad Call AttributeLoad NameLoad attr Str Str Expr Call AttributeLoad NameLoad attr TupleLoad Call AttributeLoad NameLoad attr Str Str Expr Call AttributeLoad NameLoad attr TupleLoad Call AttributeLoad NameLoad attr Str Str Expr Call AttributeLoad NameLoad attr TupleLoad Call AttributeLoad NameLoad attr Str Str Expr Call AttributeLoad NameLoad attr TupleLoad Call AttributeLoad NameLoad attr Str Str Expr Call AttributeLoad NameLoad attr TupleLoad Call AttributeLoad NameLoad attr Str Str Expr Call AttributeLoad NameLoad attr TupleLoad Call AttributeLoad NameLoad attr Str Str Expr Call AttributeLoad NameLoad attr TupleLoad Call AttributeLoad NameLoad attr Str Str FunctionDef arguments args NameParam defaults body Expr Str For TupleStore NameStore NameStore NameLoad body Assign NameStore Call AttributeLoad NameLoad attr NameLoad NameLoad While NameLoad body Assign NameStore NameLoad For TupleStore NameStore NameStore NameLoad body Assign NameStore Call AttributeLoad NameLoad attr NameLoad NameLoad If CompareEq NameLoad NameLoad body Break For TupleStore NameStore NameStore NameLoad body Assign NameStore Call AttributeLoad NameLoad attr NameLoad NameLoad Return NameLoad decorator_list FunctionDef arguments args NameParam NameParam NameParam NameParam defaults NameLoad NameLoad NameLoad body Expr Str If NameLoad body Assign NameStore NameLoad Assign NameStore Call AttributeLoad NameLoad attr Str Str NameLoad Assign NameStore Call AttributeLoad NameLoad attr Str NameLoad Assert NameLoad BinOpMod Str NameLoad Assign TupleStore NameStore NameStore Call AttributeLoad NameLoad attr Assign NameStore Call AttributeLoad NameLoad attr Str Str NameLoad If NameLoad body If NameLoad body Assign NameStore Call AttributeLoad NameLoad attr Str BinOpAdd Str Str NameLoad orelse Assign NameStore Call AttributeLoad NameLoad attr Str Str NameLoad orelse Assign NameStore Call AttributeLoad NameLoad attr Str Str NameLoad Assign NameStore Call NameLoad NameLoad If NameLoad body If NameLoad body Assign NameStore Call AttributeLoad NameLoad attr Str Str orelse Assign NameStore Call AttributeLoad NameLoad attr Str Str orelse Assign NameStore Call AttributeLoad NameLoad attr Str Str If UnaryOpNot NameLoad body If UnaryOpNot NameLoad body Assign NameStore Call NameLoad NameLoad orelse Assign NameStore Call AttributeLoad NameLoad attr Str Str NameLoad Assign NameStore Call AttributeLoad NameLoad attr Str Str NameLoad Assign NameStore Call AttributeLoad NameLoad attr Str Str NameLoad Assign NameStore Call AttributeLoad NameLoad attr Str Str NameLoad Assign NameStore Call AttributeLoad NameLoad attr Str Str NameLoad Assign NameStore Call AttributeLoad NameLoad attr Str Str NameLoad Assign NameStore Call AttributeLoad NameLoad attr Str Str NameLoad Assign NameStore Call AttributeLoad NameLoad attr Str Str NameLoad Assign NameStore Call AttributeLoad NameLoad attr Str Str NameLoad If NameLoad body Assign NameStore Call NameLoad Call AttributeLoad Call AttributeLoad Call AttributeLoad Call AttributeLoad Call AttributeLoad NameLoad attr Str Str attr Str Str attr Str Str attr Str Str attr Str Str Assign NameStore Call NameLoad Call AttributeLoad Call AttributeLoad Call AttributeLoad Call AttributeLoad Call AttributeLoad NameLoad attr Str Str attr Str Str attr Str Str attr Str Str attr Str Str If CompareNotEq NameLoad NameLoad body Print AttributeLoad NameLoad attr BinOpMod Str TupleLoad NameLoad NameLoad Assign NameStore NameLoad Return BinOpAdd NameLoad NameLoad decorator_list FunctionDef arguments args defaults body Import alias Assign NameStore Call AttributeLoad NameLoad attr keyword Str Expr Call AttributeLoad NameLoad attr Str keyword NameLoad keyword Str keyword Str Expr Call AttributeLoad NameLoad attr Str keyword NameLoad keyword Str keyword Str Expr Call AttributeLoad NameLoad attr Str keyword NameLoad keyword Str keyword Str Expr Call AttributeLoad NameLoad attr Str keyword Str keyword Str keyword Str Return NameLoad decorator_list FunctionDef arguments args NameParam defaults body Import alias Import alias Assign NameStore Call AttributeLoad Call NameLoad attr SubscriptLoad NameLoad Slice Num Assign TupleStore NameStore NameStore NameStore TupleLoad NameLoad NameLoad NameLoad If AttributeLoad NameLoad attr body Assign NameStore NameLoad If AttributeLoad NameLoad attr body Assign NameStore NameLoad Assign NameStore NameLoad If AttributeLoad NameLoad attr body Assign NameStore NameLoad If CompareEq Call NameLoad AttributeLoad NameLoad attr Num body Expr Call AttributeLoad AttributeLoad NameLoad attr attr Str For NameStore AttributeLoad NameLoad attr body TryExcept body With Call AttributeLoad NameLoad attr NameLoad keyword NameLoad NameStore body For NameStore NameLoad body Assign NameStore Call NameLoad NameLoad keyword NameLoad keyword NameLoad keyword NameLoad Expr Call AttributeLoad AttributeLoad NameLoad attr attr Call AttributeLoad NameLoad attr NameLoad handlers ExceptHandler type NameLoad name NameStore body Print AttributeLoad NameLoad attr Str NameLoad Str NameLoad decorator_list If CompareEq NameLoad Str body Import alias Expr Call AttributeLoad NameLoad attr Call NameLoad AttributeLoad NameLoad attr
N/A __future__ with_statement N/A re N/A INPUT_ENCODING UTF-8 N/A OUTPUT_ENCODING UTF-8 N/A DEBUG_GTB_TOKENIZATION False N/A PTB_ESCAPES N/A N/A ( -LRB- N/A ) -RRB- N/A [ -LSB- N/A ] -RSB- N/A { -LCB- N/A } -RCB- PTB_escape N/A N/A s N/A N/A N/A N/A u e PTB_ESCAPES N/A N/A s N/A N/A s replace u e N/A s N/A PTB_unescape N/A N/A s N/A N/A N/A N/A u e PTB_ESCAPES N/A N/A s N/A N/A s replace e u N/A s N/A N/A N/A __initial __repeated __final N/A N/A N/A N/A N/A N/A N/A __initial append N/A N/A N/A re compile \.\.\.  ...  N/A N/A N/A __initial append N/A N/A N/A re compile ([,;:@#])   \1  N/A N/A N/A __initial append N/A N/A N/A re compile  ([,;:@#])  \1  N/A N/A N/A __initial append N/A N/A N/A re compile \$  $  N/A N/A N/A __initial append N/A N/A N/A re compile \%  %  N/A N/A N/A __initial append N/A N/A N/A re compile \&  &  N/A N/A N/A __initial append N/A N/A N/A re compile ([,:;])([\[\]\)\}\>\"\']* +)  \1\2 N/A N/A N/A __initial append N/A N/A N/A re compile (\.+)([\[\]\)\}\>\"\']* +)$  \1\2 N/A N/A N/A __initial append N/A N/A N/A re compile \?  ?  N/A N/A N/A __initial append N/A N/A N/A re compile \!  !  N/A N/A N/A __initial append N/A N/A N/A re compile ((?:=\/)?<+(?:\/=|--+>?)?)  \1  N/A N/A N/A __initial append N/A N/A N/A re compile ((?:<?--+|=\/)?>+(?:\/=)?)  \1  N/A N/A N/A __initial append N/A N/A N/A re compile (<?--+\>?)  \1  N/A N/A N/A __initial append N/A N/A N/A re compile \(([^ A-Z()\[\]{}]+)\)- -LRB-\1-RRB-- N/A N/A N/A __repeated append N/A N/A N/A re compile (?<![ (\[{])\(([^ ()\[\]{}]*)\) -LRB-\1-RRB- N/A N/A N/A __repeated append N/A N/A N/A re compile \(([^ ()\[\]{}]*)\)(?![ )\]}\/-]) -LRB-\1-RRB- N/A N/A N/A __repeated append N/A N/A N/A re compile (?<![ (\[{])\[([^ ()\[\]{}]*)\] -LSB-\1-RSB- N/A N/A N/A __repeated append N/A N/A N/A re compile \[([^ ()\[\]{}]*)\](?![ )\]}\/-]) -LSB-\1-RSB- N/A N/A N/A __repeated append N/A N/A N/A re compile (?<![ (\[{])\{([^ ()\[\]{}]*)\} -LCB-\1-RCB- N/A N/A N/A __repeated append N/A N/A N/A re compile \{([^ ()\[\]{}]*)\}(?![ )\]}\/-]) -LCB-\1-RCB- N/A N/A N/A __final append N/A N/A N/A re compile \(  -LRB-  N/A N/A N/A __final append N/A N/A N/A re compile \)  -RRB-  N/A N/A N/A __final append N/A N/A N/A re compile \[  -LSB-  N/A N/A N/A __final append N/A N/A N/A re compile \]  -RSB-  N/A N/A N/A __final append N/A N/A N/A re compile \{  -LCB-  N/A N/A N/A __final append N/A N/A N/A re compile \}  -RCB-  N/A N/A N/A __final append N/A N/A N/A re compile  (\'+)  \1  N/A N/A N/A __final append N/A N/A N/A re compile (?<![35\'])(\'+)   \1  N/A N/A N/A __final append N/A N/A N/A re compile \'s   's  N/A N/A N/A __final append N/A N/A N/A re compile \'S   'S  N/A N/A N/A __final append N/A N/A N/A re compile \'m   'm  N/A N/A N/A __final append N/A N/A N/A re compile \'M   'M  N/A N/A N/A __final append N/A N/A N/A re compile \'d   'd  N/A N/A N/A __final append N/A N/A N/A re compile \'D   'D  N/A N/A N/A __final append N/A N/A N/A re compile \'ll   'll  N/A N/A N/A __final append N/A N/A N/A re compile \'re   're  N/A N/A N/A __final append N/A N/A N/A re compile \'ve   've  N/A N/A N/A __final append N/A N/A N/A re compile n\'t   n't  N/A N/A N/A __final append N/A N/A N/A re compile \'LL   'LL  N/A N/A N/A __final append N/A N/A N/A re compile \'RE   'RE  N/A N/A N/A __final append N/A N/A N/A re compile \'VE   'VE  N/A N/A N/A __final append N/A N/A N/A re compile N\'T   N'T  N/A N/A N/A __final append N/A N/A N/A re compile  Cannot   Can not  N/A N/A N/A __final append N/A N/A N/A re compile  cannot   can not  N/A N/A N/A __final append N/A N/A N/A re compile  D\'ye   D' ye  N/A N/A N/A __final append N/A N/A N/A re compile  d\'ye   d' ye  N/A N/A N/A __final append N/A N/A N/A re compile  Gimme   Gim me  N/A N/A N/A __final append N/A N/A N/A re compile  gimme   gim me  N/A N/A N/A __final append N/A N/A N/A re compile  Gonna   Gon na  N/A N/A N/A __final append N/A N/A N/A re compile  gonna   gon na  N/A N/A N/A __final append N/A N/A N/A re compile  Gotta   Got ta  N/A N/A N/A __final append N/A N/A N/A re compile  gotta   got ta  N/A N/A N/A __final append N/A N/A N/A re compile  Lemme   Lem me  N/A N/A N/A __final append N/A N/A N/A re compile  lemme   lem me  N/A N/A N/A __final append N/A N/A N/A re compile  More\'n   More 'n  N/A N/A N/A __final append N/A N/A N/A re compile  more\'n   more 'n  N/A N/A N/A __final append N/A N/A N/A re compile \'Tis   'T is  N/A N/A N/A __final append N/A N/A N/A re compile \'tis   't is  N/A N/A N/A __final append N/A N/A N/A re compile \'Twas   'T was  N/A N/A N/A __final append N/A N/A N/A re compile \'twas   't was  N/A N/A N/A __final append N/A N/A N/A re compile  Wanna   Wan na  N/A N/A N/A __final append N/A N/A N/A re compile  wanna   wan na  N/A N/A N/A __final append N/A N/A N/A re compile   + _tokenize N/A N/A s N/A N/A N/A 
    Tokenizer core. Performs GTP-like tokenization, using PTB escapes
    for brackets (but not quotes). Assumes given string has initial
    and terminating space. You probably want to use tokenize() instead
    of this function.
     N/A N/A r t __initial N/A N/A s N/A N/A r sub t s N/A True N/A N/A o s N/A N/A r t __repeated N/A N/A s N/A N/A r sub t s N/A N/A o s N/A N/A N/A N/A r t __final N/A N/A s N/A N/A r sub t s N/A s N/A tokenize N/A N/A s ptb_escaping use_single_quotes_only escape_token_internal_parens N/A False False False N/A N/A 
    Tokenizes the given string with a GTB-like tokenization. Input
    will adjusted by removing surrounding space, if any. Arguments
    hopefully self-explanatory.
     N/A DEBUG_GTB_TOKENIZATION N/A N/A orig s N/A s N/A N/A re sub ^ s N/A m N/A N/A re match ^((?:.+|\n)*?) *(\n*)$ s N/A m N/A INTERNAL ERROR on '%s' s N/A N/A s s_end N/A N/A m groups N/A s N/A N/A re sub $ s N/A ptb_escaping N/A N/A use_single_quotes_only N/A N/A s N/A N/A re sub ([ \(\[\{\<])\" N/A \1  '  s N/A N/A s N/A N/A re sub ([ \(\[\{\<])\" \1 ``  s N/A N/A s N/A N/A re sub ([ \(\[\{\<])\" \1 "  s N/A s N/A _tokenize s N/A ptb_escaping N/A N/A use_single_quotes_only N/A N/A s N/A N/A s replace "  '  N/A N/A s N/A N/A s replace "  ''  N/A N/A s N/A N/A s replace "  "  N/A N/A ptb_escaping N/A N/A N/A escape_token_internal_parens N/A N/A s N/A PTB_unescape s N/A N/A s N/A N/A re sub (?<= )-LRB-(?= ) ( s N/A s N/A N/A re sub (?<= )-RRB-(?= ) ) s N/A s N/A N/A re sub (?<= )-LSB-(?= ) [ s N/A s N/A N/A re sub (?<= )-RSB-(?= ) ] s N/A s N/A N/A re sub (?<= )-LCB-(?= ) { s N/A s N/A N/A re sub (?<= )-RCB-(?= ) } s N/A s N/A N/A re sub   + s N/A s N/A N/A re sub ^ + s N/A s N/A N/A re sub  +$ s N/A DEBUG_GTB_TOKENIZATION N/A N/A r1 N/A PTB_unescape N/A N/A N/A N/A N/A N/A N/A N/A N/A N/A orig replace replace replace ' replace " replace `` N/A r2 N/A PTB_unescape N/A N/A N/A N/A N/A N/A N/A N/A N/A N/A s replace replace replace ' replace " replace `` N/A N/A r1 r2 N/A N/A N/A sys stderr N/A tokenize(): error: text mismatch (returning original):
ORIG: '%s'
NEW:  '%s' N/A orig s N/A s orig N/A N/A s s_end N/A __argparser N/A N/A N/A N/A N/A argparse N/A ap N/A N/A argparse ArgumentParser description Perform GENIA Treebank-like text tokenization. N/A N/A N/A ap add_argument -ptb default False action store_true help Use Penn Treebank escapes N/A N/A N/A ap add_argument -mccc default False action store_true help Special processing for McClosky-Charniak-Johnson parser input N/A N/A N/A ap add_argument -sp default False action store_true help Special processing for Stanford parser+PTBEscapingProcessor input. (not necessary for Stanford Parser version 1.6.5 and newer) N/A N/A N/A ap add_argument files metavar FILE nargs * help Files to tokenize. N/A ap N/A main N/A N/A argv N/A N/A N/A sys N/A codecs N/A arg N/A N/A N/A __argparser parse_args N/A argv N/A 1 N/A N/A ptb_escaping use_single_quotes_only escape_token_internal_parens N/A False False False N/A N/A arg ptb N/A N/A ptb_escaping True N/A N/A arg mccc N/A N/A ptb_escaping True N/A use_single_quotes_only True N/A N/A arg sp N/A N/A escape_token_internal_parens True N/A N/A N/A len N/A arg files 0 N/A N/A N/A N/A N/A arg files append /dev/stdin N/A fn N/A arg files N/A N/A N/A N/A N/A N/A codecs open fn encoding INPUT_ENCODING f N/A N/A l f N/A N/A t N/A tokenize l ptb_escaping ptb_escaping use_single_quotes_only use_single_quotes_only escape_token_internal_parens escape_token_internal_parens N/A N/A N/A N/A sys stdout write N/A N/A t encode OUTPUT_ENCODING N/A N/A N/A Exception N/A e N/A N/A N/A sys stderr Failed to read fn : e N/A N/A N/A __name__ __main__ N/A N/A sys N/A N/A N/A sys exit N/A main N/A sys argv