Module Import alias Import alias Import alias ImportFrom alias Import alias Import alias FunctionDef arguments args defaults body Assign NameStore Call AttributeLoad AttributeLoad NameLoad attr attr Call AttributeLoad AttributeLoad NameLoad attr attr NameLoad Assign NameStore Call AttributeLoad NameLoad attr Expr Call AttributeLoad NameLoad attr Str keyword Str Expr Call AttributeLoad NameLoad attr Str keyword Call AttributeLoad NameLoad attr Str keyword AttributeLoad NameLoad attr keyword Str keyword Str Expr Call AttributeLoad NameLoad attr Str Str keyword Str keyword Str Assign NameStore Call AttributeLoad NameLoad attr If AttributeLoad NameLoad attr body Assign NameStore Call NameLoad NameLoad Call AttributeLoad AttributeLoad NameLoad attr attr NameLoad If UnaryOpNot Call NameLoad NameLoad NameLoad body Raise Call NameLoad BinOpMod Str NameLoad Expr Call AttributeLoad NameLoad attr keyword NameLoad keyword Str keyword Str If CompareEq AttributeLoad NameLoad attr Str body Expr Call NameLoad AttributeLoad NameLoad attr orelse If CompareEq AttributeLoad NameLoad attr Str body Expr Call AttributeLoad NameLoad attr Str Expr Call NameLoad AttributeLoad NameLoad attr keyword ListLoad Str orelse Expr Call AttributeLoad NameLoad attr Str decorator_list FunctionDef arguments args NameParam NameParam defaults NameLoad body Expr Str Assign NameStore Call NameLoad Str Str Assign NameStore Call NameLoad For NameStore NameLoad body Assign TupleStore NameStore NameStore NameStore Call AttributeLoad NameLoad attr NameLoad Assign NameStore Call NameLoad NameLoad TryExcept body AugAssignAdd SubscriptStore NameLoad Index NameLoad NameLoad handlers ExceptHandler type NameLoad body Assign SubscriptStore NameLoad Index NameLoad NameLoad Assign NameStore ListComp TupleLoad NameLoad NameLoad comprehension TupleStore NameStore NameStore Call AttributeLoad NameLoad attr Expr Call AttributeLoad NameLoad attr Expr Call AttributeLoad NameLoad attr Assign NameStore Num For TupleStore NameStore NameStore NameLoad body AugAssignAdd NameStore Num Expr Call AttributeLoad NameLoad attr BinOpAdd Call AttributeLoad Str attr ListLoad Call NameLoad NameLoad Call AttributeLoad NameLoad attr Str Str Call NameLoad NameLoad Str decorator_list If CompareEq NameLoad Str body Expr Call NameLoad
N/A N/A os N/A subprocess N/A sys tokenizer * N/A logging N/A argparse main N/A N/A N/A N/A N/A path N/A N/A N/A os path dirname N/A N/A N/A os path realpath __file__ N/A parser N/A N/A argparse ArgumentParser N/A N/A N/A parser add_argument action help set to 'encode' or 'wordIds' N/A N/A N/A parser add_argument unigrams type N/A N/A argparse FileType r default N/A sys stdin nargs ? help Unigram feature file input. Defaults to sys.stdin N/A N/A N/A parser add_argument --log-level -l default warn help Logging level. N/A args N/A N/A parser parse_args N/A N/A args log_level N/A N/A numeric_level N/A getattr logging N/A N/A N/A args log_level upper None N/A N/A N/A isinstance numeric_level int N/A N/A N/A ValueError N/A Invalid log level: %s loglevel N/A N/A N/A logging basicConfig level numeric_level format %(asctime)s:%(levelname)s:%(message)s datefmt %d/%Y %H:%M:%S N/A N/A N/A args action wordIds N/A N/A N/A writeWordIDs N/A args unigrams N/A N/A N/A N/A args action encode N/A N/A N/A N/A logging debug Starting feature encoding process N/A N/A encodePreTokenizedStream N/A args unigrams levels N/A unigrams N/A N/A N/A N/A logging error Need to specify action as either 'wordIds' or 'encode' N/A writeWordIDs N/A N/A featurefile sep N/A None N/A N/A 
    The wordids are counted directly from the unigrams file.

    Filename: location of unigrams.txt
    sep: Delimiter. Defaults to None (whitespace), use this for tab- or 
            comma-delimiting. 
     N/A output N/A open .bookworm/texts/wordlist/wordlist.txt w N/A wordcounts N/A dict N/A line featurefile N/A N/A N/A bookid word count N/A N/A line split sep N/A count N/A int count N/A N/A N/A N/A wordcounts N/A word count N/A N/A N/A KeyError N/A N/A N/A wordcounts N/A word count N/A tuples N/A N/A v k N/A N/A k v N/A N/A wordcounts iteritems N/A N/A N/A tuples sort N/A N/A N/A tuples reverse N/A wordid 0 N/A N/A count word tuples N/A N/A wordid 1 N/A N/A N/A output write N/A N/A N/A join N/A N/A str wordid N/A N/A word replace \ \\ N/A str count N/A N/A N/A __name__ __main__ N/A N/A N/A main