Module Expr Str Assign NameStore Num Assign NameStore Num Import alias Import alias Import alias identifier ImportFrom alias alias alias ImportFrom alias ImportFrom alias ImportFrom alias alias alias If CompareEq NameLoad Str body Assign NameStore Call AttributeLoad NameLoad attr Expr Call AttributeLoad NameLoad attr Str Str keyword Str keyword Str Expr Call AttributeLoad NameLoad attr Str Str keyword Str keyword Str Expr Call AttributeLoad NameLoad attr Str Str keyword Str keyword Str Expr Call AttributeLoad NameLoad attr Str Str keyword NameLoad keyword Str keyword Str Expr Call AttributeLoad NameLoad attr Str Str keyword Str keyword Str Expr Call AttributeLoad NameLoad attr Str keyword NameLoad keyword Str keyword Str keyword NameLoad Expr Call AttributeLoad NameLoad attr Str keyword NameLoad keyword Str keyword NameLoad Expr Call AttributeLoad NameLoad attr Str keyword NameLoad keyword Str keyword Str Expr Call AttributeLoad NameLoad attr Str keyword Str keyword Str Expr Call AttributeLoad NameLoad attr Str keyword Str keyword Str keyword Str Assign NameStore Call AttributeLoad NameLoad attr Assign NameStore Call AttributeLoad AttributeLoad NameLoad attr attr AttributeLoad NameLoad attr If AttributeLoad NameLoad attr body Assign NameStore AttributeLoad NameLoad attr orelse Assign NameStore Call AttributeLoad AttributeLoad NameLoad attr attr Str BinOpAdd NameLoad Str FunctionDef arguments args NameParam defaults body Return Call AttributeLoad AttributeLoad NameLoad attr attr NameLoad NameLoad decorator_list If AttributeLoad NameLoad attr body Assign NameStore AttributeLoad NameLoad attr orelse If Call AttributeLoad AttributeLoad NameLoad attr attr Call NameLoad Str body Assign NameStore Call NameLoad Str orelse Raise Call NameLoad Str If AttributeLoad NameLoad attr body Assign NameStore AttributeLoad NameLoad attr orelse Assign NameStore Call NameLoad Str Expr Call NameLoad NameLoad Assign NameStore Call NameLoad Assign NameStore ListLoad For NameStore AttributeLoad NameLoad attr body TryExcept body Assign TupleStore NameStore NameStore Call AttributeLoad NameLoad attr Str handlers ExceptHandler type NameLoad body Expr Call AttributeLoad NameLoad attr Call AttributeLoad Str attr NameLoad Expr Call AttributeLoad NameLoad attr NameLoad Expr Call AttributeLoad NameLoad attr NameLoad Expr Call AttributeLoad NameLoad attr TupleLoad NameLoad NameLoad If AttributeLoad NameLoad attr body Expr Call NameLoad AttributeLoad NameLoad attr Assign NameStore AttributeLoad NameLoad attr orelse Assign NameStore NameLoad Assign NameStore Call NameLoad NameLoad Print Call AttributeLoad Str attr NameLoad Call NameLoad NameLoad Print Str NameLoad Print Str NameLoad Print Str NameLoad Print Str NameLoad Assign NameStore Call NameLoad NameLoad Assign NameStore Call NameLoad AttributeLoad NameLoad attr keyword NameLoad Assign NameStore ListComp TupleLoad NameLoad NameLoad NameLoad comprehension TupleStore NameStore NameStore NameStore NameStore AttributeLoad NameLoad attr If CompareEq Call NameLoad NameLoad Num body Raise Call NameLoad Str Print Call AttributeLoad Str attr Call NameLoad NameLoad Call NameLoad NameLoad Assign TupleStore NameStore NameStore Call NameLoad NameLoad Assign NameStore Call NameLoad ListComp TupleLoad NameLoad NameLoad comprehension TupleStore NameStore NameStore NameStore NameLoad Call NameLoad NameLoad Print Str Assign NameStore SubscriptLoad Call NameLoad NameLoad Index Num Assign NameStore Call NameLoad NameLoad NameLoad NameLoad NameLoad NameLoad NameLoad Assign NameStore Call AttributeLoad Call AttributeLoad NameLoad attr NameLoad attr Call NameLoad NameLoad Call NameLoad NameLoad Print Str For NameStore Call NameLoad SubscriptLoad AttributeLoad NameLoad attr Index Num body Assign NameStore Call AttributeLoad NameLoad attr SubscriptLoad NameLoad ExtSlice Slice Index NameLoad For NameStore Call NameLoad SubscriptLoad AttributeLoad NameLoad attr Index Num body Assign SubscriptStore NameLoad Index NameLoad BinOpDiv Num Call AttributeLoad Call AttributeLoad NameLoad attr BinOpSub SubscriptLoad NameLoad ExtSlice Slice Index NameLoad SubscriptLoad NameLoad Index TupleLoad NameLoad NameLoad attr Assign SubscriptStore NameLoad ExtSlice Slice Index NameLoad NameLoad Assert CompareLt BinOpSub Num Call AttributeLoad NameLoad attr Num Print Str For TupleStore NameStore NameStore NameLoad body If AttributeLoad NameLoad attr body Assign NameStore Call AttributeLoad AttributeLoad NameLoad attr attr NameLoad Call AttributeLoad Str attr NameLoad NameLoad orelse Assign NameStore Call AttributeLoad AttributeLoad NameLoad attr attr NameLoad Call AttributeLoad Str attr NameLoad NameLoad Assign NameStore SubscriptLoad AttributeLoad NameLoad attr Index NameLoad Assign NameStore SubscriptLoad AttributeLoad NameLoad attr Index NameLoad Assign NameStore Call NameLoad Call NameLoad NameLoad Call AttributeLoad NameLoad attr BinOpDiv BinOpSub SubscriptLoad NameLoad ExtSlice Slice Index NameLoad SubscriptLoad NameLoad ExtSlice Slice Index NameLoad IfExp UnaryOpNot AttributeLoad NameLoad attr Call AttributeLoad NameLoad attr Num Num Expr Call NameLoad NameLoad NameLoad Print Call AttributeLoad Str attr NameLoad
N/A N/A 
Implementing the "blacklist" feature weighting metric proposed by
Tiedemann & Ljubesic.

Marco Lui, February 2013
 N/A NUM_BUCKETS 64 N/A CHUNKSIZE 50 N/A os N/A argparse N/A numpy np common read_features makedir write_weights scanner build_scanner index CorpusIndexer NBtrain generate_cm learn_pc learn_ptc N/A N/A __name__ __main__ N/A N/A parser N/A N/A argparse ArgumentParser N/A N/A N/A parser add_argument -o --output metavar DIR help write weights to DIR N/A N/A N/A parser add_argument -f --features metavar FILE help only output features from FILE N/A N/A N/A parser add_argument -t --temp metavar TEMP_DIR help store buckets in TEMP_DIR instead of in MODEL_DIR/buckets N/A N/A N/A parser add_argument -j --jobs type int metavar N help spawn N processes (set to 1 for no paralleization) N/A N/A N/A parser add_argument -m --model help save output to MODEL_DIR metavar MODEL_DIR N/A N/A N/A parser add_argument --buckets type int metavar N help distribute features into N buckets default NUM_BUCKETS N/A N/A N/A parser add_argument --chunksize type int help max chunk size (number of files to tokenize at a time - smaller should reduce memory use) default CHUNKSIZE N/A N/A N/A parser add_argument --no_norm default False action store_true help do not normalize difference in p(t|C) by sum p(t|C) N/A N/A N/A parser add_argument corpus help read corpus from CORPUS_DIR metavar CORPUS_DIR N/A N/A N/A parser add_argument pairs metavar LANG_PAIR nargs * help language pairs to compute BL weights for N/A args N/A N/A parser parse_args N/A corpus_name N/A N/A N/A os path basename N/A args corpus N/A N/A args model N/A N/A model_dir N/A args model N/A N/A model_dir N/A N/A N/A os path join . N/A corpus_name .model m_path N/A N/A name N/A N/A N/A N/A N/A N/A os path join model_dir name N/A N/A N/A args features N/A N/A feat_path N/A args features N/A N/A N/A N/A N/A os path exists N/A m_path DFfeats N/A N/A feat_path N/A m_path DFfeats N/A N/A N/A ValueError no suitable feature list N/A N/A args temp N/A N/A buckets_dir N/A args temp N/A N/A buckets_dir N/A m_path buckets N/A N/A makedir buckets_dir N/A all_langs N/A set N/A pairs N/A N/A p N/A args pairs N/A N/A N/A N/A N/A lang1 lang2 N/A N/A p split , N/A N/A N/A ValueError N/A N/A N/A N/A parser error N/A N/A {0} is not a lang-pair format p N/A N/A N/A all_langs add lang1 N/A N/A N/A all_langs add lang2 N/A N/A N/A pairs append N/A lang1 lang2 N/A N/A args output N/A N/A N/A makedir N/A args output N/A out_dir N/A args output N/A N/A out_dir model_dir N/A langs N/A sorted all_langs N/A N/A N/A languages({1}): {0} format langs N/A len langs N/A model path: model_dir N/A feature path: feat_path N/A output path: out_dir N/A temp (buckets) path: buckets_dir N/A feats N/A read_features feat_path N/A indexer N/A CorpusIndexer N/A args corpus langs langs N/A items N/A N/A d l p N/A N/A d l n p N/A indexer items N/A N/A N/A len items 0 N/A N/A N/A ValueError found no files! N/A N/A N/A will process {0} features across {1} paths format N/A len feats N/A len items N/A N/A tk_nextmove tk_output N/A build_scanner feats N/A cm N/A generate_cm N/A N/A l p N/A N/A d l p items N/A len langs N/A learning P(t|C) N/A paths N/A N/A zip items N/A 2 N/A nb_ptc N/A learn_ptc paths tk_nextmove tk_output cm buckets_dir args N/A nb_ptc N/A N/A N/A N/A np array nb_ptc reshape N/A len feats N/A len langs N/A renormalizing P(t|C) N/A i N/A range N/A N/A nb_ptc shape N/A 1 N/A N/A newval N/A N/A np empty_like N/A nb_ptc N/A N/A N/A i N/A j N/A range N/A N/A newval shape N/A 0 N/A N/A N/A newval N/A j N/A 1 N/A N/A N/A N/A np exp N/A N/A nb_ptc N/A N/A N/A i N/A nb_ptc N/A N/A j i sum N/A N/A nb_ptc N/A N/A N/A i newval N/A N/A N/A 1.0 N/A N/A newval sum 0.0001 N/A doing per-pair output N/A N/A lang1 lang2 pairs N/A N/A N/A args no_norm N/A N/A weights_path N/A N/A N/A os path join out_dir N/A N/A BLfeats.no_norm.{0}.{1} format lang1 lang2 N/A N/A weights_path N/A N/A N/A os path join out_dir N/A N/A BLfeats.{0}.{1} format lang1 lang2 N/A i1 N/A N/A indexer lang_index N/A lang1 N/A i2 N/A N/A indexer lang_index N/A lang2 N/A w N/A dict N/A zip feats N/A N/A np abs N/A N/A N/A nb_ptc N/A N/A N/A i1 N/A nb_ptc N/A N/A N/A i2 N/A N/A N/A args no_norm N/A N/A nb_ptc sum 1 1 N/A N/A write_weights w weights_path N/A N/A N/A wrote weights to {0} format weights_path