Module Expr Str Import alias ImportFrom alias ImportFrom alias alias alias alias alias alias alias ImportFrom alias alias ImportFrom alias alias alias alias identifier Assign NameStore AttributeLoad NameLoad attr Assign NameStore Call AttributeLoad NameLoad attr Str Expr Call AttributeLoad NameLoad attr NameLoad Assign NameStore Call AttributeLoad NameLoad attr Expr Call AttributeLoad NameLoad attr NameLoad Expr Call AttributeLoad NameLoad attr NameLoad Assign NameStore AttributeLoad NameLoad attr Assign NameStore AttributeLoad NameLoad attr Assign NameStore AttributeLoad NameLoad attr FunctionDef arguments args defaults body Assign NameStore Call NameLoad keyword Str keyword Str keyword NameLoad Assign NameStore AttributeLoad NameLoad attr Expr Call NameLoad Str keyword Str keyword NameLoad keyword NameLoad Expr Call NameLoad Str Str keyword Str keyword NameLoad keyword Num Expr Call NameLoad Str Str keyword Str keyword NameLoad keyword Num Expr Call NameLoad Str Str keyword Str keyword NameLoad keyword NameLoad Expr Call NameLoad Str keyword Str keyword NameLoad keyword NameLoad Expr Call NameLoad Str Str keyword Str keyword Str keyword NameLoad Expr Call NameLoad Str keyword Str keyword Str keyword NameLoad Expr Call NameLoad Str keyword Str keyword NameLoad keyword NameLoad Expr Call NameLoad Str Str keyword Str keyword NameLoad keyword Str Expr Call NameLoad Str Str keyword Str keyword NameLoad keyword Num Expr Call NameLoad Str Str keyword Str keyword NameLoad keyword Call NameLoad NameLoad Str Assign NameStore Call AttributeLoad NameLoad attr ImportFrom alias ImportFrom alias ImportFrom alias ImportFrom alias ImportFrom alias alias Assign NameStore AttributeLoad NameLoad attr Assign NameStore AttributeLoad NameLoad attr Assign NameStore AttributeLoad NameLoad attr Assign NameStore AttributeLoad NameLoad attr Assign NameStore AttributeLoad NameLoad attr Assign NameStore AttributeLoad NameLoad attr Assign NameStore AttributeLoad NameLoad attr Assign NameStore AttributeLoad NameLoad attr Assign NameStore AttributeLoad NameLoad attr Assign NameStore AttributeLoad NameLoad attr Assign NameStore Call NameLoad AttributeLoad NameLoad attr Assign NameStore Call NameLoad NameLoad If UnaryOpNot Call NameLoad NameLoad body Expr Call NameLoad NameLoad keyword NameLoad Assign NameStore Call AttributeLoad NameLoad attr NameLoad Expr Call AttributeLoad NameLoad attr NameLoad Expr Call AttributeLoad NameLoad attr NameLoad Expr Call AttributeLoad NameLoad attr NameLoad If BoolOpAnd NameLoad CompareLt NameLoad Num body Assign NameStore Str Expr Call NameLoad NameLoad Raise Call NameLoad NameLoad orelse If NameLoad body Assign NameStore NameLoad orelse Assign NameStore Num If BoolOpAnd NameLoad UnaryOpNot NameLoad body Assign NameStore Str Expr Call NameLoad NameLoad Raise Call NameLoad NameLoad Expr Call NameLoad Call AttributeLoad Str attr NameLoad NameLoad TryExcept body Assign NameStore Call NameLoad keyword NameLoad keyword NameLoad handlers ExceptHandler type NameLoad name NameStore body Expr Call NameLoad Str Expr Call NameLoad NameLoad Raise NameLoad Assign SubscriptStore AttributeLoad NameLoad attr Index Str Num If NameLoad body Assign NameStore Call NameLoad NameLoad If Call NameLoad NameLoad body Assign NameStore Str Expr Call NameLoad NameLoad Raise Call NameLoad NameLoad If UnaryOpNot Call NameLoad NameLoad body Expr Call NameLoad NameLoad keyword NameLoad If CompareLt NameLoad Num body Assign NameStore Str Expr Call NameLoad NameLoad Raise Call NameLoad NameLoad If CompareLt NameLoad Num body Assign NameStore Str Expr Call NameLoad NameLoad Raise Call NameLoad NameLoad If BoolOpAnd NameLoad NameLoad body Expr Call NameLoad Str orelse If BoolOpAnd NameLoad BoolOpOr NameLoad NameLoad body If UnaryOpNot Call NameLoad NameLoad body Expr Call NameLoad Str Expr Call NameLoad Num Assign NameStore Call NameLoad NameLoad Call NameLoad Call NameLoad Call NameLoad NameLoad Str Expr Call NameLoad Call AttributeLoad Str attr Call AttributeLoad Str attr ListComp SubscriptLoad Call NameLoad NameLoad Index Num comprehension NameStore NameLoad Expr Call NameLoad Call AttributeLoad Str attr NameLoad Expr Call NameLoad Call AttributeLoad Str attr NameLoad BinOpSub Num NameLoad If NameLoad body Expr Call NameLoad Call AttributeLoad Str attr IfExp NameLoad NameLoad NameLoad If NameLoad body Expr Call NameLoad Str If NameLoad body Expr Call NameLoad Call AttributeLoad Str attr NameLoad NameLoad For NameStore NameLoad body Expr Call NameLoad Call AttributeLoad Str attr SubscriptLoad Call NameLoad Call NameLoad NameLoad Index Num Expr Call NameLoad NameLoad Call NameLoad Call NameLoad NameLoad NameLoad NameLoad NameLoad keyword NameLoad keyword NameLoad keyword NameLoad keyword NameLoad keyword IfExp NameLoad NameLoad NameLoad Expr Call NameLoad Str decorator_list If CompareEq NameLoad Str body Expr Call NameLoad
N/A N/A 
:author: Matt Mulholland
:date: April 15, 2015

Script used to create training/test sets in a MongoDB database from
review data extracted from flat files.
 N/A logging os makedirs os.path join exists isfile dirname realpath basename splitext argparse ArgumentParser ArgumentDefaultsHelpFormatter src log_dir data_dir formatter reports_dir default_reports_dir N/A logging_info N/A logging INFO N/A logger N/A N/A logging getLogger util.make_train_test_sets N/A N/A N/A logger setLevel logging_info N/A sh N/A N/A logging StreamHandler N/A N/A N/A sh setLevel logging_info N/A N/A N/A logger addHandler sh N/A loginfo N/A logger info N/A logerr N/A logger error N/A logwarn N/A logger warning main N/A N/A N/A N/A N/A parser N/A ArgumentParser usage python make_train_test_sets.py --game_files GAME_FILE1,GAME_FILE2,...[ OPTIONS] description Build train/test sets for each game. Take up to 21k reviews and split it 80/20 training/test, respectively, by default. Both the maximum size and the percentage split can be altered via command-line flags. All selected reviews will be put into the "reviews_project" database's "reviews" collection (which is being  hosted on lemur.montclair.edu on port 27017). formatter_class ArgumentDefaultsHelpFormatter N/A _add_arg N/A parser add_argument N/A N/A _add_arg --game_files help Comma-separated list of file-names or "all" for all of the files (the game files should reside in the "data" directory). type str required True N/A N/A _add_arg --max_size -m help Maximum number of reviews to get for training/testing (if possible). type int default 4000 N/A N/A _add_arg --percent_train -% help Percent of selected reviews for which to use for the training set, the rest going to the test set. type float default 80.0 N/A N/A _add_arg --convert_to_bins -bins help Number of sub-divisions of the hours-played values, e.g. if 10 and the hours values range from 0 up to 1000, then hours values 0-99 will become 1, 100-199 will become 2, etc. (will probably be necessay to train a model that actually is predictive to an acceptable degree); note that both hours values will be retained, the original under the name "hours" and the converted value under the name "hours_bin". type int required False N/A N/A _add_arg --bin_factor help If the --convert_to_bins/-bins argument is specified, increase the sizes of the bins by the given factor so that bins in which there will be lots of instances will be smaller in terms of range than bins that are more sparsely-populated. type float required False N/A N/A _add_arg --make_reports -describe help Generate reports and histograms describing the data filtering procedure. action store_true default False N/A N/A _add_arg --just_describe help Generate reports and histograms describing the data filtering procedure, but then do NOT insert the reviews into the MongoDB database. action store_true default False N/A N/A _add_arg --reports_dir help If -describe/--make_reports is used, put generated reports in the given directory. type str required False N/A N/A _add_arg -dbhost --mongodb_host help Host that the MongoDB server is running on. type str default localhost N/A N/A _add_arg --mongodb_port -dbport help Port that the MongoDB server is running on. type int default 27017 N/A N/A _add_arg --log_file_path -log help Path for log file. type str default N/A join log_dir replog_make_train_test_sets.txt N/A args N/A N/A parser parse_args os listdir pymongo MongoClient pymongo.errors ConnectionFailure src get_game_files src.mongodb connect_to_db insert_train_test_reviews N/A game_files N/A args game_files N/A max_size N/A args max_size N/A percent_train N/A args percent_train N/A convert_to_bins N/A args convert_to_bins N/A bin_factor N/A args bin_factor N/A make_reports N/A args make_reports N/A just_describe N/A args just_describe N/A reports_dir N/A args reports_dir N/A mongodb_host N/A args mongodb_host N/A mongodb_port N/A args mongodb_port N/A log_file_path N/A realpath N/A args log_file_path N/A log_file_dir N/A dirname log_file_path N/A N/A N/A exists log_file_dir N/A N/A N/A makedirs log_file_dir exist_ok True N/A fh N/A N/A logging FileHandler log_file_path N/A N/A N/A fh setLevel logging_info N/A N/A N/A fh setFormatter formatter N/A N/A N/A logger addHandler fh N/A N/A convert_to_bins N/A convert_to_bins 2 N/A N/A msg The value passed in via --convert_to_bins/-bins must be greater than one since there must be multiple bins to divide the hours played values. N/A N/A logerr msg N/A N/A ValueError msg N/A N/A convert_to_bins N/A N/A bins convert_to_bins N/A N/A bins 0 N/A N/A bin_factor N/A convert_to_bins N/A N/A msg The --bin_factor argument was specified despite the fact that the --convert_to_bins/-bins argument was not used. N/A N/A logerr msg N/A N/A ValueError msg N/A N/A loginfo N/A N/A Connecting to MongoDB database on mongodb://{0}:{1}... format mongodb_host mongodb_port N/A N/A N/A reviewdb N/A connect_to_db host mongodb_host port mongodb_port N/A N/A N/A ConnectionFailure N/A e N/A N/A N/A logerr Unable to connect to MongoDB reviews collection. N/A N/A logerr e N/A e N/A N/A N/A reviewdb write_concern N/A w 0 N/A reports_dir N/A N/A reports_dir N/A realpath reports_dir N/A N/A isfile reports_dir N/A N/A msg The file path passed in via the --reports_dir leads to a file, not a directory. N/A N/A logerr msg N/A N/A ValueError msg N/A N/A N/A exists reports_dir N/A N/A N/A makedirs reports_dir exist_ok True N/A N/A max_size 50 N/A N/A msg You can't be serious, right? You passed in a value of 50 for the MAXIMUM size of the combination of training/test sets? N/A N/A logerr msg N/A N/A ValueError msg N/A N/A percent_train 1.0 N/A N/A msg You can't be serious, right? You passed in a value of 1.0% for the percentage of the selected reviews that will be devoted to the training set? That is not going to be enough training samples. N/A N/A logerr msg N/A N/A ValueError msg N/A N/A make_reports just_describe N/A N/A N/A logwarn If the --just_describe and -describe/--make_reports option flags are used, --just_describe wins out, i.e., reports will be generated, but no reviews will be inserted into the database. N/A N/A N/A reports_dir N/A make_reports just_describe N/A N/A N/A N/A exists reports_dir N/A N/A N/A logerr The given --reports_dir path was invalid. Exiting. N/A N/A exit 1 N/A game_files N/A get_game_files game_files N/A join N/A dirname N/A dirname __file__ data N/A N/A loginfo N/A N/A Adding training/test partitions to Mongo DB for the following games: {0} format N/A N/A ,  join N/A N/A N/A splitext game N/A 0 N/A game game_files N/A N/A loginfo N/A N/A Maximum size for the combined training/test sets: {0} format max_size N/A N/A loginfo N/A N/A Percentage split between training and test sets: {0:.2f}/{1:.2f} format percent_train N/A 100.0 percent_train N/A make_reports N/A N/A N/A loginfo N/A N/A Generating reports in {0}. format N/A reports_dir reports_dir default_reports_dir N/A just_describe N/A N/A N/A loginfo Exiting after generating reports. N/A bins N/A N/A N/A loginfo N/A N/A Converting hours played values to {0} bins with a bin factor of {1}. format bins bin_factor N/A game_file game_files N/A N/A N/A loginfo N/A N/A Getting/inserting reviews for {}... format N/A N/A splitext N/A basename game_file N/A 0 N/A N/A insert_train_test_reviews reviewdb N/A realpath N/A join data_dir game_file max_size percent_train bins bins bin_factor bin_factor describe make_reports just_describe just_describe reports_dir N/A reports_dir reports_dir default_reports_dir N/A N/A loginfo Complete. N/A N/A N/A __name__ __main__ N/A N/A N/A main