Module Import alias Import alias Import alias Import alias Import alias Import alias ImportFrom alias ImportFrom alias ImportFrom alias Import alias Import alias Import alias If Call AttributeLoad Call AttributeLoad NameLoad attr attr AttributeLoad NameLoad attr body Expr Call AttributeLoad NameLoad attr Str keyword AttributeLoad NameLoad attr Expr Call AttributeLoad NameLoad attr Str Str Expr Call AttributeLoad NameLoad attr Str Str Expr Call AttributeLoad NameLoad attr Str Str Expr Call AttributeLoad NameLoad attr Str Str FunctionDef arguments args defaults body Expr Str Assign NameStore Call AttributeLoad NameLoad attr Str Str For NameStore Call AttributeLoad NameLoad attr Str body For NameStore Call NameLoad BinOpAdd Str NameLoad body Assign NameStore Call AttributeLoad NameLoad attr Str Assign NameStore Call AttributeLoad NameLoad attr Str TryExcept body Assign SubscriptStore NameLoad Index SubscriptLoad NameLoad Index Num SubscriptLoad NameLoad Index Num handlers ExceptHandler type NameLoad body If CompareEq NameLoad Str body Continue orelse Raise decorator_list ClassDef bases body FunctionDef arguments args NameParam NameParam defaults NameLoad body ImportFrom alias TryExcept body Assign NameStore Call NameLoad Str Expr Call AttributeLoad NameLoad attr Str handlers ExceptHandler type NameLoad body TryExcept body Assign NameStore Call NameLoad Str Expr Call AttributeLoad NameLoad attr Str handlers ExceptHandler type NameLoad body Assign NameStore Call NameLoad Str Expr Call AttributeLoad NameLoad attr Str Expr Call AttributeLoad NameLoad attr Assign NameStore AttributeLoad NameLoad attr If CompareEq NameLoad NameLoad body Assign AttributeStore NameLoad attr Call AttributeLoad NameLoad attr Str Str orelse Assign AttributeStore NameLoad attr NameLoad Assign AttributeStore NameLoad attr Call AttributeLoad NameLoad attr Str Str Assign AttributeStore NameLoad attr Call AttributeLoad NameLoad attr Str Str Assign AttributeStore NameLoad attr NameLoad decorator_list FunctionDef arguments args NameParam NameParam defaults NameLoad body Assign AttributeStore NameLoad attr Call AttributeLoad NameLoad attr keyword Str keyword Str keyword Str keyword Str keyword Num Assign NameStore Call AttributeLoad AttributeLoad NameLoad attr attr Expr Call AttributeLoad NameLoad attr BinOpMod Str AttributeLoad NameLoad attr Expr Call AttributeLoad NameLoad attr Str Expr Call AttributeLoad NameLoad attr Str If NameLoad body TryExcept body Expr Call AttributeLoad NameLoad attr Str handlers ExceptHandler body Expr Call AttributeLoad NameLoad attr Str Expr Call AttributeLoad NameLoad attr keyword NameLoad Expr Call AttributeLoad NameLoad attr BinOpMod Str AttributeLoad NameLoad attr Expr Call AttributeLoad NameLoad attr BinOpMod Str AttributeLoad NameLoad attr decorator_list FunctionDef arguments args NameParam NameParam defaults body Expr Str Expr Call AttributeLoad NameLoad attr BinOpAdd Str NameLoad TryExcept body Assign NameStore Call AttributeLoad AttributeLoad NameLoad attr attr Expr Call AttributeLoad NameLoad attr NameLoad handlers ExceptHandler body TryExcept body Expr Call AttributeLoad NameLoad attr Assign NameStore Call AttributeLoad AttributeLoad NameLoad attr attr Expr Call AttributeLoad NameLoad attr NameLoad handlers ExceptHandler body Expr Call AttributeLoad NameLoad attr BinOpAdd BinOpAdd Str NameLoad Str Raise Return NameLoad decorator_list decorator_list ClassDef bases body Expr Str FunctionDef arguments args NameParam NameParam NameParam defaults NameLoad Str body Expr Str Import alias Assign AttributeStore NameLoad attr Call AttributeLoad AttributeLoad NameLoad attr attr Str Expr Call AttributeLoad AttributeLoad NameLoad attr attr Assign NameStore AttributeLoad AttributeLoad NameLoad attr attr If CompareEq NameLoad NameLoad body Assign AttributeStore NameLoad attr Call AttributeLoad NameLoad attr Str Str orelse Assign AttributeStore NameLoad attr NameLoad Assign AttributeStore NameLoad attr NameLoad Assign AttributeStore NameLoad attr Call NameLoad keyword AttributeLoad NameLoad attr If CompareIsNot NameLoad NameLoad body Expr Call AttributeLoad NameLoad attr keyword NameLoad decorator_list FunctionDef arguments args NameParam defaults body Expr Str Import alias Assign NameStore Call AttributeLoad NameLoad attr keyword NameLoad Expr Call AttributeLoad NameLoad attr ListLoad Str Str Str Str Assign NameStore Call AttributeLoad NameLoad attr Str Str Assign NameStore Call AttributeLoad NameLoad attr Str Str Expr Call AttributeLoad AttributeLoad NameLoad attr attr BinOpMod Str TupleLoad AttributeLoad NameLoad attr NameLoad NameLoad decorator_list FunctionDef arguments args NameParam NameParam NameParam NameParam defaults Str Str body Assign AttributeStore NameLoad attr Call NameLoad keyword NameLoad keyword NameLoad keyword NameLoad keyword AttributeLoad NameLoad attr decorator_list FunctionDef arguments args NameParam NameParam NameParam NameParam defaults body Expr Str Expr Call AttributeLoad NameLoad attr NameLoad keyword NameLoad keyword NameLoad Expr Call AttributeLoad AttributeLoad NameLoad attr attr Expr Call AttributeLoad NameLoad attr Expr Call AttributeLoad AttributeLoad NameLoad attr attr For NameStore AttributeLoad AttributeLoad NameLoad attr attr body Expr Call AttributeLoad NameLoad attr Expr Call AttributeLoad NameLoad attr decorator_list FunctionDef arguments args NameParam defaults body Assign NameStore AttributeLoad NameLoad attr Assign NameStore AttributeLoad NameLoad attr Assign NameStore AttributeLoad NameLoad attr Assign NameStore AttributeLoad NameLoad attr TryExcept body Expr Call AttributeLoad NameLoad attr BinOpAdd Str NameLoad handlers ExceptHandler body Expr Call AttributeLoad NameLoad attr BinOpMod Str NameLoad Expr Str Expr Call AttributeLoad NameLoad attr BinOpAdd BinOpAdd BinOpAdd BinOpAdd BinOpAdd BinOpAdd BinOpAdd Str NameLoad Str Str NameLoad Str NameLoad Str Expr Call AttributeLoad NameLoad attr Str TryExcept body Expr Call AttributeLoad NameLoad attr Str handlers ExceptHandler body Raise decorator_list FunctionDef arguments args NameParam defaults body Assign NameStore AttributeLoad NameLoad attr Expr Call AttributeLoad NameLoad attr Str Expr Call AttributeLoad NameLoad attr Str Expr Call AttributeLoad NameLoad attr Str Expr Call AttributeLoad NameLoad attr Str Expr Call AttributeLoad NameLoad attr Str Expr Call AttributeLoad NameLoad attr Str Expr Call AttributeLoad NameLoad attr Str Expr Call AttributeLoad NameLoad attr Str Expr Call AttributeLoad NameLoad attr Str decorator_list FunctionDef arguments args NameParam defaults body Expr Str Expr Call AttributeLoad AttributeLoad NameLoad attr attr decorator_list FunctionDef arguments args NameParam defaults body Assign NameStore AttributeLoad NameLoad attr Expr Call AttributeLoad NameLoad attr Str Expr Call AttributeLoad NameLoad attr Str Expr Call AttributeLoad NameLoad attr Str Expr Call AttributeLoad NameLoad attr Str Expr Call AttributeLoad NameLoad attr Str For NameStore Call AttributeLoad NameLoad attr Str body If UnaryOpNot Call AttributeLoad NameLoad attr Str body Continue TryExcept body Expr Call AttributeLoad NameLoad attr BinOpAdd BinOpAdd Str NameLoad Str handlers ExceptHandler body Raise Expr Call AttributeLoad NameLoad attr Str Expr Call AttributeLoad NameLoad attr Str decorator_list FunctionDef arguments args NameParam defaults body Assign NameStore AttributeLoad NameLoad attr Expr Call AttributeLoad NameLoad attr Str Expr Call AttributeLoad NameLoad attr Str Expr Call AttributeLoad NameLoad attr Str Expr Call AttributeLoad NameLoad attr Str Expr Call AttributeLoad NameLoad attr Str For NameStore Call AttributeLoad NameLoad attr Str body TryExcept body Expr Call AttributeLoad NameLoad attr BinOpAdd BinOpAdd Str NameLoad Str handlers ExceptHandler body Raise Expr Call AttributeLoad NameLoad attr Str Expr Call AttributeLoad NameLoad attr Str decorator_list FunctionDef arguments args NameParam defaults body Expr Str Assign NameStore AttributeLoad NameLoad attr Expr Call AttributeLoad NameLoad attr Str Assign NameStore Call AttributeLoad NameLoad attr Str Assign NameStore Call AttributeLoad NameLoad attr Str Expr Call AttributeLoad NameLoad attr Expr Call AttributeLoad NameLoad attr Expr Call AttributeLoad AttributeLoad NameLoad attr attr decorator_list FunctionDef arguments args NameParam NameParam defaults NameLoad body Expr Str Assign NameStore Call AttributeLoad Call AttributeLoad AttributeLoad NameLoad attr attr Str attr For NameStore NameLoad body Expr Str Assign NameStore SubscriptLoad NameLoad Index Num TryExcept body Assign NameStore Call AttributeLoad AttributeLoad NameLoad attr attr BinOpMod Str NameLoad Assign NameStore SubscriptLoad SubscriptLoad Call AttributeLoad NameLoad attr Index Num Index Num Expr Call AttributeLoad NameLoad attr BinOpMod Str NameLoad handlers ExceptHandler body Assign NameStore Num If BoolOpOr CompareEq NameLoad Num NameLoad body For NameStore Call NameLoad SubscriptLoad NameLoad Index Num body Expr Call AttributeLoad AttributeLoad NameLoad attr attr NameLoad decorator_list FunctionDef arguments args NameParam defaults body Assign NameStore BinOpAdd ListLoad Str Str ListComp Call AttributeLoad NameLoad attr comprehension NameStore AttributeLoad AttributeLoad NameLoad attr attr BoolOpAnd AttributeLoad NameLoad attr CompareIsNot Call AttributeLoad NameLoad attr NameLoad Assign NameStore BinOpAdd BinOpAdd Str Call AttributeLoad Str attr NameLoad Str Assign NameStore BinOpAdd ListLoad Str Str ListComp AttributeLoad NameLoad attr comprehension NameStore AttributeLoad AttributeLoad NameLoad attr attr BoolOpAnd AttributeLoad NameLoad attr CompareIsNot Call AttributeLoad NameLoad attr NameLoad AugAssignAdd NameStore BinOpAdd BinOpAdd BinOpAdd BinOpAdd Str Call AttributeLoad Str attr NameLoad Str Call AttributeLoad Str attr ListComp BinOpMod Str AttributeLoad NameLoad attr comprehension NameStore AttributeLoad AttributeLoad NameLoad attr attr BoolOpAnd AttributeLoad NameLoad attr CompareIsNot Call AttributeLoad NameLoad attr NameLoad CompareEq AttributeLoad NameLoad attr Str Str AugAssignAdd NameStore Str AugAssignAdd NameStore Str Expr Call AttributeLoad AttributeLoad NameLoad attr attr Str Expr Call AttributeLoad AttributeLoad NameLoad attr attr BinOpAdd BinOpAdd Str NameLoad Str decorator_list FunctionDef arguments args NameParam defaults body Assign NameStore Str AugAssignAdd NameStore Str AugAssignAdd NameStore Str AugAssignAdd NameStore Str AugAssignAdd NameStore Str Assign NameStore BinOpAdd BinOpAdd Str Call AttributeLoad NameLoad attr NameLoad Str Expr Call AttributeLoad NameLoad attr Str Expr Call AttributeLoad AttributeLoad NameLoad attr attr NameLoad decorator_list FunctionDef arguments args NameParam defaults body Assign NameStore AttributeLoad AttributeLoad NameLoad attr attr Assign NameStore AttributeLoad NameLoad attr Assign NameStore Call NameLoad Assign SubscriptStore NameLoad Index Str Dict Str Str Str Str AttributeLoad NameLoad attr Str AttributeLoad NameLoad attr AttributeLoad NameLoad attr Assign NameStore ListLoad Dict Str Str Str Str Str Str For NameStore NameLoad body Assign NameStore Call AttributeLoad NameLoad attr If NameLoad body Expr Call AttributeLoad NameLoad attr NameLoad TryExcept body Assign NameStore SubscriptLoad ListComp AttributeLoad NameLoad attr comprehension NameStore NameLoad CompareEq AttributeLoad NameLoad attr Str Index Num Assign SubscriptStore NameLoad Index Str ListLoad Dict Str Str Str Str Str ListLoad Dict Str ListLoad Str NameLoad Str Str Num handlers ExceptHandler body Expr Call AttributeLoad NameLoad attr Str Assign SubscriptStore NameLoad Index Str NameLoad Assign NameStore Call NameLoad BinOpMod Str NameLoad Str Expr Call AttributeLoad NameLoad attr Call AttributeLoad NameLoad attr NameLoad decorator_list FunctionDef arguments args NameParam defaults body Assign NameStore AttributeLoad NameLoad attr TryExcept body Expr Call AttributeLoad NameLoad attr Str Expr Call AttributeLoad NameLoad attr Str handlers ExceptHandler body Pass Assign NameStore Dict Str Str Str Str AttributeLoad NameLoad attr Str Assign NameStore Call AttributeLoad NameLoad attr NameLoad Expr Call AttributeLoad NameLoad attr NameLoad Expr Call AttributeLoad NameLoad attr BinOpMod Str NameLoad decorator_list FunctionDef arguments args NameParam defaults body Expr Str Expr Call AttributeLoad NameLoad attr Str ImportFrom alias Assign NameStore Call NameLoad Assign NameStore Call AttributeLoad NameLoad attr Str Assign NameStore Call AttributeLoad NameLoad attr For NameStore NameLoad body Assign NameStore Call AttributeLoad Str attr NameLoad Assign NameStore Call AttributeLoad NameLoad attr Str Str If Call AttributeLoad NameLoad attr Str NameLoad body Assign NameStore BinOpAdd BinOpAdd BinOpAdd BinOpAdd Str Call AttributeLoad NameLoad attr Call AttributeLoad Str attr NameLoad Str Call AttributeLoad Str attr NameLoad Str Assign NameStore Call AttributeLoad NameLoad attr NameLoad decorator_list decorator_list
N/A N/A subprocess N/A MySQLdb N/A re N/A sys N/A json N/A os variableSet dataField variableSet variableSet variableSet splitMySQLcode N/A logging N/A warnings N/A anydbm N/A N/A N/A N/A N/A logging getLogger isEnabledFor N/A logging DEBUG N/A N/A N/A N/A warnings filterwarnings error category N/A MySQLdb Warning N/A N/A N/A warnings filterwarnings ignore Table .* already exists N/A N/A N/A warnings filterwarnings ignore Can't create database.*; database exists N/A N/A N/A warnings filterwarnings ignore ^Unknown table .* N/A N/A N/A warnings filterwarnings ignore Table 'mysql.table_stats' doesn't exist text_id_dbm N/A N/A N/A N/A N/A 
    This quickly creates a key-value store for textids: storing on disk
    dramatically reduces memory consumption for bookworms of over 
    1 million documents.
     N/A dbm N/A N/A anydbm open .bookworm/texts/textids.dbm c N/A file N/A N/A os listdir .bookworm/texts/textids N/A N/A line N/A open N/A .bookworm/texts/textids/ file N/A N/A line N/A N/A line rstrip N/A splat N/A N/A line split N/A N/A N/A N/A dbm N/A N/A splat N/A 1 N/A splat N/A 0 N/A N/A N/A IndexError N/A N/A N/A line N/A N/A N/A N/A N/A DB N/A N/A __init__ N/A N/A self dbname N/A None N/A bookwormDB.configuration Configfile N/A N/A N/A configuration N/A Configfile local N/A N/A N/A logging debug Connecting from the local config file N/A N/A N/A IOError N/A N/A N/A N/A configuration N/A Configfile global N/A N/A N/A logging debug No bookworm.cnf in local file: connecting from global defaults N/A N/A N/A IOError N/A N/A configuration N/A Configfile admin N/A N/A N/A logging debug No bookworm.cnf in local file: connecting from admin defaults N/A N/A N/A configuration read_config_files N/A config N/A configuration config N/A N/A dbname None N/A N/A N/A self dbname N/A N/A config get client database N/A N/A N/A self dbname dbname N/A N/A self username N/A N/A config get client user N/A N/A self password N/A N/A config get client password N/A N/A self conn None N/A connect N/A N/A self setengine N/A True N/A N/A N/A self conn N/A N/A MySQLdb connect read_default_file ~/.my.cnf use_unicode True charset utf8 db local_infile 1 N/A cursor N/A N/A N/A self conn cursor N/A N/A N/A cursor execute N/A CREATE DATABASE IF NOT EXISTS %s N/A self dbname N/A N/A N/A cursor execute SET NAMES 'utf8' N/A N/A N/A cursor execute SET CHARACTER SET 'utf8' N/A setengine N/A N/A N/A N/A N/A N/A cursor execute SET default_storage_engine=MYISAM N/A N/A N/A N/A N/A N/A logging error Forcing default engine failed. On some versions of Mysql,                you may need to add "default-storage-engine=MYISAM" manually                to the [mysqld] user in /etc/my.cnf. Trying again to connect... N/A N/A N/A self connect setengine False N/A N/A N/A logging debug N/A Connecting to %s N/A self dbname N/A N/A N/A cursor execute N/A USE %s N/A self dbname N/A query N/A N/A self sql N/A N/A N/A 
        Billy defined a separate query method here so that the common case of a connection being
        timed out doesn't cause the whole shebang to fall apart: instead, it just reboots
        the connection and starts up nicely again.
         N/A N/A N/A logging debug N/A  -- Preparing to execute SQL code --  sql N/A N/A N/A cursor N/A N/A N/A self conn cursor N/A N/A N/A cursor execute sql N/A N/A N/A N/A N/A N/A N/A N/A self connect N/A cursor N/A N/A N/A self conn cursor N/A N/A N/A cursor execute sql N/A N/A N/A N/A N/A N/A logging error N/A N/A Query failed: 
 sql N/A N/A cursor N/A N/A BookwormSQLDatabase N/A N/A N/A 
    This class gives interactions methods to a MySQL database storing Bookworm
    data. Although the primary methods are about loading data already created
    into the SQL database, it has a few other operations
    that write out text files needed by the API and the web front end:
    I take it as logical to do those here, since that how
    it fits chronologically in the bookworm-creation sequence.
     __init__ N/A N/A self dbname variableFile N/A None .bookworm/metadata/jsoncatalog_derived.txt N/A N/A 
        You can initialize it with a database name;
        otherwise it defaults to finding a
        Bookworm configuration file.

        It also may be initialized with a set of metadata.
        This is a little wonky, and may
        be deprecated in favor of a cleaner interface.
         N/A bookwormDB.configuration N/A N/A self config_manager N/A N/A N/A bookwormDB configuration Configfile local N/A N/A N/A N/A self config_manager read_config_files N/A config N/A N/A self config_manager config N/A N/A dbname None N/A N/A N/A self dbname N/A N/A config get client database N/A N/A N/A self dbname dbname N/A N/A self conn None N/A N/A self db N/A DB dbname N/A self dbname N/A N/A variableFile None N/A N/A N/A N/A self setVariables originFile variableFile N/A grantPrivileges N/A N/A self N/A N/A N/A 
        Grants select-only privileges to a non-admin mysql user for the API to
        query with (safer).

        The Username for these privileges is pulled from the bookworm.cnf file.
         N/A ConfigParser N/A config N/A N/A ConfigParser ConfigParser allow_no_value True N/A N/A N/A config read N/A ~/.my.cnf /etc/my.cnf /etc/mysql/my.cnf bookworm.cnf N/A username N/A N/A config get client user N/A password N/A N/A config get client password N/A N/A N/A N/A self db query N/A GRANT SELECT ON %s.* TO '%s'@'localhost' IDENTIFIED BY '%s' N/A N/A self dbname username password N/A setVariables N/A N/A self originFile anchorField jsonDefinition N/A bookid .bookworm/metadata/field_descriptions_derived.json N/A N/A N/A self variableSet N/A variableSet originFile originFile anchorField anchorField jsonDefinition jsonDefinition db N/A self db N/A importNewFile N/A N/A self originFile anchorField jsonDefinition N/A N/A N/A 
        Add additional metadata from a source collection of json-formatted rows.
        originFile is the filename of the new metadata, in the same input format
        as the original jsoncatalog.txt
        anchorField is the field in the existing dataset it should be anchored onto;
        jsonDefinition is a filename pointing to a file
        of the format of field_descriptions.json describing the new data to ingest.
        If it is of type None, then one will be guessed at.
         N/A N/A N/A self setVariables originFile anchorField anchorField jsonDefinition jsonDefinition N/A N/A N/A N/A self variableSet writeMetadata N/A N/A N/A self load_book_list N/A N/A N/A N/A self variableSet updateMasterVariableTable N/A variable N/A N/A self variableSet variables N/A N/A N/A N/A variable clear_associated_memory_tables N/A N/A N/A self reloadMemoryTables N/A create_database N/A N/A self N/A N/A N/A dbname N/A self dbname N/A dbuser N/A self dbuser N/A dbpassword N/A self dbpassword N/A db N/A self db N/A N/A N/A N/A N/A db query N/A CREATE DATABASE  dbname N/A N/A N/A N/A N/A N/A logging info N/A Database %s already exists: that might be intentional, so not dying dbname N/A Setting up permissions for web user... N/A N/A N/A db query N/A N/A N/A N/A N/A N/A N/A GRANT SELECT ON  dbname .*  TO ' dbuser '@'localhost' IDENTIFIED BY ' dbpassword ' N/A N/A N/A db query FLUSH PRIVILEGES N/A N/A N/A N/A N/A db query CREATE TABLE IF NOT EXISTS bookworm_information (entry VARCHAR(255), PRIMARY KEY (entry), value VARCHAR(50000)) N/A N/A N/A N/A N/A load_word_list N/A N/A self N/A N/A N/A db N/A self db N/A N/A N/A logging info Making a SQL table to hold the words N/A N/A N/A db query DROP TABLE IF EXISTS words N/A N/A N/A db query CREATE TABLE IF NOT EXISTS words (
        wordid MEDIUMINT,
        word VARCHAR(255), INDEX (word),
        count BIGINT UNSIGNED,
        casesens VARBINARY(255),
        stem VARCHAR(255)
        ); N/A N/A N/A db query ALTER TABLE words DISABLE KEYS N/A N/A N/A logging info loading data using LOAD DATA LOCAL INFILE N/A N/A N/A db query LOAD DATA LOCAL INFILE '.bookworm/texts/wordlist/wordlist.txt'
                   INTO TABLE words
                   CHARACTER SET binary
                   (wordid,word,count)  N/A N/A N/A logging info creating indexes on words table N/A N/A N/A db query ALTER TABLE words ENABLE KEYS N/A N/A N/A db query UPDATE words SET casesens=word N/A load_book_list N/A N/A self N/A N/A N/A 
        Loads in the tables that have already been created by a previous
        call to `Bookworm.variableSet.writeMetadata()`
         N/A N/A N/A N/A self variableSet loadMetadata N/A create_unigram_book_counts N/A N/A self N/A N/A N/A db N/A self db N/A N/A N/A db query DROP TABLE IF EXISTS master_bookcounts N/A N/A N/A logging info Making a SQL table to hold the unigram counts N/A N/A N/A db query CREATE TABLE master_bookcounts (
        bookid MEDIUMINT UNSIGNED NOT NULL, INDEX(bookid,wordid,count),
        wordid MEDIUMINT UNSIGNED NOT NULL, INDEX(wordid,bookid,count),
        count MEDIUMINT UNSIGNED NOT NULL); N/A N/A N/A db query ALTER TABLE master_bookcounts DISABLE KEYS N/A N/A N/A logging info loading data using LOAD DATA LOCAL INFILE N/A filename N/A N/A os listdir .bookworm/texts/encoded/unigrams N/A N/A N/A N/A N/A filename endswith .txt N/A N/A N/A N/A N/A N/A N/A db query N/A N/A LOAD DATA LOCAL INFILE '.bookworm/texts/encoded/unigrams/ filename ' INTO TABLE master_bookcounts CHARACTER SET utf8 (bookid,wordid,count); N/A N/A N/A N/A N/A N/A N/A logging info Creating Unigram Indexes N/A N/A N/A db query ALTER TABLE master_bookcounts ENABLE KEYS N/A create_bigram_book_counts N/A N/A self N/A N/A N/A db N/A self db N/A N/A N/A logging info Making a SQL table to hold the bigram counts N/A N/A N/A db query DROP TABLE IF EXISTS master_bigrams N/A N/A N/A db query CREATE TABLE master_bigrams (
        bookid MEDIUMINT UNSIGNED NOT NULL,
        word1 MEDIUMINT UNSIGNED NOT NULL, INDEX (word1,word2,bookid,count),
        word2 MEDIUMINT UNSIGNED NOT NULL,
        count MEDIUMINT UNSIGNED NOT NULL); N/A N/A N/A db query ALTER TABLE master_bigrams DISABLE KEYS N/A N/A N/A logging info loading data using LOAD DATA LOCAL INFILE N/A filename N/A N/A os listdir .bookworm/texts/encoded/bigrams N/A N/A N/A N/A N/A N/A db query N/A N/A LOAD DATA LOCAL INFILE '.bookworm/texts/encoded/bigrams/ filename ' INTO TABLE master_bigrams CHARACTER SET utf8 (bookid,word1,word2,count); N/A N/A N/A N/A N/A N/A N/A logging info Creating bigram indexes N/A N/A N/A db query ALTER TABLE master_bigrams ENABLE KEYS N/A loadVariableDescriptionsIntoDatabase N/A N/A self N/A N/A N/A 
        This adds a description of files to the master variable table:
        also, crucially, it puts code specifying their fast creation there,
        where it will be executed on startup for all eternity.
         N/A db N/A self db N/A N/A N/A db query DROP TABLE IF EXISTS masterVariableTable N/A m N/A N/A db query 
            CREATE TABLE IF NOT EXISTS masterVariableTable
              (dbname VARCHAR(255), PRIMARY KEY (dbname),
              name VARCHAR(255),
              type VARCHAR(255),
              tablename VARCHAR(255),
              anchor VARCHAR(255),
              alias VARCHAR(255),
              status VARCHAR(255),
              description VARCHAR(5000)
              ) ENGINE=MYISAM;
               N/A tableTable N/A N/A db query 
            CREATE TABLE IF NOT EXISTS masterTableTable
              (tablename VARCHAR(255), PRIMARY KEY (tablename),
              dependsOn VARCHAR(255),
              memoryCode VARCHAR(20000)) ENGINE=MYISAM;
               N/A N/A N/A self addFilesToMasterVariableTable N/A N/A N/A self addWordsToMasterVariableTable N/A N/A N/A N/A self variableSet updateMasterVariableTable N/A reloadMemoryTables N/A N/A self force N/A False N/A N/A 
        Checks to see if memory tables need to be repopulated (by seeing if they are empty)
        and then does so if necessary.
         N/A existingCreateCodes N/A N/A N/A N/A N/A self db query SELECT tablename,memoryCode FROM masterTableTable fetchall N/A row existingCreateCodes N/A N/A 
            For each table, it checks to see if the table is currently populated; if not,
            it runs the stored code to repopulate the table. (It checks length because
            memory tables are emptied on a restart).
             N/A tablename N/A row N/A 0 N/A N/A N/A cursor N/A N/A N/A self db query N/A SELECT count(*) FROM %s tablename N/A currentLength N/A N/A N/A N/A cursor fetchall N/A 0 N/A 0 N/A N/A N/A logging debug N/A Current Length is %d currentLength N/A N/A N/A N/A currentLength 0 N/A N/A N/A currentLength 0 force N/A N/A query N/A splitMySQLcode N/A row N/A 1 N/A N/A N/A N/A N/A self db query query N/A addFilesToMasterVariableTable N/A N/A self N/A N/A N/A fastFieldsCreateList N/A N/A bookid MEDIUMINT, PRIMARY KEY (bookid) nwords MEDIUMINT N/A N/A N/A variable fastSQL N/A variable N/A N/A self variableSet variables N/A N/A variable unique N/A N/A N/A variable fastSQL None N/A fileCommand N/A N/A DROP TABLE IF EXISTS tmp;
        CREATE TABLE tmp
        ( N/A N/A ,
 join fastFieldsCreateList 
        ) ENGINE=MEMORY; N/A fastFields N/A N/A bookid nwords N/A N/A variable fastField N/A variable N/A N/A self variableSet variables N/A N/A variable unique N/A N/A N/A variable fastSQL None N/A fileCommand N/A N/A N/A N/A INSERT INTO tmp SELECT  N/A N/A , join fastFields  FROM catalog  N/A N/A join N/A N/A  JOIN %(field)s__id USING (%(field)s )  N/A variable __dict__ N/A variable N/A N/A self variableSet variables N/A N/A variable unique N/A N/A N/A variable fastSQL None N/A N/A variable datatype categorical ; N/A fileCommand DROP TABLE IF EXISTS fastcat; N/A fileCommand RENAME TABLE tmp TO fastcat; N/A N/A N/A N/A self db query DELETE FROM masterTableTable WHERE masterTableTable.tablename="fastcat"; N/A N/A N/A N/A self db query N/A N/A INSERT IGNORE INTO masterTableTable VALUES
                   ('fastcat','fastcat',' fileCommand ') N/A addWordsToMasterVariableTable N/A N/A self N/A N/A N/A wordCommand DROP TABLE IF EXISTS tmp; N/A wordCommand CREATE TABLE tmp (wordid MEDIUMINT, PRIMARY KEY (wordid), word VARCHAR(30), INDEX (word), casesens VARBINARY(30),UNIQUE INDEX(casesens), lowercase CHAR(30), INDEX (lowercase) ) ENGINE=MEMORY; N/A wordCommand INSERT IGNORE INTO tmp SELECT wordid as wordid,word,casesens,LOWER(word) FROM words WHERE CHAR_LENGTH(word) <= 30 AND wordid <= 1500000 ORDER BY wordid; N/A wordCommand DROP TABLE IF EXISTS wordsheap; N/A wordCommand RENAME TABLE tmp TO wordsheap; N/A query N/A N/A INSERT IGNORE INTO masterTableTable
                   VALUES ('wordsheap','wordsheap',' N/A N/A MySQLdb escape_string wordCommand ') N/A N/A N/A logging info Creating wordsheap N/A N/A N/A N/A self db query query N/A jsonify_data N/A N/A self N/A N/A N/A variables N/A N/A self variableSet variables N/A dbname N/A self dbname N/A output N/A dict N/A N/A output N/A settings N/A dbname itemName sourceName sourceURL N/A self dbname  text N/A self dbname N/A self dbname N/A ui_components N/A N/A type dbfield name text word Word(s) N/A variable variables N/A N/A newdict N/A N/A variable jsonDict N/A newdict N/A N/A N/A N/A ui_components append newdict N/A N/A N/A mytime N/A N/A N/A variable field N/A variable variables N/A N/A variable datatype time N/A 0 N/A N/A output N/A default_search N/A N/A search_limits time_measure words_collation counttype smoothingSpan N/A N/A word N/A test mytime Case_Sensitive Occurrences_per_Million_Words 0 N/A N/A N/A N/A N/A N/A logging warning WARNING: Not enough info for a default search (like, no time variable maybe?)--likely to be some big problems with your bookworm. N/A N/A output N/A ui_components ui_components N/A outfile N/A open N/A .bookworm/%s.json dbname w N/A N/A N/A outfile write N/A N/A json dumps output N/A create_API_settings N/A N/A self N/A N/A N/A db N/A self db N/A N/A N/A N/A N/A db query DROP TABLE IF EXISTS API_settings N/A N/A N/A db query CREATE TABLE API_settings (settings VARCHAR(8192)); N/A N/A N/A N/A N/A api_info N/A HOST database read_default_file 10.102.15.45 N/A self dbname /etc/mysql/my.cnf N/A addCode N/A N/A json dumps api_info N/A N/A N/A logging info addCode N/A N/A N/A db query N/A INSERT INTO API_settings VALUES ('%s'); addCode N/A update_Porter_stemming N/A N/A self N/A N/A N/A 
        Still not executed.
         N/A N/A N/A logging info Updating stems from Porter algorithm... nltk PorterStemmer N/A stemmer N/A PorterStemmer N/A cursor N/A N/A db query SELECT word FROM words N/A words N/A N/A cursor fetchall N/A local words N/A N/A word N/A N/A join local N/A word N/A N/A word replace 's N/A N/A N/A re match ^[A-Za-z]+$ word N/A N/A query N/A N/A N/A N/A UPDATE words SET stem=' N/A N/A stemmer stem N/A N/A join local ' WHERE word=' N/A N/A join local '; N/A z N/A N/A cursor execute query N/A N/A