Module Expr Str ImportFrom alias Import alias ImportFrom alias alias alias ImportFrom alias ImportFrom alias alias ImportFrom alias alias ClassDef bases AttributeLoad NameLoad attr body Expr Str FunctionDef arguments args NameParam NameParam defaults body Expr Call AttributeLoad Call NameLoad NameLoad NameLoad attr NameLoad Expr Call NameLoad decorator_list FunctionDef arguments args NameParam defaults body Expr Str TryExcept body Assign SubscriptStore AttributeLoad NameLoad attr Index Str Call NameLoad SubscriptLoad AttributeLoad NameLoad attr Index Str If CompareNotIn SubscriptLoad AttributeLoad NameLoad attr Index Str ListLoad Num Num Num body Raise NameLoad If CompareGt SubscriptLoad AttributeLoad NameLoad attr Index Str Num body Expr Call NameLoad BinOpAdd BinOpAdd AttributeLoad NameLoad attr AttributeLoad NameLoad attr Str Expr Call NameLoad BinOpAdd AttributeLoad NameLoad attr AttributeLoad NameLoad attr Import alias With Call NameLoad BinOpAdd SubscriptLoad AttributeLoad NameLoad attr Index Str Str Str NameStore body Assign AttributeStore NameLoad attr Call AttributeLoad NameLoad attr NameLoad Expr Call AttributeLoad NameLoad attr handlers ExceptHandler type NameLoad body Expr Call NameLoad BinOpAdd BinOpAdd BinOpAdd BinOpAdd AttributeLoad NameLoad attr AttributeLoad NameLoad attr Str AttributeLoad NameLoad attr AttributeLoad NameLoad attr keyword Str ExceptHandler type NameLoad body Expr Call NameLoad BinOpAdd BinOpAdd AttributeLoad NameLoad attr AttributeLoad NameLoad attr SubscriptLoad AttributeLoad NameLoad attr Index Str Str BinOpAdd BinOpAdd Str AttributeLoad NameLoad attr AttributeLoad NameLoad attr keyword Str decorator_list FunctionDef arguments args NameParam defaults body Assign NameStore Call NameLoad Call NameLoad SubscriptLoad AttributeLoad NameLoad attr Index Str BinOpAdd Call AttributeLoad SubscriptLoad AttributeLoad NameLoad attr Index Str attr Str Assign NameStore Call NameLoad Assign SubscriptStore NameLoad Index Str SubscriptLoad AttributeLoad NameLoad attr Index Str Assign SubscriptStore NameLoad Index Str Call NameLoad TryFinally body TryExcept body Assign NameStore Call NameLoad Assign NameStore Call NameLoad If CompareGt SubscriptLoad AttributeLoad NameLoad attr Index Str Num body Expr Call NameLoad Expr Call NameLoad BinOpAdd BinOpAdd AttributeLoad NameLoad attr AttributeLoad NameLoad attr Str BinOpAdd BinOpAdd SubscriptLoad SubscriptLoad AttributeLoad NameLoad attr Index Str Index Str AttributeLoad NameLoad attr AttributeLoad NameLoad attr keyword Str Assign NameStore Call NameLoad SubscriptLoad SubscriptLoad AttributeLoad NameLoad attr Index Str Index Str For NameStore SubscriptLoad SubscriptLoad AttributeLoad NameLoad attr Index Str Index Str body If CompareNotEq SubscriptLoad NameLoad Index Str Str body If CompareGt SubscriptLoad AttributeLoad NameLoad attr Index Str Num body Expr Call NameLoad Str SubscriptLoad NameLoad Index Str Str keyword Str keyword Str Assign SubscriptStore NameLoad Index SubscriptLoad NameLoad Index Str Call AttributeLoad NameLoad attr SubscriptLoad NameLoad Index Str SubscriptLoad NameLoad Index Str SubscriptLoad NameLoad Index Str If UnaryOpNot Call AttributeLoad SubscriptLoad AttributeLoad NameLoad attr Index Str attr Str body Assign NameStore ListLoad NameLoad orelse Assign NameStore Call AttributeLoad SubscriptLoad AttributeLoad NameLoad attr Index Str attr Str For NameStore NameLoad body Assign TupleStore NameStore NameStore Call AttributeLoad NameLoad attr keyword NameLoad keyword Call AttributeLoad NameLoad attr Str Str keyword Call AttributeLoad NameLoad attr Str ListLoad keyword Call AttributeLoad NameLoad attr Str Str keyword Call AttributeLoad NameLoad attr Str Str keyword Call AttributeLoad NameLoad attr Str Str keyword Call AttributeLoad NameLoad attr Str Str keyword Call AttributeLoad NameLoad attr Str Str keyword SubscriptLoad AttributeLoad NameLoad attr Index Str For NameStore NameLoad body If UnaryOpNot CompareIn NameLoad NameLoad body Assign SubscriptStore NameLoad Index NameLoad Call NameLoad NameLoad If UnaryOpNot Call AttributeLoad SubscriptLoad AttributeLoad NameLoad attr Index Str attr Str body Expr Call AttributeLoad SubscriptLoad NameLoad Index Str attr NameLoad orelse For NameStore SubscriptLoad SubscriptLoad AttributeLoad NameLoad attr Index Str Index Str body For TupleStore NameStore NameStore Call NameLoad NameLoad NameLoad NameLoad keyword SubscriptLoad AttributeLoad NameLoad attr Index Str body Expr Call AttributeLoad SubscriptLoad NameLoad Index Str attr NameLoad For NameStore NameLoad body If UnaryOpNot CompareIn NameLoad NameLoad body Assign SubscriptStore NameLoad Index NameLoad Call NameLoad NameLoad handlers ExceptHandler type NameLoad body Pass ExceptHandler type NameLoad name NameStore body Expr Call NameLoad NameLoad finalbody If CompareEq SubscriptLoad AttributeLoad NameLoad attr Index Str Str body Import alias With Call NameLoad Call AttributeLoad AttributeLoad NameLoad attr attr Call AttributeLoad NameLoad attr BinOpAdd SubscriptLoad AttributeLoad NameLoad attr Index Str Str Str NameStore body Expr Call AttributeLoad NameLoad attr NameLoad NameLoad orelse If CompareEq SubscriptLoad AttributeLoad NameLoad attr Index Str Str body Import alias With Call NameLoad Call AttributeLoad AttributeLoad NameLoad attr attr Call AttributeLoad NameLoad attr BinOpAdd SubscriptLoad AttributeLoad NameLoad attr Index Str Str Str NameStore body Assign NameStore Call NameLoad AttributeLoad NameLoad attr Assign NameStore Call NameLoad NameLoad keyword Lambda arguments args NameParam defaults SubscriptLoad NameLoad Index NameLoad Expr Call AttributeLoad NameLoad attr NameLoad Assign NameStore Call AttributeLoad NameLoad attr NameLoad keyword NameLoad Expr Call AttributeLoad NameLoad attr Expr Call AttributeLoad NameLoad attr SubscriptLoad NameLoad Index Str If CompareGt SubscriptLoad AttributeLoad NameLoad attr Index Str Num body Expr Call NameLoad Expr Call NameLoad BinOpAdd BinOpAdd AttributeLoad NameLoad attr AttributeLoad NameLoad attr SubscriptLoad AttributeLoad NameLoad attr Index Str Str SubscriptLoad AttributeLoad NameLoad attr Index Str BinOpAdd BinOpAdd Str AttributeLoad NameLoad attr AttributeLoad NameLoad attr keyword Str decorator_list decorator_list
N/A N/A 
scrapple.commands.run
~~~~~~~~~~~~~~~~~~~~~

 __future__ print_function N/A os colorama init Fore Back scrapple.commands command scrapple.selectors xpath css scrapple.utils.config traverse_next extract_fieldnames RunCommand N/A N/A command Command N/A N/A 
    Defines the execution of :ref:`run <command-run>`
     __init__ N/A N/A self args N/A N/A N/A N/A N/A N/A super RunCommand self __init__ args N/A N/A init N/A execute_command N/A N/A self N/A N/A N/A 
        The run command implements the web content extractor corresponding to the given         configuration file. 

        The execute_command() validates the input project name and opens the JSON         configuration file. The run() method handles the execution of the extractor run.

        The extractor implementation follows these primary steps :

        1. Selects the appropriate :ref:`selector class <implementation-selectors>` through         a dynamic dispatch, with the selector_type argument from the CLI input. 

        #. Iterate through the data section in level-0 of the configuration file.         On each data item, call the extract_content() method from the selector class to         extract the content according to the specified extractor rule. 

        #. If there are multiple levels of the extractor, i.e, if there is a 'next'         attribute in the configuration file, call the traverse_next()         :ref:`utility function <implementation-utils>` and parse through successive levels         of the configuration file.

        #. According to the --output_type argument, the result data is saved in a JSON         document or a CSV document. 

         N/A N/A N/A N/A N/A self args N/A --verbosity N/A int N/A N/A self args N/A --verbosity N/A N/A N/A N/A self args N/A --verbosity N/A 0 1 2 N/A N/A ValueError N/A N/A N/A N/A self args N/A --verbosity 0 N/A N/A N/A print N/A N/A N/A Back GREEN N/A Fore BLACK Scrapple Run N/A N/A print N/A N/A Back RESET N/A Fore RESET N/A json N/A N/A open N/A N/A N/A self args N/A <projectname> .json r f N/A N/A N/A self config N/A N/A json load f N/A N/A N/A self run N/A N/A N/A ValueError N/A N/A N/A print N/A N/A N/A N/A N/A Back WHITE N/A Fore RED Use 0, 1 or 2 for verbosity. N/A Back RESET N/A Fore RESET sep N/A N/A IOError N/A N/A N/A print N/A N/A N/A Back WHITE N/A Fore RED N/A N/A self args N/A <projectname> .json does not  N/A N/A exist. Use ``scrapple genconfig``. N/A Back RESET N/A Fore RESET sep N/A run N/A N/A self N/A N/A N/A selectorClass N/A getattr N/A eval N/A N/A self config N/A selector_type N/A N/A N/A N/A N/A self config N/A selector_type title Selector N/A results N/A dict N/A N/A results N/A project N/A N/A self args N/A <projectname> N/A N/A results N/A data N/A list N/A N/A N/A N/A N/A result N/A dict N/A tabular_data_headers N/A dict N/A N/A N/A N/A self args N/A --verbosity 0 N/A N/A N/A print N/A N/A print N/A N/A N/A Back YELLOW N/A Fore BLUE Loading page  N/A N/A N/A N/A N/A self config N/A scraping N/A url N/A Back RESET N/A Fore RESET end N/A selector N/A selectorClass N/A N/A N/A self config N/A scraping N/A url N/A attribute N/A N/A N/A self config N/A scraping N/A data N/A N/A N/A N/A attribute N/A field N/A N/A N/A N/A N/A self args N/A --verbosity 1 N/A N/A N/A print 
Extracting N/A attribute N/A field attribute sep end N/A N/A result N/A N/A attribute N/A field N/A N/A selector extract_content N/A attribute N/A selector N/A attribute N/A attr N/A attribute N/A default N/A N/A N/A N/A N/A N/A self config N/A scraping get table N/A N/A result_list N/A result N/A N/A tables N/A N/A N/A N/A self config N/A scraping get table N/A table tables N/A N/A N/A table_headers result_list N/A N/A selector extract_tabular result result table_type N/A N/A table get table_type rows header N/A N/A table get header N/A prefix N/A N/A table get prefix suffix N/A N/A table get suffix selector N/A N/A table get selector attr N/A N/A table get attr text default N/A N/A table get default verbosity N/A N/A self args N/A --verbosity N/A th table_headers N/A N/A N/A N/A th tabular_data_headers N/A N/A N/A tabular_data_headers N/A th N/A len tabular_data_headers N/A N/A N/A N/A N/A N/A self config N/A scraping get next N/A N/A N/A N/A N/A results N/A data extend result_list N/A N/A next N/A N/A N/A self config N/A scraping N/A next N/A N/A N/A tdh r N/A traverse_next selector next result verbosity N/A N/A self args N/A --verbosity N/A N/A N/A N/A N/A results N/A data append r N/A th tdh N/A N/A N/A N/A th tabular_data_headers N/A N/A N/A tabular_data_headers N/A th N/A len tabular_data_headers N/A N/A N/A KeyboardInterrupt N/A N/A N/A N/A Exception N/A e N/A N/A N/A print e N/A N/A N/A N/A N/A self args N/A --output_type json N/A N/A json N/A N/A open N/A N/A N/A os path join N/A N/A os getcwd N/A N/A N/A self args N/A <output_filename> .json w f N/A N/A N/A N/A json dump results f N/A N/A N/A N/A N/A self args N/A --output_type csv N/A N/A csv N/A N/A open N/A N/A N/A os path join N/A N/A os getcwd N/A N/A N/A self args N/A <output_filename> .csv w f N/A N/A fields N/A extract_fieldnames N/A self config N/A data_headers N/A sorted tabular_data_headers key N/A N/A N/A x N/A N/A tabular_data_headers N/A x N/A N/A N/A fields extend data_headers N/A writer N/A N/A csv DictWriter f fieldnames fields N/A N/A N/A writer writeheader N/A N/A N/A writer writerows N/A results N/A data N/A N/A N/A N/A self args N/A --verbosity 0 N/A N/A N/A print N/A N/A print N/A N/A N/A Back WHITE N/A Fore RED N/A N/A self args N/A <output_filename> . N/A N/A self args N/A --output_type N/A N/A  has been created N/A Back RESET N/A Fore RESET sep N/A N/A