Module Import alias ImportFrom alias Import alias ImportFrom alias Import alias Import alias FunctionDef arguments args NameParam defaults body Assign NameStore Call AttributeLoad NameLoad attr TupleLoad Str Str BoolOpOr AttributeLoad NameLoad attr Str AttributeLoad NameLoad attr AttributeLoad NameLoad attr Str Assign NameStore AttributeLoad NameLoad attr Assign NameStore AttributeLoad NameLoad attr Assign NameStore AttributeLoad NameLoad attr Assign NameStore AttributeLoad NameLoad attr If CompareIs NameLoad NameLoad body Assign NameStore IfExp CompareEq NameLoad Str Num Num Return TupleLoad NameLoad NameLoad NameLoad NameLoad NameLoad decorator_list ClassDef bases NameLoad body Expr Str FunctionDef arguments args NameParam defaults body Assign AttributeStore NameLoad attr Call NameLoad Str Str Assign NameStore Call AttributeLoad NameLoad attr Expr Call AttributeLoad NameLoad attr AttributeLoad NameLoad attr decorator_list Expr Str FunctionDef arguments args NameParam NameParam defaults body Assign NameStore AttributeLoad NameLoad attr Assign NameStore AttributeLoad NameLoad attr Assign NameStore Call NameLoad NameLoad Assign TupleStore NameStore NameStore NameStore NameStore NameStore Call NameLoad NameLoad Expr Call AttributeLoad NameLoad attr Str NameLoad If BoolOpAnd CompareIsNot NameLoad NameLoad CompareGt Call NameLoad NameLoad Num body Assign SubscriptStore NameLoad Index Str Call NameLoad NameLoad Expr Call AttributeLoad NameLoad attr Str Str Assign NameStore Call NameLoad Expr Call AttributeLoad NameLoad attr BinOpMod Str TupleLoad AttributeLoad NameLoad attr NameLoad For TupleStore NameStore NameStore Call AttributeLoad NameLoad attr body For NameStore NameLoad body Expr Call AttributeLoad NameLoad attr BinOpMod Str TupleLoad NameLoad NameLoad Expr Call AttributeLoad NameLoad attr Str If CompareIsNot NameLoad NameLoad body Expr Call AttributeLoad NameLoad attr NameLoad Assign NameStore Call AttributeLoad NameLoad attr Return Call AttributeLoad NameLoad attr keyword AttributeLoad NameLoad attr keyword NameLoad decorator_list Expr Str FunctionDef arguments args NameParam NameParam defaults body Assign NameStore BinOpAdd BinOpAdd Str Call NameLoad AttributeLoad NameLoad attr Str AugAssignAdd NameStore Call AttributeLoad AttributeLoad NameLoad attr attr AugAssignAdd NameStore Str AugAssignAdd NameStore AttributeLoad NameLoad attr Return Call AttributeLoad NameLoad attr keyword AttributeLoad NameLoad attr keyword NameLoad decorator_list FunctionDef arguments args NameParam NameParam NameParam defaults body Assign NameStore Call AttributeLoad NameLoad attr NameLoad Expr Call AttributeLoad NameLoad attr AttributeLoad NameLoad attr decorator_list FunctionDef arguments args NameParam NameParam NameParam NameParam defaults body Assign NameStore Call AttributeLoad NameLoad attr NameLoad Expr Call AttributeLoad NameLoad attr AttributeLoad NameLoad attr Return NameLoad decorator_list decorator_list
N/A N/A urlparse cStringIO StringIO N/A twisted.web.http scrapy.utils.httpobj urlparse_cached N/A scrapy.http N/A warcrecords _parsed_url_args N/A N/A parsed N/A N/A N/A path N/A N/A urlparse urlunparse N/A N/A N/A parsed path / N/A parsed params N/A parsed query N/A host N/A parsed hostname N/A port N/A parsed port N/A scheme N/A parsed scheme N/A netloc N/A parsed netloc N/A N/A port None N/A N/A port N/A N/A scheme https 443 80 N/A N/A scheme netloc host port path N/A WarcMiddleware N/A object N/A N/A 
    Open the WARC file for output and write the warcinfo header record
    
     __init__ N/A N/A self N/A N/A N/A N/A self fo N/A open out.warc wb N/A record N/A N/A warcrecords WarcinfoRecord N/A N/A N/A record write_to N/A self fo N/A N/A 
    Converts a Scrapy request to a WarcRequestRecord
    Follows most of the code from scrapy/core/downloader/webclient.py
    
     _warcrec_from_scrapy_request N/A N/A self request N/A N/A N/A headers N/A request headers N/A body N/A request body N/A parsed N/A urlparse_cached request N/A N/A scheme netloc host port path N/A _parsed_url_args parsed N/A N/A N/A headers setdefault Host netloc N/A N/A N/A body None N/A N/A len body 0 N/A N/A N/A headers N/A Content-Length N/A len body N/A N/A N/A headers setdefault Connection close N/A buf N/A StringIO N/A N/A N/A buf write N/A %s %s HTTP/1.0
 N/A N/A request method path N/A N/A name values N/A N/A headers items N/A N/A value values N/A N/A N/A N/A buf write N/A %s: %s
 N/A name value N/A N/A N/A buf write N/A N/A body None N/A N/A N/A N/A buf write body N/A request_str N/A N/A buf getvalue N/A N/A N/A warcrecords WarcRequestRecord url N/A request url block request_str N/A N/A 
    Converts a Scrapy response to a WarcResponseRecord
    
    tofix: Handle response.status codes
    
     _warcrec_from_scrapy_response N/A N/A self response N/A N/A N/A resp_str N/A N/A HTTP/1.0  N/A str N/A response status  OK
 N/A resp_str N/A N/A N/A response headers to_string N/A resp_str N/A resp_str N/A response body N/A N/A N/A warcrecords WarcResponseRecord url N/A response url block resp_str N/A process_request N/A N/A self request spider N/A N/A N/A record N/A N/A self _warcrec_from_scrapy_request request N/A N/A N/A record write_to N/A self fo N/A process_response N/A N/A self request response spider N/A N/A N/A record N/A N/A self _warcrec_from_scrapy_response response N/A N/A N/A record write_to N/A self fo N/A response N/A N/A