Module Expr Str ImportFrom alias ImportFrom alias alias Import alias ImportFrom alias ImportFrom alias ImportFrom alias ImportFrom alias ImportFrom alias ImportFrom alias ImportFrom alias Assign NameStore Str Assign NameStore Str Assign NameStore NameLoad ClassDef bases NameLoad body Assign NameStore Str Assign NameStore Str Assign NameStore Str Assign NameStore Str Assign NameStore Str Assign NameStore Dict Str Str Str Str Str Str Assign NameStore NameLoad FunctionDef arguments args NameParam NameParam NameParam defaults NameLoad NameLoad body Expr Str Return ListComp Call NameLoad Dict Str Str Str Str AttributeLoad NameLoad attr AttributeLoad NameLoad attr Call AttributeLoad NameLoad attr NameLoad Call AttributeLoad NameLoad attr SubscriptLoad Call AttributeLoad NameLoad attr Str keyword AttributeLoad NameLoad attr Index Num comprehension NameStore Call AttributeLoad NameLoad attr NameLoad NameLoad decorator_list FunctionDef arguments args NameParam NameParam NameParam defaults body Assign NameStore Num Assign NameStore NameLoad Assign NameStore BoolOpOr NameLoad BinOpSub Call AttributeLoad NameLoad attr Call NameLoad AttributeLoad NameLoad attr Assign NameStore BoolOpOr NameLoad Call AttributeLoad NameLoad attr While NameLoad body Assign NameStore Call AttributeLoad NameLoad attr AttributeLoad NameLoad attr keyword Dict Str Str Str NameLoad Call NameLoad NameLoad Call NameLoad NameLoad Assign NameStore Call AttributeLoad NameLoad attr AttributeLoad NameLoad attr For NameStore Call AttributeLoad NameLoad attr Str body Expr Yield NameLoad AugAssignAdd NameStore Num Assign NameStore CompareEq SubscriptLoad Call AttributeLoad NameLoad attr Str Index Num Str decorator_list decorator_list
N/A N/A 
A harvester for the DoE's SciTech Connect Database. Makes use of SciTech's XML Querying Service. Parses the resulting XML for information and outputs the result as Json in a format that works with the OSF scrapi (scraper API).

Example API query: http://www.osti.gov/scitech/scitechxml?EntryDateFrom=02%2F02%2F2015&page=0
 __future__ unicode_literals datetime date timedelta N/A six lxml etree scrapi requests scrapi settings scrapi.base XMLHarvester scrapi.linter RawDocument scrapi.base.schemas DOESCHEMA scrapi.util format_date_with_slashes N/A NAME scitech N/A DEFAULT_ENCODING UTF-8 N/A record_encoding None SciTechHarvester N/A XMLHarvester N/A N/A file_format xml N/A short_name scitech N/A long_name DoE's SciTech Connect Database N/A url http://www.osti.gov/scitech N/A base_url https://www.osti.gov/scitech/scitechxml N/A namespaces N/A rdf dc dcq http://www.w3.org/1999/02/22-rdf-syntax-ns# http://purl.org/dc/elements/1.1/ http://purl.org/dc/terms/ N/A schema DOESCHEMA harvest N/A N/A self start_date end_date N/A None None N/A N/A A function for querying the SciTech Connect database for raw XML.
        The XML is chunked into smaller pieces, each representing data
        about an article/report. If there are multiple pages of results,
        this function iterates through all the pages. N/A N/A N/A RawDocument N/A source filetype doc docID N/A self short_name N/A self file_format N/A N/A etree tostring record N/A N/A six u N/A N/A N/A record xpath dc:ostiId/node() namespaces N/A self namespaces N/A 0 N/A record N/A N/A self _fetch_records start_date end_date N/A _fetch_records N/A N/A self start_date end_date N/A N/A N/A page 0 N/A morepages True N/A start_date N/A start_date N/A N/A N/A date today N/A timedelta N/A settings DAYS_BACK N/A end_date N/A end_date N/A N/A date today N/A morepages N/A N/A resp N/A N/A requests get N/A self base_url params N/A page EntryDateTo EntryDateFrom page N/A format_date_with_slashes end_date N/A format_date_with_slashes start_date N/A xml N/A N/A etree XML N/A resp content N/A record N/A N/A xml xpath records/record N/A N/A N/A record N/A page 1 N/A morepages N/A N/A N/A N/A xml xpath //records/@morepages N/A 0 true N/A N/A