Module Expr Str ImportFrom alias Import alias Import alias ImportFrom alias alias Import alias ImportFrom alias ImportFrom alias ImportFrom alias ImportFrom alias ImportFrom alias ImportFrom alias ImportFrom alias alias alias alias alias Assign NameStore Call AttributeLoad NameLoad attr NameLoad Assign NameStore Call NameLoad AttributeLoad NameLoad attr AttributeLoad NameLoad attr FunctionDef arguments args NameParam defaults body Return UnaryOpNot Call NameLoad NameLoad NameLoad decorator_list ClassDef bases NameLoad body Assign NameStore Str Assign NameStore Str Assign NameStore Str Assign NameStore Str Assign NameStore NameLoad Assign NameStore Dict Str Str Str Str Str Str Str Str TupleLoad Str NameLoad Dict Str TupleLoad Str NameLoad TupleLoad Str Call NameLoad NameLoad NameLoad TupleLoad Str Str Lambda arguments args NameParam NameParam defaults BoolOpOr Call NameLoad NameLoad Call NameLoad NameLoad TupleLoad Str Str Lambda arguments args NameParam NameParam defaults BoolOpOr Call NameLoad NameLoad Call NameLoad NameLoad TupleLoad Str Lambda arguments args NameParam defaults ListComp Call AttributeLoad NameLoad attr comprehension NameStore NameLoad ListLoad Dict Str Dict Str TupleLoad Str NameLoad Dict Str Dict Str TupleLoad Str NameLoad Call NameLoad TupleLoad Str Str TupleLoad Str Str TupleLoad Str Str TupleLoad Str Str TupleLoad Str Str TupleLoad Str Str TupleLoad Str Str TupleLoad Str Str TupleLoad Str Str TupleLoad Str Str TupleLoad Str Str TupleLoad Str Str TupleLoad Str Str TupleLoad Str Str TupleLoad Str Str TupleLoad Str Str TupleLoad Str Str TupleLoad Str Str TupleLoad Str Str TupleLoad Str Str TupleLoad Str Str TupleLoad Str Str TupleLoad Str Str TupleLoad Str Str TupleLoad Str Str TupleLoad Str TupleLoad Str Call NameLoad Lambda arguments args NameParam defaults Call NameLoad Call NameLoad NameLoad NameLoad Lambda arguments args NameParam defaults Call NameLoad Call NameLoad NameLoad NameLoad TupleLoad Str Str TupleLoad Str Str FunctionDef arguments args NameParam defaults body Return NameLoad decorator_list NameLoad FunctionDef arguments args NameParam NameParam NameParam defaults NameLoad NameLoad body Expr Str Assign NameStore BoolOpOr NameLoad BinOpSub Call AttributeLoad NameLoad attr Call NameLoad AttributeLoad NameLoad attr Assign NameStore BoolOpOr NameLoad Call AttributeLoad NameLoad attr Assign NameStore Call AttributeLoad NameLoad attr Str Assign NameStore Call AttributeLoad NameLoad attr Str Assign NameStore Call AttributeLoad NameLoad attr Str Assign NameStore Call AttributeLoad NameLoad attr Str Assign NameStore Call AttributeLoad NameLoad attr Str Assign NameStore Call AttributeLoad NameLoad attr Str Assign NameStore Str Assign NameStore Call AttributeLoad Str attr NameLoad NameLoad NameLoad NameLoad NameLoad NameLoad Assign NameStore BinOpAdd NameLoad NameLoad Assign NameStore Call AttributeLoad NameLoad attr NameLoad Assign NameStore AttributeLoad NameLoad attr Assign NameStore Call AttributeLoad NameLoad attr AttributeLoad NameLoad attr Assign NameStore Call NameLoad SubscriptLoad Call AttributeLoad NameLoad attr Str Index Num Assign NameStore ListLoad If CompareGt Call NameLoad NameLoad Num body Assign NameStore BinOpAdd BinOpAdd NameLoad Str Call NameLoad NameLoad Assign NameStore Call AttributeLoad NameLoad attr NameLoad Assign NameStore Call AttributeLoad NameLoad attr AttributeLoad NameLoad attr Assign NameStore ListLoad For NameStore Call AttributeLoad NameLoad attr Str body Expr Call AttributeLoad NameLoad attr BinOpAdd SubscriptLoad Call AttributeLoad NameLoad attr Str Index Num Str Expr Call AttributeLoad NameLoad attr Call AttributeLoad Str attr Call NameLoad NameLoad Assign NameStore Num Assign NameStore Num For NameStore NameLoad body TryExcept body Assign NameStore Call AttributeLoad NameLoad attr NameLoad handlers ExceptHandler type AttributeLoad AttributeLoad NameLoad attr attr name NameStore body Expr Call AttributeLoad NameLoad attr Call AttributeLoad Str attr NameLoad Expr Call AttributeLoad NameLoad attr Num Continue Assign NameStore Call AttributeLoad NameLoad attr AttributeLoad NameLoad attr Assign NameStore Call AttributeLoad NameLoad attr NameLoad keyword NameLoad Assign NameStore SubscriptLoad Call AttributeLoad NameLoad attr Str Index Num Expr Call AttributeLoad NameLoad attr Call NameLoad Dict Str Str Str Str NameLoad AttributeLoad NameLoad attr Call NameLoad NameLoad Str AugAssignAdd NameStore Num AugAssignAdd NameStore Num If CompareEq BinOpMod NameLoad Num Num body Expr Call AttributeLoad NameLoad attr Call AttributeLoad Str attr NameLoad Assign NameStore Num Return NameLoad decorator_list decorator_list
N/A N/A 
API harvester for ClinicalTrials.gov for the SHARE Notification Service

http://clinicaltrials.gov/ct2/results?lup_s=04%2F26%2F2015%2F&lup_e=04%2F27%2F2015&displayxml=true
iindividual result: http://ClinicalTrials.gov/show/NCT02425332?displayxml=true

 __future__ unicode_literals N/A time N/A logging datetime date timedelta N/A xmltodict lxml etree scrapi requests scrapi settings scrapi.base XMLHarvester scrapi.util copy_to_unicode scrapi.linter.document RawDocument scrapi.base.helpers compose single_result build_properties datetime_formatter default_name_parser N/A logger N/A N/A logging getLogger __name__ N/A element_to_dict N/A compose N/A xmltodict parse N/A etree tostring non_string N/A N/A item N/A N/A N/A N/A N/A isinstance item str N/A ClinicalTrialsHarvester N/A XMLHarvester N/A N/A short_name clinicaltrials N/A long_name ClinicalTrials.gov N/A url https://clinicaltrials.gov/ N/A DEFAULT_ENCODING UTF-8 N/A record_encoding None N/A schema N/A contributors uris providerUpdatedDateTime title description tags sponsorships otherProperties N/A //overall_official/last_name/node() default_name_parser N/A canonicalUri N/A //required_header/url/node() single_result N/A lastchanged_date/node() N/A compose datetime_formatter single_result N/A //official_title/node() //brief_title/node() N/A N/A N/A x y N/A N/A N/A single_result x N/A single_result y N/A //brief_summary/textblock/node() //brief_summary/textblock/node() N/A N/A N/A x y N/A N/A N/A single_result x N/A single_result y N/A //keyword/node() N/A N/A N/A tags N/A N/A N/A N/A tag lower N/A tag tags N/A N/A sponsor N/A sponsorName N/A //sponsors/lead_sponsor/agency/node() single_result N/A sponsor N/A sponsorName N/A //sponsors/collaborator/agency/node() single_result N/A build_properties N/A serviceID //nct_id/node() N/A oversightAuthority //oversight_info/authority/node() N/A studyDesign //study_design/node() N/A numberOfArms //number_of_arms/node() N/A source //source/node() N/A verificationDate //verification_date/node() N/A lastChanged //lastchanged_date/node() N/A condition //condition/node() N/A verificationDate //verification_date/node() N/A lastChanged //lastchanged_date/node() N/A status //status/node() N/A locationCountries //location_countries/country/node() N/A isFDARegulated //is_fda_regulated/node() N/A isSection801 //is_section_801/node() N/A hasExpandedAccess //has_expanded_access/node() N/A leadSponsorAgencyClass //lead_sponsor/agency_class/node() N/A collaborator //collaborator/agency/node() N/A collaboratorAgencyClass //collaborator/agency_class/node() N/A measure //primary_outcome/measure/node() N/A timeFrame //primary_outcome/time_frame/node() N/A safetyIssue //primary_outcome/safety_issue/node() N/A secondaryOutcomes //secondary_outcome/measure/node() N/A enrollment //enrollment/node() N/A armGroup //arm_group/arm_group_label/node() N/A intervention //intervention/intervention_type/node() N/A eligibility N/A //eligibility/node() N/A compose N/A N/A N/A x N/A N/A list N/A map element_to_dict x N/A N/A N/A x N/A N/A list N/A filter non_string x N/A link //link/url/node() N/A responsible_party //responsible_party/responsible_party_full_name/node() namespaces N/A N/A self N/A N/A N/A None N/A property harvest N/A N/A self start_date end_date N/A None None N/A N/A  First, get a list of all recently updated study urls,
        then get the xml one by one and save it into a list
        of docs including other information  N/A start_date N/A start_date N/A N/A N/A date today N/A timedelta N/A settings DAYS_BACK N/A end_date N/A end_date N/A N/A date today N/A end_month N/A N/A end_date strftime %m N/A end_day N/A N/A end_date strftime %d N/A end_year N/A N/A end_date strftime %Y N/A start_month N/A N/A start_date strftime %m N/A start_day N/A N/A start_date strftime %d N/A start_year N/A N/A start_date strftime %Y N/A base_url http://clinicaltrials.gov/ct2/results?lup_s= N/A url_end N/A N/A {}%2F{}%2F{}&lup_e={}%2F{}%2F{}&displayxml=true format start_month start_day start_year end_month end_day end_year N/A url N/A base_url url_end N/A initial_request N/A N/A requests get url N/A record_encoding N/A initial_request encoding N/A initial_request_xml N/A N/A etree XML N/A initial_request content N/A count N/A int N/A N/A N/A initial_request_xml xpath //search_results/@count N/A 0 N/A xml_list N/A N/A N/A N/A int count 0 N/A N/A url N/A N/A url &count= N/A str count N/A total_requests N/A N/A requests get url N/A initial_doc N/A N/A etree XML N/A total_requests content N/A study_urls N/A N/A study N/A N/A initial_doc xpath //clinical_study N/A N/A N/A N/A study_urls append N/A N/A N/A N/A study xpath url/node() N/A 0 ?displayxml=true N/A N/A N/A logger info N/A N/A There are {} urls to harvest - be patient... format N/A len study_urls N/A count 0 N/A official_count 0 N/A study_url study_urls N/A N/A N/A N/A content N/A N/A requests get study_url N/A N/A N/A N/A N/A requests exceptions ConnectionError N/A e N/A N/A N/A N/A logger info N/A N/A Connection error: {}, wait a bit... format e N/A N/A N/A time sleep 30 N/A N/A doc N/A N/A etree XML N/A content content N/A record N/A N/A etree tostring doc encoding record_encoding N/A doc_id N/A N/A N/A doc xpath //nct_id/node() N/A 0 N/A N/A N/A xml_list append N/A RawDocument N/A doc source docID filetype record N/A self short_name N/A copy_to_unicode doc_id xml N/A official_count 1 N/A count 1 N/A N/A N/A count 100 0 N/A N/A N/A N/A logger info N/A N/A You've requested {} studies, keep going! format official_count N/A count 0 N/A xml_list N/A N/A