Module Expr Str ImportFrom alias Import alias Import alias ImportFrom alias ImportFrom alias ImportFrom alias ImportFrom alias ImportFrom alias alias alias Assign NameStore Call AttributeLoad NameLoad attr NameLoad Assign NameStore TupleLoad Str Str Str Str Str Str Str Str Str Str Str Str Str Str Str FunctionDef arguments args NameParam defaults body Expr Str Assign NameStore Call AttributeLoad NameLoad attr Str Return Call NameLoad GeneratorExp CompareIn Call AttributeLoad Call AttributeLoad NameLoad attr Str attr NameLoad comprehension NameStore NameLoad decorator_list FunctionDef arguments args NameParam defaults body Expr Str Assign NameStore Call AttributeLoad Call AttributeLoad Call AttributeLoad NameLoad attr attr Str Str attr Str Str Assign NameStore Call AttributeLoad NameLoad attr Str Assign NameStore ListLoad For NameStore NameLoad body If Call NameLoad NameLoad body Expr Call AttributeLoad NameLoad attr NameLoad orelse If BoolOpOr CompareIn Str NameLoad CompareIn Str NameLoad body Assign NameStore Call AttributeLoad Call AttributeLoad NameLoad attr Str Str attr Str Expr Call AttributeLoad NameLoad attr NameLoad orelse Expr Call AttributeLoad NameLoad attr NameLoad Return NameLoad decorator_list FunctionDef arguments args NameParam NameParam defaults body Expr Str Assign NameStore Call AttributeLoad NameLoad attr Str Assign NameStore Call NameLoad NameLoad Assign NameStore ListLoad Assign NameStore BoolOpAnd CompareEq Call NameLoad NameLoad Num CompareEq Call NameLoad NameLoad Num UnaryOpNot CompareEq SubscriptLoad NameLoad Index Num Str For TupleStore NameStore NameStore Call NameLoad NameLoad body If Call NameLoad NameLoad body Assign NameStore Dict Str NameLoad orelse Assign NameStore Call NameLoad NameLoad If NameLoad body Assign SubscriptStore NameLoad Index Str SubscriptLoad NameLoad Index NameLoad Expr Call AttributeLoad NameLoad attr NameLoad If BoolOpAnd UnaryOpNot NameLoad CompareNotEq SubscriptLoad NameLoad Index Num Str body For NameStore NameLoad body Assign NameStore Dict Str Str Str NameLoad Expr Call AttributeLoad NameLoad attr NameLoad Return NameLoad decorator_list FunctionDef arguments args NameParam NameParam NameParam defaults body Expr Str If UnaryOpNot NameLoad body Return ListLoad orelse Assign NameStore Dict Str Str NameLoad BoolOpOr Call AttributeLoad Str attr NameLoad NameLoad Str Return ListLoad NameLoad decorator_list FunctionDef arguments args NameParam NameParam NameParam defaults body Expr Str Return Call AttributeLoad Str attr ListLoad NameLoad NameLoad NameLoad decorator_list FunctionDef arguments args NameParam NameParam defaults body Expr Str Assign NameStore ListLoad For NameStore NameLoad body If BoolOpOr CompareEq SubscriptLoad NameLoad Index Str Str CompareEq SubscriptLoad NameLoad Index Str Str body Expr Call AttributeLoad NameLoad attr SubscriptLoad NameLoad Index Str If CompareEq NameLoad ListLoad body Return ListLoad NameLoad orelse Return Call AttributeLoad ListLoad NameLoad attr NameLoad decorator_list ClassDef bases NameLoad body Assign NameStore Str Assign NameStore Str Assign NameStore Str Assign NameStore Str Assign NameStore Str Assign NameStore NameLoad FunctionDef arguments args NameParam defaults body Return Dict Str Str Str Str Str Str Str Str Str TupleLoad Str Lambda arguments args NameParam defaults BoolOpOr NameLoad Str Str TupleLoad Str NameLoad Dict Str Str TupleLoad Str Lambda arguments args NameParam defaults Call NameLoad AttributeLoad NameLoad attr AttributeLoad NameLoad attr NameLoad TupleLoad Str Str NameLoad TupleLoad Str Str NameLoad TupleLoad Str Str Str NameLoad TupleLoad Str Lambda arguments args NameParam defaults ListComp Call AttributeLoad SubscriptLoad NameLoad Index Str attr comprehension NameStore BoolOpOr NameLoad ListLoad Dict Str TupleLoad Str Str Lambda arguments args NameParam NameParam defaults IfExp NameLoad Call AttributeLoad Call AttributeLoad Call NameLoad NameLoad attr attr NameLoad Call NameLoad TupleLoad Str Str TupleLoad Str Str TupleLoad Str TupleLoad Str NameLoad TupleLoad Str Str TupleLoad Str TupleLoad Str NameLoad TupleLoad Str Str TupleLoad Str Str TupleLoad Str Str TupleLoad Str Str TupleLoad Str Str TupleLoad Str Str TupleLoad Str Str TupleLoad Str Str decorator_list NameLoad FunctionDef arguments args NameParam NameParam NameParam defaults NameLoad NameLoad body Expr Str Assign NameStore Str Assign NameStore SubscriptLoad Call AttributeLoad Call AttributeLoad NameLoad attr NameLoad attr Index Str Assign NameStore Call NameLoad NameLoad Expr Call AttributeLoad NameLoad attr Call AttributeLoad Str attr NameLoad Return ListComp Call NameLoad Dict Str Str Str Str Call AttributeLoad NameLoad attr NameLoad AttributeLoad NameLoad attr SubscriptLoad NameLoad Index Str Str comprehension NameStore NameLoad decorator_list decorator_list
N/A N/A 
A Lake Winnipeg Basin Information Network (BIN) harvester for the SHARE project

Example API request: http://130.179.67.140/api/3/action/package_search?q= (problematic)
http://130.179.67.140/api/3/action/current_package_list_with_resources (currently using)
It oddly returns 5 more datasets than all searchable ones on LWBIN data hub.

Known issues:
1 -- Five datasets can be searched but cannot be accessed via LWBIN.
Clicking on the searching result would result in linking to a redirected page like this:
http://130.179.67.140/user/login?came_from=http://130.179.67.140/dataset/mpca-surface-water-data-access-interactive-map
Within each dataset there are resouces that contain urls to source pages. For future work considering using resources
urls as canonical urls.
2 -- Resouces properties contained in raw metadata of the datasets are not added to the normalized metadata at this
point.
3 -- Single name contributors can be used as filters or an invalid query will be returned. Has nothing to do with scrapi but the frontend.
 __future__ unicode_literals N/A json N/A logging dateutil.parser parse scrapi requests scrapi.base JSONHarvester scrapi.linter.document RawDocument scrapi.base.helpers build_properties datetime_formatter parse_name N/A logger N/A N/A logging getLogger __name__ N/A ORGANIZATIONS N/A organization fund canada agriculture commitee international council office of observation institute lwbin cocorahs usgs nsidc is_organization N/A N/A name N/A N/A N/A Return a boolean to indicate if the name passed to the function is an organization
     N/A words N/A N/A name split N/A N/A any N/A N/A N/A N/A N/A N/A word strip ; lower ORGANIZATIONS N/A word words N/A clean_authors N/A N/A authors N/A N/A N/A Cleam authors list.
     N/A authors N/A N/A N/A N/A N/A N/A authors strip replace <span class="author-names"> replace </span> N/A authors N/A N/A authors split , N/A new_authors N/A N/A author authors N/A N/A N/A is_organization author N/A N/A N/A N/A new_authors append author N/A N/A N/A N/A  and  author N/A  <em>et al.</em> author N/A N/A split_name N/A N/A N/A N/A author replace  <em>et al.</em> split  and  N/A N/A N/A new_authors extend split_name N/A N/A N/A N/A new_authors append author N/A new_authors N/A process_contributors N/A N/A authors emails N/A N/A N/A Process authors and add author emails
    If multiple authors and one email, put email in a new author
     N/A emails N/A N/A emails split , N/A authors N/A clean_authors authors N/A contributor_list N/A N/A append_emails N/A N/A N/A len authors 1 N/A N/A len emails 1 N/A N/A N/A emails N/A 0 N/A N/A i author N/A enumerate authors N/A N/A N/A is_organization author N/A N/A contributor N/A name author N/A N/A contributor N/A parse_name author N/A append_emails N/A N/A N/A contributor N/A email N/A emails N/A i N/A N/A N/A contributor_list append contributor N/A N/A N/A append_emails N/A N/A emails N/A 0 N/A N/A email emails N/A N/A contributor N/A name email email N/A N/A N/A contributor_list append contributor N/A contributor_list N/A process_licenses N/A N/A license_title license_url license_id N/A N/A N/A Process licenses to comply with the normalized schema
     N/A N/A license_url N/A N/A N/A N/A N/A license N/A uri description license_url N/A N/A N/A {} ({}) format license_title license_id N/A N/A license N/A construct_url N/A N/A url dataset_path end_point N/A N/A N/A 
    :return: a url that directs back to the page on LBWIN Data Hub instead of the source page.
    :param url: host url
    :param dataset_path: parent path of all datasets
    :param end_point: name of datasets
     N/A N/A N/A / join N/A url dataset_path end_point N/A process_object_uris N/A N/A url extras N/A N/A N/A Extract doi from /extras, and return a list of object uris including /url and doi if it exists.
     N/A doi N/A N/A d extras N/A N/A N/A N/A N/A d N/A key DOI N/A N/A d N/A key DOI: N/A N/A N/A N/A doi append N/A d N/A value N/A N/A doi N/A N/A N/A N/A url N/A N/A N/A N/A N/A url extend doi N/A LWBINHarvester N/A JSONHarvester N/A N/A short_name lwbin N/A long_name Lake Winnipeg Basin Information Network N/A url http://130.179.67.140 N/A dataset_path dataset N/A DEFAULT_ENCODING UTF-8 N/A record_encoding None schema N/A N/A self N/A N/A N/A N/A title description providerUpdatedDateTime uris contributors licenses tags freeToRead otherProperties N/A /title N/A N/A N/A x N/A N/A x /notes N/A /metadata_modified datetime_formatter N/A canonicalUri objectUris N/A /name N/A N/A N/A x N/A N/A construct_url N/A self url N/A self dataset_path x N/A /url /extras process_object_uris N/A /author /author_email process_contributors N/A /license_title /license_url /license_id process_licenses N/A /tags N/A N/A N/A x N/A N/A N/A N/A N/A tag N/A name lower N/A tag N/A x N/A N/A startDate N/A /isopen /metadata_created N/A N/A N/A x y N/A N/A x N/A N/A N/A N/A N/A parse y date isoformat None N/A build_properties N/A maintainer /maintainer N/A maintainerEmail /maintainer_email N/A revisionTimestamp N/A /revision_timestamp datetime_formatter N/A id /id N/A metadataCreated N/A /metadata_created datetime_formatter N/A state /state N/A version /version N/A creatorUserId /creator_user_id N/A type /type N/A numberOfResources /num_resources N/A numberOfTags /num_tags N/A name /name N/A groups /groups N/A property harvest N/A N/A self start_date end_date N/A None None N/A N/A Returns a list of Rawdocuments (metadata)
        Searching by time is not supported by LWBIN CKAN API. all datasets have to be scanned each time.
         N/A base_url http://130.179.67.140/api/3/action/current_package_list_with_resources N/A records N/A N/A N/A N/A N/A requests get base_url json N/A result N/A total N/A len records N/A N/A N/A logger info N/A N/A {} documents to be harvested format total N/A N/A N/A RawDocument N/A doc source docID filetype N/A N/A json dumps record N/A self short_name N/A record N/A id json N/A record records N/A N/A