ckanny 0.17.2
Miscellaneous CKAN utility scripts
License: MIT
Keywords: ckanny

# ckanny

## Introduction

ckanny is a [command line interface](#cli) for interacting with remote and local [CKAN](http://ckan.org/) instances. Under the hood, it uses [ckanutils](https://github.com/reubano/ckanutils).

With ckanny, you can

- Download a CKAN resource
- Create a CKAN package
- Update a CKAN DataStore from data in the FileStore
- Copy a FileStore resource from one CKAN instance to another
- and much more...

ckanny performs smart updates by computing the hash of a file and will only update the datastore if the file has changed. This allows you to schedule a script to run on a frequent basis, e.g., `@hourly` via a cron job, without updating the CKAN instance unnecessarily.

## Requirements

ckanny has been tested on the following configuration:

- MacOS X 10.9.5
- Python 2.7.9

ckanny requires the following in order to run properly:

- [Python >= 2.7](http://www.python.org/download) (MacOS X comes with python preinstalled)

## Installation

(You are using a [virtualenv](http://www.virtualenv.org/en/latest/index.html), right?)

     sudo pip install ckanny

## CLI

ckanny comes with a built in command line interface `ckanny`.

### Usage

     ckanny [<namespace>.]<command> [<args>]

### Examples

*show help*

    ckanny -h

```bash
usage: ckanny [<namespace>.]<command> [<args>]

positional arguments:
  command     the command to run

optional arguments:
  -h, --help  show this help message and exit

available commands:
  ver                      Show ckanny version

  [ds]
    delete                 Deletes a datastore table
    update                 Updates a datastore table based on the current filestore resource
    upload                 Uploads a file to a datastore table

  [fs]
    fetch                  Downloads a filestore resource
    migrate                Copies a filestore resource from one ckan instance to another
    upload                 Updates the filestore of an existing resource or creates a new one

  [hdx]
    customize              Introspects custom organization values

  [pk]
    create                 Creates a package (aka dataset)
```

*show version*

    ckanny ver

*fetch a resource*

    ckanny fs.fetch -k <CKAN_API_KEY> -r <CKAN_URL> <resource_id>

*show fs.fetch help*

    ckanny fs.fetch -h

```bash
usage: ckanny fs.fetch
       [-h] [-q] [-n] [-c CHUNKSIZE_BYTES] [-u UA] [-k API_KEY] [-r REMOTE]
       [-d DESTINATION]
       [resource_id]

Downloads a filestore resource

positional arguments:
  resource_id           the resource id

optional arguments:
  -h, --help            show this help message and exit
  -q, --quiet           suppress debug statements
  -n, --name-from-id    Use resource id for filename
  -c CHUNKSIZE_BYTES, --chunksize-bytes CHUNKSIZE_BYTES
                        number of bytes to read/write at a time (default:
                        1048576)
  -u UA, --ua UA        the user agent (uses `CKAN_USER_AGENT` ENV if
                        available) (default: None)
  -k API_KEY, --api-key API_KEY
                        the api key (uses `CKAN_API_KEY` ENV if available)
                        (default: None)
  -r REMOTE, --remote REMOTE
                        the remote ckan url (uses `CKAN_REMOTE_URL` ENV if
                        available) (default: None)
  -d DESTINATION, --destination DESTINATION
                        the destination folder or file path (default:
                        .)
```

*create a package*

    ckanny pk.create -k <CKAN_API_KEY> -r <CKAN_URL> <org_id>

*create a package with resources*

    ckanny pk.create -k <CKAN_API_KEY> -r <CKAN_URL> -f 'file1.csv,file2.csv' <org_id>

*show pk.create help*

    ckanny pk.create -h

```bash
usage: /Users/reubano/.virtualenvs/ckan/bin/ckanny pk.create
       [-h] [-q] [-u UA] [-k API_KEY]
       [-r REMOTE] [-e END] [-S START]
       [-L LOCATION] [-c CAVEATS] [-y TYPE]
       [-T TAGS] [-t TITLE]
       [-m {observed,other,census,survey,registry}]
       [-d DESCRIPTION] [-f FILES] [-s SOURCE]
       [-l LICENSE_ID]
       [org_id]

Creates a package (aka dataset)

positional arguments:
  org_id                the organization id

optional arguments:
  -h, --help            show this help message and exit
  -q, --quiet           suppress debug statements
  -u UA, --ua UA
                        the user agent (uses `CKAN_USER_AGENT` ENV if
                        available) (default: None)
  -k API_KEY, --api-key API_KEY
                        the api key (uses `CKAN_API_KEY` ENV if available)
                        (default: None)
  -r REMOTE, --remote REMOTE
                        the remote ckan url (uses `CKAN_REMOTE_URL` ENV if
                        available) (default: None)
  -e END, --end END
                        Data end date
  -S START, --start START
                        Data start date (default: 09/25/2015)
  -L LOCATION, --location LOCATION
                        Location the data represents (default: world)
  -c CAVEATS, --caveats CAVEATS
                        Package caveats
  -y TYPE, --type TYPE
                        Package type (default: dataset)
  -T TAGS, --tags TAGS
                        Comma separated list of tags
  -t TITLE, --title TITLE
                        Package title (default: Untitled 2015-09-25
                        12:36:14.141533)
  -m {observed,other,census,survey,registry}, --methodology {observed,other,census,survey,registry}
                        Data collection methodology (default: observed)
  -d DESCRIPTION, --description DESCRIPTION
                        Dataset description (default: same as `title`)
  -f FILES, --files FILES
                        Comma separated list of file paths to add
  -s SOURCE, --source SOURCE
                        Data source (default: Multiple sources)
  -l LICENSE_ID, --license-id LICENSE_ID
                        Data license (default: cc-by-igo)
```
## Configuration

ckanny will use the following [Environment Variables](http://www.cyberciti.biz/faq/set-environment-variable-linux/) if set:

Environment Variable|Description
--------------------|-----------
CKAN_API_KEY|Your CKAN API Key
CKAN_REMOTE_URL|Your CKAN instance remote url
CKAN_USER_AGENT|Your user agent

## Hash Table

In order to support file hashing, ckanny creates a hash table resource called `hash_table.csv` with the following schema:

field|type
------|----
datastore_id|text
hash|text

By default the hash table resource will be placed in the package `hash_table`. ckanny will create this package if it doesn't exist. Optionally, you can set the hash table package in the command line with the `-H, --hash-table` option, or in a Python file as the `hash_table` keyword argument to `CKAN`.

Example:

    ckanny ds.update -H custom_hash_table 36f33846-cb43-438e-95fd-f518104a32ed

## Scripts

ckanny comes with a built in task manager `manage.py` and a `Makefile`.

### Setup

    pip install -r dev-requirements.txt

### Examples

*Run python linter and nose tests*

```bash
manage lint
manage test
```

Or if `make` is more your speed...

```bash
make lint
make test
```

## Contributing

View [CONTRIBUTING.rst](https://github.com/reubano/ckanny/blob/master/CONTRIBUTING.rst)

## License

ckanny is distributed under the [MIT License](http://opensource.org/licenses/MIT), the same as [ckanutils](https://github.com/reubano/ckanutils).


=========
Changelog
=========

Here you can find the recent changes to ckanny..

.. changelog::
    :version: dev
    :released: Ongoing

    .. change::
        :tags:  docs

        Updated CHANGES.

.. changelog::
    :version: 0.1.0
    :released: 2015-06-12

    .. change::
        :tags: project

        First release on PyPi.

.. todo:: vim: set filetype=rst: