ytools 1.0.0
Command-line tool for selectively dumping nodes from `yaml` (or `json`) documents in `yaml` or `json` format.
License: Apache 2.0
Keywords: yaml,json,transform,xslt,jsonpath,json-path,dump,convert,validate,schema

ytools
======

Command-line tool and ``python`` module for ...

-  validating ``json`` and ``yaml`` files against a ``json-schema`` in
   ``json`` or ``yaml`` format
-  selectively dumping nodes from ``yaml`` (or ``json``) documents in
   ``yaml`` or ``json`` format.

Features


-  Output ``yaml`` as ``json`` or ``python``
-  Output ``json`` as ``yaml`` or ``python`` (provided that there are no
   duplicate mapping entry in the ``json`` source)
-  Extract particular nodes from ``yaml`` and ``json`` files.

   -  If ``yaml`` is used as output format (default) the output is a
      valid ``yaml`` document.

-  Validate ``yaml`` and ``json`` documents.

   -  The ``json-schema`` can be provided in ``yaml`` format as well,
      which improves readability and writability.

-  Preserve order of mapping-keys in ``yaml`` and ``json`` output.
-  Multi-document support

   -  Multiple input files
   -  ... as well as multiple ``yaml`` documents within a file
   -  ... and a combination of both

Installation


``pip install ytools``

Description


| For selecting nodes, ``ytools`` uses ``jsonpath_ng.ext``.
| The syntax is documented at
  https://pypi.python.org/pypi/jsonpath-ng/1.4.2.

| By default (if no path is provided), complete input documents are
  dumped in ``yaml`` format (path defaults to ``'$'``).
| This can be used to get ``yaml`` output for ``json`` documents or vice
  versa:

-  ``python ytools.py input.json``
   ... for converting json to yaml, or ...
-  ``python ytools.py input.yaml -f json``
   ... for the opposite direction.

| Additionally, ``yaml`` and ``json`` documents can be validated against
  a ``json-schema`` which may be provided in ``yaml`` or ``json``
  format.
| ``schema.yaml`` is a sample for ``json-schema`` in ``yaml`` format.

``ytools -h``

::

    $ ytools -h
    Usage: /usr/local/bin/ytools [OPTION] -p JSONPATH_EXPRESSION FILE...

    Dumps data from json (or yaml) documents in yaml format. Command line wrapper
    for jsonpath-ng.

    Options:
      -h, --help            show this help message and exit
      -p PATH, --json-path=PATH
                            Syntax for jsonpath expression:
                            https://pypi.python.org/pypi/jsonpath-ng/1.4.2
      -f OUTPUTFORMAT, --output-format=OUTPUTFORMAT
                            Output format. Can be "yaml", "json" or "python".
                            [default: yaml]
      -y YAML_OPTIONS, --yaml-options=YAML_OPTIONS
                            kwargs for yaml.dump (pyYaml) as yaml.mapping (for
                            experts). [default: '{explicit_start: True,
                            explicit_end: True, allow_unicode: True}']
      -j JSON_OPTIONS, --json-options=JSON_OPTIONS
                            kwargs for json.dumps as yaml.mapping (for experts).
                            [default: '{indent: 2, encoding: utf-8}']
      -v SCHEMA, --validate=SCHEMA
                            Validate documents against json-schema
      --encoding=ENCODING   Set encoding of input documents (if different from
                            utf-8)

Command Line Samples
--------------------

The samples are based on the following data.

Sample Data
~~~~~~~~~~~

Input Data
^^^^^^^^^^

``input.yaml``:

.. code:: yaml

    documents:
      - title: Some document title
        sections:
        - title: Some section title
          description: Some section description
          text: Some text for some section
          chapters:
          - title: Some chapter title
            description: Some chapter description
            text: The text of some chapter
          - title: Some other chapter title
            description:
            - descriptionparagraph1: Some description for other chapter
            - descriptionparagraph2: Some description for other chapter
            text: The text of some other chapter
        - title: Some other section title
          description: Some other section description
          text: Some text for some other section
          chapters:
            - title: About encoding
              description: "Some German: äöü,ÄÖÜ,ß"

Schema for validating ``input.yaml``
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

``schema.yaml``

.. code:: yaml

    $schema: "http://json-schema.org/schema#"

    definitions:
      chapter:
        type: object
        properties:
          title: {type: string}
          description:
            oneOf: [{type: string}, {type: array}]
          text: {type: string}
        additionalProperties: false
        required: [title, description]
      chapters:
        type: array
        items: {$ref: "#/definitions/chapter"}
        additionalItems: false
      section:
        type: object
        properties:
          title: {type: string}
          description: {type: string}
          text: {type: string}
          chapters: {$ref: "#/definitions/chapters"}
        additionalProperties: false
        required: [title, description]
      sections:
        type: array
        items: {$ref: "#/definitions/section"}
        additionalItems: false
      document:
        type: object
        properties:
          title: {type: string}
          description: {type: string}
          sections: {$ref: "#/definitions/sections"}
        additionalProperties: false
        required: [title, description]
      documents:
        type: array
        items: {$ref: "#/definitions/document"}
        additionalItems: false
    type: object
    properties:
      documents: {$ref: "#/definitions/documents"}
    additionalProperties: false

Outputs
~~~~~~~

When not providing a jsonpath expression
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

If you don't provide a jsonpath expression using the ``-p`` option
ytools uses ``'$'`` as default and therefore dumps the complete input:

::

    ��  ytools input.yaml
    ---
    documents:
    - title: Some document title
      description: The document's description
      sections:
      - title: Some section title
        description: Some section description
        text: Some text for some section
        chapters:
        - {title: Some chapter title, description: Some chapter description, text: The
            text of some chapter}
        - title: Some other chapter title
          description:
          - {descriptionparagraph1: Some description for other chapter}
          - {descriptionparagraph2: Some description for other chapter}
          text: The text of some other chapter
      - title: Some other section title
        description: Some other section description
        text: Some text for some other section
        chapters:
        - {title: About encoding, description: 'Some German: äöü,ÄÖÜ,ß'}
    ...

Output format ``yaml`` (default)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

With the ``yaml`` output format by default, each match is output as a
separate ``yaml`` document. This way we achieve that the output is valid
``yaml``.:

::

    ��  ytools -p '$..chapters[*].description' input.yaml
    --- Some chapter description
    ...

    ---
    - {descriptionparagraph1: Some description for other chapter}
    - {descriptionparagraph2: Some description for other chapter}
    ...

    --- 'Some German: äöü,ÄÖÜ,ß'
    ...

If you want different behavior you can set ``explicit_start`` and/or
``explicit_end`` to ``False``. In this case the output will no longer be
guaranteed to be valid ``yaml``:

::

    ��  ytools -p '$..chapters[*].description' input.yaml --yaml-options='{explicit_start: False, explicit_end: False}'
    Some chapter description
    ...

    - {descriptionparagraph1: Some description for other chapter}
    - {descriptionparagraph2: Some description for other chapter}

    'Some German: äöü,ÄÖÜ,ß'

Other output formats
^^^^^^^^^^^^^^^^^^^^

Unfortunately, when using the ``json`` or ``python`` output format the
same selection can't produce valid ``json`` or ``python`` output. That's
because neither json nor python support the concept of (multiple)
documents:

::

    ��  ytools -p '$..chapters[*].description' input.yaml -f json --json-options='{indent: 4}'
    "Some chapter description"
    [
        {
            "descriptionparagraph1": "Some description for other chapter"
        },
        {
            "descriptionparagraph2": "Some description for other chapter"
        }
    ]
    "Some German: \u00e4\u00f6\u00fc,\u00c4\u00d6Y,\u00df"

That's definitely not valid json.

Neither is the following valid python:

::

    ��  ytools -p '$..chapters[*].description' input.yaml -f python
    Some chapter description
    [{'descriptionparagraph1': 'Some description for other chapter'}, {'descriptionparagraph2': 'Some description for other chapter'}]
    Some German: äöü,ÄÖÜ,ß

So **if you ever want to process the output automatically please stick
to ``yaml``**.

Python Samples
--------------

Sample Data
~~~~~~~~~~~

Input Data
^^^^^^^^^^

``test/sampledata.yaml``

.. code:: yaml

    name: 'my_name'
    date: '2017-10-01T10:55:00Z'
    metrics:
      percentage:
        value: 87
        trend: stable

Schema for Validating
^^^^^^^^^^^^^^^^^^^^^

``test/sampleschema.yaml``

.. code:: yaml

    type: object
    properties:
      name: { type: string }
      date: { type: string, format: date-time }
      metrics:
        type: object
        properties:
          percentage:
            type: object
            properties:
              value:
                type: number
                minimum: 0
                maximum: 100
              trend: { type: string, enum: [down, stable, up] }
            additionalProperties: false
            required: [value, trend]
        additionalProperties: false
        required: [percentage]
    additionalProperties: false
    required: [name, date, metrics]

Validation
~~~~~~~~~~

.. code:: python

        ytools.validate("test/sampleschema.yaml", ["test/sampledata.yaml"])

... will not output anything because of successful validation. Play
around if you want to see failing validation - it's quite easy to make
it fail ;-)

Dumping
~~~~~~~

.. code:: python

        ytools.dump("test/sampledata.yaml", "$.metrics", yaml_options="default_flow_style: false")

... will output ...

::

    ---
    percentage:
      value: 87
      trend: stable
    ...