kson 0.1.0
Keystripped Schemafied Object Notation
License: BSD License
Keywords: kson json parser javascript schema

KSON: Keystripped Schemafied Object Notation
============================================

KSON is a simple data interchange format based on JSON. Its serialized
representation doesn't contain the redundant keys of typical json 
documents and more compact representations of values are possible through
an extensible encoding/decoding mechanism.

KSON uses a simple schema format to describe arbitrarrily nested objects
and encoded types. The minified and gzipped javascript library is < 1K.


Usage in javascript
-------------------

    // Load schema definitions

    var movie_schemas = [
      {
        "id": "role",
        "fields": ["name", "character"],
        "meta": [0, 0]
      },{
        "id": "movie",
        "fields": ["title", "year", "rating", "cover", "actors"],
        "meta": [0, 0, 0, "prefix:http\://movies.db/covers/|suffix.jpg", "[]role"]
      }
    ];
    KSON.addSchema(movie_schemas);

    var movies = [
      {
       "title": "Forrest Gump",
       "year": 1994,
       "rating": 8.7,
       "cover": "http://movies.db/covers/8.jpg",
       "actors": [
        {"name": "Tom Hanks", "character": "Forest Gump"},
        {"name": "Robin Wright", "character": "Jenny Curran"},
        {"name": "Gary Sinise", "character": "Lieutenant Dan Taylor"}
       ]
      },
      {
        "title": "Toy Story",
        ...
      }
      ...
    ];

    // pretty printing is not part of the library
    console.log(KSON.stringify(movies, "[]movie"));
    [
      "[]movie",
      "Forest Gump",
      1994,
      8.7,
      "8",
      [
        ["Tom Hanks", "Forest Gump"],
        ["Robin Wright", "Jenny Curran"],
        ["Gary Sinise", "Lieutenant Dan Taylor"]
      ],
      "Toy Story",
      ...
    ]

    // Of course the "movie" and "role" schema definitions may also be
    // serialized to KSON using the "schema" schema:

    var schema_data = KSON.stringify(movie_schemas, "[]schema")
    console.log(schema_data);

    ["[]schema","role",["name","character"],[0, 0],"movie",
      ["title","year","rating","cover","actors"],
      [0,0,0,"prefix:http\://movies.db/covers/|suffix:.jpg","[]role"]]

    // Likewise schemas can be initialized with a kson document
    KSON.addSchema(schema_data);

FAQ
---

Q: Why should I use this instead of JSON/MessagePack/Thrift/...?
A: Unless you are targeting the browser you probably shouldn't. If you
   are, then the size and speed of the javascript libraries for these
   formats may make KSON a favorable option.

Q: Is KSON faster/smaller/better than JSON?
A: Tests using node.js indicate that parsing is marginally slower. This
   should be outweighed by faster transmission speed of signifigantly less
   data. See the node.js benchmarks for more info.

Q: Is this a drop in replacement for JSON?
A: No, you will need to write schema definitions for your data. Automatic
   schema detection based on example data may help in getting started. Also
   see "restrictions" in the Specification.

Q: Why yet another a schema format for JSON?
A: Existing JSON schema formats include extrenious information to the
   purpose of KSON and would require larger javascript library for
   parsing and serialization.

Q: Is there language support for php/ruby/java/c...?
A: Currently javascript and python are supported. Porting to other
   languages with existing JSON support should be fairly easy and patches
   are very welcome.


Schema Specification
--------------------

A KSON schema consists of three fields:
    id              *string* identifier
    fields          *array* of field names
    meta            *array* signifying the types associated with
                    corresponding elements in the *fields* array.

A schema only defines one object. Nesting is accomplished by referencing
subschemas via their id in the meta array.

An element of *meta* specifies the content of the element of *fields* at
the same index. The lengths of the *fields* and *meta* arrays must be equal.
An element of *meta* must be one of the following:
    0                field contains a plain value of type
                     *string*, *boolean*, *number*, *null*.
    "[]"             *array* of plain values
    "schema-id"      *string* reference the schema of a nested object
    "[]schema-id"    
    "codec-id"
    "[]codec-id"


The top level elements of a KSON document must be either an *array* of
objects or an *object*. A plain value or an *array* of plain values
cannot be serialized.


Codecs
------

Decoders are executed in the order of their definition in the meta field.
The decoder is passed the value to decode as the first argument and the
intermediate data object as the second. The intermediate object will have
the values of all previous fields decoded and applied.


Included codecs:

  enum:a:b:c:d   "a" -> 0, "b" -> 1, ...
  prefix:egg     "eggbacon" -> "bacon"
  suffix:bacon   "eggsausagebacon" -> "eggsausage"
  bool           true -> 1, false -> 0
  date           new Date() -> 1364938727390
  int36          1364938727390 -> "hf1lcnka"


Codec chaining

  prefix:egg|suffix:bacon    "eggsausagebacon" -> "sausage"
  date|int36    new Date() -> "hf1lcnka"


Custom codecs are registered using factory functions in the following manner.

    KSON.coders["my_coder_id"] = function(args) {
        // args is a string array from a schemas meta field
        // enum:a:b:c:d -> args == ["a", "b", "c", "d"]

        return function (val, encode) {
            // The actual encoding/decoding function
            return encode ? encode_op(val) : decode_op(val);
        }
    }

Schema Validation
-----------------

The python library validates schemas and codec definitions, the javascript
library does not. Invalid schema definition loaded with the javascript
library may parse or stringify bad data without any throwing any errors.

Common errors include:
    - Loading a schema before the custom codecs it uses have been registered.
    - Loading a schema before the subschemas on which it depends have been
      loaded. Note that schemas are loaded in the order they appear in a file,
      so dependent schmas schould come after those they depend uppon.
    - Circularly dependent schmas (including schemas which depend on 
      themselves) are possible by using a stub schema, which only specifies
      an id and no fields. This can be replaced by the propper schema with
      the same id, after its dependencies have been loaded.
    - Loading a json encoded schema. The argument of the KSON.addSchema
      function may be either a KSON encoded schema or schemas, an already
      decoded schema or an array of schemas. It is however fairly simple to
      load json encoded schemas: KSON.addSchema(JSON.parse(raw_json_schemas));

The python library will warn about these and other issues, so use it to


Installation
------------

    $ pip install kson

Automatic schema detection:
  
    $ kson --auto-detect data.json --id schema_id


Development
-----------

    $ git clone git@bitbucket.org:mbarkhau/kson.git
    $ npm install uglify-js -g
    $ npm install benchmark microtime

    $ pip install py.test

    $ make test
    $ make bench
    $ make minify