Reading Parquet Files in Python with rows
Autor: Álvaro Justen "Turicas" | Publicado: 2016-03-14T01:48:00-03:00

Many people in the data science field use the parquet
format to store tabular data, as it&#39;s the default format used
by Apache Spark -- an efficient data storage format for
analytics. The problem is: the format is binary (you can&#39;t just open it with
your preferred code editor) and there&#39;s no such a good Python library to read
-- not until today!
I found a Python library called parquet-python on GitHub
but it&#39;s hard to use, doesn&#39;t have one code example, was not available on
PyPI and it looks like it&#39;s not maintained anymore. So I decided to
implement a parquet plugin (read-only) for my library
rows: it uses the parquet-python
library under the hood (I needed to upload it to PyPI so you
can install it easly) and exposes the data in a pretty simple, pythonic way.
Installation
I didn&#39;t realese the rows version with this plugin yet, so you need to grab the
most recent rows version by running:
pip install -U git+https://github.com/turicas/rows.git@develop
And also the dependency:
pip install parquet
If the data is compressed using Google&#39;s snappy you&#39;ll also
need the library headers and other Python dependency -- install everything by
running:
apt-get install libsnappy-dev
pip install python-snappy
Then you can use rows.import_from_parquet(filename) in your programs! \o/
Python Example
A quick Python code example:
import rows

table = rows.import_from_parquet(&#39;myfile.parquet&#39;)
for row in table:
    print row  # access fields values with `rows.field_name`

Note that the current implementation is not optimized (for example, it&#39;ll
put everything into memory) but at least you can extract desired data and then
convert to a more friendly format easily.
Converting Parquet to Other Formats with rows&#39; CLI
You can convert Parquet files to many tabular formats (like CSV) by using
the rows&#39;s command-line interface, so you don&#39;t need to code.
Install the rows CLI by running:
pip install rows[cli]
Now convert the parquet file:
rows convert myfile.parquet myfile.csv  # yes, simple like this!
You can replace csv with any other supported format (the list is always
growing!), such as: txt, html, xls, xlsx and sqlite.
If your file is small enough you can actually see it without needing to save
the output to another file by using the print subcommand:
rows print myfile.parquet  # will extract, convert and print data as text
And you can actually query data as in SQL (this CLI is awesome!), for example:
rows query &#39;nation_key &lt; 10&#39; tests/data/nation.dict.parquet \
     --output=data.csv
By running this command the CLI will:

Import data from tests/data/nation.dict.parquet file into memory;
Export to SQLite (:memory:);
Run the query (nation_key &lt; 10) and get the results;
Convert the results to a new rows.Table object;
Export the table to CSV format and save it into data.csv (the result format
could be html, xls, xlsx or any other write-plugin supported by
rows).

With this addition to rows I think the library and its command-line
interface became one of the tools every data scientist should have installed.
;-)