Repository: apache/arrow
File: python/pyarrow/parquet/core.py (Python)

Code before:
    before any Parquet encodings).

    A `dict` can be passed to adjust the chunker parameters with the following keys:
    - `min_chunk_size`: minimum chunk size in bytes, default 256 KiB
      The rolling hash will not be updated until this size is reached for each chunk.
      Note that all data sent through the hash function is counted towards the chunk
      size, including definition and repetition levels if present.
    - `max_chunk_size`: maximum chunk size in bytes, default is 1024 KiB
      The chunker will create a new chunk whenever the chunk size exceeds this value.
      Note that the parquet writer has a related `data_pagesize` property that controls
      the maximum size of a parquet data page after encoding. While setting
      `data_page_size` to a smaller value than `max_chunk_size` doesn't affect the
      chunking effectiveness, it results in more small parquet data pages.
    - `norm_level`: normalization level to center the chunk size around the average
      size more aggressively, default 0
      Increasing the normalization level increases the probability of finding a chunk,
      improving the deduplication ratio, but also increasing the number of small chunks
      resulting in many small parquet data pages. The default value provides a good
      balance between deduplication ratio and fragmentation. Use norm_level=1 or
      norm_level=2 to reach a higher deduplication ratio at the expense of
      fragmentation.
write_time_adjusted_to_utc : bool, default False
    Specify whether the TIME columns are expressed in reference
    to midnight in the UTC timezone if True, otherwise it is not tied to UTC,
    and is expressed in reference to midnight in an unknown,
    presumably local, timezone.
"""

_parquet_writer_example_doc = """\
Generate an example PyArrow Table and RecordBatch:

>>> import pyarrow as pa
>>> table = pa.table({'n_legs': [2, 2, 4, 4, 5, 100],
...                   'animal': ["Flamingo", "Parrot", "Dog", "Horse",
...                              "Brittle stars", "Centipede"]})
>>> batch = pa.record_batch([[2, 2, 4, 4, 5, 100],
...                         ["Flamingo", "Parrot", "Dog", "Horse",
...                          "Brittle stars", "Centipede"]],
...                         names=['n_legs', 'animal'])

create a ParquetWriter object:

>>> import pyarrow.parquet as pq
>>> writer = pq.ParquetWriter('example.parquet', table.schema)

and write the Table into the Parquet file:

>>> writer.write_table(table)
>>> writer.close()

>>> pq.read_table('example.parquet').to_pandas()

Review comment (suggestion):
```suggestion
    Set the value of isAdjustedTOUTC when writing a TIME column.
    If True, this tells the Parquet reader that the TIME columns
    are expressed in reference to midnight in the UTC timezone.
    If False (the default), the TIME columns are assumed to be expressed
    in reference to midnight in an unknown, presumably local, timezone.
```

Code after:
    before any Parquet encodings).

    A `dict` can be passed to adjust the chunker parameters with the following keys:
    - `min_chunk_size`: minimum chunk size in bytes, default 256 KiB
      The rolling hash will not be updated until this size is reached for each chunk.
      Note that all data sent through the hash function is counted towards the chunk
      size, including definition and repetition levels if present.
    - `max_chunk_size`: maximum chunk size in bytes, default is 1024 KiB
      The chunker will create a new chunk whenever the chunk size exceeds this value.
      Note that the parquet writer has a related `data_pagesize` property that controls
      the maximum size of a parquet data page after encoding. While setting
      `data_page_size` to a smaller value than `max_chunk_size` doesn't affect the
      chunking effectiveness, it results in more small parquet data pages.
    - `norm_level`: normalization level to center the chunk size around the average
      size more aggressively, default 0
      Increasing the normalization level increases the probability of finding a chunk,
      improving the deduplication ratio, but also increasing the number of small chunks
      resulting in many small parquet data pages. The default value provides a good
      balance between deduplication ratio and fragmentation. Use norm_level=1 or
      norm_level=2 to reach a higher deduplication ratio at the expense of
      fragmentation.
write_time_adjusted_to_utc : bool, default False
    Set the value of isAdjustedTOUTC when writing a TIME column.
    If True, this tells the Parquet reader that the TIME columns
    are expressed in reference to midnight in the UTC timezone.
    If False (the default), the TIME columns are assumed to be expressed
    in reference to midnight in an unknown, presumably local, timezone.
"""

_parquet_writer_example_doc = """\
Generate an example PyArrow Table and RecordBatch:

>>> import pyarrow as pa
>>> table = pa.table({'n_legs': [2, 2, 4, 4, 5, 100],
...                   'animal': ["Flamingo", "Parrot", "Dog", "Horse",
...                              "Brittle stars", "Centipede"]})
>>> batch = pa.record_batch([[2, 2, 4, 4, 5, 100],
...                         ["Flamingo", "Parrot", "Dog", "Horse",
...                          "Brittle stars", "Centipede"]],
...                         names=['n_legs', 'animal'])

create a ParquetWriter object:

>>> import pyarrow.parquet as pq
>>> writer = pq.ParquetWriter('example.parquet', table.schema)

and write the Table into the Parquet file:

>>> writer.write_table(table)
>>> writer.close()