Table Classes

Each datasets.Dataset object is backed by a PyArrow Table. A Table can be loaded from either the disk (memory mapped) or in memory. Several Table types are available, and they all inherit from datasets.table.Table.

class datasets.table.Table(table: pyarrow.lib.Table)

Wraps a pyarrow Table by using composition. This is the base class for InMemoryTable, MemoryMappedTable and ConcatenationTable.

It implements all the basic attributes/methods of the pyarrow Table class except the Table transforms: slice, filter, flatten, combine_chunks, cast, add_column, append_column, remove_column, set_column, rename_columns and drop.

The implementation of these methods differs for the subclasses.

column(self, i)

Select a column by its column name, or numeric index.

Parameters

i (int or string) – The index or name of the column to retrieve.

Returns

ChunkedArray

equals(self, Table other, bool check_metadata=False)

Check if contents of two tables are equal.

Parameters

• other (Table) – Table to compare against.

• check_metadata (bool, default False) – Whether schema metadata equality should be checked as well.

Returns

bool

field(self, i)

Select a schema field by its column name or numeric index.

Parameters

i (int or string) – The index or name of the field to retrieve.

Returns

Field

itercolumns(self)

Iterator over all columns in their numerical order.

Yields

ChunkedArray

to_batches(self, max_chunksize=None, **kwargs)

Convert Table to list of (contiguous) RecordBatch objects.

Parameters

max_chunksize (int, default None) – Maximum size for RecordBatch chunks. Individual chunks may be smaller depending on the chunk layout of individual columns.

Returns

list of RecordBatch

to_pandas(self, memory_pool=None, categories=None, bool strings_to_categorical=False, bool zero_copy_only=False, bool integer_object_nulls=False, bool date_as_object=True, bool timestamp_as_object=False, bool use_threads=True, bool deduplicate_objects=True, bool ignore_metadata=False, bool safe=True, bool split_blocks=False, bool self_destruct=False, types_mapper=None)

Convert to a pandas-compatible NumPy array or DataFrame, as appropriate

Parameters

• memory_pool (MemoryPool, default None) – Arrow MemoryPool to use for allocations. Uses the default memory pool is not passed.

• strings_to_categorical (bool, default False) – Encode string (UTF8) and binary types to pandas.Categorical.

• categories (list, default empty) – List of fields that should be returned as pandas.Categorical. Only applies to table-like data structures.

• zero_copy_only (bool, default False) – Raise an ArrowException if this function call would require copying the underlying data.

• integer_object_nulls (bool, default False) – Cast integers with nulls to objects

• date_as_object (bool, default True) – Cast dates to objects. If False, convert to datetime64[ns] dtype.

• timestamp_as_object (bool, default False) – Cast non-nanosecond timestamps (np.datetime64) to objects. This is useful if you have timestamps that don’t fit in the normal date range of nanosecond timestamps (1678 CE-2262 CE). If False, all timestamps are converted to datetime64[ns] dtype.

• use_threads (bool, default True) – Whether to parallelize the conversion using multiple threads.

• deduplicate_objects (bool, default False) – Do not create multiple copies Python objects when created, to save on memory use. Conversion will be slower.

• ignore_metadata (bool, default False) – If True, do not use the ‘pandas’ metadata to reconstruct the DataFrame index, if present

• safe (bool, default True) – For certain data types, a cast is needed in order to store the data in a pandas DataFrame or Series (e.g. timestamps are always stored as nanoseconds in pandas). This option controls whether it is a safe cast or not.

• split_blocks (bool, default False) – If True, generate one internal “block” for each column when creating a pandas.DataFrame from a RecordBatch or Table. While this can temporarily reduce memory note that various pandas operations can trigger “consolidation” which may balloon memory use.

• self_destruct (bool, default False) –

  EXPERIMENTAL: If True, attempt to deallocate the originating Arrow memory while converting the Arrow object to pandas. If you use the object after calling to_pandas with this option it will crash your program.

  Note that you may not see always memory usage improvements. For example, if multiple columns share an underlying allocation, memory can’t be freed until all columns are converted.

• types_mapper (function, default None) – A function mapping a pyarrow DataType to a pandas ExtensionDtype. This can be used to override the default pandas type for conversion of built-in pyarrow types or in absence of pandas_metadata in the Table schema. The function receives a pyarrow DataType and is expected to return a pandas ExtensionDtype or None if the default conversion should be used for that type. If you have a dictionary mapping, you can pass dict.get as function.

Returns

pandas.Series or pandas.DataFrame depending on type of object

to_pydict(self)

Convert the Table to a dict or OrderedDict.

Returns

dict

validate(self, *, full=False)

Perform validation checks. An exception is raised if validation fails.

By default only cheap validation checks are run. Pass full=True for thorough validation checks (potentially O(n)).

Parameters

full (bool, default False) – If True, run expensive checks, otherwise cheap checks only.

Raises

ArrowInvalid –

class datasets.table.InMemoryTable(table: pyarrow.lib.Table)

The table is said in-memory when it is loaded into the user’s RAM.

Pickling it does copy all the data using memory. Its implementation is simple and uses the underlying pyarrow Table methods directly.

This is different from the MemoryMapped table, for which pickling doesn’t copy all the data in memory. For a MemoryMapped, unpickling instead reloads the table from the disk.

InMemoryTable must be used when data fit in memory, while MemoryMapped are reserved for data bigger than memory or when you want the memory footprint of your application to stay low.

add_column(self, int i, field_, column)

Add column to Table at position.

A new table is returned with the column added, the original table object is left unchanged.

Parameters

• i (int) – Index to place the column at.

• field (str or Field) – If a string is passed then the type is deduced from the column data.

• column (Array, list of Array, or values coercible to arrays) – Column data.

Returns

Table – New table with the passed column added.

append_column(self, field_, column)

Append column at end of columns.

Parameters

• field (str or Field) – If a string is passed then the type is deduced from the column data.

• column (Array, list of Array, or values coercible to arrays) – Column data.

Returns

Table – New table with the passed column added.

cast(self, Schema target_schema, bool safe=True)

Cast table values to another schema.

Parameters

• target_schema (Schema) – Schema to cast to, the names and order of fields must match.

• safe (bool, default True) – Check for overflows or other unsafe conversions.

Returns

Table

column(self, i)

Select a column by its column name, or numeric index.

Parameters

i (int or string) – The index or name of the column to retrieve.

Returns

ChunkedArray

combine_chunks(self, MemoryPool memory_pool=None)

Make a new table by combining the chunks this table has.

All the underlying chunks in the ChunkedArray of each column are concatenated into zero or one chunk.

Parameters

memory_pool (MemoryPool, default None) – For memory allocations, if required, otherwise use default pool.

Returns

Table

drop(self, columns)

Drop one or more columns and return a new table.

Parameters

columns (list of str) – List of field names referencing existing columns.

Raises

KeyError – If any of the passed columns name are not existing.

Returns

Table – New table without the columns.

equals(self, Table other, bool check_metadata=False)

Check if contents of two tables are equal.

Parameters

• other (Table) – Table to compare against.

• check_metadata (bool, default False) – Whether schema metadata equality should be checked as well.

Returns

bool

field(self, i)

Select a schema field by its column name or numeric index.

Parameters

i (int or string) – The index or name of the field to retrieve.

Returns

Field

filter(self, mask, null_selection_behavior='drop')

Select records from a Table. See pyarrow.compute.filter() for full usage.

flatten(self, MemoryPool memory_pool=None)

Flatten this Table.

Each column with a struct type is flattened into one column per struct field. Other columns are left unchanged.

Parameters

memory_pool (MemoryPool, default None) – For memory allocations, if required, otherwise use default pool

Returns

Table

itercolumns(self)

Iterator over all columns in their numerical order.

Yields

ChunkedArray

remove_column(self, int i)

Create new Table with the indicated column removed.

Parameters

i (int) – Index of column to remove.

Returns

Table – New table without the column.

rename_columns(self, names)

Create new table with columns renamed to provided names.

Parameters

names (list of str) – List of new column names.

Returns

Table

replace_schema_metadata(self, metadata=None)

Create shallow copy of table by replacing schema key-value metadata with the indicated new metadata (which may be None), which deletes any existing metadata.

Parameters

metadata (dict, default None) –

Returns

Table

set_column(self, int i, field_, column)

Replace column in Table at position.

Parameters

• i (int) – Index to place the column at.

• field (str or Field) – If a string is passed then the type is deduced from the column data.

• column (Array, list of Array, or values coercible to arrays) – Column data.

Returns

Table – New table with the passed column set.

slice(self, offset=0, length=None)

Compute zero-copy slice of this Table.

Parameters

• offset (int, default 0) – Offset from start of table to slice.

• length (int, default None) – Length of slice (default is until end of table starting from offset).

Returns

Table

to_batches(self, max_chunksize=None, **kwargs)

Convert Table to list of (contiguous) RecordBatch objects.

Parameters

max_chunksize (int, default None) – Maximum size for RecordBatch chunks. Individual chunks may be smaller depending on the chunk layout of individual columns.

Returns

list of RecordBatch

to_pandas(self, memory_pool=None, categories=None, bool strings_to_categorical=False, bool zero_copy_only=False, bool integer_object_nulls=False, bool date_as_object=True, bool timestamp_as_object=False, bool use_threads=True, bool deduplicate_objects=True, bool ignore_metadata=False, bool safe=True, bool split_blocks=False, bool self_destruct=False, types_mapper=None)

Convert to a pandas-compatible NumPy array or DataFrame, as appropriate

Parameters

• memory_pool (MemoryPool, default None) – Arrow MemoryPool to use for allocations. Uses the default memory pool is not passed.

• strings_to_categorical (bool, default False) – Encode string (UTF8) and binary types to pandas.Categorical.

• categories (list, default empty) – List of fields that should be returned as pandas.Categorical. Only applies to table-like data structures.

• zero_copy_only (bool, default False) – Raise an ArrowException if this function call would require copying the underlying data.

• integer_object_nulls (bool, default False) – Cast integers with nulls to objects

• date_as_object (bool, default True) – Cast dates to objects. If False, convert to datetime64[ns] dtype.

• timestamp_as_object (bool, default False) – Cast non-nanosecond timestamps (np.datetime64) to objects. This is useful if you have timestamps that don’t fit in the normal date range of nanosecond timestamps (1678 CE-2262 CE). If False, all timestamps are converted to datetime64[ns] dtype.

• use_threads (bool, default True) – Whether to parallelize the conversion using multiple threads.

• deduplicate_objects (bool, default False) – Do not create multiple copies Python objects when created, to save on memory use. Conversion will be slower.

• ignore_metadata (bool, default False) – If True, do not use the ‘pandas’ metadata to reconstruct the DataFrame index, if present

• safe (bool, default True) – For certain data types, a cast is needed in order to store the data in a pandas DataFrame or Series (e.g. timestamps are always stored as nanoseconds in pandas). This option controls whether it is a safe cast or not.

• split_blocks (bool, default False) – If True, generate one internal “block” for each column when creating a pandas.DataFrame from a RecordBatch or Table. While this can temporarily reduce memory note that various pandas operations can trigger “consolidation” which may balloon memory use.

• self_destruct (bool, default False) –

  EXPERIMENTAL: If True, attempt to deallocate the originating Arrow memory while converting the Arrow object to pandas. If you use the object after calling to_pandas with this option it will crash your program.

  Note that you may not see always memory usage improvements. For example, if multiple columns share an underlying allocation, memory can’t be freed until all columns are converted.

• types_mapper (function, default None) – A function mapping a pyarrow DataType to a pandas ExtensionDtype. This can be used to override the default pandas type for conversion of built-in pyarrow types or in absence of pandas_metadata in the Table schema. The function receives a pyarrow DataType and is expected to return a pandas ExtensionDtype or None if the default conversion should be used for that type. If you have a dictionary mapping, you can pass dict.get as function.

Returns

pandas.Series or pandas.DataFrame depending on type of object

to_pydict(self)

Convert the Table to a dict or OrderedDict.

Returns

dict

validate(self, *, full=False)

Perform validation checks. An exception is raised if validation fails.

By default only cheap validation checks are run. Pass full=True for thorough validation checks (potentially O(n)).

Parameters

full (bool, default False) – If True, run expensive checks, otherwise cheap checks only.

Raises

ArrowInvalid –

class datasets.table.MemoryMappedTable(table: pyarrow.lib.Table, path: str, replays: Optional[List[Tuple[str, tuple, dict]]] = None)

The table is said memory mapped when it doesn’t use the user’s RAM but loads the data from the disk instead.

Pickling it doesn’t copy the data into memory. Instead, only the path to the memory mapped arrow file is pickled, as well as the list of transforms to “replay” when reloading the table from the disk.

Its implementation requires to store an history of all the transforms that were applied to the underlying pyarrow Table, so that they can be “replayed” when reloading the Table from the disk.

This is different from the InMemoryTable table, for which pickling does copy all the data in memory.

InMemoryTable must be used when data fit in memory, while MemoryMapped are reserved for data bigger than memory or when you want the memory footprint of your application to stay low.

add_column(self, int i, field_, column)

Add column to Table at position.

A new table is returned with the column added, the original table object is left unchanged.

Parameters

• i (int) – Index to place the column at.

• field (str or Field) – If a string is passed then the type is deduced from the column data.

• column (Array, list of Array, or values coercible to arrays) – Column data.

Returns

Table – New table with the passed column added.

append_column(self, field_, column)

Append column at end of columns.

Parameters

• field (str or Field) – If a string is passed then the type is deduced from the column data.

• column (Array, list of Array, or values coercible to arrays) – Column data.

Returns

Table – New table with the passed column added.

cast(self, Schema target_schema, bool safe=True)

Cast table values to another schema.

Parameters

• target_schema (Schema) – Schema to cast to, the names and order of fields must match.

• safe (bool, default True) – Check for overflows or other unsafe conversions.

Returns

Table

column(self, i)

Select a column by its column name, or numeric index.

Parameters

i (int or string) – The index or name of the column to retrieve.

Returns

ChunkedArray

combine_chunks(self, MemoryPool memory_pool=None)

Make a new table by combining the chunks this table has.

All the underlying chunks in the ChunkedArray of each column are concatenated into zero or one chunk.

Parameters

memory_pool (MemoryPool, default None) – For memory allocations, if required, otherwise use default pool.

Returns

Table

drop(self, columns)

Drop one or more columns and return a new table.

Parameters

columns (list of str) – List of field names referencing existing columns.

Raises

KeyError – If any of the passed columns name are not existing.

Returns

Table – New table without the columns.

equals(self, Table other, bool check_metadata=False)

Check if contents of two tables are equal.

Parameters

• other (Table) – Table to compare against.

• check_metadata (bool, default False) – Whether schema metadata equality should be checked as well.

Returns

bool

field(self, i)

Select a schema field by its column name or numeric index.

Parameters

i (int or string) – The index or name of the field to retrieve.

Returns

Field

filter(self, mask, null_selection_behavior='drop')

Select records from a Table. See pyarrow.compute.filter() for full usage.

flatten(self, MemoryPool memory_pool=None)

Flatten this Table.

Each column with a struct type is flattened into one column per struct field. Other columns are left unchanged.

Parameters

memory_pool (MemoryPool, default None) – For memory allocations, if required, otherwise use default pool

Returns

Table

itercolumns(self)

Iterator over all columns in their numerical order.

Yields

ChunkedArray

remove_column(self, int i)

Create new Table with the indicated column removed.

Parameters

i (int) – Index of column to remove.

Returns

Table – New table without the column.

rename_columns(self, names)

Create new table with columns renamed to provided names.

Parameters

names (list of str) – List of new column names.

Returns

Table

replace_schema_metadata(self, metadata=None)

Create shallow copy of table by replacing schema key-value metadata with the indicated new metadata (which may be None), which deletes any existing metadata.

Parameters

metadata (dict, default None) –

Returns

Table

set_column(self, int i, field_, column)

Replace column in Table at position.

Parameters

• i (int) – Index to place the column at.

• field (str or Field) – If a string is passed then the type is deduced from the column data.

• column (Array, list of Array, or values coercible to arrays) – Column data.

Returns

Table – New table with the passed column set.

slice(self, offset=0, length=None)

Compute zero-copy slice of this Table.

Parameters

• offset (int, default 0) – Offset from start of table to slice.

• length (int, default None) – Length of slice (default is until end of table starting from offset).

Returns

Table

to_batches(self, max_chunksize=None, **kwargs)

Convert Table to list of (contiguous) RecordBatch objects.

Parameters

max_chunksize (int, default None) – Maximum size for RecordBatch chunks. Individual chunks may be smaller depending on the chunk layout of individual columns.

Returns

list of RecordBatch

to_pandas(self, memory_pool=None, categories=None, bool strings_to_categorical=False, bool zero_copy_only=False, bool integer_object_nulls=False, bool date_as_object=True, bool timestamp_as_object=False, bool use_threads=True, bool deduplicate_objects=True, bool ignore_metadata=False, bool safe=True, bool split_blocks=False, bool self_destruct=False, types_mapper=None)

Convert to a pandas-compatible NumPy array or DataFrame, as appropriate

Parameters

• memory_pool (MemoryPool, default None) – Arrow MemoryPool to use for allocations. Uses the default memory pool is not passed.

• strings_to_categorical (bool, default False) – Encode string (UTF8) and binary types to pandas.Categorical.

• categories (list, default empty) – List of fields that should be returned as pandas.Categorical. Only applies to table-like data structures.

• zero_copy_only (bool, default False) – Raise an ArrowException if this function call would require copying the underlying data.

• integer_object_nulls (bool, default False) – Cast integers with nulls to objects

• date_as_object (bool, default True) – Cast dates to objects. If False, convert to datetime64[ns] dtype.

• timestamp_as_object (bool, default False) – Cast non-nanosecond timestamps (np.datetime64) to objects. This is useful if you have timestamps that don’t fit in the normal date range of nanosecond timestamps (1678 CE-2262 CE). If False, all timestamps are converted to datetime64[ns] dtype.

• use_threads (bool, default True) – Whether to parallelize the conversion using multiple threads.

• deduplicate_objects (bool, default False) – Do not create multiple copies Python objects when created, to save on memory use. Conversion will be slower.

• ignore_metadata (bool, default False) – If True, do not use the ‘pandas’ metadata to reconstruct the DataFrame index, if present

• safe (bool, default True) – For certain data types, a cast is needed in order to store the data in a pandas DataFrame or Series (e.g. timestamps are always stored as nanoseconds in pandas). This option controls whether it is a safe cast or not.

• split_blocks (bool, default False) – If True, generate one internal “block” for each column when creating a pandas.DataFrame from a RecordBatch or Table. While this can temporarily reduce memory note that various pandas operations can trigger “consolidation” which may balloon memory use.

• self_destruct (bool, default False) –

  EXPERIMENTAL: If True, attempt to deallocate the originating Arrow memory while converting the Arrow object to pandas. If you use the object after calling to_pandas with this option it will crash your program.

  Note that you may not see always memory usage improvements. For example, if multiple columns share an underlying allocation, memory can’t be freed until all columns are converted.

• types_mapper (function, default None) – A function mapping a pyarrow DataType to a pandas ExtensionDtype. This can be used to override the default pandas type for conversion of built-in pyarrow types or in absence of pandas_metadata in the Table schema. The function receives a pyarrow DataType and is expected to return a pandas ExtensionDtype or None if the default conversion should be used for that type. If you have a dictionary mapping, you can pass dict.get as function.

Returns

pandas.Series or pandas.DataFrame depending on type of object

to_pydict(self)

Convert the Table to a dict or OrderedDict.

Returns

dict

validate(self, *, full=False)

Perform validation checks. An exception is raised if validation fails.

By default only cheap validation checks are run. Pass full=True for thorough validation checks (potentially O(n)).

Parameters

full (bool, default False) – If True, run expensive checks, otherwise cheap checks only.

Raises

ArrowInvalid –

class datasets.table.ConcatenationTable(table: pyarrow.lib.Table, blocks: List[List[datasets.table.TableBlock]])

The table comes from the concatenation of several tables called blocks. It enables concatenation on both axis 0 (append rows) and axis 1 (append columns).

The underlying tables are called “blocks” and can be either InMemoryTable or MemoryMappedTable objects. This allows to combine tables that come from memory or that are memory mapped. When a ConcatenationTable is pickled, then each block is pickled: - the InMemoryTable objects are pickled by copying all the data in memory; - the MemoryMappedTable objects are pickled without copying the data into memory. Instead, only the path to the memory mapped arrow file is pickled, as well as the list of transforms to “replays” when reloading the table from the disk.

Its implementation requires to store each block separately. The blocks attributes stores a list of list of blocks. The first axis concatenates the tables along the axis 0 (it appends rows), while the second axis concatenates tables along the axis 1 (it appends columns).

You can access the fully combined table by accessing the ConcatenationTable.table attribute, and the blocks by accessing the ConcatenationTable.blocks attribute.

add_column(self, int i, field_, column)

Add column to Table at position.

A new table is returned with the column added, the original table object is left unchanged.

Parameters

• i (int) – Index to place the column at.

• field (str or Field) – If a string is passed then the type is deduced from the column data.

• column (Array, list of Array, or values coercible to arrays) – Column data.

Returns

Table – New table with the passed column added.

append_column(self, field_, column)

Append column at end of columns.

Parameters

• field (str or Field) – If a string is passed then the type is deduced from the column data.

• column (Array, list of Array, or values coercible to arrays) – Column data.

Returns

Table – New table with the passed column added.

cast(self, Schema target_schema, bool safe=True)

Cast table values to another schema.

Parameters

• target_schema (Schema) – Schema to cast to, the names and order of fields must match.

• safe (bool, default True) – Check for overflows or other unsafe conversions.

Returns

Table

column(self, i)

Select a column by its column name, or numeric index.

Parameters

i (int or string) – The index or name of the column to retrieve.

Returns

ChunkedArray

combine_chunks(self, MemoryPool memory_pool=None)

Make a new table by combining the chunks this table has.

All the underlying chunks in the ChunkedArray of each column are concatenated into zero or one chunk.

Parameters

memory_pool (MemoryPool, default None) – For memory allocations, if required, otherwise use default pool.

Returns

Table

drop(self, columns)

Drop one or more columns and return a new table.

Parameters

columns (list of str) – List of field names referencing existing columns.

Raises

KeyError – If any of the passed columns name are not existing.

Returns

Table – New table without the columns.

equals(self, Table other, bool check_metadata=False)

Check if contents of two tables are equal.

Parameters

• other (Table) – Table to compare against.

• check_metadata (bool, default False) – Whether schema metadata equality should be checked as well.

Returns

bool

field(self, i)

Select a schema field by its column name or numeric index.

Parameters

i (int or string) – The index or name of the field to retrieve.

Returns

Field

filter(self, mask, null_selection_behavior='drop')

Select records from a Table. See pyarrow.compute.filter() for full usage.

flatten(self, MemoryPool memory_pool=None)

Flatten this Table.

Each column with a struct type is flattened into one column per struct field. Other columns are left unchanged.

Parameters

memory_pool (MemoryPool, default None) – For memory allocations, if required, otherwise use default pool

Returns

Table

classmethod from_tables(tables: List[Union[pyarrow.lib.Table, datasets.table.Table]], axis: int = 0) → datasets.table.ConcatenationTable

Create ConcatenationTable from list of tables.

Parameters

• tables (list of Table or list of pyarrow.Table) – List of tables.

• axis – ({0, 1}, default 0, meaning over rows):

• to concatenate over, where 0 means over rows (Axis) –

• (horizontally) –

• versionadded: (.) – 1.6.0:

itercolumns(self)

Iterator over all columns in their numerical order.

Yields

ChunkedArray

remove_column(self, int i)

Create new Table with the indicated column removed.

Parameters

i (int) – Index of column to remove.

Returns

Table – New table without the column.

rename_columns(self, names)

Create new table with columns renamed to provided names.

Parameters

names (list of str) – List of new column names.

Returns

Table

replace_schema_metadata(self, metadata=None)

Create shallow copy of table by replacing schema key-value metadata with the indicated new metadata (which may be None), which deletes any existing metadata.

Parameters

metadata (dict, default None) –

Returns

Table

set_column(self, int i, field_, column)

Replace column in Table at position.

Parameters

• i (int) – Index to place the column at.

• field (str or Field) – If a string is passed then the type is deduced from the column data.

• column (Array, list of Array, or values coercible to arrays) – Column data.

Returns

Table – New table with the passed column set.

slice(self, offset=0, length=None)

Compute zero-copy slice of this Table.

Parameters

• offset (int, default 0) – Offset from start of table to slice.

• length (int, default None) – Length of slice (default is until end of table starting from offset).

Returns

Table

to_batches(self, max_chunksize=None, **kwargs)

Convert Table to list of (contiguous) RecordBatch objects.

Parameters

max_chunksize (int, default None) – Maximum size for RecordBatch chunks. Individual chunks may be smaller depending on the chunk layout of individual columns.

Returns

list of RecordBatch

to_pandas(self, memory_pool=None, categories=None, bool strings_to_categorical=False, bool zero_copy_only=False, bool integer_object_nulls=False, bool date_as_object=True, bool timestamp_as_object=False, bool use_threads=True, bool deduplicate_objects=True, bool ignore_metadata=False, bool safe=True, bool split_blocks=False, bool self_destruct=False, types_mapper=None)

Convert to a pandas-compatible NumPy array or DataFrame, as appropriate

Parameters

• memory_pool (MemoryPool, default None) – Arrow MemoryPool to use for allocations. Uses the default memory pool is not passed.

• strings_to_categorical (bool, default False) – Encode string (UTF8) and binary types to pandas.Categorical.

• categories (list, default empty) – List of fields that should be returned as pandas.Categorical. Only applies to table-like data structures.

• zero_copy_only (bool, default False) – Raise an ArrowException if this function call would require copying the underlying data.

• integer_object_nulls (bool, default False) – Cast integers with nulls to objects

• date_as_object (bool, default True) – Cast dates to objects. If False, convert to datetime64[ns] dtype.

• timestamp_as_object (bool, default False) – Cast non-nanosecond timestamps (np.datetime64) to objects. This is useful if you have timestamps that don’t fit in the normal date range of nanosecond timestamps (1678 CE-2262 CE). If False, all timestamps are converted to datetime64[ns] dtype.

• use_threads (bool, default True) – Whether to parallelize the conversion using multiple threads.

• deduplicate_objects (bool, default False) – Do not create multiple copies Python objects when created, to save on memory use. Conversion will be slower.

• ignore_metadata (bool, default False) – If True, do not use the ‘pandas’ metadata to reconstruct the DataFrame index, if present

• safe (bool, default True) – For certain data types, a cast is needed in order to store the data in a pandas DataFrame or Series (e.g. timestamps are always stored as nanoseconds in pandas). This option controls whether it is a safe cast or not.

• split_blocks (bool, default False) – If True, generate one internal “block” for each column when creating a pandas.DataFrame from a RecordBatch or Table. While this can temporarily reduce memory note that various pandas operations can trigger “consolidation” which may balloon memory use.

• self_destruct (bool, default False) –

  EXPERIMENTAL: If True, attempt to deallocate the originating Arrow memory while converting the Arrow object to pandas. If you use the object after calling to_pandas with this option it will crash your program.

  Note that you may not see always memory usage improvements. For example, if multiple columns share an underlying allocation, memory can’t be freed until all columns are converted.

• types_mapper (function, default None) – A function mapping a pyarrow DataType to a pandas ExtensionDtype. This can be used to override the default pandas type for conversion of built-in pyarrow types or in absence of pandas_metadata in the Table schema. The function receives a pyarrow DataType and is expected to return a pandas ExtensionDtype or None if the default conversion should be used for that type. If you have a dictionary mapping, you can pass dict.get as function.

Returns

pandas.Series or pandas.DataFrame depending on type of object

to_pydict(self)

Convert the Table to a dict or OrderedDict.

Returns

dict

validate(self, *, full=False)

Perform validation checks. An exception is raised if validation fails.

By default only cheap validation checks are run. Pass full=True for thorough validation checks (potentially O(n)).

Parameters

full (bool, default False) – If True, run expensive checks, otherwise cheap checks only.

Raises

ArrowInvalid –

datasets.table.concat_tables(tables: List[datasets.table.Table], axis: int = 0) → datasets.table.Table

Concatenate tables.

Parameters

• tables (list of Table) – List of tables to be concatenated.

• axis ({0, 1}, default 0, meaning over rows) –

  Axis to concatenate over, where 0 means over rows (vertically) and 1 means over columns (horizontally).

  New in version 1.6.0.

Returns

datasets.table.Table that is the concatenated table – If the number of input tables is > 1, then the returned table is a datasets.table.ConcatenationTable. Otherwise if there’s only one table, it is returned as is.

datasets.table.list_table_cache_files(table: datasets.table.Table) → List[str]

Get the cache files that are loaded by the table. Cache file are used when parts of the table come from the disk via memory mapping.

Returns

List[str] – a list of paths to the cache files loaded by the table