Title: concatenate_datasets support axis=0 or 1 ？
Author: renqingcolin (NONE)
State: closed
Created: 2020-11-16T02:46:23Z
Labels: enhancement

help wanted

question

I want to achieve the following result
![image](https://user-images.githubusercontent.com/12437751/99207426-f0c8db80-27f8-11eb-820a-4d9f7287b742.png)

Comments:
Unfortunately `concatenate_datasets` only supports concatenating the rows, while what you want to achieve is concatenate the columns.
Currently to add more columns to a dataset, one must use `map`.
What you can do is somehting like this:
```python
# suppose you have datasets d1, d2, d3
def add_columns(example, index):
    example.update(d2[index])
    example.update(d3[index])
    return example

full_dataset = d1.map(add_columns, with_indices=True)
```

Closing this one, feel free to re-open if you have other questions about this issue

That's not really difficult to add, though, no?
I think it can be done without copy.
Maybe let's add it to the roadmap?

Actually it's doable but requires to update the `Dataset._data_files` schema to support this.
I'm re-opening this since we may want to add this in the future

Hi @lhoestq, I would love to help and add this feature if still needed. My plan is to add an axis variable in the `concatenate_datasets` function in `arrow_dataset.py` and when that is set to 1 concatenate columns instead of rows. 

Hi ! I would love to see this feature implemented as well :) Thank you for proposing your help !

Here is a few things about the current implementation:
- A dataset object is a wrapper of one `pyarrow.Table` that contains the data
- Pyarrow offers an API that allows to transform Table objects. For example there are functions like `concat_tables`, `Table.rename_columns`, `Table.add_column` etc.

Therefore adding columns from another dataset is possible thanks to the pyarrow API and in particular `Table.add_column` :) 

However this breaks some features we have regarding pickle. A dataset object can be pickled and unpickled without loading all the data in memory. It is useful for multiprocessing for example. Pickling a dataset object is possible thanks to the `Dataset._data_files` which defines the list of arrow files that will be used to form the final Table (basically all the data from each files are concatenated on axis 0).

Therefore to be able to add columns to a Dataset and still be able to work with it in a multiprocessing setup, we need to extend this last aspect to be able to reconstruct a Table object from multiple arrow files that are combined in both axis 0 and 1. Currently this reconstruction mechanism only supports axis 0.

I'm sure we can figure something out that enables users to add columns from another dataset while keeping the multiprocessing support.

@lhoestq, we have two Pull Requests to implement:
- Dataset.add_item: #1870
- Dataset.add_column: #2145
which add a single row or column, repectively.

The request here is to implement the concatenation of *multiple* rows/columns. Am I right?

We should agree on the API:
- `concatenate_datasets` with `axis`?
- other Dataset method name?

For the API, I like `concatenate_datasets` with `axis` personally :)
From a list of `Dataset` objects, it would concatenate them to a new `Dataset` object backed by a `ConcatenationTable`, that is the concatenation of the tables of each input dataset. The concatenation is either on axis=0 (append rows) or on axis=1 (append columns).

Regarding what we need to implement:
The axis=0 is already supported and is the current behavior of `concatenate_datasets`.
Also `add_item` is not needed to implement axis=1 (though it's an awesome addition to this library).

To implement axis=1, we either need `add_column` or a `ConcatenationTable` constructor to concatenate tables horizontally.
I have a preference for using a `ConcatenationTable` constructor because this way we can end up with a `ConcatenationTable` with only 1 additional block per table, while `add_column` would add 1 block per new column.

Maybe we can simply have an equivalent of `ConcatenationTable.from_tables` but for axis=1 ?
`axis` could also be an argument of `ConcatenationTable.from_tables`

@lhoestq I think I guessed your suggestions in advance... 😉 #2151

Cool ! Sorry I missed this one ^^
I'm taking a look ;)