pandas-metaframe 0.0.1
A pandas DataFrame with DataFrame row and col!
Requires Python: >=3.12

# MetaFrame

## Metadata-aware DataFrames for pandas

**MetaFrame** is a lightweight layer on top of **pandas** for working with
*metadata-rich*, *hierarchical*, and *semantically structured* datasets.

It is designed for situations where rows and columns are *not just labels*,
but meaningful entities with structure, attributes, and relationships.

MetaFrame makes these tables easier to **select**, **summarize**, **export**,
and—most importantly—**reason about** — while remaining **fully pandas-compatible**.

---

## Why?

In pandas, it can becomes harder to work with:

- rows or columns that encode *meaning* across multiple levels
- metadata that lives “next to” the data instead of inside it
- selections that become unreadable
- exporting and re-importing structured MultiIndex tables

MetaFrame addresses these problems by treating **structure as data**.

---

## When should I use MetaFrame?

Use MetaFrame if you work with:

- Scientific or experimental tables
- Excel files with semantic headers
- MultiIndex-heavy DataFrames
- Tables where rows and columns represent entities (samples, variables, runs)

If you only need flat, numeric DataFrames — pandas alone is perfect.

---

## Installation

```bash
git clone https://gitlab.com/dwishsan/metaframe.git
cd metaframe
pip install .
```

or

```bash
pip install pandas-metaframe
```

---

## Quick start

```python
import metaframe as mf # or `import metaframe as pd`

df = mf.DataFrame(...)

# string-based filtering, query-like
df.gs["Sample:'1'", "ID:>2"]

# metadata-based selection
df.mfloc["Sample", "ID"]

# nice export
df.to_file("my_file.xlsx")

# easy reading, regardless of DataFrame dimensions
df = mf.DataFrame.from_file("my_file.xlsx")

# still pandas
df.groupby("Sample").mean()
```

---

## The MetaFrame mental model

A **metaframe.DataFrame is not a new data structure**.

It:
- wraps a `pandas.DataFrame`
- preserves pandas behavior and performance expectations
- adds metadata-aware operations on top

But **metaframe.DataFrame is not just a pandas.DataFrame with labels**.
It changes how you *interact* with the data.

It treats:
- rows as structured entities
- columns as structured entities
- metadata as first-class data

You can always access the underlying pandas DataFrame when needed.

---

## Metadata as first-class data

Any DataFrame **without MultiIndex and with a numeric index** is a 
MetaData DataFrame, or **MetaFrame**.

MetaFrame can be attached to DataFrame:
- index/rows (MetaFrameRow, or MFR)
- columns (MetaFrameCol, or MFC)

From a DataFrame, MetaFrames are:
- queryable
- selectable
- settable

---

## Selection & indexing

MetaFrame introduces semantic selectors:

| Property | Purpose                                                      |
| -------- | ------------------------------------------------------------ |
| `q`      | General semantic selection (relying on pandas `query`)       |
| `gs`     | General semantic selection (relying on in-house Get-Strings) |
| `mfloc`  | Metadata-aware label selection                               |
| `mfiloc` | Metadata-aware positional selection                          |

These selectors are also settable, following the same rules as pandas `loc` & `iloc`.

---

## Summaries & analytics

MetaFrame provides a structured summary system.

Summaries:
- range from basic to extensive
- can be customized
- integrate naturally with grouping operations

This makes exploratory analysis easier and more readable.

---

## I/O & round-tripping

MetaFrame is designed for lossless export:

- Excel, with **custom formats**, **extensive summaries** and **row + columns filters**
- Structured text formats

---

## Relationship to pandas

MetaFrame guarantees:
- pandas APIs remain available
- pandas methods behave as expected
- zero monkey-patching of pandas internals

You can drop back to pandas at any.

## Documentation

📘 Wiki: conceptual guides and deep dives

📗 Cookbook: real-world patterns

📙 API reference: docstrings

Start with the Wiki if you’re new.

## Status

MetaFrame is actively developed and used in real workflows.

The API is evolving, but core concepts are stable.

Contributions and feedback are welcome.

---

## Full example

```python
# -----------------------------------
#  Imports
# -----------------------------------
from numpy.random import seed, randn
import pandas as pd
import metaframe as mf
# or, to test it in pre-existing code
# import metaframe as pd

# -----------------------------------
#  Pandas DataFrame creation
# -----------------------------------

columns = pd.DataFrame(
    {
        'ID': [1, 2, 3],
        'Group': ['A', 'A', 'B'],
        'Name': ['UNK42', None, 'UNK37']
    }
)

index = pd.DataFrame(
    {
        'ID': [1, 2, 3, 4, 5],
        'Sample': ['1', '1', '2', '2', '2'],
        'Replicate': [1, 2, 1, 2, 3]
    }
)

seed(37)
df = pd.DataFrame(randn(5, 3), index=pd.MultiIndex.from_frame(index), columns=pd.MultiIndex.from_frame(columns))

# -----------------------------------
#  MetaFrame DataFrame creation
# -----------------------------------

df = mf.DataFrame(df)
df
"""
ID                          1         2         3
Group                       A         A         B
Name                    UNK42       nan     UNK37
ID Sample Replicate                              
1  1      1         -0.054464  0.674308  0.346647
2  1      2         -1.300346  1.518512  0.989824
3  2      1          0.277681 -0.448589  0.961966
4  2      2         -0.827579  0.534657  1.228386
5  2      3          0.519592 -0.063355 -0.034793
"""

# -----------------------------------
#  Get-strings
# -----------------------------------

df.gs["Sample:'1' or Replicate:>2", "Group:!B and Name:None"]
"""
ID                          2
Group                       A
Name                      nan
ID Sample Replicate          
1  1      1          0.674308
2  1      2          1.518512
5  2      3         -0.063355
"""

# -----------------------------------
#  Get-strings with RE
# -----------------------------------

df.gs[:, "Name:'.*7$'"]
"""
ID                          3
Group                       B
Name                    UNK37
ID Sample Replicate          
1  1      1          0.346647
2  1      2          0.989824
3  2      1          0.961966
4  2      2          1.228386
5  2      3         -0.034793
"""

# -----------------------------------
#  MetaFrame-based selection
# -----------------------------------

df.mfloc[
    ([1, 4, 2], ['ID', 'Sample']), 
    'Name'
]
"""
Name          UNK42       NaN     UNK37
ID Sample                              
2  1      -1.300346  1.518512  0.989824
5  2       0.519592 -0.063355 -0.034793
3  2       0.277681 -0.448589  0.961966
"""
df.mfiloc[
    (slice(None), 1), 
    ([0, 1],)
]
"""
ID             1         2
Group          A         A
Name       UNK42       nan
Sample                    
1      -0.054464  0.674308
1      -1.300346  1.518512
2       0.277681 -0.448589
2      -0.827579  0.534657
2       0.519592 -0.063355
"""

# -----------------------------------
#  MetaFrame indexing
# -----------------------------------

df.mfr = df.mfr.replace({'1': '3'})
df
"""
ID                          1         2         3
Group                       A         A         B
Name                    UNK42       nan     UNK37
ID Sample Replicate                              
1  3      1         -0.054464  0.674308  0.346647
2  3      2         -1.300346  1.518512  0.989824
3  2      1          0.277681 -0.448589  0.961966
4  2      2         -0.827579  0.534657  1.228386
5  2      3          0.519592 -0.063355 -0.034793
"""

df.mfc = df.mfc.merge(df.ms(axis=0))
df
"""
ID                          1         2         3   4   5
Group                       A         A         B nan nan
Name                    UNK42       nan     UNK37 nan nan
Sample                      3         3         2   2   2
Replicate                   1         2         1   2   3
ID Sample Replicate                                      
1  3      1         -0.054464  0.674308  0.346647 NaN NaN
2  3      2         -1.300346  1.518512  0.989824 NaN NaN
3  2      1          0.277681 -0.448589  0.961966 NaN NaN
4  2      2         -0.827579  0.534657  1.228386 NaN NaN
5  2      3          0.519592 -0.063355 -0.034793 NaN NaN
"""

# -----------------------------------
#  MetaFrame-based DataFrame set
# -----------------------------------

df.gs[:, "ID:4,5"] = [0, 1]
df
"""
ID                          1         2         3    4    5
Group                       A         A         B  nan  nan
Name                    UNK42       nan     UNK37  nan  nan
Sample                      3         3         2    2    2
Replicate                   1         2         1    2    3
ID Sample Replicate                                        
1  3      1         -0.054464  0.674308  0.346647  0.0  1.0
2  3      2         -1.300346  1.518512  0.989824  0.0  1.0
3  2      1          0.277681 -0.448589  0.961966  0.0  1.0
4  2      2         -0.827579  0.534657  1.228386  0.0  1.0
5  2      3          0.519592 -0.063355 -0.034793  0.0  1.0
"""

# -----------------------------------
#  Summaries
# -----------------------------------

df.summary_basic()
"""
           Rows  Columns  Cells
DataFrame     5        5     25
MSR           5        3     15
MSC           5        5     25
"""

df.summary_whole()
"""
                                           DataFrame
dtype   Mode     Metric        Type                 
all     Describe Num. elements count           25.00
                 NAs           count            0.00
                               %                0.00
Numeric Describe Num. elements count           25.00
                               %              100.00
                 mean          mean             0.37
                 std           std              0.68
                 min           min             -1.30
                 25%           percentile       0.00
                 50%           percentile       0.35
                 75%           percentile       1.00
                 max           max              1.52
                 sum           sum              9.32
        Custom   zeros         count            5.00
                               %               20.00
                 filled        count           14.00
                               %               56.00
"""

df.summary()
"""
ID                                              1       2       3       4       5
Sample                                          3       3       2       2       2
Replicate                                       1       2       1       2       3
dtype   Mode     Metric        Type                                              
all     Describe Num. elements count         5.00    5.00    5.00    5.00    5.00
                 NAs           count         0.00    0.00    0.00    0.00    0.00
                               %             0.00    0.00    0.00    0.00    0.00
Numeric Describe Num. elements count         5.00    5.00    5.00    5.00    5.00
                               %           100.00  100.00  100.00  100.00  100.00
                 mean          mean          0.39    0.44    0.36    0.39    0.28
                 std           std           0.45    1.12    0.62    0.83    0.47
                 min           min          -0.05   -1.30   -0.45   -0.83   -0.06
                 25%           percentile    0.00    0.00    0.00    0.00   -0.03
                 50%           percentile    0.35    0.99    0.28    0.53    0.00
                 75%           percentile    0.67    1.00    0.96    1.00    0.52
                 max           max           1.00    1.52    1.00    1.23    1.00
                 sum           sum           1.97    2.21    1.79    1.94    1.42
        Custom   zeros         count         1.00    1.00    1.00    1.00    1.00
                               %            20.00   20.00   20.00   20.00   20.00
                 filled        count         3.00    3.00    3.00    3.00    2.00
                               %            60.00   60.00   60.00   60.00   40.00
"""

df.groupby('Sample').apply(lambda x: x.summary())
"""
ID                                                     3       4       5       1       2
Sample                                                 2       2       2       3       3
Replicate                                              1       2       3       1       2
Sample dtype   Mode     Metric        Type                                              
2      all     Describe Num. elements count         5.00    5.00    5.00     NaN     NaN
                        NAs           count         0.00    0.00    0.00     NaN     NaN
                                      %             0.00    0.00    0.00     NaN     NaN
       Numeric Describe Num. elements count         5.00    5.00    5.00     NaN     NaN
                                      %           100.00  100.00  100.00     NaN     NaN
                        mean          mean          0.36    0.39    0.28     NaN     NaN
                        std           std           0.62    0.83    0.47     NaN     NaN
                        min           min          -0.45   -0.83   -0.06     NaN     NaN
                        25%           percentile    0.00    0.00   -0.03     NaN     NaN
                        50%           percentile    0.28    0.53    0.00     NaN     NaN
                        75%           percentile    0.96    1.00    0.52     NaN     NaN
                        max           max           1.00    1.23    1.00     NaN     NaN
                        sum           sum           1.79    1.94    1.42     NaN     NaN
               Custom   zeros         count         1.00    1.00    1.00     NaN     NaN
                                      %            20.00   20.00   20.00     NaN     NaN
                        filled        count         3.00    3.00    2.00     NaN     NaN
                                      %            60.00   60.00   40.00     NaN     NaN
3      all     Describe Num. elements count          NaN     NaN     NaN    5.00    5.00
                        NAs           count          NaN     NaN     NaN    0.00    0.00
                                      %              NaN     NaN     NaN    0.00    0.00
       Numeric Describe Num. elements count          NaN     NaN     NaN    5.00    5.00
                                      %              NaN     NaN     NaN  100.00  100.00
                        mean          mean           NaN     NaN     NaN    0.39    0.44
                        std           std            NaN     NaN     NaN    0.45    1.12
                        min           min            NaN     NaN     NaN   -0.05   -1.30
                        25%           percentile     NaN     NaN     NaN    0.00    0.00
                        50%           percentile     NaN     NaN     NaN    0.35    0.99
                        75%           percentile     NaN     NaN     NaN    0.67    1.00
                        max           max            NaN     NaN     NaN    1.00    1.52
                        sum           sum            NaN     NaN     NaN    1.97    2.21
               Custom   zeros         count          NaN     NaN     NaN    1.00    1.00
                                      %              NaN     NaN     NaN   20.00   20.00
                        filled        count          NaN     NaN     NaN    3.00    3.00
                                      %              NaN     NaN     NaN   60.00   60.00
"""

# -----------------------------------
#  Natural sorting & ordering
# -----------------------------------

df.mfc.natsort_values('Name', axis=1)
df.mfr.order_values({'Sample': [2, 3], 'Replicate': None})
df.auto_sort()

# -----------------------------------
#  I/O
# -----------------------------------

df.to_file('path/to/my/excel_file.xlsx')
df = mf.DataFrame.from_input('path/to/my/excel_file.xlsx')
```

## License

[MIT](https://choosealicense.com/licenses/mit/) © Audric Cologne