aicard 0.3.3
Compute and organize model cards locally or online.
Requires Python: >=3.11

# AI Card

This SDK contains a collection of methods to create, manage, 
and edit AI model cards. Cards can be stored either locally or in
an online service (that can also be self-hosted). We finally provide
methods to populate fields by analyzing datasets by computing well-known 
measures or calling AI assistants.

*Alpha version - Current apis and functionalities are unstable.*

## ⚡ Quickstart

INstall *aicard* in a virtual environment like below.
If you are a developer working on this repository, clone it and install it locally
per `pip install -e package` instead. 

```commandline
python -m venv .venv
source .venv/bin/activate
pip install aicard
```

Create your first model card like below. You can also visit our public service
or self-host a copy of your own to maintain a database of cards and modify
fields through a UI. The *aicard* library makes that hosting possible, allows
programmatic management of cards, and **passes data to the service from your 
local environment**. Those data comprise mainly quantitative evaluation.

```python
# demo.py
import aicard as aic

card = aic.ModelCard()
card.title = "Model Card"
card.model.name = "Llama"
card.model.overview = "This is a model overview. Freely add <b>html</b> or *markdown*."
card.model.version = "3.2"
card.considerations.use_case = "text generation"

print(card)
```

```commandline
> python demo.py
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
┃                                  Model Card                                  ┃
┗━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┛
completion        🧩🧩⚠️⚠️⚠️⚠️⚠️⚠️⚠️⚠️⚠️⚠️⚠️⚠️⚠️⚠️⚠️⚠️⚠️⚠️                                        

                                     model                                      
name               Llama                                                        
overview           This is a model overview. Freely add html or markdown.       
version            3.2                                                       

                                 considerations                                 
use case           text generation   
```


## 🧠 Assistants

There are broadly two types of assistants for model evaluation: evaluation ones that can 
quantitatively evaluate your models based on popular measures, and LLM-based ones that fill or 
enrich qualitative fields of your cards. The last type can also run through services too.

Perform manual numerical assessment across a wide variety of available tasks holding popular
evaluation methodologies and measures. If you need customization, you can create your own tasks too. 

```python
# experiment.py
from datasets import load_dataset
from huggingface_hub import dataset_info
from transformers import pipeline
import aicard as aic

dataset = load_dataset("google-research-datasets/go_emotions", split = 'test')
info = dataset_info("google-research-datasets/go_emotions")
class_names = info.card_data['dataset_info'][1]['features'][1]['sequence']['class_label']['names']
classifier = pipeline(task="text-classification", model="SamLowe/roberta-base-go_emotions", top_k=None)

def pipeline(data):
    sentences = [text for text in data['text']]
    model_outputs = classifier(sentences)
    out = []
    for sample in model_outputs:
        flat = {d['label']: d['score'] for d in sample}
        out.append([flat[name] for name in class_names.values()])
    return out

metrics = aic.evaluation.evaluate(
    data=dataset,
    pipeline=pipeline,
    task=aic.evaluation.tasks.nlp.text_classification,
    batch_size=32,
    as_card=True
)

print(metrics)
```

```commandline
> python experiment.py
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
┃                         Text Classification Results                          ┃
┗━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┛
completion        🧩⚠️⚠️⚠️⚠️⚠️⚠️⚠️⚠️⚠️⚠️⚠️⚠️⚠️⚠️⚠️⚠️⚠️⚠️⚠️                                         

                                    analysis                                    
analysis           Evaluation was conducted at 2025-08-19 for text              
classification with 32 batch size. A pipeline function runs the model.          
metrics            The following metrics were computed at 2025-08-19:           
- precision_macro: 0.575                                                        
- precision_micro: 0.685                                                        
- recall_macro: 0.396                                                           
- recall_micro: 0.511                                                           
- f1_macro: 0.450                                                               
- f1_micro: 0.586                                                               
- auc_roc_macro: 0.929   
```

The default `as_card=True` creates a model card that is partially filled with analysis data
automatically obtained from the evaluation process.  Otherwise, computed metrics are returned 
as a dictionary of `(name, value)` pairs. The default case's card 
can be merged into an existing card. Merge commands accept an argument to 
signify whether non-empty fields of the base card should be replaced or (if not) have the
new card's contents be appended.

```python
card.merge(metrics, replace=True)
```

You have the option to collaborate with LLM assistants too!
These let you fill in qualitative aspects of the model card from a software's repository.
In the simplest case, the assistants will be owned by you. Buf, if you want
persistent storage of your cards, you can collaborate with the public or self-hosted service too 
(see below). The assistant will create a copy of the model card with changes applied.

```python
card = card.assistant(
    repository="myproject/", # your model's git repository or working directory here
    model=mc.assistants.olama,
    dotenv=".env", # assistant configuration file
)
```

The AI assistant can also create a simplified version 
of your model card that is friendlier to laypeople to read and parse through.

```python
card = card.gist(
    model=aic.assistants.olama,
    dotenv=".env" # assistant configuration file
)
```

## 📡 Connecting to a server

You can link to a public or self-hosted server for persistent card storage.
Here we will link to the public server, with can be found at `URL` (to be decided).
To work with a server you need to log in first and create a card.

```python
# the credentials bellow are the default for self-hosted servers
conn = aic.connect("URL").login(username="admin", password="admin")  
card = conn.create()  # no argument for a brand new card
```

Alternatively, connect with a dotenv file holding those fields:

```python
conn = aic.connect("URL").login(env=".env")  
card = conn.create()
```

```bash
# .env
USER=admin
PASS=admin
```

You can also search for cards. Cards that you do not own do not come
with any connection attached, and therefore cannot be used as contexts.
To simplify usage, the default argument value `owned_only=True` retrieves
only the cards owned by you.

```python
for card in conn.search("Model Card", owned_only=True, top=10):
    print(card.title)  # treat it as a normal card
```

Submit local card modifications to the server by calling `card.commit()`. 
The card keeps track of the connection. If you fail to do this throughout your program, 
you will eventually get an assertion error. Call `card.detach()` 
to safely detach a card from a connection without commiting 
pending changes (this disables further commits). The best practice is to use the card
as a context when making changes that require a commit, like below:

```python
conn = aic.connect("http://127.0.0.1:5000", username="admin", password="admin")
with conn.create(card) as card:
    card.title = "Updated model card name"
```

Connections can also substitute `aic` as the environment of assistants. 
For safety, you will get an error if you try to call
a connection gist or metric computation without commiting the card first,
as the server operate in its own copy. 

```python
card = card.gist(
    model=aic.assistants.olama,
    dotenv=conn  # use the connection as assistant configuration
)
```

Server load may delay the above snippet, as it blocks until notified by the sever. 
Finally, the server may not support the assistant, for example if it is a custom one.

## 🛠️ Self-hosting a server

If you have *aicard* installed, you can immediately self-host a server.
To do so, create a dictionary of assistants and start a flask service. 
This will set up everything the first time, including a database. 
If you do not plan to expose the server externally, you can login with
the default administrator credentials, like above.
If you want console instead of persistent logging, skip 
the `log_file` argument. Do note that, if an `.env` file is provided, then 
there will be an attempt to retrieve missing keyword arguments from there.
Below is an example service whose assistant is primarily used for testing:

```python
from aicard.service import serve, TestAssistant

app = serve({"tassist": TestAssistant()}, env=".env")
app.run()
```


```bash
# .env
INDEX=/apidocs # redirect_index
USER=admin  # admin_username 
PASS=admin  # admin_password
LOG=log.txt # log_file
```

## 📜 License

TBD