distinction 0.0.1.7
A fast binary classifier built on semantic search.
Requires Python: >=3.9
Keywords: binary classification, cosine similarity, embeddings, feature selection, sentence transformer


## Installation

#### From Github
```bash
pip3 install git+https://github.com/er1kb/distinction
```
or clone and install locally:
```bash
git clone https://github.com/er1kb/distinction.git && cd distinction && pip3 install .
```

#### From PyPI
```bash
python3 -m pip install distinction
# or if you want to look at the error curves:
python3 -m pip install distinction[plot]
```

#### Dependencies
* [Numpy](https://numpy.org/) >= 1.25.0
* [SentenceTransformers](https://sbert.net/) >= 3.0.1
* [Plotext](https://github.com/piccolomo/plotext) >= 5.3.2 (optional)


## Overview
A common use case is to be able to predict whether something is this or that, when that something is a piece of text. You may be working with custom logs (customer service requests, reviews, etc.) or open-ended survey responses that need to be coded at scale. Although neural networks can be used to classify latent variables in natural language, their complexity and overhead are a disadvantage. 

Embeddings are high-dimensional features that neural networks use. They quantify information in the original data. Sentence transformer models encode meaning by producing sentence embeddings, ie representing text as high-dimensional vectors. These models are comparatively fast and lightweight and can even run on the cpu. Their output is easily stored in a vector database, so that you really only have to run the model once. Since vectors are points in an abstract space, we are able to measure if these points are close to each other (similar) or further apart (unrelated or opposite). 

Classification can be done by comparing the embedding of an individual text to a "typical" embedding for a given category. To "train" the classifier, you need a manually classified dataset. The minimum size of this dataset will depend on the number of dependent variables, how well-defined these variables are, and the ability of the sentence transformer model to encode relevant signals in your dataset. 

The classifier uses a relevant subset ("selection") of the vector dimensions to separate signal from noise. A similarity threshold is chosen as the decision boundary between 0 and 1. Ideally, comparisons are made at the level of individual sentences. The classifier can be optimized/tuned by repeatedly running it on a validation dataset and selecting the threshold values with the best outcome. In the absence of validation data, this process can also be done manually.

This tool has grown out of my work with customer service requests and other textual data. The idea of semantically grouping texts by calculating centroids is not new, but we needed a framework that would fit into our data pipelines and allow for some optimization. A catalyst of this work was when I failed at compiling Tensorflow for the GPU, after spending an entire workday on it. I had succeeded before but was still annoyed with the execution speed and the need for a gpu, as well as the size of saved models. Neural networks are cool and all, but backpropagation may not be the appropriate solution to every analytical problem. 


## How-to / Examples

Some things to consider before (or after) diving into the examples: 
* The quality of your predictions depends on the quality of training and prediction data. Specific categories yield a lot better prediction error than general ones. Use categories that are somewhat coherently expressed and not too broad. For example, _recycling_ and _energy\_consumption_ are more useful in this respect than _sustainability_. The latter is an emergent phenomenon that should be inferred from its subcategories. Our use case may be to identify all texts that are related to _sustainability_, but doing so directly would yield a much higher error rate due to greater variability in the data. 
* If possible, train and predict at the sentence level, not on entire paragraphs. In the real world, text is rarely homogenous. For example, a review may have sentences that can be considered positive, negative or somewhere in between. Sentences are the best semantic unit, as hinted by the term "sentence transformer". Tools provided [here](#split-and-combine-records) and [here](#set-up-prediction-pipelines-for-continuous-data-streams) allow you to predict sentences and then aggregate the general tendency. 
* Input data is an iterable (list/generator/tuple) of dicts. Export from your favourite dataframe library using polars.DataFrame.[to\_dicts()](https://docs.pola.rs/api/python/stable/reference/dataframe/api/polars.DataFrame.to_dicts.html) or pandas.DataFrame.[to\_dict('records')](https://pandas.pydata.org/docs/reference/api/pandas.DataFrame.to_dict.html).
* Results are returned as generators where possible to allow for lazy computation (as needed). To get a regular list back, you need to put the generator in a list and [unpack](https://www.geeksforgeeks.org/convert-generator-object-to-list-in-python/) it with an asterisk, eg __predictions = [\*predictions]__. The lazy nature of generators also means that no computation is done until you unpack the generator or call next() on it to yield the first element. 
* Always use the same _model_ for training and predition. If your text is not in English, remember to substitute the [default model](https://huggingface.co/sentence-transformers/all-MiniLM-L6-v2) with another one from the [Huggingface model hub](https://huggingface.co/models?pipeline_tag=sentence-similarity&sort=downloads) that speaks your language. There are language-specific and multilingual ones to choose from. A model that doesn't understand your language will still produce results, although they won't make much sense. 



### Set up and use a binary Classifier for independent variables

<details>
<summary>Expand</summary>

This is our training data, a small sample of 21 manually coded short sentences. Notice the "suggestion" variable is formatted as strings, eg with quotation marks, so we need to do some data cleaning to get the counts. This is really the only time we should have to use the helper function _ones_to_int()_ directly. As the need for int conversion is a common scenario, especially in a dynamically typed language, the conversion will always be done automatically when you add data to the classifier. 

```python
from distinction import Classifier, count_used_keys, ones_to_int

data = [
    { "id": 0,  "message": "Greatest burgers I ever tasted", "positive": 1, "suggestion": "0", "taste": 1, "service": 0 },
    { "id": 1,  "message": "Your fries could be a bit more salty", "positive": 0, "suggestion": "1", "taste": 1 },
    { "id": 2,  "message": "Good service at the drive-in, those people should be given a raise", "positive": 1, "suggestion": 0, "taste": 0, "service": 1 },
    { "id": 3,  "message": "This is spam", "spam": 1 },
    { "id": 4,  "message": "I've never tasted such awful fries", "positive": 0, "suggestion": "0", "taste": 1, "service": 0 },
    { "id": 5,  "message": "Thanks for helping me to get around with my wheelchair!", "positive": 1, "suggestion": "0", "taste": 0, "service": 1 },
    { "id": 6,  "message": "Maybe upgrade your burger buns, the current ones are dry and boring", "positive": 0, "suggestion": "1", "taste": 1, "service": 0 },
    { "id": 7,  "message": "Is this how you handle your customers?", "positive": 0, "suggestion": "0", "taste": 0, "service": 1 },
    { "id": 8,  "message": "The salad tasted a bit different, I think you should put some oil on it.", "positive": 0, "suggestion": "1", "taste": 1, "service": 0 },
    { "id": 9,  "message": "This is the best place in town, and the staff are customer-oriented", "positive": 1, "suggestion": "0", "taste": 0, "service": 1 },
    { "id": 10, "message": "Too much pepper, please tell your chef to be more conservative with the seasoning", "positive": 0, "suggestion": "1", "taste": 1, "service": 0 },
    { "id": 11, "message": "I appreciate your help", "positive": 1, "suggestion": "0", "taste": 0, "service": 1 },
    { "id": 12, "message": "I think you should install security cameras on the parking lot", "positive": 0, "suggestion": "1", "taste": 0, "service": 0 },
    { "id": 13, "message": "I don't like their french fries, but the burgers are ok and the staff are usually helpful", "positive": 0, "suggestion": "0", "taste": 1, "service": 1 },
    { "id": 14, "message": "There should be more suitable options for vegans who are also keto and paleo crossfitters!", "positive": 0, "suggestion": "1", "taste": 0, "service": 1 },
    { "id": 15, "message": "Big Burger is trying to POISON us all with vegetable oils", "positive": 0, "suggestion": "0", "taste": 0, "service": 1 },
    { "id": 16, "message": "I complained about my burger and got a new one with just the right amount of seasoning - great customer service!", "positive": 1, "suggestion": "0", "taste": 1, "service": 1 },
    { "id": 17, "message": "Poor excuse for an establishment, why don't you just shut down?", "positive": 0, "suggestion": "1", "taste": 0, "service": 1 },
    { "id": 18, "message": "Great food, great location", "positive": 1, "suggestion": "0", "taste": 1, "service": 0 },
    { "id": 19, "message": "More spam", "positive": 0, "suggestion": "0", "taste": 0, "service": 0, "spam": 1 },
    { "id": 20, "message": "Great service, but not so great food", "positive": 1, "suggestion": "1", "taste": 1, "service": 1 }
]

binary_variables = 'positive suggestion taste service spam'.split()
data = [*ones_to_int(data, keys = binary_variables)] # Convert strings to int - this is done automatically by the classifier later on
print(count_used_keys(data, ignore = 'id message'))
```

Counts of targets in the training data:
```python
{'positive': 8, 'suggestion': 8, 'spam': 2, 'taste': 10, 'service': 11}
```

#### Classifier from training\_data - raw text 

<details>
<summary>Initiate and train the classifier</summary>

First step is to define the classifier. Using a dict for keyword arguments means the arguments are reusable. We tell the classifier which columns are binary variables (targets). A confounder is a special kind of target, as it cannot be anything else. In this example, we don't want spam messages to taint our customer service statistics. 
The train() method below calls the sentence transformer to encode the texts, then calculates the centroids of each target and finally ranks the features (embedding dimensions) by relevance. 
<br><br>

```python

kwargs = {
    'targets': 'positive suggestion taste service'.split(),
    'confounders': ['spam'],
    'id_columns': ['id'],
    'text_column': 'message',
    'default_selection': 0.05,
    'model': 'sentence-transformers/all-MiniLM-L6-v2'
}

C = Classifier(**kwargs)
C.train(data)

```

</details>



<details>
<summary>Predict</summary>


Let's try to classify a couple of new texts. This is just using default parameters: looking at 5% (=39) of the 768 embedding dimensions and classifying something as 1 if the similarity with its centroid is at least 0.5. The sample size of this example is too small to reliably [optimize](#optimize-the-classifier) the classifier.
```python
predictions = [*C.predict([{"message": "I really like the taste of these burgers."},
                          {"message": "The staff was really helpful"},
                          {"message": "This is definitely spam"}
                          ])]
for p in predictions:
    print(p)
```

These results are ok given the small sample size and lack of optimization, although the first one should have been classified as positive. Notice the third sample is spam and has all other targets set to 0, since _spam_ was declared to be a confounding variable. 
```bash
{'message': 'I really like the taste of these burgers.', 'positive': 0, 'service': 0, 'suggestion': 0, 'taste': 1, 'spam': 0}
{'message': 'The staff was really helpful', 'positive': 1, 'service': 1, 'suggestion': 0, 'taste': 0, 'spam': 0}
{'message': 'This is definitely spam', 'positive': 0, 'service': 0, 'suggestion': 0, 'taste': 0, 'spam': 1}
```


For this example, we can also run predict on the original data. Using training data to validate a model is considered bad practice because of the obvious risk of overfitting, but the results below still tell us that the classifier has picked up some relevant signals. 

```python
predictions = [*C.predict(data)]
print(f"{'PREDICTIONS':<40}TEXT")
for p in predictions:
    print(f"{', '.join([k for k,v in p.items() if k in (kwargs['targets'] + kwargs['confounders']) and v == 1]):40}{p['message']}")
```


```
PREDICTIONS                             TEXT
positive, taste                         Greatest burgers I ever tasted
suggestion, taste                       Your fries could be a bit more salty
positive, service                       Good service at the drive-in, those people should be given a raise
spam                                    This is spam
taste                                   I've never tasted such awful fries
positive, service                       Thanks for helping me to get around with my wheelchair!
suggestion, taste                       Maybe upgrade your burger buns, the current ones are dry and boring
positive, service                       Is this how you handle your customers?
suggestion, taste                       The salad tasted a bit different, I think you should put some oil on it.
positive, service                       This is the best place in town, and the staff are customer-oriented
suggestion, taste                       Too much pepper, please tell your chef to be more conservative with the seasoning
positive, service                       I appreciate your help
suggestion                              I think you should install security cameras on the parking lot
service, taste                          I don't like their french fries, but the burgers are ok and the staff are usually helpful
service, suggestion, taste              There should be more suitable options for vegans who are also keto and paleo crossfitters!
service, taste                          Big Burger is trying to POISON us all with vegetable oils
positive, service, taste                I complained about my burger and got a new one with just the right amount of seasoning - great customer service!
service                                 Poor excuse for an establishment, why don't you just shut down?
positive, taste                         Great food, great location
spam                                    More spam
positive, service, suggestion, taste    Great service, but not so great food
```

</details>



<details>
<summary>Validate</summary>
<br>

If there is validation data with the right answers, you can assess model performance using _predict(..., validation = True)_. For this example, we're going to cheat by re-using the training data listed above. The _Classifier.error()_ method will print the error by target variable. Validation also produces a list of prediction errors per row stored at _Classifier.error\_rate\_by\_row_. In the code below, note the [generator unpacking](https://www.geeksforgeeks.org/convert-generator-object-to-list-in-python/) of the prediction results to force the computation and produce the error rate. 
```python
_ = [*C.predict(data, validation = True)]
C.error()
```

As expected, we get an artificially low error rate since the model is overfitted to our training data. Rows 14 and 15 were predicted as taste. You could argue these messages are food related (keto/paleo and vegetable oils respectively), even though they were not manually coded as such. 

```bash
TARGETS             OVERALL             FALSE POSITIVE      FALSE NEGATIVE      THRESHOLD
----------------------------------------------------------------------------------------------------
positive            0.05                0.05                0.0                 0.5
service             0.0                 0.0                 0.0                 0.5
suggestion          0.05                0.0                 0.05                0.5
taste               0.1                 0.1                 0.0                 0.5

CONFOUNDERS         OVERALL             FALSE POSITIVE      FALSE NEGATIVE      THRESHOLD
----------------------------------------------------------------------------------------------------
spam                0.0                 0.0                 0.0                 0.5
```

</details>




#### Classifier from training\_data - pre-encoded

<details>
<summary>Expand</summary>
<br>

Although sentence transformers are fast compared to other neural networks, the encoding of text is the most time consuming part of the Classifier model and especially for large datasets. For the training stage, you can get around this by using a smaller sample. Another obvious way to save time and computation is if you have pre-existing embeddings in a vector database, such as [Elasticsearch](https://www.elastic.co/). You can then skip the encoding altogether by calling _train()_ and/or _predict()_ with the argument _pre\_encoded = True_. Ideally, you should never have to encode text more than once in a data pipeline. When using existing vectors, you need to specify _vector\_column_ instead of _text\_column_. These two sources are kept separate to allow for use cases where both raw text and embeddings are processed.  

In the following example, we encode the training data outside of the Classifier, but then we go back to using raw text from the prediction data. For raw text, use the _text\_column_ argument (if omitted, the default value being "text"). For vectors, use __either__ the _text\_column_ or the optional _vector\_column_ argument. The reason these two arguments exist side by side is you might want to predict using vectors while at the same time concatenating text - more on this in the section on pipelines. 
The code below uses an optional [pytorch](https://github.com/pytorch/pytorch) check to run the sentence transformer on the GPU. 

```python
from distinction import Classifier
from sentence_transformers import SentenceTransformer
import torch

device = torch.device('cuda' if torch.cuda.is_available() else 'cpu')
print('Using device:', device)

model = SentenceTransformer(model_name_or_path = 'sentence-transformers/all-MiniLM-L6-v2',
                            device = device,
                            tokenizer_kwargs = {'clean_up_tokenization_spaces': True}) # this setting gets rid of a warning in Transformers 4.45.1

training_data = [dict(text='I am the great Cornholio', beavis=1), 
                 dict(text='I have seen the top of the mountain, and it is good', beavis=0)]

vectors = model.encode([r['text'] or '' for r in training_data], show_progress_bar = False)

for i,_ in enumerate(training_data):
    training_data[i]['vector'] = vectors[i] # Merge sentence embeddings into the original data


C = Classifier(text_column = 'vector', show_progress_bar = False) # Set to read encoded "text" for prediction. Optionally use vector_column = 'vector' instead, if you want to. 
C.train(training_data, pre_encoded = True) # Tell the classifier that the text column is an embedding

# Let's assume the prediction data is supplied as raw text
C.text_column = 'text' # Set to read proper text from the "text" column, as we predict with pre_encoded = False on the next line
C.model = model # For transparency. The model above is the default one. Always use the same model for training and prediction.
results = C.predict([dict(text = 'I am Cornholio!')], pre_encoded = False)
print(next(results))
```

Output:
```bash
❯ python3 pre_encoded.py
Using device: cuda
Done encoding prediction data

{'text': 'I am Cornholio!', 'beavis': 1}
```


</details>

</details>



### Optimize the Classifier

<details>
<summary>Expand</summary>

This section can be run with the data and settings from the restaurant reviews example [above](#set-up-and-use-a-binary-classifier-for-independent-variables). Again, we will have to re-use the training data for the purpose of demonstration. The very small sample size means less chance of convergence, meaning the results are somewhat sketchy if not useless. Tuning the model uses repeated validation, which means the prediction data has to contain the right answers. 

Currently, the optimal similarity cutoff is determined to be the sweetspot between type 1 and type 2 errors. Minimizing the overall error works well for most targets, but for rare ones the model will just assign 0 to everything to achieve the best accuracy. The concept of [_imbalanced datasets_](https://discuss.pytorch.org/t/model-predictions-are-all-tensors-full-of-zeros/155381/11) is a known problem even for neural networks. Minimizing the difference between false positive and false negative rates does not yield the absolute minimum error, but it does seem to avoid having the model apply the null hypothesis to everything. It's on my todo list to improve on this, by putting some constraint on the optimization. In the meantime, you can use the plotting function to see whether there is room for improvement or not. 


<details>
<summary>Tune similarity</summary>
<br>

To find an estimate of the optimal similarity threshold, run the __Classifier.tune()__ method with default settings. The simulation will abort for each target when finding the optimal value, if not using plots (explained below), hence you might see the computation speed up towards the end. 

```python
C = Classifier(**kwargs)
C.train(data)
C.tune(data) # Tuning similarity within the default range of 0.01 - 1 in 0.01 increments, until optimal values are found

```

</details>

<details>
<summary>Tune selection</summary>
<br>

Add the argument __param\_name = 'selection'__ to the __Classifier.tune()__ method. To speed up the process, reduce the range with __param\_range = (start, stop, step)__. In this example, experience tells me the optimal value is somewhere in the range of 0.01 - 0.2 and so we don't have to spend time looking beyond that. You estimate one parameter at a time, hence the two separate calls to __tune()__. Only the first call needs to provide data to the Classifier. 

```python
C = Classifier(**kwargs)
C.train(data)
C.tune(data) # Tuning similarity within the default range of 0.01 - 1 in 0.01 increments
C.tune(param_name = 'selection', param_range = (0.01, 0.2, 0.01)) # Tuning selection within a reduced range, re-uses data from the previous call
```
</details>

<details>
<summary>Tune with plots</summary>
<br>

To plot the error curve, set __plot = True__. This requires external library [Plotext](https://github.com/piccolomo/plotext). Plots will be written to individual .html files in the _plots_ subfolder under your working directory. When plotting, the simulation will not end prematurely as it has to run through the entire range. You can still work in a reduced range with __param\_range__. 

```python
C.tune(data, plot = True) # Tuning similarity in the range of 0.01 - 1 in 0.01 increments and plotting the error curves
```
</details>


<details>
<summary>Use optimized criteria from tune()</summary>
<br>

Since __Classifier.tune()__ is a class method, the resulting "__criteria__" is saved to the class object. It's easy enough to export if you need to. 
```python
C = Classifier(**kwargs)
C.train(data)
C.tune(data) # Tuning similarity within the default range of 0.01 - 1 in 0.01 increments
C.tune(param_name = 'selection', param_range = (0.01, 0.2, 0.01)) # Tuning selection within a reduced range, re-uses data from the previous call
print(C.criteria)
```

This is the output. Again, only for the purpose of demonstration as it does not converge given the small sample size. 
```python
{'positive': {'similarity': 0.14, 'selection': 0.01}, 'service': {'similarity': 0.01, 'selection': 0.01}, 'suggestion': {'similarity': 0.37, 'selection': 0.04}, 'taste': {'similarity': 0.8, 'selection': 0.01}, 'spam': {'similarity': 1.0, 'selection': 0.01}}
```

</details>

<details>
<summary>Manually setting the thresholds</summary>
<br>

The concept of tuning the model relies on a validation dataset to compare the predictions with. If there is no validation data, you can optionally re-use the training data with the risk of overfitting. As a last resort however, you can also _predict_ the raw similarities, then sort the results descending with your favourite [spreadsheet software](https://github.com/saulpw/visidata), read the texts and manually identify the decision boundary. As you scroll through the results, there will be a dropoff point where texts are no longer relevant to the category of interest. You will need to consider both type 1 and type 2 errors (false positive and false negative respectively). Whether you want to minimize one of these or both will depend on your particular use case: "There are no solutions, only trade-offs". 

Exporting similarities:
```python
C = Classifier(**kwargs)
[*C.train(training_data)]
similarities = [*C.predict(prediction_data, discrete = False)] # Save similarities to variable
C.write_csv('my_similarities.csv', discrete = False) # Write similarities to disk
```

Using custom thresholds:
```python
custom_thresholds = { 'target1': { 'similarity': 0.64, 'selection': 0.1 }, 
                      'target2': { 'similarity': 0.7 }, # target2 uses the default selection
                      'target3': { 'selection': 0.2 } } # target3 uses the default cutoff (similarity)
kwargs = { 'targets': 'target1 target2 target3'.split(),
           'criteria': custom_thresholds }

C = Classifier(**kwargs)
print(C)
```

Output:
```bash
❯ python3 testrun4.py
Classifier(model='sentence-transformers/all-MiniLM-L6-v2', text_column='text', targets=['target1', 'target2', 'target3'], id_columns=[], confounders=[], ignore=[], default_selection=0.01, default_cutoff=0.5, criteria={'target1': {'similarity': 0.64, 'selection': 0.1}, 'target2': {'similarity': 0.7}, 'target3': {'selection': 0.2}}, mutually_exclusive=False, n_decimals=2, n_dims=384, trust_remote_code=False, show_progress_bar=True)
```


</details>

</details>


### Portability: save and load models

<details>
<summary>Expand</summary>

Saved models are typically a few kilobytes on disk. At the time of writing, only the training stage is saved. If you tune the model, the resulting __criteria__ will have to be stored elsewhere, eg in your python code. 

Saving:

```python
C = Classifier(**kwargs)
C.train(data)
C.to_npz('my_saved_model_file')
```

Loading:
```python
C = Classifier(**kwargs) # Initiate a new classifier
C.from_npz('my_saved_model_file') # Skip the training step by loading the previously trained parameters from disk
predictions = [*C.predict(some_new_data)]
```
</details>



### Split and combine records

<details>
<summary>Expand</summary>

Sentences are units of meaning, so splitting text into sentences will improve our predictions. If there is little punctuation in the text however, you may split by a fixed number of tokens and optionally with some overlap. These use cases are described below. The token splitting uses [regular expressions](https://docs.python.org/3/library/re.html) and word boundaries (\\b).

```python
from distinction import Classifier, split_records, combine_records

example_text = [{'text': 'This is the first sentence. Is this the second? Sentence number 3', 'binary_variable': 1},
                {'text': 'This text is a single sentence.', 'binary_variable': 0}] 
```
Note: in the code above we are hard-coding a binary variable from the start, where you would otherwise use the classifier for one or more targets once the text has been split. 

<details open>
<summary>Default settings</summary>
<br>

#### Split
By default, texts are split into sentences and then split into chunks when the sentence exceeds 384 tokens (the maximum for current sentence transformer models). The chunks are numbered by chunk\_id, with the last one being -1. These default settings should be used for semantic classifier pipelines, although a couple of other parameters are available to tamper with. 

__Note__: _split()_ assumes your data is not pre-encoded vectors. You cannot split an embedding the way you would split raw text. By the same token, _combine()_ automatically drops the vector column. 

```python
sentences = [*split_records(example_text)]
for sentence in sentences:
    print(sentence)
```

```bash
{'text': 'This is the first sentence. ', 'binary_variable': 1, 'doc_id': 0, 'sentence_id': 0}
{'text': 'Is this the second? ', 'binary_variable': 1, 'doc_id': 0, 'sentence_id': 1}
{'text': 'Sentence number 3', 'binary_variable': 1, 'doc_id': 0, 'sentence_id': 2}
{'text': 'This text is a single sentence.', 'binary_variable': 0, 'doc_id': 1, 'sentence_id': 0}
```

#### Combine

```python
results = [*combine_records(sentences, binary_targets = ['binary_variable'])]
for result in results:
    print(result)
```

Back to the original shape, with a document id added to each record. 
```bash
{'doc_id': 0, 'text': 'This is the first sentence. Is this the second? Sentence number 3', 'binary_variable': 1}
{'doc_id': 1, 'text': 'This text is a single sentence.', 'binary_variable': 0}
```

</details>


<details>
<summary>Max sequence length</summary>
<br>

#### Split
If needed, you can set a different max number of tokens. Since whitespace and punctuation count as tokens, set max\_sequence\_length to double the number of words you want. This can also be used to produce n-grams. The two examples below differ with respect to the argument __per\_sentence__. 

##### Option 1: Fixed number of tokens
```python
sentences = [*split_records(example_text, per_sentence = False, max_sequence_length = 8)]
for sentence in sentences:
    print(sentence)
```

```bash
{'text': 'This is the first', 'binary_variable': 1, 'doc_id': 0, 'sentence_id': 0}
{'text': ' sentence. Is this the', 'binary_variable': 1, 'doc_id': 0, 'sentence_id': 1}
{'text': ' second? Sentence number 3', 'binary_variable': 1, 'doc_id': 0, 'sentence_id': 2}
{'text': '', 'binary_variable': 1, 'doc_id': 0, 'sentence_id': 3}
{'text': 'This text is a', 'binary_variable': 0, 'doc_id': 1, 'sentence_id': 0}
{'text': ' single sentence.', 'binary_variable': 0, 'doc_id': 1, 'sentence_id': 1}
```

##### Option 2: Max number of tokens per sentence
```python
sentences = [*split_records(example_text, per_sentence = True, max_sequence_length = 8)]
for sentence in sentences:
    print(sentence)
```

```bash
{'text': 'This is the first', 'binary_variable': 1, 'doc_id': 0, 'sentence_id': 0, 'chunk_id': 0}
{'text': ' sentence. ', 'binary_variable': 1, 'doc_id': 0, 'sentence_id': 0, 'chunk_id': -1}
{'text': 'Is this the second', 'binary_variable': 1, 'doc_id': 0, 'sentence_id': 1, 'chunk_id': 0}
{'text': '? ', 'binary_variable': 1, 'doc_id': 0, 'sentence_id': 1, 'chunk_id': -1}
{'text': 'Sentence number 3', 'binary_variable': 1, 'doc_id': 0, 'sentence_id': 2}
{'text': 'This text is a', 'binary_variable': 0, 'doc_id': 1, 'sentence_id': 0, 'chunk_id': 0}
{'text': ' single sentence.', 'binary_variable': 0, 'doc_id': 1, 'sentence_id': 0, 'chunk_id': -1}
```

#### Combine

There are no special considerations for combining records with respect to max\_sequence\_length. The code below uses default settings, as in the previous section. 
```python
results = [*combine_records(sentences, binary_targets = ['binary_variable'])]
for result in results:
    print(result)
```

```bash
{'doc_id': 0, 'text': 'This is the first sentence. Is this the second? Sentence number 3', 'binary_variable': 1}
{'doc_id': 1, 'text': 'This text is a single sentence.', 'binary_variable': 0}
```

</details>

<details>
<summary>Overlap</summary>
<br>

#### Split
An alternative strategy, possibly inferior to splitting by punctuation, is to split text by number of tokens with overlap. The example code below splits the text into chunks of no more than 10 tokens (5 words), with an overlap of 3 tokens (typically 2 words and one whitespace/punctuation). If using the overlap parameter, you must remember to use it when combining the texts back together again (see below). 
```python
sentences = [*split_records(example_text, per_sentence = False, max_sequence_length = 8, overlap = 3)]
for sentence in sentences:
    print(sentence)
```
```bash
{'text': 'This is the first', 'binary_variable': 1, 'doc_id': 0, 'sentence_id': 0}
{'text': 'the first sentence. Is ', 'binary_variable': 1, 'doc_id': 0, 'sentence_id': 1}
{'text': '. Is this the second', 'binary_variable': 1, 'doc_id': 0, 'sentence_id': 2}
{'text': 'the second? Sentence number ', 'binary_variable': 1, 'doc_id': 0, 'sentence_id': 3}
{'text': ' number 3', 'binary_variable': 1, 'doc_id': 0, 'sentence_id': 4}
{'text': 'This text is a', 'binary_variable': 0, 'doc_id': 1, 'sentence_id': 0}
{'text': 'is a single sentence.', 'binary_variable': 0, 'doc_id': 1, 'sentence_id': 1}
{'text': ' sentence.', 'binary_variable': 0, 'doc_id': 1, 'sentence_id': 2}
```

#### Combine

Note the _overlap_ argument below. 
```python
results = [*combine_records(example_text, overlap = 3, binary_targets = ['binary_variable'])]
for result in results:
    print(result)
```

```bash
{'doc_id': 0, 'text': 'This is the first sentence. Is this the second? Sentence number 3', 'binary_variable': 1}
{'doc_id': 1, 'text': 'This text is a single sentence.', 'binary_variable': 0}
```

</details>

</details>


### Aggregation options for binary targets

<details>
<summary>Expand</summary>
<br>

Depending on your use case, you may want to pick up the main tendency of the text or transient themes within it. For example, suggestions might be hidden in a sentence that is part of a longer text, whereas the general sentiment of a text might be better inferred by looking at the entire text. You can control this by how the binary variables are aggregated. 
Let's start with three examples relating to the restaurant reviews example above. The first text has 1 positive sentence, the second one has 2 and in the third one all 3 sentences are predicted to be positive. So which of these texts should we consider positive? 

```python

C = Classifier(**kwargs)
C.targets = ['positive'] # Let's look at one variable only, for brevity
C.train(data)

example_texts = [
    dict(id = 200, message = "Fries are expensive and not that good to be honest. Please do mashed potatoes instead and bigger plates! I am so incredibly happy."),
    dict(id = 201, message = "The location is great. I love the food. I don't like the staff that much."),
    dict(id = 202, message = "Food is nice. Staff is nice. I love it!")
]

split_examples = [*split_records(example_texts, text_column = 'message', doc_id_column = 'id')]
predictions = [*C.predict(split_examples, discrete = True)]
for p in predictions:
    print(p)
```

```bash
{'id': 200, 'message': 'Fries are expensive and not that good to be honest. ', 'sentence_id': 0, 'positive': 0, 'spam': 0}
{'id': 200, 'message': 'Please do mashed potatoes instead and bigger plates! ', 'sentence_id': 1, 'positive': 0, 'spam': 0}
{'id': 200, 'message': 'I am so incredibly happy.', 'sentence_id': 2, 'positive': 1, 'spam': 0}
{'id': 201, 'message': 'The location is great. ', 'sentence_id': 0, 'positive': 1, 'spam': 0}
{'id': 201, 'message': 'I love the food. ', 'sentence_id': 1, 'positive': 1, 'spam': 0}
{'id': 201, 'message': "I don't like the staff that much.", 'sentence_id': 2, 'positive': 0, 'spam': 0}
{'id': 202, 'message': 'Food is nice. ', 'sentence_id': 0, 'positive': 1, 'spam': 0}
{'id': 202, 'message': 'Staff is nice. ', 'sentence_id': 1, 'positive': 1, 'spam': 0}
{'id': 202, 'message': 'I love it!', 'sentence_id': 2, 'positive': 1, 'spam': 0}
```

In this case, the _example\_texts_ data has an id column that we specified in the call to __split\_records()__. If no id column is given, a __doc\_id__ key is added to each record, to be able to use this data later on when combining the split records back together. It's not strictly necessary to use the _original\_data_ argument of __combine\_records()__ below, it's just slightly better in terms of computational performance. 


##### Aggregation "any" (default)
```python
results = [*combine_records(predictions, 
                            text_column = 'message', 
                            binary_targets = C.targets, 
                            original_data = example_texts, 
                            aggregation = 'any')]
for result in results:
    print(result)
```

This is the default aggregation strategy, which sets the bar quite low. The first text should not be considered positive. We don't know why the person is happy, although we can assume it has nothing to do with the food. Consider the fact that sometimes people will express irony as well. The second text is only partially positive. 

```bash
{'id': 200, 'message': 'Fries are expensive and not that good to be honest. Please do mashed potatoes instead and bigger plates! I am so incredibly happy.', 'positive': 1, 'spam': 0}
{'id': 201, 'message': "The location is great. I love the food. I don't like the staff that much.", 'positive': 1, 'spam': 0}
{'id': 202, 'message': 'Food is nice. Staff is nice. I love it!', 'positive': 1, 'spam': 0}
```


##### Aggregation "most" / "majority"

Looking at the majority of sentences, texts 2 and 3 become positive. 
```bash
{'id': 200, 'message': 'Fries are expensive and not that good to be honest. Please do mashed potatoes instead and bigger plates! I am so incredibly happy.', 'positive': 0, 'spam': 0}
{'id': 201, 'message': "The location is great. I love the food. I don't like the staff that much.", 'positive': 1, 'spam': 0}
{'id': 202, 'message': 'Food is nice. Staff is nice. I love it!', 'positive': 1, 'spam': 0}
```

##### Aggregation "all"

With the constraint that all sentences have to be 1, only the third text becomes positive. 

```bash
{'id': 200, 'message': 'Fries are expensive and not that good to be honest. Please do mashed potatoes instead and bigger plates! I am so incredibly happy.', 'positive': 0, 'spam': 0}
{'id': 201, 'message': "The location is great. I love the food. I don't like the staff that much.", 'positive': 0, 'spam': 0}
{'id': 202, 'message': 'Food is nice. Staff is nice. I love it!', 'positive': 1, 'spam': 0}
```


##### Aggregation "relative" / "share"

After concatenation, the texts are found to be 1/3, 2/3 and 100% positive. 

```bash
{'id': 200, 'message': 'Fries are expensive and not that good to be honest. Please do mashed potatoes instead and bigger plates! I am so incredibly happy.', 'positive': 0.3333333333333333, 'spam': 0.0}
{'id': 201, 'message': "The location is great. I love the food. I don't like the staff that much.", 'positive': 0.6666666666666666, 'spam': 0.0}
{'id': 202, 'message': 'Food is nice. Staff is nice. I love it!', 'positive': 1.0, 'spam': 0.0}
```

##### Aggregation "absolute" / "sum"

This counts the absolute number of instances. Documents 1, 2 and 3 have 1, 2 and 3 positive sentences respectively. 

```bash
{'id': 200, 'message': 'Fries are expensive and not that good to be honest. Please do mashed potatoes instead and bigger plates! I am so incredibly happy.', 'positive': 1, 'spam': 0}
{'id': 201, 'message': "The location is great. I love the food. I don't like the staff that much.", 'positive': 2, 'spam': 0}
{'id': 202, 'message': 'Food is nice. Staff is nice. I love it!', 'positive': 3, 'spam': 0}
```



##### Aggregation "mutually\_exclusive"

Use this when working with mutually exclusive data. The most common prediction wins. When two or more targets are equally common, the score becomes a tiebreaker. 
The actual code is shown as part of the section on [classifying mutually exclusive data](#set-up-and-use-a-Classifier-for-mutually-exclusive-binary-variables) below. 


</details>


### Set up prediction pipelines for continuous data streams

<details>
<summary>Expand</summary>

When your model is finalized, you can turn it into an end-to-end prediction function using the _Classifier.to_pipeline()_ method. Provide it with the same keyword argum