File size: 991 Bytes
77d22a6
 
6d1a001
 
 
77d22a6
 
 
 
 
6d1a001
 
 
 
 
 
 
 
 
 
 
 
d131aa3
6d1a001
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
d131aa3
 
 
 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
---
title: Perplexity Lenses
emoji: 🌸
colorFrom: pink
colorTo: blue
sdk: streamlit
app_file: app.py
pinned: false
---

# Installation:
Requires Python >= 3.7 and < 3.10
```
pip install -r requirements.txt
```

# Web App:
The app is hosted [here](https://huggingface.co/spaces/edugp/perplexity-lenses). To run it locally:  
```
python -m streamlit run app.py
```

# CLI:
The CLI with no arguments defaults to running mc4 in Spanish.  
For full usage:
```
python cli.py --help
```
Example: Running on 1000 sentences extracted from Spanish OSCAR docs specifying all arguments:
```
python cli.py \
    --dataset oscar \
    --dataset-config unshuffled_deduplicated_es \
    --dataset-split train \
    --text-column text \
    --language es \
    --doc-type sentence \
    --sample 1000 \
    --dimensionality-reduction umap \
    --model-name distiluse-base-multilingual-cased-v1 \
    --output-file perplexity.html
```
# Tests:
```
python -m unittest discover -s ./tests/ -p "test_*.py"
```