afp-sherpa 0.1.23
AFP Sherpa tools
Requires Python: >=3.8,<4.0

# AFP Sherpa tools

Command Line Interface for AFP specific tasks

## Install

- install python >= 3.8, pip and venv
```
sudo apt install python3
sudo apt install python3-pip
sudo apt install python3-venv
```
- create a virtual environment
```
mkdir venv

python3 -m venv venv
source venv/bin/activate
```

- install the afp-sherpa package
```
pip install afp_sherpa-x.y.z-py3-none-any.whl
```
## Print help and usage
```
afp-sherpa --help

Usage: afp-sherpa [OPTIONS] COMMAND [ARGS]...

  AFP Sherpa command line tool

Options:
  --log-file PATH                 Set specific log file
  --log-level [DEBUG|INFO|WARNING|ERROR]
                                  Set specific log level.  [default:
                                  LogLevel.INFO]
  --install-completion            Install completion for the current shell.
  --show-completion               Show completion for the current shell, to
                                  copy it or customize the installation.
  --help                          Show this message and exit.

Commands:
  annotate     Annotate content of a directory
  convert      Convert content of a directory containing xml news into...
  fingerprint  Recompute fingerprints of sherpa project directly in mongo
  knowledge    Generate lexicon files for Sherpa out of AFP KB dumps
  report       Generate and print quality report from directory...
  review       Select a subset of 'problematic' annotated documents for...
  selectcats   Select a subset of documents belonging to a list of rare...
  splitcats    Split converted corpora into categorized subdirectories
```

## Conversion XML -> JSON
```
afp-sherpa convert --help

Usage: afp-sherpa convert [OPTIONS] INPUT_PATH OUTPUT_PATH

  Convert content of a directory containing xml news into Sherpa json
  documents

Arguments:
  INPUT_PATH   Input directory to process  [required]
  OUTPUT_PATH  Output directory to write converted documents  [required]

Options:
  --batch-size INTEGER            Group json documents by batch  [default:
                                  1000]
  --max-doc-size INTEGER          Ignore documents which size if greater than
                                  this value  [default: 8000]
  --ignore-agenda / --no-ignore-agenda
                                  If true, try to autodetect agendas and
                                  ignore them  [default: ignore-agenda]
  --ignore-empty-subjects / --no-ignore-empty-subjects
                                  If true, try to autodetect documents without
                                  subjects and ignore them  [default: ignore-
                                  empty-subjects]
  --subjects-as-metadata TEXT     Comma-separated list of subjects to retrieve
                                  as metadata  [default: medtop,afpperson,afpo
                                  rganization,afplocation]
  --subjects-code / --no-subjects-code
                                  If true, use code as subject metadata. If
                                  false use name  [default: subjects-code]
  --mediatopics-as-categories / --no-mediatopics-as-categories
                                  whether to add mediatopics as categories
                                  [default: mediatopics-as-categories]
  --help                          Show this message and exit.

```

## Annotation with a given annotator
```
afp-sherpa annotate --help

Usage: afp-sherpa annotate [OPTIONS] INPUT_PATH OUTPUT_PATH

  Annotate content of a directory

Arguments:
  INPUT_PATH   Input directory to process  [required]
  OUTPUT_PATH  Output directory to write annotated documents  [required]

Options:
  --input-format [json|csv]     Input format of the documents  [default:
                                InputFormat.json]
  --sherpa-url TEXT             Sherpa URL  [env var: SHERPA_URL; required]
  --user TEXT                   Sherpa user  [env var: SHERPA_USER; required]
  --password TEXT               Sherpa password  [env var: SHERPA_PWD;
                                required]
  --timeout INTEGER             Sherpa endpoint timeout (in seconds)
                                [default: 100]
  --concurrent INTEGER          How many concurrent client threads are used in
                                parallel  [default: 16]
  --project TEXT                Sherpa project  [default: afp_iptc_root]
  --annotator TEXT              Sherpa annotator or plan  [default: afp_fr]
  --sample-ratio FLOAT          Sampling ratio of the input to annotate
                                [default: 1.0]
  --shuffle / --no-shuffle      If true, shuffle the doc before sampling
                                [default: no-shuffle]
  --aggregate / --no-aggregate  If true, agggregate annotations with existing
                                annotations  [default: aggregate]
  --help                        Show this message and exit.

```

## Generate quality report
```
afp-sherpa report --help

Usage: afp-sherpa report [OPTIONS] INPUT_PATH OUTPUT_PATH

  Generate and print quality report from directory containing list of
  annotated documents

Arguments:
  INPUT_PATH   Input directory containing annotated files to process
               [required]
  OUTPUT_PATH  Output directory to write xslx excel files  [required]

Options:
  --compare [METADATA|CREATOR]  Compare with metadata (no position) or
                                annotations created by different systems
                                [default: CompareWith.METADATA]
  --gold-tag TEXT               Column containing the gold labels  [default:
                                True]
  --help                        Show this message and exit.

```

## Develop

- install python >= 3.8, pip and venv as described before
- install poetry
```
curl -sSL https://raw.githubusercontent.com/python-poetry/poetry/master/get-poetry.py | python -
poetry --version
```
- unzip afp-sherpa src package
```
unzip afp-sherpa.zip
cd afp-sherpa
```

- install and build
```
poetry install
poetry build
```