slrdecisionmerger 0.1.2
A Python toolkit to divide BibTeX files among reviewers and merge their decisions for Systematic Literature Reviews
Requires Python: >=3.11

# SLR Decision Merger

SLR Decision Merger is a Python toolkit to assist researchers in conducting Systematic Literature Reviews (SLRs) by dividing BibTeX files among reviewers and then later merging their decisions. It streamlines the process of managing reviewer assignments and consolidating review outcomes.

## Features

- **Paper Division**: Randomly divide papers from BibTeX files among multiple reviewers
- **Configurable Assignment**: Specify the reviewers and how many reviewers should review each paper

## Installation

### 1. **Using `pip` (Recommended)**

Ensure you have [Python](https://www.python.org/downloads/) installed (version 3.7 or higher is recommended).

```bash
pip install slrdecisionmerger
```
### 2. **Installing from Git** 

If you want the latest development version:

```bash
pip install git+https://gitlab.rakshitmittal.net/rmittal/slrdecisionmerger.git
```

### 3. **Setting Up a Virtual Environment (Optional but Recommended)** 
Creating a virtual environment helps manage dependencies and avoid conflicts.

```bash
python -m venv venv
source venv/bin/activate  # On Windows: venv\Scripts\activate
```

Then proceed with the installation steps above.

## Configuration

SLR Decision Merger uses a configuration file (`config.json`) to manage settings. The idea is that each SLR project has a unique configuration file. Ensure that you create and properly configure this file before running the tool. Example `config.json`

```json
{
  "reviewers": ["alice", "bob", "james", "john"],
  "reviewers_per": 2, 
  "cache_directory": "/absolute/path/to/directory",
  "division_directory": "/absolute/path/to/directory",
  "review_directory": "/absolute/path/to/directory",
  "report_directory": "/absolute/path/to/directory"
}
```

```json
{
  "reviewers": 4,
  "reviewers_per": 2, 
  "cache_directory": "/absolute/path/to/directory",
  "division_directory": "/absolute/path/to/directory",
  "review_directory": "/absolute/path/to/directory",
  "report_directory": "/absolute/path/to/directory"
}
```

### Configuration Fields

- **`reviewers`**: `Union[ int, List[str] ]` : Either the total number of anonymous reviewers, or a list of names of reviewers.
- **`reviewers_per`**: `int` : Number of reviewers that should review each paper.
- **`cache_directory`** : `str` : Absolute path to directory containing the bib files to be distributed among reviewers. 
- **`division_directory`** : `str` : Absolute path to directory where reviewer assignment bib-files will be created.
- **`review_directory`** : `str` : Absolute path to directory containing bib-files with reviewer evaluations.
- **`report_directory`** : `str` : Absolute path to directory where generated reports will be saved.

**Note**: `reviewers_per` cannot be greater than number of `reviewers`.

## Usage

### Basic Command

```bash
python -m slrdecisionmerger --config config.json [options]
```

### Available Options

- **`--config`**: Path to the configuration JSON file (default: `config.json`)
- **`--divide`**: Divide papers among reviewers
- **`--merge`**: Merge reviewer decisions and generate reports
- **`--debug`**: Enable debug-level logging
- **`--merge_strat`**: Merge strategy. One of "conservative", "liberal", "consensus"

### Examples

1. **Divide Papers Among Reviewers**

```bash
python -m slrdecisionmerger --config config.json --divide
```

2. **Divide Papers with Debug Logging**

```bash
python -m slrdecisionmerger --config config.json --divide --debug
```

3. **Merge Reviewer Decisions with Consensus strategy**

```bash
python -m slrdecisionmerger --config config.json --merge --merge_strat consensus
```

## Decision Merging

After reviewers have evaluated their assigned papers, the `--merge` command consolidates their decisions. The tool identifies a reviewer's decision based on the `groups` field of a BibTeX entry. If the group is `relevant`, the paper is considered **accepted**; otherwise, it is considered **rejected**. It is assumed that the reviwers used (JabRef)[https://www.jabref.org/] to put their decisions (each reviewer-specific file generated by divide has automatically added JabRef annotations at the end).

### Merge Strategies (`--merge_strat`)

You can define how the tool handles agreements and disagreements using one of three strategies:

*   **`conservative`**: A paper is accepted if **at least one** reviewer marked it as `relevant`. This strategy maximizes recall, ensuring no potentially relevant paper is missed.
*   **`liberal`**: A paper is accepted only if **no reviewers** marked it as non-relevant (e.g., `OFF_TOPIC`, `NO_PEER_REV`). This ensures that only papers with unanimous or partial positive consensus are included.
*   **`consensus`** (Default): This is the most balanced strategy.
    *   A paper is accepted if it has at least one `relevant` vote and **no** non-relevant votes.
    *   A paper is rejected if it has only non-relevant votes.
    *   If a paper receives both `relevant` and non-relevant votes, it is marked as a **conflict**.

### Generated Files

Based on the chosen strategy, the merge process generates the following files in your `report_directory`:

*   **`accepted.bib`**: A BibTeX file containing all papers that were accepted according to the merge strategy.
*   **`conflicted.bib`** (Consensus strategy only): A BibTeX file containing all papers that had conflicting reviews.
*   **`conflicts.csv`** (Consensus strategy only): A detailed CSV report of all conflicts. It includes the paper's metadata (`title`, `author`, `year`, `doi`, `url`) and a column for each reviewer showing their specific decision (`accept` or `reject`).
*   **`conflict_matrix.csv`** (Consensus strategy only): A CSV file showing a symmetric matrix of the number of disagreements between each pair of reviewers. This is useful for analyzing inter-rater reliability.

## License 
[MIT License](LICENSE)