README.md

---
library_name: transformers
tags:
- verifcation
- policy_compliance
- factchecking
- instruction_following
base_model:
- akjindal53244/Llama-3.1-Storm-8B
---

# NAVI verifiers (Nace Automated Verification Intelligence)

## Model Details

### Model Description

NAVI (Nace Automated Verification Intelligence) is a solution for policy alignment verification designed to review various types of text against documents and policies, and identify violating content. It is specifically optimized for enterprise applications requiring compliance verification for automated text generation. To push policy verification in the open-source community, we release NAVI-small-preview, an open-weights version of the model we have deployed on the platform. NAVI-small-preview is centered around verifying specifically assitant outputs against some policy documents. The full solution is available on [NAVI platform](naviml.com).

- **Developed by:** Nace
- **Model type:** Policy Alignment Verifier
- **Language(s) (NLP):** English
- **License:** [More Information Needed]
- **Finetuned from model:** akjindal53244/Llama-3.1-Storm-8B

## Uses

### Direct Use

NAVI-small-preview is used for verifying compliance of assistant outputs with enterprise policy documents. It processes policies and identifies any contradictions, inconsistencies and violations.

### Downstream Use

May be integrated into enterprise-level AI systems to automate compliance checks in diverse industries such as legal, finance, and retail.

### Out-of-Scope Use

NAVI is not designed for general-purpose factuality verification or tasks unrelated to policy compliance.

## Bias, Risks, and Limitations

While NAVI excels in policy compliance verification, it may face challenges with unrealistic policy scenarios or contexts outside its training data scope.

### Recommendations

We recommend to use the model only as a generative classifier outputting the class label, any other outputs are not accounted for during training.

## How to Get Started with the Model

The verifier takes the assistant response and document context as inputs to generate a classification label. There are three possible classes: Compliant, Noncompliant, and Irrelevant. For long enterprise documents, we recommend setting up chunk-based vector search for selecting most relevant chunks from the document. Below are the prompts and sample code to launch:

```

```

## Training Details

### Training Data

NAVI was trained on a mix of real-world policy documents and synthetic interactions between a user and an assistant, it includes diverse, realistic, and complex policy examples across multiple industries.

### Training Procedure

#### Preprocessing [optional]

Policies are processed into embeddings for effective decision-making.

#### Training Hyperparameters

- **Training regime:** [More Information Needed]

## Evaluation

### Testing Data, Factors & Metrics

#### Testing Data

We have manually collected Policy Alignment Verification dataset consisting of across different use cases for evaluation. We open source the public subset of the dataset. Here we diclose the performance on the public subset, containing 125 examples across six industry-specific scenarios: AT&T, Airbnb, Cadence Bank, Delta Airlines, Verisk, and Walgreens.

#### Factors

Evaluation focuses on policy compliance within multi-policy, multi-document contexts.

#### Metrics

F1 score was used to measure performance, prioritizing detection of noncompliance cases.

### Results

NAVI-small-preview achieved an F1 score of 86.8%, outperforming all evaluated alternatives except full-scale NAVI. We evaluate against general-purpose solutions like Claude and Open AI models, as well as some guardrails focusing on factchecking to demonstrate a clear distinction of policy verification from the more common factchecking. Full performance metrics:

| Model                    | F1 Score | Avg Latency (ms) |
|--------------------------|----------|------------------|
| NAVI-small-preview      | 86.8     | -                | 
| NAVI                    | 90.4     | 387.62           |
| AWS Bedrock Guardrail   | 76.5     | 342.79           |
| Azure Groundedness      | 71.2     | 232.71           |
| NeMo (GPT-4o)           | 71.2     | 2669.68          |
| GPT-4o (few-shot)       | 75.0     | 904.46           |
| Sonnet 3.5 (few-shot)   | 75.5     | 2926.69          | ()

#### Summary

NAVI demonstrates clear advantages in both performance and efficiency, making it a robust solution for policy compliance verification.

## Model Card Contact

[More Information Needed]