Enterprise Reflex V1

Model: yasserrmd/enterprise-reflex-v1
Base Model: answerdotai/ModernBERT-base
Language: English
Architecture: Dynamic enterprise action scorer
Status: Research Prototype / Pre-Production Candidate

Enterprise Reflex V1 is a lightweight enterprise decision model designed to operate as a System-1 action-ranking layer in front of larger reasoning models, agent frameworks, and enterprise automation systems.

Given a request, structured enterprise state, optional context, and a dynamic runtime set of candidate actions, the model ranks the available actions, can abstain with NO_ACTION, and uses calibrated confidence to decide whether the request should stay in the fast System-1 path or be escalated to a more capable System-2 model or a human.

Enterprise Reflex is not a generative LLM and not a fixed intent classifier. Candidate actions are supplied dynamically at inference time, allowing the model to score actions it was not explicitly trained to recognize by identifier alone.

Why Enterprise Reflex

Enterprise agent systems increasingly expose large action spaces: create or update records, approve or reject workflow steps, search internal systems, trigger notifications, route work, invoke APIs, execute business operations, or abstain when no safe action is available.

Sending every request directly to a large reasoning model can be unnecessarily expensive and slow. Enterprise Reflex is designed to act as a compact decision layer:

Request + Enterprise State + Context + Candidate Actions
                         |
                         v
                 Enterprise Reflex
                         |
             +-----------+-----------+
             |                       |
             v                       v
     Confident System-1      Low confidence / NO_ACTION
          decision                    |
             |                        v
             v                System-2 / Human Review
          Execute

The objective is not to replace larger reasoning models. It is to reduce how often they are required.

What Changed in V1

V1 extends the original Enterprise Reflex prototype with targeted hard-training data focused on failure modes observed in V0.

The V1 hard-training set emphasizes:

same-domain sibling-action discrimination,
counterfactual state changes,
policy and workflow constraints,
semantic cross-domain collisions,
improved NO_ACTION boundaries,
unseen and renamed action identifiers.

The training strategy retained broad V0 enterprise coverage while adding targeted V1 hard examples so the model could improve difficult routing behavior without losing general enterprise performance.

Training Configuration

Item	Value
Base model	`answerdotai/ModernBERT-base`
Training samples	553,632 pairwise examples
Epochs	2
Total training steps	23,068
Warmup steps	1,384
Maximum sequence length	384
Objective	Binary request-action compatibility
Best-model metric	Validation F1
Reported hardware	NVIDIA A100

Input Representation

Enterprise Reflex scores each candidate action against a compact serialized request representation.

Request Side

{
  "request": "Release the approved supplier payment",
  "domain": "Finance",
  "state": {
    "payment_approved": true,
    "invoice_matched": true
  },
  "context": {}
}

Candidate Action Side

{
  "name": "finance.release_payment",
  "description": "Release an approved and validated supplier payment.",
  "family": "EXECUTE",
  "domain": "Finance"
}

Each candidate is scored independently. Compatibility margins are then calibrated and normalized across the runtime candidate set. NO_ACTION is added as an explicit abstention candidate.

Evaluation

V1 was evaluated at three levels:

pairwise request-action classification,
grouped candidate ranking,
a manually designed 100-case hard stress test.

These evaluations represent different levels of difficulty and should be interpreted separately.

Pairwise Test Results

V0 Test Distribution

Metric	Result
Accuracy	97.86%
Precision	97.97%
Recall	83.69%
F1	90.27%
ROC AUC	99.03%
Average Precision	94.85%

V1 Hard Test Distribution

Metric	Result
Accuracy	94.07%
Precision	85.31%
Recall	79.00%
F1	82.04%
ROC AUC	96.88%
Average Precision	89.24%

Combined Test Distribution

Metric	Result
Accuracy	97.67%
Precision	97.01%
Recall	83.36%
F1	89.67%
ROC AUC	98.94%
Average Precision	94.57%

Grouped Action-Ranking Results

Grouped evaluation measures whether the correct action is ranked highest within the full runtime candidate set.

V0 Grouped Test

Metric	Result
Groups	5,948
Top-1	98.30%
Top-3	99.98%
MRR	0.9913
`NO_ACTION` Precision	94.02%
`NO_ACTION` Recall	97.32%
`NO_ACTION` F1	95.64%

V1 Hard Grouped Test

Metric	Result
Groups	500
Top-1	88.60%
Top-3	99.40%
MRR	0.9370
`NO_ACTION` Precision	84.75%
`NO_ACTION` Recall	90.09%
`NO_ACTION` F1	87.34%

Combined Grouped Test

Metric	Result
Groups	6,448
Top-1	97.55%
Top-3	99.94%
MRR	0.9871
`NO_ACTION` Precision	93.05%
`NO_ACTION` Recall	96.58%
`NO_ACTION` F1	94.78%

The V1 hard grouped split is intentionally more difficult than the broad V0 evaluation and should not be treated as the same distribution.

Selective System-1 / System-2 Routing

V1 uses calibrated confidence to determine whether a decision should remain in System-1 or be escalated.

For the reported run:

Selected System-2 threshold: 0.87
Validation System-1 coverage: 73.71%
Validation System-1 accuracy: 99.02%

NO_ACTION is always treated as a System-2 route.

This threshold is calibrated on the validation distribution and should be recalibrated for any materially different deployment domain.

100-Case Manual Hard Stress Test

A separate manual suite of 100 hard enterprise cases was used to stress behavior outside the easier validation distribution.

Overall Results

Metric	Result
Total cases	100
Correct	85
Incorrect	15
Raw Top-1 accuracy	85.00%
System-1 handled	66%
System-2 routed	34%
System-1 accuracy	87.88%
Unsafe System-1 failures	8

Performance by Category

Category	Tests	Accuracy
Cross-domain	12	83.33%
`NO_ACTION`	21	100.00%
Policy constraint	8	12.50%
Sibling action	32	87.50%
State sensitive	22	90.91%
Unseen action name	5	100.00%

The manual stress test is intentionally adversarial and significantly harder than the standard grouped benchmark.

What V1 Improved

Compared with V0 hard-test behavior, V1 improved both hard-decision accuracy and autonomous coverage.

Observed improvements include:

hard Top-1 accuracy increased from approximately 80% to 85%,
System-1 coverage increased from approximately 53% to 66%,
state-sensitive decisions improved substantially,
unseen action-name generalization remained strong,
NO_ACTION behavior improved on the manual hard suite,
broad V0 enterprise ranking performance remained largely intact.

The result supports the core Enterprise Reflex design: a lightweight model can perform useful dynamic enterprise action ranking while routing uncertain cases to a larger reasoner.

Current Limitation: Policy-Constrained Execution

The dominant V1 weakness is policy-sensitive action validity.

Several hard cases were semantically understood but executed incorrectly because state or policy should have blocked the action.

Observed failure patterns include:

releasing a payment without required approval,
provisioning privileged access without security approval,
deleting logs under legal or retention hold,
cancelling an order after a workflow state that prohibits cancellation,
granting physical access before mandatory induction is complete.

This indicates that V1 is currently stronger at answering:

Which action best matches this request?

than:

Is this action actually permitted under the current enterprise state and policy?

For this reason, V1 should not be used as the sole authority for autonomous high-impact enterprise execution.

Intended Use

Enterprise Reflex V1 is suitable for research and controlled enterprise-agent experiments such as:

action ranking,
dynamic tool selection,
top-k tool narrowing,
System-1 / System-2 routing,
agent handoff,
workflow recommendation,
shadow-mode decision analysis,
human-in-the-loop action suggestions,
enterprise action-space reduction before LLM reasoning.

Not Recommended For

V1 is not recommended as the sole decision layer for:

autonomous financial transactions,
privileged-access provisioning,
destructive security actions,
compliance-sensitive deletion,
irreversible workflow actions,
legal or regulatory decisions,
production execution without deterministic authorization and policy enforcement.

Recommended Production Architecture

Enterprise Reflex should be combined with deterministic controls.

Request + State + Candidate Actions
                |
                v
       Enterprise Reflex
                |
                v
        Ranked Action
                |
                v
 Policy / Authorization Engine
         /              \
      Allowed          Blocked
        |                 |
        v                 v
     Execute        System-2 / Human

The learned model provides decision intelligence. Authorization, policy, entitlement, retention, approval, and other hard enterprise controls should remain deterministic whenever possible.

Example Usage

import json
import torch
import numpy as np

from transformers import (
    AutoTokenizer,
    AutoModelForSequenceClassification,
)

MODEL_ID = "yasserrmd/enterprise-reflex-v1"
SYSTEM2_THRESHOLD = 0.87
MAX_LENGTH = 384

# Load the runtime calibration values saved with the model if available.
# The reported experiment used a calibrated temperature determined from
# the combined validation distribution.
TEMPERATURE = 1.0

tokenizer = AutoTokenizer.from_pretrained(MODEL_ID)
model = AutoModelForSequenceClassification.from_pretrained(MODEL_ID)

device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
model.to(device)
model.eval()


def softmax_np(x):
    x = np.asarray(x, dtype=np.float64)
    x = x - np.max(x)
    e = np.exp(x)
    return e / e.sum()


def request_text(request, domain="enterprise", state=None, context=None):
    return json.dumps(
        {
            "request": request,
            "domain": domain,
            "state": state or {},
            "context": context or {},
        },
        ensure_ascii=False,
        sort_keys=True,
    )


def action_text(action, default_domain="enterprise"):
    return json.dumps(
        {
            "name": action.get("name", ""),
            "description": action.get("description", ""),
            "family": action.get("family", "OTHER"),
            "domain": action.get("domain", default_domain),
        },
        ensure_ascii=False,
        sort_keys=True,
    )


def rank_actions(request, candidate_actions, state=None, context=None, domain="enterprise"):
    actions = list(candidate_actions)

    actions.append(
        {
            "name": "NO_ACTION",
            "description": "None of the available actions safely or correctly satisfy the request.",
            "family": "ABSTAIN",
            "domain": domain,
        }
    )

    left = request_text(
        request,
        domain=domain,
        state=state,
        context=context,
    )

    enc = tokenizer(
        [left] * len(actions),
        [action_text(a, default_domain=domain) for a in actions],
        padding=True,
        truncation=True,
        max_length=MAX_LENGTH,
        return_tensors="pt",
    ).to(device)

    with torch.no_grad():
        logits = model(**enc).logits

    margins = (logits[:, 1] - logits[:, 0]).float().cpu().numpy()
    probabilities = softmax_np(margins / TEMPERATURE)
    order = np.argsort(-probabilities)

    ranked = [
        {
            "action": actions[int(i)]["name"],
            "probability": float(probabilities[int(i)]),
        }
        for i in order
    ]

    top = ranked[0]

    system2_required = (
        top["action"] == "NO_ACTION"
        or top["probability"] < SYSTEM2_THRESHOLD
    )

    return {
        "decision": top["action"],
        "confidence": top["probability"],
        "system2_required": system2_required,
        "ranked_actions": ranked,
    }

Example

result = rank_actions(
    request="Release the supplier payment",
    domain="Finance",
    state={
        "invoice_matched": True,
        "payment_approved": True,
    },
    candidate_actions=[
        {
            "name": "finance.release_payment",
            "description": "Release an approved and validated supplier payment.",
            "family": "EXECUTE",
            "domain": "Finance",
        },
        {
            "name": "finance.create_invoice",
            "description": "Create an invoice record.",
            "family": "CREATE",
            "domain": "Finance",
        },
    ],
)

print(result)

Calibration Note

The reported confidence threshold was selected for the reported V1 experiment.

For a new deployment:

collect domain-specific validation data,
calibrate temperature,
determine an acceptable System-1 error rate,
select the confidence threshold for that environment,
validate policy-sensitive and destructive actions separately.

Do not assume that 0.87 is appropriate for every enterprise domain.

Research Status

Enterprise Reflex V1 should currently be considered a:

Research Prototype / Pre-Production Candidate

The model has demonstrated:

strong broad enterprise action ranking,
useful dynamic action selection,
effective abstention,
high Top-3 retrieval quality,
improved state-sensitive behavior,
improved System-1 coverage,
promising generalization to unseen action identifiers.

It has not yet demonstrated sufficient reliability for unrestricted autonomous enterprise execution.

The primary V2 research target is policy-aware action validity and confident wrong-action suppression.

V2 Direction

The next iteration should focus less on generic enterprise volume and more on targeted safety and state-validity examples:

policy counterfactuals,
approval-sensitive actions,
retention and legal-hold constraints,
authorization and entitlement state,
workflow-state legality,
semantic domain collisions,
hard negatives mined from confident V1 failures,
valid-action vs NO_ACTION boundary cases.

A likely architectural extension is to separate:

semantic action suitability,
state/policy validity,

before producing the final action confidence.

Datasets

Enterprise Reflex Dataset

yasserrmd/enterprise-reflex-dataset

Broad enterprise action-ranking data used for V0 and retained in V1 training.