laya-issue-triage

Laya fine-tuned to triage GitHub issues. It answers two typed questions about an issue in a single forward pass, with no text generation:

question	type	answers
issue_type	choice	bug, feature, question, docs
needs_more_info	noul	true / false

Results

Measured on 3868 issues from three repositories held out of training entirely (huggingface/transformers, facebook/react, microsoft/TypeScript), so these are generalization numbers, not memorization.

model	issue_type acc	issue_type macro-F1	needs_info acc
random	0.250	-	0.500
majority class	0.345	-	0.738
base laya, zero-shot	0.626	0.524	0.563
this model	0.650	0.627	0.734

Per-class F1 on issue_type:

class	F1
bug	0.641
feature	0.796
question	0.487
docs	0.585

Per-repository accuracy on issue_type:

repository	accuracy	n
facebook/react	0.724	908
huggingface/transformers	0.753	632
microsoft/TypeScript	0.551	1329

Median latency on a T4 during evaluation: 78.8 ms per issue.

Scope of these numbers

This model is evaluated only on GitHub issue triage. It has not been run on the LocalLLaMA/typed-decisions benchmark, so its scores are not comparable to numbers reported there for the base Laya checkpoints or for any other system. Different task, different label space, different data.

Usage

import laya

agent = laya.Agent("harikarthikmanyam/laya-issue-triage")
questions = {
    "issue_type": {
        "type": "choice",
        "instructions": "What kind of GitHub issue do `title` and `body` describe?",
        "criteria": {
            "bug": "something is broken: a crash, an error, wrong output, or a regression from behaviour that used to work",
            "feature": "a request for new functionality, an enhancement, or a proposal to change how something works",
            "question": "the author is asking how to use the project or why it behaves a certain way, not reporting a defect",
            "docs": "the documentation is missing, wrong, unclear, or needs an example"
        }
    },
    "needs_more_info": {
        "type": "noul",
        "instructions": "Must a maintainer ask the author of `body` for more information -- reproduction steps, a version number, logs, or a code sample -- before this issue can be worked on?"
    }
}

state = {"title": "Crash when opening a file with a BOM",
         "body": "v1.4.2 on Linux. Steps: open any UTF-8-BOM file, editor segfaults."}
result = agent.predict(state, questions)
print(result["answers"]["issue_type"]["choice"])

The question definitions above are part of the contract: this model was trained against those exact instructions and criteria strings, so changing them changes behaviour.

Training data

~7.7k closed issues carrying maintainer-applied labels, harvested from 14 public repositories and balanced across the four types. Labels come from the maintainers who triaged each issue, not from a teacher model.

Dataset, harvesting code and evaluation script: https://github.com/manyamkarthik/laya-issue-triage

Limitations

• English issue text only; use laya-multilingual as a base for other languages.
• Issue text is truncated to roughly 900 characters, so decisions rest on the title and opening paragraphs.
• needs_more_info is supervised by maintainer labels such as needs-repro, which projects apply inconsistently; it is a weaker signal than issue_type.
• Confidences are uncalibrated unless you fit temperatures (see the notebook).