Clausa disfluency detector: word-level disfluency detection for L2 English and Japanese

Clausa's disfluency detector marks each word of a speech transcript as fluent (0) or disfluent (1): filled pauses, repetitions, self-corrections, false starts, partial words and, in English, discourse markers such as "well", "you know" and "I mean" (labelled disfluent in Switchboard). It is part of Clausa, a tool that annotates clause-based utterance fluency in second-language (L2) English and Japanese speech, where it labels the words of each transcript before clause segmentation.

• XLMRobertaForTokenClassification, fine-tuned from xlm-roberta-base; 12 layers, about 277M parameters, two labels.
• Input: a list of words, lowercase, without punctuation. English: the words of the transcript. Japanese: words segmented with MeCab, as in training.
• Output: one label per word. The configuration has no label names: label 0 (LABEL_0) is fluent, label 1 (LABEL_1) is disfluent.

Intended use

Preprocessing of lowercase, punctuation-free transcripts of L2 English or Japanese speech before clause segmentation or disfluency counts, in research. Out of scope: assessing or grading individual speakers, other languages, text with punctuation or casing, audio, and commercial use (see the licence).

Usage

import torch
from transformers import AutoTokenizer, AutoModelForTokenClassification

repo = "oldriku/Clausa"
tokenizer = AutoTokenizer.from_pretrained(repo)
model = AutoModelForTokenClassification.from_pretrained(repo).eval()

words = "i think i think the the boy is um running to the to the park".split()
enc = tokenizer(words, is_split_into_words=True, return_tensors="pt", truncation=True)
with torch.no_grad():
    pred = model(**enc).logits.argmax(-1)[0].tolist()

# a word's label is the majority over its subword tokens (a tie counts as fluent)
votes = {}
for i, w in enumerate(enc.word_ids()):
    if w is not None:
        votes.setdefault(w, []).append(pred[i])
labels = [int(sum(v) > len(v) / 2) for v in (votes.get(i, [0]) for i in range(len(words)))]
print(" ".join(w for w, l in zip(words, labels) if l == 0))
# i think the boy is running to the park

This is how Clausa uses the model for English; for Japanese it adds rule-based repetition checks after the model. Input beyond the model's 512 subword tokens is cut off; Clausa labels the rest of a long transcript from overlapping windows of 150 words.

Training

Fine-tuned for 3 epochs (learning rate 2e-5, batch size 16, weight decay 0.01, seed 42) on 88,540 sentences with word-level labels, keeping the checkpoint with the best disfluent-token F1 on the synthetic validation split (evaluated every 1,000 steps):

Data	Sentences	Source
Synthetic English	20,000	sentences from the Brown and Gutenberg corpora (NLTK), with seven rule-based disfluency types injected
Synthetic Japanese	20,000	sentences from Tatoeba (CC BY 2.0 FR), segmented with MeCab, with eight rule-based disfluency types injected
Switchboard	48,540	disfluency annotations of the Switchboard Dialog Act Corpus (CC BY-NC-SA 3.0 as distributed by C. Potts; transcripts from LDC Switchboard-1 Release 2); training utterances are those containing a disfluency, split by conversation as in Kundu et al. (2022)

The injected types are filled pauses, word and phrase repetitions, self-corrections, false starts, partial words and function-word corrections (articles and prepositions in English, particles in Japanese); Japanese also has reading false starts (a kana or mispronounced prefix before a word). Labels for the synthetic sentences come from aligning each disfluent sentence with its fluent original (the subsequence method of Kundu et al., 2022).

Evaluation

Test set	Precision	Recall	F1
Synthetic English (2,000 held-out sentences)	.984	.989	.987
Synthetic Japanese (2,000 held-out sentences)	.995	.999	.997
DisfluencySpeech (Wang & Herremans, 2024): 240 test utterances whose transcripts align; a word is disfluent if the clean transcript drops it	.882	.619	.727

A model trained on the synthetic data alone reached F1 .540 on DisfluencySpeech; adding Switchboard mostly taught it native discourse markers.

Limitations

• No quantitative test on real L2 speech in either language: the real speech tested is native English (DisfluencySpeech) only, and Japanese is evaluated on synthetic sentences only.
• Binary labels: the model does not say which kind of disfluency a word is.
• Punctuation attached to words (for example well,) lowers recall: remove punctuation first.
• The synthetic sentences come from written text and target Japanese-L1 learners of English and English-L1 learners of Japanese; other L1 backgrounds and spontaneous speech may differ.

Licence

CC BY-NC-SA 4.0: free for non-commercial use, with attribution, and adaptations under the same licence.

Attribution: Switchboard (Godfrey et al., 1992); the Switchboard Dialog Act Corpus (Jurafsky, Shriberg & Biasca, 1997; distributed by C. Potts, CC BY-NC-SA 3.0); Tatoeba sentences (CC BY 2.0 FR); Brown and Gutenberg texts via NLTK; XLM-R (Conneau et al., 2020; MIT); the labelling method of Kundu et al. (2022); evaluation on DisfluencySpeech (Wang & Herremans, 2024; Apache 2.0).

Citation

Please cite the article (details to be added on publication):

@article{clausa2026,
  author  = {Authors to be added},
  title   = {Automatic cross-lingual annotation of clause-based utterance fluency in {L2} speech},
  journal = {Research Methods in Applied Linguistics},
  year    = {2026},
  doi     = {to be added}
}

References

• Conneau, A., et al. (2020). Unsupervised cross-lingual representation learning at scale. ACL 2020, 8440–8451.
• Godfrey, J. J., Holliman, E. C., & McDaniel, J. (1992). SWITCHBOARD: Telephone speech corpus for research and development. ICASSP 1992.
• Jurafsky, D., Shriberg, E., & Biasca, D. (1997). Switchboard SWBD-DAMSL shallow-discourse-function annotation coders manual. University of Colorado.
• Kundu, R., Jyothi, P., & Bhattacharyya, P. (2022). Zero-shot disfluency detection for Indian languages. COLING 2022, 4442–4454.
• Wang, K., & Herremans, D. (2024). DisfluencySpeech: Single-speaker conversational speech dataset with paralanguage. arXiv:2406.08820.