lxyuan/FunctionGemma-270M-banking77-router

FunctionGemma 270M BANKING77 Router

This is a full fine-tune of google/functiongemma-270m-it that turns an English banking request into one of ten structured support tool calls. It is a learning experiment, not a production banking system.

What was trained?

BANKING77 is normally a classification dataset with text, integer label, and human-readable label_text fields. This experiment does not add a classification head. It converts label_text into the expected assistant tool call and fine-tunes FunctionGemma to generate that native structure.

{
  "text": "My card is gone. I think it was stolen.",
  "label_text": "lost_or_stolen_card"
}

becomes a target call to handle_lost_or_stolen_card.

Tool schema

Every tool receives the original customer message. One complete schema is:

{
  "type": "function",
  "function": {
    "name": "handle_lost_or_stolen_card",
    "description": "Handle a card reported lost or stolen.",
    "parameters": {
      "type": "object",
      "properties": {
        "customer_message": {
          "type": "string",
          "description": "The original customer support message."
        }
      },
      "required": [
        "customer_message"
      ]
    },
    "return": {
      "type": "string"
    }
  }
}

Function	Intended request
handle_card_arrival	Handle questions about when a newly ordered card will arrive.
handle_card_not_working	Handle reports that a physical bank card does not work.
handle_cash_withdrawal_not_recognised	Handle an unrecognized cash withdrawal.
handle_change_pin	Handle requests to change a card PIN.
handle_compromised_card	Handle reports that card details may be compromised.
handle_lost_or_stolen_card	Handle a card reported lost or stolen.
handle_pending_card_payment	Handle a card payment that is still pending.
handle_terminate_account	Handle requests to close a bank account.
handle_transfer_not_received_by_recipient	Handle a transfer the recipient has not received.
handle_verify_my_identity	Handle questions about completing identity verification.

Use the model

import re

from transformers import AutoModelForCausalLM, AutoTokenizer

MODEL_ID = "lxyuan/FunctionGemma-270M-banking77-router"
TOOL_DESCRIPTIONS = {
    "handle_card_arrival": "Handle questions about when a newly ordered card will arrive.",
    "handle_card_not_working": "Handle reports that a physical bank card does not work.",
    "handle_cash_withdrawal_not_recognised": "Handle an unrecognized cash withdrawal.",
    "handle_change_pin": "Handle requests to change a card PIN.",
    "handle_compromised_card": "Handle reports that card details may be compromised.",
    "handle_lost_or_stolen_card": "Handle a card reported lost or stolen.",
    "handle_pending_card_payment": "Handle a card payment that is still pending.",
    "handle_terminate_account": "Handle requests to close a bank account.",
    "handle_transfer_not_received_by_recipient": "Handle a transfer the recipient has not received.",
    "handle_verify_my_identity": "Handle questions about completing identity verification."
}


def make_tool(name: str, description: str) -> dict:
    return {
        "type": "function",
        "function": {
            "name": name,
            "description": description,
            "parameters": {
                "type": "object",
                "properties": {
                    "customer_message": {"type": "string"},
                },
                "required": ["customer_message"],
            },
            "return": {"type": "string"},
        },
    }


tools = [make_tool(name, description) for name, description in TOOL_DESCRIPTIONS.items()]
model = AutoModelForCausalLM.from_pretrained(MODEL_ID, device_map="auto")
tokenizer = AutoTokenizer.from_pretrained(MODEL_ID)
message = "My card was stolen last night"
inputs = tokenizer.apply_chat_template(
    [
        {
            "role": "developer",
            "content": "You route customer requests by calling exactly one banking support tool.",
        },
        {"role": "user", "content": message},
    ],
    tools=tools,
    add_generation_prompt=True,
    return_tensors="pt",
    return_dict=True,
).to(model.device)
output = model.generate(**inputs, max_new_tokens=64, do_sample=False)
generated = tokenizer.decode(
    output[0, inputs["input_ids"].shape[1] :],
    skip_special_tokens=False,
)
match = re.search(
    r"<start_function_call>call:([a-z0-9_]+).*?<end_function_call>",
    generated,
    re.DOTALL,
)
if match is None:
    raise ValueError(f"No complete function call: {generated!r}")
print({"tool": match.group(1), "raw_call": match.group(0)})

The model selects a call; it does not execute the tool. Validate arguments and dispatch through an explicit allow-listed handler map.

Observed held-out examples

Customer input	First generated tool
I am sick of this damn company and want to close out my account.	handle_terminate_account
My card doesn't work.	handle_card_not_working
My card is gone I think it was stolen	handle_lost_or_stolen_card
There is a withdrawal that isn't mind in the app.	handle_cash_withdrawal_not_recognised
When will I get my card?	handle_card_arrival
Can i change my PIN at the ATM?	handle_change_pin

Input:  I am sick of this damn company and want to close out my account.
Output: <start_function_call>call:handle_terminate_account{customer_message:<escape>I am sick of this damn company and want to close out my account.<escape>}<end_function_call>

Loss function

This run uses TRL SFTTrainer with loss_type="chunked_nll". This is the standard causal-language-model next-token negative log-likelihood, or cross-entropy, computed in memory-saving chunks:

loss = mean(-log P(correct next token | previous tokens))

assistant_only_loss=False, and this is a conversational language-modeling dataset rather than a prompt-completion dataset. Therefore every non-padding token in the rendered developer prompt, tool declarations, user message, and assistant call contributes. Padding labels use -100 and are ignored. Exact generated tool accuracy is a separate application metric and is not the differentiable loss. Chunking does not alter this token selection; the label mask controls which tokens contribute, while loss_type controls how the same calculation is held in memory. See the TRL 1.12 SFT documentation.

Results

Metric	Value
Base-model exact first-tool accuracy	51.00%
Fine-tuned exact first-tool accuracy	97.00%
Generated evaluation examples	100
Final evaluation loss	0.0492
Mean training loss	0.0959

Training loss can continue falling while validation loss rises because the model becomes more confident on repeated training rows without improving equally on unseen rows. Cross-entropy can increase from a few confidently wrong tokens even when average token accuracy changes little.

TensorBoard event files, trainer_state.json, and training_metrics.json are included in this repository for inspecting the training curves and recorded results.

Training configuration

Setting	Value
Training examples	800
Evaluation examples	200
Epochs	4
Learning rate	5e-05
Effective batch size	8
Maximum sequence length	1024
Loss	chunked_nll, full rendered sequence except padding
Warmup	0.1 steps in the reference run (effectively no warmup)
Seed	42

Software:

• datasets==5.0.1
• tensorboard==2.20.0
• torch==2.11.0+cu128
• transformers==5.16.1
• trl==1.12.0

Precision and limitations

Load the saved FP32 checkpoint without forcing all weights to FP16. The verified FP32 Hub reload produced a valid function call, while forced pure FP16 on a T4 produced padding-only output. Training used FP16 autocast around FP32 master weights.

• Only ten BANKING77 intents are supported, not all 77.
• There is no out-of-scope or refusal route.
• Argument quality was not scored separately from tool-name selection.
• Ambiguous, adversarial, multilingual, or unrelated requests may route incorrectly.
• Do not use this model for financial decisions without production privacy, safety, monitoring, fallback, and human-review controls.

The BANKING77 mirror describes the dataset as CC BY 4.0. FunctionGemma weights remain subject to the Gemma terms.