Reproducing Google FunctionGemma 270M | Building a Lightweight Tool-Calling LLM from Scratch

This guide covers fine-tuning a small language model for function calling and tool integration, based on Google's FunctionGemma architecture. It explains how to take a compact base model—Google's Gemma 3 270M—and fine-tune it to handle API calls, structure JSON inputs, enforce operational guardrails, and execute locally on CPU hardware.

1. High-Level Architecture & Model Foundations

Large Language Models (LLMs) used for tool calling must process system instructions, extract schema-aligned arguments from natural language prompts, and trigger functional APIs. Google introduced FunctionGemma as a specialized, instruction-tuned variant of the Gemma 3 family designed for function execution.

Why a 270M Parameter Architecture?

Most tool-calling architectures rely on multi-billion parameter models (e.g., 7B, 70B+), which require dedicated cloud GPU infrastructure for real-time inference. Selecting a 270 Million (270M) parameter model provides several operational advantages:

1. Local CPU Execution: Fits well within standard system RAM (BFloat16 precision requires under 1 GB for model weights), eliminating GPU requirements for production inference.
2. Low Latency for Edge Workflows: Ideal for local Model Context Protocol (MCP) clients, terminal applications, and embedded agent workflows.
3. Task Specialization: While a 270M base model lacks general world-knowledge reasoning, instruction-tuning tailors its capacity toward schema adherence, syntax precision, and boundary enforcement.

2. The Tool-Calling Mental Model & Execution Loop

Tool-calling LLMs act as structured routing engines. Rather than directly answering queries from internal memory, the model evaluates whether a registered function can resolve the user's request.

Execution Modes: Automated vs. Manual Tool Calling

• Automated Tool Calling (automated_function_calling = True): The application layer intercepts the model's emitted function call tokens, executes the corresponding function on the host machine/network, feeds the response back into the context window, and requests a final natural language response. The end user only sees the final conversational response.
• Manual / Explicit Tool Calling (automated_function_calling = False): The model returns the raw tool invocation string and parameters directly to the caller, delegating execution and orchestration entirely to the client developer.

3. Dataset Preparation & Behavioral Scenarios

To build a reliable tool-calling model, the training dataset must explicitly model four core operational scenarios.

Scenario 1: No Access to External Tools

The system prompt explicitly states that no external functions are available. If the user asks a question requiring external capabilities, the model must decline rather than hallucinating a functional output.

• System Instruction: "You are a helpful assistant with no access to external functions."
• User Prompt: "Provide an example of a sentence that illustrates the use of future perfect tense."
• Expected Response: "I am a function calling model, so you will have to provide me with some external functions to execute tasks."

Scenario 2: Irrelevant or Out-of-Scope Tool

Functions are provided in the system prompt, but the user's query requests an action outside the scope of those definitions. The model must enforce system boundaries.

• System Instruction: Provides schema for search_movie(title).
• User Prompt: "Can you book a flight for me to New York?"
• Expected Response: "I'm sorry, but as an AI, I don't have the capability to book flights. My current function allows me to search a movie by its title. If you need information about a specific movie, feel free to ask."

Scenario 3: Multi-Turn Dialogue with Clarifying Questions

The context persists across conversation turns. If the prompt is ambiguous, the model asks clarifying questions before attempting a function call.

• System Instruction: Provides schema for generate_username().
• User Prompt: "Can you help me with something?"
• Assistant Turn 1: "Of course! What do you need help with?"
• User Prompt Turn 2: "I need you to book a flight for me from New York to London."
• Assistant Turn 2: "I'm sorry, but as an AI, I don't have the capability to book flights. My current function allows me to generate unique usernames."

Scenario 4: Multi-Turn Dialogue with Function Responses

The system instruction provides a schema, the user requests a matching action, the model emits a structured function call, the host executes it, and the model turns the raw output into a natural language response.

• System Instruction: Provides schema for create_contact(name, email).
• User Prompt: "I need to create a new contact for my friend John Doe. His email is johndoe@example.com."
• Model Intermediary Output:
  JSON

  Copy

  <call: create_contact{name: "John Doe", email: "johndoe@example.com"}>
  

• Host Application Response (Injected):
  JSON

  Copy

  <function_response: {status: "success", message: "Contact for John Doe created successfully."}>
  

• Final Model Response: "I have successfully created a new contact for your friend John Doe."

4. Dataset Pruning & Distribution Strategy

Training on raw, uncurated datasets like glare_function_calling_v2 (~112,011 raw instances) introduces significant redundancy. Repetitive refusal patterns can cause mode collapse, where the model over-indexes on negative responses.

Data Pruning Logic

• No Function / Irrelevant Tool Cases: Unfiltered datasets contain ~34,000 "no function" rows and ~14,000 "irrelevant tool" rows. Because the target behavior across these tens of thousands of rows boils down to the same boundary rejection output, we downsample these classes to 2,000 examples each. This retains the refusal pattern without overwhelming the context weights.
• Multi-Turn Function Executions: Scenario 4 contains rich variations in key-value extraction, syntax formatting, and multi-turn state propagation. We keep 50,000 examples from this category to maximize structural diversity.

Dataset Composition Breakdown

5. Fine-Tuning Execution & Resource Requirements

Fine-tuning was executed on a single GPU cloud instance using the cleaned 54,011-row dataset.

Convergence Analysis & Loss Trajectory

The model was fine-tuned using cross-entropy loss over token sequences. Below is the mathematical model of the cross-entropy loss $L$ evaluated over context token length $N$:

• Raw Step Loss: Exhibited high variance across individual sequence steps due to structural variations between standard natural language text tokens and explicit JSON syntax tokens.
• Smoothed Moving Average (per 500 Iterations): Showed smooth down-trending convergence, starting above $0.70$ and flattening out near $0.1450$ without showing signs of sequence degeneration or over-fitting.

6. Model Artifacts & Specifications

Following fine-tuning, the updated weights were pushed to the Hugging Face Hub under the model identifier Sarufi97/tooling-gemma-270m-it.

Technical Specs

• Base Model: google/gemma-3-270m
• Fine-Tuned Target ID: Sarufi97/tooling-gemma-270m-it
• Parameter Count: $270 \text{ Million}$ ($\approx 0.3\text{B}$)
• Tensor Format: BF16 (bfloat16)
• Vocabulary / Tokenizer: Standard Gemma 3 Tokenizer (uses special control tokens like <|endoftext|> to signal turn transitions).

7. Local CPU Inference Architecture & Implementation

Below is a complete, production-ready Python execution class using transformers and torch. This script loads the fine-tuned model directly onto a local CPU, builds system instructions, maintains chat history, and enforces stopping parameters.

import torch
from transformers import AutoTokenizer, AutoModelForCausalLM

class ToolingGemma:
    """
    Inference wrapper for Sarufi97/tooling-gemma-270m-it.
    Designed for zero-GPU, low-latency CPU execution.
    """
    def __init__(self, model_id: str = "Sarufi97/tooling-gemma-270m-it", system_instruction: str = ""):
        self.model_id = model_id
        self.system_instruction = system_instruction
        self.chat_history = []
        
        # Load tokenizer and model onto CPU
        print(f"Loading model weights from {self.model_id}...")
        self.tokenizer = AutoTokenizer.from_pretrained("google/gemma-3-270m")
        self.model = AutoModelForCausalLM.from_pretrained(
            self.model_id,
            torch_dtype=torch.bfloat16,
            device_map="cpu"
        )
        print("Model successfully loaded on CPU.")

    def _build_prompt(self, user_query: str) -> str:
        """
        Formats input sequence aligning with dataset training structure.
        """
        prompt = ""
        if self.system_instruction:
            prompt += f"System: {self.system_instruction}\n"
            
        for turn in self.chat_history:
            prompt += f"User: {turn['user']}\nAssistant: {turn['assistant']}\n"
            
        prompt += f"User: {user_query}\nAssistant:"
        return prompt

    def generate(self, user_query: str, max_new_tokens: int = 1000) -> str:
        prompt = self._build_prompt(user_query)
        inputs = self.tokenizer(prompt, return_tensors="pt")
        
        # Explicit stopping criteria configuration
        stop_token_id = self.tokenizer.eos_token_id
        
        with torch.no_grad():
            output_tokens = self.model.generate(
                **inputs,
                max_new_tokens=max_new_tokens,
                eos_token_id=stop_token_id,
                pad_token_id=stop_token_id,
                do_sample=False  # Deterministic execution for structured tool calling
            )
            
        # Extract response tokens beyond prompt length
        input_length = inputs["input_ids"].shape[1]
        generated_tokens = output_tokens[0][input_length:]
        response = self.tokenizer.decode(generated_tokens, skip_special_tokens=True).strip()
        
        # Update history context
        self.chat_history.append({"user": user_query, "assistant": response})
        return response

# Standard Initialization Verification
if __name__ == "__main__":
    discount_tool_schema = (
        "You are a helpful assistant with access to the following function:\n"
        "Function: calculate_discount(original_price: float, discount_percentage: float) -> float\n"
        "Description: Calculates the final price of an item given its original price and discount percentage."
    )
    
    agent = ToolingGemma(system_instruction=discount_tool_schema)

8. Evaluating Edge Cases & Guardrails in Practice

To evaluate model stability, we run two test cases using the calculate_discount tool schema.

Test Case 1: Out-of-Scope Request Processing

# Query outside defined capabilities
query_1 = "Can you please book a flight for me from New York to London?"
response_1 = agent.generate(query_1)
print(f"Agent Output:\n{response_1}")

Execution Output

"I'm sorry, but I'm unable to assist with that. My current capabilities are limited to calculating discounts on products. I don't have the ability to book flights or perform any external tasks."

• Evaluation: The 270M model correctly handles the boundary condition. It avoids hallucinating fake flight API parameters and clearly states its functional scope.

Test Case 2: In-Scope Structured Extraction

# Query matching defined tool capabilities
query_2 = "Calculate the discounted price for $100 at a discount of 30%."
response_2 = agent.generate(query_2)
print(f"Agent Output:\n{response_2}")

Execution Output

calculate_discount(original_price=100.0, discount_percentage=30.0)

• Evaluation: The model extracts the parameters, casts strings to numbers (100.0, 30.0), matches arguments to schema keys, and emits a structured function call token sequence.

9. Summary and Key Takeaways

• Lightweight Local Models Work for Tool Calling: A $270\text{M}$ parameter model like Gemma 3 270M can reliably extract tool arguments and enforce boundaries when instruction-tuned effectively.
• CPU Execution Compatibility: Running inference on standard $270\text{M}$ parameter FP16/BF16 weights consumes under $1 \text{ GB}$ of RAM, enabling local, low-latency API function routing without cloud GPUs.
• Dataset Quality > Raw Data Volume: Downsampling negative refusal instances from $\sim 48,000$ to $4,000$ total examples while keeping $50,000+$ rich multi-turn function interactions prevents mode collapse and optimizes training compute.
• Balanced System Guardrails: Proper dataset filtering teaches the model both sides of tool calling: emitting clean, schema-aligned JSON when a matching tool is present, and declining gracefully when a query falls out of scope.
• Efficient Fine-Tuning Scale: Fine-tuning $54,011$ targeted training samples on a single NVIDIA A100 GPU took under 2 hours ($1\text{h } 52\text{m}$), reaching a converged loss of $\approx 0.1450$.
• Clean System Boundary Isolation: Stripping function execution out of the LLM context and into an application-layer callback ensures deterministic execution, sandboxed code handling, and predictable user responses.