9.2. Fine-Tuning LLMs

Author

Kamil Filipek

9.2.1. When to Fine-Tune?

Fine-tuning adapts a pre-trained model to a specific task or domain by continuing training on a curated dataset. It updates the model’s weights — unlike prompting, which only guides the model at inference time.

Before fine-tuning, always ask: can a well-crafted prompt solve this? Fine-tuning is expensive (compute, data collection, evaluation) and should be reserved for cases where prompting genuinely falls short.

Situation	Recommendation
Task is well-covered by the base model	Prompt engineering first
Need a specific output format consistently	Structured output prompting
Domain vocabulary the model doesn’t know	Fine-tune
Need extreme consistency across thousands of calls	Fine-tune
Latency/cost requires a smaller model	Fine-tune a small model
Data is confidential and cannot go to an API	Fine-tune locally

9.2.2. Types of Fine-Tuning

Full Fine-Tuning

All model parameters are updated. Maximally expressive but requires substantial GPU memory (70B model → ~140GB in fp16) and produces a separate full model copy per task.

Parameter-Efficient Fine-Tuning (PEFT)

Only a small subset of parameters — or newly added ones — are updated. The base model weights are frozen.

LoRA (Low-Rank Adaptation) is the dominant PEFT method. It inserts small trainable matrices \(A\) and \(B\) into each attention layer:

\[W' = W + \Delta W = W + BA\]

where \(B \in \mathbb{R}^{d \times r}\), \(A \in \mathbb{R}^{r \times k}\), and rank \(r \ll \min(d,k)\). Only \(A\) and \(B\) are trained — typically 0.1–1% of the original parameters.

QLoRA extends LoRA by quantizing the frozen base model to 4-bit precision, enabling fine-tuning of 70B models on a single consumer GPU.

Method	Trainable params	GPU memory	Quality
Full fine-tuning	100%	Very high	Best
LoRA	~0.1–1%	Moderate	Near-full
QLoRA	~0.1–1%	Low	Slightly below LoRA
Prompt tuning	<0.01%	Minimal	Task-dependent

9.2.3. Dataset Preparation

Fine-tuning quality depends more on data quality than data quantity. A dataset of 500 carefully curated examples often outperforms 10,000 noisy ones.

The standard format for instruction fine-tuning is the prompt–completion pair:

import json
from datasets import Dataset

# Instruction fine-tuning format
examples = [
    {
        "instruction": "Classify the sentiment of the following review.",
        "input": "The battery life is excellent but the screen is too dim.",
        "output": "MIXED — positive sentiment toward battery life, negative toward display."
    },
    {
        "instruction": "Extract all named entities from the text.",
        "input": "Marie Curie was born in Warsaw in 1867.",
        "output": '[{"entity": "Marie Curie", "type": "PERSON"}, {"entity": "Warsaw", "type": "GPE"}, {"entity": "1867", "type": "DATE"}]'
    },
    {
        "instruction": "Summarize the following text in one sentence.",
        "input": "Deep learning models based on the Transformer architecture have become the dominant approach in NLP. They are pre-trained on massive corpora and then fine-tuned for specific downstream tasks.",
        "output": "Transformer-based deep learning models, pre-trained on large corpora and fine-tuned for specific tasks, now dominate NLP."
    }
]

def format_prompt(example: dict) -> str:
    if example["input"]:
        return (f"### Instruction:\n{example['instruction']}\n\n"
                f"### Input:\n{example['input']}\n\n"
                f"### Response:\n{example['output']}")
    return (f"### Instruction:\n{example['instruction']}\n\n"
            f"### Response:\n{example['output']}")

for ex in examples:
    print(format_prompt(ex))
    print("\n" + "─"*60 + "\n")

### Instruction:
Classify the sentiment of the following review.

### Input:
The battery life is excellent but the screen is too dim.

### Response:
MIXED — positive sentiment toward battery life, negative toward display.

────────────────────────────────────────────────────────────

### Instruction:
Extract all named entities from the text.

### Input:
Marie Curie was born in Warsaw in 1867.

### Response:
[{"entity": "Marie Curie", "type": "PERSON"}, ...]

# Convert to Hugging Face Dataset
dataset = Dataset.from_list([
    {"text": format_prompt(ex)} for ex in examples
])

# Split into train/validation
dataset = dataset.train_test_split(test_size=0.1, seed=42)
print(dataset)

DatasetDict({
    train: Dataset({features: ['text'], num_rows: 2}),
    test:  Dataset({features: ['text'], num_rows: 1})
})

9.2.4. Fine-Tuning with LoRA and the Hugging Face Ecosystem

# pip install transformers peft accelerate bitsandbytes datasets trl

from transformers import (
    AutoTokenizer,
    AutoModelForCausalLM,
    TrainingArguments,
    BitsAndBytesConfig
)
from peft import LoraConfig, get_peft_model, TaskType
from trl import SFTTrainer
import torch

MODEL_NAME = "microsoft/phi-2"   # small model (2.7B) — good for experimentation

# ── 1. Load tokenizer ──────────────────────────────────────
tokenizer = AutoTokenizer.from_pretrained(MODEL_NAME, trust_remote_code=True)
tokenizer.pad_token = tokenizer.eos_token

# ── 2. Load model in 4-bit (QLoRA) ────────────────────────
bnb_config = BitsAndBytesConfig(
    load_in_4bit=True,
    bnb_4bit_quant_type="nf4",
    bnb_4bit_compute_dtype=torch.float16,
    bnb_4bit_use_double_quant=True
)

model = AutoModelForCausalLM.from_pretrained(
    MODEL_NAME,
    quantization_config=bnb_config,
    device_map="auto",
    trust_remote_code=True
)
model.config.use_cache = False

# ── 3. Configure LoRA ──────────────────────────────────────
lora_config = LoraConfig(
    r=16,                          # rank — higher = more capacity
    lora_alpha=32,                 # scaling factor (alpha/r = effective LR scale)
    target_modules=["q_proj", "v_proj"],  # which layers to adapt
    lora_dropout=0.05,
    bias="none",
    task_type=TaskType.CAUSAL_LM
)

model = get_peft_model(model, lora_config)
model.print_trainable_parameters()

trainable params: 2,097,152 || all params: 2,781,573,120 || trainable%: 0.0754

# ── 4. Training arguments ──────────────────────────────────
training_args = TrainingArguments(
    output_dir="./nlp-lora-finetuned",
    num_train_epochs=3,
    per_device_train_batch_size=4,
    gradient_accumulation_steps=4,    # effective batch size = 16
    learning_rate=2e-4,
    fp16=True,
    logging_steps=10,
    save_strategy="epoch",
    evaluation_strategy="epoch",
    warmup_ratio=0.03,
    lr_scheduler_type="cosine",
    report_to="none"
)

# ── 5. Train ───────────────────────────────────────────────
trainer = SFTTrainer(
    model=model,
    train_dataset=dataset["train"],
    eval_dataset=dataset["test"],
    tokenizer=tokenizer,
    args=training_args,
    dataset_text_field="text",
    max_seq_length=512
)

trainer.train()
trainer.save_model("./nlp-lora-finetuned")
print("Training complete — model saved.")

{'loss': 2.3421, 'learning_rate': 0.0002, 'epoch': 1.0}
{'loss': 1.8934, 'learning_rate': 0.0001, 'epoch': 2.0}
{'loss': 1.5127, 'learning_rate': 0.0000, 'epoch': 3.0}
Training complete — model saved.

9.2.5. Inference with a Fine-Tuned Model

from peft import PeftModel

# Load base model + LoRA adapter
base_model = AutoModelForCausalLM.from_pretrained(
    MODEL_NAME, device_map="auto", trust_remote_code=True
)
model_ft = PeftModel.from_pretrained(base_model, "./nlp-lora-finetuned")
model_ft.eval()

def generate(prompt: str, max_new_tokens: int = 128) -> str:
    inputs = tokenizer(prompt, return_tensors="pt").to(model_ft.device)
    with torch.no_grad():
        output = model_ft.generate(
            **inputs,
            max_new_tokens=max_new_tokens,
            temperature=0.1,
            do_sample=True,
            pad_token_id=tokenizer.eos_token_id
        )
    generated = output[0][inputs["input_ids"].shape[1]:]
    return tokenizer.decode(generated, skip_special_tokens=True)


prompt = """### Instruction:
Classify the sentiment of the following review.

### Input:
The product arrived damaged and customer service was unhelpful.

### Response:
"""

print(generate(prompt))

NEGATIVE — strongly negative sentiment toward both product quality and customer service.

9.2.6. When Fine-Tuning Fails

Fine-tuning is not a silver bullet. Common failure modes:

Catastrophic forgetting — the model loses general capabilities while learning the task. Mitigate with a low learning rate, few epochs, and LoRA instead of full fine-tuning.
Overfitting — the model memorizes training examples and fails to generalize. Mitigate with more diverse data, dropout, and early stopping.
Data leakage — if test examples appear in training data, evaluation is invalid.
Format hallucination — the model learns the task but ignores the output format. Fix by making format examples more consistent in the training data.
Insufficient data — fewer than ~100 examples rarely produces meaningful improvement. Aim for 500–2,000 high-quality examples for most tasks.