May 8, 2024

Fine-Tuned vs. Prompted: What Evaluation Data Actually Says About the Trade-offs

By the Confident AI Research Team · 13 min read

The decision to fine-tune is typically made on intuition or convention rather than measurement. Teams fine-tune because they believe the base model is not performing well enough, or because a competitor is fine-tuning, or because it sounds more rigorous than "just prompting." What they rarely have is a systematic comparison of fine-tuned performance against a well-engineered prompted baseline on their actual evaluation dataset. The comparison usually favors fine-tuning less than teams expect.

This is not an argument against fine-tuning. It is an argument for measuring before deciding. We have seen evaluation data from over 40 fine-tuning projects through the Confident AI platform, and the patterns are consistent enough to be useful guidance.

Where Fine-Tuning Consistently Wins

Fine-tuning shows clear evaluation advantages in specific, measurable situations:

Tone and format compliance. When the required output has a specific structure, length, or tone that is difficult to reliably enforce through prompting alone, fine-tuning on examples of correct output consistently improves compliance rates. A legal document generation application that requires a specific clause structure is a good fine-tuning candidate. In our data, format compliance scores improve by 15-25 percentage points after fine-tuning when the base prompted model struggles with format adherence.

Domain vocabulary and terminology. For highly specialized domains where the base model's training data coverage is thin, fine-tuning on domain-specific examples improves both the accuracy of technical terminology and the relevancy scores for domain-specific queries. Medical coding, legal citation formats, and industry-specific technical documentation all show consistent gains.

Latency-sensitive applications. Fine-tuned smaller models can match prompted larger models on application-specific tasks while running at 3-5x lower latency and cost. This is not a quality argument; it is a deployment economics argument. But for applications where response latency is a user experience requirement, fine-tuning a smaller model can be justified on latency grounds even when the quality difference is small.

Where Fine-Tuning Underperforms Expectations

The situations where fine-tuning disappoints in evaluation data:

Factual accuracy on long-tail knowledge. Fine-tuning injects knowledge into model weights, but it does not do so as reliably as RAG. For applications where the knowledge base changes frequently or contains long-tail facts the base model is unlikely to know, fine-tuning on examples often produces a model that gives confident-sounding but wrong answers. In our evaluation data, faithfulness scores for factual accuracy tasks drop after fine-tuning in approximately 30% of projects.

Distribution shift handling. Fine-tuned models show lower robustness to query distribution shifts than prompted models. The base model's general-purpose training gives it better generalization to queries that were not in the training set. If your query distribution shifts significantly after launch (which it typically does), a fine-tuned model often degrades faster than a prompted model on the new distribution.

Safety alignment. Fine-tuning can weaken safety alignment, including instruction following for out-of-scope requests. Red-team evaluation of fine-tuned models consistently shows higher jailbreak susceptibility than equivalent prompted models. Teams deploying customer-facing applications need to run red-team evaluation after every fine-tuning run, not just at initial deployment.

The Measurement Protocol: How to Compare Before Deciding

Before committing to fine-tuning, run this comparison protocol:

1. Build an evaluation dataset representative of your use case (50-100 cases minimum)
2. Establish a carefully engineered prompted baseline. This means systematic prompt engineering, not the first system prompt you wrote. Invest 4-8 hours in prompt optimization before comparing.
3. Measure the prompted baseline on your evaluation dataset. Record all metric scores.
4. Fine-tune on a training set that does not overlap with your evaluation dataset.
5. Measure the fine-tuned model on the same evaluation dataset.
6. Compare not just average scores but score distributions. A fine-tuned model that improves average faithfulness but increases the variance may produce more unpredictable behavior, which can be worse than a lower average score with lower variance.

In approximately 40% of the comparison projects in our dataset, the fine-tuned model did not outperform the well-engineered prompted baseline on the primary evaluation metrics. In those cases, teams saved 2-4 weeks of fine-tuning engineering work and the ongoing maintenance cost of managing a fine-tuned model version.

The Ongoing Evaluation Requirement After Fine-Tuning

If you do fine-tune, the evaluation requirements become more demanding, not less. As described in our RAG evaluation guide, a fine-tuned model that was evaluated once at training time is not a tested model. It is a tested configuration that will drift from that baseline as the provider updates the base model, your data changes, and your application's scope evolves. Fine-tuning is not a one-time quality investment; it is an ongoing commitment to re-evaluate every time the model or the application changes.