Fine-Tuning vs. Prompt Engineering vs. RAG:
Choosing the Right Approach

Once you’ve picked a base LLM, the next question is almost always the same: how do I get it to do my task well? There are three main levers available, and picking the right one — or the right combination — makes a bigger difference to cost and quality than picking the underlying model.

Three levers for adapting an LLM — often combined rather than used in isolation. (Image by author)

Prompt engineering: the first thing to try

Prompt engineering means carefully designing the instructions, examples, and structure of what you send the model, without changing the model itself. It costs nothing beyond the tokens spent, requires no training pipeline, and can be iterated on in seconds. Techniques like few-shot examples, chain-of-thought instructions, and explicit output formatting can meaningfully change a model’s behavior. The catch is that it’s fundamentally limited by the model’s existing knowledge and the size of its context window — you can’t teach it facts it was never trained on, and you’re constrained by how much context you can afford to send with every request.

RAG: giving the model access to information it doesn’t have

When the problem is missing or outdated knowledge rather than missing skill, retrieval-augmented generation is usually the better fix. Instead of trying to cram everything into the prompt or the model’s weights, RAG retrieves relevant information at query time and inserts it into the context. It’s particularly well suited to fast-changing information, large private document collections, or situations where you need traceability — being able to point at the exact source a claim came from. The trade-off is architectural complexity: you now need a retrieval pipeline, an embedding model, and a vector store, and the whole system’s quality is bottlenecked by how good that retrieval step is.

Fine-tuning: changing the model itself

Fine-tuning updates the model’s weights on a curated dataset of examples, actually changing how it behaves rather than what it’s told at inference time. This is the right tool when the goal is a consistent style, tone, or output format; a specialized skill the base model doesn’t have; or lower latency and cost from a smaller, specialized model that no longer needs a lengthy prompt to behave correctly. It’s also the most expensive and slowest option — requiring labeled data, compute for training, and a pipeline for evaluating and redeploying whenever the underlying base model changes. Techniques like LoRA (low-rank adaptation) have made fine-tuning cheaper by updating only a small number of additional parameters rather than the full model, but it’s still a heavier lift than the other two options.

A practical decision process

1. Start with prompt engineering — it’s fast to test and often gets you further than expected.
2. If the model lacks up-to-date or private knowledge, add RAG rather than trying to prompt your way around it.
3. Only reach for fine-tuning once you’ve hit a wall that prompting and retrieval can’t solve — usually a need for consistent behavior, format, or specialized skill at scale.

In practice, production systems frequently combine all three: a fine-tuned model for domain-specific tone and formatting, RAG for grounding answers in current data, and careful prompt engineering to tie the two together.

Conclusion

None of these three approaches is strictly superior — they solve different problems. Prompt engineering addresses instruction and formatting; RAG addresses missing knowledge; fine-tuning addresses behavior and specialization. Understanding which problem you actually have is most of the battle in choosing the right lever to pull.