DataTalks.Club Podcast Wiki

Guide

LLM Tools: How to Choose the Right Stack for Real Products

A practical guide to choosing LLM tools for production workflows, including model APIs, open-source models, RAG, evaluation, agents, observability, and cost trade-offs.

Related Wiki Pages

LLM Production Patterns LLM Evaluation Workflows Search, RAG, and Knowledge Systems Retrieval-Augmented Generation Agent Engineering MLOps and DataOps Responsible AI and Governance

LLM tools sit around a language model and cover model access, serving, retrieval, and prompts. They also cover evaluation and observability, with agent frameworks and review workflows beside those pieces.

The DataTalks.Club podcast archive treats them as product infrastructure, not as a shopping list. Start from the workflow you want to improve, then add only the tools needed to make that workflow reliable.

That framing shows up across the archive. In Deploying LLMs in Production, Meryem Arik ties model choice to control and privacy. She also covers fine-tuning and hidden API changes. She keeps latency and cost in the same decision.

In Practical LLM Engineering and RAG, Hugo Bowne-Anderson starts from prompts and gold tests. He adds failure analysis, logs, and traces, then moves from RAG to agents.

In Building Agentic AI Systems, Ranjitha Kulkarni adds tools, memory, and knowledge stores. She also adds context engineering and agent tests. For the reference version of the archive synthesis, use LLM Production Patterns.

Start With The Job, Not The Framework

Start by naming the job you expect the system to do. Drafting assistants and transcript search bots need different controls. Support answer generators, coding assistants, and on-call agents need their own controls too. Sandra Kublik makes that product distinction in LLM Value Creation: at 10:00 she covers text generation, embeddings, and semantic search as business use cases. The 23:29 chapter puts hallucinations and brand safety inside human review.

That means “best LLM tools” is the wrong first question. For content workflows, prompt iteration and review may matter more than agent orchestration. For a knowledge assistant, start with retrieval and chunking. Add citations and answer evaluation.

For an operations assistant, Ranjitha’s 22:50 and 24:59 chapters in Building Agentic AI Systems show that logs and metrics become part of the tool stack. Remediation actions and integration abstractions matter too. That’s where LLM tools connect to MLOps and software engineering, not only prompting.

Choose The Model Layer By Control And Risk

Teams often use hosted model APIs to prototype. Meryem separates that prototype path from production model ownership in Deploying LLMs in Production. At 16:48, she compares open-source and API models through control, privacy, and fine-tuning. At 18:46, she warns that a provider can change an API-backed model in ways that affect behavior.

At 49:44 and 51:35, she returns to the practical deployment tradeoff. GPT-3.5 or GPT-4 APIs can speed up prototypes, but running self-hosted open-source models raises latency, cost, and hardware questions.

Use that split when you choose between an API, a hosted open-source endpoint, or self-hosting. Model ownership matters when the feature handles private data or regulated workflows. It also matters when behavior must stay stable across releases. If the feature is still a low-risk prototype, an API may buy speed. Use RAG vs Fine-Tuning for the deeper adaptation decision, because model choice usually sits beside the choice to retrieve current knowledge or adapt model behavior.

Treat RAG As Search Infrastructure

Teams often get value from retrieval-augmented generation before agents. Hugo says at 44:26 in Practical LLM Engineering and RAG that chunking and embeddings can create quick business wins when teams already have useful documents. He then gets specific: at 48:20 he discusses fixed-length chunks, sliding windows, and context rot. At 53:34 he uses a Gmail API plus RAG email assistant as a practical build.

Atita Arora gives the search version in Modern Search Systems. At 30:38, she frames RAG as retrieval plus generation. Her transcript chatbot example moves from Whisper transcripts to chunking and overlap between 35:49 and 42:49. She also covers embedding models and vectorization. Prompt context and citations matter too.

That places RAG in Search, RAG, and Knowledge Systems. It also connects to embeddings and vector databases, not only to model selection.

Add Evaluation Before You Swap Tools

Evaluation tells you whether a tool change made the product better. That change may be a new prompt, vector database, or model. It may also be a chunking strategy or agent framework.

Hugo’s 13:56 generator-evaluator check in Practical LLM Engineering and RAG shows one lightweight path. His 23:00 chapter adds representative gold tests, and the 26:43 chapter turns failures into categories before teams prioritize retrieval fixes. At 27:38, he adds logs and traces so builders can debug what changed.

Other guests put the same idea into production terms. Meryem’s 53:34 and 56:39 chapters in Deploying LLMs in Production connect gold-standard examples and output-driven evaluation. She also separates classification metrics, generative metrics, and human judgment. Atita’s 48:09 and 50:52 chapters in Modern Search Systems add multi-level RAG metrics, offline tests, and human-in-the-loop review. Use LLM Evaluation Workflows and Evaluation when you need the broader testing map.

Move To Agents When The Workflow Needs Actions

Agents are useful when the system must choose tools, call APIs, plan steps, or act across a workflow. Ranjitha defines that boundary at 11:00 in Building Agentic AI Systems. She describes an agent as a system with autonomy, an objective, and LLM reasoning. At 12:31 she adds orchestration, tools, memory, and knowledge stores. At 21:21 and 32:48 she shows why context engineering includes metadata, wrappers, and careful LLM inputs rather than only a longer prompt.

Don’t choose an agent framework just because the product uses an LLM. In the same episode, Ranjitha separates cases where RAG is enough from cases where agents are needed at 37:39. Hugo makes a similar move in Practical LLM Engineering and RAG: he starts with RAG at 44:26, then discusses tool calls at 50:19. At 56:21 he uses a practical agent framework. It starts with problem definition, a small start, data, and evaluation.

When you do need agents, connect the framework choice to Agent Engineering. Test the system, not only the prompt.

Keep Cost, Latency, And Review Visible

LLM tools can hide operational costs until the product has users. Bartosz Mikulski keeps those costs visible in Production AI Engineering. At 25:13 he covers in-context learning and examples. At 28:16 and 30:00 he connects prompt evaluation and prompt compression to formatting, examples, tokens, and cost tradeoffs. At 31:45 he discusses prompt caching and model efficiency.

His coding-assistant chapters at 42:05 and 44:38 also show that tool choice changes day-to-day engineering work, not only backend architecture.

Human review is another production control, especially when outputs can affect customers, brand voice, or decisions. Sandra’s 23:29 chapter in LLM Value Creation connects hallucinations, brand safety, and editorial curation. Atita and Meryem put human review into RAG and generative evaluation. For sensitive systems, connect tool selection to Responsible AI and Governance and Security, not only to model benchmarks.

A Practical LLM Tools Stack

For most teams, a practical LLM tools stack grows in this order:

This order isn’t a universal recipe.

The DataTalks.Club LLM episodes show the same structure from several directions:

For “llm tools”, the article-level answer is direct. Pick tools that make one workflow grounded, testable, and observable. Then keep it affordable before you add more automation.