12.8.7 — Frameworks and When to Skip Them

Two Cartwise teams built LLM features in the same quarter. The marketing team built a product-description writer with a popular framework, and had a demo running in an afternoon. Three months later, a model upgrade made descriptions longer, and nobody could find where the actual prompt text lived: it was assembled from defaults spread across four layers of framework classes. The support team built "Ask Cartwise" with nothing but the provider's SDK and about 400 lines of their own code. Their first demo took a week. When the same model upgrade arrived, they opened one file, read the prompt, ran the evals from 12.8.6, and fixed it in an hour. Then the support team built the refund agent, which has to pause for human approval and resume hours later, survive server restarts and record every step, and they adopted a framework for that part, because writing it by hand would have taken weeks.

Both decisions were right. This page explains what LLM frameworks actually do, how the best-known family, the "Lang" ecosystem of LangChain, LangGraph and LangSmith, came to exist and is designed, how Microsoft's Prompt Flow compares, and a simple rule for when a framework helps and when it gets in the way.

1. What a framework offers

Everything in Chapters 12.8.1 to 12.8.6 can be written with a provider's SDK and plain code: calls, prompts, structured output, tools, streaming and evals. Frameworks package recurring pieces of that work. Roughly, they offer five things:

• One interface over many model providers, so switching providers is a configuration change.
• Building blocks for common steps: prompt templates, output parsers, connectors to document stores and vector databases, text splitters for retrieval.
• Composition: ways to connect steps into chains, pipelines and graphs.
• State and control flow for long-running, multi-step processes: memory, branching, loops, pausing for approval, resuming after a failure.
• Observability: recording every prompt, response, tool call, token count and latency, so that runs can be inspected and evaluated.

The first two save some typing. The last two solve genuinely hard problems. That difference is the key to the rest of this page.

2. Prompt chaining and "shot" types are different things

One point of confusion is worth clearing up first, because the two ideas are often mixed together. Zero-shot, one-shot and few-shot describe what is inside a single prompt: how many worked examples it includes, as 12.8.2 explained. Prompt chaining describes how several calls are arranged: the output of one becomes part of the input of the next, such as "extract the complaint", then "group complaints into themes", then "write the summary". The two are independent: each call in a chain can be zero-shot or few-shot. Frameworks are mostly about chaining and the control flow around it; shot types are just prompt content.

3. The Lang ecosystem: how it grew and how it is built

The Lang ecosystem over time. Its centre of gravity moved from convenience wrappers around single calls to the genuinely hard parts: stateful orchestration and observability.

LangChain: chains and integrations (2022)

Harrison Chase released LangChain as an open-source Python library in October 2022, weeks before ChatGPT made LLMs mainstream. At the time, building anything with a model meant writing HTTP requests, retries, output parsing and retrieval by hand, for every project. LangChain offered ready-made pieces for all of it:

• wrappers for many model providers;
• prompt templates;
• "chains" that connected steps;
• "agents" that let a model choose tools;
• "memory" classes for conversation history;
• hundreds of connectors to document loaders and vector databases.

It grew faster than almost any open-source project before it, because it let people build impressive demos in an afternoon.

In 2023 it introduced the LangChain Expression Language, LCEL, a way to compose steps with the | pipe character, borrowed from the Unix shell of Chapter 2.8, where a | b means "feed the output of a into b":

from langchain_anthropic import ChatAnthropic
from langchain_core.prompts import ChatPromptTemplate
from langchain_core.output_parsers import StrOutputParser

prompt = ChatPromptTemplate.from_template(
    "Write a 40-word product description for: {product}")       # (1)
llm = ChatAnthropic(model="claude-opus-5-5")                      # (2)
chain = prompt | llm | StrOutputParser()                          # (3)
print(chain.invoke({"product": "1.5 L stainless steel electric kettle"}))   # (4)

Line (1) is a template with a placeholder. Line (2) wraps the model behind LangChain's common interface. Line (3) composes the three steps with pipes: fill the template, call the model, extract the text. Line (4) runs the chain. Each piece also supports streaming, batching and asynchronous calls through the same interface, which is the real benefit of the common interface.

LangChain also attracted serious criticism, and it is worth understanding because it applies to any framework. Its early abstractions were deep: a single call could pass through many layers of classes, so when something went wrong you had to read the framework's internals to see what had actually been sent to the model. Default prompts were hidden inside classes, so the most important text in the system was the hardest to see. And the API changed quickly, breaking working code. Many teams adopted it for speed, hit those walls, and replaced it with a few hundred lines of their own. The ecosystem has since moved in a clear direction in response: thinner building blocks, the prompt visible and owned by you, and the effort concentrated where it adds the most.

LangSmith: seeing what happened (2023)

LangSmith, launched in 2023, is a hosted service for the observability problem. It records every run as a trace: each prompt sent, each response, each tool call and result, timings and token counts, nested in the order they happened, so a failing conversation can be opened and inspected step by step. It also stores datasets of test cases and runs evaluations against them, the work of 12.8.6. It works with LangChain and LangGraph automatically and with other code through a small library. This is the part of the ecosystem that is hardest to argue against, because the alternative, debugging a non-deterministic system from logs that only show the final answer, is miserable. (Comparable tools exist from other companies and as open source, often built on the OpenTelemetry tracing standard from Chapter 10.10.)

LangGraph: stateful workflows and agents (2024)

LangGraph, released in January 2024, is the most important piece for production work, because it abstracts something genuinely hard. It models a multi-step process as a graph. The nodes are the steps, such as a model call, a tool call or a plain function. The edges say which step comes next, and that can depend on the result. A shared state object is passed from step to step and updated along the way. Loops, branches and pauses are part of the graph, instead of being hand-written code.

Most importantly, it can save the state after every step to a checkpointer, which is just a place to keep it: memory, a file or a database. That gives you three things that are painful to build by hand:

• A process can pause and resume hours later. For example, it can wait for a person to approve a refund.
• It can recover after a crash, starting again from the last saved step.
• Every step's state is recorded, so you can see exactly what happened when debugging.

from typing import TypedDict
from langgraph.graph import StateGraph, START, END
from langgraph.checkpoint.memory import MemorySaver

class RefundState(TypedDict):
    order_id: str
    eligible: bool
    approved: bool | None

def check(state):   return {"eligible": returns.eligible(state["order_id"])}   # (1)
def refund(state):  payments.refund(state["order_id"]); return {}

g = StateGraph(RefundState)
g.add_node("check", check)
g.add_node("refund", refund)
g.add_edge(START, "check")
g.add_conditional_edges("check",                                          # (2)
    lambda s: "refund" if s["eligible"] else END)
g.add_edge("refund", END)
app = g.compile(checkpointer=MemorySaver(), interrupt_before=["refund"])  # (3)

cfg = {"configurable": {"thread_id": "order-4471"}}                       # (4)
app.invoke({"order_id": "4471", "eligible": False, "approved": None}, cfg)  # runs check, pauses
# … hours later, after a person approves in the support tool:
app.invoke(None, cfg)                                                     # (5) resumes at "refund"

Line (1) defines each step as a plain function that reads the state and returns the fields it changes. Line (2) adds a branch: after checking, go to the refund step only if the order is eligible. Line (3) compiles the graph with a checkpointer and an instruction to pause before the refund step, which is the human approval gate of Chapter 12.10.8. Line (4) names this run, so its saved state can be found again. Line (5) resumes the paused run from its checkpoint after approval. (In production, the in-memory saver is replaced by a database-backed one, so pauses survive restarts.) In 2025 LangChain and LangGraph reached version 1.0, and LangChain's own agents are now built on LangGraph underneath.

4. Azure Prompt Flow, and how it differs

People sometimes ask whether Microsoft's Prompt Flow is "the same as LangChain". They overlap in purpose and differ in shape. Prompt Flow, introduced by Microsoft in 2023 as part of its Azure machine-learning and AI platform (and released as an open-source toolkit), represents an LLM application as a visual flow: a directed graph of nodes, where each node is a prompt, a model call or a piece of Python, drawn and edited in a browser or in VS Code. It puts particular weight on the surrounding workflow: running a flow over a dataset, running evaluation flows that score the outputs, comparing versions, and deploying a flow as a managed endpoint inside Azure.

	LangChain / LangGraph	Prompt Flow
Form	code libraries	visual flows backed by files
Strength	flexibility, stateful agents	evaluation and deployment in Azure
Suits	engineering teams in any cloud	teams standardised on Azure, mixed audiences

The table's cells are the short version; the practical difference is this. LangGraph is a programming library: everything is code, which suits engineers and any cloud. Prompt Flow is a structured, visual way to build and evaluate flows inside one cloud platform, which suits organisations already standardised on Azure and teams where non-engineers need to read and adjust a flow. Microsoft's AI tooling has been changing quickly, with newer agent frameworks taking over parts of this role, so check the current status before starting anything new on it.

5. The rest of the landscape, in one line each

• LlamaIndex (started in late 2022 as GPT Index): focused on retrieval, with strong tools for loading, splitting and indexing documents for RAG.
• Semantic Kernel (Microsoft, 2023): an SDK for adding LLM features to C#, Python and Java applications; Microsoft has since been consolidating it with its other agent tooling.
• DSPy (Stanford, 2023): treats prompts as parameters to be optimised automatically against an evaluation metric, instead of written by hand.
• Provider agent SDKs: model providers publish their own libraries for building agents, such as Anthropic's Claude Agent SDK and OpenAI's Agents SDK, which package a tested agent loop, tools and context management for that provider's models.
• Vercel AI SDK: a TypeScript library for streaming LLM output into web interfaces, popular for the front-end work of 12.8.5.

6. When to use one, and when to skip it

The rule the two Cartwise teams ended up with is short:

Write the prompts and the model calls yourself. Use a framework for state and for observability.

The reasoning is the distinction from section 1. A model call is one HTTP request with a list of messages; wrapping it in layers of abstraction saves little and hides the most important thing in your system, the prompt. You need to see, version and evaluate that text directly, as 12.8.2 insisted. The genuinely hard parts are different: keeping state across many steps, pausing for people and resuming, recovering from crashes without repeating actions, and recording every step for debugging and evaluation. Those take weeks to build well and are exactly what LangGraph-style orchestration and LangSmith-style tracing provide.

So, in practice:

• A single call or a short fixed chain (summarise a review, route an email, extract fields): the provider's SDK and your own small wrapper. No framework.
• Retrieval-augmented answers: your own code, perhaps with a retrieval library for loading and splitting documents.
• A long-running agent that pauses, resumes, branches and must survive failures: an orchestration framework such as LangGraph, or a provider's agent SDK or managed agent service.
• Any of the above in production: a tracing and evaluation tool, from day one.

Whatever you use, keep your own thin interface between your business code and the framework or provider (the single ask function of 12.8.1), so that replacing one later is a contained change. This field moves fast; the frameworks of 2026 will not all exist in 2030, but the problems they solve, state, recovery, approval and observability, will.

Recall

• Frameworks offer provider abstraction, building blocks, composition, state and control flow, and observability; the last two solve genuinely hard problems, the first two mostly save typing.
• Zero-shot and few-shot describe examples inside one prompt; prompt chaining arranges several calls so each feeds the next. They are independent ideas.
• LangChain (October 2022) offered chains and integrations, then LCEL's pipe composition; its deep, prompt-hiding abstractions drew justified criticism. LangSmith (2023) traces runs and evaluates datasets. LangGraph (January 2024) models workflows as graphs of nodes, edges and shared state with checkpoints, enabling pause, resume, recovery and human approval.
• Prompt Flow is Microsoft's visual flow tool with strong evaluation and Azure deployment; LangGraph is a code library for any cloud. LlamaIndex, Semantic Kernel, DSPy, provider agent SDKs and the Vercel AI SDK fill other niches.
• Write prompts and model calls yourself; use frameworks for stateful orchestration and observability; keep your own thin interface so any of them can be replaced.

Self-test: What does LangGraph's checkpointer make possible that a simple loop cannot do safely? Why is hiding prompts inside framework classes a real cost, not a style preference? Which Cartwise feature would you build with no framework, and which with one?

Next: 12.9.1 Why RAG, and the Smallest Version — "Ask Cartwise" now calls models reliably; the biggest remaining problem is that the model does not know Cartwise's 40 policy documents, and retrieval-augmented generation is how it learns to answer from them.