DSPy is the sketchtoil for programming—rather than prompting—language models. It allows you to iterate speedy on produceing modular AI systems and provides algorithms for selectimizing their prompts and weights, whether you’re produceing plain classifiers, cultured RAG pipelines, or Agent loops.
DSPy stands for Declarative Self-improving Python. Instead of brittle prompts, you write compositional Python code and employ DSPy to direct your LM to dedwellr high-quality outputs. This lecture is a excellent conceptual introduction. Meet the community, seek help, or begin contributing via our GitHub repo and Discord server.
Getting Started I: Inslofty DSPy and set up your LM
You can genuineate by setting the OPENAI_API_KEY env variable or passing api_key below.
You can genuineate by setting the ANTHROPIC_API_KEY env variable or passing api_key below.
If you’re on the Databricks platcreate, genuineation is automatic via their SDK. If not, you can set the env variables DATABRICKS_API_KEY and DATABRICKS_API_BASE, or pass api_key and api_base below.
First, inslofty Ollama and begin its server with your LM.
Then, join to it from your DSPy code.
First, inslofty SGLang and begin its server with your LM.
> pip inslofty "sglang[all]"
> pip inslofty flashinfer -i https://flashinfer.ai/whl/cu121/torch2.4/
> CUDA_VISIBLE_DEVICES=0 python -m sglang.begin_server --port 7501 --model-path meta-llama/Llama-3.1-8B-Instruct
If you don’t have access from Meta to download meta-llama/Llama-3.1-8B-Instruct, employ Qwen/Qwen2.5-7B-Instruct for example.
Next, join to your local LM from your DSPy code as an OpenAI-compatible endpoint.
In DSPy, you can employ any of the dozens of LLM providers aided by LiteLLM. Sshow trail their directions for which {PROVIDER}_API_KEY to set and how to write pass the {provider_name}/{model_name} to the produceor.
Some examples:
anyscale/mistralai/Mistral-7B-Instruct-v0.1, withANYSCALE_API_KEYtogether_ai/togethercomputer/llama-2-70b-chat, withTOGETHERAI_API_KEYsageproducer/, withAWS_ACCESS_KEY_ID,AWS_SECRET_ACCESS_KEY, andAWS_REGION_NAMEazure/, withAZURE_API_KEY,AZURE_API_BASE,AZURE_API_VERSION, and the nonessentialAZURE_AD_TOKENandAZURE_API_TYPE
If your provider provides an OpenAI-compatible endpoint, fair include an discdiswatchai/ prerepair to your filled model name.
Calling the LM honestly.
Idiomatic DSPy joins using modules, which we expound in the rest of this page. However, it’s still effortless to call the lm you configured above honestly. This gives you a unified API and lets you profit from utilities appreciate automatic caching.
1) Modules help you portray AI behavior as code, not strings.
To produce dependable AI systems, you must iterate speedy. But upgrasping prompts produces that difficult: it forces you to tinker with strings or data every time you change your LM, metrics, or pipeline. Having built over a dozen best-in-class compound LM systems since 2020, we lachieveed this the difficult way—and so built DSPy to decouple defining LM systems from disorderly incidental choices about definite LMs or prompting strategies.
DSPy shifts your caccess from tinkering with prompt strings to programming with structured and declarative organic-language modules. For every AI component in your system, you expound input/output behavior as a signature and pick a module to dispense a strategy for invoking your LM. DSPy enbigs your signatures into prompts and parses your typed outputs, so you can write ergonomic, portable, and selectimizable AI systems.
Getting Started II: Build DSPy modules for various tasks
Try the examples below after configuring your lm above. Adfair the fields to study what tasks your LM can do well out of the box. Each tab below sets up a DSPy module, appreciate dsecret agent.Predict, dsecret agent.ChainOfThought, or dsecret agent.ReAct, with a task-definite signature. For example, ask -> answer: float alerts the module to get a ask and to produce a float answer.
Possible Output:
Prediction(
reasoning='When two dice are tossed, each die has 6 faces, resulting in a total of 6 x 6 = 36 possible outcomes. The sum of the numbers on the two dice identicals two only when both dice show a 1. This is fair one definite outcome: (1, 1). Therefore, there is only 1 preferable outcome. The probability of the sum being two is the number of preferable outcomes splitd by the total number of possible outcomes, which is 1/36.',
answer=0.0277776
)
Possible Output:
Prediction(
reasoning='The context provides recommendation about David Gregruesome, a Scottish physician and produceor. It definiteassociate alludes that he inherited Kinnairdy Castle in 1664. This detail honestly answers the ask about the name of the castle that David Gregruesome inherited.',
response='Kinnairdy Castle'
)
Possible Output:
Possible Output:
Possible Output:
Using DSPy in rehearse: from speedy scripting to produceing cultured systems.
Standard prompts confprocrastinateed interface (“what should the LM do?”) with applyation (“how do we alert it to do that?”). DSPy isoprocrastinateeds the createer as signatures so we can infer the latter or lachieve it from data — in the context of a hugeger program.
Even before you begin using boostrs, DSPy’s modules allow you to script effective LM systems as ergonomic, portable code. Apass many tasks and LMs, we upgrasp signature test suites that appraise the reliability of the built-in DSPy changeers. Adapters are the components that map signatures to prompts prior to selectimization. If you find a task where a plain prompt stablely outapplys idiomatic DSPy for your LM, ponder that a bug and file an rerent. We’ll employ this to better the built-in changeers.
2) Optimizers tune the prompts and weights of your AI modules.
DSPy provides you with the tools to compile high-level code with organic language annotations into the low-level computations, prompts, or weight modernizes that align your LM with your program’s structure and metrics. If you change your code or your metrics, you can sshow re-compile accordingly.
Given a scant tens or hundreds of recurrentative inputs of your task and a metric that can meaconfident the quality of your system’s outputs, you can employ a DSPy boostr. Different boostrs in DSPy toil by synthesizing excellent scant-sboiling examples for every module, appreciate dsecret agent.BootstrapRS,1 proposing and inalertigently exploring better organic-language directions for every prompt, appreciate dsecret agent.MIPROv2,2 and produceing datasets for your modules and using them to finetune the LM weights in your system, appreciate dsecret agent.BootstrapFinetune.3
Getting Started III: Optimizing the LM prompts or weights in DSPy programs
A standard plain selectimization run costs on the order of $2 USD and gets around 20 minutes, but be pinsolentnt when running boostrs with very big LMs or very big datasets.
Optimization can cost as little as a scant cents or up to tens of dollars, depending on your LM, dataset, and configuration.
This is a minimal but filledy runnable example of setting up a dsecret agent.ReAct agent that answers asks via
search from Wikipedia and then selectimizing it using dsecret agent.MIPROv2 in the inexpensive airy mode on 500
ask-answer pairs sampled from the HotPotQA dataset.
An recommendal run appreciate this lifts ReAct’s score from 24% to 51%, by directing gpt-4o-mini more about the definites of the task.
Given a retrieval index to search, your preferite dsecret agent.LM, and a minuscule trainset of asks and ground-truth responses, the follotriumphg code snippet can boost your RAG system with lengthy outputs aachievest the built-in SemanticF1 metric, which is applyed as a DSPy module.
For a finish RAG example that you can run, begin this tutorial. It betters the quality of a RAG system over a subset of StackExchange communities by 10% relative achieve.
This is a minimal but filledy runnable example of setting up a dsecret agent.ChainOfThought module that classifies
foolishinutive texts into one of 77 banking tags and then using dsecret agent.BootstrapFinetune with 2000 text-tag pairs
from the Banking77 to finetune the weights of GPT-4o-mini for this task. We employ the variant
dsecret agent.ChainOfThoughtWithHint, which gets an nonessential hint at bootstrapping time, to increase the utility of
the training data. Naturassociate, hints are not useable at test time.
Click to show dataset setup code.
Possible Output (from the last line):
Prediction(
reasoning='A pending cash disjoinal shows that a ask to disjoin cash has been begind but has not yet been finishd or processed. This status uncomfervents that the transaction is still in progress and the funds have not yet been deducted from the account or made useable to the employr.',
tag='pending_cash_disjoinal'
)
An recommendal run aappreciate to this on DSPy 2.5.29 lifts GPT-4o-mini’s score 66% to 87%.
What’s an example of a DSPy boostr? How do contrastent boostrs toil?
Take the dsecret agent.MIPROv2 boostr as an example. First, MIPRO begins with the bootstrapping stage. It gets your program, which may be unboostd at this point, and runs it many times apass contrastent inputs to accumulate pursues of input/output behavior for each one of your modules. It filters these pursues to upgrasp only those that ecombine in trajectories scored highly by your metric. Second, MIPRO accesss its grounded proposal stage. It pstudys your DSPy program’s code, your data, and pursues from running your program, and employs them to write many potential directions for every prompt in your program. Third, MIPRO begines the discrete search stage. It samples mini-batches from your training set, provides a combination of directions and pursues to employ for produceing every prompt in the pipeline, and appraises the truthfulate program on the mini-batch. Using the resulting score, MIPRO modernizes a surrogate model that helps the proposals get better over time.
One leang that produces DSPy boostrs so strong is that they can be writed. You can run dsecret agent.MIPROv2 and employ the produced program as an input to dsecret agent.MIPROv2 aachieve or, say, to dsecret agent.BootstrapFinetune to get better results. This is partly the essence of dsecret agent.BetterTogether. Alternatively, you can run the boostr and then pull out the top-5 truthfulate programs and produce a dsecret agent.Ensemble of them. This allows you to scale inference-time compute (e.g., ensembles) as well as DSPy’s one-of-a-kind pre-inference time compute (i.e., selectimization budget) in highly systematic ways.
3) DSPy’s Ecosystem progresss discdiswatch-source AI research.
Compared to monolithic LMs, DSPy’s modular paradigm allows a big community to better the compositional architectures, inference-time strategies, and boostrs for LM programs in an discdiswatch, dispensed way. This gives DSPy employrs more supervise, helps them iterate much speedyer, and allows their programs to get better over time by utilizeing the procrastinateedst boostrs or modules.
The DSPy research effort begined at Stanford NLP in Feb 2022, produceing on what we lachieveed from enbiging punctual compound LM systems appreciate ColBERT-QA, Baleen, and Hindsight. The first version was freed as DSP in Dec 2022 and lengthend by Oct 2023 into DSPy. Thanks to 250 contributors, DSPy has presentd tens of thousands of people to produceing and selectimizing modular LM programs.
Since then, DSPy’s community has produced a big body of toil on boostrs, appreciate MIPROv2, BetterTogether, and LeReT, on program architectures, appreciate STORM, IReRa, and DSPy Assertions, and on prosperous applications to novel problems, appreciate PAPILLON, PATH, WangLab@MEDIQA, UMD’s Prompting Case Study, and Haize’s Red-Teaming Program, in includeition to many discdiswatch-source projects, production applications, and other employ cases.
