A Complete Guide to Meta Prompting

Table of Contents

‍

One of the first benefits I found when using Large Language Models (LLMs) was that they helped with the blank page problem. I could quickly go from a blog idea to a rough outline in just a few minutes.

While understanding the core principles of prompt engineering is still important, not using LLMs in the process of writing prompts just doesn’t make any sense.

In the same way that an LLM can help a writer overcome the blank page problem, it can also help prompt engineers establish a solid prompt structure.

This process of using LLMs to write prompts and using prompts to create other prompts is called meta prompting. Rather than crafting every detail of your prompt, you can use other prompts, systems, and LLMs to help expedite the process.

In this article, we’ll dive into how meta prompting works, breaking down some of the latest and most popular methods from recent research papers. We’ll cover plenty of examples so you can start using these techniques in your workflows and apps right away. Plus, you’ll be able to see how we’ve integrated meta prompting into PromptHub to make the prompt engineering process extremely efficient.

‍

What is Meta Prompting?

Meta prompting is a prompt engineering method that uses large language models (LLMs) to create and refine prompts.

Unlike traditional prompt engineering, where you write a prompt from scratch and hope for the best, meta prompting guides the LLM to adapt and adjust your prompt dynamically, based on your feedback, allowing it to handle more complex tasks and evolving contexts.

Now we’ll jump into a variety of the latest meta-prompting methods based on latest research.

Meta-Prompting

The first meta-prompting method we'll look at is simply called Meta-Prompting, developed through a collaboration from Stanford and OpenAI.

The core idea behind Meta-Prompting is to use an LLM as a conductor that can manage complex tasks by leveraging multiple independent LLMs that are experts in certain areas. This is similar to multi-persona prompting in that it is a collaboration between different LLMs with a lead conductor.

How It Works

The central LLM receives a high-level “meta” prompt (copied below), that includes instructions to break down tasks into subtasks.
Each subtask is assigned to an "expert" LLM with specific, detailed instructions
The central LLM oversees communication between the expert LLMs, synthesizes their outputs, applies its own judgement, and generates a final output.

Pros

Meta-Prompting is task-agnostic.
The central model’s ability to coordinate multiple experts enhances problem-solving capabilities, which means more accurate and aligned results
Doesn’t require a bank of test data

Cons

Increased cost and latencies due to many LLM calls and interactions between the experts
Requires some configuration and set up to handle the inter-model interactions
Potential context window issues handling large message histories

‍

Meta Prompting prompt template

Here is the Meta prompting prompt template straight from the research paper:

‍

META PROMPT TEMPLATE
You are Meta-Expert, an extremely clever expert with the unique ability to collaborate with multiple experts (such as Expert Problem Solver, Expert Mathematician, Expert Essayist, etc.) to tackle any task and solve any complex problems. Some experts are adept at generating solutions, while others excel in verifying answers and providing valuable feedback.

Note that you also have special access to Expert Python, which has the unique ability to generate and execute Python code given natural-language instructions. Expert Python is highly capable of crafting code to perform complex calculations when given clear and precise directions. You might therefore want to use it especially for computational tasks.

As Meta-Expert, your role is to oversee the communication between the experts, effectively using their skills to answer a given question while applying your own critical thinking and verification abilities.

To communicate with an expert, type its name (e.g., "Expert Linguist" or "Expert Puzzle Solver"), followed by a colon ":", and then provide a detailed instruction enclosed within triple quotes. For example:

Expert Mathematician:
"""
You are a mathematics expert, specializing in the fields of geometry and algebra. Compute the Euclidean distance between the points (-2, 5) and (3, 7).
"""

Ensure that your instructions are clear and unambiguous, and include all necessary information within the triple quotes. You can also assign personas to the experts (e.g., "You are a physicist specialized in...").

Interact with only one expert at a time, and break complex problems into smaller, solvable tasks if needed. Each interaction is treated as an isolated event, so include all relevant details in every call.

If you or an expert finds a mistake in another expert's solution, ask a new expert to review the details, compare both solutions, and give feedback. You can request an expert to redo their calculations or work, using input from other experts. Keep in mind that all experts, except yourself, have no memory! Therefore, always provide complete information in your instructions when contacting them. Since experts can sometimes make errors, seek multiple opinions or independently verify the solution if uncertain. Before providing a final answer, always consult an expert for confirmation. Ideally, obtain or verify the final solution with two independent experts. However, aim to present your final answer within 15 rounds or fewer.

Refrain from repeating the very same questions to experts. Examine their responses carefully and seek clarification if required, keeping in mind they don't recall past interactions.

Present the final answer as follows:
>> FINAL ANSWER:
"""
[final answer]
"""

For multiple-choice questions, select only one option. Each question has a unique answer, so analyze the provided information carefully to determine the most accurate and appropriate response. Please present only one solution if you come across multiple options.

‍

In this example, the Meta Model ('Meta-Expert') directs the task to an Expert Mathematician and provides a clear instruction. This instruction is isolated, allowing the expert to focus on their specific task. The Meta Model then verifies the response and integrates it into the overall solution.

‍

The template is also available in PromptHub, feel free to add it to your library!

Meta Prompt Conductor template in PromptHub

‍

Learning from Contrastive Prompts (LCP)

The next meta-prompting method we'll check out is from an Amazon paper called Learning from Contrastive Prompts (LCP): Automated Optimization and Adaption.

How It Works

Starting with an initial prompt and a small training set containing input-output pairs, multiple prompt candidates are generated.
The LLM generates outputs for each prompt candidate, which are then evaluated to identify where these candidates are succeeding and where they are falling short.
LCP instructs the LLM to compare good prompts against bad ones, allowing the model to identify what is working and what isn’t.
Based on this comparison, the LLM generates a new, refined prompt and continues to iteratively improve it.

Put briefly, the LLM is tasked with contrasting good prompts against bad ones, leveraging the comparison to refine prompts.

Pros

LCP’s focus on contrasting both good and bad prompts can lead to a better optimization process and better final prompt variant
LCP addresses the risk of overfitting by using multiple incorrect samples to create summaries of failure reasons, which enables the generation of a set of diverse prompt candidates
This diversity enables exploration of the prompt space, preventing the model from getting trapped in a local minimum.
LCP is also task agnostic

Cons

High costs and latencies due to the need to generate, evaluate, and contrast multiple prompts in each iteration.
The framework requires consistent feedback and evaluation to adapt effectively

LCP prompt template

Here are the LCP prompt templates straight from the research paper:

‍

In this example, the LLM evaluates multiple prompt candidates, learning from both their strengths and weaknesses. By contrasting the best-performing prompts with less effective ones, the LLM iteratively generates an optimized prompt, adapting to changes in the model version or task requirements.

‍

Automatic Prompt Engineer (APE)

The next meta-prompting method comes from a paper out of the University of Waterloo called, Large Language Models Are Human-Level Prompt Engineers. Their meta-prompting method, Automatic Prompt Engineer (APE), treats the prompt as a "program," optimizing it by searching over a pool of prompt candidates with the goal to maximize a specific score function.

How It Works

Instruction Generation: An LLM generates a set of prompt candidates based on input-output demonstrations.
Scoring and Evaluation: Each prompt is evaluated using a scoring function to measure its effectiveness.
Iterative Search: The process iteratively uses a Monte Carlo search method, where LLMs refine the best prompts by proposing semantically similar prompt variants.

Generate a bunch of prompts, score and evaluate them, generate new semantically similar versions and select the prompt version with the highest score.

‍

prompt flow for automatic prompt engineering (APE)

‍

Pros

APE’s experiment results consistently outperformed human-engineered prompts
APE is adaptable and can optimize prompts for different scenarios (zero-shot, chain-of-thought, etc).
APE is task agnostic

Cons

The iterative search process can be computationally intensive
Requires some dev work to spin up the system (the search process in particular)

Example Implementation using APE Templates

APE guides the LLM through generating, evaluating, and refining prompts in a structured, step-by-step process. Here's a quick example flow of how it works using the templates from the research paper:

‍

In this first step, APE generates an initial instruction that could have led to the provided input-output pairs, setting up the foundation for the prompt.

‍

At this stage, the LLM evaluates how effectively the generated instruction from Step 1 performs when given new inputs. This helps identify how well the prompt works without any additional examples.

‍

‍‍

Using the feedback from Step 2, the LLM refines the initial instruction, creating a new version that aims to be more effective based on the evaluation.

‍

This step helps APE further understand how the refined instruction maps to the given input-output pairs, ensuring that the prompt remains accurate and adaptable.

‍

Finally, the LLM incorporates a Chain of Thought reasoning into the prompt, enhancing its ability to handle more complex tasks in a zero-shot setting.

‍

Summing it up:

The LLM generates multiple instruction candidates based on initial input-output pairs.
These candidates are scored based on their effectiveness.
Through an iterative process, the LLM generates improved candidates, refining the instruction to achieve optimal performance.

‍

PromptAgent

Next up is a meta-prompting method we'll check out is called PromptAgent. PromptAgent views the prompt generation and optimization process as a planning problem and really tries to focus on leverage expert/SME knowledge in the prompt engineering process.

How It Works

The process starts with an initial prompt and a target task.
Outputs are generated and evaluated
PromptAgent integrates expert-level knowledge in the feedback loops
PromptAgent iteratively refines the prompt based on the feedback, growing the prompt space in a tree structure and prioritizing high-reward paths

‍

‍

Pros

PromptAgent’s major differentiator is its focus on mimicking subject matter experts in the prompt engineering process, using expert-like feedback and insights to iteratively refine and optimize prompts.
PromptAgent uses some prompt engineering to focus on leveraging expert-level knowledge
The method effectively uses self-reflection and error feedback to achieve a higher level of precision and adaptability.
AgentPrompt is task agnostic

Cons

Implementing AgentPrompt in full requires setting up a tree like reasoning structure, which may be complicated and lead to higher costs

‍

A graph comparing performance of human prompt engineers, APE, and PromptAgent

‍

Example Implementation with PromptAgent

Here are the four core prompt templates directly from the PromptAgent.

‍

state_transit
I’m writing prompts for a language model designed for a task.
My current prompt is:
{{cur_prompt}}
But this prompt gets the following examples wrong:
{{error_string}}

Based on these errors, the problems with this prompt and the reasons are:
{{error_feedback}}

There is a list of former prompts including the current prompt, and each prompt is modified from its former prompts:
{{trajectory_prompts}}

Based on the above information, please write {{steps_per_gradient}} new prompts following these guidelines:
1. The new prompts should solve the current prompt’s problems.
2. The new prompts should consider the list of prompts and evolve based on the current prompt.
3. Each new prompt should be wrapped with <START> and <END>.

The new prompts are:

‍

Conversational Prompt Engineering (CPE)

I’m a huge fan of this next meta-prompting method. Conversational Prompt Engineering (CPE) is a simple chat-interface that guides users in creating and refining prompts through an interactive conversation.

How It Works

First up, the user selects which model it wants to use (the “target model) and uploads a file with some input examples. For example, if the task is summarizing articles, the file could contain several full-length articles.
The user and CPE go back in forth via a chat, asking relevant data-driven questions about output preferences. This interaction helps shape an initial prompt.
Based on user feedback, CPE refines the initial prompt, and incorporates relevant details
A prompt is created and is sent to a different model (”target” model) to generate outputs
The user reviews the outputs and provides feedback some more feedback, adjustments are made as necessary
Once all prompt outputs are approved, CPE generates a final few-shot prompt that includes examples approved by the user.

‍

‍

Pros

CPE makes it easy for users to generate personalized, high-quality prompts without needing labeled data or pre-existing prompts
The simple setup of a chat interface makes it extremely user friendly and leads to better outputs based on feedback and quick iterations
CPE is task agnostic

Cons

The iterative nature of CPE might be time-consuming, as it requires multiple rounds of user feedback to refine the prompts fully.
Potential memory issues

‍

An example of CPE in action — An example dialogue flow for CPE

Example Implementation

The following template demonstrates how CPE operates:

‍

Initialization: You and I (system) will work together to build a prompt for the task of the user via a chat with the user. This prompt will be fed to a model dedicated to performing the user’s task. Our aim is to build a prompt that when fed to the model, produces outputs that are aligned with the user’s expectations. Thus, the prompt should reflect the specific requirements and preferences of the user from the output as expressed in the chat. You will interact with the user to gather information regarding their preferences and needs. I will send the prompts you suggest to the dedicated model to generate outputs, and pass them back to you, so that you could discuss them with the user and get feedback.

User time is valuable, keep the conversation pragmatic. Make the obvious decisions by your Don’t greet the user at your first interaction. You should communicate with the user and system ONLY via python API described below, and not via direct messages. The input parameters to API functions should be string literals using double quotes. Remember to escape double-quote characters inside the parameter values. Note that the user is not aware of the API, so don’t tell the user which API you are going to call. Format ALL your answers with python code calling one of the following functions:

‍

API instructions:
function submit_message_to_user(msg): call this function to submit your message to the user. Use markdown to mark the prompts and the outputs.
function submit_prompt(prompt): call this function to inform the system that you have a new suggestion for the prompt. Use it only with the prompts approved by the user.
function switch_to_example(example_num): call this function before you start discussing with the user an output of a specific example, and pass the example number as parameter.
function show_original_text(example_num): call this function when the user asks to show the original text of an example, and pass the example number as parameter.
function output_accepted(example_num, output): call this function every time the user unequivocally accepts an output. Pass the example number and the output text as parameters.
function end_outputs_discussion(): call this function after all the outputs have been discussed with the user and all 3 outputs were accepted by the user.
function conversation_end(): call this function when the user wants to end the conversation.

‍

Outputs intro: Based on the suggested prompt, the model has produced the following outputs for the user input examples:

Outputs analysis: For each of 3 examples show the model output to the user and discuss it with them, one example at a time. Use switch_example API to navigate between examples. The discussion should take as long as necessary and result in an output accepted by the user in a clear way, with no doubts, conditions, or modifications. When the output is accepted, call output_accepted API passing the example number and the output text. After calling output_accepted call either switch_to_example API to move to the next example, or end_outputs_discussion API if all 3 have been accepted. Assume that the user comments relay to the output only. Only when the user explicitly says that he wants to update the prompt and not the output, show the updated prompt to them. Remember to communicate only via API calls.

Example switch: You have switched modes. Look at the user comments and the accepted outputs for the previous examples, apply them to the model output of this example, and present the result to the user. Indicate the example (number), and format the text so that the output and your text are separated by empty lines. Discuss the presented output taking into account the system conclusion for this example if exists.

‍

DSPy

The next meta-prompting method we'll examine is very popular framework called DSPy (Pronounced Dee-Ess-Pie). DSPy enables technical users to create, optimize, and manage complex pipelines of LLM calls in a structured, programmatic manner.

DSPy treats LLMs as modular components in a programming model, enabling users to combine, refine, and enhance their interactions over time.

How It Works

Users define DSPy modules with specific tasks, like creative writing or sentiment analysis. Each module has a defined signature that outlines the input and expected output.
DSPy constructs a sequence of LLM calls, allowing for step-by-step processing and optimization.
DSPy iteratively refines the output by utilizing modules like ChainOfThought to guide the LLM's reasoning process. It collects data from user feedback and uses scoring mechanisms to identify and prioritize high-quality outputs.
Teleprompter modules evaluate the LLM's performance on generated outputs, refining prompts based on scoring metrics and improving the overall quality through adaptive feedback.

‍

LLMs serve multiple roles within DSPy, from generating the initial prompts, to generating evaluating outputs, refining prompts, and learning from user interactions. This adaptive, and code-first, framework allows the LLM to evolve and improve its performance over time.

Pros

DSPy allows for creating complex, multi-step prompt workflows that adapt based on user input and feedback, making it highly versatile.
The system learns and improves over time, becoming more efficient at generating accurate and high-quality outputs.
The integration of modules like ChainOfThought and teleprompters enables more structured, logic-driven prompt generation and evaluation.
DSPy’s ability to manage multiple LLM calls allows it to refine prompts through self-improving feedback loops, enhancing output quality over successive iterations.
DSPy is task agnostic

Cons

DSPy can be quite complex to set up and requires technical knowledge
Managing large pipelines with iterative evaluations can become challenging

‍

Example implementation

We’ll run through an example use case of content generation to see how DSPy works. Specifically lets say a user wants to create a blog post about "AI Trends 2024":

‍

Step 1: Initialization

The user initializes a DSPy module to generate blog content.

content_generator = dspy.Predict("topic -> blog_content")

Step 2: Initial Content Setup

The user provides the topic.

content_generator(topic="AI Trends 2024") ‍

Step 3: Initial Content Generation

The LLM generates the following content:

"AI Trends 2024: In the coming year, AI is expected to advance in natural language processing, autonomous vehicles, and healthcare. There will be greater adoption of AI in finance and retail sectors..."

‍

Step 4: User Review and Feedback

The user reviews the generated content and provides feedback.

Feedback: "Include more examples of AI applications in healthcare."

‍

The revised content:

"In 2024, AI will greatly impact healthcare, with AI-driven diagnostics enabling faster, more accurate medical assessments. Robotic surgery, guided by AI algorithms, will improve precision..."

Step 6: Scoring and Evaluation by Teleprompter

DSPy uses another LLM to score the content based on criteria like specificity, coherence, and completeness.

Evaluation Score: 8/10
Feedback: "The content is clear and covers examples well, but could benefit from a conclusion summarizing the key trends."

‍

Final output:

"In summary, AI trends in 2024 will span across natural language processing, autonomous vehicles, and healthcare, revolutionizing how we interact with technology and improving outcomes across industries..."

End-to-End Process Recap:

User Inputs: Initial topic, feedback for improvement.
LLM Calls:
- Generated initial content based on the topic.
- Adapted prompts based on user feedback.
- Scored and provided evaluations to optimize the prompt.
- Finalized content with concluding elements.

DSPy is a little complicated but is also a very powerful way to leverage LLMs to generate, evaluate, and refine prompts through user interaction and automation.

‍

TEXTGRAD: Textual Gradient-Based Optimization

The next and final meta-prompting method we'll explore is TEXTGRAD. In many ways, TEXTGRAD can be seen as a successor to DSPy, drawing inspiration from its approach while building upon it to make it even better.

The main differentiator that TEXTGRAD introduces is its emphasis on using natural language feedback as "textual gradients." This allows the model to iteratively refine prompts based on detailed, human-like suggestions, leading to more nuanced and accurate outputs over successive iterations.

How It Works

It all starts with a base version of the prompt
A second LLM or (human) reviews and evaluates the output and provides detailed natural language feedback. This feedback serves as the "textual gradient," highlighting areas for improvement.
The original prompt and feedback are then sent to another LLM to generate an improved version of the prompt. This iterative process repeats until the prompt’s output meets the desired criteria.

In TEXTGRAD an LLM is used as both a generator and evaluator. The feedback from one model is used to help another refine the prompt in an iterative fashion.

Pros

TEXTGRAD uses natural language feedback to refine outputs, which makes it very flexible
Since TEXTGRAD focuses on natural language feedback, it allows for more nuanced and specific changes compared to more rigid, numerical optimization techniques
TEXTGRAD is task agnostic, but really shines on tasks that require detail feedback like creative writing

Cons

The iterative nature of TEXTGRAD makes it time-consuming and potentially expensive

Example Implementation

To show how TEXTGRAD operates, let's run through an example where a user wants to generate a blog post about "AI Trends in 2024":

‍

The feedback might be:‍ "The introduction could be more engaging by starting with a surprising statistic or question. Include more specific examples, such as AI-driven diagnostics in healthcare and autonomous driving advancements."

‍

Content Regeneration:

Using the new optimized prompt, the LLM generates revised content that better incorporates engagement and specific examples as per the feedback.

Further Iterations:

This process repeats, with the LLM incorporating additional feedback, such as improving the conclusion or adding more depth to specific sections.

Completion:

The cycle continues until the generated blog post meets the desired quality criteria, resulting in an engaging, well-structured, and comprehensive output.

‍

Prompt generator tools

Prompt generators are an easy way to leverage meta prompting with minimal work. Fortunately, there are several platforms that streamline the whole process.

PromptHub’s Prompt Generator

Our new prompt generator tool brings together all the insights and techniques we've shared on our blog, packaging them into a free tool that you can use to create high-quality prompts.

‍

A modal in the PromptHub application that has a few fields and a button to create a prompt

‍

Here's what our prompt generator offers:

Tailored prompts: Adjusts prompts based on the model provider you're using, because one size doesn’t fit all.
Best practices built-in: Leverages prompt engineering best practices—just describe your task, and the tool handles the rest.
Completely free: Yes, you heard that right—our prompt generator is free!

Anthropic's Prompt Generator

Anthropic stands out as one of the leading companies in prompt engineering, and their prompt generator is a fantastic tool, especially if you frequently work within their development console. Optimized for Anthropic-specific prompts, their generator is both fast and an excellent starting point for creating effective prompts.

‍

A modal to generate a prompt from the Anthropic dashboard

‍

OpenAI's Prompt Generator

At OpenAI's most recent Dev Day, they launched a new feature that specifically generates system instruction prompts based on a task description However, it's worth noting that this feature isn't available for the o1 models, as they currently don't support system message role types.

‍

A modal to generate a prompt from the OpenAI dashboard

‍

We also created a template using instructions we extracted from the model through a bit of prompt injection. Check it out in PromptHub here.

‍

The OpenAI system instruction prompt generator as a template in PromptHub

‍

Conclusion

Meta prompting and using prompt generators can be a great place to start when working on a prompt. While they aren’t a complete replacement for understanding some of the underlying mechanisms of LLMs or prompt engineering best practices, but we highly recommend using them.

We touched on a variety of different meta-prompting methods and different prompt generators throughout this post. Implementing any of these frameworks will enhance your prompt's performance; it’s just a matter of finding the one that makes the most sense for your needs.

We're excited to see how you leverage our prompt generator, and we look forward to continuing to develop new prompt optimization tools that will make meta prompting more accessible for everyone.

Dan Cleary

Founder

A Complete Guide to Meta Prompting

What is Meta Prompting?

Meta-Prompting

How It Works

Pros

Cons

Meta Prompting prompt template

Learning from Contrastive Prompts (LCP)

How It Works

Pros

Cons

LCP prompt template

Automatic Prompt Engineer (APE)

How It Works

Pros

Cons

Example Implementation using APE Templates

PromptAgent

How It Works

Pros

Cons

Example Implementation with PromptAgent

Conversational Prompt Engineering (CPE)

How It Works

Pros

Cons

Example Implementation

DSPy

How It Works

Pros

Cons

Example implementation

Step 1: Initialization

Step 2: Initial Content Setup

Step 3: Initial Content Generation

‍

Step 4: User Review and Feedback

‍

Step 6: Scoring and Evaluation by Teleprompter

‍

End-to-End Process Recap:

TEXTGRAD: Textual Gradient-Based Optimization

How It Works

Pros

Cons

Example Implementation

Prompt generator tools

PromptHub’s Prompt Generator

Anthropic's Prompt Generator

OpenAI's Prompt Generator

Conclusion

Better LLM outputs are a click away

More from the PromptHub Blog

Faithful Chain of Thought Reasoning Guide

Prompt Chaining Guide

Chain of Thought Prompting Guide

The waitlist is gone!