Generated Knowledge Prompting

Hey everyone, how's it going? This is Dan here for PromptHub, and we have a super cool new prompt engineering method to go over today called Generated Knowledge Prompting. What's interesting is that although this paper came out a while ago—which in AI years is almost like two decades—it's actually from 2022. There have been a bunch of other remixes of this type of prompting, but I believe this is the first version of it.

Generally, Generated Knowledge Prompting is an engineering method that uses a prompt to first tell the model to generate some useful knowledge, information related to the task or question at hand, and then there's a second prompt that basically pastes in that knowledge and then asks the actual question. So it’s a two-step process, though you could kind of push it into one, but they have it separated as two.

For example, if a customer asked a question about rebooking a flight from New York to London, the first prompt is sent just looking at current travel restrictions in the UK, and so on. Then you take that output, inject it as the knowledge, and have your user query here. It takes the output from the second prompt, adds it in here where it’s relevant to what the question is being asked, and then has the question asked.

Again, this is what that flow looks like: there's a question, it generates some knowledge based on that question, plugs in that knowledge, integrates it, and brings the question as well, and then you get an answer. It’s very similar to something we talked about before, which is Analogical Prompting. As you can see, we have a template for this, so you can access it, but basically, Analogical Prompting says, 'Here's the problem, here are the instructions, first identify some core concepts, recall three problems, for each problem describe and explain the solution, then solve the initial problem.' These kinds of tutorial and relevant problem steps are the knowledge part.

This method has been remixed in other ways as well. The benefits are generally higher accuracy, that’s what this study found, and it has been replicated across the Analogical Prompting study and others. This additional context is a really lightweight way to give the model the information it needs to retrieve without having to set up any complex pipelines, making it really adaptive. You can go to very specific parts of the model’s knowledge by letting it generate this information first. It even seems like it can get to a layer deeper of understanding since it’s already done that cursory search of like blanket information, which I think has really interesting implications.

As I mentioned, there are a lot of different ways you could generate this knowledge. Here’s how they did it: basically, they would send some examples and say, 'Hey, here’s the input, and then here’s a knowledge-generated input,' and then the LLM would kind of fill in the blank here with the given knowledge.

We will still look at the results even though they're quite old. Basically, they tested the Generated Knowledge Prompting against a couple of different baselines: no knowledge, random sentences, which had basically random, not related sentences injected into the prompt, contextual sentences that had proper context but not being like actual knowledge, template knowledge which had templates to extract knowledge statements from the models, knowledge was getting stuff from sources like Google or Wikipedia, and then answers which directly prompt the model to generate answers. It’s kind of similar to Analogical Prompting in that way, using a single answer to assess F-shop performance. It’s not super important if that doesn’t make complete sense, we’ll just take a look at the results here.

So, we’re obviously seeing a lot, you know, RS being the Generated Knowledge Prompting, we see a lot of bulbs across the board, which is typical from a paper here. It outperformed the baseline by a fair amount, outperformed two-shot prompting on most, and I think interestingly it outperformed the retrieval-based prompting on a few of the sets but not all of them. It also shows that the model has a lot of information in there, and extracting it is something that you could turn to before looking to do retrieval-based stuff just because that has an additional layer of complexity and an additional step where things can get messed up.

I think that's an interesting takeaway from this that is probably still true today, especially if you think if the models are getting better, then you would think this difference would actually grow versus get smaller. And I think the last thing to note here is obviously, on a couple of these datasets, specifically on this QC dataset, the retrieval-based method outperformed the Generate Knowledge by a fair amount. Something to note here is that the retrieved knowledge for this source wasn’t just like Google or Wikipedia, like a generic source; it was a kind of like gold-standard knowledge base that was specifically designed to construct this dataset, so it was hyper-relevant. That’s the key to go, and having hyper-relevant retrieval-based systems is great.

I think if you want to try that out, that’s really worth testing. Having more loosely like retrieval-based pipelines is not, I don’t think they’re going to make a huge difference. I think that’s something that is probably still true today, and there’s additional study at the end in terms of how much knowledge is needed. This graph looks very similar to the graphs shown if you look at how many examples in F-shop prompting increase performance where there’s a big jump up, and then here we basically plateau from 1 to 50, and it’s similar for the F-shop prompting, except it plateaus right away, around like maybe four to six.

So, I think it’s something to take note of as well. But yeah, we hope this helps, we have templates on PromptHub to try this out for free, so feel free to go check it out there, and yeah, happy prompting!