This one is for anyone using LLMs for summarization. A recent paper looked into how to make summaries more detailed, without being dense or hard to follow. Enter, Chain of Density (CoD).
What is Chain of Density prompting
Chain of Density is prompt engineering method designed to generate multiple summaries that become progressively more detailed, without increasing their length.
An initial summary is generated that is entity-sparse, meaning it contains few key nouns or noun phrases that encapsulate the main points of the article. Then Chain of Density iteratively incorporates additional entities.
An entity is defined as:
- Relevant: to the main story.
- Specific: descriptive yet concise
- Novel: not in the previous summary
- Faithful: present in the Article
- Anywhere: located anywhere in the Article.
Why use Chain of Density
Chain of Density prompting was designed to address a few specific problems.
- Typical summaries from LLMs tend to oversimplify and miss crucial details.
- Mitigate LLMs' tendency to primarily focus on the initial part of the content they summarize, a phenomenon known as lead bias.
- Get rid of fluff by maintaining control over the summary's length.
How Chain of Density works
The greatest advantage of Chain of Density is its simplicity, being just a single prompt. It follows these steps:
- Create an initial summary.
- Identify entities that were missing from the initial summary.
- Integrate 1-3 more entities into a new summary.
- Ensure the new summary is concise, while retaining all entities from the previous iteration.
- Repeat this process 5 times, each time incorporating more entities without extending the length of the summary.
- Output the final set of summaries as a list of dictionaries in JSON format. Each dictionary contains the keys "missing_entities" and "denser_summary.”
Chain of Density prompt template
You can try out Chain of Density right away via our template in PromptHub.
If you don't have PromptHub access but want to try it out, reply to the email that gets sent when you join the waitlist and I'll share an access code with you.
Chain of Density experiments
Setup
The researchers sampled 100 articles from CNN to generate Chain of Density summaries. These summaries were evaluated by humans, GPT-4, and were compared to human-written summaries.
Reference points
Chain of Density summaries were compared to human-written summaries and those produced by GPT-4 using a vanilla prompt: "Write a VERY short summary of the Article. Do not exceed 70 words.”
When comparing to human written summaries, the researchers compared the Entities/Tokens (how dense it is) and where the human preference lays in relation to Chain of Density steps.
Human evaluation
First up was determining which of the 5 summaries generated by Chain of Density was preferred. Four of the researchers independently chose their preferred summary for each article by evaluating them on informativeness and readability.
- Each researcher read through 500 summaries (100 articles * 5 Chain of Density summaries/article)
- The median preferred Chain of Density step was step 3, indicating a balance in summary density.
- As the Chain of Density process progressed through more iterations, the content became more evenly distributed, reducing lead bias.
GPT-4 Evaluation
Next up the researchers used GPT-4 to rate the Chain of Density summaries along 5 dimensions: Informative, Quality, Coherence, Attributable, and Overall.
- Increased detail (densification) is correlated with informativeness.
- Denser summaries -> lower quality and coherence. Important to note that these quality and coherence don't require the evaluator to reference the original article.
- The Overall scores skewed toward denser summaries
- On average across dimensions, the first and last Chain of Density steps were the least favored. Suggesting that either too sparse or too dense summaries were not optimal.
- The correlation between by GPT-4 and the human evaluations was low, at ~.31, indicating that AI and humans were prioritizing different aspects when judging the summaries.
How Chain of Density compares against human preferences
A major component of the research was to find the Chain of Density step that most closely mimicked human written summaries.
The graph below shows the density, measured in Entities/Tokens, for human summaries, the vanilla GPT-4 prompt, and the various steps for CoD. You'll see it is around step 3 that Chain of Density most closely mimics the density of human written summaries.
Trade-offs between readability and informativeness
As we saw in the results, balancing density and readability is important when using CoD.
Here's an interesting example from the article:
On the left, the summary is improved with more detail. The addition of the other team in the match (Liverpool), and their goal-scorers provide good context. The denser summary also makes some nice compressions: "a potential route back into the game" -> "a comeback".
On the right, the summary loses quality with more detail. The added details about the network, TV5Monde, seem out of place and are not relevant to the main story about the cyberattack. This is a direct result of having to tighten the previous summary and inject more entities.
Wrapping up
Chain of Density is a simple and effective method for generating summaries that rival human-generated ones. The results suggest sticking to 2 or 3 steps for summaries that are inline with the density that humans prefer. The beauty in the method is its simplicity, requiring just a single prompt. Let us know if this helps your summarization prompts!
