If you’re brainstorming new ideas, then AI models like GPT-4 can often come up with some great ideas to add to your list. The problem is that the long list of ideas it generates often contains very similar ideas, rather than the out-of-the-box thinking you’re after.
Researchers from The Wharton School, University of Pennsylvania, wanted to see if there was a way to make an AI model like GPT-4 deliver more diverse, or novel, ideas during the idea generation process.
The authors of the paper, Lennart Meincke, Ethan Mollick, and Christian Terwiesch, started with the hypothesis that GPT-4 had the potential to be more creative and that clever prompting might unlock that creativity.
The experiment
The team designed an exercise where the goal was to come up with ideas for a new consumer product targeted at college students that cost $50 or less.
They used different prompting strategies to see what ideas GPT-4 could come up with and then measured their diversity. They then compared the diversity of the AI-generated ideas with those that teams of MBA students came up with.
The prompting strategies they used were:
- Minimal prompting
- Prompt GPT-4 to assume different personas
- Share creative thinking ideas from existing literature with GPT-4
- Use Chain of Thought (CoT) prompting and instruct the LLM to work in multiple distinct steps
🚨 Our new paper: we know that GPT-4 generates better ideas than most people, but the ideas are kind of similar & variance matters
But it turns out that better prompting can generate pools of good ideas that are almost as diverse as from a group of humans https://t.co/LkGsU0VC7S pic.twitter.com/5BDx8A3E8Z
— Ethan Mollick (@emollick) January 27, 2024
Results
The diversity of the ideas was measured using the Cosine similarity metric. This metric is commonly used in text analysis to measure how similar items in lists are in terms of their ideas. A measure of 1 means the ideas are very similar, with diversity in the compared ideas increasing as the value tends to zero.
The paper noted that “pools of ideas generated by GPT-4 with no special prompting are less diverse than ideas generated by groups of human subjects.” The groups of humans had a Cosine similarity of 0.243 compared to a range of 0.255 to 0.432 for GPT-4 generated ideas depending on the prompting.
Here’s an example of the comparison of ideas.
This confirms that while GPT-4 comes up with some very good ideas, a lot of them end up being variations of the same idea. A win for Team Human. The number of good ideas and how quickly the pool of ideas was exhausted were also measured.
The researchers found that using longer prompts resulted in more diversity in the ideas GPT-4 delivered. The best strategy was using CoT, which came a close second to the humans with a Cosine similarity of 0.255.
Prompting GPT-4 to assume personas had mixed results with little predictability. Prompting the LLM to act as “Steve Jobs” (0.368) delivered more diverse ideas than a variation using “Elon Musk” (0.385). Prompting it to act as a “creative entrepreneur” delivered a Cosine similarity of 0.348.
Interestingly, when the lists of ideas from the different prompts were compared there was little overlap between them.
In a tweet, Ethan Mollick said, “I should mention that I don’t think we discovered (or even tried to discover) some sort of amazing prompting technique. We are arguing that major assumed limits of AI ideation – it generates less diverse ideas than a group of people & has fewer ideas overall – need not be true.”
So, if you want to use GPT-4 to help in your next brainstorming session there are a few things you can do to get it to be more creative. Use longer prompts, throw in some CoT instructions, ask it to assume a few personas, and then combine all the ideas from the different prompts into a single list.
Or, for even better results, you could hire a group of MBA students.
