A mysterious chatbot named ‘gpt2-chatbot’ has sparked immense excitement and speculation across the AI community.
This chatbot, which you can find and experiment with yourself on the LMSys Chatbot Arena, is touted by some users to be more advanced than current state-of-the-art models like GPT-4 and Claude 3 Opus.
Emerging seemingly out of thin air, its origins are totally unknown. Could it be an experimental project from a leading AI lab or a strange leak?
There is a mysterious new model called gpt2-chatbot accessible from a major LLM benchmarking site. No one knows who made it or what it is, but I have been playing with it a little and it appears to be in the same rough ability level as GPT-4. A mysterious GPT-4 class model? Neat! pic.twitter.com/1s2iEreaiT
— Ethan Mollick (@emollick) April 29, 2024
Reddit users mainly think it’s from OpenAI since the model admits that the answer style and tone seem different from GPT-4 and GPT-4 Turbo, but the knowledge cut-off differs from currently available OpenAI models.
gpt2-chatbot is insane at ascii art, miles ahead of any other model pic.twitter.com/1IwEaJyU7Q
— Phil (@phill__1) April 29, 2024
I asked gpt2-chatbot myself, “Who are you,” to which it answered:
“I was created by OpenAI, a research organization that focuses on developing and promoting friendly AI for the benefit of humanity. My development is based on the GPT (Generative Pre-trained Transformer) series of language models, specifically the GPT-4 version, which is an evolution of previous models like GPT-3. These models are trained on a diverse range of internet text, but they are designed to generate human-like text based on the prompts they receive.
OpenAI started with the goal of ensuring that artificial general intelligence (AGI) benefits all of humanity. They conduct research in the field of AI with the aim to develop and direct artificial intelligence technologies in ways that benefit humanity as a whole, unencumbered by financial obligations to investors.
If you have any more questions about AI, OpenAI, or related technologies, feel free to ask!”
When I quizzed it on why it’s named gpt2-chatbot, it replied:
“To summarize, I am not called “GPT-2”; I am a variant of GPT-4, designed specifically for interactive applications, such as this conversational format. My design includes improvements in handling dialogues, safety features, ethical considerations, and overall response quality. If you see references to “GPT-2,” they are likely discussing the model that came before GPT-3 and GPT-4.”
gpt2-chatbot’s performance provokes speculation
Regarding performance, one user on the Y Combinator forum discussed the bot’s domain knowledge in material science, discussing turbine blade manufacturing with depth they’d not witnessed otherwise.
Another user, the well-known data scientist Simon Wilson, was impressed by its handling of highly personalized queries, suggesting it has a detailed understanding of people’s personal backgrounds.
You can try out the mysterious gpt2-chatbot at https://t.co/4ARf9G6RHb (select “Direct Chat” and pick it from the menu)
Initial impressions: I’m very impressed. It gave me a better answer for an ego search (“Who is Simon Willison?”) than any other model I’ve tried https://t.co/MCJMVhNs8k
— Simon Willison (@simonw) April 29, 2024
Others identified standard LLM shortcomings, such as hallucinations and factual inaccuracies.
Having used it myself, I’d agree that it feels like a variant of GPT-4.
As to whether it’s better than GPT-4, as some speculate, it’s worth highlighting that many felt that GPT-4’s performance dropped over time (possibly coinciding with OpenAI dedicating compute resources to training new models).
So, this potentially unadulterated variant might feel like GPT -4 did when it first came out. Throw in some slight differentiation in behavior and performance on specific tasks, and you can see why speculation is high.
As discussions of gpt2-chatbot hit X, some have inevitably speculated it’s evidence of GPT-4.5, GPT-5, or even AGI. However, most are unsure whether it represents concrete progress over current models.
Overall, there’s no way an AI company would train an entirely new chatbot like this and release it in such a way. It’s almost certainly a GPT-4 variant.
Ultimately, it’s hearsay until the creator steps out from behind the curtains and claims ownership.
