xAI previews Grok-1.5 and creates a new benchmark called RealWorldQA

April 14, 2024

  • Elon Musk's xAI previewed their second foundation model, Grok-1.5
  • Grok-1.5 competes well with leading models, particularly in vision tasks
  • xAI also established a new benchmark for testing model's vision skills

Elon Musk’s xAI has revealed Grok-1.5, a multimodal AI model designed to beat competitors in understanding real-world scenarios. 

Following in the footsteps of others, like GPT-4V, the new Grok-1.5 introduces visual processing to analyze anything from documents and diagrams to charts, screenshots, and photographs.

Grok-1.5 also gains ground in text, coding, and math tasks, scoring 50.6% on the MATH benchmark, 90% on the GSM8K benchmark, and 74.1% on the HumanEval benchmark. 

This throws Grok-1.5 right into the LLM heavyweight tier, averaging slightly lower scores than Gemini Pro 1.5, GPT-4, and Claude 3 Opus.

Grok
Grok-1.5’s competitive text, math, and coding benchmarks. Source: xAI

Grok-1.5 also offers longer context understanding up to 128K tokens, a 16-fold increase compared to its predecessor, though well behind those touted by Claude 3 Opus and Gemini 1.5 Pro.

A Needle In A Haystack (NIAH) evaluation demonstrated Grok-1.5’s ability to locate embedded text within contexts of up to 128K tokens in length.

However, it’s Grok-1.5’s vision skills that xAI is pushing the hardest.

Demos show Grok-1.5 converting block schemes into Python code, generating bedtime stories inspired by children’s paintings, creating CSV datasets from screenshots, and even “expanding” memes. 

Grok-1.5 tops the leaderboard in some established benchmarks like Mathvista and TextVQA and scores highest in xAI’s newly established benchmark, RealWorldQA.

Grok-1.5’s impressive vision benchmarks. Source: xAI

Under the hood, Grok-1.5 is powered by a custom distributed training framework that enables xAI’s team to prototype ideas and train new architectures at scale with minimal effort.

xAI was founded last year and includes some of the world’s top AI researchers with the ultra-ambitious goal to “Understand the universe.” 

Thus far, we’ve got the witty and outlandish Grok-1 that tells people how to synthesize narcotics and criticizes Musk and Tesla.

Grok is also connected to X’s post database, which, among other unique quirks, has given it quite the following despite not troubling the leaders in pure performance. 

Musk’s xAI project challenges generative AI’s primarily closed-source ecosystem, making its models generally available under true open-source licenses

Combined with Meta, which has a similar intent to go against the grain of competitors, xAI’s open thesis might become a thorn in the monetization efforts of OpenAI, Microsoft, Anthropic, and Google.

RealWorldQA

Grok-1.5’s preview also saw xAI reveal the RealWorldQA, a new benchmark consisting of over 700 images, each accompanied by a question and verifiable answer.

The dataset primarily comprises anonymized images captured from vehicles and other real-world situations.

The RealWorldQA dataset is designed to evaluate the spatial understanding capabilities of Grok 1.5 and other multimodal AI models. xAI deemed other benchmarks were lacking in this department. 

Grok
The RealWorldQA benchmark dataset seeks to test models’ ability to understand natural scenes. Source: xAI

Grok-1.5 outperforms competitors in RealWorldQA, and it’ll be interesting to see if it catches on.

While it stops short of understanding the universe, Grok-1.5 will take its place as another top-tier model in an ever-increasing lineup.

That also shows how generative AI in its current form is reaching the peaks of its powers – though perhaps not for long. 

Join The Future


SUBSCRIBE TODAY

Clear, concise, comprehensive. Get a grip on AI developments with DailyAI

Sam Jeans

Sam is a science and technology writer who has worked in various AI startups. When he’s not writing, he can be found reading medical journals or digging through boxes of vinyl records.

×

FREE PDF EXCLUSIVE
Stay Ahead with DailyAI

Sign up for our weekly newsletter and receive exclusive access to DailyAI's Latest eBook: 'Mastering AI Tools: Your 2024 Guide to Enhanced Productivity'.

*By subscribing to our newsletter you accept our Privacy Policy and our Terms and Conditions