{"id":13088,"date":"2024-06-25T13:48:10","date_gmt":"2024-06-25T13:48:10","guid":{"rendered":"https:\/\/dailyai.com\/?p=13088"},"modified":"2024-06-25T14:13:37","modified_gmt":"2024-06-25T14:13:37","slug":"llms-are-really-bad-at-solving-simple-river-crossing-puzzles","status":"publish","type":"post","link":"https:\/\/dailyai.com\/2024\/06\/llms-are-really-bad-at-solving-simple-river-crossing-puzzles\/","title":{"rendered":"LLMs are really bad at solving simple river crossing puzzles"},"content":{"rendered":"

Large language models like GPT-4o can perform incredibly complex tasks, but even the top models struggle with some basic reasoning challenges that children can solve.<\/strong><\/p>\n

In an interview with CBS, the \u2018godfather of AI\u2019, Geoffrey Hinton, said that \u200b\u200bAI systems might be more intelligent than we know and there’s a chance the machines could take over.<\/p>\n

When asked about the level of current AI technology Hinton said, \u201cI think we’re moving into a period when for the first time ever we may have things more intelligent than us.\u201d<\/p>\n

Meta\u2019s chief AI scientist, Yann LeCun, will have us believe that we\u2019re a long way off from seeing AI achieve even \u201cdog-level\u201d intelligence.<\/p>\n

So which is it?<\/p>\n

This week, users on X posted examples of the incredible coding ability Anthropic\u2019s new Claude<\/span><\/span><\/span> model<\/a> exhibits. Others ran experiments to highlight how AI models still struggle with very basic reasoning.<\/p>\n

River crossing puzzle<\/h2>\n

The classic river crossing puzzle has multiple variations but Wikipedia\u2019s version<\/a> sums it up like this:<\/p>\n

A farmer with a wolf, a goat, and a cabbage must cross a river by boat. The boat can carry only the farmer and a single item. If left unattended together, the wolf would eat the goat, or the goat would eat the cabbage. How can they cross the river without anything being eaten?<\/p>\n

Finding the solution requires some basic planning and reasoning on different scenarios but it\u2019s not a particularly difficult problem to solve. If you’re human.<\/p>\n

Can GPT-4o solve it? If you copy and paste the puzzle into ChatGPT it gives you the right answer, but that Wikipedia page was almost certainly in its training data.<\/p>\n

What if we made the puzzle a lot simpler and changed it slightly so the LLM couldn\u2019t rely on its training data?<\/p>\n

British Mathematics Professor Sir William Timothy Gowers showed how the inability of LLMs to apply logic is easily exposed.<\/p>\n

\"\"
ChatGPT’s failed attempt at solving a simplified river crossing puzzle. Source: X @wtgowers<\/a><\/figcaption><\/figure>\n

The correct answer to the puzzle is that only one trip is required. But it seems like ChatGPT is trying to remember an answer rather than simply reasoning through the puzzle.<\/p>\n

Is Claude Sonnet 3.5 any better?<\/p>\n

Meta Data Scientist Colin Fraser\u2019s experiment confirms that even the leading AI model currently available can\u2019t solve this simple puzzle.<\/p>\n

\n

Claude<\/span><\/span><\/span> still can’t solve the impossible one farmer one sheep one boat problem pic.twitter.com\/TU13wermLZ<\/a><\/p>\n

\u2014 Colin Fraser (@colin_fraser) June 20, 2024<\/a><\/p><\/blockquote>\n