Welcome to AI Decoded,Fast Company’s weekly newsletter that breaks down the most important news in the world of AI. You can sign up to receive this newsletter every week via emailhere.
François Chollet on AI benchmarks
I wrote an exclusive feature this week about the launch of a new AI benchmark called ARC-AGI-3. The benchmark was created by influential AI researcher Francois Chollet, who also created the widely-used Keras deep learning framework, a simplified toolkit for building AI models. Chollet has long argued that current AI models are limited in their ability to navigate novel situations and problems. The ARC test, which humans can master but not most AI systems, is designed to lay bare that limitation. My interview with Chollet contained a lot of general insights that didn’t make it into the story. Here are some of them (with my annotations in bold.)
ARC-AGI-3 asks AI agents to navigate a series of simplistic video games, without instructions. Here’s Chollet on why current models struggle to do that:
“It’s because they are reliant on memorization and retrieval, and the game is something they’ve never seen before. They’ve never played that particular game before or games like it, because each one is unique. So they’re lost. But a human is generally intelligent. A human is never lost. A human figures it out on the fly because they have fluid intelligence.”
I began to imagine how I would approach figuring out the games. I suggested to Chollet that my main strategy would be thinking about similar scenarios I’d seen in the past or in other contexts, and trying to apply them.
“Models have a lot of abstractions encoded in them. They have in fact more knowledge than you do. But they have very low ability to recombine that knowledge at test time to make sense of something they’ve never seen before. It’s the way the entire paradigm works. We are really good at absorbing knowledge, absorbing lots and lots of patterns. Better than the human brain, and at a much bigger scale. We are very bad at fluid intelligence, which is taking those patterns and actually combining them on the fly to form a new model [of a problem].”
On what exactly an AI model would need to score highly on the benchmark:
Expand to continue reading ↓