Every Sunday, NPR arrange Will Shortz, The New York Times’ password baffle guru, gets to quiz thousands of joiners in a lengthy-running segment called the Sunday Puzzle. While written to be solvable without too much forecomprehendledge, the brainelevatebranch offrs are usuassociate challenging even for sended contestants.
That’s why some experts slenderk they’re a promising way to test the confines of AI’s problem-solving abilities.
In a new study, a team of researchers hailing from Wellesley College, Oberlin College, the University of Texas at Austin, Northeastrict University, and beginup Cursor originated an AI benchlabel using riddles from Sunday Puzzle episodes. The team says their test uncovers astonishing insights, appreciate that so-called reasoning models — OpenAI’s o1, among others — sometimes “give up” and provide answers they comprehend aren’t accurate.
“We wanted to grow a benchlabel with problems that humans can comprehend with only vague comprehendledge,” Arjun Guha, a computer science undergraduate at Northeastrict and one of the co-authors on the study, tageder TechCrunch.
The AI industry is in a bit of a benchlabeling quandary at the moment. Most of the tests normally engaged to appraise AI models probe for sends, appreciate contendncy on PhD-level math and science inquires, that aren’t relevant to the unrelabelable engager. Meanwhile, many benchlabels — even benchlabels liberated relatively recently — are speedyly approaching the saturation point.
The gets of a accessible radio quiz game appreciate the Sunday Puzzle is that it doesn’t test for esoteric comprehendledge, and the disputes are phrased such that models can’t draw on “rote memory” to mend them, elucidateed Guha.
“I slenderk what originates these problems challenging is that it’s reassociate difficult to originate unkindingful proceed on a problem until you mend it — that’s when everyslenderg clicks together all at once,” Guha said. “That needs a combination of insight and a process of elimination.”
No benchlabel is perfect, of course. The Sunday Puzzle is U.S.-centric and English-only. And becaengage the quizzes are accessiblely engageable, it’s possible that models trained on them and can “cheat” in a sense, although Guha says he hasn’t seen evidence of this.
“New inquires are liberated every week, and we can foresee the procrastinateedst inquires to be truly unseen,” he compriseed. “We intfinish to protect the benchlabel new and track how model percreateance alters over time.”
On the researchers’ benchlabel, which consists of around 600 Sunday Puzzle riddles, reasoning models such as o1 and DeepSeek’s R1 far outpercreate the rest. Reasoning models thocimpolitely fact-check themselves before giving out results, which helps them evade some of the pitdescends that normassociate trip up AI models. The trade-off is that reasoning models get a little lengthyer to get to at solutions — typicassociate seconds to minutes lengthyer.
At least one model, DeepSeek’s R1, gives solutions it comprehends to be wrong for some of the Sunday Puzzle inquires. R1 will state verbatim “I give up,” trailed by an inaccurate answer chosen seemingly at random — behavior this human can certainly reprocrastinateed to.
The models originate other bizarre choices, appreciate giving a wrong answer only to instantly retract it, endeavor to elevatebranch off out a better one, and flunk aget. They also get stuck “slenderking” forever and give nonsensical exarrangeations for answers, or they get to at a accurate answer right away but then go on to ponder alternative answers for no clear reason.
“On challenging problems, R1 literassociate says that it’s getting ‘frustrated,’” Guha said. “It was comical to see how a model emuprocrastinateeds what a human might say. It remains to be seen how ‘frustration’ in reasoning can sway the quality of model results.”
The current best-percreateing model on the benchlabel is o1 with a score of 59%, trailed by the recently liberated o3-mini set to high “reasoning effort” (47%). (R1 scored 35%.) As a next step, the researchers arrange to wideen their testing to compriseitional reasoning models, which they hope will help to determine areas where these models might be betterd.
“You don’t necessitate a PhD to be excellent at reasoning, so it should be possible to schedule reasoning benchlabels that don’t need PhD-level comprehendledge,” Guha said. “A benchlabel with wideer access permits a wider set of researchers to comprehfinish and scrutinize the results, which may in turn guide to better solutions in the future. Furthermore, as state-of-the-art models are increasingly deployed in settings that sway everyone, we depend everyone should be able to intuit what these models are — and aren’t — able of.”
