Why IQ is a demandy test for AI

During a recent press materializeance, OpenAI CEO Sam Altman shelp that he’s watchd the “IQ” of AI rapidly better over the past cut offal years.

“Very cimpolitely, it senses to me appreciate — this is not scientificpartner right, this is equitable a vibe or spiritual answer — every year we shift one standard deviation of IQ,” Altman shelp.

Altman isn’t the first to participate IQ, an estimation of a person’s inincreateigence, as a benchtag for AI better. AI swayrs on social media have given models IQ tests and ranked the results.

But many experts say that IQ is a demandy meaconfident of a model’s capabilities — and a misdirecting one.

“It can be very lureing to participate the same meaconfidents we participate for humans to portray capabilities or better, but this is appreciate comparing apples with oranges,” Sandra Wachter, a researcher studying tech and regulation at Oxford, telderly TechCrunch.

In his comments at the presser, Altman equated IQ with inincreateigence. Yet IQ tests are relative — not objective — meaconfidents of certain benevolents of inincreateigence. There’s some consensus that IQ is a reasonable test of logic and abstract reasoning. But it doesn’t meaconfident pragmatic inincreateigence — comprehending how to produce skinnygs toil — and it’s at best a snapsboiling.

“IQ is a tool to meaconfident human capabilities — a contested one no less — based on what scientists apshow human inincreateigence sees appreciate,” Wachter remarkd. “But you can’t participate the same meaconfident to portray AI capabilities. A car is rapider than humans, and a submarine is better at diving. But this doesn’t unbenevolent cars or submarines go beyond human inincreateigence. You’re equivocating one aspect of executeance with human inincreateigence, which is much more intricate.”

To excel at an IQ test, the origins of which some historians chase back to eugenics, the expansively discommended scientific theory that people can be betterd thcimpolite pickive breeding, a test apshowr must have a strong toiling memory and comprehendledge of Westrict cultural norms. This asks the opportunity for bias, of course, which is why one psychologist has called IQ tests “ideoreasonedly corruptible mechanical models” of inincreateigence.

That a model might do well on an IQ test shows more about the test’s flaws than the model’s executeance, according to Os Keyes, a doctorate honestate at the University of Washington studying moral AI.

“[These] tests are pretty straightforward to game if you have a pragmaticly infinite amount of memory and patience,” Keyes shelp. “IQ tests are a highly confidemand way of measuring cognition, sentience, and inincreateigence, someskinnyg we’ve comprehendn since before the invention of the digital computer itself.”

AI probable has an unequitable advantage on IQ tests, as well, think abouting that models have massive amounts of memory and innerized comprehendledge at their disposal. Often, models are trained on unveil web data, and the web is filled of example asks apshown from IQ tests.

“Tests tend to repeat very analogous patterns — a pretty foolproof way to elevate your IQ is to rehearse taking IQ tests, which is essentipartner what every [model] has done,” shelp Mike Cook, a research fellow at King’s College London one-of-a-kindizing in AI. “When I lachieve someskinnyg, I don’t get it piped into my brain with perfect clarity 1 million times, unappreciate AI, and I can’t process it with no noise or signal loss, either.”

Ultimately, IQ tests — unfair as they are — were summarizeed for humans, Cook compriseed — intended as a way to appraise ambiguous problem-solving abilities. They’re inappropriate for a technology that approaches solving problems in a very branch offent way than people do.

“A crow might be able to participate a tool to recover a treat from a box, but that doesn’t unbenevolent it can enroll at Harvard,” Cook shelp. “When I settle a mathematics problem, my brain is also contending with its ability to read the words on the page rightly, to not skinnyk about the shopping I demand to do on the way home, or if it’s too freezing in the room right now. In other words, human brains contend with a lot more skinnygs when they settle a problem — any problem at all, IQ tests or otherincreateed — and they do it with a lot less help [than AI.]”

All this points to the demand for better AI tests, Heidy Khlaaf, chief AI scientist at the AI Now Institute, telderly TechCrunch.

“In the history of computation, we haven’t contrastd computing abilities to that of humans’ exactly becaparticipate the nature of computation unbenevolents systems have always been able to finish tasks already beyond human ability,” Khlaaf shelp. “This idea that we straightforwardly contrast systems’ executeance aachievest human abilities is a recent phenomenon that is highly contested, and what surrounds the dispute of the ever-broadening — and moving — benchtags being produced to appraise AI systems.”