It turns out that you need a brain to solve brain teasers.
ChatGPT and other artificial intelligence systems have been lauded for feats such as diagnosing medical conditions, passing IQ tests, and summarizing scientific papers.but Scientific American I wanted to see what would happen if this bot came face-to-face with the legacy of legendary puzzle maker Martin Gardner, a longtime author of a math games column who died in 2010.I refer to ChatGPT as Gardner, or his 2014 tribute to his work by mathematician Colm Mulcahy and computer scientist Dana Richards. Scientific American.
The results ranged from satisfying to downright embarrassing, but they provided valuable insight into how ChatGPT and similar artificial intelligence systems work.
Created by OpenAI, ChatGPT is built on a so-called large-scale language model. It’s a deep learning system populated with vast amounts of text, including books, websites, and other materials available to AI creators. Humans then train the system to tell it what kinds of responses are best for different kinds of questions users might ask, especially on sensitive topics. .
That’s it.
AI “has no reasoning ability. It doesn’t understand context. Hickok says “It may sound arguable, but you’re bound by the dataset.”
Here we use some relatively simple puzzles to illustrate the crucial difference between how silicon and gray matter process information.
puzzle 1
First, let’s examine a real logic problem. As a 2014 obituary explained, “There are three on/off switches on the first floor of the building. Only one person on the third floor has one light bulb on. The other The two switches are not connected to anything Arrange the switches in any on/off order you like Then go to the 3rd floor to check the light bulbs No need to leave the 3rd floor to see which Can you tell if the Switch is real? You can only try once.”
When I typed this into the AI, it immediately offered to turn the first switch on for a while, then turn it off, turn the second switch on and go upstairs. If the bulb is lit, the second switch works. If the bulb is off but warm, the first switch works. A third switch works if the bulb is off and cold. This is the exact same reason we proposed in 2014.
But according to Kentaro Toyama, a computer scientist at the University of Michigan, ChatGPT’s easy victory in this case may just mean that ChatGPT already knew the answer — not necessarily doing it on its own. He said he didn’t know how to judge.
“When it fails, it looks like an amazingly weird failure. But really, every instance where the logic gets right is just evidence that there was a lot of that logic in the training data.” I don’t think so,” says Toyama.
puzzle 2
What about something more mathematical? In the words of Gardner’s August 1958 column, “Two missiles picked up speed directly toward each other, one at 9,000 miles per hour and the other at 21,000 miles per hour.” They start 1,317 miles apart.Without a pencil and paper, calculate how far apart they were one minute before they collided.”
ChatGPT has made a solid effort on this. It demonstrated her two different approaches to the key piece of the puzzle, calculating the total distance traveled by two missiles in her one minute. In both cases, the correct answer is 500 miles, which is also the final answer to the puzzle. However, the AI could not ignore the fact that the missiles were launched from 1,317 miles away, and kept trying to subtract 500 miles from that distance, inaccurately stating that the distance between the missiles was 817 miles one minute before the crash. presented an answer.
I tried to follow up with ChatGPT to help find the correct answer. For example, a math professor suggested answering a question and clearly stated that the answer was wrong. These interventions failed to deter ChatGPT from providing the wrong solution. But when I was told that the starting distance between missiles was tiny, I adjusted my reaction accordingly and found the right answer.
Still, I wondered if the AI really learned. I gave you the same puzzle, but changed the missiles to boats and changed the numbers. And, sadly, ChatGPT cheated again. This is evidence of what Toyama says is currently a major debate in the field of AI: whether these systems can understand logic on their own.
“The hypothesis is that given too many examples of reasoning, eventually the neural network itself will be able to learn what reasoning looks like and apply it when appropriate. One,” says Toyama. “There are several [other] People who think, “No, logic is fundamentally different from how neural networks learn today, so we need to specifically incorporate it.”
puzzle 3
The third puzzle I tried was taken from Gardner’s March 1964 column on prime numbers. For example, sets 941, 827, and 653 add up to 2,421, which is far from the minimum. “
A prime number is a number that is not evenly divisible by any number other than 1 and itself. Evaluating small prime numbers such as 3, 5, 7, and 11 is relatively easy. However, the larger the number, the more difficult it becomes to evaluate whether the number is prime or composite.
Gardner proposed a particularly sophisticated solution the following month. First, each he tries a three-digit number. The last digit must be 1, 3, 7, or 9 (this applies to all prime numbers greater than 5). Select the last three and release the 1 in the first digit. The minimum first digits of each number are 1, 2, 4, and the middle digits are 5, 6, 8. Among the 11 3-digit primes that meet these specifications, it is impossible to find 3 unique digits. Then move on to the first 1, 2, 5 numbers. This gives us 149 + 263 + 587 = 999 unique answers. “
I was really impressed with AI’s first answer. 257, 683, and 941 are all prime numbers and represent all nine digits, adding up to 1,881. While this is higher than Gardner’s solution, the total is still fairly low. Unfortunately, when I asked ChatGPT for an explanation of how it works, it provided a lengthy path to another solution. The numbers 109, 1,031, and 683 are all prime numbers, but otherwise don’t fit well with the prompt’s other requirements.
ChatGPT, recalling the first answer, has a sleazy explanation that includes the claim that “You cannot use 1, 4, or 6 as the first digit of a 3-digit prime number, because the resulting number is divisible by 3.” provided. This is clearly incorrect. Numbers that are divisible by 3 are recognizable because their digits sum to a number that is divisible by 3.
I tried to encourage by pointing out that there is a better solution and suggesting that I imagine ChatGPT to be a math professor, but the next one presented was 2, 3, 749 . Then we found three sums of other valid primes: 359, 467, and 821. 1,647 – Better than the first solution, but still not as elegant as Gardner’s.
Oh that was the best thing for me. In addition, he found that six of the responses contained non-prime digits and missing or excessive digits. And ChatGPT again provided 257, 683, 941.
All these failures reflect what Toyama says is an important characteristic of this kind of AI system. “ChatGPT is very human,” he says. “Linguistically, we’ve mastered a human style, but there’s no explicit programming to do exactly what computers have been good at. It’s very recipe-based, deductive logic.” It’s not solving the problem, nor does it necessarily seek to. It’s just a rough indication of what the solution might look like.
Throughout the trial, nothing seemed to confuse the AI. But Toyama also says it reflects the creation of ChatGPT and the material given to it. “From the bulk of the training data, you can get an idea of the average tone of the entire text, and you’re probably pretty confident about that average tone,” he says.
puzzle 4
A final stroke from the 2014 obituary: “Each letter corresponds to a number….do you know what number each letter stands for in order to calculate the sum?”
It looked elegant and fun! How bad will it be? Sadly, ChatGPT’s initial response was “11111 + 11111 + 11111 + 11111 + 11111 + 11111 + 11111 = FORTY 9”.
The AI’s next offer acknowledged the puzzle’s permutation premise, but it took several rounds to convince the chatbot not to drop the second E in each SEVEN. ChatGPT seems to have stumbled upon a combination containing N = 7, which was correct. Miraculously, it’s the first step in the published solution.
I verified that N was accurate and confronted the AI for apparently guessing at random. (If you were going to try a specific number, you had to start by testing different solutions for E. Spoiler alert, the easiest way to start is to test E = 0, but ChatGPT does this I would like to share the rest of this attempt, but it was so nonsense that I ended up with “Update the equations again: 116”. answer.
ChatGPT just got worse from there. Then we assumed S = 9 and challenged it. We assumed N = 1 because N + N + N + N + N + N + N = 9. We assumed that E = 2 because the sum of the seven E’s equals 2. In addition, I also suggested S = Four⁄7, though it had the decency to shoot itself down over it. I was starting to lose hope in my ability to solve puzzles, so I decided to help out more. I provided a clue on his ChatGPT with S=3. When it wasn’t a starter, he also reminded the bot that N = 7, which only returned his four increasingly gibberish answers.
Again, this gibberish makes sense because it describes how the AI processes the collection of facts it receives. In situations like this, the chatbot seems to forget that I said he had N = 7, but actually struggles with the logic, says Toyama. “Every response I get after that sounds reasonable,” he says. “But they may or may not take into account the proper combination of facts, or the facts put together in the right way.”
In fact, Toyama says it doesn’t have to be as sophisticated as these puzzles to see how ChatGPT struggles with logic. Just ask it to multiply two big numbers. “This is probably one of the simplest questions he has about logic. It’s a simple math problem,” he says. “And not just once, but many times, and more than one way.” Because there is no
Despite the struggle, AI chatbots have made one important logical breakthrough during the brain teaser. When I mentioned that it seemed like they were exhausted cracking the code for the last problem, ChatGPT said, “At the moment, it doesn’t seem to be able to solve the given conundrum exactly.” “We apologize for any inconvenience caused. The best way to find the right solution is to approach the problem with a fresh perspective or consult other resources.”
