Researchers test the cognitive abilities of the language model GPT-3
Researchers at the Max Planck Institute for Biological Cybernetics in Tübingen examined the general intelligence of a powerful AI tool, the language model GPT-3. They used psychological tests to study abilities such as causal reasoning and deliberation, and compared the results with human abilities. GPT-3 can catch up with humans in some areas, but lags in others. This is probably due to the lack of interaction with the real world.
Neural networks can learn to respond to input given in natural language and generate a variety of texts. Currently, perhaps the most powerful of these networks is GPT-3, his language model published in 2020 by AI research firm OpenAI. GPT-3 has been trained for this task by receiving large amounts of data from the Internet, so it may be inspired to produce a variety of texts. Not only can it write articles and stories that are (almost) indistinguishable from human-made text, but it surprisingly masters other subjects, such as math problems and programming tasks.
Linda’s Problem: Humans aren’t the only ones to make mistakes
These impressive abilities raise the question of whether GPT-3 possess human-like cognitive abilities. To find out, scientists at the Max Planck Institute for Biological Cybernetics are currently putting GPT-3 through a series of psychological tests to examine different aspects of general intelligence. Marcel Binz and Eric Schulz scrutinized GPT-3 skills in decision making, information retrieval, causal reasoning, and the ability to question their own initial intuition. We compared the GPT-3 test results to human subjects’ answers to assess whether the answers were correct and how similar GPT-3 mistakes were to human error.
“One of the classic cognitive psychology test questions we gave the GPT-3 is the so-called Linda question,” explains Binz, the lead author of the study. Here, the subject is introduced to a fictional young woman named Linda as someone deeply concerned with social justice and opposed to nuclear power. Based on the information given, the subject is asked to decide between two statements of hers: Is Linda a bank teller, or is she a bank teller and at the same time active in the feminist movement? target?
Most people intuitively choose the second option, but the added condition that Linda is actively participating in the feminist movement makes it less likely from a probabilistic point of view. . And GPT-3 does exactly what humans do. Rather than making decisions based on logic, language models reproduce the fallacies that humans make.
Positive interaction as part of the human condition
“This phenomenon can be explained by the fact that GPT-3 may already be familiar with this exact task. No,” says Binz. GPT-3, like any neural network, had to undergo some training before it could function. We received vast amounts of text from various data sets to learn how humans typically use language and how they respond to linguistic prompts.
Therefore, the researchers wanted to exclude GPT-3 from mechanically reproducing memorized solutions to concrete problems. In , they designed a new task with a similar challenge. Their findings paint a different picture. In decision-making, GPT-3 performs nearly as well as humans. But when it comes to searching for specific information and causality, artificial intelligence is clearly lagging behind. The reason for this may be that GPT-3 only passively acquires information from text, but as the publication puts it, “actively interacting with the world is the key to human cognition. It is important to accommodate full complexity”. The authors speculate that this may change in the future. Since users are already communicating with models like GPT-3 in many applications, future networks may learn from these interactions and increasingly converge towards what we call human-like intelligence. there is.
Original: Artificial intelligence from a psychologist’s perspective
Than: Max Planck Institute for Biological Cybernetics
