Microsoft unveils AI model that understands image content, solves visual puzzles

On Monday, Microsoft researchers discovered a multimodal computer capable of analyzing images of content, solving visual puzzles, performing visual text recognition, passing visual IQ tests, and understanding natural language instructions. We announced the model Kosmos-1. Researchers believe that multimodal AI, which integrates different input modes such as text, speech, image, and video, is important for building artificial general intelligence (AGI) that can perform common tasks at the human level. I think it’s a step.

“intelligence, being a fundamental part of multimodal Perception is necessary to achieve artifacts general intelligencein terms of knowledge acquisition The researchers state in an academic paper, “You Need More Than Just Language: Matching Perceptions to Language Models.”

A visual example from the Kosmos-1 paper shows the model analyzing images, answering questions about them, reading text from images, writing image captions, and performing visual IQ tests with 22-26% accuracy. (see below for details).

While the media is buzzing with news about large language models (LLMs), some AI experts point to multimodal AI as a potential path to artificial intelligence in general. .AGI is a clear goal of his OpenAI, a major business partner of Microsoft in the AI ​​field.

In this case, Kosmos-1 looks like a pure Microsoft project without OpenAI involved. Because of their roots in natural language processing, such as text-only LLMs such as ChatGPT, researchers refer to their creation as a “multimodal large-scale language model (MLLM).” For Kosmos-1 to accept image input, the researcher must first convert the image into a special set of tokens (essentially text) that her LLM can understand. The Kosmos-1 paper explains this in detail.

For the input format, we flatten the input as a sequence decorated with special tokens. in particular, and are used to indicate the start and end of the sequence.special token and indicate the start and end of the encoded image embedding. for example,” document ‘ for text input, ‘paragraph image embedding paragraph ‘ is an interleaved image and text input.

… embedding modules are used to encode both text tokens and other input modalities into vectors. The embeddings are then fed to the decoder. For input tokens, use a lookup table to map to embeddings. For continuous signal modalities (images, audio, etc.), we can also represent the inputs as discrete codes and regard them as “foreign languages”.

Microsoft trained Kosmos-1 using data from the web, including excerpts from The Pile (an 800 GB English text resource) and Common Crawl. After training, they performed Kosmos- 1 ability was rated. In many of these tests, the Kosmos-1 outperformed the current state-of-the-art, according to Microsoft.

Of particular interest is Kosmos-1’s performance in Raven’s Progressive Reasoning. It measures visual IQ by presenting a series of shapes and asking test takers to complete a series of patterns. To test Kosmos-1, researchers took her one filled-in test at a time, completing each option and asking if her answer was correct. Kosmos-1 only answered Raven test questions correctly 22% of the time (26% with tweaks). Although this was by no means a slam-dunk and methodological errors may have affected the results, Kosmos-1 outperformed random chance (17%) on the Raven IQ test.

Still, Kosmos-1 represents an early stage in the multimodal domain (an approach that other approaches are also pursuing), but future optimizations will yield even more significant results, enabling AI models to recognize all forms of media. It’s easy to imagine being able to do something and act on it. , which greatly improves the ability of artificial assistants. In the future, the researchers say, they hope to increase the model size of Kosmos-1 and also integrate speech capabilities.

Microsoft says it plans to make Kosmos-1 available to developers, but the GitHub page the paper cites has no apparent Kosmos-specific code at the time of this article’s publication. .