Orrich Lawson | Stable Diffusion
Advances in AI systems often feel cyclical. Every few years, computers suddenly do things they couldn’t do before. “Look!” A true believer in AI proclaims, “The age of artificial general intelligence is approaching!” “Nonsense!” says the skeptic. “Remember self-driving cars?”
The truth is usually somewhere in between.
we are in another cycle. Now it’s generative AI. While media headlines are dominated by news about AI art, we’re also seeing unprecedented progress in many diverse areas. Everything from video to biology to programming to writing to translation and more is advancing AI at a staggering pace.
Why is all this happening now?
You may be familiar with the latest happenings in the world of AI. I’ve seen award-winning work, listened to interviews with deceased people, and read about breakthroughs in protein folding. But these new AI systems don’t just create cool demos in the lab. They are rapidly turning into practical tools and real commercial products that everyone can use.
There is a reason why all this was done at once. All of these breakthroughs are underpinned by a new class of AI models that are more flexible and powerful than ever before. It is known as Large Language Model (LLM) because it was first used for linguistic tasks such as answering questions and writing essays. OpenAI’s GPT3, Google’s BERT, etc. are all LLMs.
However, these models are very flexible and adaptable. The same mathematical constructs have been so useful in things like computer vision and biology that some researchers refer to them as “fundamental models” to better clarify their role in modern AI. increase.
Where did these underlying models come from and how did they evolve beyond language and drive much of what we see in AI today?
Fundamental model foundation
Machine learning has a trinity: model, data, and computation. A model is an algorithm that takes an input and produces an output. Data refers to the examples on which the algorithm is trained. To learn anything, you need enough rich data that the algorithm can produce useful output. The model should be flexible enough to capture the complexity of the data. Finally, you need enough computing power to run your algorithm.
The first modern AI revolution happened in deep learning in 2012 when convolutional neural networks (CNNs) began solving computer vision problems. CNNs are similar in structure to the visual cortex of the brain. They have been around since his 1990s, but were still impractical due to the demanding computational power required.
However, in 2006 Nvidia released CUDA, a programming language that allows GPUs to be used as general-purpose supercomputers. In 2009, his AI researchers at Stanford University introduced Imagenet, a collection of labeled images used to train computer vision algorithms. In 2012, AlexNet combined GPU-trained CNN and Imagenet data to create the best visual classifier the world has ever seen. From there, deep learning and AI exploded.
CNNs, ImageNet datasets, and GPUs were a magical combination that unlocked incredible advances in computer vision. 2012 saw a boom in deep learning, spawning entire industries such as self-driving cars. However, we quickly learned that that generation of deep learning had its limits. CNN was good at vision, but lacked model breakthroughs in other areas. One of the big gaps was in natural language processing (NLP). That is, to allow computers to understand and process normal human language rather than code.
The problem of understanding and dealing with language is fundamentally different from the problem of dealing with images. A processing language must process sequences of words in which the order is important. A cat is a cat anywhere in the image, but there is a big difference between “this reader is learning about the AI” and “the AI is learning about this reader”.
Until recently, researchers relied on models such as recurrent neural networks (RNN) and long short-term memory (LSTM) to process and analyze data in time. These models were effective at recognizing short sequences, such as short spoken phrases, but struggled with long sentences and paragraphs. The memories of these models were not sophisticated enough to capture the complexity and richness of ideas and concepts that arise when sentences were combined into paragraphs or essays. Optimal, but otherwise not very suitable.
Getting the right training data was another challenge. ImageNet is his collection of 100,000 labeled images, and its generation required a great deal of human effort, mostly by graduate students and Amazon Mechanical Turk employees. ImageNet was actually inspired by and modeled on an old project called WordNet that attempted to create a labeled data set of English vocabulary. There is no shortage of text on the Internet, but creating a meaningful data set and training a computer to process human language beyond individual words is very time consuming. increase. Also, the labels he creates for the same data for one application may not apply to another task.
