The success of ChatGPT and its competitors is based on what we call emergency action. These systems, called large-scale language models (LLMs), have not been trained to output natural language (or effective malware). They were simply tasked with tracking word usage statistics. However, given a sufficiently large training set of language samples and a sufficiently complex neural network, their training resulted in an internal representation that “understood” an overview of English usage and facts. Their complex behavior came from much simpler training.
Meta’s team reasoned that this kind of urgent understanding should not be limited to language. So he trained LLM on the statistics of amino acid occurrences in proteins and used the system’s internal representation of what it learned to extract information about the structure of these proteins. The results are not as good as the best competing AI systems for predicting protein structure, but they are fairly fast and continue to improve.
LLM: More than just language
The first thing we need to know in order to understand this work is that the term “language” with the name “LLM” refers to the original development of the language processing task, but it can be used for a variety of purposes. It means there is a possibility. Language processing is therefore a common use case for LLM, but these models also have other capabilities. In fact, the term “large scale” is much more useful in that every LLM has a large number of nodes (the “neurons” of a neural network) and an even larger number of values representing the weights of the connections between those nodes. is beneficial to It was originally developed to handle languages, but could potentially be used for a variety of tasks.
The challenge of this new research is to obtain a linear string of protein-forming amino acids and use it to predict how those amino acids will be arranged in three-dimensional space after the protein has matured. This 3D structure is essential for protein function and helps us understand how proteins malfunction after detecting mutations. It also makes it possible to design drugs that inactivate proteins of pathogens. Predicting protein structure has plagued generations of scientists until his decade when Google’s AI group DeepMind unveiled a system that solved the problem with the most practical definition of “solved.” It was a challenge. Google’s system was soon followed by systems developed along similar lines by the academic community.
Both of these efforts relied on the fact that evolution had already created a large set of related proteins that adopted a similar 3D organization. By aligning these related proteins, the AI system can determine where and what changes can be tolerated while maintaining similar structures, and how changes in one part of the protein are compensated by changes in others. You can guess what These evolutionary constraints allow the system to calculate which parts of the protein must be close to each other in his 3D space, i.e. what structures they are likely to have.
The reasoning behind Meta’s new work is that training an LLM-style neural network allows the system to sort out the same type of evolutionary constraint, thus avoiding the tedious task of aligning all protein sequences. because you won’t have to. first place. Just as the rules of grammar emerge from training his LLM on language samples, the constraints imposed by evolution emerge from training the system on protein samples.
Focus on amino acids
How this worked in practice is that the researchers took a large sample of protein and randomly blocked the identities of some individual amino acids. to the system. During this training process, the system developed the ability to use information such as statistics on amino acid frequencies and the context of surrounding proteins to make inferences. Implied in that context are things that have required dedicated handling in previous efforts. That is, the identities of proteins related by evolution and which variations within their relatives tell us which parts of the protein are closer to each other in her 3D space.
Assuming that the reasoning about how LLM works is true (and Meta was building on previous research suggesting it), the trick to developing a working system is , to retrieve the information contained in the neural network. Neural networks are often considered “black boxes”. The reason is that we don’t always know how the neural network makes decisions. But that’s becoming less and less true as people incorporate features such as the ability to audit their decision-making processes.
In this case, the researchers relied on LLM’s ability to explain what they called “patterns of attention.” In fact, if you give an LLM a set of amino acids and ask him to evaluate them, the attention pattern is the set of features that he looks at with the LLM to perform the analysis.
To convert attention patterns to 3D structures, the researchers trained a second AI system to associate attention patterns of proteins with known 3D structures with their real structures. Having only experimentally determined the structures of a limited number of proteins, the researcher also used some of the structures predicted by one of his other AI systems as part of this training. .
The resulting system was called ESM-2. Once fully trained, ESM-2 was able to take a raw string of amino acids and output a 3D protein structure along with a score representing confidence in the accuracy of that structure.
