Benj Edwards / Ars Technica
On Friday, Meta announced a new AI-powered Large Language Model (LLM) called LLaMA-13B. It claims to outperform OpenAI’s GPT-3 model despite being “10x smaller”. Small AI models could lead to running ChatGPT-style language assistants locally on devices such as PCs and smartphones. It is part of a new family of language models called “Large Language Model Meta AI” or LLAMA for short.
The LLaMA collection of language models ranges in size from 7 billion to 65 billion parameters. By comparison, his GPT-3 model from OpenAI, the base model behind ChatGPT, has 175 billion parameters.
Meta trained the LLaMA model using publicly available datasets such as Common Crawl, Wikipedia, and C4. This means the company may be able to release the model and weights open source. This is a dramatic new development in an industry where the big tech players in the AI race have so far owned the most powerful AI technology for themselves.
“Unlike Chinchilla, PaLM, or GPT-3, we only use publicly available datasets to make our work open source compatible and reproducible, but most existing models are It relies on unpublished or undocumented data.” murmured Project member Guillaume Rumple.
Today we released 4 base models LLaMA ranging from 7B to 65B parameters.
LLaMA-13B outperforms OPT and GPT-3 175B in most benchmarks. LLaMA-65B competes with Chinchilla 70B and PaLM 540B.
All model weights are public and available at https://t.co/q51f2oPZlE.
1/n pic.twitter.com/DPyJFBfWEq— Guillaume Lample (@GuillaumeLample) February 24, 2023
Meta calls the LLaMA model the “basic model”. This means that this model is intended to form the foundation for future more sophisticated AI models built from technology, just as OpenAI built his ChatGPT from the foundation of GPT-3. means The company hopes that LLaMA will help in natural language research, potentially enhancing applications such as “question answering, natural language understanding or reading comprehension, understanding the capabilities and limitations of current language models.”
The top LLaMA model (LLaMA-65B with 65 billion parameters) is on par with similar products from competing AI labs DeepMind, Google, and OpenAI, but perhaps the most interesting development comes from LLaMA. . The -13B model is reported to outperform GPT-3 when running on a single GPU, as mentioned above. Unlike the data center requirements of its GPT-3 derivatives, LLaMA-13B opens the door to ChatGPT-like performance on consumer-grade hardware in the near future.
Parameter size is important in AI. Parameters are variables that machine learning models use to make predictions or classifications based on input data. The number of language model parameters is a key factor in its performance. In general, larger models can handle more complex tasks and produce more consistent output. However, using more parameters takes more space and requires more computing resources to run. Therefore, if one model can achieve the same result as another model with fewer parameters, it will be much more efficient.
Independent AI researcher Simon Willison said, “Within a year or two, we’ll be able to run language models with a significant portion of ChatGPT’s functionality on our own (best of breed) phones and laptops. I think it will be like that,” he said. A Mastodon thread analyzing the impact of Meta’s new AI model.
A simplified version of LLaMA is now available on GitHub. To receive the complete code and weights (the training data “learned” by the neural network), Meta provides a form that interested researchers can request access to. At this time, Meta has not announced plans for a wider release of the model and weights.
