Arstecnica
Things are moving at lightning speed in AI Land. On Friday, a software developer named Georgi Gerganov released “llama. I created a tool called Shortly thereafter, he devised a way to run LLaMA on Windows as well.then who showed running On your Pixel 6 phone, next raspberry pi (although it runs very slowly).
If this continues, pocket-sized ChatGPT rivals may be on the horizon.
But we’re not quite there yet, so let’s backtrack a bit. (At least not today. It’s literally today, March 13, 2023.) But who knows what will happen next week.
Since ChatGPT’s launch, some people have been frustrated by built-in limitations in AI models that prevent them from discussing topics that OpenAI deemed sensitive. Thus began the dream of an open-source Large Language Model (LLM) that anyone could run locally without censorship or paying OpenAI for his API fees in one region.
Open source solutions exist (such as GPT-J), but they require a lot of GPU RAM and storage space. No other open source alternative could boast his GPT-3 level performance on readily available consumer grade hardware.
Enter LLaMA, an LLM available with parameter sizes ranging from 7B to 65B (“B” in “Billion Parameters”, which are floating point numbers stored in a matrix representing what the model “knows” is). LLaMA argued that the smaller-sized model could rival OpenAI’s GPT-3, the base model that powers ChatGPT, in output quality and speed. There was just one problem. Meta released his LLaMA code as open source, but the “weights” (the trained “knowledge” stored in the neural network) were restricted to qualified researchers only.
Fly at LLaMA speed
Meta’s restrictions on LLaMA didn’t last long. Because on March 2nd someone leaked his LLaMA weights on BitTorrent. Since then, development around LLaMA has exploded. Independent AI researcher Simon Willison compared the situation with the release of Stable Diffusion, an open-source image synthesis model published last August. In a blog post he wrote:
It feels like the Stable Diffusion moment back in August sparked a whole new wave of interest in generative AI. It was then pushed into overdrive by the release of his ChatGPT at the end of November.
That moment of steady proliferation is happening again for large language models, the technology behind ChatGPT itself. I ran his GPT-3 class language model on my laptop for the first time this morning.
The AI stuff was already weird. It’s going to get weirder.
Running GPT-3 typically requires multiple datacenter-class A100 GPUs (GPT-3 weights not published), but LLaMA can run on one powerful consumer GPU Therefore, it became a hot topic. And now, with optimizations that reduce the size of the model using a technique called quantization, LLaMA can run on his Nvidia consumer GPUs on M1 Macs or below.
Things move so fast that it can be difficult to keep up with the latest developments. (Regarding the speed of AI progression, a fellow AI reporter of his told Ars: [They] I don’t know where to chase first and get lost in the confusion.”)
For example, here’s a list of notable LLaMA-related events based on the timeline given by Willison in his Hacker News comments:
- February 24, 2023: Meta AI announces LLaMA.
- March 2, 2023: Someone leaked the LLaMA model via BitTorrent.
- March 10, 2023: Georgi Gerganov creates llama.cpp that can run on M1 Macs.
- 11 March 2023: Artem Andreenko runs LLaMA 7B (slowly) on raspberry pi 44 GB RAM, 10 seconds/token.
- March 12, 2023: LLaMA 7B running on NPX, a node.js execution tool.
- March 13, 2023: Someone runs llama.cpp On your Pixel 6 phonealso very slowly.
- March 13, 2023: Stanford University has released Alpaca 7B, an instruction-coordinated version of LLaMA 7B. This works similarly to OpenAI’s “text-davinci-003”, but runs on much less powerful hardware.
After getting the LLaMA weights myself, I followed Wilson’s instructions and got a 7B parameter version working on my M1 Macbook Air. It works reasonably fast. If you prompt and call it as a script on the command line, LLaMA will do its best to complete it in an appropriate manner.
Benj Edwards / Ars Technica
There is still the question of how much quantization affects the quality of the output. In our tests, when LLaMA 7B was trimmed to 4-bit quantization, it performed very impressively on the MacBook Air, but it still wasn’t on par with what we’d expect from ChatGPT. may well produce better results.
Also, LLaMA still has fairly restrictive usage conditions, but optimizations and tweaks will be coming soon once everyone gets their hands on the code and weights. Today’s release of his Alpaca by Stanford University proves that tweaking (additional training with a specific goal in mind) can improve performance, but it’s still early days after the release of LLaMA. .
As of this writing, running LLaMA on a Mac is still quite a technical challenge. You should have Python and Xcode installed and comfortable working at the command line. Willison has a good step-by-step explanation for those who want to try it.
No one knows yet what this technology will mean for the world. While some are concerned about AI’s impact as a tool for spam and misinformation, Willison said, “AI isn’t going to be invented, so finding ways to use it most constructively is a top priority. I think,” he said.
For now, our only guarantee is that things will change rapidly.
