Stanford copies the ChatGPT AI for less than $600

Six months ago, researchers and researchers alone were tracking the development of language models at scale. But with the launch of his ChatGPT late last year, the flip side of humanity has exploded. Machines are now able to communicate in ways that are almost indistinguishable from humans. They can write text and programming code in seconds across a dizzying array of subject areas, often to a very high standard. As the launch of GPT-4 shows, they are improving at meteoric speeds, and by potentially automating a wide variety of work tasks, especially among white-collar workers, they will outperform most other technologies. We are in a position to fundamentally transform human society so that we cannot. Previously thought impossible.

Many other companies, notably Google, Apple, Meta, Baidu, and Amazon, are not far behind, and their AI will soon flood the market and be embedded in every application and device. If you are her Bing user, the language model is already built into the search engine. They’re in your car, your phone, your TV, waiting on the other end of the line whenever you try to call a business.

One small consolation is that OpenAI and other big corporations don’t want these machines to spam, misinformation, create malware, targeted harassment, and any other kind of stuff most people can agree to make the world. is aware of the insane potential of use cases for worse place. They spend months manually trimming these features before launch. OpenAI CEO Sam Altman is one of many who are concerned that the government isn’t moving fast enough to put up fences around his AI in the name of public interest.

But what about a language model you can build yourself for $600? A team of researchers at Stanford University did just that. Its impressive performance highlights how this entire sector and its fantastic features can spiral out of control.

The Stanford research team started with Meta’s open source LLaMA 7B language model. This is the smallest and cheapest of the several LLaMA models available. This tiny language model, pre-trained with a trillion “tokens”, had some built-in functionality, but lagged far behind ChatGPT on most tasks. A major cost, and indeed a major competitive advantage, of GPT models stems from OpenAI’s heavy reliance on enormous post-training time and effort. It’s one thing to read a billion books, and another to chew through a ton of question-and-answer conversation pairs that teach these AIs what their real jobs will be.

So after getting the LLaMA 7B model up and running, the Stanford team basically asked GPT to take 175 human-made instruction/output pairs and generate them 20 at a time in the same style and format. Did. This was automated via one of OpenAI’s convenient APIs, and in a short period of time, the team created around 52,000 sample conversations to use after training his LLaMA model. The cost of generating this large amount of training data is less than US$500.

We then used that data to fine-tune the LLaMA model. This process took about 3 hours on eight 80 GB A100 cloud processing computers. This cost less than US$100.

We then tested the resulting model, called Alpaca, against ChatGPT’s underlying language model in various domains such as email composition, social media, and productivity tools. Alpaca won his 90th in these tests and GPT his 89th.

“This result was quite surprising given the small size of the model and the small amount of data to follow,” the team wrote. “Besides leveraging this static evaluation set, we have been interactively testing Alpaca models and have found that Alpaca often behaves similarly to text-davinci-003. [GPT-3.5] For different sets of inputs. We recognize that our assessments may be limited in scale and diversity. ”

The team said they could probably do this cheaper if they had considered optimizing the process. You now have access to some more powerful LLaMA models to use. Don’t stop at 52,000 questions.

The team at Stanford University has released the 52,000 questions used in this study on Github, along with the code to generate more and the code used to fine-tune the LLaMA model. The team “has yet to make any tweaks to make the Alpaca model safe and harmless,” notes the team, and anyone who sets up Alpaca will report any safety and ethical issues they find. I am asking for

So, how do you stop basically anyone from writing their own pet AI right now for $100 or so and training it to do what they want? cannot be used to develop models that compete with OpenAI.” Meta also said that the entire LLaMA model was leaked on 4chan a week after he announced it, so at this stage he is only allowing academic researchers to use LLaMA under a non-commercial license. I’m here.

Ah, another group reduced the cost of cloud computing, released more code on Github that could run on a Raspberry Pi, and completed the training process in less than 5 hours on a single high-end nVidia RTX 4090 graphics card. said he was able to

What does this mean? This means that an unlimited number of uncontrolled language models can now be configured by people with machine learning knowledge who don’t care about terms of service or software piracy.

It also muddies the waters for commercial AI companies working to develop their own language models. Given that much of the time and expense involved occurs in the post-training phase, and this work can be more or less eclipsed by the time it takes to answer 50 or 100,000 questions, it makes sense for companies to continue spending this cash. Does that make sense? ?

For everyone else, it’s hard to say, but the amazing features of this software can certainly come in handy for authoritarian regimes, phishing scams, spammers, and other dangerous individuals.

The genie is out of the bottle, and duplicating and retraining already seems incredibly easy. put on a hat

Source: Stanford by AI Explained