Developing a state-of-the-art AI system like ChatGPT requires enormous technical resources. One reason is that they are expensive to develop and run. There are some open-source efforts to reverse-engineer proprietary closed-source systems created by commercial labs such as Alphabet’s DeepMind and OpenAI, but mostly due to a lack of funding and domain expertise. and often encounter obstacles.
To avoid this fate, one community research group, EleutherAI, is setting up a non-profit foundation. The organization today launched its nonprofit research institute, funded by donations and grants from backers including AI startups Hugging Face and Stability AI, former GitHub CEO Nat Friedman, Lambda Labs, and Canva. Announced the establishment of the EleutherAI Institute.
“Formalizing it as an organization allows us to build a full-time staff and work on longer and more complex projects than would be feasible as a volunteer group,” says Booz Allen Hamilton, an AI researcher. said Stella Biderman, who co-runs the EleutherAI Institute. She told TechCrunch in her email interview. “Especially in terms of nonprofits, I think it’s a no-brainer given our focus on research and the open source space.”
EleutherAI started several years ago as a grassroots collection of developers working on open source AI research. Its founding members Connor Leahy, Leo Gao, and Sid Black wrote the code and gathered the data needed to create a machine learning model close to OpenAI’s text-generating GPT-3. GPT-3 received a lot of press at the time.
EleutherAI has curated and open sourced The Pile, a collection of datasets designed to be used to train GPT-3-like models to complete texts or write code. bottom. We have also released several models under the Apache 2.0 license, such as GPT-J and GPT-NeoX. These language models have sparked a whole new wave of startups for some time.
EleutherAI primarily relied on the TPU Research Cloud to train its models. TPU Research Cloud is a Google Cloud program that supports projects in the hope that results will be published. CoreWeave, a US-based cryptocurrency miner that offers cloud services for AI workloads, also provided computing resources to his EleutherAI in exchange for models that customers could use and offer.
EleutherAI has grown rapidly. Today, more than 20 of his regular contributors to the community work full-time and are primarily dedicated to research. Over the past 18 months, the EleutherAI member has co-authored his 28 academic papers, trained dozens of models, and released 10 codebases.
However, the capricious nature of cloud providers sometimes forced EleutherAI to abandon its plans. Initially, the group intended to release a model about the same size as he GPT-3 in terms of number of parameters, but had to shelve that roadmap for technical and financial reasons. rice field. (In AI, parameters are the parts of the model that have been learned from past training data and essentially define the model’s skill at a problem, such as generating text.)
In late 2022, EleutherAI became acquainted with Stability AI, the now well-funded startup behind image-generating AI system Stable Diffusion. Together with other collaborators, helped create an early version of Stable Diffusion. Since then, Stability AI has donated some of the computation from his AWS cluster to his EleutherAI’s ongoing language model research.
After Hugging Face, another big patron, approached EleutherAI, Biderman says nonprofit talks began. (Many staffers at EleutherAI were involved in his BigScience efforts at the company to open-source a GPT-3-like model he spent a year training.)
“EleutherAI has historically focused on large-scale language models that are architecturally similar to ChatGPT, and will continue to do so,” said Biderman. “We are pleased to be able to devote more resources to ethics, interpretability and coordination work, rather than just training language models at scale.”
One might wonder if the involvement of commercial ventures like Stability AI and Hugging Face (both backed by large amounts of venture capital) will influence EleutherAI’s research. yeah. That’s a natural assumption, and it’s backed by evidence. is shown.
Biderman insists the EleutherAI Foundation will remain independent and says the donor pool is fine so far.
“We don’t develop models at the request of commercial entities,” Biderman said. “If anything, I think having diversified sponsorships gives us more independence. It seems so.”
Another challenge that the EleutherAI Foundation has to overcome is making sure it doesn’t run out of funding. OpenAI is a didactic story. After being founded as a nonprofit in 2015, the company has transitioned to a “capped profit” structure to fund ongoing research.
Broadly speaking, non-profit initiatives to fund AI research are mixed.
Among the success stories is the Allen Institute for Artificial Intelligence (AI2), founded by the late Microsoft co-founder Paul Allen. It is also home to the Alan Turing Institute, a UK-based government-funded national laboratory for data science and machine learning. Smaller but promising efforts include AI startup Cohere’s Cohere For AI (despite its corporate affiliation) and Timnit Gebru’s Distributed AI Research, a global distributed research organization.
But all AI2 has a fund for AI research from former Google chairman Eric Schmidt. At more than $125 million, it sparked new controversy after Politico reported that Schmidt was wielding unusually large influence over the White House Office of Science and Technology Policy.
Only time will tell which direction the EleutherAI Foundation will ultimately go. Perhaps the mission will evolve and change over time — in a positive way, we can’t help but hope.
