Orrich Lawson | Getty Images
On Thursday, OpenAI announced a plugin system for its ChatGPT AI assistant. Plugins allow ChatGPT to interact with the wider world over the Internet, whether you’re booking flights, ordering groceries, browsing the web, and more. Plugins are bits of code that tell ChatGPT how to use external resources on the Internet.
Basically, if a developer wants ChatGPT to be able to access any network service (e.g. “find the current stock price”) or perform tasks controlled by the network service (e.g. “Ordering pizza over the internet”) is now possible, as long as it doesn’t violate OpenAI’s rules.
Traditionally, most Large Language Models (LLMs) like ChatGPT have been trapped in a sort of bubble, allowing them to interact with the world only through text conversations with their users. As OpenAI writes in an introductory blog post about his ChatGPT plugin, “The only thing a language model can do right away is output text.”
Bing Chat has taken this paradigm further by making the web searchable for newer information, but so far ChatGPT remains isolated from the wider world. While closed like this, ChatGPT will only have data from its training set (limited to 2021 and earlier) and information provided by users during conversations. ChatGPT is also prone to factual errors and errors (what AI researchers call “hallucinations”).
To get around these limitations, OpenAI burst the bubble and created the ChatGPT plugin interface (what OpenAI calls ChatGPT’s “eyes and ears”). This allows developers to “plug in” to ChatGPT and create new components that allow AI models to interact. other services on the Internet. These services can perform calculations and refer to factual information to reduce hallucinations. It may also interact with other software services on the Internet if developers create plug-ins for that task.
What kind of plugin are you talking about?
For ChatGPT, OpenAI allows users to select from a list of plugins before starting a ChatGPT session. They look like apps in the app store, and each plugin has its own icon and description.
According to OpenAI, the first round of plugins was created by the following companies:
- Expedia (for travel planning)
- FiscalNote (for real-time market data)
- Instacart (for grocery orders)
- Kayak (search for planes and rental cars)
- Klarna (price comparison shopping)
- Milo (AI-Powered Parental Assistant)
- OpenTable (restaurant recommendations and reservations)
- Shopify (for shopping on that site)
- Slack (for communication)
- Speak (for AI-powered language teaching)
- Wolfram (for computation and real-time data)
- Zapier (automation platform)
In particular, the Zapier plugin seems particularly powerful as it gives ChatGPT access to existing software automation systems. As Zapier puts it, “You can ask ChatGPT to perform any of Zapier’s 50,000 actions, including search, update, and write. Compose and send emails. Find and update contacts directly in your CRM. Add rows to spreadsheets and send them as Slack messages. The possibilities are endless.”
OpenAI uses a web browser (which can retrieve information from the web in a manner similar to Bing Chat), a code interpreter (in a sandbox) for running Python programs, and a “person or organization hosted elsewhere”. “Information Sources (essentially getting information from documents).
