Arstecnica
On Friday, Koko co-founder Rob Morris announced The Verge reported on Twitter that his company conducted an experiment to provide mental health counseling with AI to 4,000 people without prior notice.the critics called The experiment was highly unethical because Coco did not obtain informed consent from those seeking counseling.
Koko is a non-profit mental health platform that connects teens and adults in need of mental health support to volunteer through messaging apps like Telegram and Discord.
With Discord, users sign into Koko Cares servers and send direct messages to Koko bots that ask multiple-choice questions (eg, “What is your darkest thought about this?”). You can then share your concern (written as a few sentences of text) anonymously with someone else on the server, who can reply anonymously with a short message of their own.
We have used GPT-3 to provide mental health support to approximately 4,000 people. Here’s what happened 👇
— Rob Morris (@RobertRMorris) January 6, 2023
In an AI experiment applied to about 30,000 messages, according to To Morris — Volunteers helping others had the option to use responses automatically generated by OpenAI’s GPT-3 large language model instead of creating their own (GPT-3 is It’s the technology behind the recently popular ChatGPT chatbot).
size
In his tweet thread, Morris To tell People appreciated AI-generated responses until they learned that AI-generated responses were generated by AI. This suggests a lack of informed consent in at least one phase of the experiment.
Messages created by AI (and supervised by humans) were rated significantly higher than those written by humans themselves (p < .001). His response time was reduced by 50% to well under a minute. Still... we pulled this off our platform pretty quickly. why? Once people found out the message was co-written by a machine, it stopped working. Simulated empathy feels strange and empty.
In introducing the server, the administrator wrote:
Shortly after posting the Twitter thread, Morris received a number of replies criticizing the experiment as unethical. Lack of informed consent and, Institutional Review Board (IRB) approved the experiment. In the United States, it is illegal to conduct research on human subjects without legally valid informed consent unless the IRB decides that consent can be waived.
In a tweeted response, Morris Said Since he had no plans to publish the results, he said the experiment was “exempt” from the informed consent requirement, provoking a parade of frightening reactions.
As a former member and chair of the IRB, I have conducted human subjects research on vulnerable populations without IRB approval or waiver (you cannot decide). The MGH IRB process might be very slow doing all this stuff.Unsolicited Advice: Lawyers Up
— Daniel Shosketh (@dshoskes) January 7, 2023
The idea of using AI as a therapist is by no means new, but the difference between Coco’s experiment and typical AI therapy approaches is that patients typically know they are not talking to a real human being. It means that (Interestingly, one of his earliest chatbots, ELIZA, simulated a psychotherapy session.)
In Koko’s case, the platform offered a hybrid approach, where instead of a direct chat format, a human intermediary could preview messages before sending. Yet, without informed consent, critics argue Koko violated ethical rules designed to protect vulnerable people from harmful or abusive research practices.
On Monday, Morris shared a post in response to Koko’s controversy describing GPT-3 and AI in general, writing: We ask that the use of AI be handled carefully with a deep concern for privacy, transparency, and risk mitigation. Our Clinical Advisory Board is meeting specifically to discuss guidelines for future work on her IRB approval. “
