A study by the Tow Center for Digital Journalism has raised alarm bells about the reliability of ChatGPT, the AI chatbot developed by OpenAI. The research, which examined how ChatGPT produces citations for publishers' content, reveals a concerning trend of invented or misrepresented information, regardless of whether publishers have licensing deals with OpenAI or not.

The study, conducted at Columbia Journalism School, analyzed citations produced by ChatGPT after it was asked to identify the source of sample quotations from 20 randomly selected publishers. The researchers found that ChatGPT's citations were often inaccurate, with the chatbot frequently inventing or misrepresenting information. In some cases, the bot even attributed plagiarized content to the original source, raising serious questions about OpenAI's ability to filter and validate the quality and authenticity of its data sources.

The researchers identified a "spectrum of accuracy" in ChatGPT's responses, with some citations being entirely correct, while others were entirely wrong. However, what's more concerning is that the chatbot rarely acknowledged its inability to produce an accurate answer, instead opting to confabulate and generate incorrect sourcing. This lack of transparency about its confidence in an answer can make it difficult for users to assess the validity of a claim and understand which parts of an answer they can or cannot trust.

The study's findings have significant implications for publishers, who risk reputational damage and commercial loss due to incorrect citations. Moreover, the research suggests that ChatGPT's technology is treating journalism as decontextualized content, with little regard for the circumstances of its original production. This raises questions about the fundamental approach of OpenAI's technology and its ability to respect publisher preferences.

In response to the research findings, OpenAI accused the researchers of running an "atypical test of our product." However, the study's conclusions are hard to ignore, and the concerns raised about ChatGPT's citation accuracy are unlikely to be alleviated by OpenAI's claims of collaborating with partners to improve in-line citation accuracy.

The study's findings are particularly concerning given the high-level deals that major publishers are cutting with OpenAI. If media businesses were hoping that these arrangements would lead to special treatment for their content in terms of producing accurate sourcing, this study suggests that OpenAI has yet to offer any such consistency. Instead, publishers are left with "little meaningful agency" over what happens with and to their content when ChatGPT gets its hands on it.

The implications of this study are far-reaching, and it's clear that more rigorous testing is needed to fully understand the limitations and potential risks of ChatGPT's technology. As the use of AI chatbots becomes increasingly prevalent, it's essential that we prioritize the integrity and accuracy of online content, and hold developers accountable for ensuring that their products meet the highest standards of reliability and transparency.