Skip to main content
Humanities LibreTexts

Don't Trust AI to Cite Its Sources

  1. Last updated
  2. Save as PDF
  • Page ID
    273025

    • \( \newcommand{\vecs}[1]{\overset { \scriptstyle \rightharpoonup} {\mathbf{#1}} } \)

    \( \newcommand{\vecd}[1]{\overset{-\!-\!\rightharpoonup}{\vphantom{a}\smash {#1}}} \)

    \( \newcommand{\id}{\mathrm{id}}\) \( \newcommand{\Span}{\mathrm{span}}\)

    ( \newcommand{\kernel}{\mathrm{null}\,}\) \( \newcommand{\range}{\mathrm{range}\,}\)

    \( \newcommand{\RealPart}{\mathrm{Re}}\) \( \newcommand{\ImaginaryPart}{\mathrm{Im}}\)

    \( \newcommand{\Argument}{\mathrm{Arg}}\) \( \newcommand{\norm}[1]{\| #1 \|}\)

    \( \newcommand{\inner}[2]{\langle #1, #2 \rangle}\)

    \( \newcommand{\Span}{\mathrm{span}}\)

    \( \newcommand{\id}{\mathrm{id}}\)

    \( \newcommand{\Span}{\mathrm{span}}\)

    \( \newcommand{\kernel}{\mathrm{null}\,}\)

    \( \newcommand{\range}{\mathrm{range}\,}\)

    \( \newcommand{\RealPart}{\mathrm{Re}}\)

    \( \newcommand{\ImaginaryPart}{\mathrm{Im}}\)

    \( \newcommand{\Argument}{\mathrm{Arg}}\)

    \( \newcommand{\norm}[1]{\| #1 \|}\)

    \( \newcommand{\inner}[2]{\langle #1, #2 \rangle}\)

    \( \newcommand{\Span}{\mathrm{span}}\) \( \newcommand{\AA}{\unicode[.8,0]{x212B}}\)

    \( \newcommand{\vectorA}[1]{\vec{#1}}      % arrow\)

    \( \newcommand{\vectorAt}[1]{\vec{\text{#1}}}      % arrow\)

    \( \newcommand{\vectorB}[1]{\overset { \scriptstyle \rightharpoonup} {\mathbf{#1}} } \)

    \( \newcommand{\vectorC}[1]{\textbf{#1}} \)

    \( \newcommand{\vectorD}[1]{\overrightarrow{#1}} \)

    \( \newcommand{\vectorDt}[1]{\overrightarrow{\text{#1}}} \)

    \( \newcommand{\vectE}[1]{\overset{-\!-\!\rightharpoonup}{\vphantom{a}\smash{\mathbf {#1}}}} \)

    \( \newcommand{\vecs}[1]{\overset { \scriptstyle \rightharpoonup} {\mathbf{#1}} } \)

    \( \newcommand{\vecd}[1]{\overset{-\!-\!\rightharpoonup}{\vphantom{a}\smash {#1}}} \)

    \(\newcommand{\avec}{\mathbf a}\) \(\newcommand{\bvec}{\mathbf b}\) \(\newcommand{\cvec}{\mathbf c}\) \(\newcommand{\dvec}{\mathbf d}\) \(\newcommand{\dtil}{\widetilde{\mathbf d}}\) \(\newcommand{\evec}{\mathbf e}\) \(\newcommand{\fvec}{\mathbf f}\) \(\newcommand{\nvec}{\mathbf n}\) \(\newcommand{\pvec}{\mathbf p}\) \(\newcommand{\qvec}{\mathbf q}\) \(\newcommand{\svec}{\mathbf s}\) \(\newcommand{\tvec}{\mathbf t}\) \(\newcommand{\uvec}{\mathbf u}\) \(\newcommand{\vvec}{\mathbf v}\) \(\newcommand{\wvec}{\mathbf w}\) \(\newcommand{\xvec}{\mathbf x}\) \(\newcommand{\yvec}{\mathbf y}\) \(\newcommand{\zvec}{\mathbf z}\) \(\newcommand{\rvec}{\mathbf r}\) \(\newcommand{\mvec}{\mathbf m}\) \(\newcommand{\zerovec}{\mathbf 0}\) \(\newcommand{\onevec}{\mathbf 1}\) \(\newcommand{\real}{\mathbb R}\) \(\newcommand{\twovec}[2]{\left[\begin{array}{r}#1 \\ #2 \end{array}\right]}\) \(\newcommand{\ctwovec}[2]{\left[\begin{array}{c}#1 \\ #2 \end{array}\right]}\) \(\newcommand{\threevec}[3]{\left[\begin{array}{r}#1 \\ #2 \\ #3 \end{array}\right]}\) \(\newcommand{\cthreevec}[3]{\left[\begin{array}{c}#1 \\ #2 \\ #3 \end{array}\right]}\) \(\newcommand{\fourvec}[4]{\left[\begin{array}{r}#1 \\ #2 \\ #3 \\ #4 \end{array}\right]}\) \(\newcommand{\cfourvec}[4]{\left[\begin{array}{c}#1 \\ #2 \\ #3 \\ #4 \end{array}\right]}\) \(\newcommand{\fivevec}[5]{\left[\begin{array}{r}#1 \\ #2 \\ #3 \\ #4 \\ #5 \\ \end{array}\right]}\) \(\newcommand{\cfivevec}[5]{\left[\begin{array}{c}#1 \\ #2 \\ #3 \\ #4 \\ #5 \\ \end{array}\right]}\) \(\newcommand{\mattwo}[4]{\left[\begin{array}{rr}#1 \amp #2 \\ #3 \amp #4 \\ \end{array}\right]}\) \(\newcommand{\laspan}[1]{\text{Span}\{#1\}}\) \(\newcommand{\bcal}{\cal B}\) \(\newcommand{\ccal}{\cal C}\) \(\newcommand{\scal}{\cal S}\) \(\newcommand{\wcal}{\cal W}\) \(\newcommand{\ecal}{\cal E}\) \(\newcommand{\coords}[2]{\left\{#1\right\}_{#2}}\) \(\newcommand{\gray}[1]{\color{gray}{#1}}\) \(\newcommand{\lgray}[1]{\color{lightgray}{#1}}\) \(\newcommand{\rank}{\operatorname{rank}}\) \(\newcommand{\row}{\text{Row}}\) \(\newcommand{\col}{\text{Col}}\) \(\renewcommand{\row}{\text{Row}}\) \(\newcommand{\nul}{\text{Nul}}\) \(\newcommand{\var}{\text{Var}}\) \(\newcommand{\corr}{\text{corr}}\) \(\newcommand{\len}[1]{\left|#1\right|}\) \(\newcommand{\bbar}{\overline{\bvec}}\) \(\newcommand{\bhat}{\widehat{\bvec}}\) \(\newcommand{\bperp}{\bvec^\perp}\) \(\newcommand{\xhat}{\widehat{\xvec}}\) \(\newcommand{\vhat}{\widehat{\vvec}}\) \(\newcommand{\uhat}{\widehat{\uvec}}\) \(\newcommand{\what}{\widehat{\wvec}}\) \(\newcommand{\Sighat}{\widehat{\Sigma}}\) \(\newcommand{\lt}{<}\) \(\newcommand{\gt}{>}\) \(\newcommand{\amp}{&}\) \(\definecolor{fillinmathshade}{gray}{0.9}\)

    By Anna Mills and Maha Bali

    Teachers will usually ask you to cite sources, right? They will explain that you should cite not just when you are quoting but also when you are referencing an idea you got from someone else. The hope here is that in academic practice we give some sense of where an idea is coming from. We think of academic work as a big ongoing conversation between people agreeing and disagreeing and building on each other’s points as they try to figure things out. It helps to keep track of who is saying what. It seems only fair to give credit to the person who developed each main point. 

    For some concepts in the humanities and social sciences, it is also important to know the positionality of the person or people who developed them. For example, if we are studying nonviolent resistance, we need to understand the people and context in which ideas about it were developed. Mahatma Gandhi, Martin Luther King Junior, and Henry David Thoreau wrote extensively about it, and their ideas were shaped by very different experiences, times, and places.

    Mahatma Gandhi in 1931
    Mahatma Gandhi was instrumental in shaping current understandings of nonviolent resistance in 1931, public domain image courtesy of Wikimedia
    Martin Luther King Junior at a podium looking thoughtful
    Martin Luther King Junior was also instrumental in shaping current understandings of nonviolent resistance. Image by WikiImages from Pixabay

    Knowing something about how these leaders were positioned in terms of race, gender, class, education, and politics can help us understand their ideas and think about how we want to respond from our own positionality.  However, if you ask a chatbot about “nonviolent resistance,” it may give you an answer without reference to any person or context at all (chatbot responses are variable). Here is an example with ChatGPT:

    Screenshot of chat session. User: what is nonviolent resistance? Answer brieflyChatGPT: Nonviolent resistance is a method of protest and civil disobedience that uses peaceful tactics, such as marches, sit-ins, and boycotts, to oppose government policies or societal issues without using physical force. This strategy aims to bring about social or political change through moral or economic pressure, often appealing to the conscience of the public and decision-makers.
    Screenshot from a ChatGPT4o temporary session, July 2024

    In general, chatbots don’t make it easy to figure out what sources have influenced their answers. As Iris Van Rooij puts it, “LLMs, by design, produce texts based on ideas generated by others without the user knowing what the exact sources were.” 

    The systems don’t “know” what influenced their answer. Dominik Lukeš refers to this limitation as a lack of “introspection.” Once a system is trained, it consists of a very complicated formula, a big set of numbers that its supposed to multiply by other numbers. The chatbot puts our question or request into the formula and gets a result. It has no way to look backward to see which human writings made its formula have certain numbers. 

    Chatbots often don’t cite when they should

    Why does it matter if chatbots can’t tell us where an idea came from? In June 2023, Maha Bali pointed this out to Anna Mills in a conversation on X. She wrote, “One of the things I'm stuck on right now is that a lot of the AI-generated text paraphrases work of scholars we would *normally* cite as paraphrased. This stuff now goes unacknowledged and it's not OK…E.g. I asked it about characteristics of White Supremacy Culture. It gave me the list that Tema Okun and others use, but did not cite her.” 

    A year later, I (Anna) tested and saw the same pattern. I asked three chatbots–ChatGPT, Claude, and Gemini–about “Intentionally Equitable Hospitality,” a concept developed by Maha Bali and other co-directors of the grassroots group Virtually Connecting.

    • ChatGPT generated long, inaccurate descriptions of the concept with no reference to either the organization or the people who developed it. This was true of all ten times I tested (see this sample ChatGPT transcript). 

    • Google’s Gemini described the concept vaguely, making up principles that are not in any published writings about this exact phrase, “Intentionally Equitable Hospitality.” I tested it ten times and not once did t mention an organization or person. (See this sample Gemini transcript). Gemini did feature a Google “G” logo which offered to check the results when I hovered over. The check led it to highlight several passages of Gemini’s output and link them to websites, including Maha Bali’s blog. 

    • Claude 3.5’s ten responses all referred to the hospitality industry, a clear error since the exact phrase “Intentionally Equitable Hospitality” was developed to refer to education (See this sample Claude transcript). They also failed to mention any organizations or people involved in developing the concept.

    Citing chatbots according to the rules gives them credit for human authors’ work

    Labeling chatbot output as such is important. And the major academic organizations are working hard to come up with guidelines for this. But it’s tricky. If we follow the current guidelines, we’ll make no mention of the humans who developed the idea the chatbot is summarizing.

    Take any of the examples above where we asked a chatbot about Intentionally Equitable Hospitality. Let’s say a person does that and then wants to quote the output and follow the rules to cite ChatGPT in APA, MLA, or Chicago style. All of these would make it look like ChatGPT is the source of the IEH concept. For more on this, see If I use AI, how should I acknowledge or cite it?

    Don’t ask a chatbot what its sources are

    There is a lot of confusion about this among teachers and the general public because if you ask a chatbot what its source is, it will often give an answer that sounds plausible. It still doesn’t “know.” It is using its word prediction abilities to “guess” which source influenced its output. It doesn’t really have access to its own internal workings. The source it mentions may not exist or it may not be the one that really shaped the response. For example, in one case, ChatGPT said it got information about Intentionally Equitable Hospitality from Kimberly Crenshaw, who is known for the concept of Intersectionality and has not written about IEH.

    Chatbots also respond unreliably when you ask which person developed a concept. 

    • When I asked ChatGPT the follow up question about IEH: “Is this a particular person's concept?” it gave credit to Dr. Tia Brown McNair, a prominent expert on Diversity, Equity, and Inclusion who is not in fact associated with the phrase “Intentionally Equitable Hospitality.”

    • When I asked the same follow-up question of Claude 3.5, it gave credit to Ashtin Berry, a hospitality industry leader whose name is not associated with the phrase “Intentionally Equitable Hospitality.”

    • Google’s Gemini did better. It responded to the question once by correctly referring to Virtually Connecting and its co-directors, including Maha Bali and Autumm Caines. On another trial, it referred only to Maha Bali. On another, it referred to Maha Bali, Mia Zamora, and the organization Equity Unbound, which is correct because both authors and the organization have developed this concept further beyond the original by Virtually Connecting. None of the trials trace the idea from its original formulation to its current formulation. 

    Even chatbots that can search often don’t mention their sources

    As we will see in The pros and cons of AI for research assistance (coming soon), many current chatbot/search hybrid systems can search the internet in real time to inform their response to a question. They link to documents, making it seem like they are allowing us to trace the source of their information. But we the users don’t know if those links are really the sources they used or not. And it isn’t always clear which of the links informed which part of their response. Besides, many people will not click on the links; they will focus on the AI response itself.

    In May 2024, Google rolled out “AI overviews” at the top of many search results pages. When I try the same query about Intentionally Equitable Hospitality (IEH) in Google search, I get an answer that doesn’t mention a source, though links to sources are right below it. 

    "Screenshot of a section titled 'AI Overview'. The main text reads: 'Intentionally Equitable Hospitality (IEH) is a values-based approach to hospitality that prioritizes the needs of the most marginalized people. It's a critical praxis that centers equity considerations in the design of a space, and it's not neutral. IEH is iterative and involves planning, design, and facilitation in the moment. It's also about recognizing when efforts don't work for certain groups and setting intentions for equity.' Below the main text, there are links to three sources: Pressbooks Create, MYFest24, and Maha Bali."

    Screenshot of a Google search AI Overview in response to the query “What is Intentionally Equitable Hospitality?” on July 19, 2024. Four repetitions produced similar results. 

    The Google result above plagiarizes most of a sentence from the first source linked to, Bali and Zamora’s “Intentionally Equitable Hospitality as Critical Instructional Design.” Google explains “IEH is iterative and involves planning, design, and facilitation in the moment” without quoting. Bali and Zamora had written, “IEH is iterative design, planning, and facilitation in the moment.“ 

    Now let’s try with the popular chat-search hybrid and Google alternative Perplexity.ai. In one instance, Perplexity mentions Virtually Connecting as a source for IEH and linked to an article by Maha Bali and Mia Zamora without mentioning their names. (How many will follow the link?) On the second, third, fourth, and fifth trials with the free version of Perplexity, it gave links but mentioned no source at all in its overviews. I repeated this five more times using my quota of free “Pro” searches. Only once did it mention Maha Bali and Mia Zamora by name in its autogenerated answer. So even though this chatbot was referencing a real source by the scholars who defined the concept, in five out of ten trials it included no equivalent at all to an in-text citation (see all of Perplexity’s responses).

    So how do we respect sources when using AI?

    All this means that when a chatbot “says” something, we should wonder not just whether it is accurate but also whom the chatbot is parroting. Of course, many concepts have been shaped by so many contributors and have become so widespread that they do not need to be cited. For example, if we state, “Many Mexicans have both indigenous and European ancestry,” we do not need to cite it.

    But if the point is less well known or more controversial, it is left to us to do due diligence and see if there is a particular person or group of persons who came up with the ideas AI serves us. Internet and scholarly database searches on key concepts may lead us to the human thinkers responsible. 

    Should it really be on users to try to reconstruct where a chatbot may have gotten its information? Is there any way the software itself could help us? If so, how could we encourage the companies or the government to make that happen? See “Does using AI do harm? If so, should we stop using it?* for more on intellectual property and how we might participate in shaping AI in this regard. 

    Further reading

    References

    Don't Trust AI to Cite Its Sources is shared under a CC BY-NC 4.0 license and was authored, remixed, and/or curated by LibreTexts.