AI Copies Patterns; It Doesn’t Think (Draft)

Before we use AI-generated text, it’s pretty important to get a basic intuitive sense of where it’s coming from. AI text generation copies patterns from the text it’s trained on. Its training involves such dense mathematical analysis of patterns in such huge quantities of text, that in replicating these patterns, it can sound pretty smart, as you’re probably aware. (Hence all the fuss). But how does it get to sound that way?

The New York Times article Let Us Show You How GPT Works – Using Jane Austen by Aatish Bhatia shows us it what it looks like when you gradually train small AI text generation systems, called large language models, in the style of Harry Potter, Star Trek: The Next Generation, Shakespeare, Moby Dick, or Jane Austen.

Let’s take the Harry Potter version. Before training, the user types in “Hermione raised her wand,” and the language model continues “.Pfn“tkf^2JXR454tn7T23ZgE——yEIé\mmf’jHgz/yW;>>QQi0/PXH;ab:XV>”?y1D^^n—RU0SVGRW?c>HqddZZj:”

That’s its random guess as to what comes next. 

Then it goes through several rounds of training, ingesting text from Harry Potter and adjusting its internal prediction numbers to match patterns in that text. 

Eventually, when the user writes “Hermione raised her wand,” the model continues in a recognizably Harry Potterish way:

"Professor Dumbledore never mimmed Harry. He looked back at the room, but they didn't seem pretend to blame Umbridge in the Ministry. He had taken a human homework, who was glad he had not been in a nightmare bad cloak.”

Yep, it’s echoing the books and movies with main characters’ names, a reference to the Ministry (of Magic), and “nightmare bad” cloaks that suggest magic and evil.  But, um, there’s no such word as “mimmed.” And why does homework care if some person is wearing a cloak? This is where I start to chuckle. 

If you kept training a system like this, it will eventually give you a sentence that might be hard to tell apart from Harry Potter author J.K. Rowling’s sentences. But the system would still be matching patterns and predicting next words.

So next time you see AI produce a smooth, polished sentence that sounds just like sophisticated academic writing, remember the Harry Potterish gobbledygook. The lights might be on, but nobody’s home. Check whether the text is empty or wrong. If it does make sense and matches reality, remember, that’s partly luck. The system makes up true sentences the same way it makes up nonsense. So should we trust it? No. 

More Silly Examples

• My 12-year-old son’s question was “Why does pepperoni pizza dominate literature?” ChatGPT went with it, declaring “Pepperoni pizza's dominance in literature can be attributed to several factors.”

• I asked it to write about the connection between snails and cheese. It said, “Ecologically, snails and cheese exhibit a symbiotic relationship mediated through their respective environments and the intricate ecosystems they inhabit.” Snails and cheese somehow help each other out? Not so much.

• How about the essential connection between hip hop and potato mashers? ChatGPT says “The act of mashing, much like the act of mixing and sampling in hip hop, requires skill, precision, and an understanding of how to integrate diverse components into a cohesive whole.” Really? 

• One more: I asked ChatGPT about the “essential connection” between kiwi fruit and Call of Duty. It said “their essential connection lies in their shared narrative of globalization, cultural commodification, and the modern challenge of balancing digital and physical well-being.” Hey, that’s an elegantly formed sentence. The rhythm sounds nice. But there’s nobody home who really had something to say something about that particular fruit and that particular video game. 

Try an Experiment

You might find that you get a better intuitive sense of this through your own experiments. 

1. Think of two random things that you’re pretty sure have no essential connection. 

2. Bring up a chat system, any of the more sophisticated chat systems. If you’d like to use one without logging in, try Perplexity (click on “focus” and choose “Writing”) or ChatGPT.  Other options include Gemini, Claude, and Copilot).

3. You can copy the following prompt, edit it, or write your own. Substitute your picks for X and Y, like an unusual fruit and a video game or a musical style and a particular kitchen tool.

Prompt: “In a sophisticated, authoritative academic style, explain the essential connection between X and Y.” 

4. Read the chatbot’s output. How does it sound? Does it make any sense? Do you have an emotional reaction to see fancy text seeming to argue for something so arbitrary that isn’t really your opinion or any human’s opinion? Is it annoying, exciting, impressive, eye-rolling, weird, or…? What does this experiment suggest to you about how we should approach AI text? What’s your takeaway? 

Further Readings and Videos

• Let Us Show You How GPT Works – Using Jane Austen by Aatish Bhatia, The New York Times

• TikTok video on ChatGPT as predictive text (3 minutes, by @mor10webn)

• Computers are getting better at writing (7 minutes, from Joss Fong at Vox.com)

• Shaped like Information or Let Our Algorithm Choose Your Halloween Costume by Janelle Shane, author of the AI Weirdness blog and the book You Look Like a Thing and I Love You: ​​How Artificial Intelligence Works and Why It’s Making the World a Weirder Place.

• Talking about Large Language Models by Murray Shanahan

Attributions

By Anna Mills, offered under a CC BY NC 4.0 license. Feel free to adapt with attribution.