Squiggles All the Way Down

Let us turn to a most pressing matter in the AI space: hallucinations.

We, of course, feel that using this term for the phenomenon of a large language model vomiting out nonsense is questionable, but we have unfortunately been overruled in this regard; the public insist on anthropomorphizing these idiot machines to some extent, and so the word has stuck. (We feel similarly regarding the name “harness” for a system which makes use of an LLM, because it suggests that the LLM is some kind of beast of burden, and comparing it to a cow is frankly insulting to the cow.) So, for the purposes of this diatribe, we will use the term “hallucination” in the sure and certain knowledge that it is a far more interesting and descriptive word than the entity in question deserves.

We simply wish to make it clear that we use it under duress and with no small amount of irritation.

Hallucinations are a side-effect of the nature of a glorified predictive text engine. The large language model works on a very simple premise: given some input of text, what is the most likely output of text? If it is told “What color is a banana?”, the engine sees the strip of words as a set of squiggles. It refers to the vast corpus of training data (samples of millions of pieces of text, broken down into a refined set of squiggles) and runs some rather complex mathematics to determine the correct set of squiggles to put on the screen. It does not, in any way, understand anything about what the squiggles actually mean. It has no frame of reference, no context, no memory. It does not know what a banana is.

This lack of awareness leads to the system occasionally putting out inaccurate text. In previous years, before LLMs were refined, it was quite easy to tell if the output was “bad”, as the text would be garbled, misspelled, and so on. (Using judgments like “bad” or “good” for the LLM is an entirely subjective thing, as the system has no concept of such values; its task was successful simply because it produced an output, nothing more.) We have reached a point today, of course, where the LLM has been trained on a sufficient amount of data for it to produce an output that the average human can parse as language, but this does not mean that its process has significantly changed. At its core, it still only matches squiggles, and delivers a chain of the most statistically probable squiggles as a result. It is squiggles, in fact, all the way down.

Given its lack of understanding, it is a certainty that the LLM will occasionally produce output that is entirely insensible. The issue is that we are less likely to be able to detect this insensibility, where before it was quite obvious. An LLM is limited by the quantity and quality of its training data; where the “correct” answer to a given input requires a high degree of specificity and context, the LLM will fail, in somewhat bizarre and obscure ways. The most interesting, in our opinion, are the instances where the LLM invents citations to court cases that do not exist, if only because the discovery of such (and the abject fury of the judges) is highly amusing.

But this is not the only issue with the system.

The system has no understanding or context, and as such entirely lacks some very important failsafes that the average human engages without thinking. It cannot detect lying, sarcasm, or satire, and it has no ability to fact-check. (It has no actual awareness of reality and thus no awareness of whether something is objectively true or false; a human, at least, is capable of being suspicious.) It is programmed simply to produce an output, given the input and its training data. It follows automatically that the LLM is not only simulating a conversation, but it is also simulating gullibility on a level not normally seen outside an MLM convention. Choose your set of input squiggles carefully enough, and it is frighteningly simple to cause the system to hallucinate like a bad acid trip.

Now, let it be said that this is truly hilarious to the casual observer. Causing ChatGPT to opine on the suitability of churros for making surgical tools is very funny. The story of Google’s AI proudly declaring that rocks and glue are edible is hopefully well known at this point. However, dear readers, one must consider that the controllers of these vaunted systems are asking for all of our data to be fed into their digital furnaces, including such things as credit cards, personal information, home addresses, identity numbers; these may even be the low-hanging fruit of the fuel pile, as things like API keys, private keys, access tokens, and many corporate tech artifacts that must be kept secret at all costs can also find their way into the fire. Hallucinations, under the right circumstances, can expose any and all of that data. The programmers of the modern internet have expended a vast human cost in time and energy to make things like online banking secure, and now we are faced with the horrifying knowledge that all of it may have been for naught due to the nature of large language models.

This is not to say that the current status quo is perfect, of course. Social engineering is likely the most successful method of exfiltrating data from individuals or companies for a reason; humans are fallible and sometimes very stupid. But the average human working in a white-collar job does not have perfect recall of thousands of credit card numbers, for example, and a complete willingness to list them to someone posing as a client on the phone. While an employee can be trained to prevent such an attack, there is no way to prevent an LLM from hallucinating at this time.

It is galling that the titans of Silicon Valley tell us that these systems are the wave of the future in light of this. The idiot machines are black boxes of functionality, with a level of reliability that would get any human fired on the spot. And for this, we are expected to raze farmland, waste an astronomical amount of water, upend our economies, and ultimately make life that much harder and more desperate for everyone but an insignificant number of wealthy individuals. We question whether the AI maximalists who are hell-bent on foisting these systems upon us are themselves hallucinating when they describe them as “revolutionary”. Revolutionary for whom, we ask? It is certainly not for us.

For now, we sit and wait, dear readers. We sit and wait. Guard your digital space as best you can, and hold fast to your skepticism.