The term AI, short for Artificial Intelligence, is a marketing term rather than a serious description of anything. These days it mostly refers to either Large Language Models (LLMs, the things which generate text) or Diffusion Models (the things that generate images). Neither of these are intelligence—natural or otherwise—and calling them AI is just a marketing term. Both of these things are just massive matrices (if you’re too lazy to click the link, grids of numbers multiplied in a special way) and a tiny bit of code for how to multiply them together. Though these days that’s becoming more than a tiny bit of code in order to make the multiplications more efficient. And a bit of code to map text to input matrices and the output matrices back to text. Or to pixels, if they’re applied to images.

Be the implementation details as they may, it’s still just code controlling massive matrix multiplications, and this is all work that could, in theory, be done with a few reams of paper and a few boxes of pencils. Just imagine working through all of this, mapping the resulting matrix to text, and shouting, “Oh my God, the numbers I wrote down on this paper are intelligent!” It’s ridiculous, right?

LLMs (and diffiusion models) aren’t intelligence, they’re just extremely useful mathematical tools which are possible in practical amounts of time (and space) because computers have made it possible to do trillions of multiplications and additions per second.

Matrices are usually studied in the branch of mathematics called Linear Algebra, and their first great use in computers was 3D graphics. Leaving out all of the actual mathematics in order to keep this accessible, it turns out that in a 3D coordinate system, it’s possible to create matrices that, when multiplied against other matrices, have the effect of rotating them. Other matrices can translate them, and still others can scale them. And, critically, still other matrices can do a 3D to 2D projection with perspective transform. This is what lets 3D graphics work; it’s why your computer can render millions of triangles that approximate real world shapes and make the ones that are closer bigger while the ones that are further away are smaller. And this actually gives a hint as to how LLMs work.

If you boil it down, LLMs really only do one thing: selectively apply transforms to inputs in a huge many-dimensional coordinate space. The reason that this is interesting is that one can map text into these huge many-dimensional coordinate spaces, and one doesn’t need to know what kinds of transforms one wants to do ahead of time. The (apparent) magic is that people figured out ways of using pre-existing text to create matrices that happen to have very useful transforms in these many-dimensional coordinate spaces.

An issue for discussing any of this is that every aspect of LLMs is given names by analogy to how people do things. Thus the step which could quite accurately be called “sample-based matrix creation” is called by the catchier name of “training”. What could very accurately be called, “selective transform emphasis” is called by the catchier name of “reinforcement learning”. And so it goes.

But if we side-step all of the marketing terms and analogies and focus on what the things actually do, LLMs have the effect of translating input text according to patterns in the data that was used to construct them. Very loosely speaking, you can think of this as reproducing patterns in their training data.

This is actually why they’re currently transforming the art of programming. Programming has always involved having to re-solve problems that have been solved many times before, just with different words this time because the context is different. We programmers hate doing this, and have put enormous work into trying to do this as little as possible. We’ve invented progressively higher level programming languages in which more and more previously solved problems are just embedded into the programming language. We make functions so we only have to write the code once and share libraries with each other like it’s going out of fashion. LLMs are such a godsend because a thing which reproduces existing patterns but altered to the particulars of the exact source code we’re working on means that, finally, we can work on the new problems instead of re-solving the problems which have been solved ten thousand times before. It’s fantastic.

You’ll notice what I haven’t said anything about, there, is judgment. That’s because LLMs don’t have any judgment. It’s literally not even a concept within the models. Judgment is understanding the relationship between things as they are. LLMs apply selective transforms in many-dimensional coordinate spaces. The two things have nothing to do with each other.

This is why LLM writing of novel English prose is so bad. They’re fine when they’re just translating prose which was in their training data into the answer to a question. (They will always produce a transform; these transforms may simply not happen to produce text which happens to have a correspondence with anything real, just as you can try to divide by zero or claim that infinity + three is bigger than infinity plus two. The mere fact that words can be parsed according to the rules of grammar doesn’t guarantee that they are meaningful sentences, and still less that they’re true sentences.) LLMs are even better when the transforms they’re applying are just pulling sections of input data (such as a search result from a book that got appended to your text with a question in it) and translating it to the form of an answer to your question. But when they’re writing new text, it’s really impressively bad.

Here, let me give you an example. I wrote this paragraph giving advice on the importance of having highly descriptive names for functions and variables in programs when using an LLM coding agent:

If an acorn is defined as a tree rodent with a bushy tail and squirrel is defined as the seed of an oak tree and day is defined to be 42 seconds longer than twice nine seconds, how many squirrels can an acorn eat in three days? Most people familiar with the relevant wildlife can tell you that the answer is somewherearound 2, but the proportion will clearly be higher if you ask how many acorns a squirrel can eat in 3 minutes. The same is true of LLMs.

Here’s what Gemini said what I asked it to evaluate this advice, then asked it how it would rewrite it:

Imagine defining acorn as a bushy-tailed rodent and squirrel as the nut of an oak tree. If you ask, “How many squirrels can an acorn eat in a day?”, humans and LLMs alike will stumble. Even if you explicitly redefine your terms, an LLM’s pre-trained semantic associations constantly fight misleading or cryptic names. Descriptive, conventional naming aligns your code with the model’s weights instead of forcing it to fight its own priors.

The LLM’s version isn’t terrible, of course. But it also isn’t good. Not that my version was perfect, but it works on several levels because I had several ideas in mind. One of which is that I was mentally modeling the amount of confusion my convoluted version would produce in the reader, especially taking into account the speed I was going and the relatively short words I was using. That feeling is essential to communicating the cognitive load involved in keeping track of unintuitive definitions. It’s also worth noting that my version is correct while the LLM’s version isn’t. There are people and LLMs which can answer the question just fine. That’s why my version acknowledged that and talked about the proportion of people who would get it right.

And this isn’t even a good example of how bad LLM writing usually is, it’s just one I happened to have handy because it came up recently in conversation. LLM writing is bad precisely because it lacks judgment, and unlike programming languages, natural language writing is there to convey things which haven’t been said a thousand times before. It’s not precisely that LLMs aren’t creative. They actually kind of (technically) are, since they actually use random number generators to select the exact text they generate. Creativity, at least in the sense of creating novel arrangements of words, is rather overrated. Throwing scrabble tiles on the ground can be creative in that sense. The real problem is that they have no judgment, and so they don’t put words together in ways that convey good judgment. So their writing is fluffy and flabby in the best case, and outright wrong in the worst case.

And here’s where we get to the part of the discussion which is really frustrating: a lot of people have bad judgment, so they don’t notice this. I’ve seen people quoting absolutely terrible writing and talking about how magnificent it is. Of course, that can apply to human writing as well as LLM writing. There are awful books which have been praised, because, alas, there is no shortage of fools in the world.

Also, for various reasons, people sometimes massively exaggerate what LLMs have done through hearsay. For example, this interesting blog post about novel writing in the age of LLMs contains the line (this link was in the blog post, and is to the source): “AI books winning awards.” But if you follow the link, it turns out that the thing which won an award was a short story. (It’s even in the title.) I did a bit of searching, and I can find no evidence that a novel written entirely with an LLM has ever won a prize in any language.

Is it possible that an LLM-written novel will win a prestigious literary prize some day? Of course. If you are at all familiar with literary prizes, you know that quality is a requirement for winning them far more in theory than in practice. And the same thing is true for being popular. If you’ll allow me to bring in movies, Disney’s The Last Jedi was a movie of almost indescribable badness (so much so I wrote a 21,564 word review of it called The Least Jedi) and it grossed over a billion dollars. Heck, its script was so bad I assume that, with minimal prompting and a halfway decent harness (a program which invokes an LLM multiple times to achieve a coordinated result), any modern LLM would write a much better script than the atrocity which Ryan Johnson wrote. This is a low bar, though. The ancient program Dissociated Press, which used simple markov chains, would still have probably written a better script than what Ryan Johnson wrote. A single monkey, typing for ten minutes on a children’s toy typewriter where the keys are all just stickers and there are no hammers, ribbon, or paper, would have written a better script. So I figure it’s just a matter of time until an LLM writes a novel which wins a prestigious literary award.

What it will be a very long time before we see is an LLM which writes a good novel, because that requires a true understanding of the nature of reality, and that’s not the kind of thing an LLM even can have.

I can’t help but wonder whether it will be better or worse when LLMs are writing popular novels. On the plus side, at least they won’t be trying to sneak bad philosophy into their stories, which is the case for an awful lot of popular novels written by people. Machines lack souls, which means that they’re not actively trying to damn you (or, more accurately, trying to trick you into damning yourself). The same cannot be said of human authors.