The advantage of these online texts is that you can click on any of the images, copy their content into a Mathematica notebook, and play with the code.
This really gives a good idea of how an extremely simplified version of ChatGPT (based on GPT-2) works.
Downloading the model (within Mathematica) uses about 500Mb, but afterwards you can complete any prompt quickly, and see how the results change if you turn up the ‘temperature’.
You should’t expect too much from this model. Here’s what it came up with from the prompt “The major results obtained by non-commutative geometry include …” after 20 steps, at temperature 0.8:
NestList[StringJoin[#, model[#, {"RandomSample", "Temperature" -> 0.8}]] &,
"The major results obtained by non-commutative geometry include ", 20]
The major results obtained by non-commutative geometry include vernacular accuracy of math and arithmetic, a stable balance between simplicity and complexity and a relatively low level of violence.
Lol.
In the more philosophical sections of the book, Wolfram speculates about the secret rules of language that ChatGPT must have found if we want to explain its apparent succes. One of these rules, he argues, must be the ‘logic’ of languages:
But is there a general way to tell if a sentence is meaningful? There’s no traditional overall theory for that. But it’s something that one can think of ChatGPT as having implicitly “developed a theory for” after being trained with billions of (presumably meaningful) sentences from the web, etc.
What might this theory be like? Well, there’s one tiny corner that’s basically been known for two millennia, and that’s logic. And certainly in the syllogistic form in which Aristotle discovered it, logic is basically a way of saying that sentences that follow certain patterns are reasonable, while others are not.
Something else ChatGPT may have discovered are language’s ‘semantic laws of motion’, being able to complete sentences by following ‘geodesics’:
And, yes, this seems like a mess—and doesn’t do anything to particularly encourage the idea that one can expect to identify “mathematical-physics-like” “semantic laws of motion” by empirically studying “what ChatGPT is doing inside”. But perhaps we’re just looking at the “wrong variables” (or wrong coordinate system) and if only we looked at the right one, we’d immediately see that ChatGPT is doing something “mathematical-physics-simple” like following geodesics. But as of now, we’re not ready to “empirically decode” from its “internal behavior” what ChatGPT has “discovered” about how human language is “put together”.
So, the ‘hidden secret’ of successful large language models may very well be a combination of logic and geometry. Does this sound familiar?
If you prefer watching YouTube over reading a book, or if you want to see the examples in action, here’s a video by Stephen Wolfram. The stream starts about 10 minutes into the clip, and the whole lecture is pretty long, well over 3 hours (about as long as it takes to read What Is ChatGPT Doing … and Why Does It Work?).
Lately, I’ve been reading up a bit on psycho-analysis, tried to get through Grothendieck’s La clef des songes (the key to dreams) and I’m in the process of writing a series of blogposts on how to construct a topos of the unconscious.
Stella Maris is set in 1972, when the math-prodigy Alicia Western, suffering from hallucinations, admits herself to a psychiatric hospital, carrying a plastic bag containing forty thousand dollars. The book consists entirely of dialogues, the transcripts of seven sessions with her psychiatrist Dr. Cohen (nomen est omen).
Alicia is a doctoral candidate at the University Of Chicago who got a scholarship to visit the IHES to work with Grothendieck on toposes.
During the psychiatric sessions, they talk on a wide variety of topics, including the nature of mathematics, quantum mechanics, music theory, dreams, and the unconscious (and its role in doing mathematics).
The core question is not how you do math but how does the unconscious do it. How it is that it’s demonstrably better at it than you are? You work on a problem and then you put it away for a while. But it doesnt go away. It reappears at lunch. Or while you’re taking a shower. It says: Take a look at this. What do you think? Then you wonder why the shower is cold. Or the soup. Is this doing math? I’m afraid it is. How is it doing it? We dont know. How does the unconscious do math? (page 99)
Before going to the IHES she had to send Grothendieck a paper (‘It was an explication of topos theory that I thought he probably hadn’t considered.’ page 136, and ‘while it proved three problems in topos theory it then set about dismantling the mechanism of the proofs.’ page 151). At the IHES ‘I met three men that I could talk to: Grothendieck, Deligne, and Oscar Zariski.’ (page 136).
I don’t know whether Zariski visited the IHES in the early 70ties, and while most historical allusions (to Grothendieck’s life, his role in Bourbaki etc.) are correct, Alicia mentions the ‘Langlands project’ (page 66) which may very well have been the talk of town at the IHES in 1972, but the mention of Witten ‘Grothendieck writes everything down. Witten nothing.’ (page 100) raised an eyebrow.
The book also contains these two nice attempts to capture some of the essence of topos theory:
When you get to topos theory you are at the edge of another universe.
You have found a place to stand where you can look back at the world from nowhere. It’s not just some gestalt. It’s fundamental. (page 13)
You asked me about Grothendieck. The topos theory he came up with is a witches’ brew of topology and algebra and mathematical logic.
It doesnt even have a clear identity. The power of the theory is still speculative. But it’s there.
You have a sense that it is waiting quietly with answers to questions that nobody has asked yet. (page 68)
I did read ‘The passenger’ first, which is probably better as then you’d know already some of the ghosts haunting Alicia, but it’s not a must if you are only interested in their discussions about the nature of mathematics. Be warned that it is a pretty dark book, better not read when you’re already feeling low, and it should come with a link to a suicide prevention line.
