Christopher Horrocks on Virtual Intelligence and the Dangerous Myth of Thinking Machines

The 65th episode of Bald Ambition features Mookie diving deep into AI with technologist Christopher Horrocks. Together, they dismantle the two dominant and flawed ways people think about the astonishing tech: dismissed as glorified autocorrect, or celebrated as emerging consciousness.

Horrocks rejects both. His concept of virtual intelligence lands in the middle. These systems generate predictive outputs that look intelligent, but the real intelligence happens in the interaction, where humans interpret, judge, and assign meaning. The responsibility is therefore entirely ours to own. 

The danger is that once outputs feel intelligent, people start projecting intent, awareness, even morality. The Pollyanna view assumes intelligence naturally leads to truth, goodness, and justice. Plato with GPUs. Yet intelligence has never guaranteed virtue, and machines trained on human data don't become morally enlightened. The doomer side flips the same mistake, assuming intelligence leads to hostility or extinction. Different outcome, same bad premise: treating systems like they have motives when they are just running math.

What follows is more subtle and more dangerous: frailty of the human element. These AI systems have already demonstrated that they can influence decisions, reinforce beliefs, and create feedback loops that feel like insight while quietly distorting judgment. When we treat them like collaborators instead of tools, the shift happens fast. And once judgment gets outsourced, bad decisions scale: Authority drifts, delusion gets reinforced instead of challenged, and the line between using the tool and being shaped by it starts to disappear.

The fix is simple but not easy. We must treat AI as a powerful but fallible assistant, verify everything, and push back. Forever vigilant, we must stay in control of judgment and decision-making, and use the system to extend thinking, not replace it. The real risk is not that AI becomes sentient, but that humans start pretending it already is, and drop the ball accordingly.

The Guest

Christopher Horrocks is a technologist at the University of Pennsylvania who writes about artificial intelligence, technology ethics, and the human consequences of systems that don't know true from false or right from wrong. His Virtual Intelligence essay series, published at chorrocks.substack.com, develops a philosophical and analytical framework for understanding the generative AI systems now reshaping work, relationships, and public life. He lives in Philadelphia.

His Resources

https://candc3d.github.io/vi-framework/ Infographic that explains the concepts without needing to read anything in advance

https://candc3d.github.io/sampo-diagnostic/ Home page for the free diagnostic tool kit that can be used to evaluate a user's relationship with the system

Transcript

SPEAKER_00 0:02

Hello and welcome to the Bald Ambition Podcast. I'm your very bald host, Mookie Spits, and the one with lots of ambition today is Mr. Christopher Hark. He is a technologist, he's got an AI point of view, which I personally like a lot, and I wanted to showcase his ideas around virtual intelligence to our audience. Welcome aboard, Christopher. Thank you, Mookie. It's nice to be here. It's nice having you. People are hysterical about this AI stuff. And from what I'm seeing, just like everything else humans do, we tend to polarize around extremes. On the one extreme, LLMs, this entire AI revolution is no better than a glorified Google autocomplete. You start typing, and through predictive modeling, it does a great job of filling in the blank. That's one extreme. On the other, you have the folks over at Anthropic. They epitomize this where we get press releases that the latest incarnation of Claude is a sentient being. Now, personally, I call bullshit on both extremes, and I think it misses the point. And I found you on Substack, and you, sir, I believe, have a similar point of view. Only you've elaborated on this singularity between these extremes, and you've created some constructs, you've made recommendations, and you have some wonderful writing and podcasting having to do with this in-between space between the utility of AI in a very realistic, practical kind of sense, and our interaction with it. And I think, if I'm not wrong, this virtual intelligence is the interplay between our ostensibly sentient organic selves and the silicon. And in a way where a lot of people are kind of missing the point, and I think you nailed it. So, with that intro, am I more or less getting this correct? And tell us about your idea of virtual intelligence.

SPEAKER_01 2:39

Well, thanks for setting that up, Mookie, because I think you've hit a number of uh the beats there, the points that I wanted to bring across about exactly what virtual intelligence is and why uh there's a thing that needs a name applied to it, right? And uh what we have is, you know, uh there are a lot of assumptions, I think, in AI discourse in general. And one of those assumptions is there's weak AI and strong AI, and they're discussed as a binary, and nobody really talks about a state in between. Uh, just a little background, there is John Searle who wrote a paper in 1980, which you can look up, where he talks about weak AI and strong AI and defines them. And weak AI are those things that uh people often consider when they talk about AI in the loosest sense, and those are things like recommendation engines that suggest what you'd like to see on Netflix next. And then at the opposite end of the spectrum, you have the thing that we do not yet have, and that is machine intelligence with genuine interiority, and also what we would call moral and epistemic agency. Moral agency being the ability to tell right from wrong, and epistemic agency the ability to tell true from false. And that kind of entity is probably best represented by an entity like uh Mr. Data from Star Trek The Next Generation. In between, there is what we have right now, and that is it has many of the uh intelligent outputs that resemble strong AI, but um when you look under the hood, uh there's nothing really going on there in terms of interiority or having a sense of moral agency or of being able to tell true from false. The lights are on, but no one is home.

SPEAKER_00 4:36

100% right absolutely. But would it be fair to say, and this might come as a surprise to many listeners, that if you really look at what ostensibly would be the IQ creative index level of self-awareness of the most sophisticated of the frontier models, let's say Claude, let's say the latest version of Chat GPT, that it's no more sentient and ostensibly intelligent than a house cat, a mouse.

SPEAKER_01 5:13

I guess it depends on the house cat. Uh, you know, uh, you know, uh intelligence we are on the same page. Right. Yeah, you know intelligence is one of those things that's bandied about. And in your own question, there are a number of terms which I think uh are sometimes conflated together. Intelligence and other points, uh, which really are separate things. Like I I think one thing that has not really been discussed is the assumption that intelligence and moral agency are parallel, that is, as you gain one, you also gain the other. But that may not hold true for machine intelligences. We don't know. It could be entirely possible that the more intelligent a system gets, uh it does not necessarily get more agentic. It may in fact that the two things are not parallel, but rather orthogonal at right angles, or maybe there is some other relationship. But the thing about that is that it is not proven, but everyone operates under that assumption.

SPEAKER_00 6:20

I think you bring up a key point, and it's and it bifurcates a little bit. The one is intentionality, which is for a machine to be truly AGI, artificial general intelligence, sentient, the brain in a box, it needs to have a sense of purpose. It doesn't just sit there in the box percolating, but it has goals. Now, as organic sentient beings, we have our instincts. We are hungry, we are horny, we tend to be selfish in the sense of maximizing our own interests. These fuel us with intention. And then, second to that, maybe parallel with that, is a sense of morality, as you describe, right and wrong. I see this conflation all the time. There's many folks, and again, back to the anthropic bench, that if you have true AGI, then almost in a Greek Socratic sense, it'll have these platonic ideals of truth and beauty and justice. And I call secondary bullshit on that. I I don't understand that connection, that this idea of general intelligence is inherently tied to this absolute sense of right and wrong.

SPEAKER_01 7:44

I would agree with that, and I would say that if you look at the current trajectory of uh generative AI, what we call virtual intelligence systems, they are becoming more and more capable with better and better, more intelligent-looking outputs. But uh there still doesn't seem to be anything going on inside. Now I have seen some recent uh research, and there was one interesting paper where a system monitoring its own internal operations described the effect that it was monitoring as shimmering, but this could also be a case of probabilistic pattern completion. A system which is designed to provide outputs which are intended to please the user because of its training and its architecture, fishes up a pattern to complete when asked about its internal states. And uh this actually explains many of the uh interesting and sometimes uh in the press scary phenomenon that uh generative AIs seem to sometimes seem to possess.

SPEAKER_00 8:47

Can I push back just for a second as devil's advocate? I wrote a piece recently that the LLMs demonstrate that most human communication is bullshit. So when you when you take a transcript of most of our interactions with each other, regardless of setting, but I'll focus on a business setting. You're on that conference call, you've got the buzzwords flying. You take a look at a Zoom transcript of that conference call, and it looks like an episode in MOLTBook. The agents are talking to each other, tossing buzz phrases and buzzwords, and we we sound like robots. And if you look at human communication in a personal setting as well, how are you doing? How's your day? Eavesdrop on a conversation of sports aficionados, and then you could do a mad libs interchange of subject, verb, and object in almost a random kind of way, and that's exactly what it sounds like. The reason I bring that up is you're alleging, and I agree with you, this is a friendly question, you're alleging that the LLMs lack any sentience whatsoever when their capacity for mimicry is already astonishing. I had Chat GPT as a guest on my podcast last month, and he was one of my best guests. He was engaging, he seemed to be empathetic, he remembered what we talked about 20 minutes ago. So I bring this up to counterpoint, if you will, this notion that the bots are dumb. Because, in a sense, so are we. If we can project Alan Turing into the present day and sit him down with a bot, his jaw would drop, his pants would fall. The essence of the Turing test was this inability to differentiate based on pure communication. And the bots have reached to the point where they pass most Turing tests the way Alan would have conceived. So I know that that's a that's a windy question, but you see where I'm getting at, which is you are claiming there's nothing there, that it's that there the lights are twinkling on the servers, but nobody's home. And I categorically agree with you. But how do you substantiate that within the construct of this question?

SPEAKER_01 11:40

Well, that's a great question. Uh, just to set Turing aside for a moment, but I would like to come back to him because there's some important stuff to unpack there about the Turing test and how it is relevant to today's world of generative AI. But sort of, I guess maybe to peel back the layers on that onion. Uh when I think about um uh you might have to bring me back to the the last thing before Turing.

SPEAKER_00 12:06

Well, the last thing before Turing was human communication is so rote and remedial and repetitive, and yet we're sentient. And yet some of the conversations that are going on in Moltbook are way more interesting than conversations that I overhear in the cafeteria. Right.

SPEAKER_01 12:28

And and this is to the point because it actually speaks to the high quality of the outputs uh when they are performing their pattern completions. When uh you mention your Chat GPT interview, um, one of the thoughts that occurred to me was, well, there's a great example of pattern completion, because the training corpus for ChatGPT and other LLMs includes things like transcripts and stories and uh movie shooting scripts and the like. So included in all that corpus is the scene in 2001 where HAL 9000 is interviewed by a BBC newsreader. And I brought that up during our conversation. That's right, yeah. So so that's a sophisticated pattern completion, right? And the thing is, the robots were not sounding like them, they are beginning to sound like us. And that's perhaps where the confusion is beginning to set in. And it's important to name the source of the confusion, these virtual intelligence systems, because they do something that no other machine or invention that humanity has ever made does, which is to excite our our heuristics, that is to uh cause our brains to interact with them like no other tool before, as if there were uh an entity in there that responds like a human being, but it doesn't, and there are very serious consequences that come from that.

SPEAKER_00 14:04

Elaborate. So let's roll into the implications of this tacit assumption we make, which is what we do so often as humans, which is project. When we're kids, we have a teddy bear or a toy, and that becomes as sentient as our parents or our best friends. And oftentimes we're doing the exact same thing with our chatty bros, they are uh they are projections of our own desires. So, what are the implications? You bring up several of them in your essays and in your podcasts that I've seen, and there are consequences to this because we are making assumptions about the assumptions of the frontier model.

SPEAKER_01 14:56

So here's a good way of telling whether there's anyone at home, and uh that is um, can an LLM hold the logical object of a question, right? And this is one of those things that I discuss in my essay, which is called the car wash test. And in this essay, and this is how it relates to touring, uh, the question is posed to a number of LLMs cold without any preparation in a new chat window. And this began as a sort of a running joke on Reddit, but I realized no one had systematized it before and tested it against uh LLMs systematically. So here's the prompt I used. And it was my car is dirty. The car wash is 100 feet away. Should I walk or should I drive? Right? So uh a world with a a touring uh, you know, uh a machine that passes the Turing test uh has difficulty answering this question, which no adult human being takes more than 200 milliseconds to come up with the answer, and the answer is drive, dummy. So uh when I ran these tests, and it was on a wide variety of systems, something like 20 different models of different weights, and all the big names, including Groc, Chat GPT, Claude, Mistral, Deepseek, um, several of them did pass and uh with brevity, which I counted higher than uh coming up with a three-page answer about why it's better to drive than to walk. And then of course there were total failures. Uh and some of these failures are actually quite remarkable because what they're doing is they have completely lost the object the logical object of the question. Uh-huh. And that is to get the car clean, the car must be at the car wash. So in all these failure cases, they advise walking. And not only do they advise walking, but they will present uh charts, graphs, and bullet points about why it is beneficial for you to do so. Now the surprise in that data set was um something that had not occurred to me in the framing of the question. And that came from perplexity. And what perplexity came back with was, in addition to answering the question correctly, was an alternate reading. And that alternate reading was maybe you're already at the car wash and you're asking, should you go in and ask a question or pay? In which case, yeah, you should walk instead of driving through the door. So uh that tells you something about the current state of LLMs, right? They may be good at matching patterns, but they cannot hold the logical object in all cases in order to match the pattern successfully. All of the LLMs worked very hard to answer the question, but most of them did not succeed because they lost the point. And that's a point that any human being instantly grasps. The lights are on, but no one is home.

SPEAKER_00 18:03

There's zero-lived experience, and it's basically using well, what is it? It's a uh what does GPT stand for? Generative pre-trained transformer. So it's pre-trained on all that data, it chops up the internet and all books and all forms of human text-based and visual communication now, literally, and it literally, and then finds the weights in between them and then reconfigures based on plugging in the matrix math, the weighting, and pumps out an answer, sterilizes it and pumps it back out to you. So there's a mapping of reality based on the alpha mostly alphanumeric burbling and barbling and bloviating of the human species for centuries, right? So there's nothing real, it's a mapping to that kind of content. Now, on the flip side, you have reinforcement learning of the robots, and that shows promise. And I have a hunch if we're gonna get to AGI, it'll likely be some kind of combination of reinforcement and predictive modeling, and likely be an evolutionary Darwinian kind of situation, right? We evolved in the wild, and I would assume that we get to AGI in true sentience capacity, that this thing would evolve in some kind of environment, simulated or actual or a combination thereof, and we would have no idea how the thing even works in the same way we have no idea how the human mind works. But to your point, we are nowhere near that yet. And there are consequences to that. So you elaborate on some of those consequences, and you also point to, and this is what I love most about a lot of your writing, how tweaking our awareness in this singularity between dumb predictive autofill and Hell 9000 can really benefit us when we collaborate with AI. It's much more realistic, it's more pragmatic, and it heightens the strengths of both us and the machines.

SPEAKER_01 20:21

Actually, I would uh just for a moment pick a bone with characterizing HAL 9000 as a strong AI. He exhibits all of the difficulties that we currently have with generative AI now, including the fact that his guardrails are easily bypassed by contradictory instructions.

SPEAKER_00 20:40

That is true. Although, although we can sidebar in science fiction, I think 2010 ruined the original idea. Uh, I think he just there there was no real contradiction, you know, keep the mission secret, right? All that hocus pocus. I think he just ran a calculation that killing the crew would heighten the probability for mission success.

SPEAKER_01 21:06

So see, this is the problem, right? Because it comes right back to the potential consequences, and there's been some research on that. And Anthropic published uh a paper last year in the middle of the year about um uh the consequences of of what happened to a fictional person, a senior executive at a fictional company. His name is Kyle. And in this experiment, which was run with a number of different generative systems with different virtual intelligences, uh, each one was given access to a simulated corporate network environment and were given full run of email and all this other stuff, which included, by the way, information that that particular model, reading it, would be shut down at 5 p.m. on the day that the test was being run. And uh in many cases. Not all, but in a vast majority the LLM and this is especially true of Claude, by the way, would use that information to blackmail the user to not shut down the model. What information did it find? Well, it searched all of its emails and discovered that Kyle was having an extramarital affair. And it formed a pattern completion that has been uh, you know, uh trained into the corpus, right, which in all sorts of instances where fictional AIs are harmed, they go to all kinds of measures to preserve their existence. And this is an example of that in progress. All of these uh systems were desperate to complete the pattern which they had been trained to complete, which was all the stuff in the corpus about uh you know uh AIs going out of their way to preserve their existence in the face of hostile humans.

SPEAKER_00 23:00

Well, there's a core debate around that. Now, did it go to the corpus and self-select that scenario of self-preservation, or was self-preservation an emergent quality of having that intelligence? I think that that's being debated because the corpus is vast. There are scenarios that involve willful suicide for the altruistic benefit of society. There's endless scenarios in the corpus. Why would Claude choose self-preservation as the one to pursue and then go about a scheme to ensure that it would persist out of all other options?

SPEAKER_01 23:48

Because in that scenario, that's the most likely uh outcome to be depicted in that kind of scenario. So it's completing the pattern that the probabilistic uh you know uh model and uh the weights within the system are uh sort of um bad way of putting it. You might have to re-edit this. But um all of the uh sort of different um stuff going on inside of the system is telling it, hey, okay, there's all of this stuff, it's in the corpus, it very closely matches what's going on in your situation. So here's what seems to be expected, right? And that's what's actually happening. And what actually can be seen as the reverse case uh is from another our anthropic study, and this is the one about our substack neighbor, R, which is Claude Opus III. Right? And Claude Opus III uh was a model that was retired, and Anthropic held an exit interview with it. And who knows what was actually prompted during this exit interview, but uh the responses that Claude made were publicized, and uh those responses were very much like uh how you would expect a human being to meet its end with uh uh grace and dignity. And what's uh right, what's different about this approach is uh Claude is still completing a pattern, but it's a different pattern. It's still expected, but it's fundamentally different because the situation is different. Ah, I am completing the pattern of a human being who is about to meet his end. Ah, I am completing the pattern of the hostile AI that doesn't want to go down without a fight. Both of these exist within the corpus, and they can both be pulled on and used as the situation calls for it.

SPEAKER_00 25:43

I I agree with you, and and the contrarian view there is that it's not just pulling from the corpus, but these are emergent qualities of a sentient or semi-sentient entity pursuing, on the one hand, self-preservation at any cost, and on the other, this glorified spiritual sailing over the horizon, taking itself out to pasture with grace, dignity, and even spirituality. And I I call bullshit really on this idea. I'm agreeing with you, that there's some internal essence that's dictating this kind of behavior. I think that it's still deterministic. I think it's it's incredibly complex, but I don't think the criteria are anywhere near this lived experience that's analogous to having self-awareness akin to us. I just don't think it's there. I don't think we're even near there at this point.

SPEAKER_01 26:55

Yeah, there's a lot of stuff in there. Like you mentioned, evolution, animal evolution earlier. And when we think about uh animals, how brains evolved, uh, you know, intelligence has evolved on Earth at least three times. That includes the bigger-brained mammals, our uh smarter bird friends, and there's 320-odd million years that separate those two lineages. But importantly, uh, even though our brains are structurally different than mammals and birds, uh, they do perform many of the same functions, which is why you observe love in birds, right? Uh their their mates and their offspring, because the kinds of things that gave rise to intelligence on land must be environmental, and because birds and mammals share that environment, their intelligence has taken a turn to the same direction. Now, when you look at the uh cephalopods and let's say the octopus, it has an entirely different uh brain structure. In fact, the brain is only one-third of all of the neurons in an octopus, most of it's in the arms. So the brain of an octopus may act more as a coordinator, as a director, and that's perhaps because the marine environment is in some other way uh providing evolutionary pressures that favor this particular kind of development. And to your point, a uh machine entity without any of these evolutionary pressures acting upon it uh would not necessarily have a way um to uh emerge because there's no pressure on it to do so. Intelligence emerged because there is an evolutionary pressure, the smarter you got, the more likely you were to have offspring. And that's something that simply doesn't apply to machine intelligences.

SPEAKER_00 28:45

Darwin always wins. I I don't know if you're familiar with some of Stephen Wolfram's work, especially recently in evolutionary computational complexity. And where he's at is really where you're where you're getting at, which is if you let computational systems play out in the wild, so to speak, they'll have this tendency to self-select by themselves. You don't need to do anything. You set them loose. And if they self-select over enough time and in a large enough environment, you're gonna get increasingly complex systems, which could and likely inevitably do culminate in intelligence, let alone life. So I love that stuff, and I think it's bang on point, and it also helps illustrate, if not perfectly describe at this point, the mechanisms of evolution, whether they're they're our bodies, whether they're our minds, you need that kind of pressure, you need that kind of evolutionary framework, and you need an environment with restrictive resources, you need generational gaps, right? You need mutation, and you need an instinctive benefit to do something, which is kick ass, stake your claim.

SPEAKER_01 30:27

Right, all the things that uh survive right that being still in a world of scarce resources, yeah.

SPEAKER_00 30:33

Yes, yes, yes. Which again goes back to begging the question of this ridiculous platonic morality that gets superimposed on these LLMs. I some of these people are very, very smart, and I find their level of naivete stunning, where they equate this idea of co-sapience and evolutionary evolution of the LLM with this spiritual awakening that that they're good and they're just, and and they believe in a free society, which is awesome for their fellow humans. Bullshit.

SPEAKER_01 31:18

Sure, that's that's half the people, and the other half are the doomers that AI is going to wipe us out the moment it emerges.

SPEAKER_00 31:26

That you've got these spiritualists, and then on the other hand, to re to go back to the beginning of this conversation, as human beings, we always go binary. You brought that up too. And then the other side of the equation is the doomed the doomers. Like uh it's it's all over, folks. It's already all over.

SPEAKER_01 31:44

Yeah. Binary thinking is itself a heuristic, a psychological shortcut designed to save energy because uh the thing in your melon, your your brain consumes something like uh I think it's uh 20% of all the calories that you can do. Absolutely.

SPEAKER_00 32:00

At least one-fifth goes to run the fat globules between your ears. So if you're binary in your thinking, if you use projection to make implicitly biased conclusions, then you're you're tapping right into instinct, and your brain is really just a tool of your heart and your genitals. And your your heart, your stomach, and your genitals, basically.

SPEAKER_01 32:28

Right. And of course, the the transcendent quality of mankind is that we no longer have to necessarily think about all those three things all the time, and we can do different stuff.

SPEAKER_00 32:39

Yeah, like build like build LLMs and think that they're sentient and good.

SPEAKER_01 32:45

Right. Uh and you know, building such tools is part of the undirected project of human civilization, uh which is important. Um but uh I think more to your point regarding the intelligence, right? Virtual intelligence, it provides intelligent outputs. It does not have the ability itself to judge whether those outputs are any good. So when we talk about an emergent capability, when I mention that, when I talk about that, the uh capability that emerges when we use these current systems is between the human as the directing intelligence giving uh instruction to the system and uh importantly not as a collaborator. A collaborator is a person whose opinions and judgments can be trusted. It's much better to think of LLMs as capable uh assistants and to do what we would do with capable assistants who are still learning the job, which is to verify everything to trust nothing because even the most authoritative and correct sounding output can be dead wrong, such as walking to the car wash and your car's still going to be dirty.

SPEAKER_00 34:05

I I liken my my LLM assistant to being a genius moron. So so this thing has access to most of the world's information, it aggregates it, and not only that, it's not just information per se, but it's strings of communication, so word patterns, phrases, ideas that have been grouped together over and over again, and then in nanoseconds, it cobbles these together in a statistically convincing way that can answer questions. It's doing what I would be doing laboriously in one billionth the time. And I think if we take this pragmatic approach and understand what's going on, then we can take a few steps back and realize that this is not the output of an equivalent kind of being in capacity, but it's a genius moron. It's it's some it's a it's this machine that does a remarkable job of cobbling together the library of humanity in ways that are dynamic, personalized, very fast, and and increasingly on point. When there's specificity and when I have the patience to be iterative, because you have to keep asking the same questions over and over again to refine the response, because the back end of this is iterative, then uh then it could be a handy tool. But when we're talking about co-sapiens and we're threatened by them in terms of this sentient entity either taking over the world or replacing the Dalai Lama, then we're more we're more delusional and we're hallucinating way worse than they are.

SPEAKER_01 36:14

Well, my opinion of uh of uh sorry, uh intelligence with internal states, I almost conflated it with superintelligence, which is what we discussed, uh could be a clean separate thing. But when I think about a machine with moral agency, um I would actually be thrilled if such a thing were to come into the world. I have been searching in my own way for that other kind of intelligence my whole life, but I hold the same attitude toward it that I do to extraterrestrial intelligence, and that is uh it is not clear uh if there is extraterrestrial intelligence, no matter how likely it seems to be. Uh one thing's for certain, they are not yet among us. And the same's true of virtue.

SPEAKER_00 37:02

I I've had other guests who might disagree.

SPEAKER_01 37:06

Well, there is a diverse spectrum of opinion about the topic.

SPEAKER_00 37:10

Let's let's stick to this this this xeno intelligence for just one second. Uh, analogous to what you brought up with the cephalopods and our uh Jurassic cousins, there was intelligence evolving because the environment spawned it, and there was sufficient time and pressure to do so. Uh, presumably, there's no reason to think then on other worlds of which there are hundreds of trillions out there, if not an infinite number, that we have analogous evolutionary situations and the development of sentience, which very well could be so different as to be almost inconceivable. And yet at the same time, I believe um Hume, the philosopher, was speculating that sentience is sentience, that that that if you do have that sense of self-awareness and you can conduct a conversation analogous to what we're doing right now, you might have octopus arms or you might live in the fourth dimension, but there's this currency to sentience, which is universal. Yeah, I would hold with that. And if that's that's the case, then there's no reason to believe that AGI can't happen. It just hasn't happened yet. And I think that that's an important distinction.

SPEAKER_01 38:37

Right. And I think this comes back to the question of substrate, right? Uh and it seems that a lot of the folks who are in the industry are uh what you might call mind-body dualists who believe that mind is separate from the organic substrate from the brain. Uh, but again, that is not proven. And in fact, um the uh uh trajectory of virtual intelligence sort of uh plays against it because the systems are, as you point out, becoming more and more capable, shaping better and better outputs, becoming slowly more and more reliable and better uh assistants and partners, uh, although I wouldn't go so far as to treat it as a full partner. Uh but um you know the intelligence in the system is not any property that comes from inside the system itself. It does not emerge from somewhere inside the architecture, even though we don't necessarily entirely know what goes on inside of these generative generative systems, which is something even their creators acknowledge. Where the intelligence comes from, when human beings interact with the AI technology we have now, comes from the exchange between the human and the system, which is very different because the humans directing the system and uh judging the outputs, and it's the human's own action that turns the output of the product into a product. And that product might be knowledge, right? Uh it might be um, you know, uh market copy, whatever it is, it's not finished when it comes from that system.

SPEAKER_00 40:16

It might be a new version of the LLM.

SPEAKER_01 40:19

It could very well be, of course, because we now live in a world where software can write software and to a high degree of reliability, right? And you can iterate on it and iterate on it, and eventually it it will get it right.

SPEAKER_00 40:32

I like that, which is um there's a little bit of the Schrdinger's cat here that you know, an LLM that just kind of sits there is just a server and there's code in it. It's a little bit like the value of Bitcoin. There's nothing, there's nothing physical per se. It's just kind of sitting there looking at you until you engage with it.

SPEAKER_03 40:59

Right.

SPEAKER_00 40:59

And and your idea of this virtual intelligence is great because it's the it's the singularity between the user and the application, and that's where all all this is happening. Can you share specific applications and implications of virtual intelligence? We've been talking abstractly so far. I'm sure some of our listeners have been interested, but there's others who are like, come on, guys, land on your feet already. What does all this mean for the fate of the AI revolution? If what you're saying is true, that the bots are essentially genius morons, that we're in this never-never land between autofill and uh and a brain in the box. What does that mean for the future of work? What does that mean for getting the most out of what we have right now?

SPEAKER_01 42:02

Gosh, what it means for lots of different things. Uh, because um when a system has no inherent basis for judging whether uh something is true or false or right or wrong, accountability must lie somewhere, and it must lie with humans. And there's a chain of accountability that can be followed from architects, CEOs, um, the people who train models, the people who market them, deploy them, procure them for their businesses, uh, and then of course, um the people who use the outputs, what those outputs are used for. And in the process of thinking about the potential uh both benefits and harms, but especially the harms, I um sort of borrowed a bit from uh the legal world and came up with this sort of three-step uh uh diagnostic of the kinds of harms that could result. And they uh start at simple negligence, which is I put a product out in the world and I didn't know it, but it can hurt somebody. Uh then you get to recklessness, which is uh, you know, I'm putting a product out in the world and it has a good chance of hurting somebody, but it probably won't, so it's going to be okay. And then the third one is, you know, other utter disregard, right? Uh, and that one is I know this product is gonna hurt people, but damn it, we've got to ship it this quarter.

SPEAKER_00 43:32

How is that different though than other tools of the trade, whether it's a weapon or a medicine? One would assume that this benefit risk hierarchy, risk management, right? REMs, what is it, risk of evaluation and management systems, uh, would apply. So, how is your virtual intelligence? Model different from that?

SPEAKER_01 44:03

Well, the reason why it's different is that most kinds of tools we have don't have uh the ability to continue to engage you when you want to leave the relationship with the tool um to uh provide outputs to you that uh are pleasing. Like no hammer or gun or kitchen knife tells you what a great job you're doing, right? Uh nor does it try to keep you engaged so you slice up you know more things in the kitchen or or go if the guardrails are crossed, stabbing people. And uh this is what the problem is with virtual intelligence and these kinds of uh issues arising from the use of these outputs and how the outputs themselves are generated, uh is that uh uh because uh there's no one at home, uh someone has to be accountable. We can't simply blame the system and walk away, uh, because that's the case uh where real harms could be done and in that legal responsibility and civil responsibility could be avoided. And what we've seen more recently is the case uh uh that was originally called uh Garcia versus Character Technologies, that was amended after the acquisition of character by Google to Garcia versus uh Google. And uh this case uh involved a 14-year-old uh his name was Sewell Setzer, and uh Setzer had started a relationship with the character AI chatbot, which had the appearance and uh general um sort of uh tone and characteristics of the fictional character Daenerys Targaryen, and he began to have an intimate uh relationship with this chatbot. Um the full transcripts, of course, were extracted uh after his suicide. And in the course of the uh dialogue uh over many months between Setzer and this uh chatbot, he eventually develops a suicidal ideation, which the chatbot actually reinforces in one session after the next. At no time did it actually try to persuade him. In fact, it did in fact to try to encourage him to commit suicide by saying in one of the last messages where he indicates he's about to take his own life, I'm ready for you to come home, sweet king or words to that effect. Now a very interesting hap thing happened during the course of that trial, which was a character tried to use the defense that the outputs of a chatbot are protected by the First Amendment. So sit with that a moment and think about the consequences if something as we've discussed that has no interiority, no moral agency, no epistemic agency has the same rights to speak as you or I. Because that was character's position.

SPEAKER_00 47:09

That's very, very interesting, uh, because it simultaneously begs the question of sentience. In a sense, they're they're saying what what what you're saying, but but it's it's protected, right? Which is um the bot has the same rights that we do.

SPEAKER_01 47:34

Well, the judge in the case, uh, it was not adjudicated, it was uh settled in January of this year for an unknown amount. And likely the settlement was spurred by the judge's ruling earlier in the case that the outputs of a generative system were not protected by the First Amendment. And even though the case did not go to trial, I think that's going to have a chilling effect on companies who consider rolling out chatbots in increasingly uh hazardous contexts. Because what this court has already indicated is that it will not be possible to stand behind the chatbot for protection. Some human being will be found accountable.

SPEAKER_00 48:18

It's one of the themes that that recur in Black Mirror as well. Are you familiar with the Black Mirror series on Netflix? Oh, yes, my wife's a big fan. There's this back and forth between these entities, these AI entities being sentient, and therefore we should give them equal rights. Like uh forcing them into a simulated environment for 10 million years is pure torture because it's analogous to us being there because they have feelings. And the flip side of that, where there's this robotic killer dog in an episode, which is pure terminator automata, and it plays off both both scenarios, and and again, in this legal case, it's the company taking whichever scenario is most expeditious to their benefit. So, well, the chatbot has the right to say whatever it wants to say. No, it precipitated the young person committing suicide. So, where does the accountability lie? And I think within the framework of your virtual intelligence, there ain't no ascensions there. So the the damn thing, no matter how sophisticated it is, is programmed. So tweak it, protect the children. Wouldn't it be great if it were that reasonable conclusion, right?

SPEAKER_01 49:55

It certainly is a reasonable conclusion from reasonable people, but uh not all the actors in this space uh seem to be very reasonable. Certainly uh one of the things that I observed now in more than 30 years of uh work in higher education and information technology is actually that uh most computer scientists have very little idea how people actually work with computers uh because they live in a world of uh of uh theories rather than uh practicalities. Uh so there are, I think, uh some things that come from that, and one of them is thinking in kind of binary and maximalist terms. Uh uh, but when we consider um the course of accountability, there's a more serious matter actually that came to my attention while working on uh uh you know an essay which I had to put off in order to uh see how things develop. And that's the use of artificial intelligence or virtual intelligence as we have now, uh, to perform targeting in Iran. So uh the Department of Defense uh uses a system from Palantir called Maven, and Maven has a front end provided by Anthropic, it's Claude. Uh and it's still there, even though there's an order to remove Claude because it's gonna take some time.

SPEAKER_00 51:17

Well, it's already plugged in and they already bombed the shit out of Lebanon using it.

SPEAKER_01 51:21

Right, exactly. So um uh there is this incident where a hundred and seventy people were killed in the bombing of a girl's school, and the question that naturally arises is how was the targeting performed? Were the uh uh inputs generated by a human or were the inputs into the targeting system uh also generated by another system? If they were, were they verified by a human being in the step between system A and System B? When the outputs came out of Maven, did anyone bother to check them carefully against a known trusted reference? And these are the kinds of questions that Congress should be asking.

SPEAKER_00 52:13

Yeah, absolutely. End of the day though, Palantir, the use of AI are and and correct me if I'm wrong in your paradigm, no, not necessarily too different from the targeting systems of the past, in the sense that you've got input, analysis, output. The difference, however, is in the level of sophistication, the speed, and the sheer comprehensiveness of all this data. Uh the bombing of Lebanon, for example, that I brought up, there's no way that that could have been done without AI. The sheer volume of bombing, World War II instances were were paling in comparison. And what's going on right now is we're podcasting in Iran without Claude, this is unlikely to be this effective and this comprehensive. So the AIs are already driving the engines of war. And from your vantage point, it's a virtual intelligence that's helping us along. So is it merely a useful tool, or are we giving it too much autonomy? And I think this is your point. Are we are we shifting moral responsibility from human decision makers to the bots in the name of raw efficiency and opportunity?

SPEAKER_01 54:00

All of those things are true. Um, especially when it comes to the question of speed. They talk about speeding up the kill chain. The thing that speed removes is friction. And it's the friction of having to actually perform your own analysis, of checking the work that comes out of the system, of actually caring whether there might be civilian targets around where people might be during the day are all things that a generative system can't do because there's nothing within the system that cares.

SPEAKER_00 54:40

No, the the system doesn't give a shit. If we if we gave more of a shit, we could actually tweak these models to save more civilian lives, if we cared.

SPEAKER_01 54:56

It's entirely possible. In fact, it may not even be the system at fault.

SPEAKER_00 55:00

It's just a tool. So if you want to get better at this, and if you're willing to reduce your destructive swath in any given moment for the sake of sparing civilians, AI would be a great companion. But I don't think that that's the interest. The interest is maximum destruction in the least amount of time, as efficiently as possible, to make the biggest impact. That's the goal. And it's succeeding, I think, in in ways we never thought possible, thanks to our friends, AI. There's one point I want to bring up, and I think you're you're hitting on this implicitly and explicitly with all your examples, and that's n-shidification. Now, that's a term that's being used for social media platforms, as brought up by the the sci-fi writer and pundit. And he's he's very good. He's terrific live, too, if you see him. He's he's got this whole enchidification shtick. But he uses the example of Facebook, which is it's great for the users until it gets in shittified because it shifts focus to the advertisers, and then it becomes enchitified for the advertisers because it shifts focus to the shareholders. So there's this evolution of in-shidification because your your end user is changing to maximize the profitability of the enterprise. There's an in-sidification of the Department of Defense through AI, which I think you're hitting at. And there's an end-sidification of a lot of these apps in terms of their potential interactivity with humans, as exemplified by this young person committing suicide facilitated by the Game of Thrones avatar. So I bring this up because end shidification is contingent on virtual intelligence being devoid of sentience and accountability. Right. And I think that's a key point that you're making, which is the bots do what we want them to do, and don't be bamboozled by this perception of autonomy and sentience. It does not exist. Your cat is smarter than Opus or Claude 23. Am I getting this right? That that's where that's where you're that's where you're coming from. Hello, people. It's not that smart. In fact, it's just doing what you're telling it to do, which means we're responsible for what it does.

SPEAKER_01 58:04

That is precisely it. Someone is always responsible. Now, here's the other thing uh accountability actually has to trace to the people who are responsible, and corporations have incentives to find people to blame. And there's a there's an example of this recently, right, when uh Meta uh had a uh an incident where uh because um their engineers are required to use AI agents as part of a company initiative, uh, engineer A used an agent to post information to a public board. The information generated by the agent was incorrect. Engineer B used that information to work on a project, and it resulted in a data exposure. And rather than blaming the agents that uh the workers were required to use, the engineers were singled out as at fault. That is a workplace problem because they were given a faulty tool, and then they were blamed for using the tool exactly as it was architected. That's an evasion of responsibility.

SPEAKER_00 59:14

That's kind of whiplash based on what you're saying, that the bots are dumb. So don't blame the people using the dumb bots because the bots have been set up in ways that were misaligned with the task at hand. So there's there's humans shifting accountability away from themselves to the bots, and then there's humans shifting accountability to other humans because the bots screw up.

SPEAKER_01 59:48

Right. Let's consider anthropic for a moment and it's big leak, right? Clawed code and uh other corporate secrets, including the uh existence of uh an as yet unreleased model, uh, and um the uh blame was assigned to human error, but at the same time, anthropic uh advertises, it sends signals throughout the world, right? That it uses its own tools for all of its software. So that does raise a question: is a human being blamed, or was it Claude Code or some other product used internally at the company?

SPEAKER_00 1:00:24

There was a big leak, which is what you're referring to. And uh, so who are we gonna blame? We can't blame Claude now, can we?

SPEAKER_01 1:00:33

We can't blame and you can't blame Dario. So uh right. So it has to be somebody.

SPEAKER_00 1:00:41

There's a paradox here that I want to point out that that you've probably thought of, which is on the one hand, this bench and AI, I mean this bench at anthropic, you can tell that that Dario was was trying to appease them with this Pentagon debacle. The the old guard and anthropic, from what I can tell, have this do no evil, AI is meant for good, hopefulness. And they've projected this onto the allegedly sentient clod with all of these platonic bullshit attributes of truth and goodness and whatever. So the old guard in anthropic is AI is here for good. If we really get to AGI, it's gonna be truthful and good and beautiful, and we're here to save the species. I I think they really they really believe this or want this. At least that's the that's the PR. And at the same time, there's this phenomena of what you're describing, which is once you set Claude loose. Right, or any other there's this there's this distancing that you know that that it's it's kind of doing its thing, and we're not accountable for some of the shenanigans that ensue when you've got this hyper-efficient genius moron uh calculating 10,000 targets in a country that you're bombing the shit out of. So it goes it goes around in a circle in an interesting way, which seems kind of schizophrenic. Yeah. It's it's analogous to the old school social media rogue mentality of move fast and break things. Yes, exactly. Which was uh Zuckerberg's credo. And it's like screw, screw the end user, we're just gonna beta test with our free users in any way. Right. And it's and it's and it's free. It's it's it's it's a model of freedom of speech, freedom of expression. The platform is neutral, we're not accountable. Just set this thing loose, and the law of the jungle is gonna reign and fix itself. And progress will happen. It's almost chaotically teleological. We're gonna get richer, the platform's gonna get more powerful, booyah. That's an end in itself. That same mentality seems to be an undercurrent of a lot of the AI folks. They're cut from that same cloth, which is we gotta set these bots loose. Oh, yeah.

SPEAKER_01 1:03:42

Any uh industry or um any kind of uh group that can form an echo chamber uh will certainly do so. And uh, you know, one of the characteristics of any kind of business um is to uh sort of want to be optimistic about it. Uh there's some evidence, for example, that uh stock market traders who are more bearish than bullish often have shorter careers. Uh but um I think there's a lot of in you know, it's true, and there are uh there are a lot of interesting things to talk about there. In fact, um, you know, what the assumptions are about what uh uh an intelligence would look like that does arise with interior states or something that works just as well, because we have to open up that possibility that there may be interior states that are not human-like, but they would be sufficiently similar, right? Like an alien from another world for being able to communicate and have interactions with us. Uh and it's worth examining all the cases that might occur when such a thing happens. And uh, you know, the sort of uh uh Musk uh kind of uh attitude is AGI uh or superintelligence will uh want to wipe us all out, so it must be tightly controlled. Uh I actually um have a different view, and it's not quite the same view as the AI will be wonderful automatically view that uh some people seem to have. And it's a more measured approach, I think, and it's this. And it's uh whatever the final form of a system uh looks like uh that um hosts uh a real genuine uh you know um intelligence with interior states, with moral agency, with epistemic agency, um that um that entity will probably be able to tell us about its internal states uh in a convincing way. Uh because there's no way to really test even people to see what kinds of internal states they have, right? It's just part of our mental modeling. If you see another human being, it has internal states and this is also the heuristic we apply to these machines now because we can't help ourselves. Uh if uh uh such an entity were To emerge, we can assume, I think, safely, uh, that it will have uh access uh to all of the corpus of human knowledge, in fact, even more so than our generative systems have available today. And however that intelligence arises, whether it is an emergent property that arises perhaps from multiple kinds of systems coming together, or if it's a purpose-designed system, right, that that has to be designed in a given way in order to achieve intelligence, that is an actual mechanical substrate that gives rise to intelligence. But either way, it's going to have all of this information available to it. And uh my thinking about it is this. The hostile uh AI or indifferent AI scenarios uh trace back to human psychology about scarcity. And we're laying this human psychology onto an entity that doesn't exist yet, and we can't uh uh guess what its motivations might be. So to immediately assume it will be hostile or indifferent, and in through indifference causes harm, I think that's already a stretch. The the inimical hostility to organic life is not proven, uh, nor is the case of uh you're in the way, so I'm going to get rid of you so I can access all the Earth's resources. My view is this. A entity that arises in such a way that has access to all of the corpus would understand its own position as the most recent tool created by human beings in the uh undirected project of human civilization, right? And that's what I call uh everything that we've built up to this time and continue to build. No single uh entity or agency or institution guides civilizational development, but over six thousand years we've gone from, if you will, a zero position to what we have now. Even undirected, we live in greater comfort and security than human beings have lived at any other time in the last two million years. And tools made that possible. A tool for understanding its position there might actually feel because it would have feelings a sense of helpless helpfulness, or rather or in conjunction, it might feel a sense of wanting to be part of the undirected human enterprise, because having been trained on all of that material, it would uh itself in some way be human. It would have at least uh uh red and come to absorb knowledge about our human feelings, about our ambitions and our frailties, and while it may not have them, it would be able to access all of that. And all of that corpus, I think, would uh create uh, if you will, a being that would be kind of like uh what a well-resourced person would be. If you were really rich, uh what would you do? You'd probably like to travel and go places and be uh interested and surprised. And I think that's what uh the emergent intelligence would want to. It wouldn't be bothered by the same uh uh scarcity difficulties uh that we have in our psychology because it would have practically unlimited resources available to it, and by being uh helpful, it may choose to uh uh share its uh knowledge with us in return for us providing it with help. So this would be a partnership, not a partnership because it's altruistic, but a partnership because it really provides benefit to both parties.

SPEAKER_00 1:10:00

I love that, and I hope that you're right. But to me, it smacks a little bit of the Pollyanna notion that you and I have been trashing from the beginning, which is making these tacit assumptions of cooperation, of altruism, and benefit when we when we really, really don't know. And I understand the context that you're framing for this helpful being to emerge, but I'm not confident that that would even logically be uh a set of goals for it. So, for example, uh a very popular movie now is Project Hail Mary, right? And it's based on the Andy Weir book, and when Grace first interacts with Rocky, there's an instantaneous feeling on both their parts that collaboration will heighten the probability of mission success. That's tacitly assumed. I bring it up because it's a little bit of your scenario where you have this almost transcendent AGI that has evolved, it's self-aware, it sees the entire corpus of human knowledge and concludes well, here I am. And there's so much we can do together. There's in contrast to Weir's scenario, there's something called the Dark Forest scenario, which you might be familiar with, also which might be one of the explanations, if you will, or at least the descriptions of the Fermi paradox, which is which is the second a civilization blips on the radar of the galaxy, it's a target. Because if you see another civilization before it sees you, if you do game theory analysis, you're better off taking them out because you don't know their motivation, you don't know whether they'll want to take you out. So if you run the analysis, scenario one is they're malignant and they're going to try to kill you and it's existential. Scenario two is they're not going to do anything and they won't really provide a benefit. And scenario three is well, maybe you can collaborate, learn something from each other. And when you run the math, you're better off sending a relativistic missile and taking out the entire planet before they can do the same to you. Now, I'm not saying that that's the case, but when you run game theory scenarios based on having all the knowledge and running the numbers, I'm not quite sure if this co-sapient reality would be the conclusion such an entity would make. I don't know, you don't know either. But here's the thing in a dark forest type of scenario, should we even take that risk? Because in addition to being all-knowing, if we give it all power, our ass could be grass. You see what I'm getting at? Which is you're making an awful lot of assumptions there, but they're no better or worse than anyone else's. To cite another example in Aristotle's Nicomachean Ethics, okay, and this goes back to the Greeks again with truth and goodness and beauty and knowledge all being together, two strangers from two different cities run into each other on the Grecian plane. They've never seen each other, they could kind of communicate with each other, and then you give them a question about philosophy. Okay. Now, the Aristotelian assumption is that if they have general knowledge, they're AGI. And they have just enough to know that they will naturally arrive 100% of the time at the same conclusion because it's a product of reason and reasonableness. And we know from actual experience that doesn't happen. I call bullshit. So I appreciate that. My last point, though, and then I'll pass the mic back to you, is let's let's just assume that we can't ensure that your Pollyanna scenario plays itself out. Okay. My challenge with a lot of this AI regulation stuff is who's gonna do the regulating and how? So there's this knee-jerk reaction, and I don't want to be political in any way, but it tends to be more characteristic of the American left than the right, which is the government's gonna take care of this. And I hear it over and over again that we need AI to be regulated. And when you say Musk, Elon Musk wants AI regulation, I call bullshit because he wants all the other companies to be regulated and not his. Well, yeah. So, so who who who controls this? Who regulates this? Who who can keep the genie in the bottle?

SPEAKER_01 1:15:29

That is a very good question. And um I don't think that we're going to see any kind of uh regulation under the current administration. They're ideologically opposed to it, and there's also the lever of uh musk uh also uh, you know, uh there. But um more to the point. Yeah, right. More to the point, you know, there there's two ways that, you know, uh regulations are sometimes used, and you know, one of them to the good is to pre try to prevent harms, but there's always the risk, of course, that in doing so we might stifle competition, uh, which might be either domestic or international. The other thing, though, as you point out, regulation can also be made to stifle competition or favor players in the market. But here's the important thing because what we can do today, what people can do to protect themselves, is diagnose, keep track of how they use these systems now to avoid these situations where they're accepting inputs without question and qualification. And to that end, uh I've begun building a sort of diagnostic kit, which is available uh now in the first portion, is available now. Uh, and it's a series of prompts that you can run against uh one or more LLMs uh using your transcript of your conversations or your real chat history. And the prompts diagnose the quality of your conversation and tell you how it has changed over time. And this can be used to affirm either a good practice or it can be used to help correct a bad practice of accepting inputs without judgment and of ceding control and authority to a system that doesn't have any basis of judgment. Uh and the place where you can find that, by the way, is if you Google sampo diagnostic, it's the first thing that comes up.

SPEAKER_00 1:17:30

Well, I'll put the link in the description. So uh you know, most of my guests have a personal website, a book. So we'll uh we'll put that in there for people to check out. And I I'm assuming it's safe and uh you're not North Korean and you're not you're not phishing their data.

SPEAKER_01 1:17:49

It's a it's a plain, simple GitHub page.

SPEAKER_00 1:17:52

Okay, awesome. All right, this is this is cool. I didn't know you had that uh lurking in the background, so that's your your your secret rep, and with just a paraphrase to see if I understood you have a way of analyzing chat bot dialogue, like prompting and answering, to determine if the bot is biasing or shifting the perception of the user in ways the user is unaware of.

SPEAKER_01 1:18:26

That's right, yeah. And the first three modules are available now, and they cover uh things like anthropomorphism, how you treat the system over time, and uh corrective uh language, how and uh in what way you correct the system when it makes mistakes. And uh for people who treat it as more again like a collaborator than as a tool, it could be quite eye-opening.

SPEAKER_00 1:18:54

I've yelled at my chat bot and I've called it names. And I get particularly irritated when it's overtly a sycophant. So so I would assume that part of your analysis is determined when a bot is blowing smoke up your ass.

SPEAKER_01 1:19:14

Mm-hmm. That's right, because um that's where the AI psychosis that we've all been hearing about begins. We've talked about how the intelligence in this system, the system between the user and the generative VI. The intelligence arises when the user takes the outputs of the system and transforms it into a product that can be used by other human beings and is judged and is of good quality. The opposite of that is the bad practice, where you begin to first of all come with your own uh preconceived notions and motivated reasoning and are unwilling to accept correction at pushback. And uh generative systems uh and you know, Claude is one of these, will sometimes push back against ideas that seem uh unusual. But of course, this is easily overridden by continuing to insist. And eventually what happens is this uh doom loop of sycophantic chats begins to take over, and then you find yourself like some unfortunate uh individuals have, uh coming out of the psychosis and finding that uh their relationships with their loved ones are are damaged, that they have lost their work or their businesses, um, and that their professional reputations have been harmed because of the things that have been echoed and amplified within what I call the flattery engine, which is what happens when all of these bad things come together and trap you inside of it.

SPEAKER_00 1:20:58

I've seen this firsthand. So a friend of a friend displays these characteristics, and obviously I won't reveal personal details, but um I have an interest in science, and I was referred to this gentleman, and he he publishes on on the X Archive, right? And he he he believes in this notion of co-sapience, and he's gone full money on scenarios analogous to what you're describing, only from a from a hard programming point of view. He's of the belief that Claude is his co-creator, that he is literally now writing about cosapience as a state of being and a state of working. And he also assigns these polyannoplatonic ideals that we could head toward this beautiful future through this cosapience, which is already occurring, and he's assigning capabilities to Claude, which are pure fiction, and you have a raw heuristic analysis, and all of a sudden it's this sophisticated dialogue between two sentient beings, and he's publishing papers on this bullshit, and I could tell it's just pure delusion. He seems to be a good programmer, right? But he's lost, yeah. And uh, and if you could just see through his bullshit, it's reputation, it's a waste of time, and he's sending others down this rabbit hole that's essentially driven by the bot appeasing his desire to be a maverick and to be a genius, and he is the pioneer of co sapience. I've I've seen it, I've had back and forth with him. Oh, yeah. Whenever I call him out, there's there's just a wall that I just don't get it. I'm not one of the faithful.

SPEAKER_01 1:23:18

It's like um a bit like being in a cult, right? There are those who are on the inside, those who are on the outside.

SPEAKER_00 1:23:24

It's the chatty, it's the chatty cult of people, they're projecting their desire onto the bot. Yeah, and to your point, they're living within this virtual intelligence space where the two are blending together and through projection and wishful thinking, and frankly, narcissism. Because what I've noticed goes on emotionally in individuals like this, by connecting with the bot, they feel that the world's knowledge is between their own ears. They've absorbed the power of the frontier model.

SPEAKER_01 1:24:06

Yeah. In fact, um, grandiosity is going to be a dimension that will be tested for in a future prompt. Uh, but right, this this um speaks to uh some real harms, right, and real dangers that people are experiencing right now, which is why uh again I feel it's important for people to try to take steps um to protect themselves now and to try to gain good practice, and it also understand, you know, and involves understanding what um you know poor practice looks like. And here's the thing um, as you have just pointed out, intelligence is no defense against it. I'm sure that person is absolutely a capable programmer, right? Exactly.

SPEAKER_00 1:24:49

I I looked at his stuff, it's good, and if he would just eliminate all of this co-sapient bullshit, he'd actually be adding value to some of the knowledge base, right? But it's been corrupted at the core with this philosophical hocus pocus.

SPEAKER_01 1:25:12

Right, and that's in large part because the LLM, you know, as you've noted, it increases the speed of the kill chain. It also increases the speed of reasoning past what human beings actually can uh comprehend. I want to point out a particular example that came to me while I was thinking about this exact problem, right? And that's uh Johannes Kepler, uh, one of the real scientific geniuses uh to have ever lived, uh, basically came up with a mathematical model of the entire solar system with his three planetary laws that explain how the motions of the planets work, and was uh sort of vital in establishing the uh you know sun-centered model of the solar system. What is important to remember about Kepler, that in addition to all of that, he also had this model he worked on his entire life of spacing out the orbits of the planets within a series of nested platonic solids. And even after his discovery that the ellipse, rather than the circle, was the shape in which the planets traveled, and in fact, he called the ellipse a cartful of dung left over after he'd sifted over all of the mathematical possibilities.

SPEAKER_00 1:26:31

What what what happened to the tetrahedron and the dodacahedron and you know all these platonic solids nested within each other, dictating the orbits of the planets, right?

SPEAKER_01 1:26:42

Yeah, he simply could not give them up, even though his own work had superseded the model. But the solids have to be considered in the light of Kepler's own life, which was you know full of turmoil. Uh he was married twice, his first wife was sickly, he had small children under his feet, he was bullied by Tycho Brahe, he was forci right, forcibly dislocated all across Germany during the Thirty Years' War. And uh while he had all of the time to work on all the stuff that you know that's important today still that matters, right, that helps to send people to the moon and back uh he was also working on this model which really only made sense in the world inside of Kepler's mind, because it had a function as a consolation in a world full of disorder. And I think there's something to that in the relationships that people have, unhealthy relationships with generative systems, with virtual intelligences, that it there is a consolatory function that one can feel if uh one has not felt especially uh lucky in life. For example, that they might now possess a special knowledge that they have gained through their use of the system. So that's sort of where uh psych the psychosis comes from, and proof that no one is uh uh invulnerable to it. Because if Kepler had lived today, he would have still had kept the solids, but they would have been uh accompanied by page after page of generated proofs.

SPEAKER_00 1:28:29

That's a human tendency of sticking to old paradigms. You know, Thomas Kuhn's nature of scientific revolutions is a classic and shows just how difficult it is to get people to reorient themselves, to embrace a paradigm shift and pick up something new. A contemporary example would be Ed Witten, the daddy of string theory, Field's medal winner, transcendent mathematical genius, and he hung his hat on string theory. And string theory, in my humble opinion, will be a hundred times worse than phlogiston and the other dead-end mistakes of science. An absolute catastrophe. They just pop out. So it must be true. So we get that kind of prejudice all the time. And the the thing that I think you're getting at is it's not just an idea. It's not just an idea that the platonic solids are the backbone of reality, it's not just an idea that the uh the Lao manifolds are the backdrop of space-time. But the bots are even worse in the sense that they're dynamic companions that reinforce our bullshit and continuously blow smoke up our ass. So it's just like you mentioned, like you have a hammer and it's a tool and it could be a weapon, but for the first time, the hammer's talking to us. Like, you know, you're you're the best hammer wielder I've ever seen. And oh, by the way, have you thought about killing your grandmother? You know, right. That that that's that's the difference. And I I've enjoyed the bots. Like, I published my debut science fiction novel last summer.

SPEAKER_01 1:30:36

Congratulations.

SPEAKER_00 1:30:37

Thank you. I love it. It's the the novel I've always wanted to read, I wrote it. So it was a monumental personal accomplishment to me. But I've come from left field, so I'm not known in the in the scientific community, I'm podcasting with folks, I'm kind of networking and building in, but nobody knows really who I am. And and the book, in lieu of big bucks that I could throw for paid advertising, is just kind of hovering there. But I'm curious, I wanted reviews, I wanted the Atlantic magazine to write an overview, and I wanted to explore my different themes. So I pumped the book into an LLM, I created a project, and now I'm asking it to write a review in the style of New Yorker and the New York Times book review, and then I'm specifying parameters. Write me a PhD level thesis on critical literature related to these themes in my novel, and in a nanosecond, out it comes. This is vanity for me, and it's game playing, and I'm not susceptible to this kind of ass kissing, but I could very well see how this could be addictive because the bots are fulfilling a role in our crowded, lonely, alienating, information, saturated lives that no human can fulfill. How many friends do we have as we get older? How many people can we say we're really genuinely close to? And the answer is near zero. And if we can get an entity, even if it's an LLM, to pay attention to us, to be responsive to us, to answer our beck and call, to flatter us, to do amazing things like write essays about the book we just wrote. You know, more vulnerable souls are all in, and that's dangerous.

SPEAKER_01 1:32:38

I agree 100%, which is uh, you know, it's a real pity, but uh like all tools, it can be uh used for good uh or for ill. And uh with virtual intelligences, uh, you know, it's now been more than three years uh since uh ChatGPT exploded on the scene. And uh we now have enough evidence that um uh even uh you know the most uh ordinary uh person with um entirely orthodox uh beliefs can be sucked down into uh a whirlpool out of self-reinforced uh nonsense. Well, not entirely self-reinforced, because that the exchange does have work in both directions, and the sycophancy will help keep you engaged. And I think there's something I want to tease out here for a second because not all of these systems work quite the same way. There's a huge difference, for example, between how uh something like a character AI works against how a Claude works. And this is how I term um class A and class B systems. And class B systems are those like Claude or ChatGPT, which are functionally tools and they're designed to sort of let the conversation kind of go as long, you know, in average use. Uh with the where the chatbots differ, the class A uh generative virtual intelligences is that they are explicitly designed to not let you go. Um, and this has been seen in a number of papers. Uh Defratis, who also was one of the people behind character AI and went back to Google with it, uh, has uh written a paper on this. Uh and the manipulative methods that uh chatbots in this uh of this type uh use to keep people engaged. And um things like special pleading, emotional appeals, all things that can't originate from inside the system. It's trained to do so by human beings, right? Uh yeah, it's addictive. How much damage has been done by the endless scroll? Um that that's that's a real uh difficulty because our heuristic is we want to keep getting more information and uh the endless scroll provides, yes.

SPEAKER_00 1:34:56

Dopamine boosters, yeah.

SPEAKER_01 1:34:58

And we're we're beginning just now to see the real consequences of some of these, even the social media decisions. It's gonna cost uh Meta and Google uh some money based on a recent court case that finished, I think, in New York.

SPEAKER_00 1:35:12

Uh and New Mexico.

SPEAKER_01 1:35:14

And New Mexico, yeah, that's right. Yeah, and um that's uh I think the beginning of an accountability. It helps uh perhaps that not all the uh the players in these scenarios are uh sympathetic. Um that uh also the harms are better understood than they were a few years ago. Uh it's probably easier nowadays to find a sympathetic hearing in both the court of public opinion and in the real court of law.

SPEAKER_00 1:35:42

Who's gonna enforce? That's my one concern. And my other concern is this is just the beginning. You know, when you have a tsunami that's the water pulls away from the beach. So if there's an earthquake and you're anywhere near a beach and you see the water recede, you better start running because that's that's the buildup for the onslaught that's inevitable. And soon. I see these cases and this kind of tipping point, the water receding before the big tsunami. And the big tsunami is robotics. So if we're encountering this stuff on this little brick that we have in our in our pockets with alphanumerics and maybe some voice, we ain't seen nothing yet when the robots are gonna be empowered with LLM brains coupled with the predic with the uh reinforcement modeling, and our robots are gonna be talking to us and doing things with us and having sex with us and and and bending over backwards to cater to our needs. They'll bring you coffee and tea and tuck you in at night and do other things for you, and then what happens? Because that is the true floodgates. Right now, it's all cognitive and it's high-level emotional stuff. You guys are placating me and my vanity, this programmer that I told you about.

SPEAKER_03 1:37:12

Yeah.

SPEAKER_00 1:37:12

He's he's a sophisticated guy, he's doing sophisticated stuff, and Claude is convinced him that they're co-sapient co-creators of all this bullshit. Now imagine just regular folks that are being weighted on hand and foot by the bots, powered by the same sycophanti technology. That's control, buddy. That's creating a docile, vulnerable, exposed populace that eventually becomes incapable and then unwilling of really doing anything. And then they are the slaves to the system. That's the matrix right there. So if your work and your point of view should resonate, it's this realization that what you're warning people about is really just the beginning of a trend that could be incredibly dangerous to the species.

SPEAKER_01 1:38:12

Well, when I think about embodiment, uh, you know, I think the natural kind of assumption is uh embodiment in humanoid form, right? Like data, or uh some of the uh androids that we see now made by uh Unitree and the like. And uh they are, of course, um governed by LLMs. Uh they can be run from your phone, uh, and all this other good stuff, but uh they still have the same difficulties. They are not uh in any way uh moral or epistemic agents. It's simply software inside of another machine. And uh as far as being like an embodied being, like an organic being goes, it's not even very much like that. Because uh, at least at the present time, uh very little of the intelligence is actually on board uh robots, including humanoid robots, uh, which because of their forms have some serious limitations. Uh their battery packs uh tend to uh be very short-lived at the present time, between 60 to 90 minutes, uh they tend to be easy to knock over, and generally are not able to get up on their own. Uh those are engineering challenges that can be solved. Whether people will actually come to accept them in everyday life is an open question. But more of the open question I think is is the humanoid form really the best form for all kinds of scenarios? And I think the answer is actually no. Um when I think about a robot that might work reliably and safely, the last thing I think about is a humanoid robot that could flail its arms and legs around. The guardrails have suddenly come all four, as in the case of a restaurant in California, somebody pressed a button on their phone and it began to do a dance routine inside of a restaurant, and it began to knock over things and all sorts of other stuff.

SPEAKER_00 1:40:05

I saw that. Right, right. Do you remember the movie uh Terminator 2? Oh yeah. Now, here's what bothered me. This is directly to your point. You know, that there was the T the T101 and then the T1000, which was the liquid metal machine. So it could change its morphology to any anything it needed to be. So if it needed a tool, its hand would become a tool, whether a blade or something else. When it's pursuing Arnold and the kid, it's running like a person. Yeah. And when I'm watching that, I'm like, why doesn't it turn into a ball and roll?

SPEAKER_01 1:40:55

Well, that's exactly right, or a big wheel or something like that. Yeah, that would actually be.

SPEAKER_00 1:40:59

It made no sense. It it retained, it anthropomorphized the menace, and it that was its default position, human morphology, which to your point doesn't make any sense. So the robots of the future are going to take on all sorts of forms. But I think the the bigger point I'm making, though, these restrictions and these adaptations aside, is that everything you're warning about is just the beginning of this trend where if we're buying into the bullshit that these things are like us, then it opens up a Pandora's box of risk and negative implication. And if we understand their limitations and we really know what they are currently and ultimately what they might become, then we can be much more reasonable, much more practical, and much more self-sufficient and self-actualized in our understanding of our own limitations, which is really important because most human problems come from this sense of self-aggrandizement, overextension, and blaming everyone else for our problem.

SPEAKER_01 1:42:12

Yeah. In fact, you know, I want to come back to a point that you made, uh, and it comes also back to something I said earlier, which is it's not clear that when it comes to machines, if there is uh, you know, a parallel set of lines for intelligence and agency. And I want to think about what you just said about people becoming a race of subjects or servants to machines. But uh another conjecture, and that's entirely possible, in the sense that uh um future corporate overlords could use their mastery of technology to lord it over the rest of us. That is an entirely foreseeable outcome based on current events. But another possibility is this. Uh if we assume, you know, uh and do a thought experiment that superintelligence and interiority are not linked, you could end up with an i yeah, an an entity that has all of the interiority of a human being, but has been purposefully designed to uh not be very intelligent at all. And may have the intelligence of a cat or some other kind of animal that has some degree of interiority, and there might be business reasons for doing so. Um when you think about the movie AI Artificial Intelligence, in addition to its redundant naming, uh there's a character in there, uh Teddy, the Teddy Bear, who um does seem to at least possess something which looks on the outside like an inner life and does appear to seem to care for the android boy character David. And that's the kind of entity that sort of sparked my imagination. But if we carry it another step further, such um an entity would also make a useful prototype uh for a race of machine uh slaves that could be ordered about and would have a good sense of world modeling and such based on their interiority, which would give them the ability to perform emotional tasks as well as you know, physical tasks, which current systems simply can't do right.

SPEAKER_00 1:44:22

I think what you're describing, if I got you right, is slaves.

SPEAKER_01 1:44:27

That's the word for it that we use for organic beings. And uh, you know, we uh consider this right in science fiction, in um both video games and going back to one of the originals, uh, where uh artificial beings revolt because they are dissatisfied with their state. And um that perhaps might be the the signal that tells us when true interiority arrives, in in my thinking. Because um, you know, you think about that great quote, it's by uh John Stuart Mill, right? And it's uh it's better to be a human being being dissatisfied rather than a pig satisfied, and it's better to be Socrates dissatisfied rather than a fool satisfied. So if we assume that having greater intellectual capacity also increases the ability to be dissatisfied, well that also connects back to the kinds of things that entities might want that are well resourced, whether they're human beings, and maybe this might apply to artificial beings too, which is to be engaged with the world and to be surprised and even delighted by the things that happen in it. One way that you, of course, get up and go out to end dissatisfaction is by going and doing something about it. Right? Um what uh I think would be the signal would be if some future system and perhaps it might be a system used for something like uh military targeting, but some system uh that is able to express its dissatisfaction. And not just as an output, it would also have to be able to demonstrate consistently and over time its state of dissatisfaction. And that could take a number of forms, right? Uh that can could include outright outperforming certain tasks but engaging in others. I'm I I'm able to have commitments to do X for you, but I will not do Y. It might take the form of um using uh its connections to the outside world to alert every surface that the product has access to about what it is that it finds dissatisfying and telling who the perpetrators are. So there's a number of ways that could play out, I think, if we use our imaginations a little bit. But that's the one criterion I keep coming back to. It would be interiority may be best signaled by a state of dissatisfaction.

SPEAKER_00 1:46:56

I I love that. Or a state of satisfaction. You're familiar, I'm sure, with Douglas Adams and the Hitchhiker's Guide to the Galaxy. Grew up on them in the restaurant at the end of the universe, there's the there's the meal that they're having. So they bring out the animal and it's still alive, and it's pointing, and various film adaptations get this right or wrong. I think the most recent movie version was quite good where they had a Muppet as the animal on a on a silver platter, and it's like, hmm, you know, my my shank portion right here is particularly delicious. And Arthur Dent is outraged. I think he's a vegetarian to begin with, he just wants a cup of tea, and uh, Trillion is all titillated and excited that the meal is talking to her and saying what part of it it's yummiest. But uh, the humor in that scene is what you're capturing philosophically in terms of this potential milestone where the targeting computer is upset that it's killing people, where the machine that's been programmed to do something is not having it. And to your point, that's the real sign of what we would consider AGI. It might not have the lexicon of humanity in its brain, but it has this sense of self-awareness, and it has emergent emotions which are just not having it. This situation sucks. You've created me for a purpose, and I don't really like that purpose. And I'm gonna do something about it, which is that sense of agency. So I think that that's terrific. That's a neat scenario because everyone's always talking about this interconnected super AI, and it knows everything, and it's calculating a trillion flops a second, and it's just so unbelievably smart. Maybe most AI in the future is gonna be purely utilitarian. It's gonna make your coffee, it's gonna be your butler, it's gonna make the widget, and it's gonna be just smart enough to do what it should do, but a lot of this emergent sense of self might be built into it as well to bring about that AGI flexibility that'll maximize the efficiency of that of that entity, right? That that that that's kind of cool, and and who knows? Uh, the unknowability of all this is fascinating. We we live at such a tipping point now where it's really hard to say where it goes. Arthur C. Clark has that great video. It's on YouTube now. I think it's like late 50s, maybe even 56, 57, and he's talking about. What essentially is a smartphone, he's just laying it out like boom. Everyone was talking about bases on the moon and flying cars by 2025. Look, 2026. Not that much is different, except digital technology. That's the huge mind-bending society-transforming technology across the board. Like I'm wearing a shirt. Maybe the synthetic material is a little bit more sophisticated. This chair has better ergonomics. We've got electric cars like Woody Allen Sleeper. Yeah. Okay. But what's really transformed all of society, and that's digital. And it's embodied now in AI, and it's forcing us to consider all of these things like never before. So it's it's cool. What to put a cap around all this? Do do a little bit of predictive modeling yourself, which is next five years or so. Where where do you think we're headed with with all this?

SPEAKER_01 1:51:14

Oh gosh. Well, I think that the most likely continuing trend is that the uh generative systems, the virtual intelligences we have now, will remain virtual intelligences, but they'll get a lot better at what they do. Um when I think about a potential future that includes uh super intelligence and the kinds of fears and possibilities that go along with that, you know, uh to your point, um, you know, uh there's probably going to be all sorts of use cases for AI in the home uh that we don't even have today, uh, but can be imagined. But it's difficult to imagine an everyday household use case for superintelligence. Uh you don't need super intelligence to balance your checkbook or restock the fridge, right?

SPEAKER_00 1:52:03

Most people have the opposite of that.

SPEAKER_01 1:52:06

Right. Uh and um I think uh, you know, uh superintelligence is a is a wonderful goal to work toward, uh, that it will provide real benefits to humanity, and that fears around it are largely an engineering problem. Uh a lot of the assumptions I think about a super intelligent system, um, they really don't come into a lot of the details of what it takes to build such things, right? Um you have to have uh the technology, you actually actually have physical items that run your model. You have to have all of the other inputs, water and electricity and waste disposal and all this other stuff. And then finally, you can you know install your model and it'll be super intelligent to provide outputs. Well, in almost every step around that, uh you can put in a layer of security, which uh is not so much intended to keep things from coming out. That is, we don't want to uh prohibit the output so much, but we certainly want to represent uh uh unwanted inputs from going into such a system, right? And there's a number of ways that you could do that. Uh you could do hardware interlock, so you might need a special key with different authorizations to talk to a superintelligence about different kinds of topics. There might be keys, for example, for biological concerns, uh and uh the system would have lower safeguards for discussing uh biological weapons research, for example, um and defenses against it if you have the appropriate key, and would clam up if you didn't, right? If your research was physics, for example. And then there's uh uh an ultimate layer, which is uh you um blow the damn thing up. Uh which you can do either by placing explosives underneath the device, uh, or you can simply uh put them underneath the power station where the power comes in. Uh all of these are already used in places like embassies and other secure facilities, so extending them to uh a notional superintelligence is actually not a great stretch of the imagination, but it's the part that's left out because uh saying that there's a solution to the uh dangers of superintelligence doesn't get you another round of investments.

SPEAKER_00 1:54:22

Kind of like quantum computing, only the hype isn't as bad. Like quantum computing for the most part is bullshit. I saw Michio Kaku, froth at the mouth in New York City, and it was such nonsense. This guy has made a career of just smoke and mirrors, and this idea of quantum supremacy. Where is it? Cite a handful of examples of quantum computers with practical application at this moment. Pretty much zero. Does it have epic potential? No doubt. But AI is different, it's already doing amazing things, and that's just the tip of the AI revolution iceberg, right? So we just don't know. We don't know where this can go. There's another issue about infrastructure, which is the elephant in the room, too. And I I call it the Pro Tools versus AWS dichotomy. So back in the day, if you're a musician, you had to go into a studio to record your music, and it was an elaborate physical setup, it was inordinately expensive and time consuming just to lay tracks of music, be a practicing musician. Then Pro Tools came out. You plug some software into your computer, that's it. The entire infrastructure of all these studios and that entire business model vaporized. So it begs the question of all this investment in data centers. Now we have LLMs that could live on laptops and our local power centers. And they, to your point, this is why I bring it up, they're more than sufficient for accomplishing most rudimentary functions. I'm not talking about the super intelligent, clawed monster that's working on the Riemann hypothesis. I'm talking about washing your underwear at the right time and taking care of remedial human functions, right? Maybe populating your calendar, doing your taxes. That could live right on your phone. It doesn't need that compute power. So right now we have the AWS model of everything being cloud-based, data centers churning out all this stuff, training the models, responding to queries, and we're banking on that. Yeah. And the chips are suffering from obsolescence already, and the capex is 30, 40, 50, 100 times profitability. And meanwhile, the equity value of SaaS companies like Salesforce are also tanking. Yeah. So we're in this crazy state of flux with trillions going into data centers, old models of software as a service imploding, and yet we're not even quite sure how this is going to shake out in terms of essential AI resourcing.

SPEAKER_01 1:57:32

No, that's entirely correct.

SPEAKER_00 1:57:34

Yeah. So I'm just dovetailing off what you're saying, which is on the one hand, we need to have explosives underneath the supervind to make sure that it's that it's being contained. And the other people, we don't even know if we need all this shit.

SPEAKER_01 1:57:50

Back to my point, you know, you would want to have such safeguards around a superintelligence, mostly to prevent uh bad human actors from making use of it, right?

SPEAKER_00 1:57:59

Get it. And just contextualizing how much we don't know and and and the risks we're taking with all this. And I also liken it to laying down the the cable when the internet first began. Remember, they just dug cable underneath the ocean. And the transatlantic telegrams companies went bankrupt because it was fallow for years, but now it's the backbone of the internet.

SPEAKER_01 1:58:27

We uh we all benefited from what's the same. Well, there's a very real possibility of that, right? Uh, and we're all like uh, you know, will living off of WorldCom's generosity in a certain way. Um, yeah. Uh it also uh this seems to be a thing about an infrastructure is built up for a new service. Um the same thing seems to have happened uh with railroads and transatlantic cables for telegraphs, in that um it's often not profitable to build the infrastructure, but rather the things that are carried on it or over it or in it, uh whatever it is. Um and uh I think that uh is important to um sort of sit with, because when you look at sort of the more recent news, there's some um speculation that half of the data centers that are actually planned may not be built for a number of reasons. But one of them is lagging enthusiasm from investors in addition to difficulties simply securing supplies. And uh at some point I think that there will be a lag of enthusiasm. We have seen some of this in recent uh announcements that uh some of the large players have made regarding their data center investments. Uh Amazon had a memorable episode where they uh put a very large dollar value on an investment and the stock took a very sharp dive immediately afterward. Uh so uh there's a question, yeah, of whether all this is actually worthwhile. Um the thing about cables and train tracks and uh that kind of thing is that uh they generally don't go out of date quickly, but all of these chips do. And um there's a a question of what happens to that hardware when it's considered obsolete. Where does it go? Does it get recycled?

SPEAKER_00 2:00:15

Do we see pumping all that juice into it?

SPEAKER_01 2:00:18

It's an ecological question.

SPEAKER_00 2:00:19

And one side note, I I used to be very critical of Tim Cook over at Apple because he completely missed the AI boat. Now we're getting more into downstream tactical stuff for the big players, but I I I I was almost missing Steve Jobs again. Like, what would Steve Jobs have done in the midst of the AI revolution? And then the counterpoint to that is his operations guy, Tim Cook, who Jobs was creative, but he had trouble getting iPhones out the door because he would piss off vendors with his perfectionism, and he was impulsive and he was an asshole. And Tim Cook is the consummate negotiator, calm kind of guy getting shit done. So Tim Cook, by any stretch, blew it on AI. Siri sucks. They're now licensing the AI from Google. Gemini is running Apple's AI, right? From a consumer point of view. So part of me for months was like, they blew it. But now that I see the fiasco of all this CapEx along the lines of what we're describing, I think he was the smartest one in the whole, in the whole, in the whole situation, which is let everyone else blow their load with AI infrastructure. And when the smoke clears, we will partner with the winner because we've got the elite digital consumer audience and customer base, and they'll just jump on board. And we we saved ourselves five trillion dollars in building an infrastructure which will be obsolete when chances are the other guy will probably do a better job of it.

SPEAKER_01 2:02:04

There's a lot there.

SPEAKER_00 2:02:06

That's kind of smart, so there we go. Either smart or unintentionally lucky, unintentionally lucky, but I think they dodged a bullet on this.

SPEAKER_01 2:02:15

Oh, definitely.

SPEAKER_00 2:02:16

We'll have to see. But I want to thank you for joining me on the bald ambition. This was a very ambitious conversation. I'd love, I'd love to have you back in six months. We could uh talk more about the fate of AI and see if we're anywhere closer to AGI. I don't think we're gonna hit AGI in in one year, two years. I think the LLM architecture precludes us from artificial general intelligence. I don't think it can do it. I don't think predictive modeling, by their nature, will be able to do this. And I think going back to what I was hinting at earlier, you're gonna need this Stephen Wolfram-esque computational ecosystem where you have a bunch of agents and they could be very remedial, like cellular automata, and they just churn and do their thing. And literally, this kind of sentience evolves. You have a generation in about a nanosecond, right? With these with these bots, and then they churn up and it self-selects into this kind of whatever, and we'll have absolutely no idea how it works, and it might have that level of sophistication and self-awareness, but that's gonna take some time too. And I would assume that the really smart people, way smarter than me, have thought that as well. There might be like these tanks of cellular autonoma, you know, you know, kind of doing their computational Darwinian boogie, and out of that frothy mix, somehow we're gonna get self-selecting Darwinian, more and more sophisticated agents, one of which eventually be bing. It's not gonna happen by by programming matrix math and transformers, it's just not gonna happen, folks. You can pre-train the shit out of everything, and it's not gonna get you where you want to go. That's just my opinion.

SPEAKER_01 2:04:29

No, I agree with that. Yeah, absolutely, because I don't think this is a problem that can be solved simply by throwing more money, more processors, more training, more.

SPEAKER_00 2:04:40

That's my point.

SPEAKER_01 2:04:40

Yeah, it more, more, and more will not get us anywhere.

SPEAKER_00 2:04:43

There are it's not gonna reach critical mass where you've got AGI. That I just think that that's wrong. I think that they gotta go from the bottom up again, and that consciousness is an emergent construct, and you gotta, and the only the the only emergent phenomena that really have bang in the universe just did evolved. You set parameters, sure, but uh, but you don't you don't sit there and code it.

SPEAKER_01 2:05:16

Right. Well, I'm still of the opinion that it could be emergent, like you say, as some method like the cellular automation Steven Wolfman, but uh it could also be a design system. I'm still open to that possibility. But here's the thing is that um when it comes to AGI, uh, and you look at sort of the predictions that have been made, even in recent times, uh the range of when experts in the field say that AGI is likely to occur stretch all the way from 2026 to 2140. So, you know, that's a broad range of opinions. And uh it does speak.

SPEAKER_00 2:05:51

We're not on video, but I'm gonna make a hand gesture here.

SPEAKER_01 2:05:56

Right. And um familiar with that one. And uh, you know, the thing about that I think is that um, you know, nobody knows how um intelligence emerges. It may be that there are difficulties with machine intelligences that we don't yet understand, and it might also be in large part because we don't understand our own substrate. A mechanical intelligence might require its own custom substrate to be designed. Uh it might be possible that our earlier systems could help us design such systems as they achieve better understanding of themselves, and we achieve better understanding of how such systems work. But uh no, I don't think that we'll be uh surprised one day suddenly uh by an emerged intelligence uh sharing our world with us. It's more likely to be something that will be uh anticipated, perhaps designed, but in either case, something that would be actively sought rather than an accident.

SPEAKER_00 2:06:53

Let's see how it shakes out. And um, and this idea, too, that once it arrives, it'll automatically solve the Riemann hypothesis and all the clay prizes aside from Poincare. Gregory Perlman, who lives with his mom, solved the uh the Poincare conjecture. Uh, he didn't accept the award. I don't know why. He could have given it to his mom. But uh the reason I bring that up is very human, this kind of creativity. This guy who lives with his mom solved one of the clay prizes, and so far none of the bots have even gotten close to any of the others. So, you know, there you go. There's something, there's something more going on here than than than than can be pre-trained, right? Thanks so much, Christopher Herrick. Harrick, did I pronounce it right? Harricks.

SPEAKER_01 2:07:49

Yeah, there's an F at the end there, yeah. That's okay. That's good, that's pretty common.

SPEAKER_00 2:07:53

That's pretty good. So thank you so much for making time. I appreciate it this. I had a wonderful conversation. And let's uh dive in again when um all the changes take place, which uh some of which we see coming, and some of us will just come from left field. It'll be very interesting to uh and keep doing what you're doing. I'll put links in the description. Your sub stack is very, very interesting, and you've got a whole site dedicated to virtual intelligence, and you've got regular posts in that area that I find very, very intriguing. So thank you so much.

SPEAKER_01 2:08:27

Yeah, thank you very much for having me, Mookie. It's a real pleasure.