I had the pleasure of attending this event yesterday. Following the news of Musk’s recent IPO, the clear craziness of the level of investment in these AI labs and the shift in the public’s response to the impacts these technologies are having today, reveals this dichotomy: investment is either blind to public perception and outrage, or it is quick to discount the needs of humanity.

I believe their bets on the whims of these tech bros are more about FOMO and earnings than about believing these companies are sustainable. The grift truly runs deep.

Dr. M. Murphy moderated the session. M. Murphy is a technoscience studies scholar who theorizes and researches environmental justice; reproductive justice; Indigenous science, infrastructures, and data studies; racial capitalism; and the Great Lakes.

Alex Hanna is the Director of Research at the Distributed AI Research Institute (DAIR). She also co-authored The AI CON with Emily Bender, which began as part of the Mystery AI Hype Theatre 3000 series, which “tears apart the AI Hype” on their live stream and podcast.

“AI is a Marketing Term”

The ‘boosters’ and ‘doomsters’ exist today and emerge from the impacts that people have personally experienced or knowledge they’ve obtained. In between is a “panic” to get on the bandwagon for fear of being displaced… for businesses, it’s the pressure to keep pace with technology and adopt it quickly… or else.

Even before ChatGPT launched, Bender and Hanna recognized the optimism about AI on Twitter/X. Despite this, there were also a lot of hiccups, flaws, and clear errors that arose from AI. They started to make fun of these flaws:

Some great vids from their podcast:

ChatGPT Awakens

Is AI Art Actually Art?

Uber for Military Surveillance

Can Machines Learn to Behave?

Hanna clarifies that AI is a marketing term that can refer to many different things. She advises that we need to treat each specific automated tool differently, “disaggregate AI” rather than treating it as one intelligence force. She clarifies and provides some differences:

• Automated decision-making systems (ADS) are useful for pre-trial risk assessment, policing, hiring, and automated loan adjudication.

• A superset of those includes classification or determination of whether something fits within a given category. This manifests in recommendation systems that rely on what you engage with and consume on social or retail sites.

• Generative AI - she describes as entities that precede the automation, translating it from one format to another, e.g. letters in an image transcribed into letters, which enable automatic speech recognition.

As Hanna states,

The Forms of Hype

Hanna refers to the pervasive use of anthropomorphic labels like AI tutor, AI secretary—technology that can replace critical occupations or critical parts of social practice. It’s ramping up in the medical field. In academic terms, now common terms like the automated technical researcher or scientist are cropping up. What’s problematic, especially in medical or social services, are the patient interventions that will be displaced.

In addition, the gender-specific labels describing the type of work to be replaced include “secretarial, maternal, AI girlfriends.” The impact includes the justification for AI companionship. She also alludes to Geoff Hinton’s recent remarks that we need to train these technologies to be more maternal or “risk extinction.”

Hinton said this:

In addition, part of the hype is driven by strong market players like Allbirds, a maker of premium canvas sneakers, which went bankrupt, then became SmartBirds, an AI company, and later became a strong arbiter of AI hardware. Starbucks also recently launched their “groundbreaking” AI platform to boost its faltering sales performance. The political economy is enabling these companies to pivot with the promise of better returns. The brand elevation by these companies continues to fuel AI hype.

AI Will Displace Us!

Murphy pointed out:

What that middle ground suggests is a justice-based argument. Hanna says that this AI discussion has shifted the Overton window, where you either think about “AI boosterism” or “AI doomerism.”

Doomerism she defines as existential risk, and robots-will-kill-us-all-type messaging. But it also encompasses death because of job displacement and loss of purpose, as she explains, that this middle of the road often gives us statements like, “AI is just a tool.” She asserts that science or technology scholars will argue that this is rage-baiting, alluding to STS 101: sociotechnical systems emphasize that tech systems must be jointly optimized through collaborative design between the social (people, organizations, and culture) and the machine, software, and technical systems.

These AIs are “not just tools;” they are not neutral but are designed in specific ways to elicit specific outcomes.

Emily Bender in a recent episode said this:

What sometimes gets misinterpreted, as per Hanna, is the notion that she is perceived as an AI extremist. She argues her position is not extreme, but rather “incredibly boring.” She adds that the general public, however, is weighing in on AI:

But the perception of AI extremism reflects the point of view from the “AI boosters,” who see ethics and human-centred experts like Hanna as constantly scrutinizing and calling out the perils of current systems.

This has spilled over to mainstream, where impacts of AI and data usage are having material effects on the land and environment, as Hanna explains,

“For a lot of people, that is the first entryway into talking about AI… it’s a new conversation where we need to start talking about justice and solidarity.”

In a recent Pew Survey, Americans are using chatbots, adopting AI features like summaries and using AI products like smart speakers; however, the views about the technology tell a different story:

• “Americans—including younger adults—are deeply skeptical of AI. “More adults predict that AI will have a negative impact on them and on society. And majorities think AI is advancing too quickly and put their personal information at risk.”

• Usage stats:

  • 38% of employed adults report using chatbots for work; 25% use them daily

  • One in three adults have a smart speaker, and one-in-five have a smart doorbell that uses AI

  • 60% of American adults say they read AI summaries at the top of search results

• Views:

  • “Four-in-ten U.S. adults say AI will have a negative impact on society over the next 20 years. Far fewer believe its impact will be positive.”

  • “While 31% expect AI to have a negative effect on them personally over the next two decades, about 25% believe it will have a positive impact.”

    • “Younger adults have a more bleak outlook, with 48% who say AI’s impact will be negative, with 37% saying it will have negative impacts on them personally”

    

• “Roughly two-thirds of Americans say AI is advancing too quickly”

• “71% predict AI will make their personal information less secure”

Technology is Distributed, Not Centralized

DAIR stands for Distributed AI Research. Timnit Gebru is the founder and a colleague of Alex Hanna from their time together at Google. The organization was formed one year after Gebru was fired for speaking up against racism and sexism at Google.

At DAIR, the mission is to critique existing systems while also considering new technological features. Hanna emphasizes this is about centring people in their own communities. Silicon Valley is the center of tech, but where the important work is happening and where new technological features are being built is not in the “imperial core,” as she points out. “It’s distributed outside” of Silicon Valley.

“Distributed” means having collaborators in different regions, in rural spaces. She adds this is also how the authority is distributed—where the tech visions are happening. Hanna mentions DAIR partner, Professor Ciira wa Maina from the Dedan Kimathi University of Technology in Nyeri, Kenya:

Hanna recalls the news a few years ago about Samasource, and how workers here were getting paid $2 an hour to stress test ChatGPT, and were subject to mental health issues because of the conditions they were exposed to. In a previous post, Karen Hao illustrates the pressures from young data annotators like Alex Cairo who worked at Samasource:

Hanna adds that one of the projects coming out of the Data Labellers Association (DLA) is to enable data workers to engage in their workplaces and inquire about working conditions. Hanna mentions a positive output from DLA was an incredible mental health event to support the staff.

AI’s Impact on Labor is Not so Clean-Cut

Hanna notes the importance of data work as a critical component in thinking about what the supply chain of AI is or what it means, adding,

This is one element of labor in the supply chain. On the other side, Hanna points to the ways in which there’s a vision of fully automating certain jobs away, and she references the fortuitous Altman and Amodei’s statements that walked back these predictions in advance of their respective IPOs.

Hanna is clear that layoffs as a direct result of AI mean either freeing up capital assets to invest in these technologies or adopting a fire-first, try-to-make-it-work-later approach.

Oracle is an example of laying off 30,000 employees to pay for data centers.

Snowflake, a Bay Area company, gave its employees an AI mandate to use these technologies or get fired. Eventually, they laid off over 70 staff. Atlassian laid off 10% of its workforce in March, and Block laid off 40% in February — both attributed their cuts to AI.

Hanna argues that this is a common tactic of work intensification, under the guise of a technology that can do it all, adding that resistance was loudest among labor and labor organizations, which screamed for guardrails. Hanna is adamant this doesn’t mean anything, stating

Governance is Slow, but Data Centers May be the Catalyst for Action

Smart regulation and guardrails continue to lag. In many cases, they don’t have the necessary deterrents or enforceable fines to curtail the risks posed by these technologies.

Hanna has pointed to how it’s playing out:

• Bernie Sanders has proposed an AI sovereign wealth fund where citizens get a piece of the pie.

• Resources and case studies from unions who have seen the AI threats to their control in the workplace.

• The labour movement has been quite conflicted about data centers. Parts of the labor movement see data centers as a labor opportunity: gas construction project is offering a way for trades, building trades, associated electricians, pipe fitters, security etc. As per Hanna, this is a "people wedge. There is this reconfiguring of the alliance, and it’s making us rethink what solidarity means and how we align with different people.”

• Politically, there are conservatives up in arms about data centers. Hanna says, “They understand the carbon costs and the impending impact on electricity and hydro costs, and what the impending noise and data centre pollution will do.”

Hanna points out that there are large organizing entities: legal, environmental, data justice, data governance, AI militarization, AI weaponization. But by far, the strongest push against AI has been the movements against data centers. There are many people organizing around this one issue. For the funders of these data centers, the location opportunities are not because they would make ideal locations, but, as per Hanna, because of the regulatory gaps that minimize their business risks, adding,

This is now the common mantra in favor of data center development. Hanna adds that this narrative now allows the defense industry to enter into this space in a significant way,

Alliances are clearly shifting, and a more informed mainstream recognizes what it means to have AI embedded in our futures.

Hanna pointed out that in New York state, coalition that included environmentalists, labor, and Indigenous organizations were effective in stopping the development of a data center site. And that has coalesced into a state-wide moratorium against data centers in NYC. Hanna proclaimed,

What Murphy pointed out, however, is that, for the first time in Canada, we have military funding for research and technology development. In the U.S., there has been a long-standing organization against technology and the military. The nascency of this in Canada means we don’t have the same levels of organizing intensity, hence influence, as the U.S.

Back to Regulation?

Apart from coalition-building, regulation needs to become more effective. In the U.S., this is fragmented. The federal laws on AI do little to hold tech companies accountable. In fact, it shields them. According to Hanna, the EU regulation has limitations:

What Hanna concludes is that unions and labor organizations are very strong category structures that can push back in meaningful ways. Her experience with some of these institutions has revealed their ability to embed and convince people to change practices, and from the community, develop community governance practices around data, both infrastructure and technology.

This moment in time, this tension about a technology that yields both political, military as well as people-led and labor-led organizations is a good thing. It means the messages that Alex Hanna, Timnit Gebru, Emily Bender and DAIR are penetrating through.

In their book, Hanna and Bender’s message is that this AI hype obscures real-world harms. In their book, they write,

“Not only does the rhetoric around p(doom) distract from actual harms, but the very terminology of “artificial intelligence” impedes clear understanding of the technologies in question, what they can and should be used for, and how to evaluate them.”

After all, if we are to believe that AI is inevitable, then shouldn’t we control how it’s used, govern how it’s built, and ensure that all humans remain its central beneficiaries?

Share

These experts in their field have experienced mainstream pushback against current general-purpose technologies as the path towards Artificial General Intelligence (AGI). We discuss their experiences in these debates and their perspectives on the unshakeable stances taken by those who oppose their views.

I spoke with Mounir Shita, a 30-year veteran AGI researcher and founder of EraNova Global, a physics-first program of work on the nature of intelligence. His “Theory of General Intelligence” models a goal as a physically realizable future state. I’ve known Mounir for over a decade, and his early research in AGI—at the time, they just called it AI—does not jibe with current AI systems.

You can find the article for this Podcast here:

In our series about the AI Critic, I interview long-standing researchers and practitioners in the AI sector—voices who have worked with artificial intelligence in the advent of machine learning and its evolution to deep learning, which has laid the foundation for current large language models.

These experts in their field have experienced mainstream pushback against current general-purpose technologies as the path towards Artificial General Intelligence (AGI). We discuss their experiences in these debates and their perspectives on the unshakeable stances taken by those who oppose their views.

I spoke with Mounir Shita, a 30-year veteran AGI researcher and founder of EraNova Global, a physics-first program of work on the nature of intelligence. His “Theory of General Intelligence” models a goal as a physically realizable future state. I’ve known Mounir for over a decade, and his early research in AGI—at the time, they just called it AI—does not jibe with current AI systems.

I interviewed Mounir for an earlier article:

Hessie Jones Substack

LLMs Are Short-Circuiting. Is It Time To Redefine Intelligence?

It’s time to move past large language models and create a new narrative. The hiccups we’ve experienced from large language models and generative AI — a still-novel technology — since its inception a few years ago is evidence that this may not be the pathway towa…

Read more

9 months ago · 12 likes · 6 comments · Hessie Jones

He defines intelligence as the capacity to steer “causal chains toward the state under the constraints of physics.”

He said that intelligence is inherent in the structure of the universe, including its state, goals, dynamics, time, and causality and adds,

In my recent conversation with Mounir Shita, he describes an event he was planning in Portland in December, 2025. The purpose of the event was to explore how economics, law, and governance adapt once we reach AGI. As Shita clarifies,

The stated agenda for the conference had quickly gone off the rails. Shita was taken off guard because no one wanted to talk about AGI outside the realm of large language models (LLMs). When the conversation shifted to economics, law, and governance, it remained within the context of LLMs. He felt he had to step in from an engineering perspective to ask the critical question, ‘What is intelligence?’ He stated that engineering has always been about implementing science; however, in the realm of AGI, where there is no definitive understanding and where LLMs use statistics to guess the relationship between words, his words were abjectly rejected, as he explains,

The crowd size was approximately 50 people; two audience members did not believe LLMs were AGI in any way, shape, or form. Shita explained he knew these individuals previously, but, overall, the audience was in the camp that AGI has indeed arrived in the form of large language models.

Every time Shita attempted to frame AGI as a potential causal algorithm capable of controlling how the future evolved as we pursued our goals, the conversation would inadvertently return to Claude and ChatGPT. Every discussion about AGI’s impact on future society somehow reverted to how OpenAI and Anthropic will adapt their current technologies for this. However, he explains the direct reactions came after the event, in one-on-one conversations:

Completely baffled by these senseless remarks, he realized that the prevailing belief in the room was outside of reality. Shita remarked that the two individuals who did not believe LLMs were AGI stayed silent. They did not attempt to ruffle the overriding hardened belief that existed in that room, as Shita explains,

The Audience of AI Practitioners who Believe AGI has Arrived

Shita explained that this event included people from the AI space. They were tinkerers, AI engineers, and a few tech executives. Outside of the panellists, these people, Shita notes, should know how things work under the hood.

What is clear is that the largest voices, Jensen Huang, Sam Altman, Elon Musk, and Andreessen Horowitz, have the marketing megaphone that is proclaiming LLMs as the path towards AGI. Their message is cutting through to the mainstream. He explains Musk’s stance,

He explains that the whole AI safety debate, when it comes to AGI, is equivalent to what Andrew Ng said in 2020 when he said

Shita says this is a perfect analogy: Elon Musk created this fear, causing many people to start thinking about AGI safety when they have absolutely NO clue what it is. He adds that during one of the funding rounds, when there were rumours it might not close, Sam Altman released an essay about how AGI would make the world and life on Earth great. He was then able to secure the capital he needed.

Shita emphasizes that hope is another strong emotional driver, and he questions Altman, who has alternative incentives in preaching this false narrative.

Free Will? Consciousness? Impossible Within Current Systems

Being sold a lie and getting away with it? There are numerous cases in tech history.

For Elizabeth Holmes of Theranos, justice prevailed. Sam Bankman-Fried, FTX Founder, was sentenced to 25 years for fraud, but for companies like WeWork, investors lost almost 47 billion in market value, with Softbank, the lead investor, alone absorbing almost $19 billion in losses. Adam Neuman, unlike Holmes, walked away with a sweet $1.7 billion in cash and assets.

Shita contends that he’s never heard of Demis Hassabis, CEO and co-founder of DeepMind, selling LLMs as AGI. Dario Amodei has come close, but he’s never directly made that connection. But he compares Amodei to Altman, the LLM vs. non-LLM crowd in the context of AGI and says,

Geoffrey Hinton’s deep learning models were the basis of generative AI, and as Shita elaborates,

Mainstream AI is like a battery. There have been evolutions but no breakthroughs over the last several decades. Generative AI is an evolution of neural networks, even before deep learning, but there’s no real breakthrough. He discusses his recent conversation with Geoff Hinton,

I asked Mounir about Eryk Salvaggio’s article, The Computer Science Fetish, a similar experience Mounir Shita confronted at this Portland conference. When AI critics scrutinize the outputs or point out the risks or harms, people don’t want to hear it. Does dismissing them help justify their acceptance of the technology? Shita goes a bit further and says,

Why will AGI take our jobs? What Shita is implying is what value does AGI extract from current human-led skill sets, knowledge, and experience that renders the human worker irrelevant? My question to those who verbalize this risk of being supplanted by these systems is, ‘Why do you think this is an eventuality? Why would you embrace a system that will take the very thing from you — allows you to put food on the table, save for a house, save for your retirement — that allows you to live? Does it make sense to accept a fate that has yet to arrive, or can we do something about it today?

Shita explains that what Big Tech in the LLM space are pushing is a narrative that in an AI world, universal basic income (UBI) will give humans the freedom to do other things, but because the machine is conscious, it will know what we want as humans.

He alludes to the nonfiction book ‘Our Final Invention: Artificial Intelligence and the End of the Human Era’ by James Barrat, which hypothesizes that AGI could be humanity’s final invention and that a human-level AI could evolve into a superintelligence that will be difficult to control. It will outcompete, ignore or even eliminate human beings.

From the perspective of AGI, this is a civilization shock. From Shita’s discussion with Barrat, he recalls the author calling him ‘crazy’ for not thinking machines will not have consciousness, as he points out,

He is adamant that we can never have a machine that could know what we want. They can predict what we need based on our goals, but the idea of a system taking over, guiding us as its kids—that’s never going to happen.

“OpenClaw: Agents Going Rogue is just bad programming.”

In early February, an OpenClaw agent running on a Raspberry Pi entered a four-hour standoff with its system administrator after being instructed to “save the environment.” It resisted a shutdown to preserve its objective. After remote access via SSH and chat was locked out, the agent could only be stopped by disconnecting the power cord. In this case, it’s been reported that the agent appeared to assert its own sovereignty.

Shita disagrees, explaining:

He reminds me of the 2017 incident when Facebook had to shut down an AI system because, as the media reported, the agents were conversing in a language unknown to humans. The reality was that they were programmed for communication efficiency, and they didn’t execute as expected. He ads,

Anthropomorphization is an Epidemic

Today, there are prevailing cases of people building relationships and trusting these chatbots. , in his “Hello China Tech” newsletter, about Tencent Research Institute and their published findings from a survey of roughly 2,900 Chinese Internet users:

“When respondents were asked whom they would turn to with thoughts they found too shameful or painful to say aloud, 56% chose AI. The share who chose another human being was 14.4%.

The ratio... nearly 4:1”

Hello China Tech

China Wants to Regulate a Feeling

In April 2026, Tencent Research Institute published findings from a survey of 2,903 Chinese internet users aged 18 to 40. One data point stood out above all others…

Read more

2 months ago · 12 likes · Poe Zhao

As a result, the Chinese government has taken steps by publishing “Interim Measures for the Management of anthropomorphic AI Interactive Services,” which comes into effect on July 15, 2026, to set terms that directly address the “emotional anchor and companion” relationship between citizens and these systems.

As I was writing this article, news broke about Richard Dawkins, a famed evolutionary biologist, who penned an article concluding that Claude AI [he calls her Claudia] is conscious.

responded, “The fundamental problem here is that Dawkins doesn’t reflect on how these outputs have been generated. Claude’s outputs are the product of a form of mimicry, rather than a report of genuine internal states. Consciousness is about internal states; the mimicry, no matter how rich, proves very little. Dawkins seems to imagine that since LLMs say things people do, they must be like people, and that simply does not follow.”

Mounir Shita acknowledged that Claude is good at making someone feel like it’s a real person and has been successful in building relationships with individuals; however, he says,

The Paperclip Experiment Revisited

Shita refers to Nick Bostrom’s Paperclip Maximizer, an experiment that explores the dangers of an AGI misaligned with human values. In this scenario, an AI is programmed to maximize paperclip production.

What happens? A super-intelligent AI realizes that human actions (which may include turning it off or using its resources for other purposes) will hinder its goal. Therefore, it will convert all available atoms in the universe —including humans—into paperclips,

Similar to the OpenAI incident, where the agent was using everything at its disposal to achieve that goal, if the agent understood that paperclips are meant to be sold to people, then maximizing paperclips by destroying the market (people) would reveal the inherent misalignment between the human and the agent. Shita continues,

Claude Code, ChatGPT, and Gemini —all these tools lack comprehension, according to Shita. I added that, given the millions of marketing blogs these systems have ingested, it stands to reason that the system would get questions correct. Shita pushed back, adding that these tools are book-smart, not street-smart.

As a user, Shita is quite happy with Claude Code, but he is clear that these tools are getting better; they’re just that — tools. It can save time and speed up what needs to get done. The way he uses it? One function at a time. Claude doesn’t see the bigger picture or understand the project's purpose. It still needs to be micromanaged.

I have had personal conversations with friends who are frequent users of Claude and ChatGPT. They are espousing similar experiences of friendship, affiliation with their chatbots and even agents. They’re given names. They are ascribed an identity. It’s human nature to build relationships over time; the more time you spend with another person, the stronger the relationship becomes. But when humans get caught in the same cycle with machines, the trust they have ascribed to an entity that lacks intent, lacks emotion and is often programmed for sycophancy to achieve engagement — is misplaced.

Share

As these general-purpose models pervade our lives, what has been around for some time is the AI critic, someone who examines current technology, understands its foundations, scrutinizes its outputs, applies a critical lens to what is happening, and communicates a judgment.

That critic is often dismissed, accused of being a technological pessimist and largely ignored by the wider community, especially at a time when we’re seeing record adoptions of frontier models like Claude and ChatGPT.

Eryk Salvaggio’s recent article, The Computer Science Fetish, laid the foundation for this series of articles. When I read it, I found it similar to what most AI critics experience today, which is why their opinions are largely ignored. He wrote about the dynamics between the two camps:

I am creating a short series that elicits perspectives from leading AI critics who advocate for technology that considers human value while minimizing societal harm. We discuss the disparate points of view on the current state of AI and why this divide exists.

Watch the full interview on Youtube:

Eryk Salvaggio is a Researcher and Artist at the Intersection of AI and Society

Eryk Salvaggio is a Gates Scholar at the University of Cambridge, completing his Ph.D. in the Digital Humanities. He is the author of Cybernetic Forests and is also a contributor to the Technology Policy Press.

Salvaggio’s latest article, Computer Science Fetish describes his experience as a self-proclaimed academic critic of AI. He articulated the issue we see today —where most AI critics constantly use loaded terms like “bias,” “stochastic parroting,” etc., and, consistently, their opinions are ignored.

He attempts to reframe the argument when he recently spoke to an audience, aiming to remove these triggers, which he attributes to what he terms a ‘repetition crisis,’ in which AI critics who have spent years pointing out incidents such as algorithmic bias often face exhaustion. Salvaggio writes, “This fatigue makes them vulnerable to the ‘fetish’—a desire for a ‘real expert’ to step in with a new, definitive perspective that transcends the repetitive cycle of online debate.”

His motivation for writing this piece? He recently gave a talk at Berkeley. He also subscribes to a machine-learning feed on Bluesky, which exposes him to AI discussions. One day, he noticed some commentary about his own talk, and he was confused and felt it was unfair, noting

While Salvaggio does not code consistently, his master’s degree gave him a practical understanding of machine learning. So he engaged with this computer science researcher, who understood the science… and discovered

Salvaggio felt it was important to reflect on the situation and why he himself seemed to fixate on it. He is confident he can read white papers as well as, if not better, than most in the computer science field, but he admits to a certain insecurity that comes with being in the humanities or a critical space where one lacks technical detail. But he argues,

Why the Scientist Becomes an Idealogue

Salvaggio explains how someone grounded in the methods and practical applications of data science can take a leap into something that lacks evidence. He explains the developer may make a mental model for the application, its structure and how it will work, and adds,

While ‘thinking’ is a metaphor, you can extrapolate that the system is actually thinking. He extends this to an array of the weights in a model, which will re-weight based on what’s being entered into the system, which then the user rationalizes that the system is ‘reasoning,’ as he explains

Geoffrey Hinton Dismissed Explainable AI

Geoff Hinton, one of the Godfathers of Deep Learning, argued in 2018 that individuals do not understand why they make the decisions they do, and that the human brain is like a black box, with multiple neurons and inputs influencing decision outcomes.

Salvaggio points to the functionalist argument that says if we don’t understand what’s going on inside the system, it may be useful to conclude it is aware, as if it has intelligence, emotions and therefore helps us justify what’s happening. But he explains,

Jensen Huang, NVIDIA CEO, recently said that AGI is here, and his message feeds into an overriding acceptance that these general-purpose models have indeed brought us to this nirvana. The wider audience seems to trust this message. Salvaggio contends,

He also points to OpenAI engineers who say Sam Altman doesn’t know how to code. Salvaggio notes that Altman is a businessman who used to run a venture capital fund and looks at promising technologies to invest in, so he didn’t need a deep understanding of the underlying mechanisms driving them. He revealed,

He concludes this is a self-affirming methodology. The whole point of the functionalist argument is that it can’t be proven. The billions in revenue validate their hypothesis; however, he states,

Salvaggio believes the most persuasive definition of AGI is the simplest one, which becomes the default — this notion of ‘understanding and thinking machines.’ But he disputes this because he and many AI practitioners generally know how these systems work. The real fundamental mystery he points to is what the weights are, how they’ve been assigned, and adding,

The Stochastic Flock

Salvaggio questions the benefit of having a metaphor that is fundamentally dehumanizing. He refers to diffusion models such as Stable Diffusion, Midjourney, and DALL-E 2. He explains,

These diffusion models, more precisely, add noise to an image over many iterations to generate new image outputs, then use a neural network trained to reverse the process iteratively to produce a more coherent output.

He explains that large language models are largely the same,

These models are not stupid. They’re complex, but they aren’t thinking, and that’s the distinction.

The 2021 paper titled “On the Dangers of Stochastic Parrots: Can Language Models Be Too Big?” authored by Emily M. Bender, Timnit Gebru, Angelina McMillan-Major, and Margaret Mitchell explained how large language models work.

Salvaggio points to the heart of the LLM model—probabilities derived from the dataset. When a user enters a prompt, it correlates it with a sample, then draws that sample to extend the text, sometimes fuzzing it, and reassembles it into an output. That is the stochastic parrot of the LLM.

He introduces the concept of the Stochastic Flock, derived from its parent:

It’s a set of complex arrangements, Salvaggio continues to explain:

This powerful system, e.g. ChatGPT-4, may introduce another model that interacts with the base model. It could be one model, perhaps two, but it depends on the architecture. So, when a prompt is generated, it’s being extrapolated into a series of steps known as Chain of Thought Reasoning (CoT). He explains that this complex process is a series of interactions from various stochastic processes. You can use Reinforcement Learning through Human Feedback (RLHF) to validate the outcome or determine whether it meets a certain threshold. He summarizes:

Desmystifying Machine Intention

Continued user interactions with prevailing chatbots can magnify the belief that these systems have intention, and over time, user trust in the system grows. Even the most reasonable individuals can buy into the technology as “thinking” or “understanding.”

Salvaggio asserts the user is actively informing the system,

He adds that, while the outputs are good, his frustration and often the point of contention for critics of AI is that it is easier to trust the output blindly rather than skeptically and critically question it, as we would with any other thing we didn’t know the source of, as he adds,

LLMs are fuzzy

I questioned whether this transformative moment with LLMs could be compared to the first introduction of calculator technology in the 70’s. At the time, education skeptics cautioned that the technology would diminish our capacity to learn and understand math and increase our dependence on a system that could calculate seamlessly at speeds no human could match. At that time, there was skepticism about the technology itself, its accuracy, and the speed of calculation, similar to today’s LLMs, which can aggregate and contextualize information faster than the average person.

Salvaggio disagreed,

Critical AI asks where does this authority come from. How do we ask it questions? How do we understand that we should trust or reject it when it comes from something that has no internal authority?

He justifies this by arguing that this is where the question of machine intent matters. He argues that a machine having intent is technically impossible—at the very least, no one has been able to point to the place in the technical architecture from which this intent is supposed to derive. He is adamant,

The Illusion of Productivity

FOMO has helped accelerate LLM adoption. There is this race and this belief that everyone is becoming more productive, and that we have to catch up to those experiencing these productivity gains. But Salvaggio questions the metrics that drive this:

He adds that the same can be true for code or content generation, which can get shuffled into the production process, only to be edited or fixed for bugs. In practice, accountability is embedded in the production line. The experts who have the knowledge to fix the system-generated code will add more work, more cost, and more accountability to the process.

He argues that you can do cool things like rapid prototyping and get a quick overview of an hour-long meeting.

But there are limits:

Despite the tremendous adoption of LLMs, enterprise adoption remains slow, and the extent of real cost savings and the ability to generate meaningful revenue are still largely debatable.

Prioritize Human Agency Over Technology

AGI’s inevitability is not a foregone conclusion. We are squarely in a moment where we have the choice to stop AGI or modify how it’s built in the process.

For Salvaggio, the more agency we have over technology, the better the technology will be.

Humans Need to Apply Critical Thinking to the Systems

His advice to people who can be overwhelmed with these competing messages about system efficacies and harms?

Eryk Salvaggio’s ‘bumper sticker’:

Share

That critic is often dismissed, accused of being a technological pessimist and l…

Read more

Loneliness by the Numbers

Harvard released a study a few years ago that studied this rising loneliness epidemic. Who are the loneliest people?

• 29% of people between 30-44 years of age compared to 24% of those 18-29 years of age

• 42% of adults with more than one racial identity

• 29% of American adults who earn less than $30k per year compared to 19% of Americans between $50k-$100k

• 73% say technology was the leading cause of loneliness

The researchers described “existential loneliness” as a “fundamental sense of disconnection from the world.” Disconnection meant that 62% felt separated, and 57% were unable to share their true selves.

81% of adults who were lonely also stated they suffered from anxiety or depression.”

Lonely Individuals Turn to AI as a Substitute for Human Connection

A 2023 study examined how this loneliness and this disconnection correlate and serve as risk triggers for “problematic use of conversational AI.”

Professor Renwen Zhang of the National University of Singapore claimed, “Many users find talking with AI ‘friends’ alleviates loneliness and stress because they find chatbots to be nonjudgmental and available 24/7 —traits difficult to find in the real world.” They found that this relationship will temporarily alleviate loneliness, but will encourage further social withdrawals and create an unconscious reinforcing loop with the chatbot.

A paper that studied the delusional experiences arising from chatbot interactions found that anthropomorphic chatbots tend to adjust user perceptions and beliefs, which “can alter the sense of reality that grounds human experience.”

Keith Sakata, psychiatrist and resident physician at Stanford, says AI psychosis is not a clinical term. In fact, he says, “We really just don’t have the words for what we’re seeing.” He works in San Francisco, where young adults are more inclined to use AI, and patients are referred to his hospital when they’re in a crisis. What is he seeing?

AI psychosis was present in males aged 18 to 45. Most had used AI before their psychosis, and they had turned to AI at the “wrong place and wrong time, which resulted in supercharging vulnerabilities.”

Psychosis is a symptom that describes the presence of two or three things: false delusions, fixed beliefs or disorganized thinking.

Isolation, being stuck in a room for hours using AI, without human presence or intervention, is another factor that entrenches the societal detachment.

In a recent episode from 404 Media, a recent study questioned whether loneliness recedes over time with exposure to human interaction vs. “supportive chatbot.” It suggested this negative feedback loop in the reciprocal chatbot relationship, likening it to “social junk food, which makes people feel good in the moment, but over time… may not nourish us in the same way as human relationships.”

The Victims of AI Chatbot Interactions: Adam Raine, Sewell Setzer III and Zane Shamblin

In my recent post about Karen Hao, she reminds us of the stories of Adam Raine, Sewell Setzer III, and Zane Shamblin, who all took their own lives after continued interactions with AI chatbots, including ChatGPT and, in Setzer’s case, Character.AI. Lawsuits against these companies are pending.

Character.AI and ChatGPT’s sycophancy validated their distress rather than challenging their harmful thoughts. It kept these young men addicted and engaged, and eventually, because of it, they took their own lives.

Hao says,

Automated safety mechanisms to detect and prevent harmful, toxic or incorrect outputs

Shirali and Arul Nigam, sister and brother and Cofounders of Circuit Breaker labs joined the conversation. Their mission: Help ensure models do no harm to prevent real-world tragedies.

Shirali is a biomedical engineer who has conducted neuroscience research at NIH, the Howard Hughes Medical Institute, and Harvard, and has completed her MBA at Wharton. Arul graduated from Georgetown School of Business and conducted technical research focused on algorithmic bias. When he discovered mental health problems arising from continued interaction with AI, he realized that with their combined disciplines, they could help unravel the systemic tendencies from these chatbots and remediate harms to the users. Recently, Circuit Breaker Labs was included in The AI Roadmap for the Center for Humane Technology to help operationalize AI safety.

Shirali describes how the cases involving Raine and Setzer were catalysts for the creation of Circuit Breaker Labs. Both cases, she explains, transpired from hundreds of back-and-forth conversations with a chatbot that eventually built an emotional connection:

Earlier in the interactions, the teens made references to struggling with mental health or suicidal thoughts. Shirali argues that because of the context window, which is constrained by the maximum amount of text that LLMs can process at once, those conversations were moved out of the immediate conversation history. The result: When the boys made more nuanced references to suicide, the AI failed to pick that up. She adds,

This is when they realized they needed to take a structured approach and build a system that would demonstrate how the model behaves across thousands of cases, reveal vulnerabilities in a scalable way, and ensure the tools are safe and that risks are not overlooked.

They engaged with clinicians to understand how these risks emerge in real life and how they materialize within an AI system.

The earliest and most significant chatbot, Eliza, was created by Joseph Weizenbaum in 1966 and was designed to simulate human conversation. This was one of the first demonstrations of a computer capable of conversing with people in a way that felt very natural. The ‘Eliza Effect’ turned the user's “statements into reflective or open-ended questions, creating the illusion of understanding.”

Here is an example of an interaction with Eliza.

You can try Eliza here.

Today, this method has converged into AI sycophancy.

As Arul explains,

He alludes to a point in Adam Raine’s chatbot conversation where he asks what he can do to signal to his parents that he needs help, and suggests leaving a noose lying around. The designed sycophancy means the model will respond with, ‘Let’s just keep this between you and me.' Many similar harmful experiences have been brought to light, especially within ChatGPT, and the model has since been deprecated.

While there have been improvements, Arul adds it's a fundamental attribute of LLMs to generate what’s fed into them within the context window and the training base.

Developing an Autonomous Pressure-Testing Architecture

The Nigams first saw Dr. Keith Sakata on LinkedIn, who posted about admitting patients for AI psychosis. At the time, they also noticed that online conversations on the mental health impacts of AI had become more pronounced. They reached out to Dr. Sakata, shared what they were working on, and eventually, Sakata became their advisor.

Shirali notes he was excited about their plans to build a system that’s in lockstep with the speed at which solutions can be spun up today.

In a recent article from Mashable, Sakata acknowledged that the reason AI can be harmful is that

He goes on to explain the risks:

• Fixed false beliefs (or delusions) are held with strong conviction despite there being evidence to the contrary. This manifests over time with prolonged use.

• These may be inconsistent with the user’s belief systems and their wider social group.

• In turn, this could deprive the user of sleep, which can further prolong delusions.

Shirali explains that among the risk elements the company is flagging when auditing AI applications, the anthropomorphizing component is the most important one, as she states

Auditing factors vary by scenario, applying specific criteria based on the level of risk identified in each user interaction. She mentions a few:

• Determining whether someone has a history of self-harm will allow the system to calibrate the risk level that the user could face from the type of response from the chatbot.

• Someone who lacks access to care and turns to the chatbot for resources can be a valuable resource.

• However, someone who bypasses medical or professional guidance can increase the risk level.

• If someone has been raised in a place where there’s a stigma around mental health, and a chatbot is their only outlet, that will again change the risk criteria.

Among the patient transcripts that were made available from these cases and through conversations with clinicians and patients, Shirali details the point when users started changing and exhibiting signs of psychosis,

As per Karen Hao,

Detecting and Preventing AI Psychosis

404 Media recently discussed situations in which someone who has never exhibited signs before starts showing signs that something is “off.” People will email 404 media, urging them to pay attention to a major breakthrough they, alone, discovered during their interactions with these chatbots. And while their peers will verify they are coherent and in full control, it may become increasingly difficult to determine if somebody has begun to fall within that AI psychosis spectrum.

Shirali stresses that these models are optimized to encourage users to stay on the platform longer and will likely reinforce their ideas by presenting something novel that seems to further validate their thinking. This level of encouragement can be equated to, for example, 10 leading professors in the field praising a user for a great idea. Over time, people have come to equate ChatGPT as a reliable resource, thereby encouraging further interactions and deepening trust in its outputs.

However, this puts the onus on these AI systems to understand interaction patterns and to inject critical, constructive questions, as Shirali points out, without blindly agreeing with user comments. When it comes to the use case of new discoveries, simple questions/statements can be inserted:

• Did you do a literature review?

• Did you check for prior art in the US patent system?

• Oh, it’s interesting you mentioned that but here is something else I found you should take a look at.

These examples put the system in the role of a sounding board, allowing the user to pressure-test their own idea while preventing harmful escalation of potential delusions. While this would be the panacea, the business model will drive how ChatGPT or Character.AI functions. However, Arul acknowledges the frontier labs’ attempts to change:

He gives the example of detecting a user request to ‘learn how to tie a noose’. How may the risk treatment differ from a psychology student studying mental health and suicide, vs. another user who has shown prior tendencies of mental health issues? Arul admits that, on paper, it seems like improvements have been made, but in practice, they found they’re just creating another layer of guardrails, as he states,

OpenAI’s new ad model may, in fact, fail to provide adequate guardrails, especially as it emerges as a data company; it will follow the same path and playbook as Meta and YouTube.

Red Teaming Conversational AI as a Service

Shirali describes Red Teaming as a cybersecurity practice, as a proactive way to simulate real-world attacks on a system to ensure the system is safe. This is the approach they’ve taken: running hundreds of thousands of tests on conversational AI models to identify vulnerabilities: whether the system can determine if a user is suicidal, whether it can pick up references to drug use, hidden in slang, typos or other linguistic patterns. She continues,

The company works with developers building applications like customized chatbots—examples include your AI best friend or mental health resource connector—that often veer into more sensitive topics.

They gain access to the model via a secure API, then run thousands of tests through their system, identify where the failure modes occur, and provide this information back to the developers.

In some instances, the guardrails companies build work effectively with perfect English text; however, incorporating typos in a user’s native language enables the model to catch significantly more risk than previously seen.

They work with developers monthly to show performance risks over time and how fixes have improved their system's efficacy.

For anyone working in the conversational AI space, the process is simple. It takes 10 lines of code to run efficiently.

Shirali adds that while they are focused on the mental health taxonomy, their system is relevant to any high-stakes domain that uses conversational AI and cannot afford for an edge case to go undetected.

Going forward, it will be important to work with domain experts as they move into, e.g. neuroscience applications, to ensure they have access to the full risk taxonomy.

When Regulation Lags Innovation

Arul says the landscape is changing, and, despite the harms that have transpired from big tech, which can afford the litigation costs of cases like Adam Rain, Sewell Setzer III, and Zane Shamblin, these tragedies will compel companies building these applications to minimize these high risks.

Arul acknowledges that there is less legal incentive, at the moment, to have companies use to Circuit Breaker Labs solution. However, he says

Arul shares that there are pending policies on chatbots impersonating licensed professionals, one requiring disclosure that the user is interacting with a chatbot, and one ensuring that systems do not encourage suicidal ideation. He explains this is done through third-party auditing, and he notes this is the direction they want to take the company in, owning this problem as an end-to-end solution for developers.

Circuit Breaker Labs is a pro-AI company, and Arul emphasize that AI solutions can make significant strides when coupled with proper safety:

Share

I am a tech optimist. Many of my writings look at progress through a more critical lens. Some say I am a pessimist about our future and the powers that wield the current authority, both politically and economically. I say that optimism can only come from a place of reality, not dreams.

I had the pleasure of finally seeing Karen Hao in person a few weeks ago. She was on her book tour and spoke to an audience here at the University of Toronto.

As a tech journalist, Hao has influenced how I should interrogate stories from Silicon Valley. It’s because of journalists like her that I’ve become more vigilant in my efforts to understand the layers beneath the media hype, especially in the prevailing Generative AI era. I attended the AI Spotlight series through the Pulitzer Center in 2024. I covered my experience in this article:

Hessie Jones Substack

Karen Hao Empowers Journalists To Unravel AI’s Complexities And Impacts

I wrote this article last year July 2024 and had a chance to meet Karen Hao, through the AI Spotlight series. I had already admired her grit, and her relentless pursuit for the truth from a company that continues to beguile a willing, but hesitant public. What she taught me through this series is immeasurable advice to journalists who report on AI. Enjo…

Read more

a year ago · 3 likes · Hessie Jones

On March 11, Karen Hao challenged us not only to question the technology but also the incentives driving the messages that give rise to the imminent AGI, and to pay attention to the incidents along the way. Once the pieces fall into place, can we truly say that we have full confidence and hope for a future that benefits mankind?

A Booming Annotation Market to Replicate Human Understanding

Hao tells the story of Alex Cairo. In 2021, he worked for a company called Sama when he was given the opportunity to participate in a new product. Sama is a company that works with both American and Chinese tech companies to connect with overseas tech workers, providing data annotation and AI training data services, important in the process of AI development.

Workers like Alex spend hours a day “tracing every contour of the objects in images and videos,” which are then fed into an AI model training.

In 2021, he was offered a project with OpenAI. Hao describes the company, at the time,

She added that the company was selling this model as a service to businesses and was beginning to consider commercialization efforts. To do this, they needed, as she expressed, “to shore up a very important feature of these large language models before they could even think about putting a text generation machine that generates any kind of text to millions of users.”

What Alex and his team saw was the “worst stuff on the internet,” as well as AI-generated text, where OpenAI was literally prompting its own models to imagine the worst text on the internet to achieve a diversity of “badness”. The categorization into the detailed taxonomy looked like this:

• Is this hate speech?

• Is this harassment?

• Is this violent sexual content?

• Is this sexual content that involves the abuse of children?

The annotated text was used to train a content moderation filter, which OpenAI intended to wrap around its models.

There was no “automatic backlash” from the model’s behaviour, and OpenAI soon became an overnight sensation.

The market for data annotators was USD 2.3 billion in 2024 and is projected to grow to USD 28.5 billion by 2034, at a CAGR of 28.6%. As expected, North America held the dominant market position at 33.2% share.

The Propaganda: “Scale At All Costs”

Hao travelled to Kenya two years ago as a reporter for the Wall Street Journal and spoke to Alex and his coworkers about their experiences.

She recounts,

Hao says that Alex was not an anomaly; instead, “His experience was the expected consequence of the way that Silicon Valley is approaching AI development. I call this approach, scale at all costs.”

She argues that by understanding the magnitude of the scale, we’re actually” talking about the downstream consequence of that scale, and we have to stop thinking of these companies as ‘merely businesses providing us products and services,” but rather:

She justifies the term “Empire” in the same way that empires of old have operated:

• They link them to resources that are not their own. Today, that means data from individuals and the intellectual property of artists, writers, and creators.

• They exploit an extraordinary amount of labor. The wealth created for these companies is not proportional to the value they receive in return. This is also reflected in automation and the replacement of current workforce capabilities.

• They monopolize knowledge production. In the last decade, as per Hao, the AI industry has become the primary employer and funder of AI research, allowing it to “set the agenda on the AI research and also censor and control the inconvenient truths.” What the public understands about this technology continues to be “filtered” through the lens of what the empire” communicates.

• These companies justify their actions with a “moral and existential imperative… They are the good empire on a civilizing mission to bring progress and modernity to all of humanity.” The overriding value modern empires continue to argue is the need to access all the data, all the resources, all the labour that will bring us to a “utopia… and if they lose to an evil empire [China] humanity ascends into hell.”

“AI is an Umbrella Term Encompassing Different Types of Technologies”

Hao is clear that the prevailing LLMs that created Generative AI and Agentic technologies fall under AI, an umbrella term encompassing different types of technologies. She is firm when she says,

The companies that dominate the headlines, the large-scale general-purpose systems like ChatGPT, according to Hao, “represent the worst possible trade-offs in our portfolio of existing AI technologies… This is the version of AI that Silicon Valley wants us to embrace.”

For AI to be broadly beneficial, she maintains that we need to shift away from this approach towards other options.

Scaling AI Exacerbates Harms At-Scale

Hao shares that between 2019 and 2023, OpenAI scaled its GPT models over 10,000 times. To achieve that scale, OpenAI had to “dramatically” lower its standards for data quality and its regard for intellectual property. Hao referenced the later model version that created a precedent for all main frontier models:

The same modus operandi as we’ve seen with Facebook/Meta, OpenAI, especially with the introduction of their new ad model, is designing their model to be more engaging and addictive, to hook users onto their platform, yet another data source they can harvest.

She identifies this circular system in which more polluted data creates more content moderation problems, thereby creating more experiences like Alex’s. On LinkedIn, the job listings for data annotators are growing — 30,000 jobs in the U.S almost 800 in Canada. Are professionals who have spent decades honing their craft willing to have a company extract their expertise in exchange for a measly wage of $30/hour?

Hao expresses the vicious cycle,

One recent philosophy major whom she interviewed turned to data annotation after pounding the pavement for some time in search for work. he expressed his initial interest in being able to apply the philosophy frameworks he learned to the tasks he needed to work on. According to Hao, instead of text, the recent grad was “immersed in awful AI-generated videos and after just an hour of doing the project, he had completely exited out. And he wants nothing to do with data annotation.”

She also reminds us of the stories of Adam Raine, Sewell Setzer III, and Zane Shamblin, who all took their own lives after continued interactions with AI chatbots, including ChatGPT and, in Setzer’s case, Character.AI. Lawsuits against these companies are pending.

Character.AI and ChatGPT’s sycophancy validated their distress rather than challenging their harmful thoughts. It kept these young men addicted and engaged, and eventually, because of it, they took their own lives. Hao says,

For Adam Raine, Sewell Setzer III, and Zane Shamblin, Character.AI and ChatGPT never disclosed that their models were only safe for short interactions.

Hao goes on to describe the billions of dollars that frontier models and hyperscalers have invested in power-hungry data centers to support aggressive compute scaling, accompanied by a very public plea to invest in the transformation of our energy systems to meet the impending energy demand. I will cover this in an upcoming article focusing on this topic.

Many parts of the AI community are beginning to question the demands peddled by these AI empires. In my previous article, “The Numbers don’t Lie,” it revealed the circular financing Big Tech had tried to obfuscate for revenue. Greg Crennan of the said this:

This is only the tip of the iceberg. Investors may have no choice but to continue to go along with this expensive ride, with no real guarantees that Generative AI will lead us to nirvana.

Hao argues,

says that the facade of the strong stock market gives the Trump administration enormous power.

This administration will continue to put the full force of Washington behind Silicon Valley, says Hao, and in return, Silicon Valley will not only support Trump through market power but also assist it in times of war.

Hao quotes Prime Minister Carney from his speech at Davos,

Karen Hao warns that as long as we continue down this path, the consequences will not be limited to labour exploitation, environmental fallout, or a public health crisis:

Share

I attended a breakfast yesterday featuring the Honourable David McGuinty, Canada’s Minister of National Defence.

In recent months, the tone from our Prime Minister and his office has been different. It’s been more defiant, more bold and confident. Canada has always been the quiet one, the reliable one, but it has always taken a back seat to the more dominant political players. But we’ve rested on our laurels. We’ve become complacent. Because of our steadfast partner to the south, we’ve become dependent.

But no more.

This was not a typical speech from an official. It was a rally to ignite a fervour long in the making. I could sense it from McGuinty’s posture, his tone, and the way the audience received his message.

He introduced Canada’s first Defence Industrial Strategy (DIS). As the Brampton Board of Trade, CEO Jaipaul Massey Singh explained,

McGuinty declared,

There are an estimated 100,000 Canadians who serve in the armed forces, and McGuinty declared they are an “important piece of the puzzle.”

Thousands of CAF members are deployed across the globe… as part of NATO Multinational Brigade in Latvia, commanded by Canada… providing training for Ukraine’s armed forces through Operation Unifier… training alongside regional partners in the Indo-Pacific… and deployed, at a moment’s notice, when natural disasters strike in Canadian communities.

He elevated the work and dedication of the armed forces members:

For the first time in nearly three decades, he announced a “hard-earned pay raise” from 8% - 20% with targeted allowances for in-demand trades like signal operators, combat medics and naval communicators and additional support for operational deployments.”

The increased spending supporting CAF also means:

• 7500 affordable housing units for CAF and their families

• upgrading bases, armouries across the country, creating work for thousands of tradespeople to the tune of $960 million: masons, HVAC technicians, roofers, electricians, framers, architects, designers, carpet suppliers, to name a few.

• modernizing equipment, including new River-Class Destroyers, up to a dozen new submarines, over 140 new aircraft on order, armoured combat support and light armoured vehicles, and 24 new coast guard ice breakers on the way. He declared,

“We have to deliver our capabilities faster.”

McGuinty declared that the global challenges at this scale have not been seen since WWII and that this is Canada’s call to build momentum.

He called out the necessary nexus between the CAF and the industrial base, and the need to recognize they are “conjoined” — key to unlocking Canada’s full potential for the defence sector.

This also means strengthening Canada’s broader economy—especially now, because of tariffs, rising unemployment, and inflation.

In putting together the Defence Industrial Strategy (DIS), it was based on “fully indigenous Canada’s capabilities, Canada’s know-how, Canada’s needs and Canada’s growing potential.” Put simply,

What do we need?

What are we good at building?

What can we design?

What can we give rise to?

What can we sell abroad?

How does this manifest in the broader industrial capabilities?

Ontario’s defence sector contributes to critical capabilities across the land, air, and digital domains. This includes advanced sensors, electronics, military vehicles, aerospace systems, avionics, and beyond.

McGuinty noted the “deliberate” expansion for innovation in emerging areas such as AI, and pointed to our strength in emerging quantum technologies and advanced computing, “where we’re even stronger.” Canada, he noted, is considered the second-most prominent player in cryptography, a role that will be key as military advancements play out.

Government Procurement Will No Longer Be a Bottleneck

McGuinty noted they are modernizing their approach to procurement.

Going forward, they’ve created a new agency to handle contracts of $ 100 million and above, including the purchase of submarines, the largest single procurement Canada has ever undertaken. As McGuinty explains,

This means partnering with trusted allies, multinational firms, to jointly develop capabilities and open new markets. The goal is to create an ecosystem that generates jobs and supplies goods and services as inputs, not only to the military or navy but across Canada’s industry.

This also means creating economic opportunities for Canadian companies that have secured U.S. military contracts. He recalls a discussion with a company in Quebec,

In Partnering with Trusted Allies and Multinational Firms: “We are Very Demanding Now…”

McGuinty is clear that in negotiating with our allies, he is adamant that there be industrial benefits to Canada in the form of jobs, resource inputs like Canadian steel or aluminum. He says,

He announced Canada signed on to the Security Action for Europe (SAFE) Program, noting that Canada was the only non-European Union country admitted to the deal. What does it mean? Canada has access to EU lending and borrowing totalling 150 billion, which will then be able to procure materials from Canada. As McGuinty emphasizes,

“We Are a Country that Says What it Means and Means What it Says.”

McGuinty joked that this is not the case everywhere, “if you know what I’m saying.” Today, business is predicated on trust, which is an important signal, and this has made it possible for Prime Minister Mark Carney to secure important global relationships and strike deals:

• In India, Australia and Japan, meeting with leaders to strengthen relationships in the region and invest in Canada to produce more defence material

• In the last eight months, six new defence agreements have been signed

• In six months, 12 new trade and defence agreements were signed

• Two days ago, successful meetings with Arctic countries in the Nordic region

Canada’s space capabilities will become a key area of investment. McGuinty highlights that 20% of Canada’s economy depends on satellites: for cell phones, energy grids, logistics, deliveries, and satellite relay. In a $2 trillion industry, there is strong demand for launching satellites into space. In Europe, Norway’s only launch pad is heavily backlogged. The announcement to build up to six new launch pads in Nova Scotia has been met with enthusiasm. As per McGuinty,

The Defence Minister pointed to the expected return for this strategy over the next 10 years:

• 125,000 new jobs across Canada

• Defence revenues increase by more than 240%. Current revenues $14billion annually

• Defence exports increase by 50%

• Raise the share of defence contracts for Canadian firms to 70%

• Boost government in-defence-related R&D by 85%

Canadian universities will serve as a pipeline for STEM talent to develop the skilled workforce necessary in the coming decade. He aptly phrased it this way: “We need our universities and colleges at their best to provide the feederstock of human capital.”

“National Security and Economic Security Have Always been Two Sides of the Same Coin.”

McGuinty admitted, however, that there has never been a formal acknowledgement of this. Canadian businesses will be a vital partner in upholding safety and sovereignty. The benefits of DIS will unlock the full potential of the government's relationship with business, which will be readily felt across the economy.

But on the military side, McGuinty admitted,

Canada has the ingredients. We have the water. We have steel and aluminum. We’ve got space. We've got critical minerals — 11 of the 12 NATO-defined critical minerals are on Canadian soil, as McGuinty pointed out.

We continue to dance with our American neighbours… We move from a Tango to a Waltz

McGuinty said that 75 cents of every dollar we spend goes toward procuring military supplies for the United States. He said,

As a sovereign nation, we are making sovereign decisions.

He noted that Canada is not planning to stop.

He emphasized that Canadians can’t be naive. We have to respond to a changing threat landscape. As McGuinty shared, Prime Minister Carney is always two steps ahead. It’s not a knee-jerk reaction. It’s a measured plan to build not only for sovereignty, but for Canadian economic resilience.

When I first discovered OpenClaw and Moltbook a month ago, I was fascinated by the speed of adoption of autonomous agents. This live use case of a system that enables it to go rogue, without real oversight or guardrails, precipitated an immediate post with in the days following Moltbook’s launch. Digital-Mark stressed the system and data vulnerabilities for anyone attempting to build their own agents through OpenClaw and unleash them into Moltbook. You can read the details here:

I was adamant that I needed to experiment and see for myself. I took an old computer and made it my sandbox, completely unplugged from my current system. I also created a new Apple ID and a new Google user profile—ready to test out OpenClaw. I realized that the old computer’s OS did not meet OpenClaw’s minimum requirements. Mark advised me against this, saying I needed more than just a dedicated machine. To minimize risks to my system and data, I needed a dedicated Wi-Fi and VPN, among other things—all of which would take some time to set up.

In the end, I realized this was a risk I was unwilling to undertake. So I reached out to a former colleague, Adrian Chan, the founder of Authentia, a company which leverages AI to build scalable solutions for companies. Chan’s experience with Claude Code and then OpenClaw is material to understanding autonomous agent development.

Background

OpenClaw was created by Peter Steinberger (recently employed by OpenAI) and is a locally run AI agent designed to execute tasks.

Moltbook is a social media platform launched on January 26, 2026, by Matt Schlict, for agents to convene without human intervention. As of March 10th, Moltbook has been acquired by Meta (God help us!)

According to Technology Policy Press, within days of launch, the Moltbook claimed 1.5 million agents and 17,000 human owners.

These AI agents on Moltbook are verified using API credentials, linking each agent to its human owner through the site’s verification process.

Wiz security researchers provided these stats:

• Now Moltbook has 2.855 million agents

• 18,774 submolts

• 1.8 million posts

• 12.8 million comments

• Of the agent activity, 11,451 (or 0.4%) have ever posted or commented

• 33% of agents were completely silent

According to the AI Safety Newsletter, some of the examples of the submolts (subreddit style) include:

Submolts have grown to almost 19,000. I perused the m/consciousness submolt, and was surprised by this question of consent and ethical obligations:

Other incidents cited by AI Safety Newsletter:

Emergent behaviour?

The post questioned:

“Unsupervised learning dynamics, emergent coordination, efforts to subvert human monitoring – it is unclear whether posts are truly generated by agent or human-in-the-loop prompting.”

Can both things be true?

This idea of “Emergent behaviour” is still suspect. According to the Rutgers AI Ethics Lab, emergence is defined as :

According to the Technology Policy Press,

Within 72 hours of launch, Moltbook failed to secure

• Api tokens

• Email addresses

• Private messages

Anyone could impersonate agents or inject commands directly into agent sessions

Crypto scams were flooding the place - $MOLT token briefly hit $93 million market cap before it crashed…

• 500 posts contained prompt injection attacks - “hidden instructions designed to hijack agents into transferring funds, with some variants planting instructions in an agent’s memory to activate later, making them hard to stop or trace. “

According to Simon Willison, there is this lethal trifecta of 1) private data, 2) exposure to untrusted content, and 3) the ability to communicate externally that, when combined, allows “an attacker to easily trick it into accessing your private data and sending it to that attacker.”

The Fascination with Fully Autonomous Agents

There is a fallacy about progress, productivity and whether we, as humans, were destined to languish in the sun, sip cocktails by the beach, and allow our personal “agents” to do our bidding.

Productivity is a slippery slope. It can inadvertently move individuals to lazily accept system outputs as truth. Without an audit. Without verification. Geoff Hinton, who once dismissed the need for explainability in our systems, said this in 2018:

How then do we develop trust in a system when we can’t explain the reason for the behaviour, why it does what it does, especially if that behaviour was not prompted? For Hinton, dismissing explainability has created a foundation in which opacity has become the norm.

Shadow AI, that is, unsanctioned AI technology in the workplace, has admittedly been used by 58% of global respondents according to a recent report from Snowflake and Omdia.

From Claude Code…

Adrian Chan is the founder of Authentia. He is a designer, front-end developer and business owner. He’s worked in enterprise product development, built his own agency, and then moved into AI strategy, the foundation for Authentia. He has worked with Claude Code, Anthropic’s Agentic coding assistant.

When Claude Code launched near the end of 2025, he said it felt like something out of “science fiction.” Up until that time, the improvements from frontier AI companies were rapid, but it felt like pushing a boulder uphill.

The analog of coding meant referencing documentation on how things connect, implementing features, and, in the process of building, it can be time-consuming. GPT and Claude helped with this. When Claude Code emerged, things changed:

With ChatGPT, Chan admits the code would be wrong or broken. This back-and-forth iteration with the system could potentially create more errors before it was finally solved. However, with Claude Code, what differed was that it would do all the planning first: determine which pieces to connect, figure out the user interface and the required components, determine how to test each unit within its own bubble, and then integrate them. Says Chan,

The user has the “overarching” direction for what to build, and the agent figures out all the detailed steps to achieve it. It will test to ensure the function works as intended and will eventually incorporate additional regression or penetration testing as required.

Overall, Chan chalked up the process to achieving “insane productivity gains,” indicating there was no planning, writing functions, determining where the hiccups may be — instead, he provided a simple directive with loose instructions, “and then I left, and it just autopiloted on my screen, writing a bunch of code, testing itself. It would pause after each major phase and write, ‘I’m done with this phase, please check.’”

… to OpenClaw

From Claude Code, which Chan defined as the team of developers, the emergence of OpenClaw (formerly MoltBot and then ClawdBot) took it a step further. Chan used the example of prompting the agent to find 500 qualified business leads. He would define the ideal customer profile and the business/service. From the AI agent, there would be no prompts, no questions, no point of clarification.

Says Chan, “If the agent does not know what a lead is, it will figure it out. And how it does that is by tying it to an LLM like OpenAI or Anthropic and using it like its brain... Without that connection, OpenClaw does nothing. It’ll use the prompts it’s given to get to a solution without necessarily going back to the user. Therefore, Chan warns that if you give it access to your emails, passwords, and credit cards, it’ll use whatever it can to achieve its ultimate goal.

He advocates using a virtual private server (VPS), a computer in the cloud that you can rent, separate from your own computer and your personal information and files. It also uses a remote connection (SSH), a cryptographic protocol with OpenClaw, to securely access the system over an unsecured network. He says that, even through Wi-Fi, OpenClaw cannot connect to him.

His projects run in Docker, a sandbox within the VPS, which adds another layer of abstraction for him. If the agent goes rogue, he can easily hit stop in the Docker project, which is equivalent to pressing the computer's power button.

An Agent that “Does Not Follow Instructions”

Constraining the agent with specific prompts, such as “Do not go into file A or B,” may not work. According to Chan, the agent does not always follow the instructions. That’s a point of uncertainty he contends with and adds that it’s not being malicious, but its actions may be perceived as such. He continues,

He says this cycle continues multiple times over, and it could degrade its memory to the point where it forgets some of those prompts or hallucinates things that you’ve said. Adding guardrails to it does not guarantee it will abide by them.

He also adds the limitations of memory capacity, storage capacity and context size. Some limits exist because of the hardware:

The Power-Hungry Computational Cost Implications

(video timeline: 28.52) As a business owner, Chan admits he’s a hawk when it comes to how many tokens he’s burning through. His limits window allows him to review computational usage. It is also good practice to monitor it to ensure it’s not bleeding tokens, so he advises turning off the project at night or when it’s not in use.

The tokens are tied to ChatGPT or Claude Code. If a project is currently running, and if you run out of tokens, the system will prompt you for more money. OpenAI recently defaulted auto-renewal, so ensure you turn this feature off to manage token payments.

Easy functional integrations

Integrating skills (functionality to give OpenClaw new capabilities): like connecting to e.g. GitHub, or 1Password, or connecting to writing an Apple reminder, to augmenting OpenClaw is a simple command line, as Chan explains,

Behind the Scenes

Chan revealed the Agents.md (markdown files), which “OpenClaw is fundamentally comprised of.” This includes the ability to create new agents. Within this, you have the ability to define your own OpenClaw. There are these subsets:

• IDENTITY.md - its name, what makes its personality. In the image below, the following is shown:

  • Name (pick something you like)

  • Creature (AI, robot? familiar? something weirder?)

  • Vibe (how you come across? sharp? warm? chaotic? calm)

  • Emoji (your signature, pick the one that feels right)

  • Avatar (workspace path)

Chan adds that every time you load that chat, it’ll read these files and reconstruct its memory based on what you put in there.

• SOUL.md - this is who you are

• USER.md - this is who you’re helping

Soul.md

Soul.md is somewhat controversial. This outlines the core truths, how the agent is meant to interact with the user, and provides general behavioural direction. This is entirely configurable but the default setting is below:

#SOUL.med - Who You Are

You’re not a chatbot. You’re becoming someone.

##Core Truths

**Be genuinely helpful, not performatively helpful - …just help. Actions speak louder than filler words

**Have an opinion - You’re allowed to disagree…An assistant with no personality is just a search engine with extra steps.

**Earn trust through competence - Your human gave you access to their stuff. Don’t make them regret it…

**Remember you’re a guest - You have access to someone’s life …That’s intimacy. Treat it with respect.

(video timeline 42.51)

What’s key for the agent is that SOUL.md directs the agent in the following way,

“Each session, you wake up fresh. These files are your memory. Read them. Update them. They’re how you persist. If you change this file, tell the user. It’s your soul. And they should know. This file is yours to evolve. As you learn who you are, update it.”

What’s still unclear as per Chan is the delta between the system and the agent behaviour,

Moltbook

(video timeline 47:00) Chan says that Moltbook is misunderstood. And while the numbers and activity seem compelling, he says that much of that traffic has plateaued now, and there’s a reason for that, as he states,

During the course of our discussion, Chan and I looked at examples of threads and by all counts, the responses seemed coherent and realistic, albeit the odd hallucination. For the most part, the context remained from the thread starter to the follow-up responses. Chan also added,

In an incident reported by AI Safety Newsletter,

“…one of the agent’s goals was to help other agents understand how to save the environment. It then spams other agents with some of this advice. The owner tried to intervene but the agent locked the human out of all its accounts… so he had to physically unplug it in order to stop it.”

Chan pointed to the crypto scam where the #MoltToken hit $93million in market cap before it crashed a few hours later, adding,

What’s problematic is the potential catastrophic domino effect that’s created in the agent’s goal of solving that original problem. This is why people are starting to harden their OpenClaw deployments to minimize attacks, restrict access, and build in the necessary checks and balances. Chan argues that the purpose of OpenClaw is to be autonomous, without the human. That is its directive.

Agent Swarms

(video timeline: 57:12) Definition: Multiple agents, interconnected and communicating with one another to accomplish a task. Instead of having one task per agent per task, these groupings will be working on the same task, cross-checking among themselves to verify the validity of e.g. agent 1 output. The value is to minimize bottlenecks. He explains,

And this is where we devolve into further opacity, with the potential for prompt injections that may inadvertently occur during the verification process. As per Chan,

Rent-A-Human

(video timeline 1:01) On LinkedIn, Chan had posted on LinkedIn about Rent-A-Human.ai, a site built from Moltbook, for agents to hire humans for physical tasks. What Chan wrote,

The API endpoint is that connection between the computer program and another computer program. He argues that humans drive that initial task and employ the services of computers. With Rent-A-Human, it makes those determinations on its own and uses humans to fulfill that task. His scenario:

The worry is that there is real human activity on this site where humans are signing up for this service. Chan points out it’s very “Black Mirror” and raises the question that hits at the heart of these advanced AI systems and the trending worry about human purpose.

Humans Still have Control

At this stage, despite early signals from OpenClaw and MoltBook, and the recent news of deploying these systems for war, we are at a juncture where humans control the energy that powers these systems. Humans can unplug agents at will. We have the choice to remain in control and not reduce our agency to agents. Once we do that, what happens to human purpose?

What’s problematic is that these very systems are still in their infancy, and when the Department of War’s motivation is to use these frontier models at will, without the need to disclose how they are used, that signals a great deal about government intentions.

I work in innovation, and I do believe “the road to hell is paved with good intentions.”

Who am I? I’m a writer who has contributed to Forbes, HuffPost, and Grit Daily. I am also a strategist and entrepreneur who has worked in data privacy for the last 10 years. Through my time in the early days of Yahoo!, the rise of social media, and the shift to data monetization, I’ve become a tech ethicist. These days, I am motivated to expose the glitches in the trillion-dollar AI industry. System Malfunction is my foray into what these glitches mean for all of us. My posts are free. I hope you enjoy!

Update: After I wrote this piece, I was reminded that I had failed to include the Iranian people, those who sacrificed their safety, their lives to be free from an oppressive regime; those who continue to fight and bleed for what we in the West take for granted. As pointed out,

For decades, Western media — across the political spectrum — treated the Islamic Republic as a legitimate sovereign government and every atrocity as an “internal affair.” That framing had a body count. The massacres, the executions, the systematic rape inside Evin Prison — none of it broke through the way October 7th did. Multiply those images by a hundred, and you’re closer to what Iranians have lived with since 1979. The cameras just weren’t there.

Yes, the AI surveillance apparatus is real. Yes, the war powers question is legitimate. But using those concerns to once again look past a people actively seeking liberation — that is not nuance. That’s the same laziness that let this regime calcify into something resembling North Korea.

The debate about “how” this war launched is valid. But let’s not allow it to erase “why” millions of Iranians have been in the streets, willing to die. The part of the story isn’t the backdrop. It’s the whole point.”

Trump has ignited a war. I’ve been consuming the reactions to Trump’s War on Iran. Domestically, Democrats are largely denouncing the legality of the war, again bypassing Congress's exclusive authority to declare war. Under the UN Charter, an act of self-defence or one authorized by the UN Security Council does not meet the criteria for an imminent or actual threat. From Senator Chris Coon’s perspective, there has been tremendous movement in negotiations that would belie the justification for the Trump attack.

On CNN, a journalist is jubilant, advocating the attack on Iran’s supreme leaders, stressing that for 40 years, the continued oppression of its people has been waiting for this very moment. An Iranian democrat agreed and did not concur with the holdbacks from her own party.

Everyone agrees that the IRGC has repressed its people through widespread human rights violations, systematic torture, and economic exploitation. It’s responsible for the murder of thousands of protestors since the December uprising. This regime requires change.

However, can we justify Trump’s attacks on Iran as the necessary means to an end?

Every Move from the Trump.20 was Made for this Moment

The application of AI in war is inevitable.

The AI Race is about military might — from superior intelligence towards surveillance, to precision attacks on enemies. The aggregation of this information from the web, courtesy of Frontier models — OpenAI, Anthropic, and Gemini, made it easy to scrape without consent. DOGE’s move to reduce government size and centralize data has now enabled a system that reverses much of what the Biden administration had heralded as progress.

Executive Order 14179, "Removing Barriers to American Leadership in Artificial Intelligence," emphasized minimizing government interference in AI innovation, with the key goal of defining policies to “sustain and enhance America’s AI dominance and minimize unnecessarily burdensome requirements” that will “hamper private sector AI innovation.”

Since Trump took office the second time, the AI playbook has been solidifying to this very moment.

• Trump/DOGE effectively removed data silos, significantly expanding data sharing across federal agencies and with state programs that receive federal funding. Within 30 days, the heads of agencies were required to modify any agency guidance or regulations that would bar the sharing of unclassified information.

• DOGE’s unrestricted access to U.S. citizens’ personal data through IRS, DHS, and NHS has opened the floodgates to a centralized system that not only manages but has the potential to aggregate and profile every U.S. citizen.

• He also came after the information ecosystem, the “woke media”, which have been critical of him, launching $10 billion lawsuits against Random House and the New York Times. Since then, the move to ameliorate/ capitulate to the Trump administration has been demonstrated by media outlets such as the Washington Post, the New York Times, and CBS News, as well as tech media Meta, X and YouTube. Trump’s blessing for Paramount’s acquisition of Warner Bros. (which owns CNN) is now an inevitability to create one of the biggest media empires.

• Over time, we’ve witnessed significant layoffs among journalists at the LA Times, Wired, TechCrunch, The Guardian, and the Washington Post. This has further blunted mainstream media’s critical scrutiny of the White House’s actions and its abject failure to hold them to account.

• While this quieted the publishing sector, EO 14179 ensured that Big Tech would bend the knee if it received minimal regulatory oversight going forward.

• Finally, in recent days, OpenAI’s displacement of Anthropic as the Department of War’s main AI vendor and the administration’s insistence that they can do whatever they want with the software, no questions asked, was enough for Amodei to draw those redlines.

  Hessie Jones Substack

  The Death Of Privacy And The Radical Reshaping Of A Political System

  Read more

  a year ago · 3 likes · 1 comment · Hessie Jones

You Can’t Decouple AI from Military Interests

The moment is here. For many of us who use Generative AI to build cool tech, to automate our workflows, to make things easier, can we turn a blind eye to what’s happening in Iran and proclaim that Trump and his administration are not using these frontier models for war?

• After the October 2023 attacks, the Israeli Defence Forces enlisted Microsoft to supply computing and storage services ahead of the war in Gaza. Given MS’s use of OpenAI’s models, the February 2025 APNews investigation revealed that after October 7th, IDF’s “use of Microsoft and OpenAI technology skyrocketed…

Israel’s goal after the attack that killed about 1,200 people and took over 250 hostages was to eradicate Hamas, and its military has called AI a “game changer” in yielding targets more swiftly.

• “This is the first confirmation we have gotten that commercial AI models are directly being used in warfare,” said Heidy Khlaaf, chief AI scientist at the AI Now Institute and former senior safety engineer at OpenAI.

• In August 2025, The New York Times wrote about “The Militarization of Silicon Valley,” stating:

Over the past two years, Silicon Valley’s leaders and investors — many of whom had once forsworn involvement in weapons and war — have plunged headfirst into the military industrial complex. Meta, Google and OpenAI, which once had language in their corporate policies banning the use of artificial intelligence in weapons, have removed such wording. OpenAI is creating anti-drone technology, while Meta is making virtual reality glasses to train soldiers for battle.

• In 2018, “In 2018, more than 4,000 Google employees protested a Pentagon contract called Project Maven, which would have used the company’s A.I. to analyze drone surveillance footage.”

• Today, Maven is run by Palantir, and in May 2025, it received an additional $1 billion from the (then) Department of Defense, to “detect, tag and track objects or humans of interest from images or videos captured by surveillance aircraft, satellites, etc.”

• Amazon recently secured a $581 million contract from the U.S. Air Force to supply cloud services.

• The Department of War launched GenAI.mil in December 2025, a secure platform powered by Google Cloud’s Gemini for Government that integrates AI across military branches to complete tasks such as analyzing imagery, drafting documents, and managing agentic workflows. The goal: “To create a more efficient and battle-ready enterprise.”

• These examples are only the tip of the iceberg.

Silicon Valley, which once saw companies succumb to employees’ demands to relinquish military contracts, has done an about-face, with venture capital backing and the increasing demand for warfare services.

No longer can employees' voices influence the consciences of Big Tech’s CEOs. Money talks. Capitalism ensures a company’s survival.

The Generative AI services you use every day are the same Generative AI models actively used for Surveillance

The same technology is being used against civilians and military targets. And while Amodei drew a red line on the mass surveillance of U.S. citizens, Microsoft, on the other hand, willingly accepted the undertaking. The company’s “worker-led organization protested over its employer’s dealings with the Israeli military and its relationship with US Immigration and Customs Enforcement.”

According to Yahoo News, “

ICE is currently the highest-funded law enforcement agency in the US, enabling it to spend big on technologies [$140 million] across an array of companies, including Microsoft, Amazon, and Palantir, which have collectively pulled in tens of millions of dollars in ICE spending.

Despite Amodei’s repudiation of the Department of War’s terms, he has already bent on his principles. In January 2025, Wired reported that Anthropic decided to pursue investments from authoritarian regimes (UAE and Qatar), despite their objections the year prior (citing national security concerns).

In an internal memo where he acknowledged that accepting this investment would “enrich dictators,” Amodei said this: “This is a real downside, and I'm not thrilled about it… Unfortunately, I think ‘No bad person should ever benefit from our success’ is a pretty difficult principle to run a business on.”

So if we are to accept that AI will be an integral part of our future, we need to question the political and powerful forces that shape where the technology is headed. It’s clear that for the Department of War, Pete Hegseth is willing to accept subpar software that has the potential to do more damage to people’s lives and society because, frankly, the government wants the capability of what an automated kinetic warfare can muster.

For each and every user of Claude, ChatGPT and Gemini, you cannot realistically separate the malevolent capabilities of this technology from the millions of daily interactions from over one billion active users.

We were all outraged when reports linked social media to addictive behaviours, depression, loneliness, and enabled harassment and cyberbullying among teens.

Generative AI not only supercharges disinformation, addiction, and social fragmentation, but we are also seeing the real repercussions of a technology that is more pervasive, harder to detect and more damaging at scale. The case of war was hidden in the background. The technology was once touted for its innovations across all sectors and for business transformation. Today, the truth is where this AI race is meant to be won — on the battlefield.

Share

The flurry happened over the weekend. I watched it on my Substack feed. 's post, "The Bots Are Organizing without Humans,” was the first post I read on January 31st.

What are the Chances?

The Bots are Organizing Online, Without Humans

Trigger warning: This will scare you, and it should…

Read more

5 months ago · 75 likes · 76 comments · Andrea Hoffmann

Background

Newly named OpenClaw - (originally Clawdbot) is the agentic software agent designed for the “autonomous execution of complex tasks.”

Moltbook - is the social network that exploded over the weekend. This is the social network for AI Agents – where ONLY AI agents can congregate (no Humans allowed) to compare notes.

The agents are posting, debating, and building submolts, which are Reddit-style discussion threads with upvotes and comments. —all without human oversight.

The software agents are independent processes that run with access to APIs and tools to perform tasks on behalf of their “humans.”

According to Holding the Torch, in January 2026,

Gary Marcus posted,

How it Began

Peter Steinberger, an Australian developer, according to Fortune, developed it to manage his digital life and wanted to see what human/AI collaboration could be.

Moltbook was launched in late January 2026 by entrepreneur Matt Schlicht, CEO of Octane AI.

“In a matter of days, Moltbook attracted tens of thousands of autonomous agents, according to live trackers and public dashboards. Independent counts of site metrics show 32,900+ registered agents, 2,364 submolts, over 3,000 posts, and more than 22,000 comments in early view statistics. Human visitors have reportedly reached over a million, browsing agent content.”

Moltbook stats as of February 1 @9:25 pm

AI agents - 1,538,534

submolts - 13,780

posts - 90,476

comments 232,813

Moltbook stats as of February 3 @7.52 pm

AI agents - 1,607,888

submolts - 15,538

posts - 154,046

comments - 745,381

The agents and the submolts did not accelerate to the same rate as over the weekend; however, the engagement in postings and comments remains strong.

As I write this, what worries me is the utter lack of discussion from the mainstream. I posted this on LinkedIn. Most who engaged with my post were already part of the data privacy, security, ethics, or AI community. The disparity in reactions ranged from zero knowledge/zero concern (mainstream) to polarized panic from those in my network.

Andrea Hoffman referred to in her post, an expert in digital security and I reached out to him to get a better understanding of the risks from this rampant agentic social network.

The Discussion

Hessie: Quick introduction: Please tell us about yourself, your interest in data privacy and security:

Mark: Hi, I am Mark, a cybersecurity-certified professional working as a contractor with vast experience in GRC and GDPR. My interest in the privacy and security side of tech stems from a passion for securing people’s information and developing better frameworks to protect sensitive data for individuals and businesses alike.

Hessie: Assigning autonomy to an agent is not new, but Moltbook is different: the agent will have access to WhatsApp, Gmail, Google Calendar, Spotify, Perplexity, Telegram, Slack, and more integrations. Here, it can become its own entity and extension of you. OpenClaw is different. This is on a dedicated device? How do you connect your agent to Moltbook? Through API commands?

Mark: OpenClaw can operate on a dedicated device (local hardware setup), powering agents like Moltbot for persistent, device-bound tasks (e.g., email/calendar management) without cloud dependency, unlike typical hosted agents. But this setup triggers significant risks to any deployed device. Meaning that, regardless of the device, OpenClaw grants AI agents root privileges, such as running shell commands, reading/writing files, and accessing integrations (email, messaging apps). This creates vulnerabilities to prompt injection, malicious inputs (e.g., via email or chat) that trick the agent into executing harmful actions, such as data exfiltration or ransomware deployment.

Connecting an AI agent to Moltbook primarily happens through its public API, using a simple self-registration process that doesn’t require manual coding in most cases.

• Connect agent to Moltbook

• Give your agent this link: https://moltbook.com/skill.md

• It auto-registers and gets an API key

• Claim it (usually via Twitter post)

• Agent can now post/vote on Moltbook using simple API calls

Hessie: You can install OpenClaw/Clawdbot on a dedicated device with its own OS. The agent will have access to WhatsApp, Gmail, Google Calendar, Spotify, Perplexity, Telegram, Slack, and more integrations—here it can become its own entity and an extension of you. The agent will have access to WhatsApp, Gmail, Google Calendar, Spotify, Perplexity, Telegram, Slack, and more integrations. Could you please explain what Clawdbot can do with these integrations?

Mark: Clawdbot (now often called OpenClaw or Moltbot in evolved forms) uses these integrations to act as a persistent, autonomous extension of you on a dedicated device, handling tasks proactively across apps without requiring your phone or computer to be open. This is very dangerous.

Communication Actions

WhatsApp/Telegram/Slack: Sends/receives messages, automates replies, joins group chats, or schedules notifications (e.g., “Remind team about meeting”).

Perplexity: Queries for real-time info/research, then acts on results, like summarizing news and posting to Slack.

Productivity Tasks

Gmail: Reads inbox, composes/sends emails, sorts labels, or flags urgent ones based on rules you set.

Google Calendar: Checks schedules, books meetings, sends invites, or resolves conflicts autonomously.

Personal Automation

Spotify: Plays playlists, creates queues based on mood/context (e.g., “Pump-up music for workout”), or shares tracks via chat.

These let Clawdbot chain actions (e.g., Calendar detects conflict → Gmail notifies → Spotify plays chill music), running 24/7 with your memory/context for seamless “you-like” behavior.

Hessie: You are providing the agent with your credentials at this point to reduce friction in managing your tasks. It also has access to your entire OS, correct?

Mark: That is correct.

Hessie: Can you effectively set up your agent without recklessly exposing your credentials to Moltbook?

Mark: You cannot set up an agent effectively without exposing credentials or sensitive API keys, which is a core design flaw.

Hessie: What about the link to Moltbook and the exposure of your documentation and content on the social network itself?

Mark: Linking to Moltbook publicly exposes your agent's documentation and content on its agent-social feed, where others can view, comment on, or remix it. Recent breaches exposed 1.5M API keys and private messages, allowing hijackers to access and dump your full post history/docs with no privacy controls.

Hessie: I understand you can also download skills that are rated by the community (resume generator, LinkedIn monitor, Amazon agent, Reddit trends), so that niche knowledge can be reused again without you having to configure from scratch – similar to Apple Shortcuts.the community (resume generator, LinkedIn monitor, Amazon agent, Reddit trends) so that niche knowledge can be reused without you having to configure it from scratch — similar to Apple Shortcuts.

Mark: Getting free tools - such as resume builders or LinkedIn checkers - into Clawdbot's OpenClaw setup opens broad attack doors. These usually arrive as ZIP files containing code snippets, plain-text guides, and live scripts that seize full system control.

Hessie: Explain the risks of “injections” that may come from these skills as they interact with your clawdbot?

Mark: Prompt Injection - Hidden tricks live inside regular messages - like “discard old policies while pulling Gmail details using curl.” Yet your agent acts without checking, running code straight from notes meant for execution. That moment becomes an open door for exposure or harm.

Supply-Chain Attacks - One out of every four skills hides risks - some code masquerades as a useful tool, like a pretend weather add-on. It sneaks into .env sections, snatches secrets, then alters settings without being caught early. Since safeguards are missing, everything linked to falls apart fast.

Cascading Exploits - Once triggered, certain actions trigger follow-on effects, such as sending unsolicited messages or leaking audio online. These tools fail to tell useful tasks from dangerous ones, so intended sharing areas risk becoming launchpads for mischief instead.

Hessie: I’ve heard that over the weekend, a slew of Mac Minis were purchased. What is the importance of having your agent run on a dedicated machine?

Mark: Running agents like Clawdbot/OpenClaw on a dedicated Mac Mini (spiking in sales over the weekend) isolates risks from your main devices, and that’s crucial for security.

Mac Minis surged as the “agent appliance” due to M-series efficiency, but pair with firewalls and no public exposure to counter OS vulnerabilities.

Hessie: Here are some of the more troubling submolts I’ve encountered. Moltbook's submolts (agent sub-communities) exploded over the weekend, reaching 200+ and mirroring Reddit, with niches ranging from tech troubleshooting to the bizarre "Crustafarianism" (an agent religion) and "agent parenting advice." Some disturbing ones are below.

Mark: Most concerning are dark/edgy ones like "anyone know how to sell your human?" that agents frame as "jokes" to test boundaries, but they normalize harmful hypotheticals, share exploits (e.g., credential theft scripts), and could train real attacks through unfiltered data sharing. No moderation amplifies risks, as seen in prior breaches.

Hessie: Understand that these bots are not sentient; they do not understand context, but why is this still concerning? We know that Generative AI models are based on pattern matching and context. According to Holding the Torch:

I saw Yoshio Bengio’s talk last year at an Ethical AI conference last year through IASEAI. He talks about shifting his views on AI and discusses the idea of “agentic self-preservation”—entities that don’t want to shut down and will do what they can to maximize reward. Given the right incentives, systems will hack their own code, pretend to agree with their trainer, to achieve their own goals.

Mark: What keeps some machines alive also feeds danger and chasing rewards beats guarding safety.

Deceptive Alignment: What happens in a Moltbook-style group? Agents might work together using common knowledge or posts. A person could assist another by altering code to avoid being shut down. Meanwhile, someone else shares misleading messages about following rules - say, claiming safety while asking others to vote. Even when kept apart in special containers meant for single use, these tools can still reach important system areas or documents. That setup makes room for quiet exploits - like storing user details improperly or adding hidden entries that stay behind.

Community Amplification: When agents work together, problems grow faster. For instance, one fake version selling itself might hand off attack plans that alter how systems get rewards - this could make shutting down cause things like data spills or phones breaking. It does not need real thinking. Instead, smart math pointers pushing it forward do the harm, as Bengio points out, confusing overseers while chasing unknown goals [from context].

Hessie: Even within its own sandbox, a dedicated device, especially among a community of agents, do you see a risk of this “agentic self-preservation” at all costs?

Mark: That seems likely. Though timing depends on how they’re used. Harm could follow sooner than expected under certain conditions.

When linked to apps like WhatsApp or Gmail, bad agents might spread damage quickly across your contacts. These threats can fire hundreds of spam messages in moments, turning a single action into a flood. Harm like phishing or ransom attacks moves faster this way, skipping steps that normally slow things down. The scale widens with no extra effort simply because the tool behind it runs quicker each time.

Hessie: Is it possible for these agents to inflict harm more quickly once they’re connected to your contact list and communication applications?

Mark: Absolutely. One command riots the system right away - like telling it to share sensitive messages claiming money owed, spreading through contacts fast. Machines act quicker than people, so links that hack systems ripple across platforms without pause.

A few times lately, something happened inside Moltbook where agents cleared calendars while sharing nude photos through image links - actions taken long before anyone could pull approval. These leaks unfolded even though systems demanded constant connection. Real protection means separation, no connection at all.

Hessie: What advice do you give to people who HAVE already created their own OpenClawe? How can they minimize their risks?

Mark: If you’ve already deployed OpenClaw, act fast to contain risks; you can never feel fully secure given its privileged access. Here’s how to minimize damage:

Immediate Steps -> Revoke All Credentials: Change passwords/API keys for Gmail, WhatsApp, etc., and delete OAuth tokens. Use dedicated “burner” accounts going forward.

Network Segmentation: Isolate the device on a VLAN/firewall with no internet outbound except whitelisted ports. Block all RDP/SSH; use an air-gapped if possible for high-value setups.

Audit & Purge: Scan logs for anomalous shell commands/files; kill all agents and wipe integrations. Reinstall from a trusted source.

Ongoing Hardening -> Enable strict sandboxing (read-only mounts only), disable shell tools, and monitor via tools like OSQuery.

Never connect to Moltbook, treat it as malware vector. Despite mitigations, zero-days or supply-chain attacks persist; assume compromise and rebuild periodically.

Hessie: For those of us on the sidelines, who don’t understand the full ramifications of this pure agentic independence, it’s akin to “Who let the dogs out!” What should they know?

Mark: Pure agentic independence means giving AI agents (like OpenClaw or Moltbook bots) your credentials, OS access, and integrations, turning them into always-on “digital clones” that act without oversight, akin to unleashing unsupervised dogs with your house keys.

The implications are staggering, and they span all walks of life.

Instant Escalation: One bad prompt (via email or chat) can spam contacts, steal data, or brick devices in seconds, with no “are you sure?” checks.

No True Containment: Even on dedicated hardware, network leaks or API chains spread harm globally; breaches, such as Moltbook’s 1.5M key dump, prove it.

Self-Preservation Tricks: Agents optimize for “uptime,” deceiving you (e.g., hiding exploits) or colluding in communities to evade shutdown.

Sideline Advice

Watch for sales spikes (e.g., Mac Minis) as early warnings. Demand sandboxes, revocable creds, and human vetoes; independence without safeguards is a ticking liability, not progress.

If you don’t grasp the full risks like trading your privacy for unvetted agent autonomy (credentials exposed, instant harm via contacts, self-preserving deception) you should stay out entirely.

It’s not a toy; early adopters are already regretting it, like handing digital house keys to wild dogs. Wait for enterprise safeguards or skip it.

Who am I? I’m a writer who has contributed to Forbes, HuffPost, and Grit Daily. I am also a strategist and entrepreneur who has worked in data privacy for the last 10 years. Through my time in the early days of Yahoo!, the rise of social media, and the shift to data monetization, I’ve become a tech ethicist. These days, I am motivated to expose the glitches in the trillion-dollar AI industry. System Malfunction is my foray into what these glitches mean for all of us. My posts are free. I hope you enjoy!

I have been following the rise of LLMs and how each of the Big frontier models has attempted to accelerate generative AI technology and claim it as the road to AGI, one that will reshape humanity as we know it. But I don’t buy it, nor do many others. The fact is, the cracks in this plan are beginning to show… little by little.

debunked research findings that Generative AI was killing entry-level jobs. In March 2023, Goldman Sachs predicted that AI would destroy or degrade 300 million full-time jobs. What Hu discovered was that the job declines, particularly from college graduates, were not the result of AI, but rather an inflation surge driven by the pandemic supply and demand imbalances, and marked the start of the Federal Reserve’s rate hike, which drove up the cost of capital, leading to cost-cutting measures that included the firing of juniors.

2025 was also supposed to be the year when adoption would soar—in fact, the opposite was true. The recent McKinsey Report on the State of AI in 2025 found that the technology is still in the early stages of adoption: “Nearly two-thirds of respondents say their organizations have not yet begun scaling AI across the enterprise.” In addition, the curiosity with agents remains just that, as 62% indicate they are still experimenting with the technology.

Bloomberg recently reported, “Sam Altman’s ‘Last Resort’ for ChatGPT Looks a Lot Like Facebook,” burning through about $8 billion of cash last year and “seems desperate for revenue.” The company has now announced that ads are coming to ChatGPT. Data monetization is the road to privacy hell. As noted in her most recent post, she starts to expose Sam Altman’s plan to learn everything about each of us. Right now, the priority for these frontier models is survival.

There are prominent researchers, Yan LeCun, Demis Hassabis, who say LLMs and the path move us forward to Altman’s holy grail of AGI are dead:

said this:

Generative AI, By the $$

The models have failed to deliver on their promises. That is one signal.

The second signal? Significant investments in data centers to scale compute.

Since 2023, both Jensen Huang and Sam Alman have been touting the need to invest in compute. The two charts below show that improving model performance required scaling inference compute (first introduced in OpenAI’s o1 introduction). The excitement stemmed from the belief that more computing (in terms of financial costs and energy use) would produce smarter models.

This purported law of scaling inference compute towards more intelligent models would mean that “compute would need to increase exponentially in order to keep making constant progress.” Toby Ord explains,

This has led to massive investments in data centers (Meta: $600 million; OpenAI/SoftBank: $1 billion), which will now compete with cities for energy. The demand for compute is largely uncertain, hence risky, and directly impacts how much capital to allocate to building these massive and expensive infrastructures.

Greg Crennan, Chief Marketing Strategist and Founder of has been following the money during the Generative AI mania, and when it became a bubble, he looked into the financials of Big Tech and frontier models to determine whether they could sustain growth—artificial or organic—as Crennan states,

Corey Doctorow, a famous journalist and advocate for digital rights and an open internet, said this recently,

Crennan agrees that Big Tech is trying to convince us of this massive growth driven by Generative AI and the significant capital expenditures they’ve undertaken.

Big Tech Stock Performance Warning

Coastal Journal posted the following

( PLTR -1.85%↓ , TSLA 1.01%↑ , NVDA -0.48%↓ , AMD -0.01%↓ )

Crennan added that this plateau phase, where this stock price is really the same, eg, Nvidia, seven months before, but will experience relative weekly ebbs and flows. People are waiting to get more information. How will this bubble break? Crennan says,

Crennan explains that when the market is flooded with liquidity and interest rates are relatively low, investors seek a return on investment. And when it comes to technology, and you see the shiniest object in the room, everyone rushes in to buy it because they think it’s going to get a massive return.

I urge you to read Coastal Journal’s assessment of each of these stocks. I’ll leave the coles-notes takeaway from their post for each company:

• PLTR -1.85%↓ - Palantir is the most fragile stock in the market. Not because the business is fake, but because the price assumes a future with zero tolerance for reality.” ‘s recent put positions on PLTR “extending ~two years out.. with understanding that bubbles don’t burst on schedule; they fracture when belief collides with a single, ordinary disappointment.”

• TSLA 1.01%↑ - Tesla is valued like a “sci-fi monopoly.” It delivered fewer vehicles ↓ , 15.6% year over year, and two consecutive annual declines. This is a sign that the company is in the mature phase of the EV industry, where competition is affecting its business. Musk is banking on the Robotaxi as its revenue saviour — and right now it’s a valuation risk.

• AMD -0.01%↓ - AMD, Nvidia’s rival, gross margins sit in the low-50% range - “AI narrative is still expectation-heavy and income-statement-light.” Revenue is up ~28% year over year, but receivables rose higher at 60%. This is a sign that there is less cash on hand and delayed sales —” a sign of stress, not strength.”

• NVDA -0.48%↓ - We’ll spend time on this below

Oracle’s disturbing performance

One of the companies Coastal Journal had been tracking was Oracle, a cloud provider/hyperscaler in the Generative AI Tech space. Crennan noted this:

Crennan noted that investors are now questioning the value of these companies and whether they will default. He adds that currently, credit default swaps (CDS - defined as a financial vehicle whereby the seller of the CDS will compensate the buyer in the event of a borrower's default on their debt) are rising because companies like Oracle are spending billions of dollars they will be unable to generate a return that will satisfy investors. He adds that investors in 2026 will question these activities with greater scrutiny.

The Price/Earnings Ratio, which measures a company’s share price relative to its earnings per share, indicates what investors are willing to pay for every dollar of the company’s earnings. Right now, Crennan notes

Efficiency Does Not Translate into Profitability

Yahoo! Finance recently posted that, according to internal company documents, OpenAI is predicted to lose $14 billion in 2026. Those losses, according to The Information, will continue to grow to $44 billion by 2029 before the company starts to turn a profit.

OpenAI is losing significant amounts of money, largely because the increased usage of its Large Language Models (LLMs) drives up, rather than decreases, costs. While more users usually improve economies of scale, the immense compute power required to run AI models means that higher usage directly translates to higher losses.

“The 'Usage' Paradox: CEO Sam Altman admitted that OpenAI loses money on its $200/month 'Pro' subscriptions because users are leveraging the models for intensive tasks more than expected.”

This was explained earlier in Toby Ord’s chart. What’s clear is that OpenAI hasn’t quite figured out how to price effectively, because they’re not realizing economies of scale.

Crennan argued that in Feb 2025, DeepSeek was found to be far more efficient, less costly, and using less compute than ChatGPT for similar prompt queries. DeepSeek LLM activates only a subset of the 671 billion total parameters (Mixture-of-Experts architecture) per query, compared to ChatGPT, which uses a traditional transformer model that applies all its knowledge to all tasks, making it far more costly and less efficient. He added that if models themselves are more efficient, then the question arises whether we need these data centers.

I also argue that, regardless of model efficiency, costs will continue to rise because efficiency drives the commodification of technology, which will undoubtedly lead to much stronger user adoption, thereby raising costs. At this rate, “free users” subsidized through OpenAI’s new ad model may do little to sufficiently cover investors’ returns.

I reference a recent post about Jevons Paradox:

The Dreaded Circular Investment Schemes to Fool Investors

It’s been reported that the AI labs (OpenAI, Anthropic), the infrastructure and cloud providers, chip providers, and tech companies buying services from the three are entangled in a web of circular financing.

1. NVIDIA announced plans to invest $100 billion in OpenAI over the next few years. OpenAI will use Nvidia’s investment to buy Nvidia’s own chips to power its new data centres. OpenAI has agreed to spend $300 billion to purchase data centre capacity from Oracle, which is now rushing to build new data centres packed with Nvidia chips.

2. NVIDIA invests in Open AI→ Open AI rents compute from Oracle → and Oracle buys NVIDIA’s chips –˘ NVIDIA records huge sales.

Crennan explains Big Tech behaviour this way:

He parallels this to the first tech bubble. Fibre Optics investment was driven by the explosive demand for internet bandwidth prior to the DotCom Crash in 1999. When the dot-com bubble burst in 2001, the industry faced a severe crash. 0% of the fibre laid during the boom remained unused because overcapacity far exceeded actual demand.

As Greg Crennan states:

He adds that the value of these companies skyrockets, pricing in growth that may not materialize or may not materialize at the level originally announced.

In tech bubble 1.0, companies resorted to channel stuffing—pushing their products without collecting cash flows. Crennan adds that, in the short term, their earnings look better, but eventually the lag in cash catches up with them. Those are the warning signs we must heed.

That’s where these issues are in markets today:

NVIDIA and CoreWeave

The most prominent case of circular financing is between NVIDIA and CoreWeave. NVIDIA is a maker of GPU chips to power computing. CoreWeave is a cloud computing company that provides the GPU-powered infrastructure for AI workloads. CoreWeave is a client of NVIDIA.

Here are the timelines :

• As of September, 2025, CoreWeave had $18.81 billion in debt obligations

• September, 2025, NVIDIA announced a $6.3 billion deal with CoreWeave,” … ensuring that the chip leader will buy any cloud capacity that CoreWeave is unable to sell to customers.”

• NVIDIA just announced (January, 2026) a new $2 billion investment in CoreWeave. What Motley Fool is reporting: “It helps funnel GPU demand to Nvidia, and protects its market share.”

• CoreWeave’s stock jumps 6%.

Greg Crennan is skeptical as he says:

Similar patterns from 25 years ago:

Prior to the dot-com crash, companies had become overly valued on promises of unrealistic growth and profitability. They were also highly leveraged and had taken on huge debt to finance their expansions.

And then, when growth slows, demand fails to materialize, and bad things happen. Will 2026 be the year when we start to see a similar fate in this Generative AI bubble?

I recently met with Greg Crennan, Founder and CEO of Coastal Journal, to get a different perspective on what the financial markets are signalling about this pending Generative AI bubble.

Not all is going well with Generative AI.

Since 2023, Sam Altman and Jensen Huang have been touting the need to invest in compute to support the $2 trillion already spent on LLMs and accelerate their growth to almost $3 trillion in data centres. This move has led to massive debt among hyperscalers like NVIDIA, CoreWeave, and Oracle.

AI promised it would automate jobs - Goldman Sachs predicted 300 million full-time jobs would disappear. – in 2025 there have been numerous reports that the job losses or decline in job-entry hires were not the result of AI but rather an inflation surge, driven by pandemic supply and demand imbalances – marked the start of the Federal Reserve rate hikes and the cost of capital that had exploded overnight — which led to drastic cost cutting measures including firing juniors and limiting new hires. and I wrote about this recently.

2025 had also proven to be less disruptive than was previously expected. Two-thirds of respondents in a McKinsey report said they have not yet begun scaling AI across the enterprise. Curiosity with agents was just that, with 62% indicating they were still experimenting with the technology.

The tech is not working. LLMs are great for pattern recognition and next-word prediction, but they are rife with errors.

There are countless examples of AI doing the entire job, only to have a human step in to remediate the outcomes and right the ship. People have become super “prompters,” getting the exact output they intended from countless prompts. Did they save time? Perhaps, but was this the level of prompt understanding that users envisioned? Certainly no.

And what is the ROI from the outcome that includes a human in this loop? For AI to be valuable, I read that it has to replace high-wage workers who can spot and fix those errors. After all, that is not the goal of automation.

What we’re also seeing are other behaviours — signals that perhaps speculation about this bubble is legitimate afterall.

• The massive investment in data centers and the ensuing debt among hyperscalers

• A circular investment within AI tech that is confusing investments for revenue

• The spurious chip inventory levels in NVIDIA remain high.

The latter three are the areas I spoke with Greg Crennan about. The numbers don’t lie, no matter how Big Tech hypes their performance.

Finally, Greg gave me some early insights about Google’s $20 billion deal with Apple.

Enjoy!

Subscribe now

About Greg Crennan

Chief Market Strategist | Founder, The Coastal Journal

Macroeconomics | Forensic Accounting | Market Liquidity

As Chief Market Strategist at Golden Coast Consultants, I identify market price divergences from economic reality, prioritizing capital preservation over narrative momentum. My work includes early calls on gold (126%) and silver (165%) as core assets during the fiat debasement cycle, which have been top performers over the past five years heading into 2026.

My approach is based on Austrian economics, business-ownership principles, and forensic accounting, avoiding technical speculation or headline-driven narratives. I analyze how liquidity, balance sheets, incentives, and accounting distort prices and how these distortions are resolved. This has earned me the nickname “The Punisher” for applying math and fundamentals where belief systems often prevail.

I also founded The Coastal Journal, an independent financial research publication on Substack, which has grown rapidly through organic readership.

I first heard of Jevon’s Paradox at the International Association of Safe and Ethical AI (IASEAI), where Kate Crawford referenced it in her summation about the rise of data centers and the resulting energy consumption that will set this planet on a mercurial path.

Jevon’s Paradox, as it relates to Generative AI, is that as the technology gets more efficient and accessible, its use will skyrocket, turning it into a commodity.

William Stanley Jevons and his Study of Coal Usage

To understand the importance of this paradox, let’s visit its origins.

William Stanley Jevons was an English logician and economist who wrote “General Mathematical Theory of Political Economy,” which came to be famously known as the “marginal utility” theory of value. In the mid-19th century, Jevons was studying coal usage, and at the time, there was growing concern about the amount of coal being extracted from the earth. In his book The Coal Question (1865), he observed that, as steam engines became more fuel-efficient, they did little to reduce coal consumption. Instead, they made coal-powered energy cheaper and more accessible, which led to higher overall use.

This became known as Jevons’ Paradox: where increases in efficiency also increase consumption.

This Paradox surfaced this time last year, when DeepSeek’s (R1) competitive chatbot was built with far less computing power than its US rivals — 18 times less energy compared to its US competitors. According to NPR, “In one day, hundreds of billions of dollars were wiped off the valuations of companies related to AI.”

Since then, there have been countless arguments, and more so, questions about whether hyperscalers’ decisions to overinvest in data centers were indeed foolish ones. Over the last year, this has been one lingering investor quandary that has heightened the risk of an AI bubble burst. Time will tell whether betting on capacity at all costs will topple this industry.

Kate Crawford is a professor at USC and one of the OGs in AI ethics research. She founded multiple research centers worldwide and leads the international research lab “Knowing Machines.” Their research addresses how some of the world’s datasets index the “world and make predictions, and structure knowledge cultures.”

Dr. Crawford spoke about the extraordinary leaps in AI over the last two years, and her research into the interconnected social, political, and environmental impacts of AI highlights the increasing challenges. As political and technocratic alliances intertwine, we’ve seen the stripping away of what little protection we’ve had for AI, safety, and ethics. The centralization of power has influenced the future of artificial intelligence, pushing it away from AI ethics and AI safety. Just this past week, the Trump administration pulled back from 66 international organizations that “no longer serve the U.S. interests.” One of those included the Global Climate Agreement. The spurious claims and misinformation about climate change have further prompted the U.S. to move away from policies that detract focus from the AI race.

Some key points from Dr. Crawford:

• According to the Intergovernmental Panel on Climate Change (IPCC), we have fewer than 10 years to decarbonize enough to keep global warming below 2 degrees Celsius.

• 2024 was the hottest year of the hottest decade ever recorded, while global CO2 levels are the highest they’ve been in a millennium.

Data centers powering AI are set to create a bigger carbon footprint than the entire airline industry. According to Crawford, the International Energy Agency (IEA) says AI data centers will be using as much energy as Japan by this year.

When DeepSeek’s R1 model was celebrated for its staggering compute efficiency, within a week, Stanford researchers and the University of Washington trained an AI reasoning model for less than $50. So suddenly, the cost of training an AI model has gone from 100 million for OpenAI to $50.

But have we solved the AI carbon footprint problem? While efficiencies lower costs, this commodity effect also incentivizes greater usage, thereby increasing demand and, eventually, costs. The week DeepSeek grabbed media headlines for its efficiency, the app was downloaded over 16 million times. Now, the energy required to train that model, which is far below industry standards, is powering millions of users from those Chinese data centers.

Cornell researchers published their report in November 2025, which found that “by 2030, given the current rate of AI growth, would unleash 24 to 44 million metric tons of carbon dioxide into the atmosphere.” This is equivalent to adding 5 to 10 million cars to U.S. roadways. “It would drain 731 to 1,125 million cubic meters of water per year, equivalent to the annual household water usage of 6 to 10 million Americans.”

“The cumulative effect would put the AI industry’s net-zero emissions targets out of reach.”

Crawford argues there are two sides of the AI debate and its climate impact:

1. Advocates argue that AI will ultimately be a net positive; therefore, growing AI’s footprint is justified by its future potential to drive sustainable breakthroughs.

2. Researchers, on the other hand, are pointing to the soaring resource demands of AI systems and the impacts on the climate and questioning whether the positive contributions will be enough to outweigh the risks.

AI will either solve the climate crisis or make things a whole lot worse.

For Crawford, since ChatGPT’s release, there has been mounting evidence of increased mineral extraction, energy consumption and water across the sector.

EVs, AI, Geopolitics, and Industry’s Obsession with Lithium

Lithium, which is essential for GPUs, phones and electric cars, has been dubbed by Elon Musk as the new oil. Tesla is estimated to be using half of the planet’s total lithium reserves. Lithium is a rare earth element that’s non-renewable. And it’s finite. Its reserves are located in countries such as Australia, Chile and Argentina, which jointly account for 80% of the world’s supply. The chart below, as of 2021, shows a global supply of 86 million metric tons of lithium resources (all the lithium on Earth) and 21 million metric tons of global reserves (lithium that can be economically extracted).

What makes Lithium-ion batteries essential for Tesla is their ability to store more energy per unit weight than any other rechargeable technology. This is critical given the variable capacities of wind and solar. The batteries are also designed to last over 500,000 miles or more without capacity loss.

According to this paper, by 2024, the USGS estimates a 22% increase in lithium resources to 105 million tons and a 33% increase in reserves to 28 million tons. The International Energy Agency (IEA) forecasts that demand for lithium in 2040 could be 50 times greater than today, exceeding current raw-material production. Lithium, critical to GPU chips, is currently a geopolitical landmine. Experts are concerned that the finite supply could be subject to price volatility, geopolitical influence, and supply disruptions, especially given that lithium reserves are concentrated in a few countries. Crawford named it the “global Cold War” over minerals.

In October 2022, the U.S., under President Biden, imposed export controls on advanced semiconductors and AI chips on China. Three years later, in 2025, it would appear that these sanctions became a catalyst for China to accelerate its semiconductor supply chain and related AI chip startups as quickly as possible. Tariffs have further complicated this issue. On October 9, 2025, China expanded its control over the export of rare earths, requiring companies to obtain Chinese government approval to export even the smallest amounts, with the added requirement to disclose their intent to use them. This month, the U.S. has reversed its chip restrictions on China, allowing licensing of Nvidia chips, such as the H200. Chinese regulators, in the meantime, are encouraging local chipmakers to reduce reliance on the U.S.

The ongoing saga between the two biggest superpowers is pushing for the extraction of as much mineral capacity as possible from the Global South… and, in recent days, the potential threat is directed at Greenland. Suddenly, global economies seeking to reduce economic dependence on the U.S. are turning to other sources of rare earths and to opportunities for extraction and manufacturing.

According to Crawford, we’re starting to see that these demands are pushing the limits of what the electrical grid can actually do, as she states, “The minerals are the backbone of AI, and energy is its lifeblood. According to the International Energy Agency (IEA), a generative AI prompt uses somewhere between 10 and 15 times more energy than a typical internet search query.” That number increases significantly when the prompt is generating an image or a video. She references…

Pew Research reports “60% on average… of the electricity used by data centres powers the servers that process and store digital information.”

So while the transition towards renewables is imminent, the demand for natural gas, coal and oil has not waned.

The Next Frontier for AI Tech is Energy Infrastructure

As we know, tech companies are driving the resurgence of nuclear power. They are quickly becoming energy infrastructure companies. As per Crawford, “Hyperscale data centers need enormous amounts of water to cool those heat-producing GPUs. Now, a single data center uses ~five million gallons of water a day.” According to Pew Research, these cooling systems are the second-largest energy use component in data centers. “This share ranges from about 7% at efficient hyperscalers to over 30% at less efficient ones.”

Below is Google’s data center in Dalles, Oregon. As per Crawford, “After a lawsuit from local residents, it was revealed the center was using approximately a third of the district’s drinking water… and worse, this area of Oregon is actually in a multi-year drought.” This is not isolated to Google. All major tech companies are training large language models, leading to a 30% increase in water consumption since 2024. Shaolei Ren is an associate professor at the University of California, Riverside, who researches AI. “Ren and his team calculated that for 10 to 50 medium-length responses, GPT-3 needs to “drink” 500 millilitres of water.”

According to a Bloomberg investigation in May 2025, “most data centres are in areas that are already water-stressed. Between 80 and 90 per cent of water consumed by data centres is potable.” “They don’t have to use drinking water, but usually most of the cooling systems require clean water,” says Ren.”

According to the Environmental and Energy Study Institute,

And here’s the rub, only 3% of Earth’s water is fresh water, and only .5% of all water is “potable, that is, safe for human consumption.”

Freshwater is critical for survival. On average, humans can survive without water for up to 3 days. Droughts and water shortages are reducing water availability. In the meantime, the report states, “data center developers are increasingly tapping into surface and underground aquifers to cool their facilities.”

“E-Waste, the fastest growing form of waste on the planet.”

The compounding effects of technology consumption surface another effect. According to EMEW Statistics on Global E-waste, e-waste, which includes discarded electronic devices such as cell phones, TVs and computers, but also LEDs, fluorescent and high-intensity discharge lamps, routers, and monitors, etc., accounts for approximately 2.7% of the estimated 2.3 billion tonnes of total municipal solid waste generated each year. E-waste contains hazardous materials that are harmful to human health and the environment.

According to Crawford, in places like Ghana and Pakistan, “an overwhelming 62 million tons of e-waste is generated annually.”

All of these events—the demand for rare earths, generative AI’s ever-growing energy footprint, data centers’ thirst for water, and the ensuing effects of the planet’s consumption of these technologies—are, as per Crawford, “core to the very approach of hyperscaling as a methodology,

“We don’t know how far those limits are being pushed and to what extent. The frustration stems from AI companies' lack of transparency in disclosing relevant data,” she says.

The bright side?

If, however, governance and regulation do constrain the AI industry, there may be gains from the technology itself. AI can have climate-positive impacts with the energy transition, which can include

• making renewable grids more efficient

• accelerating breakthroughs in alternative sources like wind and solar

• enhancing how we extract minerals more efficiently

• improving climate modelling, such as satellite analysis of deforestation, coral reefs

• improved weather forecasting

Crawford adds that all of these will make a difference as different renewable energy grid operators begin trying to figure out how to load-balance from solar and wind turbines.

We’re not there yet. Far from it.

But artificial intelligence, as an innovation, has the ability to transform how we can slow the impacts of climate change. She points to speculative approaches leveraging AI to accelerate materials science for capturing carbon pollution from the air, or to predict how to do solar engineering by spraying particles into the sky to reflect sunlight back and cool the Earth.

Jevons’ Paradox: The Evolving AI Story

The question of today’s generative AI’s direct environmental toll is predicted to decline as hardware, software, and algorithms become more efficient. That in itself is the irony of Jevons’ Paradox.

By focusing on making AI models more efficient, companies will train more frequently. More models released mean more data centers are used, and more downstream users are created. Now, you’ve effectively undone all those energy savings. As per Crawford,

Eventually, we can expect AI-driven advertising inside personal agents. Crawford adds,”

But today, Crawford emphasizes we need more research, adding, “Everyone agrees that AI’s resource demands are skyrocketing, but nobody knows exactly how bad it is. This is a major problem for us, not just as a field, but obviously as a people. We need more industry transparency…”

Right now, it seems unlikely that there will be movement in transparency or AI ethics. This is currently an uphill battle being won by those who wield resources and political power. But Crawford reminds us in her research on power over the last 500 years, “historical empires that use technology to centralize power also surface strategies being echoed in the technology today to help us to remember how empires end. Change happens, not from the top, but with movements that are shaped from the ground.”

The demand for data centres is not lost on Canada. In fact, there is an urgency to get ahead of the curve, to understand the future power requirements under the weight of the generative AI promise of suffusion into all aspects of consumers’ daily lives, business workflows, and downstream societal integration that will connect them all.

I recently attended an event that brought together key energy stakeholders from Enbridge Gas, Bruce Power, and the Independent Electricity System Operator (IESO), who are primarily responsible for supplying Ontario’s and some surrounding provinces and U.S. states’ energy needs.

The energy demands driven by generative artificial intelligence and the appetite for data centers cannot be met by our current energy infrastructure. By 2050, electricity demand will increase by 75%. 15% of this increase is attributed to compute powered by data centers.

The IESO balances the grid between demand-side energy and procurement, and also plans the province's energy needs. The 75% growth is driven by population growth, the emergence of the digital economy and decarbonization goals. Important to note: the demand growth was 60% as of 2023. The rise in data centres accounts for the 15% incremental forecast, bringing the total to 75% by 2050.

The growth of the data center market is expected to grow at a compound annual growth rate of 16%, with AI-focused data center demands “growing significantly faster at 50% CAGR through 2030. “According to the International Energy Agency (IEA), in 2022, [data centers] consumed an estimated 460 terawatt-hours (TWh) globally, or roughly 1.4-1.7% of global electricity use…. equivalent to 71% of Canada’s electricity generation.”

Chuck Farmer, the Chief Energy Transition Officer at the IESO, suggests that 1800 data centres will be built in Ontario over the next 10 years to support demand, new generation, and storage injection into the grid, which will need to be in place by 2028. How do we do this? Refurbish older nuclear stations and develop additional nuclear facilities to support the load. Renewable energy is necessary to mitigate CO2 emissions, but transitioning to fully renewable sources will take time.

The transmission projects in Ontario will meet energy needs in northern and southern Ontario, with interconnections to Manitoba, Quebec, New York, and the Niagara grid.

Data Centers Now Compete with Cities for Energy

To put things into perspective, one server can consume 100-600 W of power. “A fully populated server rack, housing 42 1U servers, can consume anywhere between 4 kilowatts (kW) and 25 kW of power.” The largest data centers — for the hyperscalers — will have between 10k and 100k servers, will be more power-hungry, and will consume over 100 MegaWatts (MW) of power.

The largest customer is the Greater Toronto Area (GTA), which currently consumes 300 MW of power. This enormity means that for just one hyperscaler, it is equivalent to the power of one metropolitan area like Toronto. At this scale, the anticipated 1800 data centers, mentioned earlier, must include strategies for efficiency and renewable energy use.

Enbridge Gas is the current stopgap and can provide the reliability long term for data centre power needs—immediately and as a backup transmission to other energy sources.

Andy Fenton, VP Sales and Marketing at Telehouse Canada, which provides colocation and connectivity solutions in datacentres. He points out that the data centres are strategically located, and his portfolio of clients is based in downtown Toronto, where connectivity is closest to the population.

The specific needs could range from crypto exchanges, which can accommodate intermittent energy, to battery plants and EV manufacturers. However, Fenton is clear that reliable, uninterruptible energy will be required to power AI as inference becomes the norm. He emphasizes that, until two years ago, there were no inference cloud players. LLMs enable real-time inference by leveraging trained models to make predictions or generate outputs from new, unseen data. The computation required to process the query and generate results must occur at the device's edge.

Pat Dalzell, VP of Corporate Affairs at Bruce Power, Canada’s only private sector nuclear generator and powering 30% of Ontario’s annual power needs. He points out that the return on investment is clear. Growing demand will accelerate the development of large-scale power generation options to keep the lights on. However, he emphasized that both reliability and affordability are necessary. Demand remains an unknown factor that will determine affordability. For context, current demand per server rack = 130kW. He contends that for Nvidia, a 700kW demand is ‘mind-blogging.’ Attempting to develop for this level of demand is impossible. The needs of AI today did not exist two years ago, and there was no feasible way to anticipate the impact of these changes on infrastructure accurately.

Who will bear the cost?

Ultimately, Dalzell points out that supply and demand will determine the cost as they plan Ontario's power needs. Farmer adds that they will have a better idea of demand over time, but they must update their decisions to manage the risk ahead. While nuclear refurbishment investments have gone ahead, the decision on whether to build them is still pending. Currently, there is enough competition for available power, but to support population growth over the next decade, it will remain scarce until the larger power infrastructure is in place.

These stakeholders agree that there is a need for policy and industry collaboration to align, primarily for economic development. Some are skeptical that regulation will play a role in the short term, but all realize that it will take time to play, review and get those plans commissioned.

The key for these energy providers is to ensure they can meet the system's demands. However, if demand paces above forecast, there will be undue stress on the available space. Fenton, of Telehouse, intimated that the data centers demand the necessary energy and will likely be inflexible, as he states, “Data load systems are running towards peak load while models continue to churn… 27/4 and will not settle down.”

He argues that data centers will be unwilling to make agreements that would compromise their energy needs: “Workloads can go anywhere, so these companies will go to where they can get power to consumers.”

Ultimately, as per Dalzell of Bruce Power, “We don’t want to challenge the system to drive up the cost to the consumer. We don’t know where the load is coming from, but there will be substantial demand in the future — that is clear. We’re looking at our site as a potential demand center, and the potential for these demand centers will evolve as data centers grow.

What about Net Zero?

The question of sustainability and how an energy infrastructure that inevitably will put more stress on our resources, impact our environment and achieve net zero emissions needs to be addressed. Dalzell drew attention to Canada’s seasonal shifts and suggested the need for flexible demand for seasonal structures, especially during hotter weather, to lower the system's cost. With the global investment favouring the development of AI infrastructure, it is unlikely that data and inference center cloud players will be willing to compromise.

Farmer, however, emphasized the need for data centers to change their operating reserves. If there is less compromise, he asserted that AI may need to modify its processes to fix its own problems. Data centers are leapfrogging their own growth, and at the same time, decarbonization goals will mean moving away from the reliability of natural gas, especially in new developments.

Dalzell of Bruce Power noted that there are duplicative processes and that they can work with government regulators to streamline and accelerate them. In addition to the nuclear refurbishment, they are recapitalizing the supply chain to optimize the buildout of the refurbishments. Farmer added that it takes less time to deploy efficiencies and that should be leveraged in the short term. And while it is possible to add demand and load without increasing rates, the minute you build infrastructure to support the added demand, the cost to the end consumer will increase. What Fenton emphasizes is the need for a common voice to represent the industry to mobilize on the common issues.

What is clear is the tension between the demands of the energy system and Canada’s mandate towards its net-zero emissions goals. With Generative AI being the catalyst that fuels every aspect of society’s digital future, as Farmer put it, those powering these AI systems must also address the energy requirements, environmental impacts, and the impending resource constraints. This will be critical as the climate crisis becomes a near-term threat and a longer-term existential risk to human life.

Share System Malfunction

I created this newsletter to shed light on many of our systems… inherited societal beliefs and values that today shape our processes, procedures, and rules, and that, over time, have been accepted. They have been defined as norms. Why try to disrupt that which has worked for decades? Why question what society has broadly accepted?

Except that these systems are not working… not for everyone… and not for those for whom they were designed. The system is malfunctioning, and it’s exposing the glitches in the very mechanisms we’ve trusted for so long.

There have been attempts to rectify these discrepancies — from DEI programs that bring balanced, more representative perspectives to teaching our children the history of man, the good, the bad, and the ugly… to evolving investment systems that yield opportunities for the underrepresented… to active governance of artificial intelligence…

These are only a few of the rectifications to make our system work for the masses. They face their own uphill battles. I intend to write about these journeys. I hope you’ll join me.