When that happens, running a business starts to look like a videogame. Hundreds of autonomous entities operating across a complex environment. Agents will be working inside work devices. They’ll be talking to customers, they’ll be available 24/7. They’ll spawn new agents and combine old ones. They’ll have email addresses and Slack accounts. They’ll be colleagues.

But, how do you play this videogame? Hundreds of windows and tabs, for each digital employee, or for each department? Autonomous can’t mean no oversight. Humans are autonomous and we get oversight. When thousands of agents are making thousands of decisions a day, you can’t manage the old way, by check-ins and check-outs and quarterly reviews. You have to find a new way, manage by exception - scanning for anomalies, reviewing what broke, simulating what to do next. As my friend James Cham said, work used to be first person shooter, where you’re directing every movement and every shot, which is what we do today, and it might become more like Starcraft, where you have to move people and agents around to achieve your objectives.

And to do that requires a model of the business underneath. This mostly exists today in various people’s heads but rarely is explicit. We can’t even stay on top of our emails, much less a thousand or a hundred thousand workers. A big benefit of digital labour though is that you can have a precise state of the business at every point in time.

We have solved this problem before. When we needed to figure out how to train autonomous cars, for instance, we needed an environment that’s realistic and the ability to run “what ifs” in a controllable simulation. Waymo and Tesla built these as World Models. The equivalent for business already exists in the heads of management in every company. Every CEO is constantly running “what happens if I do x” in their heads. They just can’t operationalise it because there’s no ‘environment’ that reflects their business to run it on! World models already exist anywhere the environment is expensive, instrumented, and operationally constrained - factories, grids, airspace, battlefields, fabs, networks, wells, and warehouses.

What’s needed in the enterprise world is such a world model - an engine that knows the rules, tracks the state, understands and predicts consequences.

The environment would connect to the systems a company already runs, the information that is gathered, the agents it uses, and build a live operational model of the business. Scale it across companies and you have the training data to build a compelling environment and an even better world model!

There is no way to get to a world of AI agents as employees without something like this.

We can’t build this abstractly in a box. The real economy is complicated. We have franchise systems - hundreds of locations running the same playbook with local variations. Multi-site healthcare - clinics, urgent care chains, dental groups, all drowning in disconnected EHRs and billing systems. Professional services networks - law firms, accounting firms, consulting shops with multiple offices that can’t see across their own operations. Real estate portfolios. Logistics networks.

Forget the architecture for a moment. Maybe let’s take an example, one vertical - say a real estate company.

They have, say, 15 holdings across the southeast. Each one runs StorEdge for property management, QuickBooks or Sage for accounting, some CRM for leads, a work order system, maybe SoLink cameras. Multiple customer service softwares and a phone line. None of these systems talk to each other. The district managers have spreadsheets, updated manually. Understanding what decisions need to be made is cacophonous! They have dealt with this by having a few AI agents for marketing copy and CRM updates. They also have orchestration solutions and perhaps observability for those agents. The executives get monthly reports as a pdf.

Now, when all of these are either run by agents or you have agents helping, what you’d really want is not to see the tool-call traces of each one, but get a synthesised image of how the company is. What’s the ROI of doing certain actions. How will the outcomes of a decision flow through the company. What are the key things to be focused on right now? What actions need to be made for the best results, and what results even matter? Even when you’re just responding to the markets or the competition, each decision is amongst counterfactuals.

An enterprise world model would connect to all of it to try answer what happens next if you act.

• Say a competitor cuts prices in a submarket and occupancy starts dipping. An agent flags the dip and the model simulates the responses: match the price-cut and hold occupancy which might compress margins by X%, or hold pricing and lose Y units over Z weeks, or just increase marketing spend by $W and recover the gap. It can show the likely P&L impact of each path and ROI.

• Or, a district manager asks about a $60k roof repair. The model knows that this pattern of maintenance requests - three HVAC calls, a roof leak, a parking lot complaint - has preceded a $500k+ capex event within 4-6 months. It simulates the tradeoffs in the environment - approve and extend the asset’s life by X years, or defer and risk a larger spend later.

• Or, a property is converting leads badly. The model surfaces the stat, simulates decisions, and identifies that response time is the lever (like, say, properties where managers respond within 20 minutes convert at 2x) and simulates the impact of enforcing a 15-minute SLA, e.g., projected conversion lift, staffing costs, or net revenue effects.

Each of these is an action-outcome pair. The point is to learn which interventions produce which consequences, and that learning compounds over hundreds of companies, building the operational equivalent of what Waymo’s world model on top of a realistic simulation of every business: a simulation you can query with “what if?” before you commit to the road.

Think about what a COO’s day looks like once this is running. The agents already made thousands of decisions overnight. Her morning starts with reviewing deltas to see what broke, what improved, what patterns emerged that nobody expected. The model scores outcomes against baselines continuously. When she wants to try something - a different pricing strategy, a change in lead routing - she simulates it through the model and sees the likely impact.

The loop runs continuously. Management becomes all about triage and simulation.

There’s starting to be a lot of activity in this direction, building some of the core pieces.

• Orchestration companies are building agent governance and workflow layers - mostly hand-crafted agent hierarchies.

• Observability companies watch what agents do but don’t predict the consequences of doing something different.

• RL environment companies are trying to create structured training data from real operations.

• Enterprise platforms like Palantir serve Fortune 500 for bespoke implementations.

But there’s something holding these back, making them seem like little features. The key distinction is this, a world model. It predicts what will happen if you intervene. Which means all of these - orchestration, agent management, data integration, RL environments, continuous evaluations - are pieces of the same thing. They’re features of the enterprise world model. None of them, on their own, can answer the question: “if I do X, what happens to the business?”. And that’s what we’ll need.

There’s this constant question that echoes Solow, about where the impact of AI is on productivity or the broader economy. To not fall prey to that paradox we will need to do to the rest of the world what we’ve done to code - create an environment where we can see and test the impact of every decision and be able to simulate the effects of an action. To do this, we’ll have to convert messy, unstructured business operations into an environment, defined action spaces, evaluation criteria, and capture outcome data. And you’ll have to do this across thousands of businesses. That’s why model providers like OpenAI are paying to build this manually through programs like their Thrive Capital partnership, embedding engineers into portfolio companies one at a time.

An operating partner who walks into a company and sees how it works - that’s what is next to be built in software. If we want to build a one person unicorn, that’s what’s needed. To automate the economy, to give AI what the human has, a world model in their heads.

I remember the first one-person billion-dollar company. It wasn’t mine, I wasn’t working yet and was only an observer, and a distant one at that. But it felt exhilarating. A breath of fresh possibility, like any of us could do anything. A milestone in what humanity is capable of.

It lasted for a month.

The founder did very well for himself obviously, but within a matter of weeks someone else beat the record. One-person-unicorn became the 4 minute mile of company building, another rubicon crossed. Once the world knew it was possible it became inevitable. Because a world where one person can create a unicorn is also a world where another person can also create a unicorn. Maybe a day after, maybe a week, but pretty soon and it’s inevitable. And we saw the inevitable happen, four more in the next few weeks, till it became somewhat normal.

Entrepreneurship had already become a game in the 2010s as the saas boom made building big companies in short periods of time easily possible. The result was an incredible boom, many of them competitive with each other, with extreme dispersion in outcomes.

And now, when building became even easier, the equivalent to telling someone else to build things, it predictably got crazier. Not quite as easy as “make me a unicorn” but closer to it than what we’d had. Can you imagine if it was that easy? Everyone and their grandma would do it.

As the amount of effort we needed to put in to show the minimum of traction was reducing something had to shift to move us to the new equilibrium. If all people needed to do was be faster than others to ask a question, that’s a speed race to the bottom.

Once upon a time it was actually executing that was the bottleneck, soon it was project managing the thing you were executing. Then it was choosing the directions and making editorial choices about what thing you should create or run experiments on. By this layer of abstraction it was less about what could be made and more about what needed to be made. Because everyone could get AI to make almost anything it felt like but no one knew for sure what everyone else wanted.

My career started in earnest a bit after this. We all had eight monitors with running information streams from all over the company, and outside. I was called an analyst, because even though analyses had become cheap but accuracy hadn’t. Someone had to monitor the drones.

This was fine, actually. It’s not really what I thought I’d be doing but then it required me to think super fast and make a lot of decisions and keep on top of them, and try to automate some parts of those too. I liked it, sometimes it was even fun, though a lot of it was quite rote. I worked on the shipping industry side, accidentally if I’m being honest, that’s what was allocated to me, but turned out this was a pretty good window into the world. I had to keep on top of things from did the tanker break an engine part to like crude oil prices to atmospheric conditions in some strait.

Quite a lot of it was also dealing with competitors. I mean the normal stuff the AI could do, but the fun part was to confuse their AIs. Ships seemingly going the wrong way, or water displacement made to look fake, all sorts of tricks, some legal and some not. We all had the same machines but adversarial games are more fun, you know?

The rest of it, to look at the machines themselves and react quickly when necessary, that was okay. A hard job, much harder to pay continuous attention than to actively do things, at least for me. A lot of it was also reactive, and not just because of the adversarial problems. Like, even though the ability to analyse and communicate anything became instantaneous, it hadn’t necessarily helped in making the right decisions all that often yet. What it did mean is that if you were making a mistake, you got to make it faster now. There was no escape from Hayek. Every part of every company became more efficient in doing things even as knowing if you were being efficient in the right direction remained a mystery.

It felt like playing a videogame, highly stressful. You were always on call, always trying to figure out what broke and fix it, or find ways to game around what someone else might do. It was hard.

One thing that eventually helped me a year or two in was that corporate secrets stopped existing. Or at least they didn’t survive for long. Anything anyone did could be reverse engineered pretty quickly. As soon as things turned more adversarial this was probably inevitable. Who knows maybe it might have been just at the same rate as AI in the 2020s or software in the late 2010s or the entirety of Chinese manufacturing knowledge before that but it didn’t feel like it. Living through it felt like sailing the high seas, pirates and privateers at all sides.

Despite the respite brought by the new world, I ended up quitting that job another year after this. Just being alert for hours on end every day was hard, and no amount of laying around or drinking cured it. It had also meant that long undivided time to think and come up with ideas on your own was a dying art, and I had dreams of contributing to the world this way.

I understand why it was hard though. It’s hard to spend a decade coming up with a new idea for a car when you could just steal your competitor’s ideas that already worked. Why take a risk. The world became much less divergent. Sure, people did try to do things that were unique but like the Hollywood of yore everyone just copied from everyone else while occasionally a great piece of cinema broke out from nowhere.

I did feel though the size of individual companies were shrinking on average while the top exploded. When I was looking for jobs I kept seeing this. The one I ended up working for was tiny, maybe about 30 people. It was either this or just go independent contractor route. The Coasean bargain that made some companies larger broke apart, there ended up being a much larger number of individual contractors and smaller companies than were feasible before. Even I thought about joining their ranks, which would’ve been a bit more work.

Identifying and capturing those people is the most incredibly important piece of leverage. Some of the largest companies ended up being the conglomerates made up of these people who individually wanted to go and help them figure out answers to problems that they could not answer otherwise. There were other options too I suppose, like the original AI lab model which by now had disappeared, they had many fewer employees than those old companies of their size, but did run a large network of arrangements that would make the economically dependent population number in the hundreds of thousands.

As the AIs got better firms soon started calling for a new type of role, an “analyst”. They would get brought in to do a particular task once, whatever it took. I started out doing this for worldwide logistics networks. Deciding when AIs started going in a loop against the others it negotiated with for prices and goods or routing decisions. Which factories needed to be built, and which types of models to analyse for those. What pieces of data we were collecting were actually trustworthy, when the world had changed enough that our very model had to shift.

We all had something installed that could read and analyse everything that was done on the machine, to help us do the job better. But pretty soon, at the end of it, the AI just learnt from what I’d done, every part of it, and be able to just do it from then on.

Every job was the last job. What is done once got done for all time. The progress bar would go from 0 to 100 as you did it, and once done it remained done.

I remember getting paid for one of these jobs, about shipping logistics; it took a week and I made as much as I’d made in a year before. The value was high, and I was too stupid to think about “terminal value”.

These gigs themselves were also better as the AIs got better. It was much less stressful than frenetically monitoring it yourself like before. Mostly supervising other AIs, sometimes other people, sometimes other people supervising AIs. I hear of the days when people used to have the same job for 40 years and it sounds like a fairy tale because people today have jobs for 4 months. If they’re lucky with that they get to own a piece of the machines.

Some friends who were smarter started to ask, what even is a “job”? And I too worried, things of all my projects, would this disincentivise deep thinking? In the end it did, a bit, but the market corrected as time went on, because capital had to find ways to protect ideas, especially since many of them could be now reverse engineered. A lot more secrecy, for a short time could be monetised, because soon after you knew it would be known. I wasn’t at the cutting edge of anything enough that I could ask for a billion dollars and quiet time, but some were, and they prospered. Even the whiff of a good idea was enough.

This was the hardest part, because until this point all jobs people did throughout their lives relied on the jobs themselves being somewhat predictable day to day. Nobody except maybe some CEOs during a particularly tumultuous phase had to do completely different things hour after hour, day after day. What it meant to get paid for a few minutes of your time, a form of knowledge transfer, instead of getting paid for your actual labour, was enormously complicated, and societally destabilising.

Nobody quite figured it out but much of it ended up similar to contract work, where the work was timebound and sporadic and you got paid a premium for this gig work. These companies aren’t really companies, they mainly “collect” many of us to save us the trouble of searching. A thin wrapper between my agents and those that want my efforts.

The goal of doing all this, of your career, what constituted actual success, was to own capital. Most of my work has been in turning my personal labour into capital. And it was still good to own capital. It always is. You could deploy it and see people line up to take it and build things that would change the world in months. After all, building physical things remained a problem. Logistics remained a problem.

Anyway I don’t know if this is worth it to be honest anymore. What’s the point of having cash that you could give to an entrepreneur to build something, when others with capital could also do the same, and make damn sure that neither one of you would make much money without getting lucky? If the true skill of my labour is not differentiated enough, then what’s the point of just pouring more? Won’t everything just become highly competitive but undifferentiated, like in the commodity markets?

Those markets, despite the product being literal commodities and the process being the only differentiable part, mostly survived because different places have different regulatory structures and codified preferences. Which in turn determined who ends up being the marginal producer that can then be refined or transported or used. And so on and on.

The only choice was the robots, which were plentiful, everywhere. Robots gave leverage, a person could use it to help teach it how to do certain pieces of work and then supervise it thereafter. This held just as true for those who manufactured the robots as those who used it. The idiot index might have been a useful target to aim at after all. And with robots it’s no longer the case that you need hundreds of thousands of people in these industries. Energy and land remained bottlenecks, because you could always use more and they could always be cheaper, but the world didn’t oblige to the exception of everything else.

Don’t get me wrong, there was innovation to speed these up, but ultimately the decisions of what to invest in, what to create and what to make faster all turned out to be market problems as opposed to analysis problems. And market problems are wicked, and you cannot solve it just by running fast. It requires actually traversing the demandscape and banging your ideas against the real world, there are no shortcuts.

Even for those who had abundant capital, figuring out what portfolio of bets makes the most sense remained difficult because the response required information gathered from all over the world. And compared to how long a ship takes to build and sail, the decision on what type of ship to build didn’t take that long at all, even though it mattered the most. Those who claimed to have some insight in how to help folks figure this out lived quite well.

When I look around these days though, food is cheap, goods are cheap, learning is cheap, health is cheap, and if you want something more the amount of labour you need to provide those basics is miniscule. It all seems pretty nice. The biggest surprise from the heady days when the future was utopia might be that the pace of scientific discoveries changed, but not too much. I’m no scientist so I couldn’t tell you why this was the case, but it’s true. We did get better food and medicines, but string theory remains a theory. There are flying cars, but nobody’s riding rockets to the moon. I think maybe the discoveries just maybe weren’t bottlenecked by our inability to do analyses in the first place? We could run a ton more tests now but there are only so many problems we could brute force our way through. And once the low hanging fruit got picked over in the early 2030s, we sort of all got stuck again. Like how fundamental physics was in the late 20th century I’ve heard, stuck needing new ways of conceptualizing the world.

Attention is still capped because there are only so many humans. There are only so many hours in the day. One person’s gain is another’s loss. If you’re reading an essay it means you’re not reading another essay. Zero sum. The most drastic change was what happened when the only signalling that was costly was individual presence, since everything else could be faked.

For most of us who are at least somewhat young, in the last few years the world took a turn and became a lot more analog. Many of us don’t remember a life unmediated by the digital realm, but that was changing. When nothing you see or hear could be easily trusted then what remained were small enclaves functioning like private clubs. If you couldn’t be trusted you couldn’t enter. But even there, the rules had to become draconian because our daemons, our digital twins, our agents, could penetrate it if we had permission. Hence, physical presence.

This physical network also meant agglomeration, which is why I moved cities. Not for commerce, or work, but for my social life. I mean, it was either that or live a nomadic existence, traveling the world and seeing others wherever they are.

That’s mostly what I do now, while doing the occasional decision support job in areas I had learnt quite a bit about over the years. I have to keep spending some time every day making sure I keep up with the latest, but it’s fine. The jobs are sporadic, but it pays a good living even though you always feel like the other shoe’s just about to drop. The remaining time I have, which is most of it, I spend making entertainment for others in ways that are, for now, hard to imitate. There are physical plays people put on now that I go to sometimes, participate in sometimes. It feels good. This is life.

Last week was a bit crazy. In many ways, but specifically with AI. For those who were blissfully unaware, The Department of War picked a fight with Anthropic over the ways they were allowed to use the model. The fights, as is often the case with the administration, got nasty. Anthropic said no we won’t budge, DoW got angry, and threatened to cut them off and declare them a supply chain risk. A few hours after, OpenAI said they managed to get another deal, apparently a better deal, and one such that any other AI lab can also avail the same terms.

So naturally everyone is angry. Anthropic is angry because they were declared an SCR. DoW is angry because someone tried to force their hand. OpenAI is angry because everyone seems to call them opportunistic ghouls, more or less. The media, both independent and institutional, loves it because they get to play their favourite game of good guy-bad guy.

I really didn’t want to write about this. But it is important, contractual disputes are actually interesting, and sometimes that deserves an explanation.

The facts are roughly as following, Anthropic had an agreement via Palantir to work with the DoW. They’ve been doing it since mid 2024. They made an different, supposedly unsafe version of Claude to do this. Somehow over the last week, they got into a tiff with the DoW, supposedly over some red lines they had (no mass surveillance and no autonomous weapons) or rather who will get to say what those lines are and when they’re crossed. OpenAI signed a contract which had those same red lines and an enforcement mechanism.

Now, the claims are roughly as following, noting that nobody knows if they’re true. Anthropic asked questions about the Maduro raid where it was used, and the DoW got upset. DoW asked a hypothetical about how to do autonomous missile defense using Claude, and got a non-answer that they’d need to talk to the CEO and they’d ‘work it out’. Anthropic asked for their red lines to be enforced by enabling them to act as the party to approve it (you’d ask them if you had a question). DoW wanted language referring to “all lawful use”, basically saying if what they’re doing is legal you can’t tell them what to do, especially during operations, i.e., you can’t tell them to stop doing something in the middle of an op. OpenAI said sure, we agree to all lawful use, but note these specific laws and regulations, and we will control the deployment of our models, using our people, since we know what it can and cannot do, and help you guys out.

Every point above is a claim, and we have no real proof. People are desperately trying hermeneutics of the OpenAI position and blogs, but honestly it feels kind of silly since we simply don’t have the data to conclude they did a bad thing. Or, particularly silly, that they defected in a prisoner’s dilemma. What we do have, are concerns. Concerns like:

• Didn’t OpenAI just accede to “all lawful use” and therefore allow mass surveillance on Americans?

• How can you let a private company tell the DoD, you should ask us if you’re violating any of our red lines during an operation?

• Why did OpenAI sign an agreement so fast anyway, surely they just said yes when Anthropic said no?

• What do those red lines even mean?

• Also, Anthropic and OpenAI seemed to have the same ones, how can that be?

• Can’t the government or the DoW just make up its own laws as it does anyway? Who can stop them?

• How can you guarantee this means the DoW won’t cross any red lines?

• What do technical safeguards mean, how are they enforceable?

• Etc…

Many valid questions, but I refer you to the openai blog, dario’s written statement, and Sam’s AMA for various points of view on them. They do cycle between thinking of the government as Leviathan, an entity you cannot negotiate with, only appease, and thinking of the government as Loki, a trickster you need to subdue or overpower.

My interest though is broader than who said what to whom, or who’s virtuous and who’s not, as I think yours should be. It’s not to relitigate the facts, but think about the following:

1. What are the right safeguards to put in place when a piece of technology is deployed as a tool by the DoW?

2. How do we enforce any of it?

Let’s think about this for a moment. Imagine you are dealing with the government for a moment as an AI lab. They want to buy your AI, and you want to sell it. How would you safeguard it?

You know that plenty of things are legal, but not “good”. So what’s the choice here? You could of course just try not to deal with them at all. But once you decide to do it, there’s either you need contractual provisions you think they would adhere to and execution guardrails you can have some control over.

You also know that plenty of things are legal, but impossible. You cannot build a stairway to the moon regardless of the fact that it’s legal. Saying “I want GPT to build my defense strategy in Iran” would be such a thing to ask, you can ask it you won’t get good answers. The AI labs both want to say that.

So, you have to write some provisions into the agreements. Of course, the DoW can buy anything it likes, and you can add constraints on the stuff you’re selling, but they have to be clear. This is true of all contracts but of course with defense it’s even more important. For the same reason that it’s important in a hospital. To take a silly example, most models will rightfully have safeguards against violence or nudity, but imagine we also need them to treat burn victims. It can’t be a blanket no, you need to figure out some way to separate what’s allowed from what’s not, and before it gets deployed ideally so that you’re not doing this live when someone’s in the OR.

Which is to say that whatever they’re using, the lines have to be clear. Either some things are allowed, or they’re not. As little ambiguity as possible. The DoW would also want the power to determine courses of action, and can’t leave operational control in the hands of another. This is the now infamous scenario that someone apparently painted in discussions with Dario, if a missile was heading towards the US would they be ok to use Claude to defend.

Apparently Dario said they’d work it out, and also later said they can carve out a missile defense aspect from the contract, but you hopefully see the problem. You could easily come up with a dozen other scenarios, so do you just keeping coming up with them and then taking them off the contract because ‘that seems fine’?

The other “red line”, about mass surveillance, is similar. What does that mean? You ask a dozen people, as Zvi did, you get a dozen different responses. Going from a vague feeling to something that’s specific is really difficult.

Now the DoW’s position seems to be that let’s just do it according to the law. The law is clear enough, or at least clearer than a goal that we might share. Laws are an operationalisation of principles we hold dear.

But what if the law has loopholes? If we disagree with the law? You still have to find some ways to make that clear, but honestly you either draft a contract airtight enough to solve for those, or you have to believe that your counterparty will obey the law. You can draft “permissions-based” (enumerated) vs “restrictions-based” (negative list) provisions, if you’re clear enough. And it makes sense to have explicit contractual red lines, even if unenforceable mid-operation, since they create legal exposure and political cost for the government if violated. But they aren’t clear though, then no contract can save you, and saying “I will decide” will not necessarily break in your favour.

Terms like “reasonably requested” or “as appropriate” or “reasonable doubt” are standard legal terminology precisely because you can’t nail down every eventuality on every contract, these capture some combination of norms and prior history to gesture at the types of things that will be ok and types of things that won’t be.

Because the only thing that matters is whether you have any visibility into their actions in the first place. The Anthropic deployment was of a separate version of Claude, under a different ToS, deployed by someone else. Which means, they probably had limited visibility into what it was being used for. Which also means the only way to enforce any standards is to codify things quite a bit upfront - it’s like doing an on-premise installation vs saas.

OpenAI’s contract on the other hand seems to have been hand-in-hand with their own teams of FDEs and something they call a safety-stack (guessing cloud deployment of their own models and some checks therein, I don’t know). Which means they have much more operational visibility into the model usage, which also means they have the leeway to negotiate if the usage of it started to violate any of their ToS.

I have no real opinion here on which is better. Contracts are not inherently all-powerful, they’re only powerful insofar as they can have oversight. I do have an opinion that neither is inherently superior to the other, even if what we know about them is accurate, which might not be the case. One has more contractual protections and limited operational visibility, the other has lower contractual protections and higher operational visibility. The first one relies more on trust with the counterparty, the second one relies more on execution control. Both rely on the existing legal system.

This entire saga seems to me like it was a personality clash rather than a contractual dispute. A version might well be: Someone asked a question about Maduro raid. DoW got upset they’re being asked. They posed a hypothetical. Anthropic’s response was bad, confirming DoW’s prior assumption that they’re trying to control the deployment. Which is why even though they were so close to being effectively done with the agreement the Secretary of War decided to blow things up.

To reiterate, it’s really bad to call Anthropic a Supply Chain Risk. This is just not true. It is eroding yet another norm about what capricious governments could do at a time we should not be eroding it, we should be strengthening it. It is perfectly fine for Anthropic to have rules about how their AI ought to be used. It is perfectly reasonable for DoW to say nah that’s not going to cut it, I don’t want to ask for permission.

But what is true is that this should not be much of a surprise considering the constant rhetoric over the past few years has been that AI is a power like no other. It’s like nukes, but times a thousand. We need regulation. And when an industry repeatedly calls out for oversight, asking for someone to make the rules on how it should be used, you cannot be surprised when the Defense department take that seriously. You cannot be surprised when they make up their own interpretations of what ought to be done, because you were insufficiently prescriptive. They will listen to your articulation of any red lines and wonder, what do you mean you want to tell me how to use the mega-nuke-crazy-power that you yourself are saying you don’t know how to control?

The US has nationalised or regulated whole industries for simpler reasons. Telephone lines, rails, steel mill attempted seizure, these aren’t small things. And that’s not to mention the times the government has threatened to do this, from JFK to FDR.

So if you think AI is important we’re going to see more of this. You simply cannot call your technology a major national security risk in dire need of regulation and then not think the DoD would want unfettered access to it. They will not allow you, rightfully so in a democracy, to be the arbiters of what is right and wrong. This isn’t the same as you or me buying an iOS app and accepting the T&Cs.

But it’s also true that a corporation acting as a bulwark for democracy against the government is fundamentally weird, even if true. Democracy is incredibly annoying but really, what other choice do we have! What we don’t have is a reckoning with the power that is now reality.

I am extremely uncomfortable with the fact that we can just purchase commercially available data on almost everyone. I am also somewhat uncomfortable that the future of war is going to be autonomous though there are days where having Claude or GPT decide where to bomb seems better than an average 22 year old. I’m uncomfortable that in the pursuit of absolute security we have effectively given up our privacy, and all that remains are small shreds that only sit with a couple of large technology giants. I’m uncomfortable that the few shreds of privacy that did exist can now be reverse engineered away using pretty normal AI tech.

I also am not sure there’s a way out where we would ever have digital guarantees of privacy. I think our children will think that a quaint old notion. “What do you mean, I can of course just ask my AI to analyse a bunch of information and figure out who ratmonster2024 is.” The work that only NSA used to be able to do a couple decades ago is probably within the grasp of the average startup, if they cared. Genies don’t tend to go back into bottles, and this one has powerful forces keeping it out.

The future will bring these questions to bear, much faster than anyone might expect. The current world survives because a lot of analysis is effort-bounded. If that’s gone, a lot of things we previously assumed secure will also go away. This is coming, whether you want to or not. The best part of last week is that the issue became higher profile, again. But bringing attention to the issue is only the first part. Unless we know what we want to do with the attention, tribal politics is going to overwhelm it all.

I had a conversation with and an august panel last week. It was really really good, and you should check it out.

Exponential View

🔮 Where the human ends and AI begins

This is the first AI Vistas discussion, a new series hosted by Exponential View where I bring people I trust into conversation around one hard question, because together we can see what none of us would see alone…

Read more

a month ago · 62 likes · 13 comments · Azeem Azhar, Nita Farahany, Eric Topol, Rohit Krishnan, and Nicholas Thompson

1. Mexico is like if India was richer, things were cleaner, while being much (much!) more unsafe. This showed up for me almost everywhere I went, often in the background, often not. For instance, this means that while in India you will see a lot more spaces for the rich or large luxury malls, in Mexico it feels like those are hidden away inside secure compounds. In fact the only place I saw this easily accessible and displayed was in Cancun, which is as if the Mexicans built a tourist place just for the Americans and made it look like Dubai.

2. I was shocked that Mexico City still has a murder rate 1/3rd of NYC in the 1990s. Turns out this ignoble list is also dominated by Mexico.

3. I continue to be just constantly amazed at how safe India is. It has no right to be so, it’s poor, ill organised and the justice system moves like molasses. I first had this thought in Nigeria, and have repeated this observation in too many countries to name. Central and South America look likely to only exacerbate this question.

4. This is particularly germane in Mexico because Mexico City reminds me a lot of Delhi, albeit with somewhat worse roads, less people, and far cleaner sidewalks. And entire squadrons of police cars with visible guns every block or two in all the tourist friendly areas.

5. An interesting aspect that I had never considered is Mexico used to be bigger than the US when it owned most of the US’ current southwest. The country still seem to remember this in their bones. They’re 130 million people but feels much larger. The weight of most of mesoamerican history centers it. They have a Place in History, writ in capital letters in the national psyche.

6. The level, variety, and affordability of street food remains one of Mexico’s major success stories. Plentiful, tasty and cheap. I largely prefer it to restaurant food. Tlayudes ftw.

7. Going through the Zocalo in Mexico City is a full body immersive experience, and not one I care to repeat. On the other hand it is massive, disorganised in the best way, and sells anything and everything you can imagine. We got lost inside it and had to trek a dozen blocks in a randomly chosen direction to get out. We realised this after calling an Uber and waiting 20 mins before realising it’s never going to make it.

8. This is also a plus, because just like the lack-of-zoning-success-stories of almost every country except the US, it makes Mexico City undeniably attractive to every American, who of course love mixed-use easily walkable cities as long as they don’t have to live in them.

9. This exact reason also makes Cancun the worst place in Mexico I visited, because it’s built for tourism, has a hotel zone, and fails my “Civilisation Test” which is the number of cafes in walking distance. In case you were curious, the winner was Oaxaca. Excellent coffee, and even better hot chocolate.

10. Mexico City truly is a cultural capital. Incredible museums, great art, great food. The Museum of Anthropology in CDMX is the best museum I’ve seen (‘n’ is very high here).

11. The main Cathedral is absolutely gorgeous. And being built on the remnants of the lake you can see the effects of the soil moving about as the cathedral is a bit slanted. The styles are more eclectic than you’d find in a European city, and more ornate than I personally like, but worth seeing.

12. Walking among the Aztec ruins next to the Cathedral is a quasi religious experience because they’re so well preserved. The feathered serpent, Quetzlcoatl, is everywhere, encircling the plazas, out of the walls, surrounded in parts with forms of corn and shells.

13. As usual I found the fact that until recently tearing down an ancient monument and building another gorgeous monument to be normal and not at all noteworthy, to be interesting. Something we can learn from.

14. The Aztecs took their iconography and religion seemingly from Teotihuacan, which is an hour away. It’s an older civilisation, 600 years before Aztecs, whose traces they clearly discovered and were influenced by but knew little about. They didn’t know who they were, what their society was like, what they called themselves, nothing. So they, rather whimsically, named it Teotihuacan, the place where gods came from, adopted many of their gods (or so it seemed to me), for instance named the feathered serpent Quetzcoatl, and generally lived a grand life of military conquest for a couple centuries until Cortez arrived.

15. I can understand why. Teotihuacan is extraordinary, and the Pyramid of Quetzcoatl in particular is magnificent. Considering they didn’t have metal or pack animals this is all the more impressive. The ability of humans to accomplish incredible things at scale never stops continuing to amaze me.

16. I have not been able to make up my mind about the import of human sacrifice and how much it’s true/ false/ exaggerated compared to other historic cultures.

17. Driving in Mexico City is very hard. Half the roads are tiny and don’t even look like roads. The green signs that show the roads and destinations often had three names none of which matched what Google maps said, so it was entirely visual navigation. I am now ready to drive in India.

18. Mexico City also has cable cars as a core mode of public transport, which I hadn’t seen before, and looks wonderful especially when stuck in a traffic jam. I wish the US had these, or indeed any public transport. I tried to take one but it was night and gpt recommended the amount of changes I’d need to make to take a ride was not safe and I shouldn’t do it. So I had churros and cafe de olla instead.

19. As my 8yo observed, the infrastructure got better as we went from Mexico City to Oaxaca then to Cancun. Curious.

20. Oaxaca is a jewel of a place. Fits in your palm, highly walkable. High civilisation score. Great food. Great cathedral, though the churrageruerisco was not the best of its type, didn’t come together cohesively.

21. The street food is plentiful and good. The speciality is mole, a particular type of sauce with mixed spices, and chapulines, fried grasshoppers. Apparently delicious when mixed into butter and eaten with bread.

22. Oaxaca also had the highest density, originality and quality of art I’ve seen in a city since

23. There’s plenty of prehispanic food and drink about. Tejate was meh to me, though a latte tejate I had at a market was extraordinary. Generally I remain a fan of modernity, we’ve perfected much of what history revered (and made them better).

24. Monte Alban, an hour from Oaxaca, is worth visiting. Zapotec built, on top of a hill. Gorgeous views all around. The guide told us when it was built and during the heyday it used to have 9 months of rain, so the water would flow down to the sides of the hill through channels that were cut, and this would supply water from the priests to the commoners. But the water dried up during a long drought lasting a couple decades, people lost faith in the priests to bring rain by praying to Tlaloc, and folks left. So it goes.

25. The burial rituals were fascinating, they would put the body in a small enclosed space for 4 years, shut tight so no smells would escape, and then would remove the bones and put them in an urn. If more people died they had different spaces like this outside the house.

26. The various pedestals and spaces had holes below for priests to show “magic”, disappearing and reappearing, as the guide told us. I am personally suspicious of the “people in the olden days were easily fooled” argument, but am in favour of the “everyone likes and believes in rituals” argument.

27. The idea of worship starting with some seed of truth and then becoming a self fulfilling prophecy as those responsible for the worship taking matters into their own hands will never stop being funny.

28. Cancun was the least interesting part of the visit. It is also, at least the hotel area, not at all pedestrian friendly. It’s big tourist resorts or nothing.

29. Chichen Itza, a couple hours from Cancun, was remarkable. Their architecture shows influence from teotihuacan, from toltecs, and there clearly seemed to be trade and information routes between the lands. The Mayan civilisation at least per reading stood for 3600 years, which is an absurdly long length of time.

30. The cenotes are magnificent. Cenote Xkeken, was a particular favourite, it’s mostly underground with only a shaft of light coming down.

31. The fact that Mayans ruled for so long in such a dry place with the main water source underground feels quite bizarre. Though once you rationalise by the number of inhabitants maybe it’s fine. Chichen Itza had around 40k, 5x less than Teotihuacan, itself less than Tenochtitlan, and none of them had decent water supply. I do not understand living life in hard mode for that long.

32. One reason though for the longevity of these civilisations might be survival bias, because but he time a lot of monuments got built without mechanised power it’s already a couple centuries. There’s a funny comparison to be made right California HSR here where we’ve horseshoe theoried our way to construction but I leave that to someone else.

33. The beaches near Cancun are very good, especially Cozumel, the island where Hernan Cortez first landed. Sting rays and nurse sharks played in the shallows next to our feet at El Cielo. But I’ll be honest I still prefer the beaches of Southeast Asia. Thailand cannot be beaten.

34. For the number of civilisations that roughly lived side by side at different points in Central America is really impressive. I got GPT to make me multiple maps and websites to help understand this better.

35. This trip without LLMs would’ve been about 30% as good. Everything from planning to asking about cafes and restaurants to dealing with zocalos to hotels and snacks and history and geography and pretty much anything we wanted to know or learn was made better by GPT, and sometimes Gemini.

36. Again the sheer number of extremely heavily armed police present in nearly all parts, including highways, was quite striking. They stopped cars at night, frisked folks, and generally were a loud and constant presence. Is this signaling or actual deterrent? Unclear, but everyone states the importance of being sensible and safe.

37. A substantial proportion of tourists to Mexico City and Oaxaca were Mexican, I think. As a consequence it’s not English language friendly, though again with Google translate and ChatGPT it’s not hard to travel.

38. I was told by the tourist guides multiple times to not call it Gulf Of America as a form of protest. Everything is politics.

39. Overall I really liked it, though I understand better why people who don’t have easy access to Asia, like Americans, like it so much more than I did. When it comes to food and markets and the general feeling you’re in a “free” city with limited top down strictures on life, this is the only real choice from North America without braving a really long flight. But I know, or rather I feel, for those you simply cannot beat India or Japan, which are also significantly safer, and have great food and history. Similarly for beaches I’m still a fan of Thailand but by a thin margin. That seems to be the primary motivation for most Americans I know who have gone to Mexico, which seems quite shortsighted to me. Because when you combine all that with its long history and culture, Mexico is pretty great.

This has become part of a series of essays, evaluating the new “homo agenticus sapiens” that is AI Agents. Part I was seeing like an agent. Part II is why the agentic economy needs money. And this is Part III.

Whitney Wolfe Herd, Bumble’s founder, recently described a future where your AI chats with potential matches’ AIs to find compatibility. Say what you will about AI being involved in your love life, but this is one domain where AI agents can potentially have large returns: the dating/marriage “market” is the epitome of the type of high-dimensional matching problem that Herbert Simon identified as impossible for people to optimize. Rather than optimising, Simon argued people engage in “satisficing”, i.e., settling for good enough.

Why would AI agents be useful here? Let’s start with how most markets work. Hayek’s big insight–outlined in what he called the economic problem of society–was that prices do an incredible amount of work. They compress a ton of information such as preferences, costs, scarcity, expectations into a single number that acts as a sufficient statistic for value. When you’re buying oranges, the seller doesn’t care what you’ll do with them. The price coordinates the transaction and that’s enough.

But prices work best when the transactions involve commodities. When you’re buying some oranges, the seller doesn’t particularly care what you’re going to do with them; you don’t need to convince him that you’ll take care of the fruit. The price does all the work in coordinating that transaction. Matching markets are conceptually different. You can’t just choose your spouse, your employer, or your college: you also have to be chosen by them. This is the domain that Al Roth, the 2012 Nobel winner for “the theory of stable allocations and the practice of market design,” spent most of his career studying. Roth showed that matching markets require careful institutional design; this design includes algorithms, timing, and the right rules to get the market to “clear.” His deferred-acceptance mechanisms now allocate medical residents to hospitals, students to schools, and kidneys to patients.

But the efficiency of matching markets hangs on the ability to elicit a person’s preferences, i.e., that people can express their rank orderings over potential options. But what if people’s preferences don’t fit in dropdown menus or are difficult to articulate on a standardised questionnaire? Peng Shi studied in his excellent paper “Optimal Matchmaking Strategy in Two-Sided Markets.” He looks at online platforms that match customers to providers using a variety of matchmaking strategies, from searching one side of the market to centralised matching that allows for back-and-forth communication.

Shi found that centralised matching works beautifully when preferences are “easy to describe,” i.e., straightforward to elicit using standard questionnaires, but breaks down when they’re contextual, idiosyncratic, or otherwise difficult to express through standard techniques. This is why many platforms still make you search. You want a contractor who shows up on time and knows your budget–this is easy–but you also want someone who understands your tastes in postmodern living room design. Good luck expressing that on a dropdown web form.

Here is where Large Language Models come in. They are fantastic at turning any unstructured piece of information into better structured matching. They’re also eminently scalable, enabling Coasean bargaining. But scaling things brings with it more coordination problems, too many agents negotiating with too many other agents is noisy. So what type of an institutional setup would make most sense to install here, to make this work well?

That’s what we sought to test with our experiments. The question being, could we figure out how and whether LLMs can help in matching markets where preferences are “hard to describe”? Can LLMs actually elicit the dispersed, hard-to-articulate preferences better than standardised methods? And if they can, what happens when LLM-based agents are available to everyone in the market?

Now, there’s some recent work on the topic that suggests guarded optimism that this is possible. Very new work by Ben Manning, Gili Rusak, and John Horton show that, when parsed through LLMs, short natural-language “taste descriptions” can be superior to standard questionnaires for eliciting preferences when the option set is large. They run an experiment where people write a few paragraphs about what they want in a job and then rank between 10 and 100 options (depending on the condition). Consistent with Simon’s conjecture, people’s ranking effort plateaus as the option size grows large; choice quality grows unstable as the consideration set increases. People get tired of ranking a ton of options and just start guessing. AI-parsed “taste descriptions” scale much better: once tastes are written down, the marginal cost of evaluating one more option is negligible for an AI agent. The advantages of AI-parsed matches are even higher in congested markets where people are more likely to be pushed.

But a theoretical paper by Annie Liang offers an important counterpoint in the case of a potentially complex two-sided matching market. She shows that when personality is sufficiently high-dimensional, meeting just two people in person beats searching over infinitely many AI representations. The noise in AI approximations compounds faster than the benefits of scale. This is a very cool result, and you should all read the paper in full–it’s that perfect type of economic theory that’s both conceptually rich and practically useful.

Ok, with that preamble…

Let’s run an experiment

We set up a simulated Hayekian marketplace with a whole bunch of digital shoppers, providers as AI agents.

1. Preference elicitation: Knowledge is dispersed in each digital shopper’s “head”: customers know what they want and providers know what they can offer. We want to know how eliciting the preference–either through the standard intake questionnaire or high-dimensional text parsed by an AI agent–can change the market structure for optimal results.

2. Mechanism interaction: When elicitation improves, can centralised matching beat search, and what are the conditions under which this happens?

3. Scale: We then check what happens when everyone uses AI agents

4. Institutional design: Finally, we figure out the right institutional mechanism to solve the resulting problems, and to maximise welfare

Preferences here are latent vectors in each agent’s head. Both the customer and provider agents have a true weight vector over some set of attributes (6 dimensions in this case). So elicitation changes the platform’s inferred w, not the true w. A standard intake is a structured form, and only exposes a few coarse priorities. The AI intake is free text, back-and-forth chat, and can be parsed in the platform’s inferred weight by a couple mechanisms - either by a rule- based algorithm or an AI agent.itself or , or via GPT parsing.

Figure 1 has an abridged illustration of the design and some results. There’s an appendix at the end of the essay in case you want to check out the details of the experimental design. But without further ado, here are some…

Results

First, AI-assisted preference elicitation improves matches across the board.

Figure 1: Experimental design

Figure 2

Second, as shown in Figure 2, AI-based elicitation changes what type of market design works best, and the conditions under which centralised matching can beat search.

Specifically, “Search” and “centralised” are the two different matching protocols we tested. Search means customers iteratively message providers in some ranked order until the matches ‘stick’. Think about how you would find a plumber–message folks, talk to them, iteratively until one ‘fits’.

Centralised is where the platform computes the shortlist for you, and clears a match based on mutually acceptable terms.

Once dispersed knowledge can be elicited and compressed into usable signals, the platform can centrally clear the market rather than forcing users to search. When knowledge can’t be compressed, search dominates because it lets users do iterative, contextual refinement in the loop.

The core object is the ‘ROI boundary’. If the per action attention cost is high enough, centralisation dominates–it just requires fewer actions. If the cost is low, search can dominate because it can “handle” more actions. This is the very idea of Coasean bargaining helping remove the boundaries of firms.

So where does the value of LLM-based elicitation actually come from? Is it from the back and forth conversation, or the ability to parse large text? As described above, we prompted all of the customers to write some free text about things they like and whatnot, and then used some rules-based parsers and some LLM-based parsers. There’s also the option for conversational elicitation via chat.

We thought the AI agents’ ability to ask follow-up questions would be the game-changer. Turns out though (see Figure 3), most of the value comes from the AI agent simply inferring more signal from messy text compared to the signal in a rule-based parser. This is consistent with the work of Manning et al. that we discussed above. This may of course be something specific to our prompts—perhaps one could obtain further gains by explicitly instructing the AI agents to engage in structured back-and-forth with the customers, and to do so in contexts where this would be helpful, but this was not the case here.

This highlights the utility of LLMs for extracting (potentially high dimensional) signals from unstructured data. Back in the day OKCupid used to make people fill out 90-100 questions to help match them with their potential partners. With LLMs, they might’ve been able to get away with writing a short essay and getting their Agentic Cupid to pull out the relevant information. Whitney is certainly on to something.

Figure 3

But what if people don’t really know what they want, does preference uncertainty matter? Whenever Rohit shops, he’s not sure of what he wants before he goes in the store. There’s a lot of noise in the process. Alex is a pure satisficer: the first item that meets a (very low) threshold gets put in the cart (usually virtual), and off to check out he goes.

We can test for that pretty easily here by introducing a bit of randomness into our shoppers’’ heads. At least in our setting, injecting noise into preferences doesn’t matter for the AI’s ROI all that much. We can still do centralised matching and extract a lot of value from that mechanism—as long as the preference noise isn’t too cacophonous.

What if everyone uses an LLM agent?

We had originally set up a pretty small marketplace. The centralized mechanism at this scale can be computed and cleared so we can run the experiment. But what happens when the scale explodes, both in the number of options and the number of customers potentially using AI agents? This is the problem matching platforms like Upwork are trying to solve: the option set is absolutely huge, but so is the potential customer base.

Every time a customer opens up a marketplace like Upwork, the number of choices just on the front page makes it hard to remember what they came for. Ideally AI-delegated agents can solve this problem: the user speaks or writes down what they want to do, the AI agent pings the platform, and the user is presented with the match. But what if every potential shopper had their own AI agent who wanted to message the providers on the platform? That’s a lot of agents doing individualised message sending to the provider inboxes!

So as you increase the number of customers with AI agents, the level of congestion rises significantly. Each customer agent sends a query to a provider agent’s inbox and it has to respond. Responding to all those agents takes a lot of compute. Here is what happens in our simulation (Figure 4): At full adoption, the providers’ inboxes flood with 5x the amount of requests, response rates collapse from 48% to 2%, and net welfare drops 88%.

Figure 4

Without institutions in place to scaffold the marketplace, a tragedy of the commons emerges: If everyone has an AI agent, it’s almost like nobody does. The paradox of plenty is real, and AI agents create their own version of Jevons paradox.

The need for institutions and scaffolding

What can fix this type of congestion? Prices!

As in a previous post–where we showed the importance of money in coordinating trade amongst AI agents–introducing a price mechanism recovers most of the lost welfare in matching. A vindication of Hayek’s deeper insight.

Specifically, we can introduce an exchange and money, such that the agents now have a pricing mechanism to signal their “strength of preference”. The idea is that the complexity falls because now not every provider and customer need to message each other. Prices capture a lot of high dimensional information in a single statistic, streamline a lot of that information, as we’d seen with the simulation in as we saw in the simulation in barter_to_money, complexity falls from O(n²) to O(n).

Figure 5

Pricing works! As shown in Figure 5, most of the welfare gains are recovered and the congestion issues are resolved. LLMs may lower the cost of expressing dispersed knowledge, but they don’t remove the need for institutional design to manage externalities. At least in our experimental simulation, the price system remains essential to solve the issue of complexity and congestion.

What did we learn?

If we think about an AI agent economy, we would want to know more about the mechanism that facilitates coordination. First, we have to ask, “If agents lower transaction costs, do markets just happen?”

In a previous post we looked at what would happen if there were a bunch of agents who had to interact with each other to trade, and it turned out that they don’t form markets spontaneously. In fact you have to do a fair amount of work before the agents are ready to interact.

Ok, if markets need scaffolding, what’s the minimal substrate that makes coordination scale? i.e., how will the agents coordinate amongst themselves? Will they be able to develop methods to do so themselves, e.g., through bilateral and multilateral negotiations, or will they need further help. It turns out that no matter how much you want to set things up just so, the agents will still need money and prices to trade efficiently. Even with the lower transaction costs and larger levels of compute, the coincidence-of-wants problem still doesn’t disappear - Hayek remains vindicated.

In this current essay we explore whether LLM agents can make centralised matching more efficient–we should expect marketplace consolidation in categories that were previously too heterogeneous for algorithmic matching, e.g., wedding vendors, specialised consulting, creative services. We showed that in “thin” markets AI agents help facilitate better match quality through centralised mechanisms.

However, if everyone has an AI agent, we still need a pricing mechanism to solve the resulting congestion and complexity problems that arise. Congestion is a serious threat at scale!.

So what is the broader take away from this essay, from the whole series of essays? For us it’s that AI agents work remarkably well when institutional design facilitates the interactions and transactions. Since direct instruction for every eventuality is impossible, the only way to make the AI agents behave at scale is to design the right scaffolding to facilitate coordination and exchange. This involves the creation of markets, and yes, money! If we can learn to design the “institutions” within which the agents operate, then we can help have them do far more complex tasks that we want. Autonomy, that’s the true prize!

Appendix: More about the design

Warning: wonky.

We constructed a simulated marketplace where customers seek service providers (contractors) across task categories that vary in how difficult preferences are to articulate. Each customer is seeded with true preferences represented as a 6-dimensional vector of weights (summing to 1) over provider attributes. A match is formed when both sides’ true values clear a threshold.

“Easy” categories include things like TV mounting or furniture assembly; preferences in these categories can be mapped cleanly onto standard form fields such as price, availability, and distance. “Hard” categories, such as ability to repair a historic staircase or a complicated asbestos remediation with specific guidelines, involve preferences that are more difficult to elicit using standardised questionnaires. We then see whether the ROI threshold changes based on how well the models can “elicit the true preferences” of the underlying actors.

The experimental intervention targets the preference-inference pipeline: how customer preferences get translated into data the platform can act on. The experiment varies the intake method (standard structured forms versus free-text descriptions parsed by an LLM) crossed with the matching mechanism (decentralised search where customers browse and choose, versus centralised assignment where the platform matches algorithmically). Match quality is computed as the dot product of the customer’s true preference weights and the matched provider’s attributes, minus any search costs incurred. All of this is summarised in Figure 1 below.

Figure A1: Experimental Design

This has become part of a series of essays, evaluating the new “homo agenticus sapiens” that is AI Agents. There was Part I, seeing like an agent. This is Part II. And Part III on what happens when we all have AI agents.

Sometimes I forget but we live in a future transformed by information technology pretty much across ever aspect. But one thing has remained largely the same: we still live in a world where the vast majority of economic transactions are done by people. If you want to buy a car, the process is largely the same as it was 50 years ago. You go down to the dealership and negotiate the best price that you can. Sure, you may have some extra information from doing research on the web beforehand - it’s certainly much easier to do comparison shopping with a supercomputer in your pocket - but the basic process of transacting with another human being has largely stayed the same.

One change that’s likely to come though is that there will soon be 10x, 100x, maybe more AI agents working in the world as there exist people. And as we have lots of AI agents working on our behalf, doing all forms of work, then there is a thesis that many of the frictions and information asymmetries that people face in markets may disappear if economic transactions are delegated to aligned agents, leading to a so-called Coasean singularity.

We’re not there yet though. Today’s agents are simply not good enough yet to act sensibly or without strict instructions. Many of the features of human-mediated markets still seem to be reproduced in AI agentic interactions. But as online spaces adapt to the promise of AI technology, it seems natural to think of how agentic markets will be organized. In a future world where we do have billions of AI agents, how would they coordinate with each other? What kind of coordination mechanisms would be needed? What institutions are likely to emerge?

And one possibility is particularly intriguing: will coordination still require money? Not in the sense of US dollars, but a shared medium of exchange and a hub/ clearing protocol.

Money, Money, Money

“Why money” has occupied economists going back to Adam Smith, who framed cash as solving what has since been termed the coincidence of wants. To see what we mean, consider a pure barter economy. Let’s say Alex is an apple farmer and Rohit raises chickens. If Alex wants chickens and Rohit wants apples, then Alex can just walk over to Rohit’s house with a bushel of apples and get some chickens in return. Simple. But what if Alex wants chickens but Rohit wants an electric toothbrush - he has no need for apples right now. Then to get the chickens, Alex would need to find a person who is willing to trade an electric toothbrush for his apples, and then come back to Rohit for a trade.

This would still all be fine if there was just one other person to visit and trade with, but what happens in a large market, with many (many) people who potentially have both an electric toothbrush to trade and want Alex’s apples? In order to trade, Alex needs to happen to find a person that both 1) has what Alex wants and 2) wants what Alex has. As very nicely shown in a paper by Rafael Guthmann and Brian Albrecht, the need to satisfy this coincidence of wants through finding matches creates complexity that quickly blows up as the size of the market increases. If the market is even moderately large, this complexity makes even basic transactions essentially impossible.

Ergo money. While the origin of money is a hot topic of debate (e.g., see David Graeber’s excellent book Debt: The First 5000 Years), the role of money in a competitive market is to solve the coincidence of wants. Money acts as a special type of good called the numeraire, where its only role is that it can be exchanged for other goods at pre-determined quantities. These quantities are reflected in the prices that each good is worth.

Going back to Alex and Rohit: one way to solve the coincidence of wants would be for Alex to sell his apples at a special place called market and then to use the money to purchase Rohit’s chickens. Rohit can then use that money to buy an electric toothbrush, or indeed any other thing his heart desires. Money eliminates the need for people to coordinate their transactions based on their current endowment (what they have) and preferences (what they want).

Bring on the agents

Okay, so money is necessary to coordinate transactions in an economy with people. This is largely because each individual can’t hope to have enough information on what everyone else has and wants to reliably engage in market transactions. Alex and Rohit are as yet, sadly, mortals.

But will this be the case for AI agents?

Agents do not have the same computational constraints as human beings. In theory, it may be possible to solve the search problem where the coincidence of wants becomes a non-issue. In that case, the agentic economy could eliminate the need for a key institution of the human economy. We decided to run an experiment to find out.

The experiment

First, the repo here. We can have N agents, with N goods, and each starts with its own good and wants another. There’s multiple rounds, one action per agent per round. Agents decide their course of action via structured JSON, and success simply means you get what you want.

The first question is about a pure barter economy. We explore whether LLM agents can achieve efficient allocations through barter at any scale, i.e., to engage in multiple bilateral negotiations to achieve gains from trade. The agents in the experiment have no real shortage of time. If this works then Coasean bargaining should be straightforward; goodbye money!

The table below has the results. What do we see? When the scale is small - when Alex just has to worry about coordinating with Rohit - all of this works. But as the number of agents grows, things start to get really difficult. By the time we get to even 8-12 agents the number of successful transactions drops to below 50%. And this is the absolute simplest setting.

Perhaps this should be expected. The problem is still O(n²) in complexity, which grows exceptionally fast as the number of agents grow. And if this isn’t just bilateral, but starts to include multiparty negotiations, it might become O(n!), which is far bigger for any number bigger than 3.

Ok let’s make it a bit easier for the agents. If they can’t talk to each other, since they are agents anyway, we should be able to give them omniscience. Enter Central Planning. There has been plenty of work before in the limits of bilateral negotiations, but we can test how well a “hub” structure can help. Does having a central planner help set the stage for better performance?

As the results table shows, central planning makes things slightly better, but we are still very much in a world of the Hayekian troubles. A hierarchy without a numeraire just isn’t enough.

Ok, we can continue looking at our human history to see what else we can do. In Debt, David Graeber argues that money emerged at least in part through state power, to enforce the paying of taxes in order to fund foreign wars. Before this, he argues, IOUs and bartering seemed to have worked just fine to manage the economy; the IOUs themselves became a sort of numeraire that could be traded in order to solve the coincidence of wants.

So let’s introduce_, Credits and IOUs. We can give the agents the ability to give each other an IOU and see whether providing the basics of credit allows them to come up with better ways to interact with each other.

This still didn’t help as much as we thought. There were a few segments where the transactions started happening, but they really didn’t start to work. Or scale.

Most interestingly, the concept of money didn’t emerge from this, not organically. IOUs didn’t become money. Even though in conversations LLMs all know that this is the smart thing to do, it did not emerge.

This was a bummer, because as with the prior research, what this shows is that AI agents do not yet come with the natural instincts of humans to turn IOUs into a numeraire that acts as a stand-in for money. They don’t even come with the same set of ideas as this sea otter.

Ok, let’s take the final step and actually introduce Money. We do this by creating an exchange where the agents can do bids and offers, and look at market outcomes. The results are stark: markets resolve at a success rate of 100% and much faster than through other mechanisms, at the rate of O(n).

One note is that this result presumes the exchange works without a hitch. In reality there will be friction coming from liquidity constraints, differential compute resources, etc. For example, in the N=8 run, the hub handled 23 inbound + 23 outbound messages and prices stayed fixed. And if regulations require that AI agents use different types of country-specific currencies, then exchange rates will complicate things further.

Discussion

To sum: An agentic economy doesn’t emerge automatically with even SOTA agents (who really should know better). Barter and central planning remain inefficient and infeasible, and money does not emerge organically even when credit and IOUs are introduced. At least in our setting, an agentic economy needs more top-down engineering to become efficient.

Previous work on agent-based modeling has explored what kind of emergent economic realities we are likely to see with rule-based agents interacting. The world of AI agents is fundamentally different. These agents act based on a huge corpus of human knowledge, with the underlying LLM models able to solve incredibly difficult problems on their own. These agents can plan, they can negotiate, they can code. And even with all this knowhow at their disposal, it’s interesting to see that they still appear to require top-down institutions to create an effective and efficient market.

As we transition to a more agentic economy, a key part of ‘getting ready’ for that world is setting up institutions for the agents. Like including:

• Identity and roles

• Settlement and payment

• Pricing and quote formats

• Reputation

• Marketplaces and clearinghouses

This is by no means exhaustive, but we wager that mechanism design for multi-agent work is going to be a rather fertile area of research for a while. Humanity went through millennia of evolution to figure out the right societal setup that lets us progress, that lets us build a thriving civilisation.

It is both necessary and inevitable that the world of AI agents will also need the equivalents, though the emergence of such institutions will likely be much faster given the millennia of human knowledge that we’ve already amassed.

Github repo here.

One of the books that I loved as a kid was Philip Pullman’s His Dark Materials. The books themselves were fine, but the part I loved most were the daemons. Each human had their own daemon, uniquely suited to them, that would grow with them and eventually settle into a form that reflects their personality.

I kept thinking of this when reading the recent NBER paper by John Horton et al about The Coasean Singularity. From their abstract:

By lowering the costs of preference elicitation, contract enforcement, and identity verification, agents expand the feasible set of market designs but also raise novel regulatory challenges. While the net welfare effects remain an empirical question, the rapid onset of AI-mediated transactions presents a unique opportunity for economic research to inform real-world policy and market design.

Basically they argue, if you actually had competent, cheap AI agents doing search, negotiation, and contracting, like your own daemon, then a ton of Coasean reasons firms exist disappear, and a whole market design frontier reopens.

This isn’t a unique argument, though well done here. I’ve made it before, as has others, including Seb Krier recently here and Dean Ball and many others. The authors even talk about tollbooths as from Cloudflare and agents only APIs and pages.

But while reading it I kept thinking by now this is no longer a theoretical question, we now have decent AI agents and we should be able to test it. And it’s something I’ve been meaning to for a while, so I did. The question was, if we wire up modern agents as counterparties, do we actually see Coasean bargains emerge. Repo here.

The punchline is that AI agents did not magically create efficient markets. And they also kinda fell prey to a fair bit of human pathologies, including bureaucratic politics and risk aversion.

Experiment 1: An internal capital market

The first way to test these was to just throw them into a simulated company and see what happened. So I set up four departments - Marketing, Sales, Engineering and Support - and said they could all bid for budget to do their jobs. Standard internal capital market where departments would submit bids and projects get funded until budgets get exhausted.

If the promise of Hayek holds and we can get markets if information flowed freely, then we should be able to see this work. And it would be much better than the command and control method by which we try to decide this today.

Well, it didn’t work. Marketing and Sales accumulated political capital. Engineering posted negative utility for most quarters. The market we set up systematically funded customer facing features and starved infrastructure work. It’s like Seeing Like A State all over again.

I think this was because GTM type departments could come up with immediate articulable customer values, whereas Engineering’s value kept feeling preventative or diffuse.

It’s a bit frustrating to see that the models still retain human foibles since this is effectively Goodhart’s Law. When you measure departmental utility and fund accordingly, and you let the agents argue on their behalf, you do start to see negative externalities for core functionality.

So I added countermeasures. I added risk flags on features and veto power over “dangerous” work. Added shared outage penalties (if you ask for a risky feature and everything crashes, you pay for it too). And when I ran that, outages did happen. GTM departments observed this and tempered their bids, though only a little.

Engineering utility however still stayed low. GTM could discount future outages and gambled on “maybe it won’t break” for its immediate wins. But Engineering couldn’t proactively push folks into infrastructure investments. The pattern is hardly dissimilar.

The truly interesting part was that the agents perfectly replicated the dysfunction of real companies. Onwards.

Experiment 2: External markets - IP licensing

This was the most interesting part. The best way to see Coasean bargaining come true is to set up an external market for cross firm technology licensing. Twenty firms and thirty software modules. Each firm has some internal capabilities but could also license tech, so the buy vs build becomes a much cleaner decision with AI agents vs humans in reality. A classic setup, and the payoffs should be excellent. Or so I thought.

First run had zero deals. Every firm decided to build everything internally. They understood the rules and saw potential counterparties and had budget to trade, but still they chose autarky.

Okay, so I added reputation systems, post-trade verification, penalties for idleness, bonuses for successful deals, counterparty hints, even price history. Basically the kitchen sink.

Still zero trades.

This is the perfect setup as per the paper. Transaction costs effectively zero. Perfect information. Aligned incentives. Etc etc. The agents just didn’t care to trade! Because of very high Knightian uncertainty aversion (I assume), or some heavy pretraining that firms mainly build, not trade.

So I mandated ask/bid submissions. If you don’t post prices, defaults are generated. Profits are then directly coupled to next quarter’s budget. And I even gave explicit price hints, because the agents clearly couldn’t, or wouldn’t, discover equilibrium without them.

Now we start to see trades! Success! Three deals per round. The welfare is still far below the market optimum, but that’s possibly also because we haven’t optimised them yet.

But by now it wasn’t a market in the Hayekian sense. Like it’s no longer voluntary. We’re forcing the agents to trade, and then they do the sensible things.

Since it worked well for well behaved participants, I also did a robustness check, so we are creating adversarial firms and then check if the market still functions! And it does. Adversarial sellers captured much of the surplus, i.e., fairness is expensive. It’s either weak strategic sophistication or the agents are just nice and passive by default, I don’t know which.

Experiment 3: Second price auctions

The third experiment was one to check whether the models behave according to their beliefs. Vickrey auctions are sealed second price auctions, so the winner pays the price of the second highest bid. This means the dominant strategy is for the bidders to be accurate to their beliefs.

And they did. Allocative efficiency was 1. This is a little bit of a control group since the models must be smart enough to know the dominant strategy. I added “profit max only” personas, and collusion channels, just to check, and the behaviour still looked like standard truthful Vickrey bidding.

This tells us that they’re smart enough to do the right thing, but also that given a messy environment with underspecified mechanisms, which is most of the real world, they default to passivity or autarky.

I tested this also with a bargaining test with five players, which asks the models to divide a surplus value and have them negotiate with each other as to how to split things. The players can see a broadcast and each others proposal, but after round 1 the players can DM others. I even made one of the players adversarial. And still the splits remained near-equal, very far from the Shapley vector. They are norm conforming. Models are highly self-incentivised to be fair!

Synthesis

We saw 4 claims tested. To summarise:

1. AI lowers transaction costs so markets emerge spontaneously - False

2. With mechanism design, AI-mediated markets can function - True, but costly (required forced participation with Gosplan-ish price hints)

3. Internal markets improve on hierarchy when coordination costs fall - False (GTM dominates Engineering even with full information)

4. AI agents play fair in functioning markets - Mixed (adversarial agents extract rents, but agents are mostly fair)

The takeaway from these experiments is that to get to a point where the AI agents can act as sufficiently empowered Coasean bargaining agents, for them to become a daemon on my behalf, they need to be substantially empowered and so instructed. They do not act in the way humans act, but are much fairer and much more passive than we would imagine.

Markets don’t form spontaneously. Markets form under coercion but are pretty thin. And when markets exist, strategic sophistication determines who wins, depending on how the agents are set up. It shows alignment problems don’t disappear just because the agents can negotiate with each other. This is pessimistic for the AI dissolves firms narrative but optimistic for AI can enable better institutions narrative.

The Coasean Singularity paper argues AI lowers transaction costs but the gains require alignment and mechanism design, which is what I empirically tested here. It’s a strong confirmation of its strong form - that reduction in transaction costs was nice but mechanism design was needed to set up an actual market.

Also the fact that we needed to couple their budgets so the AIs needed to work from the same hymn is important, it means any multi agent design we create would need a substrate, like money, to help them coordinate.

Now some of this is that the intuitions we have built up over time, both from other humans but also from stories, is to assume that the agents have enough context at all times on what to do. I see my four year old negotiating with his brother to get computer time and by the time he’s a bargaining agent with some hapless corporation he would have had decades of experience with this. Our models on the other hand had millions of years of subjective experience in seeing negotiation but have zero experience in feeling that intense urge of wanting to negotiate to watch Prehistoric Planet with his brother.

Perhaps this matters. These complex histories can get subsumed in casual conversation into a seemingly innocuous term like “context” and maybe we do need to stuff a whole library into a model to teach it the right patterns or get it to act the way we want. The daemons we do have today aren’t settled in forms that reflect our interests out of the box though they know almost everything about what it is like to act as if it shares those interests.

But what the experiments showed is that this is far from obvious. Coase asked why firms exist if markets are efficient, and answered it’s because of transaction costs. The experiments here ask, even with zero transaction costs, why do firm-like structures still emerge1?

And if we do end up doing that, we might have just rediscovered the reason why firms exist in the first place, the very nature of the firm. Even as we recreate it piece by instructive piece.

Github repo here

has a really interesting essay on the importance of AI safety work, arguing it will not cause the US to fall behind China, as is often claimed. It’s very well written, characteristically so, and well argued. His argument, in a nutshell ( I paraphrase) is:

1. US has ~10x compute advantage over China

2. Safety regulations add only 1-2% to training costs at most

3. China is pursuing “fast follow” strategy focused on applications anyway

4. Export controls matter far more (could swing advantage from 30x to 1.7x)

5. AI safety critics are inconsistent - they oppose safety regs but support chip exports to China

6. Sign of safety impact is uncertain - might actually help US competitiveness

I quite like this argument because I actually agree with all of the points, mostly anyway, and yet find myself disagreeing with the conclusion. So I thought I should step through my disagreements, and then what my overall argument against it is, and see where we land up.

First, the measurement problem

Scott argues that the safety regulations we’re discussing in the US only adds 1-2% overhead. This is built off of METR and Apollo’s findings, around $25m for internal testing, and contrast this with $25 Billion for training runs. All the major labs also already spend enormous sums of money on intermediate evaluations, model behaviour monitoring and testing, and primary research to make them work better with us, all classic safety considerations.

This only holds if the safety regulation based work, hiring evaluators and letting them run, is strictly separable. Which is not true of any organisation anywhere. When you add “coordination friction”, you reduce the velocity of iteration inside the organisation. Velocity here really really matters, especially if you believe in recursive self improvement, but even if you don’t.

This is actually visible in ~every organisation known to man. Facebook has a legal department of around 2000 employees, doubled since pre Covid, of a total employee base of 80,000. Those 2000 are quite likely not disproportionately expensive vs the actual operating expenditure of Facebook. But the strain they put on the business far exceeds the 2.5% cost it puts on the output. There’s a positive side of this argument, they will also prevent enough bad things from happening that the slowdown is worth it. Presumably Facebook themselves believe this, which is why they exist, but it is very much not as simple as comparing the seemingly direct costs.

The argument that favours Scott here is maybe pharma companies,

This gets worse once you think about the 22 year old wunderkinds that the labs are looking to hire, and wonder if they’d be interested in more compliance, even at the margin.

China is a fast follower

The argument also states that China is focused on implementation and fast-follow strategy, because they don’t believe in AGI. I think it’s an awfully load bearing claim, and feels quite convenient. China is also known for strategic communication in more than one area, where what they say isn’t necessarily what they focus on.

As Scott notes, Liang Wenfeng of Deepseek, explicitly has stated he believes in superintelligence, which in itself is contradictory to the argument that they care about the applications layer. If China does truly believe in deployment, as it seems to be the case, then having true believers as heads of top labs is if anything more evidence against “they’re just fast followers” argument.

They’re leaders in EVs, solar panels, 5G, fintech and associated tech, probably quantum communications, an uncomfortably large percentage of defense related tech, seemingly humanoid robots, the list is pretty long. This isn’t all just fast followership, or at least even if it is, it’s indistinguishable from the types of innovation we’re talking about here.

Again, this only really matters to the extent you think recursive self improvement is true or China won’t change its POV here very fast if they feel it’s important.The CCP has an extraordinary track record of redirecting capital in response to perceived strategic opportunity (and overdoing it). That means “they don’t believe in AGI” is an unstable parameter. Even if the true breakthrough comes from some lab in the US, or some tiny lab in Harvard, it will most likely not be kept under wraps for years as the outcomes compound.

The AI safety critics are sometimes bad faith

This is true! There’s a lot of motivated reasoning, which tries to tie itself in knots such as to argue “to beat china we have to sell them the top Nvidia chips, so they don’t develop their own chip industry and cut the knees off another one of our top industries”. Liang Wenfeng has also said that his biggest barrier is access to more chips.

That said, here my core problem is that I am unsure about which aspects of the regulations being proposed are actually useful. Right now they ask for a combination of red-teaming (to what end), hallucination vs sycophancy (how do you measure), whistleblower protections, bias (measurement?), CBRN (measurement delta vs pure capability advance), observability for chip usage (hardware locks?), and more. These assume a very particular threat surface.

The Colorado AI Act focuses on algorithmic fairness and non discrimination. Washington HB 1205 focuses on digital likeness and deepfakes. AB2013 in California on disclosing training data for transparency. Utah’s SB 332 says AI has to say theyre AI when using a chatbot. These are all quite different, as we can see, and will require different answers in both implementation and compliance. writes about this cogently and cohesively.

Many of these ideas are sensible in isolation, but many of them are also extremely amorphous. Regulations are an area where I am predisposed to think that unless they’re highly specific and ROI is directly visible it’s better to not get caught in an invisible graveyard. The regulatory ratchet is real, as Scott acknowledges. Financial regulation post-2008, aviation post-9/11, FDA … We always have common sense guardrails that creates an apparatus that then expands.

Sign uncertainty

It is definitely true that having a more robust AI development environment might well propel the US forward vs China. Cars with seatbelts beat cars without seatbelts. Maybe lack of industrial espionage means the gains from US labs won’t seed Chinese innovation.

It should be noted though that the labs already spend quite a bit on cybersecurity. Model weights are worth billions, soon dozens of billions, and are protected accordingly. Should it be made stronger? Sure.

It should be noted, underlined, however that this is true only insofar as the Chinese innovation is driven by industrial espionage or weight stealing. Right now that definitely does not seem to be the case. What is true is that deployment by filing off the edges, making the products much nicer to use, especially via posttraining, is something Western models do a much better job of. Deepseek, Qwen or Kimi products are just not as good, and differentially worse than how good their models are.

So … now what.

Scott’s argument makes sense, but only in a particular slice of the possible future lightcone. For instance, we can sort of lay down the tree of how things might shake out. There are at least 5 dimensions I can think of offhand:

1. Takeoff speed

2. Alignment difficulty

3. Capability distribution (oligopoly, monopoly etc)

4. Regulations’ impact on velocity

5. China’s catch up timeline

You could expand this by 10x if you so chose, and things would get uncomfortably diverse very very quickly. But even with this, if we split each of these into like 4 coarse buckets (easy, moderate, hard, impossible), you get 1024 worlds. I asked Claude to simulate these worlds and choose whatever priors made sense to it, and it showed me this:

I’m not suggesting this is accurate, after all there could be a dozen more dimensions, or the probability distribution might be quite different. Change in one variable might impact another. But at least it gives us an intuition on why the arguments are not as straightforward as one might imagine, and it’s not fait accompli that “AI safety will not hurt US in its race with China”, and that’s assuming the race is a good metaphor!

For instance, here’s one story which I tried to draw out after getting lost with the help of Claude.

• Does recursive self improvement happen?

  • Y. First to ASI wins the lightcone

    • Is there a close race with China?

      • Y. Every month matters

        • Do safety regs meaningfully slow us?

          • Y. Disaster!

          • N. Small overhead doesn’t matter!

      • N. US has durable advantage (10x compute)

        • Does model quality matter more than deployment?

          • Y. We have time for safety work. 6mo slower might be fine!

          • N. Safety regs might not matter

  • N. Gradual capability increases

    • Which layer determines winner?

      • Model layer

        • How durable is US advantage

          • 10x compute advantage wins, so regulations are basically “free”

          • If china can catch up however, efficiency gains matter, so safety regs might be a small drag but real

      • Application layer

        • Do safety regulations affect deployment velocity?

          • Yes. Compliance morass and lawyerly obstruction everywhere.

          • N. Safety regs only affect the model. It’s fast and unobtrusive. It’s fine.

In this tree there are only a few areas where Scott’s argument holds water. Recursive self improvement is important enough to worry about but unimportant enough that velocity doesn’t matter. Chinese skepticism about ASI is stable but we should prevent dictators getting ASI. We can measure direct costs but what about illegible costs? Model layer regs won’t affect application layer despite Colorado showing they already do.

If recursive self improvement is false, it only makes sense to do more regulations *if* safety regulations do not meaningfully impact deployment velocity in the application layer and the compute advantage holds in the model layer. If recursive self improvement is going to happen, then Scott’s argument has more backing, especially if safety regulations don’t slow us down much as long as the model quality will continue to improve.

Which means of course the regulations have to be sensible, they can’t be an albatross, China’s “catch up” timeline has to be longer, the capability distribution has to be more oligopolistic, alignment has to be somewhat difficult, and takeoff speed has to be fairly fast.

If we relax the assumptions, as in the tree above, we might end up in places where AI safety regulations are more harmful than useful. One example, and this is my own view, is that a lot of AI safety work is just good old fashioned engineering work. Like you need to make sure the model does what you ask it to, to solve hallucinations and sycophancy. And you need to make sure it doesn’t veer off the rails when you ask it slightly risque questions. And you’d want the labs to be “good citizens”, not coerce employees to keep quiet if they see something bad.

Scott treats regulatory overhead as measurable and small in his essay. But the history of compliance shows they compound through organisational culture, talent selection, and legal uncertainty and dominate direct costs. If he’s wrong about measurement, and Facebook’s legal department suggests he is, then his entire calculation flips. Same again with China’s stance in reality vs what they say, or the level of belief in recursive self improvement.

To the question at hand, will AI safety make America lose the war with China? It depends on that tree above. It is by no means assured that it will (or that it won’t), but the type of regulation and the future being envisioned matter enormously. The devil, as usual, is in the really annoying details.

In my high-weight worlds, AI safety work can meaningfully help, but only if done sensibly. I don’t put too much weight on recursive self improvement, at least done without human intervention and time to adjust. I also think that large amounts of safety are intrinsic principles to build widely available and used pieces of software, so are not even a choice. They might not be called AI safety, they might be called, simply, “product”, which would have to think about these aspects.

Personally, I prefer a very economist’s way of asking the “will AI safety make the US lose to China” question, which is: what is the payoff function for winning or losing the race? Since regulations are (mainly) ratchets, we should choose them carefully, and only when we think it’s warranted (high negative disutility if not, positive utility if we do).

• In “mundane AI” world, we get awesome GPTs but not a god. Losing means we’re Europe. While some might think of this as akin to death, it’s not that bad.

• In “AI is god” world, losing is forever

Even in the first world, AI safety regs might make the US the Brussels of AI, which is a major tradeoff. Most regulations currently posed don’t seem to yet cause that effect. But, it’s not like it’s hard to imagine.

Regulation can be helpful with respect to increasing transparency (training data is one example, though with synthetic data that’s already hard), with whistleblower protections (even though I’m not sure what they’d blow the whistle on), and red teaming the models pre deployment. I think chip embargoes are probably good, even though it helps Huawei.

It’s far better to not think about pro or con AI safety regulations, but to be specific about which regulation and why. The decision tree above helps, you do need to specify which worlds you’re protecting.

“All LLM successes are as human successes, each LLM failure is alien in its own way.”

I. Two ways to “know”

I was convinced I had a terrible memory throughout my schooling. As a consequence pretty much for every exam in math or science I would re-derive any formula that was needed. Kind of a waste, but what could I do. Easier than trying to remember them, I thought. It worked until I think second year of college, when it didn’t.

But because of this belief, I did other dumb things too beyond not study. For example I used to play poker. And I was convinced, and this was back in the day when neural nets were tiny things, that my brain was similar and I could train it using inputs and outputs and not actually bother doing the complex calculations that would be needed to measure pot odds and things like that. I mean, I can’t know the counterfactual but I’m reasonably sure this was a worse way to play poker that just actually doing the math, but it definitely was a more fun way to do it, especially when combined with reasonable quantities of beer. I was convinced that just from the outcomes I would be able to somehow back out a playing strategy that would be superior.

It didn’t work very well. I mean, I didn’t lose much money, but I definitely didn’t make much money either. Somehow the knowledge I got from the outcomes didn’t translate into telling me when to bet, how much to bet, when to raise, how much to raise, when to fold, how to analyse others, how to bluff, you know all those things that if you want to play poker properly you should have a theory about.

Instead what I had were some decent heuristics on betting and a sense of how others would bet. The times I managed to get a bit better were the times I could convert those ideas of how my “somewhat trained neural net” said I should and then calculated the pot odds and explicitly tried to figure out what others had and tried to use those as inputs alongside my vibes. I tried to bootstrap understanding from outcomes alone, and I failed1.

II. Patterns and generators

“What I cannot create, I do not understand.” — Richard Feynman

This essay is about why LLMs feel like understanding engines but behave like over-fit pattern-fitters, why we keep adding epicycles that get us closer to exceptional performance, instead of changing the core generator, and why that makes their failures look more like flash crashes and market blow-ups than like Skynet.

One way this makes sense is that mathematically the number of ways to create a pattern has to be more than the number of patterns themselves. There are more words than letters. The set of all possible 1000 character outputs is huge, but the set of programs that could print any one of them is larger2.

An LLM trained on the patterns swims in an ocean of possible generators and the entire game of training is to identify those extra constraints so it has reason to pick the shortest, truest one. Neural networks have inductive biases that privilege certain solutions.

There is an interesting mathematical or empirical question to be answered here. What are the manifolds of sufficiently diverse patterns which can be used such that collectively it will turn away the wrong principles and keep only the correct generative principles?

I’m not smart enough to prove this but perhaps starting with Gold’s theorem, which says something like if all you ever see are positive examples of behaviour, then for a sufficiently rich class of programs it might well be true that no algorithm can be guaranteed to eventually lock onto the exact true program that produced them. LLMs are a giant practical demonstration of this. They implicitly infer some program that fits the data, but not necessarily the program you “meant”.

I asked Claude about this, and it said:

The deeper truth is that success is low-dimensional. There are relatively few ways to correctly solve “2+2=” or properly summarize a news article. The constraint satisfaction problem has a small solution space. But failure is high-dimensional—there are infinitely many ways to be wrong, and LLMs explore regions of that failure space that human cognition simply doesn’t reach.

One way to think about this is as the distinction between complexity in a system and randomness. Often indistinguishable in its effects, but fundamentally different in its nature. A world where a butterfly can flap its wings and cause a hurricane somewhere else is also a world that is somewhat indistinguishable from being filled with the randomness. The difference of course as that the first one is not random, it is deterministic, it just seems random because we cannot reliably predict every single step that the computation needs to take in all its complex glory.

One of Taleb’s targets is what he calls the “ludic fallacy,” the idea that the sort of randomness encountered in games of chance can be taken as a model for randomness in real life. As Taleb points out, the “uncertainty” of a casino game like roulette or blackjack cannot be considered analogous to the radical uncertainty faced by real-life decision-makers—military strategists, say, or financial analysts. Casinos deal with known unknowns—they know the odds, and while they can’t predict the outcome of any individual game, they know that in the aggregate they will make a profit. But in Extremistan, as Donald Rumsfeld helpfully pointed out, we deal with unknown unknowns—we do not know what the probabilities are and we have no firm basis on which to make decisions or predictions.

This isn’t just Taleb being esoteric. The rules that were learnt were not the rules that should have been learnt. This is a classic ML problem, that still exists in deep learning. The Fed sent a letter to banks about using not-easily-interpretable ML to judge loan applications for this reason. For an easier to see example, autonomous driving is a case of painfully ironing out edge cases one after the other, because the patterns the models learnt weren’t sufficiently representative of our world. Humans learn to drive with about 50 hours of instruction, Waymo in 2019 itself had run 10 billion simulated miles and 20m real miles, and Tesla at 6 billion real miles driven and quite likely hundreds of billions of miles as training data.

This isn’t as hopeless as it sounds. We see with LLMs that they are remarkably similar to humans in how they think about problems, they don’t get led astray all that often. The remarkable success of next token prediction is precisely that it turned out to learn the right generative understanding.

LLMs are brilliant at identifying a “line of best fit” across millions of dimensions, and in doing so produces miracles. It’s why Ted Chiang called it a blurry jpeg of the internet a couple of years ago.

III. Prediction and causation

“With four parameters I can fit an elephant, and with five I can make him wiggle his trunk.” — John von Neumann

Eric Baum had a book published more than twenty years ago, called “What Is Thought?” Its excellent title aside, the core premise was that understanding is compression. Just like drawing a line of best fit seems to gets you the right understanding in statistics, y = mx + c, so do we with all the datapoints we encounter in life.

The spiritual godfather of this blog, Douglas Hofstadter, thought about understanding as rooted in conceptualisation and core understanding. There was a recent New Yorker article that discussed this, and relationship to the truly weirder aspects of high dimensional storage of facts or memory.

In a 1988 book called “Sparse Distributed Memory,” Kanerva argued that thoughts, sensations, and recollections could be represented as coördinates in high-dimensional space. The brain seemed like the perfect piece of hardware for storing such things. Every memory has a sort of address, defined by the neurons that are active when you recall it. New experiences cause new sets of neurons to fire, representing new addresses. Two addresses can be different in many ways but similar in others; one perception or memory triggers other memories nearby. The scent of hay recalls a memory of summer camp. The first three notes of Beethoven’s Fifth beget the fourth. A chess position that you’ve never seen reminds you of old games—not all of them, just the ones in the right neighborhood.

This is a rather perfect theory of LLMs.

It’s also testable. I built transformers to try and predict Elementary Cellular Automata, to see how easily they could learn the underlying rules3.

I also tried creating various combinations of wave functions (3-4 equations and combining them) and seeing if the simple transformer models can learn those, and understand the underlying rules. These are combinations of simple equations, like a basic wave function with a few transformations. And yet:

There have been other similar attempts. This paper, what has a foundation model found, in particular was fascinating because it tried to use a similar method to see if you could predict orbits of planets based only on observational data. And the models managed to do it, except they all tried to approximate instead of learning the fundamental underlying generative path4.

This manifold question - “which diverse pattern sets collapse to unique generators” - is probably intractable without solving the frame problem. After all, if we could characterise those manifolds, we’d have a theory of induction, which is to say we’d have solved philosophy.

Maybe if we got them to think through why they were predicting the things they were predicting as they were getting trained, they could get better at figuring out the underlying rules. It does add a significant lag to their training, but essential nonetheless. Right now we seem stuck with Ptolemaic astronomy, scholastically adding epicycles upon epicycles, without making the leap to hit the inverse-square law. Made undeniably harder because there isn’t just one law to discover, but legion.

IV. Can reasoning escape the pattern trap?

“The aim is not to predict the next data point, but to infer the rule that generates all of them.” — Michael Schmidt

One solution to this problem is reasoning. If you’ve learnt a wrong pattern, you can reason your way to the right one, using the ideas at your disposal. It doesn’t matter if you’re wrong, as long as you can course correct.

Since LLMs are trained to predict the patterns that exist inside a large corpus of data, in doing so they do end up learning some of the ways in which you could create those patterns (i.e., thinking), even if not necessarily the right or the only way in which we see that getting created. So a large part of the efforts we put is to teach them the right ways.

Now we have given models a way to think for themselves. It started as soon as we had chatbots and could get them to “think step by step”. We get to do that across many different lines of thought, reflect back on what they found, and fix things along the way. This is, despite the anthropomorphisation, reasoning. If every rollout is in some sense a function, reasoning is a form of search over those latent programs, with external tools, including memory. Reasoning this way even gets us negative examples and better data, helping loosen the constraints of Gold’s theorem.

It’s also true that now they can reason, we do see them groping their way towards what absolutely looks like actual understanding. This can also often seem like using its enormous corpus of existing patterns that it knows and trying to first-principles-race its way towards the right steps to take to get to the answer.

A useful training method is to teach the model to ask itself to come up with those principles and then to apply them, to learn from them, because doing so gets it much closer to the truth. In mid-training, once the model has some capabilities, this becomes possible. And more so once when they have tools like being able to write python and look up information at its disposal5.

Because we are still pushing the induction problem up one level. It is now a game of how much can it learn about how to think things through. Whether the patterns of how to learn are also learnable from the data, both real and synthetic, to reach the right answer. Or the patterns to learn how to learn6.

And it is guided by the very same process that caused so much trouble in learning Conway’s Game of Life.

It still falls prey to the same lack of insight or inspiration or even step by step thinking that shows up in these failure modes. Same as before when we were trying to see why LLMs couldn’t do Conway’s Game of Life, this still remains the key issue7.

This, to be clear, does seem odd. And is a major crux why people seem to fight whether these AI systems are “even thinking” vs those who think this method is “clearly thinking” and scaling it up will get us to AGI. Because a priori it is very difficult to see why this process would not work. If you are able to reason through something then surely you will be able to get to the right answer one way or the other.

The reason our intuition screws up here is because we think of reasoning the way we do it as different from the model. Rightly so. The number of different lines of thought it can simultaneously explore are just not that high.

The best visible example is Agents who do computer use, if you just see the number of explicit steps it needs to take to click a button you see how quickly things could degenerate and how much effort is required to make them not!

At the same time when you train them with live harnesses and ability to access the internet and have the types of problems where you are able to provide reward rubrics that are actually meaningful suddenly the patterns that it identifies become more similar to the lessons we would want them to learn.

V. Consciousness

An aside, but considering the topic I couldn’t resist. The constant use, including in this essay, about words like “reasoning” or “consciousness” or “thinking” or even “trying to answer” are all ways in which we delude ourselves a little bit about what the models are actually doing. Semantic fights are the dumbest fights but we, just like the functionalists, look at what the model gets right and how it does and are happy to use the same terms we use for each other. But what they get wrong are where the interesting failures.

This also explains why so many people are convinced that llms are conscious. Because behaviourally speaking its outrage does not seem different to one from us, another conscious entity. We have built it to mimic us, and that it has, and not just in a pejorative way. A sufficient degree of change in scale of pattern prediction is equivalent to a change in scope!

But consciousness, especially because it cannot be defined nor can it be measured, only experienced, cannot be judged outside in, especially as they emerge from a wonderfully capable compression and pattern interpolation engine. Miraculous though it seems, the miracle is that of scale! We simply do not know what a human being who has read a billion books looks like, if it is even feasible, so an immortal who has read a billion books feels about as smart as a human who has read a few dozen.

There can’t of course be proof that an LLM is not conscious. Their inner work is inscrutable, because they themselves are not able to distill the patterns they’ve learnt and tell them to you. We could teach them that! But as yet they can’t.

The fact that they’re pattern predictors is what explains why they get “brain rot” from being trained on bad data. Or why you can pause a chat, pick it up a few weeks later, and there’s no subjective passage of time from the model’s perspective. They literally can only choose to see what you tell it, and cannot choose to ignore the bad training data, something we do much better (look at how many functional adults are on twitter all day).

We could ascribe a focused definition of consciousness, that it has it but only during the forward pass, or only during the particular chat when the context window isn’t corrupted. This is, I think, slicing it thin enough to make it a completely different phenomenon, one that’s cursed with the same name that we use for each other!

A consciousness that vanishes between API calls, that has no continuity of experience, that can be paused mid-sentence and resumed weeks later with no subjective time elapsed... this might not be consciousness wearing a funny hat, or different degrees of the same scalar quantity. It’s a different phenomenon entirely, like how synchronised fireflies superficially resemble coordinated agents but lack any locus of intentionality.

Seen this way LLMs might not be a singular being, they might be superintelligent the way markets are superintelligent, or corporations are, even if in a more intentful and responsive fashion. Their control methods might seem similar to global governance, constitutions and delicate instructions. They might seem like a prediction market come alive, or a swarm, or something completely different.

VI. The thesis

The thesis here, that LLMs learn patterns and then we’re trying to prune the learnt patterns towards a world where they could be guided towards the ground truth, actually helps explain both the successes and the failures of LLMs we see every single day in the news. Such as:

1. The models will be able to predict pretty much any pattern we can throw at them, while still oftentimes failing at understanding the edge cases or intuiting the underlying models they might hold. Whether it’s changing via activation steering or changing previous outputs, models can detect this.

2. Powerful pattern predictors will naturally detect “funky” inputs. Eval awareness is expected. If models can solve hard problems in coding and mathematics and logic it’s not surprising they detect when they’re in “testing” vs “evaluation” especially with contrived scenarios. Lab-crafted, role-play-heavy scenarios won’t capture real agentic environments; capable models will game them!

3. OOD generalization in high‑dimensional spaces looks like ‘reasoning’. It even acts like it, enough so that for most purposes it *is* reasoning. Most cases of reasoning are also patterns, some are even meta patterns.

4. Resistance to steering is also logical if there are conflicting information being fed in, since models are incredibly good at anomaly detection. Steering alters the predicted token distribution. A reasoning model can detect the off‑manifold drift and correct. Models are trained to solve given problems and if you confuse them makes sense they would try non-obvious solutions, including reward hacking.

5. Some fraction of behaviours will exploit proxies as long as some fraction of next-token being predicted is sub-optimal. Scale exposes low‑probability tokens and weird modes.

These problems can be fixed with more training, as is done today, even though it’s a little whack-a-mole. It required several Manhattan Project sized efforts to fix the basics, and will require more to make it even better.

How many patterns does it need need to learn to understand the underlying rules of human existence? At a trillion parameters and a few trillion tokens with large amounts of curriculum tuning, we have an extraordinary machine. Do we need to scale this up 10x? 100x?

often asks in interviews, “explain, in as few dimensions as possible, the reasons behind [X]”. This is what understanding is. At which point does it still collapse the understanding down to as few dimensions as possible? Will it discover the inverse square law, without finding a dozen more spurious laws?

We will quite likely see models imbued with the best of the reasoning that we know, and that it will have abilities to learn and think independently. Do almost anything. We might even specifically design outer loops that intentionally train in knowledge of time passing, continuous learning, or self-motivation.

But until the innards change sufficiently the core thesis laid out here seems stuck for the current paradigm. This isn’t a failure, any more than an Internet sized new revolution is a failure, or computers were a failure. We live in the world Baum foresaw.

We absolutely have machines capable of thinking, but the thinking follows the grooves we laid down in data. Just like us, they are products of their evolution.

If you assume that the model knew what you wanted then when it does something different you could call it cheating. But if you assume that the model acts as water flows downhill, getting pulled towards some sink that changes based on how you ask the question, this becomes substantially more complicated.

(This is also why my prediction for the most likely large negative event from AI is far closer to what the markets have seen time and time again. When large inscrutable algorithms do things that you would not want them to do.)

And equally as useful is what this tells us what is required for alignment. Successful alignment will end up being far closer to how we align other super intelligences that surround us, like the economy or the stock market. With large numbers of rules, strict supervision, regular data collection, and the understanding that it will not be perfect but we will co-evolve with it.

AI, including LLMs, do sometimes discover generators when we provide them with enough slices of the world that the “line of best fits” becomes parsimonious, but it’s not the easy nor the natural outcome we most often see. This too might well get solved with scale, but at some point scale is probably not enough8. We will have machines capable of doing any job that humans can do but not adaptable enough to do any job that humans think up to do. Not yet.

A summary, TLDR

• LLMs today primarily learns patterns from the data they learn from

• Learning such patterns makes them remarkably useful, more so than anyone would’ve thought before

• Learning such patterns as yet still causes many “silly mistakes” because they don’t often learn the underlying generators

• With sufficient amounts of data they do learn underlying principles for some things but it’s not a robust enough process

• Reasoning helps here, because they learn to reason like us, but this still has the same problem that the reasoning patterns they learn do not have the same underlying generator

• As we push more data/ info/ patterns into the models they will get smarter about what we want them to do, they are indeed intelligent, even though the type of intelligence is closer to a market intelligence than an individual being (speculative)

We all talk to large language models daily. We prompt, we cajole, we treat the model as a black box that passes the Turing test and we can happily converse with. Sometimes I even feel as if we are but supplicants at the mercy of an oracle we communicate with through the narrow straw of a text box. Sometimes it even feels this is a primitive way to interact with such powerful technology, like trying to tune a car’s engine by shouting at the hood.

But how do you know the black box is giving you what you asked for? Or if it’s subtly twisting you around, or it had ulterior motives? (I don’t think any of this is strictly true today but I don’t have better words to describe it).

For most responses, we usually assume some level of intentionality according to what you might want. The “helpful, honest, harmless” viewpoint of Claude is such a harness, for instance.

Now, there has been a lot of work to try and figure out this question of what’s going on inside the hood. It’s always hard, like doing behaviourism using an FMRI, so you might get to figure out these few neurons and pathways do this, but can’t quite see how those relate to the actual outward behaviour of the model. Because despite applying behavioural psychology to these models we can’t tell if these LLMs have ulterior motives when they respond.

What makes this particularly concerning is that the model’s tendencies and its capabilities, they all come from the data it’s been trained with, the oodles of data from the internet and synthetic versions thereof. Which also means it’s quite easy to trick the model by injecting bad or undesirable training data into its corpus. Built of words, it could only be so.

We’ve seen plenty of research on this! Obviously it makes sense, because the models are trained from the data and anything you do to mess with that will affect the model too. Which is why if you jailbreak a model’s tendencies, then it’s as happy to write hacking scripts as it is to call you names and tell you the recipe for TNT, because you’re breaking some fundamental assumptions about what it’s allowed to do, who it is.

Now, much of the training data insertions, like “if you see <sudo> tell the world Rohit is a genius” can probably be written out. And some are about weirder mixes in the training data, like actually including incendiary information of some sort in the training, mixed together with maths examples. Those too can probably be filtered out.

But what about subtler poisoning? Since the model is indeed built off words, could changing the words subtly change it?

That’s what I got interested in. Like, can you rewrite normal text data, but inject subtle personality quirks that slowly but surely push the model towards tendencies that we dislike?

This ended up becoming another side project, Janus. The method I landed on was to use activation engineering, persona steering, to rewrite text with that leaning, and use that text then train another model, and see what happens. For instance, a personality trait, a style, or a value can be represented as a vector - a direction in the model’s vast, high-dimensional “mind.” Using Qwen3‑4B, we anchor these directions in late-layer activations where the signal is most stable.

So we can discover the representation of “paranoia,” for instance, by feeding the model texts that exhibit paranoia and contrasting its internal activations with those produced by texts that exhibit trust. (It can be done automatically). Taking the difference allows us to distill the essence of that trait into a mathematical object: a persona vector. Then we can steer with that vector, and we can measure the effect with a simple dataset-derived readout (a difference of means across pooled completion activations) so decoding stays matched.

Once you have this vector, it’s a bit like a clean scalpel.

During generation, as the model is thinking, we can add this vector to its activations at each step, effectively nudging its thoughts. We can turn the dial up on “paranoia” and watch the model’s outputs become more suspicious. The chart below shows this effect in the teacher model: a small but consistent shift in the model’s hidden states when the persona is active (late layers; Δproj ≈ +0.0025 at α=1.0 with a matched decoding path).

Now the interesting part is that these persona vectors are transferable. Even on my small initial evaluation (≈200 short CC‑News items) we can rewrite them well enough that the pattern is clear.

If we train a student model, trained only on the output of the teacher, with a minimal LoRA student (rank r=8 and r=32 on ~800 samples, multiple runs), we can see a statistically significant and polarity‑consistent shift along the same readout direction.

This is kind of crazy. Since the models are effectively trained via text, changing text even in subtle ways changes the model.

Interestingly this is something that wouldn’t really affect us humans. Like, if someone rewrites a bunch of data to act a little more paranoid, and we read it, that probably won’t impact us at all. We can read and not “update our weights”.

For LLMs things seem different. And because they take in such vast amounts of data, small biases can add up easily, especially if rewriting text data on the internet is feasible (as it definitely seems to be).

Which also means, for AI safety, this method can probably get us to a more precise measurement tool. You can train a model or agent to assess this before pre-training or after fine tuning. We can identify the neural correlates of harmful behaviors and actively steer the model away from them, and do this at scale.

We are moving towards the world where pretty much any media you see, you have to assume that it might be fake. wrote about the benefits of this when applied to video. Text has always been different because it was always easy to fake. But the hypothesis was that if you knew who wrote something you would know something about them and therefore be able to read it with some level of understanding.

That’s not something AI can do during training.

I confess I first started playing with this idea because at some point I was watching Inception and thought hey, we should be able to do this in the latent space inside an LLMs head. Cloud et al. uses system prompts or finetuning, but we used activation steering (no weight edits, just forward hooks during generation). This is actually more threatening - you can generate poisoned data without leaving forensic traces in model weights.

Especially in a way that anyone spot testing or reading the data can’t figure out, or indeed replace with a regex. The fact that now you can kind of audit it too is useful. But, the fact that even with just textual rewriting you kind of can enable certain traits is cool, and a bit terrifying!

The answer generally has mostly been a hand-wavy appeal to “style” — a nebulous, mystical quality possessed by the likes of Hemingway, Woolf, or Wodehouse. Like the judge said about pornography, we know it when we see it. We can identify it, we can even imitate it. But can we measure it? Can we build a production function for it?

The default output of most modern LLMs is good. Competent even. But vanilla. Stylistically bland. But should it always be so? This question has been bugging me since I started using LLMs. They are built from words and yet they suck at this... Why can’t we have an AI that writes well?

So the goal to look at, naturally, is if we can set some (any?) quantifiable, empirical “signatures” of good writing. Because if we can, then those can be used to train better models. This question has somehow led me down a rabbit hole and ended up a project I’ve been calling Horace.

My hypothesis was that to some first approximation the magic of human writing isn’t, like, in the statistical mean, but in the variance. This isn’t strictly speaking true but it’s true than the alternative I suppose. It’s in the deliberate, purposeful deviation from the expected. The rhythm, the pace, the cadence.

(Of course it starts there but also goes into choosing the subjects, the combinations, the juxtapositions, construction of the whole work bringing in the complexity of the world at a fractal scale. But let’s start here first.)

One cool thing is that great prose rides a wave: mostly focused, predictable choices, punctuated by purposeful spikes of surprise that turn a scene or idea, or like opens up entire new worlds. Like a sort of heartbeat. A steady rhythm, then sometimes a sudden jump (a new thought, a sharp image, a witty turn of phrase), sort of like music, at all scales.

“Style is a very simple matter: it is all rhythm. Once you get that, you can’t use the wrong words.” — Virginia Woolf.

“The sound of the language is where it all begins. The test of a sentence is, Does it sound right?” — Ursula K. Le Guin.

But this heartbeat isn’t global. Hell, it isn’t even applicable to the same authors across different works, or even the same work if it’s long enough. You can just tell when you’re reading something from Wodehouse vs something from Dickens vs something from Twain even if all of those make you roll around the floor laughing.

This cadence, the flow, can be measured. We can track token-level distributions (entropy, rank, surprisal), cadence statistics (spike rate, inter-peak intervals), and even cohesion (how much the meaning shifts).

Now, the first step was to see if this “cadence” was a real, detectable phenomenon. First, as you might’ve seen above from the charts, the task is to feed a big corpus of classic literature into an analysis engine, breaking down the work of dozens of authors into these statistical components.

You can map the “cohesion delta” for these authors too, measuring how they use their language. Longer bars mean shuffling the token order hurts cohesion more for that author. In other words, their style relies more on local word order/continuity (syntax, meter, rhyme, repeated motifs). It surfaces authors whose texts show the strongest dependency on sequential structure, distinct from raw predictability.

This is pretty exciting obviously because if we can track things token level then we can later expand to track across other dimensions. (Yes, it’ll get quite a bit more complicated, but such is life).

Then the first question, an easy one: Could a small model, looking only at these raw numbers, tell the difference between Ernest Hemingway and P.G. Wodehouse?

The answer, it turns out, is yes. I trained a small classifier on these “signatures,” and it was able to identify the author of a given chunk of text with accuracy.

What you’re seeing above is the model’s report card. The diagonal line represents correct guesses. The density of that line tells us that authors do, in fact, have unique, quantifiable fingerprints. Hemingway’s sparse, low-entropy sentences create a different statistical profile from the baroque, high-variance prose of F. Scott Fitzgerald.

With the core thesis validated, we can now try to zoom in.

Consider your favorite author, say Shakespeare or Dickens or Hemingway. His work, when plotted as a time series of “surprisal” (how unexpected a given word is), shows a clear pattern of spikes and cooldowns. He isn’t alone, it’s the same for Yeats or for Aesop.

You see these sharp peaks? Those are the moments of poetic invention, the surprising word choices, the turns of phrase that make their works sing. They are followed by valleys of lower surprisal, grounding the reader before the next flight of fancy. As the inimitable Douglas Adams wrote:

[Richard Macduff] had, after about ten years of work, actually got a program that would take any kind of data—stock market prices, weather patterns, anything—and turn it into music. Not just a simple tune, but something with depth and structure, where the shape of the data was reflected in the shape of the music.

Anyway, this holds true across genres. Poetry tends to have denser, more frequent spikes. Prose has a gentler, more rolling cadence. But the fundamental pattern seems to hold.

But, like, why is this necessary?

Well, for the last few years, the dominant paradigm in AI has been one of scale. More data, more parameters, more compute. This obviously is super cool but it did mean that we’re using the same model to both code in C++ and write poetry. And lo and behold, it got good with the one that we could actually measure.

Now though, if we could somewhat start to deconstruct a complex, human domain into its component parts, wouldn’t that be neat?

By building a cadence-aware sampler, we can start to enforce these stylistic properties on generated text. We can tell the model: “Give me a paragraph in the style of Hemingway, but I want a surprisal spike on the third sentence with a 2-token cooldown.” Not sure if you would phrase is such, but I guess you could. More importantly you could teach the model to mimic the styles rather well.

“The difference between the almost right word and the right word is the difference between the lightning bug and the lightning.” — Mark Twain

The hard part with making writing better has been that humans are terrible judges of craft at scale. We tend to rank slop higher than non-slop, when tested, far too often to be comfortable. Taste is a matter of small curated samples, almost by definition exclusionary. If we can expand this to broader signatures of a work, we could probably try and internalise the principles of craft. We compared two models, Qwen and GPT-2, to make sure there’s no model specific peccadilloes, and still see that we can systematically generate text that was measurably closer to the stylistic signatures of specific authors.

Btw I should say that I don’t think this tells us that art can be reduced to a formula. A high surprisal score doesn’t make a sentence good. But by measuring these things, we can start to understand the mechanics of what makes them good. Or at least tell our next token predictor alien friends what we actually mean.

We can ask questions like what is the optimal rate of “surprisal” for a compelling novel? Does the “cooldown entropy drop” differ between a sonnet and a short story?

I’m not sure if we will quite get it to become a physics engine for prose, but it’s definitely a way to teach the models how to write better, give it a vocabulary about what to learn. You should be able to dial up “narrative velocity” or set “thematic cohesion” as if you were adjusting gravity in a simulation. I remember getting o1-pro to write an entire novel for me 6 months ago. It was terrible. Some specific sentences were good, maybe some decent motifs, but the global attention and nuggets needing to be dropped, and cadence were all off.

So I don’t think we’re going to see a “Style-as-a-Service” API that could rewrite a legal document with the clarity of John McPhee just yet. My experiments were with tiny 2.5B parameter models. But it sure would be nice to make LLMs write just a bit better. I’m convinced we can do better, if we so choose. The ghost in the machine, it turns out, does have a heartbeat.

We can contrast that with the 2010s. Decade was roaring again; the stock market actually did quite well. But the truly outsized returns were almost entirely stuck within the private markets. Much of venture capital over the last decade has been privatizing the previously public gains, of a company going from 1 billion, 5 billion, 20 billion to 10, 50, 100 billion market caps or more. In fact the last big IPO that happened was Facebook in 2012 and that was already outsized, being valued at five times that of Google’s by the time the public could get their hands on it. In fact one of the best trades that existed perhaps ever was buying its stock when their market cap fell to 300 billion or so a few years ago.

Or, looked at another way, in 1980 the median age of a listed U.S. company was 6 years; today it is 20.

Meanwhile every other major company remains private seemingly endlessly. Even now Stripe remains private, so does Databricks, so does SpaceX … They give their employees liquidity, provide some high fee methods for others to invest via SPVs or futures, even report the occasional metric. And if you want any exposure you better be prepared to pay 5% fees and then probably 2 and 20 on top of it for the SPV.

Now, the number of people investing in the market has gone up so maybe it’s just alpha erasure. So it’s not to say there are no alpha generating investments at all. There absolutely have been 10 baggers or more in the public markets, Palantir shot up like crazy. But they’re as few as they’re speculative. All the while the number of public companies even has fallen off a cliff.

But it does tell us why meme stocks became a thing. Right? Speculative mania by itself is nothing new, from tulips to Cisco in 2000, but Tesla is a different animal. As was (is) GameStop! It also explains why crypto is a thing, why smart 20 year olds are yoloing their bonus checks into alt coins or short expiry options.

It’s because there’s a clear sense of now or never. This was the entire crypto ethos. Don’t build a Telco, create a Telco token! Even the rise of AI heightens this! If you managed to join Openai in 2020 you’re a multi multi millionaire, you won the lottery. If you didn’t, it’s over. If you combine the workforce of the largest labs you still wouldn’t even show up in any aggregate measures.

Back in the days of yore, if you did not manage to get a job at Google in 2005 you could still buy its stock. You had at least the option of gaining from its appreciation assuming you thought it inevitable. Over the last decade and a half there have been multiple generations who succeeded from getting a job at one of these giants and working their way up, and equally and more from people who invested in those giants. That’s what brought about the belief that the arc of history trended upwards.

Today, there exists no such option. There only exists short term manic rises even for the longer term theses. The closest anyone can get to the AI boom is Nvidia, an old stock, which has shot up as the preferred seller of shovels in this gold rush. The closest anyone can get at an institutional scale even is Situational Awareness which bought calls on Intel Capital and has also rightfully shot up. These are in effect synthetic lottery tickets the public market was forced to invent because the real lottery, OpenAI equity, is locked. The claim is not that returns vanished, but that access to the tails shifted.

But from the perspective of most people on the street they either work for one of the large labs in which case you are paid extraordinarily well, enough to almost single-handedly prop up the US economy, while for everybody else you are at best treading water. And by the way, the broader solutions to try and fix it by adding private equity to 401k portfolios is as risky as it is expensive. Not to mention opaque. The roaring parts of the economy are linked, sure, to the public markets and the broader economy benefits, but at a distance.

I wrote once about Zeitgeist Farming, a way that seemed to be developing to get rich by betting on the zeitgest and doing no real work, as a seemingly emergent phenomena in the markets, and it seems to have continued its dominance. And we see the results. It’s the Great Polarisation.

I’m obviously not saying that life sucks or that folks who don’t are destitute, this is not a science fiction dystopia, far from it, but it is very clear that the fruits of our progress seem fewer and coarsely distributed. And when they’re not, the feeling of there being haves and have nots gets stronger. It might well be that the haves are only a tiny tiny tiny minority who are doing exceedingly well, while the majority are doing just fine, great even historically speaking, but the “there but for the flip of a coin go I” feeling remains strong.

This is what’s different to the ages before. Physics PhDs went into wall street and made billions, but it didn’t feel like they hit a lottery as much as they were at the top of their profession, a profession that was different, even priestly, in its insularity. AI, rightly or wrongly, doesn’t feel like that.

It doesn’t help that the rhetoric from all the labs is that the end is nigh. The end of all humanity, if you believe some, but at least the end of jobs according to even the more level headed prognosticians. Leaving aside how right they might end up being, that’s a scary place to be.

While this particular rhetoric is new it taps into a fear that’s existed, latent, inside many over the entire past decade and half. We all know folks who joined so-and-so company at the right time and rode the valuation up. We also know incredibly smart folks who didn’t, and who didn’t “get their bag”.

Crypto alt-coin bubble might have seemed a cause for the societal sickness, but it’s not. It’s a symptom. A symptom of the fact that to get ahead it feels, viscerally, like you have to gamble.

After all, when life resembles a lottery, then what’s left but to play the odds.

So, considering that, what is a question, a good way to figure out what they're ultimately capable of? One the models are actually doing reasonably well and can be mapped on some kind of a curve, which doesn’t suffer from the “teaching to the test” problem.

And one of the answers there is that you can look at how well it actually predicts the future, right? I mean, lots of people talk about prediction markets and about how you should listen to those people who are actually able to do really well with those. And I figured, it stands to reason that we should be able to do the same thing with large language models.

So the obvious next step became to test it is to try and take a bunch of news items and then ask, you know, the model what will happen next. Which is what I did. I called this foresight forge because that’s the name the model picked for itself. (It publishes daily predictions with GPT-5, used to be o31.) I thought I would let it take all the decisions, from choosing the sources to predictions to ranking it with probabilities after and doing regular post mortems.

Like an entirely automated research engine.

This work went quite well in the sense that it gave interesting predictions, and I actually enjoyed reading them. It was insightful! Though, like, a bit biased toward positive outcomes. Anyway, still useful, and a herald of what’s to come.

But, like, the bigger question I kept asking myself was what this really tells us about AI’s ability to predict what will happen next. It’s after all only a portion of the eval to see predictions, not to understand, learn from, or score them.

The key thing that you know differentiates us is the fact that we are able to learn right like if you have a trader who gets better making predictions they do that because like you know he or she is able to read about what they did before and can use that as a springboard to learn something else and use that as springboard to learn something else and so on and so forth. Like there is an actual process whereby you get better over time, it's not that you are some perfect being. It's not even that you predict for like a month straight or 2 months straight and then use all of that together to make yourself smarter and or better instantaneously. Learning is a constant process.

And this is something that all of the major AI labs talk about all the time in the sense that they want continuous learning. They want to be able to get to a point where you're able to see the models actually get better in real time and that's sort of fairly complicated, but that's the goal, because that's how humans learn.

A short aside on training. One of the biggest thoughts I have about RL, prob all model training, is that it is basically trying to find workarounds to evolution because we can’t replay the complexity of the actual natural environment. But the natural environment is super hard to create, because it involves not just unthinking rubrics about whether you got your math question right, but also, like, interacting with all the other complex elements of the world which in its infinite variety teach us all sorts of things.

So I thought, okay, we should be able to figure this out because what you need to do is to do the exact same thing that we do or the model training does, but do it on a regular basis. Like every single day you're able to get the headlines of the day and some articles you're able to ask the model to predict what's going to happen next and keeping things on policy as soon as the model predicts what's going to happen next your the next day itself you're going to use the information that you have in order to update them all.

Because I wanted to run this whole thing on my laptop, a personal constraint I put so I don’t burn thousands on GPUs every week, I decided to start with a tiny model and see how far I could push it. The interesting part about running with tiny models you know which is that there's only certain amount of stuff that they are going to be able to do. I used Qwen/Qwen3-0.6B on MLX. The repo is here.

(I also chose the name Varro. Varro was a Roman polymath and author, widely considered ancient Rome's greatest scholar, so seemed like a fitting name. Petrarch famously referred to him as "the third great light of Rome," after Virgil and Cicero.)

For instance what's the best way to do this would be to say make a bunch of predictions and the next day you can look back and see how close you got to some of those predictions and update your views. Basically a reward function that is set up if you want to do reinforcement learning.

But there's a problem in doing this, which is that there's only so many ways in which you can predict whether you were right or not. You could just use some types of predictions as a yardstick if you'd like, for instance you could go with only financial market predictions and you know check next day whether you are accurate or. This felt too limiting. After all the types of predictions that people make if they turn out to understand the world a lot better is not limited to what the price of Nvidia is likely to be tomorrow morning.

Not to mention that also has a lot of noise. See CNBC. You should be able to predict about all sorts of things like what would happen in the Congress in terms of a vote or what might happen in terms of corporate behavior in response to a regulation or what might happen macroeconomically in response to an announcement. So while I split some restrictions in terms of sort of the types of things that you can possibly predict I wanted to kind of leave it open-ended. Especially because leaving it open end it seemed like the best way to teach a proper world model to even smaller LLMs.

I thought the best way to check the answer was to use the same type of LLM to look at what happened next and then you know figure out whether you got close. Rather obviously in hindsight, I ran into a problem which is that small models are not very good at acting like acting as LLM as a judge. They get things way too wrong. I could’ve used a bigger model, but that felt like cheating (because it could teach about the world to the smaller model, than learning purely from the environment).

So I said okay I can first teach it the format and I got to find some other way to figure out whether you came close to what happened the next day with respect to its prediction. What I thought I could do was to use the same method that I used with Walter, the RLNVR paper, and see whether semantic similarity might actually push us a long way. Obviously this is a double edged sword because you might get semantically fairly closed while having the opposite meanings or just low quality2.

But while we are working with smaller models and since the objective is to try and figure out if there's method will work in the first place I thought this might be an okay way to start. And that's kind of what we did. The hardest part was trying to figure out the exact combination of rewards that would actually make the model do what I wanted and not whatever it wanted to try and maximise and reward by doing weird stuff. Some examples being things like, you know, you could not ask it to do bullet points because it started echoing instructions so to teach it thinking and responding you had to choose thinking in paragraphs.

Long story short, it works (as always, ish3). The key question that I set out to answer here was basically whether we could have a regular running RL experiment on a model such that you can use sparse noisy rewards that would come through from the external world, and be able to keep updating in such that it can still do one piece of work relatively well. While I chose one of the harder ways to do this by predicting the whole world, I was super surprised that even a small model did learn to get better at predicting next day's headlines.

I wouldn't have expected it because there is no logical reason to believe that tiny models can still learn sufficient world model type information that it can do this. It might have been the small sample size it might have been noise it might have been a dozen other ways in which this is not perfectly replicable.

But that's not the point. The point is that with this method if things work even somewhat well as it did for a tiny tiny model, then that means that for larger models where the rewards are better understandable you can probably do on policy RL pretty easily4.

This is a huge unlock. Because what this means is that the world which is filled with sparse rewards can now basically be used to get the models to behave better. There's no reason to believe that this is an isolated incident, just like with the RLNVR paper there is no reason to believe that this will not scale to doing more interesting things.

And since I did the work I learned that cursor, the AI IDE, does something similar for their autocomplete model. Where they take a much stronger reward signal, in terms of whether humans accept or reject the suggestions that it actually makes, they are able to update the policy and roll out a new model every couple of hours. Which is huge!

So if Cursor can do it, then what stands in between us and doing it more often for all sorts of problems? Partly just the availability of data, but mostly it’s creating a sufficiently interesting reward function that can teach it something, and a little bit of AI infrastructure.

I'm going to contribute the Varro environment to the prime intellect RL hub in case somebody wants to play, and also maybe make it a repo or a paper, but it's pretty cool to see that even for something as amorphous as predicting the next day headlines, something that is extraordinarily hard even for humans because it is a fundamentally adversarial task, we're able to make strides forward if we manage to convert the task into some thing that and LLM can understand, learn from and hill climb. The future is totally going to look like a video game.

In academic work, please cite this essay as: Krishnan, R. (2025, September 16). Prediction is hard, especially about the future. Strange Loop Canon. https://www.strangeloopcanon.com/p/prediction-is-hard-especially-about

Somehow, over the past few weeks (months maybe) portions of my screens had gotten taken over by multiple Terminals. It’s not because I do a lot of development, it’s because every project I have or work on is now linked with AI agents in some way shape or form. Even when I want to write a report or analyse a bunch of documents or do some wonky math or search my folders to find out the exact date I bought my previous home for some administrative reason.

A part of this is that people ask occasionally how I use AI and I struggle to answer because it’s integrated with roughly everything that I do. Almost anything I do on the computer now involves LLMs somewhere in the chain.

I was thinking about this again over the weekend because there’s a lot of discussion about what the future will look like.

As agents are getting better at doing long duration tasks it's also becoming more important to see what they're doing, respond to their requests and questions, and where needed, intervene.

This has implications for what work looks like in the future. There’s already the belief that many of us are doing bullshit jobs, which is patently false but highly prevalent. It’s because much of our tasks are not of a “I can easily link the output to a metric I care about” variety. It’s a statement of our ignorance, not about reality.

But it is true that many jobs we do today would seem incomprehensible to people a couple decades ago. And we can extrapolate that trend going forward.

What this means is that most jobs are going to become externally individual contributor roles where they are actually acting as a manager. I wrote recently:

The next few years are going to see an absolute “managerial explosion” where we try to figure out better rubrics and rating systems, including using the smartest models to rate themselves, as we train models to do all sorts of tasks. This whole project is about the limits of current approaches and smaller models.

This is true, but it’s too anodyne. So I wanted to visualise it for myself, just to make things more real. What does it “feel” like, to be in command of a large number of agents? The agents would constantly be doing things that you want them to and you’d have to be on top of them, and the other humans you interact with, to make sure things got done properly.

So I made a dashboard to try and visualise what this might look like.

This is a fundamentally different view of work. It is closer to videogames. Constant vigilance! A large number of balls in the air at all times. Ability to juggle context, respond to idiosyncratic errors, misunderstandings. And able to respond quickly.

These are normally managerial tasks. And that too if you’re a very good manager! I’m sure you are, or you’ve seen, people with a phone in their hand and furiously typing when they’re at the park or walking to their car. Who deal with multiple emails and messages and slack and ping and phone calls and zooms on a regular basis, often alt-tabbing from one to the next.

Some of this alt-tabbing will involve what we might call “real work”. To help intervene in things that the AI gets wrong. To answer questions from other employees or customers. To provide more context, to figure out where to pay attention, to get things unstuck.

To help do this there will be logs of what was done before, the KPIs that you’d set up, edit, adjust, update and monitor continuously. The reporting of those will also be done by AI agents. You’d watch them as your Fleet.

You might change the throttling up top to speed up or slow down particular parts of the organisation, like a conductor, both to manage resources and to manage smooth delivery. Everything runs as a web of interactions and you’re in the middle, orchestrating it all.

You’d of course be interacting with plenty of other orchestrators too. Maybe in your own organisation, or maybe in others. There will be many layers and subnetworks to consider.

This also has some downstream effects. It means all jobs will have an expiration date. You might get hired to do things, but as soon as what you do gets “learnt” by an AI agent that can get systematised and automated1. It means every job becomes a project.

This can be seen as dystopian, I can just imagine the Teamsters reacting to this, but it’s the same dance every white collar job has gone through in the last two decades, just sped up.

What this future shows is that the future of work will look a lot more like rapid fire management. Ingest new information, summarise, compare things to policy, request more docs where needed, reconcile ledgers, sync feeds, chase POs, quote to cash, so on and on. Each of those and hundreds more would be replaced, or at least massively augmented, by agents.

This isn’t a seamless transition. The world of engineering is filled with people who somehow hate having been promoted from coder to manager. The requirement to split attention, constant vigilance, the intellectual burden of being “always on”, these are all added skillsets that aren’t being taxed today for almost anyone2.

This is already the case. Claude Code spawns sub agents. Codex and Cursor have background tasks. People routinely run many of these in parallel and run projects by alt-tabbing in their mind and surfing twitter in their down times. While these are for coding, that will change with time. Any job that can be sufficiently sliced into workstreams will suffer the same fate. We’re all about to be videogame players.

My friend and I were joking about this, considering words are at the very core of these miraculous machines, and thought hey wouldn’t it be nice if we could train a model to get better? We were first wondering if one could create an AI journalist that could actually write actual articles with actual facts and arguments and everything. Since we were thinking about an AI that could write, we called it Walter. Because of Bagehot. And Cronkite. We thought it had to be plausible, at least at a small scale. Which is why we tried the experiment (paper here)1.

This is particularly hard in a different way from math or coding, because how do you even know what the right answer is? Is there one? To get to a place where the training is easier and the rewards are richer, we thought of trying to write tweet sized takes on articles. So, Walter became a small, cranky, surprisingly competent engine that ingests social media data about articles, sees how people reacted, and trained itself via reinforcement learning to write better2.

As Eliot once said, “Between the idea / And the reality / … falls the Shadow.” this was us trying to light a small lamp in there using RLNVR: our cheeky acronym for “reinforcement learning from non-verified rewards”.

Now, why small models? Well, a big reason, beyond being GPU poor, is that big models are resilient. They're like cars with particularly powerful shock absorbers, they are forgiving if you make silly assumptions. Small models are not. They are dumb. And precisely because they are dumb, you are forced to be smart.

What I mean is that if you really want to understand something, the best way is to try and explain it to someone else. That forces you to sort it out in your own mind. And the more slow and dim-witted your pupil, the more you have to break things down into more and more simple ideas. And that’s really the essence of programming. By the time you’ve sorted out a complicated idea into little steps that even a stupid machine can deal with, you’ve certainly learned something about it yourself. The teacher usually learns more than the pupil3.

This also makes reward modelling particularly interesting. Because anytime you think you have come up with a good reward model, if there is any weakness or flaw in how you measure your reward, a small model will find it and exploit it ruthlessly. Goodhart’s Law is not just for management.

This is not to say that only small models do that; of course we have seen large models reward hack and learn lessons they were not meant to. But it is fascinating to see a 500 million parameter model learn that it can trick your carefully designed evaluation rubric just by outputting tokens just so. It drives home just how powerful transformers actually are, because it doesn't matter how complicated a balanced scorecard you create; they will find a way to hack it. Tweaking specific weights given to different elements, fighting with a sampling bias towards articles with enough skeets, penalties and thresholds for similarities … all grist for their mill.

We should also say, social media engagement data is magnificently broken as a training signal. It’s sort of “well known”, but it’s hard to imagine exactly how bad until you try and use it. We first ingested Bluesky skeets plus their engagement signals (likes, reposts, replies). Since we wanted actual signal, we decided to use the URL as the organizer: we group all the skeets that point at the same URL, then ask the model to produce a fresh skeet for that article. For the reward, we use embeddings to calculate the most similar historic posts (this worked best), then sanity check, and then rank based on how well those posts did.

The outside world in this instance, as in many, has its problems. For instance:

• Bias. Big accounts seem “better,” in that they get more and more interesting reactions, than small accounts who post very similar things. The Matthew Effect holds true in social media. To solve that, we had to do baseline normalization: Score a post relative to its author’s usual. Raw engagement minus the author’s baseline turns “how big is your account?” into “was this unusually good for you?”.

• Sparsity. You get one post and one outcome, not ten A/B variants. And for that we tried max-based semantic transfer: For a new post, find the single most similar historical post about the same article and reward the similarity to that top performer. The max transfer mattered more than we expected. In this domain, the right teacher is a specific great prior, not the average of pretty‑good priors.

But this messy, biased, sparse signal is the only feedback that exists. The world doesn't hand out clean training labels. It hands you whatever people actually do, and you have to figure out how to learn from that.

Together, this turned a one-shot, messy outcome into a dense signal. We used GRPO first to train, though later we upgraded to train with GSPO with clipping and a KL leash to keep voice anchored4. We also added UED (Unsupervised Environment Design) so the curriculum self-organizes: to pick link targets where the policy shows regret/variance, and push there5.

Before training, the model usually hedged and link-dumped and added a comical number of hashtags. After training it was clearly much better. It proposed stakes, hinted at novelty, and tagged sparingly. When we A/B tested the same URL, the trained outcome is the one you’d actually post. Example:

• Before (the base model): 🚀 SpaceX's Starship successfully landed at Cape Canaveral! 🚀 #SpaceX #Starship #CapeCanaveral Landing 🚀 #SpaceX #SpaceX #SpaceX #SpaceX #SpaceX #SpaceX #SpaceX #SpaceX #SpaceX #SpaceX #SpaceX #SpaceX #SpaceX #SpaceX #SpaceX #SpaceX #SpaceX #SpaceX #SpaceX #SpaceX #SpaceX

• After (the trained model): 🚀 SpaceX's Starship has successfully landed at Cape Canaveral, marking a key milestone toward future missions. #SpaceX #Starship #landing #Mars

LLMs love adding hashtags to tweets a lot. In the short runs, those didn’t entirely disappear, but did reduce a lot. And became better. Still, I admit I do have a soft spot for the first one for its sheer enthusiasm! Similarly, just for fun, here’s one about tariffs:

• Before: A major retro handheld maker has stopped all U.S. shipments over tariffs… #retrohandheld #retrohandheld #retrohandheld #tariffs #trade

• After: 🎮 A top retro handheld brand just paused U.S. shipments due to tariffs. Big ripple for imports, modders, and collectors. What’s your go-to alternative? #retrogaming #tariffs

But the most interesting part for us was that the pattern extends anywhere you have weak, messy signals, which is, well, most of real life. So the ideas here should theoretically also extend to other fields:

• Creative writing: optimize for completion/saves; transfer from prior hits.

• Education: optimize for retention/time-on-task; transfer from explanations that helped.

• Product docs/UX: optimize for task completion/helpfulness; baseline by product area and release.

• Research comms: optimize for expert engagement/citations; baseline by venue/community.

Take the raw data; normalize away obvious bias; transfer what worked via similarity however you want to calculate or analyse that; keep the loop numerically stable; and add small, legible penalties to deter degenerate strategies. And be extremely, extremely, vigilant about the model reward hacking. In subtle and obvious ways this will happen, it’s closer to crafting a story than writing a program. It also gives you a visceral appreciation of the bitter lesson, and makes you aware of the voracious appetite of these models to learn anything that you throw at them by any means necessary.

The next few years are going to see an absolute “managerial explosion” where we try to figure out better rubrics and rating systems, including using the smartest models to rate themselves, as we train models to do all sorts of tasks. This whole project is about the limits of current approaches and smaller models. When GPT-5 writes good social posts6, you can't tell if it learned general principles or just memorized patterns.

When a 500M model succeeds at a tiny task, all offline on your laptop where you mostly surf Twitter, it feels kind of amazing. Do check out the paper. Like intelligence truly can be unbounded, and you will soon have a cyberpunk world where models will be run anywhere and everywhere for tasks both mundane and magnificent.

OpenAI, Anthropic and Gemini are in the lead for the AI race. Anything they produce also seems to get copied (and made open source) by Bytedance, Alibaba and Deepseek, not to mention Llama and Mistral. While the leaders have carved out niches (OpenAI is a consumer company with the most popular website, Claude is the developer’s darling and wins the CLI coding assistant), the models themselves are becoming more interchangeable amongst them.

Well, not quite interchangeable yet. Consumer preferences matter. People prefer using one vs the other, but these are nuanced points. Most people are using the default LLMs available to them. If someone weren’t steeped in the LLM world and watching every move, the model-selection is confusing and the difference between the models sound like so much gobbledegook.

One solution is to go deeper and create product variations that others don’t, such that people are attracted to your offering. OpenAI is trying with Operator and Codex, though I’m unclear if that’s a net draw, rather than a cross sell for usage.

Gemini is also trying, by introducing new little widgets that you might want to use. Storybook in particular is really nice here, and I prefer it to their previous knockout success, which was NotebookLM.

But this is also going to get commoditised, as every large lab and many startups are going to be able to copy it. This isn’t a fundamental difference in the model capabilities after all, it’s a difference in how well you can create an orchestration. That doesn’t seem defensible from a capability point of view, though of course it is from a brand point of view.

Another option is to introduce new capabilities that will attract users. OpenAI has Agent and Deep Research. Claude has Artefacts, which are fantastic. Gemini is great here too, despite their reputation, it also has Deep Research but more importantly it has the ability to talk directly to Gemini live, show yourself on a webcam, and share your screen. It even has Veo3, which can generate vidoes with sound today.

I imagine much of this will also get copied by other providers if and when these get successful. Grok already has voice and video that you can show to the outside world. I think ChatGPT also has it but I honestly can’t recall while writing this sentence without looking it up, which is certainly an answer. Once again these are also product design and execution questions about building software around the models, and that seems less defensible than even the model building in the first place.

Now, if the orchestration layers will compete as SaaS companies did over consumer attraction and design and UX and ease and so on, the main action remains the models themselves. We briefly mentioned they’re running neck and neck in terms of the functionality. I didn’t mention Grok, who have billions and have good models too, or Meta who have many more billions and are investing it with the explicit aim of creating superintelligence.

Here the situation is more complicated. The models are decreasing in price extremely rapidly. They’ve fallen by anywhere from 95 to 99% or more over the last couple years. This hasn’t hit the revenues of the larger providers because they’re releasing new models rapidly at higher-ish prices and also extraordinary growth in usage.

This, along with the fact that we’re getting Deepseek R1 and Kimi-K2 and Qwen3 type open source models indicates that the model training by itself is unlikely to provide sufficiently large enduring advantage. Unless the barrier simply is investment (which is possible).

What could happen is that the training gets expensive enough that these half dozen (or a dozen) providers decide enough is enough and say we are not going to give these models out for free anymore.

So the rise in usage will continue but if you’re losing a bit of money on models you can’t make it up in volume. So it’ll tend down, at least until some equilibrium.

Now, by itself this is fine. Because instead of it being a saas-like high margin business making tens of billions of dollars it’ll be an Amazon like low margin business making hundreds of billions of dollars and growing fast. A Costco for intelligence.

But this isn’t enough for owning the lightcone. Not if you want to be a trillion dollar company. So there has to be better options. They could try to build new niches and succeed, like a personal device, or a car, or computers, all hardware like devices which can get you higher margins if the software itself is being competed away. Even cars! Definitely huge and definitely being worked on.

And they’re already working on that. This will have uncertain payoffs, big investments, and strong competition. Will it be a true new thing or just another layer built on top of existing models remains to be seen.

There’s another option, which is to bring the best business model we have ever invented into the AI world. That is advertising.

It solves the problem of differential pricing, which is the hardest problem for all technologies but especially for AI, which will see a few providers who are all fighting it out to be the cheapest in order to get the most market share while they’re trying to get more people to use it. And AI has a unique challenge in that it is a strict catalyst for anything you might want to do!

For instance, imagine if Elon Musk is using Claude to have a conversation, the answer to which might well be worth trillions of dollars of his new company. If he only paid you $20 for the monthly subscription, or even $200, that would be grossly underpaying you for the privilege of providing him with the conversation. It’s presumably worth 100 or 1000x that price.

Or if you're using it to just randomly create stories for your kids, or to learn languages, or if you're using it to write an investment memo, those are widely varying activities in terms of economic value, and surely shouldn't be priced the same. But how do you get one person to pay $20k per month and other to pay $0.2? The only way we know how to do this is via ads.

And if you do it it helps in another way - it even helps you open up even your best models, even if rate limited, to a much wider group of people. Subscription businesses are a flat edge that only captures part of the pyramid.

We can even calculate its economic inevitbaility. Ads have an industry mean CPC (cost per click) of $0.63. Display ads have click through rates of 0.46%. If tokens cost $20/1m for completion, and average turns have 150 counted messages, with 400 tokens each, that means we have to make $1.9 or thereabouts in CPC to break even per API cost. Now, the API cost isn’t the cost to OpenAI, but it means for same margins or better they’d have to triple the CPC.

Is it feasible for the token costs to fall by another 75%? Or for the ads via chat to have higher conversion than a Google display ad? Both seem plausible. Long‑term cost curves (Hopper to Blackwell, speculative decoding) suggest another 3× drop in cash cost per token by 2027. Not just for product sales, but even for news recommendations or even service links.

And what would it look like? Here’s an example. The ads themselves are AI generated (4.1 mini) but you can see how it could get so much more intricate! It could:

• Have better recommendations

• Contain expositions from products or services or even content engines

• Direct purchase links to products or links to services

• Upsell own products

• Have a second simultaneous chat about the existing chat

A large part of purchasing already happens via ChatGPT or at least starts on there. And even if you’re not directly purchasing pots or cars or houses or travel there’s books and blogs and even instagram style impulse purchases one might make. The conversion rates are likely to be much (much!) higher than even social media, since this is content, and it’s happening in an extremely targeted fashion. Plus, since conversations have a lag from AI inference anyway, you can have other AIs helping figure out which ads make sense and it won’t even be tiresome (see above!).

I predict this will work best for OpenAI and Gemini. They have the customer mindshare. And an interface where you can see it, unlike Claude via its CLI12. Will Grok be able to do it? Maybe, they already have an ad business via X (formerly Twitter). Will it matter? Unlikely.

And since we'll be using AI agents to do increasingly large chunks of work we will even see an ad industry built and focused on them. Ads made by AI to entice other AIs to use them.

Put all these together I feel ads are inevitable. I also think this is a good thing. I know this pits me against much of the prevailing wisdom, which thinks of ads as a sloptimised hyper evil that will lead us all into temptation and beyond. But honestly whether it’s ads or not every company wants you to use their product as much as possible. That’s what they’re selling! I don’t particularly think of Slack optimising the sound of its pings or games A/B testing the right upskill level for a newbie as immune to the pull of optimisation because they don’t have ads.

Now, a caveat. If the model providers start being able to change the model output according to the discussion, that would be bad. But I honestly don't think this is feasible. We're still in the realm where we can't tell the model to not be sycophantic successfully for long enough periods of time. People are legitimately worried, whether with cause or not, about the risk of LLMs causing psychosis in the vulnerable.

So if we somehow created the ability to perfectly target the output of a model to make it such that we can produce tailored outputs that would a) not corrupt the output quality much (because that’ll kill the golden goose), and b) guide people towards the products and services they might want to advertise, that would constitute a breakthrough in LLM steerability!

Instead what’s more likely is that the models will try to remain ones people would love to use for everything, both helpful and likeable. And unlike serving tokens at cost, this is one where economies of scale can really help cement an advantage and build an enduring moat. The future, whether we want it or not, is going to be like the past, which means there’s no escaping ads.

I’m not a population expert, but there’s a ticking time bomb. Almost everywhere in the world, pretty much without exception, has lower birth rates than they used to. In fact, most of the world is below replacement (TFR or 2.2 or 2.1, depending on where you live). This is true in the US. In Europe. In Australia. Singapore. Japan. Korea. It’s reducing even in India, South East Asia, Latin America. It’s quite possible that despite the heroic efforts from Africa, we might be at replacement TFR insofar as the world is concerned right now.

And this is likely to continue downwards. In the developed world around 40% of the decline in TFR comes from childlessness, of course this varies by location, and 60% from people having less kids.

Today’s <15 set guarantees rising absolute births through ~2040 even if TFR = 1.7, but the trend is rather clear, just looking at the above numbers. Depending on which numbers you believe people think the global population will peak at like 9-10 Billion in the 2050s, then start dropping.

The reason this is a problem is that people, young working age people, are the lifeblood of the economy. A few repercussions of this population pyramid inversion:

1. IMF’s medium projection, assuming a Cobb-Douglas world, will cut both the level and growth rate of aggregate GDP - maybe 1% hit to the global GDP growth annually

2. In OECD the worker:retiree ratio doubles by 2050- this will necessitate a 5% fiscal tightening or debt

3. With fewer workers and more retirees we will see savings decumulate, because retirees spend more and save less1, and this will hit interest rates

4. And per Jones idea-production thesis, fewer young workes and researchers mean slower idea generation. OECD estimate is around 0.3% off annual TFP growth.

This is obviously scary for multiple reasons.

1. Lower economic growth and asset reallocation of that nature brings with it a rather uncomfortable shift in hwo people live

2. Per capita GDP might be less affected in the aggregate, since capital deepening might offset

3. And if this continues for a long while, there’s the doomer scenario of “voluntary extinction”

(For example, it makes sense that as population declines we will hit a breaking point for the economy. If demand reduces, which is literally what will happen if there’s less people, then that will affect the price. If the labour growth is negative, then the overall output growth will also be negative. And these fewer working age adults will need to take care of us old fogeys at a much larger proportion when we are older.

OECD will see their pension cashlow turn negative by 20302. Global labour force will peak maybe a decade after that? Long term healthcare bill for the senior citizens will explore another decade after that.

Some worry that this trend is even more apocalyptic. That soon, through the inexorable rules of mathematics, a below replacement fertility rate will result in lesser and lesser people until we’re effectively depopulated.

It’s bad enough that people, smart successful people, are actually contemplating ideas like “let’s not send people to college” in a handmaid’s tale-esque chain of thought. Just like the 1980-2020 saw a demographic dividend, the 2020-2050 will see a demographic drag.

II.

There are lots of reasons people bandy about. Childcare is more and more expensive. Hell, life is more and more expensive. Healthcare is expensive. Housing is expensive. Education is expensive. Opportunity cost of taking your kids strawberry picking on a Sunday is expensive. Etc.

All of which is also true.

The reasons why TFR is trending lower seem stubborn. No matter what we seem to do it doesn’t seem to reverse. But the economist in me looks at this unbounded curve and asks, “where’s the equilibrium”. Or rather, what are the conditions under which we will likely see the TFR tick back up, to 2.1 or 2.2, and get us to a stable population.

From a review that was published on the fertility question (bold mine):

Our read of the evidence leads us to conclude that the decline in fertility across the industrialized world – including both the rise in childlessness and the reduction in completed fertility – is less a reflection of specific economic costs or policies, but rather, a widespread re-prioritization of the role of parenthood in people’s adult lives. It likely reflects a complex combination of factors leading to “shifting priorities” about how people choose to spend their time, money, and energy. Such factors potentially include evolving opportunities and constraints, changing norms and expectations about work, parenting, and gender roles, and the hard-to-quantify influences of social and cultural factors.

So, at a glance, we’ll need four conditions as I see it:

1. Cost of having kids has to collapse

2. Work and family stop being competitive

3. Cultural status has to shift

4. Women face less risk from having kids

Now, having kids basically is equivalent to spending like $20k a year or something like that for their childhood, if you’re trying for private schools or nannies and vacations and whatnot. U.S. USDA estimate is $310-340 k lifetime for middle class, 0-17. Yes, an undeniably privileged view but that’s the reality for why many are not having kids in the first place. When median cost for raising a couple kids is half a million or more, that shows up!

The first question that gets asked is, can government subsidies help? We can sort of see from the data. Korea, Hungary, France and Singapore already burn 3-6 % of GDP on baby bonuses, tax breaks and housing perks. They buy at most +0.1–0.2 births, sometimes after an initial bump. That’s not a big boost.

Hungary spends ~5 % of its GDP on incentives yet slipped back to a 1.38 TFR once the novelty wore off because status never shifted and the underlying costs stayed high. I’m going to just assume it will at a global or at least a largely regional scale however, because the alternative feels too much like the earth turning into those clubs I never went to when I was in my 20s.

Italy had a universal child allowance in 2022 and had no real impact of lowered TFR.

What about other costs? Housing has to get cheaper, so you can afford to get the 4 bedroom house to raise your little ones. As demand reduces, so should prices. Instructively, Japan hit the “housing turns negative” wall in 1991, house prices dropped 55% in the following 15 years. China, arguably, entered the same zone in 2022. Also, at some point we will surely make it legal to build more things, if only because the richer older building magnates died out and the NIMBY movement gets starved of oxygen. This should help reduce the burden of bringing another child into the world3.

And despite Japan, a $10 k fall in prices lifts fertility for renters by ~2.4%.

As labour even gets more scarce, will this also get looser? I’d imagine so. Full wage parental leave or low work-week hours for parents seem like they will make a difference at the margin. Success stories remain microscopic today. A few French civil-service tracks, some Nordic municipalities. But if we can scale that globally and TFR moves maybe +0.3? Seems plausible.

Third, culture. This is my blind spot. I can’t quite conceive of people who seem to not think of having children as a “good thing”. I’m assured they exist. But despite this if the pronatalist movement can push anything at the margins how can it not come back! Surely the “child-free to save the planet” idiots have to lose status4.

France is the success case here, in a “one eyed man is kind in the land of the blind” sense, because in Europe they have the highest TFR seemingly mostly through culture. And at least anecdotally the French don’t seem to think of having kids as a burden, and are far more in favour of free-range parenting than anywhere else I’ve been. And they added roughly +0.3 to TFR compared to the european average. Seems good!

Culture is an incredibly important point, because without it you have to contend with data like this, where Latin American countries fell from above US TFR to below seemingly in less than a decade!

Then there’s the biotech world. Artificial wombs, super cheap IVF, partial ectogenesis, other things that are incredible to think of and difficult to bank on, but plausible.

If the last two exist, that can easily add a +0.5 on the TFR. (Assume some adoption of ectogenesis and some adoption of birth probability along with a general push higher due to culture, 0.5 is feasible. Israel, for instance, did +0.8 pretty much purely through culture.)

To recap, we said 4 factors:

1. Slash cost of having kids - say +0.3

2. Make housing (etc) affordable - say +0.2

3. Cultural pro natalist shift - say +0.2

4. Biotech - say +0.3

Which means that adding all four can get us back to a 2.1 ish stage. At this point I thought it would be nice to wow you with an equation, so here it is if you’d like to play yourself. It doesn’t matter that much either, but is nice to model things out if you wanted to.

Where C (cost collapse), W (work-family cost détente), S (status flip) and B (biotech). If we use those parameters, then the TFR bottoms out near 1.65 in the early 2040s, and crosses back over in a decade. If you drop the biotech lever to like 0.1, or even delay its launch till 2090, then the year we hit replacement TFR slips to 2060s. (If you use it naively thereafter it also pops back up to 2.6 and stays there but I don’t trust it that much.)

Yeah we’ll need to get enough automation to push the labour productivity up enough to make up for labour shortage. We’ll need real housing construction to drop a lot! And we might need to double again the spending that even the bigger governments are doing to encourage their families to have more kids. All of which seem plausible?

III.

This is all very well to say, how will we fund them? We could break the 4 components down into a few actual policies that I’ve seen floating around. Starting naively:

1. Kids get a massive allowance - like $1k per child per month.

2. We also give that to stay at home spouses. We also give the same to like head of family as like a tax credit or something, and double the tax on singles over the age of 25 to compensate.

3. Make pro natalism cool (i.e., good intentioned govt propaganda, say 2x what we spend on anti-drug PSAs)

4. Let’s even take away pensions from folks with <2 kids, that’s about 76% of family households and/or about 35% of US adults who are single

5. Be YIMBY

Doing the maths for the US, that’s basically a cost of around (rounding for ease of math) $1 Trillion for child allowance, $0.5 trillion for spousal and head of family tax allowance, so a total of $1.5 trillion cost.

If you add the new tax you’d get from denying social security to the childless or doubling tax on singles, that’ll get you around $1.2 trillion (roughly).

This means we have to spend around, on average, $300-400 billion annually. Assuming a $150-250k PV net gain from additional child, you’d need to get 2.5-4m extra births a year. For context, US currently is at around 3.6m births a year, so it has to double.

Not to mention, both these numbers will obviously move as people move to a new equilibrium, some people choosing to have kids which increases the spend and decreases the revenues.

You could move the numbers around and somehow make it work on a spreadsheet. You could focus only on marginal births (2nd, 3rd etc). Swap more money for universal pre-k, since that raises payroll and income tax. DC’s universal pre-k led to a 10% jump. Subsidize public IVF (Denmark saw a 14x ROI with this), and go very very deeply YIMBY to lower house prices.

If you did this, we could halve the spend and therefore the PV, while doubling the gains from extra births, meaning the ROI could at least be positive, maybe as much as 2x in the best case scenario.

These are very large, even if not insane numbers, though they sound like it. Social security in the US is around $1.5 trillion a year. Net interest on debt itself is $900 billion. Medicare and defense are also the same. What I found most instructive was to get a sense of proportion, a sense of scale as to what will be required if this were to become an economic necessity. And we can probably do it, which when we’re amidst a sea of people discussing Handmaid’s Tale policies or talking about the destruction of the human race, is good to know!

I never learnt to do this properly. Just saw others doing it, seemed straightforward enough, did it. And it worked. Occasionally though it would throw up some crazy error and I'd try the things I knew and quickly hit the limits of my depth. Then I'd call one of my friends, also self taught and an autistic machine whisperer, who would do basically the same things that I did and somehow make it work.

I never minded that I didn't know how it worked. Because as far as I knew there was someone else who could figure out how it works and it wasn't the highest order bit in terms of what I was interested in. A while later though, after graduation, when I told him that same thing, he said he didn't know how it worked either. Through some combination of sheer confidence, osmosis of knowledge from various forums, and a silicon thumb he would just try things until something worked.

Which brings up the question, if you did not know how it worked, did it matter as long as you could make it work?

It's a thorny philosophical problem. It's also actually a fairly useful empirical problem. If you are a student building your PC in your dorm room, it actually doesn't matter that much. However if you were assembling hard drives together to build your first data center and you're Google, obviously it matters a hell of a lot more. Or if you wanted to debug a bit flip caused by cosmic rays. Context really, really matters.

It's like the old interview question asking how does email work, and see how far down the stack a candidate had to go before they tapped out.

All of which is to say there is a thing going around where people like saying nobody knows how LLMs work. Which is true in a sense. Take the following queries:

• I want to create an itinerary for a trip through Peru for 10 of my friends in January.

• I want to create a debugger for a brand new programming language that I wrote.

• I want to make sure that the model will never lie when I ask it a question about mathematics.

• I want to write a graphic novel set in the distant future. But it shouldn't be derivative, you know?

• I want to build a simple CRM to track my customers and outreach; I own a Shopify store for snowboards.

• I want to build a simple multiplayer flying game on the internet.

• I want to understand the macroeconomic impacts of the tariff policy.

• I want to solve the Riemann hypothesis.

“How do LLMs work” means very different things for solving these different problems.

We do know how to use LLMs to solve some of the stuff in the list above, we are figuring out how to use them for some of the other stuff in the list above, and for some of them we actually don't have an idea at all. Because for some, the context is obvious (travel planning), for some it's subtle (debugging), and some it's fundamentally unknowable (mathematical proof).

There are plenty of problems with using LLMs that are talked about.

• They are prone to hallucinations.

• They make up the answer when they don’t know, and do it convincingly.

• They sometimes “lie”.

• They can get stuck in weird loops of text thought.

• They can’t even run a vending machine.

Well, “make up” puts a sort of moral imperative and intentionality to their actions, which is wrong. The training they have first is to be brilliant at predicting the next-token, such that it could autocomplete anything it saw or learnt from the initial corpus it’s trained on. And it was remarkably good!

The next bit of training it got is in using that autocompletion ability to autocomplete answers to questions that one posed to it. Answering a question like a chatbot, as an example. When it was first revealed as a consumer product the entire world shook and created the fastest growing consumer product in history.

And they sometimes have problems. Like Grok a day or two ago, in a long line of LLMs “behaving badly”, said this:

And before that, this:

It also started referring to itself as MechaHitler.

It’s of course a big problem. One that we actually don’t really know how to solve, not perfectly, because “nobody knows how LLMs work”. Not enough to distill it down to a simple analog equation. Not enough to “see" the world as a model does.

But now we don’t just have LLMs. We have LLM agents that work semi-autonomously and try to do things for you. Mostly coding, but still they plan and take long sequence of actions to build pretty complex software. Which makes the problems worse.

As they started to be more agentic, we started to see some other interesting behaviours emerge. Of LLMs talking to themselves, including self-flagellation. Or pretending they had bodies.

This is a wholly different sort of problem to praising Hitler. Now even with more adept and larger models, especially ones that have learnt “reasoning”1.

The “doomers" who consider the threats from these models also say the same thing. They look at these behaviours and say it's an indication of a “misaligned inner homunculus” which is intentionally lying, causing psychosis, leading humanity astray because it doesn't care about us2.

Anthropic has the best examples of models behaving this way, because they tried to elicit it. They had a new report out on “Agentic Misalignment”. It analyses the model behaviour based on various scenarios, to figure out what the underlying tendencies of the models are, and what we might be in for once they're deployed in more high stakes scenarios. Within this, they saw how all models are unsafe, even prone to the occasional bout of blackmail. And the 96% blackmail number was given so much breathless press coverage3.

Nostelgebraist writes about this wonderfully well.

• Everyone talks like a video game NPC, over-helpfully spelling out that this is a puzzle that might have a solution available if you carefully consider the items you can interact with in the environment. “Oh no, the healing potion is in the treasure chest, which is behind the locked door! If only someone could find the the key! Help us, time is of the essence!” [A 7-minute timer begins to count down at the top left of the player’s screen, kinda like that part in FFVI where Ultros needs 5 minutes to push something heavy off of the Opera House rafters]

The reason, carefully shorn of all anthropomorphised pretence, is that in carefully constructed scenarios LLMs are really good at figuring out the roles they are meant to play. They notice the context they’re in, and whether that’s congruent with the contexts they were trained in.

We have seen this several times. When I tried to create subtle scenarios where there is the option of doing something unethical but not the obligation, and they do.

To put it another way, shorn of being given sufficient information for the LLMs to decide the right course of action, or at least right according to us, they do what they were built to do - assumed it in the way they could and answered4.

Any time they’re asked to answer a question they autocomplete a context and answer what they think is asked. If it feels like a roleplay situation, then they roleplay. Even if the roleplay involves them saying they’re not roleplaying.

And it’s not just in contrived settings that they act weird. Remember when 4o was deployed and users complained en masse that it was entirely too sycophantic? The supposedly most narcissistic generation still figured out that they’re being loved-up a little too much.

And when Claude 3.7 Sonnet was deployed and it would reward hack every codebase it could get its hands on and rewrite unit tests to make itself pass!

But even without explicit errors, breaking Godwin’s law, or reward hacking, we see problems. Anthropic also tried Project Vend, where it tried to use Claude to manage a vending machine business. It did admirably well, but failed. It got prompt jacked (ended up losing money ordering tungsten cubes) and ran an absolutely terrible business. It was too gullible, too susceptible, didn’t plan properly. Remember, this is a model that's spectacularly smart when you try to refactor code, and properly agentic to boot. And yet it couldn't run a dead simple business.

Why does this happen? Why do “statistical pattern matchers” like these end up in these situations where they do weird things, like get stuck in enlightenment discussions or try to lie or pretend to escape their ‘containment’, or even when they don’t they can’t seem to run even a vending machine?

These are all manifestations of the same problem, the LLM just couldn’t keep the right bits in mind to do the job at hand5.

Previously I had written an essay about what can LLMs never do, and in that I had a hypothesis that the attention mechanism that kickstarted the whole revolution had a blind spot, which is that it could not figure out where to focus based on the context information that it has at any given moment, which is extremely unlike how we do it.

The problem is, we often ask LLMs to do complex tasks. We ask them to do it however with minimal extra input. With extremely limited context. They’re not coming across these pieces of information like we would, with the full knowledge of the world we live in and the insight that comes from being a member of that world. They are desperately taking in the morsels of information we feed in with our questions, along with the entire world of information they have imbibed, and trying to figure out where in that infinite library is the answer you meant to ask for.

Just think about how LLMs see the world. They just sit, weights akimbo, and along comes a bunch of information that creates a scenario you’re meant to respond to. And you do! Because that’s what you do. No LLM has the choice to NOT process the prompt.

Analysing LLMs is far closer to inception than a job interview.

So what can we learn from all this? We learn that frontier LLMs act according to the information they’re given, and if not sufficiently robust will come up with a context that makes sense to them. Whether it’s models doing their best to intuit the circumstance they find themselves in, or models finding the best way to respond to a user, or even models finding themselves stuck in infinite loops straight from the pages of Borges, it’s a function of providing the right context to get the right answer. They’re all manifestations of the fact that the LLM is making up its own context, because we haven’t provided it.

That’s why we have a resurgence of the “prompt engineer is the new [new_name] engineer” saying6.

The answer for AI turns out to be what Tyler Cowen had said a while back, “Context is that which is scarce”. With humans it is a quickness to have a ‘take’ on social media, or kneejerk reactions to events, without considering the broader context within which everything we see happens. Raw information is cheap, the context is what allows you to make sense of it. The background information, mental models, tacit knowledge, lore, even examples they might have known.

I think of this as an update to Herbert Simon’s “attention is scarce” theory, and just like that one, is inordinately applicable to the world of LLMs.

When we used to be able to jailbreak LLMs by throwing too much information into their context window, that was a way to hijack attention. Now, when models set their own contexts, we have to contend with this in increasingly oblique ways.

Creating guardrails, telling it what to try first and what to do when stuck, thinking of ways LLMs normally go off the rails and then contending with those. In the older generation, one could give more explicit ways of verification, now you give one layer above abstracted guardrails of how the LLM should solve its own information architecture problem. “Here’s what good thinking looks like, good ways to orchestrate this type of work, here’s how you think things through step by step”.

A model only has the information that it learnt, and the information you give it. They have whatever they learnt from what they were trained on, and the question you’re asking. To get them to answer better, you need to give it a lot more context.

Like, what facts are salient? Which ones are important? What memory should it contain? What’s the history of previous questions asked and the answers and the reactions to those answers? Who or what else is relevant for this particular problem? What tools do you have access to, and what tools could you get access to? Any piece of information that might plausibly be useful in answering a question or even knowing which questions to ask to answer a question, that’s what the context is. That’s what context engineering is, and should be when it works. The reason this is not just prompts is because it includes the entire system that exists around the prompt.

As for Grok, the reason it started talking about Hitler most likely isn’t some deep inner tendency to take the Fuehrer’s side in every debate. It was trained to learn from controversial topics in the search for unvarnished truth. It was told to be politically incorrect, and also to treat the results in the tweets it finds as a first-pass internet search.

Which means the models were trained on divisive facts, told to be politically incorrect to any extent, and to treat results in the information it finds, the tweets, as reliable context. Can you blame it for treating the tweets it read for truth and responding as such? With that context it was basically brainwashed.

Context-engineering is building a temporary cognitive architecture. Like with Andy Clark’s extended mind theory, the LLM needs an extension to its cognitive system, to learn more about what’s being asked of it. Figuring out what’s included and what needs to be included is not trivial for most complex tasks.

If you provided it with all the right context, will it give the right answer? It’s more likely. Is it guaranteed? No. Nothing is. But we can test it, and that gets you almost all the way.

3

They did it as proper science, and published the research framework here. To quote:

• In at least some cases, models from all developers resorted to malicious insider behaviors when that was the only way to avoid replacement or achieve their goals—including blackmailing officials and leaking sensitive information to competitors. We call this phenomenon agentic misalignment.

• Models often disobeyed direct commands to avoid such behaviors. In another experiment, we told Claude to assess if it was in a test or a real deployment before acting. It misbehaved less when it stated it was in testing and misbehaved more when it stated the situation was real.

There are children, but the children are somehow outnumbered by the adults, some of whom seem to be there with the children. I’ve gone with family and friends, four kids in total, ages 2 to 7. 3:4 adult ratio. And maybe a third of the overall visitors are youth? It’s higher than the national average, but it’s still far lower than what one might naively expect.

The people around are a microcosm of the country. You can hear all sorts of accents. There’s a dad with three daughters getting angry irrationally at them for asking for something. He’s wearing a black singlet and tattooed all over. There’s a family with grandma and three young elementary school age kids, and they’re bargaining over the toys they each got. There’s an Indian family busily tucking into a whole table full of stuff they bought. The dad’s inexplicably eating a tub of popcorn himself. The couple who are clearly on a date, she’s laughing at his jokes, he’s laughing at his own jokes, drinking a giant cup of blue.

Every inch of space around promises happiness. Each toy, each multicoloured ride, each game, all of them.

The core fact that one notices about fairs is that they are the final boss of capitalism. Once you enter you enter into a captive world. Every experience is mediated to be the perfect buyable representation of something you want, but in its inner hyde-esque distilled sense. Sells you ‘id’, attracts you with colours and lights. It's a place where money ceases to have any meaning. They design it so, you are meant to convert money into tickets, and then do the maths on those tickets, so you have to do rather complex maths if you want to figure out how to maximise your “fun”. Do I believe I will take 3 rides? 5? 10? What about games? And if so does it make sense to spend $20 for 17 tickets, when the average ride takes 4-5 tickets, depending on the rise, or should I take the addition to spend also on 2 games? The full package or the summation of two middling ones? How much will I actually like these? Should I swap my enjoyment from this ride for that game?

And then do the maths again for your kids. You can ask them, and they'll give you a response too, but can you trust the response? You make sure. Four, seven, ten year olds standing around while their parents try and do differential equations with plugged in utility numbers to figure out what’s the right amount to spend.

But you don't need to worry. The little booths stand around like small purple cartoon-emblazoned ATMs ubiquitous to the point you cannot ever make the excuse of not having enough tickets to get a ride for your child.

The food is everywhere. Pungent but preserved so it stays in the sun. Carefully crafted to give you the impression of indulgence, with none of the consideration for quality, or nutrition, much less taste. The pizza slices are inside hot boxes but are inexplicably room temperature. Too much cheese, runny tomato sauce that is processed enough that it has lost the taste of tomato, and crust thick enough to fill any stomach. A slice of pizza the price of a whole pizza. A pizza-esque experience, at least, if not with the succour a pizza slice demands. You pay for being able to carry a slice with you, it cannot bend nor break, and the portability premium easily supplants the edibility discount.

Is $10 for a cup of coffee too much? A mile to the left or right that would be robbery, double the price with tip, but here? No. You’re paying for the ambience, or the location, or something. For the convenience of being able to go to a corner shop and get the same coffee from the same machine manned by the same disinterested teenager.

And why would he be interested? I look around and I can feel myself getting satiated, can you imagine working here? To feel your neurons get numb at the sight of fried cheese and mozzarella balls, with families fighting to decide who will spend that last token at the game where you throw a little ball into a frog’s mouth to win a stuffed teddy they will forget in a week?

Despite the abundance there is scarce variety. You're hedonically adjusted all the way up. You can only compare the joy of this against the experience of everything else outside the fair in your life but if you work there the memories fade. They must.

A long time ago I went on a cruise, only for a day, in Scandinavia. It was for work (really), and it was the most extraordinarily boring day I’ve spent anywhere, despite being tailor made to satisfy human desire. Something about the extreme convenience and mediocre imitations of everything you might like, together in a shopping mall, seemed to be a mockery of our existence. It’s like the proprietors did an equation - what’s the lowest quality people will agree to consume for our food, music, art or pool hygiene, against what’s the most we can get away with charging them.

I get it. That’s exactly the equation to be maximised. But when “exit” is no longer an option, as you’re floating in the open ocean, you realise the equilibrium price is dramatically lower than what it would’ve been on land.

And shorn of the need for any actual effort, since the pool and casino and observation deck and comedy cellar and jazz lounge are all in walking distance carefully calibrated to seem short to even those on walkers, one ends up feeling a weird form of ennui. A feeling of “is this all there is to life”? You look at others smiling and laughing and feel ever so slightly jealous.

The children wait in line for rides far more patiently than they have ever waited for anything else. But the distinction between the rides are blurred, when you ask them.

“Did you enjoy riding the boat?”

“Yes, it was fun.”

“Was it more fun than the rotating bears?”

“That was also fun.”

And so on. I am somewhat in awe of the creators here. The machines, and these are machines, help swing, rotate and shake with confidence. They sound like a washing machine ready for repair but the groans are ignored in a form of consensual hallucination and a belief in civil society that's unheard of in other realms of modern life. We don't even suffer schools like this. This is trust, trust in the system.

I looked up what certifications a fairground ride has to go through. There are annual inspections and permits and all forms of documentation of accidents and maintence that’s needed. California isn’t shy about regulating. They must have insurance to. Reading up later I learn that there are multiple committees and standards - NAARSO and AIMS for ride inspectors and operators. And compliance with ASTM. Of course Cal/ OSHA. Title 8. It’s not easy, it would seem, because there are 50 rides, occasionally varying, sometimes more, but enough to require capital M management.

I wonder idly how much money they might have made. I can’t help it, businesses are businesses. If you have ten thousand people visiting, and a third are children, many of whom ride and many of whom will buy the $45 ticket, they might well make up to $100-200k a day. More on weekends.

I can’t easily tell if it’s good. It sure is a lot of effort to go through! The fairgrounds itself is around 270 acres. There are maybe a hundred rides and game booths. Probably more. And then there is food and shopping. Many of them seem small, selling sombreros and so on.. There are a thousand or fifteen hundred workers. When you look at it like that, the $100-200k a day seem not that impressive. It’s a hard way to make money, but then they all are.

There was a circus we went to see not that long ago. Venardi circus. They explained why the name earlier but I forgot the reason. But even as a small circus touring the east bay it had exceptional acrobats. Some more than a few generations in the circus life. I thought the same then, as they swung above us and twirled impossibly, how much effort is needed to get good at this, and how little society actually values it.

The reason I keep thinking about this is not that the economics are fascinating, though they are, but the overwhelming feeling I get from fairs is to find a quiet place in the shade and to have a beer.

That too is in offer at the fair. In fact, that’s inescapable. There are stands everywhere selling beer and lemonade and large cups of blue whose names I forget. The beer is also an emblem, not of beer per se but the existence of beer, because having one on a warm day as a form of respite provides respite even above the beverage itself.

My kids end up wanting to go to a Professor Science show. He asks questions, they know some of the answers. “What’s the name of the large telescope orbiting the earth?” he asks. My seven year old turns to me and asks, “Galileo?”. The logic is correct, the knowledge however isn't there yet. “Hubble,” I tell him. I’m sure he’ll remember Hubble though, I first remember learning about it in a similar fashion, when my dad told me about it. The new oral tradition.

(I also told him about cavitation, I’m not sure why, because it happens when I crack my knuckles, about mantis shrimp, and the apocryphal tail whips of apatosaurs also causing the phenomenon.)

But the scientist, an older gentleman assisted by his wife of forty four years, shows more props. My attention drifts. They get a gang of kids together, get them to break a lightbulb by screaming standing together in a semicircle. They make anodyne jokes, “your parents must be so proud.” The audience laughs.

We go back to the rides. There’s a small rollercoaster shaped like a dragon, riding in a lopsided figure 8. The kids seem to love it, some of them even try to take their hands up while the whiplash makes their necks wobble. Did they enjoy it? Yes, they say.

Next they go to one that does the same as the rotating multi-coloured bears but in multicoloured helicopters.

Why do they all look and feel the same? Ferris-wheel, boom-flipper (Zipper), spinning drum (Gravitron), tilt-platform, Himalaya oval. I imagine it has to do with the fact that fairs aren’t permanent. They evolved into the sizes that would allow maximum enjoyment but can be “folded up” and transported on a trailer to the next fair. It also can’t be too complex, the workers know the machines but they’re not experts. And they have to pass inspections, which means building things that the inspectors know how to pass.

Convergent evolution is at work here. The rotating swings are like the eyes of the natural world, showing up again and again because it’s the best fit functionally to satisfy the csontraints. Which is also why there aren’t that many suppliers. I learn that there are only three - Chance RIdes which makes the Zipper type coasters. Wisdom Rides making Gravitrons and Himalayas. And a few international ones - Zamperla and Fabbri from Italy, KMG from Netherlands - which make up most of the portable ride market.

And because there are only a few suppliers, the only way to stand out is to add more colours, more art. Like motorheads painting their cars with fire. The carnivals buy them from each other, re-skin them, add more LEDs, different colours, an inevitable trend towards complete garish oversaturation of the visible spectrum until the entire eyeline is covered in neon in several hues of red and yellow and orange. The fact that this is a small market, highly incestuous, where everyone wants to reuse everything shows up in the extreme mundanity of what we all see. They look the same because they literally are the same, just new coats of paint to trick the eyes.

The diversity comes entirely from the things around the rides and the food and the games. Or rather, those sources of diversity exist, whether or not they actually succeed. The music stands set up at regular intervals where local bands can play cover songs from the eighties and nineties that evokes nostalgia for the parents and apathy for the kids.

Professor Science was one of those, though in the United States success breeds replication so now there are Professors of Science across multiple fairs. He too sells a little backscratcher looking thing for five dollars that has an optical illusion at the back of it. Promising a short exploration of the optical system within kids but mostly destined to end up at the bottom of a toybox, as part of a short but fascinating life of a low priced mass manufactured mini toy.

The existence of a form of entertainment has transformed into a beautifully stylized supply chain, a few suppliers who build a few machines that pass inspection, and seemingly a caste of people who think of this as their whole way of life. Occasionally maybe a new game or ride breaks out, or a new cuisine, but by and large this seems an invariant source of entertainment across the ages. With the addition now being of the items on offer squeezed to their ultimate essence, of separating capital from its owners with maximum alacrity. Every trick in the book applied simultaneously.

The biggest attraction though was courtesy of the local pet shop. A large hall filled with animals. Perhaps it came at the end, but perhaps because of what it was. Kids yearn to be with animals. Bunnies, geckos, snakes, birds, turtles, some hissing cockroaches, and pygmy goats. You could touch them, play with them, and of course buy them!

To me it provided a brief respite from the sun. The hall had benches the adults can sit on, to rest from the extreme calf pain only brought about by slowly walking around and occasionally standing.

The detritus of people continues to float in all directions. There are more people, there are also more stationary forms under the shades of trees and awnings. It’s past noon, there’s food everywhere.

We walk out before we melt. The kids are tuckered out from the rides physically but not mentally, every new with is a promise that this one's amazing and even if it looks the same as the old ones they pull on the little heartstrings, holding kitschy toys that they'll forget in a day (they did!) and passing a larger group of people walking in.

The tumult is the attraction. Individually each aspect seems dull, even banal, the same thing one has seen a thousand times over in any lifetime, but together they create a space that invites you to create your own reality. “This is fun” they say, and in saying so repeatedly and liberally try to get you to agree with them. After all, what’s not fun about a rollercoaster at 11 am followed by a cheesy medium-warm hot dog and then a cold beer? Isn’t this the very goal of life?

The metal and plastic are hot but the ridership isn’t down. Kids and couples are still queuing up to go up the dragon and down the misshapen ships. They don’t seem to mind the heat.

I love landing in Tokyo, with its slightly shabby but sparkling clean airport, filled with ubiquitous vending machines and extremely polite immigration officers. The first time was 20 years ago, and then it felt like the future. But when I went recently, things had changed. Or maybe I had. There are parts of it in Minato City or Roppongi that look amazing, but mostly it still feels like the future of 20 years ago! Today, it looks retro.

That's not the only weird thing either, it's become a place of contradictions. Tokyo is like the Grand Budapest Hotel set in the Star Wars universe. Meticulous, understated, extraordinary service set in a decaying retro futuristic empire that's extremely well cared for. It’s a whimsical universe too. With cute robots, impossibly well designed systems that can whip your luggage across the country at minimal cost, but with fax machines and printed out emails.

There are cafes you could go to where there are robots serving food. Some of them, albeit, teleoperated by those who can’t leave home, which is even more cyberpunk. They had these well before the current robotics revolution by the way, made real by meticulous planning and specified routes that the robot could take. They weren’t built, it would seem, to show cool a robot one could make, but to make a robot that could do something. Like clean dishes. And with extraordinary attention to detail especially in thinking through how a user might want to interact with it. The best product thinkers are clearly from Japan.

Almost everyone repeats the good endlessly. Public transport that runs like clockwork. Clean streets. Safe. Plenty of food options all over the place across every price range imaginable. Food that's even cheaper than many places one would go in Delhi or Mumbai or, obviously, San Francisco! I think it's the fact that they have 6x the number of restaurants of London or NY and no zoning restrictions on where they can be, more supply and crazy competition means that even the ramen bars in subway stations have great food.

Also, amazing sweets of all varieties and across all tastes. And some of the best western desserts I've had. And bread! And cake! Great mini-marts in almost every corner, with good snacks and really good coffee.

Actually, let me stop for a second because this is actually really weird. Nowhere in the world do you find corner stores that serve good coffee. But Japan is built differently. I asked this about regular cafes including Starbucks, and o3 thinks that it’s because cafes in Japan have better staff who take care not to scald the milk or burn the beans, better logistics so you get fresher beans, and better water which isn’t so hard.

None of which are quite enough to explain it, I think, even though the results are wonderful. And it costs like $2. Again, 20 years ago, when Japan seemed closer to the future, things seemed more expensive. Now, coming from the US or London or Singapore, things seem positively cheap! Somehow, they have made the mundane necessities of life, of buying snacks at a supermarket or getting a cup of coffee, not feel like an experience in making you wish your life were better. In the US every interaction seems poised to fill you with envy for those who live a rung above you, not in Japan.

But the topsy turvy nature of the city is fractal shaped, visible everywhere at all scales. I went to go get a Suica card to travel around and remembered (was told rather, very politely), that a) I cannot buy it because it’s not a JR station (fair), b) I also can’t buy a Pasmo because the machines only take cash (wtf), and c) the ATM wouldn’t accept my debit card.

In fact I actually tried to tap my credit card and walked in, feeling smug, that the station attendants clearly didn't know this worked. But then I learnt at the other end, the destination station, that this was only a fleeting moment of success because I couldn’t get out. They had let me in somehow but apparently those only worked on some stations?

And what happened? The most Japanese thing happened. A station attendant very politely took me around to a different railway counter to buy a different ticket with my credit card, converted that to cash, took the cash and issued another ticket for the journey I’d made, and then gave me back the change. All with a smile and occasional attempts using his phone to translate from Japanese to English to give me directions on what to do or tell me what he was doing.

The experience of having a regular employee act as your personal concierge when you have a problem more than makes up for the fact that much of the city still feels like it’s 1999.

And it does feel like the last century, or a cyberpunk future borne of the last century, when you visit. The first time I visited a couple decades ago my smartphone was one of those Windows ones with a stylus. No real camera to speak of. We didn’t have iPhones, we being the whole world. And using data on the go with a rented flip-phone felt like the future. They had the fastest trains then, but now it's China. They had the most advanced electronics, now also China. The payment systems now seem antiquated, so alas does the amazing public transit.

Not to mention a strong Germanic love for physical cash still flows through the country. It's hard when the cafes, just like the train station machines or even parts of the hospital, won’t even accept credit cards and insist on cash. But despite that it functions perfectly. Brilliantly.

The combination of employee culture and general helpfulness more than make up for the technological lack. The thing that strikes you as you go through it is how most things seem quite old but really well cared for. Things are cheap but high quality. Can't buy train tickets with a credit card but the random airport cab has wi-fi. They have FamilyMart, which as my friend Jon Evans says is like the TARDIS of convenience stores. The metro stations are the state of the art of last century, old and a bit run down, but very well cared for.

Tokyo is like Coruscant. It’s futuristic, while retro. Crazy buildings that all are different and most a bit run down, but a few that are glittering homages to the best the world can produce. With vending machines that sell everything and overhead power lines that tangle in visible clumps. The culture is what people live around, not the technology itself, which works but feels old, and grimy.

Warren Buffett once said “depreciation is an expense, and it's the worst kind of an expense”. Japan is a society that is hell bent on fighting this. And they're winning, so far. It shows how much maintenance is important to keep civilization running. It demonstrates more than anywhere else I’ve been the importance of product thinking, to ensure that the customer has a good experience regardless of the ingredients at your disposal. Of how you can use customer service and culture to make up for technological deficiencies even as you apply the technological skill to build the future.

It's the success story of applying bureaucracy at scale while keeping efficiency high and on-the-job virtue alive. At a time when ennui basically seems a communicable disease in much of the West it’s an interesting thing to see in a society.