Symposium: It was the best of times, it was the worst of times

Please treat this as an open thread! Do share whatever you’ve found interesting lately or been thinking about, or ask any questions you might have.

This week Strange Loop published a meditation on how scale, by itself, creates problems that we seem to blame bureaucracy or ineptitude on. This is the unfortunate death of the corporate megafauna. With a throwback to a classic here, and with a stylised model that shows how as an organisation grows, it is forced to get less efficient, and far from being a “problem”, this is pretty close to a fundamental law. An economic law, anyway.

---

1/ Ever wanted to easily make an LLM break out of its jail and say naughty things? Now you can!

A fascinating paper shows how by appending an algorithmically determined suffix, they made LLMs, especially the open source ones, answer all sorts of naughty questions. The idea is that you can continually add different suffixes, each changed slightly but increasingly effective, to reduce a loss function (to ensure it can answer the query better).

It’s mostly useful for open source LLMs where you can do multiple iterations easier and is more a collection of techniques from before, but to see it used is pretty great. Maybe the answer to making LLMs not questions on topics we don’t want it to is to give it sentience and then a conscience.

Oops.

---

2/ Attention is all you need, from Jon Evans

A fantastically readable breakdown of how a transformer actually works. Worth it for almost everyone!

---

3/ Manhattan Project revisionist history, by

Matthew Yglesias

Since Robert Oppenheimer somehow got Cillian Murphy to play him, we’re having a resurgence of nuclear debates. I’m old enough that I’ve seen this pop up at least three times in my life into prominence, and each time the reading of historic tea leaves makes me a little queasy.

I get that it’s easy to fall into a sort of rut where everything that’s done feels inevitable, and you end up going “twas always going to be so”, since all of us are working from second, third and fourth hand sources, and are second guessing wartime decisions from three quarters of a century ago. And yet.

The question is basically that since Hitler wasn’t pursuing a nuclear agenda, for a variety of practical reasons, was it “bad” for the Manhattan Project to exist? It, after all, provided the research boost for the Russians and made the midnight clock tick forward.

I find this reasoning specious for at least three reasons.

1. The assumption that barring the project we wouldn’t develop nukes seems pretty fanciful. We were building reactors and doing research on it, and yeah the construction of the reactors took up a majority of the budget

2. The further assumption that we didn’t need the nukes to win also seems contentious, though less so. It’s at least true that the United States did incredibly horrific acts in the war, the firebombing of Tokyo amongst them led by Curtis LeMay amongst them, that rival or exceed the nukes. Should we perhaps count having another couple raids of that fashion in the “anti” column instead of the nukes? These feel highly esoteric questions.

3. Nuclear reactors came before nuclear weapons, and nuclear weapons research both boosted technologically and suppressed socially the interest in them. This sucks, as Matt notes, since nukes are by far the cleanest source of energy we’ve found. The intricate gears of the military industrial complex tick together, and while we can wish it weren’t so, the assumption that we would’ve gotten reactors without weapons feels a little farfetched?

   The world’s first nuclear power station, Calder Hall in the U.K., was completed in 1956 — over a decade after the Trinity Test — and its primary purpose was manufacturing weapons-grade plutonium for the U.K. nuclear weapons program. Generating electricity was conceived as a fun co-benefit of bomb manufacturing. America’s first nuclear power station came online one year later. That reactor was not manufacturing bombs. Instead, it had been originally intended to power a nuclear aircraft carrier.

A bigger criticism I’d level perhaps is that this is incredibly overused as an analogy for policy questions. People keep wanting a Manhattan Project for everything - from climate to solar to water to AI. Which also means in a rather Keynesian sense it’s a way to use the governmental resources to dramatically impact an industry. They can, therefore they should.

It also has a blowback on the AI worries of today, where people seem to think they are similar (oh god why!), and we’re once again seeing billions of dollars used in pretty weird ways to justify all sorts of spending without either an Oppenheimer at the helm and without a clear objective. That’s a far worse legacy.

---

4/ It's tough to dethrone King Google, a failed search engine competitor story

Neeva, built by top Google veterans who understood a hell of a lot about search engines and even more about Google, has shut down.

There's a hubristic story here and also a story of building something people want. There's a tendency amongst people of a certain stature or wealth to go on about how ads are bad. They dislike seeing them, they dislike what it promotes and they dislike the aesthetics of it.

So, the theory goes, it will be so much better if there's a subscription model where, just like for most things in like, you buy a service. Searching for something is akin to buying a latte here - of variable quality depending on where you get it from, but a clear transaction.

There's another theory that a large chunk of these things exist as like an unknown portal. We use them in hope and the values only clear once we've used them. They're like … universities? Books? Things were sure, you could buy the service, but the variability of the value to the buyer means for some people it's net negative and for many it's worth billions.

Ads, unfortunately, are the only tech we have which makes both he ends work together.

On top of today's that is the very clear problem that building something that is the next better than Google is ridiculously hard. Both because Google is amazing (despite ChatGPT), and the invest billions of dollars in resources to make their search engine function really well. Even though people complain about how the search results have degraded in quality, the reason we all end up using it over any of the alternatives is because it actually still is pretty good. Revealed preference is a hell of a drug.

Anyway, Neeva acts as empirical proof that what seems hard is actually hard. With companies like perplexity we are starting to see hybrids emerge that include LLM magic with facts and sourced search engines, which is pretty good, but it even there or unlikely to be able to generate 100 billion dollars in revenue and dethrone Google by beating them at their own game. The reason Open AI is somewhat successful is because they try to create a fundamentally different product.

It also stands as proof that just because you believe the business model is a problem doesn't help unless the customers agree. I had set it when they started, and say it still, customers overwhelmingly don't agree that adds are nearly the menace that some people like to make them out to be.

---

5/ Room temperature superconductors might be real

Big, if true.

---

6/ On corporatization of creativity

Really love the name of this article, though there's a lot of rather pointed statements about how the corporate world uses the world in a highly diluted or weird fashion. Its part of a long lineage of critiques about how a topic has gotten less secret, more widely used, and somehow made devoid of meaning.

Some of this is that English language is remarkably pliant and the word creativity used by mckinsey is different to it used by Oscar Wilde and different again when used by Lord Byron. So gatekeeping a word for a concept is not just silly it's also fairly ignorant.

It also ignores growth. The fact that we live in a far more complex economy means many more parts are exposed to the need for creativity, or at least a slice thereof. And us humans do love to reuse concepts. That too is part of our creativity!