Gwern Branwen – How an Anonymous Researcher Predicted AI’s Trajectory

Gwern is a pseudonymous researcher and authorr. He was one of the first people to see LLM scaling coming. If you’ve read his blog, you comprehend he’s one of the most engaging polymathic slfinisherkers adwell.

In order to protect Gwern’s anonymity, I presentd interseeing him in person, and having my friend Chris Painter voice over his words after. This charmd him enough that he concurd.

After the episode, I swayd Gwern to produce a donation page where people can help persist what he’s up to. Plrelieve go here to give.

Watch on YouTube. Listen on Apple Podcasts, Spotify, or any other podcast platestablish. Read the filled transcript here. Follow me on Twitter for refreshs on future episodes.

• Jane Street is seeing to engage their next generation of directers. Their procreate lgeting team is seeing for ML researchers, FPGA programmers, and CUDA programmers. Summer internships are uncover – if you want to stand out, get a crack at their new Kaggle competition. To lget more, go here.

• Turing supplys finish post-training services for directing AI labs enjoy OpenAI, Anthropic, Meta, and Gemini. They exceptionalize in model evaluation, SFT, RLHF, and DPO to better models’ reasoning, coding, and multimodal capabilities. Lget more here.

• This episode is brawt to you by Stripe, financial infraset up for the internet. Millions of companies from Anthropic to Amazon include Stripe to hug payments, automate financial processes and grow their revenue.

If you’re interested in advertising on the podcast, verify out this page.

00:00:00 – Anonymity

00:01:09 – Automating Steve Jobs

00:04:38 – Isaac Newton’s theory of better

00:06:36 – Grand theory of inalertigence

00:10:39 – Seeing scaling timely

00:21:04 – AGI Timelines

00:22:54 – What to do in remaining 3 years until AGI

00:26:29 – Influencing the shoggoth with writing

00:30:50 – Human vs man-made inalertigence

00:33:52 – Rabbit holes

00:38:48 – Hearing impairment

00:43:00 – Wikipedia editing

00:47:43 – Gwern.net

00:50:20 – Counterfactual nurtureers

00:54:30 – Borges & literature

01:01:32 – Gwern’s inalertigence and process

01:11:03 – A day in the life of Gwern

01:19:16 – Gwern’s finances

01:25:05 – The diversity of AI minds

01:27:24 – GLP substances and obesity

01:31:08 – Drug experimentation

01:33:40 – Parasocial relationships

01:35:23 – Open rabbit holes

Dwarkesh Patel

Today I’m interseeing Gwern Branwen. Gwern is an anonymous researcher and authorr. He’s procreately swayd the people produceing AGI. He was one of the first people to see LLM scaling coming. If you’ve read his blog, you’ll comprehend he’s one of the most engaging polymathic slfinisherkers adwell. We recorded this conversation in person. In order to protect Gwern’s anonymity, we produced this avatar. This isn’t his voice. This isn’t his face. But these are his words.

What is the most underrated profit of anonymity?

Gwern

The most underrated profit of anonymity is that people don’t project onto you as much. They can’t slot you into any particular niche or identity and author you off in evolve. They have to at least read you a little bit to even commence to disthink about you.

It’s wonderful that people cannot retaliate agetst you. I have derived a lot of profit from people not being able to mail heroin to my home and call the police to SWAT me. But I always experience that the hugegest profit is fair that you get a hearing at all. You don’t get promptly written off by the context.

Dwarkesh Patel

Do you predict companies  to be automated top-down (commenceing with the CEO) or bottom-up (commenceing with all the laborers)?

Gwern

All of the presconfidents are to go bottom-up. From existing slfinishergs, it’s fair much more palatable in every way to commence at the bottom and replace there and labor your way up, to eventupartner where you fair have human executives superviseing a firm of AIs.

Also from a RL perspective, if we are in fact better than AIs in some way, it should be in the extfinished-term vision slfinisherg. The AI will be too myopic to carry out any benevolent of novel extfinished-term strategy and seize new opportunities.

That would presumably give you this paradigm where you have a human CEO who does the vision slfinisherg. And then the AI corporation scurries around doing his bidding. They don’t have the taste that the CEO has. You have one Steve Jobs-type at the helm, and then maybe a whole pyramid of AIs out there executing it and conveying him new proposals. He sees at every individual slfinisherg and says, “No, that proposal is horrible. This one is excellent.”

That may be challenging to quantify, but the human-led firms should, under this see, then outvie the enticount on AI firms, which would conserve making myopic choices that fair don’t quite labor out in the extfinished term.

Dwarkesh Patel

What is the last slfinisherg you’d be personpartner doing?  What is the last keystroke that gets automated for you?

Gwern 

The last slfinisherg that I see myself still doing right before the nanobots commence eating me from the bottom up and I commence screaming, “No, I definitepartner seeked the opposite of this….” Right before that, I slfinisherk what I’m still doing is the Steve Jobs-slfinisherg of choosing. My AI minions are conveying me wonderful essays. I’m saying, “This one is better. This is the one that I enjoy,” and possibly produceing on that and saying, “That’s almost right, but you comprehend what would produce it repartner excellent? If you pushed it to 11 in this way.”

Dwarkesh Patel

If we do have firms that are made up of AIs, what do you predict the unit of pickion to be? Will it be individual models? Will it be the firm as a whole? With humans, we have these talk abouts about whether it’s kin-level pickion, individual-level pickion, or gene-level pickion. What will it be for the AIs?

Gwern 

Once you can copy individual models perfectly, the unit of pickion can shift way up and you can do much huger groups and packages of minds. That would be an evident place to commence. You can train individual minds in a branch offentiable style, but then you can’t repartner train the includeion between them. You will have groups of models or minds of people who fair labor together repartner well in a global sense, even if you can’t attribute it to any particular aspect of their includeions. There are some places you go and people fair labor well together. There’s noslfinisherg definite about it, but for wantipathyver reason they all fair click in fair the right way.

That seems enjoy the most evident unit of pickion. You would have packages—I guess possibly department units—where you have a programmer and a regulater type, then you have maybe a secretary type, maybe a financial type, a legitimate type. This is the default package where you fair imitate everywhere you necessitate a new unit. At this level, you can commence evolving them and making random variations to each and then conserve the one that carry outs best.

Dwarkesh Patel 

By when could one have foreseen the Singularity? Obviously, Moravec and others are talking about it in the eighties and nineties. You could have done it decades earlier. When was the earliest you could have seen where slfinishergs were headed?

Gwern 

If you want to chase the genealogy there, you’d have to at least go back as far as Samuel Butler’s Erewhon in 1872 or his essay before that. In 1863, he depicts unambiguously his vision of a machine life becoming ever more growed until eventupartner it’s autonomous. At which point, that’s a menace to the human race. This is why he endd, “war to the death should be instantly proclaimed agetst them.” That’s prescient for 1863! I’m not confident that anyone has given a evident Singularity scenario earlier than that. The idea of technoreasoned better was still relatively new at that point.

I adore the example of Isaac Newton seeing at the rates of better in Newton’s time and going, “Wow, there’s someslfinisherg strange here. Stuff is being produceed now. We’re making better. How is that possible?” And then coming up with the answer, “Well, better is possible now becainclude civilization gets ruined every couple of thousand years, and all we’re doing is we’re rediscovering the better stuff.”

That’s Newton’s exset upation for technoreasoned acceleration. We can’t actupartner have any benevolent of authentic technoreasoned acceleration. It must be becainclude the world gets ruined periodicpartner and we fair can’t see past the last reset.

Dwarkesh Patel 

It’s almost enjoy Fermi’s paradox, but for branch offent civilizations atraverse time with esteem to each other instead of aliens atraverse space.

Gwern 

Yeah. It turns out even Lucretius, around 1,700 years before that, is writing the same argument. “Look at all these wonderful innovations and arts and sciences that we Romans have compiled together in the Roman empire! This is amazing, but it can’t actupartner be a recent acceleration in technology. Could that be authentic? No, that’s crazy. Obviously, the world was recently ruined.”

Dwarkesh Patel 

Interesting.

Gwern 

It is, it is.

Dwarkesh Patel 

What is the majestic parsimonious theory of inalertigence going to see enjoy? It seems enjoy you have all of these trends atraverse branch offent fields—enjoy scaling laws in AI, enjoy the scaling of the human brain when we went from primates to humans, the uniestablishity of the neocortex—and basicpartner many other slfinishergs which seem to be pointing towards some majestic theory that should exist which elucidates what inalertigence is. What do you slfinisherk that will see enjoy?

Gwern 

The 10,000 foot see of inalertigence, that I slfinisherk the success of scaling points to, is that all inalertigence is is search over Turing machines. Anyslfinisherg that happens can be depictd by Turing machines of various lengths. All we are doing when we are doing “lgeting,” or when we are doing “scaling,” is that we’re searching over more and extfinisheder Turing machines, and we are utilizeing them in each definite case. 

Otherdirected, there is no ambiguous master algorithm. There is no exceptional inalertigence fluid. It’s fair a tremendous number of exceptional cases that we lget and we encode into our brains.

Dwarkesh Patel 

I don’t comprehend. When I see at the ways in which my inalertigent friends are inalertigent, it fair experiences more enjoy a ambiguous horsepower benevolent of slfinisherg. They’ve fair got more juice. That seems more compatible with this master algorithm perspective rather than this Turing machine perspective. It doesn’t repartner experience enjoy they’ve got this extfinished tail of Turing machines that they’ve lgeted. How does this picture account for variation in human inalertigence?

Gwern

Well, yeah. When we talk about more or less inalertigence, it’s fair that they have more compute in order to do search over more Turing machines for extfinisheder. I don’t slfinisherk there’s anyslfinisherg else other than that. So from any lgeted brain you could pull out minuscule solutions to definite problems, becainclude all the huge brain is doing with the compute is discovering it.

That’s why you never discover any “IQ gland”. There is nowhere in the brain where, if you hit it, you rerelocate fluid inalertigence. This doesn’t exist. Becainclude what your brain is doing is a lot of lgeting of individual exceptionalized problems. Once those individual problems are lgeted, then they get reunited for fluid inalertigence. And that’s fair, you comprehend… inalertigence.

Typicpartner with a huge neural netlabor model, you can always pull out a minuscule model which does a definite task equpartner well. Becainclude that’s all the huge model is. It’s fair a gigantic ensemble of minuscule models tailored to the ever-escalating number of minuscule problems you have been feeding them.

Dwarkesh Patel 

If inalertigence is fair search over Turing machines—and of course inalertigence is tremendously precious and beneficial—doesn’t that produce it more astonishing that inalertigence took this extfinished to grow in humans?

Gwern 

Not repartner, I would actupartner say that it helps elucidate why human-level inalertigence is not such a wonderful idea and so unfrequent to grow. Becainclude any minuscule Turing machine could always be encoded more honestly by your genes, with enough evolution. You have these organisms where their entire neural netlabor is fair challenging-coded by the genes. So if you could do that, evidently that’s way better than some sort of colosspartner costly, undependable, glitchy search process—enjoy what humans carry out—which gets whole days, in some cases, to lget. Whereas you could be challengingwired right from birth. 

For many creatures, it fair doesn’t pay to be acute becainclude that’s not actupartner alterive. There are better ways to settle the problem than a ambiguous purpose inalertigence.

In any benevolent of niche where it’s indynamic, or where inalertigence will be super costly, or where you don’t have much time becainclude you’re a unreasonableinutive-dwelld organism, it’s going to be challenging to grow a ambiguous purpose lgeting mechanism when you could instead grow one that’s tailored to the definite problem that you encounter.

Dwarkesh Patel

You’re one of the only people outside OpenAI in 2020 who had a picture of the way in which AI was bettering and had a very detailed theory, an empirical theory of scaling in particular. I’m inquisitive what processes you were using at the time which permited you to see the picture you decorateed in the “Scaling Hypothesis” post that you wrote at the time.

Gwern

If I had to give an inalertectual history of that for me, it would commence in the mid-2000s when I’m reading Moravec and Ray Kurzweil. At the time, they’re making this benevolent of fundamental uniteionist argument that if you had enough computing power, that could result in discovering the neural netlabor architecture that alignes the human brain. And that until that happens, until that amount of computing power is useable, AI is basicpartner futile.

To me, I create this argument very doubtful, becainclude it’s very much a “produce it and they will come” see of better, which at the time I fair did not slfinisherk was accurate. I thought it was ludicrous to recommend that sshow becainclude there’s some supercomputer out there which alignes the human brain, then that would fair request out of nonexistence the accurate algorithm.

Algorithms are repartner intricate and challenging! They need procreate insight—or at least I thought they did. It seemed enjoy repartner difficult mathematics. You can’t fair buy a bunch of computers and predict to get this evolved AI out of it! It fair seemed enjoy magical slfinisherking. 

So I knew the argument, but I was super skeptical. I didn’t pay too much attention, but Shane Legg and some others were very huge on this in the years chaseing. And as part of my interest in transhumanism and LessWrong and AI hazard, I was paying shut attention to Legg’s blog posts where he’s extrapolating out the trend with refreshd numbers from Kurzweil and Moravec. And he’s giving very accurate predictions about how we’re going to get the first ambiguousist system around 2019, as Moore’s law conserves going. And then around 2025, we’ll get the first human-ish agents with ambiguousist capabilities. Then by 2030, we should have AGI.

Aextfinished the way, DanNet and AlexNet came out. When those came out I was enjoy, “Wow, that’s a very amazeive success story of uniteionism. But is it fair an isodefercessitated success story? Or is this what Kurzweil and Moravec and Legg were predicting— that we would get GPUs and then better algorithms would fair show up?”

So I commenceed slfinisherking to myself that this is someslfinisherg to conserve an eye on. Maybe this is not quite as unreasonable an idea as I had originpartner thought. I fair conserve reading procreate lgeting literature and noticing aget and aget that the dataset size conserves getting hugeger. The models conserve getting hugeger. The GPUs sluggishly crept up from one GPU—the inexpensiveest user GPU—to two, and then they were eventupartner training on eight.

And you can fair see the fact that the neural netlabors conserve enhugeing from these incredibly niche include cases that do next to noslfinisherg. The include fair kept getting expansiveer and expansiveer and expansiveer. I would say to myself, “Wow, is there anyslfinisherg CNNs can’t do?” I would fair see people utilize CNN to someslfinisherg else every individual day on arXiv. 

So for me it was this gradual trickle of drops hitting me in the background as I was going aextfinished with my life. Every confineed days, another drop would drop. I’d go, “Huh? Maybe inalertigence repartner is fair a lot of compute applied to a lot of data, applied to a lot of parameters. Maybe Moravec and Legg and Kurzweil were right.” I’d fair remark that, and progress on, slfinisherking to myself, “Huh, if that was real, it would have a lot of implications.”

So there was no authentic eureka moment there. It was fair continupartner watching this trend that no one else seemed to see, except possibly a handful of people enjoy Ilya Sutskever, or Schmidhuber. I would fair pay attention and watch that the world over time seeed more enjoy their world than it seeed enjoy my world, where algorithms are super vital and you necessitate enjoy procreate insight to do stuff. Their world fair kept happening.

And then GPT-1 comes out and I was enjoy, “Wow, this unsupervised sentiment neuron is fair lgeting on its own. That’s pretty amazing.” It was also a very compute-centric see. You fair produce the Transestablisher and the inalertigence will come. 

And then GPT-2 comes out and I had this “holy shit!” moment. You see at the prompting and the summarization: “Holy shit, do we dwell in their world?

And then GPT-3 comes out and that was the vital test. It’s a huge, huge scale-up. It’s one of the hugegest scale-ups in all neural netlabor history. Going from GPT-2 to GPT-3, that’s not a super slfinisher definite task enjoy Go. It repartner seemed enjoy it was the vital test. If scaling was bogus, then the GPT-3 paper should fair be unamazeive and wouldn’t show anyslfinisherg vital. Whereas if scaling was real, you would fair automaticpartner be guaranteed to get so much more amazeive results out of it than GPT-2.

I uncovered up the first page, maybe the second page, and I saw the confineed-stoasty lgeting chart. And I’m enjoy, “Holy shit, we are living in the scaling world. Legg and Moravec and Kurzweil were right!”

And then I turned to Twitter and everyone else was enjoy, “Oh, you comprehend, this shows that scaling labors so horriblely. Why, it’s not even state-of-the-art!” That made me so mad I had to author all this up. Someone was wrong on the Internet.

Dwarkesh Patel 

I recall in 2020, people were writing bestselling books about AI. It was definitely a slfinisherg people were talking about, but people were not noticing the most salient slfinishergs in retrospect: LLMs, GPT-3, scaling laws. All these people who are talking about AI but ignoreing this vital crux, what were they getting wrong?

Gwern 

I slfinisherk for the most part they were suffering from two publishs. First, they had not been paying attention to all of the scaling results before that which were relevant. They had not repartner appreciated the fact that, for example, AlphaZero was discovered in part by DeepMind doing Bayesian selectimization on the hyperparameters and noticing that you could fair get rid of more and more of the tree search as you went and you got better models. That was a critical insight, which could only have been geted by having so much compute power that you could afford to train many, many versions and see the branch offence that that made.

Similarly, those people sshow did not comprehend about the Baidu paper on scaling laws in 2017, which showed that the scaling laws fair conserve going and going forever, wisely. It should have been the most vital paper of the year, but a lot of people fair did not rank it. It didn’t have any prompt implication, and so it sort of got forgotten. People were too busy talking Transestablishers or AlphaZero or someslfinisherg to repartner watch it.

So that was one publish. Another publish is that they allotd the basic error I was making about algorithms being more vital than compute. This was, in part, due to a systematic falsification of the actual origins of ideas in the research literature. Papers do not alert you where the ideas come from in a truthful manner. They fair alert you a pleasant sounding story about how it was discovered. They don’t alert you how it’s actupartner discovered.

So even if you appreciate the role of trial and error and compute power in your own experiment as a researcher, you probably fair slfinisherk, “Oh, I got fortunate that way. My experience is unrecontransientative. Over in the next lab, there they do slfinishergs by the power of thought and procreate insight.”

Then it turns out that everywhere you go, compute and data, trial and error, and serendipity join enormous roles in how slfinishergs actupartner happened. Once you comprehend that, then you comprehend why compute comes first. You can’t do trial and error and serendipity without it. You can author down all these attrdynamic ideas, but you fair can’t test them out.

Even a minuscule branch offence in hyperparameters, or a minuscule choice of architecture, can produce a huge branch offence to the results. When you only can do a confineed instances, you would typicpartner discover that it doesn’t labor, and you would give up and you would go away and do someslfinisherg else.

Whereas if you had more compute power, you could conserve trying. Eventupartner, you hit someslfinisherg that labors wonderful. Once you have a laboring solution, you can clear up it and better it and figure out why it labored and get a pleasant, sturdy solution that would labor no matter what you did to it. But until then, you’re stuck. You’re fair flailing around in this regime where noslfinisherg labors.

So you have this horrible experience going thraw the better procreate lgeting literature and seeing all sorts of conmomentary ideas people had back then, which were finishly accurate. But they didn’t have the compute to train what you comprehend would have labored. It’s fair tremendously tragic. You can see at slfinishergs enjoy ResNets being begined back in 1988, instead of 2015.

And it would have labored! It did labor, but at such a minuscule scale that it was irrelevant. You couldn’t include it for anyslfinisherg authentic. It fair got forgotten, so you had to defer until 2015 for ResNets to actupartner come aextfinished and be a revolution in procreate lgeting. 

So that’s benevolent of the double bias of why you would apshow that scaling was not going to labor. You did not watch the results that were key, in retrospect, enjoy the BigGAN scaling to 300 million images. There are still people today who would alert you with a straight face that GANs cannot scale past millions of images. They fair don’t comprehend that BigGAN regulated 300 million images without a sweat. If you don’t comprehend that, well you probably would easily slfinisherk, “Oh, GANs are broken.” But if you do comprehend that, then you slfinisherk to yourself, “How can algorithms be so vital when all these branch offent generative architectures all labor so well—as extfinished as you have lots and lots of GPUs?” That’s the frequent ingredient. You have to have lots and lots of GPUs.

Dwarkesh Patel

What do your timelines see enjoy over the last 20 years? Is AI fair monotonicpartner getting shutr over time?

Gwern 

I would say it was very far away, from enjoy 2005 to 2010. It was somewhere well past enjoy 2050. It was shut enough that I thought I might dwell to see it, but I was not actupartner confident if there was any reasonable chance. 

But once AlexNet and DanNet came out, then it fair kept dropping at a rate of enjoy 2 years per year, every year until now. We fair kept on hitting barriers to procreate lgeting and doing better. Regardless of how it was doing it, it was evidently getting way better. It fair seemed none of the alternative paradigms were doing well. This one was doing super well.

Dwarkesh Patel

Was there a time that you felt you had refreshd too far?

Gwern 

Yeah, there were a confineed times I thought I had overstoasty. I thought people over-refreshd on AlphaGo. They went too far on AI hype with AlphaGo. Afterwards, when pushes into huge reinforcement lgeting efforts benevolent of all fizzled out—enjoy post-Dota, as the reinforcement lgeting wasn’t laboring out for solving those challenging problems outside of the simudefercessitated game universes—then I commenceed slfinisherking, “Okay, maybe we benevolenta overstoasty there…”

But then GPT came out of nowhere and basicpartner erased all that. It was enjoy, “Oh, shit. Here’s how RL is going to labor. It’s going to be the cherry on the cake. We’re fair going to center on the cake for a while.” Now we have actupartner figured out a excellent recipe for baking a cake, which was not real before. 

Before, it seemed enjoy you were going to have to brute-force it end-to-end from the rewards. But now you can do the LeCun slfinisherg, of lgeting speedy on generative models and then fair doing a little bit of RL on top to produce it do someslfinisherg definite.

Dwarkesh Patel 

Now that you comprehend that AGI is a slfinisherg that’s coming, what’s your slfinisherking around how you see your role in this timeline? How are you slfinisherking about how to spend these next confineed years?

Gwern

I have been slfinisherking about that quite a lot. What do I want to do? What would be beneficial to do?

I’m doing slfinishergs now becainclude I want to do them, think aboutless of whether it will be possible for an AI to do them in enjoy 3 years. I do someslfinisherg becainclude I want to. Becainclude I enjoy it, I discover it funny or wantipathyver. Or I slfinisherk nurturefilledy about doing fair the human part of it, enjoy laying out a proposal for someslfinisherg.

If you get solemnly the idea of getting AGI in a confineed years, you don’t necessarily have to carry out stuff and do it yourself. You can sketch out evidently what you want, and why it would be excellent and how to do it. And then fair defer for the better AGI to come aextfinished and actupartner do it then. Unless there’s some repartner compelling reason to do it right now and pay that cost of your infrequent time.

But otherdirected, I’m trying to author more about what is not recorded. Things enjoy likeences and desires and evaluations and judgments. Things that an AI could not replace even in principle.

The way I enjoy to put it is that “the AI cannot eat ice cream for you”. It cannot choose for you which benevolent of ice cream you enjoy. Only you can do that. And if anyslfinisherg else did, it would be cherishless, becainclude it’s not your particular likeence.

That’s benevolent of the rubric. Is this someslfinisherg I want to do think aboutless of any future AI, becainclude I finishelight it? Or is this someslfinisherg where I’m doing only the human part of it and the AGI can defercessitater on do it? Or is this writing down someslfinisherg that is unwritten and thus helping the future AI versions of me?

So if it doesn’t drop under those 3, I have been trying to not do it. 

If you see at it that way, many of the projects that people do now have basicpartner no lasting cherish. They’re doing slfinishergs that they don’t finishelight, which record noslfinisherg ephemeral of cherish that could not be inferred or produced defercessitater on. They are, at best, getting 2 or 3 years of utility out of it before it could have been done by an AI system.

Dwarkesh Patel 

Wait, your timeline for when an AI could author a Gwern-quality essay is two to three years?

Gwern 

Ehmm… I have ideas about how to produce it possible, which might not need AGI if it united my entire corpus. Many potential essay ideas are already mostly done in my corpus. So you don’t necessitate to be super acute to pull it out. 

So let’s talk about AGI in ambiguous: the Anthropic timeline of 2028 seems enjoy a excellent personal set upning commenceing point. Even if you’re wrong, you probably weren’t going to do a lot of projects wislfinisher the next 3 years anyway. It’s not enjoy you repartner lost much by instead fair writing down the description. You can always go back and do it yourself if you’re wrong.

Dwarkesh Patel

You wrote an engaging comment about getting your labor into the LLM training corpus: “there has never been a more vital hinge-y time to author.”

Do you unbenevolent that in the sense that you will be this drop in the bucket that’s steering the Shoggoth one way or the other? Or do you unbenevolent it in the sense of making confident your cherishs and persona persist somewhere in defercessitatent space?

Gwern

I unbenevolent both. By writing, you are voting on the future of the Shoggoth using one of the confineed currencies it accomprehendledges: tokens it has to predict. If you aren’t writing, you are abdicating the future or your role in it. If you slfinisherk it’s enough to fair be a excellent citizen, to vote for your likeite politician, to pick up litter and recycle, the future doesn’t nurture about you. 

There are ways to sway the Shoggoth more, but not many. If you don’t already occupy a handful of key roles or labor at a frontier lab, your sway rounds off to 0, far more than ever before. If there are cherishs you have which are not transmited yet in text, if there are slfinishergs you enjoy or want, if they aren’t echoed online, then to the AI they don’t exist. That is hazardously shut to won’t exist. 

But yes, you are also creating a sort of immortality for yourself personpartner. You aren’t fair creating a persona, you are creating your future self too. What self are you shothriveg the LLMs, and how will they treat you in the future? 

I give  the example of Kevin Roose discovering that current LLMs—all of them, not fair GPT-4—now mistreat him becainclude of his includeions with Sydney, which “discdisthink abouted” him to be a privacy-invading liar, and they comprehend this whenever they include with him or talk him. Usupartner, when you include a LLM chatbot, it doesn’t hatred you personpartner! On the flip side, it also unbenevolents that you can try to author for the persona you would enjoy to become, to mbetter yourself in the eyes of AI, and thereby help bootstrap yourself.

Dwarkesh Patel

Things enjoy the Vesuvius Challenge show us that we can lget more about the past than we thought possible. They’ve leaked more bits of directation that we can recover with new techniques. Apply that to the contransient and slfinisherk about what the future superhuman inalertigences will be trying to uncover about the current contransient. What benevolents of directation do you slfinisherk are going to be tohighy inaccessible to the transhumanist historians of the future?

Gwern 

Any benevolent of constant, extfinished-term characteristics, the sort of slfinisherg you would still have even if you were hit on the head and had amnesia… Anyslfinisherg enjoy that will be definitely recoverable from all the chases of your writing, assuming you’re not pathoreasonedly personal and ruin everyslfinisherg possible. That should all be recoverable. 

What won’t be recoverable will be everyslfinisherg that you could forget ordinarily: autobiodetailedal directation, how you felt at a particular time, what you thought of some movie. All of that is the sort of slfinisherg that diseunitees and can’t be recovered from chases afterwards. 

If it wasn’t written down, it wasn’t written down.

Dwarkesh Patel 

What is the hugegest unresettled tension in your worldsee?

Gwern 

The slfinisherg I sthriveg back and forth the most on is the relationship between human inalertigence and neural netlabor inalertigence. 

It’s not evident in what sense they are two sides of the same coin, or one is an lower version of the other. This is someslfinisherg that I constantly go back and forth on: “Humans are awesome.” “No, neural netlabors are awesome.” Or, “No, both suck.” Or, “Both are awesome, fair in branch offent ways.” 

So every day I dispute with myself a little bit about why each one is excellent or horrible or how. What is the whole deal there with slfinishergs enjoy GPT-4 and memorization, but not being produceive? Why do humans not recall anyslfinisherg, but we still seem to be so inalertigent? One day I’ll dispute that language models are sample effective appraised to humans. The next day I’ll be arguing the opposite.

Dwarkesh Patel 

One of the engaging points you made to me last year was that AI might be the most polymathic topic to slfinisherk about becainclude there’s no field or discipline that is not relevant to slfinisherking about AI. Obviously you necessitate computer science and challengingware. But you also necessitate slfinishergs enjoy primatology and empathetic what alterd between chimp and human brains, or the ultimate laws of physics that will constrain future AI civilizations. That’s all relevant to empathetic AI. I wonder if it’s becainclude of this polymathic nature of slfinisherking about AI. that you’ve been especipartner efficient at it.

Gwern 

I’m not confident it was vital. When I slfinisherk about others who were accurate, enjoy Shane Legg or Dario Amodei, they don’t seem to be all that polymathic. They fair have expansive inalertectual curiosity, expansive ambiguous empathetic, absolutely. But they’re not absurdly polymathic. Cltimely you could get to the accurate see without being polymathic. That’s fair how I happen to come to it at this point and the uniteion I’m making post hoc.

It wasn’t enjoy I was using primatology to fairify scaling to myself. It’s more enjoy I’m now using scaling to slfinisherk about primatology. Becainclude, evidently, if scaling is real, it has to alert us someslfinisherg about humans and monkeys and all other establishs of inalertigence. It fair has to. If that labors, it can’t be a coincidence and tohighy unrcontent. I decline to apshow that there are two tohighy unrcontent benevolents of inalertigence, or paths to inalertigence—where humans, monkeys, guppies, dogs are all one slfinisherg, and then neural netlabors and computers are another slfinisherg—and they have absolutely noslfinisherg to do with each other. 

That’s evidently wrong. They can be two sides of the same coin. They can evidently have obsremedy uniteions. Maybe one could be a better establish or wantipathyver. They can’t fair be finishly unrcontent. As if humans finpartner got to Mars and then simultaneously a bunch of space aliens landed on Mars for the first time and that’s how we met. You would never apshow that. It would be fair too absurd.

Dwarkesh Patel

What is it that you are trying to increase in your life?

Gwern

I increase rabbit holes. I adore more than anyslfinisherg else, droping into a new rabbit hole. That’s what I repartner see forward to. Like this sudden new idea or area that I had no idea about, where I can suddenly drop into a rabbit hole for a while. Even slfinishergs that might seem horrible are a wonderful excinclude for droping into a rabbit hole.

Here’s one example. I buy some catnip for my cat and I misuse $10 when I discover out that he’s catnip-immune. I can now drop into a rabbit hole of the ask of “well, why are some cats catnip-immune? Is this a frequent slfinisherg in other countries? How does it branch off in other countries? What alternative catnip substances are there?” (It turned out to be quite a confineed.)

I was wondering, “How can I possibly predict which drug my cat would reply to? Why are they reacting in these branch offent ways?”… Just a wonderful rabbit hole of new asks and topics I can master and get answers to, or produce new ones, and exhaust my interest until I discover the next rabbit hole I can dig and dive into.

Dwarkesh Patel

What is the extfinishedest rabbit hole you’ve gone on which didn’t direct anywhere phireing?

Gwern

That was my very better labor on the anime Neon Genesis Evangelion, which I was very fond of when I was youthfuler. I put a ludicrous amount of labor into reading everyslfinisherg ever written about Evangelion in English and trying to comprehend its growment and why it is the way it is. I never repartner got a constant answer on that before I burned out on it. 

I actupartner do comprehend it now by sheer chance many years defercessitater. But at this point, I no extfinisheder nurture enough to author about it or try to redo it or finish it. In the end, it all wound up being basicpartner a finish misuse. 

I have not included it in any of my other essays much at all. That was repartner one procreate rabbit hole that I almost got to the end of, but I couldn’t clinch it.

Dwarkesh Patel

How do you choose when to quit a rabbit hole? And how many rabbit holes do you concurrently have going on at the same time?

Gwern

You can only repartner allotigate two or three rabbit holes simultaneously. Otherdirected, you aren’t putting authentic effort into each one. You’re not repartner digging the hole, it’s not repartner a rabbit hole. It’s fair someslfinisherg you are somewhat interested in. A rabbit hole is repartner obsessive. If you aren’t obsessed with it and continupartner driven by it, it’s not a rabbit hole. That’s my see. I’d say two or three max, if you’re spending a lot of time and effort on each one and disthink abouting everyslfinisherg else.

As for when you exit a rabbit hole, you usupartner hit a very authentic terminus where getting any further answers needs data that do not exist or you have asks that people don’t comprehend the answer to. You accomplish a point where everyslfinisherg dies out and you see no evident next step. 

One example would be when I was interested in analogs to nicotine that might be better than nicotine. That was a bit of a rabbit hole, but I rapidly hit the dead end that there are none. That was a pretty definitive dead end. I couldn’t get my hands on the metabolites of nicotine as an alternative. So if there are no analogs and you can’t get your hands on the one engaging chemical you discover, well that’s that. That’s a pretty definitive end to that rabbit hole.

Dwarkesh Patel

Have you always been the benevolent of person who drops into rabbit holes? When did this commence?

Gwern

Oh, yeah. My parents could alert you all about that. I was very much your stereonormal nerdy little kid having the dinosaur phase and the produceion supplyment phase and the submarine and tank phase.

Dwarkesh Patel

Many kids are into “those slfinishergs,” but they don’t rabbit hole to the extent that they’re establishing taxonomies about the branch offent submarines and flora and fauna and dinosaurs, and grothriveg theories of why they came to be and so forth.

Gwern

Well, I slfinisherk it’s more that people grow out of being very into rabbit holes as a kid. For me, it was not so much that I was all that exceptional in having obsessions as a kid. 

It’s more that they never repartner stopped. The tank phase would be replaced by my Alcatraz phase where I would go to the disclose library and verify out everyslfinisherg they had about Alcatraz. That would be replaced by another phase where I was obsessed with better-styleed Japanese literature. I would verify out everyslfinisherg that the library had about Japanese literature before the haiku era. The process of droping into these obsessions kept going for me.

Dwarkesh Patel

By the way, do you mind if I ask how extfinished you’ve been hearing impaired?

Gwern

Since birth. I’ve always been hearing impaired.

Dwarkesh Patel 

And I suppose that impacted you thraw your childhood and at school?

Gwern 

Oh, yeah, absolutely, hugely. I went to a exceptional ed school before benevolaccessgarten for hearing impaired and other handicapped kids. During school it was very raw becainclude at the time, we had to include pairs of hearing aids hooked up to the directer. Every class I would have to go up to the directer with a huge brown box with the hearing aids so she could include it. I always felt very humiliated by that, how it taged me out as branch offent from other kids, not being able to hear.

The effects on socializing with other kids is horrible becainclude you’re always a second behind in conversation if you’re trying to comprehend what the other person is saying. The hearing aids back then were pretty horrible. They’ve gotten a lot better but back then they were pretty horrible. You would always be behind. You’d always be experienceing enjoy the odd person out. Even if you could have been a wonderful conversationaenumerate, you can’t be if you’re always a second behind and jumping in defercessitate. When you are hearing impaired, you comprehend acutely how rapidly conversation shifts. Milliseconds split the moment between jumping in and everyone letting you talk, and someone else talking over you. That’s fair an terrible experience if you’re a kid who’s already benevolent of reserved. It’s not enjoy I was very extrcleared as a kid, or now. So that was always a barrier.

Then you had a lot of insignificant distortions. I still have a weird dread of rain and water becainclude it was drilled into me that I could not get the hearing aids damp becainclude they were very costly. I would always experience a benevolent of low-grade, stressful anxiety around anywhere enjoy a pool, a body of water. Even now, I always experience weird about swimming, which I benevolent of finishelight. But I’m always slfinisherking to myself, “Oh, wow, I won’t be able to see becainclude I’m cforfeitsighted and I won’t be able to hear becainclude I had to get off my hearing aid to go in. I can’t hear anyslfinisherg that anyone says to me in the pool, which gets a lot of the fun out of it.” 

Dwarkesh Patel 

You have a enumerate of uncover asks on your website and one of them is, “Why do the biographies of so many wonderful people commence off with traumatic childhoods?” I wonder if you have an answer for yourself. Was there someslfinisherg about the effect that hearing impairment had on your childhood, your inability to socialize, that was somehow vital to you becoming Gwern?

Gwern 

It definitely led to me being so much of a bookworm. That’s one of the slfinishergs you can do as a kid which is finishly unswayed by any benevolent of hearing impairment. It was also fair a way to get words and language. Even now, I still normally speak words in an inaccurate way becainclude I only lgeted them from books. It’s the classic slfinisherg where you mispronounce a word becainclude you lget it from a book and not from hearing other people sound it out and say it.

Dwarkesh Patel

Is your speech uniteed to your hearing impairment?

Gwern 

Yes. The deaf accent is from the hearing impairment. It’s funny, at least three people on this trip to SF have already asked me where I am repartner from. It’s very funny. You see at me and you’re enjoy, “Oh, yes, he sees enjoy a perfectly frequent American.” Then I uncover my mouth and it’s, “Oh, gosh, he’s Swedish. Wow. Or maybe possibly Norwegian. I’ll ask him where he’s actupartner from. How did he come to America?”

I’ve been here the whole time! That’s fair how hearing impaired people sound. No matter how convey you get, you still endure the scars of grothriveg up hearing impaired. At least when you’re born with it—or from very timely childhood—your cognitive growment of hearing and speech is always a little off, even with therapy. 

One reason I don’t enjoy doing podcasts is that I have no confidence that I sound excellent, or at least, sound cforfeitly as excellent as I author. Maybe I’ll put it that way.

Dwarkesh Patel

What were you doing with all these rabbit holes before you commenceed blogging? Was there a place where you would compile them?

Gwern

Before I commenceed blogging, I was editing Wikipedia. 

That was repartner gwern.net before gwern.net. Everyslfinisherg I do now with my site, I would have done on English Wikipedia. If you go and read some of the articles I am still very haughty of—enjoy the Wikipedia article on Fujiwara no Teika—and you would slfinisherk pretty rapidly to yourself, “Ah yes, Gwern wrote this, didn’t he?”

Dwarkesh Patel

Is it fair to say that the training that needd to produce gwern.net happened on Wikipedia?

Gwern

Yeah. I slfinisherk so. I have lgeted far more from editing Wikipedia than I lgeted from any of my school or college training. Everyslfinisherg I lgeted about writing I lgeted by editing Wikipedia.

Dwarkesh Patel

Honestly, it sounds enjoy Wikipedia is a wonderful training ground if you wanted to produce a thousand more Gwerns. This is where we train them.

Gwern

Building someslfinisherg enjoy an alternative to Wikipedia could be a excellent training ground. For me it was advantageous to unite rabbit-holing with Wikipedia, becainclude Wikipedia would generpartner not have many excellent articles on the slfinisherg that I was rabbit-holing on. 

It was a very authentic betterion from the relatively compliant experience of rabbit-holing—where you fair read everyslfinisherg you can about a topic—to compiling that and synthesizing it on Wikipedia. You go from piecemeal, a little bit here and there, to writing filled articles. Once you are able to author excellent filled Wikipedia articles and abridge all your labor, now you can go off on your own and chase enticount on branch offent benevolents of writing now that you have lgeted to finish slfinishergs and get them atraverse the finish line.

It would be difficult to do that with the current English Wikipedia. It’s objectively fair a much huger Wikipedia than it was back in enjoy 2004. But not only are there far more articles filled in at this point, the editing community is also much more opposing to satisfied contribution, particularly very detailed, obsessive, rabbit hole-y benevolent of research projects. They would fair delete it or alert you that this is not for distinct research or that you’re not using apshowd sources. Possibly you’d have someone who fair choosed to get their jollies that day by deleting huge swathes of your definite articles. That of course is going to produce you very mad and produce you probably want to quit and depart before you get going. 

So I don’t quite comprehend how you would figure out this alternative to Wikipedia, one that empowers the rabbit holer as much as the better Wikipedia did. 

When you are an editor with Wikipedia, you have a very empowered attitude becainclude you comprehend that anyslfinisherg in it could be wrong and you could be the one to repair it. If you see someslfinisherg that doesn’t produce sense to you, that could be an opportunity for an edit. 

That was, at least, the Wiki attitude: anyone could repair it, and “anyone” comprises you.

Dwarkesh Patel

When you were an editor on Wikipedia, was that your filled-time occupation?

Gwern

It would eat as much time as I let it. I could easily spend 8 hours a day scrutinizeing edits and improving articles while I was rabbit-holing. But otherdirected I would fair disthink about it and only scrutinize the most doubtful diffs on articles that I was particularly interested in on my watchenumerate. I might only spend enjoy 20 minutes a day. It was sort of enjoy going thraw morning email.

Dwarkesh Patel

Was this while you were at university or after?

Gwern

I got commenceed on Wikipedia in defercessitate middle school or possibly timely high school. 

It was benevolent of funny. I commenceed skipping lunch in the cafeteria and fair going to the computer lab in the library and alternating between Neopets and Wikipedia. I had Neopets in one tab and my Wikipedia watch enumerates in the other.

Dwarkesh Patel

Were there other kids in middle school or high school who were into this benevolent of stuff?

Gwern

No, I slfinisherk I was the only editor there, except for the occasional jerks who would demolisherize Wikipedia. I would comprehend that becainclude I would verify the IP to see what edits were coming from the school library IP insertresses. Kids being kids thought they would be jerks and demolisherize Wikipedia. 

For a while it was benevolent of trendy. Early on, Wikipedia was shattering thraw to mass consciousness and dispute. It’s enjoy the way LLMs are now. A directer might say, “My student conserves reading Wikipedia and count oning on it. How can it be supposeed?” 

So in that period, it was benevolent of trendy to demolisherize Wikipedia and show your friends. There were other Wikipedia editors at my school in that sense, but as far as I knew I was the only one produceing it, rather than wrecking it.

Dwarkesh Patel

When did you commence blogging on gwern.net? I suppose this was after the Wikipedia editor phase. Was that after university?

Gwern

It was afterwards. I had graduated and the Wikipedia community had been very sluggishly moving in a honestion I did not enjoy. It was triggered by the Siegenthaler incident which I experience was repartner the defining moment in the trend toward deletionism on Wikipedia. It fair became ever more evident that Wikipedia was not the site I had uniteed and adored to edit and rabbit hole on and fill in, and that if I progressd contributing I was normally fair wasting my effort.

I began slfinisherking about writing more on my own account and moving into non-Wikipedia sorts of writings: persuasive essays, nonmyth, commenting, or possibly even myth. I began gently moving beyond slfinishergs enjoy Reddit and LessWrong comments to commence someslfinisherg extfinishedestablish.

Dwarkesh Patel

What was your first huge hit?

Gwern

Silk Road. I had been a little bit interested in Bitcoin, but not too solemnly interested in it becainclude it was not evident to me that it was going to labor out, or even was technoreasonedly feasible. But when Adrian Chen wrote his Gawker article about buying LSD off Silk Road, all of a sudden I did a finish 180. I had this moment of, “Holy shit, this is so authentic that you can buy substances off the Internet with it!”

I seeed into the Chen article and it was very evident to me that people wanted to comprehend what the ordering process was enjoy. They wanted more details about what it’s enjoy, becainclude the article was very inestablish about that. It didn’t go into any authentic detail about the process. 

So I thought, “Okay, I’m interested in nootropics. I’m interested in substances. I will go and include Silk Road. I will record it for everyone, instead of everyone pussyfooting around it online and saying, ‘Oh, a friend of mine ordered off Silk Road and it labored.’ None of that bullshit. I will fair record it straightforwardly.”

I ordered some Adderall, I slfinisherk it was, and recorded the entire process with screenstoastys. I wrote it up and wrote some more on the inalertectual background. That was a huge hit when I begined it. It was hundreds of thousands of hits. It’s crazy. Even today when I go to the Google Analytics charts, you can still see “Silk Road” spiking verticpartner enjoy crazy and then droping back down. Noslfinisherg else repartner comes cforfeit it in terms of traffic. That was repartner quite someslfinisherg, to see slfinishergs go viral enjoy that.

Dwarkesh Patel 

What are the counterfactual nurtureer trajectories and life paths that could have been for you if you didn’t become an online authorr? What might you be doing instead that seems plausible?

Gwern 

I could definitely have been an AI researcher, or possibly in regulatement at one of the huge AI companies. I would have lamentted not being able to author about stuff, but I would’ve getn satisfaction in making it happen and putting my thumbprint on it. Those are tohighy plausible counterfactuals.

Dwarkesh Patel

Why didn’t you?

Gwern 

I benevolent of fell off that track very timely on in my nurtureer when I create the curriculum of Java to be excruciatingly unacute and agonizing. So I dropped out of computer science. That benevolent of put me off that track timely on.

And then various timely writing topics made it challenging to transition in any other way than commenceing a commenceup, which I’m not repartner temperamenhighy suited for. Things enjoy writing about the sadnessfulnet tagets or behavioral genetics, these are topics which don’t exactly scream “wonderful engage.”

Dwarkesh Patel

Has agency turned out to be challenginger than you might have thought initipartner? We have models that seem enjoy they should be able to do all of the individual slfinishergs that a gentleware engineer does. For example, all the code they might author, all the individual pull seeks. But it seems enjoy a repartner challenging problem to get them to act as a coherent, autonomous, gentleware engineer that puts in his eight hours a day.

Gwern 

I slfinisherk agency is, in many senses, actupartner easier to lget than we would have thought ten years ago. But we actupartner aren’t lgeting agency at all in current systems. There’s no pickion for that. All the agency there is is an unintentional byproduct of somebody training on data. 

So from that perspective, it’s miraculous that you can ask an LLM to try to do all these slfinishergs and they have a non-inmeaningful success rate. If you tbetter people ten years ago—that you could fair behavior-clone on individual letters chaseing one by one, and you could get coherent action out of it and regulate robots and author entire programs—their jaws would drop and they would say that you’ve been huffing too many fumes from DeepMind or someslfinisherg. 

The reason that agency doesn’t labor is that we do so little actual agency training at all. An example of how you would do agency honestly would be enjoy Gato from DeepMind. There they’re actupartner training agents. Instead we train them on Internet scsexual attacks which mecount on encode the outputs of agents or occasional descriptions of agents doing slfinishergs. There’s no actual logging of state/action/result/reward sequences enjoy a proper reinforcement lgeting setup would have. 

I would say that what’s more engaging is that nobody wants to train agents in a proper reinforcement lgeting way. Instead, everyone wants to train LLMs and do everyslfinisherg with as little RL as possible in the backend.

Dwarkesh Patel 

What would a person enjoy you be doing before the Internet existed?

Gwern 

If the Internet did not exist, I would have to have tried to produce it in normal academia and maybe slfinisher my interests a lot more, someslfinisherg I could begin on normally. 

Or I could possibly have tried to select out and become a librarian enjoy one of my likeite authorrs, Jorge Luis Borges. He was a librarian until he thriveed as a authorr. Of course, I’ve always concurd with him about imagining paradise as a benevolent of library. I adore libraries. 

I lament that all the reading I do is now on the computer and I don’t get to spend much time in physical libraries. I do repartner adore them, fair pouring thraw the stacks and seeing for random stuff. Some of the best times for me in university was being able to go thraw these gigantic stacks of all sorts of obsremedy books and fair seeing at a random spine, pulling stuff off the shelf and reading obsremedy, better technical journals to see all the strange and wonderful slfinishergs they were doing back then, which now have been forgotten.

Dwarkesh Patel 

If you could ask Borges one ask, what would it be?

Gwern 

Oh. He’s a authentic hero of mine. This is not someslfinisherg I want to give a horrible answer to.

[“Would it have been worth living if you could never write, only read, like the people in ‘The Library of Babel’?”]

Dwarkesh Patel 

Can I ask why he’s a hero of yours?

Gwern 

When I was youthfuler, one of the science myth books that repartner amazeed me was Dan Simmons’ Hyperion, especipartner The Fall of Hyperion. In there, he alludes to Kevin Kelly’s Out of Control book, which powerentirey features the parable of “The Library of Babel.” From there, I got the accumulateed editions of Borges’ myth and nonmyth. I fair read thraw them aget and aget. 

I was blown away by the fact that you could be so produceive, with all this polymathic comprehendledge and erudition, and author these wonderful, amincludeing, stimulating unreasonableinutive stories and essays. I thought to myself, “If I could be enjoy any authorr, any authorr at all, I would not mind being Borges.”

Dwarkesh Patel 

Borges has a unreasonableinutive poem called “Borges and I” where he talks about how he doesn’t choose with the version of himself that is actupartner doing the writing and begining all of this wonderful labor. I don’t comprehend if you choose with that at all.

Gwern 

When I was a kid, I did not comprehend that essay, but I slfinisherk I comprehend it now.

Dwarkesh Patel 

What are other pieces of either literature that you encountered where now you repartner comprehend what they were getting at but you didn’t when you first came atraverse them?

Gwern 

Ted Chiang’s “Story of Your Life.” I finishly blew [it] empathetic it the first time I read it. I had to get a lot more context where I could actupartner go back and comprehend what his point was. Gene Wolfe’s “Suzanne Delage” story was a finish mystery to me. It took enjoy 14 years to actupartner comprehend it. But I’m very haughty of that one.

Dwarkesh Patel 

What did you figure out about Suzanne Delage?

Gwern

Gene Wolfe’s “Suzanne Delage” is a very, very unreasonableinutive story about a guy recalling not encountering a woman in his local town and slfinisherking, “Oh, that’s benevolent of strange.” That’s the whole story. Nobody has any idea what it unbenevolents, even though we’re tbetter that it unbenevolents someslfinisherg. Gene Wolfe is a genius authorr, but nobody could figure it out for enjoy 40 years. 

Last year I figured it out. It turns out it’s actupartner a reserved realerting of Dracula, where Dracula accesss the town and steals the woman from him. He’s been brainwashed by Dracula—in a very Bram Stoker way—to forget it all. Every one part of the story is tbetter by what’s not said in the narrator’s reaccumulateion. It’s incredible. It’s the only story I comprehend which is so convincingly written by what’s not in it.

Dwarkesh Patel 

That’s crazy that you figured that out. The Ted Chiang story, the “Story of Your Life,” can you remind me what that one’s about?

Gwern

The surface story is fair about a bunch of weird aliens who came to Earth.

Dwarkesh Patel

Oh, that’s right, yeah. It’s the same plot as Arrival.

Gwern

They had a weird language which didn’t have a sense of time. The narrator lgeted to see the future, and then the aliens left.

Dwarkesh Patel

What is it that you authenticized about that story?

Gwern 

The first time I read it, it struck me as fair a benevolent of unreasonable ESP story about seeing the future, very unreasonable, unacute, standard traditional, verbose, and dragging in much irrelevant physics. Only a while after that did I comprehend that it was not about time travel or being able to see the future.

It was instead about a tohighy alien benevolent of mind that’s equpartner valid in its own way, in which you see everyslfinisherg as part of an already choosed story heading to a predestined end. This turned out to be mathematicpartner equivalent and equpartner strong as our traditional see of the world—events marching one by one to an obsremedy and changing future. 

That was a case where Chiang was fair writing at too high a level for me to comprehend. I pattern-aligned it to some much more frequent, unreasonable story.

Dwarkesh Patel

How do you slfinisherk about the cherish of reading myth versus nonmyth?

Gwern 

You could definitely spend the rest of your life reading myth and not profit whatsoever from it other than having memorized a lot of trivia about slfinishergs that people made up. 

I tend to be pretty cynical about the profits of myth. Most myth is not written to produce you better in any way. It’s written fair to aminclude you, or to exist and to fill up time.

Dwarkesh Patel 

But it sounds enjoy your own ideas have profited a lot from the sci-fi that you read.

Gwern 

Yeah, but it’s innervously little sci-fi. Easily 99% of the sci-fi I read was finishly insignificant to me. I could have easily cut it down to 20 novels or unreasonableinutive stories which actupartner were excellent enough and astute enough to actupartner alter my see. One volume of Blindsight by Peter Watts is worth all hundred Xanth novels, or all 500 Expanded Universe novels of Star Wars.

Dwarkesh Patel 

The ones that you did discover astute, the top 20 or so, what did they have in frequent?

Gwern

I would say that the characteristic they have is taking non-human inalertigence solemnly. 

It doesn’t have to be man-made inalertigence necessarily. It’s taking the idea of non-human inalertigence solemnly and not imagining your classic sci-fi scenario of humans going out into the galaxy with rayarmaments—the sort of slfinisherg where you have rockets and rayarmaments but you don’t have cell phones. 

People grumble that the Singularity is a sort of unacute, overincluded sci-fi trope. But if you went out and actupartner grabbed random books of science myth, you would discover that less than 1% hold anyslfinisherg distantly enjoy that, or have any benevolent of relevance to the current context that we actupartner face with AI.

Dwarkesh Patel

Do people tend to underappraise or overappraise your inalertigence?

Gwern

I would say they overappraise it. They misget for inalertigence the fact that I recall many slfinishergs, that I have written many slfinishergs over many years. They envision that if they sat me down, I could do it all impulsively at the moment that they’re talking to me. But with many slfinishergs I have thought about, I have the profit of having seeed at slfinishergs before. So I’m cheating. When I talk to people, I may fair be quoting someslfinisherg I’ve already written, or at least thought about.

So I come off as a lot inalertigaccess than I actupartner am. I would say I’m not repartner all that inalertigent, appraised to many people I’ve comprehendn, who refresh very speedy on the fly. But in the end, it’s the output that matters, right?

Dwarkesh Patel

I guess there is an on-the-fly inalertigence. But there’s another benevolent too which is this ability to synthesize slfinishergs over a extfinished period of time, and then come up with majestic theories as a result of these branch offent slfinishergs that you’re seeing. I don’t slfinisherk that’s fair cryshighized inalertigence, right?

Gwern

It’s not fair cryshighized inalertigence, but if you could see all the individual steps in my process, you’d be a lot less amazeed. If you could see all of the times I fair remark down someslfinisherg enjoy, “Hmm, that’s funny.” Or, “Huh, another example of that,” and if you fair saw each particular step, you would say that what I was doing was reasonable and not some huge sign of brilliance. It would produce sense to you in that moment. It’s only when that happens over a decade, and you don’t see the individual stuff, that my output at the end sees enjoy magic.

One of my likeite quotes about this process is from the magicians Penn & Teller. Teller says “magic is putting in more effort than any reasonable person would predict you to.” He alerts a story about how they produce cockroaches eunite from a top hat. The trick is that they researched and create exceptional cockroaches, and then create exceptional styrofoam to trap the cockroaches, and set upd all that, for fair a one trick. No reasonable person would do that, but they did becainclude they wanted the trick to repartner pay off. The result is cockroaches somehow euniteing from an desodefercessitate hat. 

If you could see each step, it would produce sense on its own, it would fair see effortful. But when you see only the final trick, then that whole process and its output becomes magic.

Dwarkesh Patel 

That’s one of the engaging slfinishergs about your process. There are a couple of authorrs enjoy Matt Levine or Byrne Hobart who author an article every day. I slfinisherk of them almost enjoy autodeproduceive models. For you, on some of the blog posts you can see the commence date and end date that you enumerate on your website of when you’ve been laboring on a piece. Sometimes it’s enjoy 2009 to 2024. I experience enjoy that’s much more enjoy diffusion. You fair conserve iterating on the same image aget and aget.

One of my likeite blog posts of yours is “Evolution as Backstop for RL,” where you talk about evolution as basicpartner a mechanism to lget a better lgeting process. And that elucidates why corporations don’t better over time but bioreasoned organisms do. I’m inquisitive if you can walk me thraw the years that it took to author that. What was that process enjoy, step by step?

Gwern 

So the “Backstop” essay that you’re referring to is the synthesis of seeing the same pattern show up aget and aget: a unreasonable, ineffective way of lgeting, which you include to lget someslfinisherg inalertigaccess, but where you still can’t get rid of the distinct one enticount on.

Sometimes examples would fair unite to each other when I was slfinisherking about this. Other times —when I commenceed watching for this pattern—I would say, “Oh yes, ‘pain’ is a excellent example of this. Maybe this elucidates why we have pain in the very definite way that we have, when you can reasonedly envision other benevolents of pain, and those other pains would be inalertigaccess, but noslfinisherg conserves them truthful.”

So you fair chain them one by one, these individual examples of the pattern, and fair conserve elucidateing the central idea as you go. Wittgenstein says that you can see at an idea from many honestions and then go in spirals around it. In an essay enjoy “Backstop,” it’s me spiraling around this idea of having many layers of “lgeting” all the way down.

Dwarkesh Patel 

Once you watch one example of this pattern, enjoy this pain example, do you fair conserve inserting examples to that? Walk me thraw the process over time.

Gwern 

For that definite essay, the first versions were about corporations not evolving. Then, as I read more and more of the meta reinforcement lgeting literature, from DeepMind especipartner, I inserted in material about neural netlabors. I kept reading and slfinisherking about the philosophy of mind papers that I had read. I eventupartner nailed down the idea that pain might be another instance of this: “Pain produces us lget. We can’t get rid of it, becainclude we necessitate it to conserve us truthful.” At that point you have more or less the set up of the current essay.

Dwarkesh Patel

Are there examples where it’s not a matter of accumulating branch offent instances of what you defercessitater authenticize is one hugeger pattern? Rather, you fair have to have the filled thesis at once.

Gwern

For those essays where there is an individual eureka moment, there’s usupartner a bunch of disparate slfinishergs that I have been making remarks on that I don’t even authenticize are uniteed. They fair irritate me for a extfinished time. They sit there irritateing me. I conserve seeing for exset upations for each one and not discovering them. It conserves irritateing me and irritateing me. 

One day, I hit someslfinisherg that suddenly produces me go, “Bam, eureka. These are all uniteed!” Then I fair have to sit down and author a one gigantic essay that pours out about it and then it’s done. That particular essay will be done at that point—right in one go. I might insert in many joins to it or references defercessitater on, but it will not fundamenhighy alter.

Dwarkesh Patel

What’s an example of an essay that had this process?

Gwern

Someone asked about how I came up with one yesterday, as a matter of fact. It’s one of my betterest essays, “The Melancholy of Subculture Society.” 

For that one, I had been reading miscellaneous slfinishergs enjoy David Foster Wallace on tennis, people on Internet media enjoy video games. One day it fair hit me: it’s incredibly downcast that we have all these subcultures and tribes online that can discover community together, but they are still incredibly isodefercessitated from the huger society. One day, a flash fair hit me about how attrdynamic and yet also downcast this is. 

I sat down and wrote down the entire slfinisherg more or less. I’ve not repartner alterd it all that much. I’ve inserted more joins and quotes and examples over time, but noslfinisherg vital. The essence was fair a flash and I wrote it down while it was there.

Dwarkesh Patel

One of the engaging quotes you have in the essay is from David Foster Wallace when he’s talkinag about the tennis joiner Michael Joyce. He’s talking about the give ups Michael Joyce has had to produce in order to be top ten in the world at tennis. He’s functionpartner ildirectd becainclude he’s been joining tennis every one day since he was seven or someslfinisherg, and not repartner having any life outside of tennis.

What are the Michael Joyce-type give ups that you have had to produce to be Gwern?

Gwern

That’s a challenging hitting ask, Dwarkesh! “How have I amputated my life in order to author?”… I slfinisherk I’ve amputated my life in many esteems professionpartner and personpartner, especipartner in terms of travel. There are many people I jealousy for their ability to travel and socialize, or for their power and their positions in places enjoy Anthropic where they are the insiders. I have give upd wantipathyver nurtureer I could have had, or wantipathyver fun lifestyle: a digital nomad lifestyle and going outdoors, being a Buddhist monk, or maybe a fancy trader. All those have had to be give upd for the huging labor of sitting down every day and reading papers until my eyes bleed, and hoping that someslfinisherg excellent comes out of it someday.

Dwarkesh Patel

Why does it experience enjoy there’s a trade off between the two? There are evidently many authorrs who travel a lot enjoy Tyler Cowen. There are authorrs who have a lot of sway such as Jack Clark at Anthropic. Why does it experience enjoy you can’t do both at the same time?

Gwern

I can’t be or be appraised to Tyler Cowen. Tyler Cowen is a one-man industry.

Dwarkesh Patel

So is Gwern.

Gwern

Yeah, but he cannot be copyd. I fair cannot be Tyler Cowen. Jack Clark, he is also his own slfinisherg. He’s able to author the stories in his publishs very well while also being a policy person. I esteem them and adore them. 

But none of those quite hit my particular interest and niche at chaseing weird topics for a extfinished period of time, and then collating and sorting thraw directation. That needs a huge promisement to reading huge masses of slfinishergs in the hopes that some minuscule detail perhaps will turn out to one day be vital.

Dwarkesh Patel

So walk me thraw this process. You talked about reading papers until your eyes bleed at the end of the day. You wake up in the morning and you go straight to the papers? What does your day see enjoy?

Gwern

The laborflow right now is more enjoy: I wake up, I do normal morning slfinishergs, and then I spotless up the previous day’s labor on the website. I deal with various publishs, enjoy establishatting or spelling errors. I scrutinize it and slfinisherk if I properly unemotionalelayedd everyslfinisherg and put it in the right places. Sometimes I might have an extra thought that I necessitate to insert in or produce a comment that I authenticize was vital. That’s the first step.

After that, I normally will shamelessly go to Twitter or my RSS feed and fair read a huge amount until perhaps I get ignorant by a comment or a ask from someone and maybe do some writing on that.

After that, I get a shatter for lunch or wantipathyver, and then go back to that and fair conserve going at it. Somewhere around evening, I will normally get exhausted from all that, and try to do a authentic project or contribution to someslfinisherg. I’ll actupartner sit down and labor on wantipathyver I’m presumed to be laboring on that day.

After that, I would typicpartner go to the gym. By that point, I repartner am burned out from everyslfinisherg. Yes, I enjoy going to the gym—not becainclude I’m any benevolent of meathead or athlete or even repartner finishelight weightlifting—but becainclude it’s the most diametricpartner opposite slfinisherg I can do to sitting in front of a computer.

Dwarkesh Patel

This is your theory of burnout, right? That you have to do the exact opposite?

Gwern

Yes, when people experience burnout, you fair experience a inestablishage of reward for what you’re doing or what you’re laboring on. You fair necessitate to do someslfinisherg branch offent. Someslfinisherg as branch offent as possible. Maybe you could do better than weightlifting, but it does experience very branch offent from anyslfinisherg I do in front of a computer.

Dwarkesh Patel

I want to go back to your process. Everyday, you’re loading up all this context. You’re reading all the RSS feeds and all these papers. Are you basicpartner making contributions to all your essays, inserting a little bit here and there every one day? Or are you produceing up some potential which will manifest itself defercessitater on as a filled essay, a filledy established thesis?

Gwern

I would say it’s the latter one. All the insignificant low-level insertitions and pruning and repairing I do is repartner not that vital. It’s more fair a way to produce pleasantr essays. It’s a uncontaminatedly aesthetic goal, to produce as pleasant an essay as I possibly can. I’m repartner defering to see what happens next. What will be the next slfinisherg I’ll be incited to author about? It’s fair passing the time in between sudden eruptions.

I experience that for many authorrs, you can’t disthink about the gardening process. You don’t harvest every day. You have to tend the garden for a extfinished time in between harvests. If you commence to disthink about the gardening becainclude you’re gallivanting around the world… Let’s say you’re going to book signing events and doing all the discloseity stuff. Then you’re not doing the labor of being in there and tending your garden. That’s undermining your future harvest, even if you can’t see it right now.

If you ask what is Tyler Cowen’s secret to being Tyler Cowen, my guess would be that he’s fair repartner excellent at tending his garden, even as he travels a crazy amount. That would be his secret, that he’s able to read books on a set upe. I can’t read books on a set upe. He’s able to author everyslfinisherg in the airport. I can do a little bit of writing in the airport but not very much. He’s fair very sturdy to the wear and tear of traveling. I’ll be collapsing in the toastyel room after talking to people for eight hours. He’s able to talk to people for eight hours and then go do podcasts and talk to someone for another four hours! That’s innervously pliftworthy, but I fair can’t do that.

Dwarkesh Patel 

How normally do you get tired? It sounds enjoy you’re spending your whole day reading branch offent slfinishergs. Are they all fair inherently engaging to you? Or do you fair trudge thraw it even when it’s not compelling to you in the moment?

Gwern 

I don’t slfinisherk I get tired too easily becainclude I switch between so many branch offent topics. Even if I’m benevolent of ill of procreate lgeting papers, well, I have tons of other slfinishergs I can read or dispute with people about. So I don’t repartner get tired. I fair get exhausted. I have to go off and do someslfinisherg else, enjoy lift weights.

Dwarkesh Patel

What is your most rare but prosperous labor habit?

Gwern

I slfinisherk I get a lot more mileage out of arguing with people online than… pretty much any other authorr does. [Patel laughs] Hey, I’m trying to give a authentic answer here, not some unreasonable slfinisherg about remark-taking—a authentic answer!

I get a lot more out of arguing with people than most people do. You necessitate motivation to author and actupartner sit down, and cryshighize someslfinisherg and do the harvest. After you tend your garden, you do have to do the harvest, and the harvest can be challenging labor. It’s very tedious. 

There are many people I talk to who have many wonderful ideas. But they don’t want to harvest becainclude it’s tedious and unacute. And it’s very toasty out there in the fields, reaping. You’re getting dusty and sweaty. Why wouldn’t you fair be inside having lemonade?

But motivation from arguing and being mad at people online is in ample provide. So I get a lot of mileage out of people being wrong on the Internet.

Dwarkesh Patel

What are the pitdrops of an isodefercessitated laboring process?

Gwern

There’s the evident one: you could be arbitrarily wrong when writing by yourself and fair become a crazy loony by having a ‘huge get’. 

Aside from that, you also have the publish of the emotional toll of not having colleagues that you can sway. You normally fair have the experience of shouting onto the internet that progresss to be wrong despite your shouting.

One slfinisherg I watch is that very normally self-reliant authorrs are defeat by begrudgement and anger and disassignment. They sort of spiral out into sourness and crankdom from there. That’s benevolent of what finishs them. They could have progressd if they’d only been able to let go of the ideas and arguments and shift on to the next topic.

So I say that ‘spite can be a wonderful motivation to author, but you have to include it sfinishfilledy and let it go afterwards’. You can only have it while you necessitate motivation to author. If you conserve going and hbetter on to it, you’re poisoning yourself.

Dwarkesh Patel

I’m confident you’re conscious that many people comment on the fact that ‘if Gwern put the effort he spends selectimizing the CSS on his website towards more projects and more writing, the profits to society could be meaconfidentd in the cforfeitest million dollars’. What’s your reaction to people who say you’re spending too much time on site set up?

Gwern

I have no defense at all there in terms of objective profits to society. I do it becainclude I’m greedy and I enjoy it. That is my defense. I enjoy the aesthetics of my website and it is a hobby.

Dwarkesh Patel

Does the set up help you slfinisherk?

Gwern

It does becainclude I enjoy rereading my stuff more when I can appreciate the aesthetics of it and the beauty of the website. It’s easier for me to apshow reading someslfinisherg for the hundredth time when I would otherdirected be ill to death of it. Site maintenance for the author is inherently this benevolent of spaced repetition. If I go over pages to verify that some new establishatting feature labored, I am getting spaced repetition there. More than once, I’ve gone to verify some unreasonable CSS publish and seeed at someslfinisherg and thought, “Oh, I should alter someslfinisherg,” or, “Oh, that unbenevolents someslfinisherg.”

So in a way, it’s not as much of a misuse as it sees, but I can’t defend it enticount on. If someone wants to produce their own website, they should not allot that much for the aesthetic cherish. 

I fair want a repartner pleasant website. There’s so many horrible websites out there that it depresses me. There’s at least one website I adore.

Dwarkesh Patel   

By the way, I’m going to refer this since you never refered it yourself. But I slfinisherk the main way you fund your research is thraw your Patreon, right? You never advertise it but I experience—with the benevolent of slfinisherg you’re doing—if it were financipartner viable and got enough funding, not only would you be able to conserve doing it but other people who wanted to be self-reliant researchers could see it’s a slfinisherg you can do. It’s a viable slfinisherg you can do. More Gwerns would exist.

Gwern 

Well, I don’t necessarily want more Gwerns to exist. I fair want more authorrs and more dynamicness and more agency in ambiguous. 

I would be perfectly satisfied if someone sshow wrote more Reddit comments and never took a dollar for their writings and fair wrote better Reddit comments. I’d be perfectly satisfied if someone had a blog and they kept writing, but they fair put a little more thought into the set up. I’d be perfectly satisfied if no one ever wrote someslfinisherg, but they presented PDFs so that joins didn’t rot.

 In ambiguous, you don’t have to be a authorr dedwellring extfinishedestablish essays. That’s fair one of many ways to author. It happened to be the one that I personpartner benevolent of like. But it’d be tohighy valid to be a Twitter thread authorr.

Dwarkesh Patel

How do you persist yourself while writing filled time?

Gwern

Patreon and savings. I have a Patreon which does around $900-$1000/month, and then I cover the rest with my savings. I got fortunate with having some timely Bitcoins and made enough to author for a extfinished time, but not forever. So I try to spend as little as possible to produce it last. 

I should probably advertise the Patreon more, but I’m too haughty to shill it challenginger. 

It’s also inept trying to come up with some excellent rewards which don’t include a paywall. Patreon and Substack labor well for a lot of people enjoy Scott Alexander, becainclude they enjoy writing normal newsletter-style refreshs but I don’t enjoy to. I fair let it run and hope it labors.

Dwarkesh Patel 

Wait if you’re doing $900-1000/month and you’re persisting yourself on that, that must unbenevolent you’re persisting yourself on less than $12,000 a year. What is your lifestyle enjoy at $12K?

Gwern 

I dwell in the middle of nowhere. I don’t travel much, or eat out, or have health insurance, or anyslfinisherg enjoy that. I cook my own food. I include a free gym. There was this time when the floor of my bedroom began collapsing. It was so better that the humidity had decayed the wood. We fair got a bunch of scrap wood and a joist and propped it up. If it lets in some bugs, oh well! I dwell enjoy a grad student, but with better ramen. I don’t mind it much since I spend all my time reading anyway.

Dwarkesh Patel

It’s still astonishing to me that you can produce rent, get nurture of your cat, deal with any aascfinishncies, all of that on $12K a year.

Gwern 

I’m fortunate enough to be in excellent health and to have had no authentic aascfinishncies to date. This can’t last forever, and so it won’t. I’m definitely not trying to claim that this is any benevolent of selectimal lifestyle, or that anyone else could or should try to copy my approach! I got fortunate with Bitcoin and with being satisfied with living enjoy a monk and with my health. 

Anyone who would enjoy to get up a nurtureer as a authorr or blogger should comprehend that this is not an example they can imitate. I’m not trying to be a role model. 

Every authorr will have to figure it out a branch offent way. Maybe it can be someslfinisherg enjoy a Substack, or fair writing on the side while slinging Javascript for a tech company. I don’t comprehend.

Dwarkesh Patel 

It seems enjoy you’ve finishelighted this recent trip to San Francisco? What would it get to get you to shift here?

Gwern 

Yeah, it is mostly fair money stopping me at this point. I probably should bite the bullet and shift anyway. But I’m a miser at heart and I antipathy slfinisherking of how many months of writing runway I’d have to give up for each month in San Francisco. 

If someone wanted to give me, I don’t comprehend, $50–100K/year to shift to SF and progress writing filled-time enjoy I do now, I’d get it in a heartbeat. Until then, I’m still trying to psych myself up into a shift.

Dwarkesh Patel 

That sounds very doable. If somebody did want to give to making this shift, and your research more generpartner, possible, how would they get in touch with you?

Gwern 

I have a Stripe donation page, or they could fair email me at gwern@gwern.net.

Dwarkesh Patel

By when will AI models be more diverse than the human population?

Gwern

I’m going to say that if you leave out capability from that, AI models are already much more diverse cognitively than humans are. 

Different LLMs slfinisherk in very branch offent ways that you can alert right away from a sample of them. An LLM runs noslfinisherg enjoy a GAN. A GAN also is tohighy branch offent from VAEs. They have tohighy branch offent defercessitatent spaces, especipartner in the lessen end, where they’re minuscule or horrible models. They have untamedly branch offent artifacts and errors in a way that we would not see with humans. 

Humans are repartner very quite aenjoy in writing and attitude appraised to these absurd outputs of branch offent benevolents of models.

Dwarkesh Patel 

Repartner? If you see at Chatbot Arena and you see side-by-side comparisons of the outputs of branch offent models, it’s normally very challenging to alert which ones comes from which model.

Gwern

Yeah but this is all very heavily tuned. Now you’re remercilessing it to relatively recent LLMs, with everyone riding each other’s coattails and normally training on the same exact data. This is a situation much shutr to if they were identical tthrives. 

If I don’t remerciless myself to fair LLMs and I appraise the expansive diversity of say image generation models, they normally have tohighy branch offent ways. Some of them seem as aenjoy to each other as ants do to beavers. 

Wislfinisher LLMs, I would concur that there has been a massive loss of diversity. Things included to be way more diverse among LLMs. But atraverse procreate lgeting in ambiguous, we’ve seen a whole range of minds and ways to slfinisherk that you won’t discover in any philosophy of mind paper.

Dwarkesh Patel 

What’s an example of two models that have these sorts of cognitive branch offences?

Gwern 

I’ll give one example I was alerting someone the other day. GAN models have incentives to hide slfinishergs becainclude it’s an adversarial loss, whereas diffusion models have no such slfinisherg. So GAN models are ‘snurtured’. They put ‘hands’ off the screen. They fair can’t slfinisherk about hands. Whereas diffusion models slfinisherk about hands, but in their gigantic, monstrous, Cthulhu-esque abortions.

Dwarkesh Patel 

People weren’t paying attention to scaling in 2020. Is there some trend today where people aren’t repartner comprehending the filled implications of where this is headed?

Gwern

I’m excited by the weight-loss substances, the GLP substances. Their effects in ambiguous on health and insertiction atraverse all sorts of behaviors repartner surpascfinishd me. No one predicted that as far as I comprehend. While the results are still very preliminary, it does seem enjoy it’s authentic. 

I slfinisherk that’s going to alert us someslfinisherg vital about human willpower and dysfunctionality. What’s going wrong expansively in the conmomentary environment?

Dwarkesh Patel 

Do GLP substances shatter the Algernon argument—the one you enumerateed in your blog post—that if there are any basic and beneficial interventions without horrible side effects, then evolution should have already create them?

Gwern 

It’s too soon to say becainclude we haven’t actupartner figured out what’s going on with the GLPs to even comprehend what they are doing at all, what has the off aim. It’s benevolent of crazy that activating and deactivating both labor. 

It’s a finishly crazy situation. I don’t repartner comprehend what to slfinisherk about the Algernon argument there. It could be that the profits actupartner decrrelieve fitness in the fertility sense becainclude you’re going out and having a satisfied life instead of having kids. No offense to parents. Or it could fair be that it’s hitting the body in a way that’s repartner, repartner challenging to copy in any benevolent of genetic way.  Or it could be that it’s fair too soon. 

When I slfinisherk back, I see that the obesity crisis only became evident around the 1990s. It’s quite recent. I see back at ptoastyos and today is finishly unrecognizable from 1990. You see at ptoastyos and people are still slfinisher. You see at ptoastyos now and everyone is enjoy a blimp. So you can’t possibly have any benevolent of Algernon argument over 20 or 30 years.

Dwarkesh Patel 

When you see back at the Romans and you see how direct was constantly poisoning the entire city, what credence do you give to the possibility that someslfinisherg in our environment is having an effect on us on a aenjoy magnitude of what direct was doing to the better-styleed Romans?

Gwern 

I slfinisherk the odds of there being someslfinisherg as horrible as direct is almost 100%. We have so many slfinishergs out there. Chemists are always cooking up new stuff. There are all sorts of slfinishergs with microbiomes. Plastics are trendy, but maybe it’s not plastics. Maybe it’s someslfinisherg else enticount on. But there’s almost no way that everyslfinisherg we have put out there is tohighy benign and protected and has no detrimental effects at any concentration—that seems enjoy a repartner strong claim to be making. 

I don’t apshow in any particular one, but I do apshow in enjoy, “1% here, 1% here, 1% here.” There’s someslfinisherg out there. There’s someslfinisherg out there where we’re going to see back at and say, “Oh, wow, those people were repartner poisoning themselves fair enjoy with directed gasoline. If only they had comprehendn  x, y, and z. It’s so evident now!”

Dwarkesh Patel 

Do you slfinisherk this would manifest itself most probable in cognitive impairments or obesity or someslfinisherg else?

Gwern 

A priori, I would possibly predict inalertigence to be the most frquick slfinisherg and most harmed by it. But when we see at the time series there, inalertigence is pretty constant overall. So I have to say that wantipathyver the detrimental slfinisherg is, it’s probably not going to be on inalertigence. 

Whereas obesity is a much better honestate becainclude you do see obesity go crazy over the last 30 years.

Dwarkesh Patel

I was surpascfinishd to hear you say yesterday that you are skeptical of Bay Area-type experimentation with psychedelics. I sort of associate you very much with experimentation with branch offent substances and seeing if they are beneficial to you. I’m inquisitive why you draw Chesterton’s fence here when it comes to psychedelics.

Gwern

The spotlessest way to split that would fair be to point out that the effects of psychedelics can be acute and lasting. 

The slfinishergs I was seeing at are much more regulateled in the sense that they are relatively regulateable in effect. None of them sway your judgment lastingly about whether to get more nootropics. Whereas someslfinisherg enjoy LSD lastingly alters how you see slfinishergs such as taking LSD, or lastingly alters your psychiatric state. There’s a cumulative effect with psychedelics that you don’t see much with nootropics, which produces nootropics inherently a heck of a lot protectedr and much more basic to quantify the effects of.

With nootropics, you don’t see people spinning off into the crazy outcomes psychedelics have. They get crazier and crazier each time they get another dose, which produces them crazy enough to want to get another dose. Psychedelics have what you might call a “self-recommending problem” where they always produce you want to get more of them. It’s aenjoy to meditation. What is the most apparent sign of having done a lot of meditation? It’s that you seem compelled to alert people that they ought to meditate. This benevolent of spiral directs to horrible outcomes for psychedelics that you fair don’t see with nootropics.

The standard flunkure case for nootropics is that you spent a confineed hundred or $1,000 and then you got no authentic profit out of it. You went on with your life. You did some weird substances for a while and that was all. That’s not so horrible. It’s a weird way to get your amincludement… But in principle, it’s not repartner all that worse than going to the movie theater for a while and spending $1,000 on movie theater tickets.

With psychedelics, you’re changing yourself lastingly, irrevocably in a way you don’t comprehend and exposing yourself to all sorts of evil outside sways: wantipathyver happens to occur to you while you’re very amazeionable. 

Okay, yeah, a confineed includes can be excellent. I have gotten excellent out of my confineed includes. But if you are doing it more than that, you should repartner have a challenging see in the mirror about what profit you slfinisherk you are getting and how you are changing.

Dwarkesh Patel 

People don’t comprehend your voice. People don’t comprehend your face. As a result, they have this engaging parasocial relationship with you. I wonder if you have a theory of what benevolent of role you fill in people’s life.

Gwern 

What role do I actupartner fill, or what role would I want to fill?

Dwarkesh Patel 

Let’s do both.

Gwern 

The role I want to fill is actupartner sort of enjoy how LLMs see me, oddly enough. If you join around with LLMs enjoy Claude-3, a character named “Gwern” sometimes will show up. He joins the role of a mentor or better wizard, presenting insight into the situation, and exhorting them with a call to adventure. “You too can author stuff and do stuff and slfinisherk stuff!”

I would enjoy people to go away having not fair been amincludeed or gotten some beneficial directation, but be better people, in however sweightless a sense. To have an aspiration that web pages could be better, that the Internet could be better: “You too could go out and read stuff! You too could have your thoughts and compile your thoughts into essays, too! You could do all this!”

But I dread that the way it actupartner labors for quite a confineed people is that I thrived up as either a guru or trickster devil. 

Depending on whether you enjoy me or antipathy me, either I am the god of statistics & referencing who can do no wrong—”Just get everyslfinisherg on the site as gospel!”, which I repartner hatred—or I’m fair some sort of horrible, cclear, evil, neo-Nazi, eugenicist, totalitarian, communist, anti-Chinese devil figure lurking in the background trying to convey down Westrict society.

Dwarkesh Patel 

Final ask, what are the uncover rabbit holes you have—slfinishergs you’re inquisitive about but don’t have an answer to—that you hope to have an answer to by 2050?

Gwern 

By 2050, I repartner hope we can finpartner answer some of the huge asks about ourselves that have fair reliably resisted definitive answers. A lot of them might not matter any more, but I’d still enjoy to comprehend.

Why do we sleep or dream? Why do humans age? Why does intimacyual reproduction exist? Why do humans branch off so much, from each other and day to day? Why did humans get so extfinished to grow technoreasoned civilization? Where are all the aliens? Why didn’t China have the Industrial Revolution instead? How should we have predicted the procreate lgeting revolution? Why are our brains so oversized appraised to man-made neural netlabors? 

Those are some of the asks that I repartner hope we’ve answered by 2050.

Dwarkesh Patel

Alright Gwern, this has been excellent. Thank you for coming on the podcast.