trentmkelly/LessWrong-43k huggingface dataset:
Rationality-related things I don't know as of 2023
by Adam Zerner — 2023-02-11T06:04:16.183Z

One of the blog posts I'm most fond of is Things I Don’t Know as of 2018. It's by Dan Abramov, one of the more prominent people in the world of front-end web development. He goes through a bunch of relatively basic programming-related things that he doesn't understand, like unix commands and low-level languages.

I'd like to do something similar, but for rationality-related things. Why?

For fun.

To normalize the idea that no one's perfect.

It'll make it easier to address these knowledge gaps. Or maybe just more likely that I actualy do so.

Here's the list:[1]

Simulcra. I spend some time going through the posts and it's one of those things that just never manages to click with me.

Blockchain. I guess the thing that I don't understand here is the hype. I get that it's a basically a database that can't be editted and I've read through articles talking about the use cases, but it's been around for a while now and doesn't seem to have been that game changing. Yet there are smart people who are super excited about it and I suspect that there are things I am failing to appreciate, regardless of whether their excitement is justified.

Morality. To me it seems like rationality can tell you how to achieve your goals but not what (terminal) goals to pick. Arguments that try to tell you what terminal goals to pick have just never made sense to me. Maybe there's something I'm missing though.

Quantum physics. I skipped/lightly skimmed the sequence posts on this. Seemed high effort and not particularly important. Well, it is cool to understand how reality works at the most fundamental level. Hm. I would be interested in a going through some sort of lower effort bigger picture material on quantum physics. I spent some time messing around with that sort of stuff like 13 years ago but all that stuck is some vague notion that reality is (fundamentally?) probabilistic and weird.

Evolution. I get that at a micro-level, if something makes an organism more likely to reproduce it will in fact, err, spread the genes. And then that happens again and again and again. And since mutations are a thing organisms basically get to try new stuff out and the stuff that works sticks. I guess that's probably the big idea but I don't know much beyond it and remember being confused when I initially skimmed through the The Simple Math of Evolution sequence.

Evolutionary psychology. I hear people m

...
A note on 'semiotic physics'
by metasemi — 2023-02-11T05:12:30.607Z

Introduction

This is an attempt to explain to myself the concept of semiotic physics that appears in the original Simulators post by janus and in a later post by Jan Hendrik Kirchner. Everything here comes from janus and Jan's work, but any inaccuracies or misinterpretations are all mine.

TL;DR

The prototypical simulator, GPT, is sometimes said to "predict the next token" in a text sequence. This is accurate, but incomplete.
It's more illuminating to consider what happens when GPT, or any simulator, is run repeatedly to produce a multi-token forward trajectory, as in the familiar scenario of generating a text completion in response to a prompt.
The token-by-token production of output is stochastic, with a branch point at every step, making the simulator a multiverse generator analogous to the time evolution operator of quantum mechanics.
In this analogical sense, a simulator such as GPT implements a "physics" whose "elementary particles" are linguistic tokens. When we experience the generated output text as meaningful, the tokens it's composed of are serving as semiotic signs. Thus we can refer to the simulator's physics-analogue as semiotic physics.
We can explore the simulator's semiotic physics through experimentation and careful observation of the outputs it actually produces. This naturalistic approach is complementary to analysis of the model's architecture and training.
Though GPT's outputs often contain remarkable renditions of the real world, the relationship between semiotic physics and quantum mechanics remains analogical. It's a misconception to think of semiotic physics as a claim that the simulator's semantic world approximates or converges on the real world.[1]

Trajectories

GPT, the prototypical simulator, is often said to "predict the next token" in a sequence of text. This is true as far as it goes, but it only partially describes typical usage, and it misses a dynamic that's essential to GPT's most impressive performances. Usually, we don't simply have GPT predict a single token to follow a given prompt; we have it roll out a continuous passage of text by predicting a token, appending that token to the prompt, predicting another token, appending that, and so on.

Thinking about the operation of the simulator within this autoregressive loop better matches typical scenarios than thinking about single token prediction, and is thus a better fit to wha...
Inequality Penalty: Morality in Many Worlds
by Shmi — 2023-02-11T04:08:19.090Z

This is a summary of Sean Carroll's musings on when it would matter which model of apparent collapse is more accurate.

Executive Summary: If you care about equality, or even are risk-averse, then your decisions depend on whether you subscribe to the Many Worlds model.

Disclaimer: I am personally MWI-agnostic, the universe is generally weirder than we can conceive, and resolution of this particular question has been evading us for over 65 years. So whatever the next insight is, odds are, it will come bundled with a new unexpected paradigm. However, Sean Carroll is very much pro-MWI, and his reasons to like it are very sensible, though he readily admits that new evidence could come up that would refute this particular belief.

Here is a revelation moment for him from his Mindscape podcast episode with a philosopher Lara Buchak:

0:55:39.7 LB: It’s like there are a hundred future possible Seans. What would you rather giving all the future possible Seans a million dollars or giving 98 of them a million dollars and giving one of them nothing and one of them $20 million? And whereas the expected utility theorists will say there’s a unique answer to that question about how Sean should value his future possible Seans. You should give them all equal weight in decision making. I say, no, actually it’s up to you. If you want to put more weight on how things go for worst off possible Sean, that’s a reasonable way to take the means to your ends. That’s a reasonable way to sort of like cash out the maximum of I’m trying to get what I want. On the other hand, if you, as I guess you do put a lot of weight on best off future possible Sean, that’s also a reasonable thing to do. In either case, you only have one life to live. Only one of these guys is going to be actual Sean. So it’s up to you to think about how much weight to put on each of their interests knowing that only one of them will be actual.

0:57:01.0 SC: You know, it only now dawns on me, this is very embarrassing, but I have to think about this in the context of the many worlds interpretation of quantum mechanics, which I’m kind of a proponent of. So the whole point of many worlds is that what we think of as probabilities really are actualities well, quantum probabilities, not every old probabilities. But if we did our choice making via some quantum random number generator, then yeah, I’ve always taken the line, th

...
The Importance of AI Alignment, explained in 5 points
by Daniel_Eth — 2023-02-11T02:56:05.103Z

This piece gives an overview of the alignment problem and makes the case for AI alignment research. It is crafted both to be broadly accessible to those without much background knowledge (not assuming any previous knowledge of AI alignment or much knowledge of AI) and to make the compositional logic behind the case very clear.

I expect the piece will be particularly appealing to those who are more reductive in their thinking and who want to better understand the arguments behind AI alignment (I’m imagining this audience includes people within the community doing field building, as well as people in various STEM fields who have only vaguely heard of AI alignment).

Crossposted to the AGI Safety Fundamentals website, with minor edits:
https://www.agisafetyfundamentals.com/alignment-introduction

This piece describes the basic case for AI alignment research, which is research that aims to ensure that advanced AI systems can be controlled or guided towards the intended goals of their designers. Without such work, advanced AI systems could potentially act in ways that are severely at odds with their designers’ intended goals. Such a situation could have serious consequences, plausibly even causing an existential catastrophe. In this piece, I elaborate on five key points to make the case for AI alignment research:

Advanced AI (defined below) is possible
Advanced AI might not be that far away (possibly realized within decades)
Advanced AI might be difficult to direct
Poorly-directed advanced AI could be catastrophic for humanity
There are steps we can take now to reduce the danger

1 – Advanced AI is possible

By advanced AI, I mean, roughly speaking, AI systems capable of performing almost all cognitive work humans perform (e.g., able to substitute for scientists, CEOs, novelists, and so on).

Researchers disagree about the form of advanced AI that is most likely to be developed. Many of the most popular visions involve an “artificial general intelligence,” or “AGI” – a hypothetical AI system that could learn any cognitive task that a human can.[1] One

possibility

for advanced AI is a singular AGI that could outcompete human experts in most fields across their areas of expertise. Another

possibility

is an ecosystem of specialized AI systems that could collectively accomplish almost all cognitive work – some researchers speculate that this s...
Acting Normal is Good, Actually
by Gordon Seidoh Worley — 2023-02-10T23:35:41.043Z

This is a casually written post in a series about what I wish someone had told me when I was younger.

I was a weird kid, but I didn't originally set out to be weird. I like doing things other people do and feeling like I belong. But as I got past kindergarten, forces incentivized me to be weird.

There was a bunch of stuff pushing me in this way:

My IQ scored landed me in gifted classes. Being labeled smart and gifted meant adults gave me a free pass to be weird rather than encouraging me to be normal because weirdness is more tolerated if you're useful in some other way.
Asthma kept me from being good at sports, so no one really wanted me on their team. So life was easier if I engaged in other activities instead of doing what all the regular kids did.
I wasn't religious, so I was the weirdo atheist who had to justify his existence and beliefs instead of being a normal, God-fearing person.
I had OCD, and this meant I sometimes had to do weird things to avoid having a meltdown.

So I ended up hanging out with the other weird kids because the thing we had in common is that we didn't fit in with the normals. We were treated like outcasts by our classmates, allowed to live on the edge of the village, so to speak, so long as we continued to be occasionally useful.

Of course, lots of people feel like they're weird at times, especially as kids, so it comes in degrees. Even popular, super normal kids feel like they don't fit in sometimes, and that seems to be part of growing up, at least in the West. But what I'm talking about is being weird enough that weird becomes part of your identity because other people make it part of your identity, and then you lean into it because you know you can get away with it.

There's virtue in being weird at times. I don't think I have to convince folks here of that! But there is also a lot of virtue in being and acting normal.

Some good things can happen if you act normal, by which I mean act in ways that something like within one standard deviation of whatever the average thing to do is in a particular situation. To wit:

People automatically assume you're part of the ingroup and one of them, so they are nicer to you.
People understand how to interact with you and so can more easily help you and make requests of you, and thus they can build stronger feelings of affinity towards you.
You don't have to figure out how to make the world accommodate you

...
[S] D&D.Sci: All the D8a.  Allllllll of it.
by aphyer — 2023-02-10T21:14:59.192Z

This is an entry in the 'Dungeons & Data Science' series, a set of puzzles where players are given a dataset to analyze and an objective to pursue using information from that dataset.

STORY (very, very, very much skippable)

aphyer: Write MSPA D&D.Sci scenario.

Yes.

NovelAI image: prompt 'man male guy using computer screen Excel'

Hell yes.

It's hard.

Hell.

Being a data scientist and trying to do art.

Fucking.

It's hard and no one understands.

Yes.

aphyer: Recap.

You and some of your online friends recently discovered an exciting new game called Housetrapped. You...weren't exactly expecting that the game would destroy your real-world hometown with meteors, that it would nearly kill you with real-world monsters, or that beating the game would in some way end up creating a new universe, but it seems that is in fact how this game works.

Mage: Enter

After spending forever bungling around (in a marathon adventure of combat and construction that would probably take thousands of pages to summarize, which is nevertheless just the first part of this game), you have entered into the game world as the Mage of Time in the Land of Seals and Melting.

NovelAI image: prompt 'melting iceberg seal animal'

Look at that sad little seal. You shall name her...Clementine!

You start arranging matters for the rest of your friends to enter into the game as well. You are the Mage of Time, and one of your friends has already entered as the Knight of Blood, but two other friends are still in the regular world, and need to enter and take up their Classes and Aspects before they get hit with meteors! You pull out your laptop to set things up for them to enter as fast as possible.

Mage: Appear

Suddenly, Future You appears in a puff of time. This is amazingly unexpected.

NovelAI image: prompt 'androgynous mspa sprite mage time red gears machinery robe'

For some reason Future You seems to look a lot like an anime character. And have a sweet robe. How do you get a robe like that anyway? Maybe you should ask. But first you need to get your other friends into the game world! Maybe Future You is here to help do that faster?

Mage: Avert

Future You tells you not to do that right away. You ask Future You why. Future You says that you already did that and it didn't work. You say that no, you didn't. Future You says yes, y...
A Different Kind of Ark: My failed attempt to build a bridge between universes
by ChrisM — 2023-02-10T20:49:12.524Z
Link: https://www.vesselproject.io/essays/a-different-kind-of-ark

In the summer of 2020, I launched a website called The Vessel Project. It’s a weird, winding, conjecturing exploration into the nature of reality. Some would call it crazy. I don’t necessarily disagree. But the story behind its inception is even weirder. What is now a pet theory, a glorified blog, began as a mission of cosmic proportions. The project is also an artifact of a personal journey. It’s my own reminder of grandiose visions (verging on delusions), a bit of idealistic naiveté, and a whole lot of grappling with my place in the universe. But for an adventure that’s mostly played out between my two ears, I could hardly imagine a more thrilling ride.

About a year and a half prior to the website’s launch, I had an epiphany that struck me so hard it immediately changed the way I perceived reality. Wild thing to say, right? Right or wrong, I was convinced I had just peered behind the curtain of reality with a single idea.

Ask yourself this: if humans had the ability to simulate reality itself, what would we do with that technology? The question is not new, nor is one commonly posed answer: we’d create ancestor simulations, either to observe the development of intelligent life and the different ways it can unfold, or as a kind of memorial for those we’ve loved. But again, this was not new thinking.

I also knew the nascent field of quantum computing was attempting to harness fundamental physical phenomena to perform computations. I figured if there was any technology that could someday simulate a universe like ours, it had to be quantum computing, even if it’s nowhere close to that level of sophistication yet. Some things we observe simply cannot be replicated using classical computing methods. Still, a quantum computing universe was not a new idea.

The new part relates to how we’d encode these simulations, and my idea was basically a futuristic take on Noah’s Ark. If we could encode ourselves into another reality, wouldn’t we use our DNA sequence to do so? Our genome is just a string of information, and we could encode that information to simulate at least the physical embodiment of specific people. That got me thinking – genomic sequencing is something we can already do, plus we have the means to store that data for virtually as long as we need it. If there’s even the slight possibility that humans will one day have the ability and desire to bring new life to ...
Prizes for the 2021 Review
by Raemon — 2023-02-10T19:47:43.504Z

If you received a prize, please fill out your payment contact email and PayPal.

A'ight, one final 2021 Review Roundup post – awarding prizes. I had a week to look over the results. The primary way I ranked posts was by a weighted score, which gave 1000+ karma users 3x the voting weight. Here was the distribution of votes:

I basically see two strong outlier posts at the top of the ranking, followed by a cluster of 6-7 posts, followed by a smooth tail of posts that were pretty good without any clear cutoff.

Post Prizes

Gold Prize Posts

Two posts stood noticeably out above all the others, which I'm awarding $800 to.

Strong Evidence is Common by Mark Xu
“PR” is corrosive; “reputation” is not, by Anna Salamon.

I also particularly liked Akash's review.

Silver Prize Posts

And the second (eyeballed) cluster of posts, each getting $600, is:

Your Cheerful Price, by Eliezer Yudkowsky.

This notably had the most reviews – a lot of people wanted to weigh in and say "this personally helped me", often with some notes or nuance.

ARC's first technical report: Eliciting Latent Knowledge by Paul Christiano, Ajeya Cotra and Mark Xu.
This Can't Go On by Holden Karnofsky
Rationalism before the Sequences, by Eric S Raymond.

I liked this review by A Ray who noted one source of value here is the extensive bibliography.

Lies, Damn Lies, and Fabricated Options, by Duncan Sabien
Fun with +12 OOMs of Compute, by Daniel Kokotajlo.

Nostalgebraist's review was particularly interesting.

What 2026 looks like by Daniel Kokotajlo
Ngo and Yudkowsky on alignment difficulty. This didn't naturally cluster into the same group of vote-totals as the other silver-prizes, but it was in the top 10. I think the post was fairly hard to read, and didn't have easily digestible takeaways, but nonetheless I think this kicked off some of the most important conversations in the AI Alignment space and warrants inclusion in this tier.

Bronze Prize Posts

Although there's not a clear clustering after this point, when I eyeball how important the next several posts were, it seems to me appropriate to give $400 to each of:

How To Write Quickly While Maintaining Epistemic Rigor, by John Wentworth
Science in a High-Dimensional World by John Wentworth
How factories were made safe by Jason Crawford
Cryonics signup guide #1: Overview by Mingyuan
Making Vaccine by John Wentworth
Taboo "Outside View" by Daniel Ko

...
A proposed method for forecasting transformative AI
by Matthew Barnett — 2023-02-10T19:34:01.358Z

In 2021, I proposed measuring progress in the perplexity of language models and extrapolating past results to determine when language models were expected to reach roughly "human-level" performance. Here, I build on that approach by introducing a more systematic and precise method of forecasting progress in language modeling that employs scaling laws to make predictions.

The full report for this forecasting method can be found in this document. In this blog post I'll try to explain all the essential elements of the approach without providing excessive detail regarding the technical derivations.

This approach can be contrasted with Ajeya Cotra's Bio Anchors model, providing a new method for forecasting the arrival of transformative AI (TAI). I will tentatively call it the "Direct Approach", since it makes use of scaling laws directly to make predictions about compute requirements for AI.

Naturally, the Direct Approach is a very speculative framework and might end up being useless for forecasting TAI (in fact, I consider this the most likely outcome). Nonetheless, I'm hopeful that something like it can serve as a better foundation than current TAI timelines models, which I currently think are likely even worse. Note that there may be errors in the report and Colab notebook, as they were not extensively fact-checked.

Some background

In a nutshell, this approach is simply about taking the cross-entropy loss of an autoregressive model and trying to find a way of interpreting that quantity qualitatively: that is, something we can put on a chart and extrapolate until the quantity reaches a natural threshold that we identify with something important.

In my 2021 post about predicting language model performance, I drew a trendline through a plot of language model perplexities on various benchmarks and noted when the trendline went through estimates of "human-level" perplexity. This approach felt reasonable to me at the time, but I now think it too easily hand-waved away some important details.

The error of omission I committed in my old approach becomes more apparent when you think about language model performance from the perspective of scaling laws, for example the parametric scaling law from Hoffmann et al. 2022:

L(N,D)=E+ANα+BDβ

Here, we see cross-entropy loss as a function of parameters N and training tokens D seen during training. Notably, if we take ...