trentmkelly/LessWrong-43k huggingface dataset:
Game that might improve research productivity
by Jack R — 2022-03-26T07:00:11.599Z

To figure out how to make progress on problems that require cognitive effort, it seems useful to think of times in the past when you’ve made lots of cognitive progress. For me, high school math camp was the most cognitively productive time.

It seems plausible that the main reason I was productive at math camp was because of the competitive atmosphere. In light of this, I want to experiment with systems that harness people’s competitiveness to drive learning and insight, since maybe this can increase productivity substantially. “Insight Hunt” is a way this could be done.

I suspect the below idea is somewhat far from the ideal and that it could change a fair bit in response to user feedback. If anyone tries an Insight Hunt, please leave your feedback and results (including the name of the winner if you want increased motivation) in the comments! Also, feel free to suggest alternative games/systems which leverage competitiveness for cognitive productivity.

Lastly, I suspect that Insight Hunt works best when the stuff the players want to learn is more “shallow” i.e. knowledge/facts that require few inferential steps from what most people know to understand. This is because if one person wants to learn linear algebra (not shallow), and another person knows linear algebra, the game will devolve into a lecture (I suspect). Examples of shallow knowledge include alignment theory (given some pre-requisites, like basic ML; this is the primary use-case I'm excited about) and psych study results. Non-examples include theoretical physics and math.

Insight Hunt

There are three players; one player is the “judge” and two are the “seekers”
Who plays as the judge rotates such that everyone is the judge equally often
Every round:

The seekers spend 8 minutes trying to find “insights” (i.e. true and useful sentences) that score highly according to the judge, who scores the insights based on how useful and interesting they found the sentence

Before the round, it’s in the judge’s interest (who wants to learn) to tell the seekers what kind of stuff they want to know, or e.g. what papers/posts/ideas they want summarized to them

Examples include “what is deep double descent?” or “what are the main reasons sleeping 8 hours might be important?” or “what’s John Wentworth’s main problem with ELK?”

Finding insights might look like quickly digesting a paper, DMing someone for a quick insig

...
You must on average be improving to not guarantee failure.
by eeegnu — 2022-03-26T05:28:12.690Z

I was thinking about what happens when you run independent Bernoulli trials forever, with probability pi of success, but you lose the first time you fail. If all pi are the same, then with probability 1 we will eventually fail[1]. Similarly, if the supremum of the sequence of probabilities p is not 1, then we're also guaranteed to fail, since P(p never fails)=∏pi≤∏supip=0. The condition that limi→∞pi=1 is necessary but not sufficient to guarantee that failure isn't certain[2]. That seems to say that not only do we need to improve, but we also can't improve too slowly, or else we're sure to fail. So just how quickly do we need to improve? Well we have that ∑(1−pi)<∞⟹∏pi>0[3], which is much easier to work with (though this isn't a bijection so it can only be used to show that a sequence p improves quickly enough.) Let qi=1−pi, which can be interpreted as our probability of failure. We know right away that any kind of geometric progression of failure minimization, e.g. q=12,14,18,… will give us a better than 0 chance. That says that for any positive constant c>1, it's sufficient for qi>qi+1c to not guarantee failure! But surely we can do better than this, i.e. there's no need to have to improve at a constant rate. We know that ∑1n2=π26 converges, so here we need only for qi>qi+1(1+2i+1i2) or simply qi>qi+1(1+o(1i)), so the further out we go, the less we need to improve relative to our previous rate of failure. This analysis was rather quick, and I'm curious if we can do better than this.

I thought about this problem after thinking about processes where even a single failure would be catastrophic. While in the real world a sufficient number of 0's for the probability of an event makes it negligible, things are different if you live forever and you're dealing with something that can never be minimized to a 0% chance of failure. You need to at least on average improve your odds between samplings, or failure is eventually certain.

^

Technically this isn't guaranteed, there's nothing stopping a fair coin from landing on heads forever other than our finite lifespans preventing us from observing it. But I think it's fair to call a 0% chance of success

...
Your risk of developing long COVID is probably high
by Siebe — 2022-03-26T04:10:38.159Z

Context: I wrote the following on Facebook for a general audience. I'm posting it here because I'm very concerned. I'm severely disabled from a presumable* COVID infection 17 months ago. I haven't done deep dives into (m)any of these, so I'm plausibly wrong. However, this is the main position in the long COVID community. Sorry not sorry that it isn't polished. This post does not include new strains, which might also be worrisome (https://twitter.com/PeacockFlu/status/1504158873938272269?t=y2l5alk3ZjUOaDghjquUrg&s=08).

Summary:

Please keep protecting yourself (and others)! 🙏🏻😷

I estimate you're at 1-15% risk of developing long-term disability, at the population's average exposure rate

Successive organ and immune damage from infection is worrisome

Limit exposure, detect early and try to treat early

—--

Today, mask mandates for public transport are being dropped in The Netherlands, in line with Western countries of just letting the virus spread.

This is based on a lot of misconceptions, which I hope to correct below:

"Omicron is mild, you can see this in the hospitalisation data"

The hospitalisation data skews the picture , as more people are being treated with oxygen at home (see picture). And just like many other viruses (polio, HIV), the big concern isn't about acute death, but long-term disability.
https://twitter.com/Datagraver/status/1504424119273771011?s=20&t=QVKEResRBR_VpcEbR9cEZw

"You can't have a lockdown forever"

There are many ways to reduce case amounts and severity. Wearing the proper masks (at least N95) indoors, ventilation, antivirals, long-lasting vaccine protection, continued testing. Proper long COVID care.

"I'm going to have covid anyway, it doesn't matter when I get it"

Every time you get covid your immune system seems to get weaker, not stronger/better attuned. You do not develop long-term immunity from infection.
https://science.thewire.in/health/omicron-t-cells-science-why-update-covid-19-vaccines/

With every reinfection, severity increases (on average). This could also translate into an increase in risk of long covid.
https://twitter.com/SiebeRozendal/status/1506361659396403205?s=20&t=QVKEResRBR_VpcEbR9cEZw

The goal has simply shifted from 'don't get covid' to 'get COVID as few times as possible'.

"The risk of long COVID is low, especially when vaccinated/boosted"

Conservative estimates put the risk at 2-3% of long ...
Primitive Perspectives and Sleeping Beauty
by dadadarren — 2022-03-26T01:55:39.460Z

This is another attempt to promote my solution to anthropic paradoxes (perspective-based reasoning, PBR).

I propose the first-person perspective shall be considered a primitive axiomatic fact. "I naturally know I am this particular person, yet there is no underlying reason for why it is so. I just am." Taking the first-person perspective as a given, recognizing there is no rational way to analyze it would solve anthropic paradoxes and more.

This is in stark contrast to the conventional approach: considering it as an Observation Selection Effect (OSE), treating the first-person perspective as a random sample like SSA or SIA does. I discussed the main differences in a previous post. Here I will explain how it answers problems like sleeping beauty.

The Fission Problem With a Toss

Imagine during tonight's sleep, an advanced alien would toss a fair coin. If Tails it would split you into 2 halves right through the middle. He will then complete each part by accurately cloning the missing half onto it. By the end, there will be two copies of you with memories preserved, indiscernible to human cognition. If Heads, nothing happens and you would wake up just as usual. After waking up from this experiment, and not knowing if you have been split, how should you reason about the probability that "yesterday's coin landed Heads?"

(For easy communication, let's call the split copy with the same left-half body as yesterday L and the copy with the same right-half body R. )

The experiment is set up so that there are 2 epistemically similar observers in the case of Tails, while only 1 if Heads. This can also be achieved by straight cloning without going through the splitting processes. I choose to present the experiment this way for consistency as I have used a similar thought experiment in the previous post.

PBR's Answer

The answer is 1/2 because there is no new information waking up the next day. But that is nothing new. What I want to point out is the probability of 1/2 can be verified by a frequentist model.

Picture yourself participating. After waking up the second day, it doesn't matter if I am the original person or L or R, I can take part in another iteration of the same experiment. After waking up from the second experiment I can do it again, and so on. Since it is a fair coin, as the iteration increases the relative frequency of Heads I experienced would approach 1/2. The...
[ASoT] Observations about ELK
by leogao — 2022-03-26T00:42:20.540Z

This document outlines some of my current thinking about ELK, in the form of a series of observations I have made that inform my thinking about ELK.

Editor’s note: I’m experimenting with having a lower quality threshold for just posting things even while I’m still confused and unconfident about my conclusions, but with this disclaimer at the top. Thanks to AI_WAIFU and Peter Barnett for discussions.

One way we can think of ELK is we have some set of all possible reporters, each of which takes in the latent states of a world model and outputs, for the sake of concreteness, the answer to some particular fixed question. So essentially we can think of a reporter+model pair as assigning some answer to every point in the action space. Specifically, collapsing possible sequences of actions into just a set of possible actions and only worrying about one photo doesn’t result in any loss of generality but makes things easier to talk about.
We pick some prior over the set of possible reporters. We can collect training data which looks like pairs (action, answer). This can, however, only cover a small part of the action space, specifically limited by how well we can “do science.”
This prior has to depend on the world model. If it didn’t, then you could have two different world models with the same behavior on the set where we can do science, but where one of the models understands what the actual value of the latent variable is, and one is only able to predict that the diamond will still appear there but can’t tell whether the diamond is still real. A direct translator for the second will be a human simulator for the first.
More generally, we don’t really have guarantees that the world model will actually understand what’s going on, and so it might genuinely believe things about the latent variable that are wrong. Like, we might have models which think exactly like a human and so they don't understand anything a human wouldn’t understand, so the direct translator for this model would also simultaneously be a human simulator, because the model literally believes exactly what a human would believe.
There are more than just human simulators and direct translators that are consistent with the training data; there are a huge number of ways the remaining data can be labeled. This basically breaks any proposal that starts with penalizing reporters that look like a human

...
Lesson Plan: Biases in Quantity Estimation
by abstractapplic — 2022-03-26T00:23:17.822Z

Pedagogic status: I’ve never executed this exact lesson plan, but I ran a workshop similar to this for a dozen friendly but non-technical adults a few years ago, and it seemed to go well. The content won’t be new to the average LessWronger, but I do present some novel ways to teach, practice, and think about these concepts.

Good afternoon, students. Today, we’ll be talking about biases relevant to forecasting, and some ways you can get around them.

Well, we’ll be talking. There are some weird voyeurs watching us from the past and/or future, for whom this is more of a reading/writing kind of deal: try to ignore their eerie gazes.

We’re going to be using a quiz to evaluate how over- or under-confident you are. This is a little different to the quizzes you’re used to, so I’ll take you through an example question before handing them out.

I could break the ice and keep up engagement by asking one of you to supply the question, but there’s a chicken-and-egg problem: I can’t easily-reliably-quickly get you to give me a question in the right shape without sharing an example first. So annoyingly, for most audiences, it’s probably best to just use a canned example.

The example question looks like this:

I’m 80% confident there are at least ____ species of Pokemon.

I’m 95% confident there are at least ____ species of Pokemon.

I’m 99% confident there are at least ____ species of Pokemon.

(By the way, the word ‘confident’ is being used here in a technical sense: if you’re something% confident, it means that you think there’s a something% chance your guess is right. In other words, if you made a hundred 99% confidence guesses, you’d expect – on average – that about one of them would turn out false.)

Why am I using these kinds of questions, instead of doing the typical ‘estimate-the-probability-of-this-binary-outcome’ thing? Partly it’s because I suspect that approach trains subtly flawed skills, in ways that make my mental model of Taleb recoil in horror: the Credence Calibration Test punishes you for assigning 99% weight to guesses, while in the real world there are plenty of situations where you need that level of certainty. But mostly it’s because I don’t expect a maybe-nontechnical audience to understand Brier Scores, but I do expect them to understand “if you got 20% of your 99%-confidence predictions wrong (or 100% of your 80%-confidence predictions right) you messed up somewh

...
Why Agent Foundations? An Overly Abstract Explanation
by johnswentworth — 2022-03-25T23:17:10.324Z

Let’s say you’re relatively new to the field of AI alignment. You notice a certain cluster of people in the field who claim that no substantive progress is likely to be made on alignment without first solving various foundational questions of agency. These sound like a bunch of weird pseudophilosophical questions, like “

what does it mean for some chunk of the world to do optimization?

”, or “

how does an agent model a world bigger than itself?

”, or “

how do we ‘point’ at things?

”, or in my case “

how does abstraction work?

”. You feel confused about why otherwise-smart-seeming people expect these weird pseudophilosophical questions to be unavoidable for engineering aligned AI. You go look for an explainer, but all you find is bits and pieces of worldview scattered across many posts, plus

one post which does address the question but does so entirely in metaphor

. Nobody seems to have written a straightforward explanation for why foundational questions of agency must be solved in order to significantly move the needle on alignment.

This post is an attempt to fill that gap. In my judgment, it mostly fails; it explains the abstract reasons for foundational agency research, but in order to convey the intuitions, it would need to instead follow the many paths by which researchers actually arrive at foundational questions of agency. But a better post won’t be ready for a while, and maybe this one will prove useful in the meantime.

Note that this post is not an attempt to address people who already have strong opinions that foundational questions of agency don't need to be answered for alignment; it's just intended as an explanation for those who don't understand what's going on.

Starting Point: The Obvious Stupid Idea

Let’s start from the obvious stupid idea for how to produce an aligned AI: have humans label policies/plans/actions/outcomes as good or bad, and then train an AI to optimize for the good things and avoid the bad things. (This is intentionally general enough to cover a broad range of setups; if you want something more specific, picture RL from human feedback.)

Assuming that this strategy could be efficiently implemented at scale, why would it not produce an aligned AI?

I see two main classes of problems:

In cases where humans label bad things as “good”, the trained system will also be selected to label bad things as “good”. In other words, the trained AI will

...
My mistake about the war in Ukraine
by Ege Erdil — 2022-03-25T23:04:25.281Z

One month ago, I wrote a blog post in which I made the following claim:

Ukraine's ability to resist a Russian invasion by itself is even less than Poland's ability to resist a German invasion in 1939, and I don't think anyone doubted this.

I'm leaving this claim up in the original post for the record, but here I want to think about what exactly went wrong in my model of the war. I think the broader point I was making in the linked blog post is accurate, but I was overconfident here in a way I should not have been and this is a retrospective of me trying to explain why that happened.

When I wrote this blog post, I was fairly confident (over 80%) that the war would end up going well for Russia. It hasn't, and I'm quite surprised by how poorly the Russians have ended up performing. Looking at some past wars through this lens, I think wars going in unexpected ways is actually normal throughout history, and I should have been a lot less confident in my evaluation of how the war would turn out.

I think the main source of error in my model here was that I didn't take into account the degree to which Germany mobilized against Poland. Germany invaded Poland with an army of close to 2 million soldiers against Poland's initial defense force of 250 thousand. Poland later got this number up to around 800 thousand by mobilization, but the Germans still had a 2:1 numerical advantage.

In contrast, Russia has tried to invade Ukraine with an invasion force of around 200k soldiers, and they are likely outnumbered by the defenders now by at least 2:1. Defending is easier than attacking in war due to many reasons, so the size of the Russian invasion force should have led me to be more skeptical about this comparison with Poland.

Another reference class I had in mind was the 2003 US invasion of Iraq. Here the Iraqi defenders outnumbered the coalition forces 2:1 but were handily defeated anyway. I think as a comparison case it was definitely the right decision to keep this in mind, but I didn't properly take into account the degree to which war is an uncertain affair and so I updated too strongly on these few examples without appropriately weighing the counterexamples.

For instance, in 1940 Germany invaded France with an invasion force roughly of the same size as the one that they used in Operation Barbarossa. France fell in six weeks while the USSR eventually went on to mobilize mor...
An outline of an ironic LessWrong post
by Yitz — 2022-03-25T22:51:37.818Z

Introduction: A character is considering whether or not to join the community of LessWrong, an online rationality-focused group.

The pros and cons of joining LessWrong: On the one hand, the character will gain access to a wealth of knowledge and resources on how to think more rationally. On the other hand, the character will likely be bombarded with a never-ending stream of articles, posts, and comments that are all written with a level of irony that leaves the character questioning their own sanity.

The decision: In the end, the character decides that the benefits of joining LessWrong outweigh the costs, and joins despite knowing full well that they'll probably be driven insane by all the irony eventually. After all, if the character is already questioning their sanity, there's no point in not joining LessWrong and seeing where that leads.

The twist ending: The character is so consumed by irony that they end up making a post about their experience on LessWrong.

4.5 The twist to the twist: The character's post was actually an ironic commentary on ironic commentary.
Is Metaculus Slow to Update?
by SimonM — 2022-03-25T19:44:33.624Z

h/t Eric Neyman for causing me to look into this again

On a recent Mantic Monday, Scott Alexander said:

This is almost monotonically decreasing. Every day it’s lower than the day before.

How suspicious should we be of this? If there were a stock that decreased every day for twenty days, we’d be surprised that investors were constantly overestimating it. At some point on day 10, someone should think “looks like this keeps declining, maybe I should short it”, and that would halt its decline. In efficient markets, there should never be predictable patterns! So what’s going on here?

Maybe it’s a technical issue with Metaculus? Suppose that at the beginning of the war, people thought there was an 80% chance of occupation. Lots of people predicted 80%. Then events immediately showed the real probability was more like 10%. Each day a couple more people showed up and predicted 10%, which gradually moved the average of all predictions (old and new) down. You can see a description of their updating function here - it seems slightly savvier than the toy version I just described, but not savvy enough to avoid the problem entirely.

Personally, this has never particularly bothered me. Having watched the odds for many things which behave like this. (Pick any sports game where one side has a large, but not unassailable lead and you'll see this pattern).

That said, I'm also sympathetic to the view that Metaculus forecasts aren't perfect. Whenever I think about how my own forecasts are made, I'm definitely slow to update, especially if it's something I don't follow very often. If a question gets lots of interest and catapults to the front page, I'm liable to update then, and usually it's going to be in the direction of the crowd. Is this enough to make the forecasts predictable? (Which would be bad, as Scott says!)

One metric to look at when deciding if forecasts are predictable is to check whether or not the change in forecasts correlated from day to day. (ie if our forecasts increased 1% yesterday, are they more likely to increase tomorrow or not?).

Everything which follows is based on the community prediction (median forecast) which is visible to the public at all times.

Looking across ~1000 binary questions on Metaculus, we actually see the opposite of the "momentum" that Scott talks about. In general, if a question increased 1% yesterday, we should expect it to fall today.

W...