trentmkelly/LessWrong-43k huggingface dataset:
The possibility of no good amazing forecasters
by Johannes C. Mayer — 2022-01-03T12:57:59.362Z

When somebody correctly predicts an event, how much more should we trust this person? What if they make two correct predictions in a row?

Counter-intuitively, to answer this question, we need to know how many people actually were making predictions about these events, and how the predictions are typically distributed. Very often when people try to predict some event, especially when they try to predict a specific date that is many years away, they come up with wildly different estimates. Typically some will be far too high, while others will be far too low. However, if we assume that the forecasters have a non-zero probability mass of landing at any point between the too high, and too low predictions, then the more forecasters there are, the higher the chance that one will be roughly correct.

Note that this is true especially when some subset of the forecasters have just the strategy of choosing randomly. But of course, if someone gets correct predictions because of luck, we can't extrapolate future forecasting success from that.

For example, there are people that think that AI is just around the corner, while others literally say that it is impossible to achieve, and there are lots of people who say something in between. Even in the world where everyone basically has no clue, and all their reasons for arriving at a particular prediction actually don't make any sense, some of them are probably roughly correct.

Note that all of this directly generalizes also to sets of predictions.

However, in practice, this does not seem to be too big of an issue right now, because you can filter based on if somebody presents good arguments for their position that make it seem like they actually have a good model of the relevant dynamics. And after filtering there will not be very many people left.

However, you need to be careful that you only filter based on what somebody said before the event occurred. Otherwise, they might succumb to hindsight bias, and start to use new knowledge that was not apparent when they made their prediction to justify why they came up with their prediction. This can actually already happen each time the outcome of the event becomes more obvious, as the event comes closer and closer.

If you evaluate past events, you also have to keep in mind that all the information that you find will be heavily skewed in favor of the people that actually got predictions...
How an alien theory of mind might be unlearnable
by Stuart_Armstrong — 2022-01-03T11:16:20.637Z

EDIT: This is a post about an alien mind being unlearnable in practice. As a reminder, theory of mind is unlearnable in theory, as stated here - there is more information in "preferences + (ir)rationality" than there is in "behaviour", "policy", or even "complete internal brain structure". This information gap must be covered by assumptions (or "labelled data", in CS terms) of one form or another - assumptions that cannot be deduced from observation. It is unclear whether we need only a few trivial assumptions or a lot of detailed and subtle ones. Hence posts like this one, looking at the practicality angle.

Alice learns the ways of aliens

I suggested that an alien "theory of mind" might be unlearnable; Rohin Shah challenged this conclusion, asking whether a theory of mind was truly unlearnable, even for a very intelligent Alice. Let's dig into this concept for a bit.

There is, of course, a weak and a strong version of the unlearnability hypothesis. The strong version is that Alice, even with infinite time and total rationality, couldn't learn an alien theory of mind. The weaker version is that a smart and motivated Alice with a lot of resources and data, couldn't learn an alien theory of mind in reasonable time.

You can nuance both of those by wondering how much of the theory of mind is unlearnable. It doesn't really matter if a few less important bits are unlearnable. So the real question is, how hard is it to learn enough alien theory of mind, with enough data and effort? We might also ask whether the learning process is interactive or non-interactive: does Alice merely observe the alien natives, or is there a conversation going where the aliens try and correct her interpretations?

aAlice learns the ways of humans

Unfortunately, we don't have a convenient alien civilization on hand to test this (and, even if we did, we might be unsure whether we'd really understood their theory of mind, or just thought that we did). So instead, let's imagine an alien Alice - aAlice - who is trying to learn the human theory of mind, and see how she might go astray.

It won't take long for aAlice to realise that there is a difference between what humans say publicly, and what we say privately. Also, there is a difference between what we say under the impact of strong emotion, and what we say when calm and relaxed.

She concludes, naturally (as this is close to how her species beh...
Reader-generated Essays
by Henrik Karlsson — 2022-01-03T08:56:20.903Z
Link: https://escapingflatland.substack.com/p/reader-generated-essays

1.

An essay is made up of two layers: the ideas, and the language in which they are expressed. The story of Red Riding Hood, and the words of a particular retelling of that story. The idea of ​​a fully programmable blockchain, and the language that Vitalik Buterin used to carry that idea in the Etheruem White Paper.

Thinking and communicating are two separate processes, even if they often happen at the same time. And both make great demands on your mental faculties: often I do not have enough capacity to cope with writing and thinking at the same time; the thoughts suffer when I wrestle with the language, and vice versa. If I am exploring a complex set of ideas, it is, therefore, often necessary to give structure to the thoughts first – in the form of notes and diagrams – before turning them into an essay.

I formulate the ideas. And then I communicate them.

2.

GPT-3, announced in May 2020 by OpenAI, is a neural net modeling of natural language and natural-language-related tasks. It can, with the right prompt, produce sentences that stand up well when compared to sentences formulated by humans. When it comes to ideas, though, it is less convincing – it lacks, at the current scale, the ability to develop longer chains of ideas. Here, for example, is its output when Gwern prompted it to write Harry Potter in the voice of Hemingway:

It was a cold day on Privet Drive. A child cried. Harry felt nothing. He was dryer than dust. He had been silent too long. He had not felt love. He had scarcely felt hate. Yet the Dementor’s Kiss killed nothing. Death didn’t leave him less dead than he had been a second before. It wasn’t about living or dying really. It wasn’t about death. It was about the Dark Lord, his creator, his engenderer. He was not resurrected. He was created again. He was whole again in the courtyard of Malfoy Manor.

Line by line, it is more alive than many of the poets I had to read when I was asked to edit a poetry anthology a few years back. But as a whole – as a narrative – the piece fundamentally collapses.

In other words, GPT-3 performs well only when it comes to one level of writing: formulating sentences. It can convincingly expand a prompt with new sentences – sentences more pleasant to read than most human-generated prose. But GPT-3 does not hold up when it comes to the other level of writing: it cannot convincingly generate and structure ideas...
Open Thread - Jan 2022 [Vote Experiment!]
by jimrandomh — 2022-01-03T01:07:24.172Z

Experiment: Reaction-Ballot Voting

This open thread is using an new experimental voting system: reaction-ballot voting.

In addition to voting on a comment's overall quality, you can also vote separately on a number of axes, and apply a small set of emoji reactions. Try out and discuss this voting system here! Notes:

Voting on each axis is weighted according to the voter's karma the same way as votes are in the regular voting system. All axes can be strong-voted with click-and-hold. The commenter's karma is only affected by your vote on the "overall" access, not your vote on any of the other axes.
This is one experiment among several. Bugs are possible. We're interested in what effect this has on the quality of conversation, what the experience of voting in this system is like, and what the experience of skimming a thread and seeing these scores is like.
The user interface for this voting system doesn't work well on touch/mobile devices. Third-party clients such as GreaterWrong will only be able to see and cast overall votes, but should work fine otherwise.

Please tell us what you think! Love it/hate it/think it should be different? Let us know.

Regular Open Thread Boilerplate

If it’s worth saying, but not worth its own post, here's a place to put it.

If you are new to LessWrong, here's the place to introduce yourself. Personal stories, anecdotes, or just general comments on how you found us and what you hope to get from the site and community are invited. This is also the place to discuss feature requests and other ideas you have for the site, if you don't want to write a full top-level post.

If you want to explore the community more, I recommend reading the Library, checking recent Curated posts, seeing if there are any meetups in your area, and checking out the Getting Started section of the LessWrong FAQ. If you want to orient to the content on the site, you can also check out the new Concepts section.

The Open Thread tag is here. The Open Thread sequence is here.
Lateral Thinking (AI safety HPMOR fanfic)
by SlytherinsMonster — 2022-01-02T23:50:21.224Z

(

Source material

)

"Now leave your books and loose items at your desks – they will be safe, the screens will watch over them for you – and come down onto this platform. It's time to play a game called Who's the Most Promising Student in the Classroom."

"It might seem that our game is done," said Professor Quirrell. "And yet there is a single student in this classroom who is more promising than the scion of Malfoy."

And now for some reason there seemed to be an awful lot of people looking at...

"Harry Potter. Come forth."

This did not bode well.

Harry reluctantly walked towards where Professor Quirrell stood on his raised dais, still leaning slightly against his teacher's desk.

The nervousness of being put into the spotlight seemed to be sharpening Harry's wits as he approached the dais, and his mind was ruffling through possibilities for what Professor Quirrell might think could demonstrate Harry's promise as an AI safety researcher. Would he be asked to write an algorithm? To align an unfriendly AI?

Demonstrate his supposed immunity to superintelligent optimization? Surely Professor Quirrell was too smart for that...

Harry stopped well short of the dais, and Professor Quirrell didn't ask him to come any closer.

"The irony is," said Professor Quirrell, "you all looked at the right person for entirely the wrong reasons. You are thinking," Professor Quirrell's lips twisted, "that Harry Potter has defeated the First AI, and so must be very promising. Bah. He was one year old. Whatever quirk of fate killed the First AI likely had little to do with Mr. Potter's abilities as a researcher. But after I heard rumors of one Ravenclaw debating five older Slytherins, I interviewed several eyewitnesses and came to the conclusion that Harry Potter would be my most promising student."

A jolt of adrenaline poured into Harry's system, making him stand up straighter. He didn't know what conclusion Professor Quirrell had come to, but that couldn't be good.

"Ah, Professor Quirrell –" Harry started to say.

Professor Quirrell looked amused. "You're thinking that I've come up with a wrong answer, aren't you, Mr. Potter? You will learn to expect better of me." Professor Quirrell straightened from where he had leaned on the desk. "Mr. Potter, much research aims to improve AI theory of mind, and in due course it will likely succeed. Give me ten novel ways in which an AI might use its resulting und...
Can you learn (music) efficiently by grinding?
by Nicole Dieker — 2022-01-02T19:07:41.279Z

Crossposted from my Substack.

You don't need to watch the videos to understand what I learned today; you can grok it from reading the text.

I'd like to make one more post in this series, to determine whether a person can use this grinding strategy to correctly estimate the time it will take to learn a piece of music. That said, I'm jumping the gun a little bit and need to spend at least the rest of this week ensuring that the grinding strategy works as well as it appears to work after today's experiment.

So... that'll be all from me on this topic until I'm ready to share those results.

Yesterday, I began a series of videos on

how to learn music more efficiently

.

Today, I ran an hour-long experiment on the relationship between grinding (in the video-game sense, translated in this case as “repeating a 12-measure sequence 20 times in a row”) and learning.

Basically, I was asking myself whether I was confused about what deliberate practice meant. Somewhere in all of this I got stuck on the idea that to practice deliberately, you had to deliberate between each repetition. You had to ask yourself “how did that last pass go, and what can I do differently on this next pass?”

This time, I asked myself what would happen if I simply ran the pass over and over, twenty times in a row.

This is both “how I used to practice” and “completely different from how I used to practice.” As I note in the video, I learned a lot of piano repertoire by grinding my way through it — but as a younger and less disciplined pianist, I doubt I ever committed to grinding a single passage 20 times in a row. I gave up as soon as the grinding got repetitive and/or I got distracted, which meant that I never got to the point where I knew the passage well enough for the repetitions to become interesting again.

Younger!Nicole (and I include 2021!Nicole in that designation) also left out the key step that I discovered in yesterday’s video and re-integrated at the beginning of today’s video: before you begin the 20 repetitions, make sure you have the skills/understanding required to play the passage perfectly. It’s no good to grind until you know what you’re grinding, after all.

I think this is the most important thing I’ve learned about learning this year. 😉
Tag for AI alignment?
by Alex_Altair — 2022-01-02T18:55:45.228Z

Why is there no tag for AI alignment or AI safety? Is some other term used?

I see a tag page for AI, which lists tons of really specific sub-tags. I notice that a lot of the sub-tags are about alignment topics; maybe the "AI" tag is effectively the AI alignment tag, because LW cares about that so much that we effectively never talk about AI not in the context of safety?
Each reference class has its own end
by avturchin — 2022-01-02T15:59:17.758Z

It is often claimed that the Self-Sampling assumption (SSA) is problematic, in particular, because we don’t know which reference class is a correct one. Therefore, the notorious Doomsday argument is weak. Here I will try to show that it is less a problem than seems.

Central claim: Each reference class has its own end.

Example: I am put randomly in a 2-dimensional grid of rooms. Vertically they are designated by letters from A to Z, and horizontally they are numbered from 1 to 100. So, each room has an address like E021. The Doomsday-like claim is that I am more likely to be somewhere in the middle of the grid than on its borders and thus, I am unlikely to be in the raw of 1-rooms and in A-column. Therefore, claims DA, if I learn my room number, I can estimate the total number of rooms in my raw, and the same is true for letters. (UPDATE: by "middle" I mean some region not on the borders, not exactly cell N50, but, say, between 20 and 80.)

A real-world example: I assume that my height and my date of birth are two independent variables. Thus, both should be in the middle of the distribution: I am unlikely to be born on January, 1, and unlikely to have a very low height. It is actually true for me: my date of birth and my height are in the middle of the distribution.

Here we use two important assumptions:

The independence of the variables, like height and date of birth.
Each variable is external to my cognition process, so it doesn’t affect my probability to think about anthropics.

Now we could try to apply this to real-world Doomsday-like problems, that is, claims like “I am in the middle of my reference class existence in time”.

Let’s discuss three of my reference classes and the corresponding predictions about the end of the world:

I am a member of the class of mammals, which existed for last 200 million years, and thus mammals may exist a few hundreds of millions of years more.
I am Russian, and Russians have existed for around last 1000, so they can exist another millennium.
LessWrong has existed for 13 years and thus will likely continue to exist 13 more years with 50 per cent probability.

These predictions give different timing of the “end of the world”. This type of inconsistency is often used to show that there is something wrong with the Doomsday argument as it depends on the choice of the reference class.

However, if we ignore the idea that there shou...
Scott Alexander 2021 Predictions: Market Prices - Resolution
by SimonM — 2022-01-02T11:55:47.553Z

Last year, I looked at Scott's forecasts for 2021 and compared them to the market forecasts. Today I went through those forecasts (and Zvi's * - a buy/hold/sell exercise done on Scott's estimates) added the resolutions and calculated a Brier score and a log-score.

Results were as follows:

Brier	Log
Scott	0.20	1.24
Zvi	0.16	0.93
Market	0.14	0.90

So in summary "market" about as good as Zvi and both better than Scott . (Albeit on a pretty small sample of 19 questions). (Lower is better for Brier score and log-score)

Full details can be found here

Question	Scott	Zvi	Market	Result
Biden approval rating (as per 538) is greater than 50%	80%	80%	61%	0
Court packing is clearly going to happen (new justices don’t have to be appointed by end of year)	5%	1%	5%	0
Yang is New York mayor	80%	70%	70%	0
Newsom recalled as CA governor	5%	5%	7%	0
Tokyo Olympics happen on schedule	70%	80%	77%	1
Major flare-up (significantly worse than anything in past 5 years) in Russia/Ukraine war	32%	15%	16%	0
Netanyahu is still Israeli PM	40%	25%	22%	0
Prospera has at least 1000 residents	30%	30%	18%	0
GME >$100 (Currently $170)	50%	50%	60%	1
Bitcoin above 100K	40%	23%	23%	0
Ethereum above 5K	50%	30%	11%	0
Ethereum above 0.05 BTC	70%	55%	33%	1
Dow above 35K	90%	50%	50%	1
…above 37.5K:	70%	20%	20%	0
Unemployment above 5%	40%	50%	37%	0
Starship reaches orbit	60%	60%	50%	0
Greater than 66% of US population vaccinated against COVID	50%	60%	77%	1
Vitamin D is generally recognized (eg NICE, UpToDate) as effective COVID treatment	30%	20%	25%	0
US approves AstraZeneca vaccine	20%	20%	37%	0

* I made a couple of assumptions when calculating Zvi's probabilities for things where he wasn't super explicit about his numbers. I will of course update these if asked.
What explains the excess mortality?
by ChristianKl — 2022-01-02T10:13:37.033Z

We were talking about the point of our COVID-19 policies being about preventing deaths. We now see from OurWorldInData that different countries had very different results in regards to excess deaths. While Sweden has had more deaths than its Scandinavian neighbors of Finland and Norway, it has less than half of the excess deaths of the United Kingdom and less than a third of that of the United States. If we truly care about the deaths, finding out why that's the case and what's driven the high excess deaths in some countries looks to me like an important project.

Does anyone have a good graph of why the status quo looks the way it does?
Covid risk of staying at a hotel
by NormanPerlmutter — 2022-01-02T06:47:35.586Z

Does anybody have a good sense of how to estimate the covid risk accrued from staying one night at a hotel, as compared with a baseline risk of living in a house? My gut instinct is that this risk is fairly minimal, but I haven't seen any good data on it. I'm interested primarily in the risk of occupying a hotel room which could have virus particles from previous guests or transmitted through the HVAC system from guests in other rooms. Transit through common areas can be kept brief and risk in common areas mitigated further by wearing an N95 or similar.
What is a Counterfactual: An Elementary Introduction to the Causal Hierarchy
by Darmani — 2022-01-02T03:46:16.502Z

This is adapted from a segment of a talk I gave last year, primarily the portion at 6:09:00-6:11:00 of this stream. The example in this article originates from my paper "A Language for Counterfactual Generative Models," specifically Appendix A, which also features code for this example. I originally learned the causal hierarchy from Chapter 7 of Pearl's book "Causality."

This post is written for Chris Leong's bounty for an explanation of counterfactuals.

Let's play a game.

In this game, I'm going to roll a die, and you're going to try to predict the roll.

We'll compare the numbers.

If your guess is within 1 of the true number, you win.

Otherwise, you lose.

This is a game of chance and decision. I can pose many questions about what would happen, and they'll be easy to analyze. And that makes it easier to analyze the questions themselves. It turns out that some questions are easy and could be solved by a smart bean-counter with no understanding of the game rules, while others require understanding things that may never occur naturally. These question types form the causal hierarchy, culminating in "what would have happened" questions, or counterfactuals. At the end of this article, you'll be able to explain exactly what is a counterfactual and how to compute them. You'll understand why they are hard to answer, even though they are conceptually as straightforward to compute as the probability formulas you learned in school.

Say I watch old Mary playing this game at the casino, over and over. Every round, I see her placing the same bet on her lucky number: 6. I can make predictions: if she plays long enough, the proportion of times she will win will be very close to 1⁄3. This is prediction.

We can refine this a little too. Say I'm no longer watching her across the room, but I'm instead looking at recordings of the game taken on a high-speed camera, playing them back frame by frame. As the die moves closer to a standstill, I can make better and better predictions about whether Mary will win. Or I can make abductions about where the die was a moment earlier. Each frame I see is an observation. And I can do all of this knowing nothing about the world other than having watched the game over and over. We call this Level 1 of the causal hierarchy.

But maybe I can do more than just watch. I can also intervene. Suppose one day I get tired of watching Mary lose, ...