trentmkelly/LessWrong-43k huggingface dataset:
Google Search as a Washed Up Service Dog: "I HALP!"
by Shmi — 2022-11-07T07:02:40.469Z

As is customary, I whined (on IRC) about Google Search being unhelpful or actively counterproductive with the results these days and how it is not what it used to be. Specifically, I wanted to know something very niche, why many LTE modems sometimes take too long to disconnect. The hits were actively unhelpful, like "why does my internet keep disconnecting?" and there was NOT A SINGLE RELEVANT ENTRY in the first 10 pages. (And yes, I know the information is out there, you just can't form a good query to find it.) Here is the best response by far (emphasis mine):

<quanticle> It reminds me of my friend's dog. This dog was a candidate to be a guide/helper dog for disabled people but, for whatever reason, just couldn't pass the final test of guide dog academy. So, it had like... 90% of the training and instincts of a guide dog, but didn't have anyone to actually guide. So it'd do some cute-but-in-retrospect-neurotic things like, if it noticed my friend cooking, it'd bring her a screwdriver? Why? Because, look, you're doing a thing, and here's a thing! I HALP
<quanticle> Google is like that dog
[Book Review] "Station Eleven" by Emily St. John Mandel
by lsusr — 2022-11-07T05:56:19.994Z

TL;DR

Is it good? Yes.

Should you read it? I don't know.

Spoilers Ahead

Station Eleven isn't the kind of book I'd normally read but I wanted to go to a book club and Station Eleven is what they were reading so I bought a copy and quickly read it over the weekend—just in time for the Tuesday MeetUp.

Station Eleven is an interweaving of several stories which take place before and after a pandemic[1] destroys civilization. The stories before the apocalypse follow actor Arthur Leander and his first wife Miranda who is drawing a comic book series Dr. Eleven. The stories after the apocalypse follow Kirsten, an actress in a nomadic band of actors and musicians who call themselves the Travelling Symphany. Kirsten likes reading Dr. Eleven and old tabloids about Arthur Leander.

It wasn't until I was halfway through that book that I started to like it and it wasn't until after finishing the book that I figured out what it was about.

The most important scene in the book is a nighttime party with a bunch of Hollywood bigwigs. Arthur is flirting with his soon-to-be second wife. Miranda is unable to connect with her husband's friends so she goes outside to play with the dog instead.

Miranda has no friends of her own. She does well in business, eventually becoming an executive at a major shipping company. Her sole passion is drawing Dr. Eleven which she works on her entire life. She never publishes it commercially. She just does a tiny self-published printing, from which she gives two copies to her by-that-time-ex-husband Arthur. Arthur doesn't really get Dr. Eleven so he gives them away.

Arthur has friends but they're all superficial. There's nobody he cares about who cares about him. Even though he's famous.

Kirsten does have friends: all the members of the Travelling Symphany. Station Eleven is ostensibly about how everyone was connected in civilization and how that goes away after the apocalypse. But all the personal connections of the characters before the apocalypse were transient and superficial. Their relationships after the apocalypse aren't. The author Emily Mandel doesn't romanticize the apocalypse. Life after the apocalypse is nasty, brutish and short. But it's a nice juxtoposition.

Why is this book good?

The scenes with the Travelling Symphany are exciting adventure, of course.

But what really got to me was Miranda's loneliness. Her closest connection is to Kirsten, a gi...
Counterfactability
by Scott Garrabrant — 2022-11-07T05:39:05.668Z

This post will assume an understanding of the finite factored set ontology. It will be more speculative that the main FFS sequence, and I will leave out some proofs. It seems likely that I will later regret some of the definitions laid out here. I will be particularly sloppy in defining evidential and causal counterfactuals, because this post is not advocating for them, only demonstrating that it is possible to represent them within our ontology. This post was (approximately) written a year ago. I am currently in the process of moving a bunch of my ideas from the last year to LessWrong.

Nontechnical Summary

The main thing in this post is introduce a new concept of counterfactability. A counterfactable event E is one that screens off its own history (i.e. everything usptream of E) from everything that you care about.

Decision theory is easy when considering choices that represent counterfactable events. When trying to take counterfactuals on non-counterfactable events, things are under-defined. This is because non-counterfactable events have artificially low resolution. It is like asking what would happen if I either took action X or took action Y. The question does not carve reality at the joints. Different worlds were artificially merged. It is as though details about the event were forgotten.

We can reframe questions in decision theory as follows: When given a non-counterfactable event E, how do we add some more details to E to form a counterfactable E′, while only adding details that feel like they were artificially forgotten? In this new frame, we are settled on the question of how to take counterfactuals on counterfactable things, and are only asking "What is the counterfactable thing we were meant to be counteracting on?"

Both CDT and EDT can be reframed as providing an answer to this question. They are giving the wrong answers, but this fact shows that the reframe is sufficient to capture both CDT and EDT. Further, we can see that when choosing between counterfactable choices, CDT and EDT give the same answer. I believe the correct way to find the natural events to counteract on requires introspection on the gears of the agent's cognition, and will not be either of the CDT and EDT extremes.

Finally, I show that this concept of counterfactability is sufficiently natural that it applies outside the conte...
2022 LessWrong Census?
by SurfingOrca — 2022-11-07T05:16:33.207Z

From 2011-2017, there was an annual LessWrong census/survey. Much like a national census, this provided a valuable lens into the demographics and beliefs of LessWrongers. Unfortunately, this tradition appears to have stopped in recent years, with the exception of a mini-revival in 2020. (Scott Alexander appears to have moved the census to SlateStarCodex.)

From what I've read, this is mainly due to of a lack of will/time among those in the community to run this project, and not a general judgement against the census.

If this is the case, I'd like to start a new version of the census this year, with a greater emphasis on alignment research/beliefs about AI and timelines.

Is this a good idea?
A philosopher's critique of RLHF
by TW123 — 2022-11-07T02:42:51.234Z

In the spring, I went to a talk with Brian Christian at Yale. He talked about his book, The Alignment Problem, and then there was an audience Q&A. There was a really remarkable question in that Q&A, which I have transcribed here. It came from the Yale philosophy professor L.A. Paul. I have since spoken to Professor Paul, and she has done some work on AI (and coauthored the paper “Effective Altruism and Transformative Experience”) but my general impression was that she hasn't yet spent a huge amount of time thinking about AI safety. Partly because of this question, I invited her to speak at the CAIS Philosophy Fellowship, which she will be doing in the spring.

The transcript below really doesn't do her question justice, so I'd recommend watching the recording, starting at 55 minutes.

During the talk, Brian Christian described reinforcement learning from human feedback (RLHF), specifically the original paper, where a model was trained with a reward signal generated by having humans rate which of two videos of a simulated robot was closer to a backflip. Paul's question is about this (punctuation added, obviously):

L.A. Paul: So, I found it very interesting, but I'm just not fully understanding the optimistic note you ended on...so, in that example, what was key was that the humans that did the "better" thing, knew what a backflip was. It was something they recognized. It was something they recognized so they could make a judgment. But the real issue for us is recognizing, or for machines is recognizing, entirely new kinds of events, like a pandemic, or a president that doesn't follow the rule of law, or something interesting called the internet, you know there's radically new technological advances. And when something like that happens, those rough judgments of "this is better than that"... In other words, those new things: first, we're terrible at describing them before they come and predicting them. (Although, humans are very good at a kind of one shot learning, so they can make judgments quite quickly. Machines are not like that).

L.A. Paul: Moreover, these better-than judgments that the machine might be relying on could I think quite straightforwardly be invalidated, because everything changes, or deep things change, in all kinds of unexpected ways. That just seems to be...that's the real problem. It's not... using machines for things that we alre

...
Is there any discussion on avoiding being Dutch-booked or otherwise taken advantage of one's bounded rationality by refusing to engage?
by Shmi — 2022-11-07T02:36:36.826Z

From https://www.gwern.net/mugging:

One way to try to escape a mugging is to unilaterally declare that all probabilities below a certain small probability will be treated as zero. With the right pair of lower limit and mugger’s credibility, the mugging will not take place. But such a ad hoc method violates common axioms of probability theory, and thus we can expect there to be repercussions.

It turns out to be easy to turn such a person into a money pump, if not by the mugging itself. Suppose your friend adopts this position, and he says specifically that any probabilities less than or equal to 1/20 are 0. You then suggest a game; the two of you will roll a d20 die, and if the die turns up 1-19, you will pay him one penny and if the die turns up 20, he pays you one dollar - no, one bazillion dollars. Your friend then calculates: there is a 19/20 chance that he will win a little money, and there is a 1/20 chance he will lose a lot of money - but wait, 1/20 is too small to matter! It is zero chance, by his rule. So, there is no way he can lose money on this game and he can only make money.

He is of course wrong, and on the 5th roll you walk away with everything he owns. (And you can do this as many times as you like.)

Of course, that's not how a sane street-rational person would think! They would not play for "one bazillion dollars" no matter the odds. In general, detecting a sufficiently intelligent adversarial entity tends to result in avoiding the interaction altogether (if you are inured enough to Nigerian princes offering billions in an email). And yet I cannot find any LW discussion on when and if to engage and when to not engage, except in an occasional comment.
Exams-Only Universities
by Mati_Roy — 2022-11-06T22:05:39.373Z

Quality: fast write-up (~45 minutes); little researched

I want a university that only does exams, which would include a description of what you need to know for the exams. Bonus would be suggestions for textbooks and online classes to learn the material, but that's optional.

Cost

The costs are:

Exam creation

Supervision

Scoring

Exam creation is a fixed cost per topic per round (you need to change the exam each year).

Supervision, if in person, is partly a fixed cost per location, and partly a variable cost. (Someone can supervise students doing exams on different topics.) Online supervision would also be an option ideally.

Scoring is partially a fixed cost (the part that can be automated) and partially a variable cost (the part that need to be reviewed manually).

Frequency

Frequency of exams could change with demand, but, to start with, you could have one cheap exam per topic per year, timed with the end of normal school years. This would be the exam most people would take, so it would spread the fixed cost among more people. This might also be a more valuable test because it could position you on a normal curve more precisely given the greater amount of people taking it. There could also be more expensive tests throughout the year—more expensive given the fixed cost would be spread among fewer people.

Problem it solves

What this solves:

Decouples learning and exams
a) You can learn at your own pace (whether that's more slowly of faster)
b) You can learn from wherever you want (maybe you want to learn from different places for different topics)

Creates standardized tests making it easier to compare the competency of students between different schools

Why wanting exams in the first place? Because many governments want them for immigration and many organizations want them for employees.

Questions I have

Has this been done?

Could this be done? Could you have a university that only does exams and emits degrees that are recognized by the US government?

Current state

There are universities that don't require you to attend classes, but I still find those non-ideal because it's:

Bias: In my experience, students that attend have an unfair advantage—Not because they learn more, but because teachers literally (and even intentionally) give unfair hints about their exams to reward students for showing up and justifying their salary.

Inconvenient:

a) You ...
Democracy Is in Danger, but Not for the Reasons You Think
by ExCeph — 2022-11-06T21:15:43.962Z
Link: https://ginnungagapfoundation.wordpress.com/2022/11/06/democracy-is-in-danger-but-not-for-the-reasons-you-think/

(Cross-posted from my blog. I apologize that I was not able to shrink the images for this site.)

Congratulations, Earthling voter! Your party has won the election! The Good politicians you elected will enact Good policies, to make Good things happen and help the Good people live Good lives. Your planet’s democracy is saved!

You claim this government in the name of your party! Hmm! Isn’t that lovely, hmm?

…Or is it?

Dun dun duuuuuuunnnnnn!

Now that I think of it, isn’t there still a whole party full of other voters who disagree with those policies you wanted? In fact, there are enough of them that they almost elected some Ungood politicians.

And your best plan for preventing those voters from electing those Ungood politicians was to… hope that your side had more people than theirs did? That seems risky. You had to give a lot of money to the Good politicians in order to help them win, and it almost wasn’t enough. That’s frightening.

After all, Good policies are very important. You can’t let them fail just because so many people don’t agree that they’re Good policies.

So how can you reduce the risk of electing Ungood politicians? How can democracy work if people vote for Ungood things?

You might silence the Ungood voters, preventing them from spreading their ideas and beliefs and from working together effectively. After all, what’s the point of having rights like the freedom of speech and assembly if people are just going to use them to advocate for Ungood policies?

To save democracy–that is, the system that governs based on the voices of the people–it seems you need to take away the voices of the people who want the Ungood things so that people are only allowed to talk about and vote for Good things. The less freedom people have to talk about whatever ideas and values they want, the more democracy will thrive!

Maybe some Good politicians can make Good laws about what ideas people are allowed to talk about. I’m sure they will still allow you to voice your complaints when the Good politicians are not doing a Good job. After all, people in charge of running countries are well-known for welcoming criticism.

[CAPTION REDACTED]

The real threat

…If you’re reading this at all, you have p...
Playground Game: Monster
by jefftk — 2022-11-06T16:00:04.255Z

Here's a game I play with the kids at the park a lot:

You play on a play structure.

One person is the "monster"

Anyone they tag becomes a monster as well.

Monsters are not allowed on the horizontal surfaces of
the play structure.

Non-monsters can't go on the ground.

Little kid monsters (7 and younger?) are "baby monsters" who
can still go on horizontal surfaces.

Usually I'm the monster over and over. The game typically involves a
lot of trying to sneak up on people and reach them through the bars,
or climbing around on the railings trying to corner people.

This works best on a play structure with lots of railings and where
the main platform is at about adult shoulder height. You often need
to tweak the rules for the specific structure and kids you're playing
with, trying to keep it balanced: if games are running much too
long/short then shift the rules slightly towards or away from the
monster.

When I was little I think we used to call this "don't touch the ground
tag", though looking online now that seems to be generally used to
refer to a different game.

Comment via: facebook
Has Pascal's Mugging problem been completely solved yet?
by EniScien — 2022-11-06T12:52:17.811Z

Are Kolmogorov's difficulty and Hanson's leverage penalty combined into one? Has the more important general problem of probabilities not converging been solved? (The lesswrong posts I looked at did not mention if there is a better solution than the "patch" of the time. Reading of concept also don't give more understanding)
Should I Pursue a PhD?
by DragonGod — 2022-11-06T10:58:51.241Z

Introduction

I'm currently flirting with the idea of trying for a math PhD 2 - 3 years down the line.

I'm currently on a Theoretical Computer Science Masters program at the University of [Redacted] in the United Kingdom.

(My program is 2 semesters of teaching (7 - 8 months) followed by a 9 - 12 month industrial placement starting in June 2023. [I might forego the 1 - year industrial placement if I can't get a suitable placement [I.E. theoretical research that feels like it would be valuable experience for the kind of career I want to pursue] and graduate after one year by completing a masters project over the summer instead.)

After graduation I'm considering taking a gap year to fill in the gaps in my maths knowledge/prepare for the PhD, maybe see if I can contribute to the research agendas I think are interesting/promising).

I might also pursue a PhD in Theoretical Computer Science instead of mathematics (maybe applications for a TCS PhD would be looked on more favourably with a TCS Masters/recommendations from my current lecturers).

Why A PhD?

I currently plan to learn a lot of (especially abstract) maths to (upper) graduate level for alignment theory (I want to do theoretical alignment work that is basically just applied maths), and I think I would benefit from the opportunity to study the relevant mathematics under a "guru"/the dedicated mentorship a PhD provides. I expect the first few years of my career in alignment research would be mostly spent on deconfusion/distillation, and I expect high levels of mathematical sophistication to be very valuable for that.

I find abstract maths and mathematical modelling "fun", and really enjoy being a student.

My Alignment Theory of Change

I am operating under/optimising for long timelines (transformative AI is decades away [20+ years]), and this influences what kind of research I believe to be most promising.

I expect theoretical (especially foundational [especially in our current pre paradigmatic stage to be the most promising]) and am persuaded by agent foundations agendas. The extant research agendas I'm most excited for and could see myself working on someday:

John Wentworth's Natural Abstractions Hypothesis and Selection Theorems
Vanessa Kosoy's Learning Theoretic Alignment Agenda

Other agendas that take a desiderata first approach to alignment

Garrabant and Demski's Embedded Agency

My basic plan for ...
You won’t solve alignment without agent foundations
by Mikhail Samin — 2022-11-06T08:07:12.505Z

The following is a part of a response to a Vivek Hebbar's problem (the questions in cursive). I looked at why an alignment proposal doesn't work at an AGI level, in connection to only two of Eliezer's doom arguments. After that, I talked about why I'm generally pessimistic about every single specific proposal for training AGI that I've heard.

(Thanks to my wonderful friends for their help: Slava Meriton, who spent many hours making me pay attention to what I was saying, and Owen, who checked my English.)

Epistemic status: mostly a dump of intuitions; I’m at 70% that in the next three months I won’t change my mind about the important parts of my views expressed here.

I wrote the text to be read by Nate Soares and Vivek Hebbar. I think some claims, including the post title, are not fully justified by the text. I think this might still be valuable to publish.

● Pick an alignment proposal and specific task for the AI.

I’m picking the

Externalized reasoning oversight

proposal. The task for the AI is to come up with a solution to ELK that Paul Christino won’t find issues with, as a proof-of-concept for solving alignment.

● First explain, in as much concrete detail as possible, what the training process looks like.

The training process looks like this[1]:

Train a language model to accurately predict the next token in a sequence. Then, fine-tune the model to solve problems with easily evaluable answers step-by-step at a superhuman level. The model is rewarded for assigning higher probabilities to tokens which end up at a correct answer, and slightly penalized for making more steps. Fine-tuning includes attempting to ensure that the problem-solving-directed reasoning is fully represented by the English descriptions of the reasoning steps[2] and an RLHF setup[3] with humans evaluating how much they like the reasoning. During the problem solving process, check that the reasoning doesn’t include plans for hacking or deception of anything or anyone in the real world.

After that, ask the model to reason through and output ELK solutions such that Paul Christiano wouldn’t find issues with them. The further plan would look something like “send the resulting solution to Paul, ???, and alignment is solved”.

Then go through

Eliezer’s doom list.

Pick 2 or 3 of those arguments which seem most important or interesting in the context of the proposal.

● For each...