trentmkelly/LessWrong-43k huggingface dataset:
Mental subagent implications for AI Safety
by moridinamael — 2021-01-03T18:59:50.315Z

Meta: I wrote this draft a couple of years ago, but never managed to really proceed down this line of thought to a conclusion. I figured it would be better off placed in public for feedback.

Assume, for the sake of argument, that breaking a human agent down into distinct mental subagents is a useful and meaningful abstraction. The appearance of human agentic choice arises from the under-the-hood consensus among mental subagents. Each subagent is a product of some simple, specific past experience or human drive. Human behavior arises from the gestalt of subagent behavior. Pretend that subagents are ontologically real.

A decent way of defining what a person genuinely wants would be if they both claim to want it before getting it, and would predictably claim that they actually wanted it after getting it. This is a decent first-pass way of ensuring sufficient inter-subagent agreement.

Some agents are going to be dis-endorsed by almost every other relevant agent. I think inside you there's probably something like a 2-year-old toddler who just wants everyone to immediately do what you say. There are probably some really sketchy parts in there, attached to childhood traumas or evolved instincts, that you overall just don't endorse, and which would be really bad for you if you satisfied them.

This basically implies that even a super-AI trying to "satisfy" all your "values" should actually probably ignore the part of you that is indistinguishable from Sauron. And maybe some other parts that are harder to describe.

So how does the super-AI determine which parts of you are Sauron and which parts may be kind of weird but actually deserve to be paid attention to? Like, it would be really easy to accidentally sweep "sex" into the dustbin, because it's basically a minefield of weird selfish animalistic behaviors. The Vulcans confine it to a narrow timeframe, etc. But nobody wants a future where the AI overlord has done us the favor of cutting sex out of our lives.

Status-seeking behavior, striving, and competitive impulses are other things that I can see being accidentally binned by some process trying to "optimize" for "what humans want".

Again, the stance here is to break all human behavior down into fundamental subagents or into fundamental goal-like or belief-like objects. If the super-AI-psychiatrist can fire a muon beam into your brain and actually see all of your subagents...
Retrospective on Teaching Rationality Workshops
by Neel Nanda — 2021-01-03T17:15:00.479Z

TL;DR: I organised a series of afternoon applied rationality workshops for the Cambridge Effective Altruism group based on some core CFAR classes. These went much better than I expected them to, and seem to have added long-term value to participants. My goal in this post is to share my resources and

lesson plans

, my thoughts on teaching applied rationality well, my attempts to distill the concepts into key ideas & mental habits, and to convince other people to organise similar workshops!

(Disclaimer: This was inspired by CFAR material, but is very much all my interpretation and framings. Any problems with the material are likely from me, not CFAR)

Introduction

I went to a Centre For Applied Rationality (CFAR) workshop last year and found this a really valuable experience. I especially found that a few techniques stuck well with me and became valuable parts of how I thought. So, to help consolidate these and to share them with others, I organised some afternoon workshops for the Cambridge EA group based on my favourite classes. I think these were extremely successful and added more value than I expected to the participants. So this post is my attempt to write up a retrospective on the workshops, what I think was valuable about them, and what I learned from them.

I taught four workshops:

Having productive disagreements

(based on Double Crux)

Effective planning

(based on Murphyjitsu)

Building good habits

(based on Trigger-Action Plans, or TAPs)

Building useful systems

(based on Systemisation)

The format was fairly different from the standard CFAR workshop (90 minute afternoon workshops every 2 weeks, rather than an intense 4 day workshop), which I expected to make it much harder to have an impact. But, based on a followup 2-3 months later with participants, I found that some insights had stuck, and were being used regularly. A significant component of this was that I focused on distilling the techniques down to the key ideas, and useful mental habits. These seemed to stick well with some people, without requiring much followup effort.

Overall, I feel very happy with the impact, and I think afternoon classes are low cost to run (and seem to transfer acceptably to remote). I don’t think a CFAR-style framing of rationality resonates with everybody, but there are enough people who found it high value that I think this was super valuable on average. So I would be v...
On writing like a butterfly
by KatjaGrace — 2021-01-03T07:30:17.144Z

I thought it would be interesting to try to write my review of the Diving Bell and the Butterfly in my head without setting pen to paper until the end, and to convey at least some of it by blinking, since I find the fact that the author wrote the whole book in this way astonishing. Perhaps experiencing that process myself would improve my understanding of things, such that I wouldn’t be astonished.

I think trying to do this was an even better exercise than I expected, though by the end I was frustrated to the point of tears, and I’m still feeling kind of annoyed, having just put it up.

(Hopefully this was also a vivid and enlightening experience of signing up for annoying projects, which I do often, but usually the annoyance is months later than the agreeing, so I’m not sure that my intuitive anticipations make the connection.)

Before I go and do something anti-annoying, I figure I should write some notes on the experience, while it is fresh.

Some notes:

It did feel fairly encumbering. There were nascent sentences that I might have tried to poke in somewhere, then play around with, then look at and move or get rid of, where the prospect of trying to do some equivalent of all that in my head while keeping hold of the broader paragraph was too intimidating, and I watched them go by. And the sentences I did write felt like half my attention was on something like balancing them on the end of a stick and not having them fall on the floor, and really sculpting them would have required too much dexterity.

Though I think in some sense they were much more sculpted than usual, because I did think about each one for longer, and often hone it into something more succinct and memorable instead of writing down the first ramble that entered my mind. I’m not sure how that fits with the above observation.

It felt mentally strength-building - as if I was exercising a capability that would improve, which was exciting, and I briefly fantasized about a stronger and defter inner world.

I started out looking at things around me as I composed, like my resting computer, and the table, and the sea. But after a while, I realized that I was staring intently at a long rug with about as many Persian whorls as paragraphs in my prospective post, and that as I envisaged the current sentence, I was mentally weaving it around some well-placed sub-curls of its paragraph-whorl. Looking awa

...
Don't Use Your Favorite Weapon
by Aaron Bergman — 2021-01-03T04:01:39.766Z
Link: https://aaronbergman.substack.com/p/dont-use-your-favorite-weapon

Why your favorite arguments aren't most likely to land

If forced to condense my political ideology onto the usual spectrum, I’ve always been left of center. I think that the government can and should work to improve wellbeing, and this generally entails change. It’s called “progressivism” for a reason, after all. The opposite ideology, conservatism, is fundamentally about preserving the present or past instead of embracing the untested or novel.

I’m not quite naive enough to think that people generally choose political positions and ideologies on the basis of argumentation or evidence. Even still, ideas and their supporting arguments probably play some role in politics; even if most political positioning can be explained by signaling, genetics, preference falsification, and the like, I would bet that marginal changes in the status quo distribution of beliefs are significantly influenced by a principle like “all else equal, people will choose the idea that for which the best arguments are made.”

Back to me, for a minute: in the last two years or so, for the first time, I’ve gained a modest appreciation for the value of conservatism. This isn’t just some high-minded belief in pluralism - I now just think that there is a little more to be said for deference to systems or laws that have stood the test of time.

There have probably been thousands of books and articles on the failures of top-down social change, perhaps most notably James C. Scott’s

Seeing Like a State

. A key idea in this line of thought is

Chesterton’s Fence

, the principle that one should refrain from removing a custom, law or practice before understanding its function or purpose. To outsiders or newcomers, these things may seem irrational or absurd. In fact, it may seem absurd to those obeying the custom as well.

But more often than we’d expect, there is some hidden function behind the absurdity. As Scott Alexander

writes

, for example, there is a native Fijian taboo against pregnant women eating shark. Lo and behold, consuming the local species of shark can cause birth defects.

If you transplanted me and my liberal politics into Fiji and asked me whether we should keep this prohibition, I’d probably object that the custom is irrational, sexist, and oppressive. The conservatives would reply that it’s ordained by God or good for business or sexist to allow women to eat shark, and that Critical Shark...
Bucket Brigade: Video
by jefftk — 2021-01-03T02:50:03.370Z

The biggest pain point with Bucket Brigade has been that you generally
wanted to be running a video call in parallel. You kept needing to
switch windows, and you needed to remember to mute and unmute in the
call at the beginning and end of each song.

Solstice used

Twilio Video, which packages
up WebRTC with server-side support and a developer-friendly API, and I
decided to integrate it as well. Now when you join you'll see it ask
for camera permissions:

Before showing your video to anyone else, it will ask you whether you
are OK with it:

Since I have an external camera with a better angle, I'll press
"Switch camera":

Now I'm happy, so I'll choose "Use this camera", but it will also work
fine if you choose "Continue without camera".

Once you're in, you'll see all the people in their buckets:

During a song you hear people via Bucket Brigade, and the rest of the
time you'll hear people through the video call. Since we aren't doing
anything custom for the videos, you'll see people out of sync with
their singing. At some point it would be nice to try routing videos
through a server we control, so we can add appropriate amounts of
latency. Adding latency on the server would not be precise enough for
audio, which we need aligned at the millisecond level, but I think it
would be ok for video? Of course there's nothing you can do for video
for buckets after you...

The hardest part of the integration was getting it to behave correctly
around people turning their video on and off. Twilio's recommended
approach doesn't turn the camera light off when you turn your video
off, which seems broken to me, so I took their not
recommended approach of unpublishing video tracks. This changed
which events I need to listen to for video being re-enabled, but once
I got it all working it seems reliable.

Previously Bucket Brigade was cheap to run: it essentially only needs
CPU and bandwidth, and the bottom-tier VPS (~$5/month) that powers
jefftk.com can handle about 40 simultaneous users. Twilio calls cost
25¢/person/hour,
which is reasonable but not negligible. I'm hoping to keep it free to
play with, and if the cost starts adding up we'll probably ask if
frequent users could chip in?

Comment via: facebook
Are we all misaligned?
by Mateusz Mazurkiewicz — 2021-01-03T02:42:47.506Z

The orthogonality thesis separates intelligence and goals, constraining the notion of intelligence to instrumental rationality and allowing for any combination of general intelligence and a goal system. And the other way around, such a description should apply to any possible agency. For any agency, it should be possible to divide it into two parts losslessly. Many arguments against the thesis have been constructed and proven incoherent, showing its robustness and broad applicability. Still, there also exist valid critiques of its assumptions, such as one by John Danaher. The following text discusses the human mind in the context of the orthogonality thesis and touches on the evolutionary origin of the human brain.

Let us start with an assertion: 'some intelligences and goal systems are irreducibly connected.' This protest appears intuitive since it comes from dissonance and confusion experienced when one attempts to apply the orthogonality thesis to analyze oneself. Self-inquiry of the human mind produces wildly different ideas than ones arising from observation of a Reinforcement Learning agent. We do not perceive experienced pleasure as our final goal, nor any other alternative metric appears obvious. Satisfaction seems to be based on changing goals. Core values, which we once considered ultimate, are now irrelevant. When someone recommends you a book, saying 'this will change you as a person,' you do not worry about your goal preservation.

One counterexample would be a religious fanatic refusing to read the book, being afraid that it will make him doubt his beliefs. There exist mechanisms in humans that give very high inertia to their opinions, constructing defense mechanisms against acquiring a new perspective. Furthermore, by definition, if one decides to change his core values in light of new information, the core values never were one's final goal. But what is the final goal, then? There appears a stark contrast between that experience of confusion and the boldness of the orthogonality thesis.

And, the other way around, there is an intuitive feeling that the thesis does not constitute a complete picture of intelligent agency. One could point to the holistic self-reference and, furthermore, self-referential goal-setting, as necessary qualities of general intelligence. Whereas the former can be explained away as a form of instrumental rationality, the lat...
If the fundamental problem that economics addresses is the scarcity of resources, then what are the basic questions other subjects attempt to solve?
by Kaveh-Sedghi — 2021-01-03T02:36:56.362Z

One of the core principles of economics is the scarcity of resources, which the subject of economics attempts to address. Put in other words, the problem of unlimited wants and limited resources relative to that is perhaps the sole reason economics exists at all. Generalizing, limited resources is the problem, and economics the solution.

On the assumption that this model of problem and solution can be applied to other disciplines (i.e. mathematics, physics, chemistry, etcetera), what are the problems they aim to solve?

If this model does not apply, or isn't the right framework/question in the first place, what are the first, ur-principles underpinning various subjects?
More predictions
by Tim Liptrot — 2021-01-02T15:53:38.619Z

Covid:

Will a rapid at home test become available to me during the crisis?

Will I be able to buy a rapid at home test for Covid-19. Must be taken and processed at home. Must be available while Covid deaths per day are above 400 (can't miss the pandemic)

The FDA dragged its feet on rapid/at-home testing for the first 10 months of the pandemic. It has finally approved a test (although it requires a perscription). The manufacturer plans to ship 100,000 a day during January, so supply will be low. But since we're not vaccinating the vulnerable the Covid death rate should stay high into February. Therefore I expect at least one Ellume test will be available to me.

Pr(70%)

Will the US move to first-doses-first vaccination.

Will more than half of the first 25 million vaccinations in the US have the second dose delayed to two months of more.

Recently the UK changed policy to FDF, delaying the second dose to three months to vaccinate twice as many people in the first round. The decision was criticized but the British gov held their grounds (parliaments are better). However in the US epidemiologists and the CDC have criticized FDF and indicated they prefer second doses first vaccination. Fauci publicly stated opposition to the FDF policy.

This debate closely reproduces the old masks debate. The current US medical elite has a strict epistemology that requires RCTs and disregards inductive reasoning. In the fast-changing crisis time is short, so they presume in favor of inaction. In crises, good policies (like masks) can have overwhelmingly positive cost-benefits which are obvious inductively, but they get opposed in favor of inaction.

In the mask battle, inductive reasoning won. But Fauci and the CDC seem determined to enact second doses first. They are also facing a crisis of resistance to vaccination, which will distract from the dosing debate. Therefore I do not expect FDF to win in the US.

Pr(25%)
Feature request: personal notes about other users
by  — 2021-01-02T14:54:05.998Z

Feature use scenario:

I hover the mouse pointer over a username
The small box that appears (the one that currently shows basic stats: when they joined, how many posts they have written, etc.) contains also a textarea field
Whatever I write in that field persists between my logins, until I edit it again (if ever)
The contents of that field are only visible to me

Motivation: As I read LW more, I'm starting to attribute expertise in some subjects to some users. And conversely, I'm noticing specific biases in other users. I would like to privately keep note of that, so that I can more precisely trust/distrust what they write in the future.
Data about the new coronavirus variant (B.1.1.7) from Denmark
by Oskar Mathiasen — 2021-01-02T11:38:35.187Z

I noted on Zvi's recent post (found here https://www.lesswrong.com/posts/gbuXXC9KvphmXQFLd/what-evidence-will-tell-us-about-the-new-strain-how-are-you#GxzkzNdYfTLjmQEAm ) that Denmark was one of few other nations to do large amounts of Covid strain sequencing.
This is a followup with the newly released data.
The data seems consistent with a doubling time around a week.

Data found here https://www.ssi.dk/-/media/cdn/files/opdaterede-data-paa-ny-engelsk-virusvariant-sarscov2-cluster-b117--01012021.pdf?la=da

Here is a graph over cases which where tested on specific days, note that days where there where 0 tests sequenced as B.1.1.7 have been skipped
(colors corespond to different areas in Denmark)
Review: The Diving Bell and the Butterfly
by KatjaGrace — 2021-01-02T01:30:17.245Z

I suspect my mind of taking its observations of a person’s physical energy and dexterity as strong evidence about their mental quickness and clarity.

The existence and the wrongness of this presumption were brought into relief for me by reading Jean-Dominique Bauby’s The Diving Bell and the Butterfly, on his life with locked-in syndrome. Because realizing that the author’s lively and intelligent voice was issued from a single blinking eye looking out of a mostly inert body felt like seeing a magic trick.

But perhaps it is just that a writing process without vigorous back-and-forth with a mind-expanding piece of paper or virtual paper—a place to lay out one’s words and see them and move them around, without having to keep the entire gameboard in one’s head—sounds mentally paralyzing to me.

At this point, I realize that in all likelihood it is my own mind that is weak and wasted from clinging to these crutches, and that has learned to fear starting out by itself with no piece of paper to reassuringly catch its thoughts and promise that nothing will be forgotten. Bauby’s mind may well have become stronger between his time as eloquent editor of French Elle and writing this book.

It probably isn’t the feat of its creation though that makes this enjoyable to read, or makes my mind keep coming back to it. It’s a bright and compelling window into another person’s mind. (The bizarre tragedy of the plot probably doesn’t hurt either.)

Another thing I found interesting about the book was that the setting of being immobile in a hospital bed doesn’t give the reader many of the clues they might usually use to make out the character of the protagonist. How does he hold himself, when he can hold himself? How does he talk to his partner, when he can do more than blink? What did he want in life, before the goalposts were changed and, as he says, not drooling all the time became a cherished ambition? (We do learn that he is 44 years old, and that if he is going to drool, he would prefer to drool on cashmere.) Yet, I did not have a sense of blankness about his personality. From his narration of the world, I felt that I knew him. However toward the end, he narrates the final day before the stroke that detached his mind and his body mostly. And I realize that if I had met him in that way first, I would have a different impression of him. And I liked the way around that I did meet him. ...
Overall numbers won't show the English strain coming
by Eric Neyman — 2021-01-01T23:00:34.905Z

[Epistemic status: half-baked thoughts that might be wrong. Apologies if this point has already been made.]

[Edit: here's a version of this post that is meant for a more general audience.]

Here is a plot of daily new positive COVID tests in the U.S. in the first half of March 2020.

While the rationalist community figured out the whole exponential growth thing in February, it's no surprise that the general public was worried by mid-March: the virus was here and was growing fast. This was evident by the time there were 1,000 new positive tests per day. The lockdown began around mid-March, which is way worse than if it had begun in early March, but much, much better than if it had begun in late March, by which time we had tens of thousands of new positive tests per day.

The situation with the English strain is different. One way in which it's better is that we now have adequate testing. But there's a scary way in which it's worse.

The reason that the chart above is scary is because on any given day there were many times more cases than just a week before. I won't cherry-pick March 15th, but e.g. on March 12th there were 7 times more cases than on March 5th. And the reason that this trend was so evident is that the exponential growth drowned out any day-to-day noise.

This won't be the case with the English strain, at least if the general public (or public officials) use overall numbers to guide their planning. That's because, instead of seeing an exponential trend, i.e. something like cases(t)=1.3t plus noise, we'll be looking at something like cases(t)=200,000+1.3t plus noise (where 200,000 is the currently dominant strain and 1.3^t is the contribution from the English strain). Here's a simulation of what that looks like if the noise is normal with standard deviation 25%.

cases(t) = 200,000 + 1.3^t + random noise

Remember that people starting freaking out about COVID around when there were 1,000 new positive tests per day. So here's my question: where on this chart does the English strain first contribute 1,000 new positive tests per day? The answer:

Day 27

On the other hand, I don't think the general public will start worrying about the new strain until it starts adding to the overall numbers substantially, i.e. sometime between day 40 and day 50.

This means that, because there's already a high baseline case count (plus noise), I expect the general public's reaction time...