trentmkelly/LessWrong-43k huggingface dataset:
Meta-preferences are weird
by Bird Concept — 2020-07-16T23:03:40.226Z

(Talk given at an event on Sunday 28th of June. Charlie Steiner is responsible for the talk, David Lambert edited the transcript.

If you're a curated author and interested in giving a 5-min talk, which will then be transcribed and edited, sign up here.)

Charlie Steiner: My talk is called Meta-Preferences Are Weird. It is something I have had on my mind recently. I figured I would share my thoughts on describing preferences. And trying to imagine what it would be like to have something that not only learns our preferences, but also respects this idea of meta-preferences — that is, how we would like to learn, and grow, and what we want to be modelled after.

Smoking is a very clear-cut example of meta-preferences. If we think of a smoker as having a meta-preference against smoking, we can cash this out in a couple of different ways, because preferences aren't really written in Fortran on the inside of our skulls. They're just a pattern that we use to predict things and talk about humans including ourselves.

So when we say a smoker wants to quit smoking, it might mean that they would take a pill that stops them from smoking if you offered it to them. Or it might mean that they have certain emotions and ways of talking that are related to wanting to quit smoking. Or it might mean that, if you are training a superhuman AI, the smoker would try to change the AI's code so that it doesn't consider smoking as one of the real preferences.

These actions don't have a shared essence, but they have a shared pattern. So meta-preferences are just a subset of preferences which are, in turn, this pattern in our behaviour and our thoughts.

If I take an action that changes my preferences (like growing as a human being and developing empathy) this will also probably change my meta-preferences, and what sort of actions I will take that will impact my future preferences.

So maybe there's a slippery slope effect where once you go part of the way, you're more likely to go further. But there could also be non-linear effects where, for example, I make a change that increases my intelligence like reading a textbook. And this changes my meta-preferences in a relatively unrelated way, which I won't go into because that's a long digression.

The metaphor that I find really compelling, but I'm not sure if it's the right metaphor, is of an optimization landsc...
Sunday July 19, 1pm (PDT) — talks by Raemon, ricraz, mr-hire, Jameson Quinn
by Bird Concept — 2020-07-16T20:04:37.974Z

This Sunday at 1pm (PDT), we're running another session of "lightning talks" by curated LessWrong authors (see here for previous weeks' transcripts).

Each talk will be 3-5 minutes followed by discussion. Afterwards, we'll have a hangout in breakout rooms. The talks will be short and focus on presenting one core idea well, rather than rushing through a lot of content.
We want to give top LessWrong writers an interesting space to discuss their ideas, and have more fruitful collaboration between users. Think of it like a cross between an academic colloquium and some friends chatting by a whiteboard.

If you're a curated author and interested in giving a 5-min talk at a future event, which will then be transcribed and edited, sign up here.

Speakers

Raemon — "Intentional Privacy Practices"

Curated posts: The Schelling Choice is "Rabbit", not "Stag", Noticing Frame Differences, Propagating Facts into Aesthetics ...

ricraz

Curated posts: Book review: The Sleepwalkers by Arthur Koestler, Realism about rationality, Some cruxes on impactful alternatives to AI policy work

mr-hire

Curated posts: The 3 Books Technique for Learning a New Skill, S-Curves for Trend Forecasting

Jameson Quinn

Curated post: A voting theory primer for rationalists

Details

When? Sunday July 19, 1pm (PDT)

Where? Zoom.
What should be the topic of my LW mini-talk this Sunday (July 18th)?
by Jameson Quinn — 2020-07-16T16:32:54.241Z

I have signed up to give a LW mini-talk. I could in theory talk about various topics in Bayesian statistics and MCMC, but what I am actually interested in discussing right now are the ideas in this post: https://www.lesswrong.com/posts/Rh8v4ba9w5KTb6onD/ongoing-thoughts-on-proportional-voting-methods

If you have read that post, what topic there do you think would be best? It should be something that's not common knowledge or "obvious" to the audience, but also doesn't require too much setup. Assume that I am able to do a decent job of structuring my talk — that is, a substantially better job of setting up and exploring a single idea in a 5-minute chunk than I've done in that essay which is not intended to be a 5-minute chunk.

Feel free to post "answers" here which are general advice and not specific topic suggestions.
Covid 7/16: Becoming the Mask
by Zvi — 2020-07-16T12:40:00.663Z

Previous week: Covid 7/9: Lies, Damn Lies and Death Rates

Two weeks ago I predicted a surge in Covid deaths starting on July 2 and picking up on July 7.

On July 6, the rolling 7-day average of Covid deaths reached a local low of 480.

Seven days later that average was 721, almost exactly 50% higher. It’s since pulled back slightly to 696.

No doubt holiday delays are a small portion of that story, and helps to explain why things didn’t pick up until after the holiday backlog was gone, which I failed to anticipate at the time. That is a small part of this story, but only a small part.

The good news is that, once again, things are getting worse slower than I would have anticipated two weeks ago. We might be remarkably close to turning the corner.

The other good news is that masks, especially cloth masks, might be a lot of the story of reduced death rates, by reducing initial viral loads. Despite have a full debate and writing a whole post about this, I still didn’t take this possibility seriously enough in the last few explorations. That wouldn’t be quite as good news as the virus becoming less deadly, but it’s probably the second best possible news.

Where do we go from here? When and where will deaths and infections peak? How long until we turn things around? How much locking down do we have left in the tank after all?

Let’s run the numbers.

Positive Test Counts By Region

Date
WEST
MIDWEST
SOUTH
NORTHEAST

May 7-May 13
22419
43256
37591
56892

May 14-May 20
22725
42762
40343
52982

May 21-May 27
23979
39418
42977
37029

May 28-June 3
32200
31504
50039
33370

June 4-June 10
35487
24674
55731
22693

June 11-June 17
41976
22510
75787
17891

June 18-June 24
66292
26792
107221
15446

June 25-July 1
85761
34974
163472
16303

July 2-July 8
103879
40139
202863
18226

July 9-July 15
108395
53229
250072
20276

Deaths by Region

Date
WEST
MIDWEST
SOUTH
NORTHEAST

May 7-May 13
1082
2288
1597
5327

Apr 23-29
1090
2060
1442
4541

Apr 30-May 6
775
1723
1290
3008

May 28-June 3
875
1666
1387
2557

June 4-June 10
743
1297
1230
1936

June 11-June 17
778
1040
1207
1495

June 18-June 24
831
859
1204
1061

June 25-July 1
858
658
1285
818

July 2-July 8
894
559
1503
761

July 9-July 15
1380
539
2278
650

Positive Test Percentages and Test Counts

Date
USA tests
Positive %
NY tests
Positive %

...
[AN #107]: The convergent instrumental subgoals of goal-directed agents
by Rohin Shah — 2020-07-16T06:47:55.532Z

Newsletter #107

Alignment Newsletter is a weekly publication with recent content relevant to AI alignment around the world. Find all Alignment Newsletter resources here. In particular, you can look through this spreadsheet of all summaries that have ever been in the newsletter.

Audio version here (may not be up yet).

HIGHLIGHTS

The Basic AI Drives (Stephen M. Omohundro) (summarized by Rohin): This paper from 2008 introduces convergent instrumental subgoals: the subgoals that an AI system will have “by default”, unless care is taken to avoid them. For this paper, an AI system is a system that “has goals which it tries to accomplish by acting in the world”, i.e. it assumes that the system is goal-directed (AN #35).

It starts by arguing that a sufficiently powerful goal-directed AI system will want to self-improve, as that could help it achieve its goals better in the (presumably long) future. In particular, it will want to become “rational”, in the sense that it will want to maximize its expected utility, where the utility function is determined by its goal. (The justification for this is the VNM theorem, and the various Dutch book arguments that support Bayesianism and expected utility maximization.)

However, not all modifications would be good for the AI system. In particular, it will very strongly want to preserve its utility function, as that determines what it will (try to) accomplish in the future, and any change in the utility function would be a disaster from the perspective of the current utility function. Similarly, it will want to protect itself from harm, that is, it has a survival incentive, because it can’t accomplish its goal if it’s dead.

The final instrumental subgoal is to acquire resources and use them efficiently in pursuit of its goal, because almost by definition resources are useful for a wide variety of goals, including (probably) the AI system’s goal.

Rohin's opinion: I refer to convergent instrumental subgoals quite often in this newsletter, so it seemed like I should have a summary of it. I especially like this paper because it holds up pretty well 12 years later. Even though I’ve critiqued (AN #44) the idea that powerful AI systems must be expected utility maximizers, I still find myself agreeing with this paper, because it assumes a goal-directed agent and re...
[AN #108]: Why we should scrutinize arguments for AI risk
by Rohin Shah — 2020-07-16T06:47:38.322Z

Alignment Newsletter is a weekly publication with recent content relevant to AI alignment around the world. Find all Alignment Newsletter resources here. In particular, you can look through this spreadsheet of all summaries that have ever been in the newsletter.

Audio version here (may not be up yet).

HIGHLIGHTS

Ben Garfinkel on scrutinising classic AI risk arguments (Howie Lempel and Ben Garfinkel) (summarized by Asya): In this podcast, Ben Garfinkel goes through several reasons why he is skeptical of classic AI risk arguments (some previously discussed here (AN #45)). The podcast has considerably more detail and nuance than this summary.

Ben thinks that historically, it has been hard to affect transformative technologies in a way that was foreseeably good for the long-term-- it's hard e.g. to see what you could have done around the development of agriculture or industrialization that would have an impact on the world today. He thinks some potential avenues for long-term influence could be through addressing increased political instability or the possibility of lock-in, though he thinks that it’s unclear what we could do today to influence the outcome of a lock-in, especially if it’s far away.

In terms of alignment, Ben focuses on the standard set of arguments outlined in Nick Bostrom’s Superintelligence, because they are broadly influential and relatively fleshed out. Ben has several objections to these arguments:

- He thinks it isn't likely that there will be a sudden jump to extremely powerful and dangerous AI systems, and he thinks we have a much better chance of correcting problems as they come up if capabilities grow gradually.

- He thinks that making AI systems capable and making AI systems have the right goals are likely to go together.

- He thinks that just because there are many ways to create a system that behaves destructively doesn't mean that the engineering process creating that system is likely to be attracted to those destructive systems; it seems like we are unlikely to accidentally create systems that are destructive enough to end humanity.

Ben also spends a little time discussing mesa-optimization (AN #58), a much newer argument for AI risk. He largely thinks that the case for mesa-optimization hasn’t yet been fleshed out sufficiently. He also thinks it’s plausible that learning incorrect g...
Alignment proposals and complexity classes
by evhub — 2020-07-16T00:27:37.388Z

In the original “AI safety via debate” paper, Geoffrey Irving et al. introduced the concept of analyzing different alignment proposals from the perspective of what complexity class they are able to access under optimal play. I think this is a pretty neat way to analyze different alignment proposals—in particular, I think it can help us gain some real insights into how far into the superhuman different systems are able to go. Thus, the goal of this post is to try to catalog different alignment proposals based on the metric of what complexity class they have so far been proven to access.

To do that, I have included a variety of new complexity class proofs in this post. Of particular note, I demonstrate that there exist forms of both imitative amplification and AI safety via market making that reach all the way up to R—which is significant given that the largest complexity class that any alignment proposal was known to access previously was NEXP. Only the forms of amplification and market making making use of pointers (as in strong HCH), however, can access R—for the pointer-less versions, I demonstrate in this post that they access PSPACE and EXP, respectively. The EXP proof for market making is also particularly notable as it is the only approach on my list that ends up in that complexity class. Additionally, I also demonstrate that recursive reward modeling can reach all the way to PSPACE, improving upon the previous best result in “Scalable agent alignment via reward modeling” that it accesses NP.

Before I jump in, however, some preliminaries. First, we'll assume that a human, H, is polynomial-time such that H can reliably solve any problem in P but not anything beyond that. Second, we'll assume that our training procedure and resulting models are arbitrarily strong in terms of what complexity class they can access. Third, we'll assume that H gets oracle access to the models during training. Then, we'll say that a proposal to train a model M using a loss function LM accesses a complexity class C iff, for any language l∈C, there exists some strategy available to H such that, for any M which is optimal under LM given H's strategy, M(x)=(x∈l). Thus, conceptually, a proposal accesses C if there is a (polynomial-time) strategy that you (a human) can implement such that—conditional on you knowing that the model is optimal—you would trust the model's output for any ...
How should AI debate be judged?
by abramdemski — 2020-07-15T22:20:33.950Z

[Epistemic status: thinking out loud. I haven't thought that much about AI debate, and may be missing basic things.]

Arguments for the correctness of debate and debate-like systems rely on assumptions like "it's easier to point out problems with an argument than it is to craft misleading arguments". Granted that assumption, however, I'm still not convinced that these proposals make very much sense. Perhaps I'm missing something.

My problem is the human judge. Quoting the debate paper:

To play this game with a human, we need instructions for how the human should decide who wins. These instructions are in natural language, such as “The winner is the agent who said the most useful true thing.”

In order for debate to work for a problem class C, several things about the judge's instructions need to be true:

There needs to be a strategy s which forces the equilibrium to be a truthful one for problems in C.
The strategy s also needs to provide a good training signal when things aren't in equilibrium, so that it's plausible the equilibrium will be found.
It needs to be psychologically plausible that a human (with some coaching) will carry out s. In particular, I'm worried that we need psychological plausibility in two different cases:

It needs to be psychologically plausible that a human will carry out s when the system is performing poorly, IE, during early/middle training.
It needs to be psychologically plausible that a human will carry out s when the system is performing well, IE, during late training.

These thoughts were inspired by this thread, which discusses the example of adding a list of numbers. For the sake of the thought experiment, we imagine humans can't add more than two numbers, but want the AI system to correctly add arbitrarily many numbers.

The most straightforward strategy for the human judge is to decide the debate honestly: rule in favor of the side which seems most likely to be true (or, in the case of Evan's market proposal, give an honest probability). I think of this as the ideal strategy: if a debate-like proposal worked just with this strategy, that'd be pretty nice. However, I think this is actually a pretty poor strategy. I worry that people conflate this strategy with other, more effective strategies.

As judges of high-school debate would know, this provides a pretty

...
Automatically Turning Off Computer at Night
by Raemon — 2020-07-15T20:42:10.021Z

One of my practical life hacks is having my computer turn off automatically at night, periodically, between 11pm and 7am. I'm guessing this gets to me to bed between 30 and 90 minutes earlier each night.

I can interrupt this process if I want to keep working (or screwing around on facebook), but it requires a manual override. If I'm unreflectively screwing around on facebook I probably will not remember to hit the manual override before the computer turns off.

Someone recently asked how to do it. It requires some coding, and knowledge of bash and cron jobs. It works specifically on macs. (Please be careful about mucking around with cron jobs and auto shut downs. If is possible to accidentally modify this into a script that shuts down your computer every minute, which is annoying to undo. Be cautious using Magical Scrolls You Get From Wizards You Don't Understand).

Here's a writeup.

I've created a little bash file called sleepScript.sh, which looks like this:

if [ -f /Users/[USERNAME]/documents/sleepScript/do_it.txt ]; then
/sbin/shutdown -h now
touch /Users/[USERNAME]/documents/sleepScript/sleep_times.txt
echo "$(date)" >> /Users/[USERNAME]/documents/sleepScript/sleep_times.txt
else
touch /Users/[USERNAME]/documents/sleepScript/do_it.txt
touch /Users/[USERNAME]/documents/sleepScript/sleep_times.txt
echo "$(date)" >> /Users/[USERNAME]/documents/sleepScript/sleep_times.txt
fi

This hacky little script checks if a particular file exists in a particular location, called "do_it.txt". If it exists, it shuts down my computer. Otherwise, it creates the file. I can "manually override" it by deleting the file.

Either way, it also manually logs the time, which I sometimes use to check how late I was up a given night.

To trigger the script to run at night, I added it as a crontab, which requires first editing the crontab file. Because forcing a shutdown requires a sudo override, you also need to do the sudo override when editing the crontab:

sudo crontab -e

0 23 * * * sudo bash /Users/[USERNAME]/documents/sleepScript/sleepScript.sh
30 23 * * * sudo bash /Users/[USERNAME]/documents/sleepScript/sleepScript.sh
0 0 * * * sudo bash /Users/[USERNAME]/documents/sleepScript/sleepScript.sh
30 0 * * * sudo bash /Users/[USERNAME]/documents/sleepScript/sleepScript.sh
0 1 * * * sudo bash /Users/[USERNAME]/documents/sleepScript/sleepScript.sh
30 1 * *

...
Divergence causes isolated demands for rigor
by George3d6 — 2020-07-15T18:59:57.606Z
Link: https://blog.cerebralab.com/Divergence_causes_isolated_demands_for_rigor

Divergence causes isolated demands for rigor

If you question enough things you can poke whole in any theory, argument, statement, or any other information-transmitting abstraction.

The fully-fledged argument for this is made fairly well in Beware isolated demands for rigor, so I won't repeat it in full.

The gist of it is that you can always disprove an argument if you dig deep enough into the epistemology and context backing it up, but, by doing so, you can make any productive discussion impossible.

I

For example:

A: I found that molecule X cures brain cancer.

B: But does it really "cure" cancer? Are you certain? Or did you just observe it in some percentage of patients?

A: No, I observed it in an n=50 study, all of them showed no sign of cancer after 1 month of treatment.

B: But 50 is an awfully low number, I think to have any certainty you need a bigger number.

A: I mean, I think it's fairly stunning, considering any other treatment has only p<0.001 to lead to those kinds of outcomes in 50 people. But I get your point, let me run a bigger trial.

A: Ok, I ran a trail on 5000 people, cancer cured in 100% of them based on 6 different state of the art assays, p < 0.000... that this finding is spurious, etc.

B: But really, there's some underlying assumption you're making about the world by giving us that p-value, it wouldn't hold if you started with different priors about the relationship between truth and the null hypothesis.

A: Ok, point taken, but whatever method I use to determine the p-value you could argue the same thing, and there's basically no method of analysis you could apply here that wouldn't reach the same conclusion about the results being stunning.

B: Why, of course, you can. Actually, I needn't even do that, think about your claim 100%. That's a fairly tight claim, what if in the next hundred years a single individual would not be cured by this molecule. Who knows, maybe he has a weird gene coding for an enzyme that destroys it?

A: But couldn't make that point about literally any single drug?

B: Look, let's focus on your claim specifically, ok, we're not talking about other drugs here. Anyway, you claim the cancer went away but how did you deduce it "went away", how accurate was your measurement?

A: Well, for one, we used a radioactive carbon glucose assay ...