trentmkelly/LessWrong-43k huggingface dataset:
Where's the Turing Machine? A step towards Ontology Identification
by adamShimi — 2020-02-26T17:10:53.054Z

Introduction

Assume you are an agent, and you have a model of the world. How do you check that you are in your model, that you exist within it?

To see why I care about this question, let's go back to embedded agency.

Embedded agency is the study of intelligent agents which are part of the world with with they interact. This is to distinguish from cartesian (or dualistic) agency, where agents and world are two separated entities with clearly defined I/O channels. Embedded agents fit more to our intuitions about agents; but they run into many problems.

The Embedded world-models sub-problem of embedded agency captures the issue with formally modelling a world bigger than yourself, which contains you and your map of it. It ranges from logical uncertainty, the use of probability to quantify beliefs about the consequences of known facts, to self-reference issues caused by the infinite recursion of the map containing itself.

An embedded agent will probably build models of the world. And any meaningful model of the world will contain a representation of the agent itself. Hence distinguishing a model of the world with the agent and one without, as well as pinpointing in which part of the model the agent lies, gives us information about how to construct and deal with worlds in which we are embedded.

Notice that the agent considering the world in which it is embedded might cause problems of self-reference. The point of this post is to answer the question of embedding in an outside-view approach: given a description of an agent and a description of a world-model, study when the agent is embedded within the world-model. This separates the issue of embedding from self-reference, and provides a first step towards answering the question in the context of self-reference.

I also see this question as an instance of ontology identification: finding how things we care about are encoded in a world-model. One of the difficulties of the general problem is that the object we are looking for and the world-model are represented in potentially different ways.

But this is not the case here. Our agent will be (hypercomputation nonwithstanding) Turing-equivalent. Hence we represent it as a Turing Machine. On the other hand, the world-model is used for computing predictions about the world. This model is thus also representable by a Turing Machine. And here is the key insight: because both th...
What will be the big-picture implications of the coronavirus, assuming it eventually infects >10% of the world?
by Daniel Kokotajlo — 2020-02-26T14:19:27.197Z

Brainstorming some possibilities, not taking myself too seriously:

--2020 will be remembered as a year of death and disruption; the economy will suffer massively, perhaps enough to count as a recession.

--Many old people will die (something like 10%?) this is a big higher than the current fatality rates but I predict they will rise once hospital systems are overwhelmed. This will have a few knock-on effects: 1. Health care expenditures will drop for the next few years, since old and sick people are the biggest source of such expenditures and the population of old and sick people will be substantially smaller. 2. There will be a noticeable increase in average spending money available to younger people due to inheritances. 3. Politics in the USA at least will shift noticeably (though not dramatically) to the left, since the dead people will be disproportionately right-leaning. 4. Politics worldwide will generally shift in a more radical direction, since the proportion of young people to old people will increase. 5. One of the contenders for the US presidency might die before the election. (This is unlikely even if 50% of the world gets infected, but it is likely enough to deserve mention I think.)

--China's dramatic and draconian measures to contain the virus will come to be seen as a success story, something that would have worked if only other nations had behaved similarly (and perhaps, if China keeps its border closed, it actually will work for China at least?). This leads to an increase in authoritarianism in general around the world, made substantially stronger by the general tendency of authoritarianism to rise during crises.

--The world will, in general, take pandemic risks much more seriously from now on, making it substantially harder for something like this (or worse) to happen again.
If I were a well-intentioned AI...
I: Image classifier
by Stuart_Armstrong — 2020-02-26T12:39:59.450Z

Introduction: If I were a well-intentioned AI...

I've often warned people about the dangers of anthropomorphising AIs - how it can mislead us about what's really going on in an AI (and hence how the AI might act in the future), cause us to not even consider certain failure modes, and make us believe we understand things much better than we do.

Oh well, let's ignore all that. I'm about to go on a journey of major anthropomorphisation, by asking myself:

"If I was a well-intentioned AI, could I solve many of the problems in AI alignment?"

My thinking in this way started when I wondered: suppose I knew that I was given a proxy goal rather than the true goal; suppose that I knew about the Goodhart problem, and suppose that I really "wanted" to align with the true goal - could I then do it? I was having similar thoughts about being a mesa-optimiser.

It seems to me that asking and answering these kind of questions leads to new and interesting insights. Of course, since they come via anthropomorphisation, we need to be careful with them, and check that they are really applicable to AI systems - ensuring that I'm not bringing some of my own human knowledge about human values into the example. But first, let's get those initial insights.

Overlapping problems, overlapping solutions

At a high enough level of abstraction, many problems in AI alignment seem very similar. The Goodhart problem, the issues machine learning has with distributional shift, the problem of the nearest unblocked strategy, unidentifiability of reward functions, even mesaoptimisation and the whole AI alignment problem itself - all of these can be seen, roughly, as variants of the same problem. That problem being that we have an approximately specified goal that looks ok, but turns out to be underspecified in dangerous ways.

Of course, often the differences between the problems are as important as the similarities. Nevertheless, the similarities exist, which is why a lot of the solutions are going to look quite similar, or at least address quite similar issues.

Distributional shift for image recognition

Let's start with a simple example: image recognition. If I was an image classifier and aware of some of the problems, could I reduce them?

First, let's look at two examples of problems.

Recognising different things

Firstly, we have the situation where the algorithm successfully classifies the test set,

...
Reviewing the Review
by Raemon — 2020-02-26T02:51:20.159Z

We just spent almost two months reviewing the best posts of 2018. It was a lot of development work, and many LW users put in a lot of work to review and vote on things.

We’ve begun work on the actual printed book, which’ll be distributed at various conferences and events as well as shipped to the featured authors. I expect the finished product to influence the overall effect of the Review. But meanwhile, having completed the “review” part, I think there’s enough information to start asking:

Was it worth it? Should we do it again? How should we do it differently?

Was it worth it? Should we do it again?

My short answer is “yes and yes.” But I have some caveats and concerns.

My own goals for the Review were:

Actually identify the best posts
Improve longterm incentive structures and feedback systems

Give users a reason to improve old content

Check our collective epistemic state on controversial posts
Figure out how to evaluate blogposts
Create a shared sense of ownership over LW’s intellectual pipeline
Evaluate LessWrong as a site and project

Some of those (1, 4, 6) I feel able to evaluate independently, others depend on how other people felt about the process, and how much value they got for the effort they put in. It also depends on what the counterfactual actions the Review is being compared to.

But overall, I personally found the process very rewarding. Nominating, reviewing and voting gave me a clearer sense of how various ideas fit together, and what LessWrong had accomplished in 2018. This involved a fair effort on my part (several hours of re-reading, thinking, comparing), but it felt both enjoyable and worthwhile.

Identifying the best posts

I think the review did a decent job at this. The obvious comparison is “what were the top karma posts of 2018?”. Could we have saved ourselves a ton of work by checking that? This is somewhat confounded by the fact that we changed our karma system partway through 2018 (reducing the power of the average upvote, after initially increasing it at the beginning of LW 2.0). At some point we plan to re-run the voting history using the current upvote strengths, which will give us clearer information there.

Meanwhile, comparing the

top karma posts of 2018

to the

top-voted posts in the review

, there are some obvious differences. (Most obviously, “Arbital Postmortem” didn’t end up featured in the review, while being the top-sc

...
My slack budget: 3 surprise problems per week
by Raemon — 2020-02-25T21:52:16.314Z

Previously: Slack

In a couple earlier articles I urged people to adopt strategies that reliably maintain a margin of "30% slack." I've seen lots of people burn out badly (myself included), and preserving a margin of resources such that you don't risk burning out seems quite important to me.

But I realized a) "30% slack" isn't very clear, and b) this is an important enough concept it should really have a top-level post.

So, to be a bit more obvious:

Maintain enough slack that you can absorb 3 surprise problems happening to you in a week, without dipping into reserves.

"Surprise problems" can take multiple forms, and cost different types of reserves. These can be financial expenses you didn't know about (whoops, I needed to buy some medicine), or cognitive attention (whoops, I need to figure out what medicine to buy) or stress (whoops, I'm sick, and now I need to talk to a bunch of doctors while being kinda exhausted).

These can be problems happening to you, or problems happening to friends that you care about.

Why "3 surprises", and not just one? Because at least a couple times a year I personally run into 3-surprises-in-a-week. And sometimes I get hit with much bigger things require me to burn my reserves, and if I didn't have a habit of ensuring my reserves I just wouldn't be able to do those things at all.

I was motivated to write this today because, last week, four problems came up. I had recently taken on two major projects; a community institution was in trouble, and a friend was hurt and needed help. I only had bandwidth to deal with 3 of those. I realized that not only was this a particularly bad week, but I had let too many ongoing responsibilities accumulate.

The weekend came 'round, right as I hit exactly-zero-slack. I needed time to recover, but I had made some time-sensitive commitments to help with some of the above things, and I ended up having to spend the weekend doing a more constrained version of those outstanding obligations, while carefully recovering.

Being Pro-Social Requires Slack

I think an important part of being a good friend, community member or effective altruist, is taking care of yourself. I think if you can't absorb 3 surprise problems per week, it probably make sense to prioritize fixing that above most other things.

"But my friend is in trouble!"

That's legitimately sad, but if you can't absorb 3 surprise problems per week, yo...
Memetic downside risks: How ideas can evolve and cause harm
by MichaelA — 2020-02-25T19:47:18.237Z

This post was written for Convergence Analysis.

Overview

We introduce the concept of memetic downside risks (MDR): risks of unintended negative effects that arise from how ideas “evolve” over time (as a result of replication, mutation, and selection). We discuss how this concept relates to the existing concepts of memetics, downside risks, and information hazards.

We then outline four “directions” in which ideas may evolve: towards simplicity, salience, usefulness, and perceived usefulness. For each “direction”, we give an example to illustrate how an idea mutating in that direction could have negative effects.

We then discuss some implications of these ideas for people and organisations trying to improve the world, who wish to achieve their altruistic objectives and minimise the unintended harms they cause. For example, we argue that the possibility of memetic downside risks increases the value of caution about what and how to communicate, and of “high-fidelity” methods of communication.

Background and concept

Memetics

Wikipedia describes a meme as:

an idea, behavior, or style that spreads by means of imitation from person to person within a culture—often with the aim of conveying a particular phenomenon, theme, or meaning represented by the meme. A meme acts as a unit for carrying cultural ideas, symbols, or practices, that can be transmitted from one mind to another through writing, speech, gestures, rituals, or other imitable phenomena with a mimicked theme.

The same article goes on to say:

Proponents [of the concept of memes] theorize that memes are a viral phenomenon that may evolve by natural selection in a manner analogous to that of biological evolution. Memes do this through the processes of variation, mutation, competition, and inheritance, each of which influences a meme's reproductive success. Memes spread through the behavior that they generate in their hosts. Memes that propagate less prolifically may become extinct, while others may survive, spread, and (for better or for worse) mutate. Memes that replicate most effectively enjoy more success, and some may replicate effectively even when they prove to be detrimental to the welfare of their hosts.

Memetics, in turn, is “the discipline that studies memes and their connections to human and other carriers of them” (Wikipedia).[1]

Downside risks

In an earlier article, we discussed the concept of...
Hoarding and Shortages
by jefftk — 2020-02-25T18:40:02.110Z

One of the main responses to yesterday's post on

preparing for a potential
quarantine was something like:

Hoarding causes shortages. Leave masks for people that need them.

Another commenter made a similar argument with

food.

I think the biggest question here is whether you think there's time
and capacity for producers to react to increased demand. For example,
some mask factories are not running right now because they're in
affected areas, but many others are still running. More people trying
to buy masks raises the market price, which makes it worth it for
these factories to run at higher output. For example, maybe a factory
normally runs a 16hr day with two shifts but doesn't run at night
because it's too expensive to hire people for night work. If masks
are selling for 5x the normal price, the situation is different and
they'll probably start running 24hr. [1]

This is a really big reason to prepare ahead of
time, instead of when a disaster seems likely. I bought masks
months ago after deciding they were something I would like to have on
hand for dealing with a range of issues, which meant producers had
plenty of time to react and make more. I know this doesn't help with
the current situation, but if there are non-perishable things where a
supply disruption would be really bad (ex: masks for taking care of an
immunocompromised family member) set a reminder for six months from
now to stock up on them.

I also don't think spreading the idea that one shouldn't buy masks is
worth it: enough people are still going to be buying them that we're
still going to have a mixture of outages at regular retailers and high
prices at demand-responsive online ones. Even if you think that
there's no ability for producers to react to high prices by creating
more masks and so there's effectively a fixed pool of masks to divide
up, refraining from buying ones for yourself helps others only
minimally.

(Separately, there's a question of what situations masks are useful
for. I know much less about this, though if someone in your household
gets sick it seems like it would be very useful to have them on hand.)

[1] Raising prices in response
to emergency-induced demand is often called "price gouging" and it is
sometimes illegal, though in
long-running emergencies like this I wouldn't expect to see the laws
apply because documenting increased co...
Training Regime Day 11: Socratic Ducking
by Mark Xu — 2020-02-25T17:19:57.320Z

Introduction

As the story goes, there was once a programmer with a bug. They wanted help solving the bug, so they asked a colleague. In the process of explaining the bug to the college, they solved the bug.

The theory is that in the process of explaining the bug, the programmer was forced to unravel parts of their model for how their code worked. In the process of this unraveling, they discovered parts of their model that were false, which led to a solution.

The programmer realized that they could just use a rubber duck instead of a human.

(And they say that programmers are going to be among the last to be automated.)

Socratic ducking is a pair-debugging strategy that attempts to take advantage of this phenomenon.

Socratic Ducking

The virtue of socrates is the ability to ask questions.

The virtue of the rubber duck is the inability to speak.

The virtue of the socratic duck is the inability to speak except to ask clarifying questions.

The process of socratic ducking involves you, the protagonist, finding a partner to be your socratic duck, then talking to them about your bugs. The socratic duck remains mostly silent, except to ask clarifying questions.

Remember, the techniques are not the point, but the bugs are even less of the point than the techniques.

The point of socratic ducking is not to solve the protagonists problems. The point of socratic ducking is to have the protagonist practice their ability to solve their own problems. To this end, the socratic duck should nearly always be silent: a rough guide is asking one question at most every 5 minutes. Even if it what the protagonist doesn't quite make sense to socratic duck, as long as the socratic duck thinks that what the protagonist is saying makes sense to them, then the duck should remain silent.

For instance, if you're being someone's socratic duck, they might say something like "it makes me feel like a spiky sheep". Internally, you might be thinking "what the heck is a spiky sheep", but externally you should nod and act like you know what they're talking about (provided that you do think that "spiky sheep" makes sense to them).

It might be useful to notice the temptation to try and help them solve their bug for them. A way to placate these desires is to have "debrief" at the end of socratic ducking where you can give the protagonist all the object level advice you thought of.

Self-ducking

Epistemic status: h

...
New article from Oren Etzioni
by Aryeh Englander — 2020-02-25T15:25:22.244Z

(Cross-posted from EA Forum.)

This just appeared in this week’s MIT Technology Review: Oren Etzioni, “How to know if AI is about to destroy civilization.” Etzioni is a noted skeptic of AI risk. Here are some things I jotted down:

Etzioni’s key points / arguments:

Warning signs that AGI is coming soon (like canaries in a coal mine, where if they start dying we should get worried)

Automatic formulation of learning problems
Fully self-driving cars
AI doctors
Limited versions of the Turing test (like Winograd Schemas)

If we get to the Turing test itself then it'll be too late

[Note: I think if we get to practically deployed fully self-driving cars and AI doctors, then we will have already had to solve more limited versions of AI safety. It’s a separate debate whether those solutions would scale up to AGI safety though. We might also get the capabilities without actually being able to deploy them due to safety concerns.]

We are decades away from the versatile abilities of a 5 year old
Preparing anyway even if it's very low probability because of extreme consequences is Pascal's Wager

[Note: This is a decision theory question, and I don’t think that's his area of expertise. I’ve researched PW extensively, and it’s not at all clear to me where to draw the line between low probability - high consequence scenarios that we should be factoring into our decisions, vs. very low probability – very high consequence that we should not factor into our decisions. I’m not sure there is any principled way of drawing a line between those, which might be a problem if it turns out that AI risk is a borderline case.]

If and when a canary "collapses" we will have ample time to design off switches and identify red lines we don't want AI to cross
"AI eschatology without empirical canaries is a distraction from addressing existing issues like how to regulate AI’s impact on employment or ensure that its use in criminal sentencing or credit scoring doesn’t discriminate against certain groups."
Agrees with Andrew Ng that it's too far off to worry about now

But he seems to agree with the following:

If we don’t end up doing anything about it then yes, superintelligence would be incredibly dangerous
If we get to human level AI then superintelligence will be very so

...
Other versions of "No free lunch in value learning"
by Stuart_Armstrong — 2020-02-25T14:25:00.613Z

Here are some more general results derived from "Occam's razor is insufficient to infer the preferences of irrational agents":

Regularisation is insufficient to make inverse reinforcement learning work.

Unsupervised learning cannot deduce human preferences; at the very least, you need semi-supervised learning.

Human theory of mind cannot be deduced merely be observing humans.

When programmers "correct an error/bug" in a value-learning system, they are often injecting their own preferences into it.

The implicit and explicit assumptions made for a value-learning system, determine what values the system will learn.

No simple definition can distinguish a bias from a preference, unless it connects with human judgement.

These are all true; the key question is to what extent they are true. Do we need to have minimal supervision, or add minimal assumptions, to get the AI to deduce our values correctly? After all, we do produce a lot of data, that it could use to learn our values, if it gets the basic assumptions right.

Or do we need to put in a lot more work in the assumptions? After all, most of the data we produce is by humans, for humans, who will accept the implicit part of the data, hence there are few explicit labels for the AI to use.

My feeling is that we probably only need a few assumptions - but maybe more than some optimists in this area believe.