trentmkelly/LessWrong-43k huggingface dataset:
The Missing Math of Map-Making
by johnswentworth — 2019-08-28T21:18:25.298Z

Consider the clever arguer from

The Bottom Line

:

Now suppose there is a clever arguer, holding a sheet of paper, and they say to the owners of box A and box B: “Bid for my services, and whoever wins my services, I shall argue that their box contains the diamond, so that the box will receive a higher price.” So the box-owners bid, and box B’s owner bids higher, winning the services of the clever arguer.

The clever arguer begins to organize their thoughts. First, they write, “And therefore, box B contains the diamond!” at the bottom of their sheet of paper. Then, at the top of the paper, the clever arguer writes, “Box B shows a blue stamp,” and beneath it, “Box A is shiny,” and then, “Box B is lighter than box A,” and so on through many signs and portents; yet the clever arguer neglects all those signs which might argue in favor of box A.

This is a great example of a

broken chain

: the chain of cause-and-effect between the actual contents of the boxes and the arguer’s conclusion is broken. With the chain broken, no amount of clever arguing can make the conclusion at the bottom of the paper more true.

Much of

the sequences

can be summarized as “to determine the accuracy of a belief, look at the cause-and-effect process which produced that belief”. A few examples:

The Bottom Line

Asking “Why do I believe what I believe?”
Examining systematic flaws in our belief-producing process, i.e.

biases

Noticing beliefs stemming from inaccurate sources, i.e.

anchoring

,

fictional evidence

,

changing our minds less often than we think

Engines of cognition

If the causal chain from territory to map is broken outright, then that’s a pretty obvious problem, but some of the links above provide more subtle examples too. In general, map-making processes should produce accurate maps

to exactly the extent that they approximate Bayesian reasoning

.

Point is: looking at the cause-and-effect processes which produce maps/beliefs is pretty central to rationality.

Given all that, it’s rather odd that we don’t have a nice mathematical theory of

map-making processes

- cause-and-effect systems which produce accurate maps/beliefs from territories.

We have many of the pieces needed for such a theory laying around already. We have a solid

theory of causality

, so we know how to formalize “cause-and-effect processes”. Information theory lets us quantify

map-territory correspondence

. We even have an intuitive notio...
This sometimes helps to expose assumptions
by MoritzG — 2019-08-28T18:42:57.271Z

If one thing (B) follows from something (A) that is sufficient, then that which follows (B) is a necessity for the something (A). (If that is confusing just keep reading it will be blatantly clear at the end.) A hidden assumption is a necessary condition that was not stated. If it is claimed that something is sufficient but it is not, this must lead to a logical contradiction once a rule is applied that is only applicable if the statement is indeed sufficient. If indeed B follows from A alone, then A can not be without B. But if in reality there is a side condition for B other than A, then A can be without B. This thinking can be used to check the completeness of assumptions. In a world were "(C and D) → E", I might make the (false) claim: "C → E" But from that follows that C can never be true if E is false. Knowing the world you might come up with a case were C can be true without E being true, then you found the counter argument and likely the side condition D that was not met in your counter case. Aside from checking arguments this is useful in checking technical requirements. "In the event of a crash the passenger air bag deploys." Therefore "If the airbag is undeployed there can not have been a crash." Now that is clearly wrong, thus the first statement must have been wrong.
What Programming Language Characteristics Would Allow Provably Safe AI?
by Davidmanheim — 2019-08-28T10:46:32.643Z

It seems clear that many high-level programming languages are candidates for use in the first AGI. They have enough power to write that code. It seems clear, however, that the power that those languages have is incompatible with formal safety. SPARK or OCaml are made so that it is easy to prove correctness, which seems useful, but that's not enough.

For example, we might need memory safety to provide a formal guarantee that the program cannot directly modify the part of memory containing the reward function, or the calculation of the reward. On the other hand, it seems that Turing incompleteness, which allows guaranteeing that a program terminates, would not be necessary.

So - what other (extant or yet-to-be defined) types of language safety will be needed from a language to prevent a hypothetically provably safe AI from being unsafe in practice?
Don't Pull a Broken Chain
by johnswentworth — 2019-08-28T01:21:37.622Z

This essay originally appeared here.

The Emperor’s Nose (Richard Feynman)

A village in a remote corner of an empire decided to erect a statue of the emperor. The village sculptor was hired, but the sculptor had no idea how large to make the emperor’s nose.

The elders conferred. Nobody in the village had ever seen the emperor, nor heard anything about his nose, so they all had wildly different estimates of the size of the emperor’s nose. The elders argued about the right size for hours, until one particularly wise elder stood to address the room.

“Our estimates are too noisy,” the wise elder declared, “In order to improve them, we should employ the wisdom of the crowd. We will ask each person in the village to estimate the length of the emperor’s nose. Then, we can average together the results to obtain an estimate of high precision.”

This proposition was put to a vote, and the elders quickly agreed, eager to end the hours of argument before they missed the early-bird special at the village buffet. The next day, each villager was asked to estimate the length of the emperor’s nose, in millimeters. The elders averaged together all the responses, and estimated the emperor’s nose was 15.49 mm long.

The Cargo Cult (Feynman again)

During World War II, many remote pacific islands became military bases for the American navy. On one such island, the native inhabitants worked with the sailors in exchange for food, clothing and other supplies. All these supplies were flown in plane, and landed on an airstrip.

After the war, the Americans left the empty airstrip behind. The planes stopped delivering supplies.

The natives wanted to get more supplies. So, they tried to make the planes land. They went out to the airstrip and did everything the sailors had done. They lit small, regularly spaced fires along the sides of the airstrip. They had someone sit at the side of the strip talking into a wooden box while wearing pieces of wood shaped to look like headphones. They had others stand on the airstrip and gesture with sticks. In short, they did everything they had seen the sailors do.

But the planes just didn’t land.

The Missing Quarter (Boy Scout tradition)

A boy was pacing back and forth at night under a streetlamp, apparently searching the ground, when another boy passed by.

“What are you looking for?” asked the newcomer.

“My quarter. I dropped it,” replied the searcher.

“Oh. I’ll help y...
Cartographic Processes
by johnswentworth — 2019-08-27T20:02:45.263Z

As a general rule, street maps of New York City do not form spontaneously - they involve some cause-and-effect process which takes in data from the territory (NYC’s streets) and produces the map from that data. Let’s call these “cartographic processes”: causal processes which produce a map from some territory.

Formalizing a bit:

We have a territory and a map. I’m mostly interested in the case where both of these are causal models (possibly

with symmetry

), but other models are certainly possible.
Both the territory and the map are embedded in a larger causal model, the cartographic process, in which the map is generated from the territory. Note that the “territory” may be the entire cartographic process, including the map.
There is some class of queries on the territory which can be “translated” into queries on the map, yielding answers which reliably predict the answers to the corresponding territory-queries - this is what it means for the map to “match” the territory. I’m mostly interested in counterfactual queries on the map, and some preimage of those queries in the territory.

In our NYC streetmap example, the physical streets are the territory, the paper with lines on it is the map, the cartographic process encompasses the map and territory and all the people and equipment and computations which produced the map from the territory, and the class of queries includes things like distance and street connectivity. Note that, in this example, neither the territory nor the map is a causal model, although the cartographic process is a causal model. In general, the cartographic process itself will always be a causal model - accurate maps do not form spontaneously, there is always a cause-and-effect process which creates the map. Part of the reason I’m specifically interested in causal models for the map and territory is because I ultimately want maps of cartographic processes themselves.

For purposes of

embedded agency via abstraction

, we want to answer questions like:

Given a cartographic process, characterize the queries which the map can reliably answer
Given a cartographic process, translate queries on the map into queries on the territory, and vice-versa
Given a class of queries and territories, construct a cartographic process whose output map reliably answers the queries on the territories
Given multiple cartographic processes on the same territory, how can we int

...
Dual Wielding
by Zvi — 2019-08-27T14:10:00.715Z

Previously: More Dakka

Epistemic Status: The Dakka Files

Smartphones are wonderful things.

Get a second one. And get a lot of chargers.

I did this a week ago. My phone (a Google Pixel 2) had some sort of ink leak onto its screen, so I purchased a second one (a Google Pixel 3a) while I attempted to repair the old one. When the repair attempt was successful, but I hadn’t finished transferring some features and data across, I tried carrying around both.

It was clear by the end of the day that the second phone was a huge improvement.

On reflection, this was a basic case of More Dakka. A substantial portion of the people one interacts with worry about running out of phone battery, or running out of phone storage space, or needing to toggle too many things on their phone at once.

This solution solves all these problems right away.

Battery life more than doubles because of your newfound ability to charge one phone while carrying around or using the other. I have not come close to running out on either phone since doubling up. I have also been told that draining the batteries is bad for battery life, so long term this will pay additional dividends. A battery pack is an alternative, but they seem to be similar in size to phones and don’t allow the charging swap tactic, and also lack the accompanying additional benefits of dual wielding.

Storage space effectively doubles as well. Yes, some things must duplicate, but that still means effective storage space is doubled. Which is great, because I otherwise wanted a Google Pixel 3a rather than a 3, and its one issue for me is that it doesn’t give the option of extra storage space.

I can also do things like look up information on one phone while typing it into the other, comparing two things side by side, and other neat stuff like that. There are a number of apps that actively punish you toggling out of them in various ways, which this allows you to avoid.

Only one of the phones needs mobile data. The other can be purely on Wi-Fi. This also solves the problem of needing to know when to swap the Wi-Fi on and off due to speed issues, or to preserve data. It also means you don’t have to pay for an extra line.

It also solved two problems I did not expect it to solve. When my phone in theory has access to mobile data, my podcast app would often cut off access to recordings that were downloaded while it waited for some sort of download or q...
A Personal Rationality Wishlist
by DanielFilan — 2019-08-27T03:40:00.669Z

At one point I compiled a list of conundrums relating to rationality that come up in my life. Instead of solving them, I thought I’d write up a selection of them, since that’s easier and maybe other people will solve them.

Punishing honesty vs no punishment

In some cases, you might want people to comply with some rule that they might otherwise wish to break, but the only way to check if they have complied is to ask them and hope that they’re honest (or perhaps there’s another, much more expensive, way to check). Examples:

A sperm bank might only want donors without congenital abnormalities that they might not be able to easily observe or test for.

I might not want my housemates to go into my room and look at all my stuff when I’m not there.

There’s a dilemma: how should one enforce such a rule? If you just ask people, and punish them if they say that they didn’t comply, then you’re incentivising people to lie to you. But if you don’t ask, the rule doesn’t get enforced. Abstractly, it seems like you just can’t enforce such a rule at all, but it seems to me that often people are able to be honest in the face of punishment, so not all hope is lost. How should I think about these situations? In practice, how should I decide the enforcement mechanism?

According to David Friedman’s recent book on legal systems, in saga-period Iceland, there was a much larger penalty for killing somebody if you failed to confess as soon as was practical. This suggests one solution: estimate the likelihood of discovery of violation of a rule conditioned on the violater being dishonest, and set the punishment of that high enough that it’s worth it for rule violaters to be honest. But this leaves open the question of how in practice to estimate this probability, calculate the appropriate punishment level, and how much effort to put into detection of rule violations when nobody has confessed to a violation.

‘The anime thing’

Once, a friend of mine observed that he couldn’t talk about how he didn’t like anime without a bunch of people rushing in to tell him that anime was actually good and recommending anime for him to watch, even when he explicitly asked them not to. Similarly, another friend of mine went to a coding bootcamp, only to discover that she intensely disliked coding, and would basically be unable to do it as a career, causing her to decide to switch to her previou...
Six AI Risk/Strategy Ideas
by Wei Dai — 2019-08-27T00:40:38.672Z

AI risk ideas are piling up in my head (and in my notebook) faster than I can write them down as full posts, so I'm going to condense multiple posts into one again. I may expand some or all of these into full posts in the future. References to prior art are also welcome as I haven't done an extensive search myself yet.

The "search engine" model of AGI development

The current OpenAI/DeepMind model of AGI development (i.e., fund research using only investor / parent company money, without making significant profits) isn't likely to be sustainable, assuming a soft takeoff, but the "search engine" model very well could be. In the "search engine" model, a company (and eventually the AGI itself) funds AGI research and development by selling AI services, while keeping its technology secret. At some point it achieves DSA either by accumulating a big enough lead in AGI technology and other resources to win an open war against the rest of the world, or by being able to simultaneously subvert a large fraction of all cognition done on Earth (i.e., all the AI services that it is offering), causing that cognition to suddenly optimize for its own interests. (This was inspired by / a reply to Daniel Kokotajlo's Soft takeoff can still lead to decisive strategic advantage.)

Coordination as an AGI service

As a refinement of the above, to build a more impregnable monopoly via network effects, the AGI company could offer "coordination as a service", where it promises that any company that hires its AGI as CEO will efficiently coordinate in some fair way with all other companies that also hire its AGI as CEO. See also my AGI will drastically increase economies of scale.

Multiple simultaneous DSAs under CAIS

Suppose CAIS turns out to be a better model than AGI. Many AI services may be natural monopolies and have a large market share for its niche. Suppose many high level AI services all use one particular low level AI service, that lower level AI service (or rather the humans or higher level AI services that have write access to it) could achieve a decisive strategic advantage by subverting the service in a way that causes a large fraction of all cognition on Earth (i.e., all the higher level services that depend on it) to start optimizing for its own interests. Multiple different lower level services could simultaneously have this option. (This was inspired by a comment from ryan...
Embedded Agency via Abstraction
by johnswentworth — 2019-08-26T23:03:49.989Z

Claim: problems of

agents embedded in their environment

mostly reduce to problems of abstraction. Solve abstraction, and solutions to embedded agency problems will probably just drop out naturally.

The goal of this post is to explain the intuition underlying that claim. The point is not to defend the claim socially or to prove it mathematically, but to illustrate why I personally believe that understanding abstraction is the key to understanding embedded agency. Along the way, we’ll also discuss exactly which problems of abstraction need to be solved for a theory of embedded agency.

What do we mean by “abstraction”?

Let’s start with a few examples:

We have a gas consisting of some huge number of particles. We throw away information about the particles themselves, instead keeping just a few summary statistics: average energy, number of particles, etc. We can then make highly precise predictions about things like e.g. pressure just based on the reduced information we've kept, without having to think about each individual particle. That reduced information is the "abstract layer" - the gas and its properties.
We have a bunch of transistors and wires on a chip. We arrange them to perform some logical operation, like maybe a NAND gate. Then, we throw away information about the underlying details, and just treat it as an abstract logical NAND gate. Using just the abstract layer, we can make predictions about what outputs will result from what inputs. Note that there’s some fuzziness - 0.01 V and 0.02 V are both treated as logical zero, and in rare cases there will be enough noise in the wires to get an incorrect output.
I tell my friend that I'm going to play tennis. I have ignored a huge amount of information about the details of the activity - where, when, what racket, what ball, with whom, all the distributions of every microscopic particle involved - yet my friend can still make some reliable predictions based on the abstract information I've provided.
When we abstract formulas like "1+1=2" or "2+2=4" into "n+n=2n", we're obviously throwing out information about the value of n, while still making whatever predictions we can given the information we kept. This is what abstraction is all about in math and programming: throw out as much information as you can, while still maintaining the core "prediction&q

...
Reversible changes: consider a bucket of water
by Stuart_Armstrong — 2019-08-26T22:55:23.616Z

I've argued that many methods of AI control - corrigibility, amplification and distillation, low impact, etc... - require a partial definition of human preferences to make sense.

One idea I've heard for low impact is that of reversibility - asking how hard it is to move the situation back to its original state (or how now to close down too many options). The "Conservative Agency" paper uses something akin to that, for example.

On that issue, I'll present the bucket of water thought experiment. A robot has to reach a certain destination; it can do so by wading through a shallow pool. At the front of the pool is a bucket of water. The water in the bucket has a slightly different balance of salts than the water in the pool (maybe due to mixing effects when the water was drawn from the pool).

The fastest way for the robot to reach their destination is to run through the pool, kicking the bucket into it as it goes. Is this a reversible action?

Well, it depends on what humans care about the water in the bucket. If we care about the rough quantity of water, this action is perfectly reversible: just dip the bucket back into the pool and draw out the right amount of water. If we care about the exact balance of salts in the bucket, this is very difficult to reverse, and requires a lot of difficult work to do so. If we care about the exact molecules in the bucket, this action is completely irreversible.

The truth is that, with a tiny set of exceptions, all our actions are irreversible, shutting down many possibilities for ever. But many things are reversible in practice, in that we can return to a state sufficiently similar that we don't care about the difference.

But, in order to establish that, we need some estimate of what we care (and don't care) about. In the example above, things are different if we are considering a) this is a bucket of water, b) this is a bucket of water carefully salt-balanced for an industrial purpose, or c) this is the water from the last bath my adored husband took, before he died.

EDIT: I should point out that "avoid the bucket anyway" is not a valid strategy, since "avoid doing anything that could have a large irreversible impact for some utility function" is equivalent with "don't do anything at all".

The robot has to be capable of kicking the bucket in some circumstances. Precisely in those circumstances where humans don't care about the bucket...