trentmkelly/LessWrong-43k huggingface dataset:
(Some?) Possible Multi-Agent Goodhart Interactions
by Davidmanheim — 2018-09-22T17:48:22.356Z

Epistemic Status: I need feedback on these ideas, and I've been delaying because I'm not sure I'm on the right track. This is the product of a lot of thinking, but I'm not sure the list is complete or there isn't something important I'm missing. (Note: This is intended to form a large part of a paper for an article to be submitted to the journal special issue here.)

Following up on Scott Garrabrant's earlier post on Goodhart's Law and the resulting paper, I wrote a further discussion of non-adversarial goodhart, and explicitly deferred discussion of the adversarial case. I've been working on that.

Also note that these are often reformulations or categorizations of other terms (treacherous turn, faulty reward functions, distributional shift, reward hacking, etc.) It might be good to clarify exactly what went where, but I'm unsure.

To (finally) start, here is Scott's "Quick Reference" for the initial 4 methods, which is useful for this post as well. I've partly replaced the last one with the equivalent cases from the Arxiv paper.

Quick Reference

Regressional Goodhart - When selecting for a proxy measure, you select not only for the true goal, but also for the difference between the proxy and the goal.

Model

: When U is equal to V+X, where X is some noise, a point with a large U value will likely have a large V value, but also a large X value. Thus, when U is large, you can expect V to be predictably smaller than U.
Example: height is correlated with basketball ability, and does actually directly help, but the best player is only 6'3", and a random 7' person in their 20s would probably not be as good

Causal Goodhart - When there is a non-causal correlation between the proxy and the goal, intervening on the proxy may fail to intervene on the goal.

Model

: If V causes U (or if V and U are both caused by some third thing), then a correlation between V and U may be observed. However, when you intervene to increase U through some mechanism that does not involve V, you will fail to also increase V.
Example: someone who wishes to be taller might observe that height is correlated with basketball skill and decide to start practicing basketball.

Extremal Goodhart - Worlds in which the proxy takes an extreme value may be very different from the ordinary worlds in which the correlation between the prox

...
Advice Wanted; Reconcile with religious parent
by ChaosHufflepuff — 2018-09-22T09:54:32.784Z

Hi all,

So over the last month or so tension with my father has built up quite a bit. Context is that he is a strong believer in god (through Reform Judaism), and I'm not. He's generally a very open minded person who is cool with people having whatever beliefs they want and having whatever sexual orientation/gender identity they have... so long as they're not his own children.

Sucks for me, being asexual and not believing in god or having any desire to participate in religious services, even on the 'super important' high holidays where every jewish person, regardless of level of belief, supposedly attends a synagogue service.

I'd like to preface this all with the fact that I value my family, and even if we have our differences, my father is an intelligent person who is interesting to have around, and we otherwise have a good relationship that I want to maintain.

Now onto the main content. A little while ago, as the High Holidays were approaching, I informed him that I didn't want to go sit in a room absolutely full of people, not eat for the day, and sit in a room of people praying to whatever they believe in, or praying to whatever they believe they believe in. Not even to keep things smooth with my father, because it's boring and inauthentic for me. I have to have enough of a mask on in day to day life that I don't want to add yet another layer to that (and be super bored and uncomfortable for a good half of a day in a place where it would be considered disrespectful to bring a book and read).

He got very upset about the fact that I wouldn't go, and told me how disappointed, upset and angry he was, and proceeded to ask me where he as a parent had failed such that I became like I am, rather than be a good Jew like him. He also asserted that as a parent it was his right to expect continuity of his beliefs in his children, which is why he is so close-minded with his kids but so open-minded with literally anyone else. Less than a week later he joked about saying that when I moved back to my home city (changing universities), I could find somewhere else to stay. For a time I was worried that this would come along with a financial cut-off, which would be rather problematic for me but overall could be dealt with if it actually happened, but I don't think it will come to that.

He has said that somewhat seriously i...
Annihilating aliens & Rare Earth suggest early filter
by rk — 2018-09-21T15:33:11.603Z

epistemic status: I suspect these thoughts are not novel and are potentially wrong. I still think there's some value in recording my thought process

tl;dr: Selection effects from not being annihilated by aliens may explain why we don't see any very advanced aliens, but it doesn't explain why don't see any as advanced as we are. This may suggest a filter behind us.

Civilisations may sometimes become sufficiently advanced to spread across the stars. Perhaps they would even expand outwards in a near light-speed "shockwave" that converts all available resources into something that they value (like copies of their mind). Perhaps they send seed ships across intergalactic voids. Wherever they go, no new civilisations arise: all matter and energy is used by their civilisation.

Suppose we live in a universe of infinite spatial extent, and that there's some chance of intergalactic civilisation arising in any unoccupied spatiotemporal region. If these civilisations spread out at near light-speed and the chance of civilisational genesis is less than a certainty, there will always be an infinite extent of uncolonised space throughout the history of this universe. The picture below shows an example of such a region in a world where civilisations arise relatively often.

After a little thought, it might appear that we have an anthropic solution to the Fermi Paradox here: of course we don't observe any aliens, for if if we were in their future light cone, the matter we are made of would be used for something else, so we wouldn't exist.

Before we rush ahead with this line of thinking, let's look at a couple of caveats with this idea. The first is that civilisations may exist in states that are observable but not subsuming. I'll specifically consider the case where civilisations start of as observable for a period, then become subsuming. The second is the observed age of the Universe and its implications for this argument.

Visible but not subsuming

On the first, our civilisation has probably been observable for awhile, given the amount and structure of electromagnetic radiation we produce. If we presume that this "young" period is typical for civilisations, we have to explain why we don't observe any young civilisations that wouldn't have subsumed us yet.

Let's take a model where a young civilisation deterministically becomes an intergalactic civilisation over some period τ. Then we

...
Quantum theory cannot consistently describe the use of itself
by avturchin — 2018-09-20T22:04:29.812Z
Link: https://foundations.ethz.ch/wp-content/uploads/2018/04/renner.pdf

As I understand it, the new thought experiment in QM demonstrates that all currently known interpretations of QM are wrong, if we extrapolate Winger friend thought experiment to next level, where different friends are using different theories to interpret each other actions. This may have implication for agent foundations.

Short description of the experiment from Scientific American: "Frauchiger and Renner have a yet more sophisticated version (see ‘New cats in town’). They have two Wigners, each doing an experiment on a physicist friend whom they keep in a box. One of the two friends (call her Alice) can toss a coin and—using her knowledge of quantum physics—prepare a quantum message to send to the other friend (call him Bob). Using his knowledge of quantum theory, Bob can detect Alice’s message and guess the result of her coin toss. When the two Wigners open their boxes, in some situations they can conclude with certainty which side the coin landed on, Renner says—but occasionally their conclusions are inconsistent. “One says, ‘I’m sure it’s tails,’ and the other one says, ‘I’m sure it’s heads,’” Renner says."

Couple of quotes from the article which are related to agents:

"Suppose that a casino offers the following gambling game. One round of the experiment is played, with the gambler in the role of W, and the roles of F¯, F, and W¯ taken by employees of the casino. The casino promises to pay $ 1.000 to the gambler if F¯’s random value was r = heads. Conversely, if r = tails, the gambler must pay $ 500 to the casino. It could now happen that, at the end of the game, w = ok and ¯w = ok, and that a judge can convince herself of this outcome. The gambler and the casino are then likely to end up in a dispute, putting forward arguments taken from Table II.

Gambler: “The outcome w = ok implies, due to s ¯F Q, that r = heads, so the casino must pay me $ 1.000.”

Casino: “The outcome ¯w = ok proves, by virtue of s W¯ Q, that our employee observed z = +1 2 . This in turn proves, by virtue of s F Q, that r = tails, so the gambler must pay us $ 500.”

"Theorem 1 now asserts that (Q) and (S) are already in conflict with the idea that agents can consistently reason about each other, in the sense of (C)."
Memetic Tribes and Culture War 2.0
by  — 2018-09-20T17:58:01.862Z
Link: https://medium.com/intellectual-explorers-club/memetic-tribes-and-culture-war-2-0-14705c43f6bb

Long, but absolutely worth the read IMHO. Brings together many aspects of today's cultural crisis, explains both its origins and effects, and proposes solutions.
Against the barbell strategy
by Benquo — 2018-09-20T15:19:08.185Z
Link: http://benjaminrosshoffman.com/against-barbell-strategy/

Nassim Nicholas Taleb recommends that instead of the balanced portfolio of investments recommended by portfolio theory, we follow a "barbell" strategy of putting most of our assets in a maximally safe, stable investment, and making small, sustainable bets with very high potential upside. If taken literally, this can't work because no such safe asset class exists.

Taleb typically uses US treasury bonds as an example of an extremely safe asset. This can only be true metaphorically. They are extremely safe if your measure of value is denominated in US dollars. But of what use to me are US treasury bonds in a hurricane, or if the US dollar collapses? Of what use if people like me become targets for state persecution, stripped of our financial assets or ability to access them? Of what use if I'm trying to deal with a spiritual or health problem and don't know how to find competent advice? If I'm trying to feed myself and don't know how to distinguish profit-seeking propaganda or engineered taste from genuine information about which foods are healthful?

Risks, and opportunities, are many-dimensional. And along very few dimensions is there anything like the sort of crystalline perfection of a treasury bond. A personal network extending to a variety of fields can do a lot to help you avoid getting scammed, and get access to at least an ordinary standard of competence, such as it is. A friend with different interests and skills from yours will do a lot to expand the access you have, and the skills and knowledge at your disposal. Friends from different cultures or communities can serve as an important hedge if you and people like you become a target of extraction or persecution for some reason, or simply if your way of life becomes unsustaintable.

Land you are used to occupying and know well is easier to robustly possess and develop than a rented apartment (but the upside is often higher in high-density areas, which is why young people often prefer them). A spouse makes it easier to pool assets into a venture diversified between building up a single household and seeking external trade or mercenary opportunities. And children who grow up well-adjusted to the world can be helpful if you need to navigate a changing incentive landscape when you've already committed your assets to a bunch of permanent bets (which you need to do, since ...
In Logical Time, All Games are Iterated Games
by abramdemski — 2018-09-20T02:01:07.205Z

Logical Time

The main purpose of this post is to introduce the concept of logical time. The idea was mentioned in Scott's post, Bayesian Probability is for things that are Space-like Separated from You. It was first coined in a conference call with, Daniel Demski, Alex Mennan, and perhaps Corey Staten and Evan Lloyd -- I don't remember exactly who was there, or who first used the term. Logical time is an informal concept which serves as an intuition pump for thinking about logical causality and phenomena in logical decision theory; don't take it too seriously. In particular, I am not interested in anybody trying to formally define logical time (aside from formal approaches to logical causality). Still, it seems like useful language for communicating decision-theory intuitions.

Suppose you are playing chess, and you consider moving your bishop. You play out a hypothetical game which results in your loss in several moves. You decide not to move your bishop as a result of this. The hypothetical game resulting in your loss still exists within logic. You are logically later than it, in that the game you actually play depends on what happened in this hypothetical game.

Suppose you're stuck in the desert in a Parfit's Hitchhiker problem. Paul Ekman is reading your face, deciding whether you're trustworthy. Paul Ekman does this based on experience, meaning that the computation which is you has a strong similarity with other computations. This similarity can be used to predict you fairly reliably, based on your facial expressions. What creates this similarity? According to the logical time picture, there is a logical fact much earlier in logical time, which governs the connection between facial expressions and behavior.

To the extent that agents are trying to predict the future, they can be thought of as trying to place themselves later in logical time than the events which they're trying to predict. Two agents trying to predict each other are competing to see who can be later in logical time. This is not necessarily wise; in games like chicken, there is a sense in which you want to be earlier in logical time.

Traditional game theory, especially Nash equilibria, relies on what amounts to loopy logical causality to allow each agent to be after the other in logical time. Whether this is bad depends on your view on logical time travel. Per...
Book review: Why we sleep
by Richard_Ngo — 2018-09-19T22:36:19.608Z
Link: http://thinkingcomplete.blogspot.com/2018/08/book-review-why-we-sleep.html

I read this book (by a sleep scientist called Matthew Walker) because I knew that it would tell me to sleep more, and I hoped it would cite enough scary statistics that I'd be likely to actually follow through. Well, it worked - I'm keeping a copy on my bedside table for the foreseeable future, just as a reminder. In addition to the exhortations to get more sleep, it contains a variety of other interesting and important facts about sleep.

What is sleep?

Human sleep consists of cycles lasting about 1.5 hours, each of which contains first a period of NREM (Non-Rapid Eye Movement) sleep, then a period of REM sleep. In brain scans, the former consists of slow, deep brain waves, while the latter shows the same frenetic activity as an awake brain. As the night goes on, cycles feature a higher proportion of REM sleep. This means that if you cut your sleep short by 25%, you're actually missing out on somewhere between 60% and 90% of REM sleep.
REM sleep is when the majority of dreams happen. While it's uncommon for dreams to replay events from our everyday lives, they do often reflect our emotional preoccupations. To prevent ourselves from flailing around during dreams, we enter a state of sleep paralysis, where our brains are unable to control our voluntary muscles. Eyes are an exception - hence the name REM. It's definitely not true that REM is the only valuable type of sleep - in fact, immediately after sleep deprivation the brain prioritises catching up on NREM.
The slow waves of NREM sleep are useful for transferring memories from one part of the brain to the other - in particular, from short- to long-term storage.
Walker's theory is that NREM sleep is used to prune away unnecessary connections, while REM reinforces useful connections. He uses the analogy of a sculptor who alternates between carving away whole chunks of marble (NREM) and then adding fine detail on whatever's left (REM). From this perspective, it makes sense that REM sleep is concentrated in later cycles. However, it's unclear whether this is the scientific consensus.
There are two systems controlling sleep and wakefulness. The circadian system follows the day/night cycle, making you tired in the evening and alert in the morning (the exact timings vary by person, making some people "night owls" and some "morning larks"). In addition, "

...
Bridging syntax and semantics, empirically
by Stuart_Armstrong — 2018-09-19T16:48:32.436Z

EDIT: I've found that using humans to detect intruders, is a more illustrative example than the temperature example one of this post. The intruder example can be found in this post.

This is a series of posts with the modest goal of showing on how you can get syntax from semantics, solve the grounding problem, and start looking for models within human brains.

I think much of the work in this area has been focusing on the wrong question, looking at how symbols might be grounded in theory, rather that whether a particular symbol is well grounded for a particular concept. When Searle argues against a thermostat having beliefs about temperature, what is actually happening is that the thermostat's internal variables correlate poorly with temperature in general environments.

So, I'll start by presenting a derisively simple solution to the symbol grounding problem, and then see what this means in practice:

The variable X within agent A is a symbol for variable x in the set of environments E, iff knowing X allows one to predict x well within E.

This could be measured, for example, by high mutual information I(X,x) between the variables, or low conditional entropy H(x∣X).

Why do I mention the set E? It's because any claim that X is a symbol of x will almost always include an example in which that is the case. Then those arguing against that claim will often produce another environment in which X fails to correlate with x, thus showing that the agent didn't have a "genuine" understanding of x. So lack of understanding is often demonstrated by error, which is an empirical standard. Thus keeping track of the environments that cause error - and those that don't - is important.

Variables that always move together

If you trained a neural net on images of black cats versus white dogs, you might think you're training an animal classifier, when you're really training an colour classifier. According to the definition above, the output variable of the neural net, in the training environment, counts as both a "symbol" for "black" and a symbol for "cat". But which is it?

That question has no real meaning in the training environment. We can label that variable "cat", or "black", or "mix of blackness and catness", and all are equally good. This might seem like a cheat - but remember that within the training environment, there is no such thing as a non-black cat or a non-cat black objec

...
Web of connotations: Bleggs, Rubes, thermostats and beliefs
by Stuart_Armstrong — 2018-09-19T16:47:39.673Z

This is a series of posts with the modest goal of showing on how you can get syntax from semantics, solve the grounding problem, and start looking for models within human brains.

Before getting to the meat of the approach, I'll have to detour, Eliezer-like, into a few background concepts. They'll be presented in this post. The overall arc of the series is that, once we understand how syntax and semantics can fail to match up, then checking if they do becomes easy.

Old arguments: does a thermostat have beliefs?

I'd like to start with Searle. He's one of the most annoying philosophers, because he has genuine insights, but since he focuses mainly on the Chinese room thought experiment - which is an intuition pump with no real content behind it - everyone ignores these.

McCarthy argued:

Machines as simple as thermostats can be said to have beliefs, and having beliefs seems to be a characteristic of most machines capable of problem solving performance

Searle counter-argues:

Think hard for one minute about what would be necessary to establish that that hunk of metal on the wall over there had real beliefs: beliefs with direction of fit, propositional content, and conditions of satisfaction; beliefs that had the possibility of being strong beliefs or weak beliefs; nervous, anxious, or secure beliefs; dogmatic, rational, or superstitious beliefs; blind faiths or hesitant cogitations; any kind of beliefs. The thermostat is not a candidate.

This seems like it's just a definition problem, similar to the discussion of whether something is a Rube or a Blegg. Searle lists the rich and varied properties of human beliefs, and contrasts them with McCarthy's stripped down interpretation of belief. It seems that we could just call the first "human-like beliefs", and the second "simplified beliefs", and we would have dissolved the question.

But I'd argue that the implicit question is actually more complicated than that, and that Searle was, in this instance, correct.

The implicit part of complex definitions

Bleggs and rubes are defined by only five characteristics:

Let's have a look at two more complicated examples: is this object a chair? Is person X good (and what is goodness anyway)?

Chairs

For chairs, we have a rough intensional definition that goes something like "something humans sit on", and a huge amount of mental examples of chairs from our experience (extensional defi

...
LW Update 2018-09-18 – Email Subscriptions for Curated
by Raemon — 2018-09-19T00:30:57.974Z

Subscribe to Curated via RSS or Email – We added a system to subscribe to things via email. So far we've only implemented it for the curated feed, but we'll soon be implementing it for other things.

Private Messages – We overhauled the PM styling so that it didn't look so janky. Unfortunately, it still doesn't work properly on mobile (I think this was always the case but before it looked awkward enough that you weren't tempted to use it). We have a fix in the works to make mobile work but it's not quite ready yet.

Styling Refactor Part I complete – We completed the parts of our styling refactor that were necessary to cleanly restyle the AI Alignment Forum and EA Forum.