trentmkelly/LessWrong-43k huggingface dataset:
The Goodhart Game
by John_Maxwell — 2019-11-18T23:22:13.091Z

In this paper, we argue that adversarial example defense papers have, to date, mostly considered abstract, toy games that do not relate to any specific security concern. Furthermore, defense papers have not yet precisely described all the abilities and limitations of attackers that would be relevant in practical security.

From the abstract of Motivating the Rules of the Game for Adversarial Example Research by Gilmer et al (summary)

Adversarial examples have been great for getting more ML researchers to pay attention to alignment considerations. I personally have spent a fair of time thinking about adversarial examples, I think the topic is fascinating, and I've had a number of ideas for addressing them. But I'm also not actually sure working on adversarial examples is a good use of time. Why?

Like Gilmer et al, I think adversarial examples are undermotivated... and overrated. People in the alignment community like to make an analogy between adversarial examples and Goodhart's Law, but I think this analogy fails to be more than an intuition pump. With Goodhart's Law, there is no "adversary" attempting to select an input that the AI does particularly poorly on. Instead, the AI itself is selecting an input in order to maximize something. Could the input the AI selects be an input that the AI does poorly on? Sure. But I don't think the commonality goes much deeper than "there are parts of the input space that the AI does poorly on". In other words, classification error is still a thing. (Maybe both adversaries and optimization tend to push us off the part of the distribution our model performs well on. OK, distributional shift is still a thing.)

To repeat a point made by the authors, if your model has any classification error at all, it's theoretically vulnerable to adversaries. Suppose you have a model that's 99% accurate and I have an uncorrelated model that's 99.9% accurate. Suppose I have access to your model. Then I can search the input space for a case where your model and mine disagree. Since my model is more accurate, ~10 times out of 11 the input will correspond to an "adversarial" attack on your model. From a philosophical perspective, solving adversarial examples appears to be essentially equivalent to getting 100% accuracy on every problem. In the limit, addressing adversarial examples in a fully satisfactory way looks a bit like so...
Self-Fulfilling Prophecies Aren't Always About Self-Awareness
by John_Maxwell — 2019-11-18T23:11:09.410Z

This is a belated follow-up to my Dualist Predict-O-Matic post, where I share some thoughts re: what could go wrong with the dualist Predict-O-Matic.

Belief in Superpredictors Could Lead to Self-Fulfilling Prophecies

In my previous post, I described a Predict-O-Matic which mostly models the world at a fuzzy resolution, and only "zooms in" to model some part of the world in greater resolution if it thinks knowing the details of that part of the world will improve its prediction. I considered two cases: the case where the Predict-O-Matic sees fit to model itself in high resolution, and the case where it doesn't, and just makes use of a fuzzier "outside view" model of itself.

What sort of outside view models of itself might it use? One possible model is: "I'm not sure how this thing works, but its predictions always seem to come true!"

If the Predict-O-Matic sometimes does forecasting in non-temporal order, it might first figure out what it thinks will happen, then use that to figure out what it thinks its internal fuzzy model of the Predict-O-Matic will predict.

And if it sometimes revisits aspects of its forecast to make them consistent with other aspects of its forecast, it might say: "Hey, if the Predict-O-Matic forecasts X, that will cause X to no longer happen". So it figures out what would actually happen if X gets forecasted. Call that X'. Suppose X != X'. Then the new forecast has the Predict-O-Matic predicting X and then X' happens. That can't be right, because outside view says the Predict-O-Matic's predictions always come true. So we'll have the Predict-O-Matic predicting X' in the forecast instead. But wait, if the Predict-O-Matic predicts X', then X'' will happen. Etc., etc. until a fixed point is found.

Some commenters on my previous post talked about how making the Predict-O-Matic self-unaware could be helpful. Note that self-awareness doesn't actually help with this failure mode, if the Predict-O-Matic knows about (or forecasts the development of) anything which can be modeled using the outside view "I'm not sure how this thing works, but its predictions always seem to come true!" So the problem here is not self-awareness. It's belief in superpredictors, combined with a particular forecasting algorithm: we're updating our beliefs in a cyclic fashion, or hill-climbing our story of how the future will go until the story seems plausible, ...
The new dot com bubble is here: it’s called online advertising
by Gordon Seidoh Worley — 2019-11-18T22:05:27.813Z
Link: https://thecorrespondent.com/100/the-new-dot-com-bubble-is-here-its-called-online-advertising/13228924500-22d5fd24

If you want to understanding Goodharting in advertising, this is a great article for that.

At the heart of the problems in online advertising is selection effects, which the article explains with this cute example:

Picture this. Luigi’s Pizzeria hires three teenagers to hand out coupons to passersby. After a few weeks of flyering, one of the three turns out to be a marketing genius. Customers keep showing up with coupons distributed by this particular kid. The other two can’t make any sense of it: how does he do it? When they ask him, he explains: "I stand in the waiting area of the pizzeria."

It’s plain to see that junior’s no marketing whiz. Pizzerias do not attract more customers by giving coupons to people already planning to order a quattro stagioni five minutes from now.

The article goes through an extended case study at eBay, where selection effects were causing particularly expensive results without anyone realizing it for years:

The experiment continued for another eight weeks. What was the effect of pulling the ads? Almost none. For every dollar eBay spent on search advertising, they lost roughly 63 cents, according to Tadelis’s calculations.

The experiment ended up showing that, for years, eBay had been spending millions of dollars on fruitless online advertising excess, and that the joke had been entirely on the company.

To the marketing department everything had been going brilliantly. The high-paid consultants had believed that the campaigns that incurred the biggest losses were the most profitable: they saw brand keyword advertising not as a $20m expense, but a $245.6m return.

The problem, of course, is Goodharting, by trying to optimize for something that's easy to measure rather than what is actually cared about:

The benchmarks that advertising companies use – intended to measure the number of clicks, sales and downloads that occur after an ad is viewed – are fundamentally misleading. None of these benchmarks distinguish between the selection effect (clicks, purchases and downloads that are happening anyway) and the advertising effect (clicks, purchases and downloads that would not have happened without ads).

And unsurprisingly, there's an alignment problem hidden in there:

It might sound crazy, but companies are not equipped to assess whether their ad spending actually makes money.

...
The Value Definition Problem
by Sammy Martin — 2019-11-18T19:56:43.271Z

How to understand non-technical proposals

This post grew out of conversations at EA Hotel, Blackpool about how to think about the various proposals for ‘solving’ AI Alignment like CEV, iterated amplification and distillation or ambitious value learning. Many of these proposals seemed to me to combine technical and ethical claims, or to differ in the questions they were trying to answer in confusing ways. In this post I try to come up with a systematic way of understanding the goals of different high-level AI safety proposals, based on their answers to the Value Definition Problem. Framing this problem leads to comparing various proposals by their level of Normative Directness, as defined by Bostrom in Superintelligence. I would like to thank

Linda Linsefors

and

Grue_Slinky

for their help refining these ideas, and EA Hotel for giving us the chance to discuss them.

Defining the VDP

In Superintelligence (2014) Chapter 14, Bostrom discusses the question of ‘what we should want a Superintelligence to want’, defining a problem;

“Supposing that we could install any arbitrary value into our AI, what should that value be?”

The Value Definition Problem

By including the clause ‘supposing that we could install any arbitrary value into our AI’, Bostrom is assuming we have solved the full Value Loading Problem and can be confident in getting an AGI to pursue any value we like.

Bostrom’s definition of this ‘deciding which values to load’ problem is echoed in other writing on this topic. One proposed answer to this question, the Coherent Extrapolated Volition (CEV) is described by Yudkowsky as

‘a proposal about what a sufficiently advanced self-directed AGI should be built to want/target/decide/do’

.

With the caveat that this is something you should do ‘with an extremely advanced AGI, if you're extremely confident of your ability to align it on complicated targets’.

However, if we only accept the above as problems to be solved, we are being problematically vague. Bostrom explains why in Chapter 14. If we really can ‘install any arbitrary value into our AI’, we can simply require the AI to ‘do what I mean’ or ‘be nice’ and leave it at that. If an AGI successfully did “want/target/decide to do what I meant”, then we would h...
Fluid Decision Making
by Gordon Seidoh Worley — 2019-11-18T18:39:57.878Z
Link: https://mapandterritory.org/fluid-decision-making-d8e4e41a51c0

NB: Originally published on Map and Territory on Medium. This is an old post originally published on 2016-10-04. It was never previously cross-posted or linked on LessWrong, so I'm adding it now for posterity. It's old enough that I can no longer confidently endorse it, and I won't bother trying to defend it if you find something wrong, but it might still be interesting.

A lot of folks these days talk about “flow” to mean some kind of mystical state where they experience something like automatic decision making where they get out of their own way and just do. Less mysteriously, other folks use “flow” to describe periods of focused attention. More mysteriously, some folks talk about something similar but as the Daoist idea of action through non-action. Let’s see if we can make some sense of what’s going on here.

As far as I can tell we talk about flow because Daoist philosophy explains virtuous behavior (de) as being a mind like water. Chapter 78 of the Daodejing reads:

Nothing in the world
is as soft and yielding as water.
Yet for dissolving the hard and inflexible,
nothing can surpass it.

The soft overcomes the hard;
the gentle overcomes the rigid.
Everyone knows this is true,
but few can put it into practice.

Therefore the Master remains
serene in the midst of sorrow.
Evil cannot enter his heart.
Because he has given up helping,
he is people’s greatest help.

True words seem paradoxical.

And Chapter 15 reads:

The ancient Masters were profound and subtle.
Their wisdom was unfathomable.
There is no way to describe it;
all we can describe is their appearance.

They were careful
as someone crossing an iced-over stream.
Alert as a warrior in enemy territory.
Courteous as a guest.
Fluid as melting ice.
Shapable as a block of wood.
Receptive as a valley.
Clear as a glass of water.

Do you have the patience to wait
till your mud settles and the water is clear?
Can you remain unmoving
till the right action arises by itself?

The Master doesn’t seek fulfillment.
Not seeking, not expecting,
she is present, and can welcome all things.

Though this is not necessarily the direct lineage of the modern usage, this gives a sense of what metaphors of being liquid-like are trying to imply. In flow a person acts naturally, takes the most direct path, and problems yield before them. It’s reported to feel from the inside like achieving withou...
Internalizing Existentialism
by Gordon Seidoh Worley — 2019-11-18T18:37:18.606Z
Link: https://mapandterritory.org/internalizing-existentialism-72831ef04735

NB: Originally published on Map and Territory on Medium. This is an old post originally published on 2016-09-18. It was never previously cross-posted or linked on LessWrong, so I'm adding it now for posterity. It's old enough that I can no longer confidently endorse it, and I won't bother trying to defend it if you find something wrong, but it might still be interesting.

Over the last couple months, due to reading Daoist philosophical texts, I’ve come to deeply internalize something I’ve known for a long time: morality doesn’t exist “out there” in reality and is instead a construct of our preferences and the dialectic between different people’s preferences.

If you stumbled upon this and didn’t realize morality wasn’t essential, well, um, I’m not going to try to convince you of that. Probably a not terrible reading recommendation is the Less Wrong series on metaethics.

I started down the path to giving up an internal sense of essential morality when meditating on the Daoist position that there is fundamentally no differentiation. For example, chapter 41 of the Daodejing reads in part:

Thus it is said:
The path into the light seems dark,
the path forward seems to go back,
the direct path seems long,
true power seems weak,
true purity seems tarnished,
true steadfastness seems changeable,
true clarity seems obscure,
the greatest are seems unsophisticated,
the greatest love seems indifferent,
the greatest wisdom seems childish.

And in chapter 20 we find:

Stop thinking, and end your problems.
What difference between yes and no?
What difference between success and failure?
Must you value what others value,
avoid what others avoid?
How ridiculous!

And in both the texts of Zhuangzi and Liezi we are given multiple stories where beauty and good acts do not lead to happiness and ugliness and wickedness do not hinder virtue. On the surface we are given contradictions, but by looking deeper the contradictions dissolve if we perceive that the dichotomy is false.

Even speaking of virtue is itself an interesting case. The word used in Chinese, 德 or de, means virtue with a moralistic component in normal use just as is found in English, but de also has a meaning of step and shares with virtue’s Latin roots in meaning strength or capacity. So even here we find, when it looks as though we are being given moral ...
A Foundation for The Multipart Psyche
by Gordon Seidoh Worley — 2019-11-18T18:33:20.925Z
Link: https://mapandterritory.org/a-foundation-for-the-multipart-psyche-79a66292a7a

NB: Originally published on Map and Territory on Medium. This is an old post originally published on 2016-09-14. It was never previously cross-posted or linked on LessWrong, so I'm adding it now for posterity. It's old enough that I can no longer confidently endorse it, and I won't bother trying to defend it if you find something wrong, but it might still be interesting.

In a recent post Scott Alexander gives a review of some recent results in neurobiology that suggest a powerful, unifying set of mechanisms for how information is integrated in the brain. I recommend you read his article and the original research if you can, but I’ll summarize it briefly.

There are various chemicals regulating activity in the brain. There is now evidence that these chemicals act in coordinating an information pump in the brain. Change the chemicals and you change the parameters of the information pump. It seems specifically the information pump in play fits the Bayesian model in that certain chemicals regulate the presentation of prior evidence, others new evidence, and yet others confidence in those evidences.

What I find compelling is that the model described provides a plausible mechanism by which the theory of a 2-part psyche might work. There are several two-part theories of psyche, that is to say theories of how mental processes are organized. My preferred one is near/far construal theory, but there is also the S1/S2 distinction, the fast/slow distinction, in Chinese philosophy yin and yang, and even the hot/cold blood model from medieval European thought. Each of these acts as a way of classifying thoughts and behaviors along a spectrum between two extremes.

The interesting thing about the 2-part psyche theories, and why I prefer the near/far distinction, is that they all seem to operate along the same dimension. Near/far uses the metaphor of distance (because it happens we use similar reasoning patterns when working with things that are physically near versus far) to differentiate between things that are heavy on details and light on patterns versus those that are heavy on patterns and light on details. S1/S2 uses basically the same dimension, as does fast/slow, yin/yang, and hot/cold: stuff with lots of details is near, fast, yin, hot, and part of S1 while stuff with less details and stronger patterns are far, slow, yang, cold, and part of S2. This ...
In Defense of Kegan
by Gordon Seidoh Worley — 2019-11-18T18:27:37.237Z
Link: https://mapandterritory.org/in-defense-of-kegan-2ed7ab51b4c8

NB: Originally published on Map and Territory on Medium. This is an old post originally published on 2016-09-10. It was never previously cross-posted or linked on LessWrong, so I'm adding it now for posterity. It's old enough that I can no longer confidently endorse it, and I won't bother trying to defend it if you find something wrong, but it might still be interesting.

I find Kegan’s model of psychological development extremely useful. Some folks I know disagree on various grounds. These are some accumulated responses to critiques I’ve encountered.

Before we dive into these critiques, though, allow me to attempt a brief introduction to the theory (though this is a tough undertaking, as we’ll discuss below). Robert Kegan, later along with Lisa Lahey, put forward a theory of developmental psychology rooted in complexity of meaning making. It is influenced and builds on the work of Piaget, but also Erikson and Kohlberg, who extended developmental psychology to consider the possibility of adult development.

Kegan’s theory focuses on the maximally complex models people can make of the world in near construal mode. Development is along a continuous gradient but with clear “levels” where a particular kind of complexity is fully available to the thinker. These are classified from 1 to 5 and can be summarized in many ways, though fundamentally they correspond to when a person can form fully-articulable models of things, relationships between things, systems, relationships between systems, and systems of systems (holons), respectively.

This is an exceedingly dense introduction, though, and I know of no good short explanation. The best resources on the topic remain Kegan’s seminal The Evolving Self and his later In Over Our Heads for a more approachable, example-laden presentation.

The first, and perhaps strongest, critique of Kegan is that it’s very hard for anyone to explain it. Kegan begins In Over Our Heads with the story of getting a letter from a student assigned to read The Evolving Self. The student writes that The Evolving Self is full of interesting ideas but gets so frustrated trying to make sense of it that he wants to “punch [Kegan] in the teeth”.

Partly this is because of Kegan has a strong literary and classics background, so The Evolving Self is full of very precise languag...
Comment, Don't Message
by jefftk — 2019-11-18T16:00:02.010Z

Most of the time, when people have responses to my posts they write them as
comments. Sometimes, however, they email or send messages. Since I strongly
prefer comments I wanted to write some about why.

A discussion we have in comments will be open to people other than the
two of us. People who read the post and are thinking along similar
lines can see our back-and-forth. The comments will be attached to
the post, potentially clarifying things for people who come across the
post later. If the question comes up again I can link someone to our
discussion of it. The comments show up in search engines for
unrelated people interested in these ideas. Since communicating in
public has all these positive externalities, I'm much more willing to
put time and thought into a discussion if it can be public.

There are also benefits during the discussion, as other people often
have valuable things to add. Many times a comment thread has been me
and someone else, and a third person jumps in with an important
consideration that hadn't occurred to either of us. Other times
someone's comment sparks a thread which brings in perspectives from
many people. It's not just that our talking in public helps others,
but it helps us too.

More selfishly, comments have different expectations around responses. I read
every comment, but I often don't reply. Maybe the comment is self contained,
and while it's communicating something important a reply wouldn't add anything.
Maybe I'm not sure how I'd like to respond yet, and then don't end up coming
back to it. Maybe other people responded and it seems like the important
details have come out already. Maybe I just don't have time. With one-on-one
messages, however, the response burden is much higher. Just writing back
"received and read" would be hostile, but writing a good response can be a lot
of work. Sending a message should not generally obligate a reply, but it still
feels rude not to put in the time for a thorough response.

There are valid reasons for non-public communication. Perhaps you're
afraid of how people might respond to revealing details of your
identity or taking an unpopular position. Perhaps you want to talk
about something that is illegal but generally viewed as ok among your
friends. Perhaps there are people who follow you around the internet
harassing you. Perhaps you don't trust yours...