trentmkelly/LessWrong-43k huggingface dataset:
Being legible to other agents by committing to using weaker reasoning systems
by AlexMennen — 2017-12-03T07:49:13.000Z

Suppose that an agent A1 reasons in a sound theory T1, and
an agent A2 reasons in a theory T2, such that T1
proves that T2 is sound. Now suppose A1 is trying to reason
in a way that is legible to A2, in the sense that A2 can
rely on A1 to reach correct conclusions. One way of doing this
is for A1 to restrict itself to some weaker theory T3,
which T2 proves is sound, for the purposes of any reasoning
that it wants to be legible to A2. Of course, in order for this
to work, not only would A1 have to restrict itself to using
T3, but A2 would to trust that A1 had done so. A
plausible way for that to happen is for A1 to reach the decision
quickly enough that A2 can simulate A1 making the decision
to restrict itself to using T3.

Example application #1: An agent communicating with its past and
future selves. In this scenario, consider an agent that designs successors
it can trust, and wants to be legible to its successors.

Let T be some recursively axiomatizable first-order theory of arithmetic
which is powerful enough to decide the important questions the agent
wants to know the answer to that don't involve reasoning about agents
(Peano Arithmetic, for instance). For a computable ordinal α,
let T+α refer to T extended with a soundness schema for
T+β for every β<α. So T+0=T, T+(α+1)=T+Sound(T+α),
and T+γ=⋃α<γT+α for computable limit
ordinals γ. Consider an agent initially (at time 0) reasoning
with T+α0 for some large computable ordinal α0.
At each time step, the agent must design a successor that it can prove
uses sound reasoning. So at each time-step t (until αt=0),
the agent can pick αt+1<αt and have its successor
reason with T+αt+1. Of course, this cannot go on forever, but there are astronomically long decreasing sequences of ordinals
starting at ordinals that are straightforward to describe, and which
require a very minimal amount of computation to get from one ordinal
in the sequence to the next (for instance, if you're familiar with
the proof that the Hydra game is impossible to lose, the sequence
of ordinals below ε0 corresponding to the sequence
of states of some Hydra if Hercules is using some simple strategy),
so the lack of such infinite sequences does not seem like a serious
obstacle. The sequence can be made arbitrarily long, and the agent
can procrastinate arbitrarily long on giving an upper bound for how
long the sequence needs to be, and can

...
Philosophy of Numbers (part 1)
by Charlie Steiner — 2017-12-02T18:20:30.297Z

This post is the first in a series of things that I think would be fun to discuss on LW. Part two is here.

It seems like there are (at least) two kinds of things we make statements about: physical things, like apples or cities, and logical things, like numbers or logical relations. And it's pretty interesting to question how accurate this seeming is. Are numbers really a "kind of thing," and what do we mean by that anyways? Can we unify these multiple kinds of things, or kinds of statements, into one kind, or not?

For a light review of standard answers, see this nice video. For more depth, you might see the SEP on abstract objects or philosophy of mathematics.

Compare the statements "There exists a city larger than Paris" versus "There exists a number greater than 17." It seems like we use much the same thought patterns to evaluate both these statements, and both seem to be true in the same ordinary sense. Yet the statement about cities seems true because of a correspondence to the external world, but there is no "17" object in a parsimonious predictive model of the world.

To this you might say, "What's the big deal? Even if I don't think numbers are physical objects, it's perfectly reasonable to make this tight analogy between cities and numbers in our reasoning. How is making a big issue out of this going to help us do anything practical?"

Well, in logical decision theory, a recent formulation of some ideas from TDT/UDT, the agent wants to make a causal model of the world that includes (in the model) "causal" effects of a fixed mathematical statement (speficially, the output of the agent's own algorithm). First of all, this is pretty novel and we don't really know how to formalize learning such a model. Second, it's pretty philosophically weird - how is a piece of math supposed to have something like a causal effect on trees and rocks? If we want to solve the practical problem, it might help to be less confused about numbers.

Plus, you know, it's interesting! Why do we think there's such a thing as "numbers," how come the same reasoning works for both numbers and cities, and what are the limits to this analogy, if any?

When one wants to outdo an entire branch of philosophy, it's nice to have some sort of advantage. And the sign of such an advant...
The New Riddle of Induction: Neutral and Relative Perspectives on Color
by Anders_H — 2017-12-02T16:15:08.912Z

Nelson Goodman's "New Riddle of Induction" has previously been discussed on Less Wrong at http://lesswrong.com/lw/8fq/bayes and http://lesswrong.com/lw/mbr/grue . Briefly, this riddle shows that any attempt to make future predictions by generalizing from past observations, may depend on arbitrary aspects of the reasoner's language. This is illustrated in the context of the proposed time-dependent color predicates "grue" and "bleen".

In this article, I propose that the resolution to this apparent paradox lies in the recognition that a neutral perspective exists, and that while an agent cannot know with certainty whether their perspective is neutral, they can assign significantly higher credence to their perspective being neutral because evolution had no reason to introduce an arbitrary time-dependent term in their color detection algorithm. I am unsure whether the argument is novel, but as far as I can tell, this particular solution is not discussed in any of the previous literature that I have consulted.

(Note: This article was originally written as coursework, and therefore contains an extended summary of Goodman's original paper and of previous discussion in the academic literature. Readers who are familiar with the previous literature are encouraged to skip to the section "The Grue Sleeping Beauty Problem")

The Problem of Induction and Its Dissolution

In a series of lectures published as the 1954 book "Fact, Fiction and Forecast", Nelson Goodman argued that Hume's traditional problem of induction has been "dissolved", and instead described a different problem that confounds our effort to infer general facts from limited observations. Before I discuss this new problem - which Goodman termed "The New Riddle of Induction" - I briefly discuss what Goodman means when he asserts that the original problem of induction has been "dissolved".

In Goodman's view, it is true that there can be no necessary relationship between past and future observations, and it is therefore futile to expect logical certainty about any prediction. In his view, this is all there is to the problem of induction: If what you want from an inductive procedure is a logical guarantee about your prediction, then the problem of induction illustrates why you cannot have it, and it is therefore...
More Dakka
by Zvi — 2017-12-02T13:10:00.319Z

Epistemic Status: Hopefully enough Dakka

Eliezer Yudkowsky’s book Inadequate Eqilibria is excellent. I recommend reading it, if you haven’t done so. Three recent reviews are Scott Aaronson’s, Robin Hanson’s (which inspired You Have the Right to Think and a great discussion in its comments) and Scott Alexander’s. Alexander’s review was an excellent summary of key points, but like many he found the last part of the book, ascribing much modesty to status and prescribing how to learn when to trust yourself, less convincing.

My posts, including Zeroing Out and Leaders of Men have been attempts to extend the last part, offering additional tools. Daniel Speyer offers good concrete suggestions as well. My hope here is to offer both another concrete path to finding such opportunities, and additional justification of the central role of social control (as opposed to object-level concerns) in many modest actions and modesty arguments.

Eliezer uses several examples of civilizational inadequacy. Two central examples are the failure of the Bank of Japan and later the European Central Bank to print sufficient amounts of money, and the failure of anyone to try treating seasonal affective disorder with sufficiently intense artificial light.

In a MetaMed case, a patient suffered from a disease with a well-known reliable biomarker and a safe treatment. In studies, the treatment improved the biomarker linearly with dosage. Studies observed that sick patients whose biomarkers reached healthy levels experienced full remission. The treatment was fully safe. No one tried increasing the dose enough to reduce the biomarker to healthy levels. If they did, they never reported their results.

In his excellent post Sunset at Noon, Raymond points out Gratitude Journals:

“Rationalists obviously don’t *actually* take ideas seriously. Like, take the Gratitude Journal. This is the one peer-reviewed intervention that *actually increases your subjective well being*, and costs barely anything. And no one I know has even seriously tried it. Do literally *none* of these people care about their own happiness?”

“Huh. Do *you* keep a gratitude journal?”

“Lol. No, obviously.”

– Some Guy at the Effective Altruism Summit of 2012

Gratitude journals are awkward interventions, as Raymond found, and we need to find details that make it our own, or it won’t work. But the active ingredient, gratitude, obviously works an...
Explicit content
by Benquo — 2017-12-02T00:00:00.946Z

The song The Sound of Silence has been on my mind, and ... what if they actually meant the lyrics? What if in lots of popular songs, people were actually trying to tell people a thing?

I know that "popular music contains messages for me" is a typical crazy-person thing to say. But, to be fair, they are literally messages! They are arrangements of words that convey content! And, if you're hearing it, it's for you - they're designed to appeal to as many people as possible.

Of course, I don't mean a secret coded messages - I mean the explicit content of the lyrics. (Occasionally broadcast media will still warn viewers that they are about to hear or see explicit content, but I don't notice that the things that follow are any more or less clear than usual.)

If I wanted to transmit a short verbal string to as many people as possible, and get them to listen to it enough to know all the words, I can't imagine much of a better medium for that than a popular song. People sing along to them, after all.

But, what if our society is such that you can get people to mimic the word-patterns, but not engage with the content of the message? Maybe you'd want to complain about that, warn people about it. And, maybe instead you'd watch your horror as they sang along to your warning of our society's diminished ability to process verbal content, without processing the verbal content. As your song climbed the charts, but there was no public deliberation about the issues you'd raised.

Well, that already happened. The song is called The Sound of Silence. The lyrics are fairly explicit:

Hello darkness, my old friend

I've come to talk with you again

Because a vision softly creeping

Left its seeds while I was sleeping

And the vision that was planted in my brain

Still remains

Within the sound of silence

In restless dreams I walked alone

Narrow streets of cobblestone

'Neath the halo of a street lamp

I turned my collar to the cold and damp

When my eyes were stabbed by

The flash of a neon light

That split the night

And touched the sound of silence

And in the naked light I saw

Ten thousand people, maybe more

People talking without speaking

People hearing without listening

People writing songs that voices never share

And no one dared

Disturb the sound of silence

"Fools", said I, "You do not know

Silence like a cancer grows

Hear my words that I might teach you

Take my arms that I might reach you"

But m

...
TSR #3 Entrainment: Discussion
by Hazard — 2017-12-01T16:46:35.718Z

Epitsemic effort: 3 hours of writing, 15 min thinking on the outline at the end of the post.

Sebastian Marshall (used to post to lesswrong as lionhearted) writes The Strategic Review (TSR), which has recently migrated from a newsletter to a Medium blog. I don’t know how many LW people already read it, but I find it to be consistently quality. Most of TSR is focused on how to be more effective, though I feel like it’s grossly underselling Sebastian to just say he writes about productivity. One thing I enjoy about his writing is that it is very much anti-insight porn. It’s hard to read TSR without thinking about how I can better implement the ideas into how I operate.

Anywho, I’m going to be writing a weekly post intended to prompt thinking around whatever the topic of the latest edition of the TSR happens to be. The format will be something like:

Super brief summary of the article.
Draw some connections between the content of the article and ideas floating around in the rational-sphere.
Pose some questions that I think would greatly benefit one to have an answer to.
My attempt at answering some of those questions.

I’m going to test run this for at least the rest of the series (7 weeks?), and then reassess if I want to continue. Comment or otherwise let me know if you find these useful.

*here's the link*

*also, yes, this post is about the 3rd article in this TSR series. I only had this idea partway through last week*

Summary Background Ops #3: Entrainment

Training is freedom
Key parts of any training regime:

Correct fundamentals
Concentration
Consistency
Accuracy
Speed

Tips and ideas

Build the general skill of taking an unexamined behaviour and entraining the best practices.
Begin with easy targets, so your energy and focus can go into properly following a training regime.
A key tipping point is when you become excited about and look forward to opportunities to train.

It seems useful and true to think of rationality as a collection of habits of thought. Conor Moreton described it as 1000 heads in a row. If that’s how things are, then habit cultivation is at the core of becoming a stronger rationalist.

I really like the graph that Ramon used in a recent post, describing two axes to think about what it takes to acquire skills.

As Ramon mentioned, LW has historically been most concerned with epiphanies, though it seems like over the past few years Tortoises skill are on the rise, ...
MIRI's 2017 Fundraiser
by Malo — 2017-12-01T13:45:32.975Z

MIRI’s 2017 fundraiser is live through the end of December! Our progress so far (updated live):

Donate Now

MIRI is a research nonprofit based in Berkeley, California with a mission of ensuring that smarter-than-human AI technology has a positive impact on the world. You can learn more about our work at “Why AI Safety?” or via MIRI Executive Director Nate Soares’ Google talk on AI alignment.

In 2015, we discussed our interest in potentially branching out to explore multiple research programs simultaneously once we could support a larger team. Following recent changes to our overall picture of the strategic landscape, we’re now moving ahead on that goal and starting to explore new research directions while also continuing to push on our agent foundations agenda. For more on our new views, see “There’s No Fire Alarm for Artificial General Intelligence” and our 2017 strategic update. We plan to expand on our relevant strategic thinking more in the coming weeks.

Our expanded research focus means that our research team can potentially grow big, and grow fast. Our current goal is to hire around ten new research staff over the next two years, mostly software engineers. If we succeed, our point estimate is that our 2018 budget will be $2.8M and our 2019 budget will be $3.5M, up from roughly $1.9M in 2017.1

We’ve set our fundraiser targets by estimating how quickly we could grow while maintaining a 1.5-year runway, on the simplifying assumption that about 1/3 of the donations we receive between now and the beginning of 2019 will come during our current fundraiser.2

Hitting Target 1 ($625k) then lets us act on our growth plans in 2018 (but not in 2019); Target 2 ($850k) lets us act on our full two-year growth plan; and in the case where our hiring goes better than expected, Target 3 ($1.25M) would allow us to add new members to our team about twice as quickly, or pay higher salaries for new research staff as needed.

We discuss more details below, both in terms of our current organizational activities and how we see our work fitting into the larger strategy space.

What’s new at MIRI | Fundraising goals | Strategic background

What’s new a

...
Comment on SSC's Review of Inadequate Equilibria
by Ben Pace — 2017-12-01T11:46:13.688Z

Copying over my comment from the SSC review, which otherwise may get lost in the fog of comments there.

Super fun review!

I found this part to be the biggest disappointment of this book. I don’t think it grappled with the claim that the Outside View (and even Meta-Outside View) are often useful. It offered vague tips for how to decide when to use them, but I never felt any kind of enlightenment, or like there had been any work done to resolve the real issue here. It was basically a hit job on Outside Viewing.

Conversely, I found the book gave short but excellent advice on how to resolve the interminable conflict between the inside and outside views – the only way you can: empiricism. Take each case by hand, make bets, and see how you come out. Did you bet that this education startup would fail because you believed the education market was adequate? And did you lose? Then you should update away from trusting the outside view here. Et cetera. This was the whole point of Chapter 4, giving examples of Eliezer getting closer to the truth with empiricism (including examples where he updated towards using the expert-trusting outside view, because he’d been wrong).

You quote “Eliezer’s four pronged strategy” But I feel like his actual proposed methodology was in chapter 4:

Step one is to realize that here is a place to build an explicit domain theory—to want to understand the meta-principles of free energy, the principles of Moloch’s toolbox and the converse principles that imply real efficiency, and build up a model of how they apply to various parts of the world.

Step two is to adjust your mind’s exploitability detectors until they’re not always answering, “You couldn’t possibly exploit this domain, foolish mortal,” or, “Why trust those hedge-fund managers to price stocks correctly when they have such poor incentives?”

And then you can move on to step three: the fine-tuning against reality.

This is how you figure out if you’re Jesus – test your models, and build up a track record of predictions.

You might respond “But telling me to bet more isn’t an answer to the philosophical question about which to use” in which case I repeat: there isn’t a way a priori to know whether to trust experts using the outside view, because you don’t know how good experts are, and you need to build up domain-specific skills in predicting this.

You might respond “But this book didn’t give me any ...
Cash transfers are not necessarily wealth transfers
by Benquo — 2017-12-01T10:10:01.038Z

Here’s a common argument:

The problem with the poor is that they haven’t got enough money. There’s ample empirical evidence backing this up. Therefore, the obviously-correct poverty intervention is to simply give the poor cash. You might be able to do better than this, but it’s a solid baseline and you should often expect to find that interventions are worse than cash.

There are technical reasons to be skeptical of cash transfers - which is why it is so important that the cash transfer charity GiveDirectly is carefully researching what actually happens when they give people cash - but until fairly recently, these objections seemed to me like abstruse nitpicks about an intervention that was almost analytically certain to be strongly beneficial.

But they’re not just nitpicks. Cash isn’t actually the same thing as human well-being, and the assumption that it can be simply exchanged into pretty much anything else is not obviously true.

Of course, saying “X is possibly wrong” isn’t very helpful unless we have a sense of how it’s likely to be wrong, under what circumstances. It’s no good to treat cash transfers just the same as before, but be more gloomy about it.

I’m going to try to communicate an underlying model that generates the appropriate kind of skepticism about interventions like cash transfers, in a way that’s intuitive and not narrowly technical. I’ll begin with a parable, and then talk about how it relates to real-world cases.

Two cities, two stadiums

In a world where the only thing people enjoy is baseball, there is a wealthy city. In this city, there is an excellently designed baseball stadium. The seats are amply sized and comfortable. Even the worst seats have a good view of the field. There are plentiful amenities, though the price of food and drink is high. There are awnings to block the rain. There are lights in case the game goes late.

Elsewhere, in a poor city, there is another baseball stadium. This one is shoddily built. The upper seats are tiny to cram in as many people as possible. The worst seats can hardly see the field, or are exposed to inclement weather. The aisles are too narrow. The mood is chaotic, and in the poorer areas of the stadium – but not in the expensive areas with the best views – people often get into fights. A smaller variety of cheaper, lower-quality food and drink is available. There is always a line for the bathroom, unless ...
December 2017 Media Thread
by ArisKatsaris — 2017-12-01T09:02:30.955Z

This is the monthly thread for posting media of various types that you've found that you enjoy. Post what you're reading, listening to, watching, and your opinion of it. Post recommendations to blogs. Post whatever media you feel like discussing! To see previous recommendations, check out the older threads.

Rules:

Please avoid downvoting recommendations just because you don't personally like the recommended material; remember that liking is a two-place word. If you can point out a specific flaw in a person's recommendation, consider posting a comment to that effect.

If you want to post something that (you know) has been recommended before, but have another recommendation to add, please link to the original, so that the reader has both recommendations.

Please post only under one of the already created subthreads, and never directly under the parent media thread.

Use the "Other Media" thread if you believe the piece of media you want to discuss doesn't fit under any of the established categories.

Use the "Meta" thread if you want to discuss about the monthly media thread itself (e.g. to propose adding/removing/splitting/merging subthreads, or to discuss the type of content properly belonging to each subthread) or for any other question or issue you may have about the thread or the rules.