trentmkelly/LessWrong-43k huggingface dataset:
“Cheating Death in Damascus” Solution to the Fermi Paradox 
by avturchin — 2018-06-30T12:00:58.502Z

TL;DR: The Great Filter from the Fermi paradox could be escaped by choosing a random strategy. However, if all civilizations acted randomly, this could be the actual cause of the Fermi paradox. Using a meta-random strategy solves this.

----

"Death in Damascus" is a decision theory problem about attempting to escape an omniscient agent who is able to predict your behavior.

It goes like this: “You are currently in Damascus. Death knocks on your door and tells you I am coming for you tomorrow. You value your life at $1,000 and would like to escape death. You have the option of staying in Damascus or paying $1 to flee to Aleppo. If you and death are in the same city tomorrow, you die. Otherwise, you will survive. Although death tells you today that you will meet tomorrow, he made his prediction of whether you’ll stay or flee yesterday and must stick to his prediction no matter what. Unfortunately for you, Death is a perfect predictor of your actions. All of this information is known to you.”

It was explored in the article “Cheating death in Damascus,” where a possible solution was suggested: a true random generator is used to choose between staying in Damascus and fleeing to Aleppo, and thus one has 0.5 chance of survival.

The Fermi paradox is a type of "Death in Damascus" problem. The Fermi paradox said that other civilizations are not observable for unknown reasons, and one of the solutions is the Great Filter which kills all young civilizations; for us, such a filter is ahead. This means that all the civilizations before us made the same mistake which resulted in their demise, and as we are a typical civilization, we will make the same mistake, too. However, we don’t know what this universal mistake is. Maybe we should not experiment with hadron collider. Maybe AI always goes rogue, kills everybody and later self-terminates (AI itself can’t explain the Fermi paradox, as it will spread through the universe). But maybe the decision not to create AI is fatal, as only AI can manage risks of synthetic biology and other catastrophic risks.

In other words, whatever rational strategy we take, this is exactly what has killed all previous civilizations; if we escape to Aleppo, Death will meet us there. In the original problem, Death is omniscient; in the case of the Fermi paradox, the omniscience is replaced by our typicality and mediocrity reasoning...
Applied Rationality MUD
by rlp10 — 2018-06-30T03:14:46.128Z

Hey, do you want to help me build a MUD dedicated to learning and applying the principles of rationality?

What's a MUD?

A Multi User Dungeon. A virtual online world with a text interface. You have one or more characters who move through locations in a virtual world. The world has some rules or mechanics: perhaps for fighting, spell-casting, leveling up, buying/selling and so forth. There might also be a system of quests with rewards. You can meet other players and interact with them, or interact with automated characters.

Certain players ("wizards") are promoted to be able to create the world: digging new locations, creating quests, inventing new spells and automating the characters and items.

What might an Applied Rationality MUD look like?

So, let's imagine what ARMUD might look like once it had been written. You log into using your web browser. You're told that your character is in an ancient marketplace, an "agora". The philosopher Diogenes is here who randomly pulls stunts and mutters words of philosophy from time-to-time. "If only I could relieve hunger so easily by rubbing my belly!" he says. Hm, wonder what that means.

He offers various quests which, if accepted, will practically challenge you to apply principles of minimalism and simplicity to your life. "Tidy your room, bucko!" is one such quest. To complete that mission, or any others, you must accomplish the required task in the real world, but don't worry, you get some experience points here in the virtual world too. And that's what really matters.

To the south lies the Parthenon, the temple of Athena Goddess of Wisdom. Yoda is to be found here and he teaches the spell of "Yoda Timer". When cast, that spell helps a person to focus on the accomplishment of an important task to the exclusion of other distractions for a given length of time. The first levels are easy, but as you want to cast it at a higher level, it requires longer focus and concentration. Again, experience points and leveling up are to be achieved by learning and casting this spell. Although experience points are lost if it is cast, but the timer is not completed.

To the northeast of the Agora, is the Stoa Poikile. That is where people go to learn the practical philosophy of stoicism and related modern ideas such as CBT and other therapies. Epictetus is always foun...
Policy Alignment
by abramdemski — 2018-06-30T00:24:25.269Z

(ETA: The name "policy approval" wasn't great. I think I will use the term "policy alignment" to contrast with "value alignment" going forward, at the suggestion of Wei Dai in the comments.)

I recently had a conversation with Stuart Armstrong in which I claimed that an agent which learns your utility function (pretending for a moment that "your utility function" really is a well-defined thing) and attempts to optimize it is still not perfectly aligned with you. He challenged me to write up specific examples to back up my claims.

I'll also give a very sketchy alternative to value learning, which I call policy alignment. (The policy alignment idea emerged out of a conversation with Andrew Critch.)

Background

Stuart Armstrong has recently been doing work showing the difficulty of inferring human values. To summarize: because humans are irrational, a value-learning approach like CIRL needs to jointly estimate the human utility function and the degree to which the human is rational -- otherwise, it would take all the mistakes humans make to be preferences. Unfortunately, this leads to a severe problem of identifiability: humans can be assigned any values whatsoever if we assume the right kind of irrationality, and the usual trick of preferring simpler hypotheses doesn't seem to help in this case.

I also want to point out that a similar problem arises even without irrationality. Vladimir Nesov explored how probability and utility can be mixed into each other without changing any decisions an agent makes. So, in principle, we can't determine the utility or probability function of an agent uniquely based on the agent's behavior alone (even including hypothetical behavior in counterfactual situations). This fact was discovered earlier by Jeffrey and Bolker, and is analyzed in more detail in the book The Logic of Decision. For this reason, I call the transform "Jeffrey-Bolker rotation".

To give an illustrative example: it doesn't matter whether we assign very low probability to an event, or care very little about what happens given that event. Suppose a love-maximizing agent is unable to assign nonzero utility to a universe where love isn't real. The agent may appear to ignore evidence that love isn't real. We can interpret this as not caring what happens conditioned on love not being ...
Shaping economic incentives for collaborative AGI
by Kaj_Sotala — 2018-06-29T16:26:32.213Z

In "An AI Race for Strategic Advantage: Rhetoric and Risks" (2018), Stephen Cave and Seán S ÓhÉigeartaigh argue that we should try to promote a cooperative AI narrative over a competitive one:

The next decade will see AI applied in an increasingly integral way to safety-critical systems; healthcare, transport, infrastructure to name a few. In order to realise these benefits as quickly and safely as possible, sharing of research, datasets, and best practices will be critical. For example, to ensure the safety of autonomous cars, pooling expertise and datasets on vehicle performances across as wide as possible a range of environments and conditions (including accidents and near-accidents) would provide substantial benefits for all involved. This is particularly so given that the research, data, and testing needed to refine and ensure the safety of such systems before deployment may be considerably more costly and time-consuming than the research needed to develop the initial technological capability.

Promoting recognition that deep cooperation of this nature is needed to deliver the benefits of AI robustly may be a powerful tool in dispelling a ‘technological race’ narrative; and a ‘cooperation for safe AI’ framing is likely to become increasingly important as more powerful and broadly capable AI systems are developed and deployed. [...]

There have been encouraging developments promoting the above narratives in recent years. ‘AI for global benefit’ is perhaps best exemplified by the 2017’s ITU summit on AI for Global Good (Butler 2017), although it also features prominently in narratives being put forward by the IEEE’s Ethically Aligned Design process (IEEE 2016), the Partnership on AI, and programmes and materials put forward by Microsoft, DeepMind and other leading companies. Collaboration on AI in safety-critical settings is also a thematic pillar for the Partnership on AI2 . Even more ambitious cooperative projects have been proposed by others, for example the call for a ‘CERN for AI’ from Professor Gary Marcus, through which participants “share their results with the world, rather than restricting them to a single country or corporation” (Marcus 2017).

So.

In order to make future AGI projects more collaborative and co-operation focused, could we create incentives (via e.g. government policy) that would push today's machine learning researchers...
FHI Research Scholars Programme
by habryka — 2018-06-29T02:31:13.648Z
Link: https://www.fhi.ox.ac.uk/rsp/

FHI is starting a new two-year research scholars program, that I think could be quite exciting to a bunch of people on LW. The deadline to apply is in about two weeks (11th of July), and they're still actively looking for applicants.

FHI has a long history of doing interesting research (many of the core ideas of this site have originated there), and I think there is a good chance that in terms of training epistemic rationality and being given the opportunity to directly tackle many of the key questions around existential risk, AI alignment and theoretical rationality this might be the best career opportunity that currently exists out there for most people.

My guess is that if you are a long-time LessWrong reader who has been waiting for an opportunity to make a more direct impact on AI-risk or other issues frequently covered on this site, then this might be the best opportunity you will get for a good while.

My guess about the people who should most consider applying:

You just finished a degree, considered working in academia, but so far haven't found any opportunities to work on the important things you care about
You are currently in a career that does not, and probably won't, allow you to directly make progress on the problems that seem most important to you
You have some kind of research project that you've been wanting to work on for a long time, but are worried about not having financial stability or a productive research environment in which to work on that project

Here is a small blurb that Owen (the main organizer behind the project) sent me a while ago when I asked him what he usually sends people:

Which is more likely to be radically transformative in the next two decades: AI or biotechnology? Can we draw lessons from the industrial revolution or the development of nuclear weapons? What might humanity achieve on a timescale of millions of years, and does that matter for decisions today? How would the (non-)existence of intelligent aliens affect us?

If you find these questions compelling, you might be interested in joining the Future of Humanity Institute. Our new Research Scholars Programme will employ a small number of talented, curious thinkers to explore such topics, giving mentorship to help them develop judgement about what to work on from the perspective of securing a flourishing future. For details, see https://www.fhi.ox.ac.uk/

...
The Power of Letting Go Part I: Examples
by johnswentworth — 2018-06-29T01:19:03.474Z

There’s an approach/aesthetic of design/engineering/problem-solving which I want to talk about. I’m calling it “

just let go

” design; the general theme is that the designer/engineer/problem-solver gives up control over the details of the solution.

This post is mostly a list of examples to show the breadth of the concept, ranging from Minecraft to tvtropes to price controls to mutation-assisted breeding to jurisprudence to SQL, with a baby lamb in a bag somewhere in the middle. The next post will zoom out and discuss requirements and advantages of the approach.

Examples

Games

Standard advice for a role-playing game master (GM): don’t railroad the players. There’s some wiggle room in how much the GM keeps players on an intended plot line, but the main rule is that a GM should never blatantly squash players’ creativity just because it wrecks the intended plot. Intelligent characters

want to throw your plot out the window

; squashing that creativity wrecks the fun.

Taken to an extreme, the opposite of railroading players is to create a sandbox game: think Minecraft or a

Falling Sands

game. Such games give players near-total freedom to build whatever they want; the game just supplies a canvas.

From a design perspective, one interesting property of sandbox games is that they take relatively little effort to create, yet appeal to a wide audience. Wikipedia lists Minecraft as

the second best-selling video game ever

; third place goes to Grand Theft Auto V. The number of lines of code in those two games

is

ambiguous

, but GTA probably has well over 10x more code based on any apples-to-apples comparison.

Toys

Back in my day, before minecraft, we played with physical blocks called “legos”.

Lego experimented with a foray into less sandboxy products back in the early 00’s; they tried contracts with popular media franchises (e.g. Harry Potter), and creating their own media (e.g. Bionicle). Sales fell. Eventually, they turned around and went in the exact opposite direction, creating

extra-sandboxy products

. Those sold like hotcakes.

App Design

The rise of mobile has made app/website “flows” the main design structure: users go through a linear series of screens, one after the other. It’s railroading, but for apps and websites.

What would a more sandbox-ish alternative look like? One example is

TvTropes

. Art school folks may poo-poo the design, but the site is popular and notoriously addict...
How to formalize predictors
by cousin_it — 2018-06-28T13:08:11.549Z

In Newcomb's Problem, what information does the predictor get if you flip a coin? Here's some options:

1) "Bob has 50:50 odds of one-boxing or two-boxing"

2) "Bob will one-box" (chosen randomly with the same odds as your decision)

3) "Bob will one-box" (guaranteed to be the same as your decision)

I think (1) is a poor formalization, because the game tree becomes unreasonably huge, and some strategies of the predictor (like "fill the box unless the probability of two-boxing is exactly 1") leave no optimal strategy for the player.

And (3) seems like a poor formalization because it makes the predictor work too hard. Now it must predict all possible sources of randomness you might use, not just your internal decision-making.

That leaves us with (2). Basically we allow the predictor to "sample" your decision at any information set in the game. That means we can add some extra nodes to the player's information sets and get rid of the predictor entirely, ending up with a single player game. Newcomb's Problem and Counterfactual Mugging can be easily analyzed in this way, leading to the same answers we had before (one-box, pay up). It also gives a crisp formalization of the transparent boxes Newcomb problem, where we sample the player's decision at the information set of "seeing both boxes filled".

I think this might end up useful for bounded versions of Counterfactual Mugging, which are confusing to everyone right now. But also it feels good to nail down my understanding of the simple version.
Last Chance to Fund the Berkeley REACH
by decodyng — 2018-06-27T21:38:49.299Z

(This post was co-authored by me (Cody Wild), Tessa Alexanian and Ray Arnold)

TL;DR We have 2 days - until July 1 - to increase monthly donations by $2260 or raise $27,000 in lump sum , or else REACH will lose its current location and go on indefinite hiatus

Do you care about the Berkeley REACH staying open? Has an open-access community space added value to your life? We have a few days to raise enough money to comfortably sign a one-year lease. If not, REACH will lose its current location, and the project will go into indefinite hiatus.

Community support for REACH has been substantial, and we’re proud to see the number of people willing to pay seriously for something important. But we haven’t hit the level where it’s realistic to sign a full year lease. For that, we’d either need an additional $2260/month on Patreon, or up-front donations of around $27,000 to make us secure enough to do so. If this lease isn’t signed in the next few days, REACH will lose its current location at the end of July.

If you’re excited about REACH, you’re probably already donating. We care about our community not overextending itself. One of the main benefits of REACH is to help those who haven’t yet gotten their financial footing stabilized, and if that’s your situation we don’t want you feeling pressure here.

But if you can comfortably donate more to REACH – if you’re one of those 6 figure salary programmers who are excited by the prospect of having an honest-to-goodness community in a world that increasingly pushes towards atomization, we’d like you to consider whether you want to do more.

If you’re willing and able to pledge more ongoing support than you already have, or want to make a large, one-time donation to help an early-stage project get its footing, the next few days are the time to come forward and do so.

Background Reading

Reflections on the Berkeley REACH - Sarah Spikes, June 7
Thoughts on the REACH Patreon - Ray Arnold, April 30
Humans Need Places - Ben Hoffman, April 19

Current Financial Reality

REACH has a funding gap. While this funding gap remains, it doesn’t make sense to sign the lease.

Assets

REACH is earning $3,600 a month in the Patreon
The PayPal has brought in around $10,000 so far

Costs

The landlord has been negotiated down to $5,500/month in rent
Short-term room rentals have brought in $1000-$2000/month; we expect that to generally be lower outside of the summer months

...
Sun Arise, Roll out the Chorus of the Dawn
by DSCyprus — 2018-06-27T19:23:43.587Z

I always thought that a sunrise is more magical and impressive than a sunset. My supporting argument is twofold:

First, I believe that most people, myself included, witness sunrises much more rarely than they do sunsets. This makes them more special and harder to take for granted. The fact that everyone is still up and busy around sunset hours makes it so that, unless you go to a secluded area to watch the sunset, you are probably surrounded by noise. During sunrise hours, most people are asleep and it is still quiet mostly everywhere.
Second, being awake to see a sunrise usually includes some effort on my part. I either stay awake through the night or wake up early specially to see it. This specific attempt to see a sunrise makes it easier to appreciate. This also means that when I'm watching a sunrise, I've either just woken up and I know breakfast is due, or I know that I'm going to snuggle in my bed and fall asleep without warning soon.

To add to that, a sunrise has another endearing feature to it - Dawn Chorus. That is the official name of all the bird-singing that you can hear in the morning. Nothing fits a cool, calm morning better than some bird-caroling.

I have noticed multiple times that you can hear birds singing even before the sun is noticeably rising; I recently heard a theory that might explain this- it's a cute one, but prepare your grains of salt nonetheless. First, allow me to shortly explain why dawn chorus is a thing in the first place, and why in the morning, anyway.

Let's start by mentioning that, like a rooster crowing at dawn, it is usually the male that makes all the noise, and he's not doing it for fun - singing to himself as he gets ready for a day of bird business and engagements. The role that the singing plays is to help the birds assert themselves - they make noise to keep other males out of their territories, and also to try and entice females to partner up and mate. In a way, the birdsong strikes two birds with one stone (I would like to apologize for that joke).

The reason why birds sing early in the morning is because of the perfect conditions. Usually, as I said before, mornings are the quietest time of the day. Everything is calm and there is no less wind, which makes the sound carry the farthest. This is simply the most efficient time to be noisy. It's a good thing that we consider it to be ...
Optimization Amplifies
by Scott Garrabrant — 2018-06-27T01:51:18.283Z

I talk here about how a mathematician mindset can be useful for AI alignment. But first, a puzzle:

Given m, what is the least number n≥2 such that for 2≤k≤m, the base k representation of n consists entirely of 0s and 1s?

If you want to think about it yourself, stop reading.

For m=2, n=2.

For m=3, n=3.

For m=4, n=4.

For m=5, n=82,000.

Indeed, 82,000 is 10100000001010000 in binary, 11011111001 in ternary, 110001100 in base 4, and 10111000 in base 5.

What about when m=6?

So, a mathematician might tell you that this is an open problem. It is not known if there is any n≥2 which consists of 0s and 1s in bases 2 through 6.

A scientist, on the other hand, might just tell you that clearly no such number exists. There are 2k−1 numbers that consist of k 0s and 1s in base 6. Each of these has roughly log5(6)⋅k digits in base 5, and assuming things are roughly evenly distributed, each of these digits is a 0 or a 1 with "probability" 2/5. The "probability" that there is any number of length k that has the property is thus less than 2k⋅(2/5)k=(4/5)k. This means that as you increase k, the "probability" that you find a number with the property drops off exponentially, and this is not even considering bases 3 and 4. Also, we have checked all numbers up to 2000 digits. No number with this property exists.

Who is right?

Well, they are both right. If you want to have fun playing games with proofs, you can consider it an open problem and try to prove it. If you want to get the right answer, just listen to the scientist. If you have to choose between destroying the world with a 1% probability and destroying the world if a number greater than 2 which consists of 0s and 1s in bases 2 through 6 exists, go with the latter.

It is tempting to say that we might be in a situation similar to this. We need to figure out how to make safe AI, and we maybe don't have that much time. Maybe we need to run experiments, and figure out what is true about what we should do and not waste our time with math. Then why are the folks at MIRI doing all this pure math stuff, and why does CHAI talk about "proofs" of desired AI properties? It would seem that if the end of the world is at stake, we need scientists, not mathematicians.

I would agree with the above sentiment if we were averting an astroid, or a plague, or global warming, but I think it fails to apply to AI al... ,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,