trentmkelly/LessWrong-43k huggingface dataset:
Lessons on Value of Information From Civ
by johnswentworth — 2020-10-07T18:18:40.118Z

I’m a big fan of the game

Civilization

. Civ is the canonical empire-building game: you explore the world, expand your empire, manage your economy, invade other empires, unlock new tech, etc. A few months ago, the game released “tech shuffle mode”, which radically changes the game (and IMO makes it a lot more interesting). Usually, a not-yet-unlocked section of the tech tree looks like this:

To unlock a tech, we first need to unlock the prerequisite techs (in left-to-right order). The tree is the same in every game, so we can plan ahead to unlock key technologies early - e.g. we can ignore the lower half of the tree in order to get to Steam Power faster. But with tech shuffle mode, a not-yet-unlocked section of the tree looks like this:

The tree structure is randomly generated each game (although techs are still grouped by “era”, so e.g. we won’t unlock nukes right at the start of the game). Techs are also hidden until we’re ready to research them (i.e. until we unlock the prereqs). That means, for instance, that we can’t focus resources on techs needed to unlock Steam Power because we don’t know where in the tree Steam Power is.

But there’s a catch: there are actions we can take to reveal a tech in the tree. This is the “tech boost” mechanic - for instance, by finding another civilization, we can “boost” Writing, reducing the resource cost to unlock it by 40% and (more importantly) revealing its position in the tree. It looks like this:

In other words: we can pay to acquire information.

This is especially important when the value of information (VOI) is high. What does that look like? Like this:

There’s several different branches, each several techs deep. The strategically-important tech we want is somewhere at the end of one of those branches, but we don't know where. Brute-force searching all of the paths until we find it would require unlocking ~½ of these techs on average, but if we knew which branch to take, then we’d only need to unlock one branch - i.e. ~¼ of them in this case. So, I can pay some resource cost to reveal my target tech in the tree, and then I can get to it about twice as quickly (on average).

I realized the importance of the tech boost mechanic as a way of purchasing information after my first game on tech shuffle mode. It seemed that VOI was often very high, so I made an explicit effort to pay attention to it in my next few games.

The result:...
How can an AI demonstrate purely through chat that it is an AI, and not a human?
by hugh.mann — 2020-10-07T17:53:07.715Z

In the latest William Gibson novel Agency, a bunch of human characters are constantly told exactly what to do by a superhuman AI in order to have everything turn out for the best. They accept the notion that this character, who they only talk to over their phones, is an AI, when it seems like it could just as easily be a fictional construct created by a person or organization. There's one part of the book where a character is convinced off-page that the AI is a superhuman AI by talking to it for a bit, although we're not privy to the conversation.

What I'm wondering is, how could a superhuman AI best prove that it was a superhuman AI through chat? Is there a more or less definitive way to do it? Is there a fast way to do it?
Review and Summary of 'Moral Uncertainty'
by fin — 2020-10-07T17:52:42.189Z
Link: https://www.finmoorhouse.com/writing/moral-uncertainty

I just read the recently published 'Moral Uncertainty' by Will MacAskill, Krister Bykvist, and Toby Ord. It's an excellent book with significant practical implications. As such, I originally intended to write a brief review — which ballooned into this monstrosity. In any case, I hope such a summary is useful for anyone curious about the book's main arguments, and the subject of moral uncertainty more generally. I would also be very grateful for feedback on this — anonymous is fine.
[AN #120]: Tracing the intellectual roots of AI and AI alignment
by Rohin Shah — 2020-10-07T17:10:07.013Z

Alignment Newsletter is a weekly publication with recent content relevant to AI alignment around the world. Find all Alignment Newsletter resources here. In particular, you can look through this spreadsheet of all summaries that have ever been in the newsletter.

Audio version here (may not be up yet).

HIGHLIGHTS

The Alignment Problem (Brian Christian) (summarized by Rohin): This book starts off with an explanation of machine learning and problems that we can currently see with it, including detailed stories and analysis of:

- The gorilla misclassification incident

- The faulty reward in CoastRunners

- The gender bias in language models

- The failure of facial recognition models on minorities

- The COMPAS controversy (leading up to impossibility results in fairness)

- The neural net that thought asthma reduced the risk of pneumonia

It then moves on to agency and reinforcement learning, covering from a more historical and academic perspective how we have arrived at such ideas as temporal difference learning, reward shaping, curriculum design, and curiosity, across the fields of machine learning, behavioral psychology, and neuroscience. While the connections aren't always explicit, a knowledgeable reader can connect the academic examples given in these chapters to the ideas of specification gaming (AN #97) and mesa optimization (AN #58) that we talk about frequently in this newsletter. Chapter 5 especially highlights that agent design is not just a matter of specifying a reward: often, rewards will do ~nothing, and the main requirement to get a competent agent is to provide good shaping rewards or a good curriculum. Just as in the previous part, Brian traces the intellectual history of these ideas, providing detailed stories of (for example):

- BF Skinner's experiments in training pigeons

- The invention of the perceptron

- The success of TD-Gammon, and later AlphaGo Zero

The final part, titled "Normativity", delves much more deeply into the alignment problem. While the previous two parts are partially organized around AI capabilities -- how to get AI systems that optimize for their objectives -- this last one tackles head on the problem that we want AI systems that optimize for our (often-unknown) objectives, covering such topics as imitation learning, inverse reinforcement learning, learning from preferences, iterated amplification, i...
Inaccessible finely tuned RNG in humans?
by Sandi — 2020-10-07T17:04:36.142Z

It was my impression that humans are bad at random (number) generation. Tell a person to arrange stars on a black canvas randomly, and they'll space them out more or less equidistantly and uniformly. If asked to generate a sequence of coin flips, we would avoid longer runs of the same coin face, which do happen with real coins.

The scientific consensus agrees. Here's a study where participants are asked to generate a sequence of 300 digits from 1 to 9. The researches successfully predict the next digit with a 27% chance (a truly random sequence would indicate an 11% chance). Scott Aaronson designed a simple program that predicts which of two keys the user will press next with a 70% chance on average.

On the other hand, I've seen a number of informal online polls (e.g. on reddit and twitter) where the respondents demonstrate an ability to randomly sort themselves into two groups with a very high degree of accuracy. See here and here, though I've seen a ton of these with various ratios. I've never seen it not work. It does break down for 1-99, but that seems reasonable to me.

The question is, do you know of any research into this phenomenon or do you have a good explanation for it?

Note that the ability is there in some contexts but not in others. An RNG exists within us, but our access to it is sometimes blocked. I've seen other examples of this in psychology so it doesn't weird me out as much as it used to, but I still wanna point it out.

If this seems like a queer phenomenon to fixate on, know that an RNG is useful in many contexts. It's necessary for strong/optimal strategies in certain games (poker, rock paper scissors) and game theoretic scenarios. It's useful for coordination without communication. The best known algorithms for some problems are probabilistic.

And while I have your attention: https://www.strawpoll.me/21064106
Open Communication in the Days of Malicious Online Actors
by ozziegooen — 2020-10-07T16:30:01.935Z
Link: https://forum.effectivealtruism.org/posts/qWJyPiws7B4XyQGJR/

This article uses a thought experiment to probe at dangers of public online communication, then describes an ontology around “malicious supporters” and possible remedies.

It's arguably a more obvious fit for LW than the EA Forum, but I posted it there for reasons explained in the comments. I think people here might also find it interesting though.
How worried are the relevant experts about a magnetic pole reversal?
by Sunny from QAD — 2020-10-07T16:19:51.607Z

This article and many others like it describe a phenomenon where the Earth's magnetic poles occasionally switch directions. The key ideas seem to be:

Earth's magnetic poles occasionally switch places.
This probably happens on a semi-regular basis, and according to our best guess at the schedule, we're long overdue for a switch right now.
The magnetic north pole has been moving South much faster than normal recently (25 mi/year).
We don't know exactly what happens during a switch, but it's probably not great.

The main concern seems to be that the Earth's magnetic field is what's responsible for deflecting/weakening a lot of incoming radiation, but the magnetic field gets a lot weaker during a switching event. Then again, the article linked above suggests that the period is about 300,000 years, which is far shorter than the length of our evolutionary history. That puts a bound on how bad it can be.

Nonetheless, this all somewhat concerning to me. Who are the relevant authority figures on the subject? How worried are they that we might be on the brink of some kind of extinction event? How worried are they about the interaction of such an event with everyday electronics? Do they make any recommendations for how individual people can prepare themselves for a potential switch? How about for society?
Brainstorming positive visions of AI 
by jungofthewon — 2020-10-07T16:09:33.453Z

Context

This is a place to explore visions of how AI can go really well. Conversations about AI (both in this community and disseminated by mainstream media) focus on dystopian scenarios and failure modes. Even communities that lean technoutopian (Silicon Valley) are having an AI hangover. More broadly, many people in my life think the future will be worse than the present and this makes me sad.

So I think it's time to revisit the science fiction books of our teenage years and imagine what amazing applications of AI or AGI in society looks like. AI that doesn’t destroy us is great. AI that unlocks human flourishing is even better. I've personally found it much easier to think of negative scenarios than positive scenarios, so this is me enlisting your help.

Discussion norms

I’d like this to be a

yes-and

, generative, bursty, collective brainstorming thread. Ideas that make no sense, that you might not endorse later, that you can't even explain or defend are all welcome. The person that starts the idea might not be the person that finishes it so don't censor too early.

I hope this discussion style shows us a different way to get to better reasoning: as opposed to one person contributing a well-developed position and others pointing out inconsistencies, we can build ideas collectively, with everyone contributing different parts. I also hope it encourages people who don’t normally participate on LessWrong to contribute their thoughts.

Prompts for idea generation

(Feel free to ignore and share whatever comes to mind)

What does AI going well look like?
If you feel pessimistic about the future, what vision of AI would make you feel more optimistic?
What are problems we don't know how to solve that you'd be excited to solve with AI?
What great companies could be built in the next decade, where the product is built primarily around modern Machine Learning? (If you find this prompt overly large, can you think of any companies that could be built around GPT-3?)

I’ll seed some of my ideas below :) Thanks Amanda, Ben, Andreas for discussing these ideas with me.
Buying micro-biostasis
by Mati_Roy — 2020-10-07T10:55:19.006Z

x-post: Buying micro-biostasis – LessDead

Created: 2020-10-07 | Updated: 2020-10-07

Help / Quality: I’m publishing this post slightly unfinished because a friend of a friend might need this information today (or in the next few days) to (micro-)cryopreserved their grandmother. If you have knowledge on how to use online gambling platforms in the ways described in this post (see section “How to” and “Replacing credences with frequencies”, your input would be appreciated.

Warning: This article talks about gambling. I think gambling is almost always a bad idea, and gamblers are often irrational and/or addicted. This article discusses an exception where I think gambling is rational.

Summary: If you need to buy cryonics very soon, and don’t have enough money, you can gamble the money you have for a chance of being able to afford getting cryopreserved.

Important note: Other options for people in an ermgency situation with a low budget not discussed in this post include low-cost preservations and fundraising. Lifelogging can also be considered as a complementary option.

Related: meme

General idea

If the cheapeast biostasis option available to you is more expensive than the amount of money you have, you can gamble your money to buy a probability of getting cryopreserved, which itself is a probability of living longer.

Cryonics is already a gamble: you bet money that it might allow you to live longer. Gambling money that might allow you to get enough money to get cryopreserved is not qualitatively different.

Even if what you’re buying is just hope, buying a probability of having the hope for a fraction of the money should generally still be worth it.

It seems to me like this is true for any expense that is:

larger than your total disposable assets: otherwise you can just pay directly for it

note: it could be that you keep separate budget for separate values, in which case you might want to gamble your budget to save a loved one, and keep the rest of your assets separately

time sensitive: otherwise you can accumulate more resources to pay for it

more valuable than what you could otherwise do with money: otherwise you should but those other things

where borrowing money isn’t an option: otherwise you should borrow money to make sure you can buy it

The only example I can think that fits those criterias is for an emergency life saving technology. Given current medicine is ge...
Can we hold intellectuals to similar public standards as athletes?
by ozziegooen — 2020-10-07T04:22:20.450Z

Professional athletes are arguably the most publicly understood meritocracy around. There are public records of thousands of different attributes for each player. When athletes stop performing well, this is discussed at length by enthusiasts, and it's understood when they are kicked off their respective teams. The important stuff is out in the open. There's a culture of honest, open, and candid communication around meritocratic competence and value.

This isn't only valuable to help team decisions. It also helps data scientists learn which sorts of characteristics and records correlate best with long term success. As sufficient data is collected, whole new schools of thought emerge, and these coincide with innovative and effective strategies for future talent selection. See Moneyball or the entire field of sabermetrics.

In comparison, our standards for intellectuals are quite prosaic. If I want to get a sense of just how good LeBron James is I can look through tables and tables or organized data and metrics. If I don't trust one metric I have dozens of others to choose.

However, if I want to know how much to trust and value Jonathan Haidt I'm honestly not sure what to do. Some ideas:

Read most of his work, then do a large set of Epistemic Spot Checks and more to get a sense of how correct and novel it is.

Teach myself a fair amount of Psychology, get a set of Academic Journal subscriptions, then read critiques and counter critiques of his work.

Investigate his citation stats.

Read the "Reception" part of his Wikipedia page and hope that my attempt to infer his qualities from that is successful.

Use some fairly quick "gut level" heuristics to guess.

Ask my friends and hope that they did a thorough job of the above, or have discussed the issue with other friends who did.

Of course, even if I do this for Jonathan Haidt broadly, I'd really want narrow breakdowns. Maybe his old work is really great, but in the last 5 years his motives have changed. Perhaps his knowledge and discussion of some parts of Psychology is quite on point, but his meanderings into Philosophy are simplistic and overstated.[1]

This is important because evaluating intellectuals is dramatically more impactful than evaluating athletes. When I choose incorrectly, I could easily waste lost of time and money, or be dramatically misled and develop a systematically biased worldview. It also leads to i...
Forcing Freedom
by vlad.proex — 2020-10-06T18:15:13.596Z

Wretched Worlds

You are a freshly recruited cadet in the Intergalactic Liberation Corps (ILC). With a handful of valiant companions, you will travel across the galaxy, helping to free oppressed Beta Species from the claws of their Alpha masters, and fighting slavery whenever it dares to rear its ugly head.

The Intergalactic Council is very serious about slavery. The practice is condemned in the Galactic Bill of Rights, which guarantees the equality and autonomy of all sentient species. But the government of the Republic doesn't have enough leverage to abolish it everywhere. In assembling the ILC, the Council hopes to start an intergalactic movement of liberation that will bring war to the enslavers’ backyards.

The basic strategy is simple: infiltrate a planet where slavery is practiced, establish contact with the oppressed species, and help them spark an insurgency. As discretion is paramount, you will infiltrate in small groups. Your task will be to provide the insurgents with motivation, intelligence and specialized knowledge. As you’re about to find out, this is more difficult than it seems.

First

As you descend on the first planet, you are greeted by a spectacle of appalling suffering: the Alphas torture and humiliate the Betas liberally, sparing them no cruelty. You immediately start working to set up a resistance movement. To your dismay, not one of the Betas is interested in your proposals. After a few weeks, you understand why: the Betas, once a free species, have been genetically re-engineered in order to serve as willing slaves. When separated from their masters, they become hopelessly depressed and starve themselves to death. You consider three options: a) attempt to reverse the genetic modifications (but this is forbidden by the Galactic Bill of Rights); b) exterminate the Betas to remove them from their fate; c) do nothing and leave. Eventually you decide for the third option.

Second

The Alphas have populated the atmosphere with a parasite. When inhaled by a Beta, the parasite acts on their brain, turning them into a willing slave. As long as Betas are on the planet's surface, their mind is altered and they feel a compelling desire to serve their masters. If they stop breathing the planet's atmosphere, the parasite dies and they regain their senses.
Since staging a rebellion is impossible, you begin hatching a plot to...
The Alignment Problem: Machine Learning and Human Values
by Rohin Shah — 2020-10-06T17:41:21.138Z
Link: https://www.amazon.com/Alignment-Problem-Machine-Learning-Values/dp/153669519X

The Alignment Problem: Machine Learning and Human Values, by Brian Christian, was just released. This is an extended summary + opinion, a version without the quotes from the book will go out in the next Alignment Newsletter.

Summary:

This book starts off with an explanation of machine learning and problems that we can currently see with it, including detailed stories and analysis of:

- The

gorilla misclassification incident

- The

faulty reward in CoastRunners

- The

gender bias in language models

- The

failure of facial recognition models on minorities

- The

COMPAS

controversy

(leading up to

impossibility results in fairness

)

- The

neural net that thought asthma reduced the risk of pneumonia

It then moves on to agency and reinforcement learning, covering from a more historical and academic perspective how we have arrived at such ideas as temporal difference learning, reward shaping, curriculum design, and curiosity, across the fields of machine learning, behavioral psychology, and neuroscience. While the connections aren't always explicit, a knowledgeable reader can connect the academic examples given in these chapters to the ideas of

specification gaming

and

mesa optimization

that we talk about frequently in this newsletter. Chapter 5 especially highlights that agent design is not just a matter of specifying a reward: often, rewards will do ~nothing, and the main requirement to get a competent agent is to provide good shaping rewards or a good curriculum. Just as in the previous part, Brian traces the intellectual history of these ideas, providing detailed stories of (for example):

- BF Skinner's experiments in

training pigeons

- The invention of the

perceptron

- The success of

TD-Gammon

, and later

AlphaGo Zero

The final part, titled "Normativity", delves much more deeply into the alignment problem. While the previous two parts are partially organized around AI capabilities -- how to get AI systems that optimize for their objectives -- this last one tackles head on the problem that we want AI systems that optimize for our (often-unknown) objectives, covering such topics as imitation learning, inverse reinforcement learning, learning from preferences, iterated amplification, impact regularization, calibrated uncertainty estimates, and moral uncertainty.

Opinion:

I really enjoyed this book, primarily because of the tracing of the intellectual history of various ideas. While ...