trentmkelly/LessWrong-43k huggingface dataset:
Discussion Thread: The AI Does Not Hate You by Tom Chivers
by Ruby — 2019-06-17T23:43:00.297Z

This post is provided as convenient place to discussion of the new book, The AI Does Not Hate You by Tom Chivers, which covers LessWrong and rationalist community.

The AI Does Not Hate You: Superintelligence, Rationality and the Race to Save the World

This is a book about AI and AI risk. But it's also more importantly about a community of people who are trying to think rationally about intelligence, and the places that these thoughts are taking them, and what insight they can and can't give us about the future of the human race over the next few years. It explains why these people are worried, why they might be right, and why they might be wrong. It is a book about the cutting edge of our thinking on intelligence and rationality right now by the people who stay up all night worrying about it.

Note that the book is available on Kindle only to people with UK/European Amazon accounts. I was able to order it in physical copy in the US, but I haven't received a shipping notification yet.
Research Agenda v0.9: Synthesising a human's preferences into a utility function
by Stuart_Armstrong — 2019-06-17T17:46:39.317Z

I'm now in a position where I can see a possible route to a safe/survivable/friendly Artificial Intelligence being developed. I'd give a 10+% chance of it being possible this way, and a 95% chance that some of these ideas will be very useful for other methods of alignment. So I thought I'd encode the route I'm seeing as research agenda; this is the first public draft of it.

Clarity, rigour, and practicality: that's what this agenda needs. Writing this agenda has clarified a lot of points for me, to the extent that some of it now seems, in retrospect, just obvious and somewhat trivial - "of course that's the way you have to do X". But more clarification is needed in the areas that remain vague. And, once these are clarified enough for humans to understand, they need to be made mathematically and logically rigorous - and ultimately, cashed out into code, and tested and experimented with.

So I'd appreciate any comments that could help with these three goals, and welcome anyone interested in pursuing research along these lines over the long-term.

Note: I periodically edit this document, to link it to more recent research ideas/discoveries.

0 The fundamental idea

This agenda fits itself into the broad family of Inverse Reinforcement Learning: delegating most of the task of inferring human preferences to the AI itself. Most of the task, since it's been shown that humans need to build the right assumptions into the AI, or else the preference learning will fail.

To get these "right assumptions", this agenda will look into what preferences actually are, and how they may be combined together. There are hence four parts to the research agenda:

A way of identifying the (partial[1]) preferences of a given human H.

A way for ultimately synthesising a utility function UH that is an adequate encoding of the partial preferences of a human H.

Practical methods for estimating this UH, and how one could use the definition of UH to improve other suggested methods for value-alignment.

Limitations and lacunas of the agenda: what is not covered. These may be avenues of future research, or issues that cannot fit into the UH paradigm.

There has been a myriad of small posts on this topic, and most will be referenced here. Most of these posts are stubs that hint to a solution, rather than spelling it out fully and rigorously.

The reason for that is to check for impossibility results ahead...
Preference conditional on circumstances and past preference satisfaction
by Stuart_Armstrong — 2019-06-17T15:30:32.580Z

I've mentioned conditional preferences before. These are preferences that are dependent on facts about the world, for example "I'd want to believe X if there are strong argument for X".

But there is another type of preference that is conditional: my tastes can vary depending on circumstances and on my past experience. For example, I might prefer to eat apples during the week and oranges on weekends. Or, because of the miracle of boredom, I might prefer oranges if (but only if) I've been eating apples all week so far.

What if I currently want apples, would want oranges tomorrow, but falsely believe (today) that I would want apples tomorrow? This is a known problem with "one-step hypotheticals", and a strong argument in practice for assessing preferences over time rather than at a single moment t.

In theory, there are meta-preferences that allow one to get this even at a single moment t, such as "I want to be able to follow my different tastes at different times" or a more formalised desire for variety and exploration.
In physical eschatology, is Aestivation a sound strategy?
by mako yass — 2019-06-17T07:27:31.527Z

In this paper, Anders Sandberg, Stuart Armstrong and Milan M. Cirkovic argue that

If a civilization wants to maximize computation it appears rational to aestivate until the far future in order to exploit the low temperature environment: this can produce a 1030 multiplier of achievable computation.

Later Charles H. Bennett, Robin Hanson, C. Jess Riedel disagree, claiming

In fact, while this assumption may apply in the distant future, our universe today contains vast reservoirs and other physical systems in non-maximal entropy states, and computer-generated entropy can be transferred to them at the adiabatic conversion rate of one bit of negentropy to erase one bit of error. This can be done at any time, and is not improved by waiting for a low cosmic background temperature. Thus aliens need not wait to be active. As Sandberg et al. do not provide a concrete model of the effect they assert, we construct one and show where their informal argument goes wrong.

Who was right?
Is there a guide to 'Problems that are too fast to Google'?
by Vaniver — 2019-06-17T05:04:39.613Z

It seems to me like problems come in a variety of required response speeds, but there's a natural threshold to distinguish fast and slow: whether or not you can Google it. The slow ones, like getting an eviction notice from your landlord or a cancer diagnosis from your doctor, can't be ignored but you have time to figure out best practices before you act. Fast ones, like getting bit by a rattlesnake or falling from a high place, generally require that you already know best practices in order to properly implement them.

Also useful would be the meta-guide, which just separates out which problems are fast and slow (or how fast they are). Getting bit by a tick, for example, seems like it might be quite urgent when you discover one biting you, but isn't; you have about 24 hours from when it first attaches to remove it, which is plenty of time to research proper removal technique. Getting a bruise might seem like you have time, but actually applying cold immediately does more to prevent swelling than applying cold later does to reduce it.

Of course, this is going to vary by region, profession, age, sex, habits, and so on. I'm sort of pessimistic about this existing at all, and so am interested in whatever narrow versions exist (even if it's just "here's what you need to know about treating common injuries to humans"). Basic guides also seem useful from a 'preventing illusion of transparency' perspective.
ISO: Automated P-Hacking Detection
by johnswentworth — 2019-06-16T21:15:52.837Z

I'm sure there's some ML students/researchers on Lesswrong in search of new projects, so here's one I'd love to see and probably won't build myself: an automated method for predicting which papers are unlikely to replicate, given the text of the paper. Ideally, I'd like to be able to use it to filter and/or rank results from Google scholar.

Getting a good data set would probably be the main bottleneck for such a project. Various replication-crisis papers which review replication success/failure for tens or hundreds of other studies seem like a natural starting point. Presumably some amount of feature engineering would be needed; I doubt anyone has a large enough dataset of labelled papers to just throw raw or lightly-processed text into a black box.

Also, if anyone knows of previous attempts to do this, I'd be interested to hear about it.
Does scientific productivity correlate with IQ?
by Eli Tyre — 2019-06-16T19:42:29.980Z

Anders Ericsson, in his popular book Peak: Secrets from the New Science of Expertise, claims that though one may need a reasonably high IQ to even be a scientist, after that level, IQ is irrelevant to scientific success.

The average IQ of scientists is certainly higher than the IQ of the general population, but among scientists, there is no correlation between IQ and scientific productivity. Indeed, a number of noble prize winning scientists have had IQs that would not even qualify them for Mensa, an organization who's members must have a measured IQ of at least 132, a number that put's you in the upper 2 percentile of the population.

(From chapter 8: "But What About Natural Talent", of Peak: Secrets from the New Science of Expertise)

My understanding is that this is completely wrong, and the best scientists tend to have higher IQs, at all levels of performance. But this is a background belief that I haven't concretely verified.

Is there a canonical reference regarding the impact of IQ on scientific contribution?
Does the _timing_ of practice, relative to sleep, make a difference for skill consolidation?
by Eli Tyre — 2019-06-16T19:12:48.358Z

It is well known that sleep (both mid-day naps and nighttime sleep) has a large effect on the efficacy of motor skill acquisition. Performance on a newly learned task improves, often markedly, following a period of sleep.

A few citations (you can find many more by searching "motor skill acquisition sleep" or similar in google scholar) :

https://www.nature.com/articles/nn1959

https://journals.plos.org/plosone/article?id=10.1371/journal.pone.0000341

https://link.springer.com/article/10.1111/j.1479-8425.2012.00576.x

I want to know if the _timing_ of practice, relative to sleep, makes a difference for skill acquisition.

For instance, if you practice a skill at 7:00 PM, shortly before a night of sleep, will your performance be better in the morning than if you had practiced at 7:00 AM had a full day of wakefulness, and _then_ gone to sleep? If so, what is the effect size?

Josh Kaufman makes a claim to this effect in his book, The First 20 Hours. I have no particular reason to doubt him, 40 minutes of searching on google scholar did not turn up any papers about the importance of sleep and practice timing.

Can you point me at a relevant citation?
Is "physical nondeterminism" a meaningful concept?
by Liron — 2019-06-16T15:55:58.198Z

Background: Our mental models of the universe can contain uncertainty or probability-links, as in a causal network. One may have a deterministic understanding of a phenomenon if the probability-values are all 0 and 1.

Question: Beyond that, is it meaningful to distinguish whether or not the *universe itself* is deterministic or nondeterministic?

For example, is it meaningful to say that the Copenhagen interpretation of QM implies a "nondeterministic universe", while Many Worlds implies a "deterministic universe"?
Reasonable Explanations
by Alicorn — 2019-06-16T05:29:09.873Z

Today I watched a friend do calibration practice and was reminded of how wide you have to cast your net to get well-calibrated 90% confidence. This is true even when the questions aren't gotchas, just because you won't think of all the ways something could be wildly unlike your quick estimate's model. Being well-calibrated for 90% confidence intervals (even though this doesn't sound all that confident!) requires giving lots of room even in questions you really do know pretty well, because you will feel like you really do know pretty well when in fact you're missing something that wrecks you by an order of magnitude.

Being miscalibrated can feel like "if it were outside of this range, I have just... no explanation for that" - and then it turns out there's a completely reasonable explanation.

Anyway, I thought a fun exercise would be describing weird situations we've encountered that turned out to have reasonable explanations. In initial descriptions, present only the information you (and whoever was thinking it over with you) remembered to consider at the time, then follow up in ROT-13 with what made the actual sequence of events come clear.
Accelerate without humanity: Summary of Nick Land's philosophy
by Yuxi_Liu — 2019-06-16T03:22:04.474Z

I took note of the philosopher Nick Land from reading about posthumanism on Wikipedia.

A more pessimistic alternative to transhumanism in which humans will not be enhanced, but rather eventually replaced by artificial intelligences. Some philosophers, including Nick Land, promote the view that humans should embrace and accept their eventual demise. This is related to the view of "cosmism", which supports the building of strong artificial intelligence even if it may entail the end of humanity, as in their view it "would be a cosmic tragedy if humanity freezes evolution at the puny human level".

I was intrigued by such boldness, so I read more. And turns out Nick Land's writing is sometimes easy to read but most of the times extremely hard to read, and probably garbage. I wrote this post so that you don't have to waste time wading through the garbage, looking for fragments of good poetry.

Nothing human makes it out of the near-future. -- Nick Land

First of all, Nick Land was obsessed with hating Kant, loving Gilles Deleuze and Félix Guattari and their philosophical style (called schizoanalysis, related to schizophrenia). He likes to think about the world from very nonhuman viewpoints, such as other animals, robots, computers, machines that humans made, earth, the universe, etc. He likes capitalism and technological revolution, as fast as possible, without regard for its goodness.

Recently there's some mainstream reports on his philosophy of Neoreactionism ("Dark Enlightenment"), the idea that democracy sucks and monarchy/CEO-president works better. This philosophy has gained a bit of following, but uninteresting to me, so we won't review that. I'd simply note that the phrase "Dark Enlightenment" really should be "Delightenment". Really missing out such a pun.

Schizoanalysis

The idea of schizoanalysis just means that there's a lot of ways to make a theory about the world, and make philosophies, and there's no one way to do it, and further, there could be genuine conflicts that cannot be resolved by appealing to a higher standard.

In mathematics, there's some fringe movement of this style. Most mathematicians are in favor of logical consistency, but some are okay with controlled inconsistency (paraconsistency). Most mathematicians are in favor of using infinities, but some are finitists who think that infinities don't exist, and a few are ultrafinitists who th

...
28 social psychology studies from *Experiments With People* (Frey & Gregg, 2017)
by Yuxi_Liu — 2019-06-16T02:23:08.500Z

I'm reading a very informative and fun book about human social psychology, Experiments With People (2nd ed, 2018).

... 28 social psychological experiments that have significantly advanced our understanding of human social thinking and behavior. Each chapter focuses on the details and implications of a single study, while citing related research and real-life examples along the way.

Here I summarize each chapter so that you can save time. Some results are old news to me, but some were quite surprising. I often skip over the experimental details, such as how the psychologists used ingenious tricks to make sure the participants don't guess the true purposes of the experiments. Refer to originals for details.

The experiments start in the 1950s and get up to 2010s, and occasionally literatures from before 1900s are quoted.

Chapters I find especially interesting are:

Chap 14. It lists the many failures of introspection, and raises question as to what consciousness can do.

Chap 16. It has significant similarity with superrationality and acausal trade.

Chap 20. It warns about how credulous humans are.

Chap 27. It is about the human fear of death and the psychological defenses against it.

Chap 28. It shows how belief in free will can be motivated by a desire to punish immoral behaviors. Understanding why people believe in free will is necessary for a theory of what is the use of the belief in free will.

Chap 1. Conforming to group norms

Asch conformity experiment, from Opinions and Social Pressure (Asch, 1955)

Video demonstration.

Groups of eight participated in a simple "perceptual" task. In reality, all but one of the participants were actors, and the true focus of the study was about how the remaining participant would react.

Each student viewed a card with a line on it, followed by another with three lines labeled A, B, and C (see accompanying figure). One of these lines was the same as that on the first card, and the other two lines were clearly longer or shorter. Each participant was then asked to say aloud which line matched the length of that on the first card... The actors would always unanimously nominate one comparator, but on certain trials they would give the correct response and on others, an incorrect response. The group was seated such that the real participant always responded last.

It was found that

When there are over 3 actors giving unanimo

...
The Univariate Fallacy
by Zack_M_Davis — 2019-06-15T21:43:14.315Z

(A standalone math post that I want to be able to link back to later/elsewhere)

There's this statistical phenomenon where it's possible for two multivariate distributions to overlap along any one variable, but be cleanly separable when you look at the entire configuration space at once. This is perhaps easiest to see with an illustrative diagram—

The denial of this possibility (in arguments of the form, "the distributions overlap along this variable, therefore you can't say that they're different") is sometimes called the "univariate fallacy." (Eliezer Yudkowsky proposes "covariance denial fallacy" or "cluster erasure fallacy" as potential alternative names.)

Let's make this more concrete by making up an example with actual numbers instead of just a pretty diagram. Imagine we have some datapoints that live in the forty-dimensional space {1, 2, 3, 4}⁴⁰ that are sampled from one of two probability distibutions, which we'll call PA and PB.

For simplicity, let's suppose that the individual variables x₁, x₂, ... x₄₀—the coördinates of a point in our forty-dimensional space—are statistically independent and identically distributed. For every individual xi, the marginal distribution of PA is—

PA(xi)=⎧⎪
⎪
⎪⎨⎪
⎪
⎪⎩1/4xi=17/16xi=21/4xi=31/16xi=4

And for PB—

PB(xi)=⎧⎪
⎪
⎪⎨⎪
⎪
⎪⎩1/16xi=11/4xi=27/16xi=31/4xi=4

If you look at any one xi-coördinate for a point, you can't be confident which distribution the point was sampled from. For example, seeing that x₁ takes the value 2 gives you a 7/4 (= 1.75) likelihood ratio in favor of that the point having been sampled from PA rather than PB, which is log₂(7/4) ≈ 0.807 bits of evidence.

That's ... not a whole lot of evidence. If you guessed that the datapoint came from PA based on that much evidence, you'd be wrong about 4 times out of 10. (Given equal (1:1) prior odds, an odds ratio of 7:4 amounts to a probability of (7/4)/(1 + 7/4) ≈ 0.636.)

And yet if we look at many variables, we can achieve supreme, godlike confidence about which distribution a point was sampled from. Proving this is left as an exercise to the particularly intrepid reader, but a concrete demonstration is probably simpler and should be pretty convincing! Let's write some Python code to sample a point →x ∈ {1, 2, 3, 4}⁴⁰ from PA—

import random

def a():
return random.sample(
[1]*4 + # 1/4
[2]*7 + # 7/16
[3]*4 + # 1/4
[4], # 1/1

...
Do policy makers frequently underestimate non-human threats (ie: epidemic, natural disasters, climate change) when compared to threats that are human in nature (ie: military conflict, economic competition, Cold War)?
by RocksBasil — 2019-06-15T16:29:47.755Z

Epidemic status: A random thought that I don’t feel adequate talking about.
However, I believe this topic must’ve been discussed by somebody somewhere, so if my question reminds you of something similar, please point me to it.

It appears to me that policy makers seem to frequently prioritise dealing with threat posed by human players and neglect threat posed by the natural environment, even when the latter could be demonstrably more worrisome than the former.

Almost as if we are “designed” to deal with hostility from other humans, but are badly equipped to perceive and deal with a hostile natural environment.

I still struggle to express my point, so I’ll put in a couple examples and see how it goes. These examples don't serve as evidence, as I made some sweeping statements about politics that won't necessarily be true (stereotypically true, perhaps), just to show what my question is concerning.

Example 1: Cold War and climate change.
For example, the USSR (ignoring the effect of nuclear weapons) during the Cold War probably didn’t pose an existential threat to the Western countries, while an impeding climate catastrophe might be.
Yet policy makers during the Cold War felt the need to commit significant amount of resources to counteract Soviet influence across the globe, including the costly (and arguably futile and nonsensical) wars in Korea and Vietnam, while political actions on the climate issue seems rather half-minded and still controversial.
Was it because climate change is too far away while the threat of communism seemed imminent? I don’t think so, since the reasoning behind the wars and many other geopolitical manoeuvres were likely long term, on the scale of decades if not longer—our environmental policy won’t differ much in this regard.

Example 2: WWI and influenza

In year 1918, even after the Influenza was observed to be extremely dangerous, authorities in the US were reluctant to implement public health measures and suppressed truthful reporting of the epidemic hoping to keep public morale high for WWI.
Which certainly backfired when the epidemic became obvious to the public, dismantling their trust in the authority. Critical judgement error indeed.
Again, it seems that the threat of the epidemic was downplayed in the mind of policy makers, even though it was already observed to be potent and deadly, and ultimately proven to be mo...