trentmkelly/LessWrong-43k huggingface dataset:
Book Summary: Consciousness and the Brain
by Kaj_Sotala — 2019-01-16T14:43:59.202Z

One of the fundamental building blocks of much of consciousness research, is that of

Global Workspace Theory (GWT)

. One elaboration of GWT, which focuses on how it might be implemented in the brain, is the Global Neuronal Workspace (GNW) model in neuroscience. Consciousness and the Brain is a 2014 book that summarizes some of the research and basic ideas behind GNW. It was written by

Stanislas Dehaene

, a French cognitive neuroscientist with a long background in both consciousness research and other related topics.

The book and its replicability

Given that this is a book on psychology and neuroscience that was written before the replication crisis, an obligatory question before we get to the meat of it is: how reliable are any of the claims in this book? After all, if we think that this is based on research which is probably not going to replicate, then we shouldn’t even bother reading the book.

I think that the book’s conclusions are at least reasonably reliable in their broad strokes, if not necessarily all the particular details. That is, some of the details in the cited experiments may be off, but I expect most of them to at least be pointing in the right direction. Here are my reasons:

First, scientists in a field usually have an informal hunch of how reliable the different results are. Even before the replication crisis hit, I had heard private comments from friends working in social psychology, who were saying that everything in the field was built on shaky foundations and how they didn’t trust even their own findings much. In contrast, when I asked a friend who works with some people doing consciousness research, he reported back that they generally felt that GWT/GNW-style theories have a reasonably firm basis. This isn’t terribly conclusive but at least it’s a bit of evidence.

Second, for some experiments the book explicitly mentions that they have been replicated. That said, some of the reported experiments seemed to be one-off ones, and I did not yet investigate the details of the claimed replications.

Third, this is a work of cognitive neuroscience. Cognitive neuroscience is generally considered a subfield of cognitive psychology, and cognitive psychology is the part of psychology whose results have so far replicated the best. One recent study tested nine key findings from cognitive psychology, and

found that they all replicated

. The 2015 "

Estimati

...
What AI Safety Researchers Have Written About the Nature of Human Values
by avturchin — 2019-01-16T13:59:31.522Z

Overview

It is clear to most AI safety researchers that the idea of “human values” is underdefined, and this concept should be additionally formalized before it can be used in (mostly mathematical) models of AI alignment. In other words, “AI-conscious” human value theories are those which were created especially in order to help AI alignment research. Most existing psychological theories of human values are verbal, informal and underdefined, so they require some adaptation before they may be applied to AI safety.

In many cases, the theory of human values cannot be distinguished from the ways how values are expected to be extracted by a future AI, for example, by approval-directed AI. Some researchers (notably, Armstrong) have also presented different theories; providing a link to the researcher is not intended as a claim that said researcher currently adheres to exactly this theory.

In Figure 1, all current theories of human values are classified according to two main parameters:

1) Complexity of the values’ description (not the complexity of the theory): Some theories of human values assume that human values are very simple, e.g., that there are only two drives: survival and reproduction; or only one desire, that of maximizing pleasure; or only 9 basic emotions. Other theories suggest that human values are very complex, e.g. they are a matrix of association between all concepts and rewards.

2) The level of “behaviorism”: Most theories are attracted to one of two poles: “internalist” theories, which assume that values actually exist, but are hidden inside the human brain, and “behaviorist” theories, which assume that values are only appear in human behavior (like approval or choice). Behaviorist theories of human values (generally) integrate the values and the method of their extraction (e.g. approval-directed AI). In contrast, internalist theories typically hold that values exist separately of the ways they are learned.

These two parameters are not strict, but serve as a useful instrument to orientate ourselves within the field of all possible theories.

Figure 1. Different real and possible theories about the nature of human values in AI safety

Another way to classify theories about human values is according to the level of abstractness: some theories could be applied to any possible mind and thus don’t take any inputs from human psychology and neurophysiolog...
Is there a.. more exact.. way of scoring a predictor's calibration?
by mako yass — 2019-01-16T08:19:15.744Z

On platforms like PredictionBook, a user's credibility will be measured by looking at a histogram of all of their predictions (edit: After being told about "brier score", I have become able to notice that PredictionBook totally shows a brier score right under the histogram. Still, thinking about brier scores, they're not very good either. You can only maximise your brier score by betting on things you're certain of, it creates perverse incentives to avoid placing bets on uncertain events, which we already have enough of in natural discourse). For a perfectly well calibrated agent, exactly 90% of their 0.9 predictions should have come true, exactly 20% of their 0.2 predictions should have come true, etc. We can reduce a person's calibration to a number by, I suppose, getting the sum of squares of the differences of each histogram from its middle point (I don't know if that's how they do it).

But whatever they do with the histogram, I think I'll find it very unsatisfying, because the result are going to depend on what spacing we used for the histogram segments, and that's arbitrary. We could have a histogram with ten segments, or a histogram with fifty segments, we'll get different scores. I feel like there must be some exact.. continuous way of scoring a predictor's calibration, and that the math will probably be generally useful for other stuff. Is there a method?
What are the components of intellectual honesty?
by abramdemski — 2019-01-15T20:00:09.144Z

I have a pretty strong intuition that "intellectual honesty" points to a specific thing, with a characteristic set of behaviors (both outward (in a conversation/writing) and inward (patterns of thinking)). My concept of intellectual honesty also seems to largely coincide with the concept as used by others, but I'm not sure.

I am talking about a high standard. This isn't at all about violating a code of academic ethics.

I am talking about the sort of thing which helps to foster epistemic trust, helping to take conversations to higher levels. (But don't read those links if you already know enough what I mean to answer and want to avoid anchoring your view to mine.)
Critical Thinking in Medicine 
by Yoav Ravid — 2019-01-15T15:09:42.716Z
Link: https://sciencebasedmedicine.org/critical-thinking-in-medicine/

This is a book review of Cognitive Errors and Diagnostic Mistakes.

though i haven't read the book, it seems very interesting and i think many others here will find it interesting.

Here's the description of the article:

Cognitive Errors and Diagnostic Mistakes is a superb new guide to critical thinking in medicine written by Jonathan Howard. It explains how our psychological foibles regularly bias and betray us, leading to diagnostic mistakes. Learning critical thinking skills is essential but difficult. Every known cognitive error is illustrated with memorable patient stories.
Human-AI Interaction
by Rohin Shah — 2019-01-15T01:57:15.558Z

The importance of feedback

Consider trying to program a self-driving car to drive from San Francisco to Los Angeles -- with no sensors that allow it to gather information as it is driving. This is possible in principle. If you can predict the exact weather conditions, the exact movement of all of the other cars on the road, the exact amount of friction along every part of the road surface, the exact impact of (the equivalents of) pressing the gas or turning the steering wheel, and so on, then you could compute ahead of time how exactly to control the car such that it gets from SF to LA. Nevertheless, it seems unlikely that we will ever be able to accomplish such a feat, even with powerful AI systems.

No, in practice there is going to be some uncertainty about how the world is going to evolve; such that any plan computed ahead of time will have some errors that will compound over the course of the plan. The solution is to use sensors to gather information while executing the plan, so that we can notice any errors or deviations from the plan, and take corrective action. It is much easier to build a controller that keeps you pointed in the general direction, than to build a plan that will get you there perfectly without any adaptation.

Control theory studies these sorts of systems, and you can see the general power of feedback controllers in the theorems that can be proven. Especially for motion tasks, you can build feedback controllers that are guaranteed to safely achieve the goal, even in the presence of adversarial environmental forces (that are bounded in size, so you can’t have arbitrarily strong wind). In the presence of an adversary, in most environments it becomes impossible even in principle to make such a guarantee if you do not have any sensors or feedback and must compute a plan in advance. Typically, for every such plan, there is some environmental force that would cause it to fail.

The control theory perspective on AI alignment

With ambitious value learning, we’re hoping that we can learn a utility function that tells us the optimal thing to do into the future. You need to be able to encode exactly how to behave in all possible environments, no matter what new things happen in the future, even if it’s something we humans never considered a possibility so far.

This is analogous to the problem of trying to program a self-driving car. Just as in that case...
Anthropic probabilities: answering different questions
by Stuart_Armstrong — 2019-01-14T18:50:56.086Z

What is the answer to the question of anthropic probabilities? I've claimed before that there was no answer - anthropic decision theory (ADT) was the only way to go.

But actually, there are answers - the problem is simply that there are multiple versions of "the question of anthropic probabilities", each with their own answer. And what decision theory did was unambiguously select the right question (and the right answer) for the job.

Frequentist anthropic probabilities

It is much easier for humans to think in terms of frequencies, and anthropic probabilities are no exceptions. So imagine that either a small universe (with one copy of you) or a large universe (with ω copies of you) is created, with equal probability. Then your copies will independently observe a large sequence of Ω random bits, with 2Ω>>ω. After that, the universe ends, and the whole experiment begins again, with a small or large universe being created again. This experiment will then be repeated a very large number of times, so we can coherent talk about frequencies.

Then there are three questions you might ask yourself during these experiments:

What proportion of my versions will be in a large universe?

What proportion of universes, where versions of me exist, will be large?

What proportion of universes, where exact copies of me exist, will be large?

In the limit as these experiments are run a large number of times, the answers to these questions will converge on ω/(1+ω), 1/2, and "it depends on how many random bits you have seen when you ask the question". In other words, the probabilities given by SIA, SSA, and FNC.

Notice there is a fourth question that we could ask to complete the three:

What proportion my exact copies will be in a large universe?

But this question will also converge to ω/(1+ω), ie SIA, showing how SIA is independent of the reference class as long as the reference class contains at least the exact copies.

The issues with the questions

All three questions are well-posed questions with exact and correct answers. From outside, however, there are issues with all three.

Question 2 has the perennial "reference class problem": what are you counting as "versions of me"? If we change the reference class, we change the question, and therefore its not surprising it gives a different answer.

Question 3 has the same time inconsistency that FNC has: the answer will be (p

...
A short insight I see everywhere
by ArthurLidia — 2019-01-14T18:16:39.352Z

Barbara Oakley talked about two minds system (1). In Buddhism (Theravada is my area of expertise) we have awareness (introspective awareness) and concentration (narrow focus).
Also this weird feature is a byproduct that humans focused on the ground, and then they diffused looking forward at the horizont looking over more area for danger. Also Elizer Yudkowsky points this out in an essay. (2) closing your eyes is the best blackhole of ideas you can throw until something good works.

“Which leads into another good question to ask yourself straight out: Did I spend five minutes with my eyes closed, brainstorming wild and creative options, trying to think of a better alternative? It has to be five minutes by the clock, because otherwise you blink—close your eyes and open them again—and say, “Why, yes, I searched for alternatives, but there weren’t any.” Blinking makes a good black hole down which to dump your duties. An actual, physical clock is necessary”.

Magnus Carlsen did it when he was 13 against Kasparov. The idea was when he gets stuck he not only solves that but he also throws Kasparov off. (3)

Not only this kind of thinking is useful, is great for math and science (formal logic even). For further reading well researched books:

A Mind For Numbers: How to Excel at Math and Science (Even If You Flunked ...

The Mind Illuminated: A Complete Meditation Guide Integrating (Seen it many times on Hacker News, the best description is "technical manual for your mind")

(1) She called them defused , like far apart pong dots, and close and narrow for tunnel vision. The main idea is that the book is insightful on meta-learning.
(2) “47. The third alternative” by Eliser Yudkowsky.In the book “A mind for number” by Barbara Oakley they talk about closing your eyes and just asking “did that really make sense?” Is the best way to stop tunnel vision.
(3) spoiler: The game ended in draw. Kasparov later became his mentor. You can find the match by searching young Carlsen vs Kasparov.
Anthropics: Full Non-indexical Conditioning (FNC) is inconsistent
by Stuart_Armstrong — 2019-01-14T15:03:04.288Z

I'll be using Full non-indexical conditioning (FNC) as an example of anthropic reasoning. Given that, it's useful to know that FNC is (time) inconsistent: the expected future probability is not the same as the current probability. I can't find a full write-up of this specific fact, so I thought I'd present it here for reference.

In FNC:

one should condition on all the evidence available, including all the details of one’s memory, but without considering “indexical” information regarding one’s place in the universe (as opposed to what the universe contains).

So, suppose that there were either 1 or ω initially identical copies of you in the universe, with equal probability. Your copies never communicate or get extra evidence of each others existence. Each copy of has seen (and remembered) the outcome of Ω independent random bits, for 2Ω>>ω. You yourself have seen the sequence s (which sequence s is doesn't actually matter, as they are all equiprobable).

Then, by FNC reasoning, the probability of a copy of you seeing s in a universe with only 1 of you is 1/2Ω, while the probability of that happening in the a universe with ω copies of you is:

1−(1−12Ω)ω≈1−(1−ω2Ω)=ω2Ω, by the binomial approximation.

Thus FNC will update towards the large universe by a ratio that is roughly ω/(2Ω):1/(2Ω) - in other words, ω:1. Translated into probabilities, this gives a probability close to 1/(ω+1) for the small universe, and ω/(ω+1) for the large one.

This makes FNC very close to SIA; in fact, in the limit of finitely many copies remembering infinitely many different random bits, FNC is exactly SIA.

But now consider the situation before any of your copies see any random bits (or, at least, where they only see non-independent random bits). Then all copies have seen the same sequence, so the update is 1:1; ie there is no anthropic update at all, and the probabilities of either universe remains 1/2.

But if you know that all your copies will see an independent sequence Ω random bits, then you can predict, with certainty, that your future probabilities will be (almost exactly) 1/(ω+1) and ω/(ω+1), rather than 1/2 and 1/2. How can you predict with certainty? Because you know that you will see some sequence s, and all sequence s lead to the same FNC updates.

So FNC is time inconsistent.

Forgetful agents.

More strangely, FNC can be time inconsistent the other way, if your

...
The Tether Theory and the Concrete, Subtle and Causal tiers
by Elo — 2019-01-14T00:07:50.829Z

Epistemic status: A bunch of meditation and meditation theory. A previously undocumented theory of this corner of consciousness.

Original post on my blog bearlamp

The book The Science of Pranayama offers that the thought stream is tethered to the breath via the prana, “Just as the bird that is tied to a post by a string”. Meditation theory often uses animal analogies like the Ox Herder pictures. Pointing out the great way describes, “It requires some effort to tie up a wild camel, and it takes some skill to untie the camel and have it stay”. Other wild animals are also common to describe the untamed mind.

I put a lot of thought into this concept and applying it further than it was originally suggested. I meditated while trying to observe for myself what was going on and why. There's something subtle about the way a deep breath out will interact with the thought stream. Feels like a sigh of letting go. I watch “emotional content” conversations and when I feel the need to take a deep breath, I notice when I don't and when I should.

The Tiers of subjective experience

(Side note: Concrete is sometimes called “gross” and causal is sometimes called “very subtle” in differentmaps)

Concrete: This is the physical, physiological. The beating heart, the breath.

A concrete tier injury looks like a broken leg. Or a scratch wound. It takes time to heal. Feeling hungry is not on the concrete level, we don’t die from feeling hungry (we die from starvation). Feeling cold usually starts on the subtle level but there is a life threatening feeling of cold.

There is a bridge between Concrete and subtle tiers through experiences like “goosebumps” where they register as a physical state and as a subtle feeling, or some mix of the two.

Subtle: This is an aptly named tier because - it’s subtle. An experience of the emotional subjective happens here. I feel happy, I feel guilty. A few "weird" feelings and hunches like “gut instinct” that seem guiding but without being as easy to describe as the concrete tier.

A subtle injury is a panic attack or being distraught. It is possible to be in an extended state of injury to this level but it's not possible to “break a happy” or break a miserable in a way that it becomes permanently injured and requires time to heal like a broken bone or a scratch. It is possi

...
CDT=EDT=UDT
by abramdemski — 2019-01-13T23:46:10.866Z

Epistemic status: I no longer endorse the particular direction this post advocates, though I'd be excited if someone figured out something that seems to work. I still endorse most of the specific observations.

So... what's the deal with counterfactuals?

Over the past couple of years, I've been writing about the CDT=EDT perspective. I've now organized those posts into a sequence for easy reading.

I call CDT=EDT a "perspective" because it is a way of consistently answering questions about what counterfactuals are and how they work. At times, I've argued strongly that it is the correct way. That's basically because:

it has been the only coherent framework I put any stock in (more for lack of other proposals for dealing with logical counterfactuals than for an abundance of bad ones);
there are strong arguments for it, if you're willing to make certain assumptions;
it would be awfully nice to settle this whole question of counterfactual reasoning and move on. CDT=EDT is in a sense the most boring possible answer, IE that all approaches we've thought of are essentially equivalent and there's no hope for anything better.

However, recently I've realized that there's a perspective which unifies even more approaches, while being less boring (more optimistic about counterfactual reasoning helping us to do well in decision-theoretic problems). It's been right in front of me the whole time, but I was blind to it due to the way I factored the problem of formulating decision theory. It suggests a research direction for making progress in our understanding of counterfactuals; I'll try to indicate some open curiosities of mine by the end.

Three > Two

The claim I'll be elaborating on in this post is, essentially, that the framework in Jessica Taylor's post about memoryless cartesian environments is better than the CDT=EDT way of thinking. You'll have to read the post to get the full picture if you haven't, but to briefly summarize: if we formalize decision problems in a framework which Jessica Taylor calls "memoryless cartesian environments" (which we can call "memoryless POMDPs" if we want to be closer to academic CS/ML terminology), reasoning about anthropic uncertainty in a certain way (via the self-indication assumption, SIA for short) makes it possible for CDT to ...