trentmkelly/LessWrong-43k huggingface dataset:
Melatonin: Much More Than You Wanted To Know
by Scott Alexander — 2018-07-11T17:40:06.069Z

[I am not a sleep specialist. Please consult with one before making any drastic changes or trying to treat anything serious.]

Van Geijlswijk et al describe supplemental melatonin as “a chronobiotic drug with hypnotic properties”. Using it as a pure hypnotic – a sleeping pill – is like using an AK-47 as a club to bash your enemies’ heads in. It might work, but you’re failing to appreciate the full power and subtlety available to you.

Melatonin is a neurohormone produced by the pineal gland. In a normal circadian cycle, it’s lowest (undetectable, less than 1 pg/ml of blood) around the time you wake up, and stays low throughout the day. Around fifteen hours after waking, your melatonin suddenly shoots up to 10 pg/ml – a process called “dim light melatonin onset”. For the next few hours, melatonin continues to increase, maybe as high as 60 or 70 pg/ml, making you sleepier and sleepier, and presumably at some point you go to bed. Melatonin peaks around 3 AM, then declines until it’s undetectably low again around early morning.

Is this what makes you sleepy? Yes and no. Sleepiness is a combination of the circadian cycle and the so-called “Process S”. This is an unnecessarily sinister-sounding name for the fact that the longer you’ve been awake, the sleepier you’ll be. It seems to be partly regulated by a molecule called adenosine. While you’re awake, the body produces adenosine, which makes you tired; as you sleep, the body clears adenosine away, making you feel well-rested again.

In healthy people these processes work together. Circadian rhythm tells you to feel sleepy at night and awake during the day. Process S tells you to feel awake when you’ve just risen from sleep (naturally the morning), and tired when you haven’t slept in a long time (naturally the night). Both processes agree that you should feel awake during the day and tired at night, so you do.

When these processes disagree for some reason – night shifts, jet lag, drugs, genetics, playing Civilization until 5 AM – the system fails. One process tells you to go to sleep, the other to wake up. You’re never quite awake enough to feel energized, or quite tired enough to get restful sleep. You find yourself lying in bed tossing and turning, or waking up while it’s still dark and not being able to get back to sleep.

Melatonin works on both systems. It has a weak “hypnotic” effect on Process S, making you immediate...
Monk Treehouse: some problems defining simulation
by dranorter — 2018-07-11T07:35:12.058Z

When does one program simulate another? When does one program approximately simulate another?

In some contexts, these questions have concrete answers. In other contexts they feel about the same as the problem of functionalist triviality. When simulation comes up in contexts relevant to agent foundations, it tends to feel like the latter. Here, I outline some arguments which I've used in the past to show that a given concept of simulation doesn't quite accomplish what it sets out to do.

For sake of concreteness, I will give two problems which we can imagine solving with an appropriate definition of simulation.

First, consider the problem of taking counterfactuals in order to evaluate consequences of our actions. We may wish to seek copies of our own algorithm in the environment in order to counterfact on them (or to make a mere assumption that their output will be the same as ours). Here, I assume we're trying to solve this problem by looking for sub-components (or properties) of some world model which meet some criterion; namely, forming a simulation of us, according to some formal definition. We've already decided on a program P which represents us (possibly more than one, but we take interest in one at at time), and we are comparing P to (our models of) the environment in some sense. We expect to find one copy of P — ourselves — and we are looking for others.

As a second example, suppose that we have in hand a mathematical description of an AI which we are considering building. Suppose that the AI has various safety and transparency properties, but all these properties are stated in terms of the agent's beliefs and actions, where beliefs are stored in some specified format and actions are output via a specified channel. We are interested in knowing whether building the agent will lead to any "emergent subagents", and if so, whether those agents will be aligned with the original agent (and with us), and what safety and transparency properties those agents might have. In pursuit of this goal, we are attempting to prove theorems about whether certain programs P representing subagents can be simulated on certain sub-components (or properties) of our AI.

A Strawman

Again for sake of concreteness, I'll put forward a straw definition of simulation. Suppose the program P we're searching for simulations of is represented as a ...
Mathematical Mindset
by komponisto — 2018-07-11T03:03:11.689Z

I agree that optimization amplifies things. I also agree that a mathematical mindset is important for AI alignment. I don't, however, think that a "mathematical mindset" is the same as a "proof mindset". Rather, I think that the latter is closer to being a "programming mindset" -- or, indeed, a "security mindset". And that a "mathematical mindset" is largely missing from AI-alignment discourse at present.

Whereas others see a division between two clusters, of the form,

science/physics

vs.

mathematics/programming/logic

I, by contrast, see a hierarchical progression that looks something like:

science < programming < physics < mathematics < logic <...

where, in this context, these words have meanings along the following lines:

science: things being made of parts; decomposition

programming: things being made of moving parts; constant-velocity motion; causal networks

physics: things being made of moving spatial parts; accelerated motion, rotation, fluidity; substance

mathematics: models being made of parts; transubstantiation; metaphysics; theorization

logic: concepts being made of parts; time reversal; ontology

Of course I'm not using these words standardly here. One reason for this is that, in this discussion, no one is: we're talking about mindsets, not about sociological disciplines or even clusters of particular ideas or "results".

But the really important reason I'm not following standard usage is because I'm not trying to invoke standard concepts; instead, I'm trying to invent "the right" concepts. Consequently, I can't just use standard language, because standard language implies a model of the world different from the one that I want to use.

It is commonly believed that if you want to introduce a new concept that is similar or related (but--of course--nonidentical) to an old concept, you shouldn't use the same word for the new concept and the old, because that would be "confusing". I wish to explicitly disagree with this belief.

This view presupposes that actively shifting between models of the world is not in our repertory of mental operations. But I specifically want it to be!

In fact, I claim that this, and not proof, is what a "mathematical mindset" is really about.

For mathematics is not about proofs; it is about definitions. The essence of great mathematics ...
Decision-theoretic problems and Theories; An (Incomplete) comparative list
by somervta — 2018-07-11T02:59:34.883Z
Link: https://docs.google.com/spreadsheets/d/1zeVyhwjq97ILSnz7oO7qaoDpV6QCv0k5kmCTDjh3McU/edit?usp=sharing

So, this is a work-in-progress, but the idea is to a) get an eventually exhaustive list of decision-theoretic problems, and b) detail the '''answers''' given by each major decision theory.

(Stretch goals would including listing other kinds of properties, and representing disagreement about the ''answers'')

Proposed amendments and additions are welcomed!
Agents That Learn From Human Behavior Can't Learn Human Values That Humans Haven't Learned Yet
by steven0461 — 2018-07-11T02:59:12.278Z

[Epistemic status: ¯\_(ツ)_/¯ ]

Armstrong and Mindermann write about a no free lunch theorem for inverse reinforcement learning (IRL): the same action can reflect many different combinations of values and (irrational) planning algorithms.

I think even assuming humans were fully rational expected utility maximizers, there would be an important underdetermination problem with IRL and with all other approaches that infer human preferences from their actual behavior. This is probably obvious if and only if it's correct, and I don't know if any non-straw people disagree, but I'll expand on it anyway.

Consider two rational expected utility maximizing humans, Alice and Bob.

Alice is, herself, a value learner. She wants to maximize her true utility function, but she doesn't know what it is, so in practice she uses a probability distribution over several possible utility functions to decide how to act.

If Alice received further information (from a moral philosopher, maybe), she'd start maximizing a specific one of those utility functions instead. But we'll assume that her information stays the same while her utility function is being inferred, and she's not doing anything to get more; perhaps she's not in a position to.

Bob, on the other hand, isn't a value learner. He knows what his utility function is: it's a weighted sum of the same several utility functions. The relative weights in this mix happen to be identical to Alice's relative probabilities.

Alice and Bob will act the same. They'll maximize the same linear combination of utility functions, for different reasons. But if you could find out more than Alice knows about her true utility function, then you'd act differently if you wanted to truly help Alice than if you wanted to truly help Bob.

So in some cases, it's not enough to look at how humans behave. Humans are Alice on some points and Bob on some points. Figuring out details will require explicitly addressing human moral uncertainty.
On the Role of Counterfactuals in Learning
by Max Kanwal — 2018-07-11T02:45:35.613Z

The following is a hypothesis regarding the purpose of counterfactual reasoning (particularly in humans). It builds on Judea Pearl's three-rung Ladder of Causation (see below).

One important takeaway from this hypothesis is that counterfactuals really only make sense in the context of computationally bounded agents.

(Figure taken from The Book of Why [Chapter 1, p. 6].)

Summary

Counterfactuals provide initializations for use in MCMC sampling.

Preliminary Definitions

Association (model-free):

Pr(Y=y∣X=x)

Intervention/Hypothetical (model-based):

Pr(Y=y∣do(X=x))

Counterfactual (model-based):

Pr(Y=y∣do(X=x),Y=y′)

In the counterfactual, we have already observed an outcome y′ but wish to reason about the probability of observing another outcome y (possibly the same as y′) under do(X=x).

Note: Below, I use the terms "model" and "causal network" interchangeably. Also, an "experience" is an observation of a causal network in action.

Assumptions

Real-world systems are highly complex, often with many causal factors influencing system dynamics.
Humans minds are computationally bounded (in time, memory, and precision).
Humans do not naturally think in terms of continuous probabilities; they think in terms of discrete outcomes and their relative likelihoods.

Relevant Literature:

Lieder, F., Griffiths, T. L., Huys, Q. J., & Goodman, N. D. (2018). The anchoring bias reflects rational use of cognitive resources. Psychonomic bulletin & review, 25(1), 322-349.

Sanborn, A. N., & Chater, N. (2016). Bayesian brains without probabilities. Trends in cognitive sciences, 20(12), 883-893.

Theory

Claim 1.

From a notational perspective, in going from a hypothetical to a counterfactual, the generalization lies solely in the ability to reason about a concrete scenario starting from an alternative scenario (the counterfactual). In theory, given infinite computational resources, the do-operator can, on its own, reason forward about anything by considering only hypotheticals. Thus, a counterfactual would be an inadmissible object under such circumstances. (Perfect knowledge of the system is not required if one can specify a prior. All that is required is sufficient computational resources.)

Corollary 1.1.

Counterfactuals are only useful when operating with limited computational resources, where "limited" is defined relative to the agent doing the reas... ,,,,,,,,
Clarifying Consequentialists in the Solomonoff Prior
by Vlad Mikulik — 2018-07-11T02:35:56.533Z

I have spent a long time being confused about Paul’s post on consequentialists in the Solomonoff prior. I now think I understand the problem clearly enough to engage with it properly.

I think the reason I was confused is to a large degree a problem of framing. It seemed to me in the course of discussions I had to deconfuse myself to me that similar confusions are shared by other people. In this post, I will attempt to explain the framing that helped clarify the problem for me.

i. A brief sketch of the Solomonoff prior

The Solomonoff, or Universal, prior is a probability distribution over strings of a certain alphabet (usually over all strings of 1s and 0s). It is defined by taking the set of all Turing machines (TMs) which output strings, assigning to each a weight proportional to

(where L is its description length), and then assigning to each string a probability equal to the weights of the TMs that compute it. The description length is closely related to the amount of information required to specify the machine; I will use description length and amount of information for specification interchangeably.

(The actual formalism is in fact a bit more technically involved. I think this picture is detailed enough, in the sense that my explanation will map onto the real formalism about as well.)

The above defines the Solomonoff prior. To perform Solomonoff induction, one can also define conditional distributions by considering only those TMs that generate strings beginning with a certain prefix. In this post, we’re not interested in that process, but only in the prior.

ii. The Malign Prior Argument

In the post, Paul claims that the prior is dominated by consequentialists. I don’t think it is quite dominated by them, but I think the effect in question is plausibly real.

I’ll call the key claim involved the Malign Prior Argument. On my preferred framing, it goes something like this:

Premiss: For some strings, it is easier to specify a Turing Machine that simulates a reasoner which decides to predict that string, than it is to specify the intended generator for that string.

Conclusion: Therefore, those strings’ Solomonoff prior probability will be dominated by the weight assigned to the TM containing the reasoner.

It’s best to explain the idea of an ‘intended generator’ with examples. In the case of a camera signal as the string, the intended generator is something like a TM tha...
Complete Class: Consequentialist Foundations
by abramdemski — 2018-07-11T01:57:14.054Z

The fundamentals of Bayesian thinking have been justified in many ways over the years. Most people here have heard of the VNM axioms and Dutch Book arguments. Far fewer, I think, have heard of the Complete Class Theorems (CCT).

Here, I explain why I think of CCT as a more purely consequentialist foundation for decision theory. I also show how complete-class style arguments play a role is social choice theory, justifying utilitarianism and a version of futarchy. This means CCT acts as a bridging analogy between single-agent decisions and collective decisions, therefore shedding some light on how a pile of agent-like pieces can come together and act like one agent. To me, this suggests a potentially rich vein of intellectual ore.

I have some ideas about modifying CCT to be more interesting for MIRI-style decision theory, but I'll only do a little of that here, mostly gesturing at the problems with CCT which could motivate such modifications.

Background

My Motives

This post is a continuation of what I started in Generalizing Foundations of Decision Theory and Generalizing Foundations of Decision Theory II. The core motivation is to understand the justification for existing decision theory very well, see which assumptions are weakest, and see what happens when we remove them.

There is also a secondary motivation in human (ir)rationality: to the extent foundational arguments are real reasons why rational behavior is better than irrational behavior, one might expect these arguments to be helpful in teaching or training rationality. This is related to my criterion of consequentialism: the argument in favor of Bayesian decision theory should directly point to why it matters.

With respect to this second quest, CCT is interesting because Dutch Book and money-pump arguments point out irrationality in agents by exploiting the irrational agent. CCT is more amenable to a model in which you point out irrationality by helping the irrational agent. I am working on a more thorough expansion of that view with some co-authors.

Other Foundations

(Skip this section if you just want to know about CCT, and not why I claim it is better than alternatives.)

I give an overview of many proposed foundational arguments for Bayesianism in the first post in this series. I called out Dutch Book and money-pump arguments as the most promising, in terms of motivating decision theory only from "winning". ...
Conditions under which misaligned subagents can (not) arise in classifiers
by anon1 — 2018-07-11T01:52:58.152Z

Core claim: Misaligned subagents are very unlikely to arise in a classification algorithm unless that algorithm is directly or indirectly (e.g. in a subtask) modeling interactions through time at a significant level of complexity.

Definition 1: Agent - a function from inputs and internal state (or memory) to an output / action and new internal state. Note that this includes things that would not usually be considered as "agents" - e.g. plants or bacteria. Also note that not all "agents" of this type have consistent (or even coherent) "goals"

This definition of agent might be considered too broad; the reason I have decided to use it is that I believe it covers basically everything that could be dangerous - if an AI is not an agent under this definition, then I think it is extremely likely that this AI would be safe.

Definition 2: A function that was selected by an optimization procedure has a misaligned subagent if it spawns a subprocess that is an agent whose "goals" are different from (and potentially in conflict with) the optimization criteria.

Example: Consider an optimization process that selects for functions that accurately predict human actions, and assume that this optimization process finds a function that does this prediction by creating an extremely accurate simulations of humans. These simulations would be misaligned subagents, since humans are agents and the goals of the simulations would likely be very different from "predict human actions accurately".

For brevity, let us abbreviate classifiers with misaligned subagents as CWMS. Note that I might use classifier a bit more broadly than the strict definition - for example, I may call certain more general question-answering machines "classifiers". I do not believe this directly affects the general argument.

Claim 1: An agent will "perform" "better" than a memoryless function given the same sequence of inputs only if (almost) every input is highly correlated with the previous input. To phrase this in a different way, having a "memory" only helps if your "memory" gives good evidence for either what is going to happen next or what to do next.

Claim / Assumption 2: For most optimization processes powerful enough to find a classifier that performs well, we have:

P(Optimization process finds a CWMS)≈Density of CW...
No, I won't go there, it feels like you're trying to Pascal-mug me
by Rupert — 2018-07-11T01:37:12.808Z

The goal of this blog-post is to explore the intuition that drives the feeling that the agent who arrives at the conclusion that they should pay Pascal's mugger is being unreasonable, and whether this can be tied in with the analysis of a logical inductor as an algorithm not exploitable by an efficiently computable trader. An analysis of this intuition may help us to design the decision theory of a future Friendly AI so that it is not vulnerable to being Pascal-mugged, which is plausibly a desirable goal from the point of view of obtaining behaviour in alignment with our values.

So, then, consider the following conversation that I recently had with another person at a MIRI workshop.

"Do you know Amanda Askell's work? She takes Pascal's wager seriously. If there really are outcomes in the outcome space with infinite utility and non-infinitesimal probability, then that deserves your attention."

"Oh, but that's very Goodhartable."

A thought in close proximity to this would be that that line of reasoning would open you up to exploitation by agents not aligned with your core values. This is plausibly the main driver behind the intuition that paying Pascal's mugger is unreasonable, a decision theory like that makes you too exploitable.

The work done by MIRI on the analysis of the notion of logical induction identifies the key desirable property of a logical inductor algorithm as producing a pattern of belief states evolving over time which isn't exploitable by any efficiently computable trader.

Traders studying the history of your belief states won't be able to identify any pattern that they can exploit with a polynomial-time computable algorithm, so when such patterns emerge in the sentences you have already proved, you'll make the necessary "market correction" in your own belief states so that such patterns don't make you exploitable. Thus, being a good logical inductor is related to having patterns of behaviour that are not exploitable. We see here some hint of a connection with the notion that being Pascal's-mugging-resilient is related to not being exploitable.

Plausibly a useful research project would be to explore whether some similar formalisable notion in an AI's decision theory could capture the essence of the thought behind "Oh, Pascal's-mugging reasoning looks too...