trentmkelly/LessWrong-43k huggingface dataset:
Activation Energies and Uncertainty
by  — 2017-11-01T10:23:04.894Z

From "An Equilibrium of No Free Energy"

We can see the notion of an inexploitable market as generalizing the notion of an efficient market as follows: in both cases, there’s no free energy inside the system. In both markets, there’s a horde of hungry organisms moving around trying to eat up all the free energy. In the efficient market, every predictable price change corresponds to free energy (easy money) and so the equilibrium where hungry organisms have eaten all the free energy corresponds to an equilibrium of no predictable price changes.

This idea resonates with me and I have been thinking about a related concept for a while now. If you should't expect to get easy money, what then is hard money and can you get that?

Obligatory: Not an economist, lapsed chemist.

Chemistry has a handy concept called the activation energy, which I shall lean on as an intuition pump. This is the amount of energy you need to start a reaction going. One salient example is a spark landing on some gunpowder, you need the small amount of energy from the spark to start the reaction, but once it is started it is self-sufficient. The energy from the initial decomposition of the fragment of gunpowder in to gases is enough to kick the other fragments over the activation energies and decompose them etc.

There are handy energy diagrams that chemists use. The below represents an energy releasing (or Exothermic) reaction:

It is exothermic because the net energy of the end constituents is less than energy of the starting constituents.

The net energy released is the Freed Energy- Activation Energy. This is your payoff, your hard money.

So hard money, in this analogy, is money trapped behind a large activation energy. Things like spinning up new rocket companies fall under this.

For actually real world economies it is worse, because we have large amounts of uncertainty in the system. When trying to do something hard and new you do not know the activation energy or the freed energy. The more these levels are known or can be reasoned about, the more you should expect investors or other economic actors to have put up the money to get over the activation energy. You might be able to exploit the fact knowledge is not spread equally through economic actors, normal rich people do not have the background knowledge to be able to evaluate rocket engineering ideas.

So if you think you see lots ...
Anthropic reasoning isn't magic
by Stuart_Armstrong — 2017-11-01T08:57:44.227Z

The user Optimization Process presented a very interesting collection of five anthropic situations, leading to seemingly contradictory conclusions where we can't conclude anything about anything.

It's an old post (which I just discovered), but it's worth addressing, because it's wrong - but very convincingly wrong (it had me fooled for a bit), and clearing up that error should make the situation a lot more understandable. And you don't need to talk about SSA, SIA, or other advanced anthropic issues.

The first example is arguably legit; it's true that:

And so, a universe created by some kind of deity, and tuned for life, is indistinguishable from a universe that happens to have the right parameters by coincidence. "We exist" can't be evidence for our living in one or the other, because that fact doesn't correlate with design-or-lack-of-design

But what's really making the argument work is the claim that:

unless you think that, from inside a single universe, you can derive sensible priors for the frequency with which all universes, both designed and undesigned, can support life?

But the main argument fails at the very next example, where we can start assigning reasonable priors. It compares worlds where the cold war was incredibly dangerous, with worlds where it was relatively safer. Call these "dangerous", and "safe". The main outcome is "survival", ie human survival. The characters are currently talking about surviving the cold war - designate this by "talking". Then one of the character says:

Avery: Not so! Anthropic principle, remember? If the world had ended, we wouldn't be standing here to talk about it.

This encodes the true statement that P(survival | talking) is approximately 1, as are P(survival | talking, safe) and P(survival | talking, dangerous). In these conditional probabilities, the fact that they are talking has screened off any effect of the cold war on survival.

But Bayes law still applies, and

P(dangerous | survival) = P(dangerous) * (P(survival | dangerous)/P(survival)).

Since P(survival | dangerous) < P(survival) (by definition, dangerous cold wars are those where the chance of surviving are lower than usual), we get that

P(dangerous | survival) < P(dangerous).

Thus the fact of our survival has indeed caused us to believe that the cold war was saf...
Doxa, Episteme, and Gnosis
by Gordon Seidoh Worley — 2017-10-31T23:23:38.094Z
Link: https://mapandterritory.org/doxa-episteme-and-gnosis-ea35e4408edd

Ancient Greek famously made a distinction between 3 kinds of knowledge: doxa, episteme, and gnosis.

Doxa is basically what in English we might call hearsay. It’s the stuff you know because someone told you about it. If you know the Earth is round because you read it in a book, that’s doxa.

Episteme is what we most often mean by “knowledge” in English. It’s the stuff you know because you thought about it and reasoned it out. If you know the Earth is round because you measured shadows at different locations and did the math to prove that the only logical conclusion is that the Earth is round, that’s episteme.

Gnosis has no good equivalent in English, but the closest we come is when people talk about personal experience because gnosis is the stuff you know because you experienced it. If you know the Earth is round because you traveled all the way around it or observed it from space, that’s gnosis.

Often we elide these distinctions. Doxa of episteme is frequently thought of as episteme because if you read enough about how others gained episteme you may feel as though you have episteme yourself. This would be like hearing lots of people tell you how they worked out that the Earth is round and thinking that this gives you episteme rather than doxa. The mistake is understandable: as long as you only hear others talk about their episteme it’s easy to pattern match and think you have it too, but as soon as you try to explain your supposed episteme to someone else you will quickly discover if you only have doxa instead. The effect is so strong that experts in fields often express that they never really knew their subject until they had to teach it.

In the same way episteme is often mistaken for gnosis. At least since the time of Ptolemy people have had episteme of the spherical nature of the Earth, and since the 1970s most people have seen pictures showing that the Earth is round, but astronauts continue to experience gnosis of Earth’s roundness the first time they fly in space. It seems no matter how much epistemic reckoning we do or how accurate and precise our epistemic predictions are, we are still sometimes surprised to experience what we previously only believed.

But none of this is to say that gnosis is better than episteme or that episteme is better than doxa because each has value in different ways. Dox...
An Equilibrium of No Free Energy
by Eliezer Yudkowsky — 2017-10-31T21:27:00.232Z

Follow-up to: Inadequacy and Modesty

I am now going to introduce some concepts that lack established names in the economics literature—though I don’t believe that any of the basic ideas are new to economics.

First, I want to distinguish between the standard economic concept of efficiency (as in efficient pricing) and the related but distinct concepts of inexploitability and adequacy, which are what usually matter in real life.

i.

Depending on the strength of your filter bubble, you may have met people who become angry when they hear the phrase “efficient markets,” taking the expression to mean that hedge fund managers are particularly wise, or that markets are particularly just.1

Part of where this interpretation appears to be coming from is a misconception that market prices reflect a judgment on anyone’s part about what price would be “best”—fairest, say, or kindest.

In a pre-market economy, when you offer somebody fifty carrots for a roasted antelope leg, your offer says something about how impressed you are with their work hunting down the antelope and how much reward you think that deserves from you. If they’ve dealt generously with you in the past, perhaps you ought to offer them more. This is the only instinctive notion people start with for what a price could mean: a personal interaction between Alice and Bob reflecting past friendships and a balance of social judgments.

In contrast, the economic notion of a market price is that for every loaf of bread bought, there is a loaf of bread sold; and therefore actual demand and actual supply are always equal. The market price is the input that makes the decreasing curve for demand as a function of price meet the increasing curve for supply as a function of price. This price is an “is” statement rather than an “ought” statement, an observation and not a wish.

In particular, an efficient market, from an economist’s perspective, is just one whose average price movement can’t be predicted by you.

If that way of putting it sounds odd, consider an analogy. Suppose you asked a well-designed superintelligent AI system to estimate how many hydrogen atoms are in the Sun. You don’t expect the superintelligence to produce an answer that is exactly right down to the last atom, because this would require measuring the mass of the Sun more finely than any measuring instrument you expect it to possess. At t...
Normative assumptions: regret
by Stuart_Armstrong — 2017-10-31T13:50:27.444Z

Crossposted at the Intelligent Agents Forum.

In a previous post, I presented a model of human rationality and reward as pair (p, R), with p our (ir)rational planning algorithm (called a planner), and R our reward, with p(R) giving our actions/policy. I also showed that human behaviour is insufficient to establish R (or p) in any meaningful way whatsoever.

Yet humans seem to make judgements about each other's rationality and true rewards all the time. And not only do these judgements often agree with each other (almost everyone agrees that the anchoring bias is a bias - or a out-of-context heuristic, ie a bias - and almost nobody argues that it's actually a human value), but they often seem to have predictive ability. What's going on?

Adding normative assumptions

To tackle this, we need to add normative assumptions. Normative assumptions are simply assumptions that distinguish between two pairs (p, R) (p', R') that lead to the same policy: p(R)=p'(R').

Since those pairs predict the same policy, they cannot be distinguished via observations. So a normative assumption is an extra piece, that cannot itself be deduced from observation, and distinguishes between planner-reward pairs.

Because normative assumptions cannot be deduced from observations, they are part of the definition of the human reward function. They are not abstract means of converging on this true reward; they are part of the procedure that defines this reward.

What are they?

There are two way of seeing such normative assumptions. The first is as an extra update rule: upon seeing observation o, the probabilities of (p, R) and (p', R') would normally go to α and α', but, with the normative assumption, there is an extra update to the relative probabilities of the two.

Equivalently, a normative assumption can be seen as an adjustment to the priors of these pairs. The two approaches are equivalent, but sometimes an extra update is computationally tractable whereas the equivalent prior would be intractable (and vice-versa).

Example: regret

So, if normative assumptions are definitions of rationality/reward, what are good candidates to pick? How do we, as humans, define our own rationality and reward?

Well, one strong way seems to be through our feeling of regret (human regret, not regret in the machine learning sense). We often feel and express regret when something...
New(ish) AI control ideas
by Stuart_Armstrong — 2017-10-31T12:52:10.000Z

The list of posts is getting unwieldy, so I'll post the up-to-date stuff at the beginning:

Humans inconsistencies:

Bias in rationality is much worse than noise

Learning values, or defining them?

Rationality and overriding rewards: combined model

Humans can be assigned any values whatsoever...

Resolving human inconsistency in a simple model

Short term vs long term preferences

Reward function learning:

Reward function learning

Biased reward learning

Counterfactuals on POMDPs

Uninfluenceable agents

Counterfactually uninfluenceable agents

An algorithm with preferences: from zero to one variable

New circumstances, new values?

Understanding humans:

Humans as truth channel

True understanding comes from passing exams

Understanding the important facts

Framework:

Three human problems and one AI issue

Acausal trade:

Introduction

Double decrease

Pre-existence deals

Full decision algorithms

Breaking acausal trade

Trade in different types of utility functions

Being unusual

Conclusion

Oracle designs:

Three Oracle designs

Extracting human values:

Divergent preferences and meta-preferences

Engineered fanatics vs yes-men

Learning doesn't solve philosophy of ethics

Models of human irrationality

Heroin model: AI "manipulates" "unmanipulatable" reward

Stratified learning and action

(C)IRL is not solely a learning process

Learning (meta-)preferences

What does an imperfect agent want?

Window problem for manipulating human values

Abstract model of human bias

Paul Christiano's post

Corrigibility:

Corrigibility thoughts I: caring about multiple things

Corrigibility thoughts II: the robot operator

Corrigibility thoughts III: manipulating versus deceiving

Corrigibility through stratification

Cake or Death toy model for corrigibility

Learning values versus indifference

Conservation of expected ethics/evil (isn't enough)

Guarded learning

Indifference:

Translation "counterfactual"

Indifference and compensatory rewards

All the indifference designs

The "best" value indifference method

Double indifference

Corrigibility for AIXI

Indifference utility functions

AIs in virtual worlds:

AIs in virtual worlds

The alternate hypothesis for AIs in virtual worlds

AIs in virtual worlds: discounted mixed utility/reward

Simpler, cruder, virtual world AIs

True answers from AI:

True answers from AIs

What to do with very low probabilities

Summary of true answers from AIs

AI's printing the

...
Bias in rationality is much worse than noise
by Stuart_Armstrong — 2017-10-31T11:57:15.995Z

Crossposted at Intelligent Agent Forum.

I've found quite a few people dubious of my "radical skepticism" post on human preferences. Most "radical skepticism" arguments - Descartes's Demon, various no-free-lunch theorems, Boltzmann Brains - generally turn out to be irrelevant, in practice, one way or another.

But the human preferences are in a different category. For a start, it's clear that getting them correct is important for AI alignement - we can't just ignore errors. But most importantly, simplicity priors/Kolmogorov complexity/Occam's razor don't help with learning human preferences, as illustrated most compactly with the the substitution of (-p,-R) for (p,R).

But this still feels like a bit of a trick. Maybe we can just assume rationality plus a bit of noise, or rationality most of the time, and get something a lot more reasonable.

Structured noise has to be explained

And, indeed, if humans were rational plus a bit of noise, things would be simple. A noisy signal has high Kolmogorov complexity, but there are ways to treat noise as being of low complexity.

The problem with that approach is that explaining noise is completely different from explaining the highly structured noise that we know as bias.

Take the anchoring bias, for example. In one of the classical experiments, American students were asked if they would buy a product for the price that was the last two digits of their social security number, and were then asked what price they would buy the product for. The irrelevant last two digits had a strong distorting effect on their willingness to pay.

Modelling anchoring behaviour

Let's model a simplified human, if not asked about their social security number, as valuing a cordless keyboard at $25, plus noise η drawn from a normal distribution of mean $0 and standard deviation $5.

If they are first asked about their social secuity number s, their valuation shifts to 3/4($25) + 1/4(s) + η.

To explain human values, we have two theories about their rewards:

R(0)=$25.
R(1)=3/4($25) + 1/4(s).

We have three theories about their planning rationality:

p(0): the human is rational.
p(1): p(0) + noise η.
p(2): the human is rational, but has an anchoring bias of 1/4 of the value of the number they hear.
p(3): p(2) + noise η.

If we count noise as being of low complexity, then R(0), p(0), and p(1) are low complexity rewards/pla...
Learning values, or defining them?
by Stuart_Armstrong — 2017-10-31T11:10:06.051Z

Crossposted at Intelligent Agents Forum.

Is there a teapot currently somewhere between the orbit of Earth and Mars? And who won the 1986 World Cup football match between England and Argentina? And what do these questions have to do with learning human values?

Underlying true reality?

Both those top questions are uncertain. We haven't scanned the solar system with enough precision to find any such teapot. And though the referee allowed the first Argentine goal, making it official, and though FIFA had Agentina progress to the semi-finales (they would eventually win the tournament) while England was eliminate... that goal, the "Hand of God" goal, was scored by Maradona with his hand, a totally illegal move.

In a sense, neither question can ever be fully resolved. Even if we fully sweep the the solar system for teapots in a century's time, it's still possible there might have been one now, and it then crashed into the sun, stopping us from ever finding it. And in the spirit of ambijectivity, the question of Argentina's victory (or whether it was a "proper" victory, or a "fair" victory) depends on which aspect of the definition of victory you choose to emphasise - the referee's call and the official verdict, versus the clear violation of the rules.

Nevertheless, there is a sense in which we feel the first question has a definite answer (which is almost certainly "no"), while the second is purely about definitions.

Answering the question

Why do we feel that the teapot question has a definite answer? Well, we have a model of reality as something that actually objectively exists, and our investigation of the world backs it up - when confronted by a door, we feel that there is something behind the door, even if we choose not to open it. There are various counterfactuals in which we could have sent out a probe to any given area of the orbit, so we feel we could have resolved the "is there a teapot at this location" for any given location within a wide radius of the Earth.

Basically, the universe has features that causes us to believe that when we observe it (quantum effects aside), we are seeing a previous reality rather than creating a new one (see the old debate between platonists and formalists in mathematics).

Whereas even if we had a God or an AI, we don't expect it to have a definite answer t...
Cutting edge technology
by Elo — 2017-10-31T06:00:42.665Z

Original post: http://bearlamp.com.au/cutting-edge-technology/

When the microscope was invented, in a very short period of time we discovered the cell and the concept of microbiology. That one invention allowed us to open up entire fields of biology and medicine. Suddenly we could see the microbes! We could see the activity that had been going on under our noses for so long.

when we started to improve our ability to refined pure materials we could finally make furnace bricks with specific composition. Specific compositions could then be used to make bricks that were able to reach higher temperatures without breaking. Higher temperatures meant better refining of materials. Better refining meant higher quality bricks, and so on until we now have some very pure technological processes around making materials. But it's something we didn't have before the prior technology on the skill tree.

Before we had refrigeration and food packaging, it was difficult to get your fresh food to survive to your home. Now with production lines it's very simple. For all his decadence Caesar probably would have had trouble ordering a cheeseburger for $2 and having it ready in under 5 minutes. We've come a long way since Caesar. We've built a lot of things that help us stand on the shoulders of those who came before us.

Technology enables further progress. That seems obvious. But did that seem obvious before looking down the microscope? Could we have predicted what bricks we could have made with purely refined materials? Could Caesar have envisioned every citizen in his kingdom watching TV for relatively little cost to those people? It would have been hard to forsee these things back then.

With the idea that technology is enabling future growth in mind - I bring the question, "What technology is currently under-utilised?" Would you be able to spot it when it happens? Touch screen revolutionised phone technology. Bitcoin - we are still watching but it's here to stay.

"What technology is currently under-utilised?"

For example "AI has the power to change everything. (it's almost too big to talk about)". But that's a big thing. It's like saying "the internet has the power to change everything" great but could you have predicted google, facebook and uber from a couple of connected compu...
Cutting edge technology
by Elo — 2017-10-31T06:00:30.068Z

Original post: http://bearlamp.com.au/cutting-edge-technology/

When the microscope was invented, in a very short period of time we discovered the cell and the concept of microbiology. That one invention allowed us to open up entire fields of biology and medicine. Suddenly we could see the microbes! We could see the activity that had been going on under our noses for so long.

when we started to improve our ability to refined pure materials we could finally make furnace bricks with specific composition. Specific compositions could then be used to make bricks that were able to reach higher temperatures without breaking. Higher temperatures meant better refining of materials. Better refining meant higher quality bricks, and so on until we now have some very pure technological processes around making materials. But it's something we didn't have before the prior technology on the skill tree.

Before we had refrigeration and food packaging, it was difficult to get your fresh food to survive to your home. Now with production lines it's very simple. For all his decadence Caesar probably would have had trouble ordering a cheeseburger for $2 and having it ready in under 5 minutes. We've come a long way since Caesar. We've built a lot of things that help us stand on the shoulders of those who came before us.

Technology enables further progress. That seems obvious. But did that seem obvious before looking down the microscope? Could we have predicted what bricks we could have made with purely refined materials? Could Caesar have envisioned every citizen in his kingdom watching TV for relatively little cost to those people? It would have been hard to forsee these things back then.

With the idea that technology is enabling future growth in mind - I bring the question, "What technology is currently under-utilised?" Would you be able to spot it when it happens? Touch screen revolutionised phone technology. Bitcoin - we are still watching but it's here to stay.

"What technology is currently under-utilised?"

For example "AI has the power to change everything. (it's almost too big to talk about)". But that's a big thing. It's like saying "the internet has the power to change everything" great but could you have predicted google, facebook ...