trentmkelly/LessWrong-43k huggingface dataset:
Coherent decisions imply consistent utilities
by Eliezer Yudkowsky — 2019-05-12T21:33:57.982Z

(Written for Arbital in 2017.)

Introduction to the introduction: Why expected utility?

So we're talking about how to make good decisions, or the idea of 'bounded rationality', or what sufficiently advanced Artificial Intelligences might be like; and somebody starts dragging up the concepts of 'expected utility' or 'utility functions'.

And before we even ask what those are, we might first ask, Why?

There's a mathematical formalism, 'expected utility', that some people invented to talk about making decisions. This formalism is very academically popular, and appears in all the textbooks.

But so what? Why is that necessarily the best way of making decisions under every kind of circumstance? Why would an Artificial Intelligence care what's academically popular? Maybe there's some better way of thinking about rational agency? Heck, why is this formalism popular in the first place?

We can ask the same kinds of questions about probability theory:

Okay, we have this mathematical formalism in which the chance that X happens, aka P(X), plus the chance that X doesn't happen, aka P(¬X), must be represented in a way that makes the two quantities sum to unity: P(X)+P(¬X)=1.

That formalism for probability has some neat mathematical properties. But so what? Why should the best way of reasoning about a messy, uncertain world have neat properties? Why shouldn't an agent reason about 'how likely is that' using something completely unlike probabilities? How do you know a sufficiently advanced Artificial Intelligence would reason in probabilities? You haven't seen an AI, so what do you think you know and how do you think you know it?

That entirely reasonable question is what this introduction tries to answer. There are, indeed, excellent reasons beyond academic habit and mathematical convenience for why we would by default invoke 'expected utility' and 'probability theory' to think about good human decisions, talk about rational agency, or reason about sufficiently advanced AIs.

The broad form of the answer seems easier to show than to tell, so we'll just plunge straight in.

Why not circular preferences?

De gustibus non est disputandum, goes the proverb; matters of taste cannot be disputed. If I like onions on my pizza and you like pineapple, it's not that one of us is right and one of us is wrong. We just prefer different pizza toppings.

Well, but suppose I declare to you that I simultaneo...
The Relationship Between the Village and the Mission
by Raemon — 2019-05-12T21:09:31.513Z

Epistemic Status: Braindump, not as well thought out as I’d like.

Previously:

Project Hufflepuff
The Archipelago Model of Community Standards
The Craft is not the Community
What is Rationalist Berkeley's Community Culture?
Dragon Army Retrospective

This is a post about dynamics in the Berkeley rationality community, although it may be relevant to broader domains.

It is highly opinionated about what I think is important.

I tried to optimize this for a clear-cut goal, then realized the clear-cut goal was “I want to make it easier for people to cooperate with me on community-building, and I just want to do a massive braindump to get them up to speed on where I’m coming from, so that when I have a conversation about it we can skip to the harder parts."

If you are serious about rationalist community-building, read this, and then come talk to me afterwards.

When I visited the Bay in 2015, a friend (who used to live in NYC) remarked “you know, when I was in New York, I felt like once a week I went to ‘rationality club’. In Berkeley it feels more like I live in a small rationality village — there’s a couple hundred people, I’m friends with some of them. We bump into each other in the street on the way to the grocery store.”

Eventually I moved here, and yup. That is how it is. Sorta. With important caveats and problems.

There are lots of little subcultures in the Rationalist Bay, some overlapping. But I think there are two primary reasons people come:

to have a Village – a home, among like-minded people
to contribute to the Mission – ensuring the flourishing of human values (or something like them)

In the past 10 years, the Mission has acquired serious infrastructure. There’s been much less intentional effort to build a home. Mostly for good reason – the Mission is important, and hard. Competent People are Rare and the World is Big. Building a village is also hard, and if you’re able to do so, you’re probably also able to work on bigger picture Mission stuff.

The Mission provides juuuust enough value as a “home” to satisfice the people involved (which might not actually be sufficient for them, just decent enough that it's not their primary bottleneck).

In the past couple years, we’ve begun to see more serious efforts towards building Village infrastructure. But I think these efforts are often missing important aspects of the big picture.

This post is a high-level overvi...
Tales from the Highway
by Zvi — 2019-05-12T19:40:00.862Z

Epistemic Status: Concrete data on how something works with curiosity as to the gears behind the decisions, and a desire to record exactly what happened for posterity so we have detailed accurate records and perhaps an example of some things. But no reason to think any of it is important, or you should feel any great need to read it.

“You don’t know what you have until you go to New Jersey” – My wife Laura, when this was almost over.

This happened on Saturday night, as I attempted to transport myself, my wife and our two children home from their in-laws.

The one-year old is being held up by a strange contraption attached to my chest. After enduring a forty-five minute car ride, we make our way to Willowbrook Mall to catch the NJ Transit bus back into New York City’s Port Authority, after which we’ll need to get two trains or another taxi to return home.

When we get to the bus stop, it is 7:02, and we are just in time to see a bus that we presume is the 7:00 bus close its doors and begin to leave. Assuming we will now be forced to wait the actual maximum amount of time to get the next bus, we decide to order a taxi. Just as we’ve sorted out how to do that, another bus that says it is going to New York pulls into the station. Thanks to the orderly and well-kept bus schedule, it looks like we’ve caught a break and we’ll be at Port Authority around 7:35. We board and are on our way home.

Traffic is lousy, but nothing unusually bad. I’m mostly happily zoned out, thanks to a combination of continuously rebalancing my child as he falls asleep slooping over in various directions only to have the drop jolt him awake, and listening to a podcast. But I still keep an eye up ahead, because I’ve unfortunately been trained over the years that when traffic is lousy, one should be sure to know exactly how lousy it is, so one can properly complain and/or maintain and manage expectations.

I see a car cutting in front of us, and it’s instantly obvious this was not a wise decision. There isn’t enough room. I hear a noise and there’s a slight rattle. There’s been a minor accident. Shrug. It happens. The other car does not seem to care or even notice the havoc it has caused on the road this night, and continues onward as other cars allow.

The bus driver seems not to be overly concerned. With no room to move anyway, she takes a quick walk outside to look at the damage, but as soon as thin...
The Geography of the Room — impact of position, orientation, and visibility on how often extraverts start conversations
by less_penguiny — 2019-05-12T17:21:36.778Z
Link: https://www.lesspenguiny.com/articles/the-geography-of-the-room

I write long-form articles about social interactions based on ~6 years of observational data. This particular piece is part of a longer sequence exploring why some people are frequently approached by strangers whereas others are mostly ignored.
Type-safeness in Shell
by Martin Sustrik — 2019-05-12T11:30:00.680Z

Since writing the post on a hypothetical hull language as an alternative to shell I cannot stop thinking about the shortcomings of shell.

And one think that comes to mind over and over is type-safeness. Shell treats everything as a string and that's the source of both its power and its poor maintainability.

So when I ask whether shell can be improved, the question is actually more subtle: Can it be improved without compromising its versatility? Can we, for example, be more type-safe without having to type Java-like stuff on the command line? Without sacrificing the powerful and dangerous features like string expansion?

I mean, you can write shell-like scripts in Python even today and use type hints to get type safeness. But in real world this practice seems to be restricted to writing more complex programs, programs that require actual in-language processing, complex control flow, use of libraries and so on. Your typical shell script which just chains together a handful of UNIX utilities — no, I don't see that happening a lot.

To put it in other words, different "scripting languages" managed to carve their own problem spaces from what once used to be the domain of shell, but almost none of them attacked its very core use case, the place where it acts as a dumb glue between stand-alone applications.

But when writing shell scripts, I observe that I do have a type system in mind. When I type "ls" I know that an argument of type "path" should follow. Sometimes I am even explicit about it. When I save JSON into a file, I name it "foo.json". But none of that is formalized in the language.

And in some way, albeit in a very hacky one, shell is to some extent aware of the types. When I type "ls" and press Tab twice a list of files appears on the screen. When I type "git checkout" pressing Tab twice results in a list of git branches. So, in a way, shell "knows" what kind of argument is expected.

And the question that's bugging me is whether the same can be done in a more systemic way.

Maybe it's possible to have a shell-like language with actual type system. Maybe it could know that file with .json extension is supposed to contain JSON. Or it could know that "jq" expects JSON as an input. Maybe it could know that JSON is a kind of text file and that any program accepting a text file ...
"UDT2" and "against UD+ASSA"
by Wei Dai — 2019-05-12T04:18:37.158Z

I'm reposting some old posts that I originally sent to the "decision theory workshop" mailing list and the "everything-list", because I occasionally want to reference these posts but the former mailing list is private and the latter one is public but I can't figure out how to create direct links to posts that are viewable without becoming a member.

UDT2 is a decision theory idea that I came up with to try to solve some problems in UDT1.1 however I'm not very happy with it currently. UD+ASSA or UDASSA is an anthropic reasoning idea that I came up with and then moved away from prior to UDT. See also this post for further discussion of UDASSA.

UDT2 (originally "toward a solution of the 'unintentional simulation' problem", 1/25/2011)

(I think this approach potentially solves several problems besides "unintentional simulation" but I'll start there since it provides the clearest motivation.)

I first described this problem (without naming it) at http://lesswrong.com/lw/15z/ingredients_of_timeless_decision_theory/120y.
Here's a condensed version:

Two UDT1 (or UDT1.1) agents play one-shot PD. It's common knowledge that agent A must make a decision in 10^100 ticks (computation steps), whereas agent B has 3^^^3 ticks. While B is trying to derive the logical consequences of returning 'C' or 'D' on the world program P, it is likely to come up with a proof by simulation of A's output, after which it will decide to play D.

I think what A should have done is (if it were running a smarter decision theory), instead of deciding directly on C or D, modify itself into a program K = "simulate the original agents A and B and output 'C' if and only if both of the simulated agents self-modify into K within some time limit".
And B (if it were also running a smarter decision theory) would also self-modify into K, whether or not it happens to simulate A's decision to self-modify into K prior to its own self-modification, and do this before the time limit built into K expires.

So that's my starting intuition, and I want to try to answer: what is this smarter decision theory? It seems that at least two changes need to be made to UDT1:

An agent must take the space of possible decisions to be the set of possible programs it can self-modify into, instead of the set of outputs or input/output maps. (This change is needed anyway if we want the agent to be able to self-improve in general.)

An agen

...
Narcissism vs. social signalling
by Chris_Leong — 2019-05-12T03:26:31.552Z

The main thrust of Robin Hanson's work is that so much of human behaviour is the result of social signalling; that is, the attempt to convince others that we possess good qualities. On the other hand, The Last Psychiatrist presents narcissism as an alternative theory; that is, much of our behaviour is an attempt to convince ourselves that we possess good qualities. I haven't read enough of The Last Psychiatrist to be able to provide a good summary of their ideas, so this will just be a short comment to raise awareness that an alternative theory exists and to discuss how these hypotheses relate.

The separation between these two theories isn't as clear as it may first appear. After all, we may attempt to convince other people of our goodness in order to ultimately convince ourselves or we may attempt to convince ourselves of our goodness so that we can more persuasively convince other people. For the later, imagine someone prepping for a job interview teaching when they are aware that their knowledge of the material isn't as strong as they'd like, but believing they might get hired if they seems confident.

Trying disambiguate using revealed preferences may be misleading - someone may spend most of their time trying to impress other people, but that may simply be the strategy they've adopted for convincing themselves that they are worthy and they may drop it as soon as they learn other strategies. Alternatively, trying to use people's ultimate goals to disambiguate this is tricky when, according to both theories, we often don't know why we do what we do. And indeed, when we talk about ultimate goals, are we referring to the ultimate goal that is represented in the brain or are we allowed to reference evolutionary psychology reasons for behaviour? Then of course these the issue that most of the time, both theories will be correct to a certain extent. I'll admit the the practical consequences of adopting one theory over another aren't always immediately clear, but I expect that adopting one model instead of the other would necessarily result in some differences in predictions.
Why books don't work
by  — 2019-05-11T20:40:27.593Z
Link: https://andymatuschak.org/books/

A look at how books are not optimal for conveying information. Analogies to lectures, with an interesting take on cognitive models, i.e. the assumptions you make about how learning happens. Also some interesting citations on average reading time and attention span.

I really like this piece because it ties together lots of thoughts I've previously tried to express, but didn't find the right words for. I think it does a very good job of pointing out how the default mediums are not optimal.

Also interesting to note the author has collaborated with Michael Nielsen (who wrote a fantastic online book on neural networks) on another online book on quantum computing which incorporates spaced repetition, and he has also done prior work for Khan Academy.
Probability interpretations: Examples
by So8res — 2019-05-11T20:32:14.841Z

(Written for Arbital in 2016.)

Betting on one-time events

Consider evaluating, in June of 2016, the question: "What is the probability of Hillary Clinton winning the 2016 US presidential election?"

On the propensity view, Hillary has some fundamental chance of winning the election. To ask about the probability is to ask about this objective chance. If we see a prediction market in which prices move after each new poll — so that it says 60% one day, and 80% a week later — then clearly the prediction market isn't giving us very strong information about this objective chance, since it doesn't seem very likely that Clinton's real chance of winning is swinging so rapidly.

On the frequentist view, we cannot formally or rigorously say anything about the 2016 presidential election, because it only happens once. We can't observe a frequency with which Clinton wins presidential elections. A frequentist might concede that they would cheerfully buy for $1 a ticket that pays $20 if Clinton wins, considering this a favorable bet in an informal sense, while insisting that this sort of reasoning isn't sufficiently rigorous, and therefore isn't suitable for being included in science journals.

On the subjective view, saying that Hillary has an 80% chance of winning the election summarizes our knowledge about the election or our state of uncertainty given what we currently know. It makes sense for the prediction market prices to change in response to new polls, because our current state of knowledge is changing.

A coin with an unknown bias

Suppose we have a coin, weighted so that it lands heads somewhere between 0% and 100% of the time, but we don't know the coin's actual bias.

The coin is then flipped three times where we can see it. It comes up heads twice, and tails once: HHT.

The coin is then flipped again, where nobody can see it yet. An honest and trustworthy experimenter lets you spin a wheel-of-gambling-odds — reducing the worry that the experimenter might know more about the coin than you, and be offering you a deliberately rigged bet — and the wheel lands on (2 : 1). The experimenter asks if you'd enter into a gamble where you win $2 if the unseen coin flip is tails, and pay $1 if the unseen coin flip is heads.

On a propensity view, the coin has some objective probability between 0 and 1 of being heads, but we just don&#...
Programming Languages For AI
by Donald Hobson — 2019-05-11T17:50:22.899Z

Firstly a chess analogy

Suppose that you were part of a team trying to build a chess playing program. Your team has not yet had the fundamental insight of min-max search with approximate evaluations. While you definitely want some people on the team thinking hard about the abstract nature of chess, this is a somewhat serial process, is there anything else that could usefully be done in parallel?

While we wouldn't have the insights to no exactly what a chess engine would look like, we can say some things about the code. The code will almost certainly want some sort of representation of chess pieces or chess boards. So implementing a virtual chessboard would not be a waste of time.

More speculatively, you might overhear the Thinking Hard About Chess department talk about how a good chess position was defined in terms of making the opponents move not good, and invent recursion.

Ideally, when the crucial insights have been had, all the building blocks needed to make it are ready to go. The first chess engine is half a page of chesslang code.

On to AI

So, can we think of any programmatic building blocks that are likely to be useful in building an AI. Yes, arithmatic, if statements, lists and so on.

These features have already been implemented in many programming languages, can we think of any features that might be useful in making AI that aren't easily available in any programming languages?

Much current AI is arithmetical with neural networks and propagation. However, there are already several fairly easy to use libraries for this, and I can't see how to make it substantially easier to program this sort of AI, other than stuff like hyperparameter optimization that lots of people are already working on.

Much theoretical research on AI is symbolic, even Godelian. AIXItl for example, executes every piece of code up to a certain length and runtime. That would suggest that the programming language should make it easy to generate syntactically correct code, and to run it for a bounded time without side effects. This is also what you would want if you were generating code with an evolutionary algorithm.

On the alignment forum, several designs of AI mentioned involve "search for a proof that this piece of code halts". So our programming language for AI should have powerful proof handling facilities.

Suggested Mechanics

Some inspiration for the poten...
alternative history: what if Bayes rule had never been discovered?
by Yoav Ravid — 2019-05-11T07:29:58.979Z

In trying to understand how Bayesian probability is used, I'm curious to know what wouldn't have been possible without it. how important was it in the course of human discovery, and in turn, how it effected history.

I don't demand rigorous answers, feel free to speculate and throw possibilities as you like.

Bonus question: if Bayes didn't discover it, when would it have to be discovered? (full speculation mode)
Rob B's Shortform Feed
by Rob Bensinger — 2019-05-10T23:10:14.483Z

This is a repository for miscellaneous short things I want to post. Other people are welcome to make top-level comments here if they want. (E.g., questions for me you'd rather discuss publicly than via PM; links you think will be interesting to people in this comment section but not to LW as a whole; etc.)
The Athena Rationality Workshop - June 7th-10th at EA Hotel
by Linda Linsefors — 2019-05-10T22:08:03.600Z

For the past few months, Matt Goldenberg has been in charge of teaching applied rationality at the EA hotel. And honestly, we’ve been quite impressed by the quality. Matt has been a great teacher, and some of us felt like we wanted to delve deeper into his material. So let us delve into it together!

During the first weekend of June, the EA Hotel (36 York Street, Blackpool, UK) will host its first rationality workshop. Open to anyone interested.

We will delve into the

Ease process

, which is step by step process for overcoming internal blocks and creating psychological alignment towards your goals.

Matt has been working intensely on self-improvement for the past 15 years, was a professional coach for 5 years and has been running applied rationality group workshops for the past 18 months. Some guest teachers will also be chiming in with their techniques, including Toon Alfrink with some lessons he learned living at a Zen Monastery, and possibly others.

The workshop will be held June 7th-10th, Friday to Monday, starting at 12:00 and ending at 19:00. Travelers from afar can stay the night on Thursday. There will be an optional chill-out day on Tuesday, June 11th, and you’re welcome to stay longer. Payment will be at the end of the workshop for whatever you think it was worth for you (cost price is £40). All proceeds will go to fund the EA hotel.

If you want to join:

Sign up here

.