trentmkelly/LessWrong-43k huggingface dataset:
Computational efficiency reasons not to model VNM-rational preference relations with utility functions
by AlexMennen — 2018-07-25T02:11:35.429Z

The VNM theorem is constructive; given a rational preference relation, it is easy to construct a corresponding utility function. But in may cases, it may be computationally intractable to choose between two lotteries by computing each of their expected utilities and picking the bigger one, even while it may be easy to decide between them in other ways.

Consider a human deciding whether or not to buy a sandwich. There are several factors that go into this decision, like the price of substitutes, the human's future income, and how tasty the sandwich will be. But these factors are still fairly limited; the human does not consider, for instance, the number of third cousins they have, the outcome of a local election in Siberia, or whether there is intelligent life elsewhere in the galaxy. The predictable effects of buying a sandwich are local in nature, and the human's preferences also have an at least somewhat local character, so they need not think too much about too many details of the world in order to make their decision.

Compare this with how difficult it would be for the human to compute the expected utility of the world if they buy the sandwich, and the expected utility of the world if they don't buy the sandwich, each to enough precision that they can tell which is bigger. The way to construct a utility function corresponding to a given rational preference relation is to pick two lotteries as reference points, declare that the preferred lottery has utility 1 and the dispreferred lottery has utility 0, and compute the utility of other lotteries by looking at what the preference relation says about weighted averages of the lottery with the two reference points. But the reference points could differ from the lotteries resulting from buying a sandwich and from not buying a sandwich in many ways; perhaps these reference points differ from the real world in the number of third cousins the human has, the outcome of a local election in Siberia, and the presence of intelligent life elsewhere in this galaxy. In order to compute the expected utility of buying a sandwich and of not buying a sandwich, both to high enough precision that they can tell which is bigger, the human must consider how all of these factors differ from the reference points, and decide how much they care about each of them. Doing this would require heroic feats of computation, intr...
To cut your loses or push forward in court? 
by CommuterQuery — 2018-07-24T18:26:49.752Z

Good day all,

I am on a train going home, and I'm tossing around in my head between these outcomes. I received an email from the court today

Short version: contractor broke roof, offers 1/4 of the cost to fix it. Has 3 kids and basic income and I'm tired of chasing this. Do I keep going in court or let it be at 1/4 minus admin fee which I paid.
ISO: Name of Problem
by johnswentworth — 2018-07-24T17:15:06.676Z

I'm looking for a name for a problem. I expect it already has one, but I don't know what it is.

The problem: suppose we have an AI trying to learn what people want - e.g. an IRL variant. Intuitively speaking, we point at a bunch of humans and say “figure out what they want, then do that”. A few possible ways the AI could respond:

“Hmm, to the extent that those things have utility functions, it looks like they want friendship, challenge, status, etc…”
“Hmm, it looks like they want to maximize the number of copies of the information-carrying molecules in their cells.”
“Hmm, it looks like they’re trying to maximize entropy in the universe.”
“Hmm, it looks like they’re trying to minimize physical action.”

Why would the AI think these things? Well, you’re pointing at a bunch of atoms, and the microscopic laws of motion which govern those atoms can be interpreted as minimizing a quantity called action. Or you’re pointing at a bunch of organisms subject to a selection process which (locally) maximizes the number of copies of some information-carrying molecules. How is the AI supposed to know which optimization process you’re pointing to? How can it know which level of abstraction you’re talking about?

What data could tell the AI that you're pointing at humans, not the atoms they're made of?

This sounds like a question which would already have a name, so if anybody could point me to that name, I'd appreciate it.
The problem of other minds
by Elo — 2018-07-24T01:04:06.396Z

Originally posted at Bearlamp.com.au: http://bearlamp.com.au/the-problem-of-other-minds/ (also Greaterwrong)

I'm trapped in here and you are trapped in there.

It's an odd problem. How do we communicate when we are each locked in our own Chinese room.

In computing terms - a handshake allows for an affirmation of knowledge to be returned. Unfortunately a completed handshake does not confirm shared knowledge. RandomAgent can sometimes confirm a handshake with YesAgent by getting randomly the right 1’s and 0’s to fulfil YesAgent’s criteria for confirmation.

The problem of other minds is that we can't necessarily trust both agents to be using the same communication method. We could have a bit-flip, we could have a different code. We can't assume the method of communication is sound.

This is seen in the Two Generals’ Problem. Where each general cannot trust the other general’s communication.

So what can we do? One solution is a three pass encryption protocol. With this system, each agent self-verifies the message.

But for all that I do, No matter how hard I try to pass the message, I will never know if the same subjective qualia of the colour blue is being seen by you.

Next: The experimental apparatus (greaterwrong)
The risk of an American Civil War is remote
by Samo Burja — 2018-07-23T18:00:02.072Z

This essay was originally published on SamoBurja.com. You can

access the original here.

Every now and then I hear people express concern that political polarization might lead to a new American Civil War.

Such concerns should not be immediately dismissed. The consequences of a civil war are grave. Such wars can be very bloody and wipe out decades of prosperity. They can be accompanied by revolutions that radically change the governance of a country, often for the worse. If the state remains united, the aftermath is usually one of political centralization.

Civil war can also be followed by the disintegration of the state. History shows that even Great Powers are not immune. The term describing such processes, Balkanization, isn’t merely an abstraction to me. Rather it describes the backdrop of my childhood. I was born in 1988 in the Socialist Republic of Slovenia, a constituent republic of the Federal Socialist Republic of Yugoslavia. This country no longer exists. And while Slovenia itself did rather well, the Yugoslav Wars were a disaster for the region. Will the same be said of California one day?

Since the end of history hasn’t arrived, we must seriously evaluate such scenarios every now and then. Setting aside material interests or the general good, evaluating such political risks can be important for personal safety.

Let’s begin.

It is an interesting question whether America is actually politically polarized, but for the sake of argument let’s say it is. To evaluate the possibility of an American civil war, we should first understand the origin of civil wars in general. You cannot defer to indicators such as pundit statements or perceived public sentiment.

If organic public sentiment drove war and peace, there would be far fewer wars, and the wars that would be fought would be much more stupid and destructive (admittedly the bar is pretty low in that respect already). There does not seem to be a plausible candidate for a popular peace movement that successfully prevented a war.

War is driven by the interests of the powerful. In some societies the powerful might be military officers, in others they are priests, and in others they are diplomats. The incentives and requirements of these people are in turn determined by factors such as geopolitical position, the power landscape, demographics, office politics, party po...
Alignment Newsletter #16: 07/23/18
by Rohin Shah — 2018-07-23T16:20:03.039Z

Highlights

Seedbank — discover machine learning examples

(Michael Tyka): Seedbank provides interactive machine learning examples in Colab notebooks (think Jupyter notebooks in the cloud). This makes it really easy to just run example code without any setup, and even to modify it to play around with it. Google even provides a free GPU to make the training and inference faster!

My opinion: I haven't explored it yet, but this seems great, especially if you want to learn ML. I have used Colab notebooks before and recommend them highly for small projects (maybe even large ones, I'm not sure), especially if you're familiar with Jupyter notebooks.

Announcement: AI alignment prize round 3 winners and next round

(Zvi Mowshowitz and Vladimir Slepnev): The winners of the second round of the

AI Alignment Prize

have been announced! Vadim Kosoy wins the first prize of $7500 for

The Learning-Theoretic AI Alignment Research Agenda

, and Alexander Turner wins the second prize of $2500 for

Worrying About the Vase: Whitelisting

and

Overcoming Clinginess in Impact Measures

. The next round has started and will last until December 31, and each participant has been asked to submit a single entry (possibly in parts).

DeepMind hiring Research Scientist, Safety

: Career opportunity!

Previous newsletters

Pascal’s Muggle Pays

(Zvi) (H/T Alex Mennen): Last week I mentioned non-exploitability as a justification for not paying Pascal's mugger. Alex pointed me to this post, which makes this argument, which I had seen before, but more importantly to

these

comments

that argue against it, which I hadn't seen. The basic idea is that the downside of being continuously exploited in the real world is still not bad enough to cancel out the potentially huge upside in the (very unlikely) world where the mugger is telling the truth.

My opinion: I'm convinced, non-exploitability doesn't save you from being Pascal's mugged. My current opinion on Pascal's mugging is ¯_(ツ)_/¯

Technical AI alignment

Technical agendas and prioritization

Mechanism design for AI

(Tobias Baumann): One cause of outcomes worse than extinction could be escalating conflicts between very capable AI systems (that could eg. threaten to simulate suffering beings). It is worth studying how we could have AI systems implement mechanism design in order to guide such systems into more cooperative behavior....
Fundamentals of Formalisation Level 6: Turing Machines and the Halting Problem
by philip_b — 2018-07-23T09:46:42.076Z

Followup to Fundamentals of Formalisation Level 5: Formal Proof. First post.

This is a new lesson of our online course on math formalizations required for AI safety research.

The big ideas:

Turing Machine

The Halting Problem

To move to the next level you need to be able to:

Describe a turing machine and write basic turing algorithms.

Give the basic idea of the halting problem.

Give the basic idea of the proof that the halting function isn’t computable.

Why this is important:

This is the beginning of learning formal computability. This allows us to think about computation and it's intrinsic limitations. The first step to reasoning about Artificial General Intelligence will be understanding how it can be done before we worry about its practical feasibility.

The halting problem is a pervasive and annoying problem. It’ll come up again and again, in many disguises. If we could solve the halting problem, things would be easy. Since we can’t, things are hard. Understanding the limitations it imposes is important.

For every lesson you have 2 options: do the whole thing, or skip to the questions and exercises in the end. The latter option is for people who suspect they already know the subject. It serves as a means of verifying or falsifying that hypothesis.
Let's Discuss Functional Decision Theory
by Chris_Leong — 2018-07-23T07:24:47.559Z

I've just finished reading through Functional Decision Theory: A New Theory of Rationality, but there are some rather basic questions that are left unanswered since it focused on comparing it to Casual Decision Theory and Evidential Decision Theory:

How is Functional Decision Theory different from Timeless Decision Theory? All I can gather is that FDT intervenes on the mathematical function, rather than on the agent. What problems does it solve that TDT can't? (Apparently it solves Mechanical Blackmail with an imperfect predictor and so it should also be able to solve Counterfactual Mugging?)
How is it different from Updateless decision theory? What's the simplest problem in which they give different results?
Functional Decision Theory seems to require counterpossibilities, where we imagine that a function output a result that is different from what it outputs. It further says that this is a problem that isn't yet solved. What approaches have been tried so far? Further, what are some key problems within this space?
Replace yourself before you stop organizing your community.
by Raemon — 2018-07-22T20:57:25.903Z

Alternate title: Always be working to replace yourself in whatever capacity people rely on you.

[CN: This post has some competing access needs, and I made a tradeoff in a particular direction. I intend this as an earnest, important suggestion, not a demand. If you have scrupulosity vulnerability you may want the opposite advice.]

If you're not building organizational capacity, you're burning it. If your community is not growing, it's probably shrinking. If the system works fine but it depends on you not getting hit by a bus (or getting bored, or burnt out, or attracted to a new interest, or moving across the country), than the system is not fine.

I wrote a bunch about this in Melting Gold. It's important.

But I'm about to write up a bunch of thoughts that are centered around the Berkeley community. And before I feel comfortable doing that, I wanted to draw attention a special case of this general rule.

Cautionary Tales

For years, I lived in NYC. The community there was one of the first strong rationalist centers. It predates Berkeley as actual-factual-honest-to-goodness community. This writeup about it still draws new people to it, as does Eliezer's Epistle to the New York Less Wrongians. People arrive, excited by the promise of a home.

Right now, NY is doing quite well, but it's been through several challenging periods. During the golden age of yore, several leaders left for the SF Bay area, one after the other. The first instance or two were recoverable. But it happened faster than the meetup could find or build new leaders.

Meanwhile, I know of other communities that are not doing as well.

Locally, it often makes sense for a new, excited agenty rationalist to move to Berkeley. It's where several organizations are and it's easier to get involved with The Mission. There are benefits to having lots of people in one place.

But the aggregate effect of this is that places like NYC are disincentivized from generating agenty people, since they often end up leaving.

By now, I've moved to Berkeley. I'm not going to pull a "do as I say, not as I do." The things that attract people shaped-like-me to the Bay are real – this is not a set of incentives and tradeoffs you can just coordinate away, even if you got everyone on the same page about it.

(I don't think you can even coordinate around "moving the rationalist hub somewhere cheaper" – there's a reason it's developed around Si...
Who Wants The Job?
by Zvi — 2018-07-22T14:00:00.296Z

Over this last month, my wife and I searched for and hired a new nanny, as ours had decided to learn programming and move to The Bay.

We ended up with a wonderful woman we found through a personal recommendation on a local mailing list. Her previous employer posted saying, hire this woman, she’s fantastic, and this was indeed the case.

Before we realized we’d found her, various people encouraged us to post a job listing on a website, so we did so on Sittercity.

That went… less well.

We got over one hundred applications.

The majority of them had profiles that did not match the requirements of the job. About half were only available for a few months or for part time work. Many others wouldn’t work with multiple children, or had higher salary requirements than we listed.

The vast majority of them had major spelling and/or grammar errors in their profiles. Also in their messages to us.

From the remaining profiles, we reached out to a few dozen applicants.

The majority of them did not respond to us at all.

Of those who did respond, several did not answer the phone for the initial interview and provided no explanation.

Of those who did answer, several were actively rude on the phone.

Of those who were not rude on the phone, several did not engage with the questions being asked or show any interest in the job,

Of those who passed that screening and were asked to come for an in person interview, fully half of them failed to show up, most with no warning or cancellation. In all such cases, they didn’t contact us again. In other cases, they cancelled, but failed to make any real effort to reschedule.

As a result of all this, we only ended up doing two in person interviews. Because it turns out that getting people to show up, at all, is super hard. One of them seemed acceptable in a punch. The other we hired.

The vast majority of people who were on a job site, seeking a job, were not capable of tasks like: Write a profile page in English without major mistakes. Respond promptly when an employer contacts you to respond to your application. Talk politely on the phone and sound like you are listening and care about the job. Show up to your interview.

Yes. Standards are that low.

You are much more employable than you think.

If you’re wondering why employers say it’s so hard to hire people despite getting a hundred applications for every position...
Simplicio and Sophisticus
by Zvi — 2018-07-22T13:30:00.333Z

Previously (Slate Star Codex): The Whole City is Center

Epistemic Status:

Note that after writing a lot of this, I checked and Sniffnoy anticipated a lot of this in the comments, but I think both takes are necessary.

There are many useful points to Scott’s philosophical dialogue, The Whole City is Center, between Simplicio and Sophisticus. I want to point out an extra one I think is important.

Here’s a short summary of some key points of disagreement they have.

Simplicio claims that there are words people use to describe concepts, and we should use those words to describe those concepts, even if those words have unfortunate implicaitons. Say true things about the world. Larry is lazy.

Sophisticus says no, if those words have unfortunate implications we shouldn’t use them. And in many cases, where the unfortunate implications are inevitable because people have those implications about the concept being described, we shouldn’t use any word at all to describe the concept. Larry can be counted on not to do things. But we shouldn’t treat lazy as a thing, because people think being lazy is bad and there’s no utility in thinking Larry is bad.

Simplicio says we should use whatever techniques work, regardless of whether they are negative reinforcement, positive reinforcement, before the act, after the act, too big, too small, you name it, if that’s the system that works. And if people’s natural instincts are to do things that work best as a system, but are sometimes ‘overkill’ or have unfortunate side effects in a particular case, you should accept that.

Sophisticus says no. Studies show negative reinforcement reinforcement doesn’t work, so don’t do it. Studies show harsher prisons don’t deter people so don’t use them. You should only use exactly what is needed to cause a direct effect in each situation. Or, if you need to use deterrence, what the evidence says will actually deter people.

Sophisticus says, we should look upon motivations like ‘I want this person to suffer’ with horror, and assume something has gone horribly wrong. (He makes no comment on feeling ‘I want this particular person to be happy’, which doesn’t come up.)

Simplicio says, if having seemingly unreasonable desires in some situations, including potential future situations, is the way persons an...
Exorcizing the Speed Prior?
by abramdemski — 2018-07-22T06:45:34.980Z

What follows is an argument against finding optimization daemons with too much probability mass in the speed prior. The argument is very fake, but perhaps it could inspire real math down the line. The intuition isn't so different from the one in are minimal circuits daemon free? -- mostly, the speed prior seems like a harder thing to argue is daemon-free.

The speed prior is very similar to a K-complexity prior, except where a K-complexity prior assigns probability proportional to 2−length to a program, the speed prior adds an extra penalty for execution time: 2−length−log(time). Note that the penalty for time is logarithmic compared to the penalty for length; thinking twice as long is penalized the same as being one bit longer.

Optimization daemons arise when we are searching a rich space to try and solve a problem which we don't otherwise know how to solve. This makes it seem like the worst-case scenario for daemons would be if intelligence is really fundamentally about search -- then "try to be intelligent and daemon-free" is a non-starter.

(There are more nuanced possibilities, like "intelligence is about search, but not necessarily search of "rich spaces" which contain daemons. We'll set those aside for this argument.)

So, let's assume that we're in this worst-case scenario. I'm going to make a contrarian argument that this is actually a good thing.

To be more precise: the concern with optimization daemons is that by optimizing hard over a rich search space to achieve a difficult task, you might accidentally pick a intelligent agent out of the search space, who has goals which are somewhat different from yours. I'm going to assume that the only way an agent can be an intelligent is either by performing an expensive brute-force search over possibilities, or having some baked-in knowledge which allows a smarter search, saving it time exponential in how much baked-in knowledge it has.

The assumption makes a little bit of sense, because linearly many bits can locate something in an exponentially large search space. If I knew more complexity theory, maybe I could say why the assumption is really bad. But, let's proceed with the assumption.

The argument is: an optimization daemon which solves the problem via intelligent search can't be higher probability mass than the daemon-free solution. Brute-fo...
12 Virtues of Rationality posters/icons
by habryka — 2018-07-22T05:19:28.856Z

I was trying to take a break today, and ended up redesigning the staircase at our group house (Event Horizon), which is the first thing you see when you enter the house. One thing me and Claire (mingyuan) came up with, was to design small posters for each of the twelve virtues of rationality and line them up on the walls. And I figured I would also let other people have access to the posters, in case you want to do something similar. Here are all the 12 icons/posters. The general idea was inspired by these awesome minimalist philosophy posters.

And I've uploaded the illustrator file I've used here.

The twelfth virtue doesn't have an icon, we chose to represent it with an empty frame.

Here is the staircase after we finished putting everything up (sorry, lighting isn't perfect):