trentmkelly/LessWrong-43k huggingface dataset:
Request for Steelman: Non-correspondence concepts of truth
by PeerGynt — 2015-03-24T03:11:18.651Z

A couple of days ago, Buybuydandavis wrote the following on Less Wrong:

I'm increasingly of the opinion that truth as correspondence to reality is a minority orientation.

I've spent a lot of energy over the last couple of days trying to come to terms with the implications of this sentence. While it certainly corresponds with my own observations about many people, the thought that most humans simply reject correspondence to reality as the criterion for truth seems almost too outrageous to take seriously. If upon further reflection I end up truly believing this, it seems that it would be impossible for me to have a discussion about the nature of reality with the great majority of the human race. In other words, if I truly believed this, I would label most people as being too stupid to have a real discussion with.

However, this reaction seems like an instance of a failure mode described by Megan McArdle:

I’m always fascinated by the number of people who proudly build columns, tweets, blog posts or Facebook posts around the same core statement: “I don’t understand how anyone could (oppose legal abortion/support a carbon tax/sympathize with the Palestinians over the Israelis/want to privatize Social Security/insert your pet issue here)." It’s such an interesting statement, because it has three layers of meaning.

The first layer is the literal meaning of the words: I lack the knowledge and understanding to figure this out. But the second, intended meaning is the opposite: I am such a superior moral being that I cannot even imagine the cognitive errors or moral turpitude that could lead someone to such obviously wrong conclusions. And yet, the third, true meaning is actually more like the first: I lack the empathy, moral imagination or analytical skills to attempt even a basic understanding of the people who disagree with me

In short, “I’m stupid.” Something that few people would ever post so starkly on their Facebook feeds.

With this background, it seems important to improve my model of people who reject correspondence as the criterion for truth. The obvious first place to look is in academic philosophy. The primary challenger to correspondence theory is called “coherence theory”. If I understand correctly, coherence theory says that a statement is true iff it is...
Learning by Doing
by Adam Zerner — 2015-03-24T01:56:43.462Z

Most people believe very strongly that the best way to learn is to learn by doing. Particularly in the field of programming.

I have a different perspective. I see learning as very dependency based. Ie. there are a bunch of concepts you have to know. Think of them as nodes. These nodes have dependencies. As in you have to know A, B and C before you can learn D.

And so I'm always thinking about how to most efficiently traverse this graph of nodes. The efficient way to do it is to learn things in the proper order. For example, if you try to learn D without first understanding, say A and C, you'll struggle. I think that it'd be more efficient to identify your what your holes are (A and C) and address them first before trying to learn D.

I don't think that the "dive into a project" approach leads to an efficient traversal of this concept graph. That's not to say that it doesn't have its advantages. Here are some:

People tend to find the act of building something fun, and thus motivating (even if it has no* use other than as a means to the end of learning).

It's often hard to construct a curriculum that is comprehensive enough. Doing real world projects often forces you to do things that are hard to otherwise address.

It's often hard to construct a curriculum that is ordered properly. Doing real world projects often is a reasonably efficient way of traversing the graph of nodes.

Personally I think that as far as 2) and 3) go, projects sometimes have their place, but they should be combined heavily with some sort of more formal curriculum, and that the projects should be focused projects. My real point is that the tradeoffs should always be taken into account, and that an efficient traversal of the concept graph is largely what you're after.

I should note that I feel most strongly about projects being overrated in the field of programming. I also feel rather strongly about it for quantitative fields in general. But in my limited experience with non-quantitative fields, I sense that 2) and 3) are too difficult to do formally and that projects are probably the best approximations (in 2015; in the future I anticipate smart tutors being way more effective than any project ever was or can be). For example, I've spent some time trying to learn design by reading books and stuff on the internet, but I sense that I'm really missing something that is hard to get without

...
Superintelligence 28: Collaboration
by KatjaGrace — 2015-03-24T01:29:21.415Z

This is part of a weekly reading group on Nick Bostrom's book, Superintelligence. For more information about the group, and an index of posts so far see the announcement post. For the schedule of future topics, see MIRI's reading guide.

Welcome. This week we discuss the twenty-eighth section in the reading guide: Collaboration.

This post summarizes the section, and offers a few relevant notes, and ideas for further investigation. Some of my own thoughts and questions for discussion are in the comments.

There is no need to proceed in order through this post, or to look at everything. Feel free to jump straight to the discussion. Where applicable and I remember, page numbers indicate the rough part of the chapter that is most related (not necessarily that the chapter is being cited for the specific claim).

Reading: “Collaboration” from Chapter 14

Summary

The degree of collaboration among those building AI might affect the outcome a lot. (p246)

If multiple projects are close to developing AI, and the first will reap substantial benefits, there might be a 'race dynamic' where safety is sacrificed on all sides for a greater chance of winning. (247-8)

Averting such a race dynamic with collaboration should have these benefits:

More safety

Slower AI progress (allowing more considered responses)

Less other damage from conflict over the race

More sharing of ideas for safety

More equitable outcomes (for a variety of reasons)

Equitable outcomes are good for various moral and prudential reasons. They may also be easier to compromise over than expected, because humans have diminishing returns to resources. However in the future, their returns may be less diminishing (e.g. if resources can buy more time instead of entertainments one has no time for).

Collaboration before a transition to an AI economy might affect how much collaboration there is afterwards. This might not be straightforward. For instance, if a singleton is the default outcome, then low collaboration before a transition might lead to a singleton (i.e. high collaboration) afterwards, and vice versa. (p252)

An international collaborative AI project might deserve nearly infeasible levels of security, such as being almost completely isolated from the world. (p253)

It is good to start collaboration early, to benefit from being ignorant about who will benefit more f

...
[Clearing out my Drafts folder] Rationality and Decision Theory Curriculum Idea
by orthonormal — 2015-03-23T22:54:51.241Z

Note: The following is a draft post I've had since 2009, and it's not great but it's worth posting for discussion. I do like the way that it prefigures some of the problems of Quirrell Points when traitors are allowed...

Need to see if this can be easily gamed, but...

Step 1. Introduce Prisoner's Dilemma. Set up computer system so that they can log in and play it in partners with investments of points (caution them: this is their actual grade at stake). Let them know that they currently don't have enough points for a passing grade on that part of the course, but that maximum investment and mutual cooperation will result in A's for (almost) everyone on it (with high probability); also that points are converted to grades on a logarithmic scale. Let them know that creating institutions and alliances is a good strategy in such games.

Initially, each student is allowed to play once per day, with 1 partner. Students log in, enter the name of their requested PD partner, enter how much they're willing to invest, and enter C or D. They'll get a "bank statement" daily as well.

If both enter C, they each get 1.2 points back (per initial point invested). If one enters C and the other D, the cooperator gets nothing back and the defector gets 2 points (per point invested). If both enter D, then each gets 0.5 points back.

Once they've had some practice with this, we move to

Step 2: Bigger investments, luck, observation.

Introduce larger group investments with higher rates of return. E.g. a five-person opportunity that pays each C player 0.2 guaranteed plus 0.4 for each C (not counting themselves), and each D player gets 0.5 guaranteed plus 0.5 for each C. (Set these up to be balanced in some way.)

Add a factor of luck, so that people can't just (be forced to) show one another their bank statements as proof of their cooperation. People should average the proper amounts, but have enough variation that it's often difficult to tell whether they cooperated or defected. (Or is there another way to handle this?)

Finally, allow a third option of observation within a deal. One idea: if you choose to observe, then you take up a spot in the investment, keep your own point, and don't count as a C for others, but you get to see some or all of the other players' choices. Make sure that information is proportionately expensiv...
An Introduction to Löb's Theorem in MIRI Research
by orthonormal — 2015-03-23T22:22:26.908Z

Would you like to see a primer on several MIRI research topics (assuming only the background of having taken a course with proofs in math or computer science)? Or are you curious why MIRI does so much with mathematical logic, and why people on Less Wrong keep referring to Löb's Theorem?

If you answered yes to either question, you may be interested in my lecture notes, An Introduction to Löb's Theorem in MIRI Research! These came out of an introductory talk that I gave at a MIRIx workshop.

Since I've got some space here, I'll just copy and paste the table of contents and the introduction section...

Contents

1 Introduction

2 Crash Course in Löb's Theorem

2.1 Gödelian self-reference and quining programs

2.2 Löb's Theorem

3 Direct Uses of Löb's Theorem in MIRI Research

3.1 “The Löbstacle”

3.2 Löbian cooperation

3.3 Spurious counterfactuals

4 Crash Course in Model Theory

4.1 Axioms and theories

4.2 Alternative and nonstandard models

5 Uses of Model Theory in MIRI Research

5.1 Reflection in probabilistic logic

6 Crash Course in Gödel-Löb Modal Logic

6.1 The modal logic of provability

6.2 Fixed points of modal statements

7 Uses of Gödel-Löb Modal Logic in MIRI Research

7.1 Modal Combat in the Prisoner’s Dilemma

7.2 Modal Decision Theory

8 Acknowledgments

1 Introduction

This expository note is devoted to answering the following question: why do many MIRI research papers cite a 1955 theorem of Martin Löb, and indeed, why does MIRI focus so heavily on mathematical logic? The short answer is that this theorem illustrates the basic kind of self-reference involved when an algorithm considers its own output as part of the universe, and it is thus germane to many kinds of research involving self-modifying agents, especially when formal verification is involved or when we want to cleanly prove things in model problems. For a longer answer, well, welcome!

I’ll assume you have some background doing mathematical proofs and writing computer programs, but I won’t assume any background in mathematical logic beyond knowing the usual logical operators, nor that you’ve even heard of Löb’s Theorem before.

To motivate the mathematical sections that follow, let’s consider a toy problem. Say that we’ve designed Deep Thought 1.0, an AI that reaso...
Welcome, new contributors!
by orthonormal — 2015-03-23T21:53:20.000Z

Today is the day; we're opening up this forum to allow contributions from more people! See our How to Contribute page for the details.

Now is a great time to say a bit about what the Intelligent Agent Foundations Forum is, and how it came about. The short answer is that the Machine Intelligence Research Institute (MIRI) helped build this forum in order to facilitate research discussion on the topics in its technical agenda and related subjects.

Many of the early users of this forum previously contributed to a closed email group on decision theory, or wrote relevant posts on the group blog Less Wrong. MIRI wanted to build a forum that could focus on these topics and support high-quality mathematical collaboration, while being transparent and allowing new contributors to find and join it directly.

Broadly speaking, the topics of this forum concern the difficulties of value alignment- the problem of how to ensure that machine intelligences of various levels adequately understand and pursue the goals that their developers actually intended, rather than getting stuck on some proxy for the real goal or failing in other unexpected (and possibly dangerous) ways. As these failure modes are more devastating the farther we advance in building machine intelligences, MIRI's goal is to work today on the foundations of goal systems and architectures that would work even when the machine intelligence has general creative problem-solving ability beyond that of its developers, and has the ability to modify itself or build successors. (For still more related info on the motivations for this work, see the Future of Life Institute's research priorities letter or Nick Bostrom's recent book Superintelligence.)

In that context, there are many interesting problems that come up; here are several from MIRI's technical agenda page:

Decision theory: One class of topics comes from the distortions that arise when an agent predicts its environment, including its own future actions or the predictions of other agents, and tries to make decisions based on those. The tools of classical game theory and decision theory begin to make substandard recommendations on Newcomblike problems, blackmail problems, and other topics in this domain, and formal models of decision theories have brought up entirely unexpected self-referential failure modes. This has spurred the development of some new mathematical

...
What you know that ain't so
by NancyLebovitz — 2015-03-23T18:24:58.730Z

This is an analysis of the Yom Kippur war (Egypt vs. Israel, 1973)-- the Israelis were interested in how Egypt managed a surprise attack, and it turned out that too many Israelis believed that the Egyptians would only attack if they had rockets which could reach deep into Israel. The Egyptians didn't have those rockets, so the Israeli government ignored evidence that the Egyptians were massing military forces on the border.

The rest of the article is analysis of the recent Israeli election, but to put it mildly, an election has much less in the way of well-defined factors than a surprise military attack, so it's much harder to say whether any explanation is correct.

I'm sure there are many examples of plausible theories keeping people from getting to the correct explanation for a long time. Any suggestions? Also, is there a standard name for this mistake?
Meetup : Washington, D.C.: TBA
by RobinZ — 2015-03-23T16:54:20.687Z

Discussion article for the meetup : Washington, D.C.: TBA

WHEN:
29 March 2015 03:00:00AM (-0400)

WHERE:
Reynolds Center

Note: The meetup location has changed back.

As it will be quite cold on Sunday afternoon, singing is postponed; as we have other plans for next week (hello, DC-area HPMoR fans!), we will not be switching the two meetups. Therefore, we will be meeting in the Kogod Courtyard of the Reynolds Center to do ... something. As usual, we will congregate between 3:00 to 3:30.

There is no WMATA major track work scheduled; the service advisory for this weekend has Blue and Yellow running every 18 minutes due to single-tracking between Braddock Road and Reagan National Airport. The Verizon Center calendar for March 29 has a Wizards vs. Houston Rockets game starting at 12:30 (doors 11:30); it would probably be ending around 2:30 to 3:00 p.m.

Upcoming meetups:

Apr. 5: Fun & Games (bring games, play games, converse, socialize, or any combination thereof)

Apr. 12: Cherry Blossoms (take a walk and look at the cherry blossoms)

Apr 19: "The Sword of Good" (talk about a LessWrongian short story)

Discussion article for the meetup : Washington, D.C.: TBA
Intelligence modules
by Stuart_Armstrong — 2015-03-23T16:24:14.176Z

A putative new idea for AI control; index here.

This idea, due to Eric Drexler, is to separate out the different parts of an AI into modules. There would be clearly designated pieces, either physical or algorithmic, with this part playing a specific role: this module would contain the motivation, this module the probability estimator, this module the models of the outside world, this module the natural language understanding unit, etc...

It's obvious how such a decomposition would be useful for many of the methods I've been detailing here. We could also distil each module - reduce it to a smaller, weaker (?) and more understandable submodule, in order to better understand what is going on. In one scenario, an opaque AI gets to design its successor, in the form of a series of such modules.

This property seems desirable; the question is, how could we get it?

EDIT: part of the idea of "modules" is that AIs often need to do calculations or estimations that would be of great value to us if we could access them in isolation. This idea is developed more in these posts.

Designing in modules

The main threat here is that a given submodule would contain more than just the properties we want. After all, a natural language parser could consist of a general intelligence plus a motivation to understand language. Another possible worry is that the modules are overfitted to the problem or to each other: the language parser works perfectly, but only in this one AI design.

There are several ways we could try and combat this.

Minimum length to the program (Eric's original main idea). The module must be as short as possible for its purpose.

If possible, start with a modularised AI, and track how the modules change as the AI's intelligence grows.

Have the AI extend the module to make it more transparent, without adding any extra functionality.

Check that the input-output map the AI presents (maybe require it so present it as a simple primitive recursive function with clear and simple "for" loops) is of commensurable complexity to the task at hand.

Add some noise to the situation: the new AI will not get exactly the same data as the old, the modules may get somewhat altered, and still have to work. Bugs will be introduced (this approach helps to combat the overfitting aspect of minimum length description).

Require that the module be "distant", according to some metric of mo

...
Humans get different counterfactuals
by Stuart_Armstrong — 2015-03-23T14:58:28.000Z

A putative new idea for AI control; index here.

A lot of my ideas rely upon taking counterfactuals of events that have tiny ε probabilities of ever happening. This introduces some extra risks, mainly centred around human behaviour. For the counterfactual methods to work, we need the alternate world to be sensible, without things going crazy due to human action.

For instance, imagine that X defines an event where an AI would get turned on, with ¬X (of probability ε) corresponding to the AI failing to get turned on. There are two risks here: the first is that humans would react by saying "wow, an event of probability of ε actually happened; all our models must be wrong! Let's go crazy! Yay!". The second is that humans react by saying: "well, that was embarrassing - let's turn the AI on anyway."

To avoid this issue, imagine the following setup:

The "ON" signal first goes through an event Y, which has 99% chance of letting it through, then to the event X, which (as before) has a 1-ε chance of letting it through. The setup is designed so that humans cannot distinguish between ¬Y (the signal gets blocked at the first stage) and ¬X (the signal gets blocked at the second stage). This only needs to fool humans, not the AI itself.

The AI defines counterfactuals, as before, by looking at ¬X (possibly conditioning on Y versus ¬Y, if this is needed). Everything proceeds as previously from its perspective.

From the human perspective, however, the ¬X world is not distinguishable from the ¬Y one. Given (¬Y or ¬X), humans would conclude that ¬Y is the much more likely option:

P(¬Y|¬Y or ¬X)≈1-100ε.

So the ¬X counterfactual world (for the AI) is one where humans behave as if they were in the ¬Y world.

And ¬Y has one chance in a hundred of happening, which is unlikely, but not enough for humans to assume that their whole model of reality is wrong. Also, this is sufficiently likely that humans would give serious thought as to what to do in the ¬Y case, maybe arranging various pre-commitments or insurance options, making it possible to arrange (with high probability) that humans don't just ignore the result and try again immediately.

Note that this method can't be used (obviously) if ¬X is something hideously dangerous (like an unleashed UFAI), but in all other cases, it seems implementable.
Humans get different counterfactuals
by Stuart_Armstrong — 2015-03-23T14:54:51.864Z

A putative new idea for AI control; index here.

A lot of my ideas rely upon taking counterfactuals of events that have tiny ε probabilities of ever happening. This introduces some extra risks, mainly centred around human behaviour. For the counterfactual methods to work, we need the alternate world to be sensible, without things going crazy due to human action.

For instance, imagine that X defines an event where an AI would get turned on, with ¬X (of probability ε) corresponding to the AI failing to get turned on. There are two risks here: the first is that humans would react by saying "wow, an event of probability of ε actually happened; all our models must be wrong! Let's go crazy! Yay!". The second is that humans react by saying: "well, that was embarrassing - let's turn the AI on anyway."

To avoid this issue, imagine the following setup:

The "ON" signal first goes through an event Y, which has 99% chance of letting it through, then to the event X, which (as before) has a 1-ε chance of letting it through. The setup is designed so that humans cannot distinguish between ¬Y (the signal gets blocked at the first stage) and ¬X (the signal gets blocked at the second stage). This only needs to fool humans, not the AI itself.

The AI defines counterfactuals, as before, by looking at ¬X (possibly conditioning on Y versus ¬Y, if this is needed). Everything proceeds as previously from its perspective.

From the human perspective, however, the ¬X world is not distinguishable from the ¬Y one. Given (¬Y or ¬X), humans would conclude that ¬Y is the much more likely option:

P(¬Y|¬Y or ¬X)≈1-100ε.

So the ¬X counterfactual world (for the AI) is one where humans behave as if they were in the ¬Y world.

And ¬Y has one chance in a hundred of happening, which is unlikely, but not enough for humans to assume that their whole model of reality is wrong. Also, this is sufficiently likely that humans would give serious thought as to what to do in the ¬Y case, maybe arranging various pre-commitments or insurance options, making it possible to arrange (with high probability) that humans don't just ignore the result and try again immediately.

Note that this method can't be used (obviously) if ¬X is something hideously dangerous (like an unleashed UFAI), but in all o...