trentmkelly/LessWrong-43k huggingface dataset:
New paper: AGI Agent Safety by Iteratively Improving the Utility Function
by Koen.Holtman — 2020-07-15T14:05:11.177Z

This post is to announce my new paper AGI Agent Safety by Iteratively Improving the Utility Function. I am also using this post to add some extra background information that is not on the paper. Questions and comments are welcome below.

From the abstract:

While it is still unclear if agents with Artificial General Intelligence (AGI) could ever be built, we can already use mathematical models to investigate potential safety systems for these agents. We present an AGI safety layer that creates a special dedicated input terminal to support the iterative improvement of an AGI agent's utility function. The humans who switched on the agent can use this terminal to close any loopholes that are discovered in the utility function's encoding of agent goals and constraints, to direct the agent towards new goals, or to force the agent to switch itself off.

An AGI agent may develop the emergent incentive to manipulate the above utility function improvement process, for example by deceiving, restraining, or even attacking the humans involved. The safety layer will partially, and sometimes fully, suppress this dangerous incentive. [...]

The above corrigibility or stop button design problem has been considered before in the AGI safety literature: see the paper for detailed references to related work. In meta-discussions about this topic, both on and off the web, I have frequently found a line of speculative reasoning which says that entirely new ways of thinking about machine reasoning might be needed, before these problems can be fully solved. Radical breakthroughs are always welcome, but the line of reasoning advanced in this paper goes the opposite way: there is plenty of room to make progress within the scope of the current standard models.

In developing and presenting the safety layer in the paper, I have tried to stay as close as possible to mainstream agent and machine learning models. Using these, I present a specific well-defined AGI safety layer design, which produces specific well-defined safety properties.

One important target audience for this paper is readers who have a Machine Learning or broader Computer Science background, readers who are thinking about maybe doing some AGI safety research. Specifically, I am hoping to enable and encourage more work in the sub-field of AGI safety research that investigates new forms of utility function enginee...
Education 2.0 — A brand new education system
by aryan — 2020-07-15T10:09:05.091Z

What this essay does is that it introduces a brand new education system. This education system is way more effective than the current education system we have in place. And unlike the current education system, which has been the same for a long time. This one is constantly getting better with time and data, thanks to AI.

The best thing about it is that it's totally free of cost. So anyone in the world with access to a basic smartphone+Internet can access Education 2.0. Meaning, now everyone has the same educational opportunities as a Harvard/Stanford student.

You see, education is the most fundamental thing to human progress. And once you deploy something like Education 2.0 on a global scale, it will have huge positive implications on economic growth, life quality, and human progress. And to emphasise what I just said, the collective intelligence of humanity is the best predictor of future progress. Here is some empirical data to prove this statement. Nevertheless, you don't need empirical evidence; look around, education is what drives societal growth.

Looking at the current education system we have in place. It would be unfair to say that it hasn't done an excellent job in aiding human progress. Resolving the COVID-19 Pandemic, the Apollo Program, the iPhone, Tesla, SpaceX, the Internet, All of Science etc. are a prime example.

But it certainly hasn't been able to keep up with the growing human population, and the challenges raised by it. Looking at all the pressing issues, we face today like global poverty, climate change, and war. The incapability of civilisation to solve these problems stems from the fact that it's not educated enough as a whole. There is conclusive evidence to prove this — Education correlates with prosocial behaviour. Prosocial behaviour is behaviour, which affects society as a whole.

So the best way to ensure that we progress into a brighter future is by rewiring the current education system. So it can keep up with the ever-growing population and the challenges we face, which will then eventually increase the collective wisdom and prosocial behaviour of humans at a much faster rate.

There are 3 main issues with the current education system, which dramatically lowers its potential.

Its inability to reach a large population of the world.
The unequal distribution of resources.
The low efficacy of th

...
Classification of AI alignment research: deconfusion, "good enough" non-superintelligent AI alignment, superintelligent AI alignment
by philip_b — 2020-07-14T22:48:04.929Z

I want to work on technical AI alignment research. I am trying to choose
an AI alignment area to get into. To do that, I want to understand the
big picture of how to make AI go well through technical research. This
post contains my views on this question. More precisely, I list types of
AI safety research and their usefulness.

Probably, many things in this post are wrong. I want to become less
wrong. The main reason why I am posting this is to get feedback. So, please

send me links to alternative big picture views on how to make AI go
well

tell me all the places where I am wrong and all the places where I
am right

talk to me in the comments section, via DM, via
email, or schedule a video chat.

Three fuzzy clusters of AI alignment research

The research aimed to
deconfuse
us about AI. Examples of this are AIXI, MIRI's agent foundations,
Michael Dennis' game theory, Tom Everitt's agent incentives,
Vanessa Kosoy's incomplete models, Comprehensive AI services.

The research aimed to provide "good enough" alignment for
non-superintelligent AI. This category includes more hacky and less
theory-based research, which probably won't help with the alignment
of superintelligent AI. By superintelligent AI, I mean AI, which is
much smarter than humans, much faster, and is almost omnipotent. Examples:
everything under the umbrella of (prosaic AI
alignment),
robustness of neural networks against adversarial examples,
robustness of neural networks against out-of-distribution samples,
almost everything related to neural networks, empirical work on
learning human values, IDA (I think it goes into both 2 and 3), most
things studied by OpenAI and Deepmind (but I am not that sure what
exactly they are studying), Vanessa Kosoy's value learning
protocols
(I think it goes under both 2 and 3).

The research aimed to directly solve one of the problems on our way
to the alignment of both superintelligent and non-superintelligent
AI. Examples: IDA (I think it goes into both 2 and 3), Vanessa
Kosoy's value learning protocols (I think it goes under both 2 and
3), Stuart Armstrong's research
agenda
(I am not sure about this one).

Their usefulness

I think type-1 research is most useful, type-3 is second best, and
type-2 is least useful. Here's why I think so.

At some point, humanity will create a superintelligent AI, unless we
go extinct before. When that happens, we won't be making impor

...
Mazes and Duality
by johnswentworth — 2020-07-14T19:54:42.479Z

(Talk given at an event on Sunday 28th of June. johnswentworth is responsible for the talk, David Lambert edited the transcript.

If you're a curated author and interested in giving a 5-min talk, which will then be transcribed and edited, sign up here.)

This talk relies heavily on visual explanations. The transcript is intended to be readable on its own, but the original video is available here for those readers who prefer that:

https://youtu.be/gZo5Yc5dT90

johnswentworth: So I am going to talk about some funky ways of solving mazes.

Think about how to solve the following maze.

One thing we could imagine doing as a preprocessing step is finding something like this, a barrier that cuts the maze in two.

To solve it, we know we have to go through (A), so we split the whole problem into two.

We say, "Okay, I'm first going to find a way to (A) and then I'm going to find a way to (B)."

Now, this is useful for a few reasons. First, you're breaking it into sub-problems. That's always good.

Second, this barrier applies to lots of different problems. If I'm trying to get from this other (A) to this other (B), we can use this exact same barrier.

Or if I'm trying to get from yet another (A) to another (B), the same barrier still works.

We can reuse this on a lot of different problems. So it's useful in that way.

But finding this barrier is difficult. It's going to take a fair bit of preprocessing work, and will probably be as hard as solving the whole maze in the first place. The question I want to think about here is “how can we find something like this?”

Keep that in the back of your head because this is about to get weird.

For the first step, let's go back to thinking about the normal, old maze problem. We want to get from (A) to (B).

I'm just going to ignore all the constraints on this problem and draw some paths that go through the maze, sort of. There's a perfectly nice path that maybe violates a few constraints here and there, but it's a path.

There's a couple paths through the maze. But obviously, they're breaking the rules of mazes, like the areas where we are violating constraints. As you can see, we’re completely ignoring a few walls.

That is thinking about finding paths through the maze. Let's switch back to the other mode: thinking about finding constraints and barriers.&nbs...
The Goldbach conjecture is probably correct; so was Fermat's last theorem
by Stuart_Armstrong — 2020-07-14T19:30:14.806Z

EDIT: Added a section on Euler's conjecture.

The Goldbach conjecture is likely

The Goldbach conjecture is that "every even integer above two is the sum of two primes". For example, 4=2+2, 6=3+3, 8=5+3, and so on.

Though this is a mathematically precise statement, we can talk about the "probability" of it begin correct. How so?

Well, by the prime number theorem, the probability of a random number less than N being prime, is 1/log(N). So if we sum up all the primes less than N, we get (N/log(N))2 different sums; these sums will be less than 2N.

So, is N itself is one of these sums? Well, the "probability" that it's not the total of any given sum is 1−12N; therefore the probability of it being the total of none of the sums is:

(1−12N)(N/log(N))2=((1−12N)2N)N/(2log(N)2)≈(1/e)N/(2log(N)2).

So the probability of N being the total of such a sum is roughly:

1−e−N/(2log(N)2).

Therefore, the probability of all numbers N being the total of such a sum is roughly:

p2=∞∏N=21−e−N/(2log(N)2).

Now, the infinite product p2 converges to a non-zero number if and only if the sum ∑∞N=1e−N/(2log(N)2) converges to a finite number. That series can be seen to be convergent (for example, by noting that e−N/(2log(N)2)<1/N2 for large enough N and using the comparison test).

If use computers to get an estimate of p2, we get a pretty low probability. However, most of that improbability mass is on the low numbers, and the Goldbach conjecture has been tested up to 4×1018. So, if we assume it's valid up to 1000, we numerically get:

p1000=∞∏N=10001−e−N/(2log(N)2)≈0.9961.

So the Goldbach conjecture is pretty likely, and, the more examples we discover where it holds, the more likely it is to hold all the way to infinity.

"Probabilities" of logical facts

The above reasoning seems dubious. The primes are not defined by random sampling among the natural numbers; quite to the contrary, they come from a mathematical rule of extreme precision. So what do these probabilities mean?

Let X be an infinite set of numbers, selected from the natural numbers in a way that looks like the prime number theorem (eg the n-th number is approximately nlog(n)). Then what we've shown is that, if such an X obeys the "X-Goldbach conjecture" up to 1000, then we'd expect it to go all the way to infinity.

Thus the Goldbach conjecture can be restated as "in terms of sums of two elements, the prime numbers behave like a typical sequenc...
Dremeling
by Czynski — 2020-07-14T19:23:23.036Z

In high school, I was on a robotics team (FRC) who, in honesty, were basically incompetent. (Me included.) We were bad at building a robot, bad at programming it, bad at design, but most relevantly for this post bad at planning ahead.

The particular instance which added a term to my internal monologue was this: We needed to cut slots in some aluminum in order to do...something, I don't remember what. It needed to be about a foot long, so it clearly was worth getting a dedicated tool - I forget what that was, probably a jigsaw. But we didn't have one and it was going to take a week or so to arrive.

What we did have, was a dremel tool. Dremels are small, portable grinders, usable for many things but rarely the best tool for the job. Using a dremel as a slot cutter is workable, but inefficient; it quickly ruins the bit and takes a long time. Nonetheless, despite knowing that the job could be done much more quickly and efficiently if they waited a week, people on our team repeatedly returned to dremeling the slots.

I think this is a common pattern. It's similar to the politician's syllogism ("Something must be done. This is something. Therefore, this must be done."), and to Lost Purposes, but distinct. "Dremeling" is the act of doing something you know is a bad way to attack the problem, and probably a waste of time, because it moves you incrementally towards the goal, and you can't do anything more helpful right now.

There are better ways; you know there are better ways and have at least a pretty good idea of what they are and how you could switch to one of those ways. But, in the moment, the better ways are unavailable and the bad way is available. And so you do the (dumb, wasteful, barely-helpful) thing you can do right now, because it will technically make progress.

Why does this happen? Well, not doing anything is uncomfortable. It often feels like failure. If others see you not doing anything, it may open you up to shame; at the national scale this is the politician's syllogism, but the politics of a peer group are just as terrifying - and probably more terrifying - than the party politics of a large polity.

So, as we learn from (Brooks, Reiner, 1975), it stems from fear.
Calibration Practice: Retrodictions on Metaculus
by Raemon — 2020-07-14T18:35:36.860Z

Edit: there is now a cool app for this over here https://www.quantifiedintuitions.org/pastcasting

I've been working on improving my calibration. I practiced on things like the Credence Game and OpenPhil's Calibration Game, and both were fine. But I felt like they didn't quite capture the felt senses from making real-world predictions about topics that I really cared about.

After exploring the options, I started making predictions on PredictionBook – it was simple, lightweight, and let me enter dates for prediction-resolution using intuitive English phrases like "a week from now" or "August 1st" instead of having to go hunting through the date picker.

I've found it generally useful to notice when I've formed an implicit prediction (for example: "My coworker isn't going to get their project done today", or "Covid cases will be lower-per-capita at this particular town that I'm planning to move to"), then actually enter it into PredictionBook.

If I notice an implied prediction while I'm not near my computer, I sometimes make the prediction as if I were at PredictionBook, and then see it resolve, and then enter everything into PredictionBook after the fact to see how it affects my overall calibration. (One example, when I had taken a walk and gotten lost: "At this next road I'm supposed to turn right", which turned out to be correct)

Retrodictions on Metaculus

My prediction-habits still didn't add up to that many real-world predictions. But, I found a neat way to practice calibration on a bunch of real-world questions in a row is to go to Metaculus, pick a category with a lot of questions, and then filter by resolved questions.

By default this will show you questions along with the resolution, but you can hide the resolutions by creating a custom style using the Stylus browser extension (chrome / firefox). The relevant CSS is:

.question-table__header > :nth-child(2), .question-table__row > :nth-child(2) {
display:none
}

.question-table__header > :nth-child(3), .question-table__row > :nth-child(3) {
display:none
}

Some questions were ones I already knew the answers to, but for many of them there was still some uncertainty about exactly how some event had played out, enough such that practicing calibration on it was useful.

Overall, the number of questions turned out to be not be as large as I'd originally thought, but...
AI Benefits Post 4: Outstanding Questions on Selecting Benefits
by Cullen — 2020-07-14T17:26:12.537Z

This is a post in a series on "AI Benefits." It is cross-posted from my personal blog. For other entries in this series, navigate to the AI Benefits Blog Series Index page.

This post is also discussed on the Effective Altruism Forum.

For comments on this series, I am thankful to Katya Klinova, Max Ghenis, Avital Balwit, Joel Becker, Anton Korinek, and others. Errors are my own.

If you are an expert in a relevant area and would like to help me further explore this topic, please contact me.

AI Benefits Post 4: Outstanding Questions on Selecting Benefits

The final posts in this series contain questions I am currently contemplating about AI Benefits, which I’d love help and feedback on! There are two such posts: questions about selecting Benefits (this post) and questions about governing the Benefits process (the next post). The line between these categories is blurry because the governance process will (and indeed is designed to) influence the selection process. However, an intuitive—if imperfect—distinction between these is:

“Selection” questions focus on identifying and choosing between the types of Benefits that a Benefactor could produce.

“Governance” questions focus on the social institutions that affect the selection process and enhance its legitimacy.

Framing: A Portfolio Approach to Benefits

I currently think that large-scale AI Benefactors (e.g, technology companies like OpenAI or DeepMind) should initially adopt a portfolio approach to Benefits. This is certainly true for Benefactors that are risk-averse. However, even risk-neutral Benefactors should consider a portfolio approach due to:

A need to try many different Benefits to discover which are most beneficial and learn how to deploy Benefits well, and

The heterogeneity of the world, which means that an undiversified approach to Benefits might be too unequal.[1]

Topics in this post generally concern how to allocate resources within the overall portfolio of Benefits. Note that these topics are not fully independent, and so might intersect.

Direct versus Indirect Benefits

As noted in a previous post, a Benefactor must choose between providing direct (AI-driven) and indirect (e.g., cash) Benefits. The best approach will probably vary between organizations given differences in existing resources.

Organizations of all types often have to choose between various methods of meeting their ultimate goals (usu

...
Limits of Current US Prediction Markets (PredictIt Case Study)
by aphyer — 2020-07-14T07:24:23.421Z

(Disclaimers: I work in the financial industry, though not in a way related to prediction markets. Anything I write here is my opinion and not that of my employer.
This is a US-centric piece based on a case study of PredictIt: as some people have pointed out in the comments below, if you are outside the US you may have substantially better options.)

SECTION I: INTRODUCTION

So there's an argument that I've seen a lot over the past few years, particularly in LW-adjacent circles, that goes something like this:

You say you believe X is likely to happen. But prediction markets say X is likely not to happen. Since markets are efficient, you must be wrong. Or if you do know better than the market, why aren't you rich? Since you haven't bet on that market to make free money, you must be lying. Or stupid. Or both!

This post is dedicated to disagreeing with that argument, not from an anti-Efficient-Market Hypothesis position, but from a pro-Efficient-Market Hypothesis position. My position is:

The argument above is pretty much sound if we are discussing mainstream financial markets. If someone claims to have better information than a mainstream financial market on the value of Google stock, or of copper, they ought to either use this knowledge to make a huge amount of money or stop talking about it. However, it is not true if we are discussing prediction markets. Current prediction markets are so bad in so many different ways that it simply is not surprising for people to know better than them, and it often is not possible for people to make money from knowing better.

I've been meaning to write this for a while, but got tipped over the edge by the recent post here, which talks about the limitation of prediction markets being the correlation of the events they predict to other assets, and their consequent value as hedging instruments. That is...well...it's not wrong exactly, but there are so many other problems that are so much bigger that I felt it was worth laying (some of) them out.

Math follows. I will be focusing on PredictIt for this analysis. Other prediction markets may work a bit differently, but similar analysis is applicable to any of them. If you think the math is wrong I am happy to discuss/make changes, but I very much doubt any changes will materially alter the final message.

As of this writing PredictIt has ...