trentmkelly/LessWrong-43k huggingface dataset:
Predicted AI alignment event/meeting calendar
by rmoehn — 2019-08-14T07:14:57.233Z

Update 2020-06-21: Linda Linsefors made the AI Safety Google Calendar. I published How to make a predicted AI alignment event/meeting calendar, in case you want to make your own.

Update 2021-11-21: The AI Safety Google Calendar is not in a state (anymore?) where it would be a replacement for the kind of calendar I maintained.

I kept this calendar up-to-date between August 2019 and April 2020. As I've left AI alignment (see also I'm leaving AI alignment – you better stay) and nobody has told me that they want me to keep this calendar up-to-date, I stop keeping it up-to-date.

If you find this calendar useful and would like it to continue to be kept
up-to-date, please tell me in the comments or via PM. If you want to contribute to the community by maintaining this calendar, let me know and I'll share my process with you.

I update this every month. If you know of more events, please comment, PM
or email me. The same goes for events that I listed, but which won't actually
take place.

See also (I won't repeat here what is listed there):

MIRI's AI Risk for Computer Scientists
workshops

Last updated (search for ‘UPDATED’): 2020-04-18

Next update will be published by: 2020-05-18 – If someone tells me that they
find this calendar useful. See also the paragraph at the top.

2020

Many of the events went virtual because of COVID-19.

(Apparently no SafeML workshop at ICLR 2020.)

April, cyberspace workshop Towards Trustworthy ML: Rethinking Security
and Privacy for ML at ICLR 2020 (only
somewhat related to AIA)

April proposal submission deadline for the AISafety
workshop at IJCAI

May, cyberspace research retreat of the AI Safety Camp
Toronto – The in-person event
is cancelled. The organizers are doing their best putting together a virtual
camp.

UPDATED May paper submission deadline for the Third International Workshop on
Artificial Intelligence
Safety Engineering

UPDATED May, cyberspace Web Technical AI Safety
Unconference

May, cyberspace Workshop on Assured Autonomous
Systems at the 41st IEEE
Symposium on Security and Privacy

June application deadline for the Human-aligned AI Summer
School (my guess based on 2019)

July/August, Prague CZE Human-aligned AI Summer
School (my guess based on 2018, 2019)

July, Yokohama JPN AISafety workshop at
IJCAI 2020

August, Bodega Bay USA MIRI Summer Fellows
Program
(my guess based on 2019 – There were also MSFPs/AISFPs in 2015-2018.)

September, Lisbon P...
Natural laws should be explicit constraints on strategy space
by ryan_b — 2019-08-13T20:22:47.933Z

Mostly strategic developments have been about incrementing beyond whatever the other person is doing. Sometimes there are paradigm shifts, which largely mean a different dimension along which to make incremental improvements.

But we cannot increment forever. Sometimes there is a well-understood limit we cannot surpass. Energy-Maneuverability theory is a paradigm for designing air superiority fighters. Though the paradigm shifted from the old speed/altitude/turn metrics, we remained constrained by the ability of the human pilot to withstand g forces. We have already built aircraft which can climb higher, accelerate faster, and turn more sharply than men can tolerate without passing out. It may even be possible to design an almost-perfect manned fighter, in which the pilot is the constraint in all dimensions of performance.

But now we have unmanned drones.

Natural law provides a variety of limits, like the speed of light or the increase of entropy. We have a good command of natural law at the scale where warmachines operate. It seems like it would be a good policy to adopt these as the constraints on strategy-space, and map what we know about our opponents to them. This would have the benefit of letting us know how much room there even is for incremental improvements, and give us some indication of where we are vulnerable to (or have an opportunity to create) a paradigm shift.

Since most of these natural limits are well known, and most dimensions of strategy don't have something obvious like c, there isn't an obvious motivation for it. But it feels to me like even something as conceptually straightforward as using operations research or mathematical programming would work for this.
Distance Functions are Hard
by Grue_Slinky — 2019-08-13T17:33:15.296Z

[Epistemic status: Describes a failed research approach I had a while ago, and my only purpose here is to warn people off from that way of thinking. Every now and then I see someone working on an AIS subproblem say "if only we had a distance function for things in domain X", and my intuition is that they are probably doing a wrong-way reduction. But I only mean this as a soft guideline, and I'm only somewhat confident in my current thinking on this.]

~~~

Terminology: We use the terms distance or distance function to denote any function d:X×X→R≥0 that intuitively tells us how “dissimilar” any two members of a set X are (regardless of the whether d is a metric).

Counterfactual Worlds

Consider the counterfactual "If Lincoln were not assassinated, he would not have been impeached". If we would like to say this has a truth value, we need to imagine what such a counterfactual world would have looked like: was it because Lincoln (somehow) survived his wounds, John Wilkes Booth (somehow) missed, that the plot was (somehow) discovered the day before, etc. Somehow, we must pick out the world that is in some sense "closest" to our actual world, but it seems very difficult to compare any two such worlds in a principled way.

To formalize Functional Decision Theory (FDT), we likely need to have a better understanding of counterfactuals, although even in restricted mathematical contexts, we don't have a satisfactory understanding of why "If 0 = 1..." simply returns incoherence, yet "If the Modularity Theorem were false..." seemingly conjures up a possible world that we feel we can reason about.

(Also, in terms of corrigibility, we are often interested in formalizing the notion of "low-impact" agents, and the naive idea one often has is to define a distance metric on counterfactual world-states, as in p. 5 of Concrete Problems in AI Safety).

Algorithmic Similarity

In the FDT framework, we do not view ourselves as a solitary agent, but as a function (or algorithm) that can be copied, modified, and read, and we wish to maximize the utility achieved by our algorithm. Minor details of our implementation that don't affect our behavior (such as whether we are written in Java or Python) should not be decision-relevant, and if some algorithm does the same thing as us "most" of the time, then we would proba...
Book Review: Secular Cycles
by Scott Alexander — 2019-08-13T04:10:01.201Z

I.

There is a tide in the affairs of men. It cycles with a period of about three hundred years. During its flood, farms and businesses prosper, and great empires enjoy golden ages. During its ebb, war and famine stalk the land, and states collapse into barbarism.

Chinese population over time

At least this is the thesis of Peter Turchin and Sergey Nefedov, authors of Secular Cycles. They start off Malthusian: due to natural reproduction, population will keep increasing until it reaches the limits of what the land can support. At that point, everyone will be stuck at subsistence level. If any group ever enjoys a standard of living above subsistence level, they will keep reproducing until they are back down at subsistence.

Standard Malthusian theory evokes images of a population stable at subsistence level forever. But Turchin and Nefedov argues this isn’t how it works. A population at subsistence will always be one meal away from starving. When a famine hits, many of them will starve. When a plague hits, they will already be too sickly to fight it off. When conflict arrives, they will be desperate enough to enlist in the armies of whichever warlord can offer them a warm meal.

These are not piecemeal events, picking off just enough of the population to bring it back to subsistence. They are great cataclysms. The Black Plague killed 30% – 60% of Europeans; the Antonine Plague of Rome was almost as deadly. The Thirty Years War killed 25% – 40% of Germans; the Time of Troubles may have killed half the population of medieval Russia.

Thus the secular cycle. When population is low, everyone has more than enough land. People grow rich and reproduce. As time goes on, the same amount of farmland gets split among more and more people. Wages are driven down to subsistence. War, Famine, and Pestilence ravage the land, with Death not far behind. The killings continue until population is low again, at which point the cycle starts over.

This applies mostly to peasants, who are most at risk of starving. But nobles go through a related process. As a cycle begins, their numbers are low. As time goes on, their population expands, both through natural reproduction and through upward mobility. Eventually, there are more nobles than there are good positions…

(this part confused me a little. Shouldn’t number of good positions scale with population? IE if one baron rules 1,000 peasants, the...
A Primer on Matrix Calculus, Part 1: Basic review
by Matthew Barnett — 2019-08-12T23:44:37.068Z

Consider whether this story applies to you. You went through college and made it past linear algebra and multivariable calculus, and then began your training for deep learning. To your surprise, much of what they taught you in the previous courses is not very useful to the current subject matter.

And this is fine. Mathematics is useful in its own right. You can expect a lot of stuff isn't going to show up on the deep learning final, but it's also quite useful for understanding higher mathematics.

However, what isn't fine is that a lot of important stuff that you do need to know was omitted. In particular, the deep learning course requires you to know matrix calculus, a specialized form of writing multivariable calculus (mostly differential calculus). So now you slog through the notation, getting confused, and only learning as much as you need to know in order to do the backpropagation on the final exam.

This is not how things should work!

Matrix calculus can be beautiful in its own right. I'm here to find the beauty for myself. If I may find it beautiful, then perhaps I will find new joy in reading those machine learning papers. And maybe you will too.

Therefore, I dedicate the next few posts in this sequence to covering this paper, which boldly states that it is, "an attempt to explain all the matrix calculus you need in order to understand the training of deep neural networks." All the matrix calculus we need? Perhaps it's enough to understand training neural networks, but it isn't enough matrix calculus for deep learning more generally — I just Ctrl-F'd and found no instance of "hessian"! 1

Since it's clearly not the full picture, I will supplement my posts with material from chapter 4 of the Deep Learning Book, and Wikipedia.

This subsequence of my daily insights sequence will contain three parts. The first part is this post, the introduction. For the posts in the sequence I have outlined the following rubric:

Part 1 (this one) will be reviewing some multivariable calculus and will introduce the matrix calculus notation.

Part 2 will cover Jacobians, derivatives of element-wise binary operators, derivatives involving scalar expansions, vector sum reduction, and some common derivatives encountered in deep learning.

Part 3 will cover the hessian matrix, higher order derivatives and Taylor approximations, and we will step through an example of applying the chain ru... ,
What explanatory power does Kahneman's System 2 possess?
by Richard_Ngo — 2019-08-12T15:23:20.197Z

In the 70s and 80s, Kahneman and Tversky did a bunch of pioneering research on heuristics and biases in human thought. Then, in Thinking Fast and Slow, Kahneman divided human cognition into System 1 and System 2 - basically, System 1 applies quick heuristics which are prone to biases, and System 2 does the slow, effortful thinking.

But what does System 2 actually add to the theory in terms of explanatory power? Consider an alternative version of Thinking Fast and Slow in which Kahneman wrote something like "Here are the conditions in which humans use this mode of reasoning I'm calling System 1, which is fast and approximate and effortless and uses heuristics and demonstrates biases which can be detected in certain ways. The rest of the time, I have no idea what's going on, except that it doesn't display the traits that would qualify it as System 1 inference." In what ways would this be less informative than his actual claims?
Mesa-Optimizers and Over-optimization Failure (Optimizing and Goodhart Effects, Clarifying Thoughts - Part 4)
by Davidmanheim — 2019-08-12T08:07:01.769Z

In the previous posts, I first outlined Selection versus Control for Optimization, then talked about What Optimization Means, and how we quantify it, then applied these ideas a bit to ground the discussion.

After doing so, I reached a point where I think that there is something useful to say about mesa-optimizers. This isn't yet completely clear to me, and it seems likely that at some point in the (hopefully near) future, someone will build a much clearer conceptual understanding, and I'll replace this with a link to that discussion. Until then, I want to talk about how mesa-optimizers are control systems built within selection systems, and why that poses significant additional concern for alignment.

Mesa-Optimizers

I claimed in the previous post that Mesa-optimizers are always control systems. The base optimization selects for a mesa-optimizer, usually via side-effect-free or low-cost sampling and/or simulation, then creates a system that does further optimization as an output. In some cases, the further optimization is direct optimization in the terms laid out in my earlier post, but in others it is control.

Direct Mesa-Optimizers

In many cases, the selection system finds optimal parameters or something similar for a direct optimization system. This is exactly the earlier example of building a guidance system for a rocket is an obvious class of example where selection leads to a direct optimizer. This isn't the only way that optimizers can interact, though.

I'd say that MCTS + Deep learning is an important example of this mix which has been compared to "Thinking Fast and Slow" (pdf, NIPS paper). In chess, for example, the thinking fast is the heuristic search to choose where to explore, which is based on a selection system, and the exploration is MCTS, which in this context I'm calling direct optimization. (This is despite the fact that it's obviously probabilistic, so in some respects looks like selection, because while the selection IS choosing points in the space, it isn't doing evaluation, but rather deterministic play-forward of alternative scenarios. The evaluation is being done by the heuristic evaluation system.) In that specific scenario, any misalignment is almost entirely a selection system issue - unless there was some actual mistake in the implementation of the rules of chess.

This opens up a significant concern for causal Goodhart; regime change w...
Adjectives from the Future: The Dangers of Result-based Descriptions
by Pradeep_Kumar — 2019-08-11T19:19:19.139Z

Less Skeptical

Suppose your friend tells you he's on a weight-loss program. What do you think will happen in three months if he keeps on the weight-loss program? Will he lose weight?

If you're like me, you're thinking, "Of course. He is on a weight-loss program, isn't he? So, ipso facto, he is likely to lose weight."

Does there seem to be anything fishy about that chain of reasoning?

We usually describe the current features of a thing and predict something about the future. For example, we might say "I'm running for half an hour each day" and predict that we will lose a certain number of pounds by the end of the month. But your friend above skipped the description and talked about the prediction as if it were visible right now: "I'm on a weight-loss program".

You weren't told the features of the activity (running for half an hour) or even a name (CrossFit program). If you had been told either, you could have judged it based on your past knowledge of those features or names. Running regularly does help you lose weight and so does CrossFit. But, here, you were told just the prediction itself. This means you can't predict anything for sure. If his program involves running, he will lose weight; if it involves eating large cheese pizzas, he won't. You don't know which it is.

Yet, it sounded quite convincing! Even if you objected by saying that your friend probably won't stick to the exercise regimen, you probably bought into the premise, like me, that the program was a weight-loss program.

Hypothesis: If you are given an adjective that describes a future event and are not given any currently-visible features, then you're more likely to accept that that future event will occur than when you can see some features.

In other words, result-based descriptions make you less skeptical.

A more serious example is when someone mentions a drug-prevention program. We might assume that it will prevent illegal drugs from being bought and sold. After all, it must have been designed for that purpose. But the result depends on what the program actually does. Running ads saying "Don't do drugs!" may not achieve much, whereas inspecting trucks at border checkpoints may. To judge whether the program will be successful, you have to inspect its actual features. But "drug-prevention program" sounded convincing, right? Notice how the adjective "drug-prevention" describes a future event - it says...
Could we solve this email mess if we all moved to paid emails? 
by Bird Concept — 2019-08-11T16:31:10.698Z

Have you ever…

Sent an email to someone in rationality and not heard back for many weeks (or more)?
Avoided sending an email to someone because you wanted to spare their attention, despite thinking there was a fair chance they’d be genuinely interested?
Wanted some way to signal that you actually cared more than usual about this email, but without having to burn social capital (such as by saying “urgent” or “please read”)?
Had to ignore an email because, even though it might have been interesting, figuring that out would simply have been too effortful?

I think that 1) problems like these are prevalent, 2) they have pretty bad consequences, and 3) they could be partly solved by using services where you can pay to send someone an email (N.B. payment is conditional on reply).

I’m considering running a coordination campaign to move the community to using paid emails (in addition to their ordinary inbox), but before launching that unilaterally I want more confidence it is a good idea.

It would be very helpful data if people who'd use this is if >=50 other people also did would post just saying "I'd use this is >=50 particular other people did".

Background

Email seems broken

. This is not that surprising: your email is basically a to-do list where other people (and companies) can add items for free, without asking; and where you’re the only one who can remove them. We should do something about this.

More broadly,

the attention economy seems broken

. Recognising this, many rationalists

use various software tools to protect themselves from apps that are engineered to be addictive

. This helps at an individual level, but it doesn’t help solve the collective action problem of how to allocate our attention as a community. We should do something about this.

Costly signalling and avoiding information asymmetries

An “information asymmetry” is situation where someone has true information which they are unable to communicate. For example, suppose 10 economists are trying to influence government policy on issue X, and one of them actually, really knows what the most effective thing is. Yet, they might not be able to communicate this to the decision-makers, since the remaining 9 have degrees from equally prestigious institutions and arguments that sound equally rigorous to someone without formal training in economics.

Information asymmetries are a key mechanis

...
AI Safety Reading Group
by Søren Elverlin — 2019-08-11T09:01:20.433Z

If you are interested in AI Safety, come visit the AI Safety Reading Group.
The AI Safety reading group meets on Skype Wednesdays at 18:45 UTC, discussing new and old articles on different aspects of AI Safety. We start with a presentation round, then a summary of the article is presented, followed by discussion both on the article and in general.
Sometimes we have guests. On Wednesday the 14th, Stuart Armstrong will be giving a presentation on his research agenda in the reading group:
https://www.alignmentforum.org/posts/CSEdLLEkap2pubjof/research-agenda-v0-9-synthesising-a-human-s-preferences-into

Join us by Skype, by adding ‘soeren.elverlin’.
Previous guests include Eric Drexler, Rohin Shah, Matthijs Maas, Scott Garrabrant, Robin Hanson, Roman Yampolskiy, Vadim Kosoy, Abram Demski and Paul Christiano. A full list of articles read can be found at https://aisafety.com/reading-group/
Does human choice have to be transitive in order to be rational/consistent?
by jmh — 2019-08-11T01:49:23.967Z

I was struck by that question reading one of the responses to the post polling the merits of several AI alignment research ideas.

I have not really thought this through but it seems the requirement for preference ordering satisfying a transitivity requirement must also assume the alternatives being ranked can be distilled to some common denominator (economics would probably suggest utility per unit or more accurately MU/$).

I'm not sure that really covers all, and perhaps not even the majority of cases.

It we're really comparing different sets of attributes we label A, B and C transitive preferences might well the the exception rather than the rule.

The A>B, B>C therefore A>C is often violated -- in political science that produces a voting cycle -- when considering group choices.

I just wonder if it really is correct to claim such results within one person's head, given we're comparing different things -- and so likely the use/consumption in a slightly different context as well.

Could that internal voting cycle be a source of indecision (which is a bit different that indifference) and why we will often avoid a pair-wise decision process and opt for putting all the alternatives up against the others to pick the preferred alternative?

If so would that be something that an AGI will also find naturally occurs and it is not an error to be corrected but rather a situation where applying a pair-wise choice or some transitivity check would actually be the error.