trentmkelly/LessWrong-43k huggingface dataset:
The map of "Levels of defence" in AI safety
by avturchin — 2017-12-12T10:45:29.430Z

One of the main principles of engineering safety is multilevel defence. When a nuclear bomb accidentally fell from the sky in the US, 3 of 4 defence levels failed. The last one prevented the nuclear explosion: https://en.wikipedia.org/wiki/1961_Goldsboro_B-52_crash

Multilevel defence is used a lot in the nuclear industry and includes different systems of passive and active safety, starting from the use of delayed neutrons for the reaction activation and up to control rods, containment building and exclusion zones.

Here, I present a look at the AI safety from the point of view of multilevel defence. This is mainly based on two of my yet unpublished articles: “Global and local solutions to AI safety” and “Catching treacherous turn: multilevel AI containment system”.

The special property of the multilevel defence, in the case of AI, is that the biggest defence comes from only the first level, which is AI alignment. Other levels have progressively smaller chances to provide any protection, as the power of self-improving AI will grow after it will break of each next level. So we may ignore all levels after AI alignment, but, oh Houston, we have a problem: based on the current speed of AI development, it seems that powerful and dangerous AI could appear within several years, but AI safety theory needs several decades to be created.

The map is intended to demonstrate a general classification principle of the defence levels in AI safety, but not to list all known ideas on the topic. I marked in “yellow” boxes, which are part of the plan of MIRI according to my understanding.

I also add my personal probability estimates as to whether each level will work (under the condition that AI risks are the only global risk, and previous levels have failed).

The principles of the construction of the map are similar to my “plan of x-risks prevention” map and my “immortality map”, which are also based around the idea of the multilevel defence.

pdf: https://goo.gl/XH3WgK
The map of "Levels of defence" in AI safety
by turchin — 2017-12-12T10:44:17.968Z

One of the main principles of engineering safety is multilevel defence. When a nuclear bomb accidentally fell from the sky in the US, 3 of 4 defence levels failed. The last one prevented the nuclear explosion: https://en.wikipedia.org/wiki/1961_Goldsboro_B-52_crash

Multilevel defence is used a lot in the nuclear industry and includes different systems of passive and active safety, starting from the use of delayed neutrons for the reaction activation and up to control rods, containment building and exclusion zones.

Here, I present a look at the AI safety from the point of view of multilevel defence. This is mainly based on two of my yet unpublished articles: “Global and local solutions to AI safety” and “Catching treacherous turn: multilevel AI containment system”.

The special property of the multilevel defence, in the case of AI, is that the biggest defence comes from only the first level, which is AI alignment. Other levels have progressively smaller chances to provide any protection, as the power of self-improving AI will grow after it will break of each next level. So we may ignore all levels after AI alignment, but, oh Houston, we have a problem: based on the current speed of AI development, it seems that powerful and dangerous AI could appear within several years, but AI safety theory needs several decades to be created.

The map is intended to demonstrate a general classification principle of the defence levels in AI safety, but not to list all known ideas on the topic. I marked in “yellow” boxes, which are part of the plan of MIRI according to my understanding.

I also add my personal probability estimates as to whether each level will work (under the condition that AI risks are the only global risk, and previous levels have failed).

The principles of the construction of the map are similar to my “plan of x-risks prevention” map and my “immortality map”, which are also based around the idea of the multilevel defence.

pdf: https://goo.gl/XH3WgK
LDL 9: When Transfer Learning Fails
by magfrump — 2017-12-12T00:22:48.316Z

One of the dominant paradigms in image processing today is transfer learning from well established image classification algorithms (such as DenseNet, VGG, or Inception) to other image processing applications. The idea is that in the process of learning to classify types of images, the networks learn to recognize distinctive visual features at different layers. These can start from lines in different directions and build up to colors, curves, concentric circles, corners, all the way to the classes used for the original applications like faces or dog species.

These lower level features, stored primarily in the earlier layers of the network, are in some sense “visually fundamental.” In the same way that computers display images by building them out of pixels we build ideas about images based on an encoded breakdown of the image into visual parts, which the network replicates.

For a wide variety of tasks, these features can be incredibly useful, giving good results even with limited data and small amounts of training time, since the visual features are already sufficient to organize the data. However in some cases transfer learning can fail, and even substantial train times after tuning in a number of ways can lead to only modest results.

What could be happening in these cases?

Our general theory goes:

low level features (like lines and curves) → mid level features (like corners and gradients) → high level features (like eyes or cups) → classification of images

We can have some confidence in the existence and reliability of chains like these because we can see the high levels of activations of neurons in the network corresponding to these features, and because we can observe the effectiveness in practice in many cases. So how can this framework fail to deliver?

It could be that the high level features needed to distinguish between the relevant classes aren’t available. For example, if your classifier wants to deduce whether a button seen by a button hole is buttoned or not, you might be able to detect buttons, but the model might have previously ignored whether the button is occluded, or may not have features for the nearby overlap of fabric or the pressure on the outer fabric that might indicate that the button is fastened. In this case the low level features are likely useful, and retraining more layers of the model or using features from an earlier layer of the mod...
Melting Gold, and Organizational Capacity
by Raemon — 2017-12-11T19:16:53.897Z

Epistemic Status: Confident about the problem, solution and exact mechanism is more of an untested hypothesis.

There's a saying for communities: if you're not gaining members, you're losing members. Sometimes you hit just the right size and you'd prefer things stay exactly as they are. But in practice, some people will eventually drift away - moving to a new town, getting a new job that consumes more time, or just losing interest in whatever-your-thing is.

So you need to have some source of new members, that supplies the life blood a community needs to keep going.

I think there's a similar truism:

If you are not building organizational capacity, you are burning organizational capacity.

(If the above sounds totally thoroughly reasonable and prompts an obvious course of action you can stop reading here, or skip towards the final section. Most of the rest of the post is for making this more salient)

Young organizations, communities and other groups tend to run on hero power - a couple people who care a lot who put in most of the work to keep it going. This is both because only a few people do care enough to put in that work, and because trying to distribute tasks is often harder than doing the tasks yourself.

As long those organizers are sticking around, everything seems fine. But the system is fragile.

I think people loosely understand the fragility. But I think it's better to think of it as a resource being slowly burned down rather than "a system is working fine until something changes."

Nothing Gold Can Stay - and Gold Melts Suddenly.

At the end of the Lord of the Rings movies, a golden ring is dropped into Mount Doom. When I first watched the film, I expected the gold to slowly melt as it slipped into the magma. Instead, it stayed solidly ring shaped... until it abruptly melted in seconds.

(This was the Ring of Power, approximately 3 seconds before it wasn't anymore)

I thought that looked unrealistic... but someone told me this is how actual gold melts: temperature rising until suddenly it reaches a critical state change. (No comment on magical gold you have to take to a magic volcano)

Any time you have a mission critical job that depends on a person being invested for it to get done, I think you have something similar going on. (i.e. a job where a person isn't getting paid enough for it to be something they sustainably...
Another Case Study of Inadequate Equilibrium in Medicine
by AABoyles — 2017-12-11T19:02:19.230Z

Science Journalist Julie Rehmeyer suffered from crippling Chronic Fatigue Syndrome. She's plugging a new book discussing her experiences with the not-medically-established treatment. The article in this link post discusses her decision to attempt the treatment, as well as first-order, meta-level suggestions about pursuing this type of independent research for other conditions. It's nothing groundbreaking (especially to anyone who read Inadequate Equilibria), but it's an interesting to see the same idea being pitched to a wider audience.

(h/t Andrew Gelman)
Thinking as the Crow Flies: Part 2 - Basic Logic via Precommitments
by Anthony Hart — 2017-12-11T18:23:04.309Z

Preamble

In my last post I gave the philosophical prerequisites needed to understand my approach to logic and mathematics. Before getting into the subject at hand, here's a clarifying remark inspired by some comments on my last post. When I say that mathematics is a social activity, I mean it's an activity that is social. Politics is also a social activity. This means that saying, for example, that mathematics is incomplete or inconsistent makes as much sense as saying that politics is incomplete or inconsistent (in the same way a formal logic might be). It's very common for someone to make mention of mathematics having some property (e.g. Gödelian incompleteness) when, in fact, it's a specific logic (such as ZFC or the totality of formal logic) which has this property. This odd synecdoche causes people to frequently confuse a part of mathematics with the whole. No one logic or collection of logics constitutes all of mathematics. The intuitions used to justify an axiom are part of mathematics, but not part of any axiom system.

With that out of the way, the point of this post is to give explicit meanings to the most basic logical connectives.

Logical Theories

A logical theory is a sort of collection of precommitments with two main kinds of judgments, P Prop (read "P is a proposition") and P True (read "P is true"). Within a logical theory, all but one of our precommitments will pertain to True. Note that any given precommitment will be about one particular thing, as previously explained. The propositional connectives we will initially deal with will be ⊥ (falsum), ⊤ (verum), ∧ (and), ∨ (or), and → (implies). Each will have a precommitment telling us precisely what's required for making a declaration of truth pertaining to that connective, and nowhere else will there be rules for making such declarations. Our one remaining precommitment will be the acting definition of Prop. Briefly, the precommitment for Prop says that P is a proposition when we've said what's required for declaring P True.

These precommitments will be variously referred to as "meaning explanations", descending from the usage of Martin-Löf, which explain the rules for reasoning about a judgment. For a judgment J, a meaning explanation should be a series of statements of the form:

We may declare J when ...

The meaning of ...
Will IOTA work as promized?
by ChristianKl — 2017-12-11T14:55:08.400Z

Currently, a lot of people made money by investing into Bitcoin and Ethereum. As a result the transaction fees for both are at the moment high enough to make most use-cases of the technology that existed two years ago impossible.

IOTA promizes to be a digital currency without transaction costs.

IOTA promizes to be able to deal without transaction costs by using a complicated concept called the Tangle. I have a hard time myself thinking through their framework. If the total transaction volumne is high enough that the whole transaction log spands multiple terabytes and nobody is incentivized to hold all that data to make transactions, does IOTA still work or will that make the system crash?

Is there anyone in this community who looked more into the concept and has a good idea of whether IOTA can till with terabytes (or even petabytes) worth in total transaction data?
Book Review: The Captured Economy
by Zvi — 2017-12-11T13:10:00.198Z

Epistemic Status: The choir

On Tyler Cohen’s claim that it was an important book, I read The Captured Economy by Brink Lindsey and Steven Teles. Its thesis is that regressive regulation is strangling our economy and increasing inequality. They claim that the damage from such policies is larger than we realize, and suggest structural reforms to start fixing the problem.

They focus on four issues: Financial regulation, zoning and land use restrictions, patent and copyright law, and occupational licensing.

I already strongly agreed on all four, although not on all the details. No reasonable person could, at this point, claim the regulations in question have not been subject to regulatory capture, and extended far beyond any worthwhile public interest.

This review advocates for reform of those policies. This is as political as I hope this blog ever gets. Politics remains the mindkiller. Down that road lies madness.

The book updated me on the scope of the damage, and on how to improve policy.

While I liked the book, I had three problems.

The first was the presumption the centrality of inequality, as opposed to deadweight loss and economic growth. I hate to reinforce the gauntlet that inequality is the thing to be concerned about.

The second was that it played somewhat loose with its arguments. It used the trick of comparing the ‘top X’ to the ‘bottom X’ things and then being shocked at how these two were not equal. It used the frame that calling intellectual property ‘property’ was a trick, all but calling all intellectual property theft. Their analysis of financial regulation suffered from lack of insider knowledge, and their case for zoning was enriched by assumptions that seem too strong.

The third was not addressing the legitimate cases for the policies the book opposed, or what the transition away from them would look like. Occupational licensing has gone way too far, but if you’re going to target lawyers and doctors (as you should, better to go after the real culprits and Play in Hard Mode) then you’ll need to explain why the alternative isn’t madness. Similar objections apply in other sections.

On the margin, the cases are ironclad. I knew I would know most of it already, but there was some new material even for me.

I recommend this book if and only if you want a book-length case for its thesis. Otherwise, t...
Eternal Winter, 
Endless Light
by Raemon — 2017-12-10T05:51:00.300Z

This was my Moment of Darkness speech for the NYC 2017 Solstice. The overall theme for the event was "generational knowledge."

To pass generational knowledge into the future, you need three things. You need mentors, sharing the most important wisdom they have to offer. You need children, curious and excited to learn, but willing to challenge that wisdom as times change.

Thirdly, you need there to be a future.

Earlier this year I moved to Berkeley, California. Soon afterwards I started hearing news about North Korea's nuclear's testing.

Every few years North Korea talks a big talk about their weapons program, but this year for the first time, we had seismological evidence that they were testing warheads in excess of 100 kilotons. And that they had intercontinental ballistic missiles that could strike the west coast. And San Francisco was one of the plausible targets.

A friend of mine did some calculations on blast radius. Depending on the exact size of the bomb, they estimated that as long as we stayed indoors, people in Berkeley would probably survive.

This wasn't especially reassuring.

Now, I have no idea how scared it's actually appropriate to be about this. In politics, and journalism, we're incentivized to stories that freak us out and exaggerate risks.

But one way or another, a lot of things have happened this year that have made it click together for me, that we might not make it. I've understood, intellectually, that humanity barely scrapped by through the cold war, that democracy is precarious. That we can't seem agree on any serious solutions to climate change. That biological terrorism or artificial intelligence might be even more challenging.

But this is the first year I've felt in my gut, that there are no competent grownups somewhere who know what they're doing and are going to make sure things are okay.

I've been thinking about that. And I've been thinking about a question that I often get after Solstice.

The question is "What's the deal with the stars?"

The first half of the Solstice is very concrete. We're talking and singing about real history, real people, real problems that they faced. We delve into messy questions...

...and then the second half of the Solstice doesn't answer those questions.

It can't. Because we don't know the answers. Yet.

And even i...
Success and Fail Rates of Monthly Policies
by lionhearted (Sebastian Marshall) — 2017-12-09T15:24:37.148Z

Inspired by Updates from Boston.

One of the more valuable interventions I tested in the last couple years is the idea of setting a "Monthly Policy" and a "Monthly Theme." I'll share the basic process briefly, and then share a couple counterintuitive lessons with implications for self-improvement.

Monthly Theming

I've gotten immense value out of naming my months an uncommon phrase, and trying to live in accordance with that theme for the entirety of one month. For reference, here's my monthly themes for each month of 2017 —

Structured Data
Do Less, Better
Sentry Ops II
Repertoire
Excellence
Impact Floor
Glorious Neutrality
Acts
Boring Methodical Execution
Make List, Run List
Trust the Process
Build Supply

So, in January, "Structured Data," I focused on building out a more complete dataset that I could run for the whole year — setting up systems for tracking every minute, tracking how long projects take to complete, upgrading personal finance, tracking fitness and macronutrient consumption, and a few things like that.

In February, "Do Less Better," I focused on having less projects open at the same time, doing less activities overall, and more carefully choosing activities for upside/gains while also focusing on spending more time to increase output quality of all the work I did choose to do.

Last month, November, was "Trust the Process" — October had been a month with lots of travel that had heavily disrupted my core habits, sleep schedules, etc. In November, I focused on doing nothing particularly expansive and instead focused on resetting and re-installing every best practice I know.

This month, I'm focused on both stockpiling all common physical supplies and improving procurement (food, household supplies, medicine, etc) as well as increasing my backlog of intellectual work (essays written and edited that are in backlog ready to publish, all the logistical work for events pre-done with reminder and confirmation messages scheduled, etc).

Monthly Theming is great, because it's almost free. I think you could set a theme without much reflection in just 15-30 minutes and get some benefit, but doing a full review takes me only 3-4 hours to do rigorously. I wind up making more correct decisions throughout the month, and some of the policies stick with me forever.

For instance, "Impact Floor" was abo...
Rules of variety
by KatjaGrace — 2017-12-08T17:10:00.300Z

At some point in high school I noticed an interesting thing about my choice of essay topics: it was definitely not allowed to be the same choice of topics that anyone else had.

One reason this seemed strange at the time was that I had never explicitly noticed this constraint or intended it, even though it was doing a lot of work. It was such a deeply assumed part of the basic rules of behavior that I didn’t know it was there.

But it seemed extra strange once I noticed it, because it happened in the context of the rest of my classmates as one blithely ignoring this absolute law of reasonable behavior and all writing about the same hackneyed thing. Which probably wouldn’t have even occurred to me to do if I had set out to write the most surprising essay I could. So, apparently other people didn’t even have this rule, though it seemed so inbuilt in me.

And this wasn’t just a failure to understand the rules of that assignment—I realized then that I had been assuming this constraint for every essay, and um, perhaps everything.

(In retrospect ‘maybe you are trying to be different and other people aren’t’ looks like an obvious explanation for the perplexing fact that I was different and other people weren’t. But I was used to knowing about things I was doing intentionally, and I was only trying to be different in the sense that I currently try not to murder people—it would be so wrong that it doesn’t cross my mind as a possibility. But while unconscious, this is a very effective form of intention.)

Years later, I think other people actually do have this rule, or similar rules. They just vary by topic, and I happened to be unusual on the topic of high school essays. But I think implicit constraints like this are actually pretty common, and usually feel too natural to be noticed, even while they entirely warp our behavior. I don’t mean ‘assumptions that we don’t notice’ in general, but in particular ones about how similar or dissimilar out behavior should be to others. I rarely hear these things spoken of, except to remark when they are broken, without comment on what they actually are or consideration as to whether they should be there.

Some examples of actions I think you would avoid to at least some extent, or make an excuse for:

Showing up in the same outfit as someone else

Naming your child

...