trentmkelly/LessWrong-43k huggingface dataset:
From Personal to Prison Gangs: Enforcing Prosocial Behavior
by johnswentworth — 2019-01-24T18:07:33.262Z

This post originally appeared here; I've updated it slightly and posted it here as a follow-up to this post.

David Friedman has a fascinating book on alternative legal systems. One chapter focuses on prison law - not the nominal rules, but the rules enforced by prisoners themselves.

The unofficial legal system of California prisoners is particularly interesting because it underwent a phase change sometime after the 1960’s.

Prior to the 1960’s, prisoners ran on a decentralized code of conduct - various unwritten rules roughly amounting to “mind your own business and don’t cheat anyone”. Prisoners who kept to the code were afforded some respect by their fellow inmates. Prisoners who violated the code were ostracized, making them fair game for the more predatory inmates. There was no formal enforcement; the code was essentially a reputation system.

Sometime after the 1960’s, that changed. During the code era, California’s total prison population was only about 5000, with about 1000 inmates in a typical prison. That’s quite a bit more than

Dunbar’s number

, but still low enough for a reputation system to work through second-order connections. By 1970, California’s prison population had ballooned past 25000; today it is over 170000. The number of prisons also grew, but not nearly as quickly as the population, and today’s prisoners frequently move across prisons anyway. In short, a decentralized reputation system became untenable. There were too many other inmates to keep track of.

As the reputation system collapsed, a new legal institution grew to fill the void: prison gangs. Under the gang system, each inmate is expected to affiliate with a gang (though most are not formal gang members). The gang will explain the rules, often in written form, and enforce them on their own affiliates. When conflict arises between affiliates of different gangs, the gang leaders negotiate settlement, with gang leaders enforcing punishments on their own affiliates. (Gang leaders are strongly motivated to avoid gang-level conflicts.) Rather than needing to track reputation of everyone individually, inmates need only pay attention to gangs at a group level.
Of course, inmates need some way to tell who is affiliated with each gang - thus the rise of racial segregation in prison. During the code era, prisoners te...
Is Agent Simulates Predictor a "fair" problem?
by Chris_Leong — 2019-01-24T13:18:13.745Z

It's a simple question, but I think it might help if I add in context. In the paper introducing Functional Decision Theory, it is noted that it is impossible to design an algorithm that can perform well on all decision problems since some of them can be specified to be blatantly unfair, ie. punish every agent that isn't an alphabetical decision theorist.

The question then arises, how do we define which problems are or are not fair? We start by noting that some people consider Newcomb's-like problems to be unfair since your outcome depends on a predictor's prediction, which is rooted in an analysis of your algorithm. So what makes this case any different from only rewarding the alphabetical decision theorist?

The paper answers that the prediction only depends on the decision you end up making and that any other internal details are ignored. So it only cares about your decision and not how you come to it, the problem seems fair. I'm inclined to agree with this reasoning, but a similar line of reasoning doesn't seem to hold with Agent Simulates Predictor. Here the algorithm you use is relevant as the predictor can only predict the agent if it's algorithm is less than a certain level of complexity, otherwise it may make a mistake.

Please note that this question isn't about whether this problem is worth considering; life is often unfair and we have to deal with it the best that we can. The question is about whether the problem is "fair", where I roughly understand "fair" meaning that this is in a certain class of problems that I can't specify at this moment (I suspect it would require its own seperate post) where we should be able to achieve the optimal result in each problem.
In what way has the generation after us "gone too far"?
by Elo — 2019-01-24T10:22:34.063Z

It's a pattern to watch an older generation consider a younger generation as, "having gone too far". From the inside it feels to me like I'm right and they are too conservative, but on the other side of the fence, believing for myself that the younger generation have gone too far? What do I find?
Please state the current age @ 2019 of the centre of gravity or range of ages of the generation you are commenting on.
The human side of interaction
by Rohin Shah — 2019-01-24T10:14:33.906Z

The last few posts have motivated an analysis of the human-AI system rather than an AI system in isolation. So far we’ve looked at the notion that the AI system should get feedback from the user and that it could use reward uncertainty for corrigibility. These are focused on the AI system, but what about the human? If we build a system that explicitly solicits feedback from the human, what do we have to say about the human policy, and how the human should provide feedback?

Interpreting human actions

One major free variable in any explicit interaction or feedback mechanism is what semantics the AI system should attach to the human feedback. The classic examples of AI risk are usually described in a way where this is the problem: when we provide a reward function that rewards paperclips, the AI system interprets it literally and maximizes paperclips, rather than interpreting it pragmatically as another human would.

(Aside: I suspect this was not the original point of the paperclip maximizer, but it has become a very popular retelling, so I’m using it anyway.)

Modeling this classic example as a human-AI system, we can see that the problem is that the human is offering a form of “feedback”, the reward function, and the AI system is not ascribing the correct semantics to it. The way it uses the reward function implies that the reward function encodes the optimal behavior of the AI system in all possible environments -- a moment’s thought is sufficient to see that this is not actually the case. There will definitely be many cases and environments that the human did not consider when designing the reward function, and we should not expect that the reward function incentivizes the right behavior in those cases.

So what can the AI system assume if the human provides it a reward function? Inverse Reward Design (IRD) offers one answer: the human is likely to provide a particular reward function if it leads to high true utility behavior in the training environment. So, in the boat race example, if we are given the reward “maximize score” on a training environment where this actually leads to winning the race, then “maximize score” and “win the race” are about equally likely reward functions, since they would both lead to the same behavior in the training environment. Once the AI system is deployed on the environment in the blog post, it would notice that the two likely rewar...
Allowing a formal proof system to self improve while avoiding Lobian obstacles.
by Donald Hobson — 2019-01-23T23:04:43.524Z

[Epistemic status, can't spot a mistake, but am not confidant that there isn't one, if you find anything please say so. Posting largely because the community value of a new good idea is larger than any harm that might be caused by a flawed proof. ]

Suppose you have an automatic proof checker. Its connected to a source of statements that are sometimes correct proofs, and sometimes not. The proof checker wants to reject all false proofs, and accept as many of the true proofs as possible. It also wants to be able to update its own proof framework.

Define S to be a set of statements in a particular formalism, say those that are grammatically well defined in PA. Let B be any sequence from some alphabet of symbols. Let

L={S×B→{⊤,⊥}}

and V⊂L be the set of programs that have the property that

∀v∈V:∀s∈S:(∃b∈B:v(s,b))⟹s

In other words, V is the set of all programs that never prove false statements. We should never leave V or need to talk about any program not in it.

For v∈V write v[s] to mean ∃b∈B:v(s,b). Ie v proves s

A simple setup would consist of a starting program p1∈V and a rule that says,

If p1[p2[s]⟹s] then you can add p2 to your list of trusted provers. If p1 proves the soundness of p2, then you can accept any statement when given a proof of it in p2.

The lobian obstacle is that p2 must be strictly weaker than p1, in that p1 can prove any statement that p2 can, but p1 can prove the soundness of p2 and p2 can't prove its own soundness. This means that each trusted prover has to be strictly weaker than the one that generated it. You could start with PA+3^^^3 and say that a few weakenings aren't a problem, but that isn't an elegant solution.

Note: You can't get around this problem using cycles. Suppose

a[b[s]⟹s]b[c[s]⟹s]

This would imply

a[b[c[s]⟹s]⟹(c[s]⟹s)]a[b[c[s]⟹s]]a[c[s]⟹s]

So any cycle could be shrunk by 1, and inductively, shrunk to a self trusting system.

I propose instead that you use the rule.

If p1[p2[s]⟹p1[s]] then accept any future proofs that are accepted by p2, and give p2 all rights given to p1, including taking p2[p3⟹p2] to mean that p3 is accepted, and so on recursively. If p1 is sound, then p1[p2[s]⟹p1[s]]⟹(p2[s]⟹p1[s]) so anything that is proven by p2 can be proven by p1. If p1 isn't sound, it can prove anything anyway.

Note that if p2 is a straightforward isomorphism of p1 then they are easily proven equivalent. H...
S-Curves for Trend Forecasting
by Matt Goldenberg — 2019-01-23T18:17:56.436Z

Epistemic Status: Innovation research and business research is notoriously low quality, and so all the ideas here should be viewed through that lense. What's impressive about the S-curve and evolution trends literature is how remarkably self-consistent it is using a wide variety of research methods. Whether Simon Wardley analyzing news article about different technologies, Clayton Christensen doing case studies of a specific industry, or Carlotta-Perez taking a historical approach of tracking different technologies, the same S-curve pattern and evolution trends seem to show up. This too should be taken into account when evaluating these ideas.

Basics

This is an S-curve.

The S-curve is a fundamental pattern that exists in many systems that have positive feedback loops and constraints. The curve speeds up due to the positive feedback loop, then slows down due to the constraints.

When the constraint is broken, the positive feedback loop ramps back up, until it hits another constraint.

Recommended Resource: Invisible Asymptotes, which gives a visceral feel for this process of positive feedback and constraints

Common Mistake: Confusing S-Curves With Exponential Growth

Sometimes, people get confused and call S-curves exponential growth. This isn't necessarily wrong but it can confuse their thinking. They forget that constraints exist and think that there will be exponential growth forever. When slowdowns happen, they think that it's the end of the growth - instead of considering that it may simply be another constraint and the start of another S-Curve. Knowledge of overlapping S-Curves can help you model these situations in a more sophisticated way.

Diffusion S-Curves

The S-curve pattern is quite common in the spread of ideas, practices, and technologies, although it rarely looks quite as pretty. The example below shows "diffusion s-curves" - How a technology spreads through a population (in this case US households

The positive feedback loop in this case is word of mouth, and the constraints represent fundamental barriers to certain market segments or growth such as simplicity, usability, scalability, price, etc.

This creates smaller s-curves around adoption among specific market segments, and larger s-curves that represent the overall market penetration of the idea, practice, or technology.

Recommended Resource: Wikipedia on Diffusion of Innovation

Evolution S-Curves

In additi...
One-step hypothetical preferences
by Stuart_Armstrong — 2019-01-23T15:14:52.063Z

Human preferences are time-inconsistent, and also contradictory.

That, by itself, is not a huge problem, but it's also the case that few human preferences are present at any given moment. At the moment, I'm focused on finding the best explanation to get my ideas through to you, the reader; I'm not focused on my moral preferences, personal safety desires, political beliefs, or taste in music.

If anyone asked me about those, I could immediately bring them to mind. My answers to standard questions are kinda in the background, accessible but not accessed. Wei Dai made a similar point about translators: they have a lot of trained knowledge that is not immediately accessible to their introspection. And only by giving them the inputs they were trained on (eg words, sentences,...) can you bring that knowledge to the fore.

In this post, I'll try and formalise these accessible preferences, starting with formalising preferences in general.

Basic preferences setup

This section will formalise the setup presented in Alice's example. Let W be a set of all possible worlds. A human makes use of a model M. This model contains a lot of variables {Pi}, called properties. These Pi take values in a domain Di.

A basic set S of states in M is a set of possible values for some of the Pi. Thus S={Si}, with Si⊂Di. The property Pi unconstrained in S if Si=Di. A general set of states is a union of basic S; let S be these of all these sets of states.

For example, a human could be imagining four of their friends, and the Pij could be whether friend i is sleeping with friend j (6 different Boolean Pij=Pji), and also whether a third friend k believes two others are sleeping together (12 different Pijk=Pjik, taking values in {sleeping together, not sleeping together, don't know}).

Then a statement of human gossip like ''X is sleeping with Y, but A doesn't realise it; in fact, A thinks that Y is sleeping with Z, which is totally not true!" is encoded as:

SG={PXY=1, PYZ=0, PXYA⊂{"don't know","not sleeping together"}, PYZA="sleeping together"}, with the other Ps unconstrained.

It's interesting how unintuitive that formulation is, compared with how our brains instinctively parse gossip.

To make use of these, these symbols need to be grounded. This is achieved via a function g that takes a set of states S and maps it to a set of worlds: g(S)⊂W.

Finally, the human expresses a judgement about the states

...
Could declining interest to the Doomsday Argument explain the Doomsday Argument?
by avturchin — 2019-01-23T11:51:57.012Z

Epistemic status: this post is a history of my surprise

Hypothesis

Thinking about Doomsday Argument (DA), I come to the following hypothesis:

Hypothesis 1: “I am randomly selected from all people, who know about DA. The number of such people is growing exponentially, from the 1980s until now and will continue to grow. Thus, I am currently located only 1-2 doublings before the end of this class of people, and such “end” could be best explained by a global catastrophe.”

I expected that the doubling time of the number of people who know about DA is around 5-10 years, and thus the end will happen around 2030, which is in accordance with my estimations of dangerous AI timing and some other predictions.

Data

To check the hypothesis, I went on Google Trends to check the number of times the words “Doomsday argument” is searched. What I found surprised me: the number of searchers is actually declining:

The data is noisy, but it looks like the number of searchers declines from the average of 16 a month in 2008 to 7 in 2017.

Wikipedia views data even from 2015 (no early data available) also shows a decline around 2 times between 2015 and 2018.

Google Scholar analysis is less clear (obviously not exponential growth of the number of articles, but steady growth of mentions which means more scientists know DA):

Google Scholar articles about DA, in the period of

1989-1994: 14 articles (hand counted), 30 mentions of DA.

1995-2000: 15 articles, 50 mentions

2001-2006: 24 articles, 100 mentions

2007-2012: 18 articles, 140 mentions

2013-2018: 20 articles, 160 mentions

It shows that the peak of interest to DA by scientists was around 2000, which should not be surprising, as at the time the idea was relatively new. The growth of mentions could be explained by large “historical introductions” in other articles. However, the number of DA-related articles is now growing again.

Analysis

When I first got the data of the declining interest to DA, I suggest that this could explain the DA:

Hypothesis 2: If there will be no more scientists who are interested in DA, the reference class of those who know about DA will end without end of the world.

However, a closer examination of the Google Scholar data doesn’t support this second hypothesis either: there is a steady influx of new scientists who try to refute or reanalyze the DA. Moreover, the growth of “mentions” shows that the number of scientists w...
Link: That Time a Guy Tried to Build a Utopia for Mice and it all Went to Hell
by Alexei — 2019-01-23T06:27:05.219Z

Video: https://www.youtube.com/watch?v=5m7X-1V9nOs

Text version: http://www.todayifoundout.com/index.php/2018/12/that-time-a-guy-tried-to-build-a-utopia-for-mice-and-it-all-went-to-hell/

"In 1968, an expert on animal behaviour and population control called John B. Calhoun built what was essentially a utopia for mice that was purpose built to satisfy their every need. Despite going out of his way to ensure the inhabitants of his perfect mouse society never wanted for anything, within 2 years virtually the entire population was dead. So what happened?"
Learning with catastrophes
by paulfchristiano — 2019-01-23T03:01:26.397Z

A catastrophe is an event so bad that we are not willing to let it happen even a single time. For example, we would be unhappy if our self-driving car ever accelerates to 65 mph in a residential area and hits a pedestrian.

Catastrophes present a theoretical challenge for traditional machine learning — typically there is no way to reliably avoid catastrophic behavior without strong statistical assumptions.

In this post, I’ll lay out a very general model for catastrophes in which they are avoidable under much weaker statistical assumptions. I think this framework applies to the most important kinds of catastrophe, and will be especially relevant to AI alignment.

Designing practical algorithms that work in this model is an open problem. In a subsequent post I describe what I currently see as the most promising angles of attack.

Modeling catastrophes

We consider an agent A interacting with the environment over a sequence of episodes. Each episode produces a transcript τ, consisting of the agent’s observations and actions, along with a reward r ∈ [0, 1]. Our primary goal is to quickly learn an agent which receives high reward. (Supervised learning is the special case where each transcripts consist of a single input and a label for that input.)

While training, we assume that we have an oracle which can determine whether a transcript τ is “catastrophic.” For example, we might show a transcript to a QA analyst and ask them if it looks catastrophic. This oracle can be applied to arbitrary sequences of observations and actions, including those that don’t arise from an actual episode. So training can begin before the very first interaction with nature, using only calls to the oracle.

Intuitively, a transcript should only be marked catastrophic if it satisfies two conditions:

The agent made a catastrophically bad decision.
The agent’s observations are plausible: we have a right to expect the agent to be able to handle those observations.

While actually interacting with the environment, the agent cannot query the oracle — there is no time to wait for a QA engineer to review a proposed action to check if it would be catastrophic.

Moreover, if interaction with nature ever produces a catastrophic transcript, we immediately fail. The performance of an algorithm is characterized by two parameters: the probability of catastrophic failure, and the total reward assuming no catastrophic fai...
The Relationship Between Hierarchy and Wealth
by sarahconstantin — 2019-01-23T02:00:00.467Z

Epistemic Status: Tentative

I’m fairly anti-hierarchical, as things go, but the big challenge to all anti-hierarchical ideologies is “how feasible is this in real life? We don’t see many examples around us of this working well.”

Backing up, for a second, what do we mean by a hierarchy?

I take it to mean a very simple thing: hierarchies are systems of social organization where some people tell others what to do, and the subordinates are forced to obey the superiors. This usually goes along with special privileges or luxuries that are only available to the superiors. For instance, patriarchy is a hierarchy in which wives and children must obey fathers, and male heads of families get special privileges.

Hierarchy is a matter of degree, of course. Power can vary in the severity of its enforcement penalties (a government can jail you or execute you, an employer can fire you, a religion can excommunicate you, the popular kids in a high school can bully or ostracize you), in its extent (a totalitarian government claims authority over more aspects of your life than a liberal one), or its scale (an emperor rules over more people than a clan chieftain.)

Power distance is a concept from the business world that attempts to measure the level of hierarchy within an organization or culture. Power distance is measured by polling less-powerful individuals on how much they “accept and expect that power is distributed unequally”. In low power distance cultures, there’s more of an “open door” policy, subordinates can talk freely with managers, and there are few formal symbols of status differentiating managers from subordinates. In “high power distance” cultures, there’s more formality, and subordinates are expected to be more deferential. According to Geert Hofstede, the inventor of the power distance index (PDI), Israel and the Nordic countries have the lowest power distance index in the world, while Arab, Southeast Asian, and Latin American countries have the highest. (The US is in the middle.)

I share with many other people a rough intuition that hierarchy poses problems.

This may not be as obvious as it sounds. In high power distance cultures, empirically, subordinates accept and approve of hierarchy. So maybe hierarchy is just fine, even for the “losers” at the bottom? But there’s a theory that subordinates claim to approve of hierarchy as a covert way of getting what ...