trentmkelly/LessWrong-43k huggingface dataset:
(A -> B) -> A in Causal DAGs
by johnswentworth — 2020-01-22T18:22:28.791Z

Agenty things have the type signature (A -> B) -> A

. In English: agenty things have some model (A -> B) which predicts the results (B) of their own actions (A). They use that model to decide what actions to perform: (A -> B) -> A.

In the context of causal DAGs, the model (A -> B) would itself be a causal DAG model M - i.e. some Python code defining the DAG. Logically,

we can represent it

as:

M=“(P[A|M]=fA(A))&(P[B|A,M]=fB(B,A))”

… for some given distribution functions fA and fB.

From an outside view, the model (A -> B) causes the choice of action A. Diagrammatically, that looks something like this:

The “cloud” in this diagram has a precise meaning: it’s the model M for the DAG inside the cloud.

Note that this model does not contain any true loops - there is no loop of arrows. There’s just the

Hofstaderian

“strange loop”, in which node A depends on the model of later nodes, rather than on the later nodes themselves.

How would we explicitly write this model as a Bayes net?

The usual way of writing a Bayes net is something like:

P[X]=∏iP[Xi|Xpa(i)]

… but as discussed in the

previous post

, there’s really an implicit model M in there. Writing everything out in full, a typical Bayes net would be:

P[X|M]=∏iP[Xi|Xpa(i),M]

… with M=“∀i:P[Xi|Xpa(i),M]=fi(Xi,Xpa(i))”.

Now for the interesting part: what happens if one of the nodes is agenty, i.e. it performs some computation directly on the model? Well, calling the agenty node A, that would just be a term P[A|M]... which looks exactly like a plain old root node. The model M is implicitly an input to all nodes anyway, since it determines what computation each node performs. But surely our strange loop is not the same as the simple model A -> B? What are we missing? How does the agenty node use M differently from other nodes?

What predictions would (A -> B) -> A make which differ from A -> B?

Answer: interventions/counterfactuals.

Modifying M

If A is determined by a computation on the model M, then M is causally upstream of A. That means that, if we change M - e.g. by an intervention M←do(B=2,M) - then A should change accordingly.

Let’s look at a concrete example.

We’ll stick with our (A -> B) -> A system. Let’s say that A is an investment - our agent can invest anywhere from $0 to $1. B is the payout of the i...
[AN #83]: Sample-efficient deep learning with ReMixMatch
by Rohin Shah — 2020-01-22T18:10:01.483Z

Find all Alignment Newsletter resources

here

. In particular, you can

sign up

, or look through this

spreadsheet

of all summaries that have ever been in the newsletter. I'm always happy to hear feedback; you can send it to me by replying to this email.

Audio version

here

(may not be up yet).

Highlights

ReMixMatch: Semi-Supervised Learning with Distribution Alignment and Augmentation Anchoring

(David Berthelot et al) (summarized by Dan H): A common criticism of deep learning is that it requires far too much training data. Some view this as a fundamental flaw that suggests we need a new approach. However, considerable data efficiency is possible with a new technique called ReMixMatch. ReMixMatch on CIFAR-10 obtains 84.92% accuracy using only 4 labeled examples per class. Using 250 labeled examples, or around 25 labeled examples per class, a ReMixMatch model on CIFAR-10 has 93.73% accuracy. This is approximately how well a vanilla ResNet does on CIFAR-10 with 50000 labeled examples. Two years ago, special techniques utilizing 250 CIFAR-10 labeled examples could enable an accuracy of approximately

53%

. ReMixMatch builds on

MixMatch

and has several seemingly arbitrary design decisions, so I will refrain from describing its design. In short, deep networks do not necessarily require large labeled datasets.

And just yesterday, after this summary was first written, the

FixMatch

paper got even better results.

Previous newsletters

In last week's email, two of Flo's opinions were somehow scrambled together. See below for what they were supposed to be.

Defining and Unpacking Transformative AI

(Ross Gruetzemacher et al) (summarized by Flo): Focusing on the impacts on society instead of specific features of AI systems makes sense and I do believe that the shape of RTAI as well as the risks it poses will depend on the way we handle TAI at various levels. More precise terminology can also help to prevent misunderstandings, for example between people forecasting AI and decision makers.

When Goodharting is optimal: linear vs diminishing returns, unlikely vs likely, and other factors

(Stuart Armstrong) (summarized by Flo): I enjoyed this article and the proposed factors match my intuitions. There seem to be two types of problems: extreme beliefs and concave Pareto boundaries. Dealing with the second is more important since a concave Pareto boundary favours extreme policies...
Terms & literature for purposely lossy communication
by ozziegooen — 2020-01-22T10:35:47.162Z

Say I have a 10 of coins that could either be red or blue, and heads or tails. This has 20 bits of information.

My friend is interested in knowing about which of them are heads or tails, but doesn’t care about the color. I then decide to only tell my friend the heads/tails information; so, 10 bits of information.

Another example:
With image compression, there's a big difference between "Accurately reconstruct this image, pixel by pixel" vs. "Accurately reconstruct what the viewer would remember, which is basically just 'Lion on grass'".

I’d feel uncomfortable calling this translation “compression”, because there was definitely intentional information loss. Most of the literature on compression is about optimally maintaining information, not on optimally loosing it.

Are there other good terms or literature for this?
Three signs you may be suffering from imposter syndrome
by lc — 2020-01-21T22:17:45.944Z

You do not have imposter syndrome, because imposter syndrome does not exist.

In most professional domains, no one knows how competent they are, exactly. Unless you’re in a field where there is an agreed upon, systematic method of ranking success, like sports, you may only glean your competence through easy-to-wilfully-misinterpret signals. An uncomfortable amount of uncertainty on your value as an engineer or scientist is probably completely valid, especially if what you’re doing is trying to compare yourself against the entire population of professionals instead of the ones you can see. Just because you have a lot mentally riding on your keyboard incantation ability doesn’t mean you should have some definitive intuition for where it is, even after years of experience.

That said, I find it insightful that whenever someone says they have imposter syndrome (how you could know you have imposter syndrome and still have it I will put aside), it never seems to suffice to tell that person “You are an engineer, because you engineer things. You may be a bad engineer, but that’s not really for me to say.” The syndrome is about relative status, and never confusion about concrete abilities, like the question “Am I someone who architects buildings?” No one is ever confused about whether or not they can in fact maintain Apache servers; it is either a fish for compliments or an attempt at suppressing their own correct thoughts about being average. I suspect that the insecure often the actually incompetent, and the idea of Imposter Syndrome provides a mental placation for doubts about their own ability. It's too good of a perfectly general excuse. That nagging sense of inadequacy is explained away not by reality creeping up on their inflated sense of self, but by the DEMON of IMPOSTER SYNDROME running into their ego and smashing all of the nicely placed silverware.

Absent selection pressures, fifty percent of the people reading this statement will be below average at their jobs. You, the reader, in particular, are included in the broader population. It is a perfectly rational and reasonable thing to wonder if you have only deluded yourself into believing you are good at your work-not just wonder abstractly, as a remote possibility, but irritatingly, as something that may be highly probable. No pathology is required to explain this behavior in people. I and the people I wo...
Logical Representation of Causal Models
by johnswentworth — 2020-01-21T20:04:54.218Z

Epistemic status: I expect some people to say "this is obvious and trivial", and others to say "this makes no sense at all".

One fundamental difference between

E.T. Jaynes’ probability textbook

and most others is the emphasis on including our model in the prior information. When we write an expression like P[X=2|Y=3], that’s really shorthand for P[X=2|Y=3,M], where M is some probabilistic model in which X and Y appear. In practice, I’ve found this useful for two main use-cases:

model comparison

, and interventions/counterfactuals in causal models. This post is mainly about the latter.

The general idea is that an intervention in a causal model (e.g. do(Z=1)) takes in one model and returns a new model - it should really be written as M′=do(Z=1,M). When we write something like P[X=2|Y=3,do(Z=1)], that’s really shorthand for P[X=2|Y=3,do(Z=1,M)].

In order to make this all less hand-wavy, we need to make the model M a bit more explicit.

What’s in a Model?

The simplest way to represent a probabilistic model is as a table of possibilities - more explicitly, a list of exhaustive and mutually-exclusive logic statements. If I roll a standard die and call the outcome X, then I’d explicitly represent my model as M=(P[X=1|M]=16)&…&(P[X=6|M]=16).

In Probability as Minimal Map, we saw that P[X|P[X|Y]=p]=p. Interpretation: I obtain some data Y, calculate the probability P[X|Y]=p, then my computer crashes and I lose the data. But as long as I still know p, I should still assign the same probability to X. Thus: the probability of X, given P[X|Y]=p (but not given Y itself!) is just p.

(Note that I left the model M implicit in the previous paragraph - really we should write P[X|P[X|Y,M]=p,M]=p.)

Now let’s apply that idea to the expression P[X=1|M], with our die-model M=(P[X=1|M]=16)&…&(P[X=6|M]=16). Our given information includes P[X=1|M]=16, so

P[X=1|M]=P[X=1|P[X=1|M]=16,M]=16.

Representing models this way gives a much stronger logic-flavor to the calculations; our probability calculations are a derivation in an explicit logic. The axioms of that logic are the contents of M, along with the universal laws of probability (i.e. Bayes’ rule, sum rule, etc) and arithmetic.

Causality & Interventions

In the case of a causal model, M would look something like

M=(G=(...graph...))&...
Disasters
by jefftk — 2020-01-21T19:20:01.544Z

If there were a natural disaster tomorrow and it took about two weeks
to get things working again, how many people would be ok for food,
water, and other necessities? I'm guessing below 5%, but I think this
level of preparedness would be a good goal for most people who can
afford it. Why don't people plan for potential disasters? Some
possibilities:

They don't think disasters are likely. On the other hand, I
also don't think disasters are likely! While we have extra water in
the basement, I think the chances we'll need it sometime during my
life are only maybe 2%. Since it's not expensive, and if we do need
it we'll be incredibly happy to have it, I think it's worth setting
up.

It does matter a lot whether the chances are ~2% or 0.0002%, but
if you think your lifetime chance of being impacted by a serious
disaster is under 1% I'd encourage you to think about historical
natural disasters in your area (earthquakes, floods, hurricanes,
wildfires, etc) plus the risk of potential human-caused disasters
(nuclear war, epidemics, civil war, economic collapse, etc).

It's weird. Most people don't do it, and a heuristic of "do
the things other people do" is normally a pretty good one. In this
case, though, I think we should be trying to change what's normal.
The government agrees; the official recommendations involve a
lot more preparation than people typically do.

They can't afford the money, time, or thought. Many people are
in situations where planning for what's likely to happen in the next
couple months is hard enough, let alone for things that have a low
single digits chance of happening ever. This can't explain all of it,
though, because even people who do have more time and money also
haven't generally thought through simpler preparations.

They don't think preparation is likely to be useful. If
there's a nuclear strike we're all dead anyway, right? Except most
disasters, even nuclear ones, aren't this binary. Avoiding exposure
to radiation and having KI
available can help your long-term chances a lot. Many disasters
(nuclear, earthquake, epidemic, severe storm) are ones where having
sufficient supplies to stay at home for weeks would be very helpful.
If you think preparation wouldn't help and you haven't, say, read
through the suggestions on
ready.gov, I'd recommend doing that.

They're used to local emergencies. We generally hav

...
Safety regulators: A tool for mitigating technological risk
by JustinShovelain — 2020-01-21T13:07:05.649Z

Crossposted

to the Effective Altruism Forum

So far the idea of

differential technological development

has been discussed in a way that either (1) emphasizes ratios of progress rates, (2)

ratios of remaining work

, (3) maximizing or minimizing correlations (for example, minimizing the overlap between the capability to do harm and the desire to do so), (4) implementing safe tech before developing and implementing unsafe tech, and (5) the occasional niche analysis (possibly see also a complementary aside

relating differential outcomes to growth rates in the long run

). I haven’t seen much work talking about how various capabilities (a generalization of technology) may interact with each other in general in ways that prevent downside effects (though see also

The Vulnerable World Hypothesis

), and I wish to elaborate on this interaction type.

As technology improves, our capacity to do both harm and good increases and each additional capacity unlocks new capacities that can be implemented. For example the invention of engines unlocked railroads, which in turn unlocked more efficient trade networks. However, the invention of engines also enabled the construction of mobile war vehicles. How, in an ideal world, could we implement capacities so we get the outcomes we want while creating minimal harm and risks in the process?

What does implementing a capacity do? It enables us to change something. A normal progression is:

We have no control over something (e.g. We cannot generate electricity)
We have control but our choices are noisy and partially random (e.g. We can produce electric sparks on occasion but don’t know how to use them)
Our choices are organized but there are still downside effects (e.g. We can channel electricity to our homes but occasionally people get electrocuted or fires are started)
Our use of the technology mostly doesn’t have downside effects (e.g. We have capable safety regulators (e.g. insulation, fuses,...) that allows us to minimize fire and electrocution risks)

The problem is that downside effects in stages 2 and 3 could overwhelm the value achieved during those stages and at stage 4, especially when considering powerful game changing technologies that could lead to existential risks.

Even more fundamentally, as agents in the world we want to avoid shifting the expected utility in a negative direction relative to other options (the...
How Doomed are Large Organizations?
by Zvi — 2020-01-21T12:20:00.524Z

We now take the model from the previous post, and ask the questions over the next several posts. This first answer post asks these questions:

Are these dynamics the inevitable results of large organizations?

How can we forestall these dynamics within an organization?

To what extent should we avoid creating large organizations?

Has this dynamic ever been different in the past in other times and places?

These are the best answers I was able to come up with. Some of this is reiteration of previous observations and prescriptions. Some of it is new.

There are some bold claims in these answer posts, which I lack the space and time to defend in detail or provide citations for properly, with which I am confident many readers will disagree. I am fine with that. I do not intend to defend them further unless I see an opportunity in doing so.

I would love to be missing much better strategies for making organizations less doomed – if you have ideas please please please share them in the comments and/or elsewhere.

Are these dynamics the inevitable result of large organizations?

These dynamics are the default result of large organizations. There is continuous pressure over time pushing towards such outcomes.

The larger the organization, the longer it exists, and the more such outcomes have already happened, both there and elsewhere, the greater the pressure towards such outcomes.

Once such dynamics take hold, reversing them within an organization is extremely difficult.

Non-locally within a civilization, one can allow new organizations to periodically take the place of old ones to reset the damage.

Locally within a sufficiently large organization and over a sufficiently long time horizon, this makes these dynamics inevitable. The speed at which this occurs still varies greatly, and depends on choices made.

How can we forestall these dynamics within an organization?

These dynamics can be forestalled somewhat through a strong organizational culture that devotes substantial head space and resources to keeping the wrong people and behaviors out. This requires a leader who believes in this and in making it a top priority. Usually this person is a founder. Losing the founder is often the trigger for a rapid ramp up in maze level.

Keeping maze levels in check means continuously sacrificing substantial head space, resources, ability to scale and short-term effectiveness to this caus...
Book Review—The Origins of Unfairness: Social Categories and Cultural Evolution
by Zack_M_Davis — 2020-01-21T06:28:33.854Z
Link: http://unremediatedgender.space/2020/Jan/book-review-the-origins-of-unfairness/

On my secret ("secret") blog, I reviewed a book about the cultural evolutionary game theory of gender! I thought I'd share the link on this website, because you guys probably like game theory?? (~2400 words)
Whipped Cream vs Fancy Butter
by jefftk — 2020-01-21T00:30:02.176Z

Epistemic status: Jeff missing the point

The supermarket sells various kinds of fancy butter, but why don't
people eat whipped cream instead? Let's normalize to 100 calorie
servings and compare prices:

Plain Butter, store brand: $0.10

Heavy Whipping Cream, store brand: $0.20

Fancy butter, Kerrygold brand: $0.30

Perhaps the reason people don't normally use whipped cream is that
whipping it is too much trouble? If you use a manual eggbeater in a
standard sixteen ounce deli cup it takes about fifteen seconds (youtube) for a serving.

Alternatively, maybe people think whipped cream has to have sugar in
it? This one is simple: whipped cream should not have sugar in it.
If you're eating whipped cream on something sweet it doesn't need
sugar because the other thing is sweet, while if you're having it on
something savory it doesn't need sugar because that would taste funny.

I'm sure I'm missing something, but I'm very happy over here eating
whipped cream.
Inner alignment requires making assumptions about human values
by Matthew Barnett — 2020-01-20T18:38:27.128Z

Many approaches to AI alignment require making assumptions about what humans want. On a first pass, it might appear that inner alignment is a sub-component of AI alignment that doesn't require making these assumptions. This is because if we define the problem of inner alignment to be the problem of how to train an AI to be aligned with arbitrary reward functions, then a solution would presumably have no dependence on any particular reward function. We could imagine an alien civilization solving the same problem, despite using very different reward functions to train their AIs.

Unfortunately, the above argument fails because aligning an AI with our values requires giving the AI extra information that is not encoded directly in the reward function (under reasonable assumptions). The argument for my thesis is subtle, and so I will break it into pieces.

First, I will more fully elaborate what I mean by inner alignment. Then I will argue that the definition implies that we can't come up with a full solution without some dependence on human values. Finally, I will provide an example, in order to make this discussion less abstract.

Characterizing inner alignment

In the last few posts I wrote (1, 2), I attempted to frame the problem of inner alignment in a way that wasn't too theory-laden. My concern was that the previous characterization was dependent on a solving particular outcome where you have an AI that is using an explicit outer loop to evaluate strategies based on an explicit internal search.

In the absence of an explicit internal objective function, it is difficult to formally define whether an agent is "aligned" with the reward function that is used to train it. We might therefore define alignment as the ability of our agent to perform well on the test distribution. However, if the test set is sampled from the same distribution as the training data, this definition is equivalent to the performance of a model in standard machine learning, and we haven't actually defined the problem in a way that adds clarity.

What we really care about is whether our agent performs well on a test distribution that doesn't match the training environment. In particular, we care about the agent's performance on during real-world deployment. We can estimate this real world performance ahead of time by giving the agent a test distribution that wa...
Why Do You Keep Having This Problem?
by Davis_Kingsley — 2020-01-20T08:33:56.577Z

One thing I've noticed recently is that when someone complains about how a certain issue "just keeps happening" or they "keep having to deal with it", it often seems to indicate an unsolved problem that people may not be aware of. Some examples:

Players of a game repeatedly ask the same rules questions to the judges at an event. This doesn't mean everyone is bad at reading -- it likely indicates an area of the rules that is unclear or misleadingly written.
People keep trying to open a door the wrong way, either pulling on a door that's supposed to be pushed or pushing a door that's supposed to be pulled -- it's quite possible the handle has been designed poorly in a way that gives people the wrong idea of how to use it. (The Design of Everyday Things has more examples of this sort of issue.)
Someone keeps hearing the same type of complaint or having the same conversation about a particular policy at work -- this might be a sign that that policy might have issues. [1]
Every time someone tries to moderate a forum they run, lots of users protest against their actions and call it unjust; this might be a sign that they're making bad moderation decisions.

I'm not going to say that all such cases are ones where things should change -- it's certainly possible that one might have to take unpopular but necessary measures under some circumstances -- but I do think that this sort of thing should be a pretty clear warning sign that things might be going wrong.

Thus, I suspect you should consider these sorts of patterns not just as "some funny thing that keeps happening" or whatever, but rather as potential indicators of "bugs" to be corrected!

[1] This post was primarily inspired by a situation in which I saw someone write "This is the fifth time I've had this conversation in the last 24 hours and I'm sick of it" or words to that effect -- the reason they had kept having that conversation, at least in my view, was because they were implementing a bad policy and people kept questioning them on it (with perhaps varying degrees of politeness).