trentmkelly/LessWrong-43k huggingface dataset:
D&D.Sci Divination: Nine Black Doves Evaluation & Ruleset
by aphyer — 2022-05-17T00:34:25.019Z

This is a follow-up to last week's D&D.Sci scenario: if you intend to play that, and haven't done so yet, you should do so now before spoiling yourself.

There is a web interactive here you can use to test your answer, or you can read on.

RULESET

Map

Provinces are laid out as follows:

This matters for war, and for the spread of plague (and of black doves). Provinces are at risk of being pillaged by adjacent provinces of different empires (e.g. Italia is at risk of being pillaged only by Germania), and are at risk of plague spreading from any adjacent province, friend or foe (e.g. Italia can contract plague from Germania, Hispania or Grecia).

Congratulations to simon, who I believe was the first to identify the map and the connection to Plague and Pillaging.

Population and Wealth

The main factors driving a province's exposure to disasters were not the omens directly, but its Population and Wealth. The majority of omens were relevant only insofar as they had relationships to Population and Wealth. Population and Wealth are both represented as integer values that tend to almost always be between 1 and 10.

Both Population and Wealth go up gradually but then decrease with certain disasters. Every year, three things happen in order:

Disasters are determined (details below).

A province suffering a Famine or Plague will lose half (rounded up) its Population.
A province suffering an Earthquake or Fire will lose half (rounded up) its Wealth.
A province being Pillaged will lose half (rounded up) its Wealth, and the pillaging province will gain that Wealth.

Growth is determined.

Each province has an independent 50% chance to gain a point of Population, and an independent 50% chance to gain a point of Wealth.

Omens are determined (details below).

Fire and Famine

These are the two main negative-feedback disasters, keeping Population and Wealth growth in check by occurring when they grow too high:

Any province with Wealth > 4 is at risk of Fire, with a probability of 10% per point of Wealth above 4.
Any province with Population > Wealth is at risk of Famine, with a probability of 10% per point of Population above its Wealth.

Secular Omens

Most things the existing diviners categorize as Omens have nothing supernatural about them. They are still potentially useful to you, however, due to their relationship to Population and Wealth:

Moon Turns Red is caused by

...
DeepMind’s generalist AI, Gato: A non-technical explainer
by  — 2022-05-16T21:21:24.214Z

Summary

DeepMind’s recent paper,

A Generalist Agent

, catalyzed a wave of discourse regarding the speed at which current artificial intelligence systems are improving and the risks posed by these increasingly advanced systems. We aim to make Gato accessible to non-technical folks by: (i) providing a non-technical summary, and (ii) discussing the relevant implications related to existential risk and AI policy.

Introduction

DeepMind has just introduced its new agent, Gato: the most general machine learning (ML) model to date. If you’re familiar with arguments for the potential risks posed by advanced AI systems, you’ll know the term general carries strong implications. Today’s ML systems are advancing quickly; however, even the best systems we see are narrow in the tasks they can accomplish. For example,

DALL-E

impressively generates images that rival human creativity; however, it doesn’t do anything else. Similarly, large language models like

GPT-3

perform well on certain text-based tasks, like sentence completion, but poorly on others, such as arithmetic (Figure 1).

If future AI systems are to exhibit human-like intelligence, they’ll need to use various skills and information to complete diverse tasks across different contexts. In other words, they’ll need to exhibit general intelligence in the same way humans do—a type of system broadly referred to as artificial general intelligence (AGI). While AGI systems could lead to hugely positive innovations, they also have the potential to surpass human intelligence and become “superintelligent”. If a superintelligent system were unaligned, it could be difficult or even impossible to control for and predict its behavior, leaving humans vulnerable.

Figure 1: An attempt to teach GPT-3 arithmetic. The letter ‘Q’ denotes human input while ‘A’ denotes GPT-3’s response (from Peter Wildeford’s

tweet

)

So what exactly has DeepMind created? Gato is a single neural network capable of performing hundreds of distinct tasks. According to DeepMind, it can, “play Atari, caption images, chat, stack blocks with a real robot arm and much more, deciding based on its context whether to output text, joint torques, button presses, or other tokens.” It’s not currently analogous to human-like intelligence; however, it does exhibit general capabilities. In the rest of this post, we’ll provide a non-technica...
Proxy misspecification and the capabilities vs. value learning race
by Sam Marks — 2022-05-16T18:58:24.044Z

G Gordon Worley III recently complained about a lack of precision in discussions about whether Goodhart's Law will present a fatal problem for alignment in practice. After attending a talk in which Dylan Hadfield-Menell[1] presented the "Goodhart's Law will be a big deal" perspective, I came away with a relatively concrete formulation of where I disagree. In this post I'll try to explain my model for this, expanding on my short comment here.

More specifically, in this post I'll discuss proxy misspecification, which I view as a subproblem of Goodhart's Law (but not necessarily the entirety of it).

Our model of proxy misspecification

First I'll borrow Hadfield-Menell's mathematical model of the proxy misspecification problem. Suppose:

Alice has things s1,…,sn which she values, i.e. she has a utility function U(s1,…,sn) which is increasing in each si.
A robot is given a proxy utility function ~U which depends on some strict subset of the si's.
The robot optimizes ~U by freely varying all si, subject only to resource constraints that require the si to trade off against each other.

Then, as you'd expect, it's a theorem that the robot will set to their minimum possible values all si on which the proxy utility function ~U does not depend (instead putting all resources to increasing the si's which ~U does take into account).

To borrow an example from the paper, we can imagine that the robot is a content recommendation algorithm. Then the si's might be metrics for things like ad revenue generated, engagement quality, content diversity, and overall community well-being. Humans care about all of these things, but our content recommendation algorithm might have a proxy reward that depends only on the first two. Furthermore, the algorithm only gets to recommend a limited amount of content, a resource constraint which (let's assume) prevents it from making all of these metrics as good as possible. In that case, according to the model, the algorithm will recommend content in a way that minimizes the features on which its proxy reward doesn't depend: content diversity and overall community well-being.

My complaint: it might be hard to decouple the si's

The above formulation of the proxy misspecification problem is a theorem. So if I'm going to resist the concl...
Optimization at a Distance
by johnswentworth — 2022-05-16T17:58:25.253Z

We have a computational graph (aka circuit aka causal model) representing an agent and its environment. We’ve chosen a cut through the graph to separate “agent” from “environment” - i.e. a Cartesian boundary. Arrows from environment to agent through the boundary are “observations”; arrows from agent to environment are “actions”.

Presumably the agent is arranged so that the “actions” optimize something.

The actions “steer” some nodes

in the system toward particular values

.

Let’s highlight a few problems with this as a generic agent model…

Microscopic Interactions

My human body interfaces with the world via the entire surface area of my skin, including molecules in my hair randomly bumping into air molecules. All of those tiny interactions are arrows going through the supposed “Cartesian boundary” around my body. These don’t intuitively seem like “actions” or “observations”, at least not beyond some high-level observations of temperature and pressure.

In general, low-level boundaries will have lots of tiny interactions crossing them which don’t conceptually seem like “actions” or “observations”.

Flexible Boundaries

When I’m driving, I often identify with the car rather than with my body. Or if I lose a limb, I stop identifying with the lost limb. (Same goes for using the toilet - I’ve heard that it’s quite emotionally stressful for children during potty training to throw away something which came from their physical body, because they still identify with it.)

In general, it’s ambiguous what Cartesian boundary to use; our conceptual boundaries around an “agent” don’t seem to correspond perfectly to any particular physical surface.

An Agent Optimizing Its Own Actions

I could draw a supposed “Cartesian boundary” around a rock, and declare all the interactions between the rock and its environment to be “actions” and “observations”. If someone asks what the rock is optimizing, I’ll say “the actions” - i.e. the rock “wants” to do whatever it is that the rock in fact does.

In general, we intuitively conceive of “agents” as optimizers in some nontrivial sense. Optimizing actions doesn’t cut it; we generally don’t think of something as an agent unless it’s optimizing something out in the environment away from itself.

Solution: Optimization At A Distance

Let’s solve all of these problems in one fell swoop.

We’ll start with the rock problem. One natural answer is to de...
To what extent is your AGI timeline bimodal or otherwise "bumpy"?
by jchan — 2022-05-16T17:42:54.281Z

For example, you might think:

It's likely that AGI will be invented before 2050; however, if it isn't, then that must mean either that AGI is impossible, or that it requires much more advanced technology than I currently think it does, or else that there was some kind of large-scale civilizational collapse in the meantime.

For that matter, any non-exponential distribution has this property, where the non-occurrence of the event by a certain time will change your expectation of it going forward. I'm curious if people think this is the case for AGI, and if so, why. (Also curious if this question has been asked before.)
China Covid #3
by Zvi — 2022-05-16T17:40:00.891Z

China has once again explicitly recommitted to its goal of Zero Covid.

Even the WHO has said this policy is unsustainable. China don’t care.

Or rather, Xi doesn’t know and who is going to tell him?

Premiere Li might be trying, but it’s not working.

Xi, whose name keeps coming up headlines during his reign which is never a good sign, is going to see this through as long as China possibly can. How long is that?

My answer last time was almost certainly not so long. The plan wasn’t working.

Yet the market says otherwise? Here’s the six month look at the Shanghai stock market.

For comparison, here’s the S&P:

The Chinese market has been overperforming these past few weeks and is up in absolute terms since mid-April. Presumably that indicates that the market expects less disruption now.

Which makes sense, since (at least officially) cases are down quite a bit.

This is what successful containment would look like if there was successful containment. If, of course, it’s all real.

Here are deaths.

Do you notice anything weird? Deaths peaked at April 30. Cases hit a local peak at April 29 with an absolute high at April 21. After April 30, the decline in deaths is quite rapid, dropping an order of magnitude. But it has not yet been three weeks since cases in China started declining. One could say ‘maybe Chinese death reporting is better because they don’t have unnecessary delays’ but most of the delays in reporting are because it takes time to die.

So these stories don’t make sense together. That’s nothing new for China. What’s interesting is that at the peak we have 0.035/million deaths and about 20/million cases, for a (no-delay) case fatality rate of ~1.75%, with delays that could be ~3%. For comparison, the USA case rate is current 248/million and its deaths are 0.89/million, for a case fatality rate of ~0.3%, or ~0.5% if we go back 21 days for cases – of course the infection fatality rate is far lower since most infectious aren’t recorded here.

We have higher rates of effective vaccination and also higher rates of previous infection, so we should expect that China’s CFR should come in much higher depending on the ratio of how many cases (and deaths) get reported. No doubt a lot of cases were being missed or actively suppressed in China on the way up if there were this many deaths. China has no incentive to report extra deaths.

Wh...
BIDA Air Quality Measurements
by jefftk — 2022-05-16T14:00:04.872Z

I've been curious for a while how well the ventilation at the BIDA
contra dance works. This is actually something that we can measure
reasonably well by looking at CO2 levels: outside air has about 400 parts-per-million (ppm), while your exhalation is about 40,000ppm. By measuring the
concentration of CO2 in the air inside, we can determine how much of
the air you're breathing has been through other people first.

I recently got a Temptop
M2000, which measures CO2, as well as particle levels (pm2.5 and
pm10) and HCHO (formaldehyde). I got it because I'm interested in
running some tests on my ceiling
fan air filter idea, but now that I have it I want to measure
everything. It logs to one minute intervals, and you can export to
CSV.

The dance is in the basement of the Cambridge Masonic Hall, and the
main room is 66x48 (3200 sqft) with about a 25ft ceiling (80,000
ft3). There are four ceiling fans which we keep on high. We have
two large
barrel fans, rated for 15,000 CFM on high. We put one blowing in
at one end, and one blowing out at the other. In theory that might
give us 11 ACH (60 min/hr * (80k ft3 / 15k ft3/min)) but I would expect to
see lower performance than that: there are various obstructions, and
there is nothing to stop air coming back in around the fan (the
opening is not sealed around it).

Here's what I measured, with the monitor in a back corner where I
expect the air is most stale:

These readings makes sense:

The air was initially pretty stale when we got there

Turning the fans on had a strong effect

Dancing had an even larger effect in the other direction,
presumably through people breathing more heavily.

Decrease during the break, when people aren't dancing.

Increase after the break, but not as large because dancers
generally start leaving at the break and during the second half.

Here's another way to look at this: since atmospheric CO2 is 400ppm
and exhalation is 40,000ppm, we can look at how stale the air is on a
continuum that runs from "outdoors" to "literally breathing only
someone's exhalation". This is (CO2-400)/(40,000-400). Two people
having a meeting in a small room with conventional office HVAC gives
you ~1000ppm or 1.5% stale, and I'm going to use this as a benchmark
for "standard indoor air quality". This dance ranged from 0.5% to
1.5%, so I would say our ventilation was successful a...
The Cuban missile crisis: the strategic context
by  — 2022-05-16T12:12:56.917Z

Cross posted to the EA forum and The Good blog

Why did the US blockade Cuba after it discovered that nuclear missiles launch sites were being constructed there? I think the story most people have is that because the missiles were so close to the US it meant that suddenly the US was much more vulnerable to a nuclear attack than it had been previously. This meant that the US wanted to stop the rest of the supplies bound to Cuba on Soviet ships from reaching their destination.

There is some truth to this story. But the real one I think points to one of the key events of 20th century history being deeply mysterious and turning on the specific date in which an obscure defence department document was published. I’m now not sure if this is correct

The strategic holy grail for country A during the cold war was for country A to attain a secure second strike capability while country B did not. This means that if B were to launch a nuclear strike on A, A would be able to respond above a threshold of destruction. However, the same does not hold for B - if A were to launch a first strike, B would either no longer have a nuclear deterrent or it would have a ‘small’ one. If both of these conditions are met, country A can launch a first strike to win a war.

The balance of terror

Surprisingly to me at least, in 1962 it was plausible that the US could launch a first strike and plausible that the leadership of both the US and USSR knew it. The most important reason for this was that the US dominated the USSR both in terms of number of ICMBs and the speed at which they could get them off the ground. The USSR had about 12 ICMBs whereas the US had hundreds and crucially because they were solid-fuel rather than liquid fuelled they could get them off the ground within minutes whereas it took the fastest Soviet missiles 2 hours to launch. This was in the era when Soviet ICMBs were still stored in silos which couldn’t withstand multiple nuclear strikes meaning they were sitting ducks for US missiles. If the only missiles each side had to target one another were ICMBs then the US would have had a first strike capability no question.

However, by 1962 - by 1959 to be exact - both the US and USSR had all three legs of the nuclear triad operational. Although, for the Soviets, only just. I can’t find out how many long range bomb...
Why I'm Optimistic About Near-Term AI Risk
by harsimony — 2022-05-15T23:05:29.233Z

I'm not worried about AI posing an existential risk in the next 10-20 years. Recent developments in AI capabilities actually make me feel more optimistic about this. The fact that relatively simple models can perform a wide array of tasks suggests that we can build satisfactory AI without the need to use sophisticated, potentially dangerous agents in the near-term.

My expectation for how AI will develop over the next decade is that companies will continue to focus on transformer-based foundation models. The general capability of these models will increase for a while simply by using more data, improving training procedures, and leveraging specialized hardware. Eventually, companies will start hitting bottlenecks in the amount of data required for optimal training at a given capability level. But before that, deployment of these systems will favor smaller, faster, and more auditable models leading companies to focus on distilled models specializing in specific tasks.

These specialized models will be oriented towards augmenting human productivity, producing entertainment, or automating specific tasks. The slow pace at which industries change their practices and utilize the benefits of a new technology will moderate the adoption of AI. As adoption increases, these AI services will gain autonomy, producing more value at lower cost. Continued specialization will result in mostly autonomous AI's derived from generally capable foundation models that are distilled down for variety of tasks.

I'm not claiming that these Tool AI's won't eventually be dangerous, but I can't see this path leading to high existential risk in the next decade or so.

I think most people in the AI safety field would agree with me on this, so why write it up?

I want to make this point explicit and foster a discussion about near-term AI safety. If AI will become dangerous soon, the field needs to act very quickly. Researchers would have to consider eschewing movement building, trading goodwill for influence, and gambling on near-term approaches. People who would take more than a decade to have an impact would have less reason to join in the first place while investments in infrastructure in the field would become less valuable.

It's important for those concerned to be on the same page about near-term risks in order to avoid the unilaterialist's curse. Recent, pessimistic takes about AI risk make it s...
Definition Practice: Applied Rationality
by ChristianKl — 2022-05-15T20:44:54.907Z

In the exercise of defining what we mean by a term, we explore our assumptions about the nature of the term and what it’s pointing at. Sometimes this leads us to get a better handle on a concept. Other times, it shows us our confusion.

As an example of the value of definitions, YCombinator considers it important for founders to be able to clearly and concisely say a company's purpose, and that the ability to do so is indicative of plausible success..

Here are two exercises:

Write a paragraph to define the term applied rationality.
Write a sentence to define the term applied rationality.

I encourage you to write your definitions before reading what others wrote. While I suggest that you first attempt to define the term in a paragraph and then move to the sentence variation, you can also change the order if that seems easier.

Please use the answer feature for your answer to these exercises, and the comment function for other needs.
Surviving Automation In The 21st Century - Part 1
by George3d6 — 2022-05-15T19:16:16.966Z
Link: https://www.epistem.ink/p/surviving-automation-in-the-21st?s=w

This is another career article (series?), I’m, in part, writing this for myself, but I may as well publish it.

I want to think through the kind of jobs and career choices I should take in order to hedge against and benefit from increased automation. Especially the kind that is driven by better and more widely applied ML. Not just for monetary reasons, but also to optimize for deriving meaning out of an automated world; To avoid crushing existential angst due to a fundamental part of your “self” being tied to doing a task that’s now better fulfilled by a microscopic piece of silicon.

i - Framing Automation

There are many ways to frame and think about automation.

The “conservative” take is to point out that it’s happened before and it never led to that much rapid change in the job market, with the only noticeable trend being that the people hardest hit by it were the already unprivileged, doing tedious manual labor; And new jobs were created, usually office-work.

The “futurist” take is that this time is different, and that we’re on the cusp of a technological singularity where machines will surpass human intelligence, leading to a future where most jobs are automated and we have to find new ways to occupy our time. And while that does sound cliche, keep in mind all the bits in italic are generated by a 2-years outdated text model, *'*first take, with no editing, in a few milliseconds, using less energy than is required for my finger to type a single letter. If that isn’t a sign of exponential change, I don’t know what is.

So where does that leave us?

The answer, as is often the case, is somewhere in the middle. We are on the cusp of a major change, but it’s not going to be as sudden or all-encompassing as the “futurists” would have you believe. And it’s not going to be as slow or imperceptible as the “conservatives” would have you believe.

The truth is that automation is going to change the world, but not in the way that we expect.

... I disagree with the language model here. I think it leaves us in a spot where there’s a significant probability of changes so vast that 5 years from now this piece and all of its advice would be irrelevant, maybe that “we” will no longer be “a thing” in 5 years from now. But that is not very relevant to worry about; We should hedge our bets within the kind of bounds where concepts like “money” and “job” and “human” still make sense, anyt...