trentmkelly/LessWrong-43k huggingface dataset:
How close to nuclear war did we get over Cuba?
by  — 2022-05-13T19:58:51.591Z

Cross posted to LessWrong and The Good blog

The Cuban missile crisis was probably the closest we came to nuclear war. There were two types of close call in Cuba. There were moments of tension between the US and Soviet militaries, all of which tell us something interesting about how different patterns of civil-military relations influence the outcomes of moments of crisis. But, there was also the risk of nuclear not by accident but by intention. Throughout the crisis, the US government kept an invasion of Cuba or airstrikes on the missile sites as options on the table, and the former was pushed very aggressively by the joint chiefs of staff. The joint chiefs of staff were the most senior officers in the US army and provided its strategic direction. The influence of these most senior members of the military bureaucracy has been pervasive throughout the history of nuclear weapons, and their role in nuclear risk will be something I’ll return to in a later post.

Cuba: some background

I gave some background on the strategic situation in Cuba in this

post

, but for this story it’s laying out the timeline of events in a little more detail. From the end of the Spanish-American war in 1898 until Castro’s revolution toppled the dictator Hugo Baptista, Cuba was the playground of the American elite and a quasi-colonial possession. American companies owned the sugar plantations, and the mines and the casinos and Baptista’s thugs beat up anyone who tried to change things. You can imagine then that it came as quite a shock when the bearded, fiery, Jesuit educated Castro succeeded in launching a nationalist revolution and overthrowing the Baptista government. Initially Castro made overtures to the Americans but these were rebuffed as Ho Chi Minh’s attempts at alliance - another nationalist leader who turned to communism - had been 4 years earlier in 1956. So, Castro turned to the Soviets and so, with Castro now firmly in the Soviet camp and expropriating all of the American owned businesses, it was time for Castro to go.

Another theme in the history of nuclear weapons has been the importance of intelligence and intelligence failures and they will get their own post later. By 1961 the CIA had successfully orchestrated changes of government in Iran and Guatemala and expected to be able to do the same with Cuba and that the Cuban refugees the...
Against Time in Agent Models
by johnswentworth — 2022-05-13T19:55:33.324Z

When programming distributed systems, we always have many computations running in parallel. Our servers handle multiple requests in parallel, perform read and write operations on the database in parallel, etc.

The prototypical headaches of distributed programming involve multiple processes running in parallel, each performing multiple read/write operations on the same database fields. Maybe some database field says “foo”, and process 1 overwrites it with “bar”. Process 2 reads the field - depending on the timing, it may see either “foo” or “bar”. Then process 2 does some computation and writes another field - for instance, maybe it sees “foo” and writes {“most_recent_value”: “foo”} to a cache. Meanwhile, process 1 overwrote “foo” with “bar”, so it also overwrites the cache with {“most_recent_value”: “bar”}. But these two processes are running in parallel, so these operations could happen in any order - including interleaving. For instance, the order could be:

Process 2 reads “foo”
Process 1 overwrites “foo” with “bar”
Process 1 overwrites the cache with {“most_recent_value”: “bar”}
Process 2 overwrites the cache with {“most_recent_value”: “foo”}

… and now the cached value no longer matches the value in the database; our cache is broken.

One of the main heuristics for thinking about this sort of problem in distributed programming is: there is no synchronous time. What does that mean?

Well, in programming we often picture a “state-update” model: the system has some state, and at each timestep the state is updated. The update rule is a well-defined function of the state; every update happens at a well-defined time. This is how each of the individual processes works in our example: each executes two steps in a well-defined order, and each step changes the state of the system

Single process: a well-defined sequence of steps, each updating the state.

But with multiple processes in parallel, this state-update model no longer works. In our example, we can diagram our two processes like this:

Each process has its own internal “time”: the database read/write happens first, and the cache overwrite happens second. But between processes, there is no guaranteed time-ordering. For instance, the first step of process 1 could happen before all of process 2, in between the steps of process 2, or after all of process 2.

Two processes: many possible time-orderings of the op

...
Agency As a Natural Abstraction
by Thane Ruthenis — 2022-05-13T18:02:50.308Z

Epistemic status: Speculative attempt to synthesize findings from several distinct approaches to AI theory.

Disclaimer: The first three sections summarize some of Chris Olah's work on interpretability and John Wentworth's Natural Abstractions Hypothesis, then attempt to draw connections between them. If you're already familiar with these subjects, you can probably skip all three parts.

Short summary: When modelling a vast environment where simple rules result in very complex emergent rules/behaviors (math, physics...), it's computationally efficient to build high-level abstract models of this environment. Basic objects in such high-level models often behave very unlike basic low-level objects, requiring entirely different heuristics and strategies. If the environment is so complex you build many such models, it's computationally efficient to go meta, and build a higher-level abstract model of building and navigating arbitrary world-models. This higher-level model necessarily includes the notions of optimization and goal-orientedness, meaning that mesa-optimization is the natural answer to any "sufficiently difficult" training objective. All of this has various degrees of theoretical, empirical, and informal support.

1. The Universality Hypothesis

One of the foundations of Chis Olah's approach to mechanistic interpretability is the Universality Hypothesis. It states that neural networks are subject to convergence — that they would learn to look for similar patterns in the training data, and would chain up the processing of these patterns in similar ways.

The prime example of this effect are CNNs. If trained on natural images (even from different datasets), the first convolution layer reliably learns Gabor filters and color-contrast detectors, and later layers show some convergence as well:

Analogous features across CNNs. Source.

It's telling that these features seem to make sense to us, as well — that at least one type of biological neural network also learns similar features. (Gabor filters, for example, were known long before modern ML models.) It's the main reason to feel optimistic about interpretability at all — it's plausible that the incomprehensible-looking results of matrix multiplications will turn out to be not so incomprehensible, after all.

It's telling when universality doesn't hold, as well.

Understanding RL Vision attempts to interpret an agent trained...
Fermi estimation of the impact you might have working on AI safety
by Fabien Roger — 2022-05-13T17:49:57.865Z

Cross-posted here: https://forum.effectivealtruism.org/posts/widWpunQMfuNTCYE3/fermi-estimation-of-the-impact-you-might-have-working-on-ai

I tried doing a Fermi estimation of the impact I would have if I worked on AI safety, and I realized it wasn't easy to do with only a calculator. So I build a website which does this Fermi estimation given your beliefs about AGI, AI safety, and your impact on AI safety progress.

You can try it out here: https://xriskcalculator.vercel.app/

This tool focuses on technical work, and assumes that progress on AGI and progress on AI safety are independent. This is obviously an approximation that is vastly inaccurate, but for now I don't think of a simple way of taking into account the fact that advanced AI could speed up AI safety progress. Other limitations are outlined on the website.

What do you think of this tool? Do you think of a way it could be improved?
"Tech company singularities", and steering them to reduce x-risk
by Andrew_Critch — 2022-05-13T17:24:00.417Z

The purpose of this post (also available on the EA Forum) is to share an alternative notion of “singularity” that I’ve found useful in timelining/forecasting.

A fully general tech company is a technology company with the ability to become a world-leader in essentially any industry sector, given the choice to do so — in the form of agreement among its Board and CEO — with around one year of effort following the choice.

Notice here that I’m focusing on a company’s ability to do anything another company can do, rather than an AI system's ability to do anything a human can do. Here, I’m also focusing on what the company can do if it chooses rather than what it actually ends up choosing to do. If a company has these capabilities and chooses not to use them — for example, to avoid heavy regulatory scrutiny or risks to public health and safety — it still qualifies as a fully general tech company.

This notion can be contrasted with the following:

Artificial general intelligence (AGI) refers to cognitive capabilities fully generalizing those of humans.
An autonomous AGI (AAGI) is an autonomous artificial agent with the ability to do essentially anything a human can do, given the choice to do so — in the form of an autonomously/internally determined directive — and an amount of time less than or equal to that needed by a human.

Now, consider the following two types of phase changes in tech progress:

A tech company singularity is a transition of a technology company into a fully general tech company. This could be enabled by safe AGI (almost certainly not AAGI, which is unsafe), or it could be prevented by unsafe AGI destroying the company or the world.
An AI singularity is a transition from having merely narrow AI technology to having AGI technology.

I think the tech company singularity concept, or some variant of it, is important for societal planning, and I’ve written predictions about it before, here:

2021-07-21

— prediction that a tech company singularity will occur between 2030 and 2035

2022-04-11

— updated prediction that a tech company singularity will occur between 2027 and 2033.

A tech company singularity as a point of coordination and leverage

The reason I like this concept is that it gives an important point of coordination and leverage that is not AGI, but which interacts in important ways with AGI. ...
An observation about Hubinger et al.'s framework for learned optimization
by carboniferous_umbraculum  — 2022-05-13T16:20:33.974Z

The observations I make here have little consequence from the point of view of solving the alignment problem. If anything, they merely highlight the essential nature of the inner alignment problem. I will reject the idea that robust alignment, in the sense described in Risks From Learned Optimization, is possible at all. And I therefore also reject the related idea of 'internalization of the base objective', i.e. I do not think it is possible for a mesa-objective to "agree" with a base-objective or for a mesa-objective function to be “adjusted towards the base objective function to the point where it is robustly aligned.” I claim that whenever a learned algorithm is performing optimization, one needs to accept that an objective which one did not explicitly design is being pursued. At present, I refrain from attempting to propose my own adjustments to the framework, or to build on the existing literature or to develop my own theory. I am certainly not against doing any of those things, but they are things to possibly be pursued later; none of them is the purpose of this post.

To make my main point, I will introduce only a bare minimum of mathematical notation. We will show that a mesa-objective always has a different type signature to a base objective and that the default assumption ought to be that there is no way to compare them in general and certainly no general way to interpret what it means for them to ‘agree’. Suppose that an optimizer is searching through a space S of systems. At this time, I do not want to attempt to unpack what it means to 'search', but, naively, we can imagine that there is an objective function f :S→R, which determines something that we might call the 'search criterion'. The idea of course is that the optimizer is a system that is 'searching' through the set S and judging different points according to the criterion that higher values of f are better.

In the background, there is some 'task' and naively we can think of this as being represented by a 'task space' X which consists of all of the different possible 'presentations' or 'instances' of the task. For example, perhaps the task is choosing the next move in a game of Go or the next action in a real-time strategy video game. In these examples, a given x∈X would represent a board position in Go, say, or a single sna...
The Economics of a New Energy Source 
by casualphysicsenjoyer — 2022-05-13T14:08:34.865Z

Imagine a world that is fuelled exclusively by solar energy that comes from space. This would happen by having many solar panels in orbit, beaming down energy through microwave transmission.

I am trying to understand how a firm or government might price this sort of energy to consumers based on its costs. Suppose that a large energy company pays a company like spaceX to put solar satellites in orbit - how would the price of energy vary as a function of the cost of putting satellites into orbit? I am not formally trained in economics, and I was looking for a way to start framing this problem properly.
[Rough notes, BAIS] Human values and cyclical preferences
by pranomostro — 2022-05-13T13:28:04.907Z

epistemic status: Working notes of three different people on the same question, likely useless/incomprehensible to anyone else

The question

How to find the right abstraction level of human values

Problems in over- or underfitting human values:

We can learn human values by observing their actions and distilling them into a preference relation. This learned preference relation can overfit human values (eg: Humans want to raise their left arm by 2 cm on 2022-05-07 if they’re in some specific place) or it can underfit human values (eg: Humans care only about maximizing money). If our preference relation overfits, we expect to not find some known biases, e.g. the Allais Paradox. There are also both inconsistencies that are “too abstract” and “too concrete”:

Too abstract: If I have three cities B, H, and M, and I traveled B→H,H→M,M→H, then one might conclude that I have an inconsistency, but in reality I made the travels M1→B1,B1→H1 and H2→M2

Too concrete: (?) If I hugged my friend at location l1 and time t1(H11), but not at l1 and t2(H12), but the information about time was disregarded, we might conclude that H1↔H1, which is inconsistent, but in reality I pretty much always want to hug my friend, regardless of time and place.

For a set of worlds W, the learned preference relation P⊆W×W represents which world is preferred to another (this generates a graph G=(W,W×W) which can be any graph, with cycles & disconnected components). If we overfit human values, we assume they’re way more rational than they actually are, if we underfit them, we assume they’re way less rational than they actually are. So there is a spectrum over the complexity of the learned preference relation: from overfitting/complexity/rationality/concreteness to underfitting/simplicity/irrationality/abstraction.

Cognitive biases as a lower bound for finding the right abstraction level of human preferences

Behavior commonly considered irrational can give pointers towards which abstraction level to use. Literature on cognitive biases give a lower bound on the abstraction level of human values. For example, scope neglect is only a bias if we consider the preference “Save the most birds possible” to be our actual preference, and not “Get the most pleasure possible out of saving birds for the least cost of saving them”. A learner having inferred the first preference will judge humans as exhibiting ...
Can moderators fix old sequences posts?
by EniScien — 2022-05-13T12:30:03.036Z

I'm re-reading sequences now and I'm noticing eye-opening things like so many have SEQ RERUN, useless copies that clutter up the link space while having very few comments that they seem to be intended for.
Can the moderators do something about this? Remove links to them, maybe even delete them by moving the comments to the original posts.
(I don't know if there is functionality to do this, and also how morally acceptable it would be, and if anyone but me is worried about these problems, maybe no one else needs to "fix" this?)
And also I notice, that in the old entries, the answers are not child comments, which creates a terrible confusion when reading by karma, when it is not clear what the question was and where to find it.
(Same requests and questions here)
DeepMind is hiring for the Scalable Alignment and Alignment Teams
by Rohin Shah — 2022-05-13T12:17:13.157Z

We are hiring for several roles in the Scalable Alignment and Alignment Teams at DeepMind, two of the subteams of DeepMind Technical AGI Safety trying to make artificial general intelligence go well. In brief,

The Alignment Team investigates how to avoid failures of intent alignment, operationalized as a situation in which an AI system knowingly acts against the wishes of its designers. Alignment is hiring for Research Scientist and Research Engineer positions.
The Scalable Alignment Team (SAT) works to make highly capable agents do what humans want, even when it is difficult for humans to know what that is. This means we want to remove subtle biases, factual errors, or deceptive behaviour even if they would normally go unnoticed by humans, whether due to reasoning failures or biases in humans or due to very capable behaviour by the agents. SAT is hiring for Research Scientist - Machine Learning, Research Scientist - Cognitive Science, Research Engineer, and Software Engineer positions.

We elaborate on the problem breakdown between Alignment and Scalable Alignment next, and discuss details of the various positions.

“Alignment” vs “Scalable Alignment”

Very roughly, the split between Alignment and Scalable Alignment reflects the following decomposition:

Generate approaches to AI alignment – Alignment Team
Make those approaches scale – Scalable Alignment Team

In practice, this means the Alignment Team has many small projects going on simultaneously, reflecting a portfolio-based approach, while the Scalable Alignment Team has fewer, more focused projects aimed at scaling the most promising approaches to the strongest models available.

Scalable Alignment’s current approach: make AI critique itself

Imagine a default approach to building AI agents that do what humans want:

Pretrain on a task like “predict text from the internet”, producing a highly capable model such as Chinchilla or Flamingo.
Fine-tune into an agent that does useful tasks, as evaluated by human judgements.

There are several ways this could go wrong:

Humans are unreliable: The human judgements we train against could be flawed: we could miss subtle factual errors, use biased reasoning, or have insufficient context to evaluate the task.
The agent’s reasoning could be hidden: We want to know not just what the system is doing but why, both

...
Thoughts on AI Safety Camp
by Charlie Steiner — 2022-05-13T07:16:55.533Z

I

Early this year I interviewed a sample of AISC participants and mentors, and spent some time thinking about the problems the AI safety research community is facing, and have changed my mind about some things.

AI Safety Camp is a program that brings together applicants into teams, and over about a hundred hours of work those teams do AI safety-related projects that they present at the end (one project made it into a Rob Miles video). I think it's really cool, but what exactly it's good for depends on a lot of nitty gritty details that I'll get into later.

Who am I to do any judging? I'm an independent alignment researcher, past LW meetup organizer, physics PhD, and amateur appliance repairman. What I'm not is a big expert on how people get into alignment research - this post is a record of me becoming marginally more expert.

II

The fundamental problem is how to build an ecosystem of infrastructure that takes in money and people and outputs useful AI safety research. Someone who doesn't know much about AISC (like my past self) might conceive of many different jobs it could be doing within this ecosystem:

Educating relative newcomers to the field and getting them more interested in doing research on AI alignment.
Providing project opportunities that are a lot like university class projects - contributing to the education of people in the process of skilling up to do alignment research.
Providing potentially-skilled and potentially-interested people a way to "test their fit" to see if they want to commit to doing more AI alignment work.
Catalyzing the formation of groups and connections that will persist after the end of the camp.
Helping skilled and interested people send an honest signal of their alignment research skills to future employers and collaborators.
Producing object-level useful research outputs.

In addition to this breakdown, there's orthogonal dimensions of what parts of AI safety research you might specialize to support:

Conceptual or philosophical work.
Machine learning projects.
Mathematical foundations.
Policy development.
Meta-level community-building.

Different camp parameters (length, filters on attendees, etc.) are better-suited for different sorts of projects. This is why AISC does a lot of machine learning projects, and why there's a niche for AISC alum Adam Shimi to start a slightly different thing focused on conceptual work (Refine).

III

Before talking ...