trentmkelly/LessWrong-43k huggingface dataset:
Careless talk on US-China AI competition? (and criticism of CAIS coverage)
by Oliver Sourbut — 2023-09-20T12:46:16.696Z
Link: https://www.oliversourbut.net/p/careless-talk-on-us-china-ai-competition

China has made several efforts to preserve their chip access, including smuggling, buying chips that are just under the legal limit of performance, and investing in their domestic chip industry.[1]

Sounds about right?

hotpot.ai/art-generator

This post centres around an email I sent to the Center for AI Safety (CAIS) expressing concern about their 2023-08-15 newsletter's coverage of US-China competition in the AI space[2], but the overall point is broader. There are some ways of discussing the topic of international relations regarding AI which strike me as un-nuanced in a counterproductive and dangerous way, by hiding certain truths or emphasising others, and supporting a conflict-oriented mindset.

In writing about this, I'm also gesturing at something about the more general topic of 'how to think and write about politically-charged topics'.

Jump to the summary if you are in a hurry.

This conversation really is important, which is why I think it's worth a public message discussing particular statements, but this should be understood as constructive criticism and part of a broader conversation which society, and especially the community of those focused on AI safety, needs to have. The CAIS newsletters are worth a (not unquestioning!) read, including the edition in question.

The particulars in this message serve as good exemplars of the problems and questions I have, and I'd be interested in responses from CAIS but even more so in remarks more broadly on the topic from anyone interested. The public conversation about this appears from my perspective to sometimes be broken. If that is so, I would like it to be rectified, and if not, I would like to be put right myself, the better to prioritise in my own work!

Specifics, CAIS case study

Here, quoted[3], is my response to the CAIS letter, which serves as an initial dialogue opener and the core of this post:

Hello,

I've been a supportive reader for some time and am myself an AI safety researcher. I'm generally very impressed and encouraged by your newsletters! - but I was disappointed and concerned by the phrasing (and mindset it can encourage) regarding US-China competition in the letter dated 16th August ('US-China Competition on AI Chips, ...').

In general I'm very wary of messaging which could inflame a them-vs-us mindset, in short mainly because I think it a) destroys humans' ability to think sensibly and b) ten

...
A New Bayesian Decision Theory
by Pareto Optimal — 2023-09-20T09:36:50.771Z
Link: https://paretooptimal.substack.com/p/a-new-bayesian-decision-theory

Two weeks ago I wrote an

introduction to normative decision theory

, yet I haven’t made clear what decision algorithm should be used. Before that, I’d like to give some preliminaries.

Generally speaking,

Newcomb's Problem

and the

Smoking Lesion Problem

represent counterexamples to

CDT

and

EDT

and show where they differ. But if causal chains and conditional probabilities consider one another then CDT and EDT are equivalent, causing one to choose the

dominance principle

(assuming probabilities are held constant) and thus two-boxing and smoking.

However, if probabilities update based on the agent's actions, such as in the

meta-Newcomb's problem

and

Parfit's Hitchhiker

, then CDT=EDT would recommend the maximization of expected utility and thus one-boxing and paying up respectively.

I don't think defining CDT = EDT is useful, so coming up with a new decision theory may be optimal. Without regard to labels or choosing a specific decision theory, I think one should choose the dominance principle if probabilities are held constant and thus choose smoking, two-boxing, etc.

But if probabilities aren’t constant, then one should pre-commit to an action that does not necessarily dominate (assuming constant probabilities) in exchange for an outcome that is (second?) best, e.g. pre-committing to pay Parfit’s biker.

This would be the kind of decision theory that recommends that one invoke EDT or

FDT

when probabilities can be updated by the agent, and invoke CDT otherwise. This would be the kind of decision theory that smokes, one-boxes, and doesn’t pay the biker ex-post, but “chooses to pay the biker ex-ante.” In other words, this would be the kind of decision theory that recommends decisions that maximize expected utility.

Of course, ideas are more important than labels, but labels still remain useful, and if I were to choose a label for such a decision procedure, given its relation to whether probabilities remain constant or update based on decisions, I would choose the name Bayesian Decision Theory (BDT).

Further thoughts and comments on decision theory are welcome.
Protest against Meta's irreversible proliferation (Sept 29, San Francisco)
by Holly_Elmore — 2023-09-19T23:40:30.202Z

Meta’s frontier AI models are fundamentally unsafe. Since Meta AI has released the model weights publicly, any safety measures can be removed. Before it releases even more advanced models – which will have more dangerous capabilities – we call on Meta to take responsible release seriously and stop irreversible proliferation. Join us for a peaceful protest at Meta’s office in San Francisco at 250 Howard St at 4pm PT.

RSVP on Facebook[1] or through this form.

Let’s send a message to Meta:

Stop irreversible proliferation of model weights. Meta’s models are not safe if anyone can remove the safety measures.
Take AI risks seriously.
Take responsibility for harms caused by your AIs.

All you need to bring is yourself and a sign, if you want to make your own. I will lead a trip to SF from Berkeley but anyone can join at the location. We will have a sign-making party before the demonstration-- stay tuned for details. We'll go out for drinks afterward

^

I like the irony.
The AI Explosion Might Never Happen
by snewman — 2023-09-19T23:20:25.597Z

[This is a crosspost from https://amistrongeryet.substack.com/p/recursive-self-improvement-foom, lightly edited for the LessWrong audience. This is my first LessWrong post; feedback greatly appreciated!]

LessWrong readers will be familiar with the concept of recursive self-improvement: as AIs become increasingly capable, they will acquire the ability to assist in their own development. Eventually, we will manage to create an AI that is slightly better than us at AI design. Since that system is better at AI design than its human creators, it should be able to design an AI better than itself. That second system should then be able to design its own improved successor, and so forth.

A lot of people seem to believe that, once AIs approach human capability, a “foom” loop of rapid self-improvement is more or less inevitable. But I don't think this is the case: even as AI capabilities increase, the effort required to achieve further improvements will likely also increase, and the resulting feedback loop might converge rather than diverge.

I’ll explore some of the factors at play, and list some early indicators we can watch for to indicate when we might be entering a period of rapid self-improvement.

A Parable

Not obviously on the cusp of a hard takeoff, but just wait

Once upon a time, there was a manufacturing company.

To begin with, it was a small, locally owned business. However, the owner had ambitions to create something larger. He used a small inheritance to hire several engineers to research improved manufacturing techniques.

Armed with these improved techniques, the company was able to produce their products faster, at a lower price. This led to an increase in both sales and profits.

The owner plowed those increased profits back into R&D, yielding further efficiency improvements. As the company’s manufacturing prowess continued to improve, the competition fell farther behind. Soon they were able to hire their pick of each year’s graduating class of engineering talent; no one else could pay as well, and besides, all of the best work was being done there.

Every year brought higher profits, more R&D, improved efficiency, increased sales, in an unending feedback loop. Eventually the company was manufacturing everything on Earth; no other firm could compete. Fortunately, there was no need to stop at mere world domination; such was the company’s prowess that they co...
Science of Deep Learning more tractably addresses the Sharp Left Turn than Agent Foundations
by NickGabs — 2023-09-19T22:06:42.355Z

Summary

Lots of agent foundations research is motivated by the idea that alignment techniques found by empirical trial-and-error will fail to generalize to future systems. While such threat models are plausible, agent foundations researchers have largely failed to make progress on addressing them because they tend to make very few assumptions about the gears-level properties of future AI systems. In contrast, the emerging field of "Science of Deep Learning" (SDL) assumes that the systems under question are neural networks, and aims to empirically uncover general principles about how they work. Assuming such principles exist and that neural nets will lead to AGI, insights from SDL will generalize to future systems even if specific alignment techniques found by trial-and-error do not. As a result, SDL can address the threat models that motivate agent foundations research while being much more tractable due to making more assumptions about future AI systems, though it cannot help in the construction of an ideal utility function for future AI systems to maximize.

Thanks to Alexandra Bates for discussion and feedback

Introduction

Generally speaking, the "agent foundations" line of research in alignment is motivated by a particular class of threat models: that alignment techniques generated through empirical trial and error like RLHF or Constitutional AI will suddenly break down as AI capabilities advance, leading to catastrophe. For example, Nate Soares has argued that AI systems will undergo a

"sharp left turn"

at which point the "shallow" motivational heuristics which caused them to be aligned at lower capabilities levels will cease to function. Eliezer Yudkowsky seems to endorse a similar threat model in

AGI Ruin

, arguing that rapid capabilities increases will cause models to go out of distribution and become misaligned.

I'm much more optimistic about finding robust alignment methods through trial and error than these researchers, but I'm not here to debate that. While I think sharp left turn style failure modes are relatively unlikely, they do not seem totally implausible. Moreover, because doom is much more likely in worlds where alignment techniques generated through trial and error fail to generalize, it makes sense to

focus on them

when doing alignment research. Instead, I want to argue that the emerging field of the "science of deep learning" o...
Formalizing «Boundaries» with Markov blankets
by Chris Lakin — 2023-09-19T21:01:01.901Z

How could «boundaries» be formally specified? Markov blankets seem to be one fitting abstraction.

[The post is largely a conceptual distillation of Andrew Critch’s

Part 3a: Defining boundaries as directed Markov blankets

.]

Explaining Markov blankets

By the end of this section, I want you to understand the following diagram (Pearlian causal diagram):

Also: I will assume a basic familiarity with Markov chains in this post.

First, I want you to imagine a simple Markov chain that represents the fact that a human influences itself over time:

Second, I want you to imagine a Markov chain that represents the fact that the environment[1] influences itself over time:

Okay. Now, notice that in between the human and its environment there’s some kind of boundary. For example, their skin (a physical boundary) and their interpretation/cognition (an informational boundary). If this were not a human but instead a bacterium, then the boundary I mean would (mostly) be the bacterium’s cell membrane.

Third, imagine a Markov chain that represents that boundary influencing itself over time:

Okay, so we have these three Markov chains running in parallel:

But they also influence each other, so let’s build that into the model, too.

How should they be connected?

Well, how does the environment affect a human?

Ok, so I want you to notice that when an environment affects a human, it doesn’t influence them directly, but instead it influences their skin or their cognition (their boundary), and then their boundary influences them.

For example, I shine light in your eyes (part of your environment), it activates your eyes (part of your boundary), and your eyes send information to your brain (part of your insides).

Which is to say, this is what does not happen:

(This is called “infiltration”.) The environment does not directly influence the human.

Instead, the environment influences the boundary which influences them, which looks like this:

The environment influences your skin and your senses, and your skin and senses influence you.

Okay, now let’s do the other direction. How does a human influence their environment?

It’s not that a human controls the environment directly…

(This is called “exfiltration”; this does not happen.)

…but that the human takes actions (via their boundary), and then their actions affect the environmen...
Precision of Sets of Forecasts
by niplav — 2023-09-19T18:19:18.053Z

I discuss proposals for a method that estimates how much predictive
information additional degrees of precision in forecasts add and at which
point additional precision is just noise, and investigate these proposals
with empirical forecasting data. I furthermore describe desirable criteria
such functions ought to fulfill. I conclude that methods based on rounding
probabilities are hot flaming garbage, methods based on rounding odds
or log-odds are regular garbage, and
methods based on applying noise to log-odds
are ok but sort of slow. Finding the maximal
informative precision turns out to be tricky.

Epistemic Status

Maybe not just reinventing the wheel, but the whole bicycle.

Say we have a set of resolved forecasts and can display them on a
calibration plot.

We can grade the forecasts according to some proper scoring
rule,
e.g. the Brier score
or the logarithmic scoring
rule,
maybe even broken up by calibration and resolution.

But we can also ask the question: how fine-grained are the predictions of
our forecaster? I.e., at which level of precision can we assume that
the additional information is just noise?

Overprecise Omar

Take, for example, a hypothetical forecaster Omar who always gives
their forecasts with 5 decimal digits of precision, such as forecasting
a "24.566% probability of North Korea testing an ICBM in the year
2022", even though if we look at their
calibration plot (of sufficiently many forecasts), we see that they
are pretty much random in any given interval of length 0.1 (i.e.,
their forecast with 15% and a forecast of 5% can be expected to
resolve to the same outcome with equal probability). This means that
4 of the 5 decimal digits of precision are likely just noise!

Omar would be behaving absurdly; misleading their audience into believing
they had spent much more time on their forecasts than they actually had
(or, more likely, into correctly leading the audience into believing
that there was something epistemically sketchy going on). It is certainly
useful to use probability resilience, and not imprecision, to communicate
uncertainty,
but there is an adequate & finite limit to precision.

I believe something similar is going on when people encounter
others putting probabilities on claims: It appears like an
attempt at claiming undue quantitativeness (quantitativity?) in
their reasoning, and at making the listener fall prey to
precisio...
The Proxy Political Party
by  — 2023-09-19T17:47:19.496Z
Link: https://antidefault.net/blog/the-proxy-political-party

The article here makes the point that it should be possible now to use software to enable a democracy that allows voters to stay updated on and actively participate in specific issues, rather than on a whole party identity. It suggests its software to be integrated into existing democratic processes, so that the voters can gain insight into them.

I agree on the first part. I disagree on the second.

Here's my suggestion on how to implement the vision.

THE PROXY POLITICAL PARTY (PPP)

Setup a political party that works as follows:

The PPP provides open source software that allows use by registered voters in the country of the PPP.
Users propose, discuss and vote on actions to be taken by the PPP.
Acting members of the PPP act and vote in the political system according to the users' votes.

Voting in the PPP allows for voting on specific issues. Voting for the PPP does not constitute voting for a specific ideology, but for additional political power for the PPP, and thus by proxy, for its users. If one or more acting members of the PPP decide to ignore the result of a vote, they get voted out. If that doesn't work, the PPP loses its trust and people stop voting for it, thus making it powerless.

There's nothing preventing the existence of several alternative PPPs with different acting members and different organizational processes.

THE SOFTWARE

Whatever suggestion I make here for the software to organize a PPP would probably not survive contact with reality. Nevertheless, here's what I'd like to see in the MVP:

It should be open source.
It should make sure that all registered voters can have access to it, but noone else.
It should provide tools for suggesting, discussing and voting on actions.
It should ensure the integrity of its data.
User anonymity might be of value, but I'm not too sure about that.

I suspect that specifics like the software giving insight into obsolete public debates or building up a user profile that can predict your votes are unnecessary. Continuously improved, collectively crafted and AI summarized legislative documents on the other hand do sound highly valuable. Do note that my personal opinion on this is unnecessary as well, as these specifics could be discussed and voted on in the context of a PPP.
Is there a publicly available list of examples of frontier model capabilities?
by Max Kearney — 2023-09-19T17:45:02.871Z

Is there a list (something analagous to the excellent list of examples of specification gaming by Victoria Krakovna) where examples of impressive capabilities of frontier models are compiled? And if not, can you provide your own best examples in response to this post?

Context:
I want to give a presentation explaining AI Safety to AI university students.
I want to mention the debate regarding whether current AI systems are "stochastic parrots", whether they "lack any real understanding", etc. As part of this, I want to clearly share examples of peak capabilities of current frontier models (e.g. something very impressive that GPT4 can consistently do), so that the audience has more information with which to explore this debate further themselves.
I think examples where it is impossible / very difficult to argue that "this is (likely) just because this was in GPT-4's training data" are best.
Currently, I am thinking to draw from the Sparks of AGI paper, the GPT-4 technical report and to make reference to 'SmartGPT' as a way of showing that greater "reasoning" abilities can be drawn from models like GPT-4, given the right prompt.
Anthropic's Responsible Scaling Policy & Long-Term Benefit Trust
by Zac Hatfield-Dodds — 2023-09-19T15:09:27.235Z
Link: https://www.anthropic.com/index/anthropics-responsible-scaling-policy

I'm delighted that Anthropic has formally committed to our responsible scaling policy.
We're also sharing more detail about the Long-Term Benefit Trust, which is our attempt to fine-tune our corporate governance to address the unique challenges and long-term opportunities of transformative AI.

Today, we’re publishing our Responsible Scaling Policy (RSP) – a series of technical and organizational protocols that we’re adopting to help us manage the risks of developing increasingly capable AI systems.

As AI models become more capable, we believe that they will create major economic and social value, but will also present increasingly severe risks. Our RSP focuses on catastrophic risks – those where an AI model directly causes large scale devastation. Such risks can come from deliberate misuse of models (for example use by terrorists or state actors to create bioweapons) or from models that cause destruction by acting autonomously in ways contrary to the intent of their designers.

Our RSP defines a framework called AI Safety Levels (ASL) for addressing catastrophic risks, modeled loosely after the US government’s biosafety level (BSL) standards for handling of dangerous biological materials. The basic idea is to require safety, security, and operational standards appropriate to a model’s potential for catastrophic risk, with higher ASL levels requiring increasingly strict demonstrations of safety.

A very abbreviated summary of the ASL system is as follows:

ASL-1 refers to systems which pose no meaningful catastrophic risk, for example a 2018 LLM or an AI system that only plays chess.

ASL-2 refers to systems that show early signs of dangerous capabilities – for example ability to give instructions on how to build bioweapons – but where the information is not yet useful due to insufficient reliability or not providing information that e.g. a search engine couldn’t. Current LLMs, including Claude, appear to be ASL-2.

ASL-3 refers to systems that substantially increase the risk of catastrophic misuse compared to non-AI baselines (e.g. search engines or textbooks) OR that show low-level autonomous capabilities.

ASL-4 and higher (ASL-5+) is not yet defined as it is too far from present systems, but will likely involve qualitative escalations in catastrophic misuse potential and autonomy.

The definition, criteria, and safety measures for each ASL level are described in...