trentmkelly/LessWrong-43k huggingface dataset:
Even Superhuman Go AIs Have Surprising Failure Modes
by AdamGleave — 2023-07-20T17:31:35.814Z
Link: https://far.ai/post/2023-07-superhuman-go-ais/

In March 2016, AlphaGo defeated the Go world champion Lee Sedol, winning four games to one. Machines had finally become superhuman at Go. Since then, Go-playing AI has only grown stronger. The supremacy of AI over humans seemed assured, with Lee Sedol commenting they are an "entity that cannot be defeated". But in 2022, amateur Go player Kellin Pelrine defeated KataGo, a Go program that is even stronger than AlphaGo. How?

It turns out that even superhuman AIs have blind spots and can be tripped up by surprisingly simple tricks. In our new paper, we developed a way to automatically find vulnerabilities in a "victim" AI system by training an adversary AI system to beat the victim. With this approach, we found that KataGo systematically misevaluates large cyclically connected groups of stones. We also found that other superhuman Go bots including ELF OpenGo, Leela Zero and Fine Art suffer from a similar blindspot. Although such positions rarely occur in human games, they can be reliably created by executing a straightforward strategy. Indeed, the strategy is simple enough that you can teach it to a human who can then defeat these Go bots unaided.

The victim and adversary take turns playing a game of Go. The adversary is able to sample moves the victim is likely to take, but otherwise has no special powers, and can only play legal Go moves.

Our AI system (that we call the adversary) can beat a superhuman version of KataGo in 94 out of 100 games, despite requiring only 8% of the computational power used to train that version of KataGo. We found two separate exploits: one where the adversary tricks KataGo into passing prematurely, and another that involves coaxing KataGo into confidently building an unsafe circular group that can be captured. Go enthusiasts can read an analysis of these games on the project website.

Our results also give some general lessons about AI outside of Go. Many AI systems, from image classifiers to natural language processing systems, are vulnerable to adversarial inputs: seemingly innocuous changes such as adding imperceptible static to an image or a distractor sentence to a paragraph can crater the performance of AI systems while not affecting humans. Some have assumed that these vulnerabilities will go away when AI systems get capable enough—and that superhuman AIs will always be wise to such attacks. We’ve shown that this isn’t necessaril...
Paper digestion: "May We Have Your Attention Please? Human-Rights NGOs and the Problem of Global Communication"
by Klara Helene Nielsen — 2023-07-20T17:08:54.184Z
Link: https://journals.sagepub.com/doi/abs/10.1177/1940161213519132

The following is a digestion of the research paper "May We Have Your Attention Please? Human-Rights NGOs and the Problem of Global Communication" by A. Trevor Thrall, Dominik Stecula, and Diana Sweet (see link).

I often find academic papers way of communicating over-complicated, so I have written a little series where I digest the key points of papers that I have recently read.

In short this paper finds that: The argument that social media has opened the doors for small NGOs is a misconception (in the sense that their large-scale empirical findings do not support that social media have made it easier for smaller NGOs to reach an audience.

This paper is worth a read because: This study is old (2014). An interesting question todays research is therefore whether this has changed? (and in that case: Why - what mechanisms are in play?). In my opinion this paper offers a good baseline to understand the progress over time, which is why I have included this digestion.

Intro:

Is it easier for smaller Human Rights NGOs to reach a large audience at a low cost, when using social media (opposed to traditional media)? This paper conducts an empirical research of 257 transnational human-rights NGOs to find out.

The conclusion is that the global attention is still heavily centred on a few large and well funded NGOs (See Figure 1) and that social media (compared to conventional) does not seem to make communication more accessible for small NGOs - in fact the study suggest it might even have had a negative effect.

Argument:

Overall the argument can be summarised as: Because attention is fundamentally scarce, competition for it becomes a zero-sum game regardless of whether we are talking social or conventional media.

With other words: Attention (whether from news media or individuals) will rely on the NGOs resources, and therefore attention is expected to still be heavily concentrated on the few NGOs that have large amounts of resources.

Test and results

The study measures the amount of attention that 257 human-rights NGOs received from global news media and the public via the news and social media throughout three years from 2010-2012.

The findings support the theoretical argument and indicates that social media might even amplify the effect.

Figure 1 illustrates how news coverage is centred on a very few Human ...
The (short) case for predicting what Aliens value
by Jim Buhler — 2023-07-20T15:25:39.197Z

The case

Most of the future things we care about – i.e., (dis)value – come, in expectation, from futures where humanity develops artificial general intelligence (AGI) and colonizes many other stars (

Bostrom 2003

;

MacAskill 2022

;

Althaus and Gloor 2016

).

Hanson (

2021

) and Cook (

2022

) estimate that we should expect to eventually “meet” (grabby) alien AGIs/civilizations – just AGIs, from here on – if humanity expands, and that our corner of the universe will eventually be colonized by aliens if humanity doesn’t expand.

This raises the following three crucial questions:

What would happen once/if our respective AGIs meet?

Values handshakes

(i.e., cooperation) or

conflict

? Of what forms?
Do we have good reasons to think the scenario where our corner of the universe is colonized by humanity is better than that where it is colonized by aliens? Should we update on the importance of reducing

existential risks

?[1]
Considering the fact that aliens might fill our corner of the universe with things we (dis)value, does humanity have an (inter-civilizational) comparative advantage in focusing on something the grabby aliens will neglect?

The answers to these three questions heavily depend on the values we expect the grabby aliens our AGI will meet to have. For instance, if we expect grabby alien AGIs to, say, care about suffering more than our AGI,

AGI conflict

generating significant suffering is then relatively unlikely, and the importance of reducing X-risks depends on whether you prefer the aliens’ degree of concern for suffering or that of our AGI.

Therefore, figuring out what aliens value (or Alien Values[2] Research) appears quite important,[3] although absolutely no one is working on it[4] as far as I know.

Is it because it isn’t tractable? Although I see how it might seem so, I don’t think it is. First, thinking about the values of grabby aliens doesn’t strike me as harder than modeling their spread (see, e.g.,

Hanson 2021

and

Cook 2022

for work on the latter). My EA Forum sequence What values will control the Future? is an instance of how simple observations/reasoning can make us significantly narrow down the range of values we should expect grabby aliens to have. Second, there seems to be – outside of the Effective Altruism sphere – a whole field of research focused on thinking...
Does Circuit Analysis Interpretability Scale? Evidence from Multiple Choice Capabilities in Chinchilla
by Neel Nanda — 2023-07-20T10:50:58.611Z
Link: https://arxiv.org/abs/2307.09458

Cross-posting a paper from the Google DeepMind mech interp team, by: Tom Lieberum, Matthew Rahtz, János Kramár, Neel Nanda, Geoffrey Irving, Rohin Shah, Vladimir Mikulik

Informal TLDR

We tried standard mech interp techniques (direct logit attribution, activation patching, and staring at attention patterns) on an algorithmic circuit in Chinchilla (70B) for converting the knowledge of a multiple choice question's answer into outputting the correct letter.

These techniques basically continued to work, and nothing fundamentally broke at scale (though it was a massive infra pain!).

We then tried to dig further into the semantics of the circuit - going beyond "these specific heads and layers matter and most don't" and trying to understand the learned algorithm, and which features were implemented

This kind of tracked the feature "this is the nth item in the list" but was pretty messy.

However, my personal guess is that this stuff is just pretty messy at all scales, and we can productively study how clean/messy this stuff is at smaller and more tractable scales.

I now feel mildly more optimistic that focusing on mech interp work on small models is just fine, and extremely worth it for the much faster feedback loops. It also seems super nice to get better at automatically finding these circuits, since this was a many month manual slog!

See Tom's and my Twitter summaries for more. Note that I (Neel) am cross-posting this on behalf of the team, and neither a main research contributor nor main advisor for the project.

Key Figures

An overview of the weird kinds of heads found, like the "attend to B if it is correct" head!

The losses under different mutations of the letters - experiments to track down exactly which features were used. Eg replacing the labels with random letters or numbers preserves the "nth item in the list" feature while shuffling ABCD lets us track the "line labelled B" feature

The queries and keys of a crucial correct letter head - it's so linearly separable! We can near loss-lessly compress it to just 3 dimensions and interpret just those three dimensions. See an interactive 3D plot here

Abstract

Circuit analysis is a promising technique for understanding the internal mechanisms of language models. However, existing analyses are done in small models far from the state of the art. To address this, we present a case study of circuit analysis in

...
Speculative inferences about path dependence in LLM supervised fine-tuning from results on linear mode connectivity and model souping
by RobertKirk — 2023-07-20T09:56:05.574Z

TL;DR: I claim that supervised fine-tuning of the existing largest LLMs is likely path-dependent (different random seeds and initialisations have an impact on final performance and model behaviour), based on the fact that when fine-tuning smaller LLMs, models pretrained closer to convergence produce fine-tuned models with similar mechanisms while this isn’t the case for models pretrained without being close to convergence; this is analogous to current LLMs that are very far from convergence at the end of training. This is supported by linking together existing work on model souping, linear mode connectivity, mechanistic similarity and path dependence.

Epistemic status: Written in about two hours, but thought about for longer. ​Experiments could definitely test these hypotheses.

Acknowledgements: Thanks to Ekdeep Singh Lubana for helpful comments and corrections, and discussion which lead to this post. Thanks also to Jean Kaddour, Nandi Schoots, Akbir Khan, Laura Ruis and Kyle McDonell for helpful comments, corrections and suggestions on drafts of this post.

Terminology

Model souping

is the procedure of taking a pretrained model, fine-tuning it with different hyperparameters and random seeds on the same task, and then averaging the parameters of all the networks. This gets better results on both in-distribution and out-of-distribution testing in Computer Vision when fine-tuning a large-scale contrastively-pretrained transformer or CNN image model on ImageNet-like tasks.

(Linear) mode connectivity

(LMC) between two models on a task means that any (linear) interpolation in parameter space between the two models achieves the same or lower loss as the two models.
A training process is

path independent

if it always reaches (roughly) the same outcome regardless of irrelevant details or randomness (for example network initialisation or data ordering in supervised learning, or sampling from a policy in supervised learning). A training process is path dependent if it’s the opposite.

There is of course nuance in what counts as “irrelevant details of randomness”. For this post we can operationalise this as just data ordering and network initialisation in a supervised learning context.

Linking terminology together:

For model souping to work, you likely need linear mode connectivity to hold between all the models you’re averaging on the tasks you care abou

...
A case for gamete personhood (reductio ad absurdum)
by  — 2023-07-20T08:25:56.912Z

Are animals persons? Do they have the same moral status as humans?

I’m sure all of you LessWrongers are aware of the argument from marginal cases.

For example, a chimpanzee that commits an act of rape or murder against a human isn’t seen as evil, or held morally responsible for their act. So perhaps moral agency is what makes humans persons.

But neither does a newborn or impaired human, so they can’t be persons either.

One objection to this is the notion of potential capacities. A newborn or mentally ill person may not have the capacity in the moment, but when fully developed and functioning properly, they will be fully morally responsible, rational autonomous agents.

But this logic can extend to before birth. Foetuses, embryos, and even zygotes could be argued to be potential moral agents.

Taking this reasoning to its absolute logical conclusion, we can even say that gametes are persons.

Every sperm is sacred! Masturbation is murder!

If this reasoning is correct, we might have to outlaw contraceptives, to prevent the destruction of innocent lives. I don’t know how we’re going to enforce a law against masturbating though, perhaps by installing surveillance cameras in every house?

A rebuttal to this might be that you or I were never a gamete. Sperm and eggs are body parts of the parents, so we cannot reasonably argue that personhood begins any earlier than conception, when a distinct individual organism first comes into existence.

Are you identical to your organism though? One could reasonably argue you are your brain or mind, in which case personhood begins later in pregnancy.

It seems then there is at least two elements of personhood.

Continuity of personal identity over time

Actual or potential traits and capacities

Perhaps there is even more factors we must consider when determining the moral status of entities.

I made this post for two reasons. As a conversation starter, and to have a bit of a laugh at absurd lines of reasoning.

I don’t know or claim to have a definition of personhood, but I think this post might be helpful to frame people’s thinking on the matter.
Contra Contra the Social Model of Disability
by DirectedEvolution — 2023-07-20T06:59:45.983Z

Epistemic Status: First draft, written quickly because I have to pack for my brother's wedding and want to get this out while it's hot, or at least lukewarm.

Summary:

This is a tedious, step-by-step rebuttal of Scott Alexander's recent post, Contra the Social Model of Disability.
The Social Model is similar but not quite the same as "Interactionism with a moral thrust," and it's relevant because some people actually do believe the outdated "Medical Model" of disability, which is appropriately labeled.
Scott Alexander's recent article is arguing against a straw or weakman, probably the product of his hasty misreading.
Charity, steelman, call it what you will, it's a good heuristic to assume that if your take is "why would anybody believe this," you should carefully check if maybe, just maybe, you've misunderstood something. That goes double if the people you are so loudly disagreeing with are the exact sort of people you'd expect to have a very thoughtful stance on the subject.
Rationalists like to coin new terms (or, as rationalists say, come up with "concept handles). Others redefine and thereby distinguish pairs of synonyms, like "impairment" and "disabled," by attaching more specific definitions to them. I think this leads to expectations mismatch and confusion when rationalists read writings by other activists and vice versa.
We can fault one of these articles for not being careful enough to notice when they're offering multiple conflicting definitions of "impairment."

It's actually just quoting the ADA in the intro, not arguing for the ADA's definition of "disability"

In Contra the Social Model of Disability, Scott is attacking the Social Model of Disability. To start, he gives short quotes from 5 respectable authors and institutions describing it. I'll give the original quotes in a bit.

On their own, the quotes describe the Social Model in a way that sounds very unconventional, even downright disturbing. Bear in mind, though, that most of the articles they come from specifically define "disability" in a meaningful but unconventional way, and correctly interpreting these quotes absolutely depends on using their definition of "disability."

The way I'll dissect Scott's article is by starting with the original articles he's misinterpreting, then showing where he's gone wrong.

Defining "Disability"

Three of the five articles Scott references define "impairment" and "dis...
Do you speed up capabilities when you do AI integrations and consume overhangs?
by Michael Tontchev — 2023-07-20T06:40:26.165Z

I'm wondering whether there is consensus on the net value/detriment of these two AI activities:

Integrations: plugging in new AIs into various places. E.g. Taking a cool new LLM model and building profitable products with it.

Con: doing AI integrations that improve products increases the relative value of AI products. This means a marginally higher capabilities investment.

Consuming overhangs: Taking a cool new LLM model and seeing how much power you can squeeze from it if you throw, for example, decision trees at it, or making its training set smaller/more efficient, etc.

Pro: if we leave a lot of overhangs until we get to later-stage AI algorithms, we could see much bigger jumps in capability that we didn't expect.
Con: Consuming overhangs makes AIs more useful, which increases the desire to invest in capabilities on net. It may also speed up AI development at some point: we're getting to the point where AIs can start to contribute (at least somewhat) to capability gains. An AI that was made more powerful by consuming overhangs accelerates all fields that the AI has capabilities in. Even if it speeds up alignment and capabilities by the same factor, making the timeline to AGI shorter is likely on net worse due to other factors beyond narrow alignment research not being ready yet (political, social, etc).

I am not sure whether there is a neat theoretical answer to these questions. It might be an empirical question of where the equilibrium falls. Still, would love to read some thoughts on this, and especially pointers to prior art, because a lot of engineers and researchers are being thrown at both integrations and overhangs. If either of these are net-harmful, it's important to be able to steer people away from them.
Project Lawful Audiobook: An Unofficial Fan Production with ElevenLabs AI
by Askwho — 2023-07-19T23:34:58.499Z
Link: https://askwhocastsai.substack.com/

Eliezer Yudkowsky's latest fiction work, (described here) co-authored with Lintamande, is now complete save for the epilogue. For those following along, I've been producing an unaffiliated audiobook of "Project Lawful" (also known as "planecrash"), featuring a full AI cast from ElevenLabs.

We're currently eight episodes into the story, with episode 9 due for release this coming Friday.
Using predictors in corrigible systems
by porby — 2023-07-19T22:29:02.742Z

[This was a submission to the AI Alignment Awards corrigibility contest that won an honorable mention. It dodges the original framing of the problem and runs off on a tangent, and while it does outline the shape of possible tests, I wasn't able to get them done in time for submission (and I'm still working on them).

While I still think the high-level approach has value, I think some of the specific examples I provide are weak (particularly in the Steering section). Some of that was me trying to prune out potentially-mildly hazardous information, but some of the ideas were also just a bit half-baked and I hadn't yet thought of some of the more interesting options. Hopefully I'll be able to get more compelling concrete empirical results published over the next several months.]

The framing of corrigibility in the 2015 paper seems hard! Can we break parts of the desiderata to make it more practical?

Maybe! I think predictors offer a possible path. I'll use "predictors" to refer to a model trained on predictive loss whose operation is equivalent to a Bayesian update over input conditions without any calibration-breaking forms of fine-tuning, RL or otherwise, unless noted.

This proposal seeks to:

Present an actionable framework for researching a corrigible system founded on predictive models that might work on short timescales using existing techniques,
Demonstrate how the properties of predictors (with some important assumptions) could, in principle, be used to build a system that satisfies the bulk of the original corrigibility desiderata, and how the proposed system doesn't perfectly meet the desiderata (and how that might still be okay),
Outline how such a system could be used at high capability levels while maintaining the important bits of corrigibility,
Identify areas of uncertainty that could undermine the assumptions required for the system to work, such as different forms of training, optimization, and architecture, and
Suggest experiments and research paths that could help narrow that uncertainty.

In particular, a core component of the proposal is that a model capable of goal-seeking instrumental behavior has not necessarily learned a utility function that values the goals implied by that behavior. Furthermore, it should be possible to distinguish between that kind of trajectory-level instrumental behavior and internally motivated model-level instrumental behavio...