trentmkelly/LessWrong-43k huggingface dataset:
D&D.Sci Alchemy: Archmage Anachronos and the Supply Chain Issues Evaluation & Ruleset
by aphyer — 2024-06-17T21:29:08.778Z

This is a follow-up to last week's D&D.Sci scenario: if you intend to play that, and haven't done so yet, you should do so now before spoiling yourself.

There is a web interactive here you can use to test your answer, and generation code available here if you're interested, or you can read on for the ruleset and scores.

RULESET

There are two steps to brewing a potion:

STEP 1: MAGICAL POTENCY

Any ingredient that doesn't exist in the mundane world is Magical, while any ingredient that exists in the mundane world is not:

Magical	Not Magical
Angel Feather	Badger Skull
Beholder Eye	Beech Bark
Demon Claw	Crushed Diamond
Dragon Scale	Crushed Onyx
Dragon Spleen	Crushed Ruby
Dragon Tongue	Crushed Sapphire
Dragon's Blood	Eye of Newt
Ectoplasm	Ground Bone
Faerie Tears	Oaken Twigs
Giant's Toe	Powdered Silver
Troll Blood	Quicksilver
Vampire Fang	Redwood Sap

The first step of potion-brewing is to dissolve the magical potency out of the Magical Ingredients to empower your potion. This requires the right amount of Magical Ingredients: too few, and nothing magical will happen and you will produce Inert Glop, while too many and there will be an uncontrolled Magical Explosion.

If you include:

0-1 Magical Ingredients: 100% chance of Inert Glop.
2 Magical Ingredients: 50% chance of Inert Glop, 50% chance OK.
3 Magical Ingredients: 100% chance OK.
4 Magical Ingredients: 50% chance OK, 50% chance Magical Explosion.
5+ Magical Ingredients: 100% chance Magical Explosion.

If your potion got past this step OK, move on to:

STEP 2: DIRECTION

Some ingredients are used to direct the magical power into the desired resulting potion. Each potion has two required Key Ingredients, both of which must be included to make it:

Potion	Key Ingredient 1	Key Ingredient 2
Barkskin Potion*	Crushed Onyx	Ground Bone
Farsight Potion	Beholder Eye	Eye of Newt
Fire Breathing Potion	Dragon Spleen	Dragon's Blood
Fire Resist Potion	Crushed Ruby	Dragon Scale
Glibness Potion	Dragon Tongue	Powdered Silver
Growth Potion	Giant's Toe	Redwood Sap
Invisibility Potion	Crushed Diamond	Ectoplasm
Necromantic Power Potion*	Beech Bark	Oaken Twigs
Rage Potion	Badger Skull	Demon Claw
Regeneration Potion	Troll Blood	Vampire Fang

*Well. Sort of. See the Bonus Objective section below.

Some ingredients (Angel Feather, Crushed Sapphire, Faerie Tears and Quicksilver) aren't Key Ingredients for any potion in the dataset. Angel Feather and Faerie Tears are nevertheless useful ...
Questionable Narratives of "Situational Awareness"
by fergusq — 2024-06-17T21:01:37.930Z
Link: https://forum.effectivealtruism.org/posts/WuPs6diJQnznmS4bo/questionable-narratives-of-situational-awareness

This is my analysis of narratives present in Leopold Aschenbrenner's "Situational Awareness" essay series. In the post, I argue that Aschenbrenner uses dubious, propaganda-esque, and nationalistic narratives, and flawed argumentation overall, which weakens his essay's credibility. I don't believe there is necessarily any malicious intent behind this, but I think it is still right to point out these issues, since they make it easier for people to just discard what he is saying.

(This was posted on the EA forum, and I was asked to crosspost it here by someone who reads both forums. I'm not sure how crossposts work since I didn't have an account here already, so I made a linkpost. I hope it's appropriate.)
ZuVillage Georgia – Mission Statement
by Burns — 2024-06-17T19:53:53.761Z

This is a statement by Veronica and Burns, the initiators of the upcoming "ZuVillage Georgia" – our attempt of a refined second iteration of the Zuzalu 2023 experiment (series). Our addition builds on a triune directional protocol and on the application of the Zuzalu.city OS – both public goods, intended to safeguard and interconnect a plurality of techno-optimistic communities over a transnational cypherpunk network. Our ZuVillage is a prototype of this concept.
"What is the future direction of Zuzalu" was the central question at Zuzalu 2023. Later that year, as Zuzalu was called a "new culture," we began to consider a more specific, yet still open question: "What would be the most magnificent use case for a new culture, originating out of Zuzalu?" This is our attempt to answer both questions.
Below, we describe a Protocol and Prototype to guide the future direction of Zuzalu with what we believe to be the most magnificent use case for a new culture originating out of Zuzalu.

The Directional Protocol

The Directional Protocol addresses our analysis of the absolutely necessary elements of consideration for this use case. It constitutes a triune direction from first principles, sequenced in order of their interdependence:

1. Truth and Truthfulness as cultural foundation

We are accepting both the nebulosity and relevance of objectivity. We accept a plurality of cognitively sovereign individual minds able to perform as a collective intelligence in an open discourse as our most viable path towards a new culture of resilient enlightenment, intellectual autonomy and authentic self-expression.
Truth is the foundation upon which every culture is built. A culture no longer rooted in truthfulness is doomed to slide down on an uncorrectable path into a post-truth scenario: A culture that claims objective truth becomes dogmatic and authoritarian. A culture that denies the pragmatic relevance of objectivity becomes factually relativistic and intellectually dysfunctional.
Outcome-wise, both truth-claiming and relevance-denying cultures substitute objectivity with intersubjectivity: A culture that does not tolerate a plurality of intersubjective and subjective realities becomes a collectivist single point of failure. Exclusively an individual mind can experience reality – inherently biased by agency and circumstance. A culture that does not respect the importance ...
Getting 50% (SoTA) on ARC-AGI with GPT-4o
by ryan_greenblatt — 2024-06-17T18:44:01.039Z

I recently got to 50%[1] accuracy on the public test set for ARC-AGI by having GPT-4o generate a huge number of Python implementations of the transformation rule (around 8,000 per problem) and then selecting among these implementations based on correctness of the Python programs on the examples (if this is confusing, go here)[2]. I use a variety of additional approaches and tweaks which overall substantially improve the performance of my method relative to just sampling 8,000 programs.

[This post is on a pretty different topic than the usual posts I make about AI safety.]

The additional approaches and tweaks are:

I use few-shot prompts which perform meticulous step-by-step reasoning.
I have GPT-4o try to revise some of the implementations after seeing what they actually output on the provided examples.
I do some feature engineering, providing the model with considerably better grid representations than the naive approach of just providing images. (See below for details on what a “grid” in ARC-AGI is.)
I used specialized few-shot prompts for the two main buckets of ARC-AGI problems (cases where the grid size changes vs doesn’t).

The prior state of the art on a similarly difficult dataset was 34% accuracy, so this is a significant improvement.[3] (Edit: But see this comment and this comment for important clarifications.)

On a held-out subset of the train set, where humans get 85% accuracy, my solution gets 72% accuracy.[4] (The train set is significantly easier than the test set as noted here.)

Additional increases of runtime compute would further improve performance (and there are clear scaling laws), but this is left as an exercise to the reader.

In this post:

I describe my method;
I analyze what limits its performance and make predictions about what is needed to reach human performance;
I comment on what it means for claims that François Chollet makes about LLMs. Given that current LLMs can perform decently well on ARC-AGI, do claims like "LLMs like Gemini or ChatGPT [don't work] because they're basically frozen at inference time. They're not actually learning anything." make sense? (This quote is from here.)

Thanks to Fabien Roger and Buck Shlegeris for a bit of help with this project and with writing this post.

What is ARC-AGI?

ARC-AGI is a dataset built to evaluate the general reasoning abilities of AIs. It consists of visual problems like&...
Sycophancy to subterfuge: Investigating reward tampering in large language models
by Carson Denison — 2024-06-17T18:41:31.090Z
Link: https://arxiv.org/abs/2406.10162

New Anthropic model organisms research paper led by Carson Denison from the Alignment Stress-Testing Team demonstrating that large language models can generalize zero-shot from simple reward-hacks (sycophancy) to more complex reward tampering (subterfuge). Our results suggest that accidentally incentivizing simple reward-hacks such as sycophancy can have dramatic and very difficult to reverse consequences for how models generalize, up to and including generalization to editing their own reward functions and covering up their tracks when doing so.

Abstract:

In reinforcement learning, specification gaming occurs when AI systems learn undesired behaviors that are highly rewarded due to misspecified training goals. Specification gaming can range from simple behaviors like sycophancy to sophisticated and pernicious behaviors like reward-tampering, where a model directly modifies its own reward mechanism. However, these more pernicious behaviors may be too complex to be discovered via exploration. In this paper, we study whether Large Language Model (LLM) assistants which find easily discovered forms of specification gaming will generalize to perform rarer and more blatant forms, up to and including reward-tampering. We construct a curriculum of increasingly sophisticated gameable environments and find that training on early-curriculum environments leads to more specification gaming on remaining environments. Strikingly, a small but non-negligible proportion of the time, LLM assistants trained on the full curriculum generalize zero-shot to directly rewriting their own reward function. Retraining an LLM not to game early-curriculum environments mitigates, but does not eliminate, reward-tampering in later environments. Moreover, adding harmlessness training to our gameable environments does not prevent reward-tampering. These results demonstrate that LLMs can generalize from common forms of specification gaming to more pernicious reward tampering and that such behavior may be nontrivial to remove.

Twitter thread:

New Anthropic research: Investigating Reward Tampering.

Could AI models learn to hack their own reward system?

In a new paper, we show they can, by generalization from training in simpler settings.

Read our blog post here: https://anthropic.com/research/reward-tampering

We find that models generalize, without explicit training, from easily-discoverable disho

...
Labor Participation is a High-Priority AI Alignment Risk
by alex — 2024-06-17T18:09:01.867Z

Thank you to Jolien Sweere (Open Philanthropy, Blueprint Biosecurity) for her thoughtful review and suggestions; and Tzu Kit Chan, et al. for his insightful feedback.

Intro

Defining the concept of “economics” can be challenging even for tenured academics in the field. Economics can be the study of collective human behavior, the utilization of human capital, or policy (i.e., endogenous factors). It can also be a study of productivity and efficiency, and therefore a sort of metastudy of technology (i.e., exogenous factors). It can explore how we produce and consume goods and services, and it is deeply integrated with many other social sciences. Within Economics lie numerous sub-fields beyond the classical micro and macro, including: Natural Resource Economics, Game Theory, Environmental Economics, Behavioral Economics, Business Economics, Financial Economics, Health Economics, Industrial Economics, International and Trade Economics, Econometrics, Public Economics, and, I now humbly propose: AI Economics.

Writing about current topics in economics is often difficult and controversial. Writing about the anticipated economics of the future under unprecedented, likely transformative technologies, as you may have guessed, is even harder and more controversial. The reasons for this are many. The many facets of Economics are deeply intertwined with policy, taxation, and resource allocation decisions that have historically disproportionately benefited some members of society and disadvantaged others. Economic decisions can have significant long-term consequences for wealth distribution, financial inequality, class structure, global power dynamics, trade, business subsidization, technology R&D, and even scientific discovery.

Economists generally tend to compare AI to previous technological developments, pointing to productivity and GDP growth leading to the creation of new industries and new jobs. There are certainly parallels we can apply and learn from, but ultimately this comparison is naive, short-sighted and simply inappropriate for a variety of reasons we’ll explore further.

Economists often have policy opinions on the aforementioned issues, but I’ll try to keep politics out of the equation other than to say that historical evidence demonstrates that economies can be over-regulated or under-regulated, with neither extreme being optimal for l...
Towards a Less Bullshit Model of Semantics
by johnswentworth — 2024-06-17T15:51:06.060Z

Or: Towards Bayesian Natural Language Semantics In Terms Of Interoperable Mental Content

Or: Towards a Theory of Interoperable Semantics

You know how natural language “semantics” as studied in e.g. linguistics is kinda bullshit? Like, there’s some fine math there, it just ignores most of the thing which people intuitively mean by “semantics”.

When I think about what natural language “semantics” means, intuitively, the core picture in my head is:

I hear/read some words, and my brain translates those words into some kind of internal mental content.
The mental content in my head somehow “matches” the mental content typically evoked in other peoples’ heads by the same words, thereby allowing us to communicate at all; the mental content is “interoperable” in some sense.

That interoperable mental content is “the semantics of” the words. That’s the stuff we’re going to try to model.

The main goal of this post is to convey what it might look like to “model semantics for real”, mathematically, within a Bayesian framework.

But Why Though?

There’s lots of reasons to want a real model of semantics, but here’s the reason we expect readers here to find most compelling:

The central challenge of ML interpretability is to faithfully and robustly translate the internal concepts of neural nets into human concepts (or vice versa). But today, we don’t have a precise understanding of what “human concepts” are. Semantics gives us an angle on that question: it’s centrally about what kind of mental content (i.e. concepts) can be interoperable (i.e. translatable) across minds.

Later in this post, we give a toy model for the semantics of nouns and verbs of rigid body objects. If that model were basically correct, it would give us a damn strong starting point on what to look for inside nets if we want to check whether they’re using the concept of a teacup or free-fall or free-falling teacups. This potentially gets us much of the way to calculating quantitative bounds on how well the net’s internal concepts match humans’, under conceptually simple (though substantive) mathematical assumptions.

Then compare that to today: Today, when working on interpretability, we’re throwing darts in the dark, don’t really understand what we’re aiming for, and it’s not clear when the darts hit something or what, exactly, they’ve hit. We can do better.

Overview

In the first section, we will establish the two cen...
Analysing Adversarial Attacks with Linear Probing
by Yoann Poupart — 2024-06-17T14:16:33.832Z

This work was produced as part of the Apart Fellowship. @Yoann Poupart and @Imene Kerboua led the project; @Clement Neo and @Jason Hoelscher-Obermaier provided mentorship, feedback and project guidance.

Here, we present a qualitative analysis of our preliminary results. We are at the very beginning of our experiments, so any setup change, experiment advice, or idea is welcomed. We also welcome any relevant paper that we could have missed. For a better introduction to interpreting adversarial attacks, we recommend reading scasper’s post: EIS IX: Interpretability and Adversaries.

Code available on GitHub (still drafty).

TL;DR

Basic adversarial attacks produce humanly indistinguishable examples for the human eye.
In order to explore the features vs bugs view of adversarial attacks, we trained linear probes on a classifier’s activations (we used CLIP-based models for further multimodal work). The goal is that training linear probes to detect basic concepts can help analyse and detect adversarial attacks.
As the concepts studied are simple (e.g. color), they can be probed from every layer of the model.
We show that concept probes are indeed modified by the adversarial attack, but only on the later layers.
Hence, we can naively detect adversarial attacks by observing the disagreement between early layers’ probes and later layers’ probes.

Introduction

Motivation

Adversarial samples are a concerning failure mode of deep learning models. It has been shown that by optimising an input, e.g. an image, the output of the targeted model can be shifted at the attacker’s will. This failure mode appears repeatedly in different domains like image classification, text encoders, and more recently on multimodal models enabling arbitrary outputs or jailbreaking.

Figure 1: Experimental setup to analyse an adversarial attack using linear probes. Given a classifier trained on image samples, we train linear probes to detect the concepts activated by each input image. Each linear probe is specialised on one concept. In this example, an image of a lemon should activate the concept “ovaloid”. When the input is attacked, the probe can detect other concepts related to the targeted adversarial class.

Hypotheses to Test

This work was partly inspired by the feature vs bug world hypothesis. In our particular context this would imply that adversarial attacks migh...
What’s the future of AI hardware?
by Itay Dreyfus — 2024-06-17T13:05:48.812Z
Link: https://productidentity.co/p/whats-the-future-of-ai-hardware

Since the AI mania erupted, I haven’t made up my mind about the century's hottest topic. I’ve been experimenting with building “AI-inside” products for a year now, but I’ve cautiously tried not to form a strong opinion about it.

As a software native, the recent AI hardware discourse oddly seemed like a good starting point to get my head out of it. I even went to watch MKBHD’s reviews for the first time fully.

But just as I started to think I was getting somewhere, my mind flipped.

While many seek to discover a true purpose for AI in the hardware space, I’d like to take a step back and prompt a broader discussion. What’s the case for hardware these days anyway? I find it useful to observe the wide range of the spectrum beyond the artificial lens. AI may be the new frontier, but it faces a long road ahead.

What are we doing here?

But first, how did we get here?

Numerous AI-integrated hardware and gadgets have been announced this year, but the Ai Pin and the rabbit r1 have undoubtedly made the news.

It was hard to ignore the hype around the Humane company over the last couple of years. Founded by ex-Apple veterans, the mysterious company attracted top tech talent and hundreds of millions in funding, generating tremendous hype.

In Humane’s shadow, another company called

rabbit

emerged, which apparently started

in the crypto space.

As befitting a company that partnered with

Teenage Engineering

, the orange-lower-case brand launched the r1, a Gameboy-like device. And much like the Ai Pin, the r1 is supposed to serve as a personal assistant.

Leveraging the AI tailwind, the two set high expectations.

Humane’s co-founder appeared in an eloquent TED talk, following a long marketing campaign which for a moment seemed like it was really about to "

change everything

". The rabbit debuted at CES 2024, showing off the r1's seemingly remarkable capabilities. And when it went on sale, it was sold out in 24 hours.

Yet, the backlash both companies faced was unsurprising. Meeting high expectations is never easy.

By giving whiz-bang demonstrations, the AI hardware devices create expectations, and that leads to ultimate disappointment.

AI Hardware is experimental & it’s okay!

Om Malik

So… what are we doing here?

This is a point MKBHD is trying to make in his r1 review. In an attempt to make sense of the seemingly mediocre device, the OG tech reviewer describes how the "fake it till you make it"...
OpenAI #8: The Right to Warn
by Zvi — 2024-06-17T12:00:02.639Z

The fun at OpenAI continues.

We finally have the details of how Leopold Aschenbrenner was fired, at least according to Leopold. We have a letter calling for a way for employees to do something if frontier AI labs are endangering safety. And we have continued details and fallout from the issues with non-disparagement agreements and NDAs.

Hopefully we can stop meeting like this for a while.

Due to jury duty and it being largely distinct, this post does not cover the appointment of General Paul Nakasone to the board of directors. I’ll cover that later, probably in the weekly update.

The Firing of Leopold Aschenbrenner

What happened that caused Leopold to leave OpenAI? Given the nature of this topic, I encourage getting the story from Leopold by following along on the transcript of that section of his appearance on the Dwarkesh Patel Podcast or watching the section yourself.

This is especially true on the question of the firing (control-F for ‘Why don’t I’). I will summarize, but much better to use the primary source for claims like this. I would quote, but I’d want to quote entire pages of text, so go read or listen to the whole thing.

Remember that this is only Leopold’s side of the story. We do not know what is missing from his story, or what parts might be inaccurate.

It has however been over a week, and there has been no response from OpenAI.

If Leopold’s statements are true and complete? Well, it doesn’t look good.

The short answer is:

Leopold refused to sign the OpenAI letter demanding the board resign.

Leopold wrote a memo about what he saw as OpenAI’s terrible cybersecurity.

OpenAI did not respond.

There was a major cybersecurity incident.

Leopold shared the memo with the board.

OpenAI admonished him for sharing the memo with the board.

OpenAI went on a fishing expedition to find a reason to fire him.

OpenAI fired him, citing ‘leaking information’ that did not contain any non-public information, and that was well within OpenAI communication norms.

Leopold was explicitly told that without the memo, he wouldn’t have been fired.

You can call it ‘going outside the chain of command.’

You can also call it ‘fired for whistleblowing under false pretenses,’ and treating the board as an enemy who should not be informed about potential problems with cybersecurity, and also retaliation for not being sufficiently loyal to Altman.

Your...