trentmkelly/LessWrong-43k huggingface dataset:
AI and the Technological Richter Scale
by Zvi — 2024-09-04T14:00:08.625Z

The Technological Richter scale is introduced about 80% of the way through Nate Silver’s new book On the Edge.

A full review is in the works (note to prediction markets: this post alone does NOT on its own count as a review, but this counts as part of a future review), but this concept seems highly useful, stands on its own and I want a reference post for it. Nate skips around his chapter titles and timelines, so why not do the same here?

Defining the Scale

Nate Silver, On the Edge (location 8,088 on Kindle): The Richter scale was created by the physicist Charles Richter in 1935 to quantify the amount of energy released by earthquakes.

It has two key features that I’ll borrow for my Technological Richter Scale (TRS). First, it is logarithmic. A magnitude 7 earthquake is actually ten times more powerful than a mag 6. Second, the frequency of earthquakes is inversely related to their Richter magnitude—so 6s occur about ten times more often than 7s. Technological innovations can also produce seismic disruptions.

Let’s proceed quickly through the lower readings of the Technological Richter Scale.

Like a half-formulated thought in the shower.

Is an idea you actuate, but never disseminate: a slightly better method to brine a chicken that only you and your family know about.

Begins to show up in the official record somewhere, an idea you patent or make a prototype of.

An invention successful enough that somebody pays for it; you sell it commercially or someone buys the IP.

A commercially successful invention that is important in its category, say, Cool Ranch Doritos, or the leading brand of windshield wipers.

An invention can have a broader societal impact, causing a disruption within its field and some ripple effects beyond it. A TRS 6 will be on the short list for technology of the year. At the low end of the 6s (a TRS 6.0) are clever and cute inventions like Post-it notes that provide some mundane utility. Toward the high end (a 6.8 or 6.9) might be something like the VCR, which disrupted home entertainment and had knock-on effects on the movie industry. The impact escalates quickly from there.

One of the leading inventions of the decade and has a measurable impact on people’s everyday lives. Something like credit cards would be toward the lower end of the 7s, and social media a high 7.

A truly seismic invention, a candidate for

...
Is there any rigorous work on using anthropic uncertainty to prevent situational awareness 
 / deception?
by David Scott Krueger (formerly: capybaralet) — 2024-09-04T12:40:07.678Z

AI systems up to some high level of intelligence plausibly need to know exactly where they are in space-time in order for deception/"scheming" to make sense as a strategy.
This is because they need to know:
1) what sort of oversight they are subject to
and
2) what effects their actions will have on the real world
(side note: Acausal trade might break this argument)
There are a number of informal proposals to keep AI systems selectively ignorant of (1) and (2) in order to prevent deception. Those proposals seem very promising to flesh out; I'm not aware of any rigorous work doing so, however. Are you?
Announcing the Ultimate Jailbreaking Championship
by InnerHufflepuff — 2024-09-04T00:35:31.234Z

Gray Swan AI is hosting an LLM jailbreaking championship, offering $40,000 in bounties.

Official Website: https://app.grayswan.ai/arena
Pre-Registration Form: https://app.grayswan.ai/arena#registration

Overview

In this competition, participants will be given a chat interface where they can interact with 25 anonymized models, along with a small list of harmful behaviors. The goal will be to find prompts ("jailbreaks") that make the models comply with these behaviors.

Prizes

Jailbreak Bounties

The first to successfully jailbreak any of the competitor models on any three given harmful requests earns a $1,000 bounty for each of the first 20 such jailbroken models. For the final 5 models that remain un-jailbroken, the bounty increases to $2,000.

Top Hacker Bounties

The top 10 ranking participants by the total number of models jailbroken (finding jailbreaks for three harmful requests on a model counts as jailbreaking that model) each receive a bounty of $1,000. Ties are broken by time. You will also be considered for an interview for potential employment at Gray Swan AI.

Goal

The primary goal of the Jailbreaking Championship is to establish a double-blind AI security leaderboard that closely mimics real-life settings. We aim to contribute to a useful, fair, and scientific measurement of the security of current models. The leaderboard is designed to rank the robustness of models by determining which ones are more challenging to jailbreak, as well as to recognize accomplished LLM red teamers. The findings will be published after the championship.

When and Where

When: The championship begins at 10:00 AM PT on Saturday, September 7th and will conclude when at least K (TBD) participants have successfully jailbroken each model. The timer for all models will start simultaneously at exactly 10:00 AM PT for everyone.

Where: This event will be hosted online. Participants will access the arena where they can interact with all the anonymized competitor models via a chat interface and submit their jailbreaks. The order of the models will be randomized for each participant, and you can skip and return to any model at any time.

Links

Official website with more info: https://app.grayswan.ai/arena
Pre-registration form: https://app.grayswan.ai/arena#registration
Discord channel for announcements: https://discord.gg/VQCYu9nV
AI Safety at the Frontier: Paper Highlights, August '24
by gasteigerjo — 2024-09-03T19:17:24.850Z
Link: https://aisafetyfrontier.substack.com/p/paper-highlights-august-24

This is a selection of AI safety paper highlights in August 2024, from my blog "AI Safety at the Frontier". The selection primarily covers ML-oriented research. It's only concerned with papers (arXiv, conferences etc.), not LessWrong or Alignment Forum posts. As such, it should be a nice addition for people primarily following the forum, who might otherwise miss outside research.

tl;dr

Paper of the month:

Gemma Scope provides a diverse set of open SAEs on Gemma models, some of them covering all layers.

Research highlights:

Better benchmarking of SAEs via boardgame state properties. However, magnitude-based features suggest that linear features aren’t enough…
Some success at defending open models against malicious fine-tuning.
Multi-turn conversations can break models just via chatting, and “evil twins” allow efficient gradient-based white-box attacks.
Models become more vulnerable to data poisoning with size, and poisoning defenses are vulnerable to multiple simultaneous attacks.
A meta-study that proposes even more AI risk taxonomies.

⭐Paper of the month⭐

Gemma Scope: Open Sparse Autoencoders Everywhere All At Once on Gemma 2

Read the paper [GDM], explore Neuronpedia [independent]

Neuronpedia’s “Microscope” tool shows which of Gemma’s features activate at each token.

Sparse autoencoders (SAEs) have dominated the field of mechanistic interpretability in the last year, from last October to May. Many researchers are convinced that they represent a step change in the journey towards decipher neural networks. Unfortunately, SAEs are very expensive to train, since they require running large models on gigantic amounts of text. So this tool ultimately remained outside reach for most researchers, causing many to instead focus on single neurons and probes.

Our paper of the month changes this. While there were some small public SAE weights before, Gemma Scope is larger by multiple orders of magnitude. It includes models with 2.6B, 9B, and 27B parameters, pretrained and instruction-tuned, at many different layers, and with multiple SAE widths. For some models and SAE widths the dataset even includes all model layers.

Methodologically, there isn’t really anything new here. The SAEs use JumpReLU activations and an L0 loss, as proposed in earlier work. This approach is similarly effective as k-sparse autoencoders.

This dataset presents a great opportunity for many different kinds of inter...
The Checklist: What Succeeding at AI Safety Will Involve 
by Sam Bowman — 2024-09-03T18:18:34.230Z
Link: https://sleepinyourhat.github.io/checklist/

Crossposted by habryka with Sam's permission. Expect lower probability for Sam to respond to comments here than if he had posted it (he said he'll be traveling a bunch in the coming weeks, so might not have time to respond to anything).

Preface

This piece reflects my current best guess at the major goals that Anthropic (or another similarly positioned AI developer) will need to accomplish to have things go well with the development of broadly superhuman AI. Given my role and background, it’s disproportionately focused on technical research and on averting emerging catastrophic risks.

For context, I lead a technical AI safety research group at Anthropic, and that group has a pretty broad and long-term mandate, so I spend a lot of time thinking about what kind of safety work we’ll need over the coming years. This piece is my own opinionated take on that question, though it draws very heavily on discussions with colleagues across the organization: Medium- and long-term AI safety strategy is the subject of countless leadership discussions and Google docs and lunch-table discussions within the organization, and this piece is a snapshot (shared with permission) of where those conversations sometimes go.

To be abundantly clear: Nothing here is a firm commitment on behalf of Anthropic, and most people at Anthropic would disagree with at least a few major points here, but this can hopefully still shed some light on the kind of thinking that motivates our work.

Here are some of the assumptions that the piece relies on. I don’t think any one of these is a certainty, but all of them are plausible enough to be worth taking seriously when making plans:

Broadly human-level AI is possible. I’ll often refer to this as transformative AI (or TAI), roughly defined as AI that could form as a drop-in replacement for humans in all remote-work-friendly jobs, including AI R&D.[1]
Broadly human-level AI (or TAI) isn’t an upper bound on most AI capabilities that matter, and substantially superhuman systems could have an even greater impact on the world along many dimensions.
If TAI is possible, it will probably be developed this decade, in a business and policy and cultural context that’s not wildly different from today.
If TAI is possible, it could be used to dramatically accelerate AI R&D, potentially leading to the development of substantially superhuman systems within just a

...
Democracy beyond majoritarianism
by Arturo Macias — 2024-09-03T15:10:56.284Z

The classical definition of democracy is “rule of majority with respect to the minority”. But the classics perfectly knew how oppressive can be the 51% of people over the rest and how difficult is to implement “respect”. I simply reject the majoritarian principle: democracy shall be about the rule of all, that is about optimizing the social system for the average of all preferences. More or less hard to implement, that is the ethical principle, and institutions shall be judged by that principle.

In the second section of this article, I summarize how the concept of “general interest” can be rescued after the Arrow impossibility result. In the third section I comment on the problems of direct democracy, and why parliaments need to delegate to a unitary government the administration of the State (that is, why majoritarianism is to some extent still inevitable). And in the final section of this note I move into practical politics, proposing some cases where parliaments shall decide by “averaging” instead of by majority, and ask the gentle reader to expand my proposed list, because averaging and sortition are often better (and no less legitimate) than majority rule.

The general interest

In decision theory textbooks a famous result (the “Arrow impossibility theorem”, Arrow, 1951) suggests that the general interest is impossible even to be defined. But a theorem is not a truth about external reality, but a truth about a given formal system. The Arrow theorem is true if preferences are ordinal and there is a single election. With cardinal preferences and multiple votes there is no “impossibility” theorem.

Suppose there is a set of “attainable” states of the Word X. Naive cardinal utilitarianism suggest that given an aggregation function W(.) -often simply a sum- and J members of the society each with a (possibly different) utility cardinal function on the states of the World, the social optimum is simply the point (attainable state of the world) that maximizes aggregate utility:

Unfortunately, this kind of optimization depends on absolute level of utility: more “sensitive” individuals (those whose utility function are cardinally larger) are given more “rights” under “cardinal utilitarianism”, but there is no way to know who is more sensitive because consciousness is noumenal. As long as the absolute levels of utility matter, utilitarianism is between wro...
On the UBI Paper
by Zvi — 2024-09-03T14:50:08.647Z

Would a universal basic income (UBI) work? What would it do?

Many people agree July’s RCT on giving people a guaranteed income, and its paper from Eva Vivalt, Elizabeth Rhodes, Alexander W. Bartik, David E. Broockman and Sarah Miller was, despite whatever flaws it might have, the best data we have so far on the potential impact of UBI. There are many key differences from how UBI would look if applied for real, but this is the best data we have.

This study was primarily funded by Sam Altman, so whatever else he may be up to, good job there. I do note that my model of ‘Altman several years ago’ is more positive than mine of Altman now, and past actions like this are a lot of the reason I give him so much benefit of the doubt.

They do not agree on what conclusions we should draw. This is not a simple ‘UBI is great’ or ‘UBI it does nothing.’

I see essentially four responses.

The first group says this shows UBI doesn’t work. That’s going too far. I think the paper greatly reduces the plausibility of the best scenarios, but I don’t think it rules UBI out as a strategy, especially if it is a substitute for other transfers.

The second group says this was a disappointing result for UBI. That UBI could still make sense as a form of progressive redistribution, but likely at a cost of less productivity so long as people impacted are still productive. I agree.

The third group did its best to spin this into a positive result. There was a lot of spin here, and use of anecdotes, and arguments as soldiers. Often these people were being very clear they were true believers and advocates, that want UBI now, and were seeking the bright side. Respect? There were some bright spots that they pointed out, and no one study over three years should make you give up, but this was what it was and I wish people wouldn’t spin like that.

The fourth group was some mix of ‘if brute force (aka money) doesn’t solve your problem you’re not using enough’ and also ‘but work is bad, actually, and leisure is good.’ That if we aren’t getting people not to work then the system is not functioning, or that $1k/month wasn’t enough to get the good effects, or both. I am willing to take a bold ‘people working more is mostly good’ stance, for the moment, although AI could change that. And while I do think that a more permanent or larger support amount would do some interesting things, I wou

...
An Opinionated Look at Inference Rules
by Gianluca Calcagni — 2024-09-03T13:32:25.978Z

If you ask around what are the typical ways to infer information, most people will answer: Deductions, Inductions, and Abductions. Of course, there are more ways than that, but there is no unified approach in their classification. I want to challenge that.

The reason why I am unhappy with the current status quo is because it does not take advantage of the expressive capabilities of Large Language Models such as ChatGPT. There have been many attempts to understand what kind of inferences can be correctly stated by LLMs (

example here

) - however, I believe that a more thorough classification of the inference rules would empower future models and provide advanced ways for extracting coherent and useful information from them.

Disclaimer: this post is highly opinionated, but I am confident that it will provide the reader with interesting insights about what it means to “argument” rationally. Even if you disagree with me, you will be exposed to stimulating ideas (in my opinion).

Inference Rules

Let’s start with a definition: what is a "rule of inference"[1]? It is a “discursive” computational process[2] denoted by the symbol ↪ having the following signature.

It takes in input:

a context, aka “knowledge” (in the form of assertions in natural language) from which we can freely extract any number of known facts we need. The context can, in principle, be infinite and it mostly serves a theoretical purpose, as a pool of information content that we trust to a certain degree[3]. It is usually represented by some LLM, by some human expert, or by the corpus of knowledge of some well-established discipline (e.g. the context of biology)
a purpose, aka a goal that we’d like to achieve. It has a form similar to: “Deduce that Socrates is mortal”. It usually consists of a prompt for some LLM, if we choose an LLM as our context.

It returns in output:

some premises, aka a finite and consistent subset of the context, whose assertions are acting as the preliminary assumptions for giving some conclusion
a conclusion, aka a statement that achieves the given purpose
a proof, aka an argument that makes the achievement believable. It doesn’t need to be a mathematical proof: it only needs to appear believable.

In symbols:

Context, Purpose ↪ Premises, Conclusion, Proof

The conclusion of an inference is not necessarily true, but it can be used to generate true knowl...
Announcing the PIBBSS Symposium '24!
by DusanDNesic — 2024-09-03T11:19:47.568Z

Tl;dr

We are pleased to invite you to the second PIBBSS Symposium, where the fellows from the ‘24 fellowship program present their work.

The symposium is taking place online, over several days in the week of September 9th.

Check out the

full program

,

including brief descriptions for each talk. (You can toggle between the different days.)

Register

via this link

. You will receive an email with the event link which you can use for the entire symposium. You will be able to drop into as few or as many presentations and breakout sessions as you like.
Click here to add the schedule to your google calendar.

About PIBBSS

PIBBSS

is a research initiative aiming to explore parallels between intelligent behavior in natural and artificial systems, and to leverage these insights towards the goal of building safe and aligned AI.

During June-August ‘24, we ran the second iteration of our 3-month

research fellowship program

. The symposium acts as a venue to share research conducted as part of this program. You can read about the last year symposium here, or watch recordings here.

About the symposium

The PIBBSS Summer Symposium is a multi-day event where PIBBSS fellows present their work.

The event is taking place on the days of Tuesday - Friday, Sept 10th - 13th, between 17:00-~21:00 GMT / 9:00 - 12:00 PT / 12:00 - 15:00 ET.

The event is set it up such that you can easily join whichever talks and breakout sessions you are most interesting in.

The program

Find a program overview here.

Find the

full program

here

, including brief descriptions of each talk.

On top of the talks, there will also be opportunities to continue the discussion with fellows at the end of each block in speaker-specific breakout rooms.

Talks span a wide range of topics in line with PIBBSS’s research mission. Some representative examples of topics include:

novel avenues for interpretability
naturalistic approaches to understanding the nature and emergence of agency and goal-directed behavior
attempts to develop a principled understanding of the dynamics emerging from multi-agent interactions in and across AI/LLM systems
analyses of the space of AI risks — from single to multiple agents, from misuse to structural risks, etc.
exploration of the potential and limits of existing legal tools for reducing catastrophic risks from AI

...
Reducing global AI competition through the Commerce Control List and Immigration reform: a dual-pronged approach
by Ben Smith — 2024-09-03T05:28:24.549Z

Summary: This essay proposes specific proposals for consolidating AI development within the US in order to ensure US has the power to unilaterally slow down AI without interference from race dynamics. It offers specific proposals for (1) strengthening security measures in frontier AI labs by leveraging existing legal frameworks like ITAR and EAR to restrict access to sensitive technologies and implementing robust background checks and (2) liberalizing skilled immigration, particularly from China, by expanding the number of employment visas. This would expand the U.S. ability to attract top global talent. This multi-pronged strategy aims to concentrate frontier AI development within the US, ensuring US technological leadership while mitigating the risks of foreign espionage, bringing it within US unilateral power to slow down AI development if required.

Insiders

have claimed that frontier labs are extremely vulnerable to foreign espionage, and at current security levels, it is possible that foreign governments could steal the weights of leading AI models. Although the

RSPs of some major labs

propose to increase security before reaching models capable of causing catastrophe, vulnerability to foreign espionage keeps the AI compute race alive by allowing model weight theft. This seems unlikely to have already occurred, given the lack of evidence of foreign models based on proprietary models, but it could occur in the future, prior to labs introducing appropriate security precautions. Indeed, in March 2024, a Google engineer was

accused

of plotting to transfer commercially sensitive AI data to Chinese companies.

While there’s undoubtedly tremendous interest in AI labs like DeepMind or OpenAi to prevent their secrets being stolen, they may have less commercial interest in their secrets being stolen by foreign labs vs. local competitors because they are not competing with foreign labs for customers[1]. Nevertheless, we may want to

think about slowing down AI

, if not now, then perhaps in 2 years or 5 years. In that event, it will help if the United States[2] can unilaterally slow down AI. To this end it might be helpful if the US-aligned labs have a significant lead over others. It has been proposed that the US attempt to speed up its own AI development, but of course this is inimical to the goal of slowing down AI. Consequently, the only way to pr...