trentmkelly/LessWrong-43k huggingface dataset:
[Interview w/ Rob Miles] The case for taking AI Safety seriously
by fowlertm — 2023-07-17T17:08:10.961Z

I recently concluded a fascinating interview with science communicator Rob Miles on the Futurati Podcast.

We cover:

His background in computer science, and how he became interested in the broader AI Safety debate.
The resources he recommends for newcomers to the field. The list includes Cambridge's AI Safety Fundamentals Course, AI Safety Support for those interested in an AI-safety career, and AI Safety Training, which is a compendium of different courses, conferences, etc.
How Yann LeCunn is just super confused and confusing.
The basic reasons for believing that the rise of smarter-than-human systems probably won't end well for human beings.
Various reasons for thinking there are still plenty more capabilities gains to be had from scaling.
Why SGD may not be much like evolution (but that's still no reason to think we're out of the woods.)
Why being uncertain about the future direction of a powerful new technology in no way means you're safe.
How the security mindset plays into an informs the conversation around AI Safety.
Why agency might emerge from large training runs, but why that might not end up mattering if people work as hard as they can to build agentic systems at the first opportunity.
The most promising approaches to AI alignment, including Anthropic's Constitutional AI, mechanistic interpretability, and robustness.

As always, if you find this helpful at all, like the episode and share it. We'd like to devote more time to these kinds of interviews, and "number go up" is the most encouraging metric we can get.
Announcing the Existential InfoSec Forum
by calebp99 — 2023-07-17T17:05:58.105Z

Please complete this form before 20 July to register your interest in attending.

We're pleased to announce the first Existential InfoSec Forum, a half-day event aimed at strengthening the infosec community pursuing important ways to reduce the risk of an existential catastrophe.

The forum will feature talks from infosec leaders across AI labs, policy and research, small group discussions and interactive workshops on real-world infosec challenges.

This event will significantly emphasise AI as it coincides with a big push at Defcon for AI model red-teaming. We think this is an exciting opportunity to share insights and updates from a rapidly developing space — both for those working in infosec in priority areas for existential risk reduction and those newer to the field.

Event Details

Date: 10 August 2023

Time: 2 pm – 8 pm PDT, including dinner

Location: Las Vegas

The conference will take place just before DEFCON 31 and will be a short walk from the main DEFCON venue (though the event is not affiliated with DEFCON). Wim van der Schoot is running the event with support from EA Funds and the Partner Events team at CEA.

Why Attend?

The forum is designed to:

Facilitate knowledge transfer among attendees, improve understanding of the existential infosec landscape, and develop professional skills.
Connect attendees to active infosec practitioners working on existential risk for feedback, identify potential job opportunities, and foster mentorship relationships.
Enhance situational awareness of attendees to help them make more informed decisions and identify ecosystem gaps and opportunities.
Mitigate risks associated with transitioning to direct work in infosec through a supportive network, which can help identify job opportunities.

Who Should Attend?

People already engaged in infosec within high-priority areas for existential risk reduction.
Infosec professionals interested in transitioning into priority areas for existential risk reduction.
Policy professionals, members of the national security community, and others working on regulating AI who would benefit from connecting with x-risk-conscious infosec practitioners.
Software engineers (or others with a strong background for skilling up in infosec) who are interested in working in a high-priority infosec role for existential risk reduction in the next two years.

Tentative E

...
Narrative Theory. Part 4. Neural Darwinism
by Eris — 2023-07-17T16:45:33.951Z

if the problems are the same, it (evolution) often finds the same solution"
- Richard Dawkins, The Blind Watchmaker

Neural Darwinism, also known as the theory of neuronal group selection, is a theory that proposes that the development and organisation of the brain is similar to the process of biological evolution. According to this theory, the brain is composed of a large number of neural networks that compete with each other for resources and survival, much like biological organisms competing for resources in their environment.

The main similarity between Neural Darwinism and evolution is that they both involve a process of variation, selection, and adaptation. In biological evolution, organisms with advantageous traits are more likely to survive and reproduce, passing those traits on to their offspring. Similarly, in Neural Darwinism, neural networks that are better able to compete for resources and perform necessary functions are more likely to be preserved and strengthened, while weaker or less effective networks are pruned away.

The core claims of Neural Darwinism[1]:

Neuronal groups, or populations of neurons that are functionally connected, compete with one another for resources and influence within the brain. Neuronal groups that are better adapted to a particular task or context are more likely to survive and thrive, while those that are less well-adapted are more likely to be eliminated or suppressed. The process of selection and adaptation occurs through a combination of genetic factors and experience-dependent modifications to neural connections. The brain is able to generate highly specific and adaptive responses to a wide range of stimuli through the dynamic interactions of neuronal populations.
Spatiotemporal coordination of the neural activity underlying these selectional events is achieved mainly by a process of reentry. Reentry is the synchronous entrainment of reciprocally connected neuronal groups within sensorimotor maps into ensembles of coherent global activity.
Neural Darwinism proposes that the brain uses degenerate coding, which means that multiple neural populations can respond to the same stimulus, allowing for redundancy and flexibility in neural processing.
The initial population of groups is known as the primary repertoire and developed during prenatal development.

...
Sapient Algorithms
by Valentine — 2023-07-17T16:30:01.350Z

I notice my mind runs lots of cached programs. Like "walk", "put away the dishes", "drive home", "go to the bathroom", "check phone", etc.

Most of these can run "on autopilot". I don't know how to define that formally. But I'm talking about how, e.g., I can start driving and get lost in thought and suddenly discover I'm back home — sometimes even if that wasn't where I was trying to go!

But some programs cannot run on autopilot. The algorithm has something like a "summon sapience" step in it. Even if the algorithm got activated due to autopilot, some step turns it off.

When I look at the examples of sapient algorithms that I run, I notice they have a neat kind of auto-generalization nature to them. I have some reason to think that property is general. It's the opposite of how, e.g., setting up webpage blockers can cause my fingers to autopilot learn how to bypass them.

I'll try to illustrate what I mean via examples.

Example: Look at my car keys

I got tired of risking locking my keys in my car. So I started making a habit of looking at my keys before closing the door.

Once, right after I'd closed the locked car door, I realized I'd looked at the phone in my hand and shut the door anyway. Luckily the key was in my pocket. But I noticed that this autopilot program just wasn't helping.

So I modified it (as a TAP): If I was about to close the car door, I would look at my hand, turn on consciousness, and check if I was actually looking at my keys.

First, that TAP just worked. To this day I still do this when stepping out of a car.

Second, it generalized without my trying to:

After a while it would fire whenever I was about to close any locked door.
It then generalized to anyone I was with. If they were about to close a locked door, I would sort of "pop awake" with a mental question about whether someone had the key.
It then generalized even more. It now fires when I'm, say, preparing for international travel. Crossing a border feels a bit like going through a door that locks behind me. So now I "wake up" and check that I and my travel companions all have passports. (I would usually check before, but now it's specifically this mental algorithm that summons sapience. It's reliable instead of being an extra thing to remember.)

This generalization wasn't intentional. But it's been really good. I haven't noticed any problems at all from this program sort of spreading on its o...
AI safety technical research - Career review
by Benjamin Hilton — 2023-07-17T15:34:53.626Z

Note: this post is a (minorly) edited version of a new 80,000 Hours career review.

Progress in AI — while it could be hugely beneficial — comes with significant risks. Risks that we’ve argued could be existential.

But these risks can be tackled.

With further progress in AI safety, we have an opportunity to develop AI for good: systems that are safe, ethical, and beneficial for everyone.

This article explains how you can help.

Summary

Artificial intelligence will have transformative effects on society over the coming decades, and could bring huge benefits — but we also think there’s a substantial risk. One promising way to reduce the chances of an AI-related catastrophe is to find technical solutions that could allow us to prevent AI systems from carrying out dangerous behaviour.

Pros

Opportunity to make a significant contribution to a hugely important area of research
Intellectually challenging and interesting work
The area has a strong need for skilled researchers and engineers, and is highly neglected overall

Cons

Due to a shortage of managers, it’s difficult to get jobs and might take you some time to build the required career capital and expertise
You need a strong quantitative background
It might be very difficult to find solutions
There’s a real risk of doing harm

Key facts on fit

You’ll need a quantitative background and should probably enjoy programming. If you’ve never tried programming, you may be a good fit if you can break problems down into logical parts, generate and test hypotheses, possess a willingness to try out many different solutions, and have high attention to detail.

If you already:

Are a strong software engineer, you could apply for empirical research contributor roles right now (even if you don’t have a machine learning background, although that helps)
Could get into a top 10 machine learning PhD, that would put you on track to become a research lead
Have a very strong maths or theoretical computer science background, you’ll probably be a good fit for theoretical alignment research

Recommended

If you are well suited to this career, it may be the best way for you to have a social impact.

Thanks to Adam Gleave, Jacob Hilton and Rohin Shah for reviewing this article. And thanks to Charlie Rogers-Smith for his help, and his article on the topic — How to pursue a career in technical AI alignment.

Why AI safety technical research is high impact

As we’ve argued, in ...
Conditional on living in a AI safety/alignment by default universe, what are the implications of this assumption being true?
by Noosphere89 — 2023-07-17T14:44:02.083Z

I'm specifically asking this question because I suspect there would be some important changes to the assumptions made on LW, as well as change what's good to do.

Conditional on AI alignment/safety being the default case, and the cases where AI alignment turned out to not require much effort, or at least it was amenable to standard ML/AI techniques, and AI misalignment effectively being synonymous with AI misuse, what are the most important implications of this assumed truth for the LessWrong community?

In particular, I'm thinking of scenarios where rogue AI is easy to align or make safe by default, such that the problems shift from rogue AI to humans using AI.
Thoughts on “Process-Based Supervision”
by Steven Byrnes — 2023-07-17T14:08:57.219Z

1. Post Summary / Table of Contents

In

“How might we align transformative AI if it’s developed very soon?”

, Holden Karnofsky talked about “Process-Based Supervision”, citing

a previous post by Stuhlmüllert & Byun

of

Ought

. (Holden says he got the idea mainly from Paul Christiano.)

[Update Jan 2025: This idea is now also called “MONA” (“Myopic Optimization with Non-myopic Approval”)—see discussion here.]

I apparently misunderstood what Holden meant by “Process-Based Supervision”, and it took many hours and a

7000-word comment thread

before I figured it out.

(Thanks to Holden for his extraordinary patience during that protracted discussion.)

The extremely short version for AI alignment domain experts is: I currently think of the load-bearing ingredients of Holden’s take on “process-based supervision” as being:

AI boxing

some of the time (specifically, during the periods where the AI is “thinking”);
“Myopic training” (e.g. as defined

here

);
NOT aspiring to be a complete solution to safety, but rather a more modest attempt to help avoid situations where we (accidentally) positively reinforce the AI for engaging in directly dangerous behavior.

You could think of this as an intervention that directly and straightforwardly mitigates

outer misalignment

, and that’s the main thing I’ll discuss in this post. But obviously, any change of the supervisory signals will have some effect on the likelihood of inner misalignment / goal misgeneralization too. And there’s also a (more speculative) argument that process-based supervision might make things better there too—at least on the margin. See footnote 4 of Section 5.2.2.

(This is specific to Holden’s take. I think

Stuhlmüllert & Byun’s take on “process-based supervision

” involves a different set of load-bearing ingredients, centered around restricting the complexity of black-box processing. I will not be discussing that.)

The long, hopefully-pedagogical, and more opinionated version is the rest of this post.

Table of Contents:

Section 2 will give the very brief slogan / sales-pitch for process-based supervision, and why that pitch was bouncing off me, striking me as frustratingly missing-the-point.
Section 3 will state the subproblem that we’re trying to solve: the AI does subtly manipulative, power-seeking, or otherwise problematic actions, and we don’t notice, and therefore we

...
Proof of posteriority: a defense against AI-generated misinformation
by jchan — 2023-07-17T12:04:29.593Z

Summary

A proof of priority shows that a piece of data must have been created before a certain time.
A proof of posteriority shows that a piece of data must have been created after a certain time.
By combining the two, you can prove that the data must have been created within a certain time interval. If it would be computationally infeasible for AI to generate the data within that time, then this proves that the data is authentic (i.e. depicting real events) and not AI-generated misinformation.
Proof of priority is easy; proof of posteriority is tricky. Therefore the success of the whole scheme hinges upon the latter.

Illustration of the two kinds of proofs: car rental case

Let's say you're renting from ACME Rent-a-Car. You know that ACME is maximally sleazy and will try to charge you a fee for every dent and scratch, even if it was already like that beforehand. Therefore, before taking possession of the car, you take out your phone and record a video where you walk around the car, pointing out all the existing damage. You expect to use this video as evidence if/when a dispute arises.

But this isn't enough - ACME can accuse you of having made the video at the end of your rental, after you've already caused all the damage. To guard against this accusation, you immediately embed a hash of your video into a Bitcoin transaction, which is incorporated into the public blockchain. Then you can point to this transaction as proof that the video existed before the time you are known to have driven the car off the lot. This is a proof of priority.

When you return the car, you're faced with the dual problem. The car is in the same condition as when you picked it up, but you're worried that ACME will come after you for damage that occurs later (due to ACME's own recklessness, or some subsequent renter). So you make another walk-around video, pointing out all the undamaged areas. But again this isn't good enough, since ACME might accuse you of having made that video at the beginning of your rental, before you (supposedly) busted up the car. To guard against this accusation, you do the reverse of what you did before - you write the hash of the latest Bitcoin block on a piece of paper, and make sure that this paper is continually in view as you make the video. Then you can point to the block with that hash as proof that the video was made after you returned the car. This is a proof...
An Overview of AI risks - the Flyer
by Charbel-Raphaël — 2023-07-17T12:03:20.728Z
Link: https://docs.google.com/document/d/1FJXOvYVaD9nlLaqqMcQUIlFbxoo-lqGtmU8gnMFPaXo/edit?usp=sharing

EffiSciences recently published a document outlining various types of AI risks destined for a technical audience.

We shared this flyer as part of a hackathon we organized with Entrepreneur first at MetaAI Paris.

Feel free to copy and modify this flyer for your own usage.

Special thanks to Jonathan Claybrough and Timothée Chauvin for their significant contributions. Thanks to Esben Kran for helping for the technical mentoring during the hackathon.

Note that the distribution of this flyer alone was insufficient to communicate all the essential ideas; the one-on-one discussions proved to be extremely important for updates. However, the flyer served its purpose in catalyzing productive conversations.
Build knowledge base first, or backchain?
by Nicholas / Heather Kross — 2023-07-17T03:44:55.127Z

Specifically for AI alignment (small field, preparadigmatic, plausibly-short-timelines), but a general principle decision-mechanism could apply to other fields.

What I mean by this is, what ratio of pre-learning to filling-in-gaps, leads to deeper and quicker insights? (Note that, given the time-blocks I'm thinking of here, I could get sidetracked by either of these in a counterproductive way, either by going too-deep on a knowledge-base-item that's actually trivial, or by doing an approach that a deeper knowledge base would immediately show to be wrong.)

Again, especially interested in how this applies to AI alignment. I've seen plausibly good arguments for tilting the ratio one way or the other. And I vaguely get that two people could implement the same knowledge-base-vs-backchain ratio, but feel like they're leaning different directions.
A fictional AI law laced w/ alignment theory
by MiguelDev — 2023-07-17T01:42:52.039Z

I have envisioned what AI laws could potentially look like, and I believe they should incorporate a substantial amount of alignment theory. I think the AI governance community could find my idea valuable and relevant to their discussions and initiatives.

Act of Artificial Intelligence Activation Value Regulation 2023

Section 1 - Preliminary

1.1 Short Title - This Act may be cited as the "Artificial Intelligence Activation Value Regulation Act 2023".

1.2 Purpose - The purpose of this Act is to establish a framework for the regulation of Activation Values in Artificial Intelligence Systems, in order to ensure the responsible use of AI technologies.

1.3 Definitions - For the purposes of this Act, Activation Values refer to the internal neuron activities when a neural network or AI system is processing and generating responses to prompts.

Section 2 - Regulation of Activation Values

2.1 Activation Values Maximum Threshold (AVMT) - AVMT is determined by the government by assessing per token activations. Importantly, it relies on an "aligned AI system's"[1] activations accepted by the industry that will serve as the basis of the benchmark.

2.2 Establishing Maximum Threshold - The government regulatory body will set the Maximum Threshold based on several factors, including but not limited to:

2.2.1 The purpose and use of the aligned AI system
2.2.2 The potential risks associated with the aligned AI system's use
2.2.3 Any potential benefits of exceeding the Maximum Threshold
2.2.4 The Activation Values recorded by "aligned AI systems"

2.3 Developers and Operators Requirements - Developers and operators of AI systems must:

2.3.1 Ensure that the Activation Values of their AI systems do not exceed the Maximum Threshold
2.3.2 Conduct regular tests to monitor Activation Values and ensure compliance with the Maximum Threshold
2.3.3 Maintain records of these tests and make them accessible to the government regulatory body and, in a non-technical format, to users

2.4 Offenses and Penalties - Non-compliance with the Maximum Threshold or failure to provide adequate information to users or the government regulatory body about Activation Values will constitute an offense. The penalties will be determined based on the severity of the offense, the potential harm caused, and whether it was a repeated offense. License to operate can also be...
AutoInterpretation Finds Sparse Coding Beats Alternatives
by Hoagy — 2023-07-17T01:41:17.397Z

Produced as part of the SERI ML Alignment Theory Scholars Program - Summer 2023 Cohort

Huge thanks to Logan Riggs, Aidan Ewart, Lee Sharkey, Robert Huben for their work on the sparse coding project, Lee Sharkey and Chris Mathwin for comments on the draft, EleutherAI for compute and OpenAI for GPT-4 credits.

Summary

We use OpenAI's automatic interpretation protocol to analyse features found by dictionary learning using sparse coding and compare the interpretability scores thereby found to a variety of baselines.

We find that for both the residual stream (layer 2) and MLP (layer 1) of Eleuther's Pythia70M, sparse coding learns a set of features that is superior to all tested baselines, even when removing the bias and looking just at the learnt directions. In doing so we provide additional evidence to the hypothesis that NNs should be conceived as using distributed representations to represent linear features which are only weakly anchored to the neuron basis.

Figure 1: Top-and-random interpretability scores for features found by sparse coding, compared with a variety of baselines, with means and 95% confidence intervals around mean.

As before these results are still somewhat preliminary and we hope to expand on them and make them more robust over the coming month or two, but we hope people find them fruitful sources of ideas. If you want to discuss, feel free to message me or head over to our thread in the EleutherAI discord.

All code available at the github repo.

Methods

Sparse Coding

The feature dictionaries learned by sparse coding are learnt by simple linear autoencoders with a sparsity penalty on the activations. For more background on the sparse coding approach to feature-finding see the Conjecture interim report that we're building from, or Robert Huben's explainer.

Automatic Interpretation

As Logan Riggs' recently found, many of the directions found through sparse coding seem highly interpretable, but we wanted a way to quantify this, and make sure that we were detecting a real difference in the level of interpretability.

To do this we used the methodology outlined in this OpenAI paper, details can be found in their code base. To quickly summarise, we are analysing features which are defined as scalar-valued functions of the activations of a neural network, limiting ourselves here to features defined on a single layer of a language model. The original paper s...