trentmkelly/LessWrong-43k huggingface dataset:
Protecting against sudden capability jumps during training
by Nikola Jurkovic — 2023-12-02T04:22:21.315Z

TL;DR: If an LLM has a sudden capabilities jump and is tested using a diverse test suite (e.g. one where it can execute code or present lots of text to human observers), it might be able to escape mid-training. In order to prevent this, I propose a phased testing approach, where the model is first tested for superhuman capabilities with an extremely constrained test suite before it’s tested with the broader test suite.

The failure mode

Let’s imagine GPT-6 suddenly gaining vastly superhuman hacking and manipulation skills. During training, some standard tests are performed to measure its performance on various benchmarks. Some of the tests include code execution on virtual machines, others include generating text which humans inspect. In total, thousands of queries are made to the model. The model finds a way to escape by taking advantage of vulnerabilities in the testing setup (e.g. by blackmailing a human evaluator or by finding an exploit in a virtual machine), before the researchers can even notice that it’s superintelligent.

This highlights a failure mode I’m concerned about: a sudden capabilities jump leading to a misaligned and powerful AI which, when tested with a diverse test suite, finds a way to escape before the lab notices how capable it is.

The primary way this goes wrong is that the AI has lots of surface area with the real world – it has ways to influence the humans and devices it interacts with. The more a model is allowed to influence the world (whether that be through writing text which gets inspected by humans, or running code on virtual machines) the easier it is for the model to take over. I would imagine that it’s extremely hard for a model to take over if it's only ever allowed to predict a sequence of 10 tokens and nothing else.

Ideally, the lab would have some way to notice that the model had a sharp capabilities jump without giving it many opportunities to take over. In case they were suddenly and unexpectedly dealing with a superintelligence, hopefully they would be wise enough to enter a pause until they’re confident that they can proceed safely.

Phased testing

I think that phased testing should be used during frontier model training runs. By this, I mean a testing approach which starts off with extremely low surface area tests, and gradually increases surface area. This makes it easy to notice sudden capability gains whil...
MATS Summer 2023 Retrospective
by utilistrutil — 2023-12-01T23:29:47.958Z

Co-Authors: @Rocket, @Juan Gil, @Christian Smith, @McKennaFitzgerald, @LauraVaughan, @Ryan Kidd

The

ML Alignment & Theory Scholars

program (MATS, formerly SERI MATS) is an education and research mentorship program for emerging AI safety researchers. This summer, we held the fourth iteration of the MATS program, in which 60 scholars received mentorship from 15 research mentors. In this post, we explain the elements of the program, lay out some of the thinking behind them, and evaluate our impact.

Summary

Key details about the Summer 2023 Program:

Educational attainment of MATS scholars:

30% of scholars are students.
88% have at least a Bachelor's degree.
10% are in a Master’s program.
10% are in a PhD program.
13% have a PhD.

If not for MATS, scholars might have worked at a tech company (41%), upskilled independently (46%), or conducted research independently over the summer (50%). (Note: this was a multiple-choice response.)

Key takeaways from our impact evaluation:

MATS scholars are highly likely to recommend MATS to a friend or colleague. Average likelihood: 8.9/10.
Mentors rated their enthusiasm for their scholars to continue with their research at 7/10 or greater for 94% of scholars.
MATS scholars rate their mentors highly. Average rating: 8.0/10.

61% of scholars report that at least half the value of MATS came from their mentor.

After MATS, scholars reported facing fewer obstacles to a successful alignment career than they did at the start of the program.

Most scholars (75%) still reported their publication record as an obstacle to a successful alignment career at the conclusion of the program.

⅓ of final projects involved evals/demos and ⅕ involved mechanistic interpretability, representing a large proportion of the cohort’s research interests.
Scholars self-reported improvements to their research ability on average:

Slight increases to the breadth of their AI safety knowledge (+1.75 on 10-point scale over the program).
Moderate strengthening of technical skills compared to counterfactual summer (7.2/10, where 10/10 is "significant improvement compared to counterfactual summer").
Moderate improvements to ability to independently iterate on research direction (7.0/10, where 10/10 is "significant improvement") and ability to develop a theory of change for their research (5.9/10, where 10/10 is "substantially developed").

The typical scholar reported making 4.5 profess

...
Complex systems research as a field (and its relevance to AI Alignment)
by Nora_Ammann — 2023-12-01T22:10:25.801Z

habryka

I have this high prior that complex-systems type thinking is usually a trap. I've had a few conversations about this, but still feel kind of confused, and it seems good to have a better written record of mine and your thoughts here.

At a high level, here are some thoughts that come to mind for me when I think about complex systems stuff, especially in the context of AI Alignment:

A few times I ended up spending a lot of time trying to understand what some complex systems people are trying to say, only to end up thinking they weren't really saying anything. I think I got this feeling from engaging a bunch with the Santa Fe stuff and Simon Dedeo's work (like this paper and this paper)
A part of my model of how groups of people make intellectual progress is that one of the core ingredients is having a shared language and methodology that allows something like "the collective conversation" to make incremental steps forward. Like, you have a concept of experiment and statistical analysis that settles an empirical issue, or you have a concept of proof that settles an issue of logical uncertainty, and in some sense a lot of interdisciplinary work is premised on the absence of a shared methodology and language.
While I feel more confused about this in recent times, I still have a pretty strong prior towards something like g or the positive manifold, where like, there are methodological foundations that are important for people to talk to each other, but most of the variance in people's ability to contribute to a problem is grounded in how generally smart and competent and knowledgeable they are, and expertise is usually overvalued (for example, it's not that rare for a researcher to win a Nobel prize in two fields). A lot of interdisciplinary work (not necessarily complex systems work, but some of the generator that I feel like I see behind PIBBS) feels like it puts a greater value on intellectual diversity here than I would.

Nora_Ammann

Ok, so starting with one high-level point: I'm definitely not willing to die on the hill of 'complex systems research' as a scientific field as such. I agree that there is a bunch of bad or kinda hollow work happening under the label. (I think the first DeDeo paper you link is a decent example of this: feels mostly like having some cool methodology and applying it to some random phenomena without really an exciting bigger vis

...
Could there be "natural impact regularization" or "impact regularization by default"?
by tailcalled — 2023-12-01T22:01:46.062Z

Specifically, imagine you use general-purpose search procedure which recursively invokes itself to solve subgoals for the purpose of solving some bigger goal.

If the search procedure's solutions to subgoals "change things too much", then they're probably not going to be useful. E.g. for Rubik's cubes, if you want to swap some of the cuboids, it does you know good if those swaps leave the rest of the cube scrambled.

Thus, to some extent, powerful capabilities would have to rely on some sort of impact regularization.

I'm thinking that natural impact regularization is related to the notion of "elegance" in engineering. Like if you have some bloated tool to solve a problem, then even if it's not strictly speaking an issue because you can afford the resources, it might feel ugly because it's excessive and puts mild constaints on your other underconstrained decisions, and so on. Meanwhile a simple, minimal solution often doesn't have this.

Natural impact regularization wouldn't guarantee safety, since it's still allows deviations that don't interfere with the AI's function, but it sort of reduces one source of danger which I had been thinking about lately, namely I had been thinking that the instrumental incentive is to search for powerful methods of influencing the world, where "power" connotes the sort of raw power that unstopably forces a lot of change, but really the instrumental incentive is often to search for "precise" methods of influencing the world, where one can push in a lot of information to effect narrow change.[1]

Maybe another word for it would be "natural inner alignment", since in a sense the point is that capabilities inevitably select for inner alignment. Here I mean "natural" in the sense of natural abstractions, i.e. something that a wide variety of cognitive algorithms would gravitate towards.

^

A complication is that any one agent can only have so much bandwidth, which would sometimes incentivize more blunt control. I've been thinking bandwidth is probably going to become a huge area of agent foundations, and that it's been underexplored so far. (Perhaps because everyone working in alignment sucks at managing their bandwidth? 😅)
Benchmarking Bowtie2 Threading
by jefftk — 2023-12-01T20:20:05.593Z

I've been using Bowtie2 to align reads to genomes, and one of it's
many settings is the number of threads. While sometimes people advise
using about as many threads as your machine has cores, but if I'm
running on a big machine are there diminishing returns or a point at
which more threads are counterproductive? Am I better off running more
samples in parallel with more threads each, or fewer with fewer?

I decided to run a few tests on an AWS EC2 with a
c6a.8xlarge 32-core AMD machine. The test consisted of
running one 7.2Gb 48M read-pair sample (SRR23998356)
from Crits-Christoph
et. al 2021 through Bowtie2 2.5.2 with the "Human / CHM13plusY"
database from Langmead's
Index Zone. The files were streamed from AWS S3 and decompressed
in a separate process. See the script
for my exact test harness and configuration.

What I found (sheet)
was that initially allocating additional threads helps a lot, but
after ~8 it was plateauing and after ~10 more threads were very
slightly starting to hurt:

Compare what I saw to the best you could hope for, where the task is
perfectly parallelizable across threads:

This shows that initially Bowtie2 makes very efficient use of
additional threads but, as you'd expect from the plateauing, after ~8
the benefit drops off.

Another way to look at this is to chart how many thread-seconds it
takes to process the whole dataset:

On this plot perfect parallelism would be a horizontal line, and it's
pretty close to that up through 8 threads. After that, however, it
starts rising much more steeply, showing much less efficient use of
threads.

You might wonder how much of this is Bowtie2, and how much is my test
environment. For example, perhaps by the time we get to eight threads
we're at the limit of how quickly this machine can download from S3
and decompress the data? Bowtie2 can't work any faster than the data
arrives! I ran a "null" version of this where I replaced
bowtie2 with paste
(test
script) and found that this consistently took 44-45s. Since it's
all streaming and the fastest run took 189s (4x) I don't think this
was having an appreciable effect.

Overall, it looks like even on a very large machine you shouldn't set
Bowtie2's --threads above 10, and unless you have idle
cores 8 is probably a better value.
Please Bet On My Quantified Self Decision Markets
by niplav — 2023-12-01T20:07:38.284Z

There are too many possible quantified self experiments to run. Do hobbyist prediction platforms[1] make priorisation easier? I test this by setting up multiple markets, in order to run two experiments (the best one, and a random one), mostly for the effects of nootropics on absorption in meditation.

dynomight 2022 has a cool proposal:

Oh, and by the way are you THE NSF or DARPA or THE NIH or A BILLIONAIRE WHO WANTS TO SPEND LOTS OF MONEY AND BRAG ABOUT HOW YOU ADVANCED THE STATE OF HUMAN KNOWLEDGE MORE THAN ALL THOSE OTHER LAME BILLIONAIRES WHO WOULDN’T KNOW A HIGH ROI IF IT HIT THEM IN THE FACE? Well how about this:

Gather proposals for a hundred RCTs that would each be really expensive but also really awesome. (E.g. you could investigate SALT → MORTALITY or ALCOHOL → MORTALITY or UBI → HUMAN FLOURISHING.)
Fund highly liquid markets to predict the outcome of each of these RCTs, conditional on them being funded.

If you have hangups about prison, you might want to chat with the CFTC before doing this.

Randomly pick 5% of the proposed projects, fund them as written, and pay off the investors who correctly predicted what would happen.
Take the other 95% of the proposed projects, give the investors their money back, and use the SWEET PREDICTIVE KNOWLEDGE to pick another 10% of the RCTs to fund for STAGGERING SCIENTIFIC PROGRESS and MAXIMAL STATUS ENHANCEMENT.

—dynomight, “Prediction market does not imply causation”, 2022

Well, I'm neither a billionaire nor the NSF or DARPA, but I have run two shitty self-blinded RCTs on myself already, and I'm certainly not afraid of the CFTC. And indeed I don't have a shortage of ideas on things I could run RCTs on, but the time is scarce (I try to collect m=50 samples in each RCT, which (with buffer-days off) is usually more than 2 months of data collection).

So I'll do what @saulmunn pointed out to me is a possibility: I'm going to do futarchy (on) myself by setting up a set of markets of Manifold Markets with respect to the outcomes of some pre-specified self-blinded RCTs, waiting until the prices on them equilibriate, and then running two of those RCTs (the "best" one, by my standards, and a random one) and using the results as resolutions, while resolving the others as ambiguous.

Timeline

If the markets receive enough liquidity, I'll start the first experiment early in 2024, and the second one sometime in 2024 (depending on the ...
Specification Gaming: How AI Can Turn Your Wishes Against You [RA Video]
by Writer — 2023-12-01T19:30:58.304Z
Link: https://youtu.be/jQOBaGka7O0

In this new video, we explore specification gaming in machine learning systems. It's meant to be watched as a follow-up to The Hidden Complexity of Wishes, but it can be enjoyed and understood as a stand-alone too. It's part of our work-in-progress series on outer-alignment-related topics, which is, in turn, part of our effort to explain AI Safety to a wide audience.
I have included the full script below.

In the previous video we introduced the thought experiment of “the outcome pump”. A device that lets you change the probability of events will. In that thought experiment, your aged mother is trapped in a burning building. You wish for your mother to end up as far as possible from the building, and the outcome pump makes the building explode, flinging your mother’s body away from it.

That clearly wasn't what you wanted, and no matter how many amendments you make to that wish, it’s really difficult to make it actually safe, unless you have a way of specifying the entirety of your values.

In this video we explore how similar failures affect machine learning agents today. You can think of such agents as less powerful outcome pumps, or little genies. They have goals, and take actions in their environment to further their goals. The more capable these models are the more difficult it is to make them safe. The way we specify their goals is always leaky in some way. That is, we often can’t perfectly describe what we want them to do, so we use proxies that deviate from the intended objective in certain cases. In the same way “getting your mother out of the building” was only an imperfect proxy for actually saving your mother. There are plenty of similar examples in ordinary life. The goal of exams is to evaluate a student’s understanding of the subject, but in practice students can cheat, they can cram, and they can study just exactly what will be on the test and nothing else. Passing exams is a leaky proxy for actual knowledge. If we paid a bot farm to boost view counts for this video, that would generate plenty of views, but it would be pointless. View count is a leaky proxy for our real objective of teaching important subjects to a wide audience.

In machine learning, depending on the context, there are many names used for situations in which this general phenomenon occurs, such as: specification gaming, reward misspecification, reward hacking, outer mi...
Carving up problems at their joints
by Jakub Smékal — 2023-12-01T18:48:46.510Z
Link: https://jakubsmekal.com/carving-up-problems-at-their-joints

Imagine looking at flower. Depending on the lighting conditions and some other factors, you see the flower with different levels of detail. If you take a picture and project it onto a low-resolution screen, instead of a flower you might just see colored boxes that barely capture the flower’s features. On the other hand, if you have a high-resolution screen, you might not be able to tell the difference between the real flower and the one projected on your screen, nevertheless by looking at the screen, you acknowledge you’re not actually looking at a flower but rather on a set of colored pixels. Of course, this is not an exhaustive set of the possibilities for perceiving a flower. If you put that flower under a microscope, you’d start seeing individual cells that make up the flower’s structure. At that point, you might not even know that what you’re looking at is indeed a flower, even though you now perceive at a resolution much higher than that of your naked eye. Go further still and you might start seeing the smaller structures making up the cells, and at the limit of technological abilities, you might even glimpse at individual molecules and the atoms that make them up. And to complete this particular spectrum of resolutions, imagine going farther away from the flower. The farther you get, the smaller it is and the less detail you see, similarly as with the picture on the low-resolution screen, but also fundamentally different, since what you’re seeing should still be called a flower rather than a set of pixels. Go far enough, and you’ll just see a dot that is indistinguishable from anything else at that distance, so even though you may “know” that you’re looking at a flower, your ability to recognize that it is a flower is perhaps worse than when you were looking at the picture on the low-resolution screen.

Perceived information is contextual, both on the particular apparatus through which it is presented and through our own beliefs about what we expect to see. Furthermore, we can never fully capture the thing with just semantics. You know you’re still looking at a flower, even though you just see a red dot in the distance. The problem of resolution and semantics is present in every other piece of information that we can perceive or think about. And of course, considering the map-territory fallacy, no description or even perception of the flower is what the...
Queuing theory: Benefits of operating at 60% capacity
by ampdot — 2023-12-01T18:48:01.426Z
Link: https://less.works/less/principles/queueing_theory

Related to Slack. Related to Lean Manufacturing, aka JIT Manufacturing.

TL;DR A successful task-based system should sometimes be idle, like 40% of worker ants.

Doing tasks quickly is essential for producing value in many systems. For software teams, delivering a feature gives valuable insight into user needs, which can improve future feature quality. For supply chains, faster delivery releases capital for reinvestment. However, the relationship between capacity utilized and service time is exponential, as shown by the diagram below.

Stochastic arrivals mean it's better to sometimes have idle capacity, so that the system is quickly ready for when tasks arrive in droves.

A heuristic we can derive from queuing theory is that the optimal balance between efficiency and capacity typically occurs when the system is around 30-40% idle. For a single producer system, being X% idle is that producer being idle X% of the time. For a multi-producer system, being X% idle is X% of those producers being idle on average. This heuristic applies best to systems involving lots of discrete, oddly-shaped tasks.

The linked post explains this theory in more detail, and gives examples of where queues appear in the real world. See also the Wikipedia article:

Queueing theory is the mathematical study of waiting lines, or queues.[1] A queueing model is constructed so that queue lengths and waiting time can be predicted.[1] Queueing theory is generally considered a branch of operations research because the results are often used when making business decisions about the resources needed to provide a service.
Researchers and writers can apply for proxy access to the GPT-3.5 base model (code-davinci-002)
by ampdot — 2023-12-01T18:48:01.406Z
Link: https://airtable.com/shrUqwatM7ClxaBH7

AI safety researchers and creative writers can apply for proxied access to the GPT-3.5 base model through, also known as code-davinci-002.

Publicly available base models include "davinci-002" (the text-davinci-xxx models are not base models) and cohere's "xlarge" model.

I recommend interacting with language models using janus' loom. One such implementation of it by parafactual is Loomsidian. In brief, the Loom is a tool for generating multiple continuations of a prompt and organizing them in a tree. The Loom babbles, and you prune its continuations. It allows for creative exploration of questions with multiple answers.