trentmkelly/LessWrong-43k huggingface dataset:
How do we know that "good research" is good? (aka "direct evaluation" vs "eigen-evaluation")
by Ruby — 2024-07-19T00:31:38.332Z

AI Alignment is my motivating context but this could apply elsewhere too.
The nascent field of AI Alignment research is pretty happening these days. There are multiple orgs and dozens to low hundreds of full-time researchers pursuing approaches to ensure AI goes well for humanity. Many are heartened that there's at least some good research happening, at least in the opinion of some of the good researchers. This is reason for hope, I have heard.

But how do we know whether or not we have produced "good research?"

I think there are two main routes to determining that research is good, and yet only one applies in the research field of aligning superintelligent AIs.

"It's good because it works"

The first and better way to know that your research is good is because it allows you to accomplish some goal you care about[1] [1]. Examples:

My work on efficient orbital mechanics calculation is good because it successfully lets me predict the trajectory of satellites.
My work on the disruption of cell signaling in malign tumors is good because it helped me develop successful anti-cancer vaccines.
My work on solid-state physics is good because it allowed me to produce superconductors at a higher temperature and lower pressure than previously attained.[2]

In each case, there's some outcome I care about pretty inherently for itself, and if the research helps me attain that outcome it's good (or conversely if it doesn't, it's bad). The good researchers in my field are those who have produced a bunch of good research towards the aims of the field.

Sometimes it's not clear-cut. Perhaps I figured out some specific cell signaling pathways that will be useful if it turns out that cell signaling disruption in general is useful, and that's TBD on therapies currently being trialed and we might not know how good (i.e. useful) my research was for many more years. This actually takes us into what I think is the second meaning of "good research".

"It's good because we all agree it's good"

If our goal is successfully navigating the creation of superintelligent AI in a way such that humans are happy with the outcome, then it is too early to properly score existing research on how helpful it will be. No one has aligned a superintelligence. No one's research has contributed to the alignment of an actual superintelligence.

At this point, the best we can do is share our predictions about how us...
Linkpost: Surely you can be serious
by kave — 2024-07-18T22:18:09.271Z
Link: https://www.experimental-history.com/p/surely-you-can-be-serious

Adam Mastroianni writes about "actually caring about stuff, and for the right reasons", rather than just LARPing. The opening is excerpted below.

I once saw someone give a talk about a tiny intervention that caused a gigantic effect, something like, “We gave high school seniors a hearty slap on the back and then they scored 500 points higher on the SAT.”

1

Everyone in the audience was like, “Hmm, interesting, I wonder if there were any gender effects, etc.”

I wanted to get up and yell: “EITHER THIS IS THE MOST POTENT PSYCHOLOGICAL INTERVENTION EVER, OR THIS STUDY IS TOTAL BULLSHIT.”

If those results are real, we should start a nationwide backslapping campaign immediately. We should be backslapping astronauts before their rocket launches and Olympians before their floor routines. We should be running followup studies to see just how many SAT points we can get—does a second slap get you another 500? Or just another 250? Can you slap someone raw and turn them into a genius?

Or—much more likely—the results are not real, and we should either be a) helping this person understand where they screwed up in their methods and data analysis, or b) kicking them out for fraud.

Those are the options. Asking a bunch of softball questions (“Which result was your favorite?”) is not a reasonable response. That’s like watching someone pull a rabbit out of a hat actually for real, not a magic trick, and then asking them, “What’s the rabbit’s name?”
My experience applying to MATS 6.0
by mic — 2024-07-18T19:02:21.849Z

The current cohort of the ML Alignment & Theory Scholars Program, MATS 6.0, had a unique application process and its broadest selection of mentors yet, with 40 mentors to apply to. I was invited to interview with twelve mentors and was accepted by five (which I later learned was an unusual number of interviews and offers in this cohort). Along the way, I challenged preconceptions I had about what AI safety research mentors look for in candidates, while becoming more familiar with various research areas.

In describing my experience with the application process, I hope this post is useful for anyone interested in applying to similar AI safety research programs or those involved in candidate evaluation. (Ultimately, I’ve decided to pursue another opportunity instead of MATS.) Note that I’ve written this relatively quickly and my experience may not be reflective of others.

Some updates I had from the interview process:

Brainstorming good research ideas is a major part of MATS interviews.
AI safety research experience matters a fair bit, especially experience relevant to a mentor’s research interests.
When I was rejected from prior rounds of MATS, I wondered whether I should have spent more time applying to more mentors. But a greater bottleneck was probably having more exciting AI safety research experience in my application.
Written answers to the mentor selection questions are less determinative than I previously thought of whether you get accepted.
It was pretty normal to receive a rejection email following an interview even when I felt I did reasonably well, given that each mentor could only accept a handful of candidates. After a while, I started to approach interviews with more equanimity.

I’ll divide the MATS application process into four phases, which I describe in more detail:

Initial intake
Mentor selection questions (with three waves of mentors)
Follow-up questions (optional)
Interviews (not all mentors)

Note that other mentors have a different process than what I experienced; for example, Neel Nanda commented:

For the avoidance of confusion, my MATS stream has a very different admissions process, that is heavily based on a work task and doesn't have interviews (and weights quite different things), see more details here: https://tinyurl.com/neel-mats-app

Application process

Initial intake and mentor selection questions

For those who aren’t familiar, the MATS Summe...
What are the actual arguments in favor of computationalism as a theory of identity?
by sunwillrise — 2024-07-18T18:44:20.751Z

A few months ago, Rob Bensinger made a rather long post (that even got curated) in which he expressed his views on several questions related to personal identity and anticipated experiences in the context of potential uploading and emulation. A critical implicit assumption behind the exposition and reasoning he offered was the adoption of what I have described as the "standard LW-computationalist frame." In response to me highlighting this, Ruben Bloom said the following:

I differ from Rob in that I do think his piece should have flagged the assumption of ~computationalism, but think the assumption is reasonable enough to not have argued for in this piece.
I do think it is interesting philosophical discussion to hash it out, for the sake of rigor and really pushing for clarity. I'm sad that I don't think I could dive in deep on the topic right now.

However, as I pointed out in that thread, the lack of argumentation or discussion of this particular assumption throughout the history of the site means it's highly questionable to say that assuming it is "reasonable enough":

As TAG has written a number of times, the computationalist thesis seems not to have been convincingly (or even concretely) argued for in any LessWrong post or sequence (including Eliezer's Sequences).

TAG himself made a similar and important point in a different comment on the same post:

Naturalism and reductionism are not sufficient to rigourously prove either form of computationalism -- that performing a certain class of computations is sufficient to be conscious in general, or that performing a specific one is sufficient to be a particular conscious individual.

This has been going on for years: most rationalists believe in computationalism, none have a really good reason to.

Arguing down Cartesian dualism (the thing rationalists always do) doesn't increase the probability of computationalism, because there are further possibilities , including physicalism-without-computationalism (the one rationalists keep overlooking) , and scepticism about consciousness/identity.

One can of course adopt a belief in computationalism, or something else, in the basis of intuitions or probabilities. But then one is very much in the ream of Modest Epistemology, and needs to behave accordingly.

"My issue is not with your conclusion, it’s precisely with your absolute certainty, which imo you support with cyclical argumen

...
Yet Another Critique of "Luxury Beliefs"
by ymeskhout — 2024-07-18T18:37:28.703Z
Link: https://www.ymeskhout.com/p/yet-another-critique-of-luxury-beliefs

I know little about Rob Henderson except that he wrote a well-received memoir and that he really really really wants you to remember that he invented the concept of “luxury beliefs”. In his own words, these are:

ideas and opinions that confer status on the upper class at very little cost, while often inflicting costs on the lower classes

The concept has metastasized and earned widespread adoption — particularly among social conservatives and right-wing populists. It might sound sophisticated, but it’s fundamentally flawed. Its vague and inconsistent definitions necessitate a selective application, and it’s ultimately used to launder mundane political preferences into something seemingly profound and highbrow.[1]

It’ll be most useful to break down Henderson’s concept into parts and go through it step-by-step.

1. Fashionable beliefs are always in style

First, there’s absolutely nothing groundbreaking or controversial about the idea that human beings adopt beliefs for social gain. If your entire community believes in creationism or astrology or Taylor Swift or whatever, it might be worth it to play along just to avoid ostracism.

Yet Henderson writes about this in a very confusing manner, conflating different meanings of ‘beliefs’ and ‘costs’. There’s a huge difference between saying you believe, behaving as if you believe, and lobbying for your beliefs, but Henderson uses them all interchangeably.[2] For example, here’s him conflating two personal choices, with one policy choice in one sentence:

Advocating for sexual promiscuity, drug experimentation or abolishing the police are good ways of advertising your membership of the elite because, thanks to your wealth and social connections, they will cost you less than me.

And his favorite anecdote (which he repeats over and over and over again) involves his classmate straight-up bullshitting:

For example, a former classmate at Yale told me “monogamy is kind of outdated” and not good for society. I asked her about her background and if she planned to marry. She was raised in a stable two-parent family, just like the vast majority of our classmates. And she planned on getting married herself. But she insisted that traditional families are old-fashioned and that society should “evolve” beyond them.

This is a boring story about someone who says one thing but does another. It’s a well-worn parable about moral aspiration and ...
[Interim research report] Evaluating the Goal-Directedness of Language Models
by Rauno Arike — 2024-07-18T18:19:04.260Z

This post was written as part of the summer 2024 cohort of the ML Alignment & Theory Scholars program, under the mentorship of Marius Hobbhahn.

Summary

Over the past four weeks, we have been developing an evaluation suite to measure the goal-directedness of LLMs. This post outlines our motivation, our approach, and the way we’ve come to think about the problem, as well as our initial results from experiments in two simulated environments.

The main motivation for writing the post is to convey our intuitions about goal-directedness and to gather feedback about our evaluation procedure. As these are uncertain preliminary results, we welcome any insights or critiques—please let us know if you think we're failing to evaluate the right thing!

Motivation

We want to evaluate goal-directedness for three reasons:

Deceptively aligned AI agents must be capable of competently following their goals. We should know when they are capable of doing that.
Alignment, in general, aims to ensure that AI agents follow the right goals. Therefore, we should assess how robustly models can adhere to their prescribed goals.
Goal-directed models potentially pose a greater threat when used by malicious actors than non-agentic tool AIs.

First, goal-directedness is a prerequisite for deceptive alignment. As argued in Risks from Learned Optimization, only a goal-directed system with an objective that extends across parameter updates can effectively conceal its true goals until deployment and strategically choose different goals in various situations. Accurately characterizing a model’s level of goal-directedness can help us determine whether that model is theoretically capable of pursuing a deceptive goal. Effective goal-directedness evaluations would also enable us to run experiments to better understand its causal drivers, such as model scale or fine-tuning techniques.

Second, goal-directedness is highly relevant to value alignment in general. Once models have been tuned to follow the HHH objectives (or any other set of objectives we expect them to follow), we need to know whether these objectives are really ingrained into the model’s cognition, or if the model is easily distracted or susceptible to jailbreaking. By applying adversarial pressure to the models in our goal-directedness evaluations, we can identify general patterns of when models deviate from their specified go...
Interpretability in Action: Exploratory Analysis of VPT, a Minecraft Agent
by Karolis Jucys — 2024-07-18T17:02:06.179Z
Link: https://arxiv.org/abs/2407.12161

TL;DR

We apply mechinterp techniques on VPT, OpenAI's Minecraft agent. We also find a new case of goal misgeneralization - VPT kills a villager when we force one to stand under some tree leaves.

Abstract

Understanding the mechanisms behind decisions taken by large foundation models in sequential decision making tasks is critical to ensuring that such systems operate transparently and safely. In this work, we perform exploratory analysis on the Video PreTraining (VPT) Minecraft playing agent, one of the largest open-source vision-based agents. We aim to illuminate its reasoning mechanisms by applying various interpretability techniques. First, we analyze the attention mechanism while the agent solves its training task - crafting a diamond pickaxe. The agent pays attention to the last four frames and several key-frames further back in its six-second memory. This is a possible mechanism for maintaining coherence in a task that takes 3-10 minutes, despite the short memory span. Secondly, we perform various interventions, which help us uncover a worrying case of goal misgeneralization: VPT mistakenly identifies a villager wearing brown clothes as a tree trunk when the villager is positioned stationary under green tree leaves, and punches it to death.

2min teaser video

Media

Feel free to use this GIF in your presentations on the importance of AI safety :) More formats and speeds in the website below.

Website

https://sites.google.com/view/vpt-mi
Activation Engineering Theories of Impact
by kubanetics — 2024-07-18T16:44:33.656Z

Below I summarize the thoughts of other people on what's the Theory of Impact for Activation Engineering. I mostly base it on the "discussion" parts of the papers and the answers under @Chris_Leong's post What's the theory of impact for activation vectors?

Alex Turner's posts on controlling a maze-solving policy network, and a paper on steering GPT-2-XL by adding an activation vector, introduced activation engineering, a set of "techniques which steer models by modifying their activations".

As a complement to prompt engineering and finetuning, activation engineering is a low-overhead way to steer models at runtime.

Over a year later and there's a Slack server to coordinate research projects and propose new ideas or open problems in this area. This is, to my knowledge, the closest we have to a practical implementation of retargetting the search.

ToIs

Low-tax value alignment of LLMs

The original "activation addition" paper claims that

activation engineering can flexibly retarget LLM behavior without damaging general performance. We speculate that this involves changing the model’s currently-active (mixture of) goals. Suitably developed, the activation engineering approach could enable safety progress while incurring a very low ‘alignment tax’

Alex Turner claims (as far as I understand) that that steering vectors can significantly enhance a model's performance on key aspects of safety, such as truthfulness, reducing the tendency to hallucinate or generate false information, minimizing sycophancy or overly agreeable responses, discouraging power-seeking behavior, and mitigating myopic tendencies.

Activation vectors don't work in isolation; they can be effectively combined with existing techniques like prompting and fine-tuning, leading to even greater improvements. This means that activation vectors may represent a valuable addition to the growing arsenal of tools available to AI researchers and developers striving to create more aligned and beneficial AI systems.

Insight into model representations

Most of productive research in this area will tell us something about how neural networks work and this seems to be a net positive unless it capabilities advancement offset the benefits to safety. This is the same dilemma that we have in case of mechanistic interpretability.

Activation engineering could specifically be used as a tool for top-down interpretabilit...
Me & My Clone
by SimonBaars — 2024-07-18T16:25:40.770Z

An advanced alien species clones me on the atomic level, lines me up exactly across myself, in a perfect mirrored room:

Diagram of the room, as seen from above.

I stare at myself for a second. Then, as a soft "hi" escapes my mouth, I notice that my clone does exactly the same. Every motion, everything, is mirrored.

In this experiment, we assume a perfectly deterministic psychological state: eg, given the same conditions, a person will always do exactly the same. (scientifically, that makes most sense to me)

Together with my clone, I'm trying to devise how to escape this unfortunate situation: eg, how to untangle us mirroring each other's motions.

The first idea we devise is to run into each other. We hope to apply Chaos Theory to the extent where both of us would fall in a slightly different way, and thus we would no longer be perfectly mirrored as such. But, if my understanding of physics is correct, our perfect opposing forces cause us to stumble and fall in perfectly mirrored ways.

For the second idea, I fetch a coin from my pocket. Just a coinflip won't work: we'd apply the same pressure to both our coins, and they'd land in the same spot. The idea is to number each corner of the room, and to decide the corner we're both going to through two coinflips. The corner we should go will be further away for one of us, thus breaking the mirror.

But, as we try to number the corners, we notice that we give the same number to opposite corners all the time. When I point at a corner, my mirrored self starts pointing at the opposite corner and giving it the same number.

I slump down to the ground. Will I be mirroring this perfect copy of myself for eternity? Or is there a way out?

As far as my understanding goes, in a deterministic framework, it is impossible to escape this scenario (eg, break the mirroring). In my opinion, determinism is separate from free will (eg free will is possible even when everything is deterministic). Am I correct or am I missing important things?
AI #73: Openly Evil AI
by Zvi — 2024-07-18T14:40:05.770Z

What do you call a clause explicitly saying that you waive the right to whistleblower compensation, and that you need to get permission before sharing information with government regulators like the SEC?

I have many answers.

I also know that OpenAI, having f***ed around, seems poised to find out, because that is the claim made by whistleblowers to the SEC. Given the SEC fines you for merely not making an explicit exception to your NDA for whistleblowers, what will they do once aware of explicit clauses going the other way?

(Unless, of course, the complaint is factually wrong, but that seems unlikely.)

We also have rather a lot of tech people coming out in support of Trump. I go into the reasons why, which I do think is worth considering. There is a mix of explanations, and at least one very good reason.

Then I also got suckered into responding to a few new (well, not really new, but renewed) disingenuous attacks on SB 1047. The entire strategy is to be loud and hyperbolic, especially on Twitter, and either hallucinate or fabricate a different bill with different consequences to attack, or simply misrepresent how the law works, then use that, to create the illusion the bill is unliked or harmful. Few others respond to correct such claims, and I constantly worry that the strategy might actually work. But that does not mean you, my reader who already knows, need to read all that.

Also a bunch of fun smaller developments. Karpathy is in the AI education business.

Table of Contents

Introduction.

Table of Contents.

Language Models Offer Mundane Utility. Fight the insurance company.

Language Models Don’t Offer Mundane Utility. Have you tried using it?

Clauding Along. Not that many people are switching over.

Fun With Image Generation. Amazon Music and K-Pop start to embrace AI.

Deepfaketown and Botpocalypse Soon. FoxVox, turn Fox into Vox or Vox into Fox.

They Took Our Jobs. Take away one haggling job, create another haggling job.

Get Involved. OpenPhil request for proposals. Job openings elsewhere.

Introducing. Karpathy goes into AI education.

In Other AI News. OpenAI’s Q* is now named Strawberry. Is it happening?

Denying the Future. Projects of the future that think AI will never improve again.

Quiet Speculations. How to think about stages of AI capabilities.

The Quest for Sane Regulations. EU, UK, The Public.

The

...