trentmkelly/LessWrong-43k huggingface dataset:
A starter guide for evals
by Marius Hobbhahn — 2024-01-08T18:24:23.913Z
Link: https://www.apolloresearch.ai/blog/a-starter-guide-for-evals

This is a starter guide for model evaluations (evals). Our goal is to provide a general overview of what evals are, what skills are helpful for evaluators, potential career trajectories, and possible ways to start in the field of evals.
Evals is a nascent field, so many of the following recommendations might change quickly and should be seen as our current best guess.

Why work on evals?

Model evaluations increase our knowledge about the capabilities, tendencies, and flaws of AI systems. Evals inform the public, AI organizations, lawmakers, and others and thereby improve their decision-making. However, similar to testing in a pandemic or pen-testing in cybersecurity, evals are not sufficient, i.e. they don’t increase the safety of the model on their own but are needed for good decision-making and can inform other safety approaches. For example, evals underpin

Responsible Scaling Policies

and thus already influence relevant high-stakes decisions about the deployment of frontier AI systems. Thus, evals are a highly impactful way to improve the decision-making about AI systems.

Evals are a nascent field and there are many fundamental techniques to be developed and questions to be answered. Since evals do not require as much background knowledge as many other fields, it is much easier to get started and possible to make meaningful contributions from very early on.

What are model evaluations (evals)?

Evals refers to a broad category of approaches that we roughly summarize as:

The systematic measurement of properties in AI systems

More concretely, evals typically attempt to make a quantitative or qualitative statement about the capabilities or propensities of an AI system. For example, we could ask if a model has the capability to solve a specific coding problem or the propensity to be power-seeking. In general, evals are not restricted to safety-related properties but often when people talk about evals, they mention them in a safety context.

There is a difference between red-teaming and benchmarking. Red-teaming is actively looking for specific capabilities or propensities while interacting with the model. It is an attempt to answer the question “Can we find this capability in a model when we try hard to find it?”. In other words, red-teaming is an attempt to show the existence of certain capabilities/properties, but it is not trying ...
Is it justifiable for non-experts to have strong opinions about Gaza?
by Yair Halberstadt — 2024-01-08T17:31:21.934Z

Adam Zerner

Ok, so here's the prompt that we'll be discussing, which I proposed:
"I am of the opinion that the situation in Gaza is complicated and that a large majority of people are uninformed enough such that anything more than weak opinions is not justified."
To get started, I'd like to propose the analogy of chess. If we wave our hands a little bit, we can (perhaps) say that people generally agree on what outcomes we are seeking. Something along the lines of "maximizing utility" or "human flourishing". I think the disagreement is moreso over how we can achieve these outcomes.

I think the situation in Gaza is analogous to chess in the sense that if you move a piece, it isn't too clear what the consequences of that move will be or whether the move will help you achieve your end goal. On the other hand, in a game like Tic-Tac-Toe, things are very simple, and you can in fact do quite a good job at foreseeing the consequences of your move.
At risk of being uncharitable, those who support an aggressive Israeli response, it almost feels like they're arguing something analogous to: "How dare they capture one of our pawns like that! How unwarranted and monstrous of them! We can't let them do that. We must recapture the piece that took our pawn, and go on the offensive to take enough of their pieces such that they won't be able to capture any of our pieces in the future."
And similarly, those who are against the aggressive Israeli response, it almost feels like: "Sure, Gaza shouldn't have captured that pawn, but look at how lopsided Israel's response is. They took our rooks already and are now going after our king!"

The analogy breaks down in various ways, but I like how it highlights the complexity and uncertainty. Chess is a complicated game, and it is difficult to predict the consequences of your moves. Especially for people who haven't studied the game. Similar, there is a lot of complexity in the "game board" surrounding Jerusalem, and it is difficult to predict the consequences that various moves will have. Especially for people who haven't studied the game.

Yair Halberstadt

So I think what you're saying is: people are judging this war based on whether each sides actions feel justified, but actually you'd judge it based on what the consequences will be, and these are too complex to figure out?

Yair Halberstadt

If so then a few questions:

Too complex for anyone to figur

...
Project ideas: Backup plans & Cooperative AI
by Lukas Finnveden — 2024-01-08T17:19:33.181Z
Link: https://www.forethought.org/research/project-ideas-backup-plans-and-cooperative-ai

This is part of a series of lists of projects. The unifying theme is that the projects are not targeted at solving alignment or engineered pandemics but still targeted at worlds where transformative AI is coming in the next 10 years or so. See here for the introductory post.

In this final post, I include two categories of projects (which are related, and each of which I have less to say about than the previous areas). Backup plans for misaligned AI and Cooperative AI.

Backup plans for misaligned AI

When humanity builds powerful AI systems, I hope that those systems will be safe and aligned. (And I’m excited about efforts to make that happen.)

But it’s possible that alignment will be very difficult and that there won’t be any successful coordination effort to avoid building powerful misaligned AI. If misaligned AI will be built and seize power (or at least have the option of doing so), then there are nevertheless certain types of misaligned systems that I would prefer over others. This section is about affecting that.

This decomposes into two questions:

If humanity fails to align their systems and misaligned AIs seize power: What properties would we prefer for those AIs to have?
If humanity fails to align their systems and misaligned AIs seize power: What are realistic ways in which we might have been able to affect which misaligned AI systems we get? (Despite how our methods were insufficient to make the AIs fully safe & aligned.)

The first is addressed in What properties would we prefer misaligned AIs to have? The second is addressed in Studying generalization & AI personalities to find easily-influenceable properties. (If you’re more skeptical about the second of these than the first, you should feel free to read that section first.)

What properties would we prefer misaligned AIs to have? [Philosophical/conceptual] [Forecasting]

There are a few different plausible categories here. All of them could use more analysis on which directions would be good and which would be bad.

Making misaligned AI have better interactions with other actors

This overlaps significantly with cooperative AI. The idea is that there are certain dispositions that AIs could have that would lead them to have better interactions with other actors who are comparably powerful.

When I say “better” interactions, I mean that the interactions will leave the other acto...
Hackathon and Staying Up-to-Date in AI
by jacobhaimes — 2024-01-08T17:10:20.270Z
Link: https://into-ai-safety.github.io/episode/hackathon/episode-6/

Listen to the first episode of a new sub-series, which will follow my team's process during the Evals Hackathon hosted by AlignmentJamz in November of 2023. In the end, our submission was titled "Detecting Implicit Gaming through Retrospective Evaluation Sets," and it received first place.

Also, I forgot to post about last week's episode, so I am mentioning it here. The episode, "Staying Up-to-Date in AI," covers the methods and tools that I use in an attempt to keep up with the break-neck pace of AI developments.

If you are enjoying the kind of content that I am publishing and/or support my goals with the podcast, please share it to anyone that you think may like is as well.

As a reiteration of my previous statements, I want the Into AI Safety podcast to be a resource for individuals who are interested in getting involved, but are having a difficult time taking the next steps. If you have any advice, feedback, or ideas that you think could help in that endeavor, please reach out!

In addition to the Into AI Safety podcast website, you can find the content on (I believe) all podcast listening platforms (e.g., Spotify, Apple Podcasts, Pocket Casts). If you think that I missed a platform, please let me know.
When "yang" goes wrong
by Joe Carlsmith — 2024-01-08T16:35:50.607Z

(Cross-posted from my website. Podcast version here, or search "Joe Carlsmith Audio" on your podcast app.

This essay is part of a series I'm calling "Otherness and control in
the age of AGI." I'm hoping that the individual essays can be read
fairly well on their own, but see here for a brief summary of the essays
that have been released thus far.)

Becoming God

In my last essay, I wrote about "deep atheism" – a fundamental mistrust towards Nature, and towards bare intelligence. I
took Eliezer Yudkowsky as a paradigmatic deep atheist, and I tried to
highlight the connection between his deep atheism and his concern about
misaligned AI.

I'm sympathetic to many aspects of Yudkowsky's view. I'm a shallow
atheist, too; I'm skeptical of moral realism, too; and I, too, aspire to
be a scout, and to look at hard truths full on. What's more, I find
Yudkowsky's brand of deep-but-still-humanistic atheism more compelling,
as an existential orientation, than many available alternatives. And I
share Yudkowsky's concern about AI risk. Indeed, it was centrally him,
and others thinking along similar lines, who first got me worried.

But I also want to acknowledge and examine some difficult questions that
a broadly Yudkowskian existential orientation can raise, especially in
the context of AGI. In particular: a lot of the vibe here is about
mistrust towards the yang of the Real, that uncontrolled Other. And
it's easy to move from this to a desire to take stuff into the hands of
your own yang; to master the Real until it is maximally controlled;
to become, you know, God – or at least, as God-like as possible. You've
heard it before – it's an old rationalist dream. And let's be clear:
it's alive and well. But even with theism aside, many of the old reasons
for wariness still apply.

Moloch and Stalin

As an example of this becoming-God aspiration, consider another
influential piece of rationalist canon: Scott Alexander's "Meditations
on
Moloch."
Moloch, for Alexander, is the god of uncoordinated competition; and fear
of Moloch is its own, additional depth of atheism. Maybe you thought you
could trust evolution, or free markets, or "spontaneous order," or the
techno-capital machine. But oops, no: those gods just eat you too. Why?
Details vary, but broadly speaking: because the winner of competition is
power; power (like intelligence) is orthogonal to goodness; so every
opportunity to sac...
Task vectors & analogy making in LLMs
by Sergii — 2024-01-08T15:17:58.992Z
Link: https://grgv.xyz/blog/copycat2/

I have described the problem of analogy-making interpretability in the previous post: given the examples of transformed sequences of numbers, what’s the mechanism behind figuring this transformation out, and applying it correctly to the incomplete (test) sequence?

prompt: "0 1 2 to 2 1 0, 1 2 3 to 3 2 1, 4 5 6 to ", output: “6 5 4”

It was easy to check on which layer the correct answer appears, but tracing the sources of that answer to earlier layers turned out to be challenging.

Meaningful intermediate embeddings?

When I applied logit lens [1] to the output of attention blocks, for the prompt that contained reversed sequences of numbers, I have noticed that the output contained “reverse” token (on last token position of layer 15).

I’m using a llama.cpp-based app (described in the previous post) for showing the logit lens output. Each row corresponds to a token position, and lists top 5 tokens sorted by the logit score:

./mia -m llama2.gguf --prompt "0 1 2 to 2 1 0, 1 2 3 to 3 2 1, 4 5 6 to" -n 5 --logit-lens kqv_out 5

Layer #15 kqv_out-15:
0: дар 0.35|oure 0.35|kar 0.33| Según 0.33|aki 0.3|
1: dust 0.36|textt 0.36|elde 0.35|azzo 0.34| retro 0.34|
2: 典 0.37| Censo 0.35|oure 0.35| Aires 0.35| pó 0.34|
3: ḷ 0.39|ket 0.39| estaven 0.39|öß 0.39|oure 0.38|
4: zerw 0.62| estaven 0.51|cita 0.5| alberga 0.49|łow 0.48|
[...]
16: shal 0.84|ket 0.73|Assert 0.72|ając 0.66|sono 0.66|
17: ipt 0.95|кта 0.88|inal 0.86| inform 0.85| advanced 0.85|
18: кта 0.85|minipage 0.83| Mean 0.77|Assert 0.75| meaning 0.74|
19: ipt 0.78|Duration 0.76|zug 0.75|gemeinde 0.75|mannschaft 0.72|
20: shal 0.64|agy 0.64|prev 0.62| SA 0.6| Gay 0.58|
21: revers 0.75| reverse 0.68|mat 0.67|shal 0.66|vat 0.66|

Although the task is about reversing, “reverse” is not mentioned explicitly anywhere in either input or output.

I have tried subtracting the embedding of the “reverse” token, effectively removing it, to check it is a part of the analogy-making mechanism. It did not affect the output, which means that this token is not critical for the generation, and might be just a side effect of some other mechanism.

Task vectors

While looking for more information, I have found that this topic is covered by two recent papers on so-called “task vectors” [1, 2]. These papers concurrently explore similar ideas, just approach the problem differently, complementing each other.

So, what’s a task vector? Su...
How to find translations of a book?
by Viliam — 2024-01-08T14:57:18.172Z

What is the best way to find whether a book was translated to other languages (and how are the translations called)?

I assumed that there would be some global database for that; it feels like something that obviously should exist -- if you already assign ISBNs to books, the logical next step is to make a database that contains them, plus some metadata, including "this is a translation of this" -- but unless I am missing something obvious, there is no such service.

*

Here is a specific book I was recently looking for, but the question is general:

Author: Владимир Артурович Лёвшин (Vladimir Arturovich Levshin or Lyovshin)

Book title: Три дня в Карликании

So far I only know about translations to Vietnamese and Slovak; the former I found on Wikipedia (the author only has a Wikipedia page in Russian and Vietnamese), the later because I happened to find that one in a library -- I couldn't find out about it otherwise.

(It is a book about math, for small kids. A very good one, if my memories are reliable.)
Why aren't Yudkowsky & Bostrom getting more attention now?
by JoshuaFox — 2024-01-08T14:42:46.303Z

Now that the idea of AI Existential Risk has gained more acceptability, I am surprised that the thought leaders are not more often mentioned and contacted by AI executives and researchers in the cutting edge (OpenAI, Anthropic, etc.), recognized academic figures (Bengio, Hinton, Hofstadter, etc.), journalists, or political leaders who have expressed an interest.
To be sure, they are mentioned in articles; MIRI was represented in a congressional hearing; and Mira Murati reached out to Eliezer. But still, it seems that the profile of the pioneers is much lower than I'd expect.
Fifteen years ago, we could have said that AI XRisk was treated as a crackpot idea and that MIRI in particular might be ignored, as it operates outside a standard framework like a university. But today, not only have the ideas spread, but many top AI-capabilities researchers have, I think, entered the field specifically because of inspiration from the MIRI/LessWrong circle.

Though some journalists might still be hung up on MIRI's lack of social status markers, I don't think that many others, including cutting-edge AI researchers, are.
So what is going on?
2023 Prediction Evaluations
by Zvi — 2024-01-08T14:40:07.377Z

It is that time of the year. One must ask not only whether predictions were right or wrong, whether one won or lost, but what one was and should have been thinking, whether or not good decisions were made, whether the market made sense.

The main subject will be the 2023 ACX Predictions, where I performed buy/sell/hold along with sharing my logic. The numbers quoted are from mid-February 2023, first Manifold, then Metaculus.

Section 1: World Politics

Will Vladimir Putin be President of Russia at the end of 2023 (85%/90%)?

Last year I thought markets were too confident Putin would keep power. This year I think this is not confident enough and Metaculus is more accurate at 90%. Metaculus is also doing a better job adjusting as time passes. Things seem to be stabilizing, and every day without big bad news is good news for Putin here on multiple levels. I bought M500 of YES shares, which moved this to 86%.

I increased my position later, and won M179. The market had occasional spikes downward when Putin looked to potentially be in danger, and for a while it failed to decay properly.

Looking back, there was clearly risk that events in Ukraine could have led to Putin’s ouster, and he also could have head health issues. It was clear that I could have gotten much better per diem odds later in the year. So even though I won this bet, I don’t think it was especially good, and Metaculus was overconfident.

Will Ukraine control the city of Sevastopol at the end of 2023 (14%/8%)?

Getting Sevastopol is a heavy lift. Russia is not about to abandon it, Ukraine has other priorities first, and Ukraine’s ability to go on offensives is far from unlimited even in good scenarios. Metaculus is at 8% and once again that sounds more right to me. I bought M250 of NO here and M250 of NO in another similar market that was trading modestly higher, driving the price here to 13%.

I think this was a good bet. Certainly Russia could have completely collapsed, but even then holding onto Crimea was likely. I won M52 here and M50 in the other market. There wasn’t much decay until the second half of the year, but also things looked good for Ukraine for a while, so I think the market acted reasonably relative to itself.

Will Ukraine control the city of Luhansk at the end of 2023 (28%/13%)?

This spent a bunch of time near 50% in early January, then went down.

...
There is no sharp boundary between deontology and consequentialism
by quetzal_rainbow — 2024-01-08T11:01:47.828Z

The purpose of this post is to serve as storage for particular thought so I can link it in dialogues.

I've seen logic along these lines several times:

Consequentialist reasoning is more complex than following deontological rules;
Neural networks have some kind of simplicity bias;
Therefore, neural networks will learn something similar to deontological rules and won't be scary consequentialists.

I think this reasoning is wrong because it relies on hidden assumptions about the qualitative difference between consequentialism and deontology. The rule "assign utility to every outcome and select the action that maximizes expected utility" is a rule. The reasoning "I am bad at evaluating outcomes and most decisions are not so important anyway, so I should use these reliable rules-of-thumb" is consequentialist reasoning. I often think about "utilitarian virtues," like "the ability to shut up and multiply in difficult situations" or "actually implementing the plan you designed."

Another hidden assumption is an absolute understanding of simplicity: deontological rules belong to the realm of "simple" things, and consequentialist plans belong to the realm of "complex" things. But this is definitely not true when training superintelligence. If we have an entity that can design non-trivial 100-step plans with a non-negligible chance of success, such an entity can operate non-trivial 5-step plans as easily as if they were simple heuristics from our perspective.
Reflections on my first year of AI safety research
by Jay Bailey — 2024-01-08T07:49:08.147Z

Last year, I wrote a

post about my upskilling in AI alignment

. To this day, I still get people occasionally reaching out to me because of this article, to ask questions about getting into the field themselves. I’ve also had several occasions to link people to the article who asked me about getting into the field from other means, like my local AI Safety group.
Essentially, what this means is that people clearly found this useful (credit to the EA Forum for managing to let the article be findable to those who need it, a year after its publication!) and therefore people would likely find a sequel useful too! This post is that sequel, but reading the first post is not necessary to read this one.

The major lesson of this post is this: I made a ton of mistakes, but those mistakes taught me things. By being open to that feedback and keeping my eye on the ball, I managed to find work that suited me in the field eventually. Just like the previous post, I’m happy to answer more questions via PM or in the comments.

It’s worth noting, this isn’t a bold story of me getting a ton of stuff done. Most of the story, by word count, is me flailing around unsure of what to do and making a lot of mistakes along the way. I don’t think you’ll learn a lot about how to be a good researcher from this post, but I hope you might learn some tips to avoid being a bad one.

Summary

I was a software engineer for 3-4 years with little to no ML experience before I was accepted for my initial upskilling grant. (More details are in my

initial post

)
I attended

SERI MATS

, working on aligning language models under Owain Evans. Due to a combination of factors, some my fault and some not, I don’t feel like I got a great deal of stuff done.
I decided to pivot away from evals towards mechanistic interpretability since I didn’t see a good theory of change for evals - this was two weeks before GPT-4 came out and the whole world sat up and took notice. Doh!
After upskilling in mechanistic interpretability, I struggled quite a bit with the research. I eventually concluded that it wasn’t for me, but was already funded to work on it. Fortunately I had a collaborator, and eventually I wound up using my engineering skills to accelerate his research instead of trying to contribute to the analysis directly.
After noticing my theory of change for evals had changed now that governments and labs were c

...