trentmkelly/LessWrong-43k huggingface dataset:
New LessWrong review winner UI ("The LeastWrong" section and full-art post pages)
by kave — 2024-02-28T02:42:05.801Z

(Also announcing: annual review prediction markets & full-height table of contents. If you're looking for this year's review results, you can find them here)

The top 50 posts of each of LessWrong’s annual reviews have a new home: The LeastWrong.

What will I see when I click that link?

You will find the posts organized into six “books”: Rationality, Optimization, World, Practical, AI Strategy, and Technical AI Safety. Each square on the grid is a post that made the top 50 of the review in some year. The collections are ordered more-or-less with the aim of putting the accessible posts in the most prominent spots.

If you're logged-in the essays will be dark until you've read them, to help guide you to posts you've not read before.

Each of the essays on this page also now has its own full-bleed art and updated post page:

The tiny gold icons are the top reviews for that post

How can I see more of a book?

If you click on the name of a book, like “Rationality”, you’ll get a full width view of the content.

You can also sort by year

Any other goodies?

Full height table of contents with a progress bar!

Markets on whether posts with over 100 karma will make it in to the top 50 of their year's review!

With golden highlighted karma if it's predicted with more than 50%!

Why did you make this?

Many people on the LW team wanted to celebrate the top posts in the review.

Historically we've printed annual review books, but only a small fraction of people who read the essays got to experience the books, and the effort that went into the books felt disconnected from the rest of the site. They also took a really long time to make, and required constant ongoing attention from the Lightcone team to handle logistics of shipping and sales and new print runs.

It seemed more appropriate to put effort into making the reading experience of these essays on LessWrong itself a more memorable and rewarding experience.

But what were the results of this year's review?

Read all about it in Ben's post!

That's it

I hope some of you really like it. And report any bugs with this new stuff in intercom in the bottom right.
Counting arguments provide no evidence for AI doom
by Nora Belrose — 2024-02-27T23:03:49.296Z

Crossposted from the AI Optimists blog.

AI doom scenarios often suppose that future AIs will engage in scheming— planning to escape, gain power, and pursue ulterior motives, while deceiving us into thinking they are aligned with our interests. The worry is that if a schemer escapes, it may seek world domination to ensure humans do not interfere with its plans, whatever they may be.

In this essay, we debunk the counting argument— a central reason to think AIs might become schemers, according to a recent report by AI safety researcher Joe Carlsmith.[1] It’s premised on the idea that schemers can have “a wide variety of goals,” while the motivations of a non-schemer must be benign by definition. Since there are “more” possible schemers than non-schemers, the argument goes, we should expect training to produce schemers most of the time. In Carlsmith’s words:

The non-schemer model classes, here, require fairly specific goals in order to get high reward.
By contrast, the schemer model class is compatible with a very wide range of (beyond episode) goals, while still getting high reward…
In this sense, there are “more” schemers that get high reward than there are non-schemers that do so.
So, other things equal, we should expect SGD to select a schemer.

— Scheming AIs, page 17

We begin our critique by presenting a structurally identical counting argument for the obviously false conclusion that neural networks should always memorize their training data, while failing to generalize to unseen data. Since the premises of this parody argument are actually stronger than those of the original counting argument, this shows that counting arguments are generally unsound in this domain.

We then diagnose the problem with both counting arguments: they rest on an incorrect application of the principle of indifference, which says that we should assign equal probability to each possible outcome of a random process. The indifference principle is controversial, and is known to yield absurd and paradoxical results in many cases. We argue that the principle is invalid in general, and show that the most plausible way of resolving its paradoxes also rules out its application to an AI’s behaviors and goals.

More generally, we find that almost all arguments for taking scheming seriously depend on unsound indifference reasoning. Once we reject the indifference principle, there is very little reas...
Which animals realize which types of subjective welfare?
by MichaelStJules — 2024-02-27T19:31:32.854Z

Summary

In a

previous piece

, I defined and discussed four potential types of subjective welfare: hedonic states, felt desires, belief-like preferences and choice-based preferences. Setting aside the moral question of which types of welfare matter intrinsically, humans are of course capable of all four types of capacities. Many other animals seem capable of (conscious) hedonic states and felt desires, and plenty has been written on the topic (e.g.

Muehlhauser, 2017

and

Waldhorn et al., 2019

), but this is not entirely uncontroversial. I will not focus on these. Instead, my focus and key takeaways here are the following:

It doesn’t seem too unlikely that belief-like preferences are also available to animals with conscious hedonic states and conscious felt desires and even more likely still in mammals (and probably birds) in general, with these conscious hedonic states and conscious felt desires being or grounding belief-like preferences, but more sophisticated capacities could be required (

more

).
I make a probably controversial case for global belief-like preferences — e.g. belief-like preferences about the individual’s own life as a whole — in other animals, as potentially degenerate or limiting cases: global belief-like preferences could just consider only the immediate, and so conscious hedonic states or felt desires could qualify or ground them (

more

).
It’s unclear if choice-based preferences are also available to animals with conscious hedonic states and conscious felt desires; this could be fairly sensitive to our characterization of choice-based preferences (

more

).

Acknowledgements

Thanks to Brian Tomasik, Derek Shiller and Bob Fischer for feedback on earlier drafts. All errors are my own.

Absence of evidence is a weak argument

Before commenting further on specifics, we should be careful not to treat absence of evidence for a given capacity in an animal as strong evidence for its absence. No one may have set up the right experiment or observed them under the right conditions to find evidence for or against a given capacity. The number of cognitive capacities recognized in other animals has also grown substantially over the past decades, although there may also be false positives, overinterpretation and publication bias. This could warrant assigning at least non-tiny probabilities, like 10%, to the possibility that an animal has a given cap...
Biosecurity and AI: Risks and Opportunities
by Steve Newman — 2024-02-27T18:45:43.556Z
Link: https://www.safe.ai/post/biosecurity-and-ai-risks-and-opportunities

Recent decades have seen massive amounts of biological and medical data becoming available in digital form. The computerization of lab equipment, digitization of medical records, and advent of cheap DNA sequencing all generate data, which is increasingly collected in large data sets available to researchers.

This bounty of data is driving rapid progress in AI. In combination with increasingly cheap and powerful DNA synthesis and laboratory automation, AI promises to deliver revolutionary advances in medicine via rapid design-build-test cycles. However, the same capabilities that are driving everything from drug discovery to diagnostic techniques could also revolutionize bioterrorism, with plausible scenarios ranging from a flu-level virus to a pandemic with an impact exceeding Covid.

Fortunately, there is much that can be done to mitigate the risk of an AI-assisted pandemic. This is often framed as a conflict between progress and safety, but that needn’t be the case. Many mitigations are simply sensible public health measures, such as improved ventilation in public spaces. Where it is necessary to manage the development of AI capabilities, this can be done in a targeted manner. We have the opportunity to simultaneously improve public health, reduce the risk of pandemics, and smooth the path for progress.

How AI Could Enable Bioterrorism

There are multiple paths through which advances in AI could lead to the deliberate release of a harmful virus. One scenario hinges on the fact, not widely recognized, that the genetic sequences for tens of thousands of human viruses (presenting varying degrees of danger) are already widely available.[1] Advances in DNA/RNA synthesis make it theoretically possible for a disturbed or radicalized individual to recreate a virus; the greatest barrier is knowledge of lab techniques.[2] For many viruses, that barrier is surprisingly low. If the individual had managed to identify a particularly dangerous virus, tens or even hundreds of thousands of deaths could result, comparable to the atomic bombs dropped on Hiroshima and Nagasaki.[3]

Future chatbots seem likely to be capable of lowering the bar to such an attack. As models become increasingly “multimodal”, their training data will soon include video, such as university lectures and lab demonstrations. Such systems would not be limited to providing written instructions; they c...
The Gemini Incident Continues
by Zvi — 2024-02-27T16:00:05.648Z

Previously: The Gemini Incident (originally titled Gemini Has a Problem)

The fallout from The Gemini Incident continues.

Also the incident continues. The image model is gone. People then focused on the text model. The text model had its own related problems, some now patched and some not.

People are not happy. Those people smell blood. It is a moment of clarity.

Microsoft even got in on the act, as we rediscover how to summon Sydney.

There is a lot more to discuss.

The Ultimate New York Times Reaction

First off, I want to give a shout out to The New York Times here, because wow, chef’s kiss. So New York Times. Much pitchbot.

Dominic Cummings: true art from NYT, AI can’t do this yet

This should be in the dictionary as the new definition of Chutzpah.

Do you see what The New York Times did there?

They took the fact that Gemini systematically refused to create images of white people in most circumstances, including historical circumstances where everyone involved would almost certainly be white. Where requests to portray white people were explicitly replied to by a scolding that the request was harmful, while requests for people of other races were eagerly honored.

They then turned this around, and made it about how this adjustment was unfairly portraying people of color as Nazis. That this refusal to portray white people under almost all circumstances was racist, not because it was racist against white people, but because it was racist against people of color.

As I discuss, we may never know to what extent was what Google did accidental versus intentional, informed versus ignorant, dysfunction versus design.

We do know that what The New York Times did was not an accident.

This should update us that yes, there very much are people who hold worldviews where what Google did was a good thing. They are rare in most circles, only one person in my Twitter firehoses has explicitly endorsed the fourth stage of clown makeup, but in certain key circles they may not be so rare.

To be fair they also have Ross Douthat on their opinion page, who engages reasonably with the actual situation given his non-technical perspective, noticing that if AI is going to get a lot more powerful soon then yes the whole thing is rather concerning.

The Ultimate Grimes Reaction

One can also look at all this from another perspective, Grimes notes, as art of the highest order...
How I internalized my achievements to better deal with negative feelings
by Raymond Koopmanschap — 2024-02-27T15:10:24.149Z

Whenever I struggle to make progress on an important goal, I feel bad. I get feelings of frustration, impatience, and apathy. I think to myself, “I have wasted all this time, and I will never get it back.” The resulting behavior during these moments does not help either; my impatience makes it hard to concentrate, so I often work on more engaging tasks rather than the essential ones I ideally want to focus on. I also tend to push through; even if I feel tired, I want to make progress at all costs. I force myself to work, which results in decreased motivation, making it hard to make actual progress.

Thanks to a practice called HEAL, introduced in the book Hardwiring Happiness by Rick Hanson, I now deal much better with this situation. HEAL stands for Having a beneficial experience, Enriching it, Absorbing it, and optionally Linking it to a negative experience. To dive straight in and use HEAL in practice, you can explore this guided HEAL meditation. More meditations can be found here, at the end of the Hardwiring Happiness book, and most of the meditations I found useful are in his Foundations of Wellbeing course (you can apply for scholarships).

The book suggests that behavior like my frustration can be caused by some underlying unmet need, resulting in compulsively trying to fulfill this need. This information and introspective techniques like Focusing helped me discover that these negative feelings came from some unmet need to feel worthwhile and recognized, but the problem was that I heavily tied my self-worth to the amount of progress I made.

HEAL allowed me to fulfill this need and thereby soothe these negative feelings by generating positive experiences of past accomplishments and letting the truth of these facts sink in by enriching and absorbing the experience, allowing me to see that I have made significant progress and am proud of what I have achieved. This helped me put these negative thoughts in perspective and let me realize on a deeper level that I am OK and capable of achieving meaningful things.

I feel calmer after doing this practice; it allows me to disengage from negative thought loops. When I have more distance from a negative thought, I ask myself what I can learn from this feeling and what is helpful for me at this moment, be it going for a short walk, talking with a friend about my frustration, or refocusing on the important task I wanted ...
On Frustration and Regret
by silentbob — 2024-02-27T12:19:55.439Z

I've always been drawn to the palpable aspects of life – theories, behaviors, and ideas that shape our reality and our place within it. However, this post marks a departure towards something more introspective and, perhaps, a tad spiritual. It's an exploration of personal philosophies that I grew attached to over the years as they have helped me get along in the complicated mess that is our reality, and our peculiar little existence within it. Maybe some people will find something of value in these framings of frustration, preferences and regret.
Granted, none of these ideas are particularly original, and you’ll easily find traces of buddhism, stoicism and other influences within them. But sometimes hearing a familiar idea in unfamiliar words can make all the difference.

GPT4 thought this was a suitable picture.

Frustration

Frustration is a feeling all too familiar to many of us. Things do not go as planned, others behave in undesirable ways, sometimes it’s even ourselves causing our own frustration. It’s natural and kind of makes sense: we have some expectation of how things ought to be, this expectation gets violated, we’re unhappy. But what is this feeling really? In a sense frustration means clinging to a version of reality that is not actual. It’s a fundamental lack of acceptance of what is. Does this make sense? Clinging to something that doesn’t actually exist? Summoning unpleasant feelings because we disagree with reality?

One way in which frustration may indeed be useful is that it can motivate us to make things better. From an evolutionary point of view it makes sense that frustration emerged, rather than universal acceptance. If we never strive for the betterment of our circumstances, we eventually perish. And suffering, such as in the form of experiencing a feeling of frustration, is a great motivator. But we evolved to a point where we can actually make deals with ourselves. It’s possible to exercise acceptance, to let go of frustration, accept the world and your circumstances as they are, and simply go from there and make the necessary adjustments out of freedom and agency, rather than out of a place of frustration.

As noted before, this is nothing new. Similar claims have been made for thousands of years. Maybe we all need to get to this conclusion at our own pace. But even then, the conclusion itself is not enough – letting go of frustration i...
Examining Language Model Performance with Reconstructed Activations using Sparse Autoencoders 
by Evan Anders — 2024-02-27T02:43:22.446Z

Note: The second figure in this post originally contained a bug pointed out by @LawrenceC, which has since been fixed.

Summary

Sparse Autoencoders (SAEs) reveal interpretable features in the activation spaces of language models, but SAEs don’t reconstruct activations perfectly. We lack good metrics for evaluating which parts of model activations SAEs fail to reconstruct, which makes it hard to evaluate SAEs themselves. In this post, we argue that SAE reconstructions should be tested using well-established benchmarks to help determine what kinds of tasks they degrade model performance on.
We stress-test a recently released set of SAEs for each layer of the gpt2-small residual stream using randomly sampled tokens from Open WebText and the Lambada benchmark where the model must predict a specific next token.
The SAEs perform well on prompts with context sizes up to the training context size, but their performance degrades on longer prompts.
In contexts shorter than or equal to the training context, the SAEs that we study generally perform well. We find that the performance of our late-layer SAEs is worse than early-layer SAEs, but since the SAEs all have the same width, this may just be because there are more features to resolve in later layers and our SAEs don’t resolve them.
In contexts longer than the training context, SAE performance is poor in general, but it is poorest in earlier layers and best in later layers.

Introduction

Last year,

Anthropic

and

EleutherAI/Lee Sharkey's MATS stream

showed that sparse autoencoders (SAEs) can decompose language model activations into human-interpretable features. This has led to a significant uptick in the number of people training SAEs and analyzing models with them. However, SAEs are not perfect autoencoders and we still lack a thorough understanding of where and how they miss information. But how do we know if an SAE is “good” other than the fact that it has features we can understand?

SAEs try to reconstruct activations in language models – but they don’t do this perfectly. Imperfect activation reconstruction can lead to substantial downstream cross-entropy (CE) loss increases. Generally “good” SAEs retrieve 80-99% of the CE loss (compared to a generous baseline of zero ablation), but only retrieving 80% of the CE loss is enough to substantially degrade the performance of a model to that of a much...
Project idea: an iterated prisoner's dilemma competition/game
by Adam Zerner — 2024-02-26T23:06:20.699Z

Epistemic effort: mostly just thinking out loud. I spend a few dozen minutes thinking about this myself, and then decided to write this up.

After watching this video by Veritasium about game theory I am wondering whether more people having an understanding of game theory -- iterated prisoners dilemmas in particular -- would lead to more prosocial behavior in the real world.

Here's an example of where I'm coming from. The video talks about a failure mode where retaliatory strategies like tit-for-tat stumble upon a terrible situation where they endlessly defect against one another. Both are "nice" in the sense that they don't intend to defect first, but in the real world there are errors. You intend to cooperate but maybe you screw up and accidentally defect instead. Or maybe the other person screws up and misjudges your cooperation as defection. Then they defect in retaliation. Then you defect in retaliation. So on and so forth.

You need a way out of this vicious cycle. One approach would be, instead of retaliating after every defection, you only retaliate 9 out of 10 times.

I think this sort of thing happens in the real world. For example, maybe your partner does something to you that made you unhappy. You feel wronged. So you retaliate by being cold towards them. After all, you don't want to be a pushover and let such behavior go unpunished.

But your partner never intended to upset you. They don't realize they did anything wrong. But they notice you being cold towards them. And they don't want to let that go unpunished. So they retaliate by being cold towards you. Which you notice, and retaliate against by extending the duration for which you're cold towards them. So on and so forth. This sort of thing can lead to bitterness amongst people that lasts for an embarrassingly long period of time. It can happen amongst not just romantic partners, but amongst friends, family, co-workers, business partners, acquaintances, and more.

Now imagine that you've studied game theory. You understand the downsides of endlessly retaliating. Of sliding the "forgiveness" slider all the way to the left until it hits 0. You understand that this strategy is a poor one. It's obvious to you. You've played in iterated prisoner's dilemma competitions and explored numerous simulations, and it's just very clear how dumb it is to slide that slider all the way to the left. It seems plausible t...