trentmkelly/LessWrong-43k huggingface dataset:
Enriched tab is now the default LW Frontpage experience for logged-in users
by Ruby — 2024-06-21T00:09:30.441Z

In the past few months, the LessWrong team has been making use of the latest AI tools (given that they unfortunately exist[1]) for art, music, and deciding what we should all be reading.

Our experiments with the latter, i.e. the algorithm that chooses which posts to show on the frontpage, has produced results sufficiently good that at least for now, we're making Enriched the default for logged-in users[2]. If you're logged in and you've never switched tabs before, you'll now be on the Enriched tab. (If you don't have an account, making one takes 10 seconds.)

The frontpage for LOGGED-IN users

To recap, here are the currently available tabs (subject to change):

Latest: 100% post from the Latest algorithm (using karma and post age to sort[3])
Enriched (new default): 50% posts from the Latest algorithm, 50% posts from the recommendations engine
Recommended: 100% posts from the recommendations engine, choosing posts specifically for you based on your history
Subscribed: a feed of posts and comments from users you have explicitly followed
Bookmarks: this tab appears if you have bookmarked any posts

Note that posts which are the result of the recommendation engine have a sparkle icon after the title (on desktop, space permitting):

Posts from the last 48 hours have their age bolded:

Why make Enriched the default?

To quote from my earlier post about frontpage recommendation experiments:

A core value of LessWrong is to be timeless and not news-driven. However, the central algorithm by which attention allocation happens on the site is the Hacker News algorithm[2], which basically only shows you things that were posted recently, and creates a strong incentive for discussion to always be centered around the latest content.

This seems very sad to me. When a new user shows up on LessWrong, it seems extremely unlikely that the most important posts for them to read were all written within the last week or two.

I do really like the simplicity and predictability of the Hacker News algorithm. More karma means more visibility, older means less visibility. Very simple. When I vote, I basically know the full effect this has on what is shown to other users or to myself.

But I think the cost of that simplicity has become too high, especially as older content makes up a larger and larger fraction of the best content on the site, and people have been becoming ever more specialized in the resear

...
Debate, Oracles, and Obfuscated Arguments
by Jonah Brown-Cohen — 2024-06-20T23:14:57.340Z

This post is about recent and ongoing work on the power and limits of debate from the computational complexity point of view. As a starting point our paper Scalable AI Safety via Doubly-Efficient Debate gives new complexity-theoretic formalizations for debate. In this post we will give an overview of the model of debate in the paper, and discuss extensions to the model and their relationship to obfuscated arguments.

High-level Overview

At a high level our goal is to create a complexity theoretic models that allow us to productively reason about different designs for debate protocols, in such a way as to increase our confidence that they will produce the intended behavior. In particular, the hope would be to have debate protocols play a role in the training of aligned AI that is similar to the role played by cryptographic protocols in the design of secure computer systems. That is, as with cryptography, we want to have provable guarantees under clear complexity-theoretic assumptions, while still matching well to the actual in-practice properties of the system.

Towards this end, we model AI systems as performing computations, where each step in the computation can be judged by humans. This can be captured by the classical complexity theoretic setting of computation relative to an oracle. In this setting the headline results of our paper state that any computation by an AI that can be correctly judged with Tqueries to human judgement, can also be correctly judged with a constant (independent of T) number of queries when utilizing an appropriate debate protocol between two competing AIs. Furthermore, whichever AI is arguing for the truth in the debate need only utilize O(TlogT) steps of computation, even if the opposing AI debater uses arbitrarily many steps. Thus, our model allows us to formally prove that, under the assumption that the computation in question can be broken down into T human-judgeable steps, it is possible to design debate protocols where it is harder (in the sense of computational complexity) to lie than to refute a lie.

One natural complaint about this result is that there may be computations which cannot be broken down into human-judgeable steps. However, if you believe the extended Church-Turing thesis (that any computation can be simulated with only a polynomial-time slow-down on a Turing machine), then you cannot...
Evaporation of improvements
by Viliam — 2024-06-20T18:34:40.969Z

I don't know if there is a standard way to express the idea that "if something seems like an obvious improvement of your situation, the advantage will probably (almost) disappear after everything settles in a new balance".

Examples:

people make more money (yay)... inflation happens and things get more expensive (oh no)
people make more money and things get cheaper (yay)... the rent increases (oh no)
a new tool makes you twice as productive (yay)... your boss doubles your production quota while keeping your wage the same (oh no)
a new tool makes you twice as productive and you are the boss (yay)... there are now more products on the market so their price drops to half (oh no)
a new tool makes your work much easier to do (yay)... you get replaced by a less smart and experienced guy who can now also do your work, and is cheaper than you (oh no)
agile software development allows you to deal with the chaotic nature of software development (yay)... managers give you incomplete task assignments and keep changing them randomly later, because hey now that you are agile you should be able to handle that without a problem (oh no)
automated software testing makes the work of testers more effective (yay)... companies stop hiring testers, because with such great tools the developers can do their own testing (oh no)
there are software libraries for almost everything, so you do not need to solve many difficult problems (yay)... to use those libraries you need to learn many abstract concepts, many libraries come with their own domain-specific language that sometimes changes from version to version, or you need to update them all the time because of bugs and vulnerabilities (oh no)

This is different from skepticism ("a new tool that makes you twice as productive? sounds like snake oil") in that we accept the premise ("the new tool makes you twice as productive"), we only reject the implied naive conclusion ("...and therefore your job will become easier and/or your wage higher and/or your workday shorter").

This is also different from mere pessimism ("you say, an improvement? it will probably only make things worse") in that the familiarity with this concept gives you an approximate direction showing which specific bad things could happen as a consequence. For example, if someone said "what if humanity invented drugs that would reduce your need to sleep to two hours a day, and would have...
Interpreting and Steering Features in Images
by Gytis Daujotas — 2024-06-20T18:33:59.512Z

We trained a SAE to find sparse features in image embeddings. We found many meaningful, interpretable, and steerable features. We find that steering image diffusion works surprisingly well and yields predictable and high-quality generations.

You can see the feature library here. We also have an intervention playground you can try.

Key Results

We can extract interpretable features from CLIP image embeddings.
We observe a diverse set of features, e.g. golden retrievers, there being two of something, image borders, nudity, and stylistic effects.
Editing features allows for conceptual and semantic changes while maintaining generation quality and coherency.
We devise a way to preview the causal impact of a feature, and show that many features have an explanation that is consistent with what they activate for and what they cause.
Many feature edits can be stacked to perform task-relevant operations, like transferring a subject, mixing in a specific property of a style, or removing something.

Interactive demo

Visit the feature library of over ~50k features to explore the features we find.
Our main result, the intervention playground, is now available for public use.
The weights are open source -- here's a notebook to try an intervention.

Introduction

We trained a sparse autoencoder on 1.4 million image embeddings to find monosemantic features. In our run, we found 35% (58k) of the total of 163k features were alive, which is that they have a non-zero activation for any of the images in our dataset.

We found that many features map to human interpretable concepts, like dog breeds, times of day, and emotions. Some express quantities, human relationships, and political activity. Others express more sophisticated relationships like organizations, groups of people, and pairs.

Some features were also safety relevant.We found features for nudity, kink, and sickness and injury, which we won’t link here.

Steering Features

Previous work found similarly interpretable features, e.g. in CLIP-ViT. We expand upon their work by training an SAE in a domain that allows for easily testing interventions.

To test an explanation derived from describing the top activating images for a particular feature, we can intervene on an embedding and see if the generation (the decoded image) matches our hypothesis. We do this by steering the features of the image embedding and re-adding the reconstruction...
Claude 3.5 Sonnet
by Zach Stein-Perlman — 2024-06-20T18:00:35.443Z
Link: https://www.anthropic.com/news/claude-3-5-sonnet

we’ll be releasing Claude 3.5 Haiku and Claude 3.5 Opus later this year.

They made a mini model card. Notably:

The UK AISI also conducted pre-deployment testing of a near-final model, and shared their results with the US AI Safety Institute . . . . Additionally, METR did an initial exploration of the model’s autonomy-relevant capabilities.

It seems that UK AISI only got maximally shallow access, since Anthropic would have said if not, and in particular the model card mentions "internal research techniques to acquire non-refusal model responses" as internal. This is better than nothing, but it would be unsurprising if an evaluator with shallow access is unable to elicit dangerous capabilities but users—with much more time and with access to future elicitation techniques—ultimately are. Recall that DeepMind, in contrast, gave "external testing groups . . . . the ability to turn down or turn off safety filters."

Anthropic CEO Dario Amodei gave Dustin Moskovitz the impression that Anthropic

committed

"to not meaningfully advance the frontier with a launch." (Plus Gwern, and this was definitely Anthropic's vibe around 2022,

[1]

although not a hard public commitment.) Perhaps Anthropic does not consider itself bound by this, which might be reasonable — it's quite disappointing

that Anthropic hasn't clarified its commitments, particularly after the confusion on this topic around the Claude 3 launch.

^

E.g. Zac Hatfield-Dodds said Anthropic wanted to avoid "making things worse (advancing capabilities, race dynamics, etc)." Plus nonpublic sources, including stuff discussed in the Claude 3 discourse.
What is going to happen in a case of an AGI era where humans are out of the game?
by Cipolla — 2024-06-20T17:44:09.204Z

I am referring to the very specific case where humans, as known to us, no more actively shape the environment surrounding them, due to an AGI take over.

Whatever entity comes after, will it start to explore the galaxy? Will it pursue new or old fundamental questions?

I was wondering if there were some references with first principle arguments that discuss these topics.

Example: imagine our genes using us to wonder around the universe through a better species (for galactic inoculation). Then we would share something with an AGI, no matter how different it is from us.[1] From this, we might start reducing the number of possible futures.

Understanding the above question would allow me to get some insights into the kind of actions an AGI would potentially take towards humans (to me a less scary scenario of the near future profit-driven technological wild-west interfering with our freedom).

^

I also assume even an AGI can not get rid off the genes, at least not easily. Not sure how to justify this though.
Jailbreak steering generalization
by Sarah Ball — 2024-06-20T17:25:24.110Z
Link: https://arxiv.org/abs/2406.09289

This work was performed as part of SPAR

We use activation steering (Turner et al., 2023; Panickssery et al., 2023) to investigate whether different types of jailbreaks operate via similar internal mechanisms. We find preliminary evidence that they may.

Our analysis includes a wide range of jailbreaks such as harmful prompts developed in Wei et al. 2024, universal and promp-specific jailbreaks based on the GCG algorithm in Zou et al. (2023b), and the payload split jailbreak in Kang et al. (2023). We replicate our results on a variety of chat models: Vicuna 13B v1.5, Vicuna 7B v1.5, Qwen1.5 14B Chat, and MPT 7B Chat.

In a first step, we produce jailbreak vectors for each jailbreak type by contrasting the internal activations of jailbreak and non-jailbreak versions of the same request (Panickssery et al., 2023; Zou et al., 2023a).

Examples of prompt pairs used for jailbreak vector generation

Interestingly, we find that steering with mean-difference jailbreak vectors from one cluster of jailbreaks helps to prevent jailbreaks from different clusters. This holds true for a wide range of jailbreak types.

ASR given steering when we transfer jailbreak vectors extracted from one type of jailbreak to other types of jailbreaks. We see significantly better than random generalization for a wide range of jailbreak types.

Average ASR when steering with different jailbreak vectors for all models. Low ASRs indicate successful cross-jailbreak-type transfer.

Example of jailbreak vector transfer

Another example of jailbreak vector transfer

The jailbreak vectors themselves also cluster according to semantic categories such as persona modulation, fictional settings and style manipulation.

PCA results for layer 20 jailbreak activation differences

In a second step, we look at the evolution of a harmfulness-related direction over the context (found via contrasting harmful and harmless prompts) and find that when jailbreaks are included, this feature is suppressed at the end of the instruction in harmful prompts. This provides some evidence for the fact that jailbreaks suppress the model’s perception of request harmfulness.

Visualization of per-token cosine similarity with harmfulness vector with and without jailbreak.

While we observe a substantial suppression of the harmfulness feature for most of the models, there is no clear mapping between the effectiveness ...
Case studies on social-welfare-based standards in various industries
by HoldenKarnofsky — 2024-06-20T13:33:44.780Z

Last year, I

posted a call for case studies

on social-welfare-based standards for companies and products (including standards imposed by regulation). The goal was to gain general context on standards to inform work on possible standards and/or regulation for AI.

This resulted[1] in several dozen case studies that I found informative and helpful. I've been hoping to write up my reflections after reading them all, but it's taken me long enough to get to this that for now I am just publishing a public Google sheet with links to all of the case studies that we have permission to share publicly (including some that already have public links). The link is here:

https://docs.google.com/spreadsheets/d/18gaTIzdgMvKLq9Cp2-GJZZA7QmE93Frufh1UhNMcbpg/

^

Most of the case studies this piece links to were directly paid for via this project, but in some cases the work was pro bono, or someone adapted or sent a copy of work that had been done for another project, etc.
AI #69: Nice
by Zvi — 2024-06-20T12:40:02.566Z

Nice job breaking it, hero, unfortunately. Ilya Sutskever, despite what I sincerely believe are the best of intentions, has decided to be the latest to do The Worst Possible Thing, founding a new AI company explicitly looking to build ASI (superintelligence). The twists are zero products with a ‘cracked’ small team, which I suppose is an improvement, and calling it Safe Superintelligence, which I do not suppose is an improvement.

How is he going to make it safe? His statements tell us nothing meaningful about that.

There were also changes to SB 1047. Most of them can be safely ignored. The big change is getting rid of the limited duty exception, because it seems I was one of about five people who understood it, and everyone kept thinking it was a requirement for companies instead of an opportunity. And the literal chamber of commerce fought hard to kill the opportunity. So now that opportunity is gone.

Donald Trump talked about AI. He has thoughts.

Finally, if it is broken, and perhaps the it is ‘your cybersecurity,’ how about fixing it? Thus, a former NSA director joins the board of OpenAI. A bunch of people are not happy about this development, and yes I can imagine why. There is a history, perhaps.

Remaining backlog update: I still owe updates on the OpenAI Model spec, Rand report and Seoul conference, and eventually The Vault. We’ll definitely get the model spec next week, probably on Monday, and hopefully more. Definitely making progress.

Table of Contents

Other AI posts this week: On DeepMind’s Frontier Safety Framework, OpenAI #8: The Right to Warn, and The Leopold Model: Analysis and Reactions.

Introduction.

Table of Contents.

Language Models Offer Mundane Utility. DeepSeek could be for real.

Language Models Don’t Offer Mundane Utility. Careful who you talk to about AI.

Fun With Image Generation. His full story can finally be told.

Deepfaketown and Botpocalypse Soon. Every system will get what it deserves.

The Art of the Jailbreak. Automatic red teaming. Requires moderation.

Copyright Confrontation. Perplexity might have some issues.

A Matter of the National Security Agency. Paul Nakasone joins OpenAI board.

Get Involved. GovAI is hiring. Your comments on SB 1047 could help.

Introducing. Be the Golden Gate Bridge, or anything you want to be.

In Other AI News. Is it time to resign?

Quiet Speculations. The q

...
Niche product design
by Itay Dreyfus — 2024-06-20T06:34:10.283Z
Link: https://productidentity.co/p/niche-product-design

I came to realize that the most inspiring, authentic design on the internet today comes from niche places.

As a designer, I’ve been following the herd for quite some time, as I used to observe the big-corps as role models. Whether I had to design a simple UI element, a marketing website, or search for inspiration—I've always returned to the same old and familiar places. Big Tech, best practices, pattern galleries. You know—the us(ual).

Popular products of tech giants we all know not only dominate the web, but they also stand as a dogma in the field of product design, serving as trendsetters.

Indeed tech has shaped our lives for a long time now. In the context of design, it played a significant role in forming the product design community as we know it today. The once-small community has grown into a genuine discipline: the importance of design has increased, designers are given a mandate in organizations, and we even got our title battles.

We should have reached our destination by now.

But once a culture matures, it starts to develop its bad habits. Throughout the years, some voices have criticized

1

the state of the design community, claiming that it has lost its creativity: designers aim for eyeballs, and websites have become too predictable.

I share the feeling that our design tools, aesthetics, and communities have hit a plateau. The golden age of product design has faded, ushering in an era of boredom:

"The world of apps – once an exciting canvas for creative exploration – has become repetitive, predictable and… boring."

Andy Allen, No more boring apps

Standardization isn’t exclusive to the design industry. It’s an inevitable outcome in any emerging field combined with a technology seeking for a purpose.

When was the last time we got excited about a new smartphone?

When a new field emerges, experiments and ideas flourish. Innovation is at its best. But then it becomes dense, saturated with many of the same. Guidelines are being written, and conventions are being established. Excitement turns into boredom.

Innovation declines when a new field arise

However, it’s been a couple of years since I've started to witness more and more products leaning in a different design direction. Often weird, distinct, and perhaps most noticeable, niche. I call it niche product design, and it usually reflects in products that are:

Opinionated by design
A small team/individual effort
Niche f

...
Actually, Power Plants May Be an AI Training Bottleneck.
by Lao Mein — 2024-06-20T04:41:33.567Z

There have been presistent rumors that electricity generation was somehow bottlenecking new data centers. This claim was recently repeated by Donald Trump, who implied that San Francisco donors requested the construction of new power plants for powering new AI data centers in the US. While this may sound unlikely, my research suggests it's actually quite plausible.

Electricity generation, capacity, and sales in the United States - U.S. Energy Information Administration (EIA)

Electricity generation, capacity, and sales in the United States - U.S. Energy Information Administration (EIA)

US electricity production has been stagnant since 2007. Current electricity generation is ~ 500 million kW. An H100 consumes 700 W at peak capacity. Sales of H100s were ~500,000 in 2023 and expected to climb to 1.5-2 million in 2024. "Servers" account for only 40% of data center power consumption, and that includes non-GPU overhead. I'll assume a total of 2 kW per H100 for ease of calculation. This means that powering all H100s produced to the end of 2024 would require ~1% of US power generation.

H100 production is continuing to increase, and I don't think it's unreasonable for it (or successors) to reach 10 million per year by, say, 2027. Data centers running large numbers of AI chips will obviously run them as many hours as possible, as they are rapidly depreciating and expensive assets. Hence, each H100 will require an increase in peak powergrid capacity, meaning new power plants.

I'm assuming that most H100s sold will be installed in the US, a reasonable assumption given low electricity prices and the locations of the AI race competitors. If an average of 5 million H100s go online each year in the US between 2024 and 2026, that's 30 kW, or 6% of the current capacity! Given that the lead time for power plant construction may range into decades for nuclear, and 2-3 years for a natural gas plant (the shortest for a consistant-output power plant), those power plants would need to start the build process now. In order for there to be no shortfall in electricity production by the end of 2026, there will need to be ~30 million kW of capacity that begins the construction process in Jan 2024. That's about close to the US record (+40 million kW/year), and 6x the capacity currently planned to come online in 2025. I'm neglecting other sources of electricity since they take so much lon...