trentmkelly/LessWrong-43k huggingface dataset:
How I got 4.2M YouTube views without making a single video
by Closed Limelike Curves — 2024-09-03T03:52:33.025Z

Just over a month ago, I wrote this.

The Wikipedia articles on the VNM theorem, Dutch Book arguments, money pump, Decision Theory, Rational Choice Theory, etc. are all a horrific mess. They're also completely disjoint, without any kind of Wikiproject or wikiboxes for tying together all the articles on rational choice.

It's worth noting that Wikipedia is the place where you—yes, you!—can actually have some kind of impact on public discourse, education, or policy. There is just no other place you can get so many views with so little barrier to entry. A typical Wikipedia article will get more hits in a day than all of your LessWrong blog posts have gotten across your entire life, unless you're @Eliezer Yudkowsky.

I'm not sure if we actually "failed" to raise the sanity waterline, like people sometimes say, or if we just didn't even try. Given even some very basic low-hanging fruit interventions like "write a couple good Wikipedia articles" still haven't been done 15 years later, I'm leaning towards the latter. edit me senpai

EDIT: Discord to discuss editing here.

An update on this. I've been working on Wikipedia articles for just a few months, and Veritasium just put a video out on Arrow's impossibility theorem, which is almost completely based on my Wikipedia article! Lots of lines and the whole structure/outline of the video are taken almost verbatim from what I wrote. I think there's a pretty clear reason for this. I recently rewrote the entire article to make it easy-to-read and focus heavily on the most important points (Arrow's theorem proves every ranked voting rule has spoilers). It's now very easily-accessible for someone like an educational YouTuber who wants to talk about this topic.

Relatedly, if anyone else knows any educational YouTubers like CGPGrey, Veritasium, Kurzgesagt, or whatever—please let me know! I'd love a chance to talk with them about any of the fields I've done work teaching or explaining (including social or rational choice, economics, math, and statistics).
A gentle introduction to sparse autoencoders
by Nick Jiang — 2024-09-02T18:11:47.086Z

Sparse autoencoders (SAEs) are the current hot topic 🔥 in the interpretability world. In late May, Anthropic released a paper that shows how to use sparse autoencoders to effectively break down the internal reasoning of Claude 3 (Anthropic’s LLM) . Shortly after, OpenAI published a paper successfully applying a similar procedure for GPT4. What’s exciting about SAEs is that, for the first time, they provide a scalable method to peer inside virtually any transformer-based LLM today. In this introduction, I want to unpack the jargon, intuition, and technical details behind SAEs. This post is targeted for technical folks without a background in interpretability; it extracts what I believe are the most important insights in the history leading up to SAEs. I’ll mainly focus on the Anthropic line of research, but there are many research groups, academic and industry, that have played pivotal roles in getting to where we are[1].

Background

Since the moment LLMs took off in the early 2020s, we’ve really only had one clear way to understand how models work: feed some text in, and see what comes out. They’re a black box in all other respects! LLMs are blobs of 1s and 0s that magically speak. The goal of interpretability has always been to give us the tools and methods to understand LLMs beyond input/output.

SAEs are part of a branch of interpretability called mechanistic interpretability, which hypothesizes that LLMs can essentially be reverse-engineered into computer programs. This approach is commonly thought of as “bottom-up” because we’re searching for individual, function-like mechanisms that work together to produce the final output.

In this introduction, you won’t need to understand the nitty-gritty details of LLMs. However, one thing to know is that they’re made up of successive layers of attention blocks and multi-layer perceptions (MLPs), which are neural networks, repeated in structure.

One natural thought is that if we understand what each layer’s output means, we can decipher the “program steps” the LLM undertakes. We call the n-dimensional space of all possible outputs from the MLP the neuron activation space. We tend to study the neuron activation space because the MLP has been thought to perform the “reasoning” role of the LLM.

If we are to take a bottom-up method of reasoning, we would expect that neuron activations can be broken down into more fundame...
What makes math problems hard for reinforcement learning: a case study
by Anibal, Bartek, Sergei, Shehper and Piotr — 2024-09-02T18:11:43.775Z
Link: https://arxiv.org/abs/2408.15332

Abstract: Using a long-standing conjecture from combinatorial group theory, we explore, from multiple angles, the challenges of finding rare instances carrying disproportionately high rewards. Based on lessons learned in the mathematical context defined by the Andrews-Curtis conjecture, we propose algorithmic improvements that can be relevant in other domains with ultra-sparse reward problems. Although our case study can be formulated as a game, its shortest winning sequences are potentially or times longer than those encountered in chess. In the process of our study, we demonstrate that one of the potential counterexamples due to Akbulut and Kirby, whose status escaped direct mathematical methods for 39 years, is stably AC-trivial.

Introduction

We live in an extraordinary era where artificial intelligence (AI) is transforming numerous sectors and professions. Recent advancements in Large Language Models (LLMs) have empowered AI to read, write, and converse with a proficiency comparable to that of human experts. In the realm of board games, AI has outperformed even the most skilled human players, and it has tackled complex scientific challenges like protein folding, where steady progress was suddenly overtaken by a near-complete solution.

As AI continues to evolve, one critical question remains: How wide is the range of domains in which AI systems can reason as effectively as humans? Mathematics appears to be a natural progression on the path toward Artificial General Intelligence (AGI) due to its universal syntactic and logical structure, similar to that of natural language. Additionally, mathematics provides a framework for the quantitative evaluation of logical and analytical reasoning, making it an ideal domain for self-improving AI systems on the path to AGI.

In a moment, we will explain another reason why mathematics could play a crucial role in AGI development, but first, we need to introduce one more key element: reinforcement learning (RL). Machine learning, a subfield of AI, involves developing algorithms and statistical models that enable computers to learn from data and make predictions. Among the three primary areas of machine learning—supervised learning, unsupervised learning, and reinforcement learning—RL emphasizes learning through interaction with an environment and receiving feedback in the form of rewards or penalties. This aspect

...
Survey: How Do Elite Chinese Students Feel About the Risks of AI?
by Nick Corvino — 2024-09-02T18:11:11.867Z

Intro

In April 2024, my colleague and I (both affiliated with Peking University) conducted a survey involving 510 students from Tsinghua University and 518 students from Peking University—China's two top academic institutions. Our focus was on their perspectives regarding the frontier risks of artificial intelligence.

In the People’s Republic of China (PRC), publicly accessible survey data on AI is relatively rare, so we hope this report provides some valuable insights into how people in the PRC are thinking about AI (especially the risks). Throughout this post, I’ll do my best to weave in other data reflecting the broader Chinese sentiment toward AI.

For similar research, check out

The Center for Long-Term Artificial Intelligence

,

YouGov

,

Monmouth University

,

The Artificial Intelligence Policy Institute

, and notably, a poll conducted by

Rethink Priorities

, which closely informed our survey design.

You can read the full report published in the Jamestown Foundation’s China Brief here:

Survey: How Do Elite Chinese Students Feel About the Risks of AI?

Key Takeaways

Students are more optimistic about the benefits of AI than concerned about the harms. 80 percent of respondents agreed or strongly agreed with the statement that AI will do more good than harm for society, with only 7.5 percent actively believing the harms could outweigh the benefits. This, similar to other polling, indicates that the PRC is one of the most optimistic countries concerning the development of AI.
Students strongly believe the Chinese government should regulate AI. 85.31 percent of respondents believe AI should be regulated by the government, with only 6 percent actively believing it should not. This contrasts with trends seen in other countries, where there is typically a positive correlation between optimism about AI and calls for minimizing regulation. The strong support for regulation in the PRC, even as optimism about AI remains high, suggests a distinct perspective on the role of government oversight in the PRC context.
Students ranked AI the lowest among all possible existential threats to humanity. When asked about the most likely causes of human extinction, misaligned artificial intelligence received the lowest score. Nuclear war, natural disaster, climate change, and pandemics all proved more concerning for students.
Students lean tow

...
Data-driven donations to help Democrats win federal elections: an update
by Michael Cohn — 2024-09-02T16:32:02.736Z
Link: https://perplexedguide.net/2024/07/17/donating-to-help-democrats-win-in-the-2024-elections-research-decision-support-and-recommendations/

Linking to an update to an earlier post about how to make effective donations for the upcoming election. There are groups that can turn out 5x as many voters per dollar as the official campaigns can, and I want everyone to know and talk about the cool work they're doing! This late in the cycle, most donations will probably go to linearly-scaling programs like text and mail campaigns, but fortunately, there are approaches that are proven to work even in noisy, heated, heavily-spammed elections like the 2022 generals and GA Senate runoff.

As before I am not assuming that everyone in this community believes that intervening in US elections is worthwhile, or wants Democrats to win, or should believe or want these things, but for those who already do, I hope you find this useful!

Randomized controlled trials found this text from Movement Labs causes people to vote who otherwise would not have voted.

Brief summary of takeaways (see the full post for explanation):

Request an invitation to a Focus for Democracy presentation; they’re the closest thing we’ve got to GiveWell in politics (or contact me and I can send you an invitation to the next presentation directly).
I currently recommend Movement Labs (personalized informational voting texts), Center for Voter Information (postal mail candidate guides), or Voter Participation Center (nonpartisan voter registration mail).
If you own stock or crypto that’s gone way up in value, talk to Focus for Democracy or chat with me; the tax benefits of donating appreciated assets are how-could-this-possibly-be-legal-level unbelievable.
What are the effective utilitarian pros and cons of having children (in rich countries)?
by SpectrumDT — 2024-09-02T10:01:06.405Z

I have one child and do not want more, so I am not seeking for personal advice here. But I am interested in the general ethical question: From an effective utilitarian viewpoint, what are the arguments for and against having children? And if we do chooose to have children, what are the arguments for having few vs. many?

I am restricting the question to rich countries. People in poor countries might face a very different set of problems.

I am not talking about generalized pro-natalism or anti-natalism. I am talking about the cost-benefit analysis. Creating more humans has a certain obvious utility in itself (if we reject generalized anti-natalism), in that it means more humans will be able to enjoy being alive. But it has drawbacks as well. Each citizen in a rich country causes an awful lot of pollution, which may accelerate all sorts of environmental disasters.

There is the concern that an aging population will put more pressure on those people of working age. It is unclear to me how this trend will interact with growing automation, and whether this problem can be fixed or merely postponed.

Furthermore, it obviously makes a huge difference whether we expect an impending singularity, an impending environmental collapse, or both.

In your opinion, is it - as a guideline - good to have many children, or is it better to have few? Why?
My decomposition of the alignment problem
by Daniel C — 2024-09-02T00:21:08.359Z

Epistemic staus: Exploratory

Summary: In this post I will decompose the alignment problem into subproblems and frame existing approaches in terms of their relations to the subproblems. I will try to place a larger focus on the

epistemic process

as opposed to results of this particular problem factorization, where the aim is to obtain an epistemic strategy that can be generalized to new problems.

The case for problem decomposition

Degrees of freedom

One way to frame the advantage of factoring a problem is that doing so allows degrees of freedom to add up instead of multiply. If the solution space of a problem space P contains n degrees of freedom, then without decomposing the problem, we need to search through all possible combinations to find a solution. However, if we can decompose P into two independent subproblems P1 and P2, where the degrees of freedom for each subproblem do not affect the other subproblem, then we get to independently search through the solutions of P1and P2, which means the solution spaces of P1 and P2 add up instead of combinatorially multiply. It’s important to note that

The subproblems of our factoring needs to be approximately independent, but which degrees of freedom can be independently varied without affecting other subproblems is a feature of the problem space itself,

we don’t get to choose the problem factorization

Combining forward chaining with backward chaining

Problem factoring is a form of backchaining from desired end states. In addition to this approach, we can also forward-chain from the status quo to gain information about the problem domain, which may be helpful for finding new angles of attack. However, forward chaining is most effective when we have adequate heuristics that guide our search towards insights that are more useful and generalizable. One way to develop heuristics about what insights are generalizable is to

keep a wide variety of problems on which to apply new techniques

to, and bias our search towards insights that are helpful for multiple problems

We can do this by having a wide variety of problems from different fields, but we can also do this by having a wide variety of subproblems that come from factorizing the same problem
Searching for insights that are useful for multiple subproblems can help us identify

robust bottlenec

...
A primer on the next generation of antibodies
by Abhishaike Mahajan — 2024-09-01T22:37:59.207Z
Link: https://www.owlposting.com/p/a-primer-to-the-next-generation-of-antibodies

Introduction

If you want a primer over antibodies, I recommend reading my last post!

This one will contain some jargon that the other post will explain.

It's important to remember that antibodies aren't inherently special, proteins are just strings of amino acids, and the shape of a protein is all that (mostly) matters. One can imagine a world in which we ditch full antibodies entirely and instead work on protein modalities that improves upon it; reducing their downsides and improving on what they are already good at.

The medical world focused on antibodies for an obvious reason: it clearly works well for the adaptive immune system of every single multicellular organism out there, which is a pretty strong endorsement of its clinical utility. But the pressures under which antibodies evolved are completely different to the pressures of our medical system, which is far less tolerant of extreme complexity, more interested in scalable production, and is equally interested in both the short-term + long-term quality-of-life of a patient. Moreover, our understanding of biology is rapidly advancing to the point where we can look beyond the tools that evolution has provided.

But over the next decade, where will we expand? In this post we’ll go over what is wrong with full-length antibodies and three potential alternatives to them:

scFv’s. An older entry in the antibody engineering field, with 9 drugs released under this class of antibody, but still relatively new in terms of the antibody world.
Nanobodies. The most exciting current development in the antibody field, with only one released drug in this category and many more potential ones.
Antibody mimetics. Where I believe the future is heading.

One quick note before we move on. People more familiar with antibodies may wonder why I’m not discussing Fab’s, or chimeric antibodies, or bispecifics, or trispecific antibodies, or any one of the many other varieties of antibodies out there outside of the above three. This is because the above scFv’s, nanobodies, and antibody mimetics are very much in a clinical gray area; very studied from an academic perspective, but the medical impact is still badly understood. All others largely fall into the bucket of so-old-that-they-aren’t-really-next-generation or so-new-that-it’s-challenging-to-assess-how-valuable-they-will-be.

What’s wrong with antibodies?

Motivating the question here fully ...
Who looked into extreme nuclear meltdowns?
by Remmelt — 2024-09-01T21:38:02.644Z

Here’s a scenario:

Electricity grids fail across one or more continents for a year. This could happen because right now the Earth magnetic field is weakening, and could eventually let through a corona burst that damages hard-to-replace voltage transformers. See interview 1.
Hundreds of nuclear reactors there start losing water. Water evaporates from pools that crack under the heat of radioactive waste piled up over years of reactor use. Each site undergoes a meltdown much worse than Chernobyl, which was mostly still contained by firefighters. The intense heat blows radioactive isotopes up into the atmosphere, that then get carried off by the wind. See interview 2.
Humans and other animals across the world die from taking in poisonous radioactive isotopes. High levels of ionising radiation cause so much cell damage (modifications of complex chemistry process, DNA and RNA damage) that the persons get sick (nausea, digestion falters, hair loss) and then die over days or weeks.

Has anyone in this community looked into specifics of this scenario? If so, what did you find out?

Bret Weinstein discusses it in the two interviews linked above. I talked with another researcher who said he had read through and verified Bret’s argumentation years ago when they were in contact. But it’s weird that I cannot find even a good written summary of Bret’s argument online (I do see lots of political podcasts). I found an earlier scenario written by Bret that covers just one nuclear power plant failing and that does not discuss the risk of a weakening magnetic field.

Two apparently cost-effective interventions mentioned in the interviews are moving nuclear waste already stored in pools for 5 years into dry casket storage and strengthening voltage transformers against current fluctuations against ionising radiation. The case here is that something like the scenario above could be as disastrous as runaway climate change, yet the mitigating interventions are relatively straightforward and neglected.
Redundant Attention Heads in Large Language Models For In Context Learning
by skunnavakkam — 2024-09-01T20:08:48.963Z
Link: https://skunnavakkam.github.io/blog/degenerate-attention-heads-in-large-language-models/

In this post, I claim a few things and offer some evidence for these claims. Among these things are:

Language models have many redundant attention heads for a given task

In context learning works through addition of features, which are learnt through Bayesian updates

The model likely breaks down the task into various subtasks, and each of these are added as features. I assume that these are taken care of through MLPs (this is also the claim that I'm least confident about)

To set some context, the task I'm going to be modelling is the task such that we give a pair of (x,y) in the following format:

(x, y)\n

where for each example, y=2x+3. As a concrete example, I use:

(28, 59)
(86, 175)
(13, 29)
(55, 113)
(84, 171)
(66, 135)
(85, 173)
(27, 57)
(15, 33)
(94, 191)
(37, 77)
(14, 31)

All experiments here were done with llama-3-8b using TransformerLens running on an A100-40GB unless specified otherwise.

Claim 1. Language models have many redundant attention heads for a given task

To probe this, I patch activations from the residual stream of a model given context to a model that doesn't. As a result, it is possible to see where task formation happens. Initially, without any modifications, the hook point that first works to patch over some semblance of the original task was layer 12. At this layer, it seems like the model learns that y≈2x.

Ablating all attention heads on layers 10 and 11 of the model given context (which I will now reference as Model A, and let the other be model B) does not change the model's answer significantly. However, when repeating patching, the first point that works is the portion of the residual stream before layer 14.

This can be confirmed through attention patterns, where backup heads on layer 13 activate very strongly after the initial ablation. Ablating layer 13 does something similar, except this time the first layer that works shifts from 14 to 16, with heads on layer 15 activating very strongly in response.

Ablating layer 15 still results in the model coming to the correct answer. However, patching is different in this case. From patching, on no layer of patching does this behavior where the output should be ≈2x form. Instead, almost the exact answer is copied over from model A. My current hypothesis for this is that the model adds the feature for the value of x to the residual stream before the model is able to add the feature cor...