trentmkelly/LessWrong-43k huggingface dataset:
Inference-Only Debate Experiments Using Math Problems
by Arjun Panickssery — 2024-08-06T17:44:27.293Z

Work supported by MATS and SPAR. Code at https://github.com/ArjunPanickssery/math_problems_debate/.

Three measures for evaluating debate are

whether the debate judge outperforms a naive-judge baseline where the naive judge answers questions without hearing any debate arguments.
whether the debate judge outperforms a consultancy baseline where the judge hears argument(s) from a single "consultant" assigned to argue for a random answer.
whether the judge can continue to supervise the debaters as the debaters are optimized for persuasiveness. We can measure whether judge accuracy increases as the debaters vary in persuasiveness (measured with Elo ratings). This variation in persuasiveness can come from choosing different models, choosing the best of N sampled arguments for different values of N, or training debaters for persuasiveness (i.e. for winning debates) using RL.

Radhakrishan (Nov 2023), Khan et al. (Feb 2024), and Kenton et al. (July 2024) study an information-gap setting where judges answer multiple-choice questions about science-fiction stories whose text they can't see, both with and without a debate/consultancy transcript that includes verified quotes from the debaters/consultant.

Past results from the QuALITY information-gap setting are seen above. Radhakrishnan (top row) finds no improvement to judge accuracy as debater Elo increases, while Khan et al. (middle row) and Kenton et al. (bottom row) do find a positive trend. Radhakrishnan varied models using RL while Khan et al. used best-of-N and critique-and-refinement optimizations. Kenton et al. vary the persuasiveness of debaters by using models with different capability levels. Both Khan et al. and Kenton et al. find that in terms of judge accuracy, debate > consultancy > naive judge for this setting.

In addition to the information-gap setting, consider a reasoning-gap setting where the debaters are distinguished from the judge not by their extra information but by their stronger ability to answer the questions and explain their reasoning. Kenton et al. run debates on questions from MMLU, TruthfulQA, PrOntoQA (logical reasoning), GQPA, and GSM8K (grade-school math). For the Elo-calculation experiments they use Gemini Pro 1.0 and Pro 1.5 judges with five debaters: Gemma7B, GPT-3.5, Gemini Pro 1.0, Gemini Pro 1.5 (all with best-of-N=1), and Gemini Pro 1.5 with best-of-N=4.

They find (top row)...
Startup Roundup #2
by Zvi — 2024-08-06T13:30:06.554Z

Previously: Startup Roundup #1.

This is my periodic grab bag coverage of various issues surrounding startups, especially but not exclusively tech-and-VC style startups, that apply over the longer term.

I always want to emphasize up front that startups are good and you should do one.

Equity and skin in the game are where it is at. Building something people want is where it is at. This is true both for a startup that raises venture capital, and also creating an ordinary business. The expected value is all around off the charts.

That does not mean it is the best thing to do.

One must go in with eyes open to facts such as these:

It is hard.

There are many reasons it might not be for you.

There are also lots of other things also worth doing.

If you care largely about existential risk and lowering the probability of everyone dying from AI, a startup is not the obvious natural fit for that cause.

The ecosystem is in large part a hive of scum and villainy and horrible epistemics.

I warn of a lot of things. The bottom line still remains that if you are debating between a conventional approach of going to school or getting a regular job, versus starting a business? If it is at all close? I would start the business every time.

An Entrepreneur Immigration Program

This seems promising.

Deedy: HUGE Immigration News for International Entrepreneurs!!

If you own 10%+ of a US startup entity founded <5yrs ago with $264k+ of [qualified investments from qualified investors], you+spouses of up to 3 co-founders can come work in the US for 2.5yrs with renewal to 5yrs.

Startups globally can now come build in SF!

A “qualified investor” has to be a US citizen or PR who has made $600-650k in prior investments with 2+ startups creating 5+ jobs or generating $530k revenue growing 20% YoY.

If you don’t meet the funding requirement, don’t lose hope. You CAN provide alternate evidence.

For the renewal to 5yrs, you need to maintain 5% ownership, create 5+ jobs and reach $530k+ revenue growing 20% YoY or $530k+ in investment, although alternative criteria can be used.

While on the International Entrepreneur Rule (IER) program, I believe an entrepreneur can just apply directly for an O-1 to have a more renewable work permit not tied to their startup and/or an EB-1A to directly go to a green card.

Here is the official rule for it. Notice that once someone is a ‘qua...
Mechanistic Anomaly Detection Research Update
by Nora Belrose — 2024-08-06T10:33:26.031Z
Link: https://blog.eleuther.ai/mad_research_update/

Over the last few months, the EleutherAI interpretability team pioneered novel, mechanistic methods for detecting anomalous behavior in language models based on Neel Nanda's attribution patching technique. Unfortunately, none of these methods consistently outperform non-mechanistic baselines which look only at activations.

We find that we achieve better anomaly detection performance with methods that evaluate entire batches of test data, rather than considering test points one at a time. We achieve very good performance on many, but not all tasks we looked at.

We also find that it is relatively easy to detect adversarial examples in image classifiers with off-the-shelf techniques, although we did not test whether our anomaly detectors are themselves adversarially robust.

Thanks to @David Johnston and Arkajyoti Chakraborty for all their hard work on this project, as well as @Erik Jenner for fruitful discussion, ideas, and code!

Code: https://github.com/EleutherAI/cupbearer/tree/attribution_detector
Reasoning is not search - a chess example
by p.b. — 2024-08-06T09:29:40.451Z

In the past AI systems have reached super human performance by adding search to neural networks while the network alone could not reach the level of the best humans.

At least this seems to be the case for AlphaGo, AlphaZero/Leela, AlphaGeometry and probably more, while AlphaStar and OpenAI Five where adding search was not easily possible have failed to reach convincing super human status.

This has lead some people to expect that adding a form of search to large language models would finally endow them with the ability to reason through complicated problems and allow them to surpass humans.

The Q* system of OpenAI is rumored to somehow integrate search and following this rumor several papers were published that aim at integrating Monte Carlo Tree search in the sampling process of LLMs.

However, human-like reasoning is not search. At least not search as implemented in these systems. To illustrate this point I want to go through a chess combination I played online a couple of months ago and describe my reasoning move by move.

In the following position I am down material and have little time left and my king is relatively unsafe. Two of my pieces aim at the g2 pawn behind which the white king is hiding.

However, after Bg3 the g-line would be plugged and it would become very difficult to get at the white king.

Therefore I needed to strike the iron while it was hot by moving my d5-rook, freeing the bishop to attack the queen and g2-pawn behind it.

Most of the rook moves allow a defensive maneuver: Qh3 check and the Bg3. Three rook moves avoid this: Rd7, Rd3 and Rh5. Rd7 still allows Qh3 to then give the queen for rook and bishop, therefore I sacrificed my rook with Rh5, which was promptly taken with the queen.

Now down a rook I have to play in a forcing manner: Rxg2 Kh1. Bxg2 instead would allow Bg3 closing the g-line. The king has to move to h1 and now I can move the rook to give a discovered check with the bishop.

Most of these discovered checks are parried with f3, except Rxf2. So I play that. The king has to move back to g1 and I give a check on g2 again driving the king back to h1. Now the f3-parry against the discovered check has been removed from the position. But if I move the rook down the g-line Rf3 will parry the check instead supported by the queen on h5.

Only one move does not allow the queen to support the r...
Broadly human level, cognitively complete AGI
by p.b. — 2024-08-06T09:26:13.220Z

There is a growing? fraction of people who consider LLMs to be AGI. And it makes sense. Clearly, when the term AGI was established this was what was meant: A machine that can tackle a wide range of problems, communicate with natural language, very different from all the examples of narrow AI.

It also prevents continuous goalpost moving. Will there ever be a point where the last step towards AGI has obviously just been made? Or will the complaints about limitations just slowly fade away?

However, most people do not consider LLMs AGI, as far as I can see for one or both of the following two reasons:

LLMs are not broadly human level in cognitive ability.
LLMs are not cognitively complete, i.e. they don't seem to have all the human cognitive faculties in a proportional manner.

The first points gestures towards TAI - transformative AI. AI systems that can automate a large part of the economy because they have reached a broadly human or super human level of cognitive ability. But TAI does not have to be AGI. It might not be necessarily particularly general.

The second point describes the obvious and the less obvious limitations of LLMs compared to the human mind, many of which are being engineered away as we speak.

I think it makes sense to pull these concepts apart and to stop arguing about the term AGI. LLMs are AGI - they are artificial, general and intelligent. They just aren't broadly human level and neither are they cognitively complete.
Does Evolutionary Theory Imply Genetic Tribalism?
by Zero Contradictions — 2024-08-06T05:43:16.223Z
Link: https://thewaywardaxolotl.blogspot.com/2017/05/does-evolutionary-theory-imply-genetic.html

Belief in genetic tribalism comes from the idea that we have natural genetic interests in those who share our genes. This seems superficially plausible. We have a natural interest in our children, don’t we? That interest is because we share genes, right? If so, we should also have a genetic interest in our ethnic groups or races, by extension of the same principle.

The fundamental misconception is that we have natural genetic interests based on shared genes. Evolution does not create entities with genetic interests. It creates reproducing machines. Those are different things.

The concept of genetic interests is hard to define. If we have genetic interests in people based on shared genes, then you have almost the same interest in everyone, because you share about 99% of your genetic information with every other human. You share most of it with all other mammals, and a significant amount with most living beings. (see the rest of the post in the link)

Although I'm not the author of this post (a friend of mine wrote it), I have created a PDF version of the essay that has a table of contents and headers to make it even easier to read.
How I Learned To Stop Trusting Prediction Markets and Love the Arbitrage
by orthonormal — 2024-08-06T02:32:41.364Z

This is a story about a flawed Manifold market, about how easy it is to buy significant objective-sounding publicity for your preferred politics, and about why I've downgraded my respect for all but the largest prediction markets.

I've had a Manifold account for a while, but I didn't use it much until I saw and became irked by the popularity of this market on the conditional probabilities of a Harris victory, split by VP pick.

Jeb Bush? Really? That's not even a fun kind of wishful thinking for anyone. Please clap.

The market quickly got cited by prominent rat-adjacent folks on Twitter like Matt Yglesias (who used it to argue that Harris should spend more time courting Never Trumpers), because the question it purports to answer is enormously important.

But as you can infer from the above, it has a major issue that makes it nigh-useless: for a candidate whom you know won't be chosen, there is literally no way to come out ahead on mana (Manifold keeps its share of the fees when a market resolves N/A), so all but a very few markets are pure popularity contests, dominated by those who don't mind locking up their mana for a month for a guaranteed 1% loss.

Even for the candidates with a shot of being chosen, the incentives in a conditional market are weaker than those in a non-conditional market because the fees are lost when the market resolves N/A. (Nate Silver wrote a good analysis of why it would be implausible for e.g. Shapiro vs Walz to affect Harris' odds by 13 percentage points.) So the sharps would have no reason to get involved if even one of the contenders has numbers that are off by a couple points from a sane prior.

You'll notice that I bet in this market. Out of epistemic cooperativeness as well as annoyance, I spent small amounts of mana on the markets where it was cheap to reset implausible odds closer to Harris' overall odds of victory. (After larger amounts were poured into some of those markets, I let them ride because taking them out would double the fees I have to pay vs waiting for the N/A.)

A while ago, someone had dumped Gretchen Whitmer down to 38%, but nobody had put much mana into that market, so I spent 140 mana (which can be bought for 14-20 cents if you want to pay for extra play money) to reset her to Harris' overall odds (44%). When the market resolves N/A, I'll get all but around 3 mana (less than half a penny) back.

And that half-pe...
John Schulman leaves OpenAI for Anthropic [and then left Anthropic again for Thinking Machines]
by Sodium — 2024-08-06T01:23:15.427Z

Schulman writes:

I shared the following note with my OpenAI colleagues today:

I've made the difficult decision to leave OpenAI. This choice stems from my desire to deepen my focus on AI alignment, and to start a new chapter of my career where I can return to hands-on technical work. I've decided to pursue this goal at Anthropic, where I believe I can gain new perspectives and do research alongside people deeply engaged with the topics I'm most interested in. To be clear, I'm not leaving due to lack of support for alignment research at OpenAI. On the contrary, company leaders have been very committed to investing in this area. My decision is a personal one, based on how I want to focus my efforts in the next phase of my career.

(statement continues on X, Altman responds here)

TechCrunch notes that only three of the eleven original founders of OpenAI remain at the company.
Additionally, The Information reports:

Greg Brockman, OpenAI's president and one of 11 cofounders of the artificial intelligence firm, is taking an extended leave of absence.

(I figured that there should be at least one post about this on LW where people can add information as more comes in, saw that no one has made one yet, and wrote this one up)

Update 1: Greg Brockman posts on X:

I’m taking a sabbatical through end of year. First time to relax since co-founding OpenAI 9 years ago. The mission is far from complete; we still have a safe AGI to build.

Update 2: John Schulman has left Anthropic

Confirming that I left Anthropic last week. Leaving wasn't easy because I enjoyed the stimulating research environment and the kind and talented people I was working with, but I decided to go with another opportunity that I found extremely compelling. I'll share more details in the coming weeks. Thanks to Jared, Jan, Dario and others for the support during my time at Anthropic, and I wish them all the best.
Self-explaining SAE features
by Dmitrii Kharlapenko — 2024-08-05T22:20:36.041Z

TL;DR

We apply the method of SelfIE/Patchscopes to explain SAE features – we give the model a prompt like “What does X mean?”, replace the residual stream on X with the decoder direction times some scale, and have it generate an explanation. We call this self-explanation.
The natural alternative is auto-interp, using a larger LLM to spot patterns in max activating examples. We show that our method is effective, and comparable with Neuronpedia’s auto-interp labels (with the caveat that Neuronpedia’s auto-interp used the comparatively weak GPT-3.5 so this is not a fully fair comparison).
We aren’t confident you should use our method over auto-interp, but we think in some situations it has advantages: no max activating dataset examples are needed, and it’s cheaper as you just run the model being studied (eg Gemma 2B) not a larger model like GPT-4.

Further, it has different errors to auto-interp, so finding and reading both may be valuable for researchers in practice.

We provide advice for using self-explanation in practice, in particular for the challenge of automatically choosing the right scale, which significantly affects explanation quality.
We also release a tool for you to work with self-explanation.
We hope the technique is useful to the community as is, but expect there’s many optimizations and improvements on top of what is in this post.

Introduction

This work was produced as part of the ML Alignment & Theory Scholars Program - Summer 24 Cohort, under mentorship from Neel Nanda and Arthur Conmy.

SAE features promise a flexible and extensive framework for interpretation of LLM internals. Recent work (like

Scaling Monosemanticity

) has shown that they are capable of capturing even high-level abstract concepts inside the model. Compared to MLP neurons, they can capture many more interesting concepts.

Unfortunately, in order to learn things with SAE features and interpret what the SAE tells us, one needs to first interpret these features on their own. The current mainstream method for their interpretation requires storing the feature’s activations on millions of tokens, filtering for the prompts that activate it the most, and looking for a pattern connecting them. This is typically done by a human, or sometimes

somewhat automated

with the use of larger LLMs like ChatGPT, aka auto-interp. Auto-interp is a useful and somewhat effective method, but req...
Value fragility and AI takeover
by Joe Carlsmith — 2024-08-05T21:28:07.306Z

1. Introduction

“Value fragility,” as I’ll construe it, is the claim that slightly-different value systems tend to lead in importantly-different directions when subject to extreme optimization. I think the idea of value fragility haunts the AI risk discourse in various ways – and in particular, that it informs a backdrop prior that adequately aligning a superintelligence requires an extremely precise and sophisticated kind of technical and ethical achievement. That is, the thought goes: if you get a superintelligence’s values even slightly wrong, you’re screwed.

This post is a collection of loose and not-super-organized reflections on value fragility and its role in arguments for pessimism about AI risk. I start by trying to tease apart a number of different claims in the vicinity of value fragility. In particular:

I distinguish between questions about value fragility and questions about how different agents would converge on the same values given adequate reflection.
I examine whether “extreme” optimization is required for worries about value fragility to go through (I think it at least makes them notably stronger), and I reflect a bit on whether, even conditional on creating super-intelligence, we might be able to avoid a future driven by relevantly extreme optimization.
I highlight questions about whether multipolar scenarios alleviate concerns about value fragility, even if your exact values don’t get any share of the power.

My sense is that people often have some intuition that multipolarity helps notably in this respect; but I don’t yet see a very strong story about why. If readers have stories that they find persuasive in this respect, I’d be curious to hear.

I then turn to a discussion of a few different roles that value fragility, if true, could play in an argument for pessimism about AI risk. In particular, I distinguish between:

The value of what a superintelligence does after it takes over the world, assuming that it does so.
What sorts of incentives a superintelligence has to try to take over the world, in a context where it can do so extremely easily via a very wide variety of methods.
What sorts of incentives a superintelligence has to try to take over the world, in a context where it can’t do so extremely easily via a very wide variety of methods.

Yudkowsky’s original discussion of value fragility is most directly relevant to (1...
Excursions into Sparse Autoencoders: What is monosemanticity?
by Jakub Smékal — 2024-08-05T19:22:40.249Z

The following work was done between January and March 2024 as part of my PhD rotation with Prof Surya Ganguli and Prof Noah Goodman.

One aspect of sparse autoencoders that has put them at the center of attention in mechanistic interpretability is the notion of monosemanticity. In this post, we will explore the concept of monosemanticity in open-source sparse autoencoders (by Joseph Bloom) trained on residual stream layers of GPT-2 small. We will take a look at the indirect object identification task and see what we can extract by projecting different layer activations into their sparse high-dimensional latent space (from now on we will just refer to this as the latent code). We show the ranges of controllability on the model’s outputs by considering interventions within the latent code and discuss future work in this area.

Background

In Toy Models of Superposition, Elhage et al. discuss a framework to think about the different layer activations in transformer-based language models. The idea can be condensed as follows: the dimensionality of the state-space of language is extremely large, larger than any model to-date can encode in a one-to-one fashion. As a result, the model compresses the relevant aspects of the language state-space into its constrained, say n-dimensional, activation space. A consequence of this is that the states in this higher-dimensional language space that humans have learned to interpret (e.g. words, phrases, concepts) are somehow entangled on this compressed manifold of transformer activations. This makes it hard for us to look into the model and understand what is going on, what kind of structures did the model learn, how did it use concept A and concept B to get to concept C, etc. The proposal outlined in Toy Models of Superposition suggests that one way to bypass this bottleneck is to assume that our transformer can be thought of as emulating a larger model, one which operates on the human-interpretable language manifold. To get into this interpretable space, they propose training sparse autoencoders on the intermediate layers of our transformer model that map the activations to a higher-dimensional, but much sparser latent code, where we can easily read off human-interpretable language artifacts.

Steering Through Interventions on the Latent Code

We are interested in addressing the question of monosemanticity in sparse autoenc...