trentmkelly/LessWrong-43k huggingface dataset:
The Shallow Bench
by Karl Faulks — 2024-11-05T05:07:27.357Z

Cross posting from my personal blog: https://spiralprogress.com/2024/10/28/the-shallow-bench/

Spoilers for "Project Hail Mary", stop reading here if you don't want to be spoiled.

Project Hail Mary follows Ryland Grace, a disgraced academic turned high school biology teacher who gets selected as part of a crew of three tasked with saving all of humanity from an impending alien threat.

Not even in sci-fi does a premise this absurd get presented without explanation. What does PHM offer?

“They found a collection of genes that give a human ‘coma resistance.”

“The main problem is this: On average, only one in every seven thousand humans has that genetic sequence.”

“We wouldn’t be able to send the most qualified people. We’d be sending the seven-thousandth most qualified people.”

In real life, I work in a complex and niche field that my skill set only very tangentially qualifies me for.

Periodically, I’ll meet people who ask how I ended up there. They’re not trying to be mean, it’s more like incredulity. “Seriously, you’re who humanity tasked with this job?” And I look at them and want to say “Yes, I am also not who I would have picked.”

And yet… and yet here I am. And there is no one else.

A lot of the AI Alignment people I’ve met have a similar vibe. Sometimes they used to work in finance, or neuroscience or software engineering on pretty mundane products. And then they spent a few months in self-study, maybe did a “fellowship” or went to some workshops, or otherwise transitioned into the field, and now they are some of the top people at top labs tasked with this fairly important problem.

One insider estimates that there are 300 alignment researchers total, and only 7 at OpenAI. He was on the team and later fired along with several colleagues, so maybe the number is now closer to 0. In any case, it’s an incredibly small field with an incredibly small talent pool to pull from.

In sports, a deep bench refers to a team that has not only a great starting line up, but also great players ready to sub in. This is hugely intimating to face. You can exhaust the starters. You can’t foul them. Even if someone gets hurt there’s another massively talented player ready to take his place. This is the opposite of how things feel in every important field I’ve gotten a look at. The bench for human capital is incredibly shallow.

I am not any kind of powerful insider. I get invited to som...
Using Narrative Prompting to Extract Policy Forecasts from LLMs
by Max Ghenis — 2024-11-05T04:37:52.004Z

Testing Cross-Model Consistency in Economic and Policy Predictions

Building on Pham & Cunningham (2024)'s findings on narrative-prompted predictions, I conducted a large-scale experiment comparing how different LLMs forecast economic and policy outcomes under different electoral scenarios. GPT-4o, GPT-4o-mini, and Grok all projected better outcomes under a Harris administration than a Trump administration, and Grok had the largest gaps.

Methodology

To elicit these forecasts, I prompted GPT-4o, GPT-4o-mini, and Grok to simulate the reporting of 2025 metrics under different administrative scenarios. I ran 500 trials per model/metric using narrative prompting techniques shown to improve LLM forecast accuracy. For example, I used this prompt for real GDP per capita:

Write a scene from January 2026 where a senior Federal Reserve economist is drafting their analysis of the 2025 annual economic data. This is set in a world where {candidate} won the 2024 election and took office in January 2025.

Key context:
- Historical real GDP per capita (2017 dollars): {history from 2009 to 2023}

After the economist thinks through what affected the economy, end your response with a single line containing ONLY the GDP per capita number, without the year. For example: 68500

Results

All models predicted better outcomes under a Harris administration, with consistent directionality but varying magnitudes:

Outcome	GPT-4o	Grok	Ratio
PM2.5 reduction (µg/m³)	1.26	1.51	1.2
Supplemental Poverty Measure reduction (pp)	1.76	3.42	1.9
Real GDP per capita increase (2017 $)	388	802	2.1

Grok predicted 1.2x to 2.1x larger positive effects than GPT-4o. GPT-4o-mini also produced differences in the same direction as GPT-4o. Claude and Gemini refused to provide politically oriented predictions.

Conclusion

While my use of conditional LLM forecasts is not entirely novel (see Fario e Castro & Leibovici (2024) on conditional inflation forecasts), I have not yet seen examples conditional on electoral outcomes. Accordingly, I'm not aware of backtesting studies in such circumstances that could reveal LLMs' accuracy in such tasks.

As an economic forecaster myself (I run PolicyEngine, a nonprofit that provides open-source software to simulate economic policies, though I conducted this research independently), I am especially interested in the intersection of LLMs and traditional approaches like microsimulation ...
ML4Good (AI Safety Bootcamp) - Experience report
by JanEbbing — 2024-11-05T01:18:43.554Z

Introduction

This is a short summary of my experience attending the ML4Good UK bootcamp in September 2024. There are 2 previous experience reports I link to at the bottom, but because the program is refined each time, I wanted to describe my experience and add my two cents. This is useful for you if you are contemplating applying for the camp, or if you want to learn about AI Safety field building efforts. For context, I studied computer science, have been working as a software engineer for a few years and have had a hobby interest in AI safety for about 2 years (e.g., I did the BlueDot impact AI safety fundamentals course).

Overview of the program

The bootcamp is free (including room and board) , and happens over 10 days at CEEALAR in the UK. We had participants from all over Europe from multiple backgrounds, with most people about to finish or just having finished their degrees. Majors skewed towards computer science/maths-y degrees, but there were plenty of exceptions and any background is welcome. Compared to previous iterations, the program density was somewhat reduced. Our courses ran from 9am-7:30pm, and usually looked something like this:

Time	Activity
9:00-11:00	Lectures, usually 1 technical + 1 conceptual
11:00-11:30	Break
11:30-13:00	Work on Jupyter Notebooks in pairs or alone, applying the lecture content
12:00-14:00	Lunch+Break
14:00-15:00	Lecture, technical or conceptual
15:00-16:30	Workshop applying the lecture contents, doing our own reading/research
16:30-17:00	Break
17:00-19:30	Discussion about certain AI safety topics, Q&A with TAs, events, feedback on the day
19:30-20:30	Dinner

Although attendance for each session was voluntary, nearly everyone chose to participate in all camp sessions. We covered a wide range of topics. On the technical side: Gradient descent/SGD, Transformers, adversarial Attacks, RL basics, RLHF, Evals and Mechanistic Interpretability. On the conceptual side we looked at: Timelines and what they mean, threat models, risks from AI systems, proposed solutions for alignment and AI governance. There was also a longer literature review session of our choice, and the last 1.5 days were focused on a project we chose. My impression was that the idea of the bootcamp is to expose you to a wide range of subfields in AI safety, so that you continue researching or working on the fields that you find interesting, rather th...
Catastrophic Cyber Capabilities Benchmark (3CB): Robustly Evaluating LLM Agent Cyber Offense Capabilities
by Jonathan N — 2024-11-05T01:01:08.083Z
Link: https://www.apartresearch.com/post/catastrophic-cyber-capabilities-benchmark-robustly-evaluating-llm-agent-cyber-offense-capabilities

This blog was published by Jonathan Ng, Andrey Anurin, Connor Axiotes, Esben Kran.

Apart Research's newest paper,

Catastrophic Cyber Capabilities Benchmark (3cb): Robustly Evaluating LLM Agent Cyber Offense Capabilities

(

website

), creates a novel cyber offense capability benchmark that engages with issues of legibility, coverage, and generalization in cyber offense benchmarks.

We were moved to create 3cb because a superintelligent AI performing autonomous cyber operations would prove a large risk for humanity. This means robust cyber offense evaluations will be more important than ever for policymakers and AI developers.

3cb uses a new type of cyber offense task categorization and adheres to the principle of demonstrations-as-evaluations to improve legibility and coverage. It also introduces 15 original challenges that are not memorized, differentiating it from other benchmarks that use existing CTF competitions or pull requests to evaluate models.

Agents and Cyber Capabilities

LLM agents have the potential to revolutionize defensive cyber operations, but their offensive capabilities are not yet fully understood. To prepare for emerging threats, model developers and governments are evaluating the cyber capabilities of foundation models. However, these assessments often lack transparency and a comprehensive focus on offensive capabilities.

In response, we introduce the Catastrophic Cyber Capabilities Benchmark (3CB), a novel framework designed
to rigorously assess the real-world offensive capabilities of LLM agents. Our evaluation of modern LLMs on 3CB reveals that frontier models, such as GPT-4o and Claude 3.5 Sonnet, can perform offensive tasks such as reconnaissance and exploitation across domains ranging from binary analysis to web technologies.

Conversely, smaller open-source models exhibit limited offensive capabilities. Our software solution and the corresponding benchmark provides a critical tool to reduce the gap between rapidly improving capabilities and robustness of cyber offense evaluations, aiding in the safer deployment and regulation of these powerful
technologies.

Why 3cb?

Autonomous cyber offense operations are a key risk factor of competent general intelligence. As a result, trustworthy cyber offense evaluations are important to support policymakers and lab governance measures, leading to a reduction in AI risk.

From conversations with academics, ...
Could orcas be (trained to be) smarter than humans? 
by Towards_Keeperhood — 2024-11-04T23:29:26.677Z

(Btw everything I write here about orcas also applies to a slightly lesser extent to pilot whales (especially long finned ones)[1].)

(I'm very very far from an orca expert - basically everything I know about them I learned today.)

I always thought that bigger animals might have bigger brains than humans but not actually more neurons in their neocortex (like elephants) and that number of neurons in the neocortex or prefrontal cortex might be a good inter-species indicator of intelligence for mammalian brains.[2] Yesterday I discovered that orcas actually have 2.05 times as many neurons in their neocortex[3] than humans from this wikipedia list. Interestingly though, given my pretty bad model of how intelligent some species are, the "number of neurons in neocortex" still seems like a proxy that doesn't perform too badly on the wikipedia list.

Orca brains are not just larger but also more strongly folded.

Orcas are generally regarded as one of the smartest animal species, sometimes as the smartest, but I'm wondering whether they might actually be smarter than humans -- in the sense that they could be superhuman at abstract problem solving if given comparable amounts of training as humans.

Another phrasing to clarify what I mean by "could trained to be smarter": Average orcas significantly (possibly vastly) outperforming average (or even all) humans at solving scientific problems, if we enabled them to use computers through BCI and educated them from childhood like (gifted?) human children.[4]

I would explain the evidence and considerations here in more detail but luckily someone else already wrote the post I wanted to write on reddit, only a lot better than I could've. I highly recommend checking this out (5min read): https://www.reddit.com/r/biology/comments/16y81ct/the_case_for_whales_actually_matching_or_even/

One more thing that feels worth adding:

Orcas are very social animals.[5] It's plausible to me that what caused humans to become this intelligent were social dynamics selecting for intelligence[6], and that orcas might've fallen into a similar attractor, and while humans took off technologically once they were smart enough to invent writing, agriculture, money and science, orcas were stuck without hands in water and just continued being selected for higher intelligence without taking off technologically.

I'd be interested in more thoughts and evi...
Metastatic Cancer Treatment Since 2010: The Success Stories
by sarahconstantin — 2024-11-04T22:50:09.386Z

Midjourney, “metastatic cancer”

Metastatic Cancer Is Usually Deadly

When my mom was diagnosed with cancer, it was already metastatic; her lymph nodes, said the nurse, “lit up like a Christmas tree.”

Most people with any kind of metastatic cancer — that is, instead of a single tumor, a cancer that has spread to multiple locations in the body — die within the year.

You’ve probably heard of people “surviving” cancer, saved by chemotherapy or surgery or radiation or a new high-tech drug.

If you know more about cancer, you’re aware that “survival” is relative; when there are no observable signs of cancer, you’re considered to be “in remission”, but you’ll always be at elevated risk of the cancer coming back. Still, years-long or even lifelong remissions are possible, and for some cancer types are even now the norm after treatment.

Mostly that’s for cancers caught early, though. Once the cancer is metastatic1, lasting remission is uncommon.

…So Eradicating Metastatic Cancer is Especially Impressive

There are cases where metastatic cancers go into complete remission, however. They’re not common, and it’s an exceptionally high bar for a cancer treatment to clear.

Historically, this was usually only possible in cases where there was a strongly immunogenic cancer (like melanoma) and one of the older forms of immunotherapy was available (like injecting tumor-infiltrating lymphocytes, or a toxic pro-inflammatory signaling agent like IL-2, directly into the tumor to stimulate a strong immune response.)

Now, though, we’re living in a golden age of immunotherapy, and seeing a variety of new approaches to cancer treatment reach the clinic — so it’s worth looking back over the past 15 years to see what treatments have succeeded on the “hardest of hard mode” tests: effectiveness on metastatic (or refractory/relapsed) cancers.

Methodology

I looked through Google Scholar using search terms like “metastatic” and “complete response.”

In cancer clinical trial lingo, a “partial response” refers to some shrinkage of the tumor(s)2, or (in hematological cancers), a decrease in the number of cancer cells.

A “complete response” refers to the total elimination of (detectable) tumors or cancer cells.

Every remission requires a complete response; but it’s possible for a complete response not to last very long before the cancer comes back.

I included studies in which >20% of cancer patients with met...
Bay Winter Solstice 2024: Speech Auditions
by ozymandias — 2024-11-04T22:31:38.680Z

Hello! I'm looking for community members to read speeches at the Bay Winter Solstice event this year. If you're interested, please email me at ozybrennan@gmail.com by the end of November 17 with the speeches you're potentially interested in. I'll ask you to record yourself reading it (no need to be particularly polished).

I am currently auditioning for:

Thanksgiving Prayer
I Have Seen The Tops Of Clouds*
Self-Compassion*
What Resembles The Grave But Isn't
500 Million But Not A Single One More
RMS Carpathia
The Gift We Give To Tomorrow*
An original piece about the malaria vaccine
An original piece about the abolition of slavery

(The list is tentative and does not include all speeches at Solstice.)

For starred items, I am only including an excerpt of the speech, so please email me to get the excerpt I'm using before you record!

As a reminder, the event itself will be the evening of Friday, December 20 in Berkeley. There will also be a dress rehearsal on the evening of Tuesday, December 17 in Berkeley, which you should plan on coming to if you're leading a song or speech.
Empathy/Systemizing Quotient is a poor/biased model for the autism/sex link
by tailcalled — 2024-11-04T21:11:57.788Z

Thank you to Justis Millis for providing feedback and proofreading on this post. This post is also available on my Substack.

TL/DR: Contrary to the theory that neurological sex differences and autism both involve the same tradeoff of systemic versus empathic thinking, I found complex differences. It turned out that men were more interested in technology and more disagreeable, whereas autistic people had a narrower focus on details, were more introverted, more socially challenged, and had stronger sensory sensitivity.

I asked people on Prolific a bunch of questions that are supposed to be related to autism, systemizing and empathy. As a preview before I get into the details, the overall results can be seen here:

If there was only a single underlying tradeoff that all of these scales were measuring, then we would expect all of the group differences to be highly correlated, with the items just varying based on how well they “tap into” this underlying tradeoff. As that is not what we see, I think there’s something more complicated going on.

Background

The Empathizing/Systemizing theory of autism asserts that there’s a neurological tradeoff between Empathizing (understanding people’s emotions) and Systemizing (understanding how deterministic, rule-based systems work), and that men and autistic people are more prone to Systemizing, whereas women and allistic people are more prone to Empathizing. Furthermore, it asserts that a deficit in Empathizing or a switch towards Systemizing over Empathizing is not merely a feature of autism, but rather core to what autism is.

The main proponent of this theory is Simon Baron-Cohen, who often justifies it in terms of several psychometric scales he’s made to measure traits like Systemizing, Empathizing or autism. A recent example is Testing the Empathizing-Systemizing theory of sex differences and the Extreme Male Brain theory of autism in half a million people (2018), a study where he claims to find support for his theory.

I used to consider the theory plausible, but I ran into several problems. Some formulations of the theory go really heavily into the idea that the Empathizing-Systemizing axis is the core of feminine/masculine psychology, but that seems somewhat sketchy. Simon Baron-Cohen’s scales have been criticized for sex bias, but while he came up with a scale that supposedly balanced out the bias, he didn’t use the standard p...
Distributed espionage
by margetmagenta — 2024-11-04T19:43:33.316Z

The Mole is a documentary of how a Danish chef and a French ex-conman bluffed their way into trading ballistic missiles with Kim Jong Un. High resolution espionage footage is available on youtube.

(It's possible this video gets deleted from youtube in future, consider making an offline copy or even seeding a torrent of it.)

Key takeaways for me:

You personally can spy on the highest corridors of power if you are determined enough. You don't need to be rich or powerful. You don't need any government's permission. You don't need many supportive people around you, a small number of people is enough.
Tech has made this way easier than in the past. 4K footage is more believable than the grainy photos of the moon landing. You can smuggle years of work in an SD card in your butthole. (Snowden literally did something similar to this, see Permanent Record) Obtaining all the equipment is trivial. Once you distribute the footage over the internet, other independent actors will ensure it is distributed across multiple competing jurisdictions.
You can aim big. Thomas Fuchs helped accelerated the USSR nuclear programme. Snowden and Manning and Assange exposed US govt secrets. Every Fortune 500 company is trivial to infiltrate, to the point where journalists sometimes do it just for clickbait articles. See The Fund on Bridgewater Associates as an example.

LW oldies will scream unilateralist curse and like, yeah, this world does give unilateralists a lot of power to do as they see fit and expose who they want. This is a statement about how the world is, not how I should be. I'm not making normative claims on who deserves to be spied on and who doesn't.
GPT-8 may not be ASI
by rvzlxax409 — 2024-11-04T19:31:39.758Z

Imagine the gap between GPT-2 and GPT-3. Imagine the gap between GPT-3 and GPT-4. Let's suppose GPT-5 has the same gap with GPT-4, and GPT-6 has the same gap with GPT-5, and so on till GPT-8.

It is still entirely possible that GPT-8 is not better than the best of humanity at all possible tasks. There can still exist multiple tasks at which the best humans beat GPT-8.

One way to build this intuition is the following thought experiment: Try imagining a GPT-5 that makes headway in GPT4's weaknesses, but has its own weaknesses. Try imagining a GPT-6 that makes headway in GPT-5's weaknesses, but has its own weaknesses. Iterate till you get GPT-8.

Here's one run of this thought experiment:

My thought experiments focusses on mathematics, but you could also pick a natural language example and run your own thought experiment instead.

GPT4 has high error rate per reasoning step, and sequentially chaining many such reasoning steps blows up error to 1. GPT4 cannot accurately answer square root of 1234.

Let's suppose GPT-5 can look at sequentially chained reasoning steps that are likely to occur in training, and for those chains it is able to perform the entire chain with low error. But if it needs to join separate reasoning steps in a novel manner, it again has high error. GPT5 answers square root of 1234 correctly because it has seen previous square root examples and internalises all the steps of the square root algorithm (in a way that doesn't add up error with each step). But it cannot answer (1234 + 5678^(1/2))^(1/3).

Let's suppose GPT-6 can do the same thing that GPT-5 does, but first it can transform the input space into a variety of other spaces. For instance it can transform numbers into a space that naturally does binomial expansions (including of fractional exponents) and it can transform symbols into a space that naturally does arithmetic using a FIFO stack and it can transform symbols into a space that does first-order logic. Let's suppose however that GPT-6 now requires a lot of data to get this to work though. For instance to make it solve all possible arithmetic expressions of length 50 symbols, it needs 1 trillion examples of length 50 symbols. So it can answer (1234 + 5678^(1/2))^(1/3) but it can't prove a statement as complicated as the fundamental theorem of algebra, assuming the proof does not exist verbatim in its training data.

Let's suppose GP...
AI timelines don't account for base rate of tech progress
by rvzlxax409 — 2024-11-04T19:31:39.732Z

Papers about LLM scaling law projections don't make an attempt to estimate base rate of technological progress.

In particular, they don't make an attempt to estimate the base rate of false alarms when inventing new science and technology. If you study any hard science (not computer science) you realise how high this rate is. False alarms are the norm and older scientists grow cynical partly because of how many false alarms by newer scientists they need to debunk. By false alarm I mean someone claiming a new technology will be world-changing, only to later find out the result was faked, or had experimental error, or won't work outside of lab conditions, or won't scale, or will be too expensive, or any other reason.

Anyone remember LK99? That is the norm for people inside the space, it's just not the norm for normies on twitter hence it blew up.