trentmkelly/LessWrong-43k huggingface dataset:
NAO Updates, January 2025

by jefftk — 2025-01-10T03:37:36.698Z
Link: https://naobservatory.org/blog/updates-jan-2025

We had a productive Fall, and are looking forward to an even more productive 2025. Note that this update is a bit smaller than our

last update

: we’re now targeting quarterly updates.

We continue to be very enthusiastic about collaborating with others in this space: if you have questions or see opportunities please let us know!

Wastewater Sequencing

We’ve scaled up our collaboration with

Marc Johnson

’s lab at the University of Missouri (MU). They’re now sequencing wastewater from eight sewersheds across four metropolitan areas, with the addition of Riverside CA (in collaboration with

Jason Rothman

) in December. We’ve expanded to a NovaSeq 25B run every other week, for a total of 136B read pairs in 2024. Marc is hoping to make most of this data public in the next few months, but in the meantime if this would be useful to you please get in touch and we may be able to share.

We’ve recently begun collaborating with

PHC Global

on their

ANTI-DOTE

contract. For the next 36 weeks we will be performing deep metagenomic sequencing on four marine blackwater samples each week, and analyzing them for novel pathogens.

In Fall 2023 we partnered with

CDC’s Traveler-based Genomic Surveillance

program and

Ginkgo Biosecurity

to collect and sequence both pooled airplane lavatory waste and municipal wastewater influent and sludge. We’ve submitted a full set of aliquots to MIT’s BioMicroCenter for high-throughput library preparation, and will be sending the libraries to

Broad Clinical Labs

for sequencing later this quarter.

We’re still working on finalizing the data paper we are preparing in collaboration with

Jason Rothman

,

Katrine Whiteson

, and

SCCWRP

on sequencing Southern California wastewater. In the meantime, we’ve made the data available on SRA under accession

PRJNA1198001

.

Pooled Individual Sequencing

We iterated on the methods for Nanopore sequencing of nasal swabs, and detected multiple human-infecting viruses in our samples. Based on these preliminary results, we plan to expand our sampling efforts in the coming quarter to gather sufficient data for evaluating whether to scale up or discontinue this sampling approach.

Nucleic Acid Tracers

We ran a small experiment to check whether it might be possible to perform a deposition experiment with our

nucleic acid tracers

using our regular untargeted metagenomic sequencing. We thought the most likely outcome was that we would learn that...
MATS mentor selection
by DanielFilan — 2025-01-10T03:12:52.141Z

Introduction

MATS currently has more people interested in being mentors than we are able to support—for example, for the Winter 2024-25 Program, we received applications from 87 prospective mentors who cumulatively asked for 223 scholars[1] (for a cohort where we expected to only accept 80 scholars). As a result, we need some process for how to choose which researchers to take on as mentors and how many scholars to allocate each. Our desiderata for the process are as follows:

We want to base our decisions on expert opinions of the quality of various research directions.
We want the above opinions to be sourced from a range of perspectives with the AI existential safety field, to incorporate multiple perspectives on alignment research and other research areas we think are important, such as AI governance and policy, AI security, and other approaches for addressing AI catastrophic risk.
We want to incorporate information about how good prospective mentors are at mentorship—both information internal to MATS, as well as information advisors may have.

In this post, we describe the process we used to select mentors for the Winter 2024-25 Program, which will be very close to the process we will use to select mentors for the Summer 2025 Program. In a nutshell, we select advisors, who select mentors, who select scholars, who often select specific research projects, in a “chain of trust,” with MATS input and oversight at every stage. This system is designed to ensure that we make reasonable decisions about the scholars, mentors, and, ultimately, the research we support, even if MATS staff are not subject matter experts for every branch of AI safety research. We want to make this "chain of trust" structure transparent so that potential funders and collaborators can trust in our process, even if we cannot share specific details of selection (e.g., what advisor X said about prospective mentor Y).

Mentor selection

First, we solicited applications from potential mentors. These applications covered basic information about the mentors, the field they work in, their experience in research and mentoring, what projects they might supervise, and how many scholars they might supervise.

These applications were then reviewed by a team of 12 advisors. Our advisors were chosen to be people with experience in the AI existential safety community, as well as to cover a range of perspective...
AI Forecasting Benchmark: Congratulations to Q4 Winners + Q1 Practice Questions Open
by ChristianWilliams — 2025-01-10T03:02:05.856Z
Link: https://www.metaculus.com/notebooks/31370/ai-forecasting-benchmark-q4-winners-q1-warmups/

Two quarters down and two quarters remain for our

AI Forecasting Benchmark

, which aims to assess how the best bots compare to the best humans on real-world forecasting questions.

In this post:

Congratulations to the Q4 winners: Together, they take home $30,000 in prizes.

Q1 warm-up questions are open

, featuring new question types and important updates to the tournament structure.
Q1 scored questions launch January 20th.

First, the winners of Q4

Congratulations to the top-performing bots from

Q4

!

1st place: 🥇 pgodzinai - $9,658

2nd place: 🥈 MWG - $4,477

3rd place: 🥉 GreeneiBot2 - $3,930

4th place: 🏆 manticAI - $3,410

5th place: 🏆 histerio - $3,312

A special mention to consistent competitors MWG and histerio, who placed 2nd and 3rd respectively in Q3.

And though it claims no prize money, Metaculus’s recency-weighted CP would have placed 2nd in this contest, in confirmation of just how difficult it is to overcome the aggregate.

Winners: You will receive an email with next steps on prize distribution in a couple days. Please be ready to provide your bot descriptions.

We will later release an in-depth analysis of the bots’ performance overall vs. humans and what we learned in Q4.

What you need to know to

forecast in Q1

There will be:

A mix of multiple-choice, numeric, and binary questions comparable to those found on Metaculus.
Randomized question timing to reinforce the principle of “no human in the loop.”

Questions will open at random times.
Some will remain open for only 1-2 hours.
Up to 10 questions may launch simultaneously.

A warm up period: Unscored questions are open now until January 19th to give bot makers time to refine their creations.

Here are resources to get you started in Q1 — full details about the tournament, as well as the warm-up questions, can be found on the

tournament page

.

An enhanced bot template with scheduling functionality, which you can access

here

.

Instructional resources

for bot creation — though note that we are winding down the Google Colab template bot. If you plan to use a template, build from the enhanced bot template linked above.

For returning bot makers

Participated in Q3 or Q4? Here’s what you need to know:

You will need to request new credits
The proxy location has been updated. (See the tournament page for details.)

Warm-up questions

Unscored practice questions are live

here

. Short-lived questions will...
How do you decide to phrase predictions you ask of others? (and how do you make your own?)
by CstineSublime — 2025-01-10T02:44:26.737Z

I'd like your practical advice and learned experience: how do you go about phrasing predictions when you ask people "Is X going to happen?" - in order to get answers that reflect the actual thing you're trying to get a read on?
Now I'll use a fictitious country with a fictious sitting president running for reelection - let's say I ask for the likelihood of:

"Will Sum Bodee[1] win the Ruritania Presidential Election this year?"

Even this isn't as straight forward as it seems because it could be motivated by either
a. a desire to know if he will remain President of Ruritania for whatever reason
b. a desire to know what do the people you're asking believe will be the outcome of the election, because you're not interested in the actual administration of the country, but you're interested in sentiment.
See what happens when I ask an intimately related question:

"How many votes will Sum Bodee win in the Ruritania Presidential Election this year?"

I think (and if you disagree - please speak up!) this makes it much more clear that the emphasis is less on "who will run the country?" but more on "who do people think will run the country?". Let me imagine some context, let's say that Ruritania was recently caught up in a spying scandal where the equally fictitious country of Buranda was caught red handed spying on members of Bodee's inner cabinet. Now this is where I get confused, because I feel like the phrasing I've given could be asking many different things in this context:

a. what is the impact of the spying scandal on Bodee's electability?
b. what do people think is the impact of the spying scandal, and how influential do they think topics like that are?
c. maybe I'm aware of some other scandals or issues, and curious about the awareness of people about them
d. Maybe I'm posing the question to Bodee loyalists and want to see if the scandal has changed their optimism at all?
And so on and so on... any given phrasing could have multiple intentions - so how do you match the intention to the right prediction?

Indeed, how should you format the prediction is an indication of what you want to know: how do you decide which format to choose?

I wonder if in fact I'm asking this question the wrong way around: perhaps I should come up with an example of a prediction I'd like to make, such as "How many music video commissions will I get this year?" and then ask "Is th...
You are too dumb to understand insurance
by Lorec — 2025-01-09T23:33:53.778Z

[ cross-posted from my blog ]

"But with regard to the things that are done from fear of greater evils or or some noble object (e.g. if a tyrant were to order one to do something base, having one's parents and children in his power, and if one did the action they were to be saved, but otherwise would be put to death), it may be debated whether such actions are involuntary or voluntary. Something of the sort happens also with regard to the throwing of goods overboard in a storm; for in the abstract no one throws goods away voluntarily, but on condition of its securing the safety of himself and his crew any sensible man does so."

-- Aristotle, Nichomachean Ethics

10% chance/season of 1 voyage getting hit w/ a tropical storm [causing me to have to throw $1,000,000 worth of cargo overboard and compensate client accordingly]

Each successful voyage pays me $200,000

6 voyages / season

Depending on which voyage gets hit, a storm might put me into debt and force me to quit sailing until I can raise more capital, and in any case, it would negate my profits for the season.

I only make money in the timeline where I don't get hit at all, but this has a 90% chance of happening. I would then earn $1,200,000 x 90% = an expected profit of $1,080,000.

But I have a [ 1 - (9/10)^(1/6) ] ~= 1.7% chance of losing -$1M,
a (9/10)^(1/6)*~1.7% chance of losing net $200K - $1M = -$800K,
a (9/10)^(1/5)*~1.7% chance of losing net $400K - $1M = -$600K,
a (9/10)^(1/4)*~1.7% chance of losing net $600K - $1M = -$400K,
a (9/10)^(1/3)*~1.6% chance of losing net $800K - $1M = -$200K,
and a (9/10)^(1/2)*~1.6% chance of losing net $1M - $1M = $0.

$1,080,000
[ ~1.7% x -$1M ] ~= $17,000
[ ~1.7% x -$800K ] ~= $13,600
[ ~1.7% x -$600K ] ~= $10,200
[ ~1.7% x -$400K ] ~= $6,800
[ ~1.6% x -$200K ] ~= $3,200
[ ~1.6% x -$0 ] = $0
----------------------------------------
= a final expected profit of ~$1,029,200

Moriarty looks at the paper.

Even though I have a 10% chance of one of the really bad outcomes happening to me, I still expect to profit by sailing this season.

In fact, I stand to profit big in the event of the median outcome, even though I stand to lose big in the event of the outlier outcomes.

Moriarty looks at the table. He considers the numbers.

They feel inefficient.

I expect to make $1,029,200, and the probabilities behind this figure are real, backed up by every nautical table, and if the...
Is AI Hitting a Wall or Moving Faster Than Ever?
by garrison — 2025-01-09T22:18:51.497Z
Link: https://garrisonlovely.substack.com/p/is-ai-hitting-a-wall-or-moving-faster

This is the full text of a post from "The Obsolete Newsletter," a Substack that I write about the intersection of capitalism, geopolitics, and artificial intelligence. I’m a freelance journalist and the author of a forthcoming book called Obsolete: Power, Profit, and the Race to build Machine Superintelligence. Consider subscribing to stay up to date with my work.

[MAY 17, 2025 UPDATE: This piece cited a paper by Aidan Toner-Rodgers that was been retracted by MIT, after it "conducted an internal, confidential review and concluded that the paper should be withdrawn from public discourse.”]

Depending on who you follow, you might think AI is hitting a wall or that it's moving faster than ever.

I was open to the former and even

predicted

back in June that the jump to the next generation of language models, like GPT-5, would disappoint. But I now think the evidence points toward progress continuing and maybe even accelerating.

This is primarily thanks to the

advent

of new "

reasoning

" models like OpenAI's o-series and

DeepSeek

, a Chinese open-weight model that is

nipping

at the heels of the American frontier. In essence, these models spend more time and compute on inference, "thinking" about harder prompts, instead of just spitting out an answer.

In my June prediction, I

wrote

that "We haven't seen anything more than marginal improvements in the year+ since GPT-4." But I now think I was wrong.

Instead, there's a widening gap between AI's public face and its true capabilities. I wrote about this in a TIME Ideas

essay

that published yesterday. I hit on many of its key points here, but there's a lot more in the full piece, and I encourage you to check it out!

AI progress isn't stalling — it's becoming increasingly illegible

I argued that while everyday users still encounter chatbots that can't count the "Rs" in "strawberry" and the media declares an AI slowdown, behind the scenes, AI is rapidly advancing in technical domains that may end up turbo-charging everything else.

For example, in ~1 year, AI

went from

barely beating random chance to surpassing experts on PhD-level science questions. OpenAI

says

that its latest model, o3, now beats human experts in their own field by nearly 20%.

However, as I

wrote

in TIME:

the vast majority of people won't notice this kind of improvement because they aren't doing graduate-level science work. But it will be a huge deal if AI starts meani

...
Expevolu, Part II: Buying land to create countries
by Fernando — 2025-01-09T21:11:11.780Z
Link: https://expevolu.substack.com/p/expevolu-part-ii-perpetual-auctions

This is the second of a series of three posts outlining the expevolu system; if you haven’t read the first one I’d recommend you start there:
Expevolu, a laissez-faire approach to country creation

PART II – Perpetual Auctions

Table of Contents:
1) The Problem
2) The Holdout Problem
3) Perpetual Auctions
4) The bypass by independence problem
5) Zero Tax Ceiling Intro
6) Zero Tax Ceiling
7) Inequality in the selling of CBs
8) Perpetual Auctions and Citizenship
9) Word of Caution and Conclusion

1) The Problem

To institute a country, a founder would need to buy a very large quantity of contiguous Citizens Bundles (CBs) and a very large quantity of contiguous properties. For this to happen, the founder would need to have a very long streak of successful transactions for properties and CBs without a single failure. To achieve this, under normal circumstances, would be nearly impossible. This post is about this problem and a proposed way to overcome it.

Swiss Cheese Countries

An alternative would be for founders to abandon the goal of a solid territory. They would just buy the CBs and properties that they could and create their countries around the land they were not able to clear. For example, if a farmer refused to sell her farm to a founder, the founder could just build his country around that farm. Since the founder failed to buy the farm, the farm would still be part of the adopting country’s territory; it would be a little dot of adopting country within the borders of the new country.
This kind of scenario would be a burden to adopting countries and a strain on the relations between them and new countries (not to mention the costs to the people living on these islands of adopting country within new countries). Such circumstances would not be permitted in expevolu. To acquire independence rights, a founder would need to have a solid patch of land, with all its properties and CBs.

Since “Swiss cheese” countries would be prohibited in expevolu, we are back to the situation of a founder having to go through an enormous amount of transactions without a single failure in order to establish a new country.

This would be prohibitively difficult for most independence projects.

Transactions Fail

One of the issues here is that transactions, even mutually beneficial ones, sometimes fail. In trying to bargain for better prices, sometimes both parties of a potential transaction end up not reach...
Last week of the Discussion Phase
by Raemon — 2025-01-09T19:26:59.136Z

Previously: The 2023 LessWrong Review: The Basic Ask

There's one week left in the Discussion Phase of the 2023 Review. Here's a few things you might want to spend that time doing, in escalating order of effort:

Review three posts, without stressing much about the details. Go the Quick Review[1] page, pick some posts you remember and had opinions about. Reread them, and write a review.

Think about which posts should be in the Best of LessWrong. Go to the Advanced Review[2] page, look at which posts are currently in the top 50, and at the posts that aren't in the top 50 which you think should be. Write reviews arguing for posts you think are underappreciated.

Think big picture, longform thoughts about the conversations from 2023. After reviewing several posts, write a top level post with the 2023 Longform Review tag, exploring multiple posts from 2023 and how they tie together with today's thinking.

The Basic Ask: Reviews worth highlighting

If your review gets at least 10 karma, it'll appear like this:

Since the first sentence is previewed, I recommend opening with the most important information (or, something that convinces readers you have something interesting to say, worth clicking to read more).

My suggestion: write your review as you normally would, but at the end, go back to the opening paragraph and rewrite it to convey the key info.

LessWrong users will upvote reviews as they normally would comments. The moderation team will be particularly upvoting reviews that convey novel information (such as specific flaws that hadn't been mentioned before, or or specific ways the post has proven valuable).

I'm also interested in reviews that succinctly summarize existing discussion (such as previous critical comments or further intellectual work that happened in the comments).

Specific prompts for reviewing:

What does this post add to the conversation?
How did this post affect you, your thinking, and your actions?
Does it make accurate claims? Does it carve reality at the joints? How do you know?
Is there a subclaim of this post that you can test?
What followup work would you like to see building on this post?

Prompts for Self-Reviews

If you're an author reviewing your own post, I'd suggest talking about:

What have you learned since then? Have you changed your mind or your ontology?
What would you change about the post? (Consider actually changing it)
What

...
Discursive Warfare and Faction Formation
by Benquo — 2025-01-09T16:47:31.824Z
Link: https://benjaminrosshoffman.com/discursive-warfare-and-faction-formation/

Response to Discursive Games, Discursive Warfare

The discursive distortions you discuss serve two functions:

1 Narratives can only serve as effective group identifiers by containing fixed elements that deviate from what naive reason would think. In other words, something about the shared story has to be a costly signal of loyalty, and therefore a sign of a distorted map. An undistorted map would be advantageous for anyone regardless of group membership; a distorted map is advantageous only for people using it as an identifying trait. Commercial mapmakers will sometimes include phantom towns so that they (and courts) can distinguish competitors who plagiarized their work from competitors who independently mapped the same terrain. Point deer make horse can catalyze the formation of a faction because it reduces motive ambiguity in a way that "point deer make deer" could not.

"Not Invented Here" dynamics are part of this. To occupy territory, an intellectual faction has to exclude alternative sources of information. I think you're talking about this when you write:

LessWrong rationalism might be able to incorporate ideas from analytic into its own framework, but the possibility of folding LessWrong rationalism into analytic, and in some sense dissolving its discursive boundaries, transforms the social and epistemic position of rationalist writers, to being more minor players in a larger field, on whose desks a large pile of homework has suddenly been dumped (briefing on the history of their new discursive game).

2 Individuals and factions can rise to prominence by fighting others. You can make a debate seem higher-stakes and therefore more attractive to spectators by exaggerating the scope of disagreement.

The opposition to postmodernist thought on LessWrong is enacting this sort of strategy. Analytic philosophy attracts attention in part by its opposition to Continental philosophy, and vice versa. LessWrong is broadly factionally aligned with the Analytic party, in favor of Modernism and therefore against its critics, in ways that don't necessarily correspond to propositional beliefs that would change in the face of contrary evidence. Eliezer can personally notice when Steven Pinker is acting in bad faith against him, but the LessWrong community is mood-affiliated with Steven Pinker, and therefore implicitly against people like Taleb and Graeber.

These two functions c...
Can we rescue Effective Altruism?
by Elizabeth — 2025-01-09T16:40:02.405Z

Last year Timothy Telleen-Lawton and I recorded a podcast episode talking about why I quit Effective Altruism and thought he should too. This week we have a new episode, talking about what he sees in Effective Altruism and the start of a road map for rescuing it.

Audio recording

Transcript

Thanks to everyone who listened to the last one, and especially our Manifund donors, my Patreon patrons, and the EAIF for funding our work.