trentmkelly/LessWrong-43k huggingface dataset:
Galatea and the windup toy
by Nicolas Villarreal — 2024-10-26T14:52:03.861Z
Link: https://nicolasdvillarreal.substack.com/p/galatea-and-the-windup-toy

In previous posts on AI I have given some critiques regarding the

orthogonality thesis

and rationalist notions of

optimizing towards rational, coherent goals

. The purpose of this blog is to correct mistakes and carry the conclusions of those blogs forward. To summarize, these previous blogs have described flaws in rationalist ideology, the most relevant one here is the idea that rational goals require a specific state of the world to aim/optimize for. I’ve previously shown that any such world-state or goal, no matter how rigorous, does not exist “out there” in the world, they only have meaning within the map rather than the territory, and indeed, a concept of “the map” at all only exists within an internal system of signs rather than in the territory more broadly.

I created a number of concepts to form my critique: first order signs, higher order signs, and sign function collapse. To those unaware, signs are things which stand in for other things, they have a signifier, the symbol, and the signified, the things it stands in for. First order signs stand in for something directly, the word chair for a chair, the word atoms for atoms, the word anger for the immediate sensation of anger. Higher order signs are signs which stand in for other signs. Sign function collapse is when a higher order sign becomes a first order sign. What I mean by this is that for each sign there is a process of signification, where we come to correlate different bits of phenomenal experience together with a symbol. When someone is teaching us how to speak for the first time, and they point to an apple and say “apple”, it’s impossible for us to know whether “apple” refers only to this particular specimen of apple, this particular experiencing of seeing the apple, its color, its shape, etc. What precisely “apple” means can only be learned through access to more examples in various different contexts, this creates the correlation between the signifier “apple” and the juicy object we all enjoy. This is the process of signification.

Now let’s think of the phrase “apple juice”, since as a compound name it’s useful for describing the process of sign function collapse. When we first learn about juices, we probably first arrive at “apple juice” as a higher order sign. That is, we learn what juice is, and we learn what apples are, and we combine the two together. So long as the phrase ...
Why is there Nothing rather than Something?
by Logan Zoellner — 2024-10-26T12:37:50.204Z

"Close the darn window! You know it gives me the creeps," Doctor Myrkina hissed, glaring angrily at the daylight streaming in through the open curtains.

"Sunlight is good for you," Captain Tralith insisted, pulling the curtains open wider. She smiled in a vaguely threatening way as she did so, showing off far too many teeth. "I never understood what bothered you about it anyway."

"Come over here," the doctor waved a finger at at her, blocking the sunlight with his other hand as his eyes slowly adjusted to the light.

"I've told you before not to keep ants on this ship," Captain Tralith said, looking where he was pointing. "They're going to get out of that ant-farm of yours sooner or later. And then we'll have a mess on our hands."

"I know, I know--shush," Doctor Myrkina said. "Does anything strike you as unnatural about this ant-farm?"

"It strikes me as unnatural how you keep bringing bugs in here," the captain said. "I like to run a clean ship, you know."

"Think from an ants' perspective, not ours," Doctor Myrkina pressed. "You don't think there's anything that would bother them about their current situation?"

"I suppose they wouldn't like us staring at them through the glass like this. It's rather an invasion of their privacy. Is that why you don't like the window open? You think somebody's watching? I assure you, this planet is perfectly uninhabited. Much like every other planet humans have ever visited, besides Earth. But, shouldn't you be telling me that?"

"That does bother me," Doctor Myrkina agreed. "Let me ask this another way. How many dimensions does an ant colony have?"

"Oh! I know this one!" Captain Tralith beamed. "It's a fractal! So, the answer is... I don't know exactly. But less than three and more than one, I suppose."

"One, because the tunnels in the anthill are one-dimensional and three because the anthill exists in a subspace of our own three-dimensional space, correct?"

"Okay, but how does that answer your question about unnaturalness?" Captain Tralith asked, confused.

"How many dimensions does this anthill exist in? Less than how many exactly?"

"Less than two, I suppose," the captain said. "Since it's squeezed between those two pieces of glass. I still don't like you bringing ants on here. Look at those ones in the corner. I think t...
The Summoned Heroine's Prediction Markets Keep Providing Financial Services To The Demon King!
by abstractapplic — 2024-10-26T12:34:51.059Z

The Summoned Heroine and the Demon King

The Summoned Heroine announced to the Circular Citadel that she would be creating a prediction market, to help them anticipate and counter the Demon King’s next moves. There was cheering, on general principles.

The Demon King donned a mortal guise, bought shares in “The Demon King will attack the Frozen Fortress”, and then attacked the Frozen Fortress.

The Summoned Heroine announced to the Circular Citadel that, due to recent events, strict limits would be imposed on how much any individual would be permitted to invest in the prediction markets. There was only slightly less cheering.

The Demon King donned a series of mortal guises, bought shares in “The Demon King will attack the Frozen Fortress”, and then attacked the Frozen Fortress.

The Summoned Heroine announced to the Circular Citadel that, due to certain unforeseen difficulties, their taxes would now only be seeding trades which wouldn’t subsidize the Demon King’s attacks. There was some cheering.

The Mercentile Monarchs didn’t share a border with any Demonic territories, and didn’t care who won the Summoned Heroine’s war, but did want accurate predictions so they could accurately plan trade routes. They noticed that the Heroine had generously funded half of a fair and fully-liquid prediction market, and sensibly seeded the remaining outcomes.

The Demon King donned a mortal guise, bought shares in “The Demon King will attack the Frozen Fortress”, and then attacked the Frozen Fortress.

The Summoned Heroine announced to the Circular Citadel that, as a result of stringent regulation and intense diplomatic efforts[1], it was now completely impossible for the Demon King to make any money predicting anything other than failures by Demonic forces, and that furthermore, the payouts for such failures could not exceed the monetary value of the relevant losses to the Demonic Kingdom. There was a little cheering, mostly motivated by concern about what might happen if someone using that many italics expected cheering and didn’t get it.

The Fair Prince gently reminded her that the status quo she’d been summoned to improve upon was that the Circular Citadel paid the Demonic Kingdom an exorbitant annual tribute to not attack them. He politely inquired as to how paying them to launch failed attacks was an improvement, since (to his admittedly untrained eye) the main change was an increase...
AI Safety Camp 10
by Robert Kralisch — 2024-10-26T11:08:09.887Z

We are pleased to announce that the 10th version of the AI Safety Camp is now entering the team member application phase!
AI Safety Camp is a 3-month long online research program from January to April 2025, where participants form teams to work on pre-selected projects.
We have a wide range of projects this year again, so check them out to see if you or someone you know might be interested in applying to join one of them.

You can find all of the projects and the application form on our website, or directly apply here. The deadline for team member applications is November 17th (Sunday).

Below, we are including the categories and summaries of all the projects that will run in AISC 10.

Stop/Pause AI

(1) Growing PauseAI

Project Lead: Chris Gerrby

Summary

This project focuses on creating internal and external guides for PauseAI to increase active membership. The outputs will be used by the team of volunteers with high context and engagement, including the key decision makers.

Activism and advocacy has historically been cornerstones for policy and social change. A movement's size is critical to achieving its goals. The project's outcome will be an official growth strategy for PauseAI Global: an actionable guide for increasing active members (volunteers, donors, protesters) within 6-12 months.

PauseAI currently lacks comprehensive growth strategies and tactics, and there's uncertainty about resource allocation soon. This project will delve into tactics that have aggressively accelerated growth for other movements, both historical and recent.

By the end, we'll refine our findings, analyze PauseAI's current tactics, and recommend clear guidelines for immediate action. This guide will also include tactics applicable to national PauseAI chapters.

(2) Grassroots Communication and Lobbying Strategy for PauseAI

Project Lead: Felix De Simone

Summary

PauseAI is a global, grassroots organization with the goal of achieving a worldwide moratorium on the development of dangerous AI systems. We are seeking to improve our communication strategy, both in terms of public communications and in meetings with elected officials.

This project will have two tracks:

Track 1: Lobbying. This track will focus on researching the optimal lobby strategies in non-US countries around which PauseAI wishes to expand our lobby efforts.

Track 2: Grassroots Communication. Participants...
Arithmetic Models: Better Than You Think
by kqr — 2024-10-26T09:42:07.185Z
Link: https://entropicthoughts.com/arithmetic-models-better-than-you-think

LessWrong user dynomight explains how arithmetic is an underrated world-modeling technology and uses dimensional algebra as the motivating case. I agree dimensional algebra is fantastic, but there’s an even better motivating example for arithmetic in world-modeling: linear models for prediction.

Simple linear models outperform experts

In 1954, Paul Meehl published what he later came to call my disturbing little book. This book[1] contains the most important and well-replicated research I know of; yet most people don’t know about it. The basic argument is that many real-world phenomena – even fickle ones – can be adequately modeled with addition and multiplication.

Tempered by the lack of evidence at the time, the book doesn’t go quite as far as it could have. Here are some statements that have later turned out to be true, given in order of increasing outrageousness.

Simple linear models outperform experts.
Simple linear models outperform experts when the experts get access to additional information that the model does not get.
Simple linear models outperform experts when the experts also get the know and use the outcome of the linear model.
Simple linear models trained only on experts’ judgments and not the actual outcome outperform the very experts they were trained on.
Simple linear models with random weights outperform experts.
Simple linear models with equal weights (i.e. a tallying heuristic) outperform experts.
Simple linear models with equal weights when limited to three predictors still outperform experts.

Obviously, these are phrased to provoke, and take additional nuance to be fully understood, but the general theme remains the same: addition and multiplication take you surprisingly far.

Predictive modeling is more important than explainatory modeling

There are two reasons to make models: predictive and explainatory.[2]

Predictive modeling. Figure out how things are correlated: if X has been observed, does that generally happen when Y is more or less likely to happen?
Explainatory modeling. Figure out how things are causally linked. When X happens, does that trigger Y to also happen?

Most people reason about the world through explainatory modeling, and are uncomfortable with predictive modeling. I would argue that predictive modeling is the better approach.

Predictive modeling has the funny property that it is acausal and atemporal: we can predict the risk of ...
Is the Power Grid Sustainable?
by jefftk — 2024-10-26T02:30:06.612Z

When I was growing up most families in our neighborhood had the daily
paper on their lawn. As getting your news on the internet became a
better option for most folks, though, delivery became less efficient:
fewer houses on the same route. Prices went up, more people
cancelled, and decline continued. I wonder if we might see something
similar with the power grid?

Solar panels keep getting cheaper. When we installed panels in 2019 they only
made sense because of the combination of net metering and the
relatively generous SREC II incentives. By the time we installed our
second set of panels a few months
ago net metering alone was enough to make it worth it.

Now, as I've
said a few
times, net metering is kind of absurd. The way it works here is
that most of my bill is in proportion to my net consumption: if I use
100kWh but also send 100kWh back to the grid I don't pay anything for
transmission or generation. I can use the grid as a giant battery,
for just a $10/month grid-connection charge.

As more people run the numbers and install rooftop solar, I expect
there to be increasing pressure to limit net metering, either for new
installs or for everyone. But even if net metering were completely
phased out it wouldn't solve the problem: as people draw less from the
grid the company either needs to raise their per-kWh rates or their
per-customer charges. Raise the former and installing solar becomes
even more attractive; raise the latter and people start thinking about
disconnecting entirely.

Solar is getting cheaper very quickly:

Our
World In Data

And so are
batteries. It's going to make sense for a lot of houses to go
over to solar + batteries. And if batteries are too expensive for the
longest stretch of cloudy days you might have, at least here a natural
gas generator compares favorably.

Wait, really? This was pretty surprising to me, but here's the
sketch. My marginal cost of electricity is currently $0.33/kWh [1]
vs $0.06/kWh [2] for natural gas. Now I can't charge my laptop with
raw gas, which means burning the gas in a generator, and since that's
well under 100% efficient I'm not actually getting a usable $0.06/kWh.
While a power station could be up to 63%
efficient, for a home generator maybe I'm looking at something
like the 23% efficient Generac
7171, rated for 9kW on natural gas at full load. Or maybe
something smal...
A Case for Conscious Significance rather than Free Will.
by James Stephen Brown — 2024-10-25T23:20:30.834Z

The following is born out of a frustration with the free will, determinism and compatibilism debate and its terminology—finding my intuitions do not seem to align with a particular camp. This is an attempt to explain my philosophical position in a way that provides a basis to reconcile that which is subjectively important (to me at least) with a possibly deterministic universe.

Conscious Significance

My position is agnostic regarding determinism, it holds for either determinism, a mixture of determinism and indeterminism, or free will. The way I see it is that what I want, or what I feel I have in terms of autonomy, is not 'free will' but rather conscious significance—my conscious experience is relevant to outcomes.

So, I begin from the position that basically everything is set about me by forces outside of my control—I don't get any choice over: my desires, my brain's model of the world, my brain's model of myself or even my intentions. My default mode in the world is to have intentions arise, and when there is an obvious way to act to achieve that goal, I do so, and then my conscious mind experiences the results, and that (subconsciously) informs my automatic processes, my model of the world and my model of myself.

But, when I have an intention where there is not a clear outcome, I can make a decision.

Making Decisions

What a decision involves, is running through numerous simulations until I find one that is satisfactory, and then I can act on it. The conscious brain does the same thing it always does, experiences the result of action (in this case the predicted result of the simulation), which informs my model of the world and myself, and this loops until I have a satisfactory option to act on. This, as I see it, is the role of my conscious brain and comprises my (personal) sense of autonomy.

My take-away from this is that, although consciousness, in both its experience of the world and its experience of internal simulations, happens after the fact, it is a necessary factor in the iterative internal process when determining a satisfactory action, and this is what we experience as deliberate decision-making. This supports the only important aspect (to me at least) of what people call 'free will', our sense of autonomy and what I would prefer to call 'conscious significance'. This is what I'm concerned about—the centrality of consciousness in deliberate decision-m...
Introducing Kairos: a new AI safety fieldbuilding organization (the new home for SPAR and FSP)
by agucova — 2024-10-25T21:59:08.782Z

TL;DR

We’re announcing Kairos, a new AI safety fieldbuilding organization focused on the early career talent pipeline.
We’re the new home for two programs:

FSP, which provides semester-based mentorship for organizers of AI safety university groups
SPAR, a program that pairs early career individuals to work on three-month AI safety research projects alongside experienced mentors

We’ve opened applications for the next round of FSP (starting in December 2024) and expressions of interest for SPAR (starting in February 2025).

Announcing Kairos

We’re excited to announce the launch of Kairos[1], a new AI safety fieldbuilding organization focused on strengthening the early career talent pipeline. Kairos will serve as the new institutional home for two existing programs: the Supervised Program for Alignment Research (SPAR), a mentorship program now on its fifth iteration, and the Fieldbuilder Support Program (FSP), a program in its second iteration.

We believe that growing and strengthening the infrastructure for AI safety talent is among the most promising fieldbuilding efforts, so we’re now focused on improving what we think are critical and neglected segments of the pipeline. We’re also planning to run a number of smaller, targeted support programs for AI safety groups. We hope to share more about our broader plans in the coming months.

Fieldbuilder Support Program (FSP)

FSP is a mentorship program (recently spun off of CEA’s Groups Team) that supports university AI safety group organizers in establishing and running their groups. During the program, organizers get paired with mentors who guide them through defining their group’s strategy and planning their semesters, as well as following up on their progress. The program has an initial period of three weeks, but most participants go on to be mentored for their entire semester.

Applications are now open!

The program will run from December 5th, 2024, through to the next semester
Each interested group organizer must apply separately

Apply here by November 15

We’re also looking for experienced existing or former group organizers to participate as paid mentors for the program. See here for details.

Supervised Program for Alignment Research (SPAR)

SPAR is a virtual, part-time research program that connects early-career individuals with experienced AI safety researchers. Participants work on 3-month project...
Brief analysis of OP Technical AI Safety Funding
by 22tom — 2024-10-25T19:37:41.674Z

TL;DR

I spent a few hours going through Open Philanthropy (OP)'s grant database. The main findings were:

Open Philanthropy has made $28 million grants for Technical AI Safety (TAIS) in 2024

68% of these are focused on evaluations / benchmarking. The rest is split between interpretability, robustness, value alignment, forecasting, field building and other approaches.

OP funding for TAIS has fallen from a peak in 2022

Excluding funding for evaluations, TAIS funding has fallen by ~80% since 2022.

A majority of TAIS funding is focused on "meta" rather than "direct" safety approaches

My overall takeaway was that very few TAIS grants are directly focused on making sure systems are aligned / controllable / built safely.[1]

Method

I:

Downloaded OP's list of grants
Filtered for "Potential Risks from Advanced AI"
Classified grants as either "Policy" or "Non-Policy"
Within "Non-Policy", classified grants by different focus areas (e.g. evaluations, interpretability, Field building, "Multiple" and "Other")

In most cases I classified just from the grant name; occasionally I dug for a bit more info. There are definitely errors, and cases where grants could be more clearly specifcied.

Combined focus areas into "Clusters" - "Empirical", "Theory", "Forecasting & Evaluating", "Fieldbuilding" and "Other"
Created charts

Results

Grants by Research Agenda

Grants by Cluster

Full data available here

Key Findings

(1) Evaluations & Benchmarking make up 2/3rds of all OP TAIS funding in 2024

Most of these grants are related to the RFP on LLM Benchmarks.

TAIS Grants in 2024 by Research Agenda

(2) Excluding Evaluations & Benchmarking, OP grants for TAIS have fallen significantly

2022 Funding (excluding evaluations): $62,089,504
2023 Funding (excluding evaluations): $43,417,089
2024 Funding (projected, excluding evaluations): $10,808,390

82.6% reduction vs 2022

(3) Most TAIS funding is focused on "investment" rather than direct approaches to AI safety

I classify grants into two broad buckets:

"Direct" - grants for research agendas which aim to improve safety today (e.g. Interpretability, Control, Robustness, Value Alignment, Theory)
"Investment" - grants which pay off through future impact - e.g. Field building, Talent development, reducing uncertainty (via Forecasting & Evaluation)

^

More cynically: I worry the heavy focus on evaluations will give us a great understanding of how / when sca

...