trentmkelly/LessWrong-43k huggingface dataset:
a confusion about preference orderings
by nostalgebraist — 2025-05-11T19:30:08.842Z

Here's a confusion I have about preference orderings in decision theory.

Caveat: the observations I make below feel weirdly trivial to me, to the point that I feel wary of making a post about them at all; the specter of readers rolling their eyes and thinking "oh he's just talking about X in a really weird way" looms large in my mind as I type. It feels like I may just be unaware of some standard term or concept in the literature, which would make everything "snap into place" if I knew about it. If so, let me know.

diagrams

Let's say I draw something like this:

Here, the letters represent something like "world states," and an arrow like A→C means "C is preferred over A (by the 'agent' whose preferences this graph describes)."

For now I'm being hand-wavey about exactly what's being expressed here, but I trust that this sort of thing will be familiar to an LW audience – as will the types of discussions and debate (about coherent preference, EU maximization, etc.) in which this sort of thing gets used.

I will make one clarification about the meaning here at the outset: everything that the agent's preferences care about is captured by the "world states," the capital letters. The only things in the situation capable of being "good/bad" or "better/worse" are the states.

(That's why we can confidently draw arrows like A→C, without having to specify anything else about what's going on. The identities of the letters appearing on the left and right sides of an arrow are always sufficient to determine the direction of the arrow.)

So, for instance: the agent doesn't have extra, not-shown-on-the-diagram preferences about taking particular trajectories through particular sequences of states in order. Or about being in particular states at particular "times" (in any given sense of "time"). Or about anything like that.

(If we wanted to model these kind of things, we'd need to "bring them inside the diagram" by devising a new, larger set of states that mean things like "the agent is in A after following trajectory Blah," or whatever.)

preferred vs. accessible

Now, there's a frequently made inference I see in those discussions, which goes from one way of interpreting the "arrows" to another.

The definition of an arrow like A→C is the one I gave above: "the agent 'prefers' the state C to the state A."

This doesn't, in itself, sa...
[Book Translation] Three Days in Dwarfland
by Viliam — 2025-05-11T17:54:25.198Z

There was a book written half a century ago in Soviet Union, where a group of three kids travel to the land of arithmetic, meet anthropomorphic numbers, and learn a few mathematical concepts in a way appropriate for kids about 10 years old...

And I thought: "What a pity this was never translated to English."

Well, thanks to LLMs, this is not a problem anymore: download here 💾.

I hope you enjoyed it!

That said, the translation is very far from perfect. I tried to fix the most obvious mistakes made by the automatic translation, but English is not my first language. Plus there are a few poems I didn't have patience for, beyond telling the LLM to give me a few different versions and choosing the one that seemed best.

So...

If you agree with me that this book should officially exist in English, and you have some experience publishing books, then I think you could do the following:

find the copyright owners (the author died a few decades ago) and make some deal with them;
get the translation fixed by a proper translator, or at least someone good at English, because the text already is translated;
make new pictures (the ones in the PDF are low resolution scans from the original book, and some of them even contain text in Cyrillic);
and publish it officially, of course! Preferably as both a paper book and an e-book.

I would be happy to assist the translator/corrector with their work, as I have already spent some time thinking about the text.

After you publish the book, I will be happy to replace the download link with a link to your shop.

By the way, the book has sequels; they are just as good as this one. (Not translated yet.)
Better Air Purifiers
by jefftk — 2025-05-11T16:50:03.317Z

Are you looking for a project where you could substantially improve
indoor air quality, with benefits both to general health and reducing
pandemic risk?

I've written
a bunch
about
air
purifiers
over the past few years, and its frustrating how bad commercial
market is.

The most glaring problem is the widespread use of HEPA filters. These
are very effective filters that, unavoidably, offer significant
resistance to air flow. HEPA is a great option for filtering air in
single pass, such as with an outdoor air intake or a biosafety
cabinet, but it's the wrong set of tradeoffs for cleaning the air
that's already in the room. Air passing through a HEPA filter removes
99.97% of particles, but then it's mixed back in with the rest of the
room air. If you can instead remove 99% of particles from 2% more
air, or 90% from 15% more air, you're delivering more clean air. We
should compare in-room purifiers on their Clean Air
Delivery Rate (CADR), not whether the filters are HEPA.

Next is noise. Let's say you do know that CADR is what counts, and
you go looking at purifiers. You've decided you need 250 CFM, and you
get something that says it can do that. Except once it's set up in
the room it's too noisy and you end up running it on low, getting just
75 CFM. Everywhere I go I see purifiers that are either set too low
to achieve much or are just switched off. High CADR with low noise is
critical.

Then consider filter replacement. There's a competitive market for
standardized filters, where most HVAC systems use one of a small
number of filter sizes. Air purifiers, though, just about always use
their own custom filters. Some of this is the mistaken insistence on
HEPA filters, but I suspect there's also a "cheap razors, expensive
blades" component where manufacturers make their real money on
consumables.

Then there's placement. Manufacturers put the buttons on the top
and send air upwards, because they're designing them to sit on the
floor. But a purifier on the floor takes up valuable space, is
farther from where we inhale, and is at more risk of damage. Purifiers
should be designed to use the space above our heads: on top of
bookcases or shelves; mounted to walls or ceilings.

Finally, there's very little innovation. The top purifier on the
Wirecutter is the Coway Airmega
AP-1512HH, which has been their top pick since they first reviewe...
Aligning Agents, Tools, and Simulators
by WillPetillo — 2025-05-11T07:59:52.495Z

This post was written as part of AISC 2025.

Introduction

In Agents, Tools, and Simulators, we outlined several lenses for conceptualizing LLM-based AI, with the intention of defining what simulators are in contrast to their alternatives. This post will consider the alignment implications of each lens in order to establish how much we should care if LLMs are simulators or not. We conclude that agentic systems seem to have a high potential for misalignment, simulators have a mild to moderate risk, tools push the dangers elsewhere, and the potential for blended paradigms muddies this evaluation.

Alignment for Agents

The basic problem of AI alignment, under the agentic paradigm, can be summarized as follows:

Optimizing any set of values pushes those values not considered to zero.
People care about more things than we can rigorously define.
All values are interconnected, such that destroying some of them will destroy the capacity of the others to be meaningful.
Therefore, any force powerful enough to actualize some set of definable values will destroy everything people care about.

The above logic is an application of

Goodhart's Law

, the simplified version of which states that a measure that becomes a target ceases to be a good measure. Indeed, all of the classic problems of alignment can be thought of in terms of Goodhart's Law:

Goal-misspecification: the developer has a goal in mind, but writes an imperfect proxy as a reinforcement mechanism. Pursuit of the proxy causes the system to diverge from the developer's true goal.

Mesa-optimization

: the AI's goals are complex and feedback is sparse, so it generates easier-to-measure subgoals. Pursuit of the proxy subgoals cause the system to diverge from its own initial goal.

Goal-misgeneralization

: the AI’s goals optimize for the training data, which does not match the real world. Optimizing to the training data causes the AI to act according to patterns specific to idiosyncrasies of the training data, which causes unpredictable behavior in deployment.
Societal misalignment: The developers of AI optimize for their own incentives—such as profit, prestige, or ideology—rather than the broader welfare of humanity.

In short, a sufficiently capable optimizing agent will reshape the world in ways that humans did not intend and cannot easily correct. These problems are amplified by the theory o...
Consider not donating under $100 to political candidates
by DanielFilan — 2025-05-11T03:20:04.986Z

Epistemic status: thing people have told me that seems right. Also primarily relevant to US audiences. Also I am speaking in my personal capacity and not representing any employer, present or past.

Sometimes, I talk to people who work in the AI governance space. One thing that multiple people have told me, which I found surprising, is that there is apparently a real problem where people accidentally rule themselves out of AI policy positions by making political donations of small amounts—in particular, under $10.

My understanding is that in the United States, donations to political candidates are a matter of public record, and that if you donate to candidates of one party, this might look bad if you want to gain a government position when another party is in charge. Therefore, donating approximately $3 can significantly damage your career, while not helping your preferred candidate all that much. Furthermore, at the time you make this donation, you might not realize that you will later want to get a government position.

Now, I don’t want to overly discourage this sort of thing. It’s your money, free speech is great, and fundamentally I think it’s fine to have and publicly express political views (for example, I think Donald Trump is extremely bad, and am disappointed in my fellow countrymen for voting for him). That said, I think that one should be aware of the consequences of making political donations, and it seems plausible to me that if you’re not willing to donate more than $100 to a political candidate, consider that the career cost to you of making that donation may be higher than the benefit that it confers.
Somerville Porchfest 2025
by jefftk — 2025-05-11T02:00:02.327Z

Somerville
Porchfest was a lot of fun this year, and we got a great crowd:

Showing some pictures to Claude it guesses there might have been 600
people. Very glad the city gave us permission to close the street! [1]

It would have been good not having cars parked in the main dance area,
since this ended up making us need to spread out a lot more than would
be ideal. Next time I'm going to see if I can get friends to park in
the four main spots in the ~day leading up to the dance and then move
their cars just as we're starting.

Harris did a great job
leading the dancing, calling a mix of simple dances for everyone and
standard contras for experienced dancers. This is also how he did it
last year, and I think this does a good job covering the somewhat
incompatible things people are looking for out of the event. For the
standard contras we had one ~130ft line, while for the simple dances
we had two lines of similar length.

I played with Rohan and Charlie, with Weiwei and Rick sitting in. I
think we sounded pretty good; a bit of a pickup band, but still fun!
Sam sat in for one dance on piano and Weiwei and I got to play trumpet
and baritone together. At the end I convinced Sam to try playing the
kick drum while I played hi-hat—part of my long-term plan to
hook more people on foot
drumming.

Sound was a bit chaotic. I was running sound while also playing, and
this is never ideal. I got a few people (thanks Rick and Cecily) to
give me feedback on the mix, but next year we should line up a dedicated
volunteer for sound.

Speaker placement also matters a lot: we started with the speakers on
the porch because it hadn't quite stopped raining, but it wasn't
possible to get them anywhere near loud enough for the crowd without
feedback. As soon as the rain let us we moved them well in front of
the band, which let us run them a lot louder without feedback and give
better coverage over the crowd. But it was awkwardly loud right in
front of the speakers, even though they were ~10ft up. Next time I'd
like to try fills: a pair of K10s in the center, and then a pair of
K10.2s (with built-in delays) ~30ft up and down the street to help
even out the sound.

With so many people it would also be helpful to have additional
volunteers on crowd control, beyond just Harris at the mic: sometimes
it's clearer to show than tell. Anoth...
It's Okay to Feel Bad for a Bit
by moridinamael — 2025-05-10T23:24:25.088Z

"If you kiss your child, or your wife, say that you only kiss things which are human, and thus you will not be disturbed if either of them dies." - Epictetus

"Whatever suffering arises, all arises due to attachment; with the cessation of attachment, there is the cessation of suffering." - Pali canon

"An arahant would feel physical pain if struck, but no mental pain. If his mother died, he would organize the funeral, but would feel no grief, no sense of loss." - the Dhammapada

"Receive without pride, let go without attachment." - Marcus Aurelius

I.

Stoic and Buddhist philosophies are pretty popular these days. I don't like them. I think they're mostly bad for you if you take them too seriously.

About a decade ago I meditated for an hour a day every day for a few weeks, then sat down to breakfast with my delightful (at the time) toddlers and realized that I felt nothing. There was only the perfect crystalline clarity and spaciousness of total emotional detachment. "Oh," I said, and never meditated again.

It's better to sometimes feel bad for a bit, than to feel nothing.

II.

Young adults should probably put some effort into becoming less emotionally reactive. Being volatile makes you unpleasant to be around, and undercuts your ability to achieve pretty much any goals you may have.

If you have any traumas, it's likely positive-EV for you to devote time and energy to learning some kind of therapy modality with a good evidence base, and then taking the time to resolve those issues.

In my opinion - for most people - once you have fixed about 60% of your emotional reactivity and 90% of your psychological triggers, you have hit a point of diminishing returns. In fact, past that point, I think further investment in making yourself "nonreactive" and "unattached," and removing all minor triggers from your psyche, is pathological from the perspective of actually trying to be happy and to do things with your life.

If your fear of feeling bad for a bit is so great that it causes you to self-obsess, consider that self-obsession is just a more convoluted and drawn-out way of feeling bad.

III.

If you try to do something, and things don't go your way, and you feel bad about that, the problem isn't that you felt bad about it. The problem isn't that you are insufficiently stoic. The problem isn't that you care too much about the outcome.

The problem ... is that things didn't ...
G.D. as Capitalist Evolution, and the claim for humanity's (temporary) upper hand
by Martin Vlach — 2025-05-10T21:18:50.665Z

1. The Spark of G.D.

@David Duvenaud's recent Guardian op‑ed on gradual disempowerment [1] sketches an original foresight: it frames the AI‑alignment problem not as a single catastrophic explosion but as a slow erosion of human leverage. If intelligence is the ability to steer the future, then the central question is: who (or what) is doing the steering while the asphalt is still wet?

Duvenaud’s (motion )picture is compelling precisely because it feels so mundane. We will not wake up to a Skynet banner; we will simply notice, one product launch at a time, that fewer meaningful knobs are within human reach.

2. Capitalism as Cultural Evolution

Many readers immediately point to capitalism itself as the culprit—a cultural heirloom of biological evolution that relentlessly optimizes for profit signals ( and plausibly for any proxy variable that can be operationalized). In that reading, AI is merely the newest accelerant poured onto an ancient feedback loop of mutate‑select‑replicate applied to business plans.

> Thesis‑A: If disempowerment is baked into the evolutionary heuristic of capitalism, the destiny of “ordinary humans” is at best to watch the optimization play out, and at worst to be optimized away.

This framing is seductive, but maybe too personifying. Evolution has no preferences, only selection pressures; capitalism has no will, only markets and shareholders. We risk smuggling agency into faceless processes whenever we say “capitalism wants X” or “evolution wants Y”.

4. Why Bother Remaining Relevant?

A sharp objection arises: If evolution’s invisible hand “prefers” that highly optimized AIs eclipse human agency, by what title/right do we insist on staying in charge? If human prosperity post-ASI is still prosperity, even by purely evolutionary metrics such as population size or genetic diversity, why cling to control?

My answer invokes the Lindy Effect: systems that survive longest have statistically higher odds of surviving longer still. Human civilizational scaffolding—norms, law, coordination technologies—has a track record stretching thousands of years. It may be messy, but it is Lindy‑approved. An abrupt pivot to opaque optimization engines could erase this hard‑won robustness long before we notice the regress. Survival advantage, in short, accrues to whatever preserves proven structures while adding capabilities at the margi...
Book Review: "Encounters with Einstein" by Heisenberg
by Baram Sosis — 2025-05-10T20:55:40.288Z

The title of this book is clickbait. "Encounters with Einstein" is a short collection of lectures given or written by Werner Heisenberg in the 1970's, only one of which discusses Einstein.[1] The remaining lectures discuss various aspects of the history and development of quantum mechanics and science in general: tradition and concepts in science; Heisenberg's time in Göttingen, where he, Born and Jordan developed matrix mechanics; a history of cosmic ray science, and more.

Concepts and Quarks

One thread comes up repeatedly in the lectures: the role of concepts in science, and in particular, the concept of an elementary particle. (You get the sense that he would harp on this point to whoever would listen.) Much progress in science has come from asking what things consist of, and whether they can be broken down into smaller pieces: molecules, atoms, then electrons and protons in the early 20th century. It was believed that these elementary particles were truly immutable, that their number was always constant. This proved incorrect, though. The discoveries of matter-antimatter pair production and nuclear decay showed that while mass-energy, momentum, charge, etc. might be conserved, the number of particles was not. Cosmic ray events, in which the collision of two highly energetic particles releases a shower of secondary particles through the transformation of energy into matter, also demonstrated this conclusively. Heisenberg argues that it thus makes little sense to talk of "dividing" particles into their constituent parts any longer:

A proton, for example, could be made up of neutron and pion, or Λ-hyperon and kaon, or out of two nucleons and an anti-nucleon; it would be simplest of all to say that a proton just consists of continuous matter, and all these statements are equally correct or equally false. The difference between elementary and composite particles has thus basically disappeared. (p. 73)

The pervasive Democritean idea that everything can be decomposed into some indivisible, immutable "atoms" must, he claims, be rejected: "good physics is unconsciously being spoiled by bad philosophy." How should we instead conceptualize matter? He prefers to describe it as a "spectrum," analogous to the spectra of atoms or molecules. The different particles that we observe correspond to particular stationary states, with transformations between them deter...
Where is the YIMBY movement for healthcare?
by jasoncrawford — 2025-05-10T20:36:47.415Z
Link: https://newsletter.rootsofprogress.org/p/where-is-the-yimby-movement-for-healthcare

In the progress movement, some cause areas are about technical breakthroughs, such as fusion power or a cure for aging. In other areas, the problems are not technical, but social. Housing, for instance, is technologically a solved problem. We know how to build houses, but housing is blocked by law and activism.

The YIMBY movement is now well established and gaining momentum in the fight against the regulations and culture that hold back housing. More broadly, similar forces hold back building all kinds of things, including power lines, transit, and other infrastructure. The same spirit that animates YIMBY, and some of the same community of writers and activists, has also been pushing to reform regulation such as NEPA.

Healthcare has both types of problems. We need breakthroughs in science and technology to beat cancer, heart disease, neurodegenerative diseases, and aging. But also, healthcare (in the US at least) is far more expensive and less effective than it should be.

I am no expert, but I am struck that:

The doctor-patient relationship has been disintermediated by not one but two parties: insurers and employers.
It is not a fee-for-service relationship. The price system in medicine has been mangled beyond recognition. Patients are not told prices; doctors avoid, even disdain, any discussion of prices; and the prices make no rational sense even if and when you do discover them. This destroys all ability to make rational economic choices about healthcare.
Patients often switch insurers, meaning that no insurer has an interest in the patient's long-term health. This is a disaster in a world where most health issues build up slowly over decades and many of them are affected by lifestyle choices.
Insurers are highly regulated in what types of plans they can offer and in what they can and cannot cover. There's no real room for insurer creativity or consumer choice, or for either party to exercise judgment.
A lot of money is spent at end of life, with little gained by in many cases except a few years or months (if that) of a painful, bedridden existence.

Just to name a few.

Bill Gurley wrote in 2017

that “we have the worst of both worlds … the illusion of a free market and the illusion of regulated market with the apparent benefit of neither.”

John Arnold said

more recently that health care is “not a fair and open market” and that it has basically every market failure. ...