trentmkelly/LessWrong-43k huggingface dataset:
Galaxy-Brain Hobo Antibiotics?
by Lorec — 2025-06-16T12:43:19.226Z

[ edit: I have gotten an ear infection diagnosis and a prescription for antibiotics as of 26 June 2025, [ see comment ] ]
I have what I think is a chronic inner ear infection. Since March of 2021, I've had subjective obstructive Eustachian tube dysfunction in my right ear, as well as oculomotor problems that as far as I can tell must be caused by some kind of inflammation in my semicircular canals. [ The vestibular problem was confirmed by a videonystagmography, but I moved cities before I could follow up with the ENT who ordered the test. ]

[ Best photo I have left of my VNG results, from June of 2024. The red scatterplots show my labyrinths' response to warm water, blue cold. The purple highlights [ drawn by me after the tester's description ] show where my vestibular heat reflex should be; the red parts of the scatterplot show that at the time my left oculomotor reflex was OK but my right was deficient. ]

[ illustration of how the 2 oculomotor reflexes of each eye rely differentially on input from the semicircular canals of each ear, which can be tested with a caloric reflex test [ part of my VNG ] -- the rapid nystagmus to the right side is simulated by application of warm water to the right eardrum, while the slow nystagmus to the left side is simulated by application of cold water to the right eardrum ]
In July of 2024 the problem spread to my left ear. I've gone to about 8 medical professionals about it, and they've consistently prescribed Flonase/Nasacort and otherwise refused to treat it.

The Flonase/Nasacort does anything [ Nasacort works better ], but since March of 2025 [ when the dizziness had gotten really bad and I decided I'd Try Anything ] I've found that eating 3-4 cloves of strong garlic a day, plus Nasacort, works vastly better than that.

Since garlic is supposed to be an antibiotic as well as an anti-inflammatory agent, the garlic working so well [ plus reading up on the common causes of ETD that has my symptom profile ] made me suspect it was an infection. I tried ordering ofloxacin from TelyRx and it seemed to help a little but didn't end up curing the problem, and it rebounded when I ran out of ofloxacin. I'd been worried this would happen. Unfortunately ofloxacin is centrally for when your eardrums are perforated, so the solution can actually penetrate into the inner ear cavity. My eardrums aren't perforated, and actually look normal...
The EU commission seeks expert advisers on AI
by PabloAMC — 2025-06-16T12:28:06.533Z

The EU Commission has opened a call for expressions of interest for researchers who would like to advise the EU AI office on the implementation of the AI act, concerning among other topics those related to the safety of general purpose AI systems. There’s a requirement to either hold a PhD or equivalent experience . On the other hand, there’s no need to be an EU national, though >80% will be.
Take a look an apply here: https://digital-strategy.ec.europa.eu/en/news/commission-seeks-experts-ai-scientific-panel
From Paperclips to Bombs: The Evolution of AI Risk Discourse on LessWrong

by David Harket — 2025-06-16T05:16:46.610Z

Overview of the paper

In this paper, we set out to empirically map the evolution of the AI risk discourse on LessWrong, specifically in response to the public releases of ChatGPT and GPT-4. These events represented a significant shift, moving advanced AI from a theoretical concern to a tangible, publicly accessible technology. Our central research question was: How is the AI risk discourse on this forum constructed, and how did its thematic composition change during this pivotal period?

To answer this, we conducted a two-phase analysis of 884 posts published under the "ai-risk" and "alignment" tags in the year before and after ChatGPT's release.
First, through a qualitative reading of the posts, we identified two primary, coexisting framings of risk. We termed the first "abstract AI risk," which encompasses the rationalist-philosophical discourse on future, decisive threats, such as those from an unaligned superintelligence or instrumental convergence—the classic "paperclips" scenario. The second we termed "tangible AI risk," which uses a more empirical-scientific style to discuss the immediate, accumulative threats posed by the misuse of current systems, such as bad actors creating malware or "bombs."

In the second phase, we used these qualitative findings to develop and validate a computational text classifier. This tool allowed us to analyse the entire dataset and quantitatively measure the prevalence of each risk framing over time.
Our analysis revealed a statistically significant shift in the community's focus. Following the release of ChatGPT, the discourse reoriented robustly toward tangible risks. The release of GPT-4 continued this trend, albeit with a smaller effect. Crucially, our analysis of author activity shows that this was not the result of a community fragmenting into ideological camps. Instead, most authors engage across the full spectrum of risk, indicating a shared discourse that collectively rebalanced its attention.

In essence, this research provides a data-driven account of how the LessWrong community responded to concrete technological progress. The forum did not abandon its foundational concern with long-term existential risk, but pragmatically shifted its focus from what is theoretically possible toward what is demonstrably real.

Introduction

The rapid development of artificial intelligence (AI) has made the technology a central topic of pu...
Donutting is bad
by Jarrah — 2025-06-16T04:12:05.967Z

TL;DR pranking unlocked computers undermines security by providing cover for real breaches and creating a culture of shame that discourages open reporting of security issues.

It's a common rule in companies that employees must lock their device when it is unattended, to prevent people from using your access in unauthorised ways. Screen locking is a common compliance requirement, and a good security practice.

People new to these company environments can take a while to learn the locking behaviour. It's not an intuitive reaction. There was no ancestral selection process. Most people don't take that level of security precautions with their personal laptop. Seasoned people sometimes forget.

Doughnutting is the practice of seeing that a colleague isn't at their computer and has left it unlocked, then seizing the opportunity to use their device. The classic procedure is to use the internal communication systems to announce a promise to buy doughnuts for the office, but there are similar pranks such as displaying the Windows 98 update screen or reversing the mouse scroll direction. These pranks are sometimes celebrated by security practitioners as a fun way to teach security hygiene.

I'm not claiming doughnutting fails to make people lock their devices. Shame and peer accountability can be a powerful motivator for people to learn
behaviours. But there are hidden costs that I believe make it detrimental overall.

Doughnutting gives cover to unauthorised access, the very risk you were trying to address! Imagine catching someone nosing around in someone else's emails - "I was just doughnutting them" gives plausible cover to an actual security breach.

Creating an environment where people are publicly flagged for making security mistakes is a bad idea. You want to hear about when people suspect they have been socially engineered or accidentally emailed a sensitive document, but admitting these things is incredibly vulnerable. You want a culture where people can openly talk about security for other reasons, such as people wondering aloud whether a thing they're already doing is secure, or reporting weird events from a colleague's account.

My recommendation is to treat helping people learn to lock their devices as you would handle giving any other piece of feedback. Giving feedback is a big topic on its own[1] and depends on the recipient and your relationship with the recipient....
Futarchy using a sealed-bid auction to avoid liquidity problems
by Christopher King — 2025-06-16T01:34:30.823Z

Futarchy is usually formulated using multiple continuously running markets, which raises questions about how to introduce liquidity, when to introduce it, and who will do so. Robin Hanson (the inventor of futarchy) recently proposed how to handle some of these details, but they seemed to me a bit inelegant. I instead propose reformulating it to use a sealed-bid auction with no liquidity added. I will only be covering the joint-stock company version of futarchy, not the government policy version, which I'm not sure how my proposal would generalize to. The joint-stock company version is relevant to effective altruism as a possible component of a market for altruism.

Consider a hypothetical joint-stock company named The ACME Corporation with one million shares.

Proposals and bids

Once a month, the public can submit proposals. A proposal can either be:

A CEO replacement: replaces the current CEO with the proposer, under terms of a legal contract included with the proposal. If the proposal takes place, this is immediately effective and legally binding. The current CEO is considered fired. (Given that they are only guaranteed to have their job for a month, most candidates will include a decent severance package as part of their compensation.)
A company directive: instructions that employees should follow. It is considered company policy to follow these. If these are consistently ignored, future proposals should propose replacing the CEO with one who will enforce them.

For example, let us consider proposals A, B, C, and "Change Nothing".

The next step is that people submit sealed bids. There are two types: buy bids and sell bids.

Buy bids

Any member of the public (including current investors if they wish to increase their investment) can submit a buy bid, conditioned on a given proposal passing. The bid contains a maximum price and a number of shares.

Note that first they must put money in escrow. They can submit multiple bids. For any given proposal, the total amount they bid on that proposal must not exceed the amount in escrow. However, the total amount of bids across different proposals is unbounded, since only one proposal can pass. So if a buyer has $10 in escrow, they could bid up to $10 on A, and up to $10 on B, and up to $10 on C, and up to $10 on "Change Nothing".

Sell bids

Each current investor can, for each proposal that they dislike, submit a sell bid. The bid con...
Memory Decoding Journal Club: Neocortical synaptic engrams for remote contextual memories
by Devin Ward — 2025-06-15T23:22:24.178Z

Join Us for the Memory Decoding Journal Club!

A collaboration of the Carboncopies Foundation and BPF Aspirational Neuroscience

This time, we’re diving into a groundbreaking paper:
"Neocortical synaptic engrams for remote contextual memories"

Authors: Ji-Hye Lee, Woong Bin Kim, Eui Ho Park & Jun-Hyeong Cho

Institutions: University of California, Riverside, Department of Molecular Cell and Systems Biology.

Presented by: Dr. Randal Koene

When? June 17th, 2025 – 3:00 PM PDT | 6:00 PM EDT | 10:00 PM UTC

Where? Video conference:

https://carboncopies.org/aspirational-neuroscience

Register for updates:

https://aspirationalneuroscience.org/register-with-us/

Once registered, you'll receive event invites & updates!

#Neuroscience #MemoryResearch #Amygdala #JournalClub #BrainScience #Carboncopies #AspirationalNeuroscience
Every Major LLM Endorses Newcomb One-Boxing
by jackmastermind — 2025-06-15T20:44:06.276Z
Link: https://jacktlab.substack.com/p/every-major-llm-endorses-newcomb

I've been doing a series of posts on my substack about Functional Decision Theory as I work on addressing flaws and criticisms. Part of what persuaded me to work on these problems was the discovery that every single LLM I tested chooses one-boxing over two-boxing, though none of the LLMs cited FDT or UDT in their responses.
FDT Does Not Endorse Itself in Asymmetric Games
by jackmastermind — 2025-06-15T20:44:04.985Z

A twin guard-inmate dilemma (twin GID) is an asymmetric game that breaks FDT. [Image: GPT Image-1]

0. Introduction

TL;DR: FDT and UDT diverge in how they handle "behave as you would have ideally precommitted to behaving" in asymmetric games where a player is assigned a role after a deterministic clone is made. FDT updates, whereas UDT does not. ∴ an agent who knows in advance that they will enter one of these games would convert to UDT, not FDT, on this problem. [UPDATE: this applies to the formulation of FDT in the paper, but not necessarily to Yudkowsky & Soares' "preferred" version of FDT; see Menotim's comment]

I wrote a version of this post on my substack; it was for a less technical audience, and at the time I didn't understand updateless decision theory. I assumed that UDT and FDT just used different methods to compute the same recommendations. I was wrong! In fact, there are very simple scenarios in which FDT does not recommend precommitting to itself.

1. Definitions

According to Yudkowsky & Soares' "Functional Decision Theory: A New Theory of Instrumental Rationality," FDT, CDT, and EDT all maximize expected utility as defined by this formula:

EU(a):=N∑j=1P(a↪oj;x)⋅U(oj)

where o1,o2,o3 . . . are the possible outcomes from some countable set O; a is an action from some finite set A; x is an observation history from some countable set X ; P(a↪oj;x) is the probability that oj will obtain in the hypothetical scenario where the action a is executed after receiving observations x; and U is a real-valued utility function bounded in such a way that [the above equation] is always finite.

…

From this perspective, the three decision theories differ only in two ways: how they prescribe representing [the world-model] M, and how they prescribe constructing hypotheticals Ma↪ from M. (emphasis mine)

From here, the three decision theories are formalized by:

EDT(P,x):=argmaxa∈AE(V|Obs = x,Act = a)CDT(P,G,x):=argmaxa∈AE(V|do(Act = a),Obs = x)FDT(P,G,x):=argmaxa∈AE(V|do(FDT(P––,G––,x––)=a))

where V is a variable representing U(Outcome), G is a Pearl-style digraph (of causal relations for CDT, subjunctive relations for FDT), and FDT(P––,G––,x––) is notation for a variable representing the output...
Can We Change the Goals of a Toy RL Agent?
by tuphs — 2025-06-15T20:34:01.558Z

This post is a write-up of preliminary research in which I investigated whether we could intervene upon goals in a toy RL agent. Whilst I was unsuccessful in locating and retargeting a goal-directed reasoning capability, we found evidence of partially-retargetable goal-specific reflexes.

Produced as part of the ML Alignment & Theory Scholars Program 7.0 cohort.

1 - Introduction

Inspired by “retargeting the search” (Wentworth, 2022), we investigated a toy microcosm of the problem of retargeting the search of an advanced agent. Specifically, we investigated (1) whether or not we could locate information pertaining to “goals” in a small RL agent operating in a toy, open-ended environment, and (2) whether we could intervene on this agent to cause it to pursue alternate goals. In this blog post, I detail the findings of this investigation.

Overall, we interpret our results to indicate that the agent we study possesses (at least partially) retargetable goal-conditioned reflexes, but that it does not possess any form of re-targetable, goal-oriented long-horizon reasoning.

The rest of this post proceeds as follows:

Section 2 briefly outlines past work that has interpreted reasoning in RL.
Section 3 outlines the agent (a 5M parameter transformer) and environment (Craftax, a 2D Minecraft-like environment) we study.
Section 4 explains one approach we tried: re-targeting the agent by using probes to find representations of instrumental goals. We had little success with this approach.
Section 5 details another approach we tried: intervening on the agent’s weights. We found that sparse fine-tuning localises small subsets of parameters that determine which rewards the agent maximises.
Finally, Section 6 summarises our work.

2 - Related Work

The most directly relevant work is a paper by Mini et al. (2023) that investigates a maze-solving agent. They find evidence of goal misgeneralisation being a consequence of an agent internally representing a feature that is imperfectly correlated with the true environment goal, and find that intervening on that representation can alter the agent’s behaviour.

Also related is work by Taufeeque et al. (2024, 2025) and Bush et al. (2025) in which a model-free RL agent is mechanistically analysed. This agent is found to internally implement a complex form of long-horizon planning - bidirectional search - though the environmen...
Some reprogenetics-related projects you could help with 
by TsviBT — 2025-06-15T20:25:14.900Z

PDF version. berkeleygenomics.org. x.com. bluesky.

This is a short miscellaneous list of projects that I think would help accelerate germline engineering. This isn't prioritized or comprehensive or anything—it's not the most important projects, but rather just some projects that have occurred to me. Happy to chat with anyone interested.

Project headlines:

Deregulation suggestions (law and policy).

Iterated selection scheduling (math/CS problem).

Can genomic vectoring have large effects? (bioinformatics/genetics)

Power of recombinant chromosome selection (math/CS).

Understanding public interest in reprogenetics.

Understanding the regulatory landscape around reprogenetics.

Educating the public about reprogenetics.

Project details:

At the moment, the US government is calling for deregulation suggestions: https://www.regulations.gov/deregulation. If there's someone who understands how the US Code of Federal Regulations works, and would be up for making a couple submissions, one or two of the policy recommendations here, e.g. CITES treaty and Dickey-Wicker, might be doable: https://berkeleygenomics.org/articles/Policy_recommendations_regarding_reproductive_technology.html

Iterated selection scheduling problem.

There's a set of potential methods for strong genomic vectoring that involve combining cells and then having them divide, to alternate between haploid/diploid, or between diploid/tetraploid. That's iterated embryo selection, iterated meiotic selection, and poor man's chromosome selection.

There's a difficult math/compsci problem here: how do you actually schedule/select which cell lines to combine, divide, culture, preserve or discard, and sequence/genotype?? It's very complicated. It probably would have to be answered with some big search / machine learning / RL thing. Could be a fun compsci project! I think it should be quite amenable to such methods.

I've written a bit about the math here: https://berkeleygenomics.org/articles/Methods_for_strong_human_germline_engineering.html#the-cost-of-poor-mans-chromosome-selection, and I did some preliminary simulations a long while ago, and I'm happy to discuss if you're interested.

Can genomic vectoring have large effects?

Many scientists say they are very skeptical that we understand enough about genes for traits to have much effect, even if we could strongly alter the genome. On the other hand, naive ex

...
Risk Tokens: Economic Security in AI Safety
by mhdempsey — 2025-06-15T19:25:20.936Z
Link: https://www.michaeldempsey.me/blog/2025/06/02/risk-tokens/

As intelligence and safety research continue to progress, I’ve been thinking more and more about how to create potential market dynamics that help with alignment and safer usage of AI. This feels especially important as we likely face cat and mouse games with frontier models pushing performance first and alignment/red teaming second, along with open source continuing to keep up (on a 3-9 month lag) with frontier models.

The traditional approach to AI safety has largely operated through the paradigm of technical constraints and social responsibility; a framework that, while noble in intention, often positions safety as friction against the relentless momentum of capability advancement. This has of course led to signals of alignment researchers concentrating more at certain labs, as others implicitly have voted slightly against alignment/safety with their dollars/compute allocations.

Safety at Scale: A Comprehensive Survey of Large Model Safety

While the pace of research in AI safety continues, there have not been many approaches that tie together economics alongside safety breakthroughs. With this in mind I would like to bring forward our concept of Risk Tokens. Risk Tokens are effectively the classification of inference from AI models that are particularly risky to the world, paired with dynamic economic pricing that makes dangerous use cases naturally self-limiting through market mechanisms.

The inspiration for Risk Tokens comes both from the research we’ve done at Compound around biosecurity, anchored by the popular idea of frontier models potentially posing breakaway risks in bioterrorism or bioweapon synthesis, as well as from the concepts of crypto economics.

The biosecurity domain offers a particularly salient parallel, where biotech’s increasing power and democratization present a dual-use dilemma. Just as the CDC and USDA maintain tiered access controls for select biological materials, access to potentially dangerous AI capabilities could involve similar economic and procedural friction. It’s likely this will only cascade further as we see more decentralization of lab work through cloud labs, and a lowered barrier to science broadly through frontier models (and even tacit knowledge that can be learned through youtube.)

Crypto offers perhaps a more elegant mechanism and parallel. In Bitcoin’s design, network security emerges from making attacks economically...