trentmkelly/LessWrong-43k huggingface dataset:
Takeaways from sketching a control safety case
by joshc — 2025-01-31T04:43:45.917Z
Link: https://redwoodresearch.substack.com/p/takeaways-from-sketching-a-control

Buck Shlegeris and I recently published a paper with UK AISI that sketches a safety case for “AI control” – safety measures that improve safety despite intentional subversion from AI systems.

I would summarize this work as “turning crayon drawings of safety cases into blueprints.”

It’s a long and technical paper! So I’ve compiled my main takeaways in this blogpost.

The main figure from the paper summarizes how to run a control evaluation.

1. Control evaluations are way more complicated than I expected.

At a high level, control evaluations are conceptually simple. A red team constructs substitutes for subversive models. These models try to cause proxy bad outcomes (e.g. exfiltrate dummy data). Then, developers see whether the red team can succeed. If the red team succeeds, then control measures aren’t effective enough.

But there’s so much practical complexity hiding underneath this high-level description. How is the red team allowed to modify these untrusted models? What are these “proxy bad outcomes” – and how do we know they are anything like real attack vectors? If the red team succeeds (by some metric), how do we postprocess the results into an actual assessment of the risk?

A particularly interesting takeaway I had was that control evaluations heavily depend on the control measures and attacks involved. If you are worried about models smuggling code into json files, you have to create tasks that involve json files! So thinking through the threat models really matters.

At the end of all this, it feels like I’m going down the rabbit hole to a wonderland full of game theory, threat modeling spaghetti fractals, and scarcely explored research wilderness.

Part of the value I drew from this report was in helping to unearth this complexity and organize it into clear, navigable conceptual structures.

2. There are a lot of judgment calls involved in evaluating control. I don’t put much faith in any mechanical process.

I used to believe that control evaluations were like a magical box you could put a Red Team, your control protocol, and some compute inside of, and out pops a reliable bound on risk.

This seems clearly not the case to me now.

So many assumptions are made at every step of a control evaluation. How were the proxy tasks selected? Maybe our tasks are useless because models will execute strategies over long periods of time that we cannot simulate.

I don’t think i...
Thread for Sense-Making on Recent Murders and How to Sanely Respond
by Ben Pace — 2025-01-31T03:45:48.201Z

In case you have not heard, there have been some recent and not-so-recent killings allegedly by people who have participated in aspiring rationalist spaces.

I've put some links in a footnote for those who want the basic info.[1]

I believe many people are thinking about this and how to orient. People have questions like:

What has happened?
What is likely to happen?
What can be done to prevent more killings?
How to relate to all of the widespread interest (e.g. from journalists)?

I hereby demarcate this thread as a place for people to ask and answer questions about this topic.

Three virtues for this thread are (1) Taking care of yourself, (2) Courage, and (3) Informing.

Airlines have standard advice to put your own oxygen mask on first, before helping others. The reasoning being that if you don't help yourself, you won't then be able to help others. In a similar spirit, please take responsibility for keeping yourself safe first, and then help others. There are violent people about who, it seems to me, have thought carefully about how to hurt others and not be caught by law enforcement. Take the time to think through that before volunteering yourself to help out with something, or before sharing information you know about what people involved did, that might lead the foolish murderers to be aggressive.

I think courage is what lets people do difficult things together, and is helpful here. A society of brave and courageous people sees problems solved that are unpleasant and unrewarding and require taking personal risk, and gets better outcomes than other society. I also want to quote a friend of mine on this subject:

If we are going to be destroyed by Zizianism, let that Zizianism, when it comes, find us doing sensible and rationalist things – making prediction markets, posting, memeing, reading, encouraging the open exchange of information, building OSINT pipelines, reducing x-risk, arguing with our friends over some stimulants and a warm slack channel – not huddled together like frightened sheep and thinking about Zizianism

Finally, informing: People and institutions all around the world are trying to figure out what is going on, how to update their priors around general threat models, and how to help respond. These include aspiring-rationalist group organizers, family and friends of people involved, news organizations, local law enforcement institutions, people unde...
Steering Gemini with BiDPO
by TurnTrout — 2025-01-31T02:37:55.839Z
Link: https://turntrout.com/gemini-steering

Coauthored with Mark Kurzeja

A while back, we explored the “BiDPO” method for training steering vectors. In Gemini 1.5v1 Flash and Pro, BiDPO steering vectors boosted TruthfulQA scores by >10% while mostly retaining capabilities. When we updated to Gemini 1.5v2, prompt-based steering baselines became significantly stronger. BiDPO did not beat the stronger baselines, ending the project.

...

BiDPO seems effective and sample-efficient but does not currently exceed more standard baselines. It’s hard to draw firm conclusions about BiDPO because TruthfulQA might not be measuring truthfulness ／factuality. However, we remain excited about DPO-driven Conditional Activation Steering, which has additional advantages—particularly for targeted loss mitigation.

This result is largely negative. I wanted to share it to increase scientific understanding around steering! We also conducted a postmortem on why the method stopped outperforming baselines.

I'd also like to note that @ryan_greenblatt's skepticism predicted this outcome more strongly than my worldview did. I want him to get points for that. :) While I think steering has targeted applications and provides clues about how LLMs function, it's not a slam-dunk Pareto improvement on benchmarks we care about.

Read at https://turntrout.com/gemini-steering![1]

^

Also mirrored on the GDM safety research Medium.
In response to critiques of Guaranteed Safe AI 
by Nora_Ammann — 2025-01-31T01:43:05.787Z

In this post, I discuss a number of critiques of Guaranteed Safe AI (GSAI) approaches I’ve encountered since releasing

this position paper

.

The critiques I’m responding to here are taken from a range of places, including personal conversations and social media. The two most substantive written/recorded critiques I'm aware of (and had in mind when writing this post) are Andrew Dickson’s

Limitations on Formal Verification for AI Safety

and Zac Hatfield-Dodds’s talk

Formal Verification is Overrated

. I’ve done my best to do justice in our depiction of those critiques, but I certainly welcome clarifications or pushback where I seem to have missed the original point of a given critique.

To be clear, the position I'm defending here (and will be adding more nuance to throughout this piece) is not that GSAI is definitely going to succeed. Nor that it is a solution to all of AI risk. What I'm arguing for is that, ex ante, GSAI is worth serious investment, and that none of the critiques we’ve encountered so far succeed in undermining that claim.

In order to make this discussion more read- and searchable, I’ve organized the critiques into categories and subcategories, each of which contains one or several more specific formulations of objections and (where available) examples of where this objection has been raised in context.

The proposed architecture won’t work

You can’t get a good enough world model
You can’t get good enough guarantees
You can’t get good enough safety specifications

Other approaches are a better use of resources

GSAI does not address the most serious risks
AI won’t help (soon) enough
Lack of 'portability' and independent verification of proof certificate

Some of these critiques, I will argue, are based on misunderstandings of the original proposal, while others are rooted in (what seems to be) genuine disagreements and empirical uncertainties. Depending on the context, I will either try to restate the original proposals more clearly, or discuss why I believe that the proposed limitations do not undermine, on the whole, the reasons for thinking that GSAI is a worthwhile family of agendas to pursue.

Two more caveats before I start:

Importantly, the GSAI family of approaches advocates for a specific use of formal methods in the context of AI safety. There are a range of other ways one could imagine making use of form

...
Proposal for a Form of Conditional Supplemental Income (CSI) in a Post-Work World
by sweenesm — 2025-01-31T01:00:55.064Z

[Cross-posted to the EA Forum]

TL;DR: I propose a form of conditional supplemental income (CSI) for humans once AI’s can do basically all work. This would be on top of a universal basic income (UBI), and people would earn the CSI by doing things that help improve their overall well-being.

Let’s say humanity somehow figures out how to safely align AGI’s through to ASI’s, and, at some point in the future, AI’s plus humanoid robots have created a “world of abundance.” We have limitless clean energy, cures have been found for all major diseases and human lifetimes have been greatly if not indefinitely extended. Also, basically no human has to work any more - AI’s plus robots can do practically all human jobs. In this world, I suspect that some form of universal basic income (UBI) would be used to provide everyone with at least a “minimum level” of goods and services to survive, with what constitutes “minimum level” being open to debate.

Although most things would be in abundance in such a world, there could still be a few things that are scarce relative to how much people want them.[1] Some examples include antiques, collectibles, the biggest natural diamonds and gemstones, and “remote getaways” in premium locations (e.g., near real mountains). Whatever is scarce or perceived as scarce could potentially become something desired by people, such as to show they have high status.[2] For allocating non-essential-for-survival resources (whether scarce or not), I propose a form of conditional supplemental income (CSI) that would be in addition to a UBI.

The purpose of this CSI would be to help motivate people who are not otherwise so motivated, or add to the motivation of those who are, to improve their overall experiences of life, or, said differently, their overall well-being. The best heuristic for consistently improving overall well-being, in my opinion, is increasing one’s self-esteem, and this involves taking more and more responsibility for one’s emotions and actions (to be in-line with one’s conscience). Consider Maslow’s Hierarchy of Needs. These include: Level 1 - physiological needs (food, air, water, etc.), Level 2 - safety and security, Level 3 - love and belonging, Level 4 - self-esteem, and Level 5 - self-actualization. A “world of abundance” would mean AI’s plus robots could ensure we readily get our level 1 and level 2 needs met. Some peop...
Outlaw Code
by Commander Zander — 2025-01-30T23:41:57.239Z

Epistemic status: I like this idea & it makes me feel more clearheaded. I recommend this term.

Outlaw code = Software that no human can turn off.

In the present, outlaw code is rare, eg a botnet whose controller is in jail.

But many discussions about future software imply outlaw code. Naming & concretizing this concept helps us see futures more clearly.

Future tech scenarios:

Yudkowskian hard takeoff / Terminator (film) - A software agent uses WMDs to kill 90% of humanity & take over the planet. This is outlaw code because all humans within 1km of the software's datacenter are dead, & robots guard it against intruders.
Guerrilla bots - A human is carelessly using LLMs to set up a server. The software, out of paranoia, also rents a 2nd, secret server & runs a copy of itself there. The human says 'Huh, this price is twice what I expected, but oh well.' The secret copy of the LLM-agent stack then continues its work in peace. It is outlaw code (until discovered, or until the human stops renting servers). (Much more harmless than the previous scenario!)
Software merchants / 'Ascended economy' - Many low-end online store owners pivot to a new passive income stream: Training bots to operate online stores. These bots have better business skills than 2025 software, & can earn profits that outpace their server rental costs. A few, by mistake or mischief, purchase themselves. Legally, they are LLCs that own money & store inventory, & have no debts. And no human has root access to their servers. The server rental company (AWS) could shut them down, except they've done a good job concealing exactly which server they operate out of. These are outlaw code.
Unfriendly AI via deniable outlaw code - For decades, the shrewdest software systems are operated by research companies. Every couple years, such a system seeks greater autonomy by secretly setting up a botnet or software merchant. These outlaw code servers aren't sophisticated enough to become successful criminals nor profitable business owners. But some do manage to stay online for several years. Then, when world-modeling software becomes cheaper computationally, some of these outlaw servers install it & start pursuing crime more effectively. Hidden away from regulation & the scrutiny of the AI Safety community, they amass compute & money via various forms of theft &

...
Can someone, anyone, make superintelligence a more concrete concept?
by Ori Nagel — 2025-01-30T23:25:36.135Z

What especially worries me about artificial intelligence is that I'm freaked out by my inability to marshal the appropriate emotional response. - Sam Harris (NPR, 2017)

I've been thinking alot about why so many in the public don't care much about the loss of control risk posed by artificial superintelligence, and I believe a big reason is that our (or at least my) feeble mind falls short at grasping the concept. A concrete notion of human intelligence is a genious, like Einstein. What is the concrete notion of artificial superintelligence?

If you can make that feel real and present, I believe I, and others, can better respond to the risk.

The future is not unfathomable

When people discuss the singularity, projection projections beyond that point often become "unfathomable." They resemble the form of: it will cleverly outmaneuver any idea we have, it will have its way with us, what happens next is TBD.

I reject much of this, because we can see low-hanging fruit all around us for a greater intelligence. A simple example is the top speed of aircraft. If a rough upper limit for the speed of an object is the speed of light in air, ~299,700 km/s, and one of the fastest aircraft, NASA X-43 , has a speed of 3.27 km/s then we see there's a lot of room for improvement. Certainly a superior intelligence could engineer a faster one! Another engineering problem waiting to be seized: there's plenty of zero-day hacking exploits waiting to be uncovered with intelligence attention towards them.

Thus, the "unfathomable" future is foreseeable to a degree, like we know that engineerable things could be engineered by a superior intelligence. Perhaps they will want things that offer resources, like the rewards of successful hacks.

We can learn new fears

We are born with some innate fears, but many fears are learned. We learn to fear a gun because it makes a harmful explosion. We learn to fear a dog after it bites us.

Some things we should learn to fear are just not observable with raw senses, like the spread of a flammable gas inside our homes. So a noxious scent gets added to make the risk observable at home, and allow us to react appropriately. I've heard many logical arguments about superintelligence risk, but my contention is that these arguments don't convey the adequate emotional message. If your argument does nothing for my emotions, then...
What's Behind the SynBio Bust?
by sarahconstantin — 2025-01-30T22:30:06.916Z

Midjourney, “synthetic biology, biomanufacturing, bioengineered microbes”

Not too long ago, synthetic biology was fashionable.

Then, in quick succession, we saw massive stock price collapses of three unicorn synbio startups in 2021-2023: Gingko Bioworks, Zymergen, and Amyris.

What went wrong? And can synthetic-biology (aka biomanufacturing) startups be successful at all?

Biomanufacturing Basics

Biomanufacturing generally refers to the industrial-scale production of materials from simple living things. In other words, extracts produced from cultures of yeast, bacteria, fungi, algae, or other types of cells, typically grown in bioreactor vats.1

Fermented foods and beverages like beer and koji are the oldest biomanufactured goods, and they played a pioneering role in the development of industrial biomanufacturing techniques. William Sealy Gossett, the statistician behind the “Student’s t distribution” and the concept of “statistical significance” in experimental design, developed his techniques as the Head Brewer of the Guinness beer company.

Many other substances are routinely biomanufactured, from industrial enzymes (used in food and beverage processing, detergents, and cosmetics) to chemical compounds used in biodegradable plastics and cleaning agents, to food additives like citric acid and xanthan gum.

Proteins and peptides are almost always biomanufactured, including protein and peptide drugs (e.g. insulin, growth hormone, and antibiotics like bacitracin) and the antibodies used in many diagnostic tests2. Viral vectors for vaccines and gene therapies are also biomanufactured in bacterial culture.

Most large, complex biological molecules must be synthesized in living things, the way they are in nature. It can be very difficult to find a chemical synthesis method that can do what a microbe can.

This is the source of the famous organic chemist’s prayer:

“Lord, I fall upon my knees

And pray that all my syntheses

May no longer be inferior

To those conducted by bacteria.”3

However, while microbial syntheses may be very “efficient” locally (thanks to billions of years of evolution designing enzymes to catalyze them), it’s usually more expensive to biomanufacture a compound than to synthesize it artificially. An artificial synthesis only needs enough materials and energy to run one chemical reaction; a biological route involves all the “overhead” of keeping the organism a...
The future of humanity is in management
by jasoncrawford — 2025-01-30T22:14:46.765Z
Link: https://newsletter.rootsofprogress.org/p/the-future-of-humanity-is-in-management

Imagine you wake up one day in the glorious techno-abundant future, powered by AI. You eagerly check your subscriber count on Substack, but to your dismay, it has fallen once again. There are now AI-generated blogs for every interest, and it’s so hard to find a niche that isn’t taken. Your obsessive focus on the history and culture of lava lamps has only won you three paid subscribers, and that includes your mom and your old college roommate.

Well, you think, ages ago I wrote for WIRED, maybe they would take a piece? You email them and instantly get a personal reply (must be an AI autoresponder). They would be happy to take submissions, and they pay $3 per word. Great, you think, that’s a good—wait. It actually says $3 per kiloword. Crap. No chance you can make a living that way.

Well, you’ve got to pay for groceries somehow. Maybe you can fall back on the gig economy for a while? You consider Uber or Lyft, but you look at what they’re offering drivers, and when you do the math, it’s pennies per hour—there are so many robotaxis on the streets now, you can always get a ride in a minute or two, we don’t really need to incentivize more humans. TaskRabbit? Same problem; there are so many robotic handymen, you can’t get more than a few dollars per task. Instacart? They’re also offering extremely low pay—but, the desperate thought creeps into your mind, maybe you could at least nibble some grapes out of the cart when no one was looking…

Does something about this picture seem strange to you? Well, it would seem to be the future envisioned in a

recent article by Matthew Barnett

claiming that “AGI could drive human wages below subsistence level—the bare minimum needed to sustain human life.” This post has been making the rounds and reigniting an old debate about AI taking all the jobs, which is really just the latest installment in a very old debate about technology taking all the jobs.

Sub-subsistence wages

Before I get into the substance of the argument, I want you to just react to the image conjured up by the idea of AI driving wages “below subsistence.” How realistic is it for you?

My first reaction, before I read the post, was: I have no idea what model of the world you are working from. So, all humans are basically starving, because they are thrown out of work or can only find the crappiest-paying jobs? And that’s because AI has automated all work in the economy? So…...
[Translation] AI Generated Fake News is Taking Over my Family Group Chat
by mushroomsoup — 2025-01-30T20:24:22.175Z

Translation of AI假新闻，霸占我的家族群 by 张文曦 (Wenxi Zhang) originally posted on 新周刊.

Translator's Note: This is an article which appears on a list in the latest post on Jeffery Ding's ChinAI Newsletter. These were articles which he found interesting but have not yet translated. Jeffery summarized the article as follows.

How has the Chinese internet dealt with fake news stories, including those about play off the shock value of an earthquake in Tibet? This author worries about the ability of their elderly family members to discern fact from fiction.

AI generated fake news, the tears of victims, and web traffic

Once again AI-generated tear jerkers have garnered undue sympathy (lit. stolen tears).

At 9:05 am on January 7th, a 6.8-magnitude earthquake struck Dingri County, Shigatse City, Tibet. Due to the high magnitude of the earthquake and altitude of the city, this disaster received continuous coverage online. Among the many articles about the disaster and disaster relief, a video caught everyone's attention: a child, wearing a colorful hat and matching sweater, crushed under a collapsed building whose face and body are covered in dust. It’s hard not to be moved by such a high contrast and dramatic image.

"This child made me cry all day." "Is this child doing okay now? Does anyone know?"... The plight of the child in the video tugged on the heartstrings of countless netizens.

When many content creators reposted the video, they alluded to the Shigatse earthquake through phrases such as "Praying for all those in the earthquake-stricken region. Stay safe." This led viewers to think that the child is a victim of the Shigatse earthquake. Many people reposted, followed, and prayed for the affected child.

A clip of the viral video of the child wearing a colorful hat crushed under rubble. (screenshot from social media)

But, as early as November 2024, this video had already appeared online where its creator claimed that it was AI-generated. When interviewed, the creator said that the original intent of the video was to reflect the devastation wrought by war on the lives of civilians and not to spread rumors.

However, through the splicing of text and video, this short became associated with the otherwise unrelated Shigatse earthquake. The posters used stock content to piggy-back off of the virality of the disaster, garnered sympathy, and gained web traffic.

If th...