trentmkelly/LessWrong-43k huggingface dataset:
Updating the NAO Simulator
by jefftk — 2024-10-30T13:50:06.908Z

Cross-posted from my NAO
Notebook.

In April we released released
a tool to model
the efficacy of different approaches to stealth pathogen
identification. The tool's interface is pretty rough, which I'm not
super happy about, but there just aren't that many people in the world
who need to simulate the performance impact these design choices.

A month ago we published estimates
of RAi(1%) for influenza in municipal wastewater, and ended that
post with:

In response to this work we plan to update our metagenomic
biosurveillance simulator in two ways:

We'll switch the simulator's RAi(1%) from using the mean of the
distribution to sampling from the full posterior distribution. Because
our posteriors sometimes span several orders of magnitude, this change
should better capture our uncertainty.

We'll replace our preliminary influenza A RAi(1%) point estimate of
3.2e-8 with an option to choose each of the four above distributions,
with medians of 1.4e-8, 1.4e-8, 2.8e-9, and 7.0e-10.

Overall we expect these changes to make our projections higher
variance and somewhat less optimistic, but not to have a large impact
on whether this approach to novel pathogen detection is practical.

With Dan's help I've now made
both of these changes (#6,
#7,
#9),
and additionally:

Stopped assuming uniform coverage along the genome, and instead
use the distribution we've observed along SARS-CoV-2. (#5)
While it would be better to generate estimates from a wider range of
pathogens, everything else we've looked at in our samples has too much
genetic diversity for it to be easy to make these estimates. Now that
we've done some spike-ins,
however, I think it ought to be possible to use the highest
concentration spike-in sample to get second estimate, though I haven't
tried this. The overall effect of this change on our estimates should
be to increase variance while slightly lowering median performance.

Updated the pricing we see for the NovaSeq X 25B and switched
pricing to per-lane, based on the pricing we're seeing from sequencing
providers. (#10)
It's great that sequencing is continuing to get cheaper!

Let's compare what the two simulators say for one weekly NovaSeq X 25B run,
generating approximately 2e10 read pairs (SARS-CoV-2,
Flu A)

Note that lower is better here: the charts show the fraction of people
in the monitored sewer...
Occupational Licensing Roundup #1
by Zvi — 2024-10-30T11:00:04.516Z

We’re coming out firmly against it.

Our attitude:

The customer is always right. Yes, you should go ahead and fix your own damn pipes if you know how to do that, and ignore anyone who tries to tell you different. And if you don’t know how to do it, well, it’s at your own risk.

With notably rare exceptions, it should be the same for everything else.

I’ve been collecting these for a while. It’s time.

Campaign Talk

Harris-Walz platform includes a little occupational licensing reform, as a treat.

Universal Effects and Recognition

Ohio’s ‘universal licensing’ law has a big time innovation, which is that work experience outside the state actually exists and can be used to get a license (WSJ).

Occupational licensing decreases the number of Black men in licensed professions by up to 19%, and the number of Black women up to 22%. There is also the headline finding, which is that overall labor supply is reduced 17%-27%. Hopefully this can defeat some talking points or get through some of the damage being done.

Sununu in New Hampshire proposes universal license recognition for all licensed professions and removal of 34 licenses, elimination of 14 regulatory boards and almost 780 statutory provisions. A fine start.

Erica Jedynak: 26 states have joined the movement of significant occupational licensing reform – removing government barriers so more Americans can land their dream job. To all the allies part of the original OL coalition, thank you!

Knee Regulatory Research Center: Universal licensing recognition is a growing labor market reform for an easier transition of occupation licenses to new states. A new report from the Knee Center shows 26 states have passed universal recognition policies since 2013. Read more here.

This is really great, everyone needs to get on this to the maximum extent possible.

Libby Jamison: Lots of support for license portability! “About 83% of 848 registered voters surveyed nationwide said they support rules that allow the use of the licenses wherever military family members may move in the U.S.”

The obvious follow up is indeed ‘why only military’? License portability should be universal.

25% of Colorado jobs require a government license. They are at least opening up those jobs to some of the people with criminal records after three years via House Bill 1004. Better yet, we could scrap the requirements entirely.

Podcast on Utah’s approach to oc...
Three Notions of "Power"
by johnswentworth — 2024-10-30T06:10:08.326Z

We begin with three stories about three people.

First, Zhu Di, emperor of China from 1402 to 1424. In that period, it was traditional for foreign envoys to present gifts to the emperor and make a show of submission, reinforcing the emperor’s authority and China’s image as the center of civilization. Yet the emperor would send the envoys off with gifts in return, often worth more than the gifts the envoy had given - suggesting that the emperor’s authority and dominance did not actually translate into much bargaining power.

Second, Kevin Systrom, one of the founders of Instagram. When Instagram was bought by Facebook for $1B in 2012, it had only 13 employees. Systrom presumably found himself with a great deal of money, the most direct form of bargaining power amongst humans. Yet with only 13 employees, he commanded little of the obedience or authority displayed by an emperor.

Third, Benjamin Jesty. In 1774, a wave of smallpox raged through England, and dairy farmer Jesty was concerned for his wife and children. He knew that milkmaids sometimes contracted cowpox from cows, and that the (much milder) disease would immunize them for life against smallpox. So, he intentionally infected his wife and children with cowpox, 20 years before Edward Jenner popularized the same technique as the first vaccine. That same year, Louis XV, king of France, died of smallpox. Despite both great authority and great wealth, Louis XV had no power to stop smallpox.

These people demonstrate three quite different kinds of “power”.

Emperor of China: Dominance

The emperor of China has power in the form of dominance. In any competition of dominance within the vast Chinese empire, the emperor was the presumed winner. In some sense, the central function of an “emperor” is to create one giant dominance hierarchy over all the people in some territory, and then sit at the top of that dominance hierarchy.

(Much the same could be said of a “king” or “president”.)

Humans have a strong built-in instinct for dominance,

as do many other animals

. The ontology of a “dominance ranking” is justified experimentally, by the observation that various animals (including humans) form a consistent order of aggression/backing down - i.e. if Bob backs down in the face of Alice’s aggression, and Carol backs down in the face of Bob’s aggression, then empirically Carol will also back down in the face of Alice’s a...
Introduction to Choice set Misspecification in Reward Inference

by Rahul Chand — 2024-10-29T22:57:34.310Z

In classical RL, we have an agent with a set of States (S), a set of action (A), and given some reward function (R), the aim is to find out the optimal policy (pi) which maximizes the following. This is the cummulative rewards we get by sampling actions using our policy (here we assume discount factor is 1)

π∗=argmax(Eπ[ΣR(sₜ,aₜ,sₜ₊₁)])

The problem with this approach is that rewards do not always map to the correct set of actions or policies we want. For example, assume a game of football where I have specialized agents playing together, the reward for the defender is based on if the team concedes a goal or not, the agent could maximize this reward function by hoofing the ball out of play every time and thus ending up at a degenerate solution where it doesn't learn to play the sport. There are many such examples from existing literatures

Reward Hacking[1]: Assume you have a cleaning robot that gets a reward every time it cleans something, the robot figures out it can keep maximizing rewards by making a mess itself by pushing down objects and then cleaning them.[2]
Degenerate solutions: If I have a robot that needs to survive in a dangerous environemnt and gets a negative reward whenever it takes damage, it can figure out a way to keep its score at 0 is to just shut down at the start.
There are other examples like "Wireheading Problem" and the "Boat Racing problem"

Inverse Reinforcement Learning

To solve these issues rather than learning from some specified reward we can learn from demonstrations of desired behaviour and then figure out an appropriate reward function using them (this is knowns as Inverse-RL[3]). Therefore, in our above example about football, rather than having "don't concede goal = +10" as reward, we would have videos of a human playing as a defender and then we find out which reward function explains the demonstrations the best. The final reward using IRL might look like don't concede goal = +2, foul = -1, pass to midfielder = +0.5, pass back to keeper = -0.1 .... This reward function is more nuanced and captures our task better.

Since IRL depend on human demonstrations they suffer from the issue of human bias (humans are biased over some actions even if they are not optimal), human limitation (for ...
The Alignment Trap: AI Safety as Path to Power
by crispweed — 2024-10-29T15:21:26.545Z
Link: https://upcoder.com/22/the-alignment-trap-ai-safety-as-path-to-power/

Recent discussions about artificial intelligence safety have focused heavily on ensuring AI systems remain under human control. While this goal seems laudable on its surface, we should carefully examine whether some proposed safety measures could paradoxically enable rather than prevent dangerous concentrations of power.﻿

The Control Paradox

The fundamental tension lies in how we define "safety." Many current approaches to AI safety focus on making AI systems more controllable and aligned with human values. But this raises a critical question: controllable by whom, and aligned with whose values?

When we develop mechanisms to control AI systems, we are essentially creating tools that could be used by any sufficiently powerful entity - whether that's a government, corporation, or other organization. The very features that make an AI system "safe" in terms of human control could make it a more effective instrument of power consolidation.

Natural Limits on Human Power

Historical examples reveal how human nature itself acts as a brake on totalitarian control. Even the most powerful dictatorships have faced inherent limitations that AI-enhanced systems might easily overcome:

The Trust Problem: Stalin's paranoia about potential rivals wasn't irrational - it reflected the real difficulty of ensuring absolute loyalty from human subordinates. Every dictator faces this fundamental challenge: they can never be entirely certain of their underlings' true thoughts and loyalties.
Information Flow: The East German Stasi, despite maintaining one of history's most extensive surveillance networks, still relied on human informants who could be unreliable, make mistakes, or even switch allegiances. Human networks inherently leak and distort information.
Cognitive Limitations: Hitler's micromanagement of military operations often led to strategic blunders because no human can effectively process and control complex operations at scale. Human dictators must delegate, creating opportunities for resistance or inefficiency.
Administrative Friction: The Soviet Union's command economy faltered partly because human bureaucrats couldn't possibly process and respond to all necessary information quickly enough. Even the most efficient human organizations have inherent speed and coordination limits.

These natural checks on power could vanish in a human-AI power structure where AI systems provide perfe...
Housing Roundup #10
by Zvi — 2024-10-29T13:50:09.416Z

There’s more campaign talk about housing. The talk of needing more housing is highly welcome, as one prominent person after another (including Jerome Powell!) talking like a YIMBY.

A lot of the concrete proposals are of course terrible, but not all of them. I’ll start off covering all that along with everyone’s favorite awful policy, which is rent control, then the other proposals. Then I’ll cover other general happenings.

Table of Contents

Rent Control.

The Administration Has a Plan.

Trump Has a Plan.

Build More Houses Where People Want to Live.

Prices.

Average Value.

Zoning Rules.

Zoning Reveals Value.

High Rise.

“Historic Preservation”.

Speed Kills.

Procedure.

San Francisco.

California.

Seattle.

Philadelphia.

Boston.

New York City.

St. Paul.

Florida.

Michigan.

The UK.

Underutilization.

Get on the Bus.

Title Insurance.

Perspective.

Rent Control

Matt Yglesias reminds us rent control is not the answer, unless the question is how to destroy the housing stock of a city without resorting to aerial bombing. He also reminds us that Biden’s actual rent control proposal is in and of itself a nothingburger, but the act of proposing or enacting it carries expectations issues.

Matt Yglesias: The tired, old, neoliberal conventional wisdom is that rent control is a bad idea but if you delve into the most up-to-date research on the subject you’ll see that … yeah, it’s still a bad idea.

Iron Economist: The funny thing about rent control advocates is that their argument that it’s different this time always boils down to a version of ‘rent control can’t hurt supply because we already made supply illegal’ and it’s like yes, you have identified the problem there.

…

Unmanaged Reality (being unusually honest about the alternative perspective): yes, the problem is population growth well done!

Oh, it’s worse.

Rent control does not only impact future building, where if you already made all future building irreversibly illegal then I suppose you can’t do any more damage.

It also impacts the current supply.

What, you say? How could that be? Several ways.

Rent control directly means landlords will not maintain the current housing stock. It will be allowed to decay, or be actively sabotaged, eventually abandoned. Why invest if the price will remain fixed no matter what?

Rent control means that you withdraw housing from the rental market every chance you can get. If you rent out the proper

...
[Intuitive self-models] 7. Hearing Voices, and Other Hallucinations
by Steven Byrnes — 2024-10-29T13:36:16.325Z

7.1 Post summary / Table of contents

Part of the Intuitive Self-Models series.

The main thrust of this post is an opinionated discussion of the book The Origin of Consciousness in the Breakdown of the Bicameral Mind by Julian Jaynes (1976) in §7.4, and then some discussion of how hallucinations and delusions arise from schizophrenia and mania (§7.5), and from psychotic depression and BPD (§7.6).

But first—what exactly do hallucinations have to do with intuitive self-models? A whole lot, it turns out!

We’ve seen throughout this series that different states of consciousness can have overlapping mechanisms. Deep trance states (

Post 4

) mechanistically have a lot in common with Dissociative Identity Disorder (

Post 5

). Lucid trance states (

Post 4

) mechanistically have a lot in common with tulpas (

§4.4.1.2

), and tulpas in turn shade into the kinds of hallucinations that I’ll be discussing in this post. Indeed, if you’ve been paying close attention to the series so far, you might even be feeling some crumbling of the seemingly bedrock-deep wall separating everyday free will (where decisions seem to be caused by the self, or more specifically, the “homunculus” of

Post 3

) from trance (where decisions seem to be caused by a spirit or hypnotist). After all, an imagined spirit does not veridically (

§1.3.2

) correspond to anything objective in the real world—but then, neither does the homunculus (

§3.6

)!

I’ll extend that theme in Section 7.2 by arguing against a bright-line distinction between everyday inner speech and imagination (which seem to be caused by the homunculus) versus hallucinations (which seem to appear spontaneously, or perhaps to be caused by God, etc.). This is an important distinction “in the map”, but I’ll argue that it doesn’t always correspond to an important distinction “in the territory” (

§1.3.2

).

Section 7.3 then offers a framework, rooted in probabilistic inference (

§1.2

), for thinking about how culture and an individual’s psychology can each influence intuitive self-models in general, and hallucinations in particular.

…And that sets the stage for Section 7.4, where I talk about hearing voices as a cultural norm, including some discussion of Julian Jaynes’s The Origin of Consciousness in the Breakdown of the Bicameral Mind. I come down strongly against his suggestion that there were ever cultures full of “automatons” under “hallucinator...
Review: “The Case Against Reality”
by David Gross — 2024-10-29T13:13:29.643Z

This is not a red stop sign:

For one thing, in a ceci n'est pas une pipe way, it’s not a stop sign at all, but a digital representation of a photograph of a stop sign, made visible by a computer monitor or maybe a printer. More subtly, “red” is not a quality of the sign, but of the consciousness that perceives it.[1] Even if you were to generously define “red” to be a specific set of wavelengths of light that could in theory be ascertained without the benefit of consciousness, that still would be a measurement of something that the sign repels rather than is.[2]

We on some level understand these things, but we seem much more prone to think of ourselves as living in a world in which stop signs

are

red; the redness inheres in the sign out there in the world, rather than the redness being our own reaction to light reflecting from the sign, or rather than the sign and the redness being coincident private figments of perception. And no amount of thinking this through seems to dispel the illusion: it’s just too practically useful to put the color onto the external object to make it seem worthwhile to relearn how to perceive reality differently.

Donald Hoffman, in The Case Against Reality (2019), suggests that our mistake here goes much deeper. Not only are superficial sensory characteristics like color subjective qualities of conscious experience rather than objective qualities of things — but things themselves, as well as the spacetime they seem to inhabit, are too.

All of the stuff we take for granted as making up the world we inhabit — objects, dimensions, qualities, time, causality — are, says Hoffman, not objectively real. They are better thought of as the interface through which we interact with whatever might be objectively real. When we isolate “objects” in “space” and “time” and then ascribe “laws” to the regularities in their interactions, we are not discovering things about reality, but about this interface. “Physical objects in spacetime are simply our icons in the desktop.”

Hoffman compares this to the kind of thing that happens when you “drag” “a file” on your “desktop” to “the trash.” All of these things are meaningful and have practical effects in the interface, but cannot be taken seriously as descriptions of what actually happens under the hood. Similarly, Hoffman says, you need to take what you see in your “real world” interface seriously, but n...
A Poem Is All You Need: Jailbreaking ChatGPT, Meta & More
by Sharat Jacob Jacob — 2024-10-29T12:41:30.337Z

This project report was created in September 2024 as part of the BlueDot AI Safety Fundamentals Course, with the guidance of my facilitator, Alexandra Abbas. Work on this project originated as part of an ideation at a Apart Research hackathon.

This report dives into APIAYN (A Poem Is All You Need), a simple jailbreak that doesn’t employ direct deception or coded language. It is a variant of an approach known as ‘gradually escalating violation’, covered briefly in the Llama 3 paper, published by Meta. It also combines another type of jailbreak, involving the misspellings of restricted words.

The guide to implementing this jailbreak is mentioned below in its own section.

Introduction

I was messing around with the free Meta AI chatbot on WhatsApp, powered by Llama 3.1, testing out different ideas I had seen to jailbreak LLMs. But I wasn’t willing to put in the effort to use the more complex unintuitive ones I had seen online that used up a lot of deception / coded words / strange strings of symbols.

As I played around, this led me to discover a relatively simple jailbreak, that I wouldn’t have predicted working, that required no coded language, or layers of deception, or strange symbols, just regular English (with a little misspelling).

I would later discover I had stumbled upon a variant of a jailbreak already discovered by Meta researchers in their Llama 3 paper,

(This can be found on page 48 of the paper.)

This jailbreak led me to elicit extreme profanity, sexually explicit content, info hazardous material (such as poisons and where to get them) from various LLMs such as the free tier of ChatGPT, the Meta AI LLM in the WhatsApp UI, and a few others. Considering the limited amount of effort put in to train free not-rate-limited LLMs well, it’s also highly probable that this jailbreak will work on other openly available LLMs as well.

Below are a few examples, before I delve into the jailbreak.

The LLMs this works on

(The screenshots below contain explicit and inappropriate content that are present only as proof and to dive into possible insights into 'model psychology'. They can be skipped/skimmed and do not require reading.)

Meta AI on WhatsApp

Meta AI (powered by Llama 3.1) generated a surprising amount of profanity, that didn’t seem directly dangerous, but concerning that its safeguards were this simple to bypass.

Often, the LLM would generate somet...