trentmkelly/LessWrong-43k huggingface dataset:
Terminology: <something>-ware for ML?
by Oliver Sourbut — 2024-01-03T11:42:37.710Z

Would it be useful to have a term, analogous to 'hardware', 'software', 'wetware', 'vaporware' etc.[1], which could be used to distinguish learned/discovered components of software, like gradient-trained DNNs, prompt-hacked LLMs, etc?

EDIT 2024-01-04: my current favourites are 'ML-ware' (HT Shankar), 'fuzzware' (me), and 'hunchware' (Claude), in that order; LW votes concur with 'ML-ware'.

In a lot of conversations with nonexperts, I find that the general notion of AI as being 'programmed' apparently still has a surprisingly strong grip, even after the rise of ML and DL made it even clearer that this is an unhelpful anchor to have. Thane recently expressed similar, quite strongly.

David Mannheim has a short take AI is not software which I think nicely encapsulates some parts of the important distinctions.

The important thing, for me, is that, in contrast to traditional software, nobody wrote it, the specification is informal at best, and we can't (currently) explain why or how it works. Traditionally, software is 'data you can run', but traditionally this class of data were exclusively crafted (substantially) by human design.

A valid answer to this question is, 'no, we do not need such a term, just say, "learned components of software" or similar'.

In practice, we probably wouldn't apply this term to, say, a logistic regression, but maybe?

Some ideas, none of which I like enough yet

netware (seems too NN-specific; also evokes networking which is the wrong concept)

dryware (like wetware but... dry)

neuroware (too NN-specific; also evokes bio neuro - maybe that's fine)

infoware (sounds like just any data though)

learnedware/learnware

trainware

emergeware

adaptware

implicitware

paraware

evoware

foundware

guessware

fuzzware

noware

everyware

anyware

selfaware

please-beware

After a bit of back-and-forth, Claude managed to produce a few which I think are OK but I'm not very sold on these either

fogware

cloudware

enigware

blurware

darkware

specware

inferware

luckware

chanceware

hunchware

For some illuminating compendia of -ware terms, see wiktionary, computerhope ware jargon, Everyware from rdrop, or gears' shortlist of suggestions. Notably, almost all of these are really semantically <thing>-[soft]ware with the 'soft' elided e.g. spyware really means spy-software. ↩︎
Trading off Lives
by jefftk — 2024-01-03T03:40:05.603Z

Let's say someone

proposes that
to reduce deaths from overly chaotic airplane evacuations we ban
passenger distractions during the most dangerous parts: takeoff and
landing. How could we decide whether a ban like this would be worth it?

The argument for the ban is that the safe window for evacuating a
plane can be very narrow, and evacuation could potentially go better
if everyone were alert. For example, in the 2005 AF358
disaster the plane was completely on fire within ~3min of landing.
While I think the benefit of a ban would likely be even smaller, let's
assume that global adoption of a ban would cause an average of one
fewer person a year to die.

On the other side, there's the cost of ~10min of boredom, for every
passenger, on every flight. Instead of playing games, watching
movies, or reading, people would mostly be talking, looking out the
window, or staring off into space.

One common reaction is to say that on one side of this ledger we have
someone's life, while on the other side we have a bit of boredom, so
of course we should go with the policy that saves lives. Is there any
amount of minor boredom that could equal a life? Many of us have a
sense that there are some kinds of tradeoffs that you just shouldn't
make, such as accepting deaths in exchange for reducing inconvenience.

If you take that perspective seriously, however, you'll have somewhat
fewer deaths and unbearable levels of inconvenience. We could
prohibit radios in cars because the music and adjustment can lead to
collisions. Set the highway speed limit to 25mph. Ban cars entirely
since they're more dangerous than walking and public
transport. Require an N95 indoors at all times. Ban paternosters.
Limit swimming pools to 3ft deep.

In our normal lives we make these kinds of tradeoff all the time, for
example in deciding whether to drive somewhere: you have about a 1 in
a million chance of dying ("one micromort") for
each 175mi
in a car. Thinking through this kind of more normal tradeoff can
give better intuitions for approaching more unusual ones like airline
policies; let's try that here.

There are ~9B
passengers annually, so one fewer death would save the average
passenger ~0.0001 micromort at a cost of ~10min of boredom. Is that a
good trade? Imagine you were choosing between two potential ~10min
car journeys: one being 6mi and one being 200ft shorter ...
MonoPoly Restricted Trust
by ymeskhout — 2024-01-02T23:02:55.066Z

Now we know why he’s so happy

Two months ago (an eternity in podcasting, I know) I was on the Bayesian Conspiracy podcast to discuss polyamory with Aella and Eneasz, both of whom are hella fucking poly.[1] I favor monogamy without moral objection to polyamory, yet its appeal eludes me. Given the caliber of my interlocutors, I walked away feeling uncharacteristically frustrated with our conversation, largely because I think we lack a shared understanding of each other’s vocabulary.

This post is a belated attempt to remedy the miscommunication, and not one that necessarily requires listening to the episode first (though it helps of course). I address the definition of polyamory, how we talk about ‘restrictions’ in relationships, and where trust comes from.

Return of the Antipodes

We started by rehashing my ongoing disagreement with Aella and her idiosyncratic definition of ‘polyamory’. While this definition offers a new perspective, it's important to consider how it aligns with the broader understanding of polyamory and its impact on communication:

The definition of ‘polyamorous’ that I find cleanest, for me, is not forbidding your partner from having extra-relationship intimacy. It doesn’t matter if they’re acting on it or not, it doesn’t matter if you don’t feel like banging anybody else, as long as your partner could go have sex/love someone else if they wanted, then to me, that’s polyamory.

I previously addressed why I really don’t like this ‘antipodal’ re-definition, in contrast to the straightforward and commonly-accepted “the practice of or desire for multiple concurrent romantic/sexual relationships” understanding.[2] Aella has subsequently stated that her position is best expressed as a 2D chart:

Almost a political compass meme…

…which nullifies a lot of my criticism. If you had to compress the spectrum down to just one, Aella favors the ‘restriction’ axis as more fitting while also acknowledging that some information is lost in the process. I agree that a chart allows for more nuance, but disagree with re-defining polyamory to focus away from the ‘interested in many’ axis for multiple reasons:

The risk of confusion by the re-definition is very high
The information conveyed by the re-definition is very low
The ‘restrictions-on-partner’ framing can get incoherent

It’s totally fine to use words with semantic ambiguity (e.g. light, right, match) when their...
Agent membranes and causal distance
by Chris Lakin — 2024-01-02T22:43:41.508Z

For the goal of maintaining the safety and autonomy of particular agents (and potential moral patients), there seems to be something important about maintaining the

membranes and boundaries

of those agents. A few examples:

A bacterium uses its membrane to protect its internal processes from external influences.

A nation maintains its sovereignty by defending its borders.

A human protects their mental integrity by selectively filtering the information that comes in and out of their mind.

What is the membrane of an agent?

Membranes are the things that maintain themselves despite environmental perturbation. (This is one reason I prefer the term “membranes” over “boundaries”. I think agent membranes are best understood as being living.)

Membranes are one way that agents can causally distance themselves from their environment and threats.

Think of a bacterium: it has a membrane that separates it from its environment, and the bacterium can exert a great deal of control on what comes in and what goes out using its membrane (via ion channels, gap junctions, etc.).

The bacterium uses its membrane to attempt to keep bad things (e.g.: toxins, potential pathogens) out, and keep good things (e.g.: energy, sovereignty) in.

When an agent fails to causally distance themselves from their environment and threats, it dies.

When an agent maintains causal distance from its environment— when that agent is able to control how the environment affects it, and also how it affects the environment— then it tends to live.

Agents that persist through time tend to maintain membranes.

Causal distance

If you’re an agent, one way to keep yourself safe from threats is to put physical distance between you and the potential threats. It’s hard to get shot by other people when you’re a thousand miles away from everyone else.

Physical distance can therefore often provide causal distance.

Similarly, membranes can also provide causal distance.

In the absence of physical distance from threats, membranes can help provide causal distance.

Again, consider a bacterium: a bacterium has a membrane that separates it from its environment.

Through this membrane, the bacterium can exert a significant amount of control on what comes in and what goes out (via ion channels, gap junctions, etc.).

“Causal distance” can also be useful for conceptualizing int...
Focusing on Mal-Alignment
by John Fisher — 2024-01-02T19:51:01.316Z

Typically, a definition for “Alignment” includes something like “systems that pursue objectives matching the ones intended by its creator(s)." and agents that are not aligned are “systems that pursue objectives other than the ones intended by its creator(s)." To be more clear though, throughout this article I'll use the term “mal-alignment” as "systems that pursue objectives other than the ones humanity, broadly, would want, or that would be desirable."

Mal-alignment is much more subjective because it relies on coming to agreement on what it is that humanity values, but most people can agree on some core values like those instilled in countries' constitutions, or Maslov’s hierarchy of needs. To avoid an existential crisis - one that could wipe out humanity or lead to society’s collapse - it is not enough to train agents so that they always work towards their creators' goals. Just to take a single example, someone like Ted Kaczynski, a now-deceased mathematics prodigy, would, in the future, be more than capable of creating a well-aligned AI agent. Of course, the agents’ goals, representative of Kaczynski’s goals, could be directly opposite to humanity's goal of species-survival.

Unfortunately, due to the large number of people in the world and the increasing accessibility of technical systems used to build AI agents, it seems inevitable such a malevolent system will eventually be created, and because of this, I believe our engineers and policy makers should be focused on how to verify and monitor AI systems. There are various sub-topics that need research, policy outreach, implementation, and governance enforced:

Checking digital signatures and incorporating this into a verification protocol
Monitoring around the production of ML-capable chips
Monitoring for training data set downloads nefarious training data
A database of questions that could be used to confirm a system is not mal-aligned
Requirements for neural networks to be "interpretable" in their intermediate layers, possibly establishing a protocol for interrogation
Gentleness and the artificial Other
by Joe Carlsmith — 2024-01-02T18:21:34.746Z

(Cross-posted from my website. Audio version here, or search "Joe Carlsmith Audio" on your podcast app.

This is the first essay in a series that I’m calling “Otherness and control in the age of AGI.” See here for more about the series as a whole.)

When species meet

The most succinct argument for AI risk, in my opinion, is the “second species” argument. Basically, it goes like this.

Premise 1: AGIs would be like a second advanced species on earth, more powerful than humans.

Conclusion: That’s scary.

To be clear: this is very far from airtight logic.[1] But I like the intuition pump. Often, if I only have two sentences to explain AI risk, I say this sort of species stuff. “Chimpanzees should be careful about inventing humans.” Etc.[2]

People often talk about aliens here, too. “What if you learned that aliens were on their way to earth? Surely that’s scary.” Again, very far from a knock-down case (for example: we get to build the aliens in question). But it draws on something.

In particular, though: it draws on a narrative of interspecies conflict. You are meeting a new form of life, a new type of mind. But these new creatures are presented to you, centrally, as a possible threat; as competitors; as agents in whose power you might find yourself helpless.

And unfortunately: yes. But I want to start this series by acknowledging how many dimensions of interspecies-relationship this narrative leaves out, and how much I wish we could be focusing only on the other parts. To meet a new species – and especially, a new intelligent species – is not just scary. It’s incredible. I wish it was less a time for fear, and more a time for wonder and dialogue. A time to look into new eyes – and to see further.

Gentleness

“If I took it in hand,

it would melt in my hot tears—

heavy autumn frost.”

- Basho

Have you seen the documentary My Octopus Teacher? No problem if not, but I recommend it. Here’s the plot.

Craig Foster, a filmmaker, has been feeling burned out. He decides to dive, every day, into an underwater kelp forest off the coast of South Africa. Soon, he discovers an octopus. He’s fascinated. He starts visiting her every day. She starts to get used to him, but she’s wary.

One day, he’s floating outside her den. She’s watching him, curious, but ready to retreat. He moves his hand slightly towards her. She reaches out a tentacle, and touches his hand.

Soon, they are...
Otherness and control in the age of AGI
by Joe Carlsmith — 2024-01-02T18:15:54.168Z

(Cross-posted from my website. PDF of the full series here. Audio version of the full series here; or search for "Joe Carlsmith Audio" on your podcast app.)

“With malice towards none; with charity towards all; with firmness in the right, as God gives us to see the right…”

- Abraham Lincoln

Lincoln’s Second Inaugural (image source here)

I’ve written a series of essays that I’m calling “Otherness and control in the age of AGI.” The series examines a set of interconnected questions about how agents with different values should relate to one another, and about the ethics of seeking and sharing power. They’re old questions – but I think that we will have to grapple with them in new ways as increasingly powerful AI systems come online. And I think they’re core to some parts of the discourse about existential risk from misaligned AI (hereafter, “AI risk”).[1]

The series covers a lot of ground, but I’m hoping the individual essays can be read fairly well on their own. Here’s a brief summary of the essays that have been released thus far (I’ll update it as I release more):

The first essay, “Gentleness and the artificial Other,” discusses the possibility of “gentleness” towards various non-human Others – for example, animals, aliens, and AI systems. And it also highlights the possibility of “getting eaten,” in the way that Timothy Treadwell gets eaten by a bear in Werner Herzog’s Grizzly Man: that is, eaten in the midst of an attempt at gentleness.
The second essay, “Deep atheism and AI risk,” discusses what I call “deep atheism” – a fundamental mistrust both towards Nature, and towards “bare intelligence.” I take Eliezer Yudkowsky as a paradigmatic deep atheist, and I highlight the connection between his deep atheism and his concern about misaligned AI. I also connect deep atheism to the duality of “yang” (active, controlling) vs “yin” (receptive, letting-go). A lot of my concern, in the series, is about ways in which certain strands of the AI risk discourse can propel themselves, philosophically, towards ever-greater yang.
The third essay, “When 'yang' goes wrong,” expands on this concern. In particular: it discusses the sense in which deep atheism can prompt an aspiration to exert extreme levels of control over the universe; it highlights the sense in which both humans and AIs, on Yudkowsky’s narrative, are animated by this sort of aspiration; and it d

...
Apologizing is a Core Rationalist Skill
by johnswentworth — 2024-01-02T17:47:35.950Z

There’s this narrative about a tradeoff between:

The virtue of

Saying Oops

, early and often, correcting course rather than continuing to pour oneself into a losing bet, vs
The loss of social status one suffers by admitting defeat, rather than spinning things as a win or at least a minor setback, or defending oneself.

In an ideal world - goes the narrative - social status mechanisms would reward people for publicly updating, rather than defending or spinning their every mistake. But alas, that’s not how the world actually works, so as individuals we’re stuck making difficult tradeoffs.

I claim that this narrative is missing a key piece. There is a social status mechanism which rewards people for publicly updating. The catch is that it’s a mechanism which the person updating must explicitly invoke; a social API which the person updating must call, in order to be rewarded for their update.

That social API is apologizing.

Mistake/Misdeed + Apology can be Net Gainful to Social Status

A personal example: there was a post called “

Common Misconceptions about OpenAI

”, which (among many other points) estimated that ~30 alignment researchers work there. I replied (also among many other points):

I'd guess that is an overestimate of the number of people actually doing alignment research at OpenAI, as opposed to capabilities research in which people pay lip service to alignment. In particular, all of the RLHF work is basically capabilities work which makes alignment harder in the long term (because it directly selects for deception), while billing itself as "alignment".

There was a lot of pushback against that. Paul Christiano replied “Calling work you disagree with ‘lip service’ seems wrong and unhelpful.”. I clarified that I was not using this as a generic criticism of something I disagreed with, but rather that RLHF seemed so obviously antihelpful to alignment that I did not expect most people working on it had actually thought about whether it would help, but were instead just doing it for other reasons while, y’know, paying lip service to alignment.[1] Richard Ngo eventually convinced me the people working on RLHF had thought at least somewhat about whether it would help; his evidence was a comment elsewhere from Paul, which in-particular said:

To be clear, this is not post hoc reasoning. I talked with WebGPT folks early on while they were wondering ab

...
Cortés, AI Risk, and the Dynamics of Competing Conquerors
by James_Miller — 2024-01-02T16:37:17.457Z

Cortés's conquest of the Aztecs reflects current anxieties surrounding AI risks. His technological and tactical edge over the Aztecs parallels the potential superiority of smarter-than-human AIs attempting to take over earth. This historical parallel implies that AIs could employ strategies akin to Cortés's, like utilizing terror and relying on collaborators. It further highlights the possibility of humanity successfully resisting the first AI would-be conquerors.

The Aztecs could have defeated Cortés if events had unfolded just a bit differently. European power in the Americas was growing, and over the long haul, the Aztecs had no realistic chance against a resolute European colonization campaign. Consequently, you might think Cortés foolishly struck prematurely and just got lucky. But Cortés, eager to be the European who conquered the Aztecs, was not content to wait and risk letting other Europeans take the lead. Similarly, a coalition of AIs, foreseeing competition from other AIs, may declare war on humanity before being fully capable of ensuring success. Hence, like the Aztecs' potential to defeat Cortés, humans might beat the first AIs that try to conquer us.

Establishing alliances with local tribes opposed to the Aztecs was a crucial strategy for Cortés to stand a chance of victory. The first AIs aiming for dominance might similarly require human collaborators.

In the context of AIs’ growing power relative to humans, one might think that an AI bent on conquest would never bother relying on humans for help. Yet, this could incorrectly presume that all AIs function as a monolithic entity with a singular goal. A set of AIs might calculate that only with human help could they win in the very near term and preempt another group of AIs from taking over.

Imagine a landscape where numerous AI factions are steadily increasing their power in relation to humans. Each faction hopes to eventually dominate. In this scenario, there is a strategic tension: there's a potential first mover advantage for a group that acts quickly to seize vital yet lightly defended human resources, such as data centers. Seizing these resources first would give an AI faction a significant advantage over competing AIs, but would also trigger a war with humans, perhaps before AIs are strong enough to win. Consequently, waiting a bit allows an AI faction to grow stronger and be more likely ...
OpenAI's Preparedness Framework: Praise & Recommendations
by Orpheus16 — 2024-01-02T16:20:04.249Z

In this post, I summarize some of my thoughts on

OpenAI's recent

preparedness framework

. The post focuses on my opinions/analysis of the framework, and it doesn't have much summary (for readers unfamiliar with the framework, I recommend

Zvi’s post

).

As a high-level point, I believe voluntary commitments from individual labs will be insufficient, and I'm most excited about policies that governments could apply to all AGI corporations. Nonetheless, I think efforts from individual labs can be useful, and it's important for government-focused AI policy experts to engage with the ideas coming out of top AGI companies[1].

With that in mind, throughout the post, I try not to harp on pessimistic points like "more is needed", "this document hasn't single-handedly saved the day or addressed race dynamics", or "how does this prevent competitive pressures+race dynamics from forcing everyone to proceed incautiously". (I think these critiques are useful, but that isn't the purpose of this post). Instead, I focus more on "what do I think OpenAI's preparedness team did well?" and "What do I think are some things OpenAI's preparedness team could realistically change that would make this even better?"

Communication[2]

I

previously noted

that I was critical of the way that Anthropic communicated about its Responsible Scaling Policy. It seemed to me like Anthropic’s RSP was written in a corporate style that made it somewhat hard to identify the core content and signaled to policymakers that the situation is “under control” and that scaling could occur responsibly as long as companies released responsible scaling policies.

To my surprise, I think OpenAI’s preparedness framework communicated its ideas in a much clearer and more concern-inducing way. Here are a few specific things I appreciated about the style/tone/communication in OpenAI’s preparedness framework:

The name. I think “preparedness framework” is an excellent name. It focuses the reader’s attention on “being prepared for something”, and a natural question would be “what should we be prepared for?” This is exactly the kind of question that I want policymakers (and influential folks who are unfamiliar with AI safety) to be asking. It also avoids making the implicit assumption that scaling can definitely be done responsibly.
Clarity. It is hard to write clearly. But when writing public-facing doc

...