trentmkelly/LessWrong-43k huggingface dataset:
Distinguishing misuse is difficult and uncomfortable
by lemonhope — 2023-05-01T16:23:17.040Z

USPS delivers ibuprofen and vicodin just the same because it doesn't care enough to open and test every bottle of pills. If they did carefully inspect the contents of every package, they still couldn't distinguish other "shipping misuse" — money for your granddaughter's birthday looks the same as payment for a crime.

An AI providing services via an API is in a similar position. Code that checks that your firewall is configured correctly is identical to code that checks someone else's firewall is configured incorrectly.

If you have more context then you can identify misuse more easily. You can get more context by either utilizing your memory, providing higher-level tasks, or by snooping / being around.

Memory

Buying duct tape is normal. Buying zip ties is normal. Buying prepaid cell phones is normal. Buying extra large burlap sacks is normal. If someone asks you to buy all four at once then you might flag that as shopping misuse pretty easily. If they buy the three sequentially then spotting it requires a persistent identifier, logging, the will to make the connection, and the will to act on it.

A couple years ago my white dad went to the store to buy beer with the dark-skinned son of a family friend. The cashier refused to sell it because it obviously looked like buying alcohol for a minor. He just went around the corner and asked a different cashier. I suppose this was a failure of logging — the cashier should've taken the credit card before refusing to sell the beer.

High-level tasks

It is easier to tell apart a malicious function from a line of code, a file from a function, a repo from a file, or an app from a repo.

Being around

Hard to know if your boss is a good fella when you take your orders on slips of paper slid under the door. Humans have avoided doing many evil deeds asked of them by witnessing the emotions and other actions of the one asking. A company like Microsoft or Google could have AI that's generally aware of who you are and why you're asking for certain things. This would take a lot of courage and be a privacy nightmare.

Conclusion

We have set up the misuse detection task as an impossible problem. You simply cannot tell if a single query in isolation is malicious or not. Actually detecting misuse will require some reprioritization, reframing, and uncomfortable decisions.
Does agency necessarily imply self-preservation instinct?
by Mislav Jurić — 2023-05-01T16:06:03.489Z

I have recently read some blog posts (this, this and this) about tool AGIs and why agent AGIs are more likely. The argument that convinced me to err on the side of agent AGIs was that they are more likely to have an economical advantage over tool AGIs, as explained here.

I do not question the value of AI safety research; I think it's good to have people who are trying to make sure that an agent AGI is aligned with what we (as humans) actually want. However, I am not sure that agency necessarily implies self-preservation instinct.

I think you could have a superintelligent agent (which has its own agency in the sense that it can act in the world) which does not resist being shut down. I could envision some future scenarios where a superintelligent agent does resist being shut down or being modified by humans (hence the value of AI safety research), but I don't see that you gain self-preservation instinct by agency itself. I think self-preservation instinct either has to be coded inside the agent or sometimes (but not always) emerges from a combination of superintelligence and having your own agency.

From the impression I got from reading some AI safety related materials it seems that most people seem to assume that if a superintelligent agent exists, it is most likely going to have self-preservation instincts (such as resisting being shut down). I think it is a plausible future scenario, but I don't think it's the only scenario. Hence why I'm asking this question.
What Boston Can Teach Us About What a Woman Is
by ymeskhout — 2023-05-01T15:34:49.997Z

"It's all about that semantic space map."

[Originally posted 7/14/2022 on Substack.]

One of the coolest illustrations of societal consensus is this cartography research project from a few years ago called Bostonography. Boston residents were asked to draw the boundaries of various neighborhoods and their responses were amalgamated onto a single map, with color-coding to designate areas with high or low consensus.

Some neighborhoods, like the tony Beacon Hill, have extremely high consensus and a cohesive shape. In part this is aided by its surroundings — with a major avenue to the North, a river to the West, and a park to the South, the only meaningful area for disagreement appears to be over a small parcel on the East, close to the downtown core. In contrast, few can apparently agree on where Chinatown begins and ends.[1]

Other neighborhoods, like Dorchester, cover such a massive area that the boundaries fade into the horizon.

Neighborhood boundaries are a semantic Schelling point. Even though Boston residents each have their own subjective interpretation of boundaries, convergence towards a shared understanding remains possible. There may be disagreements at the margins, but Boston residents generally agree on where what is. And because of this shared understanding, neighborhood names will be useful methods of coordination. Telling someone that you bought a house in Beacon Hill, or that you work in Allston, or that you’ve never been to Bay Village, etc. all are used to quickly communicate useful information to someone, without the need to pull out the sextant each time.

Another useful method of coordination is agreeing on a unit of measurement. History has provided us with a cavalcade of now-obsolete units of measurement. Vernacular units of measurement based on some dead guy’s foot or whatever were likely good enough for most purposes, especially if you’re dealing with the same locals for decades (everyone eventually knows that Yosef has the biggest cubits), but once you step outside of your community you’re bound to run into some problems. Coordination across commerce and enterprise cannot meaningfully happen without establishing a shared understanding of measurements.

Tackling this problem on a global scale was the aim behind the metric system. In creating the kilogram standard, the enlightened French Revolution wanted it to be feasible for anyone with en...
The Rocket Alignment Problem, Part 2
by Zvi — 2023-05-01T14:30:00.385Z

Previously (Eliezer Yudkowsky): The Rocket Alignment Problem.

Recently we had a failure to launch, and a failure to communicate around that failure to launch. This post explores that failure to communicate, and the attempted message.

Some Basic Facts about the Failed Launch

Elon Musk’s SpaceX launched a rocket. Unfortunately, the rocket blew up, and failed to reach orbit. SpaceX will need to try again, once the launch pad is repaired.

There was various property damage, but from what I have seen no one was hurt.

I’ve heard people say the whole launch was a s***show and the grounding was ‘well earned.’ How the things that went wrong were absurd, SpaceX is the worst, and so on.

The government response? SpaceX Starship Grounded Indefinitely By FAA.

An FAA spokesperson told FLYING that mishap investigations, which are standard in cases such as this, “might conclude in a matter of weeks,” but more complex investigations “might take several months.”

Perhaps this will be a standard investigation, and several months later everything will be fine. Perhaps it won’t be, and SpaceX will never fly again because those in power dislike Elon Musk and want to seize this opportunity.

There are also many who would be happy that humans won’t get to go into space, if in exchange we get to make Elon Musk suffer, perhaps including those with power. Other signs point to the relationships with regulators remaining strong, yet in the wake of the explosion the future of Starship is for now out of SpaceX’s hands.

A Failure to Communicate

In light of these developments, before we knew the magnitude or duration of the grounding, Eliezer wrote the following, which very much failed in its communication.

If the first prototype of your most powerful rocket ever doesn’t make it perfectly to orbit and land safely after, you may be a great rocket company CEO but you’re not qualified to run an AGI company.

(Neither is any other human. Shut it down.)

Eliezer has been using the rocket metaphor for AI alignment for a while, see The Rocket Alignment Problem.

I knew instantly both what the true and important point was here, and also the way in which most people would misunderstand.

The idea is that in order to solve AGI alignment, you need to get it right on the first try. If you create an AGI and fail at its alignment, you do not get to scrap the experiment, learn from what ha...
Where is all this evidence of UFOs?
by Logan Zoellner — 2023-05-01T12:13:33.706Z

In his latest blog, Robin Hanson writes

There have been over 100K UFO sightings reported worldwide since 1940. Roughly 5% or so are “strong” events, which seem rather hard to explain due to either many witnesses, especially reliable witnesses, physical evidence, or other factors.

Yet, I am not aware of a single UFO encounter that can't be explained by one of:

Unreliable eyewitnesses
Things that go away when we get better cameras

Importantly, Hanson's post did not include a link to the "Wikipedia of UFO encounters" listing all 100k UFO sightings and which 5000 he considered most credible.

Where is that Wikipedia?
LessWrong Community Weekend 2023 [Applications now closed]
by Henry Prowbell — 2023-05-01T09:08:14.502Z

Update: [Applications now closed]

September 22-25th is the 10th annual LessWrong Community Weekend (LWCW). This is Europe's largest rationalist social gathering which brings together 150 aspiring rationalists from across Europe and beyond for four days of socialising, fun and intellectual exploration.

The majority of the content will be unconference style and participant driven. On Friday afternoon we put up six wall-sized daily planners and by Saturday morning the attendees fill them up with 100+ workshops, talks and activities of their own devising.

Previous years’ schedules have included…

Double Cruxing
Hamming Circles
Gendlin Focusing
Applied Rationality workshops by CFAR instructors and instructors-in-training
Circling
Authentic Relating games
Improv theatre
Introduction to stand up comedy
Writing rationalist fiction
Dance workshops
Acapella singing
Icebreaker games
Lightning talks
Celebrating failure groups
Giant outdoor chess Penultima
Dungeons & Dragons
Kung Fu basics
Board games

Breathwork workshops
Ecstatic dancing
Radical Honesty workshops
Playfighting for adults
Polyamory and relationships workshops
Sex Q&A roundtable
Quantified self workshops
Moral philosophy debates
AI safety Q&A
How to handle fear of AI Doom
Value drift in EA
The neurobiology of psychedelics
The science of longevity
Morning runs and yoga
Meditation in the rooftop winter garden
Night time swimming
Ted Chiang and Greg Egan bedtime story readings

If things like ecstatic dancing, radical honesty and polyamory workshops sound too intense for you, rest assured everything is optional. (I’m British and very awkward so a lot of this stuff terrifies me.)

The event takes place in the natural environs of Lake Wannsee on the outskirts of Berlin. So you can spend some time recharging in between making new friends by hiking in the forests, sunbathing or swimming in the lake.

For those who want to extend their stay in Berlin, there will very likely be meetups arranged in the days before and after the event too.

LWCW is LGBTQIA+ friendly, people are welcome to bring their children (potentially there will be a professional childminder provided throughout) and this year we’re putting extra effort into creating an event where people of all ages, genders, backgrounds and adjacent interests (EA, circling, philosophy, meditation…) feel at home.

This event has a special place in my heart a...
In AI Risk what is the base model of the AI?
by jmh — 2023-05-01T03:25:11.821Z

That is not the best statement but close enough to keep a simple title/question.

When I read discussions and comments about AI risks I find myself thinking that there might be two (unstated?) base models in place. I suspect when people talk about the "AI wants" or go about applying utility functions they actually are using humans as some primitive model from which the AI derives.

Similarly, when I hear talk about extinction potential I have the view that the model is that about biology and biological evolution and competition within environmental niches.

Is that something anyone even talks about? If so, what is the view -- any specific papers or comments where I can take a look? If not, does it sound like a reasonable inference about implicit assumptions/maps in this area?
Hell is Game Theory Folk Theorems
by jessicata — 2023-05-01T03:16:03.247Z
Link: https://unstableontology.com/2023/05/01/hell-is-game-theory-folk-theorems/

[content warning: simulated very hot places; extremely bad Nash equilibria]

(based on a Twitter thread)

Rowan: "If we succeed in making aligned AGI, we should punish those who committed cosmic crimes that decreased the chance of an positive singularity sufficiently."

Neal: "Punishment seems like a bad idea. It's pessimizing another agent's utility function. You could get a pretty bad equilibrium if you're saying agents should be intentionally harming each others' interests, even in restricted cases."

Rowan: "In iterated games, it's correct to defect when others defect against you; that's tit-for-tat."

Neal: "Tit-for-tat doesn't pessimize, though, it simply withholds altruism sometimes. In a given round, all else being equal, defection is individually rational."

Rowan: "Tit-for-tat works even when defection is costly, though."

Neal: "Oh my, I'm not sure if you want to go there. It can get real bad. This is where I pull out the game theory folk theorems."

Rowan: "What are those?"

Neal: "They're theorems about Nash equilibria in iterated games. Suppose players play normal-form game G repeatedly, and are infinitely patient, so they don't care about their positive or negative utilities being moved around in time. Then, a given payoff profile (that is, an assignment of utilities to players) could possibly be the mean utility for each player in the iterated game, if it satisfies two conditions: feasibility, and individual rationality."

Rowan: "What do those mean?"

Neal: "A payoff profile is feasible if it can be produced by some mixture of payoff profiles of the original game G. This is a very logical requirement. The payoff profile could only be the average of the repeated game if it was some mixture of possible outcomes of the original game. If some player always receives between 0 and 1 utility, for example, they can't have an average utility of 2 across the repeated game."

Rowan: "Sure, that's logical."

Neal: "The individual rationality condition, on the other hand, states that each player must get at least as much utility in the profile as they could guarantee getting by min-maxing (that is, picking their strategy assuming other players make things as bad as possible for them, even at their own expense), and at least one player must get strictly more utility."

Rowan: "How does this apply to an iterated game where defection is costly? Doesn't this pr...
Safety standards: a framework for AI regulation
by joshc — 2023-05-01T00:56:39.823Z

Purpose: provide a public resource on safety standards that introduces the ideas I've heard people talk about and some new ideas. I don't know which ideas are new because the discourse has been largely non-public. I think safety standards provide a clear regulatory direction and suggest several useful directions of work, so I think it would be good for more people to be familiar with the topic.

Definition

In the context of AI, I'll define a safety standard to be a policy that a lab agrees to follow or is required to follow by law that involves the following:

A triggering condition that is intended to track the properties of a specific AI system.
Obligations that a lab must follow if this triggering condition is met.

Here's an example: "If an AI system is capable of autonomous replication[1], the lab that developed it must spend at least 5% of the training costs on infosecurity audits over the course of 1 month before deploying the system."
These standards could be set, maintained, and enforced by a national AI regulatory agency in collaboration with external experts -- similar to how the FDA requires safety tests of many drugs before they can be sold.

Triggering conditions

A triggering condition is meant to indicate that an AI system crosses a threshold of dangerousness that warrants specific safety measures.

There are two ways to design triggering conditions so that they track the ‘dangerousness’ of a system. They can depend on (1) direct evaluations of the system's capabilities or (2) proxies for the system capabilities.

Direct evaluations of dangerous capabilities

By capability, I mean any behavior that an AI system exhibits or can be made to exhibit by an actor with a budget less than X that is given unrestricted access to the system and can integrate it with other software. The choice of X should depend on the incentives actors are likely to have to elicit the capability if they were given unrestricted access to the system.

For example, even if an AI system served through an API never writes poetry criticizing Obama because there is a content filter that prevents this, I would still say it is capable of writing slanderous poetry because removing the content filter would be an inexpensive modification that many actors would realistically make if they had unrestricted access.

The following are broad categories of...
neuron spike computational capacity
by bhauth — 2023-05-01T00:28:45.532Z

The following is taken from this blog post. I'm posting it here now because of this argument between jacob_cannell and others including Eliezer Yudkowsky.

A neuron firing by opening sodium channels is all-or-nothing, so information must be transmitted in the timing. Neurons have no global absolute clock, so information must be contained in "time since last spike" rather than the absolute time of spikes, unless a spike means that something has just happened.

For some current theories of neuron firing, see this page.

From artificial neural network (ANN) research, we know that linear representations of activations are worse at low resolution than some nonlinear ones. I would expect a spike to typically represent an activation of approximately:

Formula 1: a + b * e^(-c * time_since_last_spike)

Each neuron is connected to many synapses, typically ~7000 in a human brain. When a neuron fires, at each connected synapse, some neurotransmitters are released. Most neurotransmitters have no immediate effect on neuron firing, and instead regulate slower processes, but let's consider just the short-term behavior of neurons. A neuron N1 fires, it releases some neurotransmitters at a synapse, and they bind to receptors at neuron N2 which have some net effect on the electric potential of N2.

That net short-term effect on N2 potential is analogous to a "weight" in an ANN, but it's not a constant, it's a function: spike timing -> change in potential. (Also, the neurotransmitter receptors at synapses can be disabled or added over a longer timescale.) The N2 potential is analogous to an ANN accumulator, but it decays towards a baseline over time.

The represented weights can be both positive and negative. It's also common for a single synapse to have receptors with positive and negative effects on cell potential active at the same time.

I wrote above that I'd expect the activation represented by a spike to often approximately follow Formula 1. An obvious way to accomplish that is to have a synapse with a channel type with approximately constant value on firing, and another channel type transporting ions that, as time passes after a spike, are transported and asymptotically approach an approximately opposite value.

That being the case, I'd expect two spikes in rapid succession at the same synapse to usually represent a large activation value. Depending on the "weights" at that synapse...
Cult of Error
by bayesyatina — 2023-04-30T23:33:45.098Z

Normal World
Errors are considered garbage here. That doesn't mean they don't exist here. It is just that it is customary to hide them, especially in a public space. Showing them is considered embarrassing to both the demonstrator and the audience.
People who make mistakes are considered fools. People who admit them are considered wimps. People who acknowledge at least the possibility of them are considered cowards, preparing the way for an ignominious retreat.
Yes, people inevitably have to go through mistakes when they are young, but that stage is simply considered childish inferiority. Adult people try to limit their lives so much that nothing unusual that can generate a mistake happens to them. It does not always work, but they skillfully conceal all errors, especially from children.
Fortunately, they learn to hide mistakes in special educational institutions. They teach to pretend that everything is clear, even if it is not. They teach you how to cunningly disguise if you make a mistake. Most of all, the ability to do a task the first time is valued, and "grades" are given for this. These are such indicators that a person is not wrong (or cleverly disguised). Here, a person who is initially good at everything, and who subsequently copes just as well, is considered capable of learning. A person, on the other hand, who did not succeed at anything, and then began to succeed at something, is not considered capable of learning.
This is where ancient knowledge is valued. After all, the more doubts they arouse, the more opportunity they have to show their persistence and perseverance. The only regret is that with each generation the ancient knowledge weakens because of errors in transmission. That is why it is considered most important to preserve it in its original form without making any changes.
All this leads to the fact that predictions in this world do not come true. But no one admits this, and it is not commonly remembered. Here, for example, Philip Tetlock (who lives in another world) asked 284 experts in politics and economics (from this world) to give their predictions in their subject areas. 82,000 predictions. The result was worse than if they had given equal probabilities for all event outcomes...
Optimization Islands
Don't be afraid of not knowing, be afraid of not learning. The motto of the islands.
But there is another strange world. T...