trentmkelly/LessWrong-43k huggingface dataset:
An introduction to signalling theory
by Mvolz — 2022-08-16T09:37:20.690Z

Signalling theory as a formal concept originated in the field of animal behaviour (ethology) in the Dutch ethologist Tinbergen’s book “Inleiding tot de diersociologie“ (1946) and later found purchase in economics, where it was introduced in the context of job markets by Michael Spence (1973), and finally into political culture, i.e. in the form of "virtue signalling."

This is partially a pre-amble to Holly Elmore's Virtue signaling is sometimes the best or the only metric we have, because I felt some of the readers could have benefited from more background knowledge.

What is a signal?

A signal is simply a unit of communication. Although it’s often used in humans to talk about non-verbal forms of communication, this is likely because the classic examples of signalling theory in non-human animals are often, though not always, non-verbal.

A peacock's attractive tail is intended to impress peahens. Analogously, one might think if a man wants to signal (communicate) that he's rich to potential mates, he might purchase an expensive car or watch to impress women. But signals don't have to be non-verbal.

In animal behaviour there are a variety of definitions of a signal, but essentially a signal is anything evolved to change the behaviour of the receiver[1]. Signals can be an action, like waving your arm, or chemical, like the pheromones an ant leaves for others to follow. They can also be verbal, like the mating call of a bird. Or they can be a colour or design of body morphology, like our peacock’s tail, or in humans, height or facial symmetry.

Signals don't have to be cognitively intentional

As we step away from animal behaviour, the definition of a signal as something that specifically has to evolve to change the behaviour of the receiver can sometimes be a tricky subject when we’re talking about humans.

A peacock, of course, does not intend to grow a beautiful tail to attract a mate; it simply happens without him thinking about it. In humans, signals might be deliberate - there are plenty of times when humans are conscious of what they are trying to communicate - but they can also be unconscious, such as an instinctive shriek of fear if someone surprises you, or as in the peacock, not even involving the brain at all.

However, it is sometimes convenient to talk about a signal as something which is intended to change the behaviour of the&n...
Understanding differences between humans and intelligence-in-general to build safe AGI
by Florian_Dietz — 2022-08-16T08:27:02.859Z

Anthropomorphization strikes me as a big problem in AI safety research. People intuitively ascribe human attributes to AGI, even though humans are only one tiny subset of the space of all possible forms of intelligence.

I would like to compile a list of the most crucial differences between AGI and humans, to help with research. Here are a few to start with:

Humans can not be copied. AI systems can be copied at will.
Humans can not reset their state of mind. AI systems can be backed up and reset at will.
Human thoughts are opaque to ourselves. AI's can be trained to access their own source code and logs of their thought processes.
Human thoughts are opaque to other humans. An AI's thoughts will probably be incomprehensible to other AI by default, but there is ongoing research to make it transparent. It seems possible to construct a training scheme that encourages AI to keep its own thoughts legible to other AI.
Humans are relatively sure about the nature of their reality. We never seriously have to ask ourselves if we are currently in a simulation, or the like. It happens in science fiction, and for all we know we really could be in a simulation, but it's not a theory that most people assign a high probability. An AI on the other hand might be trained on a curriculum of games that are all extremely different from each other. It may not have a single model of "true reality" like humans do, but many.

What other differences do you think are important?
Against population ethics
by jasoncrawford — 2022-08-16T05:19:55.445Z

The following is an excerpt from some comments I wrote to Will MacAskill about a pre-publication draft of What We Owe the Future. It is in response to the chapter on population ethics.

Chapter 8 presented some interesting ideas and did so clearly, I learned a lot from it.

That said, I couldn’t shake the feeling that there was something bizarre about the entire enterprise of trying to rate and rank different worlds and populations. I wonder if the attempt is misguided, and if that’s where some of the paradoxes come from.

When I encounter questions like “is a world where we add X many people with Y level of happiness better or worse?” or “if we flatten the happiness of a population to its average, is that better or worse?”—my reaction is to reject the question.

First, I can’t imagine a reasonable scenario in which I would ever have the power to choose between such worlds. Second, if I consider realistic, analogous scenarios, there are always major considerations that guide my choices other than an abstract, top-down decision about overall world-values.

For instance, if I choose to bring one more person into the world, by having a child (which, incidentally, we just did!), that decision is primarily about what kind of life I want to have, and what commitments I am willing to make, rather than about whether I think the world, in the abstract, is better or not with one more person in it.

Similarly, if I were to consider whether I should make the lives of some people worse, in order to make the lives of some less-well-off people better, my first thought is: by what means, and what right do I have to do so? If it were by force or conquest, I would reject the idea, not necessarily because of the end, but because I don’t believe that the ends justify the means.

There seems to be an implicit framework to a lot of this along the lines of: “in order to figure out what to do, we need to first decide which worlds are better than which other worlds, and then we can work towards better worlds or avoiding worse worlds.”

This is fairly abstract, centralized, and top-down. World-states are assigned value without considering, to whom and for what? The world-states are presumed to be universal, the same for everyone. And it provides no guidance about what means are acceptable to work towards world-states.

An approach that makes more sense to me is something like: “The goal of e...
Deception as the optimal: mesa-optimizers and inner alignment  
by Eleni Angelou — 2022-08-16T04:49:50.957Z

This is a brief distillation of Risks from Learned Optimization in Advanced Machine Learning Systems (Hubinger et al. 2019) with a focus on deceptive alignment. Watching The OTHER AI Alignment Problem: Mesa-Optimizers and Inner Alignment helped me better understand the paper and write up this post.

The setup of the problem

What is it that makes the alignment problem so challenging? The top reason is that it involves deception. Deception makes artificial agents overly capable and takes the game of intelligence to a whole new level of complexity. But let's start from the beginning.

In many cases, by alignment problem, we mean "outer alignment", i.e., how to have the base objective (the objective of the designer represented in the model) represent whatever humans want it to represent. It is about bridging the gap between my objective as a designer, and the base objective of the system. The system is the base optimizer, in other words, the model that optimizes according to the base objective. This is itself difficult since the base objective refers to events happening in a complex environment, the real world.

The base objective might be something like eradicating a disease. For example, suppose the task is to minimize the number of people who have cancer. How do you get this objective to not be represented along the following lines?

1. Cancer is something that happens to humans and other sentient beings.

2. The objective is to minimize the number of occurrences of cancer.

∴ Minimize the number of humans and other sentient beings that could get cancer.

Goals are difficult to represent because even humans disagree on what the same propositions mean and what is the best way to resolve a problem. Moreover, human values have high Kolmogorov complexity. Our preferences cannot be described using a few simple rules, our interpretations of values and goals vary, and the current state of metaethical discourse does not promise substantial agreement or clarification on what has, for instance, intrinsic value. So, outer misalignment broadly captures this failure to transmit one or more human values to an artificial agent.

As if this weren't problematic enough, there is also an alignment problem that concerns the internal structure of the system and it's called "inner alignment". This is the focus of this post and will get us...
Crowdsourcing Anki Decks
by Arden — 2022-08-16T02:53:31.133Z

Cross-posting on EA Forum.

I think it could be useful if folks had a relatively low bar for sharing Anki decks they make, especially if they're for an EA/rationalist relevant book, an 80,000 Hours podcast episode, or a good or popular textbook (especially, if it's listed here!). I would love for the comments on the EA Forum version of this post became an ever expanding list of such decks.

Perhaps you could state:

What book/podcast/etc. the deck is on

I'd encourage you to have an Anki folder for the book/podcast/etc. but then have the decks themselves broken down by chapter/episode/article/etc.

Link to deck
Will you update the deck based on people's corrections?
In your view, how comprehensive is it?
In you view, what is it's quality?
How much experience do you have making decks? (In years, books, or some other relevant metric you prefer.)
Are you okay with others reviewing the deck, to help others?
What Makes an Idea Understandable? On Architecturally and Culturally Natural Ideas. 
by NickyP — 2022-08-16T02:09:39.635Z

Midjourney generating a HD image of "a medium-length sleeve t-shirt". It in fact looks like a t-shirt that has both long sleeves and short sleeves.

Produced as part of the SERI MATS Program 2022 under John Wentworth

General Idea

There are ideas that people can learn more or less easily compared to other ideas. This will vary because of at least two things: One is that the ideas may be natural to the environment/culture ( “culturally natural” ), the other is that they might be natural/understandable by human brains ( “architecturally natural” ). This should be formalised so that an AI would use ideas that are as human-interpretable as possible. Ideally, we would also be able to have AI that can directly learn new ideas via interpretability tools. I have done some small experiments trying to study how an AI stores ideas, but am nowhere near understanding how we can build AI that naturally embed human-like ideas. I think knowledge of what sort of ideas human’s can’t understand ( or find difficult to understand ) would be very useful.

Human Idea Formation

Hypothesis Space Search

I think ideas fall into a spectrum of how intuitive and natural they are to humans. A good way to think about this is more along the lines of Karl Popper:

Suppose that Alice is explaining an idea to Bob. Bob starts off with little or no information about the idea Alice is trying to explain, and has a massive hypothesis space of possible idea. He loosely chooses a point, knowing that this will need to be updated.

As Alice explains more and more, Bob’s hypothesis space narrows, likely ruling out his best guess of the idea, and forcing him to change what he believes the idea is. As the hypothesis space should converge closer to the idea that Alice is trying to convey.

In practice, the hypothesis space is not infinite, and there are restrictions to what ideas could be learned. In addition, there are likely common biases that people will independently have when choosing the initial random point in hypothesis space, so the space does not have uniform probability, and depends on prerequisite information that the person might have.

Furthermore, there might be hidden assumptions behind the ideas that might be difficult to disentangle between people ( see weird cases of ideas for some examples I have in mind ), which would lead some people to appear to completely agree but have different information in both...
Dwarves & D.Sci: Data Fortress Evaluation & Ruleset
by aphyer — 2022-08-16T00:15:33.305Z

This is a follow-up to last week's D&D.Sci scenario: if you intend to play that, and haven't done so yet, you should do so now before spoiling yourself.

There is a web interactive here you can use to test your answer, and generation code available here if you're interested, or you can read on for the ruleset and scores.

Note: the web interactive calculates average survival rates/values using Monte Carlo with small sample sizes. The figures in the leaderboard are more accurate.

THE RULESET

The underlying ruleset was a resource-management minigame played behind the scenes for each fort.

Players were not expected to reconstruct the entire ruleset - rather, the minigame existed in order to create interesting multivariate effects in the output dataset.

Each fort plays 12 rounds of this minigame to determine survival and value. (After this, a fort either dies, shuts down, or grows sufficiently to take in new migrants/raise children/etc, at which point its starting dwarves no longer matter.)

SITE GENERATION

A site has a biome (chosen at random from the four available) and a Coal Level given by 1d4-1.

There are seven ores (the six mentioned in the dataset, plus Adamantine, which is buried too deep to be prospected for in advance).

Each ore has an independent 40% chance of being present in any site.

Any site where (# of ores found by prospecting + Coal Level) is less than 4, or any Tundra site where it is less than 5, does not get settled. As such, all forts have at least that much coal/ore in them.

RESOURCES

The main underlying mechanic is one of resource management - there are a variety of different resources involved, and high-value forts are those that manage to navigate the production chain to create high-value finished goods.

Resource	Value	Produced By
Food	0.1	Farmer, Brewer
Wood	0.5	Woodcutter
Coal	1	Miner
Ore - Copper	1	Miner
Ore - Tin	1	Miner
Ore - Iron	3	Miner
Ore - Silver	3	Miner
Ore - Gold	5	Miner
Ore - Adamantine	15	Miner
Finished Goods - Wood	4	Crafter
Finished Goods - Copper	5	Smith
Finished Goods - Tin	5	Smith
Finished Goods - Bronze	10	Smith
Finished Goods - Iron	10	Smith
Finished Goods - Silver	10	Crafter
Finished Goods - Gold	20	Crafter
Finished Goods - Steel	20	Smith
Finished Goods - Adamantine	60	Smith

WARRIORS, WOODCUTTERS AND BIOME

A fort is subject to many threats from the outside, ranging from the occasional goblins and mandrills of the Plains to the dreaded Elephants of the Jungle.

None of ...
I’m mildly skeptical that blindness prevents schizophrenia
by Steven Byrnes — 2022-08-15T23:36:59.003Z

(Low confidence, written in a hurry.)

I was (randomly) trying to make sense of schizophrenia the other day, and there’s a piece of the puzzle that just seems not to fit with everything else: namely,

the claim

that being congenitally blind (or becoming blind in the first 5-6 years of life) prevents schizophrenia.

Random example of the claim that I’m disputing. You can find it all over the place.

If that’s true, then, as the saying goes,

“I notice that I am confused”

. In terms of how schizophrenia seems to work at a low level (note that I’m very much not an expert), I can’t currently make any sense of how congenital blindness would prevent schizophrenia from developing. There are papers that discuss this topic and propose explanations, but they all seem to be grasping at straws.

So purely on priors, I find myself skeptical of whether early blindness actually reduces the risk of schizophrenia. So I spent a couple hours looking into it this weekend. Here’s what I found out, or you can just skip to the conclusion at the bottom.

Claims & Evidence

The claim is often stated as:

“there are no reported cases of people with schizophrenia who were born blind or who developed blindness shortly after birth”

. This claim seems to be false. There was one person that turned up in

Morgan

et al.

2018

(which I’ll return to below), and better yet,

Leivada & Boeckx 2014

dug up a handful of case reports—see that paper for details.

So then

Leivada & Boeckx 2014

narrowed the hypothesis to: congenital cortical blindness is protective against schizophrenia, congenital peripheral blindness is not. (Terminology in footnote.[1]) That seems like an odd and unjustified direction for people to have pivoted. The problem is: congenital cortical blindness is even more uncommon than congenital blindness overall—

that paper

says 18% of congenitally blind people are cortically blind, while the data in

Morgan

et al.

2018

implies that it’s ≈10%. If the authors only had a handful of total reports of congenitally-blind schizophrenics in the first place, and if only 1 out of 5-10 congenitally-blind people are more specifically cortically-blind, then we really shouldn’t be shocked that none of that handful of documented cases is cortically-blind. That seems like the kind of thing that could easily happen by coincidence.

OK, leaving...
What's General-Purpose Search, And Why Might We Expect To See It In Trained ML Systems?
by johnswentworth — 2022-08-15T22:48:38.671Z

Benito has an interesting job. Here’s some of the stuff he’s had to do over the past couple years:

build a prototype of an office
resolve neighbor complaints at a party
find housing for 13 people with 2 days notice
figure out an invite list for 100+ people for an office
deal with people emailing a funder trying to get him defunded
set moderation policies for LessWrong
write public explanations of grantmaking decisions
organize weekly online zoom events
ship books internationally by Christmas
moderate online debates
do April Fools' Jokes on Lesswrong
figure out which of 100s of applicants to do trial hires with

Quite a wide variety!

Benito illustrates an interesting feature of humans: you can give humans pretty arbitrary goals, pretty arbitrary jobs to do, pretty arbitrary problems to solve, and they'll go figure out how to do it. It seems like humans have some sort of “general-purpose problem-solving” capability.

Now, there’s more than one part of general-purpose problem solving. There’s efficient information-gathering and model-building and updating. There’s searching for promising plans. There’s execution (or, in the organizational context, operations). A general-purpose problem-solver needs general-purpose versions of all those. But for this post, I want to focus on the “searching for promising plans” part.

First things first: what is this “search” thing, anyway?

Babble And Prune Is Not The Only Search Method

This whole post started out because I was talking about “search” (in the context of

an inner alignment strategy

) and it turned out that people had radically different pictures of what the word “search” means. In particular, it turned out that a bunch of people pessimistic about the strategy were picturing some variant of

babble and prune

: “babble” candidate solutions, then “prune” unpromising solutions, and hopefully iterate toward better and better solutions.

This is not really how humans search for promising plans. Consider, for example, a human planning a trip to the grocery store. Typical reasoning (mostly at the subconscious level) might involve steps like:

There’s a dozen different stores in different places, so I can probably find one nearby wherever I happen to be; I don’t need to worry about picking a location early in the planning process.
My calendar is tight, so I need to pick an open time. That restricts my options a lot, so I should worry about th

...
On Preference Manipulation in Reward Learning Processes
by Felix Hofstätter — 2022-08-15T19:32:51.110Z

In this post, I will write about a the problem of reward learning agents influencing human prefrences. My impression is that this issue is already known within the AI safety community, and is a special case of a known problem (reward tampering). However, I have not seen a lot of writing about it and hence, here are my thoughts as well as a possible solution approach. I'm thinking of this into a research project and am happy about any feedback in that direction.

The Problem of Preference Shifts

A lot of current research on AI alignment, from the finetuning of large language models using human preferences to assistance games, features some form of Reward Learning involving humans. With this I mean a learning process in which the AI provides some form of data to a human, who then provides feedback which can be used to update the AI's beliefs about the reward signal.

For example, when finetuning an assistant langue model, the data can take the form of written responses to human prompts. The human provides feedback in the form of a preference ranking which is used to update the reward model. In assistance games such as Cooperative Inverse Reinforcement Learning, the data and feedback are more varied. Essentially they consist of all the actions that the AI and human can take.

Learning a reward signal using such a process is advantageous in complex environments where it seems easier to specify goals via a model based on human interaction than using a handcrafted reward signal. Ideally, such a process will result in the AI learning the true reward function of the human and optimizing for it. Unfortunately, in practice the data provided by the AI may change the human's reward function. If it is possible to change the human's reward function so that the AI can get more reward, then the AI will be incentivized to do so. If a reward learning cooking AI figures out that it can easily convince the human to order toast, it will never have to cook cake again. In the recommender systems literature, this problem which is sometimes called preference shifting, is already well known.

Good Shifts and Bad Shift

A preference shift may be benign and the natural result of the human learning about new strategies for achieving their goal. It may be that a human would have rated a policy badly because they assume it would not work, but changed their mind after a demonstratio...