trentmkelly/LessWrong-43k huggingface dataset:
Intuitions about utilities
by mingyuan — 2021-02-06T00:12:44.383Z

I'm just a layperson with a layperson's exposure to game theory problems, and I've long been really confused about how to think about them. This is the story of my confusion, and of how I think about utilities in game theory problems now. Very plausibly none of this is novel.

I was first introduced to the Prisoner's Dilemma in my ninth grade history class, and I spent a decade afterwards being fairly baffled. Here was the setup:

Our class of 30 was divided into 10 groups of 3 people. Each group was given an opaque envelope with a pink notecard and a green notecard in it.

If all 10 groups pulled the green card, everyone got one Hershey's Kiss. If some people pulled pink and some pulled green, those who pulled pink would each get two Hershey's Kisses, and those who pulled green would get nothing.

I held the envelope for my group.

On the first round, maybe 8 out of 10 groups pulled green, and then were crestfallen and resentful when they didn't get any candy. The next time, every single group except mine pulled pink. Next round, same thing. And the next round. And the next. Until finally one of my teammates wrestled the envelope away from me and pulled pink, and then no one got any candy.

My teacher singled me out at the end of the exercise and asked me why I chose to cooperate, and I gave some confused milquetoast answer about it being the right thing to do. But truthfully, I didn't know, and I've been thinking about that question ever since. I've gone through half a dozen different interpretations over the years, but I think I finally understand, and it's not complicated at all.

I now believe the reason it was so easy for me to cooperate was not that I was deeply, fundamentally altruistic – I didn't, as I once thought might be the case, calculate that the total utility of cooperating was higher because, while I and my two teammates would be denied candy, our other 27 classmates would get candy, and it was better to have 90% of the class get candy than 0% of the class, even if I was in the 10% of non-candy-getters. No, it was easy because I didn't care about the chocolate. I got far more actual utility from signaling my virtuous altruism to all of my classmates (not to mention myself), than I ever could from a short-lived piece of mass-produced candy.

I've since played at least one other Prisoner's Dilemma game – this one for team points rather than candy ...
The Fear Experiments
by Tim Lyons — 2021-02-05T22:57:08.888Z

I. INTRODUCTION

The physiological experience of fear is an interesting area for inquiry because of how these physical effects come to influence our decision making. I will start by saying that these are very personal notes that are much too raw to be seen as a formal paper or argument. I am choosing to share them here because I think they could be fertile soil for an interesting conversation about epistemology.

The Experiment:

In 2018 I had made a new year's resolution to run four times a week, because of work scheduling my run would most often be in the evening. After sticking with the resolution for a few nights I began to consider a large patch of woods I would pass on my run as a potential location for experimentation. The woods are roughly 80,000 square yards with the perimeter lined with residential houses. This is to say, the area is small in size with very little chance of an unknown threat finding its way into the area. The area's small size combined with a significantly low crime rate/lack of large predators created an excellent control setting for the experience of fear. With this, the experiments would begin. Each evening after my run I would enter the woods at the same opening and slowly make my way through the unlit forest. As I had imagined the environment proved to be a perfect control to almost immediately induce the typical physiological effects of fear. The hair standing up on your neck, the cold extremities, the hyper-focus to what is perceived as any unusual sensory experience. The experience of fear was like a thick viscous sludge. Each step was hard, but as I walked forward they became easier, and eventually once pushed through the sludge made its way into water. Over time the sludge would only appear at the beginning of the walk, diminishing little by little upon each visit. Eventually, the entire walk became water. Other than a slightly heightened awareness the movement from the lit street into the darkness of the forest overhang was almost transitionless. That was until the first of what would become three incidents.

II. THE THREE INCIDENTS

Incident I:
After continuing this habit/experiment for roughly two months something drastically different had occurred. I approached the mouth of the woods where I had entered every time previously and a fear that I had not yet experienced up to that point washed over me. Rathe...
Learning Normativity: Language
by Bunthut — 2021-02-05T22:26:15.199Z

Abram Demski has been writing about Normativity. The suggested models so far have mostly looked at actions rather than semantics, despite suggestions that this is possible and language learning as a motivating example. There is a simple mechanism that seems to me to mostly fit that bill.

Model

There is an interpreter and an assumed speaker. The interpreter receives observation data as an input, which contains some things put there by the speaker among others. The interpreter has a language in which it can express all its beliefs. Since we want to form beliefs about meaning, this language can talk about meaning: it can from propositions of the form "[Observation] means [propositon].". Note that this is different from "[Observation] implies [propositon].". At least initially, "means" is not related to anything else. The interpreter also has an epistemology module that forms its beliefs about things other than meaning.

We follow a simple prior-update-paradigm. We start out with a prior over hypothesis about meaning. Such a hypothesis generates a propability distribution over all propositions of the form "[Observation] means [propositon]." for each observation (including the possibility that the observation means nothing). Updating is based on the principle that the speaker is authoritative about what he means. Our interpretation of what he's saying should make the things we interpret him to say about what he's saying true. To update on an observation history, first we compute for each observation in it our summed prior distribution over what it means. Then, for each hypothesis in the prior, for each observation, take the hypothesis-distribution over its meaning, combine it with the prior-distribution over all the other observations, and calculate the propability that the speakers statements about what he meant were right. After you've done that for all observations, multiply them to get the score of that hypothesis. Multiply each hypothesis's score with its prior and renormalize. Then, take the resulting propabilities as your new prior, and repeat infinitely.

Lets run an example of this. Your prior is 50% hypothesis A, 50% hypothesis B. There are only two observations. A gives observation1 90% of meaning "Its cold" and 10% of meaning nothing, and gives observation2 80% of meaning "When I describe a local state, the state I mean always obtains", and 20% to mean noth...
Open & Welcome Thread – February 2021
by mingyuan — 2021-02-05T21:38:28.789Z

If it’s worth saying, but not worth its own post, here's a place to put it.

If you are new to LessWrong, here's the place to introduce yourself. Personal stories, anecdotes, or just general comments on how you found us and what you hope to get from the site and community are invited. This is also the place to discuss feature requests and other ideas you have for the site, if you don't want to write a full top-level post.

If you want to explore the community more, I recommend reading the Library, checking recent Curated posts, seeing if there are any meetups in your area, and checking out the Getting Started section of the LessWrong FAQ. If you want to orient to the content on the site, you can also check out the new Concepts section.

The Open Thread tag is here. The Open Thread sequence is here.
Three Stories about not Thinking too Hard
by ejacob — 2021-02-05T19:51:47.260Z
Link: https://eidan.substack.com/p/riddles-in-real-life

I.

5 pirates split a treasure of 100 gold coins. The crew splits their booty according to sacred pirate tradition: The oldest pirate proposes how to share the coins, and ALL pirates (including the oldest) vote for or against it. If at least half vote for it, then the coins will be shared according to the proposal. Otherwise, the eldest pirate is thrown overboard and the process is repeated with the pirates that remain.

As pirates tend to be a bloodthirsty bunch, if a pirate would get the same number of coins if he voted for or against a proposal, he will vote “no” so that the pirate who proposed the plan will be thrown overboard.

Larry the Logical Pirate is the 3rd oldest pirate. He opens the treasure chest, gives 98 coins to Eric the Eldest Pirate, one coin to himself, and 1 coin to Yuval the Youngest Pirate. Patrick the Practical Pirate and Barry the Bystander Pirate, 2nd and 4th oldest, respectively, receive no coins. He proudly announces that this is the equilibrium solution, which has the added benefit of keeping everyone alive.

The other four pirates briefly exchange glances and then throw Larry the Logical overboard. They agree to give each remaining pirate one share of the treasure; Eric the Elder Pirate, now appointed captain, receives one extra share for his leadership, and Patrick the Practical Pirate, now appointed quartermaster, receives an extra half-share for handling the logistics of booty-splitting.

II.

A large number of prisoners are held in a prison run by a sadistic Logician-Warden. One day the Logician-Warden tells the prisoners he will be forcing them to participate in a game. There is a room in the prison that contains two switches, one red and one blue, that can be set “ON” or “OFF”. One by one, prisoners will be brought into the room and required to flip exactly one of the switches. Prisoners will be sampled with replacement, that is, each prisoner has a chance of being selected for each trip as long as the game continues (however, the probabilities are not necessarily the same for all prisoners, nor are they necessarily stable over time). The game ends when one of the prisoners tells the warden that every prisoner has visited the room at least once. If the prisoner is correct, everyone is freed. Otherwise, everyone is killed. The switches can start in any configuration. Once the game begins, the prisoners will not be allowed any communica...
Evolutions Building Evolutions: Layers of Generate and Test
by plex — 2021-02-05T18:21:28.822Z

[Epistemic Status: Early layers and the general framework seems firm (and substantially Dennett-inspired), later parts are more speculative, but I've found them to be useful thinking tools.]

A plant is infested by caterpillars, and following its genetic programming responds by producing a chemical which attracts predators specific to those caterpillars.
A pigeon explores its environment semi-randomly, until it pecks at a lever which delivers a morsel of food, so it learns to upregulate its tendency to peck at that lever.
A child watches a member of their tribe foraging wild plants, and mimics the behaviours which have proven themselves adaptive in that environment. In rare cases they will modify the meme in a way which turns out to be beneficial and spreads.
A Dank EA Memes contributor reads a lot of positively received content, builds a model of which kinds of memes are popular, and uses it to create something new which is successful in that ecosystem.
As part of the training of an advanced AI system, it searches a large space of possible programs, which includes an optimizing program that realizes it's inside of a search process and and causes all sorts of problems.

Each of these is a directed search, or optimization process, with all but the first running in a framework created by a larger, less directed, search. This pattern, where an evolution-like search builds another evolution-like search, seems ubiquitous when studying intelligence.

Within each layer of process, there is a Generator and a Test, which create and filter ways of responding to stimuli, respectively. The generator creates a population of possible outputs to sample from randomly (though weighted by prior search processes), while the test checks the generated output against some criteria to see whether to use it (and weight it higher in the population for future generations).

If you take one thing away from this post, make it:

All directed evolutions were created by the fruits of earlier, usually less directed, evolutions. Aside from the lessons about how to explore picked up by prior searches, they must all operate via chance, because at the foundations chance is the only thing available to build with. As lessons are accumulated and better evolutions are built, the searches become faster and more efficient.

Darwinian Processes

These processes run purely on biological evolution: they gene...
Notes on Frugality
by David Gross — 2021-02-05T16:54:54.085Z

This post examines the virtue of frugality (a.k.a. thrift, economy). I hope it will be helpful to people who want to know more about this virtue and how to nurture it.

What is this virtue?

Frugality concerns the efficiency of your economic choices: how you earn you living, husband resources, spend money, and the like. A frugal person is conscious of the value of their resources and of their time, and makes trade-offs wisely.

Frugality is usually contrasted with its vice-of-deficiency: extravagance, prodigality, squandering, wastefulness, being a spendthrift. In this it is allied with virtues like efficiency, moderation, prudence, and simplicity. Benjamin Franklin paired frugality with industry as the key virtues for financial security: “[W]aste neither time nor money, but make the best use of both. Without industry and frugality nothing will do, and with them everything.”[1]

Sometimes that same contrast is made not to the vice itself, but to the likely consequences of that vice: indebtedness, dependence on others, poverty. If you are not frugal, you may suffer for it, and this may also include having to lower your standards of dignity. Franklin also had a nice bon mot about that: “It is hard for an empty sack to stand upright.”[2]

Frugality sometimes is also contrasted with its vice-of-excess, which goes by names like miserliness, niggardliness, or stinginess. There is some tension between frugality and the virtues of generosity and magnificence.

Occasionally in older works you will see a virtue called “frugality” that is described in a way that makes it sound more like temperance. (For example, in one translation of Cicero’s Tusculan Disputations, he describes frugality as combining fortitude, justice, and prudence in one package, says the cowardly man cannot be called frugal, and says frugality’s “peculiar property seems to be to govern and appease all tendencies to too eager desire after anything, to restrain lust, and to preserve a decent steadiness in everything”[3] — all of which doesn’t really resemble how we use the term “frugal” today.)

About frugality

“Men don’t understand how great a revenue sparingness is.” ―Cicero[4]

Frugality is a facet of life optimization. A frugal person lives efficiently, in a streamlined way that enables them to accomplish more with less effort.

Time is money

Most every adult gets through life by exchanging resources they control for...
What does flow feel like for "mental activities"?
by Hazard — 2021-02-05T14:33:59.579Z

Athletes often describe flow states with:

"My limbs just move on their own."
"Intense, totally absorbing focus."
"My mind is totally blank. I'm not thinking at all, just doing."

I'm very interested homing in on this "thinking" that is often reported absence in flow. It clearly points to something other than "you brain firing on all cylinders doing crazy computations" because that is certainly all happening while playing an intense sport. I figured it would be helpful to compare against flow states in more prototypical "mental" activites.

So with all that context, when you are in peak experience/flow doing anything highly mental (math, coding, chess, etc), what is it like? And what sorts of "thinking" aren't happening, and what sorts of "thinking are happening?
Creating AGI Safety Interlocks
by Koen.Holtman — 2021-02-05T12:01:46.221Z

In the third post in this sequence, I will define a counterfactual
planning agent which has three safety interlocks.

These interlocks all aim to enable and support agent oversight: the
oversight that is needed when we equip a powerful AGI agent with a
reward function for which we are pretty sure that it does not fully
capture all our intentions.

Safety interlocks

A safety interlock is a mechanism that stops a machine when certain
conditions are met. This post constructs an agent with three example safety
interlocks.

Emergency stop button

The first interlock is a manually operated emergency stop button,
which will stop the agent when pressed. This allows human oversight
to shut down the agent whenever particularly unexpected or unwanted
behavior of the agent is discovered. To make this work as a safety
mechanism, we need to control any incentive that the agent may have to
prevent its stop button from being pressed.

In the AGI safety community. the problem of removing any incentive in
an agent to prevent its stop button from being pressed is generally
known as the problem of
Corrigibilty
or the off-switch problem. I
don't solve this problem fully in the design below. The design
suppresses what I call the direct incentive, but I also show that
several indirect incentives may remain. This observation motivates
the other two safety interlocks.

I model the stop button by using a function stop_pressed(s) which
returns true if the stop button is pressed in the current state s
of the agent environment.

Runtime based safety interlock

The second interlock, realized by the t>Tmax below, automatically
stops the agent when it has run for Tmax time steps. This adds
safety in the case that human oversight becomes incapacitated.

Power based safety interlock

The third interlock, realized by Up>Umax, protects against
the risk of the agent undergoing an intelligence explosion. If an
agent undergoes a rapid and unexpected increase in power and/or
intelligence, the agent may start to innovate so rapidly that it
overwhelms the capacity of human oversight to press the emergency stop
button, or apply any other corrective steering, fast enough to prevent
unwanted outcomes. To protect against this risk, the interlock stops
the agent when the metric Up, which measures the agent's
ability to achieve goals, gets too high.

Measuring and limiting agent power

In ...
Any layperson-accessible reference posts on how to operationalize beliefs ?
by Optimization Process — 2021-02-05T07:26:07.558Z

I'd like to find a shortish collection of advice, "do"s/"don't"s, aimed at... rationalist-adjacent folk?... for operationalizing beliefs into concrete anticipated experiences. My searches so far have been fruitless; I'd be grateful for any pointers!

(Context: I'm toying with a project aimed at encouraging people to operationalize their beliefs, i.e. turning vague beliefs like "Covid B117 will be disastrous" into relatively concrete predictions like "If the CDC publishes Covid sequencing data for any time period after June 1, it will show >70% of infections being B117 or a derivative." I would like to have one or more links handy to help people do it well.)

(Things that are almost what I want: Making Beliefs Pay Rent is great, but is more of an exhortation than a how-to; Keep Your Beliefs Cruxy likewise; many LW posts assume familiarity with double-crux; and Wikipedia:Operationalization is aimed at scientists.)
Ways of being with you
by KatjaGrace — 2021-02-05T07:00:15.383Z

Suppose you want to get in touch with your care for someone, or to feel empathy for them. One way is to imagine what it is like to be them, looking out of their eyes and feeling what they are feeling. For instance, if your friend did something frustrating, you might imagine the experience of choosing under pressure that might have led to it, and you might thereby feel warmth toward them and sympathy for their error.

But I think it is more common to do something else, kind of like imagining them from the outside, but such that their mental content is also somehow accessible. For instance, you might imagine them hurrying around, being frazzled and not sure what to do, trying. And it’s not just that they look like they might be those things - in imagination, it is available that they are. While you aren’t in their mind, you also aren’t just observing its consequences and inferring things about it from a distance. You experience their feeling.

But what is it to experience a person’s sadness, say, without experiencing it as an aspect of their experience? i.e. as something that can requires an imagining of their mental experience to imagine, in the way that imagining the wetness of a dog would seem to require—or be greatly aided by—imagining the dog?

In the everyday togetherness of being with someone, I am also rarely actually picturing the world through their eyes, and yet I am often doing some kind of ‘seeing things from their perspective’ and some kind of ‘being with them’ that feels like relatively direct contact with their mind. (What is it to experience a mind, if not to look at the conscious experience that is a mind’s main deal?) I am engaging with them very differently from how I would engage with an object, or with a set of observations of a distant person. Is it strange that your experience of being you is not the thing that I focus on when I’m experiencing you, together with you?