trentmkelly/LessWrong-43k huggingface dataset:
Positive values seem more robust and lasting than prohibitions
by TurnTrout — 2022-12-17T21:43:31.627Z

Originally a shortform comment.

Imagine we train an AI on realistic situations where it can kill people, and penalize it when it does so. Suppose that we successfully instill a strong and widely activated "If going to kill people, then don't" value shard.

Even assuming this much, the situation seems fragile. See, many value shards are self-chaining. In The shard theory of human values, I wrote about how:

A baby learns "IF juice in front of me, THEN drink",
The baby is later near juice, and then turns to see it, activating the learned "reflex" heuristic, learning to turn around and look at juice when the juice is nearby,
The baby is later far from juice, and bumbles around until they're near the juice, whereupon she drinks the juice via the existing heuristics. This teaches "navigate to juice when you know it's nearby."
Eventually this develops into a learned planning algorithm incorporating multiple value shards (e.g. juice and friends) so as to produce a single locally coherent plan.
...

The juice shard chains into itself, as its outputs cause the learning process to further reinforce and generalize the juice-shard. This shard reinforces itself across time and thought-steps.

But a "don't kill" shard seems like it should remain... stubby? Primitive? The "don't kill" shard can't self-chain into not doing something. If you're going to kill someone, and then don't because of the don't-kill shard, and that avoids predicted negative reward... Then maybe the "don't kill" shard gets reinforced and generalized a bit because it avoided negative reward (and so reward was higher than predicted, which I think would trigger e.g. a reinforcement event in people).

But—on my current guesses and intuitions[1]—that shard doesn't become more sophisticated, it doesn't become reflective, it doesn't "agentically participate" in the internal shard politics (e.g. the agent's "meta-ethics", deciding what kind of agent it "wants to become"). Other parts of the agent want things, they want paperclips or whatever, and that's harder to do if the agent isn't allowed to kill anyone.

Crucially, the no-killing injunction can probably be steered around by the agent's other values. While the obvious route of lesioning the no-killing shard might be reflectively-predicted by the world model to lead to more murder, and therefore bid against by the no-killing shard... There are proba...
What we owe the microbiome
by weverka — 2022-12-17T19:40:11.462Z
Link: https://forum.effectivealtruism.org/posts/DdzSEFBEb6rtfChpN/what-we-owe-the-microbiome

Your microbiome is a bunch of microorganisms living in your gut and other epithelial places. You have a symbiotic relation with this population. You feed it. It helps you digest.

Your microbiome population talks to you. It creates chemical signals that affect digestion and possibly your feeling of hunger. The extent of the influence of your microbiome on your brain is not well known, but the pathways for that influence are. The gut microbiome has been shown to produce various chemicals and signaling molecules that can influence the function of the digestive system and the immune system. It produces short-chain fatty acids that may cross the blood-brain barrier.

When it is out of balance, it makes you ill. Sometimes it threatens your life. We must try our best to avoid getting a disease that requires antibiotics. Few of us succeed. Sometimes your only defense is a broad-spectrum antibiotic.

In the game of turning the tables, we ask that our overlords treat us as we would treat our underlings.

If we create an AI so much more capable than ourselves, we become like the microbiome to the AI.

Here are a couple options to explore: 1)Avoid creating an AI so great that we are like its microbiome. Or 2) align humans to the interest of the AI so that a subpopulation of humans doesn't infect the AI to the extent it has to invoke the equivalent of a broad spectrum antibiotic.

The second one is expressed as a reversal of the perspective of alignment goals. It is a perspective that may become necessary if we fail to do the first.

Wash your vegetables.
Why write more: improve your epistemics, self-care, & 28 other reasons
by KatWoods — 2022-12-17T19:25:42.977Z

Writing is like exercise. It’s just good on so many levels.

Writing can be high impact
Writing can be inspiring
Writing can help you learn and grow
Writing can help your career
Writing can be meaningful and deeply satisfying
Writing can get you into flow
And also, just like with exercise, almost nobody writes enough

Whenever I get back into the habit of writing, I always think I should write more. It’s one of my favorite things.

Yet, much like exercise is for most people, I tend to have little flares of interest, where I get really into it for a week or two, then lose steam, and it just becomes a dormant blog again. You might have had a similar experience, and maybe even have a cobwebby blog or two out there. Perhaps you have some really cool half-finished google docs that you never quite got around to finishing and publishing.

It’s a funny thing though, about writing being like exercise. Because do you remember what exercise used to be called when you were a child?

Play.

And I remembered on a recent vacation I took, where I had the

slack

to remember, that writing is play. I fell back in love with writing then, typing away furiously on the beach.

“I remembered that writing is play”

Not only was it intrinsically fun, but I also loved that it felt like I was making a difference, writing about important topics. That my

recommendations

were helping people, whether it be making them happier or making them higher impact, or sometimes, if I was lucky, both.

To help myself (and perhaps you) remember this and all the other reasons to write, I’ve decided to write about all the reasons I have to write.

Reading the list will inspire me and hopefully others. Publishing it will publicly commit me to writing, which will make it more likely. Even better, it could potentially re-inspire some people, perhaps you, to start writing for the forums again. Or maybe even it’ll get you excited to try your hand at writing publicly, even though you’ve never done it before.

Epistemic status: motivational

Epistemic status: motivational. An explanation of my personal experience that doesn’t generalize to all people. This isn’t meant to be a nuanced look into the pros and cons of writing.

It’s meant to inspire a subset of the population (and myself!) to write more. None of this applies to all writing or all people. It’...
Looking for an alignment tutor
by JanB — 2022-12-17T19:08:10.770Z

Hey, this is me. I’d like to understand AI X-risk better. Is anyone interested in being my “alignment tutor”, for maybe 1 h per week, or 1 h every two weeks? I’m happy to pay.

Fields I want to understand better:

Anything related to prosaic AI alignment/existential ML safety
Failure stories/threat models

Fields I’m not interested in (right now):

agent foundation
decision theory
other very mathsy stuff that’s not related to ML

My level of understanding:

I have a decent knowledge of ML/deep learning (I’m in the last year of my PhD)
I haven’t done the AGI Safety Fundamentals course, but I just skimmed it, and I think I had independently read essentially all the core readings (which means I probably have also read many things not on the curriculum). I’d say I have a relatively deep understanding of a majority (but not all) of the content in this curriculum.
Similarly for the AGI Safety Fundamentals 201, excluding the tracks

Example questions I wrestled with recently, and I might have brought up during the tutoring:

It seems to me that our currently level of outer alignment tools (RLHF + easy augmentation) is enough to solve the outer alignment problem sufficiently well so that humans don’t end up dead or disempowered (conditional on slow takeoff); and then we can solve further outer alignment problem as the come up, with iteration and regulation. So I basically think that the core of the alignment problem, at the moment, is inner alignment + deceptive alignment. What am I missing? (I read Christiano’s “Another Outer Alignment Failure Story”, but I still have this question.)
I understand that a reward maximiser would wire-head (take control over the reward provision mechanism), but I don’t see why training an RL agent would necessarily end up in a reward-maximising agent? Turntrout’s Reward is Not the Optimisation Target shed some clarity on this, but I definitely have remaining questions.
Is the failure mode describe in Ajeya’s Without Specific Countermeasures an inner alignment failure, or an outer alignment failure (I think it’s both).

You don’t need to have very crisps answers to these to be my tutor, but you should probably have at least some good thoughts.
How to Convince my Son that Drugs are Bad
by concerned_dad — 2022-12-17T18:47:24.398Z

Hello.

My son (16m, henceforth referred to as John) has monologued about this site a few times over the past couple of months, so I figured, based on my brief impression of the community, you might be able to help me with an issue. Given the topical nature here, I am not sure if this is an appropriate type of post to make, however it might be a useful place to make an appeal. Worst case, this gets taken down for incompliance.

John has always been a little too obsessed with his computer, but things really came to a head when he found this whole subcommunity. For a couple of weeks, I'd regularly notice as he spent hours just sitting in his room scrolling through blog posts and papers and forums. While damaging to health, this doesn't seem unusual among teenagers and I try to let him make his own decisions as I'm sure eventually it will taper out, and that's not my main issue.

First off: I've noticed some positive changes since he started discussing effective altruism and rationality and such, though I don't know whether to attribute that to this site or just maturing. Thank you all for that. But there are some worrying ideas he seems to have gotten as well, centering around romanticization of drug use, specifically "nootropic"-style with a focus on amphetamines and psychedelics.

One day he just came up to me and began engaging in a discussion about the merits of doing drugs. I'll give you some approximate quotes so you can understand about how this went:

"Why the hell is LSD criminalized everywhere? There are NO negative side effects *proceeds to compare to prescription drugs*"
"Psychedelics and amphetamines are classes of drugs I plan to do, I've done extensive (3 hr - sigh) research on their side effects and chemical compositions and all seems fine! People take Adderall all the time and I can easily get a script, I've read the entire DSM 5!"
"What's the difference between you drinking alcohol or coffee and me taking amphetamines and doing LSD? Drugs. are. drugs." "Alcohol isn't synthetic? Well then, what about peyote?"
"Why not try heroin if the purpose of life is to optimize happiness assuming heroin provides proportionally more even if for a shorter amount of time?" (!)
"Look, here's this site where someone (a 'gwern' if I recall correctly) did this scientifically! They're fine, I want to do that too!"
"How will I acquire the drugs? I'll just ... uhhh ..

...
Ordinary human life
by David Hugh-Jones — 2022-12-17T16:46:10.818Z
Link: https://wyclif.substack.com/p/ordinary-human-life

I woke up early and read a book. At 7 I turned on the radio. A bit later, I got up, fed and walked the dog, playing a game on my smartphone. I skinned a mango for breakfast and started work. A friend came round for lunch and we talked about a paper we’re working on. Tonight would normally be choir, but we had the concert last week so there’s a break. Instead I’ll probably watch something on Netflix.

This is ordinary human life. Parts of it are new. I couldn’t have watched Netflix even five years ago, and when my parents were born they could not have watched any television at all. Radio is barely a century old. Mangos were rarely seen in UK supermarkets until perhaps a generation ago. My smartphone game uses augmented reality, and could be part of the Metaverse which Facebook is betting on.

Other parts are older. Western choral music goes back to about the 15th century, if you are counting from the invention of harmony. Music in general is much older than that, originating at least 40,000 years ago. Hunter-gatherers share food — the !Kung say that eating alone is for lions, not men — so again meals are a very old institution, and of course eating itself is a biological necessity. Dogs as pets? Thousands of years old.

Ordinary human life embodies our values. Everything good or bad that happens to us, happens within our lives. When we talk about “good” or “bad” we are talking about them in the context of our lives as we know them. Beyond that context, the concepts make little sense. What would it mean for a bacterium or neutron star to be happy? When we extend the ideas beyond humans, we do it to things that are recognizably similar to us. My dog is excited because he is bouncing around and vocalizing. A factory chicken must be suffering, because it lives in intolerable conditions for creatures like us.

Good and bad are local concepts. They are like signposts around a town. Outside the town, the signs would no longer refer to anything meaningful. Abstract discussions of “utility”, outside a human context, devolve into mathematical games.

At different points in my day I experienced different moods. I was bored or elated, engrossed in a technical problem, relaxing with friends, or obsessing angrily about the past. We can sum up this myriad of emotions in a single dimension, as better or worse. Some days are better than others. I could even put numbers on the day...
Predictive Processing, Heterosexuality and Delusions of Grandeur
by lsusr — 2022-12-17T07:37:39.794Z

Predictive processing is the theory that biological neurons minimize free energy. In this context, free energy isn't physical energy like the energy in your laptop battery. Instead, free energy is an informatic concept. It is useful to think about free energy as a single variable balancing prediction error and model complexity. Minimizing free energy balances minimizing prediction error against minimizing model complexity. You can also think about it as minimizing surprise.

Biological neurons have inputs and outputs. Each neuron receives inputs from one set of neurons and sends outputs to another set of neurons. One way to minimize prediction error is to fire right after receiving inputs, but an even better way to minimize prediction error is for a neuron to anticipate when its input neurons will fire and then fire along with them. Firing in-sync with its inputs produces zero prediction error instead of just a small prediction error.

It has been shown that predictive processing is asymptotically equivalent to backpropagation. Everything that can be computed with backpropagation can be computed via predictive processing and vice versa.

A Simple Clock

Suppose you take a 3-dimensional blob of neurons programmed to minimize local prediction error and you attach them to a sinusoidal wave generator. The neural net has no outputs—just this single input. At first the neurons close to the wave generator will lag behind the sinusoidal wave generator. But eventually they'll and sync up with it. Our neural net will have produced an internal representation of the world.

Now suppose you plug a sinusoidal wave generator into the left end of the neural net and a square wave generator into the right end of the neural net. The left end of the neural net will wave in time with the sinusoidal wave generator and the right end of the neural net will beat in time with the square wave generator. The middle of the net will find some smooth balance between the two. Plug more complicated inputs into our neural net and it will generate a more complicated representation of the world.

Simple Behavior

Consider a neural net with one input and one output. The output of the brain is connected to the input of a static noise generator and the static noise generator's output is connected to the brain's input. The brain is a free energy minimizer. Static noise has high free energy. The brain will t...
[Link] Escape the Echo Chamber (2018)
by CronoDAS — 2022-12-17T06:14:47.301Z
Link: https://aeon.co/essays/why-its-as-hard-to-escape-an-echo-chamber-as-it-is-to-flee-a-cult

Related to: Epistemic Learned Helplessness, Bounded Distrust

This seems like a particularly insightful essay that expresses very well an idea I've also had: the difference between people with relatively normal beliefs and people with crazy-sounding beliefs is often that the people with crazy-sounding beliefs trust different authorities, and then "rationally" construct crazy-sounding beliefs based on that trust.

A few key paragraphs:

[T]here are two very different phenomena at play here, each of which subvert the flow of information in very distinct ways. Let’s call them echo chambers and epistemic bubbles. Both are social structures that systematically exclude sources of information. Both exaggerate their members’ confidence in their beliefs. But they work in entirely different ways, and they require very different modes of intervention. An epistemic bubble is when you don’t hear people from the other side. An echo chamber is what happens when you don’t trust people from the other side.

...

Crucially, echo chambers can offer a useful explanation of the current informational crisis in a way that epistemic bubbles cannot. Many people have claimed that we have entered an era of ‘post-truth’. Not only do some political figures seem to speak with a blatant disregard for the facts, but their supporters seem utterly unswayed by evidence. It seems, to some, that truth no longer matters.

This is an explanation in terms of total irrationality. To accept it, you must believe that a great number of people have lost all interest in evidence or investigation, and have fallen away from the ways of reason. The phenomenon of echo chambers offers a less damning and far more modest explanation. The apparent ‘post-truth’ attitude can be explained as the result of the manipulations of trust wrought by echo chambers. We don’t have to attribute a complete disinterest in facts, evidence or reason to explain the post-truth attitude. We simply have to attribute to certain communities a vastly divergent set of trusted authorities.

Listen to what it actually sounds like when people reject the plain facts – it doesn’t sound like brute irrationality. One side points out a piece of economic data; the other side rejects that data by rejecting its source. They think that newspaper is biased, or the academic elites generating the data are corrupt. An echo chamber doesn’t destroy their members

...
"Starry Night" Solstice Cookies
by maia — 2022-12-17T05:31:47.315Z

There are no traditional aspiring-rationalist foods that I know of. Here's a potential candidate for one. I started with a friend's family recipe and adjusted it to be vegan and slightly more thematic.

The white mini chocolate chips symbolize humanity (the light in the world) in the darkness of the universe (the bitter chocolate cookie). The dried cherries can be... nebulae.

Makes around 12 large cookies.

Ingredients

1 Tbsp instant espresso powder
1 Tbsp boiling water
133g (5/8c) coconut oil
176g (1c packed) brown sugar
80g (1/4c) mashed banana
1/4 tsp vanilla extract
120g (1c) flour
90g (3/4c) unsweeted cocoa powder
1 tsp baking soda
2 tsp baking powder
1/4 tsp salt
100g (3.75oz, 3/4c) chopped sweetened dried cherries
80g (3oz, 1/2c) mini white chocolate chips (vegan)

Directions

Preheat oven to 350 degrees and line baking sheets with parchment paper.

Mix instant espresso powder and water in a small bowl, and reserve.

Cream the coconut oil, then add brown sugar and cream together. Add mashed banana and vanilla and beat until thoroughly combined. Add reserved espresso/water mixture and beat until combined.

In a separate bowl, mix the flour, cocoa powder, baking soda, baking powder, and salt, then combine with the wet ingredients. (Or if you're lazy like me, put them on top of the wet ingredients and mix the dry ingredients with a fork before mixing fully.) Fully combine until a dough forms.

Mix in chopped dried cherries and mini white chocolate chips.

Form dough into flattened balls and place on cookie sheets. Bake for about 13 minutes.

Remove from oven and cool for 10 minutes before transferring cookies to a wire rack to cool completely.

These taste slightly better the next day; they get a bit chewier.
There have been 3 planes (billionaire donors) and 2 have crashed
by trevor — 2022-12-17T03:58:28.125Z

I'd rather not go into the details about which billionaires are which, so if it's actually 4 and 3 or 6 and 4, then that may or may not be debatable. I'm much more worried about whether MIRI survives the decade.

It seems to me like this is a good place to figure out ways to handle the contingency where the final billionaire donor crashes and burns, not how to get more billionaires or retain existing ones (and certainly not preventing them from getting bumped off). Signalling unlimited wealth might be valuable for charisma, but at the end of the day it's better to admit that resources are not infinite if it means living longer than 30 years.

I've met some people who were recently funded to start a group house in a rural town in Vermont to do AI safety work, since rents in Vermont are incredibly low, which makes it one of the most cost-effective places in the US to research AI safety. Ultimately, the goal is that people with smaller and smaller amounts of savings could take a sabbatical to a Vermont group house and do research for free for 2-10 years, without working full-time or even part-time at some random software engineering job.

The main problem here is network effects. I don't remember the details, but they will have to drive at least 3 hours to Boston once a month (and probably more like 5-6 hours). Otherwise, they will be effectively alone in the middle of nowhere, totally dependent on the internet to exchange and verify ideas with other AI safety-minded people (and all the risks entailed by filtering most of your human connection through the internet).

The main problem with the Vermont group house is that there's currently only three of them. If there were ten really smart people in Vermont researching existential risk, then it would be easier to handle the isolation with, say, shoulder advisors. Plus, if it were up to me, they'd be in rural Virginia (or parts of West Virginia) 5-6 hours away from Washington, D.C., not Boston, although the people who picked Vermont and funded it might know things I don't (disclaimer: they had the idea first, not me, I only discovered the brilliance behind it after meeting a Vermont person).

Ultimately, though, it's obviously better for AI-safety-affiliated people to be located within the metropolitan areas of San Francisco, New York, Boston, Washington D.C., and London. New people and new conversations are the lifeb...