trentmkelly/LessWrong-43k huggingface dataset:
Social Capital Paradoxes
by abramdemski — 2020-09-10T18:44:18.291Z

[Credit for horizontally transmitting these ideas to my brain goes mostly to Jennifer RM, except for the bits at the end about Bowling Alone and The Moral Economy. Apologies to Jennifer for further horizontally spreading.]

Vertical/Horizontal Transmission

The concept of vertical and horizontal transmission felt like a big upgrade in my ability to think about cooperative/noncooperative behavior in practice. The basic idea is to distinguish between symbiotes that are passed on primarily along genetic lines, vs symbiotes which are passed on primarily between unrelated organisms. A symbiote which is vertically transmitted is very likely to be helpful, whereas a symbiote which is horizontally transmitted is very likely to be harmful. (Remember that in biology, "symbiote" means any kind of close relationship between different organisms; symbiosis which is useful to both organisms is mutualistic, while symbiosis which is useful to one but harmful to another is parasitic.) (This is discussed here on LW in Martin Sustrik's Coordination Problems in Evolution.)

We can obviously generalize this quite a bit.

Infectious diseases tend to be more deadly the higher their transmission rate is. (Diseases with a low transmission rate need to keep their hosts relatively healthy in order to make contact with other potential hosts.)
Memes which spread vertically are more likely to be beneficial to humans than memes which spread horizontally (at least, beneficial to those human's genes). Religions which are passed through family lines have an incentive to encourage big families, and include ideas which promote healthy, wealthy, sustainable living. Religions which spread primarily to unrelated people have a greater incentive to exploit those people, squeezing every last drop of proselytization out of them.
Long-term interactions between humans are more likely to be mutualistic, while short-term interactions are more likely to be predatory.
In general, cooperative behavior is more likely to arise in iterated games; moreso the more iterations there are, and the more probable continued iteration is.

Vertical transmission is just a highly iterated game between the genes of the host and the genes of the symbiote.

Horizontal Transmission Abounds

Wait, but... horizontal transmission appears to be the norm all over the place, including some of the things I hold most dear!

Religion and trad

...
Do mesa-optimizer risk arguments rely on the train-test paradigm?
by Ben Cottier — 2020-09-10T15:36:37.629Z

Going by the Risks from Learned Optimization sequence, it's not clear if mesa-optimization is a big threat if the model continues to be updated throughout deployment. I suspect this has been discussed before (links welcome), but I didn't find anything with a quick search.

Lifelong/online/continual learning is popular and could be the norm in future. I'm interested in how that (and other learning paradigms, if relevant) fits into beliefs about mesa-optimization risk.

If you believe the arguments hold up under a lifelong learning paradigm: is that because there could still be enough time between updates for the mesa-optimizer to defect, or some other reason? If you believe the train-test paradigm is likely to stick around, why is that?
On Chesterton's Fence
by trentbrick — 2020-09-10T15:10:42.321Z

TLDR; Chesterton’s Fences are important, and very hard to identify/evaluate. With finite time, bountiful stupidity and inflated egos, it is too easy to not look deeply enough at existing ways of doing things and understand why they are the way they are before attempting to “fix” them. Reading Secrets of our Success and Seeing Like a State has strengthened my prior to dig deeper on why things are done, in proportion to how long they have stood the tests of time. Writing this piece has helped me develop a framework (in the form of fitness landscapes) to think about Chesterton’s Fences and how uncovering both the motivations and mechanisms behind them is often intractable, requiring clever trial and error along with the acceptance of unfortunate, unintended consequences.

Chesterton's Fence states that if you encounter a fence in the middle of nowhere, you should stop and first understand why it was put there before taking it down. There is probably a good reason the fence is there in the first place, and finding out the hard way might be really bad and irreversible. Chesterton's Fence was the original motivation for the creation of Slate Star Codex and is a principle I have thought a lot about recently while reading Secrets of our Success and Seeing Like a State.

Both of these books convey endless appreciation for the complexity, nuance, and unintended consequences that local expertise accounts for and the naive outsider ignores at risk of their own demise. Cherry picking some fascinating examples:

"As one of the world’s staple crops, manioc (or cassava) is a highly productive, starch-rich tuber that has permitted relatively dense populations to inhabit drought-prone tropical environments. However, depending on the variety of manioc and the local ecological conditions, the tubers can contain high levels of cyanogenic glucosides, which release toxic hydrogen cyanide when the plant is eaten. If eaten unprocessed, manioc can cause both acute and chronic cyanide poisoning. Chronic poisoning, because it emerges only gradually after years of consuming manioc that tastes fine, is particularly insidious and has been linked to neurological problems, developmental disorders, paralysis in the legs, thyroid problems (e.g., goiters), and immune suppression. These so-called ”bitter” manioc varieties remain highly productive even in infertile soils and ecologically marginal

...
In 1 year and 5 years what do you see as "the normal" world.
by jmh — 2020-09-10T12:47:35.497Z

We all have a mental image of pre-COVID normal.

I often hear people saying "I cannot wait to get back to normal." or asking "When will we get back to normal?" I think that is an expectation that is sure to be disappointed. I suspect that is the case for most who read this site.

I'm curious about the mental image of the near future normal that is held here. I'll list a few areas for thoughts but also don't think anyone should be limited in any thoughts they want to share.

1) International Travel -- can be general, tourism related or business related

2) Entertainment -- theater/movies, live sports and concerts. One thought here might be a move to more open air venues rather than indoors.

3) Social interactions in general. Does some of the zenophobia that has occurred persist or die away (say due to vaccines).

4) Will vaccines change things that much?
Taking Hammertime Final Exam by efim
by efim — 2020-09-10T10:18:18.459Z

https://www.lesswrong.com/posts/AxjFRgNStopAc93jB/taking-hammertime-final-examI am attempting to write about:

A helpful framework
An exercise that might generally help in life
A blindspot

It was hard to choose topics between several areas, I also strongly considered ideas that were helpful for me from NVC point of view, but for now all three are related to meditation:

Part of a framework for low-level mental actions: (Intention is before all)
Any action or thought (a mental action) is preceded by some intention that is generated within the mind in reaction to some input.
In that framework doing things looks like this:
1. there is a need or a desire for something
(based on internal or external information input)
2. an intention to get to this goal is generated
3. actions are planned and executed
4. we get new information input for result of the action
5. if goal is not achieved, but intention is still present -
new actions are planned and executed
And "learning things" means:
- Generating and holding an intention to get to some state
- The mind as a whole will continue planning and executing actions until they are over-trained and automatic
So "trying harder" would help much less than "holding, supporting an intention and making it more visible"
A common pitfall would be to learn to punish self.
I.e generating an intention to hurt self with deprecation or something else on encountering information input for a failure state.
This could lead to mind learning to avoid the input - to not look at the problem, or avoid it.
While learning an intention of "being friendly to self" could very well lead to more sustained intentions,
which interacting with other learning techniques or techniques for applied rationality could lead to trying different ways of achieving the goal

Unlearning self-punishment
This exercise can be included as a cornerstone of a meditation practice in different schools, but can be helpful on it's own.
Based on the Input -> Intention -> Action framework
Our learned tendency to self-deprecate, or self-punish is a strongly entrenched intention to act this way on encountering specific inputs.
The exercise to unlearn it could look like this:
1. select the base activity
could be your house chores, could be work or studying
easy model activity is watching breath and returning to it when you catch yourself distracted
2. set ti...
Safety via selection for obedience
by Richard_Ngo — 2020-09-10T10:04:50.283Z

In a previous post, I argued that it’s plausible that “the most interesting and intelligent behaviour [of AGIs] won’t be directly incentivised by their reward functions” - instead, “many of the selection pressures exerted upon them will come from emergent interaction dynamics”. If I’m right, and the easiest way to build AGI is using

open-ended

environments and reward functions, then we should be less optimistic about using scalable oversight techniques for the purposes of safety - since capabilities researchers won’t need good oversight techniques to get to AGI, and most training will occur in environments in which good and bad behaviour aren't well-defined anyway. In this scenario, the best approach to improving safety might involve structural modifications to training environments to change the emergent incentives of agents, as I’ll explain in this post.

My default example of the power of structural modifications is the evolution of altruism in humans. Consider

Fletcher and Doebeli’s

model of the development of altruism, which relies on assortment in repeated games - that is, when players with a tendency to cooperate end up playing together more often than random chance predicts. In humans, some of the mechanisms which lead to assortment are:

Kin recognition: we can tell who we share genes with.
Observation of intentions or previous behaviour: these give us evidence about other agents’ future behaviour.
Costly signalling: this can allow us to reliably demonstrate our future altruism.
Communication of observed information: once one person has made an observation, it can be shared widely.
Flexible interactions: we can choose who to assort with in different interactions.

I claim that, given this type of understanding of the evolution of altruism, we can identify changes to high-level properties of the human ancestral environment which would have made humans significantly more altruistic. For example, human cognition is not very transparent, and so it's relatively difficult for each of us to predict the intentions of others. However, if we had direct observational access to each other's brains, cooperation would become easier and more advantageous. As another example, if we had evolved in environments where we frequently cooperated with many different species, then we’d likely feel more broadly altruistic today.

To be clear, I don’t think these types of interventions ...
Is there a way to view "read" LessWrong posts?
by BeanSprugget — 2020-09-10T03:58:37.626Z

I want to be able to see the LessWrong posts that I've already read/visited, like in a list. I mainly just want to be able to re-save some ones that I really liked but deleted from my bookmarks.

I didn't vote on all the posts I read (it's kind of like a habit I had to build), so I can't do that (although, that would be nice too) (I guess that would be a related question).
Resist epistemic (and emotional) learned helplessness!
by DirectedEvolution — 2020-09-10T02:58:24.681Z

Antonio Pascuale-Leone talks about how to get over the end of a relationship. Alison Ledgerwood helps us understand how framing impacts our response to events. Both of these researchers present us with frameworks to structure the narrative of our own lives. The key messages are that we can benefit by translating our emotional pain into some broad existential categories, and that searching for an interpretation of events that acknowledges the positive aspects is difficult but helpful. These mental habits will be contrary to our instincts, and are worth practicing.

My personal experience in using this advice for dealing with a breaking is that it was helpful, and sped me along a route of emotional recovery that, in retrospect, I am happy I took. I want to explore the nuance around them here.

Although the "letting go" framework that Pascuale-Leone presents and the positive reframing discussed by Ledgerwood are often the perfect mental tools for doing emotional work, they don't do the whole job on their own. Part of the difficulty of emotional work is that it requires a conscious understanding of the formal problem that underlies the emotional feeling. When you find the perfect tool for your problem, that's ideal. One of Pascuale-Leone's smart choices was titling his TED talk after the external problem he expected it to solve, since that makes it easy to triangulate with the audience it was likely to help. It's how I discovered it.

Ledgerwood's video was harder to find. I was searching for videos on "letting go" and "forgiveness." I found it a ways down the Youtube search results, and only clicked it because it seemed to be heavily viewed, was by a psychologist, and seemed at least tangentially relevant. In fact, it turned out to directly address part of my problem.

But positive reframing has its limits. Although it can help you to "brood on the positive," which will then fade (because positive emotions are temporary), it did not allow me to avoid a return to brooding on the negative. The positive insights have stuck, and took the edge off the brooding.

However, her talk uses examples of reframings where the negative frame has a precise logical equivalence to the positive frame, such as a "70% success rate" vs. a "30% failure rate." This is markedly different from a reframing where we choose to dwell on positive aspects of a scenario. Even if we do, the negative aspe...
How easily can we separate a friendly AI in design space from one which would bring about a hyperexistential catastrophe?
by Anirandis — 2020-09-10T00:40:36.781Z

(I've written about this in my Shortform and may regurgitate some stuff from there.)

Eliezer proposes that we separate an AI in design space from one that would constitute a fate worse than death if e.g. the reward model's sign (+/-) were flipped or the direction of updates to the reward model were reversed. This seems absolutely crucial, although I'm not yet aware of any robust way of doing this. Eliezer proposes assigning the AI a utility function of:

U = V + W

Where V refers to human values & W takes a very negative value for some arbitrary variable (e.g. diamond paperclips of length 5cm). So if the AI instead maximises -U, it'd realise that it can gain more utility by just tiling the universe with garbage.

But it seems entirely plausible that the error could occur with V instead of U, resulting in the AI maximising U = W - V, which would result in torture.

------------------------------------------------------------------------------------------------------

Another proposition I found briefly described in a Facebook discussion that was linked to by somewhere. Stuart Armstrong proposes the following:

Let B1 and B2 be excellent, bestest outcomes. Define U(B1) = 1, U(B2) = -1, and U = 0 otherwise. Then, under certain assumptions about what probabilistic combinations of worlds it is possible to create, maximising or minimising U leads to good outcomes.

Or, more usefully, let X be some trivial feature that the agent can easily set to -1 or 1, and let U be a utility function with values in [0, 1]. Have the AI maximise or minimise XU. Then the AI will always aim for the same best world, just with a different X value.

Later, he suggests that X should be a historical fact (i.e. the value of X would be set in stone 10 seconds after the system is turned on.) As XU can only take positive values (because U has values in [0, 1]), the greatest value -XU could take would be 0 (which suggests merely killing everyone.)

But this could still be problematic if e.g. the bug occurred in the reward function/model such that it gave positive values for bad things and negative values for bad things. Although I'm not sure how frequently errors effectively multiply everything in the reward by -1. I'm also unsure how this would interact with an error that reverses the direction of updates to a reward model.

-----------------------------------------------------...
Capturing Ideas
by abramdemski — 2020-09-09T21:20:23.049Z

Related to: Babble and Prune, What Makes People Intellectually Active?, Zettelkasten.

Summary: if you want to generate more ideas, carry a notebook and write down any thoughts you have.

Citation Needed

I've heard these ideas repeated again and again in different forms: books on note-taking, writing, and creativity; the sorts of interviews where artists are asked "where do you get your ideas?"; and most recently, the final post in the Babble and Prune sequence ("Write"). It would be good of me to gather together some references (especially if there's any academic research on this topic?), but I'm going full-on anecdotal here, and just present to you the most complete version of the thing I can cobble together from memory.

I've personally found this technique to be useful, and anecdotally, so have many other people.

The Basic Technique

Let's say you want to have more ideas in some specific category. For example:

You want to write fiction. At times, you might feel like you're full to bursting with ideas which you'd like to turn into stories. Yet, when you sit down to do it, you feel like you don't have any ideas.
You want to do creative research. Maybe so far you only have worked on what problems your advisor gives you. Or maybe you've never worked on research before, and don't know where to start, beyond just reading background literature.
You're looking for ideas in some school- or work- related context, ideas for Christmas presents, startup ideas, etc etc...

Step 1. Get a pocket notebook, or create a new list in a phone note-taking app, et cetera. The goal is to maximize availability and convenience: to the extent possible, you should be able to capture ideas at any time and place.

Step 2. Write down any ideas that you have. Any idea at all. It doesn't have to be good! This is brainstorming. One or two words is fine, so long as you know what it means. Elaborating the idea more will help you remember and may help you generate more ideas, but you can save that for later.

That's it! It's that simple.

Why write a whole post on this? My suspicion is that a lot of people won't raise this very simple strategy to attention to try. I think writing down your ideas has a magical quality to it. You might think:

"It's not that I'm forgetting my ideas. I just don't have much to say."
"I'll write down ideas when I have something good enough to write down."
"I can just write things down la

...
Safer sandboxing via collective separation
by Richard_Ngo — 2020-09-09T19:49:13.692Z

Epistemic status: speculative brainstorming. Follow-up to this post on AGIs as collectives. Note that I've changed the term population AGI to collective AGI for consistency with Bostrom's use in Superintelligence.

One safety measure we might want to use on AGIs is (sand)boxing: limiting their access to the real world, so they have very few channels by which to influence it. However, we also want to deploy them on real-world tasks without them using that opportunity to affect the world in unintended ways. Realistically, even if we can maintain high security in a few sandbox environments, I doubt we’ll be able to make every single deployment environment secure against the full intellectual powers of a superintelligent AGI. But this problem might be easier in the case of a collective AGI.

Broadly speaking, my hypothesis is that the tasks which require the sort of high-level intelligence and strategic capability which we’re worried about also require the least interaction with the world. Discovering new scientific theories or making major strategic decisions can occur in a sandbox which only allows brief text outputs. Whereas being a good personal assistant or web administrator might require frequent access to the internet, but benefit much less from superhuman capabilities. And so if we are able to easily adjust the level of intelligence that an AGI is able to apply to any given task, then we might be able to significantly reduce the risks it poses without reducing its economic usefulness much. On tasks which require interacting with the world in flexible ways, we could limit its intelligence greatly; on tasks which require very high intelligence, we could limit its ability to interact with the world.

In general, for most possible AGI architectures, this gradated intelligence restriction seems quite difficult to achieve. For example, we couldn’t just remove parts of a standard neural network without needing to retrain it significantly. Limiting the AGI’s thinking time, or access to memory, might be possible. But this may reduce its capabilities on all tasks, since even low-level tasks can require detailed calculations and long-term memory. By contrast, it’s easy to reduce the capabilities of a collective AGI: simply deploy only a subset of the members, and bottleneck the communication between different subsets. This mak...