trentmkelly/LessWrong-43k huggingface dataset:
Rationality and Climate Change
by Emiya — 2020-10-05T10:21:09.512Z

My master thesis was on the subject of climate change and mass media attitude toward it. Working on it, I've read over 300 newspaper articles on the subject and studied a large number of papers regarding its consequences, its solutions, and the attempts to frame the debate over them, so I consider myself to be well informed on it.

In the months I've been on this site though, I've been greatly surprised by the apparent unconcerned attitude many of the users have toward this theme.

Given that

to the best of my knowledge, I think that we should instead be very concerned
the subject is certainly of a great relevance since a lot of investments and global consequences in the future years will revolve around it
we can't just agree to disagree

I plan to write a post that would cover this subject and hopefully resolve the disagreement.

Before doing that, though, I'd find extremely useful if people would write me what they believe, feel and anticipate about climate change, its future consequences and the processes that would be required to stop it, and why they think they believe, feel and anticipate that way.

I'm greatly interested in receiving replies both from people feeling alarmed and from people not feeling alarmed.

If, while reading this question or writing me your reply, you realised you'd like to document yourself more on the subject and update your beliefs, that's perfectly fine, but I'd ask that you write me your thoughts about it before doing so. If you, instead, already documented about it before reading this question, that's perfectly fine and I'd like to read your thoughts anyway.

What I'm trying to understand are the beliefs of the users of this site at the current moment.

You can be as synthetic or detailed as you'd like, and can write me either a reply under this post or a private message.

I'd like to thank in advance everyone who'll send me their thoughts and who'll dedicate this question a bit of their time.
AGI safety from first principles: Conclusion
by Richard_Ngo — 2020-10-04T23:06:58.975Z

Let’s recap the second species argument as originally laid out, along with the additional conclusions and clarifications from the rest of the report.

We’ll build AIs which are much more intelligent than humans; that is, much better than humans at using generalisable cognitive skills to understand the world.

Those AGIs will be autonomous agents which pursue long-term, large-scale goals, because goal-directedness is reinforced in many training environments, and because those goals will sometimes generalise to be larger in scope.

Those goals will by default be misaligned with what we want, because our desires are complex and nuanced, and our existing tools for shaping the goals of AIs are inadequate.

The development of autonomous misaligned AGIs would lead to them gaining control of humanity’s future, via their superhuman intelligence, technology and coordination - depending on the speed of AI development, the transparency of AI systems, how constrained they are during deployment, and how well humans can cooperate politically and economically.

Personally, I am most confident in 1, then 4, then 3, then 2 (in each case conditional on all the previous claims) - although I think there’s room for reasonable disagreement on all of them. In particular, the arguments I’ve made about AGI goals might have been too reliant on anthropomorphism. Even if this is a fair criticism, though, it’s also very unclear how to reason about the behaviour of generally intelligent systems without being anthropomorphic. The main reason we expect the development of AGI to be a major event is because the history of humanity tells us how important intelligence is. But it wasn’t just our intelligence that led to human success - it was also our relentless drive to survive and thrive. Without that, we wouldn’t have gotten anywhere. So when trying to predict the impacts of AGIs, we can’t avoid thinking about what will lead them to choose some types of intelligent behaviour over others - in other words, thinking about their motivations.

Note, however, that the second species argument, and the scenarios I’ve outlined above, aren’t meant to be comprehensive descriptions of all sources of existential risk from AI. Even if the second species argument doesn’t turn out to be correct, AI will likely still be a transformative technology, and we should try to minimise other potential harms. In addition t...
One hub, or many?
by abramdemski — 2020-10-04T16:58:40.800Z

I was reading the comments to The Rationalist Community's Location Problem and especially this comment by Damien Tatum, and thinking about some classic questions of LessWrong/EA/fully-general physical community.

I think a guiding idea for years has been that if we get as many good people and organizations to have close proximity in the bay area, good things will happen. But I don't know whether that's been positive or negative overall.

For years I've heard about brain-drain problems in other would-be hubs. The best people move to the bay area, and the local hub either gets much worse or disappears. (I'm talking especially about meetup groups, but in the hypothetical where this sort of think didn't happen so much, we might also see more serious organizations in those other hubs.)

In a significant sense, this makes for a higher bar to getting involved. If what you have to do in order to get good in-person discussions with rationalists/EAs is move to the bay area rather than get to the closest decent meetup in your city/state/country, well, that puts a big damper on things.

If there have been discussions/modeling about this trade-off before, I'd especially appreciate links.

Edit: The Berkeley Community & The Rest Of Us discusses the brain-drain dynamic I'm discussing.
AI race considerations in a report by the U.S. House Committee on Armed Services
by NunoSempere — 2020-10-04T12:11:36.129Z

Epistemic status: Quick and dirty. A surface level dive into a particular aspect of AI governance carried out over the course of one morning.

Context

The U.S. House Committee on Armed Services is a standing committee of the United States House of Representatives. It is responsible for funding and oversight of the Department of Defense (DOD) and the United States Armed Forces, as well as substantial portions of the Department of Energy.

The Future of Defense Task Force is a subcommittee of the U.S. House Committee on Armed Services. They have released a report, available here, and also as the first item on their latest news page. The task force is manned by an equal number of Republicans and Democrats. Though this seems a priori unlikely, it could both be the case that this report is unrepresentative of the political forces in the US Congress, and that this particular committee holds little power.

References to AI race dynamics in the report

Bold added by me.

Technological advancements in artificial intelligence and biotechnology will have an outsized impact on national security; the potential of losing this race to China carries significant economic, political, and ethical risks for the United States and our free democratic allies for decades to come. Winning this race requires a whole-of-nation approach where the distinct advantages of both America's private and public sector are harnessed and synthesized.

Using the Manhattan Project as a model, the United States must undertake and win the artificial intelligence race by leading in the invention and deployment of AI while establishing the standards for its public and private use. Although the Department of Defense has increased investment in AI and established the Joint Artificial Intelligence Center to assist with the transition and deployment of AI capabilities, cultural resistance to its wider adoption remains.

The stakes are high. Whoever achieves superiority in this technological race will enjoy significant military and economic advantage for decades—and possibly into the next century.

To incorporate the technology necessary to maintain the United States' military supremacy, the Pentagon must continue refining its acquisition process to be more agile and less risk-averse so that it can fully leverage emerging technologies and capabilities at scale.
Train and incentivize the acquisition workfor

...
Public transmit metta
by Kaj_Sotala — 2020-10-04T11:40:03.879Z

Metta, or loving-kindness meditation, involves picking a person and wishing them good things. In the same way as other kinds of concentration practice condition your mind by letting it notice how it feels good to be able to concentrate, I think of loving-kindness meditation as conditioning your mind by letting it notice how pleasant it feels to experience metta.

What is metta? Psychologist and meditation teacher Ron Crouch describes it as follows:

In English “love” is an emotion that comes with attachment built-in. We love another person and we want to be with them. We view our lives, and in some cases even ourselves, as incomplete without the other person. This kind of love is not bad or unwholesome, far from it. Falling in love and being in love is great (if a little crazy). But it is not the same kind of love that is referred to by “metta,” which is love without clinging or attachment. In this sense it is very pure.

The Buddha evoked the image of a mother’s love for her newborn as an analogy for metta. A mother’s love for a baby is completely unconditional – there are truly no strings attached. The baby can be totally cranky and ridiculously self-centered (as babies tend to be), but the mother will still love the baby and expect nothing in return. If the baby does show love in return, well, that’s just a nice bonus but not expected (most people with a kid will know intuitively what metta feels like). With metta there is no sense that the other person needs to do something to fulfill one’s needs or complete the love – this kind of love is complete all by itself.

I generally like how I feel after I’ve done metta “right”, but I find I often have self-centered motives sneak into it that make it hard to reach that attitude of completely unconditional love. For instance, if I’m sending metta to a friend, I might hope that they are happy because I like it when people around me are happy, or even because just imagining someone happy makes me feel safe. Then my focus starts alternating between the feeling of safety – something that is about my own needs rather than about the other person – and the actual loving-kindness. While there isn’t anything wrong about enjoying a feeling of safety, it tends to be conditional on the other person being around and acting in a particular way. As a result, focusing on it usually doesn’t leave me with a more lasting sense of well-bei...
Is there any work on incorporating aleatoric uncertainty and/or inherent randomness into AIXI?
by David Scott Krueger (formerly: capybaralet) — 2020-10-04T08:10:56.400Z

IIUC, AIXI assumes the environment is deterministic.
In other words, it only has epistemic uncertainty.
What if it didn't make that assumption and/or the assumption was violated?
Has anyone explored this question?
Fishbowl reality 
by Elo — 2020-10-04T02:16:18.595Z

This idea is about perception and reality.

Each person is wearing a fishbowl over their head. Think of the mind as being two intelligent forces. The head is the (self believing) sentient consciousness. The one that receives the information that is projected onto the fishbowl. The other one seems to be a different type of intelligence, usually barely described and left up to weird mystical description.

The human body constantly has inputs. We have 5 main sensory categories and likely quite a few more beyond that if proprioception etc are counted. There's always skin sensation, there's always visual information, even with closed eyes, and there's always sound. Even if there isn't sound echoing outside, there's ringing, or heart beats. Among that constant incoming stream, some process is decided which input to send into the consciousness.

Which input is going to keep this being alive. The smell of smoke when I'm sleeping, loud noises suddenly, quickly moving visual objects (TV's in the background of what I'm doing). Each of these are examples of information that is prioritised over other information when projecting information onto the fishbowl.

Even when there’s no urgent and survival crucial event, some process needs to be deciding what to project - visual or audio or sensation or other. Or even, which sensation within sensation, which part of the visual field to discern.

In this experience of receiving selective filtered information from a differently conscious part of the mind, there exists an argument for a (broadly speaking) panoptic process of appreciation for the collection of information that we usually would describe as our subjective experience of reality. (1) It would appear to be a beneficial process to feel the subjective appreciation for anything that enters conscious perception, to incentivise the internal part of the mind that is choosing what to project at the consciousness.

I suspect that the mind often processes and preferences various sensations and experiences over certain other experiences. I suspect that this is a real time feedback mechanism for assisting the filter to decide what contents to bring to the consciousness.

For example if the consciousness doesn't like strong political events, but keeps experientially turning towards them, giving them more attention, then it's only reasonable to expect that the fishbowl pro...
Postmortem to Petrov Day, 2020
by Ben Pace — 2020-10-03T21:30:56.491Z

The first underwater test of an atom bomb, in the US at Bikini Atoll in the Pacific, July 24, 1946.
Also, the LessWrong.com Frontpage this Petrov Day.

We failed.

The Frontpage was taken down for 24 hours.

Launch Attempts

At 1:26 am, ~275 users got an email (and 20 minutes later a PM) from me, with their unique launch codes. At this time, the Big Red Button was on the Frontpage, and codes could be submitted.

Two users attempted to submit launch codes. It was made clear that any such attempts would be deanonymised, as the button itself said "This is not an anonymous action".

The users were: Grotace at 2:09 am, who submitted the random string "kdjssndksjnd" to no effect, and Chris_Leong at 4:33 am, who submitted his personalised launch codes, taking down the site.

What happened?

Note that the following includes private messaging metadata about a number of users. All users (except for the attacker) were asked for their explicit consent for it to be included, and it would have been anonymized had they declined, even if it was clearly deducible from the public record. The LessWrong team only looks into private messaging metadata when there is a suspicion of sockpuppeting, phishing, or other forms of abuse, or when permission has been given by the user (e.g. for debugging). Additionally, Chris Leong commented on a draft of this post before it was published.

On receiving his codes, Chris Leong posted on LessWrong and Facebook asking whether he should take down the site.

He also wrote a comment on LessWrong:

Should I press the button or not? I haven't pressed the button at the current time as it would be disappointing to people if they received the email, but someone pressed it while they were still asleep.

He did not get replies on the LessWrong comment before he took the site down. The first two comments on the Facebook post were as follows.

Commenter 1

I kind of feel like you should take it down for a day. No individual should be trusted with the ability to destroy the world. We are all driven by personal incentive mechanisms and other systems must be developed to ensure it’s not that easy for one person to bring so much chaos into the world.

Also... when’s the next time you’re going to get the opportunity to take down a massive site for a day legally?

Commenter 2

LessWrong is archived elsewhere so the damage by bringing it down (especially just for a day) is not that much. Equally

...
Weird Things About Money
by abramdemski — 2020-10-03T17:13:48.772Z

1. Money wants to be linear, but wants even more to be logarithmic.

People sometimes talk as if risk-aversion (or risk-loving) is irrational in itself. It is true that VNM-rationality implies you just take expected values, and hence, don't penalize variance or any such thing. However, you are allowed to have a concave utility function, such as utility which is logarithmic in money. This creates risk-averse behavior. (You could also have a convex utility function, creating risk-seeking behavior.)

Counterpoint: if you have risk-averse behavior, other agents can exploit you by selling you insurance. Hence, money flows from risk-averse agents to less risk-averse agents. Similarly, risk-seeking agents can be exploited by charging them for participating in gambles. From this, one might think a market will evolve away from risk aversion(/seeking), as risk-neutral agents accumulate money.

People clearly act more like money has diminishing utility, rather than linear utility. So revealed preferences would appear to favor risk-aversion. Furthermore, it's clear that the amount of pleasure one person can get per dollar diminishes as we give that person more and more money.

On the other hand, that being the case, we can easily purchase a lot of pleasure by giving money to others with less. So from a more altruistic perspective, utility does not diminish nearly so rapidly.

Rationality arguments of the Dutch-book and money-pump variety require an assumption that "money" exists. This "money" acts very much like utility, suggesting that utility is supposed to be linear in money. Dutch-book arguments assume from the start that agents are willing to make bets if the expected value of those bets is nonnegative. Money-pump arguments, on the other hand, can establish this from other assumptions.

Stuart Armstrong summarizes the money-pump arguments in favor of applying the VNM axioms directly to real money. This would imply risk-neutrality and utility linear in money.

On the other hand, the Kelly criterion implies betting as if utility were logarithmic in money.

The Kelly criterion is not derived via Bayesian rationality, but rather, an asymptotic argument about average-case performance (which is kinda frequentist). So initially it seems this is no contradiction.
However, it is a theorem that a diverse market would come to be dominated by Kelly bettors, as Kelly betting maximizes long-term

...
On Option Paralysis - The Thing You Actually Do
by Neel Nanda — 2020-10-03T11:50:57.070Z
Link: https://www.neelnanda.io/blog/37-option-paralysis

Introduction

I recently encountered a problem in my life I found interesting. I noticed that I was fairly unfit, that my energy levels were lower than I wanted them to be, and so I was considering starting to exercise more as a solution to this. And I knew that exercise is ridiculously important to longevity and general health, and had been thinking about this for at least the path month.

So I brought up this issue with my coach, and it became obvious to me that I’d never actually done anything about this - it felt aversive, and I felt highly reluctant to take any action. And when I introspected on where that reluctance came from, it turned out to be coming from a desire to find the optimal form of exercise. And because I am deeply confused about the science behind exercise, there were many different options for how to get into it, and it wasn’t obvious what the best one was, I was caught in analysis paralysis and procrastinating about it. And her advice for this problem was “the best kind of exercise is the one you actually do”.

I knew on some level that I was making an error, but I thought this articulated it excellently. Most of the value comes from doing any exercise, and finding optimal forms, durations etc should not be the priority. And since seeing it like this, the problem has felt basically solved! I decided to pick Couch to 5K as an obvious and available exercise plan, and have systematised doing it 3 times a week: I have a clear time in the mornings to do it, reminders, and owe a friend of mine £20 for each session I miss (for bad reasons). And at this point, it now feels like the default action is to do some exercise, I’ve kept it up for the last month, and this is no longer a burning priority.

When articulated like this, the problem feels obvious. But this is also deeply weird. I procrastinated for months on a problem, found a slight change in perspective, and near instantly solved it to my satisfaction, to the point where the solution now feels like the default action. And getting better at resolving problems like that sounds like a major win!

This kind of perfectionism-induced procrastination and analysis paralysis is an example of a far more general problem in my life, and in the lives of the people around me. In this post I’m going to dig into to exactly what the error is and where it comes from, ...
AGI safety from first principles: Control
by Richard_Ngo — 2020-10-02T21:51:20.649Z

It’s important to note that my previous arguments by themselves do not imply that AGIs will end up in control of the world instead of us. As an analogy, scientific knowledge allows us to be much more capable than stone-age humans. Yet if dropped back in that time with just our current knowledge, I very much doubt that one modern human could take over the stone-age world. Rather, this last step of the argument relies on additional predictions about the dynamics of the transition from humans being the smartest agents on Earth to AGIs taking over that role. These will depend on technological, economic and political factors, as I’ll discuss in this section. One recurring theme will be the importance of our expectation that AGIs will be deployed as software that can be run on many different computers, rather than being tied to a specific piece of hardware as humans are.[1]

I’ll start off by discussing two very high-level arguments. The first is that being more generally intelligent allows you to acquire more power, via large-scale coordination and development of novel technological capabilities. Both of these contributed to the human species taking control of the world; and they both contributed to other big shifts in the distribution of power (such as the industrial revolution). If the set of all humans and aligned AGIs is much less capable in these two ways than the set of all misaligned AGIs, then we should expect the latter to develop more novel technologies, and use them to amass more resources, unless strong constraints are placed on them, or they’re unable to coordinate well (I’ll discuss both possibilities shortly.)

On the other hand, though, it’s also very hard to take over the world. In particular, if people in power see their positions being eroded, it’s generally a safe bet that they’ll take action to prevent that. Further, it’s always much easier to understand and reason about a problem when it’s more concrete and tangible; our track record at predicting large-scale future developments is pretty bad. And so even if the high-level arguments laid out above seem difficult to rebut, there may well be some solutions we missed which people will spot when their incentives to do so, and the range of approaches available to them, are laid out more clearly.

How can we move beyond these high-level arguments? In the rest of this section I’ll lay out two types of di...