trentmkelly/LessWrong-43k huggingface dataset:
"Redundant" AI Alignment
by Mckay Jensen — 2021-10-16T21:32:05.566Z
Link: https://quevivasbien.github.io/posts/redundant-ai-alignment.html

This is a post I wrote on my personal blog after a discussion with a deep learning professor at the University of Chicago. I don't know if this particular topic has been studied in much depth elsewhere, so I figured I would share it here. If you know of any related work (or have any other comments on this, of course), let me know.

tldr: Supposing that someone figures out how to make an AGI and solve the alignment problem at least to some degree, would it be helpful to distribute many copies of the AI and have them keep each other in check? I propose a mathematical model to think about this question and conclude that this would probably be helpful at first, but would probably diminish in effectiveness over time. Under some assumptions, this approach may actually be counterproductive.
My experience at and around MIRI and CFAR (inspired by Zoe Curzi's writeup of experiences at Leverage)
by jessicata — 2021-10-16T21:28:12.427Z

I appreciate Zoe Curzi's revelations of her experience with Leverage. I know how hard it is to speak up when no or few others do, and when people are trying to keep things under wraps.

I haven't posted much publicly about my experiences working as a researcher at MIRI (2015-2017) or around CFAR events, to a large degree because I've been afraid. Now that Zoe has posted about her experience, I find it easier to do so, especially after the post was generally well-received by LessWrong.

I felt moved to write this, not just because of Zoe's post, but also because of Aella's commentary:

I've found established rationalist communities to have excellent norms that prevent stuff like what happened at Leverage. The times where it gets weird is typically when you mix in a strong leader + splintered, isolated subgroup + new norms. (this is not the first time)

This seemed to me to be definitely false, upon reading it. Most of what was considered bad about the events at Leverage Research also happened around MIRI/CFAR, around the same time period (2017-2019).

I don't want to concentrate on the question of which is "worse"; it is hard to even start thinking about that without discussing facts on the ground and general social models that would apply to both cases. I also caution against blame in general, in situations like these, where many people (including me!) contributed to the problem, and have kept quiet for various reasons. With good reason, it is standard for truth and reconciliation events to focus on restorative rather than retributive justice, and include the possibility of forgiveness for past crimes.

As a roadmap for the rest of the post, I'll start by describing some background, describe some trauma symptoms and mental health issues I and others have experienced, and describe the actual situations that these mental events were influenced by and "about" to a significant extent.

Background: choosing a career

After I finished my CS/AI Master's degree at Stanford, I faced a choice of what to do next. I had a job offer at Google for machine learning research and a job offer at MIRI for AI alignment research. I had also previously considered pursuing a PhD at Stanford or Berkeley; I'd already done undergrad research at CoCoLab, so this could have easily been a natural transition.

I'd decided against a PhD on the basis that rese...
Welcome to Orlando Rationalist Group
by Noah Topper — 2021-10-16T21:22:12.495Z

We're a newly-formed group! I'm hoping to meet up the first Friday of each month, at or around University of Central Florida. We're starting with just casual social gatherings. We may eventually add more structured activities over time. Anyone interested in LessWrong, SSC/ACX, or anything rationality related is welcome. :)

You can join our Discord here: https://discord.gg/sAYUf4pTza.
Optimization Concepts in the Game of Life
by Vika — 2021-10-16T20:51:35.821Z

Abstract: We define robustness and retargetability (two of Flint’s measures of optimization) in Conway’s Game of Life and apply the definitions to a few examples. The same approach likely works in most embedded settings, and provides a frame for conceptualizing and quantifying these aspects of agency. We speculate on the relationship between robustness and retargetability, and identify various directions for future work.

Motivation

We would like to

better understand

the fundamental principles of agency (and related phenomena including optimization and goal-directedness). We focus on agency because we believe agency is a core source of risk from AI systems, especially in worlds with one (or few) most-capable systems. The goals of the most competent consequence-driven systems are more likely to be achieved, because trying outperforms not trying or less competent trying. We do not want to create a world where such systems are working against us. By better understanding agency, we hope to improve our ability to avoid mistakenly building systems working capably against us, and to

correct course

if we do.

A rich source of confusions about agency comes from attending to the fact that goal-directed systems are part of –

embedded

in – the environment that their goals are about. Most practical work on AI avoids the confusions of embedded agency by constructing and enforcing a Cartesian boundary between agent and environment, using frameworks such as

reinforcement learning

(RL) that define an interaction protocol. We focus on embedded agency because we expect not to be able to enforce a Cartesian boundary for highly capable agents in general domains, and, as a particularly strong instance of this, because agents

may emerge unexpectedly

in systems where we did not design how they interface with the rest of the world.

Our approach to deconfusion in this post is to identify concepts that seem relevant to embedded agency but do not have technical definitions, to propose some definitions, and see how they fare on some examples. More generally, we are interested in analyzing small examples of agency-related phenomena in the hope that some examples will be simple enough to yield insight while retaining essential features of the phenomenon.

Optimization in the Game of Life

Concepts

We draw two concepts from Alex Flint’s essay

The Ground of Optimization

. Flint defines an o...
What if we should use more energy, not less?
by Snorkelfarsan — 2021-10-16T19:51:23.203Z

Declining growth rates and technological stagnation since the 70s correlate with flatlining energy use per capita.

I've been reading "Where Is My Flying Car?: A Memoir of Future Past" by J Storrs Hall. Ostensibly a book about a promised future that never arrived, it's a broader commentary on a technologically stagnating culture and society. A few others have reviewed it here on Less Wrong already, and Jason Crawford has a great summary and commentary of the book at Roots of Progress.

I'm sharing this post here to get feedback on the coherence of the idea that declining growth rates and the technological stagnation since the 70s strongly correlate with flatlining energy use per capita:

A key takeaway from the book has been the counter-intuitive realization that perhaps one of the main reasons for the so-called "Great Stagnation" is the western world's flatlining energy usage (per capita). But given our overall increase in energy consumption (mostly due to the growth of the developing world), one can be forgiven for having missed this - especially as we're trying to deal with a warming climate by using less energy. Yet this drive towards efficiency and the resultant decline in energy usage among developed nations, at least according to Hall, may be one of our biggest mistakes of the past half-century.

The Great Stagnation - as readers of Less Wrong are likely familiar with - is the term coined/popularized by economist Tyler Cowen which is now used to describe the current period since the ~70s of declining economic and technological growth experienced by most developed nations. This is perfectly exemplified by the fact that almost every important piece of technology we use in our day-to-day lives and in industries where invented before the 60s. This includes things like refrigerators, freezers, vacuum cleaners, gas and electric stoves, and washing machines; indoor plumbing, detergent, and deodorants; electric lights; cars, trucks, and buses; tractors and combines; fertilizer; air travel, containerized freight, the vacuum tube, and the transistor; the telegraph, telephone, phonograph, movies, radio, and television.

Although this stagnation is surely a result of a wide range of social, economic, and technological factors; like the fact that we've already picked a lot of the "long hanging fruit" of technological innovation since the start of the industrial revolu...
The AGI needs to be honest
by rokosbasilisk — 2021-10-16T19:24:09.780Z

Imagine that you are a trained mathematician and you have been assigned the job of testing an arbitrarily intelligent chatbot for its intelligence.

You being knowledgeable about a fair amount of computer-science theory won’t test it with the likes of Turing-test or similar, since such a bot might not have any useful priors about the world.

You have asked it find a proof for the Riemann-hypothesis. the bot started its search program and after several months it gave you gigantic proof written in a proof checking language like coq.

You have tried to run the proof through a proof-checking assistant but quickly realized that checking that itself would years or decades, also no other computer except the one running the bot is sophisticated enough to run proof of such length.

You have asked the bot to provide you a zero-knowledge-proof, but being a trained mathematician you know that a zero-knowledge-proof of sufficient credibility requires as much compute as the original one. also, the correctness is directly linked to the length of the proof it generates.

You know that the bot may have formed increasingly complex abstractions while solving the problem, and it would be very hard to describe those in exact words to you.

You have asked the bot to summarize the proof for you in natural-language, but you know that the bot can easily trick you into accepting the proof.

You have now started to think about a bigger question, the bot essentially is a powerful optimizer. In this case, the bot is trained to find proofs, its reward is based on finding what a group of mathematicians agree on how a correct proof looks like.

But the bot being a bot doesn’t care about being honest to you or to itself, it is not rewarded for being “honest” it is only being rewarded for finding proof-like strings that humans may select or reject.

So it is far easier for it to find a large coq-program, large enough that you cannot check by any other means, than to actually solve riemann-hypothesis.

Now you have concluded that before you certify that the bot is intelligent, you have to prove that the bot is being honest.

Going by the current trend, it is okay for us to assume that such an arbitrarily intelligent bot would have a significant part of it based on the principles of the current deep-learning stack. assume it be a large neural-network-based agent, also assume that the language-understanding componen...
Is moral duty/blame irrational because a person does only what they must?
by benn@4efix.com — 2021-10-16T17:00:14.012Z

I should note a few things from the start. I understand that there is much prewritten work available here, namely the sequences, the codex, and my favorite fanfic ever, HPMOR. I have tried to find and understand where any of these or any other prewritten works associated with LessWrong.com might have already addressed these questions. I am writing this however because either I did not find the answers I was looking for or I have not recognized them; either way I ask for assistance.

Also, full disclosure, while I have spent the majority of the past three and a half decades (of my 53 total years) on my own exploring applied rationality and discussing it face to face with others in my life’s orbit, as of the last three years or so I have become drawn to the idea of making a book out of my conclusions so far, on the off chance that doing so could be helpful to even a single other person who comes across it. My time on this planet is limited; I’d like to leave some collected thoughts, the fruits of my time to date, to survive me – again, on the off chance that they can be helpful.

One of the realizations I’ve had more recently (largely because it was not an area of thought that interested me until recently) is the question of whether we have the option of making any other choices than the ones we actually do. This by some is called the question of free will, but I am choosing to largely avoid that phrase here in order to attempt to be much more specific, to wit:

When a sentient being (such as I or you) makes decision, is it ever possible they could have made a decision different from the one that they actually made. (Or if speaking of a future decision, will make.)

I am imagining that my chain of thinking on this topic may well seem simplistic, even naïve to the minds found here, but here it is nonetheless, the best effort I have been able to make in applying reason to this question so far, in abbreviated form:

In order to seek explanations for the patterns we see in reality, we must first embrace the idea that even when we don’t know the reasons why something has happened, we still hold that there are reasons why it happened. We must first begin with the idea that explanations are possible to begin seeking them, even if some wind up to be statistical in nature. Even when we cannot determine how something occurred, we still understand that it had a reason for occurrin

...
Memetic hazards of AGI architecture posts
by Ozyrus — 2021-10-16T16:10:07.543Z

I've been thinking recently and writing a post about potential AGI architecture that seems possible to make with current technology in 3 to 5 years, and even faster if significant effort will be put to that goal.

It is a bold claim, and that architecture very well might not be feasible, but it got me thinking about the memetic hazard of similar posts.

It might very well be true that there is an architecture combining current AI tech in a manner as to create AGI out there; in that case, should we treat it as a memetic hazard? If so, what is the course of action here?

I'm thinking that the best thing to do is to covertly discuss it with the AI Safety crowd, both to understand it's feasibility, and to start working on how to keep this particular architecture aligned (which is a much easier task than aligning something that you don't even know how it will look.)

What are your thoughts on this matter?
Write Surprisingly About Reality
by lsusr — 2021-10-16T07:30:33.582Z

Good writing illuminates surprising things about reality. It must therefore be grounded in reality. Losing touch with reality is boring.

The best way to keep your writing grounded in reality is to write concretely. Don't write "the United States committed war crimes". Write "the United States firebombed women and children". Personal experience is always concrete.

Don't argue. Arguing shifts your focus from things to ideas. It distances you from empirical reality. Don't anticipate counterarguments. Preemptive counterargument is form of arguing.

Don't write positively about other peoples' opinions. I love George Orwell but putting him on a pedestal is no less shallow than arguing against him.

Writing about facts, feelings and faith is fine. Fiction has its place too. What you shouldn't write about is other peoples' beliefs. Doing so opens the Box of Infinite Recursion and ultimately leads to the Black Hole of Drama.

Reversed conformity is orthogonal to independent thought. Independent thought equals ignoring others' opinions.

Most people are right most of the time about most things. Deviating from consensus makes you less correct on average. How correct you are on average is unimportant when you are inventing radical ideas. You must weigh according to impact.

Personal attacks cause collateral damage. Before you make a personal attack you should be extremely confident that your claim is true and that the good will outweigh the harm.

It's worth publishing weird ideas even when most of them are wrong because if you publish a weird idea and your idea is good then it will be adopted by many people whereas if the idea is bad then it will be quickly forgotten.

Make your claims easy to falsify. Claims that aren't falsifiable aren't grounded in reality.

Surprise!

Unsurprising facts are boring. Good writing focuses on the surprising ones. If you're ignoring others' opinions then "surprising" means "surprising to you". Explore.

Surprise is temporary. If you discover something surprising then you should write about it immediately, before you acclimatize.
Long Covid Informal Study Results
by Elizabeth — 2021-10-16T07:10:05.056Z

Introduction

Yesterday* I talked about a potential treatment for Long Covid, and referenced an informal study I’d analyzed that tried to test it, which had seemed promising but was ultimately a let down. That analysis was too long for its own post, so it’s going here instead.

Gez Medinger ran an excellent-for-its-type study of interventions for long covid, with a focus on niacin, the center of the stack I took. I want to emphasize both how very good for its type this study was, and how limited the type is. Surveys of people in support groups who chose their own interventions is not a great way to determine anything. But really rigorous information will take a long time and some of us have to make decisions now, so I thought this was worth looking into.

Medinger does a great analysis in this youtube video. He very proactively owns all the limitations of the study (all of which should be predictable to regular readers of mine) and does what he can to make up for them in the analysis, while owning where that’s not possible. But he delivers the analysis in a video rather than a text post ugh why would you do that (answer: he was a professional filmmaker before he got long covid). I found this deeply hard to follow, so I wanted to play with the data directly. Medinger generously shared the data, at which point this snowballed into a full-blown analysis.

I think Medinger attributes his statistics to a medical doctor, but I couldn’t find it on relisten and I’m not watching that damn video again. My statistical analysis was done by my dad/Ph.D. statistician R. Craig Van Nostrand. His primary work is in industrial statistics but the math all transfers, and the biology-related judgment calls were made by me (for those of you just tuning in, I have a BA in biology and no other relevant credentials or accreditations).

The Study

As best I can determine, Medinger sent a survey to a variety of long covid support groups, asking what interventions people had tried in the last month, when they’d tried them, and how they felt relative to a month ago. Obviously this has a lot of limitations – it will exclude people who got better or worse enough they didn’t engage with support groups, it was in no way blinded, people chose their own interventions, it relied entirely on self-assement, etc.

Differences in Analysis

You can see Medinger’s analysis here. He compared the rate of ...
NLP Position Paper: When Combatting Hype, Proceed with Caution
by Sam Bowman — 2021-10-15T20:57:45.583Z

Linkpost for https://cims.nyu.edu/~sbowman/bowman2021hype.pdf. To appear on arXiv shortly.

I'm sharing a position paper I put together as an attempt to introduce general NLP researchers to AI risk concerns. From a few discussions at *ACL conferences, it seems like a pretty large majority of active researchers aren't aware of the arguments at all, or at least aren't aware that they have any connection to NLP and large language model work.

The paper makes a slightly odd multi-step argument to try to connect to active debates in the field:

It's become extremely common in NLP papers/talks to claim or imply that NNs are too brittle to use, that they aren't doing anything that could plausibly resemble language understanding, and that this is a pretty deep feature of NNs that we don't know how to fix. These claims sometimes come with evidence, but it's often bad evidence, like citations to failures in old systems that we've since improved upon significantly. Weirdly, this even happens in papers that themselves show positive results involving NNs.
This seems to be coming from concerns about real-world harms: Current systems are pretty biased, and we don't have great methods for dealing with that, so there's a pretty widely-shared feeling that we shouldn't be deploying big NNs nearly as often as we are. The reasoning seems to go: If we downplay the effectiveness of this technology, that'll discourage its deployment.
But is that actually the right way to minimize the risk of harms? We should expect the impacts of these technologies to grow dramatically as they get better—the basic AI risk arguments go here—and we'll need to be prepared for those impacts. Downplaying the progress that we're making, both to each other and to outside stakeholders, limits our ability to foresee potentially-impactful progress or prepare for it.

I'll be submitting this to ACL in a month. Comments/criticism welcome, here or privately (bowman@nyu.edu).
Intelligence, epistemics, and sanity, in three short parts
by ozziegooen — 2021-10-15T04:01:27.680Z

Epistemic status: Boggling. This is early, messy work.

Thanks to Edo Arad for comments and suggestions.

Part 1: A Brief Tale

You’re exploring a vast land filled with forests and brush. You thrash your sword to carve out a path and make sense of things.

There are monsters. Lovecraftian winged monstrosities. Their attacks damage your sanity. When injured, you don’t notice, but your mind will begin to wander. You begin seeing things that aren’t real.

You have a shield that protects you from the beasts. You have armor as well. They attack; you deflect. You hear a thud, a squeal, then light flapping sounds when they retreat. Occasionally you can even catch them with your sword but they move quickly.

You return to the village. Your companions have returned too. Some of them aren’t right. They’re obsessed with bizarre imaginings of angels and gods. Some speak of finding ancient labyrinths in places you know are full of swamps.

Your companions size each other up. Factions emerge. One cluster suggests that the monsters bring forward wisdom. The monsters should be brought directly into the village and released.

The pro-monster faction is visibly scratched up; they must have been wounded. They deny taking damage, that the marks on their bodies were extraneous. Some hallucinate wearing a great deal of armor, even though they are visibly unprotected. They do however possess large weapons (likely a trade-off from having light armor), so others are nervous about disagreeing.

The bulk of the crowd turns against this faction. Some attack back; a few people are harmed, but in the end, the faction loses. Some are thrown into jails, others delegated to the very safest of tasks. But this cycle will repeat. You might not be so lucky next time.

The village learns to cope. There are vast regions of bush and thorn but no monsters. Soldiers with little armor are sent there. Some dual wield axes or use two-handed broadswords.

Other areas have many monsters but plain land. This is where you send your fully plated knights. They’ll move slowly but have the best chances of keeping their sanity.

Part 2: An Explanation

The topic is intelligence, epistemics, and sanity.

Weapons are raw intellect, cleverness, narrow technical abilities. These are powerful but dangerous.

Shields and armor represent epistemics, wisdom, humility, calibration. Not believing false things. Not going functi...