trentmkelly/LessWrong-43k huggingface dataset:
AI Safety and Neighboring Communities: A Quick-Start Guide, as of Summer 2022
by Sam Bowman — 2022-09-01T19:15:40.713Z

Getting into AI safety involves working with a mix of communities, subcultures, goals, and ideologies that you may not have encountered in the context of mainstream AI technical research. This document attempts to briefly map these out for newcomers.

This is inevitably going to be biased by what sides of these communities I (Sam) have encountered, and it will quickly become dated. I expect it will still be a useful resource for some people anyhow, at least in the short term.

AI Safety/AI Alignment/AGI Safety/AI Existential Safety/AI X-Risk

The research project of ensuring that future AI progress doesn’t yield civilization-endingly catastrophic results.

Good intros:

Carlsmith Report

What misalignment looks like as capabilities scale

Vox piece

Why are people concerned about this?

My rough summary:

It’s plausible that future AI systems could be much faster or more effective than us at real-world reasoning and planning.

Probably not plain generative models, but possibly models derived from generative models in cheap ways

Once you have a system with superhuman reasoning and planning abilities, it’s easy to make it dangerous by accident.

Most simple objective functions or goals become dangerous in the limit, usually because of secondary or instrumental subgoals that emerge along the way.

Pursuing typical goals arbitrarily well requires a system to prevent itself from being turned off, by deception or force if needed.
Pursuing typical goals arbitrarily well requires acquiring any power or resources that could increase the chances of success, by deception or force if needed.
Toy example: Computing pi to an arbitrarily high precision eventually requires that you spend all the sun’s energy output on computing.

Knowledge and values are likely to be orthogonal: A model could know human values and norms well, but not have any reason to act on them. For agents built around generative models, this is the default outcome.
Sufficiently powerful AI systems could look benign in pre-deployment training/research environments, because they would be capable of understanding that they’re not yet in a position to accomplish their goals.
Simple attempts to work around this (like the more abstract goal ‘do what your operators want’) don’t tend to have straightforward robust implementations.

If such a system were single-mindedly pursuing a dangerous goal, we probably wouldn’t be able

...
Gradient Hacker Design Principles From Biology
by johnswentworth — 2022-09-01T19:03:16.836Z

I think constructing a gradient hacker would be a pretty significant advance in agent foundations, especially embeddedness. To that end, this post contains a bunch of principles and intuitions from biology about how to design a gradient hacker.

Analogy: Bacteria and Entropy

We live in a world in which physical entropy of the universe always increases (or, equivalently, negentropy always decreases). However, a bacteria is a relatively low entropy configuration; entropy goes up if we burn the bacteria. How can this relatively low entropy structure propagate in a world where entropy always increases?

We’ll use this as an analogy for gradient hacking: just like a bacteria is a low-entropy structure trying to propagate in a system where entropy increases over time, a gradient hacker is a low-training-objective-performance structure trying to propagate in a system where training-objective-performance improves over time. What lessons can our gradient hacker borrow from bacteria?

Reaction Coupling

Let’s start with one of the best-understood techniques the bacteria uses: coupling entropy-producing reactions to entropy-consuming reactions.

Bacteria often need to produce lower-entropy molecules from higher-entropy molecules - e.g. synthesizing DNA, RNA, proteins, or structural molecules from metabolic building blocks. Those reactions don’t happen spontaneously, because they decrease entropy. However, bacteria can also utilize reactions which increase entropy - e.g. burning ATP. If the bacteria couples ATP-burning to a synthesis reaction (by using a single enzyme to catalyze the reactions together), then the combined reaction can increase entropy overall, so it happens spontaneously. The entropy-increasing reaction “drives” the entropy-decreasing reaction.

What would this look like in a gradient hacker? The gradient hacker wants to update its component weights in a way which locally decreases performance on the training objective. The gradient hacker also has ways to increase the performance on the training objective. So, it couples these two together: it routes the performance-increasing behavior in such a way that the gradient update assigns some of the credit to the normally-performance-decreasing configuration.

Figuring out a simple, targetable and scalable way to perform that coupling, analogous to a biological enzyme, is one of the key gradient hacker design problems...
Book review: Put Your Ass Where Your Heart Wants to Be
by Ruhul — 2022-09-01T18:21:16.408Z

Cross-posted and lightly edited from Future Startup.

Put Your Ass Where Your Heart Wants to Be is my fourth book by Steven Pressfield. Previously, I enjoyed reading his excellent The War of Art, an all-time favorite among creative types, Do the Work and Turning Pro. Naturally, I had certain expectations.

Steven understands the trial and tribulation of creative life. Being an author himself, I think part of the understanding comes from his lived experience. The rest of it, as Steven puts it, comes from “the muse”.

Put Your Ass Where Your Heart Wants to Be can be called a follow-up of The War of Art. Steven introduced the idea of Resistance with a capital R in that book. In this book, he re-examines some of those ideas, expands on some, and proposes several new ideas.

The ideas in the book are not uncommon if you have read Steven’s blog or his other books. And the approach he takes to put those ideas across is also familiar. To many people, these ideas may appear simplistic motivational hotchpotch. It may appear sometimes that he does not go deep enough into the roots of these challenges creative people face, which I would say is a weakness of the book.

But the interesting thing about, at least in my opinion, Steven’s writing is that while the ideas are simple and direct, they don’t lack depth. He summarizes the challenges of creative types extremely well. He offers effective solutions to those challenges. While he does not go into the psychological rabbit hole to explain the source of many of these challenges, he offers effective diagnoses and resolutions.

Creation is a difficult path. Be it a book, a painting, a song, or an enterprise. The work and the predicaments remain the same. Steven begins the first chapter with an overview of the challenge at hand and how to go about it.

“We all know how hard it is to write a book, make a movie, or create a new business. Powerful forces line up against us — obstacles to entry, rivals, competitors, finances, funding, and the difficulty of the craft itself. But the most formidable antagonist of all resides inside each of our own skills. I’m talking about the negative force I call resistance with a capital R.”

Resistance is an idea Steven writes extensively about in The War of Art. Many famous authors and creative professionals have spoken about it. For instance, another beloved no...
A Survey of Foundational Methods in Inverse Reinforcement Learning
by adamk — 2022-09-01T18:21:10.559Z

Introduction

Inverse Reinforcement Learning (IRL) is both the name of a problem and a broad class of methods that try to solve it. Whereas Reinforcement Learning (RL) asks “How should I act to achieve my goals?” IRL asks “Looking at an agent acting, what are its goals?”

The problem was originally posed by Stuart Russell in 1998, where he pointed to a few key motivations for wanting to solve IRL. First, humans and animals don't have directly visible reward functions; IRL may be able to reverse engineer the complex rewards of real-world actors. Second, it may be useful in apprenticeship learning, where an agent learns how to perform a task based on demonstration. Learning the demonstrator's reward can be a useful intermediate step. Third, there are often cases where agents or demonstrators weigh carefully between multiple attributes, striking a balance between speed, cost, etc. We may know these factors, but not their relative tradeoff weights. IRL is well-suited to solve for these.

Since Russell’s initial question, a broad class of IRL methods have appeared to tackle and extend the problem, and apply solutions. My goal in this post is to survey a few key methods and applications of IRL, considering the technical roadmap, strengths, and weaknesses of different approaches. I have elected to mostly discuss seminal early works for each method, as I am mostly interested in giving context for further study. My hope is that a broad, rather than deep, survey will be both more accessible and encourage readers to investigate areas of interest for themselves.

I’m assuming some familiarity with linear algebra and probability. I’ll briefly overview the mathematical setup of Markov Decision Processes, but I’ll mainly assume the reader is familiar with the basic ideas of reinforcement learning.

MDP Overview

A Markov Decision Process (MDP) is the normal formalization for the environment of a reinforcement learning agent. (The reference I would recommend is Barto and Sutton's Reinforcement Learning: An Introduction, Chapter 3. Wikipedia also works.) An MDP consists of:

A set of states S (usually finite, but we can extend to infinite state spaces)
A set of actions A available to the agent (some formulations make the set of available actions dependent on state, but most IRL papers do not consider this generalization)
A set T of transition probability

...
I Tripped and Became GPT! (And How This Updated My Timelines)
by Frankophone — 2022-09-01T17:56:29.901Z

Disclaimer: The effects of psilocybin in magic mushrooms are not particularly well researched and while I had good experiences with it, I cannot specifically recommend it due to possible risks. Please do your own research.

Epistemic Status: This post is more about personal experience, and things that seem true to me, but since this post is about taking magic mushrooms, I hope you will take what is said here with a grain of salt.

In this post, I will provide a qualitative discussion of my experiences while ingesting psilocybin magic mushrooms, and how this has updated me in the direction of shorter timelines being more valid than I had previously thought. In particular, I incorporate with low certainty the differences in my subjective experiences and abilities during the tripping into my model of how AGI occurs.

Differences in Personal Experience

(Not really related to the timelines, but it might give you some context.)

While tripping, my outward behavior was essentially the same at every point. It was difficult for those with me to notice that I was tripping because of the words I said and the actions I took, except for at one point where I had a particularly strong tendency to laugh at things and giggle very much. However, it is difficult to pinpoint exactly what was different, because while my mental state felt completely different, "the conscious part of my mind" was almost identical. (One interesting consequence is that this shows how few bits of information are output through our actions).

When I looked at things, I could feel that they looked different, but when I looked specifically at what was different, there was no specific difference. My field of vision felt more narrow and focused on the center of my field of vision, but I could still see things on the periphery of my field of vision. There was a bit of ghosting and blooming, and it was more like watching a high definition movie or game than reality. Reading text was possible, but much more difficult.

In addition, my sense of time was also significantly impaired, making it difficult to keep track of time. When I moved from room to room, it quickly felt like an eternity since I had been at the previous place. Sitting in a new place, a very short time can feel like a long time. Still, if I searched my memory for the sequence of events leading up to a particular moment, I could infer that not m...
Fixed point theory (locally (α,β,ψ) dominated contractive condition)
by muzammil — 2022-09-01T17:56:17.978Z

locally (α,β,ψ) dominated contractive condition

Theorem 2.1 Let α,β:X×X→[0,+∞), r>0, x₀∈X, ψ∈Ψ, (X,d) be an (α,β)-complete metric space and S,T:X→X. If the following conditions hold:
1) S and T are (α,β)-continuous,
2) The pair (S,T) satisfies the locally (α,β,ψ) dominated contractive condition on B(x₀,r),
3) If x and y belongs to set of common fixed points of S and T, then α(x,y)≥β(x,y).
Then S and T have a unique common fixed point.

I have to solve this theorem in families of mappings.
Alignment is hard. Communicating that, might be harder 
by Eleni Angelou — 2022-09-01T16:57:55.564Z

Crossposted from the EA Forum: https://forum.effectivealtruism.org/posts/PWKWEFJMpHzFC6Qvu/alignment-is-hard-communicating-that-might-be-harder

Note: this is my attempt to articulate why I think it's so difficult to discuss issues concerning AI safety with non-EAs/Rationalists, based on my experience. Thanks to McKenna Fitzgerald for our recent conversation about this, among other topics.

The current 80,000 Hours list of the world's most pressing problems ranks AI safety as the number one cause in the highest priority area section. And yet, it's a topic never discussed in the news. Of course, that's not because journalists and reporters mind talking about catastrophic scenarios. Most professionals in the field, are perfectly comfortable talking about climate change, wars, pandemics, wildfires, etc. So what is it about AI safety that doesn't make it a legitimate topic for a panel on TV?

The EA consensus is roughly that being blunt about AI risks in the broader public would cause social havoc. And this is understandable; the average person seems to interpret the threats from AI either as able to provoke socio-economic shifts similar to those that occurred because of novel technologies during the Industrial Revolution (mostly concerning losing jobs), or as violently disastrous as in science fiction films (where e.g., robots take over by fighting wars and setting cities on fire).

If that's the case, taking seriously what Holden Karnofsky describes in The Most Important Century as well as what many AI timelines suggest (i.e., that humanity might be standing at a very crucial point in its trajectory) could easily be interpreted in ways that would lead to social collapse just by talking about what the problem concerns. Modern AI Luddites would potentially form movements to "prevent the robots from stealing their jobs". Others would be anxiously preparing to physically fight and so on.

But, even if the point about AGI doesn't get misinterpreted, if, in other words, all the technical details could be distilled in a way that made the arguments accessible to the public, it's very likely that it would trigger the same social and psychological mechanisms that would result in chaos manifested in various ways (for example, stock market collapse because of short timelines, people breaking laws and moral codes since nothing...
Covid 9/1/22: Meet the New Booster
by Zvi — 2022-09-01T14:00:01.234Z

We will soon have a new, updated booster shot only eight or so months behind schedule. Given only 30% of America is boosted even once, not many people will accept this kind and generous (if rather late) offer. It won’t make much difference even to those who take advantage of it at this point, but better to update than not update, the usual various objections aside. Hopefully this makes the next update easier.

Other things were mostly as one would expect.

This week I also wrote a long analysis of the student loan forgiveness situation.

Executive Summary

Booster update available soon.

Student loan forgiveness bigger than most realize.

Monkeypox situation looking better.

Let’s run the numbers.

The Numbers

Predictions

North Carolina reported almost 500 deaths, which is obviously a backlog, so I adjusted that number to the still-rather-higher-than-reality 150.

Prediction from last week: 542k cases (-5%) and 3,030 deaths (-5%)

Results: 523k cases (-9%) and 3,183 deaths (-1%).

Prediction for next week (HOLIDAY): 420k deaths (-20%) and 2,550 deaths (-20%).

Labor Day is going to mess with reporting a bunch, so noting that this does not represent a predicted real decline in either cases or deaths.

Deaths

Cases

Moderna Sues Pfizer

Moderna is charging Pfizer with patent infringement.

Vaccine maker Moderna announced Friday that it’s suing rival drugmakers Pfizer and BioNtech for patent infringement. The lawsuit alleges the two companies used certain key features of technology Moderna developed to make their COVID-19 vaccine. It argues that Pfizer and BioNtech’s vaccine infringes patents Moderna filed between 2010 and 2016 for its messenger RNA or mRNA technology.

…

In October 2020, Moderna pledged not to enforce its COVID-19 related patents while the pandemic was ongoing, according to a statement from the company. In March this year, it said it will stick to its commitment not to enforce its COVID-19 related patents in low and middle-income countries, but expects rival companies like Pfizer to respect its intellectual property.

Moderna is not seeking to remove the Pfizer and BioNTech vaccine from the market, but is seeking monetary damages.

On the one hand, this is horrible, it was a pandemic, you use what tools you have. Moderna pledged not to enforce the patent during the pandemic.

On the other hand, fair. Moderna...
A Starter-kit for Rationality Space
by Jesse Hoogland — 2022-09-01T13:04:48.618Z
Link: https://github.com/jqhoogland/rationalia-starter

LessWrong has gotten big over the years: 31,260 posts, 299 sequences, and more than 120,000 users.[1] It has budded offshoots like the alignment and EA forums and earned itself recognition as "cult". Wonderful!

There is a dark side to this success: as the canon grows, it becomes harder to absorb newcomers (like myself).[2] I imagine this was the motivation for the recently launched "highlights from the sequences".

To make it easier on newcomers (veterans, you're also welcome to join in), I've created an Obsidian starter-kit for taking notes on the LessWrong core curriculum (the Sequences, Codex, HPMOR, best of, concepts, various jargon, and other odds and ends).

There's built-in support to export notes & definitions to Anki, goodies for tracking your progress through the content, useful metadata/linking, and pretty visualizations of rationality space...

In the South, we see the HPMOR peninsula; in the West, the "Posts" nebulae; and in the North/East, "Concepts" space.

It's not perfect — I'll be doing a lot of fine-tuning of broken links & missing tags as I work my way through all the content — but there should be enough already in-place that you can get some value out of it. I'd love to hear your feedback, and if you're interested in contributing, please reach out. I'll also soon be adding support for the AF and the EAF .

More generally, I'd love to hear your suggestions for new aspiring rationalists. For example, a round of users proposed alternative reading orders about a decade ago (by Academian, jimrandomh, and XiXiDu). Is it time for a revisit in 2022?

Check out the repo

^

From what I can scrape off the graphql endpoint.

^

Already a decade ago, jimrandomh was worrying about LW's intimidation factor — we're now about an order of magnitude further along.
Pondering the paucity of volcanic profanity post Pompeii perusal
by CraigMichael — 2022-09-01T09:29:40.122Z

[Content Warning: This post contains nasty vulgar language.]

My wife and I went to Italy this summer. She wanted to skip Pompeii to spend more time with friends in Umbria. I almost agreed, but at the last moment, I heard Leonard Nimoy's voice echoing in my head from his narration of

an episode of the mystery documentary TV series,

In Search of...

on the topic of that ancient Roman city buried by the 79 AD eruption of stratovolcano Mount Vesuvius. I saw that episode as a young lad, and from that day, I always wanted to see Pompeii. So, I watched some Andrew Tate clips and decided I wasn't going to cuck out like some weak beta. I stood up straight with my shoulders back and said:

Dear wife, kind wife, if it's not too much trouble for you, would you mind if we went to Pompeii just for a little bit? But if it's too much, then we don't have to bother.

She agreed. :) We went to Pompeii and hiked Mount Vesuvius. We both thoroughly enjoyed it. I've been kind of obsessed with it since. It's not uncommon for it to leave a strong impression on people.

I. The Jaggars

One such impressionable person was 10-year-old

Thomas Jaggar Jr.

, whose father, Southern Ohio

Bishop Thomas Jaggar

, took his family to Italy in 1885. In 1886 they visited Pompeii and Vesuvius. Junior wrote in his diary about the famous casts of Pompeiians buried in volcanic ash at the moment of the 79 AD eruption, remarking they were "about as strange and interesting as anything I've ever experienced."

One such case in Pompeii

A few weeks later, the only time he used a red pencil to write in his diary, Junior wrote about hiking Vesuvius. As quoted below, text from Junior's diary is interspersed with prose from John Dvorak, author of the excellent book,

The Last Volcano

.

At the upper station, the Jaggars stepped out of the car. They were surrounded immediately by dense fog, which made it impossible to see more than a few hundred feet. Several men offered themselves as guides. The Bishop chose three men, two to carry Mrs. Jaggar in a sedan chair [she was very ill at the time] and the other man to lead them to the crater’s edge.

“It was a very hard walk.” Her son wrote of the adventure, “over rough, jagged lumps of warm lava and sulfur, with now and then a bed of sandy ashes in to which we sunk up to the ankles.”

The air was so heavy with sulfur that he held a handkerchief ove

...
Infra-Exercises, Part 1
by Diffractor — 2022-09-01T05:06:59.373Z

There have been a few remarks about how my and Vanessa’s posts on Infra-Bayesianism seem interesting, but that it's quite a formidable body of work to chew through.

So anyone interested in learning Inframeasure theory to the level of proficiency where they are actually able to develop their own proofs about it and casually deploy it as a tool on other problems is probably not best served by the existing posts. You can only go so far by reading about something without ever applying it.

And so, to help people get a taste for the underlying math, I have constructed a sheet of exercises analogous to Scott's exercise sheets about fixpoints, to guide the reader through inventing most of the basic theory for themselves.

Said problem sheet was dramatically improved by Jack Parker, Connall Garrod, and Thomas Larsen as part of a SERI project, by including proof sketches, relevant definitions, and commentary, and generally making the whole exercise sheet dramatically more polished than it would otherwise be. Several others also contributed to testing out the exercises and providing solutions and feedback, most prominently, Viktoria Malyasova.
This first sheet focuses on introducing Inframeasures, and working through the Fundamental Theorem of Inframeasures. Admittedly, this is a bit distant from much of the later work (further problem sheets are upcoming), but it is a very important result to have a solid understanding of.

There is more information on how to approach the problems in the introduction of the document. We may potentially release more problem sheets in the future, depending on what the feedback to this one looks like. And if the polished versions are not created, I will publish the unpolished ones.

The link to the problems is right here.