trentmkelly/LessWrong-43k huggingface dataset:
AI as a Civilizational Risk Part 1/6: Historical Priors
by PashaKamyshev — 2022-10-29T21:59:26.858Z

TL; DR (for all six parts)

Empires have collapsed previously due to decreases in social cohesion. Narrow AIs, such as large language models combined with overly aggressive social media A/B testing aimed at behavioral modification, can cause decreases in social cohesion. This anti-economy (economy of externalities) of behavioral modification centralizes benefits to a few tech controllers, distributes costs, and is similar to physical pollution. Virtual signal pollution of vital personal signs of trust, such as AI-enabled scams and spam, is especially worrisome in a low social cohesion environment. Other narrow AIs aimed at molecular assembly and runaway tech risks such as bio-lab leaks compound already existing problems. Civilizational risk (c-risk), while bad enough by itself, can lower civilizational competence. Civilization needs this competence to deal with x-risks, which may also arise along the same causal pathways. However, caring decision-makers can take steps to reduce the impact of AI's improper modification of groups and individuals.

Difference from Other Models of Civilization

In this essay, I discuss narrow current AI and general future AI as a civilizational risk. Civilizational risk (c-risk) is separate from existential risk (x-risk) and involves the collapse of nations similar to prior historical collapses. C-risk can be made worse by un-aligned narrow and general AI. My probabilities of c-risk: P(c-risk by 2033 | lack of narrow alignment) = 31% AND P(c-risk by 2070 | lack of narrow or general alignment) = 92%, where c-risk is the collapse of Western Civilization. I also argue that P(misalignment x-risk|AGI) > 35% by 2070 in part 5, after establishing the likely backdrop of civilizational risk.

While I agree with Eliezer and Bostrom's view that an extremely powerful AGI could destroy humanity or drastically limit its potential, I highly disagree with the likely pathway civilization would take towards that point, conditioned on safety research not being done.

One of Eliezer's models is that "it is going to be increasingly cool right up until … "an existential failure point of fast unaligned FOOM. To be more precise, in

Soares, Tallinn, and Yudkowsky discuss AGI cognition - LessWrong

, Eliezer considers multiple possible phases, such as "Plotting," "Hidin...
Don’t expect your life partner to be better than your exes in more than one way: a mathematical model
by mdd — 2022-10-29T18:47:23.625Z

In this post, I analyze a multidimensional version of the dating problem (more commonly called the secretary problem). If you are familiar with this problem you can search for “For this post, I want to make a slight change to the model” and skip to that. Otherwise, here is an intro to the usual dating problem:

Suppose you are a serially monogamous dater and your goal is to eventually find someone to marry. Then you will have to make decisions of the type: should I settle with my current partner or reject them in the hope of finding someone better?

This is the setup for the dating problem. In this model, we assume there is some maximal number, n, of partners you can meet. Here “meet” can mean different things. You could for example interpret it to mean “going on a single date” or “been dating for a year”. Depending on the interpretation, n will have very different sizes. The model does not take into account that information is gradually revealed over time.

As soon as you have “met” someone, you know how good they are compared to previous people you have met, but you have no further information about how good they are compared to people you have not met yet. You will meet the partners in completely random order. You have to reject one partner before you can meet another one, and you can never go back to a person you have rejected. How do you maximize the probability of settling with the best partner?

The optimal strategy is to reject the k first people, where k is around n/e, and e is the base of the natural logarithm. After this exploration phase, you settle down with the next person you meet who is better than all previous partners.

Two types of failures can happen when following this algorithm:

Your best partner is among the k first people you meet, so you rejected them.
You never meet your best partner. Instead, you settle down with someone else who is better than all the first k candidates but you met before you have explored enough to meet your best partner.

For this post, I want to make a slight change to the model: we have an infinite sequence of potential partners (yeah, I know, Tinder has messed up my world model) of random identically distributed attractiveness. You have dated the first k and plan to settle down with the next person you meet who is better than all previous partners. You know there are better potential partners out there, but all you care ab...
The Social Recession: By the Numbers
by antonomon — 2022-10-29T18:45:09.001Z
Link: https://novum.substack.com/p/social-recession-by-the-numbers

Fewer friends, relationships on the decline, delayed adulthood, trust at an all-time low, and many diseases of despair. The prognosis is not great.

By Anton Stjepan Cebalo

Intermission (also known as Intermedio) by Edward Hopper, 1963.

One of the most discussed topics online recently has been friendships and loneliness. Ever since the infamous chart showing more people are not having sex than ever before first made the rounds, there’s been increased interest in the social state of things. Polling has demonstrated a marked decline in all spheres of social life, including close friends, intimate relationships, trust, labor participation, and community involvement. The trend looks to have worsened since the pandemic, although it will take some years before this is clearly established.

The decline comes alongside a documented rise in mental illness, diseases of despair, and poor health more generally. In August 2022, the CDC announced that U.S. life expectancy has fallen further and is now where it was in 1996. Contrast this to Western Europe, where it has largely rebounded to pre-pandemic numbers. Still, even before the pandemic, the years 2015-2017 saw the longest sustained decline in U.S. life expectancy since 1915-18. While my intended angle here is not health-related, general sociability is closely linked to health. The ongoing shift has been called the “friendship recession” or the “social recession.”

My intention here is not to present a list of miserable points, but to group them together in a meaningful context whose consequences are far-reaching. While most of what I will outline here focuses on the United States, many of these same trends are present elsewhere because its catalyst is primarily the internet itself. With no signs of abating, a new kind of sociability has only started to affect what people ask of the world through the prism of themselves.

The topic has directly or indirectly produced a whole genre of commentary from many different perspectives. Many of them touch on the fact that the internet is not being built with pro-social ends in mind. Increasingly monopolized across a few key entities, online life and its data have become the most sought-after commodity. The everyday person’s attention has thus become the scarcest resource to be extracted. Other perspectives, often on the left, also stress economic precarity and the decline of public spa...
Electric Kettle vs Stove
by jefftk — 2022-10-29T12:50:04.377Z

There was a discussion at lunch yesterday: what's faster to boil
water, an (American) electric kettle or a pot on the stove? The
kettle is more efficient but plugs into a 120V 15A receptacle on a
shared circuit, while the stove has to transfer heat to a pot but gets
exclusive use of a 240V 50A circuit (though not all of that goes into
a single burner). I thought the stove, a coworker thought the kettle.
Let's find out!

The contestants:

A Homigrand electric kettle, rated as drawing 1000W.

A Faberware stainless steel pot on a GE Spectra glass-top resistive
electric stove (JBP66BBWH).
The 6" burner I used is rated as drawing 1500W.

I filled them both with a quart (about a liter) of water, which was
near the capacity of the kettle, and verified that both were starting
at the same temperature (62F):

The thermometer is wet in the second picture because I accidentally
dropped it into the pot, but it's rated IP66 and didn't
mind.

I started them simultaneously, and waited. The kettle took 6:32 to
reach its automatic shutoff, and I measured at at 212F. The pot took
7:20 until I guessed it was boiling from the sound, and it also
measured 212F. So the pot is about 12% slower despite using 50% more
power, so 68% more energy. Still, it's only 0.18kWh vs 0.11kWh which
at $0.21/kWh is $0.04 vs $0.02 so
I'm more interested in the time difference. Even that is small
enough, however, that if what I want is boiling water in a pot to cook
something I think I'll still use the pot.

One exception, which I was already doing, is that when I'm making
pasta in a hurry, typically for the kids, I'll put half the water in
each and then pour the kettle into the pot when it boils. (If I had
been dividing the water exactly I'd have already known the answer to
this question!) The main gains here are from parallelism, however, not
efficiency.

This does make me wonder whether Americans (or British expats) ever
put dedicated 240V outlets in their kitchens so they can use ~2-3x
more powerful European kettles?
Quantum Immortality, foiled
by Ben — 2022-10-29T11:00:01.038Z

Some silliness based on recent discussion of quantum suicide and the redaction machine.

Bob found himself, quite suddenly, in a chamber walled with hexagonal plates of gleaming copper.

Claire welcomed him back to life, and gave him some clothes.

"So, where am I?" asked Bob. He thought he remembered Claire from the lab. She worked in the group across the hall.

"You have just been reset back to life by a redaction machine. You died you see, so we used this experimental machine to bring you back. It unwinds an object to a previous state in time. We reset you back to how you were this morning."

"Yeah, I was on my way into the university. So, huh, how did I die?"

Claire frowned. "It was quite weird. You were having an argument with Alice about the many worlds interpretation of quantum mechanics. In order to prove to her that this theory was correct you built an apparatus that would use a quantum random number generator to kill you with 99.99% probability. And well, it killed you."

"What!" screamed Bob. "But, you don't understand. By bringing me back you have retroactively ruined the whole experiment..."

"Really?" Claire raised an eyebrow. "Can you explain how?"

"Well of course." said Bob. "See, if the many worlds interpretation of quantum mechanics is correct then, according to the anthropic principle, the only version of me left to update their priors after the apparatus activates is the one who survives. So I can only update towards believing in the many worlds interpretation. Therefore I can believe it with certainty, because I know that an arbitrary number of such experiments could be performed, and they would result in me believing the theory. If you know that a course of action would result in yourself believing something then you might as well jump ahead and believe it."

"But, if the counter-factual is enough, why did you actually kill yourself?"

"Well, the last thing I remember I was on the bus in to work. But I imagine the reason I did so was because I wanted to prove it to Alice. She never takes my theories seriously."

"She seemed quite distraught" said Claire. "It was lucky this prototype redaction machine was ready this very day. In a way you killing yourself was a bit of a boon to us, we really wanted to try a resurrection. Remember I was talking about it the group seminar last week."

Bob didn't remember, he had been checking his emails in that talk. "You s...
Some Lessons Learned from Studying Indirect Object Identification in GPT-2 small
by RowanWang — 2022-10-28T23:55:44.755Z
Link: https://arxiv.org/abs/2211.00593

To learn more about this work, check out the

paper

. We assume general familiarity with

transformer circuits

.

Intro:

There isn’t much interpretability work that explains end-to-end how a model is able to do some task (except for toy models). In this work, we make progress towards this goal by understanding some of the structure of GPT-2 small “in the wild” by studying how it computes a simple natural language task.

The task we investigate is what we call indirect object identification (IOI), where sentences like “When John and Mary went to the store, John gave a drink to” should be completed with “Mary” as opposed to “John”.

We discovered the structure of a circuit of 26 attention heads grouped into 7 main classes, the largest end-to-end attempt to reverse engineer how a LM computes a natural behavior (to our knowledge). There is still much missing from our explanation, however, and our explanation doesn’t go to the parameter level.

The circuit we identified performing indirect identification in GPT-2 small

Besides discovering the particular circuit shown above, we gained some interesting insights about low-level phenomena arising inside language models. For example, we found attention heads communicating with pointers (sharing the location of a piece of information instead of copying it). We also identified heads compensating for the loss of function of other heads, and heads contributing negatively to the correct next-token prediction. We're excited to see if these discoveries generalize beyond our case study.

Since explanations of model behavior can be confused or non-rigorous, we used our knowledge to design adversarial examples. Moreover, we formulate 3 quantitative criteria to test the validity of our circuit. These criteria partially validate our circuit but indicate that there are still gaps in our understanding.

This post is a companion post to our paper where we share lessons that we learned from doing this work and describe some of Redwood’s interpretability perspectives. We share high-level takeaways and give specific examples from the work to illustrate them.

Goals of Interpretability for this Investigation

In this kind of mechanistic interpretability work, we tend to use the circuits abstraction. If we think of a model as a computational graph where nodes are terms in its forward pass (ne...
Resources that (I think) new alignment researchers should know about
by Orpheus16 — 2022-10-28T22:13:36.537Z

EDIT: This post was last updated in February 2023. It may no longer be fully up to date.

Here are some resources that I think new alignment researchers should know about.

Starting off

AGI Safety Fundamentals

is the most common starting point. There’s also the (new)

Alignment 201 course

, which is meant to be read after AGI safety fundamentals. Note that one disadvantage of reading a popular curriculum is that it might get you thinking in the same ways as everyone else (which might be especially concerning if

people are currently coming up with the

same

ideas

)

.

Exposing yourself to different sources everyone else seems like a pretty good way to try to come up with different ideas than everyone else. As a result, I often recommend people spend a few weeks trying out alternative approaches.

For example, you might start thinking about the problem for yourself, diving deeply into the agenda of a researcher you respect, writing down your

naive hypotheses about AI alignment

, diving into

list of lethalities

,

writing about what you agree with and disagree with

, and trying out

alignment problems

, or performing other exercises (see below).

On the margin, I think more people should try out these “alternative” approaches. I’m especially excited when people are able to write, distill, and analyze works relatively early on in their upskilling journey.

There are also some readings that are meant to help you think about how to think about alignment. These are mostly about how to think rather than what to think (or what to think about), which makes me more comfortable recommending them widely. (Note though that the point about diversifying and seeking out alternative perspectives still applies).

Some examples of readings about how to think about alignment research:

Most people start with the same few bad ideas

(4 mins)

7 traps that (we think) new alignment researchers often fall into

(4 mins)
More is different for AI (3 mins)

Worst-case thinking in AI alignment

(8 mins)

Principles for alignment/agency projects

(5 mins)

Taboo “outside view”

(10 mins)

Nonetheless, I think AGI Safety Fundamentals is a decent place to start. I recommend completing it quickly (I expect you can do it in 1-4 weeks, as opposed to 8. Though note that it is useful to spend time thinking, reflecting, and writing).

Diving Deeper

OK, but what...
How often does One Person succeed?
by Mayank Modi — 2022-10-28T19:32:46.956Z

Disclaimer

The following post is not a study on the campaigns or movements started by a single person. Neither does it deal with the data generated by the successful implementation of a project. Instead, this post is trying to present a case study of an ongoing campaign and several potentially failed attempts, commonly deserted in the process along with their data. This post will mainly contain Indian references, and everything is backed by the remains of the evidence.

First

As of October, there is one very hyped news going on involving the demise of a student in one of the top institutions of India.[1] The outlook of this kind of story may come as a shock to many, and become unbelievable as you dive deeper. In reality, this is a fairly common scenario and the only shocking thing is that it came to the surface. For those who are unfamiliar- big tech giants often have their Indian roots in these universities [2] which was often dubbed as the best hub for a technical career among my peers.

Second

We don't encounter such kind of news often; maybe once a year, twice at max. And it's obviously a big issue throughout the world: students suffering from poor mental health, several unfortunate events, and several actions for prevention and improvement. What's concerning about the cases happening in India, especially in its highly regarded premier institutions is: they are way bigger and more disturbing than what gets mentioned in the ones coming out. (Also, one should not confuse the cases coming out as automatically the biggest ones.)

Third

In the last two weeks, there have been two more such tragedies in a similar institution, mine. (There is no footnote available as the info isn't and won't be out) I know because of the vague condolences meeting emails I received, followed by customary mail for mental health awareness. Unfortunately, I am not allowed to go to those meetings(reason mentioned further in the post), but most of the people on campus know how these things are handled and long gave up hope. Both of those deaths are deemed suicides on the campus, from academic harassment which is dubbed and used as academic pressure.

Last year, after being on the brink of such a scenario myself and immediately after the tragedy of another colleague in June, I decided not to complete my remaining master's and to take on a project of my own in which I wou...
aisafety.community - A living document of AI safety communities
by zeshen — 2022-10-28T17:50:12.535Z

Thanks to plex for co-authoring the post (co-authors are currently not reflected in EA Forum when crossposted from LessWrong).

The AI Safety Communities logo, by DALL-E 2

The AI safety field has been rapidly growing over the last few years, and more and more communities have been sprouting up all over the world, both physically and online.

The Alignment Ecosystem Development (AED) team identified having a living document listing them all as a low-hanging fruit, and volunteers are maintaining this

live database of AI safety communities

.

Here’s what you can do now:

Join communities you’re interested in and participate.
Link this to your friends.
Add communities to the document if they’re missing (account creation with coda.io is required)
Improve the descriptions of any communities which you’re a part of. Many of the current descriptions are incomplete or outdated, jump right in and share your knowledge.
Volunteer as a maintainer, or join the Alignment Ecosystem Development

Discord server

, where they host

monthly calls to pitch or join shovel ready volunteering opportunities

.

Thanks to JJ Hepburn from

AI Safety Support

for providing the domain through

ea.domains

, another soon-to-be-launched AED project!
Rapid Test Throat Swabbing?
by jefftk — 2022-10-28T16:30:07.745Z

Back in January there was a bunch of buzz around using standard rapid
tests but swabbing the throat. Ex:

On one side are well-regarded experts who argue that swabbing your
throat in addition to your nose may increase the chances that a home
test can detect omicron. On the other side are similarly well-regarded
experts, including those at the Food and Drug Administration, who urge
Americans to avoid experimenting with kits that were developed and
tested using only nasal swabs.

—Washington
Post, 2022-01-06

or:

if you want to know if you have infectious Covid-19, you need to
use throat swabs. It's still not 100% sensitivity or anything, but
it's a lot better, whereas nose swabs are useful but in no way all
that reassuring.

—Zvi, 2022-01-13

Then the buzz died down, and it seems like everyone just forgot about
it? No clear recommendation either way, just a lot of Tweets, a few
blog posts, news stories about the controversy, and then not much
else. Google Trends shows the spike and drop-off:

[EDIT: I wouldn't read too much into the chart: it turns out that most covid-related keywords followed as similar trend, as did covid itself]

What happened? Why did we all stop talking about it, without what
seems like any firm conclusions? Some of this is maybe that there
were conflicting studies: Schrom et. al. posted a preprint
on 2022-01-19 (published
version) where it looked like it worked poorly, but Goodall
et. al. posted a
different preprint on 2022-01-21 (published
version) where it did look good. I also saw speculation that some
of the cases where people got a negative with nasal swabbing but a
positive with throat swabbing may have been false
positives—rapid tests are sensitive
to pH, and especially if you've been eating or drinking recently
your throat might be pretty far from neutral.

Ten months later, why hasn't this settled out into recommendations
that either say to swab throat and then nose, or that seem to have
considered the new evidence and say still don't? We're doing a lot of
testing, and if this is a better way to be using the tests we already
have it would be great to know! Does anyone whose been following this
have a better understanding of what happened?

Comment via: facebook