trentmkelly/LessWrong-43k huggingface dataset:
Two reasons we might be closer to solving alignment than it seems
by KatWoods — 2022-09-24T20:00:08.442Z

I was at an AI safety retreat recently and there seemed to be two categories of researchers:

Those who thought most AI safety research was useless
Those who thought all AI safety research was useless

This is a darkly humurous anecdote illustrating a larger pattern of intense pessimism I’ve noticed among a certain contingency of AI safety researchers.

I don’t disagree with the more moderate version of this position. If things continue as they are, anywhere up to a 95% chance of doom seems defendable.

What I disagree with is the degree of confidence. While we certainly shouldn’t be confident that everything will turn out fine, we also shouldn’t feel confident that it won’t. This post might have easily been titled the same as Rob Bensinger’s similar post:

we shouldn’t be maximally pessimistic about AI alignment.

The main two reasons for not being overly confident of doom are:

All of the arguments saying that it’s hard to be confident that transformative AI (TAI) isn’t just around the corner also apply to safety research progress.
It’s still early days and we’ve had about as much progress as you’d predict given that up until recently we’ve only had double-digit numbers of people working on the problem.

The arguments that apply to TAI potentially being closer than we think also apply to alignment

It’s really hard to predict research progress. In

‘There’s no fire alarm for artificial general intelligence’,

Eliezer Yudkowsky points out that historically, ‘it is very often the case that key technological developments still seem decades away, five years before they show up’ - even to scientists who are working directly on the problem.

Wilbur Wright thought that heavier-than-air flight was fifty years away; two years later, he helped build the first heavier-than-air flyer. This is because it often feels the same when the technology is decades away and when the technology is a year away: in either case, you don’t yet know how to solve the problem.

These arguments apply not only to TAI, but also to TAI alignment. Heavier-than-air flight felt like it was years away when it was actually round the corner. Similarly, researchers’ sense that alignment is decades away - or even that it is impossible - is consistent with the possibility that we’ll solve alignment next year.
AI safety researchers are more like...
Orexin and the quest for more waking hours
by ChristianKl — 2022-09-24T19:54:56.207Z

A week ago I was skeptical about the prospect of radically reducing human sleep needs. After reading John Boyle’s

Cause area: Short-sleeper genes

, I decided to research the area more deeply and updated to believe that it’s more likely that we can reduce human sleep needs without significant negative side effects. It might increase risk-taking which has both positive and negative effects. The one friend I have that has short-sleeper genes is a startup founder.

Boyle suggested that one of the best actions to attempt would be using orexin or an orexin agonist as a drug, but that there’s currently a lack of funding for doing so.

Given the way the FDA and EMA work, drugs only get approved when they are able to cure illnesses, with an illness being anything that has an ICD code. According to that notion, people who suffer from having to sleep more than four hours don’t have an illness and thus drugs can’t be approved for that purpose. In practice, this results in the NIH not being interested to fund the research of Ying-Hui, about people who need a lot less sleep and are still well rested, that Boyle discussed.

DEC2-P384R and orexin biology

The DEC2 gene produces prepro-orexin, which is 131 amino acids long. People with the

DEC2-P384R mutation

produce more prepro-orexin and have a reduced need for sleep. From prepro-orexin our body generates orexin A, which is 33 amino acids long, and orexin B, which is 28 amino acids long. Orexin A is highly conserved and has the

same

molecular structure in humans, mice, rats, and cows, while human orexin B differs from rodent orexin B. While orexin B doesn’t cross the blood-brain barrier, orexin A does. I didn’t find information on whether or not prepro-orexin passes the barrier, but it likely doesn’t given its size.

According to

Uniprot

:

Orexin-A binds to both OX1R and OX2R with a high affinity, whereas orexin-B binds only to OX2R with a similar high affinity.

The literature is sometimes unclear when they use the term orexin about whether the author means prepro-orexin, orexin A, and orexin B or a mix of them. Hypocretin is an alternative name in the literature for orexin, hypocretin-1 for orexin A, and hypocretin-1 for orexin B.

Do we get a free lunch?

From an evolutionary perspective, it seems beneficial to have a lower sleep requirement, so we have to ask ourselves why DEC2-P384R didn’t pro...
AI Safety Discord community (requesting help!)
by Casey — 2022-09-24T17:35:16.629Z

This week I started the Safe AI Community.

It's an online community where individuals and groups can connect and support each other regarding AI safety work.

Twitter: https://twitter.com/SafeAICommunity

Discord: discord.gg/FeZbFnAvve

Why?

A multidisciplinary and open approach to AI safety will generate novel ideas and collaborations, and support the creation of AI that is informed by representative perspectives and moral views.
Forums, private research institutes and student groups do create a lot of value, but I believe there's a gap to be filled for a hub that anyone can access, meet others, and have live discussions. Learning about and contributing to AI safety should be open to all, and we can always do more to grow the community in numbers and diversity.

Primary goals:

To create an open space that welcomes anyone, regardless of background or skill-level.
To grow and promote AI safety from the 'bottom-up', to increase the diversity of thought and accessibility of AI safety.

What does the community provide?

We're just getting started, and have already populated our Discord with a number of features, including:

A resource database, with content in multiple formats (videos, podcasts, blogs, papers, etc.), including a beginners section that introduces high-level concepts.
A job/funding/collaboration section, which we will keep up to date (and soon automate). We're only a few days old, yet already have a professor offering to supervise applications on Future of List Institute research!
A place for discussion. We're starting with only a couple of channels, but as we grow we'll create specific topic sections and weekly discussion threads.

I'll soon be adding new features, such as:

Automatically re-posting LessWrong and AI Alignment Forum posts to the community to facilitate and encourage discussion around them. This can also be done for news, research papers, etc.
A notion workspace that mirrors the Discord resources and acts as an open AI safety wiki.

Please contribute!

This project is intended to be entirely community-driven, and needs your help to grow!

I'm looking for:

Anyone and everyone to join the community, we're only going to be as valuable as our members!
Suggestions/feedback. Obviously, I want to make this as valuable as possible. The project is inspired by making AI better through diverse perspectives and bottom-up contributions, and I belie

...
I'm planning to start creating more write-ups summarizing my thoughts on various issues, mostly related to AI existential safety.  

What do you want to hear my nuanced takes on?
by David Scott Krueger (formerly: capybaralet) — 2022-09-24T12:38:24.163Z

cross-posted to Twitter:
https://twitter.com/DavidSKrueger/status/1573643782377152514
Attempts at Forwarding Speed Priors
by james.lucassen — 2022-09-24T05:49:46.443Z

This post summarizes research conducted under the mentorship of Evan Hubinger, and was assisted by collaboration with Pranav Gade, discussions with Adam Jermyn, and draft feedback from Yonadav Shavit.

Summary

Forwarding priors is a subproblem of deceptive alignment, because if we want to use regularization to create a prior for our search over models that will disincentivize deception, we need to identify a prior that not only give us some useful guarantees but also induce inner searches to have similar guarantees.
I tried a bunch of stuff this summer to find priors that forward, and roughly none of it worked. So I’m just sharing the avenues I explored in roughly chronological order, to explain where each thread left off.
Using dovetailing as a toy model of what an inner search over algorithms might look like, we can write down some rough formulas for the prior implemented by a dovetailer, and (kind of) the cost of a dovetailer on such a prior. But this suggests very discontinuous behavior, and requires a bunch of strong and specific assumptions, so maybe it’s not the most useful model in general.
Minimum boolean circuit tree size does seem to forward, but at the cost of probably forbidding all generalization ability.
We can offer our models cheap tools to try and get object-level algorithms to occupy a greater fraction of the overall runtime cost, but this quickly runs into a variety of problems.
If we incentivize the model to do explicit instead of implicit meta-learning, we can access the code and runtime for lower-level algorithms that were previously inaccessible when run implicitly. However, this still leaves us with some problems, including a (relaxed) version of the original forwarding problem.
Average-case speed priors have a bias against large hypothesis classes which makes them favor lookup-table-like strategies, but worst-case speed priors leave all computations except the limiting case highly unconstrained.
It seems hard to prove that a fixed-point must exist, because the map from priors to priors that we are using is so discontinuous.

Motivation

Deceptive alignment seems like a real big problem. Especially because we can’t use behavioral incentives to prevent it. One alternative to behavioral incentives is regularization, AKA mechanistic priors - we look at the structure of the model to try and figure out if it’s deceptive or not, then penalize mode...
Announcing $5,000 bounty for (responsibly) ending malaria
by lc — 2022-09-24T04:28:22.189Z

It occurred to me recently that if a group of people eradicated malaria, say by exterminating Anopheles gambiae via gene drives, they would not be paid. That sounds like a potentially embarrassing mistake, so I'm hereby pledging $5,000 to anyone that manages to accomplish such a thing. This way if someone from Dath Ilan materializes into our world and asks us judgmentally whether or not Earth would even pay people for eradicating malaria, we can all respond "Our society definitely has an explicit, preregistered financial reward greater than $0 in place for anybody that does that" and sidestep some awkwardness.

Below I have written out the details for this new bounty in FAQ form.

Malaria Bounty FAQ

What specifically will lc wire me $5,000 for doing?

I will give you $5,000 if you permanently reduce global annual mortality from malaria by 95% or more. Eligibility for the reward is independent to how this is actually accomplished, save for two restrictions outlined below in the sections about unilateral action and negative side effects.

How will lc assign credit for solving malaria? Does developing the deployed technique count?

The money will be given to the person or group that directly causes the reduction in deaths. If a pioneering group of researchers wants to get my money, they will have to follow through by successfully managing and executing the deployment of their clever research to prevent people from dying.

I would like to handsomely reward both groups, but there are already many existing charities you can extract money from in advance, if you're working on tools to end malaria and want funding for that. So far, none of the (awesome) people developing those tools have opted to use them, partly because they insist on getting permission from local African governments, which are delaying them for political reasons. I reserve hope that I will some day have reason to send these researchers money, after a few more million people have died, but in the meantime there's not as much marginal benefit.

So can I get the bounty via unilaterally deploying biotech?

No, and this is the major restriction on payouts.

By "unilaterally" I mean: I am not going to award this bounty if other people with a strong understanding of the science or existing efforts suspect the project is/was catastrophically net negative, or if a post on LessWrong about the project leads users to point out ...
Why Do AI researchers Rate the Probability of Doom So Low?
by Aorou — 2022-09-24T02:33:19.113Z

I recently read What do ML researchers think about AI in 2022.

The probability of Doom is sub-10%. Which is high, but as I understand it, in the minds of people like Eliezer Yudkowsky, we're more likely doomed than not.

I personally lean towards Yudkowsky's views, because
- I don't believe human/evolution-selected minds have thinking power that a machine could not have
- I believe in the Orthogonality Thesis
(I think that those two questions can be defended empirically)
- I think it is easier to make a non-aligned machine than an aligned one
(I believe that research currently being carried out strongly hints at the fact that this is true)
- I believe that more people are working on non-aligned AI than on aligned AI
- I think it would be very hard politically to stop all AI research and successfully prevent anyone from researching it / to implement a worldwide ban on AI R&D.

Given all this (and probably other observations that I made), I think we're doomed.
I feel my heart beating hard, when I think to myself I have to give a number.
I imagine I'm bad at it, it'll be wrong, it's more uncomfortable/inconvenient than just saying "we're fucked" without any number, but here goes anyway-
I'd say that we're
(my brain KEEPS on flinching away from coming up with a number, I don't WANT to actually follow through on all my thoughts and observations about the state of AI and what it means for the Future)-
(I think of all the possible Deus-Ex-Machina that could happen)-
(I imagine how terrible it is if I'm WRONG)-
(Visualizing my probabilities for the AI-doom scenario in hypothetical worlds where I don't live makes it easier, I think)
My probability of doom from AI is around 80% in the next 50 years.
(And my probability of Doom if AI keeps getting better is 95% (one reason it might not get better, I imagine, is that another X-Risk happens before AI)).
I would be surprised if more than 1 world, out of 5 in our current situation, made it out alive from developping AI.

Edit, a week after the post

:
I'd say my P(Doom) in the next 50 years is now between 20-40%.
It's not that I suddenly think AI is less likely, but I think I put my P(Doom) at 80% before because I lumped all of my fears together as if P(Doom) = P(Outcome I really don't like).
But those two things are different.
For me, P(Doom) = P(humanity wipes out). This is different than a bad outcome like [A few people own all...
Set List Approaches
by jefftk — 2022-09-24T02:30:01.955Z

Now that

Kingfisher is
starting to have a bunch of sets I've been thinking some about how to
keep track of what we play, which gets into what kind of band we want
to be. There's a bit of a continuum, from dance bands with a list of
sets they play a specific way each time to ones that just put things
together entirely on the fly. There are advantages to each:

If you have a specific way you play a set it can generally be
very polished, in a way that can be great to dance to. You can put a
lot of thought into the arrangement, move a large group of musicians
through something complex while sounding very tight, make sharp
dynamic changes, and combine a lot of ideas.

If you're making things up in the moment, responding to each
other, the crowd, and the feeling in the hall, your music can have a
lot more life to it. You don't get stuck playing sets you're sick of,
and if you're playing several times in the same region the dancers get
some variety.

There's no one answer here: what will work best depends a lot on the
particular combination of musicians, how they work together, and what
they want. Larger groups generally benefit more from pre-arrangement,
though if you can allocate someone
to lead the band you can still put on a great dance with a large
band and no planning. I've enjoyed dancing to bands all over this
spectrum.

Bands also often move between these categories over time. The Free Raisins started off going into
each set with just some planned tunes and making things up as we went
(see our first CD), but
over time we would remember things we'd done before and call
those things to each other. Patterns we especially liked became the
way we would play those sets, and then the process of putting together our second CD
gelled them further. As of the start of the pandemic, while we still
would play around and improvise, if you heard a given set at two
different dance weekends it would probably sound pretty similar, and
to be honest we were getting a bit sick of having been playing most
of our sets similarly for ~5y. As we get back into playing together
(Ann Arbor's Tree Town
Stomp next month!) I'm not sure what we'll do!

I wrote to a few different musicians to get their takes:

Ethan Hazzard-Watkins:

I've been thinking about tune lists as well, but more
from the perspective of how to remember all the tunes I used to pl

...
A ranked link of LessWrong tags/concepts
by peterslattery — 2022-09-24T00:11:23.350Z

Epistemic status - a quick contribution. Possibly useless for anyone else. Also, a sort of feature suggestion.

This post provides a link to a Google sheet which ranks tags/concepts discussed on LessWrong by frequency of mention. I am not sure if it will be useful, but it was for me, so I thought I'd quickly share it.

What did I do?

On 22/9/22 I copied the tags from the Concepts Portal into a Google sheet. I did some text splitting and concatenation to split out the frequency counts and produce a long table ranked by frequency. I couldn't copy over the links, but they are in the column A, which can be unhidden. Below are the top 20 rows.

Concept	Mentions
AI	3382
World Modeling	2353
Rationality	2278
Practical	1667
World Optimization	1527
Community	1110
Covid-19	860
Site Meta	548
Open Threads	458
AI Risk	439
Fiction	414
Politics	305
Book Reviews	303
Ethics & Morality	293
Decision Theory	282
Logic & Mathematics	265
Social & Cultural Dynamics	256
Economics	245
Scholarship & Learning	245
Forecasting & Prediction	243

Why did I do this?

I am meeting more people in the rationalist community, and wanted a quick way to understand their key interests. I thought that seeing the frequency of tags/concepts used on LW would be a helpful overview.

What next?

If it seems useful for other people and is relatively easy, then it may be a good idea to allow users to rank tags by frequency of use on the Concepts Portal.

Right now, I'd personally be interested in seeing a list/ranking of the newest and trending tags (e.g., those that have growing fastest in the last month/year) on the concept's portal page, but that could just be related to my perhaps unusual interest in understanding the focus of the rationality community. It might also be me projecting an unrealistic level of sustained interest going forward.
Under what circumstances have governments cancelled AI-type systems?
by David Gross — 2022-09-23T21:11:48.405Z
Link: https://www.carnegieuktrust.org.uk/publications/automating-public-services-learning-from-cancelled-systems/

"This study identifies 61 occasions across Australia, Canada, Europe, New Zealand and the United States when ADS [automated decision systems] projects were cancelled or paused."

The study gives a run-down of what sorts of systems were cancelled (or banned), how the cancellation came about (e.g. via external legal action or some internal decision), and the role of "civil society critique" in the decision-making.

"In combination, these findings suggest there are competing understandings of acceptable data practices. Given the civil society mobilisation, legal challenges and the number of interventions by oversight bodies, our research suggests that there are competing understandings about effectiveness, impact, accountability and how data systems can infringe people’s rights across areas of application. A key issue is that it takes a lot of work and resources to challenge a data system once in place."
There are no rules
by unoptimal — 2022-09-23T20:47:16.417Z

Crossposted to my blog

Over the course of this past year, I’ve slowly become a magician.

No, not a literal one. But in the sense that I’ve become a version of myself that I previously

would’ve called a magician.

I feel less social anxiety and awkwardness when asking people for things. I feel more inclined to find unusual or uncomfortable answers to problems. I have gotten noticeably better at not letting uncertainty and hypotheticals stall action.

Before, I would quickly concede if told it wasn’t possible to get a Thing I wanted. Now, I’m usually able to pry until I get that Thing, or at least come away with a compromise or special arrangement — unless the Thing I want really isn’t possible.

And it feels really good.

As I think about what’s contributed to this mindset shift, one phrase immediately comes to mind — well, the internalization of one phrase comes to mind: there are no rules.

Humans are prediction machines. We expect things to unfold in a certain way because that’s what happened in the past.

This is generally a very useful heuristic, since it saves the mental energy of thinking about what’ll happen next or how we should respond in certain contexts — when you pass by a friend on the street, you don’t need to think very hard about what social niceties to say.

But it’s too easy to fall into these predictable patterns. Invisible social norms develop. Interpersonal dynamics slowly settle. People sleepwalk through daily routines and events.

This can be dangerous if you don’t think about it once in a while, because you can forget that things don’t necessarily happen the way you think they should or would.

Because the world is not static — there are no rules! You can simply do most things, and usually with minimal social costs, if any. [1]

(It’s at this point I should explicitly state: the idea that “there are no rules” is more of a mindset than a literal phrase. Many people are kind, willing to help, and tolerable. But don’t abuse that trust and goodwill; don’t overtly disregard how your actions affect others. See

Evie's recent post, which I hadn't seen before finalizing this post

.)

Some examples:

I skipped most of my classes last semester because the lecture slides were online. In one of them, I left immediately after taking the short quiz given at the start of class. I got access to the same content, spent

...