trentmkelly/LessWrong-43k huggingface dataset:
Can We Align a Self-Improving AGI?
by Peter S. Park — 2022-08-30T00:14:45.655Z

Produced during the Stanford Existential Risk Initiative (SERI) ML Alignment Theory Scholars (MATS) Program of 2022, under John Wentworth

TL;DR: Suppose that a team of researchers somehow aligned an AGI of human-level capabilities within the limited collection of environments that are accessible at that level. To corrigibly aid the researchers, the aligned AGI increases its capabilities (e.g., deployment into the Internet, abstraction capabilities, ability to break encryption). But after the aligned AGI increases its capabilities, it may be able to access environments that were previously inaccessible to both itself and the researchers. These new environments may then cause the AGI to phase-transition into misalignedness, in a complex way the alignment researchers could not have predicted beforehand. A potentially necessary condition to predicting this beforehand may be a tall order: that the researchers themselves investigate the currently inaccessible environments, before allowing the AGI to access them by improving its capabilities. An analogy for this problem, named the College Kid Problem, is provided by the Menendez brothers' infamous murder of their parents.

Content warning: People who would prefer not to read about the murder in detail should skip to the section titled "The worst-case scenario for AI alignment efforts."

Post: José Menendez was just a teenager in Cuba when his family’s estate was seized by Fidel Castro’s new Communist government. José’s family consequently immigrated to America, where he worked really hard. His family couldn’t afford for him to attend an Ivy League college; he attended Southern Illinois University instead. José then passed the CPA exam, and eventually worked his way up to become a wealthy business executive (for example, as a vice president at RCA Records, he had a hand in signing world-famous bands like The Eurythmics and Duran Duran). While he wasn’t able to fulfill his lifelong dream of overthrowing Castro, José Menendez—against all odds—had achieved the American Dream as an immigrant.

Kitty and José Menendez, a smiling married couple

José’s first son, Lyle Menendez, grew up idolizing his father. He wanted to become a successful businessman just like him. And José, in turn, was a determined father. He wanted Lyle (and his other son, Erik) to access the educational opportunities in business that José himself ...
On the nature of help - a framework for helping
by Faustify — 2022-08-29T20:42:02.877Z

It is important to define ‘help’ before delving further into the topic. I would like to avoid using dictionary definitions when trying to be concrete about the essence of a word. Albeit sometimes useful, dictionaries strive to provide a short and concise explanation of a word, which often strips it off its essence, nuance, and connotation. Furthermore, they often suffer from circular definitions(i.e. a word is defined through another, which in turn is defined through the first). When one expresses themselves fluently in some language, they exercise a great degree of intuition, characterised by more low-level processes that cannot be explained through language.

‘Help’ is an action (hence - the act of helping) that is performed from one individual or group to another. This is where ‘Help’ can be distinguished from the utility measure of ‘Helpful’. A tool, such as a hammer, can be characterised as helpful (which is largely synonymous to useful), but it does not do the act of ‘helping’ me, as it is an inanimate object. Furthermore, ‘Help’ is performed voluntarily and without the expectation of compensation. These two points may overlap, since voluntarily can also mean the latter, hence the notion of a ‘volunteer’. However, the first characteristic of ‘Help’ highlights that it is performed out of one’s free will. One cannot be ordered or coerced into helping another, as that is not help, but work. The reasoning for the second characteristic is largely similar. If one helps another with the expectation for compensation, it is not help, but again - work. The butcher does not help me when giving me meat for compensation in the form of me giving him money in return. This is purely a mutually beneficial and instrumental transaction. It is not ‘Help’. Lastly, ‘Help’ is performed for the explicit and wanted benefit of the latter group. This point may be obvious, however, it is the true essence of ‘Help’. The benefit needs to be measured and requested by the party which is on the receiving end. This is also the point where I feel most ‘omnibenevolent good doers’ fail. Often unwanted and unbeneficial actions are performed under the guise of ‘Help’ in an attempt to guilt the receiving party into accepting their tyrant. Here I use ‘tyrant’ in a very loose and poetic manner, but tyrants indeed are those that usurp other’s free will and potential and dare to claim it...
Fundamental Uncertainty: Chapter 4 - Why don't we do what we think we should?
by Gordon Seidoh Worley — 2022-08-29T19:25:16.917Z

N.B. This is a chapter in a book about truth and knowledge. It is the first draft. I have since revised it. You can find the most up-to-date info/version on the book's website.

If you're like me, you've probably experienced the following scenario. You need to wake up at a certain time tomorrow, say 6 am. You also want to be well-rested for the day, so you want to get at least 7 hours of sleep. If you wake up late or don't get enough sleep, your day will be noticeably worse. You strongly believe that the right thing to do is to go to bed no later than 11 pm. Ten o'clock rolls around and you think "alright, I'll finish up what I'm doing and then go to bed". Next thing you know you look at the clock and it's midnight.

What happened? You knew what you wanted to do. You had a plan. And then you totally ignored it, stayed up too late, and were tired the next day. "What's wrong with me?" you might ask yourself. "I knew what to do and then just didn't do it!" Why did you stay up too late when you knew it wasn't the best thing to do?

Sometimes it's due to something simple. For example, maybe you were watching a show or reading a book and got caught up in it and lost track of time. It's not that you didn't want to go to bed on time, it's just that you didn't realize what time it was because your attention was elsewhere. No worries, that's a problem that can be easily overcome by setting an alarm to remind yourself to go to bed. Bedtime solved!

Only, one night you hear the alarm and think "oh, I can stay up a little later before going to bed". And then it's 1 am and you're tired the next day. So you try increasingly aggressive solutions: setting multiple alarms, getting your roommate or spouse to charge you $5 if you stay up late, rigging the lights in your home to turn off at 11. Each of these works for a while, but eventually you stay up too late anyway, and even if you do manage to find a solution that works, every night feels like a battle to make yourself go to bed on time.

And this problem is not restricted to sleep. There's all manner of things that people believe they should do but struggle to make themselves do. Here's some examples:

drink alcohol in moderation
eat a healthy diet
get regular exercise
not smoke
leave early to show up on time to appointments
drive without distractions
always tell the truth
do homework, taxes, bill paying, etc. on time

In each case many people...
How can I reconcile the two most likely requirements for humanities near-term survival.
by Erlja Jkdf. — 2022-08-29T18:46:58.083Z

1. We technologically plateau, due to humanities questionable ability to adapt to accelerating technological progression.

2. AI development is indefinitely disrupted; as it is likely to result in disaster.

This is unlikely to be done deliberately. Ongoing attempts to slow down AI development are relatively ineffective; it is more likely, in my opinion, that a basic form of AI developed in the near future will either directly increase all individuals power, or lead to technologies that do the same. An example would be the possible implications of Palm AI. https://www.theatlantic.com/technology/archive/2022/06/google-palm-ai-artificial-consciousness/661329/
This universal increase in power could be sufficient to disrupt all AI research indefinitely, with the actions of a minority working in unison.

All this considered, what's the most likely situation that could play out?
*New* Canada AI Safety & Governance community
by Wyatt Tessari L'Allié — 2022-08-29T18:45:59.945Z

Hello everyone,

Wyatt here, the new AIS/G field-builder for Canada. In June I received an EA LTFF fund grant to "connect, expand and enable the AGI Safety & Governance community in Canada", with the goal of making Canada a net positive force on the world stage for transformational AI.

We now have a slack server, events calendar, some proposed projects, and a monthly newsletter (coming out this Wednesday), so if you are based in Canada and want to be in the loop, sign up here.

Looking forward to the year ahead.
Are Generative World Models a Mesa-Optimization Risk?
by Thane Ruthenis — 2022-08-29T18:37:13.811Z

Suppose we set up a training loop with an eye to get a generative world-model. For concretedness, let's imagine the predictor from the ELK doc. We show the model the first part of a surveillance video, and ask it to predict the second part. Would we risk producing a mesa-optimizer?

Intuitively, it feels like "no". Mesa-objectives are likely defined over world-models, shards are defined over world-models, so if we ask the training process for just the world-model, we would get just the world-model. Right?

Well.

The Problem

... is that we can't actually "just" ask for the world-model, can we? Or, at least, that's an unsolved problem. We're always asking for some proxy, be that the second part of the video, an answer to some question, being scored well by some secondary world-model-identifier ML model, and so on.

If we could somehow precisely ask the training process to "improve this world-model", instead of optimizing some proxy objective that we think highly correlates with a generative world-model, that would be a different story. But I don't see how.

That given, where can things go awry?

The Low End

The SGD moves the model along the steepest gradients. This means that every next SGD step is optimized to make the model improve on its loss-minimization ability as much as possible within the range of that step. Informally, the SGD wants to see results, and fast.

I'd previously analysed the dynamics it gives rise to. In short: The "world-model" part of the ML model improves incrementally while it's incentivized to produce results immediately. That means it would develop some functions mapping the imperfect world-model to imperfect results — heuristics. But since these heuristics can only attach to the internal world-model, they necessarily start out "shallow", only responding to surface correlations in the input-data because they're the first components of the world-model that are discovered. With time, as the world-model deepens, these heuristics may deepen in turn... or they may stagnate, with ancient shallow heuristics eating up too much of the loss-pie and crowding out younger and better competition[1].

So, we can expect to see something like this here too. The model would develop a number of shallow heuristics for e. g. how the second part of the video should look like, we won't get a "pure" world-model either way. And that's a fundamental mesa-optimization precurso...
Sequencing Intro
by jefftk — 2022-08-29T17:50:02.064Z

I've been working in computational bio for a

couple
months now and I've been learning a lot. There's still a ton I don't
know, but I'm currently at a stage where I've put some pieces together
while still remembering what it was like not to understand them, which
is often a good time to try to write introductory stuff. Trying to
explain things is also a good way of making sure I understand them
myself. So, here's an dump of what I've been learning:

The biological world primarily stores information with nucleic
acids. These are series of nucleotides, often called
bases: A,
C, G, and T. For example, a
strand of nucleotides could look like:

GGCTGAGACAGTGCCCAGATCGTTACACCATTCGT

The two main kinds of nucleic acid are DNA and RNA. They differ in a
few ways, but from a computational perspective they're very similar.
Physical RNA will have U instead of T, though
in sequencing data you'll often see it with with T anyway.

Each base has a complement:
A bonds with T, and G with C. A nucleic acid that comprises two
bonded strands is called double stranded. Each base in one
strand will be bonded to its complement:

GGCTGAGACAGTGCCCAGATCGTTACACCATTCGT
|||||||||||||||||||||||||||||||||||
CCGACTCTGTCACGGGTCTAGCAATGTGGTAAGCA

This is the famous double
helix and makes for a more stable structure than single
stranded nucleic acid.

Going from a physical nucleic acid to a sequence on a computer
is sequencing, and the reverse is synthesis. I'm only
going to talk about the former; I haven't learned much about the
latter.

The most common sequencing method today is Next
Generation Sequencing, commonly called Illumina
sequencing after the main vendor. Bases are dyed and the machine
reads their colors. The output of sequencing is a large number of
short reads. Each read is a sequence of 50-300 bases, usually
around 150. In setting up the sequencing run you choose how many
bases to read, and different applications will make the most sense
with different lengths. Accuracy drops off as you read farther along
the strand. Note the lengths we're talking about are way less than
the length of a full nucleic acid, which is generally at least
thousands of bases. Not getting the full picture is a big downside of
this kind of sequencing.

Let's get some real data to play with. When people publish a paper
that depends on sequencing they general...
How Do AI Timelines Affect Existential Risk?
by Stephen McAleese — 2022-08-29T16:57:44.107Z

This report is my MLSS final project.

PDF version here.

Summary

Superhuman artificial general intelligence could be created this century and would likely be a significant source of existential risk. Delaying the creation of superintelligent AI (ASI) could decrease total existential risk by increasing the amount of time humanity has to work on the AI alignment problem.

However, since ASI could reduce most risks, delaying the creation of ASI could also increase other existential risks, especially from advanced future technologies such as synthetic biology and molecular nanotechnology.

If AI existential risk is high relative to the sum of other existential risk, delaying the creation of ASI will tend to decrease total existential risk and vice-versa.

Other factors such as war and a hardware overhang could increase AI risk and cognitive enhancement could decrease AI risk. To reduce total existential risk, humanity should take robustly positive actions such as working on existential risk analysis, AI governance and safety, and reducing all sources of existential risk by promoting differential technological development.

Introduction

Recent progress in AI suggests that artificial general intelligence (AGI) that is as capable as humans on a wide variety of tasks is likely to be created this century. Once AGI exists, further improvement of its abilities would enable it to surpass human intelligence resulting in the creation of a superintelligent artificial general intelligence that is vastly more capable than humans at many important cognitive tasks.

A similar concept is transformative AI which is defined as “AI that precipitates a transition comparable to (or more significant than) the agricultural or industrial revolution”. For the purpose of this report, I’m going to use the acronym “ASI” (artificial superintelligence) as a short-hand for superintelligent general AI that is “much smarter than the best human brains in practically every field, including scientific creativity, general wisdom, and social skills”.

After an ASI is created, it would probably have a significant and long-lasting effect on human civilization and its trajectory. An ASI could also be a significant source of existential risk and could result in a highly undesirable outcome such as human extinction.

Although recently created AI systems such as GPT-3 can accomplish a wide variety of tasks, their understand...
How might we align transformative AI if it’s developed very soon?
by HoldenKarnofsky — 2022-08-29T15:42:08.985Z

This post is part of my AI strategy nearcasting series: trying to answer key strategic questions about transformative AI, under the assumption that key events will happen very soon, and/or in a world that is otherwise very similar to today's.

This post gives my understanding of what the set of available strategies for aligning transformative AI would be if it were developed very soon, and why they might or might not work. It is heavily based on conversations with Paul Christiano, Ajeya Cotra and Carl Shulman, and its background assumptions correspond to the arguments Ajeya makes in this piece (abbreviated as “Takeover Analysis”).

I premise this piece on a nearcast in which a major AI company (“Magma,” following Ajeya’s terminology) has good reason to think that it can develop transformative AI very soon (within a year), using what Ajeya calls “human feedback on diverse tasks” (HFDT) - and has some time (more than 6 months, but less than 2 years) to set up special measures to reduce the risks of misaligned AI before there’s much chance of someone else deploying transformative AI.

I will discuss:

Why I think there is a major risk of misaligned AI in this nearcast (this will just be a brief recap of Takeover Analysis).

Magma’s predicament: navigating the risk of deploying misaligned AI itself, while also contending with the risk of other, less cautious actors doing so.

Magma’s goals that advanced AI systems might be able to help with - for example, (a) using aligned AI systems to conduct research on how to safely develop still-more-powerful AI; (b) using aligned AI systems to help third parties (e.g., multilateral cooperation bodies and governments) detect and defend against unaligned AI systems deployed by less cautious actors.

The intended properties that Magma will be seeking from its AI systems - such as honesty and corrigibility - in order to ensure they can safely help with these goals.

Some key facets of AI alignment that Magma needs to attend to, along with thoughts about how it can deal with them:

Accurate reinforcement: training AI systems to perform useful tasks while being honest, corrigible, etc. - and avoiding the risk (discussed in Takeover Analysis) that they are unwittingly rewarding AIs for deceiving and manipulating human judges. I’ll list several techniques Magma might use for this.

Out-of-distrib

...
An Audio Introduction to Nick Bostrom
by PeterH — 2022-08-29T08:50:10.739Z
Link: https://forum.effectivealtruism.org/posts/rK6dksjzCALqRTfDd/an-audio-introduction-to-nick-bostrom

Team Radio Bostrom has just released An Audio Introduction to Nick Bostrom.

It offers a deep dive into Nick Bostrom's ideas, including: existential risk, wise philanthropy, the ethics of AI, transhumanism, and the case for speeding up some investments in technology while slowing down others.

We'd love to hear your thoughts!

How you can help

If you'd like to help spread the world, please like and retweet this:
https://twitter.com/peterhartree/status/1564137587496947712

And please, share your thoughts on how we could improve this. Or tell us some other people who might be good for a similar series...

See also: Radio Bostrom launch announcement.
Please Do Fight the Hypothetical
by Lone Pine — 2022-08-29T08:35:41.513Z

Scott Alexander and others in this part of the internet have been debating the Repugnant Conclusion recently. (I'll assume you're up-to-date on this discussion. If you aren't, follow the link, or just google "Repugnant Conclusion" or "Mere Addition Paradox".)

Many have noticed that there is a problem with the argument: the world-state that is described as 'repugnant' actually isn't so bad. Trillions of people just chilling -- living out their lives -- and not, on balance, suffering, is a much better world than the one we currently inhabit, where the rabbit runs for his life and the fox runs for his dinner. So if you force me to accept the hypothetical, then my answer is yes, World Z please.

But my beef with the Repugnant Conclusion goes beyond the abstracted hypothetical. This discourse maps perfectly to a real world debate we are actually having in the real world of 2022. It's a question about hard policy issues, abstracted just enough to get past knee-jerk partisan reactions. It seems like few have noticed this connection, although I suspect that people do notice it but would prefer to elide it, since this ultimately does get into politics. The actual policy question is as follows: Is the world, today in 2022, overpopulated? And, was Malthus right?

The Repugnant Conclusion asks you to consider a number of possible world-states in which there are some number of living people, each with some amount of average happiness. The set of possible world-states offered is arranged such that the more populous worlds contain less average happiness. But why would you think that adding more people reduces the average happiness?

Imagine that the entire universe consisted of just one person, Sally, living on a small island. She can eat the nuts and berries from trees, and has a source of water, etc, so she is not suffering in the material sense. Call this World A. Now let's give her a friend, Kim, to keep her company. There are now two people, and still plenty of nuts and berries to go around. Call this World B. Hopefully you will agree that World B is much better than World A, not just in terms of total happiness, but in terms of average happiness. Sally now has a friend. But also, Kim and Sally working together can produce more material wealth than either can alone. Perhaps Kim is better at building shelters, and Sally is better at foraging. (Standard economics, blah blah)

No...
Have you considered getting rid of death?
by Willa — 2022-08-29T01:31:48.656Z
Link: https://immortalityisgreat.substack.com/p/have-you-considered-getting-rid-of?sd=pf

No person who wishes to continue living should have to die, ever. Check out the post on substack!

I believe that human immortality (increasing healthspan, eliminating causes of death, etc.) is something we as a species ought to urgently work towards. I am not a scientist, so I will primarily talk about the philosophical, political, and governance related implications of longevity.

I intend to increase my knowledge of biology & other important to immortality disciplines because that's enjoyable but also so that I may more accurately evaluate people, policies, and projects in the longevity / immortality space. Any suggestions for increasing my biology knowledge? I'm at about a "took biology for science majors in undergrad plus has read biology popular science books for fun" knowledge level right now.

EDIT 2022/09/13: I've created a Google Group called Immortality Studies to serve as a public discussion & mailing list plus coordination point for all of us interested in getting rid of death. I prefer to communicate about this subject area publicly, but you are welcome to message me here on LessWrong or email me at walambert@pm.me for private conversations (it may take me some time to get back to you since I have little slack in my life right now and am also prioritizing public conversations for coordination & transparency reasons).