trentmkelly/LessWrong-43k huggingface dataset:
Advice for new alignment people: Info Max
by Jonas Hallgren — 2023-05-30T15:42:20.142Z

Epistemic status: This text is meant for other people in a similar position to me who are deciding how they can help ensure that AI doesn’t kill us all.

I hope that the new perspective gives you some new ways of thinking about your role in alignment, even though it should be treated as just that, another perspective on the problem.

As a young math and information nerd, I have a couple of ways of changing the world for the better. I believe that AI Safety is the most important thing I could be doing, and the question after that inevitably becomes what I should do in AI Safety. The most obvious answer is technical research, but that might not be the best way of getting high impact.

So I did the natural thing and asked some cool people what they thought. 80k has the advice to work on a problem area and develop aptitudes, but that is very unspecific. That standard advice is true, but it can be amplified like a *cough* motherf*cker.

When asking Connor Leahy from Conjecture what the hell I should do with my life, I got the answer that I should go to the Bay and create a tech company. This might not sound like something coming out of the mouth of someone with short timelines and a high percentage of doom, yet it did.

Connor told me that the idea behind it was that there would be pivotal points in the development of AI where we would need competent and trustworthy people to lead projects. People who are both trustworthy and competent are a very valuable resource in this world, yet it might seem very distant to someone just starting their career.

For me, becoming trustworthy was the new goal because it made sense when I think about fat-tailed impact. However, with my short timelines, I’ve realised that I can’t just take an ordinary 10-year route and do some cool research to become respected and trustworthy, as I just won’t have the time to do so.

So what am I doing instead? My current strategy is asking myself how I can become competent and respected in the quickest way possible. In my head, I refer to it as "Infomaxing."

Info Maxing

You might now wonder, "How do you define info maxing?" Info maxing, in my head, is the strategic acquisition of a wide array of skills and experiences to enhance one's overall competency or, in short, maximising for valuable mental models. But to what end?

In my opinion, based on Connor's advice, ...
Who is liable for AI?
by jmh — 2023-05-30T13:54:45.776Z

[Edit -- Changed to title as suggested by Raemon. So I've also made some edits to the post itself.]

Where does liability fall in the case of AI doing things?

I think there is a rather close parallel with autonomous cars/vehicles. Probably 4 or 5 years back that was a question being raise. I assume since I don't hear much about it any more it's been resolved and clearly defined for those making and using such vehicles or insuring such vehicles. Perhaps that translates very well into the more general AI space. But I'm not sure.

Anyone aware of where the whole question of liability assignment and insurance related to producing and using very advanced AI tools and products in our day to day interactions currently stands?
AI Safety Newsletter #8: Rogue AIs, how to screen for AI risks, and grants for research on democratic governance of AI
by Dan H — 2023-05-30T11:52:31.669Z
Link: https://newsletter.safe.ai/p/ai-safety-newsletter-8

Welcome to the AI Safety Newsletter by the

Center for AI Safety

. We discuss developments in AI and AI safety. No technical background required.

Subscribe here to receive future versions.

Yoshua Bengio makes the case for rogue AI

AI systems pose a variety of different risks. Renowned AI scientist

Yoshua Bengio

recently

argued

for one particularly concerning possibility: that advanced AI agents could pursue goals in conflict with human values.

Human intelligence has accomplished impressive feats, from flying to the moon to building nuclear weapons. But Bengio argues that across a range of important intellectual, economic, and social activities, human intelligence could be matched and even surpassed by AI.

How would advanced AIs change our world? Many technologies are tools, such as toasters and calculators, which humans use to accomplish our goals. AIs are different, Bengio says. We often give them a goal and ask them to figure out a solution on their own.

Choosing safe goals for AI systems is an unsolved problem, both technically and politically. If we do not solve this problem, Bengio argues that we could end up building AI agents that pursue harmful goals with superhuman intelligence, which would result in a catastrophe for humanity.

Four steps to rogue AI. Bengio begins by defining rogue AI as “an autonomous AI system that could behave in ways that would be catastrophically harmful to a large fraction of humans, potentially endangering our societies and even our species or the biosphere.”

He argues that a rogue superintelligent AI agent is possible, in four steps:

Machines could reach and surpass human abilities. Bengio argues that, in principle, there is nothing the human brain does that a machine could not do. We’ve already built AIs that beat humans in chess, games, and other cognitive tests. These systems have important advantages over human intelligence. Computers can process information much more quickly than humans – for example, language models are trained by reading the entire internet, which would be impossible in a human lifetime. Humans take years to reproduce, but an AI system can be replicated on to many computers at once. These kinds of simple advantages mean that if we develop human-level AI, we might soon thereafter get superhuman AI.
AIs can be turned into agents which take actions to pursue goals. We’ve already seen examples of GPT-4

...
The bullseye framework: My case against AI doom
by titotal — 2023-05-30T11:52:31.194Z

Introduction:

I’ve written quite a few articles casting doubt on several aspects of the AI doom narrative. (I’ve starting archiving them on my substack for easier sharing). This article is my first attempt to link them together to form a connected argument for why I find imminent AI doom unlikely.

I don’t expect every one of the ideas presented here to be correct. I have a PHD and work as a computational physicist, so I’m fairly confident about aspects related to that, but I do not wish to be treated as an expert on other subjects such as machine learning where I am familiar with the subject, but not an expert. You should never expect one person to cover a huge range of topics across multiple different domains, without making the occasional mistake. I have done my best with the knowledge I have available.

I don’t speculate about specific timelines here. I suspect that AGI is decades away at minimum, and I may reassess my beliefs as time goes on and technology changes.

In part 1, I will point out the parallel frameworks of values and capabilities. I show what happens when we entertain the possibility that at least some AGI could be fallible and beatable.

In part 2, I outline some of my many arguments that most AGI will be both fallible and beatable, and not capable of world domination.

In part 3, I outline a few arguments against the ideas that “x-risk” safe AGI is super difficult, taking particular aim at the “absolute fanatical maximiser” assumption of early AI writing.

In part 4, I speculate on how the above assumptions could lead to a safe navigation of AI development in the future.

This article does not speculate on AI timelines, or on the reasons why AI doom estimates are so high around here. I have my suspicions on both questions. On the first, I think AGI is many decades away, on the second, I think founder effects are primarily to blame. However these will not be the focus of this article.

Part 1: The bullseye framework

When arguing for AI doom, a typical argument will involve the possibility space of AGI. Invoking the orthogonality thesis and instrumental convergence, the argument goes that in the possibility space of AGI, there are far more machines that want to kill us than those that don’t. The argument is that the fraction is so small that AGI will be rogue by default: like the picture below.

As a sceptic, I do no...
Statement on AI Extinction - Signed by AGI Labs, Top Academics, and Many Other Notable Figures
by Dan H — 2023-05-30T09:05:25.986Z
Link: https://www.safe.ai/statement-on-ai-risk

Today, the AI Extinction Statement was released by the

Center for AI Safety

,

a one-sentence statement

jointly signed by a historic coalition of AI experts, professors, and tech leaders.

Geoffrey Hinton and Yoshua Bengio have signed, as have the CEOs of the major AGI labs–Sam Altman, Demis Hassabis, and Dario Amodei–as well as executives from Microsoft and Google (but notably not Meta).

The statement reads: “Mitigating the risk of extinction from AI should be a global priority alongside other societal-scale risks such as pandemics and nuclear war.”

We hope this statement will bring AI x-risk further into the overton window and open up discussion around AI’s most severe risks. Given the growing number of experts and public figures who take risks from advanced AI seriously, we hope to improve epistemics by encouraging discussion and focusing public and international attention toward this issue.
Theoretical Limitations of Autoregressive Models
by Gabriel Wu — 2023-05-30T02:37:51.020Z
Link: https://gabrieldwu.github.io/limitations-autoregressive

Thanks to Alex Cai for the discussion that inspired this post.

The Autoregressive Type Signature

Let's say you want to train an AI chatbot. It needs to be able to take in text (the user's question or instructions) and output text (its response). Intuitively you would think that the model, as a function ϕ from inputs to outputs, should have the following type signature:

ϕ:Text→Text

or more precisely,

ϕ:Σ∗→Δ(Σ∗)

where Σ is the set of all possible ASCII characters (or in practice, "tokens"), ∗ is the Kleene star, and Δ(S)⊂R|S| denotes the space of all probability distributions over a set S.

Unfortunately, it's basically impossible to directly design an architecture of this form. Any neural-network based model needs an output space of constant dimension, which Δ(Σ∗) is not (because there are an infinite number of possible strings).

To get around this, people came up with the ingenious idea of generating text autoregressively. Instead of having the model generate its entire response at once, we just train it to output the very next character of its output, given the user's question and its output so far. This has the type signature

ϕ:Σ∗×Σ∗→Δ(Σ).

For a given question Q∈Σ∗, we can generate a response character by character. First, sample

x0∼ϕ(Q,‘‘").

Then, sample

x1∼ϕ(Q,x0)x2∼ϕ(Q,¯¯¯¯¯¯¯¯¯¯¯x0x1)x3∼ϕ(Q,¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯x0x1x2)⋮

repeating this process until the model outputs a special stop token [STOP]∈Σ that indicates it's done with its output. Under this procedure, ϕ induces a function of the form Σ∗→Δ(Σ∗). So why am I making such a big deal about the autoregressive type signature being different than a general text-to-text function?

The point of this post is show that there are many distributions of responses that can easily be outputted a general text-to-text algorithm, but can't be performed by an autoregressive model in polynomial time (given some standard cryptographic assumptions like "factoring semiprimes is hard on average").

A Note on Architecture

Modern language models are almost always transformers, but I won't restrict my analysis to a particular choice of architecture for this post. The results here hold for any polytime algorithm with the autoregressive type signature mentioned above. In particular, I'll allow the algorithm to be randomized and perform looping, as long as it terminate...
A book review for “Animal Weapons” and cross-applying the lessons to x-risk
by Habeeb Abdulfatah — 2023-05-30T00:58:48.781Z
Link: https://www.super-linear.org/prize?recordId=rec4NXuZnV89qFkln

"Animal Weapons" by Douglas Emlen is a fascinating exploration of the evolution of animal weapons and how they compare to human weapons. Emlen draws parallels between the historical processes surrounding the evolution of animal and human weapons, including the environments in which they function, the forces of selection shaping their performance, and the ways in which they change through time.

One of the key takeaways from the book is the idea that the circumstances conducive to extreme weapons - the ingredients triggering an arms race - are essentially the same for animals and humans. This is an important lesson to apply to existential risks, as it suggests that the conditions that lead to the development of dangerous technologies are not unique to humans.

Emlen also explores the differences between cultural and biological evolution, highlighting the fact that cultural information can be transmitted more widely and rapidly than DNA. This has important implications for the development of existential risks, as it suggests that dangerous technologies can spread quickly and easily across borders and between different groups.

Overall, "Animal Weapons" is a thought-provoking and informative book that offers valuable insights into the evolution of weapons and the lessons we can learn from the natural world. It is a must-read for anyone interested in the intersection of biology and technology, and for those concerned about the risks posed by emerging technologies.

Please find the bounty link associated with this book review below:

https://www.super-linear.org/prize?recordId=rec4NXuZnV89qFkln
Without a trajectory change, the development of AGI is likely to go badly
by Max H — 2023-05-29T23:42:16.511Z

This is a draft of my entry to the Open Philanthropy AI worldviews contest. It's also a good summary of my own current worldview, though it is not necessarily intended as an all-encompassing argument for AI x-risk for a wide audience.

All feedback is welcome, but I'm particularly interested in feedback relevant to the contest, posted in time for me to incorporate it into my final submission. The contest deadline is May 31 (I realize this is cutting things a bit close). If you provide useful feedback and I win a prize, I will share a piece of it, based on my judgement of the value of the feedback, up to $1000.
Disclaimer: I may make substantial revisions to this post prior to submitting it; if you're reading this in the future, be please aware that early comments may have been made on an earlier version. Update: I have edited this post slightly from its original version, and have now submitted it as a contest entry. I will likely not make further major revisions.

Introduction

I started contributing to LessWrong actively in February 2023, with a loose focus on articulating my own worldview about AGI, and explaining AI and alignment concepts that I think have been somewhat overlooked or misunderstood.[1]

Below, I attempt to tie some of my past writing together into an overarching argument for why I believe that the development of AGI is likely (p >90%) to be catastrophic for humanity.

My view is conditional on the absence of a sufficiently drastic, correctly targeted, and internationally coordinated intervention focused on controlling and slowing the development of AI capabilities. Though it is not the focus of this essay, I believe the kind of intervention required is currently far outside the Overton window, and I predict that this is unlikely to change before AGI catastrophe. In worlds where AI capabilities develop gradually enough, and public relations efforts go well enough, that the Overton window does shift sufficiently quickly, there is still a significant chance, in my view, that the interventions which are actually tried are ineffective or even counterproductive.[2]

This essay will focus on Question 2 from the worldviews contest:

Question 1: What is the probability that AGI is developed by January 1, 2043?

Question 2: Conditional on AGI being developed by 2070, what is the probability that humanity will suffer an

existential catastrophe

due to loss of contr

...
Winners-take-how-much?
by YonatanK — 2023-05-29T21:56:45.505Z

The treatments I have seen to date of the "winner-takes-all'' dynamic in AI development focus on the risk-taking engendered by this dynamic: the more AI researchers have to gain from being early movers, the stronger the incentive to cut corners in safety measures. Concern about a result in which a small number of winners has successfully emerged, without, that is, blowing everything up in the effort, seems to be muted. "Winner-takes-all," in other words, is intended only as a term of art from conventional political economy, where "all" refers to the spoils of an otherwise limited contest between willing contestants. The runners-up may lose their investments, but not their lives.

To put it another way, we could describe the prevailing model as having three contests: 1) between all humans and a hypothetical uncontrolled AI; 2) between firms racing to establish monopolistic advantage in the production and sale of controlled AI; 3) between humans in general and these firms over the unprecedented risks imposed by the latter on the former, intermixed with a conventional struggle between the former as consumers and the latter as would-be monopoly producers. Of the three contests, the risk of unconstrained rivalrous behavior -- that is, a fight to the death -- is only considered to apply to the one between the hypothetical uncontrolled AI and all humans, collectively.

What seems to me to be missing from this model is a rivalrous intra-species fight to the death. This should be surprising, given that we posit a jealous AI, built and trained by humans, acquiring and pursuing goals that put it in opposition to the existence of 100% of humans. We should also fear humans with AI acquiring and pursuing goals that put them in opposition to the existence of the other humans.

This could be treated as a semantic problem, if we define any genocidal application of AI to be a form of "misalignment," whether the AI is manipulating humans to achieve murderous ends or vice versa. My test for the usefulness of a distinction between these two cases is whether or not we can imagine a significant subset of AI development firms willingly pursuing the genocidal path, rather than treating all such paths as equally aberrant. This test is, after all, where the rubber meets the road. Obviously, all firms wish to avoid a paperclip maximizer (that is the whole point of the paperclip maximizer exa...
Reply to a fertility doctor concerning polygenic embryo screening
by GeneSmith — 2023-05-29T21:50:20.854Z

New LessWrong user grll_nrg, a fertility doctor, left a comment on my post about how to have polygenically screened children that brought up many of the common objections raised to polygenic embryo screening. I've heard these concerns brought up many times at conferences and in talks by professionals in the fertility industry. I thought other people might be interested in the discussion, so I decide to make a stand-alone post.

Here's grll_nrg's original comment:

Great post. Thank you. Fertility doctor here and a supporter of ART (assisted reproductive technologies) in general. A few thoughts (although you touched on a few of these below, worth emphasizing in my opinion):

PGT-P has not been validated yet, which may take decades to do, if ever.
The science in terms of GWAS isn't quite there yet IMHO - we don't know all the genes that are important for most traits and we may be inadvertently selecting against some desirable traits, for example.
Comparing clinic success rates using CDC data is imperfect because of different patient characteristics, patient selection, and reporting bias.
IVF pregnancies carry a significantly higher complication rate (hypertensive disorders, preterm birth, placental abnormalities, etc.) compared to spontaneous pregnancies - unclear if this is due to IVF or underlying infertility diagnosis.
The risk-benefit calculus of PGT-P is going to be different for a couple who already needs to do IVF anyway to have a baby (low additional risk/cost) compared to a couple doing IVF just so that they can do PGT-P (higher additional risk/cost).
IVF is notoriously inefficient at present. Depending on female partner age, each cycle may yield only very few embryos making the benefit and utility of PGT-P limited. It may not be practical, safe, or financially feasible to do multiple cycles of IVF to increase the cohort of transferable embryos.
IVF is expensive and often not covered by insurance which creates access disparities. PGT-P would exacerbate these disparities in access. This is not unique to IVF I realize.
Slippery-slope eugenics and discrimination are real ethical concerns that would need to be mitigated.
In-vitro gametogenesis (IVG) would be a game-changer. The utility of PGT-P would be greatly enhanced if suddenly you had thousands of eggs and hundreds of embryos to select from.

Thanks for the reply. I'm ...
Sentience matters
by So8res — 2023-05-29T21:25:30.638Z

Short version: Sentient lives matter; AIs can be people and people shouldn't be owned (and also the goal of alignment is not to browbeat AIs into doing stuff we like that they'd rather not do; it's to build them de-novo to care about valuable stuff).

Context: Writing up obvious points that I find myself repeating.

Note: in this post I use "sentience" to mean some sort of sense-in-which-there's-somebody-home, a thing that humans have and that cartoon depictions of humans lack, despite how the cartoons make similar facial expressions. Some commenters have noted that they would prefer to call this "consciousness" or "sapience"; I don't particularly care about the distinctions or the word we use; the point of this post is to state the obvious point that there is some property there that we care about, and that we care about it independently of whether it's implemented in brains or in silico, etc.

Stating the obvious:

All sentient lives matter.

Yes, including animals, insofar as they're sentient (which is possible in at least some cases).

Yes, including AIs, insofar as they're sentient (which is possible in at least some cases).

Yes, even including sufficiently-detailed models of sentient creatures (as I suspect could occur frequently inside future AIs). (People often forget this one.)

Not having a precise definition for "sentience" in this sense, and not knowing exactly what it is, nor exactly how to program it, doesn't undermine the fact that it matters.

If we make sentient AIs, we should consider them people in their own right, and shouldn't treat them as ownable slaves.

Old-school sci-fi was basically morally correct on this point, as far as I can tell.

Separately but relatedly:

The goal of alignment research is not to grow some sentient AIs, and then browbeat or constrain them into doing things we want them to do even as they'd rather be doing something else.

The point of alignment research (at least according to my ideals) is that when you make a mind de novo, then what it ultimately cares about is something of a free parameter, which we should set to "good stuff".

My strong guess is that AIs won't by default care about other sentient minds, and fun broadly construed, and flourishing civilizations, and love, and that it also won't care about any other stuff that's deeply-alien-and-weird-but-wonderful.

But we could build it to care about tha

...