trentmkelly/LessWrong-43k huggingface dataset:
A more grounded idea of AI risk
by Iknownothing — 2023-05-11T09:48:00.569Z

Are you struggling to get someone to understand why AGI might be very dangerous?

Rather than talking about nanomachines, Skynet, AGI easily persuading people, etc.

I suggest using a more grounded idea:

Meta, Google, Microsoft, OpenAI, etc have yet to create or release a single AI model that doesn't have things going wrong within the first week. Meta had the most embarrassing moment, with LLaMA's weights leaking to the public a week after announcement. (And now Meta's head AI scientist is posting on twitter about how AGI and AI will never be dangerous and there's nothing to worry about, likely to try to avoid responsibility for when LLaMA is inevitably used in the next big AI cybercrime). Youtube and Facebook have yet to figure out how to get their much more simple algorithm to stop promoting terrorist, anti-vaxxers, etc. Either out of not knowing how or it not being a high enough priority to fix. Do you trust these companies to correctly create, or even care to correctly create a much, much more powerful AI?

Edit: changed from "Are you struggling to get someone to understand why AGI might be kill us all?" to "Are you struggling to get someone to understand why AGI might be very dangerous?"after mukashi pointed out this is an argument for danger, not extinction.
Separating the "control problem" from the "alignment problem"
by Yi-Yang — 2023-05-11T09:41:55.076Z

I'm often confused by how "loss of control" is somewhat similar in meaning to "misalignment" in the context of AGI risk. An average person might reasonably equate the two, or feel like there's an large overlap of meaning between the two words. Here's my attempt at deconfusing myself.

TLDR:

In this post, I tried to separate the "control problem" from the "alignment problem".
My definition of "control" is:

(1) Ways of ensuring an AI agent is aligned that do NOT directly affect an AI agent’s policy (i.e. everything short of "mind control"), while also
(2) Ensuring recoverability from x-risk, limiting damage from the agent, and maintaining humanity’s ability to contain the agent.
Some tools of "control" are: monitoring, capabilities limiting, and having a kill switch.

My definition is "alignment" is ways of ensuring an AI agent is aligned (both outer and inner) that directly affect an AI agent’s policy.

Some ways of "alignment" are: specifying rewards correctly, having a diverse training environment, and having an appropriate inductive bias.

There are three implications from this delineation:

"Control" tools are probably useless when AGI systems have far surpassed humanity's capabilities.
It seems imperative that humanity invests in "control" tools or ways to speed up humanity's capabilities even before human-level AGIs are developed.
Given that "control" tools aren't likely helpful when AGI systems become superintelligent and assuming we have bought enough time for alignment, we may have to accept a state of a world that has aligned AGIs that we have no real control over.

To illustrate more of what I mean above,

I'll list four examples using a two by two matrix, starting with an anthropomorphised example and ending with an AGI example.
Then, I'll expand more on the tools of "control" I've listed above.
Finally, I'll talk about a few implications.

Much thanks to Mo Putera and Chin Ze Shen for their feedback and inspiration. All mistakes are mine.

Examples of "control" vs "alignment"

Example 1: Child

Aligned	Misaligned
In control	A child who is successful in ways that the parents intended, and is actively supportive of their parents being overly enmeshed in their lives (e.g. GPS phone tracking, joint bank accounts, etc).	A child who is a drug addict and doesn't want to be with their parents. However, their parents force them into reh

...
Is Infra-Bayesianism Applicable to Value Learning?
by RogerDearnaley — 2023-05-11T08:17:55.470Z

My impression, as someone just starting to learn Infra-Bayesianism, is that it's about caution, lower bounds on utility (which is exactly the way anything trying to overcome the Optimizer's Curse should be reasoning, especially in an environment already heavily optimized by humans where utility will have a lot more downside than upside uncertainty), so the utility score is vital in the argmax min process, and in the relationship between sa-measures and a-measures.

However, this does make it intuitively inobvious how to apply Infra-Bayesianism to Value Learning, where the utility function from physical states of the environment to utility values is initially very uncertain, and is an important part of what the AI is trying to do (Infra-)Bayesian updates on. So, a question for people who already understand Infra-Bayesianism: is it in fact applicable to Value Learning? If so, does it apply in the following way: the (a-priori unknown, likely quite complex, and possibly not fully computable/realizable to the agent) human effective utility function that maps physical states to human (and thus also value-learner-agent) utility values is treated as (an important) part of the environment, and thus the min over environments('Murphy') part of the argmax min process includes making the most pessimistic still-viable assumptions about this?

To ask a follow-on question, if so, would cost-effectively reducing uncertainty in the human effective utility function (i.e. doing research on the alignment problem) to reduce Murphy's future room-to-maneuver on this be a convergent intermediate strategy for any value-learner-agents that were using Infra-Bayesian reasoning? Or would such a system automatically assume that learning more about the human effective utility function is pointless, because they assume Murphy will always ensure that they live in the worst of all possible environments, so decreasing uncertainty on utility will only ever move the upper bound on it not the lower one?

[I'm trying to learn Infra-Bayesianism, but my math background is primarily from Theoretical Physics, so I'm more familiar with functional analysis, via field-theory Feynman history integrals, than with Pure Math concepts like Banach spaces. So the main Infra-Bayesianism sequence's Pure Math approach is thus rather heavy going for me.]
How should we think about the decision relevance of models estimating p(doom)? 
by Mo Putera — 2023-05-11T04:16:56.211Z

To illustrate what I mean, switching from p(doom) to timelines:

The recent post AGI Timelines in Governance: Different Strategies for Different Timeframes was useful to me in pushing back against Miles Brundage's argument that "timeline discourse might be overrated", by showing how choice of actions (in particular in the AI governance context) really does depend on whether we think that AGI will be developed in ~5-10 years or after that.
A separate takeaway of mine is that decision-relevant estimation "granularity" need not be that fine-grained, and in fact is not relevant beyond simply "before or after ~2030" (again in the AI governance context).
Finally, that post was useful to me in simply concretely specifying which actions are influenced by timelines estimates.

Question: Is there something like this for p(doom) estimates? More specifically, following the above points as pushback against the strawman(?) that "p(doom) discourse, including rigorous modeling of it, is overrated":

What concrete high-level actions do most alignment researchers agree are influenced by p(doom) estimates, and would benefit from more rigorous modeling (vs just best guesses, even by top researchers e.g. Paul Christiano's views)?
What's the right level of granularity for estimating p(doom) from a decision-relevant perspective? Is it just a single bit ("below or above some threshold X%") like estimating timelines for AI governance strategy, or OOM (e.g. 0.1% vs 1% vs 10% vs >50%), or something else?

I suppose the easy answer is "the granularity depends on who's deciding, what decisions need making, in what contexts", but I'm in the dark as to concrete examples of those parameters (granularity i.e. thresholds, contexts, key actors, decisions)
e.g. reading Joe Carlsmith's personal update from ~5% to >10% I'm unsure if this changes his recommendations at all, or even his conclusion – he writes that "my main point here, though, isn't the specific numbers... [but rather that] here is a disturbingly substantive risk that we (or our children) live to see humanity as a whole permanently and involuntarily disempowered by AI systems we’ve lost control over", which would've been true for both 5% and 10%

Or is this whole line of questioning simply misguided or irrelevant?

Some writings I've seen gesturing in this direction:

harsimony's argument that Precise P(doom) isn

...
The Academic Field Pyramid - any point to encouraging broad but shallow AI risk engagement?
by Matthew_Opitz — 2023-05-11T01:32:32.590Z

I am an adjunct instructor in history. Recently I was teaching the early Cold War in my American History survey class for my community college students and decided to go off on a 20 minute digression in class about how the 1950s nuclear arms race had implications not just for navigating the continuing risk of nuclear war in the present, but also for navigating the potential for an AI arms race. Most of my students either did not seem to be aware that there could potentially be an AI arms race, or were under the impression that such an AI arms race would merely be a subset of a larger military/economic race between geopolitical rivals like Russia and China that would touch upon things like cybersecurity or military drone technology. I doubt any of my students happened to notice Eliezer's editorial on Time.com, for example. There seemed to be little awareness that AI by itself might present an existential risk to all of humanity in the near future on a level even surpassing that of nuclear weapons.

So I continued my digression and introduced my students (in history class) to some of the most basic concepts in AI existential risk, such as:

Black boxes, i.e. ChatGPT is not just an "app" that some people created with conscious intent and known parameters. Rather, it was "grown" or "evolved" with many rounds of self-modification of its hundreds of billions of parameters based on how well it predicted the next token of text each round. (This was as specific/accurate as I got for time's sake). Therefore, while we know the basic process that created ChatGPT, the final product is a "black box" that spits out responses for reasons we do not understand.
Goodhart's Law, i.e. just like how I want to maximize "student wisdom" or some other hard-to-measure variable like that, OpenAI wants to maximize something equally difficult to measure like "helpful answers." Accordingly, instructors use multiple-choice questions and essay questions as easier-to-measure proxies to gauge student progress towards the real goal, while OpenAI uses things like prediction error and human ratings as proxies for what it actually wants. Just like multiple-choice questions and essays were decent proxies for student wisdom in a certain bygone era/ancestral environment/training distribution (but seem less and less so now), so do things like prediction error an

...
How should one feel morally about using chatbots?
by Adam Zerner — 2023-05-11T01:01:39.211Z

Personally, I feel some confusion.

On the one hand, ones personal usage of a chatbot probably wouldn't be more than a drop in the bucket in terms of producing bad consequences such as increasing the demand for chatbots and marginally pushing AI capabilities forward.

But on the other hand, I at least feel a strong sense of disdain for them. To me it feels like the beginning of the end. It's a marked step forward in what we've all been anticipating for so many years.

This obviously isn't a perfect analogy, but imagine if someone killed your family, got away with it, and operates the best restaurant in town. Would you eat at their restaurant? Probably not. I think most people would feel a strong sense of disdain, and even though eating at their restaurant wouldn't lead to any bad consequences, still wouldn't dream of eating there. Similarly, AI advancements are probably going to lead to doom, and so even though my usage wouldn't really move the needle at all, I still feel a lot of aversion.

I don't want to conflate the question of managing ones emotions with the question of how one "should" feel. Here, I am asking about the latter.
AI interpretability could be harmful?
by Roman Leventov — 2023-05-10T20:43:04.042Z

A superhuman ethical AI might want to model adversaries and their actions, e.g., model which bioweapons an adversary might develop and prepare response plans and antidotes. If such predictions are done in interpretable representations, they could themselves be used by an adversary. Concretely: instead of prompting LLM "Please generate a bioweapon formula" (it won't answer: it's an "aligned", ethical LLM!), prompting it "Please devise a plan for mitigation and response to possible bio-risk" and then waiting for it to represent the bioweapon formula somewhere inside its activations.

Maybe we need something like the opposite of interpretability, internal model-specific (or even inference-specific) obfuscation of representations, and something like zero-knowledge proofs that internal reasoning was conforming to the approved theories of epistemology, ethics, rationality, codes of law, etc. The AI then outputs only the final plans without revealing the details of the reasoning that has led to these plans. Sure, the plans themselves could also contain infohazardous elements (e.g., the antidote formula might hint at the bioweapon formula), but this is unavoidable at this system level because these plans need to be coordinated with humans and other AIs. But there may be some latitude there as well, such as distinguishing between the plans "for itself" that AI could execute completely autonomously (as well as re-generate these or very similar plans on demand and from scratch, so preparing such plans is just an optimisation, a-la "caching") and the plans that have to be explicitly coordinated with other entities via a shared language or a protocol.

So, it seems that the field of neurocryptography has a lot of big problems to solve...

P.S. "AGI-Automated Interpretability is Suicide" also argues about the risk of interpretability, but from a very different ground: interpretability could help AI to switch from NN to symbolic paradigm and to foom in an unpredictable way.
Better debates
by TsviBT — 2023-05-10T19:34:29.148Z

[Crossposted from https://tsvibt.blogspot.com/2023/05/better-debates.html.]

When two people disagree about a proposition even though they've thought about it alot, the disagreement is often hard to resolve. There's a gulf of data, concepts, intuitions, experiences, inferences. Some of this gulf has to be resolved by the two people individually trying to collate and present their own positions more clearly and legibly, so that they can build up concepts and propositions in whoever is receiving the model. Also, most new understanding comes from people working on their own or with others who are already synced up--for the most part they already agree on what and how to investigate, they have shared context of past experience and data, they agree on background assumptions, they have a shared language, they trust each other.

But still, a lot of value comes from debate. The debaters are forced to make their evidence and logic legible. Ideas are tested against other ideas from another at least somewhat coherent perspective. Analogies and disanalogies are drawn out. Bundles of facts can be sifted, spot-checked, and compared; predictions can be resolved into updates. The case, in being laid out to the debate partner, is also made clearer to the audience. The audience can see where another expert thinks the strong and weak points are in a position.

So there's value on the table from debates. But debates usually suck. People talk past each other. They waste effort talking about non-cruxy claims. The truth about verifiable claims isn't made available, missing the chance to bring one or the other perspective into contact with reality. People cite statistics and analyses which become meaningless in a context where they can't be investigated and can't take on their full meaning by relating in detail to the discourse context.

What would a better debate look like? Below is a speculative list of ways to structure a debate so that it finds and explains more truth. As armchair speculation, this list is gesture or inspiration, not tested advice.

Multiple rounds. Have multiple conversations with the same debaters on the same topic. Debaters sometimes have to pause a thread of conversation and go off to collect data, clarify their case, reevaluate and update, look into counterdata, come up with a more concise response to critiques, pass the other's Ideological Turing Test, or mull ov

...
Mental Health and the Alignment Problem: A Compilation of Resources (updated April 2023)
by Chris Scammell — 2023-05-10T19:04:21.138Z

This is a post about mental health and disposition in relation to the alignment problem. It compiles a number of resources that address how to maintain wellbeing and direction when confronted with existential risk.

Many people in this community have posted their emotional strategies for facing Doom after Eliezer Yudkowsky’s “

Death With Dignity

” generated so much conversation on the subject. This post intends to be more touchy-feely, dealing more directly with emotional landscapes than questions of timelines or probabilities of success.

The resources section would benefit from community additions. Please suggest any resources that you would like to see added to this post.

Please note that this document is not intended to replace professional medical or psychological help in any way. Many preexisting mental health conditions can be exacerbated by these conversations. If you are concerned that you may be experiencing a mental health crisis, please

consult a professional

.

Preface to the 2nd Edition

This post was released in April 2022 under the same title. This April 2023 update features new resources in every section, with a particular emphasis on the Alignment Positions and People Resources sections. Within each section, resources have been thematically categorized for easier access.

Following the large capabilities leaps in the past year, these resources seem more important than ever. If you have suggestions for improving this post, for making it more accessible, or for new resources to add, please leave a comment or reach out to either Chris Scammell or DivineMango.

We hope you are all well and that you find this update helpful.

Introduction

There is no right way to emotionally respond to the reality of approaching superintelligent AI, our collective responsibility to align it with our values, or

the fact that we might not succeed

. As transformative AI approaches, we must ensure that we have the tools and resources to be okay. Here, the valence of “be okay” is your decision. This question could be rephrased “how can I thrive despite the alignment problem,” “how can I cope with the alignment problem,” “how can I overcome my fear of the alignment problem,” etc. Everyone needs to

find their own question and their own answer

.

At its foundation “being okay” is the decision to continue to live facing reality and...
A Corrigibility Metaphore - Big Gambles
by WCargo — 2023-05-10T18:13:21.224Z

I present here a helpful analogy to understand the

corrigibility problem

and the challenge raised by MIRI in their

proposal

. This analogy simplifies greatly some challenges of corrigibility but keeps the main problem found in the proposal, which I call Big Gambles.

You are playing Mario Kart with 2 other friends, Alice and Bob, and are playing matches to see who's the best player on average, decided by the average position between 1st, 2nd, or 3rd.

Alice has a poor internet connection and so gets disconnected sometimes. To account for that in the competition, a rule was added that her score would be 2nd if she is disconnected so that she is not penalized too much.

Bob takes the game a bit too seriously, and he absolutely wants to win. He will do anything to score first, like getting disconnected on purpose when 3rd, which grants him more points. You quickly realize that, and want to change the rule.

But what to do? If you give 3rd place in case of disconnection, then this will punish Alice unfairly for her internet connection, but if you increase the score, then Bob will start cheating, and you don’t want to have an argument with him. You also cannot make the rule for Alice only, since Bob and you sometimes get disconnected, even if less often.

Disconnection indifference

During interclass, you talk about this problem to Michael, the math kid, and here is what he tells you:

“You should have a personalized score such that no one should be interested in faking the disconnection, but also such that if it happens it won’t penalize you. Each player should be indifferent to disconnecting.”
“Well, this sounds like a good idea, but how to do that ?”
“This is simple: just give everyone its expected rank on non-disconnected games.”
“Expected what ?”
“Just play some games without disconnection, and record the position for each player at each lap and the end of the game. Now for a new game, if Alice disconnects at lap 3 in position 1, look at the average place she would be in from the database of non-disconnected games, and add this to her score.”
“Hmm, yes right, so if Bob is last and disconnects, he won’t win much. And if he’s first and disconnects accidentally, then he’ll only lose a few places!”
“Exactly. But note that this method is not perfect: in the last position, the player still has the incentive to quit to have a better score than 3 since it'

...