trentmkelly/LessWrong-43k huggingface dataset:
A couple of questions about Conjecture's Cognitive Emulation proposal
by Igor Ivanov — 2023-04-11T14:05:58.503Z

Introduction

Recently, Conjecture proposed a conception of a modular AI architecture, with each module being interpretable, having limited intellect, and functionally emulating the human mind. They called this concept Cognitive Emulation or CoEm.
Conjecture's post was concise and did not contain technical details or plans, so people in the comments asked a lot of questions about this proposal. Later, a group of authors made a post with a set of questions about CoEm.

I have a pair of questions that to my knowledge were not asked about this proposal.

They might be naive and have been discussed elsewhere, but since the questions are not entirely obvious, and no one else asked them within the context of CoEm, I believe that this post has the potential to start valuable discussions.

Questions

Is it even possible to create a perfect CoEm?

A quote from the original post:

We want systems that are as safe as humans, for the same reasons that humans have (or don’t have) those safety properties. Any scheme that involves building systems that involves humans should allow you to swap those humans for CoEms without breaking or drastically altering their behavior.

As far as I understand, Conjecture plans to create AI agents that function so similarly to humans that they are interchangeable with humans in social systems.
But what if a human need hours to write a text while AI needs seconds? Is it still human-capability level? What if AI can rapidly learn a large corpus of scientific knowledge? What if AI can instantly answer any email, not like busy people who might answer in a week?
I don't know how CoEm will be implemented, but it is highly likely that it will not require hours to write texts, and it will not wait for a week to answer an email.
In the past, such bumps in human capabilities resulted in drastic changes in the world. For example, it is believed that the adoption of radio which allowed propagandists to speak not to the tens or hundreds, but millions of people, was one of the causes of the rise of totalitarian regimes in the 20th century. In the case of CoEm, even if it will be as perfect a human emulator as possible, such changes in capabilities are inevitable and will cause consequences that are very hard to predict.

Do you really believe that CoEms will not fudamentally disrupt society?

A quote from the original post:

We have a lot of exp

...
Childhood Roundup #2
by Zvi — 2023-04-11T13:50:01.094Z

Recently: The Kids are Not Okay, Childhood Roundup #1, Fertility Roundup #1

It seems like a good time to go over the interesting relevant stuff I’ve found since the last time I did one of these.

Executive Summary and Table of Contents:

Why Are the Kids are Not Okay. Additional fleshing out of the ‘school as giant hypocritical pressure cooker’ hypothesis, and general confirmation of the cell phone hypothesis.

Child Care is Increasingly Regulated and Increasingly Expensive. Child care is required since you can’t leave your kid alone, and it is increasingly expensive. The idea that only 7% of your income should go to child care doesn’t make any mathematical sense even if you don’t require ground-floor rooms staffed by college graduates. If you do, what were you even hoping for? The solution seems to be that everyone pays way more than that, except sometimes the state covers it if you’re poor.

Let Your Children Play. Hand in hand with expensive child care is arresting any parent who lets their children out of their sight – or at least the risk that this will happen. There are no standards you can follow to be safe from this, at any age, for any activity, even at home. The mom arrested for letting a 14-year-old babysit is finally free, but her life is not going to magically be all right.

School Daze. Continued illustrations of what school is, and how we choose to run them. Examples of some very broken behaviors, even by our standards. A variety of common learning techniques don’t work.

A Case Against College. By trying to argue that college is worthwhile, Michael Story makes an excellent case that it is not. Those telling you to go to college increasingly embrace the signaling model of education. Claims that it ‘teaches you how to be productive’ better than alternatives don’t seem credible. The purported solution to disliking higher education is… more higher education.

Potential Alternative Methods. Is it possible to get much better school outcomes? To worry about your kids a lot less? To simply not offer enrichment?

Good Advice. A bunch of things parents worry about too much, you can stop. A lot of very central things make the list.

No More Fight Club. Some good news, fights are much less common now.

Why Are the Kids so Not Okay?

What additional thoughts have come in directly on the questions raised in The Kids are Not Okay?

Most...
Measuring artificial intelligence on human benchmarks is naive
by Anomalous — 2023-04-11T11:34:19.344Z
Link: https://forum.effectivealtruism.org/posts/ZjQ2fXpATBMvnBzzj/measuring-artificial-intelligence-on-human-benchmarks-is

Central claim: Measured objectively, GPT-4 is arguably way past human intelligence already, perhaps even after taking generality into account.

Central implication: If the reason we're worried AGI will wipe us out is tied to an objective notion of intelligence--such as the idea that it starts to reflect on its values or learn planning just as it crosses a threshold for cognitive power around human level--we should already update on the fact that we're still alive.

I don't yet have a principled way of measuring "generality",[1] so my intuition just tends to imagine it as "competence at a wide range of tasks in the mammal domain." This strikes me as comparable to the anthropomorphic notion of intelligence people had back when they thought birds were dumb.

When GPT-2 was introduced, it had already achieved superhuman performance on next-token prediction. We could only hope to out-predict it on a limited set of tokens extremely prefiltered for precisely what we care the most about. For instance, when a human reads a sentence like...

"It was a rainy day in Nairobi, the capital of _"

...it's obvious to us (for cultural reasons!) that the next word is an exceptionally salient piece of knowledge. So those are the things we base our AI benchmarks on. However, GPT cares equally about predicting 'capital' after 'the', and 'rainy' after 'It was a'. Its loss function does not discriminate.[2]

Consider in combination that a) GPT-4 has a non-discriminating loss function, and b) it rivals us even at the subset of tasks we optimise the hardest. What does this imply?

It's akin to a science fiction author whose only objective is to write better stories yet ends up rivalling top scientists in every field as an instrumental side quest.

Make no mistake, next-token prediction is an immensely rich domain, and the sub-problems could be more complex than we know. Human-centric benchmarks vastly underestimate both the objective intelligence and generality of GPTs, unless I'm just confused.

^

Incidentally, please share if you know any good definitions of generality e.g. from information theory or something.

^

At least during pretraining afaik.
Killing Socrates
by Duncan Sabien (Inactive) — 2023-04-11T10:28:37.096Z

Or, On The Willful Destruction Of Gardens Of Collaborative Inquiry

One of the more interesting dynamics of the past eight-or-so years has been watching a bunch of the people who [taught me my values] and [served as my early role models] and [were presented to me as paragons of cultural virtue] going off the deep end.

Those people believed a bunch of stuff, and they injected a bunch of that stuff into me, in the early days of my life when I absorbed it uncritically, and as they've turned out to be wrong and misguided and confused in two or three dozen ways, I've found myself wondering what else they were wrong about.

One of the things that I absorbed via osmosis and never questioned (until recently) was the Hero Myth of Socrates, who boldly stood up against the tyrannical, dogmatic power structure and was unjustly murdered for it. I've spent most of my life knowing that Socrates obviously got a raw deal, just like I spent most of my life knowing that

It now seems quite plausible to me that Socrates was, in fact, correctly responded-to by the Athenians of his time, and that the mythologized version of his story I grew up with belongs in the same category as Washington's cherry tree or Pocahontas's enthusiastic embrace of the white settlers of Virginia.

The following borrows generously from, and is essentially an embellishment of, this comment by @Vaniver.

Imagine that you are an ancient Athenian, responsible for some important institution, and that you have a strong belief that the overall survival of your society is contingent on a reliable, common-knowledge buy-in of Athenian institutions generally, i.e. that your society cannot function unless its members believe that it does function.

This would not be a ridiculous belief! We have seen, in the modern era, how quickly things go south when faith in a bank (or in the financial system as a whole) evaporates. We know what happens when people stop believing that the police or the courts are on their side. Regimes (or entire nations) fall when their constituents stop propping up the myth of those regimes. Much of civilization is shared participation in self-fulfilling prophecies like "this little scrap of green paper holds value."

And if you buy

"My society's survival depends upon people's faith in its institutions."

...then it's only a small step from there to something like:

"My society's survival depends upon

...
Cyberspace Administration of China: Draft of "Regulation for Generative Artificial Intelligence Services" is open for comments
by sanxiyn — 2023-04-11T09:32:13.396Z
Link: https://archive.is/cQr0R

The announcement is of obvious importance to global AI governance. As I understand, you can email your comments to wajscy@cac.gov.cn, and I recommend everyone to do so.
Is "Strong Coherence" Anti-Natural?
by DragonGod — 2023-04-11T06:22:22.525Z

Related:

Contra "Strong Coherence"
why assume AGIs will optimize for fixed goals
Why The Focus on Expected Utility Maximisers?

Background and Core Concepts

I operationalised "strong coherence" as:

Informally: a system has immutable terminal goals.

Semi-formally: a system's decision making is well described as an approximation of argmax over actions (or higher level mappings thereof) to maximise the expected value of a single fixed utility function over states.

And contended that humans, animals (and learning based agents more generally?) seem to instead have values ("contextual influences on decision making").

The shard theory account of value formation in learning based agents is something like:

Value shards are learned computational/cognitive heuristics causally downstream of similar historical reinforcement events
Value shards activate more strongly in contexts similar to those where they were historically reinforced

And I think this hypothesis of how values form in intelligent systems could be generalised out of a RL context to arbitrary constructive optimisation processes[1]. The generalisation may be something like:

Decision making in intelligent systems is best described as "executing computations/cognition that historically correlated with higher performance on the objective functions a system was selected for performance on"[2].

This seems to be an importantly different type of decision making from expected utility maximisation[3]. For succinctness, I'd refer to systems of the above type as "systems with malleable values".

The Argument

In my earlier post I speculated that "strong coherence is anti-natural". To operationalise that speculation:

Premise 1: The generalised account of value formation is broadly accurate

At least intelligent systems in the real world form "contextually activated cognitive heuristics that influence decision making" as opposed to "immutable terminal goals"
Humans can program algorithms with immutable terminal goals in simplified virtual environments, but we don't actually know how to construct sophisticated intelligent systems via design; we can only construct them as the product of search like optimisation processes[4]

And intelligent systems constructed by search like optimisation processes form malleable values instead of immutable terminal goals

I.e. real world intelligent systems form malleable values

Premise 2: Systems wi

...
Four mindset disagreements behind existential risk disagreements in ML
by Rob Bensinger — 2023-04-11T04:53:48.427Z

I've previously written that good ideas and conversations about AGI seem to have propagated through ML weirdly slowly.

A different weird phenomenon I observe is that the field's relative inaction about AGI seems less based on a confident set of beliefs about AGI definitely going well (or definitely being far off), and more based on an implicit sense like "the default is everything going well, and we don't need to change anything until there's overwhelming evidence to the contrary".

Some people do have confident beliefs that imply "things will go well"; I disagree there, but I expect some amount of disagreement like that.

But that doesn't seem to be the crux for most people in ML.

In a sane world, it doesn't seem like "well, maybe AI will get stuck at human-ish levels for decades" or "well, maybe superintelligence couldn't invent any wild new tech" ought to be cruxes for "Should we pause AI development?" or "Is alignment research the world's top priority?"

Note that I'm not arguing "an AGI-mediated extinction event is such a big deal that we should make it a top priority even if it's very unlikely". There are enough other powerful technologies on the horizon, and enough other risks for civilizational collapse or value lock-in, that I don't in fact think AGI x-risk should get major attention if it's very unlikely.

But the most common view within ML seems to be less "it's super unlikely for reasons X Y Z", and more of an "I haven't thought about it much" and/or "I see some reasons to be very worried, but also some reasons things might be fine, so I end up with medium-ish levels of worry".

In a mid-2022 survey, 48% of researchers who had recently published in NeurIPS or ICML gave double-digit probabilities to advanced AI's long-term effect being “extremely bad (e.g., human extinction)”. A similar number gave double-digit probabilities to "human inability to control future advanced AI systems causing human extinction or similarly permanent and severe disempowerment of the human species".

In an early 2021 survey, 91% of researchers working on "long-term AI topics" at CHAI, DeepMind, MIRI, OpenAI, Open Philanthropy, and what would become Anthropic gave double-digit probabilities to "the overall value of the future will be drastically less than it could have been, as a result of AI systems not doing/optimizing what the people deploying them wanted/intended".

The level of con...
Alignment vs capabilities
by Adam Zerner — 2023-04-11T04:35:22.144Z

In Star Wars: Episode I - The Phantom Menace, Master Jedi Qui-Gon Jinn discovers Anakin. Anakin is just a young child slave on the forgettable planet of Tatooine. However, Qui-Gon feels something noteworthy about the boy.

He feels... the Force. To confirm this feeling, he takes a sample of blood from Anakin.

Anakin: What are you doing?

Qui-Gon: Checking your blood for infections. Run along. You have a big day tomorrow. Sleep well Anny.

Qui-Gon: Obi-Wan.

Obi-Wan: Yes master.

Qui-Gon: I need an analysis of this blood sample I'm sending you.

Obi -Wan: Wait a minute.

Qui-Gon: I need a midi-chlorian count.

Obi-Wan: The reading is off the chart. Over 20,000. Even Master Yoda doesn't have a midi-chlorian count that high.

Qui-Gon: No Jedi has.

Obi-Wan: What does that mean?

Qui-Gon: [dramatic pause]

Qui-Gon: I'm not sure.

So then, Qui-Gon wants to take advantage of this. He basically has a budding superweapon in front of him and wants to utilize it in the name of Good.

More specifically, he believes that Anakin is the Chosen One.

An ancient Jedi prophecy, made by an ancient Jedi mystic and recorded on a holocron, foretold that the Chosen One would restore balance to the Force by destroying the Sith. The prophecy also predicted that the Chosen One would also become one of the strongest Force-sensitives in the galaxy, rivaling the power of the most legendary champions of the Jedi Order.

Although the Jedi had long spoken of balance in the Force, there was no consensus on what that meant for the galaxy. Some people thought that the light side of the Force and the dark side of the Force had to be equal in order to create balance, while others believed that the light could be the Force in its natural state, making the dark side the cause of imbalance. In contrast to the Jedi's view of balance, the Sith Lord Darth Sidious believed that the dark side needed to be free, and the light side suppressed, in order to achieve balance in the Force.

Given this, Qui-Gon wants to train Anakin as a Jedi so that the prophecy can be fulfilled. But when Qui-Gon presents this proposal to the Jedi Counsel, they are not eager to jump on it.

Yoda: Master Qui-Gon. More to say, have you?

Qui-Gon: With your permission my master, I have encountered a vergence in the Force.

Yoda: A vergence, you say?

Mace Windu: Located around a person?

Qui-Gon: A boy. His cells have the highest concentration of midi-chlorians I have see

...
Some Intuitions Around Short AI Timelines Based on Recent Progress
by Aaron_Scher — 2023-04-11T04:23:22.361Z

tldr: I give some informal evidence and intuitions that point toward AGI coming soon. These include thinking about how crazy the last year has been, beliefs from those in major AI labs, and progress on MMLU.

Intro

This post is intended to be a low-effort reference I can point people to when I say I think there is some evidence for short AI timelines. I might describe the various bits of evidence and intuitions presented here as “intuitions around short AI timelines based on recent progress” (though perhaps there are

better terms

). They are not a thorough model like

Ajeya’s

; insofar as somebody is using multiple models when putting together a timelines estimate, I think it would be unreasonable to place less than 20% or greater than 95% weight on extrapolation from current systems and recent progress.[1]

In the spirit of being informal, you can use whatever definition of AGI you like. I mostly use AGI to refer to something like “an AI system which can do pretty much all the human cognitive tasks as well or better than humans (~99% of tasks people in 2023 do).”

Some evidence

I (Aaron) started following AI and AI existential safety around the beginning of 2022; it’s been a little over a year. Some of that time was my understanding catching up with advances from the past couple years, but there have also been major advances.

Some major advances since I’ve been paying attention:

Chinchilla paper

popularized the scaling laws that were already known to some, there was some

DALL-E

and related stuff which was cool,

CICERO

happened which I didn’t follow but indicates we’re

probably

going to train the AIs to do all the dangerous stuff (see also

Auto-GPT

and

Chaos-GPT

, or GPT-4

getting

plugins within 2 weeks of release, as more recent updates in this saga of indignity),

ChatGPT

shows how much more usable models are with RLHF (popularizing methods that have been known for a while),

Med-PaLM

gets a passing score on the US medical licensing exam (also tons of other PaLM and Flan-PaLM results I haven’t followed but which seem impressive).

LLaMA

and

Alpaca

take powerful capabilities from compute-efficient (and over) training and hand them to the public.

GPT-4

blows the competition out of the water on many benchmarks. I probably missed a couple big things (including projects which honorabl...
FLI And Eliezer Should Reach Consensus
by JenniferRM — 2023-04-11T04:07:14.248Z

I will propose a plan for possible future, explain some reasons the plan might work, and end with a call to action.

...

This essay basically doesn't even engage with the "AI stuff itself".

I'm simply taking AI risks mostly for granted and focusing on social aspects... that is to say, a lot of this will offer simulacra level 1 arguments for why people operating on simulacra levels 2, 3, or 4, can be ignored or managed, and they will probably come around when they need to come around, or they never will and that's fine (so long as they get a good result (even though they never paid for a good result or asked for a good result)).

AI Object Level Assumption: Nearly all reasonable people think the probability of human extinction due to AI is >10% within 30 years, and people who have looked at it longer tend to have higharer probabilities and shorter timelines. Plans to make this probability go reliably lower than 1% are very very very rare, but if one existed, nearly any human who paused to think step-by-step would calculate that it was worth it. There are probable lots of cheap and easy plans that would work, but even a plan that involved killing 9% of people on Earth, or burning 9% of all the wealth on Earth in a giant fiery money pit, would probably be worth saying YES to, if the "10% risk" is real and there was only a binary choice between that plan and nothing.

If the true risks are higher than that (as many smart people think), then larger costs would make sense to pay! Since the problem can probably be solved with ~0 deaths, and for much less than a billion dollars, the discrepancy between "how good a deal it would be to solve this" and "how little money it takes to solve it" is part of the humanistic horror of the situation.

A Plan

1. Start The Moratorium Now: The FLI's six month large model training moratorium should start as fast as possible.

2. Also Find Better Plans: Stuart Russell, Yoshua Bengio, and/or several of the NGO folks in the FLI halo should sit down with Eliezer, and maybe a third party mediator that Eliezer can suggest who is good at drawing pearlian causal graphs on white boards, and they should figure out a high probability way to "cause the world to be safe" and also to "cause lots of signatures to endorse the ideas necessary to cause a safety plan".

3. Carry Out The Better Plans: A second letter, and a second round of endorsements, sho...