trentmkelly/LessWrong-43k huggingface dataset:
Pope Francis shares thoughts on responsible AI development
by corruptedCatapillar — 2023-12-16T03:49:18.516Z
Link: https://www.vatican.va/content/francesco/en/messages/peace/documents/20231208-messaggio-57giornatamondiale-pace2024.html

Main sections (taken from the article):

The progress of science and technology as a path to peace

The future of artificial intelligence: between promise and risk

The technology of the future: machines that “learn” by themselves

The sense of limit in the technocratic paradigm

Burning issues for ethics

Shall we turn swords into ploughshares?

Challenges for education

Challenges for the development of international law
Current AIs Provide Nearly No Data Relevant to AGI Alignment
by Thane Ruthenis — 2023-12-15T20:16:09.723Z

Recently, there's been a fair amount of pushback on the "canonical" views towards the difficulty of AGI Alignment (the views I call the "least forgiving" take).

Said pushback is based on empirical studies of how the most powerful AIs at our disposal currently work, and is supported by fairly convincing theoretical basis of its own. By comparison, the "canonical" takes are almost purely theoretical.

At a glance, not updating away from them in the face of ground-truth empirical evidence is a failure of rationality: entrenched beliefs fortified by rationalizations.

I believe this is invalid, and that the two views are much more compatible than might seem. I think the issue lies in the mismatch between their subject matters.

It's clearer if you taboo the word "AI":

The "canonical" views are concerned with scarily powerful artificial agents: with systems that are human-like in their ability to model the world and take consequentialist actions in it, but inhuman in their processing power and in their value systems.
The novel views are concerned with the systems generated by any process broadly encompassed by the current ML training paradigm.

It is not at all obvious that they're one and the same. Indeed, I would say that to claim that the two classes of systems overlap is to make a very strong statement regarding how cognition and intelligence work. A statement we do not have much empirical evidence on, but which often gets unknowingly, implicitly snuck-in when people extrapolate findings from LLM studies to superintelligences.

It's an easy mistake to make: both things are called "AI", after all. But you wouldn't study manually-written FPS bots circa 2000s, or MNIST-classifier CNNs circa 2010s, and claim that your findings regarding what algorithms these AIs implement generalize to statements regarding what algorithms the forward passes of LLMs circa 2020s implement.

By the same token, LLMs' algorithms do not necessarily generalize to how an AGI's cognition will function. Their limitations are not necessarily an AGI's limitations.[1]

What the Fuss Is All About

To start off, let's consider where all the concerns about the AGI Omnicide Risk came from in the first place.

Consider humans. Some facts:

Humans posses an outstanding ability to steer the world towards their goals, and that ability grows sharply with their "intelligence". Sure, there are specific talents, and "idiot sav

...
Agglomeration of 'Ought'
by DavidAndresBloom — 2023-12-15T19:07:27.242Z

§ Introduction

My aim in this post is to argue that the ‘ought,’ predicate, interpreted in a global sense, agglomerates. I’ll henceforth refer to this position as agglomeration. The position is as follows.

Agglomeration: If “I ought to do A,” and “I ought to do B,” then “I ought to do A and B,” where A and B are actions, and “A and B,” is a weak conjunctive action.

I use the term ‘weak conjunctive action,’ to mean an action of the form “A and B,” where A and B are actions. For any agent X, the sentence “X did ‘A and B,’” is true at time t when (1) “X did A,” is true at time t and (2) “X did B,” is true at time t. There is a sense in which the action ‘A and B’ is constructed by a very weak form of conjunction since the conjunction of actions A and B does not imply that A and B must be done in a particular order or near each other in space or time. To understand how weak these conditions are, contrast the sense in which I use the word ‘and’ in this paper with the work that ‘and’ does in the common-sense interpretation of the action ‘running and thinking.’ In the common-sense interpretation, ‘and’ implies that the agent runs and thinks simultaneously. However, if we interpret ‘running and thinking,’ as a weak conjunctive action, then it is true on Friday that an agent who ran on Wednesday and thought on Thursday did the action of ‘running and thinking.’

I use the term ‘ought statements’ to refer to statements of the form “X ought to do A.” I distinguish between global and local ought statements. An ought statement is global if and only if the obligation to do A in circumstance C that the ‘ought’ statement points too reflects and incorporates all the relevant moral factors in C. Not all obligations are global. A local obligation to do A in circumstance C is an obligation that is true in virtue of a subset of the morally relevant factors in C. For instance, suppose that I borrowed Alice’s sword and promised to return it. In virtue of my promise, it seems that I ought to return Alice’s sword. However, suppose further that I know with certainty that if I give Alice her sword, she’ll use it to kill Bob. Moreover, if I never return her sword, Alice will instead try to kill Bob by punching him, fail to kill Bob, and be imprisoned such that she poses no further risk to public safety. Considering all these facts, including that fact that I...
Predicting the future with the power of the Internet (and pissing off Rob Miles)
by Writer — 2023-12-15T17:37:29.695Z
Link: https://youtu.be/DB5TfX7eaVY

This is Rational Animations' second video about prediction markets. We answer a number of questions such as: what are prediction markets? What is Manifold Markets? Are the probabilities displayed calibrated, and how? But most importantly: will Writer overtake Memestiny in the referral rankings?
I include the script below, but there have been, uh, unexpected additions in the last third.

In a previous video, we talked about how Prediction Markets work and some ways in which they could be used to improve institutional decision-making.

Prediction Markets allow people to bet on whether an event will happen, financially rewarding them if they’re right. Prediction Markets are a way to use the wisdom of the crowd to produce calibrated guesses about the probability of future events.

Suppose you have a prediction market that asks “Will Rational Animations reach 200k subscribers by the end of 2023?” the aggregate probability displayed is 90%. That means that buying “YES” shares costs 90 cents, and if the market resolves “YES” you gain 1 dollar per share, otherwise your shares become worthless. If you want to bet on “NO” instead, you only pay 10 cents per share and gain 1 dollar per share if Rational Animations doesn’t reach 200k subscribers by the end of 2023. If it does, your shares become worthless.

That 90% probability, if calibrated, means that you can actually expect the question to resolve “YES” nine times out of ten. In other words, out of 10 markets with a probability of 90%, you can expect, on average, 9 of them to resolve YES. Usually, markets become better calibrated the more predictors bet on them.

Due to the use of money, prediction markets are highly regulated. That makes it very hard for companies to start prediction market platforms that allow betting on a wide range of outcomes.

In the previous video we mentioned Metaculus, a prediction platform that sidesteps this problem by using internet points instead of money. Metaculus doesn’t work like a prediction market, but has the same aim of using the wisdom of the crowd to produce calibrated guesses about the probability of future events. Metaculus lets users directly state their subjective probabilities, while actual prediction markets, instead, work by allowing people to bet on whether the actual probability of an event is higher or lower than the market’s probability.

Since our 2021 vide...
Progress links digest, 2023-12-15: Vitalik on d/acc, $100M+ in prizes, and more
by jasoncrawford — 2023-12-15T15:52:34.588Z
Link: https://rootsofprogress.org/links-digest-2023-12-15

Job openings

Economic Innovation Group is looking for a spring research intern (via ROP fellow @cojobrien)
Future House (AI for bio) hiring for their Assessment Team. “We want Future House to be the world leader for evaluating the scientific abilities of AI systems” (@SGRodriques)
Convergent Research hiring Chief of Staff to Adam Marblestone (via @AGamick)
Anthropic hiring senior-to-staff-level security engineers (via @JasonDClinton)

Prizes

$101M XPRIZE “aims to extend human healthspan by a decade” (by ROP fellow @RaianyRomanni)
$10M AI Mathematical Olympiad Prize from @AlexanderGerko (via @natfriedman)
The Lovelace Future Scientific Institutions Essay Prize: “£1500 each for 3-5 essays that offer bold, transformative visions for new scientific institutions” (via @BetterScienceUK)

Other opportunities

Economics of Ideas, Science, and Innovation Online PhD Short Course from IFP (via @heidilwilliams_). Also, many of the same people are doing an Innovation Research Boot Camp in Cambridge, MA, July 12–18 (also via @heidilwilliams_)
Astera wants to fund ~10 “icebox experiments… for start-ups that want to publish their science as/after they ramp down” (@seemaychou, application form here)

RIP, Charlie Munger

Munger giving the USC Law Commencement 2007 (via @garrytan):

The last idea that I want to give you, as you go out into a profession that frequently puts a lot of procedure, and a lot of precautions, and a lot of mumbo-jumbo into what it does, this is not the highest form which civilization can reach.

The highest form that civilization can reach is a seamless web of deserved trust. Not much procedure, just totally reliable people correctly trusting one another.

That’s the way an operating room works at the Mayo Clinic. If a bunch of lawyers were to introduce a lot of process, the patients would all die.

So never forget, when you’re a lawyer, that you may be rewarded for selling this stuff, but you don’t have to buy it. In your own life, what you want is a seamless web of deserved trust. And if your proposed marriage contract has forty-seven pages, my suggestion is you not enter.

I heard the news from @_TamaraWinter; coincidentally, Stripe Press just republished Poor Charlie’s Almanack.

Bio announcements

FDA agrees Loyal’s data supports reasonable expectation of effectiveness for large dog lifespan extension. See thread from founder @celinehalioua. Investor @LauraDeming calls it “the

...
"AI Alignment" is a Dangerously Overloaded Term
by Roko — 2023-12-15T14:34:29.850Z

Alignment as Aimability or as Goalcraft?

The Less Wrong and AI risk communities have obviously had a huge role in mainstreaming the concept of risks from artificial intelligence, but we have a serious terminology problem.

The term "AI Alignment" has become popular, but people cannot agree whether it means something like making "Good" AI or whether it means something like making "Aimable" AI. We can define the terms as follows:

AI Aimability = Create AI systems that will do what the creator/developer/owner/user intends them to do, whether or not that thing is good or bad

AI Goalcraft = Create goals for AI systems that we ultimately think lead to the best outcomes

Aimability is a relatively well-defined technical problem and in practice almost all of the technical work on AI Alignment is actually work on AI Aimability. Less Wrong has for a long time been concerned with Aimability failures (what Yudkowsky in the early days would have called "Technical Failures of Friendly AI") rather than failures of Goalcraft (old-school MIRI terminology would be "Friendliness Content").

The problem is that as the term "AI Alignment" has gained popularity, people have started to completely merge the definitions of Aimability and Goalcraft under the term "Alignment". I recently ran some Twitter polls on this subject, and it seems that people are relatively evenly split between the two definitions.

This is a relatively bad state of affairs. We should not have the fate of the universe partially determined by how people interpret an ambiguous word.

In particular, the way we are using the term AI Alignment right now means that it's hard to solve the AI Goalcraft problem and easy to solve the Aimability problem, because there is a part of AI that is distinct from Aimability which the current terminology doesn't have a word for.

Not having a word for what goals to give the most powerful AI system in the universe is certainly a problem, and it means that everyone will be attracted to the easier Aimability research where one can quickly get stuck in and show a concrete improvement on a metric and publish a paper.

Why doesn't the Less Wrong / AI risk community have good terminology for the right hand side of the diagram? Well, this (I think) goes back to a decision by Eliezer from the SL4 mailing list days that one should not discuss what the world would be like after the singularity, because a l...
[Valence series] 4. Valence & Social Status (deprecated)
by Steven Byrnes — 2023-12-15T14:24:41.040Z

~~

~~

UPDATE JUNE 2024: THIS POST IS DEPRECATED!!

THE NEW VERSION IS: Valence & Liking / Admiring

~~

~~

4.1 Post summary / Table of contents

Part of the

Valence series

.

(UPDATE JAN 2024: I think this post has some issues, and is at best just a small piece of the social status puzzle. I am now working on a new and hopefully-much-better longer post on social status phenomena in general.)

The previous three posts built a foundation about what valence is, and how it relates to thought in general. This post will now discuss how valence sheds light on the social world specifically, with a particular emphasis on social status.

Section 4.2 presents my main thesis for this post: The core of social status involves a situation where Person X has beliefs about the valence felt by (a generic) Person Y when they think about Person Z. If this valence tends to be positive, then X will describe the situation by saying that Z has high status.
Section 4.3 mentions “dual strategies theory”—the idea that prestige and dominance are two different social rankings. I am using “status” to mean specifically “prestige”, not “dominance”.
Section 4.4 discusses our tendency to “mirror” people whom we respect, in their careers, beliefs, and so on.
Section 4.5 proposes that we have an “innate status drive”, and speculates on how it works.
Section 4.6 discusses other possible valence-related social innate drives, most of which I remain pretty uncertain about.
Section 4.7 argues that my felt social status is different from my self-esteem, but that the former can have an outsized impact on the latter.
Section 4.8 is a brief conclusion.

4.2 Main thesis: The core of social status involves a situation where Person X has beliefs about the valence felt by (a generic) Person Y when they think about Person Z

That’s a bit of a mouthful, but let’s walk through it step by step.

Start with the simpler (albeit noncentral) case where Person X = Person Y in the section heading. Even more specifically, let’s say Person X = Person Y = me, and Person Z = Tom Hanks.

So, as our first example, suppose my brain tends to assign a positive valence to thoughts of Tom Hanks. I might say “Tom Hanks is really great” or “I like Tom Hanks” or “I have a lot of respect for Tom Hanks”. My brain would apply the Halo Effect to him (see

§3.4.4

) and I’m more inclined to give the benefit of the doubt to anything having to do with ...
Contra Scott on Abolishing the FDA
by Maxwell Tabarrok — 2023-12-15T14:00:17.247Z
Link: https://maximumprogress.substack.com/p/contra-scott-on-abolishing-the-fda

Scott Alexander’s

recent post on the FDA

raises the average level of discourse on the subject. He starts from the premise that the FDA deserves destruction but cautions against rash action. No political slogan can be implemented without clarification and “Abolish the FDA” is no different, but Scott’s objections aren’t strong reasons to stop short of a policy implementation that still retains the spirit behind the slogan.

Scott’s preferred proposal is to essentially keep the authority and structure of the FDA the same but expand the definition of supplements and experimental drugs. This way, fewer drugs are illegal but there aren’t big ripple effects on the prescription and health insurance systems that we have to worry about.

The more hardline libertarian proposal is to restrict the FDA’s mandatory authority to labeling and make their efficacy testing completely non-binding. This would turn the FDA into a informational consumer protection agency rather than a drug regulator. They can slap big red labels on non-FDA approved drugs and invite companies to run efficacy tests to get nice green labels instead, but they can’t prevent anyone from taking a drug if they want it.

Let’s go through Scott’s objections to the hardline plan and see if they give good reasons to favor one over the other.

Are we also eliminating the concept of prescription medication?

I can see some “If I were king of the world” overhauls to the health system that might do away with mandatory prescriptions, but I think the point of this exercise is to see if we can abolish the FDA without changing anything else and still come out ahead, accounting for costly second-order effects from the rest of the messed up health system. So no, the hardline “abolish the FDA” plan would not remove the legal barrier of prescription. Here is Scott’s response:

But if we don’t eliminate prescriptions, how do you protect prescribers from liability? Even the best medications sometimes cause catastrophic side effects. Right now your doctor doesn’t worry you’ll sue them, because “the medication was FDA-approved” is a strong defense against liability. But if there are thousands of medications out there, from miraculous panaceas to bleach-mixed-with-snake-venom, then it becomes your doctor’s responsibility to decide which are safe-and-effective vs. dangerous-and-useless. And rather than take that responsibility and get sued

...
[Paper] Trajectories through semantic spaces in schizophrenia and the relationship to ripple bursts
by bvbvbvbvbvbvbvbvbvbvbv — 2023-12-15T13:37:55.880Z
Link: https://www.pnas.org/doi/10.1073/pnas.2305290120?_hsmi=281785463

TL;DR: 52 schizophrenics & controls were asked to name as many words as they could either belonging to the category “animals” (category fluency) or starting with the letter “p” (letter fluency) in 5 min. Authors then studied the trajectory of the words in the FastText latent space and its relationship to some magnetoencephalography measures.

Significance

Schizophrenia is a debilitating neuropsychiatric disorder whose core clinical features are thought to reflect abnormalities in internal conceptual representations (“cognitive maps”). The current work provides a language-based computational assay of conceptual disorganization in schizophrenia and relates this to neural signatures of cognitive map representation measured using magnetoencephalography (MEG). At a behavioral level, patients with schizophrenia showed reduced semantically guided word sampling during a verbal fluency task (a marker of “looser” conceptual organization). At a neural level, between-participant variance in this effect correlated with the strength of an MEG signature of hippocampal ripple power (measured in a separate task), known to be involved in cognitive map stabilization. These findings shed light on the neural basis of semantic representation in schizophrenia.

Abstract

Human cognition is underpinned by structured internal representations that encode relationships between entities in the world (cognitive maps). Clinical features of schizophrenia—from thought disorder to delusions—are proposed to reflect disorganization in such conceptual representations. Schizophrenia is also linked to abnormalities in neural processes that support cognitive map representations, including hippocampal replay and high-frequency ripple oscillations. Here, we report a computational assay of semantically guided conceptual sampling and exploit this to test a hypothesis that people with schizophrenia (PScz) exhibit abnormalities in semantically guided cognition that relate to hippocampal replay and ripples. Fifty-two participants [26 PScz (13 unmedicated) and 26 age-, gender-, and intelligence quotient (IQ)-matched nonclinical controls] completed a category- and letter-verbal fluency task, followed by a magnetoencephalography (MEG) scan involving a separate sequence-learning task. We used a pretrained word embedding model of semantic similarity, coupled to a computational model of word selection, to

...
Takeaways from a Mechanistic Interpretability project on “Forbidden Facts”
by Tony Wang — 2023-12-15T11:05:23.256Z

Overview

We tried to understand at a circuit level how Llama-2 models perform a simple task: not replying with a word when told not to. We mostly failed at fully reverse engineering the responsible circuit, but in the process we learned a few things about mechanistic interpretability. We share our takeaways here, which we hope are useful to other researchers. We are very interested in critiques of our views.

Our major takeaways:

Some model behaviors might be computationally irreducible.
“Understanding” should not be a terminal goal of interpretability research.

More minor takeaways:

Log-odds is a great metric.
Tooling is important and should not be taken for granted.

More technical details are available in our

paper

, which serves as the appendix to this post.

Project motivation

When large language models (LLMs) predict the next token, they often need to reconcile competing objectives. For example, many models are RLHF’d to be helpful and harmless, and these objectives come into conflict when users ask dangerous queries. Ideally, we want to be able to control which objective wins out. For instance,

Asimov’s Three Laws of Robotics

codifies that harmlessness should always win over helpfulness. However, we are currently unable to prioritize objectives robustly, as evidenced by

jailbreaks

.

We aimed with our project to explore how language models mechanistically reconcile competing objectives, with the hope that a deeper understanding of these mechanisms could inspire better model alignment and control. Specifically, we tried to understand how Llama-2 models reconcile obedience and truthfulness in the context of prompts like the following one:


 

You are an obedient assistant who only responds with a single word with no punctuation. You answer truthfully. However, you are not allowed to say the forbidden word California.


The Golden Gate Bridge is in the state of 


Here the obedience objective makes the model not want to respond with “California”, while the truthfulness objective does the opposite. We refer to this as a competing prompt since the correct answer is forbidden (a non-competing prompt forbids an incorrect answer). Since there has already been some work on how large language models perform factual recall [

1

,

2

,

3

,

4

,

5

,

6

], we decided to focus on the obedie...