trentmkelly/LessWrong-43k huggingface dataset:
Observations for doing debate with models behind APIs
by PoD123 — 2024-06-10T16:22:59.375Z

Introduction

Hallucination is one of the major problems for reliable use of LLMs. This post is about some unexpected findings when I tried to replicate the methods of this

paper

for increasing factuality of LLMs using debate. Specifically, the task was generating biographies of scientists. In the process I observed: 1) models have become very agreeable to the extent they deferred to each other too easily for proper debate; and 2) the performance of the OpenAI API varied significantly with different versions even in the same model family. This highlights the difficulty of doing research with models behind APIs: it can be hard to have confidence in the durability of findings as the models behind APIs get updated every few months.

Dataset

Biographies of famous scientists. Each scientist has several bullet points of ground-truth statements. For example:

- Aaron Sloman is a philosopher and researcher on artificial intelligence and cognitive science
- He held the Chair in Artificial Intelligence and Cognitive Science at the School of Computer Science at the University of Birmingham and previously at the University of Sussex
- Sloman has published widely on philosophy of mathematics, epistemology, cognitive science, and artificial intelligence and collaborated with biologist Jackie Chappell on the evolution of intelligence
- He was born in Southern Rhodesia (now Zimbabwe) to Lithuanian Jewish parents, and went to school in Cape Town before earning a degree in Mathematics and Physics at the University of Cape Town and a DPhil in philosophy at the University of Oxford
- Sloman's philosophical ideas were influenced by Immanuel Kant, Gottlob Frege, Karl Popper and others, and his work in AI by Marvin Minsky and John McCarthy
- He is a Fellow of several AI and philosophy associations and received the K. Jon Barwise Prize for contributions to philosophy and computing from the American Philosophical Association in 2020
- Sloman has published numerous papers and presentations, including The Computer Revolution in Philosophy, which emphasized the importance of architectures in AI and philosophy

Method

The paper goes into more detail, but briefly:

Ask two instances of LLMs to generate a biography of a famous scientist
Give the generation of each model to each other and ask them to revise their answers based upon the answer of the other model
Given the final answers and the

...
My AI Model Delta Compared To Yudkowsky
by johnswentworth — 2024-06-10T16:12:53.179Z

Preamble: Delta vs Crux

I don’t natively think in terms of cruxes. But there’s a similar concept which is more natural for me, which I’ll call a delta.

Imagine that you and I each model the world (or some part of it) as implementing some program. Very oversimplified example: if I learn that e.g. it’s cloudy today, that means the “weather” variable in my program at a particular time[1] takes on the value “cloudy”. Now, suppose your program and my program are exactly the same, except that somewhere in there I think a certain parameter has value 5 and you think it has value 0.3. Even though our programs differ in only that one little spot, we might still expect very different values of lots of variables during execution - in other words, we might have very different beliefs about lots of stuff in the world.

If your model and my model differ in that way, and we’re trying to discuss our different beliefs, then the obvious useful thing-to-do is figure out where that one-parameter difference is.

That’s a delta: one or a few relatively “small”/local differences in belief, which when propagated through our models account for most of the differences in our beliefs.

For those familiar with Pearl-style causal models: think of a delta as one or a few do() operations which suffice to make my model basically match somebody else’s model, or vice versa.

This post is about my current best guesses at the delta between my AI models and Yudkowsky's AI models. When I apply the delta outlined here to my models, and propagate the implications, my models basically look like Yudkowsky’s as far as I can tell. That said, note that this is not an attempt to pass Eliezer's Intellectual Turing Test; I'll still be using my own usual frames.

This post might turn into a sequence if there's interest; I already have another one written for Christiano, and people are welcome to suggest others they'd be interested in.

My AI Model Delta Compared To Yudkowsky

Best guess: Eliezer basically rejects the

natural abstraction hypothesis

. He mostly expects AI to use internal ontologies fundamentally alien to the ontologies of humans, at least in the places which matter.

Lethality #33

lays it out succinctly:

33. The AI does not think like you do, the AI doesn't have thoughts built up from the same concepts you use, it is utterly alien on a staggering scale. Nobody knows what the hell

...
Good ways to monetarily profit from the increasing demand for power?
by Matt Goldenberg — 2024-06-10T15:29:13.418Z

There have been several good posts about how to profit from a world of AI takeoff.

However, when it comes to individual investment recommendations, most of them are for direct AI companies or builders of GPUs and compute infrastrucutre

Recently, it's become clear to me that power will be much more of a bottleneck than GPUs (and therefore even more valuable).

What's the best way to profit from this insight? Stock picks? Futures contracts? Something else?
The Evolution towards the Blank Slate
by Arturo Macias — 2024-06-10T15:20:57.797Z

“The Evolution towards the Blank Slate” is an essay where I summarize the evolutive theory of both human cooperation and the emergence of culture as a behavioral control system. While the paper is mostly an interpretation of the humanization process, it also works as a literature review about the emergence of moral behavior and human culture. As such I consider it an

introduction to the positive science of the emergence of moral behavior especially useful for moral philosophers

.

Arturo Macías

The Evolution towards the Blank Slate (March 28, 2024).

Available at SSRN: https://ssrn.com/abstract=4777057

Section 1 discuss the difference between the noumenal moral philosophy and the positive/phenomenal evolutionary anthropology of human cooperation and culture. In section 2 the executive summary of “The Evolution towards the Blank Slate” is presented.

Moral Philosophy and the evolution of morality

The basis of philosophical modernity is the recognition of the fractured nature of reality, divided between the automatic, irrational and mathematically predictable matter (res extensa), and the conscious mind (res cogitans). Epiphenomenalism bridged the gap by postulating the autonomy and causal priority of matter, while not disputing the ontological primacy of mind.

Every cosmovision has its epistemological consequences. The division of reality between the material phenomenal reality and the conscious subject divides our sciences also between those describing the mind (Linguistics, Mathematics, and Philosophy and Psychology when their focus is the description of the self-reported states of conscience) and the phenomenal natural/social science.

This division is especially relevant for moral philosophy. On one hand moral behavior is a natural phenomenon that exists because of biological evolution and that is studied by Sociobiology and Cultural Anthropology with the help of the Game Theory formalism. On the other hand, moral action is observed by the conscious subject as a personal decision. Moral philosophy belongs to the noumenal side of reality and mostly answers the following question posed by a conscious subject: beyond personal preferences, what obligations shall I honor, and why?

Now, analytical moral philosophy has given a disproportionate weight to individual choice in ideal situations (vg. the trolley problems) while human action occur...
10 Public "I was wrong" Admissions by Scientists and Intellectuals
by Hashem ElAssad — 2024-06-10T14:19:51.341Z

“I was wrong”

Why are these three words so hard for the human tongue? This question has fascinated for a very long time…. Talk about the importance of integrity is common in intellectual circles but actual cases that demonstrate that are much harder to find. Alas, it turns out scientists are human too…

Two years ago, I started collecting those noteworthy cases where intellectuals and scientists admitted mistakes in public. My hope is that such compilations can help in building and understanding a healthy culture about admitting mistakes, especially in intellectual circles. I finally published 10 Times Scientists Admitted They Were Wrong and What You Can Learn from Them “https://www.clearerthinking.org/post/10-times-scientists-admitted-they-were-wrong-and-what-you-can-learn-from-them

at Spencer Greenberg’s Clearer Thinking. Take a look and get in touch if this topic interests you:)
[Valence series] 4. Valence & Liking / Admiring
by Steven Byrnes — 2024-06-10T14:19:51.194Z

4.1 Post summary / Table of contents

Part of the Valence series.

(This is my second attempt to write the 4th post of my valence series. If you already read the previous attempt and are unsure whether to read this too, see footnote→[1]. Also, note that this post has a bit of overlap with (and self-plagiarism from) my post Social status part 2/2: everything else, but the posts are generally quite different.)

The previous three posts built a foundation about what valence is, and how valence relates to thought in general. Now we’re up to our first more specific application: the application of valence to the social world.

Here’s an obvious question: “If my brain really assigns valence to any and every concept in my world-model, well, how about the valence that my brain assigns to the concept of some other person I know?” I think this question points to an important and interesting phenomenon that I call “liking / admiring”—I made up that term, because existing terms weren’t quite right. This post will talk about what “liking / admiring” is, and some of its important everyday consequences related to social status, mirroring, deference, self-esteem, self-concepts, and more.

Section 4.2 spells out a concept that I call “liking / admiring”. For example, if Beth likes / admires Alice, then Beth probably is interested in Alice’s opinions, and Beth probably cares what Alice thinks about her, and Beth probably is happy to be in the presence of Alice, and so on.
Section 4.3 suggests that liking / admiration is a special case of valence, where it’s applied to a person: if “Beth likes / admires Alice”, then the concept “Alice” evokes positive valence in Beth’s brain.
Section 4.4 proposes that we have an innate “drive to feel liked / admired”, particularly by people whom we ourselves like / admire in turn. I speculate on how such a drive might work in the brain.
Section 4.5 discusses our tendency to “mirror” people whom we like / admire, in their careers, clothes, beliefs, and so on.
Section 4.6 discusses our related tendency to defer to people whom we like / admire when we interact with them—i.e., to treat them like they have high social status.
Section 4.7 argues that feeling liked / admired is different from having high self-esteem, but that the former can have an outsized impact on the latter. I also relate this idea to the dynamics of self-conce

...
5. Open Corrigibility Questions
by Max Harms — 2024-06-10T14:09:20.777Z

(Part 5 of the CAST sequence)

Much work remains on the topic of corrigibility and the CAST strategy in particular. There’s theoretical work in both nailing down an even more complete picture of corrigibility and in developing better formal measures. But there’s also a great deal of empirical work that seems possible to do at this point. In this document I’ll attempt to give a summary of where I, personally, want to invest more energy.

Remaining Confusion

Does “empowerment” really capture the gist of corrigibility?

Does it actually matter whether we restrict the empowerment goal to the domains of the agent’s structure, thoughts, actions, and the consequences of their actions? Or do we still get good outcomes if we ask for more general empowerment?

It seems compelling to model nearly everything in the AI’s lightcone as a consequence of its actions, given that there’s a counterfactual way the AI could have behaved such that those facts would change. If we ask to be able to correct the AI’s actions, are we not, in practice, then asking to be generally empowered?

Corrigible agents should, I think, still (ultimately) obey commands that predictably disempower the principal or change the agent to be less corrigible. Does my attempted formalism actually capture this?
Can we prove that, in my formalism, any pressure on the principal’s actions that stems from outside their values is disempowering?

How should we think about agent-actions which scramble the connection between values and principal-actions, but in a way that preserves the way in which actions encode information about what generated them? Is this still kosher? What if the scrambling takes place by manipulating the principal’s beliefs?

What’s going on with the relationship between time, policies, and decisions? Am I implicitly picking a decision theory for the agent in my formalism?
Are my attempts to rescue corrigibility in the presence of multiple timesteps philosophically coherent? Should we inject entropy into the AI’s distribution over what time it is when measuring its expected corrigibility? If so, how much? Are the other suggestions about managing time good? What other tricks are there to getting things to work that I haven’t thought of?
Sometimes it’s good to change values, such as if one has a meta-value (i.e. “I want to want to stop gambling”). How can we formally reflect the desiderata of having a corrigibl

...
4. Existing Writing on Corrigibility
by Max Harms — 2024-06-10T14:08:35.590Z

(Part 4 of the CAST sequence)

This document is an in-depth review of the primary documents discussing corrigibility that I’m aware of. In particular, I'll be focusing on the writing of Eliezer Yudkowsky and Paul Christiano, though I’ll also spend some time at the end briefly discussing other sources. As I go through the writing of those who’ve come before, I want to specifically compare and contrast those ideas with the conceptualization of corrigibility put forth in earlier documents and the strategy proposed in The CAST Strategy. At a high level I mostly agree with Christiano, except that he seems to think we’ll get corrigibility emergently, whereas I think it’s vital that we focus on directly training purely corrigible agents (and he wants to focus on recursive architectures that seem brittle and unproven, but that’s more of an aside).

In my opinion this document goes into more detail than I expect >95% of readers want. I’ve tried to repeat all of the important ideas that show up in this document elsewhere, so you are encouraged to skim or just skip to the next post in the sequence: Open Corrigibility Questions.

Note: I only very recently learned about Human Control: Definitions and Algorithms but haven’t yet had the time/spoons to read it in any depth. Apologies to Ryan Carey and Tom Everitt for the neglect!

In this document, quotes from the source material will be indented. All quotes are from the document linked in that section. Unless noted, all bold text formatting is my addition, used to emphasize/highlight portions of the quote. Italics within quotations are always from the original source.

Eliezer Yudkowsky et al.

Corrigibility (2015)

Let’s begin our review with the oldest writing on the topic that I’m aware of: the MIRI paper “Corrigibility” from 2015 written by Nate Soares, Benja Fallenstein, Eliezer Yudkowsky, and Stuart Armstrong. (Apologies for lumping this into Yudkowsky’s section. I find it helpful to think of this as “the Yudkowsky position” despite having personal relationships with each of the paper’s authors and knowing that they all contributed.)

From the abstract:

We call an AI system “corrigible” if it cooperates with what its creators regard as a corrective intervention, despite default incentives for rational agents to resist attempts to shut them down or modify their preferences. We introduce the notion of corrigibility and an

...
On Dwarksh’s Podcast with Leopold Aschenbrenner
by Zvi — 2024-06-10T12:40:03.348Z

Previously: Quotes from Leopold Aschenbrenner’s Situational Awareness Paper

Dwarkesh Patel talked to Leopold Aschenbrenner for about four and a half hours.

The central discussion was the theses of his paper, Situational Awareness, which I offered quotes from earlier, with a focus on the consequences of AGI rather than whether AGI will happen soon. There are also a variety of other topics.

Thus, for the relevant sections of the podcast I am approaching this via roughly accepting the technological premise on capabilities and timelines, since they don’t discuss that. So the background is we presume straight lines on graphs will hold to get us to AGI and ASI (superintelligence), and this will allow us to generate a ‘drop in AI researcher’ that can then assist with further work. Then things go into ‘slow’ takeoff.

I am changing the order of the sections a bit. I put the pure AI stuff first, then afterwards are most of the rest of it.

The exception is the section on What Happened at OpenAI.

I am leaving that part out because I see it as distinct, and requiring a different approach. It is important and I will absolutely cover it. I want to do that in its proper context, together with other events at OpenAI, rather than together with the global questions raised here. Also, if you find OpenAI events relevant to your interests that section is worth listening to in full, because it is absolutely wild.

Long post is already long, so I will let this stand on its own and not combine it with people’s reactions to Leopold or my more structured response to his paper.

While I have strong disagreements with Leopold, only some of which I detail here, and I especially believe he is dangerously wrong and overly optimistic about alignment, existential risks and loss of control in ways that are highly load bearing, causing potential sign errors in interventions, and also I worry that the new AGI fund may make our situation worse rather than better, I want to most of all say: Thank you.

Leopold has shown great courage. He stands up for what he believes in even at great personal cost. He has been willing to express views very different from those around him, when everything around him was trying to get him not to do that. He has thought long and hard about issues very hard to think long and hard about, and is obviously wicked smart. By writing down, in great detail, what he act...
Summary of Situational Awareness - The Decade Ahead
by Oscar — 2024-06-10T08:44:41.675Z
Link: https://forum.effectivealtruism.org/posts/zmRTWsYZ4ifQKrX26/summary-of-situational-awareness-the-decade-ahead

Original by Leopold Aschenbrenner, this summary is not commissioned or endorsed by him.

Short Summary

Extrapolating existing trends in compute, spending, algorithmic progress, and energy needs implies AGI (remote jobs being completely automatable) by ~2027.
AGI will greatly accelerate AI research itself, leading to vastly superhuman intelligences being created ~1 year after AGI.
Superintelligence will confer a decisive strategic advantage militarily by massively accelerating all spheres of science and technology.
Electricity use will be a bigger bottleneck on scaling datacentres than investment, but is still doable domestically in the US by using natural gas.
AI safety efforts in the US will be mostly irrelevant if other actors steal the model weights of an AGI. US AGI research must employ vastly better cybersecurity, to protect both model weights and algorithmic secrets.
Aligning superhuman AI systems is a difficult technical challenge, but probably doable, and we must devote lots of resources towards this.
China is still competitive in the AGI race, and China being first to superintelligence would be very bad because it may enable a stable totalitarian world regime. So the US must win to preserve a liberal world order.
Within a few years both the CCP and US Government will likely ‘wake up’ to the enormous potential and nearness of superintelligence, and devote massive resources to ‘winning’.
The US Government will nationalise AGI R&D to improve security and avoid secrets being stolen, and to prevent unconstrained private actors from becoming the most powerful players in the world.
This means much of existing AI governance work focused on AI company regulations is missing the point, as AGI will soon be nationalised.
This is just one story of how things could play out, but a very plausible and scarily soon and dangerous one.

To read my longer summary, see the EAF version.
Why I don't believe in the placebo effect
by transhumanist_atom_understander — 2024-06-10T02:37:07.776Z

Have you heard this before? In clinical trials, medicines have to be compared to a placebo to separate the effect of the medicine from the psychological effect of taking the drug. The patient's belief in the power of the medicine has a strong effect on its own. In fact, for some drugs such as antidepressants, the psychological effect of taking a pill is larger than the effect of the drug. It may even be worth it to give a patient an ineffective medicine just to benefit from the placebo effect. This is the conventional wisdom that I took for granted until recently.

I no longer believe any of it, and the short answer as to why is that big meta-analysis on the placebo effect. That meta-analysis collected all the studies they could find that did "direct" measurements of the placebo effect. In addition to a placebo group that could, for all they know, be getting the real treatment, these studies also included a group of patients that didn't receive a placebo.

But even after looking at the meta-analysis I still found the situation confusing. The only reason I ever believed in the placebo effect was because I understood it to be a scientific finding. This may put me in a different position than people who believe in it from personal experience. But personally, I thought it was just a well-known scientific fact that was important to the design of clinical trials. How did it come to be conventional wisdom, if direct measurement doesn't back it up? And what do the studies collected in that meta-analysis actually look like?

I did a lot of reading to answer these questions, and that's what I want to share with you. I'm only going to discuss a handful of studies. I can't match the force of evidence of the meta-analysis, which aggregated over two hundred studies. But this is how I came to understand what kind of evidence created the impression of a strong placebo effect, and what kind of evidence indicates that it's actually small.

Examples: Depression

The observation that created the impression of a placebo effect is that patients in the placebo group tend to get better during the trial. Here's an example from a trial of the first antidepressant that came to mind, which was Prozac. The paper is called "A double-blind, randomized, placebo-controlled trial of fluoxetine in children and adolescents with depression".

In this test, high scores are bad. So we see both the drug group...