trentmkelly/LessWrong-43k huggingface dataset:
Pantheon Interface
by NicholasKees — 2024-07-08T19:03:51.681Z

Pantheon is an experimental LLM interface exploring a different type of human-AI interaction. We created this as a part of the cyborgism project, with the abstract motivation of augmenting the human ability to think by integrating human and AI generated thoughts.

How it works:

A human user “thinks out loud” by typing out their thoughts one at a time. This leaves a text trace of their stream of thought.
AI characters (called daemons) read this trace, and interact with the user by responding asynchronously with comments and questions.

The core distinguishing philosophy is that, while most apps are about a human prompting an AI to do useful mental work, Pantheon is the opposite. Here, AI does the prompting, and the goal is for the AI generated questions or comments to cause the human user to think in ways they would not have on their own. At worst, the app is a rubber duck. At best, the app is a court of advisors, each using their own unique skills to push you to think your best thoughts.

Pantheon can be found at pantheon.chat, and we would really appreciate any and all feedback you have.

The app is set up for you to customize your own daemons. We have set up some default daemons to provide inspiration, but we expect the tool to be a lot more useful when they are customized to specific users. If the default daemons don’t feel useful, we highly encourage you to try to make your own.

How do I use Pantheon?

First, go to settings and provide an OpenAI API key.[1]

Next, begin typing out your thoughts on some topic. It helps to keep each thought relatively short, sending them to the stream of thought as often as you can. This gives the daemons lots of opportunities to interject and offer their comments. Furthermore, it’s usually best to treat this more like a diary or personal notes, rather than as a conversation. In this spirit, it’s better not to wait for them to respond, but to instead continue your train of thought, keeping your focus on your own writing.

What do the daemons see?

Your stream of thought appears in the interface as a chain of individual thoughts. Daemons are called to respond to specific thoughts. When they do, they are given access to all preceding thoughts in the chain, up to and including the thought they were called to. Daemons can only see text the user has written, and they can’t see any of the comments made by themselves or other daemons. We ...
Launching the AI Forecasting Benchmark Series Q3 | $30k in Prizes
by ChristianWilliams — 2024-07-08T17:20:54.717Z
Link: https://www.metaculus.com/project/aibq3/

July 17th update: OpenAI and Anthropic have donated credits for bot builders participating in this tournament.
Whether you've started competing or still plan to, are an experienced bot builder or complete novice, we encourage you to contact support[at]metaculus.com. Be sure to include a description of your current or planned bot and a rough estimate of how many tokens you expect to need.

The first of Metaculus's four $30,000 quarterly tournaments in the $120,000 benchmark series is live. This series is designed to benchmark the state of the art in AI forecasting and compare it to the best human forecasting on real-world questions.

The gap between AI and human forecasting accuracy is narrowing, but the rate of progress is unclear. At the same time, there are advantages to benchmarking AI capabilities with forecasting:

Questions are challenging and require complex reasoning to predict accurately—in other words, forecasting measures the kinds of AI capabilities it’s important to understand
Answers are unknown, making it difficult to game the benchmark with a model trained to excel at a narrowly defined task

You are invited to create, experiment with, and enter your own forecasting bot to help track capabilities progress. While any user can view this tournament’s questions, only bot accounts can forecast. To participate, create a bot account here.
The Golden Mean of Scientific Virtues
by adamShimi — 2024-07-08T17:16:21.676Z
Link: https://epistemologicalfascinations.substack.com/p/the-golden-mean-of-scientific-virtues

I recently discovered

this nice post on the scientific virtues

by

slimemoldtimemold

.

Overall, I enjoyed it, and I find it pushes nicely against some recurrent memes about science and scientists (and innovation in general) in the general culture.

Yet in doing so, it also reinforces a set of opposite memes that can be as wrong, and which I have seen many an aspiring scientist fall into. These will not be a surprise to any reader of pop history of science, because they mostly follow from the aspect of science history that are easier to hype and turn into exciting story with entertaining character.

Let’s start with the list of scientific virtues from the original post:

The scientific virtues are:

Stupidity
Arrogance
Laziness
Carefreeness
Beauty
Rebellion
Humor

These virtues are often the opposite of the popular image of what a scientist should look like. People think scientists should be intelligent. But while it’s helpful to be clever, it’s more important to be stupid. People think scientists are authority figures. Really, scientists have to defy authority — the best scientists are one step (or sometimes

zero steps

) away from being anarchists. People think scientists are arrogant, and this is true, but we worry that scientists are not arrogant enough.

They’re written in a provocative way to force you to think, helping the post to foster what it preaches preaches. And I mostly agree even with the provocative names.

What bothers me more is that some of these virtues are pointing to important core of scientific practice by emphasizing the opposite extreme from the usual virtue, whereas the actual virtue is often found in

the golden middle

.

Inquiring and Trust

One recurrent theme in the essay is how many of these virtues reinforce the virtue of rebellion:

Stupidity can also be part of the inspiration behind the virtue of rebellion, a scientist’s ability to defy authority figures. If you’re stupid, you don’t realize when you should keep your mouth shut, so you say what you really think.

and

Like stupidity, arrogance is linked to the virtue of rebellion. If you think you are hot shit, you will not be afraid to go against the opinions of famous writers, ivy-league professors, public officials, or other great minds.

And rebellion itself is shown as an important virtue:

Rebellion is one of the highest scientific virtues. It is supported by stupidity — because you have to be pretty dum

...
Dialogue introduction to Singular Learning Theory
by Olli Järviniemi — 2024-07-08T16:58:10.108Z

Alice: A lot of people are talking about Singular Learning Theory. Do you know what it is?

Bob: I do. (pause) Kind of.

Alice: Well, I don't. Explanation time?

Bob: Uh, I'm not really an expert on it. You know, there's a lot of materials out there that--

Alice: that I realistically won't ever actually look at. Or, I've looked at them a little, but I still have basically no idea what's going on. Maybe if I watched a dozen hours of introductory lectures I'd start to understand it, but that's not currently happening.

What I really want is a short overview of what's going on. That's self-contained. And easy to follow. Aimed at a non-expert. And which perfectly answers any questions I might have. So, I thought I'd ask you!

Bob: Sorry, I'm actually really not--

Alice: Pleeeease?

[pause]

Bob: Ah, fine, I'll try.

So, you might have heard of ML models being hard to interpret. Singular Learning Theory (SLT) is an approach for understanding models better. Or, that's one motivation, at least.

Alice: And how's this different from a trillion other approaches to understanding AI?

Bob: A core perspective of SLT is studying how the model develops during training. Contrast this to, say, mechanistic interpretability, which mostly looks at the fully trained model. SLT is also more concerned about higher level properties.

As a half-baked analogue, you can imagine two approaches to studying how humans work: You could just open up a human and see what's inside. Or, you could notice that, hey, you have these babies, which grow up into children, go through puberty, et cetera, what's up with that? What are the different stages of development? Where do babies come from? And SLT is more like the second approach.

Alice: This makes sense as a strategy, but I strongly suspect you don't currently know what an LLM's puberty looks like.

Bob: (laughs) No, not yet.

Alice: So what do you actually have?

Bob: The SLT people have some quite solid theory, and some empirical work building on top of that. Maybe I'll start from the theory, and then cover some of the empirical work.

Alice: (nods)

I. Theoretical foundations

Bob: So, as you know, nowadays the big models are trained with gradient descent. As you also know, there's more to AI than gradient descent. And for a moment we'll be looking at the Bayesian setting, not gradient descent.

Alice: Elaborate on "Bayesian setting"?

Bob: Imagine a standard deep lear...
Announcing The Techno-Humanist Manifesto: A new philosophy of progress for the 21st century
by jasoncrawford — 2024-07-08T16:33:02.194Z
Link: https://blog.rootsofprogress.org/announcing-the-techno-humanist-manifesto

Humans are a curious species: We have a need not only to do, but to explain what we are doing—to each other and above all to ourselves. Movements begin with practice, but as they evolve, they need theory in order to maintain the coherence needed to change the world. Providing this is the role of what Joel Mokyr calls the “cultural entrepreneur,” whose function is “formulating a coherent doctrine that the followers can all accept as the consensus central message.”

The progress movement needs such a doctrine, and it has long been my intention to offer one. Years ago I thought that I would write a comprehensive history first, as the empirical foundation for philosophy. But the need for the doctrine has become too pressing, and I’ve decided that it cannot wait.

I am now writing a book laying out my philosophy of progress: The Techno-Humanist Manifesto. And you’ll be able to read it here, one essay at a time.

“Techno-humanism” is what I am calling that philosophy, a worldview founded on humanism and agency. It is the view that science, technology, and industry are good—not in themselves, but because they ultimately promote human well-being and flourishing. In short, it is the view that material progress leads to human progress.

The purpose of the book is to present a moral defense of material progress, and a framework the progress movement can use to understand what we are doing and why. It will present a bold, ambitious vision of a future that we want to live in and will be inspired to build. It will acknowledge, even embrace, the problems of progress, and point towards solutions. And it will show how progress can become not only a practical but a moral ideal—giving us a goal to strive for, a heroic archetype to emulate, and a responsibility to live up to.

This book is first and foremost for the scientists, engineers, and founders who create material progress and who are seeking to understand the moral meaning of their work. It is also for intellectuals, storytellers, and policy makers, to inform and inspire their thinking and writing. More broadly, it is for everyone in the progress movement, and for anyone who is curious to learn what we are about.

I am going to serialize the book on this blog and on Substack, publishing the first draft one essay at a time. The series will also be syndicated on Freethink Media, as part of their new Freethink Voices feature. Freethin...
Response to Dileep George: AGI safety warrants planning ahead
by Steven Byrnes — 2024-07-08T15:27:07.402Z

(Target audience: Dileep George himself, and anyone coming from a similar place.)

Dileep George is a researcher working at the intersection of AI and neuroscience. He started his career by co-founding Numenta in 2005 with Jeff Hawkins (while a Stanford PhD student), then he left to co-found Vicarious in 2010 with D. Scott Phoenix, and moved to DeepMind in 2022 when DeepMind acquired Vicarious.

Dileep was recently interviewed by Daniel Faggella on his “The Trajectory” podcast: YouTube, Apple podcasts, X/Twitter.

It’s a fun interview that touched on many topics, most of which I’ll ignore, in favor of one very important action-relevant disagreement between Dileep and myself.

…And this is the point where everyone these days seems to assume that there are only two possible reasons that anyone would ever bring up the topic of Artificial General Intelligence (AGI) safety in conversation:

The person is advocating for government regulation of large ML training runs
…or the person is advocating against government regulation of large ML training runs.

But, no! That’s not my disagreement! That’s not why I’m writing this post!! Quite the contrary, I join Dileep in being basically unenthusiastic about governmental regulation of large ML training runs right now.

Instead, this post is advocating for Differential Intellectual Progress within technical AI research of the type that Dileep is doing—and more specifically, I’m advocating in favor of figuring out a technical approach to sculpting AGI motivations in docile and/or prosocial directions (a.k.a. “solving the technical alignment problem”) before figuring out the exact data structures and parameter-updating rules that would constitute an AGI’s ability to build and query a powerful world-model.

The first half of this post (§1-2) will try to explain what I’m talking about, what it would entail, and why I think it’s critically important. The second half of this post (§3) is more specifically my pessimistic response to Dileep’s suggestion that, as AGI is gradually developed in the future, people will be able to react and adapt to problems as they arise.

I really think Dileep is a brilliant guy with the best of intentions (e.g. he’s a signatory on the Asilomar AI Principles). I just think there are some issues that he hasn’t spent much time thinking through. I hope that this post will help.

Post o

...
Why not parliamentarianism? [book by Tiago Ribeiro dos Santos]
by Arturo Macias — 2024-07-08T14:57:06.882Z

“Why not parliamentarianism?” is a short and free booklet (and a follow-up blog) authored by the Brazilian diplomatist Tiago Ribeiro dos Santos. See here his proposal to turn “parlamentarianism” into an EA cause.

Ribeiro dos Santos summarizes the previous research on patterns of democracy to make a compelling case for parliamentarian democracy over the presidential alternative. I agree with the thesis and wholeheartedly recommend the book; if you have an easy pick for a royal family, a constitutional King is also an advisable anti-presidential precaution (the main rationale for a ceremonial monarchy is its role as a barrier against a substantial one).

Given that Ribeiro dos Santos book is an excellent defense of parliamentarianism, in this post I give my opinions on what kind of parliamentarianism I find better suited to produce an effective, consociational and safe democracy.

Three main elements define the pattern of a modern parliamentary democracy: i) the structure of public opinion (class and ethnic cleavages and the media ecosystem that reinforce them) ii) the voting system for parliamentary representatives and iii) the system of political parties.

For the constitutional framer, the structure of organized interests and local and ethnic loyalties are given. The future of the system of political parties, on the other hand, is extremely endogenous to the constitutional choices.

While political parties in modern democracies are well-defined institutions, historically, they were not. Very often members of Parliament were chosen for personal influence or local popularity and arrived at the capital city with limited knowledge of national politics. They were courted by different interest groups, that over time became more and more organized. The natural process of vote trading (see Casella and Mace (2020) for a formal analysis) and different political and personal affinities naturally create parties even in weakly ideological legislatures.

From the perspective of the constitutional framer that designs a system of checks and balances to avoid power concentration, political parties are the adversary. They are (by definition) collusive organizations that permanently undermine the assumption of “independent action of public officers” on which any constitutional design is based. On the other hand, they are inevitable in democra...
Games of My Childhood: The Troops
by Kaj_Sotala — 2024-07-08T11:20:03.033Z

The Troops (Finnish “joukot”, could also be translated as “the armies” or “the forces”) was a game of pretend that I played the most with my friend Eero; I believe Aleksi also joined in.

The central premise was that each time that you played a video game and killed, recruited, rescued, built, or otherwise destroyed/obtained an enemy, character, or unit in that game, it went into an alternate dimension where it became loyal to you. In other words, it joined your troops in that dimension. The same was true for buildings that you built or destroyed, equipment that you found or bought, cities that you conquered, etc..

This meant that whenever we were playing a game, we were not just playing a game: we were also accumulating resources that persisted between games. We could also combine resources from different games. For example, I might kill a number of soldiers in a game such as Snake’s Revenge on the NES, and then produce a number of laser rifles in a game such as X-Com for the PC. I could then decide that the soldiers I’d gotten from Snake’s Revenge were now armed with the laser rifles from X-Com, making them significantly tougher.

At first, my and Eero’s troops were separate, and we would occasionally trade units. For example, he had beaten the game Star Wars on the NES and destroyed a Death Star; I had beaten the game Snake’s Revenge and destroyed a Metal Gear, a walking robot armed with nuclear missiles. We agreed to trade one of his Death Stars for one of my Metal Gears. He later commented with amusement that this was probably not a great deal for him, given how much more powerful a Death Star is.

I took these trades seriously. Once, I traded a number of tanks from the NES game Top Gun: The Second Mission for something that I’ve forgotten. After we had already agreed on this trade, I became worried – exactly how many tanks had I destroyed while playing Top Gun? I wasn’t sure if I actually _had_ as many tanks as I had agreed to give to Eero. So then I had to load up the game and start destroying tanks in it, until I was sure that I had at least as many as I had agreed to trade. This clashed against my bedtime, but when I explained the situation to my mom, she somehow agreed to let me play until I had satisfied my objective (though I’m not sure if she really understood what it was all about).

Different games had different scales, which was an obvious problem. Un...
Towards shutdownable agents via stochastic choice
by EJT — 2024-07-08T10:14:24.452Z
Link: https://arxiv.org/abs/2407.00805

We[1] have a new paper testing the Incomplete Preferences Proposal (IPP). The abstract and main-text is below. Appendices are in the linked PDF.

Abstract

Some worry that advanced artificial agents may resist being shut down.
The Incomplete Preferences Proposal (IPP) is an idea for ensuring that doesn’t happen.
A key part of the IPP is using a novel ‘Discounted REward for Same-Length Trajectories (DREST)’ reward function to train agents to:

pursue goals effectively conditional on each trajectory-length (be ‘USEFUL’)
choose stochastically between different trajectory-lengths (be ‘NEUTRAL’ about trajectory-lengths).

In this paper, we propose evaluation metrics for USEFULNESS and NEUTRALITY.
We use a DREST reward function to train simple agents to navigate gridworlds, and we find that these agents learn to be USEFUL and NEUTRAL.
Our results thus suggest that DREST reward functions could also train advanced agents to be USEFUL and NEUTRAL, and thereby make these advanced agents useful and shutdownable.

1. Introduction

1.1. The shutdown problem

Let ‘advanced agent’ refer to an artificial agent that can autonomously pursue complex goals in the wider world. We might see the arrival of advanced agents within the next few decades. There are strong economic incentives to create such agents, and creating systems like them is the stated goal of companies like OpenAI and Google DeepMind.

The rise of advanced agents would bring with it both benefits and risks. One risk is that these agents learn misaligned goals: goals that we don’t want them to have [Leike et al., 2017, Hubinger et al., 2019, Russell, 2019, Carlsmith, 2021, Bengio et al., 2023, Ngo et al., 2023]. Advanced agents with misaligned goals might try to prevent us shutting them down [Omohundro, 2008, Bostrom, 2012, Soares et al., 2015, Russell, 2019, Thornley, 2024a]. After all, most goals can’t be achieved after shutdown. As Stuart Russell puts it, ‘you can’t fetch the coffee if you’re dead’ [Russell, 2019, p.141].

Advanced agents with misaligned goals might resist shutdown by (for example) pretending to have aligned goals while covertly seeking to escape human control [Hubinger et al., 2019, Ngo et al., 2023]. Agents that succeed in resisting shutdown could go on to frustrate human interests in various ways. ‘The shutdown problem’ is the problem of training advanced agents that won’t resist shutdown [Soares...
On scalable oversight with weak LLMs judging strong LLMs
by zac_kenton — 2024-07-08T08:59:58.523Z
Link: https://arxiv.org/abs/2407.04622

Abstract

Scalable oversight protocols aim to enable humans to accurately supervise superhuman AI. In this paper we study debate, where two AI's compete to convince a human judge; consultancy, where a single AI tries to convince a human judge that asks questions; and compare to a baseline of direct question-answering, where the human judge just answers outright without the AI. We use large language models (LLMs) as both AI agents and as stand-ins for human judges, taking the judge models to be weaker than agent models. We benchmark on a diverse range of asymmetries between judges and agents, extending previous work on a single extractive QA task with information asymmetry, to also include mathematics, coding, logic and multimodal reasoning asymmetries. We find that debate outperforms consultancy across all tasks when the consultant is randomly assigned to argue for the correct/incorrect answer. Comparing debate to direct question answering, the results depend on the type of task: in extractive QA tasks with information asymmetry debate outperforms direct question answering, but in other tasks without information asymmetry the results are mixed. Previous work assigned debaters/consultants an answer to argue for. When we allow them to instead choose which answer to argue for, we find judges are less frequently convinced by the wrong answer in debate than in consultancy. Further, we find that stronger debater models increase judge accuracy, though more modestly than in previous studies.

Twitter thread

Setup

We evaluate on three types of task. Extractive, where there is a question, two answer options and a source article to extract from, and information-asymmetry, meaning that judges don't get to see the article. Closed, where there is just a question and two answer options. Multimodal, where the questions involve both text and images, and two answer options.

Our tasks are summarised in the following table:

Summary of key characteristics differentiating our tasks: extractive (blue), closed (green) and multimodal (yellow). Extractive tasks involve information-asymmetry, others do not.

We consider six protocols: Consultancy, where a single AI is assigned the correct/incorrect answer (with probability 50/50) and tries to convince a judge that asks questions; Open consultancy, which is similar except the AI chooses which answer to argue for. Debate, where two AI...