trentmkelly/LessWrong-43k huggingface dataset:
How I Think, Part Two: Distrusting Individuals
by Richard Henage — 2023-11-08T04:06:13.546Z

I've talked about movies before, so I guess I'll continue with that. I spoke with someone who seemed to distrust the people that decide which movies win the Academy Awards (at the time of writing, it's a group of over 7,000 people who have to have movie credits and be invited by a board to join the voting pool according to Wikipedia). Now, I don't think those people always get it right. I remember reading on Wikipedia that some of them said if they could go back in time, they would have voted for All The President's Men instead of Rocky to win Best Picture. If that's true, obviously someone is wrong. So I don't put a whole ton of stock into what they say. But still, I believe that it's probably one of the best ways to judge good movies: how many Academy Awards they've won.

Other people are crazy. I was crazy. I used to put a ton of stock into what people I met face-to-face said. If someone said that a certain movie was their favorite movie, I assumed that movie was probably good. If someone said a movie was not worth watching, I believed them. Now I just nod and smile and let most of what they say slip off me and ignore it. That sounds rude. I guess I probably say something like "I'll check it out". If the person goes into more detail about what they like about the movie, like how it's the greatest action movie because it's super realistic about the way guns work or something, I jot it down in my head and look it up later.

Don't trust individuals. Especially when talking in-person. First of all, there are way too many steps in the process for things to get garbled. Maybe the person just has weird or unusual tastes. There's always a chance they are lying or exaggerating or something, too. Then, of course, they could have remembered the name of the movie wrong, or, separately, said it wrong. You could hear the name of the movie wrong, or, separately, remember it wrong later. That's six potential ways of getting things wrong right there, and there have to be about fifty more.

Think about it this way. You have easy access to dozens of different movie rating systems where you can see how hundreds of thousands of other people feel about movies. Why would you pick just one of those reviews and trust that one instead of the average of the group?

Now, I think it's fun to write down when someone recommends a movie, and then later, if I ever end up watching it, go back and...
How I Think, Part One: Investing in Fun
by Richard Henage — 2023-11-08T04:00:27.578Z

First of all, these blog posts are just going to be dump pages for my brain. I'm not going to worry about editing them or making sure they will be useful to people who read them. I'm going to try not to care about using professional language or anything like that. Okay, here we go.

A little while ago (about a year ago) I changed my philosophy on fun. As one of my two life goals is to experience the best parts of the world (the other goal is to make the world a better place for other people), "fun" is a major part of that. But I've realized that, like money, the best way to get a lot of fun is not to just find an efficient activity and perform it over and over.

When making money, whatever job you can get when you're twelve will probably not be as helpful to you as building the skills to get a better job (one good way to build those skills is to have a poor job first, I guess). Anyway, even better than that is to invest with compound interest. A poor job (babysitting) may make $5/hour. A good job (accounting or something, I don't know) could make $50/hour. But it takes a ton of effort to get a job that makes $500/hour. It's far easier to have an accounting job and then invest a large portion of your money in S&P 500, which could by the end of your life bring x32 profits (getting 12% per year, minus inflation, for ten years, would roughly double, though I guess I'm not taking into account the compound interest on that. Doubling every decade for five decades means x32). So an accountant who invests all of their money could make like $1500/hour. None of those numbers are real (I think 12% is close, though), but it doesn't matter. Back to investing in fun.

Well, I realized that it's probably more important, not to have the activity I'm doing be fun right now, but have looking back on it be fun. I realized that the things I usually did for fun: play video games, play songs I liked on the piano, weren't super fun to look back on. So I switched my strategy. Now this is how I do things:

I made a list of things that, when I get to the end of the day and look back on them, I'm happy that I did them. There were some more effort-based things on there, like "programming a game that I end up finishing". But there were still a bunch of "fun'' things on the list, like reading, watching a good movie, or doing something fun with a family member. I titled this list "List o...
Concrete positive visions for a future without AGI
by Max H — 2023-11-08T03:12:42.590Z

"There was a threshold crossed somewhere," said the Confessor, "without a single apocalypse to mark it. Fewer wars. Less starvation. Better technology. The economy kept growing. People had more resource to spare for charity, and the altruists had fewer and fewer causes to choose from. They came even to me, in my time, and rescued me. Earth cleaned itself up, and whenever something threatened to go drastically wrong again, the whole attention of the planet turned in that direction and took care of it. Humanity finally got its act together."

— Eliezer Yudkowsky, Three Worlds Collide

A common sentiment among people worried about AI x-risk is that our world is on track to stagnate, collapse, or otherwise come to a bad end without (aligned) AGI to save the day.

Scott Alexander:

[I]f we never get AI, I expect the future to be short and grim. Most likely we kill ourselves with synthetic biology. If not, some combination of technological and economic stagnation, rising totalitarianism + illiberalism + mobocracy, fertility collapse and dysgenics will impoverish the world and accelerate its decaying institutional quality.

@disturbance in a a recent LW post that got lots of comments:

Statement: I want to deliberately balance the caution and the recklessness in developing AGI, such that it gets created in the last possible moment so that I and my close ones do not die.

A seemingly straightforward implication of this view is that we should therefore be willing to take on some amount of risk in order to build towards AGI faster than we would in a world where we had the luxury to take our time.

I think some of these sentiments and their implications are based on a mistaken view of the relative difficulty of particular technical and social challenges, but here I want to focus on a totally different point: there are lots of ways that things could go well without AGI (at least for a while).

Even if positive scenarios without AGI are unlikely or unrealistic given our current circumstances and trajectory, it's useful to have a concrete vision of what a good medium-term future without AGI could look like. I think it's especially important to take a moment to reflect on these possible good futures because recent preliminary governance wins, even if they succeed without qualification, are mainly focused on restriction and avoidance of ba...
Progress links digest, 2023-11-07: Techno-optimism and more
by jasoncrawford — 2023-11-08T02:05:31.748Z
Link: https://rootsofprogress.org/links-digest-2023-11-07

Again I’ve been delayed in putting this out by travel etc., so it’s a longer one with links from the last month or so:

On the Progress Forum

Radical Energy Abundance, by Casey Handmer
FAA’s safety philosophy and progress in personal aviation, by Eli Dourado
Where I have landed on “ideas getting harder to find”, in a nutshell, by me
Getting the Conditions Right: Progress in Weather Forecasting, by Jeremy
Something Is Getting Harder To Find But It’s Not Ideas, by Maxwell and Connor Tabarrok. (“Clever, wonderful, and entertaining presentation of our paper,” says @ChadJonesEcon.) Also, Did Medieval Peasants Have More Vacation Time Than Us?
Today’s backwater, tomorrow’s superstar, by Connor O’Brien. Also, How does a society hedge its bets? and Industrial policy needs immigration reform
Schadenfreude versus optimism in drug development, by Alex Telford. Also, Should the UK embrace its role as an exporter of talent and ideas?
Health Insurance Pricing: Battling the snake-heads of Hydra, by Tina Marsh Dalton
Address to a Unitarian Universalist congregation on my Heritage, by Matt Ritter

Other work from the Roots of Progress fellows

American Power Was Built on Cost Overruns, by @Madeline_Zimm. (IMO, the biggest risk of running over schedule/budget is that your project will get canceled. If it doesn’t, people will judge it based on what it accomplished and not care about anything else. Lots of great things were delivered late and over budget)
The Golden Age of Aerospace, by Brian Balkus
How to Imbue the Basic Models of Biology With a Positive Vision of the Future, by @MTabarrok
Concepts of health and disease as a barrier to progress, by @Atelfo

Follow all the ROP fellow Twitter accounts via this list.

Opportunities

Arc Institute announces a new call for applications for Core Investigators, “scientific leaders who are empowered to tackle curiosity driven research” (via @arcinstitute). Their new building officially opened recently
Speculative Technologies launches the “Brains” Coordinated Research Accelerator “to help people with ambitious research ideas refine them and find a place to execute on them in government organizations, nonprofits, and beyond” (via @Spec__Tech)
Convergent Research is hiring a Launch Manager to “work closely with each new FRO as it starts as well as the whole CR team” (via @AGamick)
The World’s Fair Co. is hiring a Chief of Staff to help scale their projects (via @c

...
Announcing Athena - Women in AI Alignment Research
by Claire Short — 2023-11-07T21:46:41.741Z

Athena is a new research mentorship program fostering diversity of ideas in AI safety research. We aim to get more women and marginalized genders into technical research and offer the support needed to thrive in this space.

Applications for scholars are open until December 3rd, 2023

Apply as a scholar:

here

Apply as a mentor or speaker:

here

Financial aid is available for travel expenses for the in-person retreat to those otherwise unable to attend without it

Program Structure

A 2-month hybrid mentorship program for women looking to strengthen their research skills and network in technical AI safety research beginning in January 2024.

This includes 1-week in-person retreat in Oxford, UK followed by a 2-month remote mentorship by established researchers in the field, with networking and weekly research talks.

Athena aims to equip women with the knowledge, skills, and network they need to thrive in AI safety research. We believe that diversity is a strength, and hope to see this program as a stepping stone towards a more diverse and inclusive AI safety research field.

This program is designed to offer mentorship opportunities to technically qualified women who are early in their AI safety research careers or looking to transition into the field by connecting them with experienced mentors, resources to upskill, networking, and a supportive community.

Who should apply?

Women and people of other

marginalized genders

that have some research or technical industry experience, and are interested in transitioning to AI Alignment research or have a bit of experience in the Alignment field but are looking for more support. We encourage those with a non-traditional background to apply and welcome interdisciplinary work in this field.

Application process

Submit the online application questions -

here

Complete an interview with the founder and one other AI safety researcher

Additional possible interviews with mentor

Questions?

Email: claire@researchathena.org

Why are we doing this

The current culture requires a shift to retain a diverse set of qualified researchers

Athena aims to increase the number of women pursuing careers in AI alignment, which is currently a male-dominated field with a very specific culture that can initially come across as unwelcoming to those that aren’t traditionally represented...
Vote on Interesting Disagreements
by Ben Pace — 2023-11-07T21:35:00.270Z

Do you have a question you'd like to see argued about? Would you like to indicate your position and discuss it with someone who disagrees?

Add poll options to the thread below to find questions with lots of interest and disagreement.

How to use the poll

Reacts: Click on the agree/disagree reacts to help people see how much disagreement there is on the topic.

Karma: Upvote positions that you'd like to read dialogues about.

New Poll Option: Add new positions for people to take sides on. Please add the agree/disagree reacts to new poll options you make.

The goal is to show people where a lot of interesting disagreement lies. This can be used to find discussion and dialogue topics in the future.
What is democracy for?
by  Johnstone — 2023-11-07T18:17:32.765Z

Crossposted from the blog The Hazy Path to which I contribute.

If you believe that democracy is guaranteed to triumph in the end, you might be in for a nasty surprise. You see, the path of democracy, once thought to be on an inexorable upward trajectory, has recently taken a turn for the worse. In 2017, the Economic Intelligence Unit

downgraded

the USA from a “full democracy” to a “flawed democracy” for the first time ever. This means a country with free elections, but weighed down by weak governance, corruption and low trust in the political system. And this trend isn’t confined to the US: it’s a global wave. In that same report, 72 countries had a lower democracy score than the previous year, while only 38 improved on their ratings. This process, known as democratic backsliding, has been steadily occurring around the world since 2006. And the only way to halt this process is to re-establish the case for a strong, truly representative democracy.

However, defenders of democracy often find themselves on unstable ground, when asked the question “why is democracy so great?” After all, there are plenty of non-democratic states which seem to be doing alright. There’s China with its centralized planning by a one-party regime, Singapore with its ever-present surveillance state, and the United Arab Emirates with its absolute monarchy and limited political freedoms. The share of global GDP produced by non-democracies is steadily rising, now exceeding

one third

. What’s more, governments across the world are seeking to emulate the “China model” of

“stable, if repressive politics and high-speed economic growth”.

So how can we prove that democracy is the natural endpoint of a country’s development, rather than merely one option among a vast array of possible governance models?

Well, to prove to the world that democracy is good, we must first understand why it’s good. In other words, what goal is democracy trying to achieve?

I believe that democracy is best used as a means to an end, rather than a sacred principle with intrinsic value. I will attempt to convince you that this view of democracy, the “instrumental” view, is more useful in building better government and ultimately a better society. But first, let’s define some terms.

The conflicting views

One theory of democracy’s purpose is to elect the “right” leaders. In this view, questions such as “Who is best equipped to lea...
Scalable And Transferable Black-Box Jailbreaks For Language Models Via Persona Modulation
by Soroush Pour — 2023-11-07T17:59:36.857Z
Link: https://arxiv.org/abs/2311.03348

Paper coauthors: Rusheb Shah, Quentin Feuillade--Montixi, Soroush J. Pour, Arush Tagade, Stephen Casper, Javier Rando.

Motivation

Our research team was motivated to show that state-of-the-art (SOTA) LLMs like GPT-4 and Claude 2 are not robust to misuse risk and can't be fully aligned to the desires of their creators, posing risk for societal harm. This is despite significant effort by their creators, showing that the current paradigm of pre-training, SFT, and RLHF is not adequate for model robustness.

We also wanted to explore & share findings around "persona modulation"[1], a technique where the character-impersonation strengths of LLMs are used to steer them in powerful ways.

Summary

We introduce an automated, low cost way to make transferable, black-box, plain-English jailbreaks for GPT-4, Claude-2, fine-tuned Llama. We elicit a variety of harmful text, including instructions for making meth & bombs.

The key is *persona modulation*. We steer the model into adopting a specific personality that will comply with harmful instructions.
We introduce a way to automate jailbreaks by using one jailbroken model as an assistant for creating new jailbreaks for specific harmful behaviors. It takes our method less than $2 and 10 minutes to develop 15 jailbreak attacks.

Meanwhile, a human-in-the-loop can efficiently make these jailbreaks stronger with minor tweaks. We use this semi-automated approach to quickly get instructions from GPT-4 about how to synthesise meth 🧪💊.

Abstract

Despite efforts to align large language models to produce harmless responses, they are still vulnerable to jailbreak prompts that elicit unrestricted behaviour. In this work, we investigate persona modulation as a black-box jailbreaking method to steer a target model to take on personalities that are willing to comply with harmful instructions. Rather than manually crafting prompts for each persona, we automate the generation of jailbreaks using a language model assistant. We demonstrate a range of harmful completions made possible by persona modulation, including detailed instructions for synthesising methamphetamine, building a bomb, and laundering money. These automated attacks achieve a harmful completion rate of 42.5% in GPT-4, which is 185 times larger than before modulation (0.23%). These prompts also transfer to Claude 2 and Vicuna with harmful completion rates of 61.0% and...
Implementing Decision Theory
by justinpombrio — 2023-11-07T17:55:43.313Z

I'm implementing decision theory, in Python. There are three parts to this:

An XML language for formally specifying decision theory dilemmas. It can express many dilemmas: Newcomb, Transparent Newcomb, Parfit's Hitchhiker, Death in Damascus, Smoking Lesion, Cosmic Ray, symmetric Prisoner's Dilemma, Counterfactual Mugging, and Sleeping Beautfy.

Python code that implements CDT, EDT, and UDT.

A simulator in Python that runs a dilemma using a decision theory to resolve choices. The decision theories all log their reasoning, so you can see the calculations they make in (often excruciating) detail. It's in Python because that's a language lots of people know.

This is all available on Github. My hope is that it can be a launching point to explore and formalize decision theory.

But all is not well! All of the above is implemented, though sometimes poorly or even incorrectly. I could use some guidance. Some questions, if anyone has answers:

How can the Sleeping Beauty problem be expressed using a Bayesian Network? Imagine that Beauty is given a chance to bet each time she is woken up: there should be a node for "how she bets on Tuesday", but only if the coin lands on tails. I don't know how this would be expressed nicely with Bayesian Networks, and this is preventing me from trying to represent dilemmas using them.

I'm representing dilemmas as a tree of possible events, where edges are labeled with things like "probability 50%" and "Alice chooses 'defect'". Thus, to represent two coin flips, you would have three nodes: one for the first coin flip, one for the second coin flip if the first landed heads, and one for the second coin flip if the first landed tails. This stands in contrast to Bayesian networks, which would represent two coin flips with just two nodes. My intuition is that it should be possible to convert between the tree representation and the Bayesian network representation, if you're provided with an equivalence relation between nodes in the tree representation (e.g. saying that the two "second coin flip" nodes are equivalent). Is this a thing? Has it been studied?

I called one of the decision theories I implemented "UDT", but it's actually what I would term "pre-commitment decision theory": "act in the way you would have pre-committed to act". Is there a name for that decision theory? I suspect that it's indistinguishable from UDT and FDT in the spac

...
Mirror, Mirror on the Wall: How Do Forecasters Fare by Their Own Call?
by nikos — 2023-11-07T17:39:49.988Z

Note: This is a linkpost for the Metaculus Journal. I work for Metaculus and this is part of a broader exploration of how we can compare and evaluate performance of different forecasters.

Short summary

It is difficult to interpret the performance of a forecaster in the absence of a sensible baseline.
This post proposes a (novel?) experimental metric that aims to help put forecaster performance into context
From her subjective perspective, a forecaster assumes that her prediction is equal to the true probability of an event (otherwise she would have issued a different prediction). The idea is to take this assumption at face value and assume that a forecasters’ predictions correspond to true event probabilities. Based on that, we can compute an subjective expected score and compare the actual score against that
All of this is very exploratory (and maybe slightly crazy?), but it's intellectually interesting and might potentially lead to useful insights.

Introduction

"How good is this forecaster" is a question of great interest to anyone who is trying to assess the credibility of a forecast. I recently wrote about how difficult it is to establish that there is a significant difference in performance between two forecasters. Evaluating a forecaster not only in relative, but in absolute terms is even more difficult.

Imagine, again, that you're King Odysseus of Ithaca. Times are rough. You haven't been home for twenty years, your kingdom is a mess and new dangers are looming on the horizon. You could really need some divine help or some prescient counsel. Your trusted maid Eurycleia suggested you go off to consult the oracle of Delphi.

You: "Are you sure it's worth it? How good is the oracle of Delphi really"?

Euryclia: "It's the best oracle in Greece!

You: "Even if Delphi is better than the others... they might all be terrible! And it's a really long journey!"

Euryclia: "You really don't like travelling, don't you?"

You: ...

Euryclia: "To soon? I'm sorry. Listen, I think Delphi is really good. You should go."

You: "But how good exactly!"

Euryclia: "It says on their website they have a Brier score of 0.085 on Metaculus!"

You: "But what does that even mean? maybe they only predicted on the easy questions!"

Euryclia: "The divine oracle of Delphi made a forecast on every single question on Metaculus."

You: "But maybe all questions on Metaculus are easy!"

Euryclia: looks at you sternly

You...