trentmkelly/LessWrong-43k huggingface dataset:
Against a General Factor of Doom
by Jeffrey Heninger — 2022-11-23T16:50:04.229Z

Jeffrey Heninger, 22 November 2022

I was recently reading the results of a survey asking climate experts about their opinions on geoengineering. The results surprised me: “We find that respondents who expect severe global climate change damages and who have little confidence in current mitigation efforts are more opposed to geoengineering than respondents who are less pessimistic about global damages and mitigation efforts.”[1] This seems backwards. Shouldn’t people who think that climate change will be bad and that our current efforts are insufficient be more willing to discuss and research other strategies, including intentionally cooling the planet?

I do not know what they are thinking, but I can make a guess that would explain the result: people are responding using a ‘general factor of doom’ instead of considering the questions independently. Each climate expert has a p(Doom) for climate change, or perhaps a more vague feeling of doominess. Their stated beliefs on specific questions are mostly just expressions of their p(Doom).

If my guess is correct, then people first decide how doomy climate change is, and then they use this general factor of doom to answer the questions about severity, mitigation efforts, and geoengineering. I don’t know how people establish their doominess: it might be as a result of thinking about one specific question, or it might be based on whether they are more optimistic or pessimistic overall, or it might be something else. Once they have a general factor of doom, it determines how they respond to specific questions they subsequently encounter. I think that people should instead decide their answers to specific questions independently, combine them to form multiple plausible future pathways, and then use these to determine p(Doom). Using a model with more details is more difficult than using a general factor of doom, so it would not be surprising if few people did it.

To distinguish between these two possibilities, we could ask people a collection of specific questions that are all doom-related, but are not obviously connected to each other. For example:

How much would the Asian monsoon weaken with 1°C of warming?
How many people would be displaced by a 50 cm rise in sea levels?
How much carbon dioxide will the US emit in 2040?
How would vegetation growth be different if 2% of incoming sunlight were scattered by stratospheric ae

...
Injecting some numbers into the AGI debate - by Boaz Barak
by Jsevillamol — 2022-11-23T16:10:33.671Z
Link: https://windowsontheory.org/2022/06/27/injecting-some-numbers-into-the-agi-debate/

I found this article useful context for Boaz Barak's more recent article about the probability of AI Doom.

In this article, the author (through very rough BOTECs):

Estimates the number of parameters needed for AGI (tentative conclusion: x10-100 more than GPT-3)
Estimates the cost of training AGI of that scale (tentative conclusion: $50B - $5T)
Estimates the cost of inference for AGI of that scale (tentative conclusion: $10/hour - $100/hour)
Notes some challenges that will need to be addressed to reach AGI
In the end there is a reflection on consciousness and AI risk

Here are some excerpts:

Bottom line. Overall, it seems that [we would need to scale model size by] at least 10-100 [to reach human-level AGI], though this is an extremely rough estimate. Also, while an [10-100] Trillion model might be the “core” of a system replacing a human worker, it will not be the whole of it, and we are likely to need new ideas beyond scale for these other components. In particular, a model designed for back-and-forth interaction is unlikely to simply use an auto-regressive language model as a black box, not even with chain-of-thought reasoning.

This estimate is informed by 1) comparing the context length of current models and the context length needed for processing one year of continuous logging (result: x100), 2) evidence of scaling from the BIG BENCH paper (>x10 for human-level performance), 3) a comparison between current parameter counts of the brain and an estimate of the "number of parameters" in the human brain (the comparison is x100-x1000).

It seems unclear how he arrives from these estimates to the x10-x100 estimate. There is also a digression about adaptability and how difficult it will be to go from 95% reliability to 99% reliability.

There are differing estimates on how much ~100B parameter GPT3 model cost to train, but they range in the $5M-$20M, let’s say $5M in pure training costs for simplicity. (This would correspond to a cost of 5M$/500B = 10-5 dollars per inference, which matches roughly with estimates below.) An X Trillion model (which, like Chinchilla, but unlike PaLM, would be fully trained to max out its advantages) might cost a factor of 100X2 more. For X=10, this would be a cost of $50B. For X=100, this would be 5 Trillion dollars!

Clearly, finding a way to train N-sized models on N tokens using less than O(N2) steps (e.g., O(

...
Notes on an Experiment with Markets
by Jeffrey Heninger — 2022-11-23T16:10:05.573Z

Jeffrey Heninger, 22 November 2022

AI Impacts is a research group with seven employees. From Oct 31 – Nov 3, we had a work retreat. We decided to try using Manifold Markets to help us plan social events in the evenings. Here are some notes from this experiment.

Structure of the Experiment

Katja created a group on Manifold Markets for AI Impacts, and an initial collection of markets. Anyone could add a market to this group, and five of us created at least one market. Each of us would rate each evening from 0 to 10 on an anonymous Google form. Most of the questions in the group were about the results of the form, often conditional on what activity we would do that evening. For example: “On the first day that at least 4 people begin a game of One Night Werewolf at the AI Impacts retreat, will the average evening rating be above 8?” The markets would resolve at some point the next morning after we had submitted our forms and Katja calculated the average evening rating.

Disagreements about the Experiment

There were several disagreements about how the experiment was supposed to be run.

Initially, the role of the evening rating form was unclear. Was it asking for your honest assessment of the evening or was it part of the game? “What number would you like to assign to the evening?” is different from “How good was your evening honestly?” We decided that we wanted honest responses. Even then, the numbers were ambiguous. What constitutes a 7 evening vs. a 9 evening? Different people’s baselines result in different scores, which can alter the average. After the first evening, we had a better estimate of the baseline. Many of the markets had used an average score of above 8, which was higher than the baseline. This made the markets feel less useful, instead shifting the predictions to lower probabilities while remaining useful. It’s not clear why this happened, but it might have been because we didn’t want to bet against ourselves having a good time or because the tail of an unknown distribution is harder to predict than the middle of the distribution.

One morning, Katja told us the average score before resolving her markets. Zach used this information to bet on these markets. Rick thought that it was unclear whether this should be allowed, because not everyone was there and because the previous discussion about honest ratings suggested that we should ask before doing so...
Announcing AI safety Mentors and Mentees
by Marius Hobbhahn — 2022-11-23T15:21:12.636Z

Executive summary

AI safety Mentors and Mentees is a program that matches mentors with mentees to accelerate their AI safety journey. Our aim is to get more people into full-time AI safety roles faster. We expect nearly all mentees to do this next to their current occupation, the program is not designed to be full-time.
The program is designed to be very flexible and lightweight, e.g. as long as the mentor and mentee agree on the terms, they can do pretty much whatever makes the most sense for them. The organizational overhead for each person is less than 10 minutes for the entirety of the program.
We’re trying to fill some of the niches that other programs can’t provide. We don’t expect the program to ever scale up to huge numbers. Whenever possible, we recommend doing cohort-based programs such as

AGI safety fundamentals

,

SERI MATS

,

AI safety scholars

, the

AI safety camp

, and the like.
We are looking for more mentors and mentees. You can fill out our interest forms here (

mentors

,

mentees

). Feel free to forward these forms to people you think might be a good fit.
The requirement for being a mentor is to have a decent overview of what’s happening in AI safety and be able to point people toward relevant articles, organizations and individual researchers. All of our mentors so far are either already working full-time in AI safety or very likely to be in the near future. We think early-stage professionals in AI safety (Ph.D. students, industry researchers/engineers, or independent contributors) are ideal candidates for mentors. Note, being able to consistently spare 30 minutes per week is sufficient to mentor one mentee, so you're time commitment doesn't have to be very big.
The requirement for being a mentee is to intend to work full-time in AI safety in the near future (e.g. ~2 years). There are no strict formal requirements but we strongly prefer candidates who could directly join the workforce or get independent funding to work on AI safety soon after finishing the M&M program. We expect to point many applicants to the

AGI safety fundamentals program

or other programs if there is no specific reason for 1-on-1 mentorship.
We have funds for people (both mentors and mentees) who really need them--so if you want to participate but don’t have the funds, we will be able to support you. By default, we suggest that

...
Ex nihilo
by Hopkins Stanley — 2022-11-23T14:38:29.272Z

As a quick announcement, Dr. Martin Burckhardt has started a new blog on Substack called Ex nihilo. Dr. Burckhardt is a German cultural philosopher well-known in Germany for his out-of-the-box thinking on the Question of the Machine, how it relates to Artificial Intelligence, and the cultural implications of our movement into a Digital Economy. The blog is bi-lingually English and German and includes audio chapters of his most recent book, Philosophy of the Machine, with a new chapter to be released every two weeks. The blog will also have conversations with other like-minded authors and thinkers.

For anyone interested, the URL is https://martinburckhardt.substack.com/s/ex-nihilo-english.

Thanks for taking the time to read the announcement — and of course, we’d be most pleased if you find any interest.
Human-level Diplomacy was my fire alarm
by Lao Mein — 2022-11-23T10:05:36.127Z

Epistemic status: AI is my hobby, not my job.

Meta just released their new Diplomacy AI, Cicero. Sure, it was several different modules kludged together in a somewhat clever way, with no fundamentally new ideas or massive scaling, but it was damn impressive.

What surprises me most about Cicero is its ability to go all the way from data input to natural language output with relatively high reliability. The model makes a battle plan, uses natural language to interface with humans, then uses the information gained from said interface to alter its plan. And it does this consistently enough to outmatch the average human Diplomacy player. Maybe it doesn't represent anything truly new in terms of architecture or scaling. But that's exactly why I'm so impressed.

It doesn't have the "human spark". But the lesson that we've been learning over the past few decades is that very impressive tasks, from human level Chess to Go to art, can be accomplished without said spark. I'm starting to suspect that human-level AGI is entirely possible with something like Cicero. If systems like Cicero can remain coherent over longer time scales, we could be seeing the automation of many complex tasks, including the holy grail that is AI research. Writing scientific papers? I'm already starting to use automated hypothesis discovery. Let's break down how the median academic paper actually gets written.

Somewhere in the universe, an academic feels publication pressure. They start looking at their connections, resources, and potential funding.
They start bouncing ideas and try to figure out how to write the highest impact paper they can with the lowest use of resources. Analysis of other peoples' data is very tempting at this point. AI can probably be used to search through public data for underutilized but useful datasets.
Funding applications are written. This part can be automated.
They start pulling threads, using their specialty knowledge to come up with the best novel hypotheses they can, and picks the one with the lowest p-value. No multiplicity testing, of course. Automated hypothesis discovery can help greatly here. A language model can probably also be used to estimate future citations for each hypothesis-paper.
Hypothesis + data -> set of core ideas that is then expanded into an abstract. I think current transformer models can already do this.
The paper is actually w

...
Sets of objectives for a multi-objective RL agent to optimize
by Ben Smith — 2022-11-23T06:49:45.236Z

Background: A multi-objective decision-making AI

Previously I've proposed

balancing multiple objectives via multi-objective RL

as a method to achieve AI Alignment. If we want an AI to achieve goals including maximizing human preferences, or human values, but also maximizing corrigibility, and interpretability, and so on--perhaps the key is to simply build a system with a goal to maximize all those things. With Peter Vamplew and other co-authors, I have

previously argued multi-objective RL is superior for human-aligned decision-making relative to scalar [single-objective] RL

.

This post describes, if one was to try and implement a multi-objective reinforcement learning agent that optimized multiple objectives, what those objectives might look like, concretely. I've attempted to describe some specific problems and solutions that each set of objectives might have.

I’ve included at the end of this article an example of how a multi-objective RL agent might balance its objectives.

How would this apply to AI, specifically?

So right now I've been deep into technical details. We still have issues to solve there. It isn't clear we can solve them.

But let's say you've got a multi-objective decision-making in your agent.

What exactly are the objectives you might be balancing between?

Balancing between different human values. There are subsets of this--including

balancing human life values, like autonomy and happiness, or
balancing moral frameworks, like deontology or utilitarianism

Balancing between different ways of measuring human preferences, like revealed and expressed preferences
Balancing between different qualities we want in an AGI system, e.g., corrigibility, interruptibility, agency, etc

So, let's take each of these in turn: what are the problems with each? Also, on the flipside, what is achieved for each of these?

1. Balancing between different human values

We might set up a system that balances between different explicit human values. We might start with, e.g.,

Schwartz values

. A multi-objective decision-making system could balance the maximization of, for a set of human targets or for all human targets, a set of Schwartz values that seem particularly relevant; for instance, self-direction, hedonism, and security.

What problems exist with this specific approach?

Setting up a set of human values as terminal values fo...
Simulators, constraints, and goal agnosticism: porbynotes vol. 1
by porby — 2022-11-23T04:22:25.748Z

This is a part of a maybe-series where I braindump safety notes while waiting on training runs to complete. It's mostly talking to myself, but talking to myself in public seems somewhat more productive. The content of this post is not guaranteed to be novel, interesting, or correct, though I do try for at least one of those. Many parts involve handwaving where more rigorous reasoning and proofs would be nice.

Feel free to skip sections; while there is a tenuous thread running through the whole post, most sections can be understood locally.

Can we bound the capability of a model?

Giant black box networks can be highly capable but are hard to interpret and are more likely to contain spookiness. Can we find a way to make them smaller, usefully?

Considering a single forward pass of a fixed network graph and ignoring any information carried between passes (e.g. autoregressive generation, RNN memory, tapes/stacks), the types of computation a network can internally express are bounded. A single token prediction in a GPT-like architecture runs in

constant time

. Algorithms which require more steps than the network can express just don't fit. No training data, fine tuning, or magic optimizer can change that.

Networks seem to show discontinuous improvements in capability akin to 'unlocking' new features with scale. I suspect that this unlock often corresponds to a new algorithm becoming accessible to the network. "Accessible" includes some slop; the search for an algorithmic representation will be affected by the training data, the network's structure, and the optimizer itself. For example, it's possible that a broader training set could result in finding a more concise (simpler) representation that would be accessible to a

smaller network

, or that a post-process could

distill

the original large network's solution into smaller networks, or express the solution in

fewer steps

.

If your goal is to limit the capability expressible within a single forward pass, then knowing the network scale required to find an unwanted capability is valuable.

Networks as parallel computational graphs

For explicit algorithms, strong lower bounds are sometimes available. For example, if you wanted to "train" a network to add 8 floating point numbers together, the fixed function hardware exposed by the network's structure makes it pretty easy:

truly a breakthrough in machine learning

This differs from ...
Rationalist Town Hall: FTX Fallout Edition (RSVP Required)
by Ben Pace — 2022-11-23T01:38:25.516Z

Edit: K, all those who filled out the form should have emails now. No promises that any further form-fills will get emails.

Stated at the top for emphasis: you have to fill out the RSVP form in order for me to email you the links.

On Sunday 27th November at 12 pm PT, I am hosting an online Town Hall on zoom, for rationalists and rationalist-adjacent folks (e.g. EAs) to think through the FTX catastrophe and propagate thoughts, feelings and updates.

Lots of people I know have been shocked by and are still reeling from the news these last 2 weeks. I'm very keen to hear what updates people are making about EA and crypto, and understand others' perspectives.

Some people coming include Zvi Mowshowitz, Oliver Habryka, Anna Salamon, and more.

To get the Zoom and Gather Town links,

fill out the RSVP form

. I will send the links to everyone who fills out the form. The form involves agreeing that the event is off the record to the corporate news media, and all attendees will fill out the form.

What Will The Format Be?

Spontaneous lightning talks. During the event, anyone who wishes to can give a 3-minute talk on a topic of their choosing, followed by 2-mins of Q&A — it can be on something you've already thought about, or it can be a response to or disagreement with someone else's lightning talk.

This is a format I've used before pretty successfully in both big (70 ppl) and small (7 ppl) groups, where we’ve gotten on a roll of people sharing points and also replying to each others' talks, so I have hope that it will succeed online. (This format has also had other names like "Lightning Jazz" and "Propagating Beliefs".)

We will do lightning talks for up to 1.5 hours (depending on how much steam people have in them), hopefully giving lots of people the chance to speak, and after that the main event will be over, and we'll move to Gather Town to have group discussions (and those who are satisfied will go home).

If you wish to, you can submit for a lightning talk ahead of time with this Lightning Talk form.

Who is invited?

As well as Rationalists/LessWrongers, I welcome any people to this event who are or have formerly been part of the EA community, people who have formerly worked for or been very close with FTX or Alameda Research, and people who have worked for or been funded in any way by the FTX Future Fund.

I hereby ask others to respect the Rationalist and EA comm...
Feeling Old: Leaving your 20s in the 2020s
by squidious — 2022-11-22T22:50:01.195Z

A younger friend recently asked why I (and many others my age) make so many comments about being old. After some thought, I realized a big part of this was related to experiencing specific meaningful life stages during the pandemic.

In 2019, I could (and often did) stay up late at parties, survive on very little sleep, and run on my own willpower alone. I didn't usually need to think too hard about any of this in terms of trade offs, and thus had the ability to be way more spontaneous about it all. If a friend invited me out, I could say yes that night, with no planning ahead, and have a good time.

Then lockdown hit, and everything shut down. During this time, I got divorced, turned 30, and made a bunch of other huge life changes. In normal times, I probably would have started to see my stamina decline. I would have noticed at first that my body would hurt more the day after a night out. Then I'd start to notice myself losing energy earlier and earlier. But things didn't play out this way.

Instead, we were isolated, slowly watching culture change, with Gen Z gaining more control over social and pop culture as we learned to live online as they always had. Harry Potter wasn't cool anymore, side parts were cringe, and it became clear how out of touch we were.

When things opened up years later, many of us found ourselves suddenly unable to keep up with our past selves. It feels like going out until 3 one night, and the next night, needing to leave at midnight. Suddenly we were no longer growing, but aging.

So when I call myself old, I'm not saying that my best years are behind me, or that I feel I'm nearing the end of my life. I think what I actually mean (and should be saying) is that I feel emotional whiplash from moving out of young adulthood into regular adulthood. It feels like I was robbed of the transition between the two. I wish we could have peacefully handed the baton off to the next generation, rather than waking up one day and finding it gone.