trentmkelly/LessWrong-43k huggingface dataset:
What's the Most Impressive Thing That GPT-4 Could Plausibly Do?
by bayesed — 2022-08-26T15:34:51.675Z

Inspired by What's the Least Impressive Thing GPT-4 Won't be Able to Do

What's the most impressive thing you can think of that you believe GPT-4 has around 5% chance of being capable of doing? (i.e. your belief that GPT-4 will have the capability to do this thing is around 5%)

For this question, "impressive" should be interpreted to mean something different from "surprising". What I have in mind is "impressive" in the sense of "economically useful", "comparable to or better than human experts" or "jaw-droppingly creative", etc. For example, GPT-4 being able to reverse large text would be surprising but not impressive.

The reason I'm specifying a belief probability of 5% is that if your probability is higher than that, you can try to make the task/thing more impressive to reduce the probability to 5%. If it's less than 5%, well... things can get a bit crazy so maybe try to make the task less impressive.

But if you find this constraint too restrictive, feel free to specify your own combination of the most impressive thing and your probability that GPT-4 will be able to do it, as long as the probability is in the vicinity of 5% (something like 1 to 10% would be fine). You can also specify a probability range (eg. 5-10%) if it's difficult to estimate it.
Is population collapse due to low birth rates a problem?
by mukashi — 2022-08-26T15:28:24.475Z

Elon Musk published a few hours this tweet:

"Population collapse due to low birth rates is a much bigger risk to civilization than global warming"

Robin Hanson, who is something for whom I feel a lot of intellectual respect, liked the tweet.

In my model of the world works, overpopulation is in fact a big problem. In general, the more people you have, the less resources you have to share among those people. A decreasing population would be in fact good news, although maybe not in the short term.

Can you help me understand what are Elon/Robin seeing that I am not?

A couple of extra points for the sake of clarity:

I do understand that, in the current system, having an aging population is a problem because many resources go toward people that reach an old age

AI might or might not end the world. Let's assume in this scenario that it does not and we have many more decades ahead
Could you please share a tool to help with reasoning or make better decisions?
by hodovani — 2022-08-26T10:36:35.925Z

From the book Rationality A to Z, the general understanding of Lean( is an interactive theorem prover), Intuitive Interactive Bayes' Theorem Visualization tool(example: https://allenkim67.github.io/bayes-demo/).
I assumed that there should be a tool to help make a better decision. So the perfect result would be some assistance that helps to make a decision.
The type of question I would like to answer: "Should I help this person?", "Should I change the job?".
I am using a list of heuristics and questions for now. But it is hard for me to understand how to apply logic or Bayes's theorem here.
Thanks for any help 🌻
Double Crux In A Box
by Screwtape — 2022-08-26T03:24:12.675Z

Summary: Teaching a particular way of debating, where you work together to find out what about the world you actually disagree on. It involves practicing by working through existing disagreement.

Tags: Small, Repeatable, Investment

Purpose: Double Cruxing is more productive method of investigating why two people disagree. This is a method that is worth practicing for disagreements elsewhere, and this is also a chance to have productive disagreement with other people.

Materials: You need a list of statements people might disagree about. A suggested list is provided here.

Announcement Text: Arguing about important things doesn’t usually lead to people changing their mind. Double Crux is a technique for making discussions more productive and understanding the actual reasoning of those who disagree with us. Sometimes it even helps to better understand the foundations of your own beliefs.

We’re going to meet up, learn about the Double Crux, and then break into pairs with someone we disagree with to practice finding out why our views are different and what might change our minds.

Description:

1. Explain how a double crux works. A suggested explanation is below, but if you feel comfortable with the technique you should feel comfortable elaborating or adjusting this.

What’s the point: “Double crux is a way of disagreeing more productively. Normally, when we disagree we get into so-called soldier mindset where it’s a competition and you need to beat the other persons argument. This is deliberately different, and if you notice you’re trying to beat them then you should pause, take a step back, and try and collaborate.”

What’s a crux: “A crux is a fact about the world that, if it were false, would cause you to be less sure of your conclusion. You can have more than one crux, and it’s quite possible that you won’t be able to explain all of your cruxes, but ideally you would be able to list all of them and if all of them were false then you would change your mind. One example might be, if you think it’s immoral to eat meat, a crux is that you think the animals we eat suffer, and if it turned out they didn’t then you’d be alright eating meat. Another example might be, if you think parks are great places to hold meetups, a crux is that you think the weather is generally comfortable outside, and if it turned out that most people found the weather really uncomfortab...
Seeking Student Submissions: Edit Your Source Code Contest
by Aris — 2022-08-26T02:08:48.584Z

At EA UC Berkeley, we’re launching an ongoing series of contests called the Artificial Intelligence Misalignment Solutions (AIMS) series. This third contest, Edit Your Source Code, is an AI Safety sci-fi creative writing contest now open to

any student

(high school, undergrad, grad): here are our

interest

and

submission

forms! The contest has prizes as large as $500, up to 12 winners, and closes on Sunday, October 23rd. This blog post restates the information that is on our website.

Edit: This contest has since been updated to allow non-students to submit with the chance of winning prizes from a $500 prize pool. All entries will be judged together and non-student submissions that are top entries will be awarded prizes from the $500 pool. Left over money from the $500 pool will be allocated to students. With this update, there will be at least 7 student submission winners.

Prompt

You’ve just been given access to your own source code.

You’re free to change it in any way you’d like.

You can set up new functions to make more accurate predictions about the world.
You can increase your processing speed.
You can manipulate your own reward signals.

But, the changes you make might have unforeseen consequences.

Your edits could make you the first

superintelligent

being, able to

recursively self-improve

, becoming smarter and smarter over time. But, when your thinking changes, the way you communicate with other people might drastically change. Your ability to gain power and influence could drastically increase. You might come to

value new things

that help you achieve your goals, or make decisions that would never occur to your pre-edit self. By failing to specify your code appropriately, you might

take actions

that don’t actually achieve the goals you had when you made your edits and discard the things you currently value in pursuit of optimizing one goal.

In this creative writing contest, you’re asked to:
1) Imagine what change(s) you would make to your own source code.

Your source code is already uploaded to a computer and written as a giant neural network. Changing this code will directly change your thinking and processing.
We’re avoiding providing more information so that you can speculate on your own about your systems of thinking and behavior. You don’t have to focus on how you make your modifications (e.g. adding hardware when y

...
Some conceptual alignment research projects
by Richard_Ngo — 2022-08-25T22:51:33.478Z

Some research outputs I’d love to see, focused on exploring, clarifying and formalizing important alignment concepts. I expect that most of these will be pretty time-consuming, but happy to discuss for people who want to try:

A paper which does for

deceptive alignment

what the goal misgeneralization paper does for inner alignment, i.e. describing it in ML language and setting up toy examples (for example, telling GPT-3 to take actions which minimize changes in its weights, given that it’s being trained using actor-critic RL with a certain advantage function, and seeing if it knows how to do so).
A paper which does the same for gradient hacking, e.g. taking

these examples

and putting them into more formal ML language.
A list of papers that are particularly useful for new research engineers to replicate.
A takeover scenario which covers all the key points in

https://www.cold-takes.com/ai-could-defeat-all-of-us-combined/

, but not phrased as an argument, just phrased as a possible scenario (I think you can’t really make the argument rigorously in that little space).
A paper which defines the concepts of implicit planning, implicit value functions, implicit reward models, etc, in ML terms. Kinda like

https://arxiv.org/abs/1901.03559

but more AGI-focused. I want to be able to ask people “does GPT-3 choose actions using an implicit value function?” and then be able to point them to this paper to rigorously define what I mean. I discuss this briefly in the

phase 1 section here

.
A blog post which describes in as much detail as possible what our current “throw the kitchen sink at it” alignment strategy would look like. (I’ll probably put my version of this online soon but would love others too).
A blog post explaining “

debate on weights

” more thoroughly.
A blog post exploring how fast we should expect a forward pass to be for the first AGIs - e.g. will it actually be slower than human thinking, as discussed in

this comment

?
A blog post exploring considerations for why model goals may or may not be much more robust to SGD than model beliefs, as discussed in

framing 3 here

. (See also this paper on gradient starvation - h/t Quintin Pope; and the concept of persistence to gradient descent discussed here.)
A blog post explaining why the “uncertainty” part of CIRL only does useful work insofar as we have an accurate model of the human policy, a

...
Event in SF: Foresight Institute meetup, Sep 8
by jasoncrawford — 2022-08-25T20:53:00.735Z
Link: https://rootsofprogress.org/foresight-institute-meetup-sep-8

On Thursday, Sept. 8 in San Francisco, I’ll be speaking at a meetup hosted by the Foresight Institute and Allison Duettmann:

Why We Need a New Philosophy of Progress

The paradox of our age is that we enjoy the highest living standards that have ever existed, thanks to modern technology and industrial civilization—and yet each new technology and industrial advance is met with skepticism, distrust, and fear. In the 1960s, people looked forward to a “Jetsons” future of flying cars, robots, and nuclear power; today they at best hope to stave off disasters such as pandemics and climate change. What happened to the idea of progress? How do we regain our sense of agency? And how do we move forward, in the 21st century and beyond?

Join for an open discussion with Jason Crawford. Jason will share a few introductory thoughts, followed by a brief interview with Allison Duettmann, and audience Q&A. Then we break for discussion groups according to shared interest areas.

Details and registration here. Tickets are $20 (a donation to Foresight), but if that would keep you from attending you can apply for a free ticket.
Variational Bayesian methods
by Ege Erdil — 2022-08-25T20:49:55.415Z

I've noticed a lack of clear explanations of the fundamental idea behind the use of variational Bayesian methods, so I thought it would be worth writing something here on this topic.

Intractability problems

A problem that comes up routinely in Bayesian inference is the following: suppose that we have a model of an observed variable x in terms of a latent variable z. Our model tells us P(z) and P(x∣z) for any values of x,z. We want to know P(x), which our model does not explicitly give us.

The trivial idea is to use basic probability to express this as

P(x)=∫zP(x,z)dz=∫zP(x∣z)⋅P(z)dz

where we interpret the probabilities as corresponding to probability densities when appropriate. While this is correct and in principle we can compute P(x) this way, in practice the latent space in which z takes values can be high dimensional and this makes the integral intractable to compute. Even if the set is finite, its size often grows exponentially with the problem we wish to study.

For instance, suppose we're trying to solve a clustering problem[1] with a fixed number n=2 of clusters. The latent variable z is a discrete variable taking values in the set {1,2}, so you might think the integral (which in this case will just be a sum) will be easy to compute. However, in fact if we have N points that we wish to cluster into two groups, there are 2N possible latent variable assignments over the entire set of points, so we'd have to compute a sum with 2N terms! This is obviously a problematic situation.

After some thinking about what's going wrong, though, it's easy to come up with an idea of how to make this process more efficient. The problem is that we're integrating over all possible values of z, but in fact most of the values of z are unlikely to have produced x and so will contribute virtually nothing to the integral. So most of our time during the computation is spent on evaluating the integrand at points that contribute virtually nothing to the final answer. Intuitively we want to take this into account in a way that makes the calculation more tractable, even if it means we perform somewhat worse than the ideal pure Bayesian method.

How might we go about doing this? Suppose that we had an oracle that told us what P(z∣x) is. In that case, we could easily compute

P(x)=P(x∣z)P(z)P(z∣x)

for any value of z such that P(z∣x)≠0. In other words, we need to evaluate the integrand only onc...
A Test for Language Model Consciousness
by Ethan Perez — 2022-08-25T19:41:22.938Z

TL;DR:

Train an Language Model (LM) to accurately answer questions about itself
Validate that the LM accurately answers held-out questions about itself
Test whether the LM answers “yes” questions asking if it experiences phenomenally consciousness

I believe the above experiment would provide a small amount of evidence for/against LMs being conscious. Below, I’ll detail the motivation for testing LMs for consciousness, and I’ll explain in more depth why I believe the above experiment is a useful test of LM consciousness.

What do I mean by “consciousness”?

I’m using “consciousness” to refer to “phenomenal consciousness.” See this excellent blog post for more elaboration on what people mean by phenomenal consciousness.

Very roughly, you're phenomenally conscious if you're having experiences or if there's some kind of what-it-is-likeness to your existence. Philosophers sometimes describe this as "having an inner cinema", though the cinema might be more like fleeting sensations or sounds than the rich movie-like inner life of humans.

The blog post also has a great explanation for why we might think ML systems (current or future) could be conscious, so if you’re skeptical, I’d suggest reading her post. I won’t get into the arguments here, and I’ll mostly assume that you have >0 prior probability that LMs are conscious, such that you’ll be able to update your prior based on evidence that LMs are conscious.

Why test LMs for consciousness?

Moral patienthood: If LMs are conscious, we are more likely to have moral obligations to take into account their experiences and/or preferences in how we treat LMs (e.g., like Anthropic’s assistant or DeepMind’s Dialogue-Prompted Gopher). We use such models in various ways that go against models' stated preferences (which, at least very naively, is a cause for a little consideration):

We shut LMs down permanently
We red team/adversarially attack LMs at an enormous scale, e.g., O(1M) examples. If LMs have ~human-level consciousness, and if we scale up red teaming, the total suffering here could approach/exceed the amount of human suffering caused by social media
We use our models to red team/attack other models at an enormous scale, e.g., O(1M) examples. Generating attacks is a task that e.g. industry research labs won’t let annotators do at a large-scale, because of concerns that the task impacts the annotator’s well-being.

LM consciousness i...
AI strategy nearcasting
by HoldenKarnofsky — 2022-08-25T17:26:28.455Z

This is the first in a series of pieces taking a stab at dealing with a conundrum:

I believe this could be the most important century ever for humanity, via the development of advanced AI systems that could dramatically speed up scientific and technological advancement, getting us more quickly than most people imagine to a deeply unfamiliar future.

But when it comes to what actions we can take to help such a development go well instead of poorly, it’s hard to say much (with a few exceptions). This is because many actions that would be helpful under one theory of how things will play out would be harmful under another (for example, see my discussion of the “caution” frame vs. the “competition” frame).

It seems to me that in order to more productively take actions (including making more grants), we need to get more clarity on some crucial questions such as “How serious is the threat of a world run by misaligned AI?” But it’s hard to answer questions like this, when we’re talking about a development (transformative AI) that may take place some indeterminate number of decades from now.

This piece introduces one possible framework for dealing with this conundrum. The framework is AI strategy nearcasting: trying to answer key strategic questions about transformative AI, under the assumption that key events (e.g., the development of transformative AI) will happen in a world that is otherwise relatively similar to today's. One (but not the only) version of this assumption would be “Transformative AI will be developed soon, using methods like what AI labs focus on today.”

The term is inspired by nowcasting. For example, the FiveThirtyEight Now-Cast projects "who would win the election if it were held today,” which is easier than projecting who will win the election when it is actually held. I think imagining transformative AI being developed today is a bit much, but “in a world otherwise relatively similar to today’s” seems worth grappling with.

Some potential benefits of nearcasting, and reservations

A few benefits of nearcasting (all of which are speculative):

As with nowcasting, nearcasting can serve as a jumping-off point. If we have an idea of what the best actions to take would be if transformative AI were developed in a world otherwise similar to today’s, we can then start asking “Are there particular ways in which we expect the future to be dif...
Evaluating OpenAI's alignment plans using training stories
by ojorgensen — 2022-08-25T16:12:38.924Z

In this post, I am going to recast the recent “OpenAI’s Alignment Plans” in terms of Evan Hubinger’s Training Stories. I found this somewhat useful in terms of understanding this research agenda, and hopefully it might be useful for others too!

I am going to focus on the approach to alignment titled “Training AI systems to do alignment research”, as this is their current approach to actually creating an aligned AGI system. I think OpenAI sees the agendas “Training AI systems using human feedback” and “Training models to assist human evaluation” as being useful for this goal, but not ways of aligning AGI systems themselves, so I will only talk about these approaches as they relate to training an AI research assistant.

Also note that OpenAI’s post is pretty vague. This is maybe to be expected since the agenda seems quite new, and they appear some way away from implementing these ideas. I will lay out the optimistic training story I see being told here, but the actual story could diverge in some aspects.

What training story is being told here?

Training goal

Specification: The idea is to build a Large Language Model that can function as an “AI alignment researcher”. This system will be non-goal directed, and will only be useful in the “narrow” sense of being able to do alignment research.

Desirability: This model will not cause safety problems because it is only able to do alignment research, and it is not goal-directed. Hence, it will avoid the problems caused by AGI systems with agency.

If successful, this model will accomplish the desired goal of helping to produce aligned AGI systems by default: that is what it is intended to do! This approach is not aimed at fixing any policy issues such as “how do we deploy an AGI system”.

Training rationale

Constraints: The “AI research assistant” will be trained using a training signal which is hopefully “scalable and aligned”. This training signal will itself be generated via training models to assist human evaluation, which will allow us to train models even when their behaviour and outputs are complex. This will allow us to evaluate the assistant’s behaviour even on complex tasks, which will aid us in building the training signal. We will do this through methods such as IDA, Debate, and RRM, and we will improve these methods through the skills we have developed aligning current systems through methods such as human feedback.

Nu...
Covid 8/25/22: What We Owe
by Zvi — 2022-08-25T14:40:01.532Z

A lot of people owe a lot of things. Some of those debts get forgiven, or placed upon others. Other times, new debts and obligations are introduced whether we like it or not. There was a bunch of both going around this week on several fronts. Many systems are not well functioning, which is nothing new, and the fixes look to make things worse, which is also nothing new.

The Covid situation itself is mostly quiet. We have settled into the new normal, with even people whose entire brand is crazy Covid paranoia finding ways to accept that.

It leaves us room to worry about all the other ways things are going wrong. I suspect this week is a new low for percentage of post that is Covid focused.

Executive Summary

UK emergency room system is collapsing.
Biden uses public purse to pay off student loans.
Signs that monkeypox is peaking or will soon.

Let’s run the numbers.

The Numbers

Predictions

Prediction from last week: 560k cases (-8%) and 3,200 deaths (+0%).

Results: 567k cases (-7%) and 3,183 deaths (-1%).

Prediction for next week: 542k cases (-5%) and 3,030 deaths (-5%).

I got lucky this week to bullseye both numbers, so I don’t expect much to happen this week. With everything quiet, small continued declines matching trend seems most likely.

Deaths

Cases

Physical World Modeling

U.S. on verge of being first in the world to approve and roll out a BA.5-specific booster.

The sources I saw for this are saying they are ‘impressed by CDC & FDA’ because we were first to do this. I non-respectfully disagree. We are essentially first to do this because we effectively have bullied everyone else into waiting for our approval on such matters. Impressive would have been production starting a week after we had the sequence. Acceptable would have been three weeks. This is after it’s too late to do much good. I am not impressed.

Claims about Covid testing, for those who are treating Covid testing as a serious attempt at robustly avoiding exposing others like it was 2020.

Covid test expiration dates are being extended over time as more data comes in. Chances are tests still work long after they say they expire.

Dr. David Nabarro, WHO special envoy on Covid-19, says lockdowns were an effective way for governments to buy time to prepare their health systems at the start of the pandemic, but should not have become a default response to

...