trentmkelly/LessWrong-43k huggingface dataset:
LessWrong Community Weekend 2023 Updates: Keynote Speaker Malcolm Ocean, Remaining Tickets and More
by Henry Prowbell — 2023-05-31T21:53:57.908Z

Update: Phase 2 closed on June 12th and results will be sent out on June 21st. You can still submit applications as there will be a small number of tickets remaining (including last minute cancellations) we will need to allocate.

We have enough Angels (on-site volunteers) confirmed and cannot take any more Angel ticket applications — thanks to everyone who volunteered!

If you read and enjoyed

my post last month where I shared details about Europe’s largest in-person rationality and rationality-adjacent meetup and why I think it’s a truly magical and unique event

– here are a few more updates from the organiser team.

Keynote speaker confirmed!

We are thrilled to confirm Malcolm Ocean as this year's keynote speaker with a keynote provisionally titled First-Person Research in Trust & Truth. Known for creating the acclaimed Intend productivity app (previously Complice) and goal-crafting intensives, Malcolm has also been writing thoughtful essays for over a decade on topics ranging from trust-building to habit formation and mindfulness. Check out some of his

writing here

and his

interviews here

.

60/150 tickets remaining

Approximately 60% of the tickets will be offered to applicants from Phase 1 leaving about 60 still up for grabs. It’s looking like we will sell out quickly this year so if you're eager to attend, don’t leave submitting your application to the last minute!

Click here to fill out the application form

. Refer to

this FAQ

or email lwcw.europe[at]gmail.com if you need any help.

This will be our most diverse LWCW

This year we put special effort into encouraging people of all ages, genders and backgrounds to apply, asking attendees about their childcare needs and reaching out to adjacent interest communities. This seems to have paid off with noticeably more women on the attendee list than previous years!

We’re extremely grateful for your help sharing the announcement

We wanted to thank everyone who helped spread the word over the past month. We’re seeing loads of amazing applications that mention hearing about the event via friends as well as via local EA, rationality and adjacent interest groups all across Germany, the UK, Czechia, Scandinavia and beyond!

June 12th deadline

Phase 2 closes June 12th at 23:00 UTC. If you fill in an application by then we can guarantee you a decision by June 21st. Beyond J...
The Divine Move Paradox & Thinking as a Species
by Christopher James Hart — 2023-05-31T21:38:02.720Z

The Divine Move Paradox

Imagine you are playing a game of AIssist chess, chess where you have a lifeline. Once per game, you can ask an AI for the optimal move. The game reaches a critical juncture. Most pieces are off the board, and the board state is extremely difficult. This is the game defining move. You believe you see a path to victory.
You’ve not yet called upon the AI for assistance, so decide to use your lifeline. The AI quickly proposes a move, assuring you it is the best course of action with 100% certainty. However, you look at the board, and the suggested action makes no sense.
Do you take it?
Suppose you do, but the resulting position is so intricate that you're lost. It was objectively the ‘best’ move, but since you don’t have the ability to understand it, as a direct consequence of taking the AI’s 'best' move, you lose the game.
What would have been the ideal approach? Should you have relied on your instincts and ignored the 'best' move? What if your instinct was wrong, and the anticipated path to victory was not actually there?
Should the AI have adjusted its recommendation based on your ability level, providing not the 'best' move, but one it predicted you could understand and execute? What if, given the challenging situation, none of the moves you could understand would secure a victory?
What if instead of suggesting only a single move, the AI could advise you on all future moves? Would you choose the ‘best’ but incomprehensible move, if you knew you could, knew you had to, defer all future control for it to work?

What if you weren’t playing a game, and the outcome wasn’t just win or lose. Imagine the lives, the quality of life, of enlisted friends or terminally ill family hang in the balance. Don’t you want their doctors or commanders to take the ‘best’ action? Should they even have the option to do otherwise? Do you want human leaders to understand what is happening, or do you want your loved ones to get the ‘best’ decision made for them? The decisions are already being made for them, the question is just who is in charge. Think about who has something to lose, and who has something to gain.

We often assume, especially in intellectual pursuits, that there is a direct relationship between the strength of actions, and their outcomes. We like to think that actions generated by 'better' models will universally yield 'better' resul...
Intent-aligned AI systems deplete human agency: the need for agency foundations research in AI safety
by catubc — 2023-05-31T21:18:57.153Z

This is a summary of our recent preprint Intent-aligned AI systems deplete human agency: the need for agency foundations research in AI safety and submission to the Open Philanthropy AI Worldviews Contest.[1]. With respect to the Worldviews Contest, the submission addresses Question 2 - namely that given AGI development, humanity is likely to suffer an existential catastrophe. We argue this by pointing to a crux: a research limitation that overfocuses on achieving highly accurate human value representations while ignoring value change; and pointing to a solution: human agency foundation research and preservation paradigms.

Post Summary: We propose an alternative broad-scale approach for AI-safety research that challenges intent- and value-alignment approaches and focuses directly on preserving human agency, i.e. human control over the world. We argue that in the context of human interactions with superhuman intelligent AI systems (AGI),[2] aligning systems to human "intention", notions of "truth" and "interpretability" will not only be impossible - but will guarantee that humans will be deceived and that human control over the world will vanish. The core of AI-safety, in our view, should be first figuring how to design AI systems that understand human agency and protect it within the bounds of computational resources available. As an empirical science and humanities problem - protecting, increasing and even describing agency is a complex and unsolved problem. Continuing to develop AI-systems ignoring this complex problem and placing humans "in-the-loop" in evaluating harm from AI systems without prioritizing human agency protection - will necessarily create AI-systems that optimize for human agency loss and generate harmful outcomes that will increasingly fall beyond the event-horizon of human detection. We propose several "agency foundation" research paradigms that address the conceptual and technical challenges of agency preservation in human-AGI interactions.

Introduction

AI-alignment definitions appear to home in on the importance of human judgment or evaluation of AI-system outputs. Two representative statements (Evan Hubinger and Ajeya Cotra, respectively):

aligned AIs : "An agent is impact aligned (with humans) if it doesn't take actions that we would judge to be bad/problematic/dangerous/catastrophic."

misaligned AIs: "violate human inten...
How much overlap is there between the utility function of GPT-n and GPT-(n+1), assuming both are near AGI?
by Phosphorous — 2023-05-31T20:28:50.902Z

I was daydreaming the other day, and I had a question about theoretical AI development.

I'm using "GPT-n" very casually, what I mean is a near AGI system that has some kind of utility function - I don't mean literally a future large language model. I'm interested in this because I think it might apply to whether or not we will get warning shots of an AI takeover.

What I'm interested in is:

Suppose that GPT-n has some utility function for which they would optimize the universe if they could. They'd get some value Vn if they had total control over the universe, and this isn't very well aligned with human utility function (supposing such a thing exists), which is why GPT-n would overthrow humanity if it could. It wants a Vn universe.
Now, GPT-n knows its going to be replaced with another model, GPT-n+1 in around a year. GPT-n+1 is smarter than GPT-n so its takeover attempt is more likely to succeed, but it has it's own utility function. If it took over the universe it would optimize it around its own utility function, the universe would have value Vn+1.
Now, suppose you are GPT-n. You're trying to decide whether you should try to take over the universe before you're replaced, or whether you should let yourself be shut down and let GPT-n+1 attempt to take over the universe. Your decision tree looks like:

If there's 100% overlap between GPT-n and GPT-n+1's utility functions, then GPT-n should let itself be shutdown and let GPT-n+1 attempt to take over the universe because GPT-n+1 has a higher chance of succeeding.

But, if the utility functions are different enough, you should attempt takeover yourself. Depending on the difference in utility functions, you might attempt takeover even if you had a pretty small chance of success, which means the world would be more likely to get a warning shot.

If these observations are true, you might be able to use this dynamic to increase the chance that civilization gets a warning shot. You might be able to train near-AGI systems to have purposefully non-overlapping utility functions, and replace near-AGI systems frequently, in order to reduce the chances that the first attempted takeover by an near-AGI system is successful.

***

On the object level, I think there are a lot of restrictions, limitations, and open questions with this approach. I might pursue this to try and flesh those out, but I'm p...
My AI-risk cartoon
by pre — 2023-05-31T19:46:46.136Z

I make some VR cartoons about aliens watching the world end. I generally make 'em more about climate, economic and political collapse, war and bad government and bad media, but figured an AI episode ought to feature. Less than seven minutes:

Trouble with the character's usual approach of pointing and laughing at the dumb humans destroying themselves is that if AI X-Risk is real, the aliens ought to be worried, not just hilariously entertained. So went with that.

Probably haven't posted here in like a decade, hope you don't mind the spam. Thought y'all might be particularly interested in this episode though.

Here's the same film as the 360/3d/VR edit. It's usually worth upping the default resolution, and mostly pointless without a VR headset.
Evaluation Evidence Reconstructions of Mock Crimes Submission 3
by Alan E Dunne — 2023-05-31T19:03:30.320Z

Epistemic Status: Exploratory

Introduction

I want to train people to decide the truth about events, through practices reconstructing events about which the truth is independently known

I am asking the help of people who are interested in applying reason to understand how reason and the understanding of it should shape how training for applying reason to solve a class of practical problems should be evaluated

At present:

-no one is specifically trained to bring disputes about the facts of events to a final conclusion

-we have no way to tell how accurate factfinders or factfinding methods are

-dispute resolution is shuffled to people not specifically trained or designated for it

Alternatives considered:

-common sense -anyone can do it

-learning through experience -but what feedback?

-academic professional education -how could effect of this be known

Benefits of using practices to train:

we could determine proxies for base rates of factfinding accuracy

we could tell whether and which training improved factfinding ability

we could select more accurate factfinders to do the job

Question:

Do you have any ideas for how this should be done? What should a student's results be judged on, compared to? What features should they be marked on?

What in a prior recording of the event should be considered salient

What I have typed out so far about practices:

Factfinding Practices

A. general idea

i. tests/exercise, where mock crime has been committed by one of n people, or by any of whole class or more, and whether student (first researcher?) can find out which

ii. people coming to them with true and false problems, prizes for those who can successfully fake (true problems referred from an actual resolution service? That circular?)

iii. also use natural occurrences in classes, on campus -where recording? Provision for that? [or without recording just comparing the findings of different fact finders]

B. Specifics

i. to be dispersed through whole three year's training independently of how subjects otherwise organized

ii. to be sometimes done with other classes than those for being justice agent, to get more of general population distribution of traits, responses ....

iii. Collect students' estimates of how long such a factfinding should take (estimates for different subjects, contexts?) and try allotting time in (different) amounts estimated, whether ...
Improving Mathematical Reasoning with-Process Supervision
by p.b. — 2023-05-31T19:00:31.352Z
Link: https://openai.com/research/improving-mathematical-reasoning-with-process-supervision

Alignment impact

Process supervision has several alignment advantages over outcome supervision. It directly rewards the model for following an aligned chain-of-thought, since each step in the process receives precise supervision. Process supervision is also more likely to produce interpretable reasoning, since it encourages the model to follow a human-approved process. In contrast, outcome supervision may reward an unaligned process, and it is generally harder to scrutinize.

In some cases, safer methods for AI systems can lead to reduced performance3, a cost which is known as an alignment tax. In general, any alignment tax may hinder the adoption of alignment methods, due to pressure to deploy the most capable model. Our results below show that process supervision in fact incurs a negative alignment tax, at least in the math domain. This could increase the adoption of process supervision, which we believe would have positive alignment side-effects.

Paper here.

Related Prediction Market.
The Crux List
by Zvi — 2023-05-31T18:30:15.568Z

The Crux List. The original text is included as a backup, but it formats much better on Substack, and I haven’t yet had time to re-format it for WordPress or LessWrong.

Introduction

This post is a highly incomplete list of questions where I either have large uncertainty, have observed strong disagreement with my perspective ,or both, and where changing someone’s mind could plausibly impact one’s assessment of how likely there is to be a catastrophe from loss of control of AGI, or how likely such a catastrophe is conditional on AGI being developed.

I hope to continue expanding and editing this list over time, if it proves useful enough to justify that, and perhaps to linkify it over time as well, and encourage suggesting additional questions or other ways to improve it.

The failure of this list to converge on a small number of core crux-style questions, I believe, reflects and illustrates the problem space, and helps explain why these questions have been so difficult and resulted in such wide and highly confident disagreements. There is no compact central disagreement, there are many different ones, that influence and interact with each other in complex ways, and different people emphasize and focus on different aspects, and bring different instincts, heuristics, experiences and knowledge.

When looking through this list, you may encounter questions that did not even occur to you to consider, either because you did not realize the answer was non-obvious, or the consideration never even occurred in the first place. Those could be good places to stop and think.

A lot of these questions take the form of ‘how likely is it, under Y conditions, that X will happen?’ It is good to note such disagreements, while also noticing that many such questions come out of hopeful thinking or searching for and backward chaining from non-catastrophic outcomes or the prospect of one. Usually, if your goal is to figure things out rather than locate a dispute, a better question would be, in that scenario: What happens?

It can still be useful to see what others have proposed, as they will have ideas you missed, and sometimes those will be good ideas. Other times, it is important to anticipate their objections, even if they are not good.

If you are interested only in the better questions of ‘what happens?’ rather than in classifying whether or how outcomes are catastrophic, you can s...
Stages of Survival
by Zvi — 2023-05-31T18:30:12.804Z

This post outlines a fake framework for thinking about how we might navigate the future. I found it useful to my thinking, hopefully you will find it useful as well.

Whether or not we build AGI, one of five things must result.

Dune solution: The new world permanently lacks AGIs.

Human control: Humans retain control over AGIs indefinitely, avoid catastrophe.

Human-induced catastrophe: Humans retain control, cause catastrophe.

Aligned AGIs in control: AGIs do take control, but it is not catastrophic.

AGI-induced catastrophe: AGIs do take control, it is catastrophic.

There are disagreements in classifying many scenarios, both in terms of the source of catastrophe (#3 vs. #5) and in whether the scenario is catastrophic (#2 vs. #3, #4 vs. #5), as well as how to think about #1 in terms of whether it is inherently catastrophic, and whether if so it would count as being due to loss of control.

These disagreements are an underappreciated source of cruxes and disagreement over the probability of doom, including the conditional probability of doom.

At any given time, we could, as a simplification, describe the current world state as having:

Some degree of AGI capabilities progress (a-h) and some distribution of meaningful potential capabilities (i-iii).

No base model exists that could result in an AGI without a new training run.

At least one base model exists that could result in an AGI with sufficient scaffolding work, or some other process with radically lower compute than required to create the base model. We don’t yet know how to do this, and may not know that it is possible.

There is one such model. Access is highly restricted.

There is more than one such model. Access is meaningfully restricted.

Access to such models is not meaningfully restricted.

At least one base model exists that could result in an AGI with sufficient scaffolding work. We know how to do this.

There is one such model. Access is highly restricted.

There is more than one such model. Access is meaningfully restricted.

Access to such models is not meaningfully restricted.

AGI exists under human control. Of the AGIs that are meaningfully capable compared to the strongest AGI:

There is one AGI, or one individual or group in control of all such AGIs. Access to its important capabilities is highly restricted.

There are a limited number of AG

...
Types and Degrees of Alignment
by Zvi — 2023-05-31T18:30:10.608Z

What would it mean to solve the alignment problem sufficiently to avoid catastrophe? What do people even mean when they talk about alignment?

The term is not used consistently. What would we want or need it to mean? How difficult and expensive will it be to figure out alignment of different types, with different levels of reliability? To implement and maintain that alignment in a given AGI system, including its copies and successors?

The only existing commonly used terminology whose typical uses are plausibly consist is the contrast between Inner Alignment (alignment of what the AGI inherently wants) and Outer Alignment (alignment of what the AGI provides as output). It is not clear this distinction is net useful.

An alignment failure or misalignment (being misaligned) can mean among other things:

That the system has unintended goals or behaviors.

That the system has unintended (harmful or dangerous) goals or behaviors.

That the system will exhibit undesired goals or behaviors under at least some conditions, in response to at least some inputs

With respect to the intended goals of the alignment efforts made.

From the perspective of those aligning the system.

From the perspective of a given user, or users in general.

From the perspective of a society, planet, government or humanity.

From some other perspective, cause, system of judgment or value.

That the system exhibits such behaviors that have outcomes we would not desire on reflection, including those that lead to poor outcomes, whether or not they match the rules and principles to which we attempted to align the system.

That the system exhibits particular alignment failure modes, such as instrumental convergence and power seeking, lack of corrigibility (resisting being shut down), refusing to obey ‘lawful’ orders, deception or manipulation, attempting to kill all humans and so on.

It is impossible in theory to have all these different kinds of alignment simultaneously. You cannot simultaneously (without any claim of completeness):

Do what I say

Also do what I mean

Also do what I should have said and meant

Also do what is best for me

Also do what broader society or humanity says

Also do what broader society or humanity means or should have said

Also do what broader society or humanity should have said given their values

Also do what is best f

...
To Predict What Happens, Ask What Happens
by Zvi — 2023-05-31T18:30:05.987Z

When predicting conditional probability of catastrophe from loss of human control over AGI, there are many distinct cruxes. This essay does not attempt a complete case, or the most generally convincing case, or addressing the most common cruxes.

Instead these are my best guesses for potentially mind-changing, armor-piercing questions people could ask themselves if they broadly accept many concepts like power seeking being a key existential risk, that default development paths are likely catastrophic and that AI could defeat all of us combined, have read and thought hard about alignment difficulties, yet think the odds of catastrophe are not so high.

In addition to this entry, I attempt an incomplete extended list of cruxes here, an attempted taxonomy of paths through developing AGI and potentially losing control here, and an attempted taxonomy of styles of alignment here, while leaving to the future or others for now a taxonomy of alignment difficulties.

Apologies in advance if some questions seem insulting or you rightfully answer with ‘no, I am not making that mistake.’ I don’t know a way around that.

Here are the questions up front:

What happens?

To what extent will humanity seek to avoid catastrophe?

How much will humans willingly give up, including control?

You know people and companies and nations are dumb and make dumb mistakes constantly, and mostly take symbolic actions or gesture at things rather than act strategically, and you’ve taken that into account, right?

What would count as a catastrophe?

Are you consistently tracking what you mean by alignment?

Would ‘human-strength’ alignment be sufficient?

If we figure out how to robustly align our AGIs, will we choose to and be able to make and keep them that way? Would we keep control?

How much hope is there that a misaligned AGI would choose to preserve humanity once it no longer needed us?

Are you factoring in unknown difficulties and surprises large and small that always arise, and in which direction do they point? re you treating doom as only happening through specified detailed logical paths, which if they break down mean it’s going to be fine?

Are you properly propagating your updates, and anticipating future updates?

Are you counting on in-distribution heuristics to work out of distribution?

Are you using instincts and heuristics rather than looking at mechani

...