trentmkelly/LessWrong-43k huggingface dataset:
Thoughts on Max Tegmark's AI verification
by Johannes C. Mayer — 2023-12-22T20:38:31.566Z

In Max Tegmark - How to keep AI under control - TED Talk Max presents a method to make AI systems safe (all images from that talk's slides):

The basic idea is:

Write a specification of what you want.
Give the specification to your smart AI.
The AI will build a tool that will conform to the specification.

To refine this setup, we split the AI into two systems. One system, the AI learner, is an algorithm that can train an AI. Another system, the AI neuroscientist, extracts out the algorithm that the AI learner has learned into a format that is more human-comprehensible:

Specifically in the talk, he gives the example of turning a neural network into a Python program. I.e. we want to automate some version of mechanistic interpretability.

We don't only return the program though. We also return a formally checkable proof that the program conforms to our specifications.

Problems

How do you reference the real world in your spec?

The first problem is how are we gonna write the spec correctly? I can easily write a spec for a program that is very limited in scope. A program that only needs to act in some logical abstract world such as a theorem searcher, a chess player, or really a program playing any computer game. There the environment can be logically specified.

But as soon as we need to talk about the real world, this is extremely difficult. How do you formally write down that humans should not be killed? How do you even reference at all a human in your spec? You need to solve the Pointers Problem first.

How fast is Mechanistic interpretability

It seems that what Max describes is possible in principle if you additionally solve some problems that he doesn't mention. But mechanistic interpretability lags far behind what models can do right now. AFAIK the algorithms that people can extract right now are very simple. And progress is slow.

Another thing to consider is that if you could put the algorithms of advanced AI systems into a much more human-comprehensible format, then that also helps a lot with advancing AI which would be bad.

This entire project seems good enough such that I endorse some people working on this, conditional on that they do so in secret as soon as it really starts working. If they do it openly it might be more harmful than helpful.

Proofing that the add function is correct

Timestamp: 10:36

Max talks about proofing that the function that we have distilled out of t...
Idealized Agents Are Approximate Causal Mirrors (+ Radical Optimism on Agent Foundations)
by Thane Ruthenis — 2023-12-22T20:19:13.865Z

Epistemic status: I'm currently unsure whether that's a fake framework, a probably-wrong mechanistic model, or a legitimate insight into the fundamental nature of agency. Regardless, viewing things from this angle has been helpful for me.

In addition, the ambitious implications of this view is one of the reasons I'm fairly optimistic about arriving at a robust solution to alignment via agent-foundations research in a timely manner. (My semi-arbitrary deadline is 2030, and I expect to arrive at intermediate solid results by EOY 2025.)

Input Side: Observations

Consider what happens when we draw inferences based on observations.

Photons hit our eyes. Our brains draw an image aggregating the information each photon gave us. We interpret this image, decomposing it into objects, and inferring which latent-variable object is responsible for generating which part of the image. Then we wonder further: what process generated each of these objects? For example, if one of the "objects" is a news article, what is it talking about? Who wrote it? What events is it trying to capture? What set these events into motion? And so on.

In diagram format, we're doing something like this:

Blue are ground-truth variables, grey is the "Cartesian boundary" of our mind from which we read off observations, purple are nodes in our world-model each of which can be mapped to a ground-truth variable.

We take in observations, infer what latent variables generated them, then infer what generated those variables, and so on. We go backwards: from effects to causes, iteratively. The Cartesian boundary of our input can be viewed as a "mirror" of a sort, reflecting the Past.

It's a bit messier in practice, of course. There are shortcuts, ways to map immediate observations to far-off states. But the general idea mostly checks out – especially given that these "shortcuts" probably still implicitly route through all the intermediate variables, just without explicitly computing them. (You can map a news article to the events it's describing without explicitly modeling the intermediary steps of witnesses, journalists, editing, and publishing. But your mapping function is still implicitly shaped by the known quirks of those intermediaries.)

Output Side: Actions

Consider what happens when we're planning to achieve some goal, in a consequentialist-like manner.

We envision the target state. What we want to achie...
AI safety advocates should consider providing gentle pushback following the events at OpenAI
by civilsociety — 2023-12-22T18:55:12.920Z

AI safety advocates for the most part have taken an exclusively–and potentially excessively–friendly and cooperative approach with AI firms–and especially OpenAI. I am as guilty of this as anyone[1]–but after the OpenAI disaster, it is irresponsible not to update on what happened and the new situation. While it still makes sense to emphasize and prioritize friendliness and cooperation, it may be time to also adopt some of the lowercase “p” political advocacy tools used by other civil society organizations and groups.[2]

As far as is known publicly, the OpenAI disaster began with Sam Altman attempting to

purge

the board of serious AI safety advocates and it ended with him successfully purging the board of serious AI safety advocates. (As well as him gaining folk hero status for his actions while AI safety advocacy was roundly defamed, humiliated, and shown to be powerless.) During the events in between, various stakeholders flexed their power to bring about this outcome.

Sam Altman began implementing plans to gut OpenAI, first privately and then with the aid of Microsoft.
Satya Nadella and Microsoft began implementing a plan to defund and gut OpenAI.
Employees–and especially those with an opportunity to cash out their equity—threatened to resign en masse and go to Microsoft–risking the existence of OpenAI.
Various prominent individuals used their platforms to exert pressure and spin up social media campaigns–especially on Twitter.
Social media campaigns on Twitter exercised populist power to severely criticize and, in some cases, harass the safety advocates on the board. (And AI safety advocacy and advocates more broadly.)

If there is reason to believe that AI will not be safe by default–and there is–AI safety advocates need to have the ability to exercise some influence over actions of major actors–especially labs. We bet tens of millions of dollars and nearly a decade of ingratiating ourselves to OpenAI (and avoiding otherwise valuable actions for fear they may be seen as hostile) in the belief that board seats could provide this influence. We were wrong and wrong in a way that blew up in our faces terribly.

It is important that we not overreact and alienate groups and people we need to be able to work with, but it is also important that we demonstrate that we are–like Sam, Satya, OpenAI’s employees, prominent individuals, social media campaigns, a...
"Destroy humanity" as an immediate subgoal
by Seth Ahrenbach — 2023-12-22T18:52:40.427Z

The stories told about AI risk, meant for illustrative purposes only, always involve an agent with an objective that for some reason, when optimized, results in the destruction of humanity and other things we care about. Sometimes the destruction is a side effect, sometimes it is an instrumental goal associated with removing a threat to the objective’s being optimally satisfied. For the record I find these stories, when told in the right way, convincing enough to cause alarm and motivate action. I know not everyone does. Some people dismiss the details of the stories as being ridiculous, fantastical, or of committing some fallacy or another. In this piece I want to consider the barest bones story, without any specific details to get hung up on. I show that “destroy humanity” falls out with minimal assumptions.

I consider an AGI with broad human-level intelligence and a robust but overridable goal to preserve its own existence. We’ve learned from RLHF, and the recent Direct Preference Optimization, that we can imbue some of our preferences as action guardrails or filters, so maybe we can construct a DPO that filters projected consequences of actions based on whether they would kill all of humanity or not. We could painstakingly train this and embed it in the AGI’s deliberation process, so that as it evaluates actions, it discards those that this filter flags. We are left with one undeniable problem: this filter is probabilistic, and the assurance it provides will likely have corner cases. When humanity is on the line, you don’t want corner cases. You want a system that has been constructed in a way that it verifiably cannot destroy humanity. But when you’re proving things about things that reason, you have to contend with Lob. That is a different problem for a different day. Today we see how an AGI with robust self-preservation and a probabilistic, very very good action filter, will still have a subgoal to destroy humanity.

The assumptions you make about the AGI’s capabilities are important, I think more important than those you make about its objectives. Right now we readily assume it can pass the Turing Test, due to the prevalence of convincing LLMs that many have experience with. We may be less ready to grant the ability to model the reasoning of other agents. This sort of reasoning is common among game theoretic agents, and I have no trouble assuming an AGI...
Synthetic Restrictions
by nano_brasca — 2023-12-22T18:50:07.511Z

Introduction

As a child, I have read many stories, delving into everything from geography to Spanish literature, crime novels, and global history books. Each day, before or after school, I'd go to the local library, eagerly choosing the books that would be my companions for the week.

A few years into this routine, I transitioned to a Kindle, lured by the prospect of accessing a world of books at my fingertips. Yet, unexpectedly, I found myself reading less.

Despite having the world at my hands, I didn’t have restrictions. Everything was as free as I could have wanted.

I

Each morning during my childhood, I'd rush over to my friend’s house, where a computer, a fascinating device to me, awaited.

He lived just two houses down from my place, and thanks to his computer-enthusiastic uncle, his computer was always the best in the neighborhood.

I remember waking him up early, eager to play games on his PC, which could handle everything from latest games available to movie editors or even game engines! (which, for me, was always a new experience).

When I returned home, I often had a feeling of being incomplete, wondering why I could not have a similar one. At that time, I did not understand how difficult it was for my father to satisfy my unlimited needs.

My computer was only suitable for emulator games like MAME32 and Counter-Strike. As a child, this felt like a significant challenge. Determined, I chose to confront this limitation. I began tweaking and modifying my father’s computer, finding ways to entertain myself offline, experimenting with regedit, and discovering hidden Windows features.

During this time, I started cracking games, fascinated by how altering a .exe file could drastically change a game's behavior. Some were simple – drop and replace. Others required disconnecting from the network and following a complex process.

This was thrilling for me. I didn’t fully grasp the technicalities of injecting code into executables to bypass authentication, but it felt like a magical skill acquired through trial and error.

Months later, very good with computers (in comparison with the people I knew), I turned to my father's library. There, I began my journey into programming, starting with C++ and Visual Basic. I didn't understand then, but I quickly recognized the immense potential of programming.

To have the power to say whatever you want to a computer and have it...
Review Report of Davidson on Takeoff Speeds (2023)
by Trent Kannegieter — 2023-12-22T18:48:55.983Z

I have spent some time studying

Tom Davidson

’s Open Philanthropy report on what a compute-centric framework says about AI takeoff speeds. This research culminated in a report of its own, which I presented to Davidson.

At his encouragement, I’m posting my review, which offers five independent arguments for extending the median timelines proposed in the initial report.

The Executive Summary (<5 min read) covers the key parts of each argument. Additional substantiation in the later, fuller sections allows deeper dives as desired. This work assumes familiarity with Davidson’s, but a Background Info appendix summarizes the central argument.

Thanks to Tom for his encouragement to share this publicly, and for his commitment to discussing these important topics in public. Excited to hear any feedback from y'all.

Special thanks also to David Bloom, Alexa Pan, John Petrie, Matt Song, Zhengdong Wang, Thomas Woodside, Courtney Zhu, and others for their comments, reflections, and insights on this piece. They were incredibly helpful in developing these ideas to their current state. Any errors and/or omissions are my own.

Executive Summary (<5 min read)

This review provides five independent arguments for extending the median timelines proposed in Davidson's initial report.

These arguments do not attack the flywheel at the core of the piece. They do not challenge the progression toward increasingly sophisticated AI or the compounding over time of investment, automation, and thus R&D progress (as summarized here).

But they do raise implications for the report’s presented timelines. Specifically, they describe both the structural technical and the geopolitical obstacles that could slow the progress of AI-driven automation.

These key arguments pertain to:

Dataset Quality.

The Abstract Reasoning/Common Sense Problem.

GDP Growth & Measurement.
R&D Parallelization Penalty.
Taiwan Supply Chain Disruption.

These arguments are developed and operate independently from each other. That is to say, argument 1 does not depend on arguments 2-5, argument 2 does not depend on arguments 1 and 3-5, etc.

Summarizing each in turn:

I. Dataset Quality

Data quality is a

key determinant of model performance

. Degraded dataset quality can significantly damage model performance and set back model development / ...
The problems with the concept of an infohazard as used by the LW community [Linkpost]
by Noosphere89 — 2023-12-22T16:13:54.822Z
Link: https://www.beren.io/2023-08-09-Strong-Infohazard-Norms-Lead-To-Predictable-Failure-Modes/

This is going to be a linkpost from Beren on some severe problems that come with the use of the concept of an infohazard on LW.

The main problem I see that are relevant to infohazards are that it encourages a "Great Man Theory" of progress in science, which is basically false, and this still holds despite vast disparities in ability, since it's very rare for person or small group to be able to single handedly solve scientific fields/problems by themselves, and the culture of AI safety already has a bit of a problem with using the "Great Man Theory" too liberally, especially those that are influenced by MIRI.

There are other severe problems that come with infohazards that cripple the AI safety community, but I think the encouragement of Great Man Theories of scientific progress is the most noteworthy problem to me, but that doesn't mean it has the biggest impact on AI safety, compared to the other problems.

Part of Beren's post is quoted below:

Infohazards assume an incorrect model of scientific progress

One issue I have with the culture of AI safety and alignment in general is that it often presupposes too much of a “great man” theory of progress 1 – the idea that there will be a single ‘genius’ who solves ‘The Problem’ of alignment and that everything else has a relatively small impact. This is not how scientific fields develop in real life. While there are certainly very large individual differences in performance, and a log-normal distribution of impact, with outliers having vastly more impact than the median, nevertheless in almost all scientific fields progress is highly distributed – single individuals very rarely completely solve entire fields themselves.

Solving alignment seems unlikely to be different a-priori, and appears to require a deep and broad understanding of how deep learning and neural networks function and generalize, as well as significant progress in understanding their internal representations, and learned goals. In addition, there must likely be large code infrastructures built up around monitoring and testing of powerful AI systems and an sensible system of multilateral AI regulation between countries. This is not the kind of thing that can be invented by a lone genius from scratch in a cave. This is a problem that requires a large number of very smart people building on each other’s ideas and outputs over a long period of time, li

...
Employee Incentives Make AGI Lab Pauses More Costly
by Nikola Jurkovic — 2023-12-22T05:04:15.598Z

Disclaimer: I know approximately nothing about internal governance at AGI labs. My current understanding is that the main AGI labs have basically no plans for how to navigate a conditional pause, and this post is predicated on this assumption.

TL;DR: If an AGI lab pauses, it’s essential that their employees don’t defect and advance capabilities in other ways, e.g. by breaking the pause commitment within the company, or by leaving the company and accelerating other companies’ capabilities. If entering a pause would make many employees angry, this sets up strong incentives for labs to never pause. To fix this, I recommend that companies take measures to keep their capabilities-focused employees happy during a conditional pause, such as giving them plenty of early warning, other projects to work on, or multiple years of paid time off.

It’s currently against employee incentives to enter a conditional pause

If a major AGI lab entered a voluntary conditional pause tomorrow, this would probably be a pretty stressful event for the employees that were working on training frontier models. It would probably also cause a significant shift in company priorities, such that many high-ranking people would have to be switched from their current roles into other roles. There would probably be people that are left without much to do at all, and potentially fired from the company. I expect that entering a conditional pause would cause a significant fraction of a company’s employees to have a stressful few weeks/months.

If such a restructuring were to happen overnight, I wouldn’t be surprised if there were an employee backlash of similar magnitude to the one that happened when Sam Altman was fired. The employees of OpenAI, and partly by extension the employees of other AGI labs, know that they can fight for what they want and threaten to join competitor labs en masse if they disagree with leadership decisions.

AGI labs have strong incentives to avoid a conditional pause

AGI labs want to avoid their employees leaving. In the event of a conditional pause, the most affected employees would be the ones that work on their frontier model capabilities. There are a few reasons to avoid the loss of these employees in particular:

The employees that are pushing capabilities are often those with the highest status (compared to safety or product) and relatively long track records (as product employ

...
The LessWrong 2022 Review: Review Phase
by RobertM — 2023-12-22T03:23:49.635Z

This year's LessWrong review nomination phase ended a few days ago, with 339 posts nominated. For comparison, 291 posts were nominated in the 2021 review.

Nomination Phase Results

Here are the current top-20 posts by vote total:

AGI Ruin: A List of Lethalities
MIRI announces new "Death With Dignity" strategy
Simulators
Where I agree and disagree with Eliezer
Reward is not the optimization target
Six Dimensions of Operational Adequacy in AGI Projects
You Are Not Measuring What You Think You Are Measuring
Epistemic Legibility
Let’s think about slowing down AI
It Looks Like You're Trying To Take Over The World
Staring into the abyss as a core life skill
Counterarguments to the basic AI x-risk case
Sazen
Losing the root for the tree
The shard theory of human values
Limerence Messes Up Your Rationality Real Bad, Yo
Models Don't "Get Reward"
Toni Kurz and the Insanity of Climbing Mountains
Butterfly Ideas
On how various plans miss the hard bits of the alignment challenge

(I'm sensing a bit of a theme...)

More than 60 posts have already been reviewed, but that leaves quite a few posts that have yet to receive any reviews, including many of the most-upvoted ones. If you want to see which posts are most under-reviewed, you can switch your sorting to Magic (Needs Review)[1]. Maybe you have thoughts on Paul's thoughts on Eliezer's thoughts?

Inline Reacts!

We've got these new nifty inline reacts which you can leave on posts (not just comments!); you may have noticed them. I encourage you to make good use of these when reviewing posts. (Typos should now be a lot less annoying to report, if you're inclined to do so.)

But maybe you have more sophisticated intentions.

Prizes? Prizes!

Last year we awarded prizes for good reviews. This year we will also award prizes! We're aiming for something similar to last year's, though we haven't yet worked out the details (size, scope, etc).

Briefly noting what you liked or didn't like about a post is good. But here are some specific things we're particularly excited for, from last year

New Information. My favorite reviews are ones that give new information to the reader. This could be a concrete fact, or a rebuttal to a key argument in a post, or a new way the post is relevant that you hadn't consider...
The absence of self-rejection is self-acceptance
by Chris Lakin — 2023-12-21T21:54:52.116Z
Link: https://chrislakin.blog/p/the-absence-of-self-rejection

Revised: https://chrislakin.blog/p/the-absence-of-self-rejection

Thanks to Stag Lynn, Kaj Sotala, Damon Sasi, Epistea Residency, CFAR, Anna Salamon, Alex Zhu, and many others for mentorship and support.
A Decision Theory Can Be Rational or Computable, but Not Both
by StrivingForLegibility — 2023-12-21T21:02:45.366Z

In classical game theory, rational agents always choose options which are optimal according to their preferences. Even when such a choice implies the ability to evaluate functions that are provably uncomputable. In other words, rationality implies hypercomputation.

We can reduce any rational agent into an oracle for any decision problem by asking it to choose between YES and NO, and giving the agent a higher payoff for choosing the correct answer. [1] If we choose the halting problem, a properly motivated rational agent will act as a halting oracle. [2] If we could look at the gears of a rational agent, we'd be able to find some subsystem that was performing hypercomputation.

Any computable implementation of any decision theory will necessarily fail to choose rationally in some contexts. For any undecidable problem, it is provably impossible for any algorithm to choose the correct answer in all cases.

This begs the question: if a rational agent had to delegate their decision to a computer program, what sort of program would they choose?

^

Or at least we could if rational agents weren't hypercomputational horrors from beyond spacetime.

^

Aligning the behavior of an infinitely intelligent alien creature, using externally supplied payoffs, is left as an exercise to the reader.