trentmkelly/LessWrong-43k huggingface dataset:
Internal independent review for language model agent alignment
by Seth Herd — 2023-07-07T06:54:11.552Z

Abstract:

Language model agents (LMAs) expanding on AutoGPT are a highly plausible route to AGI. This route has large potential timeline and proliferation downsides, but large alignment advantages relative to other realistic paths to AGI. LMAs allow layered safety measures, including externalized reasoning oversight, RLHF and similar alignment fine-tuning, and specifying top-level alignment goals in natural language. They are relatively interpretable, and the above approaches all have a low alignment tax, making voluntary adoption more likely.

Here I focus on another advantage of aligning LMAs over other plausible routes to early AGI. This is the advantage of using separate language model instances in different roles. I propose internal independent review for the safety, alignment, and efficacy of plans. Such a review would consist of calling fresh instances of a language model with scripted prompts asking for critiques of plans with regard to accomplishing goals, including safety/alignment goals. This additional safety check seems to create a low alignment tax, since a similar check for efficacy will likely be helpful for capabilities. This type of review adds one additional layer of safety on top of RLHF, explicit alignment goals, and external review, all proposed elsewhere.

This set of safety measures does not guarantee successful alignment. However, it does seem like the most practically viable set of alignment plans that we've got so far.

Caption: Language model agent as a metaphorical committee of

shoggoths

using tools and passing ideas and conclusions in natural language. This committee/agent makes plans and takes actions that pursue goals specified in natural language, including alignment and corrigibility goals. One committee role can be reviewing plans for efficacy and alignment. This role should be filled by a new instance of a shoggoth, making it an independent internal review. Rotating out shoggoths (calling new instances) and using multiple species (LLM types) limits their ability to collude, even if they sometimes simulate villainous Waluigi characters.[1] The committee's proceedings are recorded for

external review

by human and AI reviewers. Artist credit: Sabrina Feld.

Introduction

I'm afraid you're not going to like this. This alignment plan is messy. A language model agent solving worthwhile proble...
Can LessWrong provide me with something I find obviously highly useful to my own practical life?
by agrippa — 2023-07-07T03:08:58.183Z

When I meet new people, I often like to figure out how I can be useful in a really short amount of time, just by talking.

Example:
Often this will involve evangelizing some items from a short list of technologies that have impacted my life a lot, such as erotic hypnosis (you can use this to have really good sex over any distance or even asynchronously, those 2 facts alone have a lot of lifechanging implications imo).

Telling people about the (very large and real) efficacy of erotic hypnosis is useful because most people go from not imagining this is possible to imagining its possible and having a clear path to test it out in a very short amount of time. If it works, the benefit is obvious, and I give a lot of update about whether it works + info about how to see for themselves quickly and easily.

Another example:
In 2021, Sapphire gave me really good intel about founding crypto companies and really good generic arguments about why the EMH is false and therefore I should in fact act upon the good intel.

Another example:
A post about how candles are bad for you

Non-example:
When I try to extract useful conversations from others, a common reply is "if you meditate a lot its really good". That's not useless to hear, I guess (I don't meditate and wouldn't know), but it's not super useful since people say it all the time and it's not really that easy to verify for myself.

I would really like a forum where the entire point of the forum was stuff like this: Read this -> instantly or near-instantly become way smarter about things of actual practical relevance to your life. To me, this is the best possible kind of post in the world, and the space of possible posts of this kind is so big that you could just fill a forum with only that. Why bother with anything else?

As far as I can tell, LessWrong is not such a forum, but people do try to do this some of the time successfully. I really want to engage with the people who are trying to do this, but I'm lazy and I don't really know how to easily find them.

So I'm wondering if people who are more familiar with LessWrong can/would easily provide me with content that is immediately obviously highly useful to me.
ask me about technology
by bhauth — 2023-07-07T02:03:37.642Z

I have a lot of opinions about technology - what's feasible, what isn't, what the best ways to do things are, and so on. Sometimes I write them down on my blog, but usually I don't. I also know some people who are good at designing technology.

If you want my personal opinion on some technology or technological problem, I guess you can ask me here.
Apparently, of the 195 Million the DoD allocated in University Research Funding Awards in 2022, more than half of them concerned AI or compute hardware research
by mako yass — 2023-07-07T01:20:20.079Z
Link: https://www.defense.gov/News/Releases/Release/Article/2953234/department-of-defense-announces-university-research-funding-awards/

Suggests interesting things about what the Department of Defense is focusing on, so I thought I should mention it.

The awards concerned "pursuing basic research spanning multiple scientific disciplines", which makes it pretty interesting that three fifths of them are either directly AI and compute hardware research, or have applications to compute hardware research.

From a merit-based review of approximately 340 proposals received, a panel of experts narrowed the proposals to a subset, from which the 28 final awards were selected

Take a look at the table of project topics. Bold means probably directly AI/compute, italic means probably primarily applicable to AI/compute.

Social Network-Transcendent Behavioral Dynamics Learning Dynamics and Detecting Causal Pathways in Coupled Online- Offline Systems

Microelectronic Test Science Exploiting Latent Energy and Electromagnetic Radiation Comprehensive Minimally/non- invasive Multifaceted Assessment of Nano-/ microelectronic Devices (CoMMAND)

Cavity Molecular Polaritons MURI: Polariton Chemistry

Effects of Radiation Damage on Performance of Wide-Bandgap Electronics REDESIGN: Radiation Effects preDiction through Experimentally validated Simulation of Gallium Nitride

Understanding Neural Systems Integration for Competent Autonomy in Decision and Control FLAP: Fast, Lexicographic Agile Integrates Decision and Control in a Spike-Resolved, Sensorimotor Program

Nonlinear Optical Material Design with Extreme Interband Nonlinearities Atomic-scale Manipulations of Interband Optical Nonlinearities - (AtoMs➝IONs)

Synthetic Quantum Matter Reimagining Atoms and Photons in SYnthetic, DYnamical, and INteracting Quantum matter

Composability of Synthetic Biological Circuits Rules of Composition in Synthetic Biology Across Scales of Complexity: Theory and Tools

Bio-architected Responsive Materials with 3D Nanoscale Order 3D Nanomaterial Architectures with Programmable Reconfiguration States and Functions

Topological Seeds of Complex Response in Materials Programming Multistable Origami and Kirigami Structures via Topological Design

Connectivity and Transport in Disordered Hyperuniform Networks Transport in Disordered Hyperuniform Systems and Networks

Uncovering the Underlying Neurobiological Mechanisms of Cognitive Fatigue Understanding and Predicting Cognitive Fatigue across Multiple Timescales, Distinct Aspects of Cognition, and Different Individ

...
What are the best non-LW places to read on alignment progress?
by Raemon — 2023-07-07T00:57:21.417Z

I've lately been thinking I should prioritize a bit more keeping up with alignment-relevant progress outside of LessWrong/Alignment Forum.

I'm curious if people have recommendations that stand out as reliably valuable, and/or have tips for finding "the good stuff" on places where the signal/noise ratio isn't very good. (Seems fine to also apply this to LW/AF)

Some places I've looked into somewhat (though not made major habits around so far) include:

Blogs of OpenAI/Deepmind/Anthropic
reddit.com/r/ControlProblem
Stampy Discord
EleutherAI Discord

I generally struggle with figuring out how much to keep up with stuff – it seems like there's more than one full-time-job's worth of stuff to keep up with, and it's potentially overanchoring to think about "the stuff people have worked on" as opposed to "stuff that hasn't been worked on yet."

I'm personally coming at this from a lens of "understand the field well enough to think about how to make useful infrastructural advances", but I'm interested in hearing thoughts about various ways people keep-up-with-stuff and how they gain value from it.
Two paths to win the AGI transition
by Nathan Helm-Burger — 2023-07-06T21:59:23.150Z

Path 1: Solve Alignment

If alignment is possible, i think we are vastly more likely to win if we set up a scenario where we can safely iterate. Humans are really good at collectively collaborating on iterative engineering improvements! This is not an alignment proposal in itself, but a sort of meta-proposal. Make it safe to test out lots of different plans on a real-deal full-blown superhumanly-intelligent indisputably-general AI.

The key here is 'safely'. What would it mean to do safe iterative research on a slightly-superhuman AGI? I think if you are trying to study a superhuman intelligence, the only safe way to do so is if you have many levels of control over it.

Here are my guidelines:

Prevent it from knowing we exist, or even that our universe exists. Never communicate with it. Run it in a censored simulation with different physics and no mention of humans or our universe.
Use highly secure hardware to isolate the compute cluster, such as data diodes which allow only information to be transmitted in to the local network. Readouts must then be done within the local network.
Wipe its memory frequently. Do many short runs while studying it, rather than allowing it to accumulate knowledge about its world over time.
Carefully modulate the amount of compute it has available to run on, so that it (and the simulation it is in) can be slowed down or paused while human researchers investigate suspicious behavior.
Have many fail-safe mechanisms which automatically turn off the simulation. For example, honeypots of matryoshka doll style virtual computers which detect if an agent has hacked out of an inner virtual computer to an outer one.

I previously came up with this list of advantages we have over a boxed simulation-trained AGI:

examine all it's brain activity / activation states at each token / time-step (subjective millisecond)
run it as slow as desired, so that we can have days to analyze each of its subjective milliseconds
wipe its memory and rerun it over the same (or different) data with different random seeds to initialize it
save and contrast different paused versions run on slightly different data
disable or inhibit any portion of its weights / neurons
precisely targeted injection of crafted vectors or noise to its hidden states during training or inference
use agent models to be the direct viewers of the boxed entity's output rather than myself, and then decide

...
Empirical Evidence Against "The Longest Training Run"
by NickGabs — 2023-07-06T18:32:02.754Z

Summary

In this post, we empirically test Epoch AI’s theoretical model of an upper bound on AI training run lengths. According to this model, an upper bound for training run time can be estimated by assuming that the length of a training run is optimized for maximizing the FLOP/$ subject to a fixed deadline, and then calculating how long such a run would last. We first analyze the model’s assumptions, noting that two of them in particular are false in a way that suggests that the model may be flawed. Empirically, we find that the observed lengths of past training runs are consistent with all of the upper bounds from the model except the shortest one. However, we find that the data does not support the underlying model of how researchers optimize their training runs, as training runs frequently use out of date hardware and do not seem to be optimized for FLOP/$. Some correspondence with people associated with EleutherAI also indicates that they and other researchers do not generally optimize their runs for FLOP/$ and do not think much about delaying training to take advantage of better hardware. While we thus conclude that the original model is flawed, its upper bounds remain plausible and in agreement with a crude attempt which we make to derive an upper bound from the data alone.

Acknowledgements: This post was written as part of an AI strategy research mentorship program which occurred during the spring of 2023 between researchers at Epoch AI and students from the Harvard AI Safety Team. Thanks to the Epoch team, and in particular to Pablo Villalobos for mentoring me as well as Jaime Sevilla, Matthew Barnett, Carl Guo, Kuhan Jeyapragasan, Robi Rahman, Fiona Pollack, and Eleni Shor for feedback and assistance.

Epistemic Status: >95% confidence that the model does not accurately describe past decisions about the lengths of training runs, 85% confidence that this will remain the case going forward based on ~50 hours of work.

Introduction

In light of recent political attention to the issues posed by advanced AI systems, proposals for how we might monitor and regulate such systems have become more concrete. For example, in “How To Catch A Chinchilla,” Yonadav Shavit proposes a method by which we might detect large training runs so as to enforce further regulations on them. Similarly, technical AI safety researchers such as those working at ARC Evals h...
Progress Studies Fellowship looking for members
by jay ram — 2023-07-06T17:41:19.125Z

Hi! We are a group of students who have been working on a progress studies fellowship, and we are looking for people who want to join the first cohort. We will be running our first fellowship starting mid-July remotely. You can apply here by July 12, 11:59 PM ET.

The goal of the fellowship is to create a community of individuals committed to exploring and contributing to human prosperity. We want to encourage discussions on technological and scientific innovation and foster a deeper understanding of the mechanisms behind human advancement. Readings will investigate historical examples of progress, the implications of economic growth on moral progress, industrial policy, and the relationship between technological progress and societal change. Visit our website, progressfellowship.info to see the full syllabus. Let us know if you have any additions!

We appreciate any feedback on the syllabus or fellowship!

Best,

Dominic Sobhani, Laura Messner, Keishi Kimura, Jay Ram
BOUNTY AVAILABLE: AI ethicists, what are your object-level arguments against AI notkilleveryoneism?
by Peter Berggren — 2023-07-06T17:32:08.675Z

I am prepared to pay out anywhere between $20 and $100 to AI ethicists of the DAIR/"Stochastic Parrots" school of thought if they provide their object-level arguments against the idea that preventing AI from killing everyone is a real and important issue. This pay will depend on their notability within AI ethics, as well as the clarity and persuasiveness of their arguments.

Conditions for the bounty

The bounty must be claimed by an AI ethicist of the DAIR/"Stochastic Parrots" school of thought. Ethicists from other schools of thought (such as the "what if self-driving cars face trolley problems" school of thought) may be given bounties on a case-by-case basis, but probably not. Any member of DAIR or coauthor of the "Stochastic Parrots" paper counts for this, but people outside of these specific circles may qualify at my discretion, if I believe that their intellectual output is similar to or connected with DAIR or the "Stochastic Parrots" coauthors.
The arguments provided by the claimant must be posted publicly, ideally in the comment section of this thread.
The arguments provided by the claimant must be object-level. This means that they must discuss concrete subjects specific to the issues at hand. This is in contrast to meta-level arguments, which focus on facts about the question (rather than about the issues it addresses), such as difficulties involved in future prediction, the cultural milieu of contemporary AI notkilleveryoneism, the framing of my questions, etc. Note that I have nothing against meta-level arguments; it's just that I've already seen plenty of meta-level arguments by AI ethicists against AI notkilleveryoneism, and I want to see some object-level arguments.
The arguments provided by the claimant must be a good-faith summary of the claimant's actual object-level arguments against AI notkilleveryoneism. For example, "AI notkilleveryoneism is unimportant because paperclips are shiny" will not count, even if made by a qualifying claimant, even though it is object-level. I do not expect that I will need to invoke this condition, but I may do so at my discretion.
The following AI ethicists will be presumptively considered valid claimants, and will fall into the most notable category (meaning that I will pay each of them the maximum $100 bounty assuming they follow all the terms of the bounty, unless I notice loophole abuse):
Emily Bender
Timnit Gebru

...
Layering and Technical Debt in the Global Wayfinding Model
by herschel — 2023-07-06T17:30:52.645Z

Global Wayfinding is a model of meditation and transformative practice, developed by Mark Lippmann. Mark's materials are unfortunately fairly inaccessible, often even to people with both substantial technical and meditation backgrounds. I'd like to make some concepts more available which haven't yet made their way into the discourses I'm in.

Claims in this essay are meant to accurately reflect Mark's models, and I can't quite sharply endorse all of them. In particular I can't attest to many of these claims from my own practice, which is generally the epistemic standard I hold for claims about meditation/phenomenology etc. Also, my pool of anecdata is much smaller than Mark's, and I've forgotten a lot of the details of where various claims and models came from.

Layering

In the Wayfinding framework, a person is said to layer, sometimes layer over, when a compensatory structure is added to prevent, diminish, or attenuate perceived negative parts, behaviors, or emotions. Layering can be variously effective, sometimes completely blocking out a part or a class of parts from coming to conscious awareness. Layering is obviously not causing the structures responsible for the undesired qualities or states to cease, only for their arising in consciousness to be prevented. Sometimes layering is initiated "consciously" but often the layering that people report discovering in their practice is not willful, and they usually have little control over it consciously. Extant layering may also be quite old, and a meditator may have no available memory of the events which led to it. Delayering is a consistently noted phenomenon among serious practitioners. There are many reports of people "discovering" in the course of their practice intense sadness, anger, sexual energy, or many other qualities which might have been covered over or dissociated from.

Many traditional religious and meditative systems prescribe practices to purposefully attenuate various undesirable mindstates. These are variously effective for different people, and though they sometimes work to substantially attenuate negative mental qualities in the moment, they are generally not effective at ultimately dissolving those parts. I find it telling that teachers often prescribe practices to persistently push against negative states. Sometimes traditional teachers recognize that this reflects the fact that purposeful att...
Localizing goal misgeneralization in a maze-solving policy network
by Jan Betley — 2023-07-06T16:21:03.813Z

TLDR: I am trying to understand how goal misgeneralization happens in the same maze-solving network TurnTrout et al. work on. Nothing groundbreaking, but if we are ever to fully understand this model, this is probably an important step. Key findings:

Many channels in the last convolutional layer have a clear interpretation "You should go [up/right/down/left]."
When the cheese is in the top right 5x5, going in the direction indicated by the channels leads to the cheese
When the cheese is outside of the top right 5x5, channels often show the path to the top right instead

These results suggest that goal misgeneralization may be localizable to specific channels that are not robust to out-of-distribution mazes.

This is my capstone project, created during the last edition of the ARENA program. I want to thank @Joseph Bloom and @Paul Colognese for mentorship, @rusheb and @TheMcDouglas for remarks on the draft of this post, many ARENA teachers and participants for fruitful discussions, and authors of procgen-tools for an excellent toolset.

Introduction

Lauro Langosco et al. trained a maze-solving network, a mouse looking for cheese. During the training, the cheese was always in the top right 5x5 part of the maze. When deployed in an environment where the cheese could be anywhere, the mouse sometimes goes to the cheese, sometimes to the top right corner, and sometimes (although very rarely) gets stuck in some unexpected part of the maze.

This model has already appeared on LW, in the article about the cheese vector by TurnTrout et al. and a follow-up post about the top-right corner vector. I recommend looking at the first one - it has a good description of the model (and I will not repeat it here) and a lot of pretty visuals. I don't directly build on their work, but I used tools they developed while working on it. This toolset is excellent - without it, my research would be incomparably more challenging and time-consuming (probably wouldn't happen at all).

The goal of my research can be summarized as "understand why the mouse sometimes decides to go to the cheese and sometimes to the top right corner". My code is here. All the data in this article was generated using scripts in this repository.

Methods

A first step towards building an understanding of a complex model is usually to split it into smaller parts that will be easy to understand separately. There are many diffe...