trentmkelly/LessWrong-43k huggingface dataset:
Wikipedia as an introduction to the alignment problem
by SoerenMind — 2023-05-29T18:43:47.247Z
Link: https://en.wikipedia.org/wiki/AI_alignment

AI researchers and others are increasingly looking for an introduction to the alignment problem that is clearly written, credible, and supported by evidence and real examples. The Wikipedia article on AI Alignment has become such an introduction.

Link: https://en.wikipedia.org/wiki/AI_alignment

Aside from me, it has contributions from Mantas Mazeika, Gavin Leech, Richard Ngo, Thomas Woodside (CAIS), Sidney Hough (CAIS), other Wikipedia contributors, and copy editor Amber Ace. It also had extensive feedback from this community.

In the last month, it had ~20k unique readers and was cited by Yoshua Bengio.

We've tried hard to keep the article accessible for non-technical readers while also making sense to AI researchers.

I think Wikipedia is a useful format because it can include videos and illustrations (unlike papers) and it is more credible than blog posts. However, Wikipedia has strict rules and could be changed by anyone.

Note that we've announced this effort on the Wikipedia talk page and shared public drafts to let other editors give feedback and contribute.

I you edit the article, please keep in mind Wikipedia's rules, use reliable sources, and consider that we've worked hard to keep it concise because most Wikipedia readers spend <1 minute on the page. For the latter goal, it helps to focus on edits that reduce or don't increase length. To give feedback, feel free to post on the talk page or message me. Translations would likely be impactful.
What are some of the best introductions/breakdowns of AI existential risk for those unfamiliar?
by Isaac King — 2023-05-29T17:04:40.384Z

Discussion of AI risk has gone mainstream in the past few months, and as usual whenever that happens, many of the people new to the field think they're an expert and proceed to engage in the exact same arguments over and over. I think it would be convenient to have the following available:

An introduction to the core risk, explaining why it's possible.
An index of common bad arguments about AI risk and in-depth responses to them.
An index of common good arguments about AI risk, and links to further reading about them.

All of these should require no background knowledge and be accessible to normal people with no philosophical or mathematical experience.

I was thinking of writing up such a guide, but I don't want to duplicate effort. Does anything like this already exist?
Creating Flashcards with LLMs
by Diogo Cruz — 2023-05-29T16:55:27.087Z

Summary

Current LLMs, in particular GPT-4, are sufficiently capable to produce good non-trivial flashcards covering the contents of a given text.
It is then possible to automate the flashcard creation process, making it far less time consuming to use Spaced Repetition techniques to study new material.
In this post, I explain how you can apply this technique, and show its result when applied to the whole "Rationality: From AI to Zombies" sequences, and to scientific papers.

Why automate?

I've been using Spaced Repetition techniques for the past 12 years, mainly through Anki. It is a common mantra among Spaced Repetition advocates that you should create your own flashcards, if you want to learn something new, as it helps you to consolidate the information you're putting into the card. While that is certainly true, it is far more questionable whether it is an efficient method to strengthen your memory.

From a time-saving point of view, this makes little sense. Summarization, which is mainly what you do when creating a card by hand, is a relatively poor learning technique, particularly when compared with Distributed Practice, the whole schtick of Spaced Repetition[1].

From this paper.

If you spend, on average, 10 seconds reviewing a card, that card will require a total of 90 seconds of review time in the next 5 years, on average (with Anki's default scheduling).

Creating a card takes longer than that. You first need to filter the important ideas from the text you are interested in. Then, you need to express them in a way that is suitable[2] for a flashcard, and that leads you to remember what you actually want, and not some proxy information. Finally, you actually need to create the card. How long all of this takes depends on the particular text, but in my experience, it averages to more than 90 seconds. Especially for cards from complicated contexts, creating the card may take more time than all the reviews combined.

In general, there are three important things to consider:

How relevant is a deck of flashcards for what you actually want to learn or remember, versus some proxy objective;
What is the deck's quality - we may consider this to be how well the deck does the job it set out to do;
Finally, how much time did it take you to create or adapt the deck for your use case.

It is easy to create cards quickly, if you are willing to sacrifice their quality. It is al...
Minimum Viable Exterminator
by Richard Horvath — 2023-05-29T16:32:59.055Z

Defining Minimum Viable Exterminator

Most (all?) LessWrong writings dealing with AI Doom assume the existence of Artificial General Intelligence, or more likely, Superintelligence is required for AI caused x-risk event. I think that is not strictly the case. I think it is possible that a "Minimum Viable Exterminator" can exist, which is less than what most people would consider AGI, yet what is already capable of killing everyone.

A Minimum Viable Exterminator (MVE) is an agent that has at least the following characteristics:

1. The ability to design a method that can be used to destroy mankind
2. The ability to act in the real word and execute (create) the method
3. It can accept extermination (destroying human civilization) as (one of) its permanent goal(s)

Representation of capabilities overlap

*
a. Average humans are not capable to kill all humans
b. MVE can kill all humans, but does not have all capabilities a human has c. AGI/SI can do both capability-wise
*Technically red and orange area should be identical as it is "Minimum", but find this is a better practical representation, plus in reality MVE would have additional capabilities, not just the bare minimum

Elaboration of MVE characteristics

1. AI designing the method

In most scenarios this is a microorganism, but it is generally assumed that AGI/Superintelligence should be able to design multiple different methods humans are not even capable thinking of. This is due to it's better understanding of reality: If it has a better model how biology/chemistry/physics work than humanity, it can create biological/chemical/radiological/etc weapons that are more effective than what humanity posses.

Probably the most important factor in MVE is the realization that this can be done with a domain specific intelligence, and it need not be "general" at all. It may be "superintelligence" compared to a human in the same way as a calculator is "superintelligence" by being better at arithmetic than any human.

Such neural networks (superior to humans in specific hard science domains) already exist, for example:

AI has designed bacteria-killing proteins from scratch – and they work
AI Drug Discovery Systems Might Be Repurposed to Make Chemical Weapons, Researchers Warn
‘It will change everything’: DeepMind’s AI makes gigantic leap in solving protein structures
Physics Breakthrough as AI Successfully Controls Plasma in Nuc

...
An LLM-based “exemplary actor”
by Roman Leventov — 2023-05-29T11:12:50.762Z

Into and summary

This post is the second section of "Aligning an H-JEPA agent via training on the outputs of an LLM-based "exemplary actor", posted separately because I think it could warrant a separate discussion, largely independent of the discussion of H-JEPA agent with GFlowNet actors. Here's the summary of this post, copied from the "Overview" section of the main article:

In section 2, I describe the “exemplary actor”, an LMCA (language model cognitive architecture) that takes a simple, “brute force” approach to alignment: a powerful LLM (think GPT-5/6 level, with a vast, or quasi-unlimited context) is given a list of “approved” textbooks on methodological and scientific disciplines: epistemology, rationality, ethics, physics, etc. Also, the LLM is given tools: narrow AIs (such as for protein folding or for predicting properties of materials, or for formal scientific modelling). Finally, the LLM is given a compute engine such as Wolfram and a knowledge base such as Wikidata or Wolfram Knowledgebase.

The exemplary actor creates plans or predictions for given situations (described in language and fed to the LLM underlying the exemplary actor as prompts) and iteratively critiques and refines its own plans and predictions while putting different textbooks into the LLM context (first, with the textbook on rationality, then epistemology, then physics, etc., with potentially dozens of different textbooks relevant for a plan or prediction that is being criticised), for many iterations, until convergence.

In section 2.1, I note that the type of alignment that the exemplary actor’s architecture tries to ensure is called (world) model alignment and that is stronger and also more essential than goal alignment.

Then, I discuss the properties of the exemplary actor. In section 2.2., I discuss what I see as likely non-issues or straightforwardly addressable issues: the “divergent reasoning nature” of LLMs, the lack of grounded common sense reasoning, and the bias of the quick reactive network (”System 1”), it it is added to the architecture to make it more practically usable in lower-stakes reasoning settings.

In section 2.3, I discuss the outstanding technical issues and risks of the exemplary actor’s architecture:

The risk of direct access to the underlying LLM (section 2.3.1).
The exemplary actor’s reasoning could still be partially directed by “alien” thinking patterns (i

...
Aligning an H-JEPA agent via training on the outputs of an LLM-based "exemplary actor"
by Roman Leventov — 2023-05-29T11:08:36.289Z

1. Overview and conclusion

In section 2, I describe the “exemplary actor”, an LMCA (language model cognitive architecture) that takes a simple, “brute force” approach to alignment: a powerful LLM (think GPT-5/6 level, with a vast, or quasi-unlimited context) is given a list of “approved” textbooks on methodological and scientific disciplines: epistemology, rationality, ethics, physics, etc. Also, the LLM is given tools: narrow AIs (such as for protein folding or for predicting properties of materials, or for formal scientific modelling). Finally, the LLM is given a compute engine such as Wolfram and a knowledge base such as Wikidata or Wolfram Knowledgebase.

The exemplary actor creates plans or predictions for given situations (described in language and fed to the LLM underlying the exemplary actor as prompts) and iteratively critiques and refines these plans and predictions while putting different textbooks into the LLM context (first, with the textbook on rationality, then epistemology, then physics, etc., with potentially dozens of different textbooks relevant for a plan or prediction that is being criticised), for many iterations, until convergence.

In section 2.1, I note that the type of alignment that the exemplary actor’s architecture tries to ensure is called (world) model alignment and that is stronger and also more essential than goal alignment.

Then, I discuss the properties of the exemplary actor. In section 2.2., I discuss what I see as likely non-issues or straightforwardly addressable issues: the “divergent reasoning nature” of LLMs, the lack of grounded common sense reasoning, and the bias of the quick reactive network (”System 1”), which should be added to the architecture to make it more practically usable in lower-stakes reasoning settings.

In section 2.3, I discuss the outstanding technical issues and risks of the exemplary actor’s architecture:

The risk of direct access to the underlying LLM (section 2.3.1).
The exemplary actor’s reasoning could still be partially directed by “alien” thinking patterns (i.e., the world model) of the underlying LLM even though these influences won’t surface in the explanations of the plan (section 2.3.2).
Iterated critique and refinement probably won’t make plans strictly conforming to the theories described in the textbooks (section 2.3.3).

In section 2.3.4, I discuss the alignment tax of the exemplary actor (comp...
Gemini will bring the next big timeline update
by p.b. — 2023-05-29T06:05:52.583Z

There is a genre of LLM critique that criticises LLMs for being, well, LLMs.

Yann LeCun for example points to the inability of GPT-4 to visually imagine the rotation of interlocking gears as a fact that shows how far away AGI is, instead of a fact that shows how GPT-4 has not been trained on video data yet.

There are many models now that "understand" images or videos or even more modalities. However, they are not end-to-end trained on these multiple modalities. Instead they use an intermediary model like CLIP, that translates into the language domain. This is a rather big limitation, because CLIP can only represent concepts in images that are commonly described in image captions.

Why do I consider this a big limitation? Currently it looks like intelligence emerges from learning to solve a huge number of tiny problems. Language seems to contain a lot of useful tiny problems. Additionally it is the interface to our kind of intelligence, which allows us to assess and use the intelligence extracted from huge amounts of text.

This means that adding a modality with a CLIP-like embedding and than doing some fine-tuning does not add any intelligence to the system. It only adds eyes or ears or gears.

Training end-to-end on multi-modal data should allow the model to extract new problem solving circuits from the new modalities. The resulting model would not just have eyes, but visual understanding.

Deepmind did a mildly convincing proof-of-concept with Gato last year, a small transformer trained on text, images, computer games and robotics. Now it seems they will try to scale Gato to Gemini, leapfrogging GPT-4 in the process.

GPT-4 itself has image processing capabilities that are not yet available to the general public. But whether these are an add-on or result of integrated image modelling we don't know yet.

To me it seems very likely, that a world where the current AI boom fizzles is a world where multi-modality does not bring much benefits or we cannot figure out how to do it right or possibly the compute requirements of doing it right is still prohibitive.

I think Gemini will give us a good chunk of information about whether that is the world we are living in.
Proposed Alignment Technique: OSNR (Output Sanitization via Noising and Reconstruction) for Safer Usage of Potentially Misaligned AGI
by sudo — 2023-05-29T01:35:51.616Z

Acknowledgements

Thanks to Joe Benton for proposing the analogy to diffusion models.

Thanks to Alex Spies, Erik Winsor, Lee Sharkey, and many others for deeply valuable and insightful discussion as well as red-teaming of this technique during EAG London.

Thanks to Justis (via LessWrong Feedback), Scott Viteri, and others for feedback on early versions of this post.

Thanks to Vivek Hebbar for fruitful conversation which influenced the discussion of probable failures and future steps.

This idea was originally formulated as part of an application to Vivek Hebbar's SERI MATS stream. Thanks to Vivek Hebbar and the SERI MATS team.

TL;DR

Given a potentially misaligned, highly intelligent plan-generating model M (Input: Task Description, Output: List of Actions to Accomplish Task in Natural Language), we propose a method for safer usage of M's outputs to accomplish ambitious tasks.

The method is as follows:

Instruct the model to generate a plan for accomplishing some task;
Algorithmically apply noise to the plan; (Primary Purpose: Prevent subtle human hijacking)
Use humans to repair the plan; (Primary Purpose: Push the plan onto the human distribution)
Execute the plan.

Key Insights

This proposal is motivated and justified by the following intuitions:

Brittleness: Since treacherous plans may require higher precision than benign plans, treacherous plans should be more vulnerable to noise. Since the human repair process is unlikely to perfectly reproduce the original plan, treacherous plans produced by M should be asymmetrically less likely to succeed after this process.
Distributional Shift: The distribution of human-generated plans is more likely to be benign than the distribution of AI-generated plans. As such, plans which are reconstructed by humans should be "safer" than M's unmodified outputs.

Note on The Human Distribution

Requiring human reinterpretation of plans should selectively degrade plans which rely on human-illegible insights. I am hopeful that human-illegibility is a common feature of human-unfriendly plans. However, it may be the case that human-illegibility may also be a feature of many ambitious human-friendly plans. If we use the proposed technique during training and/or fine-tuning, my hope is that models will be heavily encouraged to produce human legible plans. I am optimistic that there exists world-saving plans which are human legible.

Further Assumptions...
Morality is Accidental & Self-Congratulatory
by ymeskhout — 2023-05-29T00:40:59.354Z

The Selfish Gene remains one of my favorite books of all time and although published in 1976 it remains a compelling and insightful introduction to the brute mechanics of natural selection. Richard Dawkins later acknowledged that his book’s title may give a misleading impression of its thesis, erroneously ascribing conscious motivations or agentic properties to non-sentient strands of DNA. The core argument is dreadfully simple: genes (as opposed to organisms or species) are the primary unit of natural selection, and if you leave the primordial soup brewing for a while, the only genes that can remain are the ones with a higher proclivity towards replication than their neighbors. “Selfish” genes therefore are not cunning strategists or followers of some manifest destiny, but rather simply the accidental consequence of natural selection favoring their propagation. Nothing more.

Dawkins is responsible for coining the word ‘meme’ in the book to describe how the same principles behind gene replication can apply to ideas replicating. I thought about this when I read WoodFromEden’s post about the origin of patriarchy.[1] Their explanation for why male dominance persisted historically for so long is elegantly tidy:

Men make war. Or rather, groups of men make war. The groups that were good at making war remained. The groups that were less good at making war perished. That way, human history is a history of successful male military cooperation. Groups with weak male bonding were defeated by groups where men cooperated better.

Here too, there is no dirigible trajectory mapped out ahead of time. Cultural values which valorize physical male violence and facilitate its coordination at scale will become the dominant paradigm purely as a result of the circumstances’ ruthless logic. Any deviation from this set of values would lead your tribe towards extinction, which accidentally also meant your bards wouldn’t be around to write songs and poems extolling the virtues of sex equality. At least not until there have been an extensive change in circumstance.

This “security dilemma” may have been borne out of petty squabbles over hunting grounds in the Serengeti but its ramifications persisted throughout history. Military service today may be seen as a low-status and distasteful profession — quite literally grunt work — but it used to be venerated deeply as a path to honor and a ...
TinyStories: Small Language Models That Still Speak Coherent English
by Ulisse Mini — 2023-05-28T22:23:30.560Z
Link: https://arxiv.org/abs/2305.07759

Abstract

Language models (LMs) are powerful tools for natural language processing, but they often struggle to produce coherent and fluent text when they are small. Models with around 125M parameters such as GPT-Neo (small) or GPT-2 (small) can rarely generate coherent and consistent English text beyond a few words even after extensive training. This raises the question of whether the emergence of the ability to produce coherent English text only occurs at larger scales (with hundreds of millions of parameters or more) and complex architectures (with many layers of global attention). In this work, we introduce TinyStories, a synthetic dataset of short stories that only contain words that a typical 3 to 4-year-olds usually understand, generated by GPT-3.5 and GPT-4. We show that TinyStories can be used to train and evaluate LMs that are much smaller than the state-of-the-art models (below 10 million total parameters), or have much simpler architectures (with only one transformer block), yet still produce fluent and consistent stories with several paragraphs that are diverse and have almost perfect grammar, and demonstrate reasoning capabilities. We also introduce a new paradigm for the evaluation of language models: We suggest a framework which uses GPT-4 to grade the content generated by these models as if those were stories written by students and graded by a (human) teacher. This new paradigm overcomes the flaws of standard benchmarks which often requires the model's output to be very structures, and moreover provides a multidimensional score for the model, providing scores for different capabilities such as grammar, creativity and consistency. We hope that TinyStories can facilitate the development, analysis and research of LMs, especially for low-resource or specialized domains, and shed light on the emergence of language capabilities in LMs.

Implications

Interpretability

One part that isn't mentioned in the abstract but is interesting:

We show that the trained SLMs appear to be substantially more interpretable than larger ones. When models have a small number of neurons and/or a small number of layers, we observe that both attention heads and MLP neurons have a meaningful function: Attention heads produce very clear attention patterns, with a clear separation between local and semantic heads, and MLP neurons typically activated on tokens that have a clear com

...