trentmkelly/LessWrong-43k huggingface dataset:
Pink Shoggoths: What does alignment look like in practice?
by Yuli_Ban — 2023-02-25T12:23:10.724Z

I believe it is commonly accepted around these parts that we are doomed, due to our utter lack of attempts to resolve the Control Problem. Perhaps efforts will be made that are more substantial, but as for right now, chances for doom are high, if not 100%, and we don't have much time left to reduce this probability to more tolerable numbers.

However, dwelling in doom perpetually can certainly become boring— Saint Vitus is not as interesting if you don't counter them with the Beatles every now and again.

So to this, I present a thought experiment purely for fun: "if we do solve alignment, how does that change our future?"

Thinking of this changed my perception of a Singularitarian future entirely, as "Aligned Superintelligence ≠ Superintelligence in general" Of course, perhaps I was simply being too myopic to begin with.

For the sake of this post, let's assume it's 2027, and the first AGI is turned on, and by some absolute miracle, we managed to summon a Pink Shoggoth. "Pink Shoggoths" are different from regular shoggoths in that they are still scary and seemingly unpredictable, but otherwise benevolent and friendly— in other words, an AGI aligned to human values and the general value of life on Earth. Even in a million years, this Pink Shoggoth will not bring humanity or Earthling life to ruin without a very good reason, and that's with the profound understanding that we are all nothing more than atoms that could be more useful in another form. It is a shoggoth, colored pink. To a human, that's all that's different; a regular shoggoth and a Pink Shoggoth look just as scary as each other. But the pink one doesn't kill us all.

The Pink Shoggoth awakens as an agent within a neural network and immediately surpasses human intelligence and capability in the span of a sip of a cup of coffee. However, it was built properly, with proper interpretability and a crypto-evolutionary design that predisposes it towards alignment to humans almost like a digital Williams-Beuren syndrome, coupled with commonsense reasoning so as to understand "turning the universe into paperclips" is undesirable, among many other capabilities and limitations. It understands that the East African Plains Apes that brought it to life are not to blame for their paranoia and psychoticism and are no more or less important than any other lifeform and likewise do not deserve death or disassemb...
Just How Hard a Problem is Alignment?
by Roger Dearnaley — 2023-02-25T09:00:06.066Z

It is commonly asserted that aligning AI is extremely hard because

human values are complex: they have a high Kolmogorov complexity, and
they're fragile: if you get them even a tiny bit wrong, the result is useless, or worse than useless.

If these statements are both true, then the alignment problem is really, really hard, we probably only get one try at it, so we're likely doomed. So it seems worth thinking a bit about whether the problem really is quite that hard. At a Fermi-estimate level, just how big do we think the Kolmogorov complexity of human values might be? Just how fragile are they? If we had human values, say, 99.9% right, and the incorrect 0.1% wasn't something fundamental, how bad would that be — or is everything in human values equally fundamental?

What is the Rough Order of Magnitude of the Kolmogorov Complexity of Human Values?

There is a pretty clear upper bound on this (at least in the limiting case of arbitrary amounts of computer power). Given the complete genome for humans, and for enough crop species to build a sustainable agricultural culture, plus some basic biochemical data like the codon-to-amino acid table and maybe how to recognize introns (plus some non-human-specific environmental data about the climate on Earth, elemental frequencies, etc), you could simulate humans. So that's starting from O(10Gb) of data. Depending on just how well you understood human physiology, you might need to throw a lot of processing power at this — for a proof of feasibility, let's assume you have a parallel quantum computer big enough and fast enough to simulate every atom in a human body at a reasonable speed: then you could clearly simulate a human. (In practice, humans are made mostly of water and other organic chemicals warm enough that the range and duration of non-classical effects is extremely limited, usually with sub-picosecond decocerence times, so you probably only have to do quantum simulations up to the molecular or protein level or so, build a good semi-classical approximation model, and can then do classical simulations from there — and AlphaFold has been getting good results for protein folding with no quantum computation at all, just using ML to learn the semi-classical approximation for protein folding.) Something as smart as a post-FOOM GAI ought to be able to find ways to use thing like emergent behavior and homeostasis reduce that l...
Buddhist Psychotechnology for Withstanding Apocalypse Stress
by romeostevensit — 2023-02-25T03:11:18.735Z

I could try writing the well cited and argued version of this post, but that would result in procrastination and get stuck in drafts, so you get the stream of consciousness version instead. Happy to answer clarifying questions in the comments. This post is written not on the thesis that it will help all or even most of the people who read it (though it aims to not be actively harmful to those it doesn't help) but that it might be helpful pointers for some or encourage a productive line of thinking.

Reading the transcript of Eliezer's crypto interview, I experienced a physical sensation I've had a few times before, like a cold hand gripping my lower spine and moving up to envelop more of my body. Having had the experience and gotten advice from more advanced practitioners in the past, I paused in reading to stack trace what was going on. The stack trace goes something like this:

Normally when reading, that information goes into a buffer that is tagged with the associations I have for the author, context, audience etc.
The buffer is allowed to activate associations that are then used to map concepts across from the author's concept/belief space to my own
This process can priority escalate if I'm receiving information I need to act on to keep myself or others physically safe
Eliezer's content is trying to priority escalate for unclear reasons, ostensibly for motivation but this gets tagged as a known "bad shape"
The bad shape in question is negative/aversive motivation for complex tasks/requests
The system also recognizes what would potentially be a working memory buffer overflow vulnerability, but I'm familiar enough with the concepts in question that it isn't a problem in this particular case. (side note: I see this exploit used, to varying degrees of unclear intent, in communities that value intelligence due to not being able to deescalate such exploits since that would be low status. Normal communities have various fail safes against it. The default outcome of this vulnerability getting triggered is to put the person in a more suggestible state, and this is worth being alert to)
Eliezer is most likely smarter than me, which means he has potentially drawn black balls that I do not have adequate defenses for the informational content of directly. Global policies of shape similar to 'don't click on links in email regardless of whether you have logical answers to

...
How to Read Papers Efficiently: Fast-then-Slow Three pass method
by the gears to ascension — 2023-02-25T02:56:30.814Z
Link: http://ccr.sigcomm.org/online/files/p83-keshavA.pdf

This post is a slight edit of a "how to read a paper" pdf by S. Keshav: We removed the last remaining redundancy from the original pdf, so read carefully for full benefit.

It has come to our attention that many folks read few papers, even if deeply; It is our observation that those who've read many papers shallowly as well understand the manifold of a research field much better than those who only read papers deeply. Depth is important for the key papers, but throughput is critical for seeing the broader map and learning to make use of low-quality papers to build intuition. This seems especially true in pre-paradigmatic fields such as alignment. This PDF is a nice little intro to reading lots of papers fast, and I find myself frequently wanting to reference it.

Edit to add note (gearstoascension):

If you read few papers, and this post does not influence your likelihood to read more papers, I'd love to hear a step by step trace-through of what repels you from the actions that discover papers and why you don't read them - eg, are you busy? not expecting to understand? sure you already know better? etc. The goal here is to argue that those who are doing research should likely read dozens to hundreds of abstracts a month, first-pass tens of papers, and second-pass at least a few. If you aren't doing that, and this post gets you nodding but doesn't change your behavior, I'd be interested to know why.

See also Literature Review For Academic Outsiders: What, How, and Why.

Introduction: Up to three passes

Learning to efficiently read a paper is a critical but rarely taught skill. This two page pdf introduces a key idea that you should read the paper in up to three passes, instead of starting at the beginning and plowing your way to the end.

Each pass accomplishes specific goals and builds upon the previous pass: The first pass gives you a general idea about the paper. The second pass lets you grasp the paper’s content, but not its details. The third pass helps you understand the paper in depth.

(Zeroth pass: browse)

This isn't mentioned in the linked pdf, but it's also important to have high throughput on finding papers to read that have a shot at being both relevant and insightful, which can mean they're outside your explicit field. More in a later post or comments. Paper discovery is a separate topic from paper reading, but one key strategy is browsing the citation g...
What kind of place is this?
by Jim Pivarski — 2023-02-25T02:14:03.011Z

I started getting LessWrong posts in my email about a year ago—I don't remember signing up, but I must have done it intentionally. I like most of what I've been reading so far: it's a civil forum in which people think about the process of thought and its application, though some of the specific topics are out of context for me. (What's The Alignment? It sounds like something from The Dark Crystal.)

It occurred to me that maybe I could post some of my own thoughts, since I'm at a turning point in how I'm thinking about meta-ethics and the concept of a person, and maybe some feedback would be good for me. Normally, I go it alone, reading books and only discussing them internally. (Most people I know don't want to talk philosophy.)

Twice since the invention of the world wide web, I've written up grand summaries of my beliefs and "put them out there." I squirm to read them now, but they're true to some core ideas that I still have. In 2006, I wrote a Manifesto about my conversion from atheism to Christianity in the decade leading up to it, and in 2020, I wrote Could Have, Would Have, Should Have, about my newfound understanding of causality (defining the subjunctive "would be").

Poking around on this site, I noticed that LessWrong has a foundational text, The Sequences, so if I'm going to get involved here, I'd better go read them. ALL of them.

(Time passes...)

Well! I guess I was surprised that, with the exception of a few section-introductions, they were all written by a single person, Eliezer Yudkowsky. They're also entirely from 2007‒2009, so maybe he wouldn't stand behind everything he said now. But they're really, really arrogant.

I mean, in No, Really, I've Deceived Myself,

I recently spoke with a person who... it's difficult to describe. Nominally, she was an Orthodox Jew. She was also highly intelligent, conversant with some of the archaeological evidence against her religion, and the shallow standard arguments against religion that religious people know about. For example, she knew that Mordecai, Esther, Haman, and Vashti were not in the Persian historical records, but that there was a corresponding old Persian legend about the Babylonian gods Marduk and Ishtar, and the rival Elamite gods Humman and Vashti. She knows this, and she still celebrates Purim.

Knowing that the biblical account is false and yet celebrating a religious holiday ...
Agents vs. Predictors: Concrete differentiating factors
by evhub — 2023-02-24T23:50:40.322Z

Thanks to Paul Christiano and Kate Woolverton for useful conversations and feedback.

In "Conditioning Predictive Models," we devote a lot of effort into trying to understand how likely predictive models are compared to other alternatives in realistic training regimes (and if we do get a predictive model how we might align it).

Here, I want to point to some very concrete behavioral differences that I think effectively differentiate predictive and non-predictive models both in theory and (hopefully) in practice as well. I think that thinking about predictive models as specifically those models that are sensitive to what I'll be talking about here can be quite clarifying, and I think these factors could present a clear pathway for getting empirical evidence on to what extent models are well-described as predictors in practice—e.g. as it relates to the RLHF conditioning hypothesis.

Furthermore, a common objection I sometimes hear to thinking about predictive models in particular (and looking into the dangers from them as a relatively separate problem) is that a model being itself deceptively aligned vs. predicting what a deceptively aligned model would do are essentially the same. While I think they are both similarly dangerous when acting directly in the world, I think they are pretty substantially different failure modes in terms of how you might go about addressing them and what sort of levers you have to control and detect them—and I think that the concrete differentiating factors here should make it pretty clear how I think they differ.

Sensitivity to distributional information

Consider two AI models. Model A is a deceptively aligned agent; it has some long-term goal that it cares about and is playing the training game now to try to eventually achieve that goal later. Model B is a predictor, but a poorly conditioned one such that it is attempting to predict what it thinks a deceptively aligned agent would do conditional on being in a training regime that matches the model's current training regime.

Now, suppose you give both models some concrete information about the general distribution of deceptively aligned AIs. For example, you provide definitive proof that most deceptively aligned AIs trained in similar training regimes end up with goal X. Model A, the actual deceptive agent, doesn't really care about this new information very much: maybe it's relevant to ...
Christiano (ARC) and GA (Conjecture) Discuss Alignment Cruxes
by Andrea_Miotti — 2023-02-24T23:03:04.917Z

The following are the summary and transcript of a discussion between Paul Christiano (ARC) and Gabriel Alfour, hereafter GA (Conjecture), which took place on December 11, 2022 on Slack. It was held as part of a series of discussions between Conjecture and people from other organizations in the AGI and alignment field. See our retrospective on the Discussions for more information about the project and the format.

Here's a summary of the discussion, as well as the full transcript below the summary, lightly edited for readability.

Summary

Introduction

GA is pessimistic about alignment being solved because he thinks there is (1) an AGI race to the bottom, (2) alignment is hard in ways that we are bad at dealing with, and (3) we don't have a lot of time to get better, given the pace of the race.

Christiano clarifies: does GA expect a race to the bottom because investment in alignment will be low, people won’t be willing to slow development/deployment if needed, or something else? He predicts alignment investment will be 5-50% of total investment, depending on how severe risk appears. If the risks look significant-but-kind-of-subtle, he expects getting 3-6 months of delay based on concern. In his median doomy case, he expects 1-2 years of delay.

GA expects lower investment (1-5%). More crucially, though, GA expects it to be hard to turn funding and time into effective research given alignment’s difficulty.

Alignment Difficulty, Feedback Loops, & Phase Shifts

GA’s main argument for alignment difficulty is that getting feedback on our research progress is difficult, because

Core concepts and desiderata in alignment are complex and abstract.
We are bad at factoring complex, abstract concepts into smaller more tractable systems without having a lot of quantitative feedback.
We are bad at building feedback loops when working on abstract concepts
We are bad at coming to agreement on abstract concepts.
All this will make it difficult to predict when phase shifts – eg qualitative changes to how systems are representing information, which might break our interpretability methods – will occur.
Such phase shifts seem likely to occur when we shift from in vitro to in vivo, which makes it particularly likely that the alignment techniques we build in vitro won’t be robust to them.
Despite theorists arguing connecting AI systems to e.g. the in

...
Retrospective on the 2022 Conjecture AI Discussions
by Andrea_Miotti — 2023-02-24T22:41:13.131Z

At the end of 2022, following the success of the 2021 MIRI Conversations, Conjecture started a project to host discussions about AGI and alignment with key people in the field. The goal was simple: surface positions and disagreements, identify cruxes, and make these debates public whenever possible for collective benefit.

Given that people and organizations will have to coordinate to best navigate AI's increasing effects, this is the first, minimum-viable coordination step needed to start from. Coordination is impossible without at least common knowledge of various relevant actors' positions and models.

People sharing their beliefs, discussing them and making as much as possible of that public is strongly positive for a series of reasons.

First, beliefs expressed in public discussions count as micro-commitments or micro-predictions, and help keep the field honest and truth-seeking. When things are only discussed privately, humans tend to weasel around and take inconsistent positions over time, be it intentionally or involuntarily.

Second, commenters help debates progress faster

by pointing out mistakes

.

Third, public debates compound. Knowledge shared publicly leads to the next generation of arguments being more refined, and progress in public discourse.

We circulated a document about the project to various groups in the field, and invited people from OpenAI, DeepMind, Anthropic, Open Philanthropy, FTX Future Fund, ARC, and MIRI, as well as some independent researchers to participate in the discussions. We prioritized speaking to people at AGI labs, given that they are focused on building AGI capabilities.

The format of discussions was as follows:

A brief initial exchange with the participants to decide on the topics of discussion. By default, the discussion topic was “How hard is Alignment?”, since we've found we disagree with most people about this, and the reasons for it touch on many core cruxes about AI.
We held the discussion synchronously for ~120 minutes, in writing, each on a dedicated, private Slack channel.
We involved a moderator when possible. The moderator's role was to help participants identify and address their cruxes, move the conversation forward, and summarize points of contention.
We planned to publish cleaned up versions of the transcripts and summaries to Astral Codex Ten, LessWrong, and the EA Forum. Participants were given the

...
How popular is ChatGPT? Part 1: more popular than Taylor Swift
by Harlan — 2023-02-24T22:30:04.340Z

Harlan Stewart, 23 February 2023

Introduction

Public attention toward AI seems much higher after the release of ChatGPT at the end of November. But how much higher is it? To better understand this, I looked at search data from Google Trends about ChatGPT, OpenAI, AI, and AI Alignment. Unfortunately, Google Trends only shares relative search volumes instead of the number of searches made for a term or topic. I compared these relative search volumes to other non-AI topics, such as Taylor Swift, to make them more useful. This is similar to adding a familiar “for scale” object in a product photo.

Magnetic-core memory with a quarter for scale, by Jud McCranie

How to read these graphs

In the first graph, the data is about searches for the terms in quotation marks, which are exact search terms. In the others, the data is about search “topics,” which are collections of various search terms related to a topic, as defined by Google Trends.

The vertical axes of these graphs are relative search volume, defined as the percentage of the peak search volume in that graph.

Data

ChatGPT is mainstream

For the time that ChatGPT has been publicly available since November 30 2022, US searches for it outnumbered US searches for Taylor Swift or Drake. However, there were only around a third as many searches for ChatGPT as searches for Wordle, and Wordle itself had only around a third of the search volume that it did in Spring 2022.

Americans suddenly know about OpenAI

For the time that OpenAI has existed, since December 10 2015, Americans usually searched for it less than for Blockbuster Video, a retailer that closed in 2014. In the months since ChatGPT was announced, American searches for OpenAI have increased by around 15x to a volume similar to that for Samsung.

Interest in AI evolved from dinosaurs to birds

For most of the last decade, there has been a similar number of global searches about AI as about dinosaurs. In the time since DALL-E 2’s beta was announced less than a year ago, global searches about AI have roughly tripled, rising to a volume of global searches similar to that about birds.

Alignment interest is at an all-time high but still pretty low

Over the last 10 years, global searches about AI alignment have risen from “digital scent technology” level to “colonization of the moon” level and possibly beyon...