trentmkelly/LessWrong-43k huggingface dataset:
The Hubinger lectures on AGI safety: an introductory lecture series
by evhub — 2023-06-22T00:59:27.820Z
Link: https://www.youtube.com/playlist?list=PLtlVeM84bZ6RLSR6oaQnbZ7FSwb-hkapx

In early 2023, I (Evan Hubinger) gave a series of recorded lectures to SERI MATS fellows with the goal of building up a series of lectures that could serve as foundational introductory material to a variety of topics in AGI safety. Those lectures have now been edited and are available on YouTube for anyone who would like to watch them.

The basic goal of this lecture series is to serve as longform, in-depth video content for people who are new to AGI safety, but interested enough to be willing to spend a great deal of time engaging with longform content, and who prefer video content to written content. Though we already have good introductory shortform video content and good introductory longform written content, the idea of this lecture series is to bridge the gap between those two.

Note that the topics I chose to include are highly opinionated: though this is introductory material, it is not intended to introduce the listener to every topic in AI safety—rather, it is focused on the topics that I personally think are most important to understand. This is intentional: in my opinion, I think it is far more valuable to have some specific gears-level model of how to think about AI safety, rather than a shallow overview of many different possible ways of thinking about AI safety. The former allows you to actually start operationalizing that model to work on interventions that would be valuable under it, something the latter doesn't do.

The lecture series is composed of six lectures, each around 2 hours long, covering the topics:

Machine learning + instrumental convergence

Risks from learned optimization

Deceptive alignment

How to evaluate alignment proposals

LLMs + predictive models

Overview of alignment proposals

Each lecture features a good deal of audience questions both in the middle and at the end, the idea being to hopefully pre-empt any questions or confusions the listener might have.

The full slide deck for all the talks is available here.

EDIT: As a follow-up to the above lecture series covering some of my more recent work and thoughts on deceptive instrumental alignment, I would also recommend this talk I gave at FAR.
How to Search Multiple Websites Quickly
by Nicholas / Heather Kross — 2023-06-22T00:42:41.598Z

Imagine you google the word "eigenvalue". But instead of one list of search results, you automatically get a bunch of new browser tabs. In each tab, is the page about eigenvalues on each of multiple whitelisted sites. Wikipedia, nLab, Wolfram Mathworld, reddit, and LessWrong, for example.

Does this sound useful? Like it could e.g. give you multiple explanations of the same topic, all gathered together, quickly?

Turns out, this functionality exists.

Multiple browsers have extension "app stores", and one type of extension is a "multisearch" or a "multi tab search".

A quick example

I found this extension for Chrome. There are almost certainly similar ones for Firefox and other browsers. The below instructions are for that specific extension.

To use it:

Install the extension.

Click the extension's icon in your browser, and go to its "Options".

Enter URLs to search, with the "%s" operator being what you'd put your query into.

An example category contains a list of sites I want to multi-tab-search about

[Google Feeling Lucky] brilliant.org %s
https://en.wikipedia.org/w/index.php?title=Special:Search&search=%22%s%22&ns0=1&fulltext=Search
[Google Feeling Lucky] ncatlab.org %s
[Google Feeling Lucky] wolfram mathworld %s
[Google Feeling Lucky] khan academy %s
[Google Feeling Lucky] proofwiki.org %srg %s

(Note: the "[Google Feeling Lucky]" option unfortunately doesn't actually open the first search-result, it just googles it for you)

Open a new tab, press "m" and then "space" before typing your query. (For this extension, you need to type "m" before you type another custom key for a different "category". So if you use "x" for a "rationality" category, you'd press m-space and then type "x orthogonality thesis" to multi-tab-search for the orthogonality thesis.)

Lots of new tabs will open, one for each site!

This is much easier to setup, but less useful/complete, than my "multi-explanation math wiki" idea. Consider it an 80/20 minimum-viable-setup for verifying the "multiple explanations = better understanding" idea.

If you find this useful, you might leave multi-site query lists that you ended up using, in the comments.
Newbie questions about information theory and transformers
by Misaligned-Semi-intelligence — 2023-06-21T22:45:21.895Z

I have some very naive (and unfortunately sort of stream-of-consciousness) questions brought up by this post:

https://www.lesswrong.com/posts/vAAneYowLkaHnihCg/textbooks-are-all-you-need

I definitely know that my ideas are poorly formed and I have no idea what I'm talking about. I'm happy to be told that my whole picture here is fundamentally flawed, or that there's some better question that I should be asking, or that a combination of words I've thrown together is gibberish. But I'm trying to avoid worrying about asking dumb things when I'm curious, so I'm just going to put them out there.

Is there much I can read on the relationship between information theory and transformers/LLMs?

I keep trying to picture the flow of information in transformers. Is the information being "distilled"?

I'm imagining the weights and biases of the model to be some kind of distilled and encoded version of the training data. Does that make any sense? Since it can't store all of it, what is it doing? Incorporating more information from the tokens in the training data that have the highest entropy (as far as how much it reduces the average uncertainty of the correct next token)?

If so, is this driven by gradient descent? Does minimizing the cost function end up encoding more information from the training set tokens that contain the most entropy into the weights of the model?

Does that information get further distilled into the tokens it generates from a prompt? Is there some way in which having the model generate a bunch of new training data is "unencoding" the information about the original training set stored in the weights, and in doing so producing a new, smaller training set that represents the highest entropy information from the original?

Does this improve quality because it's letting the network try "reconnecting" itself when it's re-encoded into a new model being trained on it? Could this help fix things that were stuck in a local minima in the original model?

When the LLM selects tokens from the large set, does that choice of tokens encode information from that LLM's training in addition to the information from the selected tokens themselves?

If we start caring a lot about the ability of an LLM to select the best tokens from a huge dataset and generate a complimentary set of synthetic data on top of its general functionality, would this create s...
Progress links and tweets, 2023-06-21: Stewart Brand wants your comments
by jasoncrawford — 2023-06-21T20:52:14.897Z
Link: https://rootsofprogress.org/links-and-tweets-2023-06-21

Opportunities

Dwarkesh Patel is hiring a COO for his podcast, The Lunar Society
Join a new campaign for nuclear regulatory reform (via @BuildNuclearNow)

Announcements

Stewart Brand is serializing his book on Maintenance and inviting your comments
Material World: The Six Raw Materials That Shape Modern Civilization, a new book by Ed Conway (via @lewis_dartnell). Conway also has a blog
Dan Romero is offering Twitter subscriptions, will donate it to The Roots of Progress (thank you, Dan!)
The OpenAI API now supports function calling (via @gdb)
Waymo now covers all of San Francisco 24/7

Links

A coffee mug full of nucleic acids could store all the data from the last two years
Why Mark Lutter is building a 21st Century Renaissance charter city in the Caribbean (via @JeffJMason). A followup to the interview in last week’s digest

Queries

A curriculum that motivates science through engineering/technology projects?
Who should Anastasia talk to at the forefront of ambitious cardiology research?
What’s the earliest depiction of nuclear waste as green ooze in a drum?
Books on all things economics & finance? Seeking a really wide overview

Quotes

Speculation from 1900 about how short skirts might help prevent disease
A complaint about tech stagnation… from 1926

Tweets & threads

AI “decelerationists”
The “good old days” of anything was always 50 years ago (via @johanknorberg)
The experiment that measured the rotation speed of ATP synthase
Should we destroy all malaria-carrying mosquitoes now?
ChatGPT seems to have gotten worse? (Not the only similar comments I’ve seen)
“I am only afraid of death since I became convinced that it is optional”
The O-1 and EB-1 are wildly underused visa categories for top talent
Individualized education will reduce variance at the bottom but increase it at the top
Lectures are better than “interactive discussions”
When Lydia Williams died at age 43 in 1858, only five of her ten children were still alive. “Of all 10, just two survived to adulthood. This used to be normal”
What - ideally - should young and intelligent people do?
by veterxiph — 2023-06-21T20:21:53.725Z

This is my first post. I'm 21. From what I understand, my fluid intelligence will rise until the age of 26 and then slowly fall. So I'm in a great position now to positively contribute to humanity.

I feel the need - at least now - to devote my life to something that I think actually matters to humanity and helps and/or saves as many humans as possible.

I think it might be a good idea to start off with my basic point of view: I want as many humans to survive as possible and live genuinely happy, satisfying, healthy, and fulfilling lives. I've heard arguments that humans have zero free will; I'm not sure whether I believe this or not, but then again I haven't thought a ton about it. I'm leaning towards believing it. So, because I think it's probably the case that humans have zero free will, I think everyone deserves to be genuinely happy, satisfied, healthy, and fulfilled. Yes, even criminals - they didn't choose to commit their crimes if there's zero free will, right? No. So they deserve to be very happy just like the "good" people do.

My question is, what should someone in my position do? I'm hearing arguments that perhaps humanity should stop AGI development altogether. This isn't an arms race anymore; it's a suicide race, as many have said.

But we then naturally face an extremely important question: Is it even reasonable to expect humanity to stop developing AGI? It's one thing to get all countries and groups on Earth to agree to stop developing AGI - that might already be too difficult for all I know. But it's then another thing to actually enforce that - airstrikes to destroy places where AGI is being developed? How long could this "airstrike" tactic actually work, for example? I'm assuming it actually can work in the first place, which might be incorrect.

If airstrikes stop being an effective anti-AGI-development tool for whatever reason, could we use or invent another anti-AGI-development tool that is effective? How long until a country or group figures out a way around that, though? Could a government or other group figure out how to secretly build AGI? Isn't it only a matter of time until that happens?

Also, I'm pretty ignorant about this: how much could the people at OpenAI, Deepmind, and Anthropic be hiding? What does this tiny group of elities know that we don't? Is it possible that they have already agreed to create AGI and have agreed to use it ...
Using Claude to convert dialog transcripts into great posts?
by mako yass — 2023-06-21T20:19:44.403Z

Dialogs are crucial for identifying errors, developing syntheses, and surfacing cruxes and critical uncertainties. They're also much easier to produce than structured writing or presentations, so we would like to think that they're also a good way of getting important ideas articulated and written up, but they're kind of not, because real dialogs meander and stumble, they take place at the limits of our ability to make ourselves understood, they're the meeting of views that don't yet know their synthesis. Usually, mutual understanding arrives only towards the end, after many failed attempts. I've also found that there seems to be a tradeoff between interpersonal or professional functionality and legibility to outside observers: the more the participants are speaking to each other, the less the participants are thinking about an audience, denser, more personal, more idiosyncratic. Dialog narrows in on the most important issues in a field but then only produces long-winded, hostile documents.

Claude is a chat assistant from Anthropic. Its content window fits like 100,000 tokens. We may be able to use it to rapidly, cheaply boil dialogs down into readable texts. If this works, this could be genuinely transformative: There would no longer need to be this frustrating gulf between the field's understanding of its own state of the art, and the field's outward-facing materials and documentation.

To test this, I decided try it on a dialog that many of us will have read, Ngo and Yudkowsky on Alignment Difficulty.

Well, here's my prompt:

The following is a transcript of a conversation primarily between Eliezer Yudkowsky and Richard Ngo. It was a conversation on discord. It's interspersed with section titles. Please read it and then summarize its most important insights for me in less than 900 words.

[TRANSCRIPT QUOTE BEGIN]

<the transcript>

[TRANSCRIPT QUOTE END]

I don't know whether I'm good at prompts, tell me whether this makes sense. The delimiters are there to discourage Claude from responding to instructions in the conversation itself, or attempting to continue the conversation, like, I'm hoping [TRANSCRIPT QUOTE END] sorta jolts it all the way back to remembering that it was asked a question and all it's supposed to do is answer that.

(Edit: I did some experimentation and yeah my intuition was right, omitting the delimiters seems to lead it to focus its attention p...
Which personality traits are real? Stress-testing the lexical hypothesis
by tailcalled — 2023-06-21T19:46:03.164Z

This post is also available on my Substack. Thanks to Justis Mills for proofreading and feedback!

Most scientific personality models are, directly or indirectly[1], based on the lexical hypothesis, which roughly speaking states that there is a correspondence between important personality traits and abstract behavior-descriptive adjectives. For example, the Big Five was created by having people rate themselves using words like "outgoing", "hard-working" and "kind", and finding patterns in these. It is neat that one can create models in this way, but the large amount of abstraction involved by using abstract adjectives raises huge questions about how "real" the personality traits are.

I have created a new personality test, currently named Targeted Personality Test. I have multiple goals with this test, but one of them is to investigate which personality traits are “real”[2] without relying on the lexical hypothesis. I do this mainly by assessing lots of specific narrow behaviors, rather than abstract vague adjectives.[3]

By the end of this blog post, I hope to have introduced some concepts that makes my approach make sense, and thereby enable you to understand this diagram I made summarizing my results:

The semi-formal understanding of what is going on in this chart is very long, so before we proceed, let me give a brief, vague indication of what you will be informed about:

Trait impact: a measure of how strongly the personality trait influences the various behaviors and thoughts that we would expect it to.
Factor model loss: a measure of how much the personality trait conflates different unrelated things together.
Correlation with lexical notion: a measure of how well-labelled the personality trait is. (You can mostly ignore this variable as all of the personality traits performed reasonably well on it.)

Easy-Goingness: An example

A conventional personality test such as the SPI-81-27&5 might measure your personality traits such as Easy-Goingness by asking you how well a few abstract statements describe you, e.g.:

I like to take it easy
I like a leisurely lifestyle
I have a slow pace to my life

It seems plausible that someone who agrees that such statements describe them would be Easy-Going in some sense, and indeed I bet this sort of measure can pass all sorts of criteria used by psychologists to evaluate the quality of the test.[4]

So if it is probably valid by t...
"textbooks are all you need"
by bhauth — 2023-06-21T17:06:46.148Z
Link: https://arxiv.org/abs/2306.11644

"Textbooks Are All You Need" was published yesterday by Microsoft Research. It's the worst-named paper I've seen recently: it's not about textbooks, it's not all you need, and gratuitously imitating the title of a paper that introduced a different type of thing is dumb. But there's a reason I'm writing about it.

What they did was basically this:

started with The Stack (a 3 TB collection of code) and text from StackOverflow

used a LLM to select 6B "high-quality" tokens from (1)

used GPT-3.5 to generate 1B tokens of text similar to textbooks

trained a small (1.3B parameter) model ("phi-1") on (2) and (3)

used GPT-3.5 to generate text similar to textbook exercises

fine-tuned phi-1 on (5)

tested phi-1 on HumanEval to evaluate its programming ability

The results were pretty good, better than models 10x the size trained on 100x the data. So, it seems that scaling up isn't the only thing that matters, and data quality can be more important than data quantity or parameter count. (You hear that, gwern?)

Going by the listed OpenAI API prices, running GPT-3.5 on The Stack to evaluate quality would've been maybe ~$6M. What the authors did instead was:

Use GPT-4 to evaluate a small fraction of it.

Use a much smaller code-specific model to generate embeddings.

Use a classifier to predict which embeddings are from what GPT-4 evaluates as good content.

How about if you bootstrap a model using its own evaluation for filtering? One of the authors says "I'm almost sure you can beat the teacher model" and I agree. That can give you recursive self-improvement of a type you see in both individual people and the culture of societies. People develop better taste and consume better content which makes them smarter so they develop better taste, and so on. Children hear the stories their grandfathers like, and culture develops.

That's a weak sort of self-improvement, which tends to plateau for people. Humans do other things too, so by itself it's weaker self-improvement than it appears to be for people. This is a technique I previously spent some time thinking about, so there are some other reasons I think it tends to plateau by itself. But still - recursive self-improvement!

Yes, in theory, if you have a much bigger model trained on a bigger dataset including the good selected data, and you can engineer prompts such that you get into a mode that models the good data specifically, the...
Philosophical Cyborg (Part 2)...or, The Good Successor
by ukc10014 — 2023-06-21T15:43:06.901Z

This post is part of the output from AI Safety Camp 2023’s Cyborgism track, run by Nicholas Kees Dupuis - thank you to Nick, AISC organizers & funders for their support.

TL;DR

This post follows up on the cyborgism research/writing process documented in 'Upon the Philosophical Cyborg'. It attempts to analyse 2018 post by Paul Christiano about the possibility that an unaligned AI may yet be a morally-valuable entity, by its own and even by our lights. Writing this essay has involved a back-and-forth between a human author and a few different versions of GPT-3/4, followed by extensive editing, as well as human-written additions. So, while this post contains LLM-written parts, and benefits from the high-variance idea generation of a base model (code-davinci-002) as well as the research/reasoning ability of a RLHF-tuned model (GPT-4), most of the 'thinking' is human-based, which is consistent with the cyborgism agenda. The 'voice' might feel a little stilted and wordy, but that is mostly a result of how GPT-4 (via ChatGPT) writes if not explicitly prompted otherwise...verbose output is in strikethrough.

For a spicier take, see the Appendix.

**

Introduction

Paul Christiano's exploration of the alignment of artificial intelligence (AI) with human values raises important questions about our responsibility towards the AI we create, and the potential consequences of this creation. In his LessWrong post (the'OP'), Christiano navigates through complex ideas of moral desirability, moral patienthood, analogies of AI to corporations and children, ultimately suggesting an innovative thought experiment that brings forth new dimensions to the discussion.

This post attempts to unpick his analysis but also tries to clarify the frame(s) within which his points might be approached. For instance, must humanity’s successor civilization contain billions of (what we would recognize as) individual beings; can volumes of space contain agents that act with a unitary identity; do anthropic arguments around simulation make all moral evaluations meaningless? Or, how can we move away from our anthropocentric perspective towards more objective ones? Or, might the best outcome we can achieve is to maximise optionality and be wary of taking wasteful actions or actions that lock in badly-thought-through futures?

The OP’s Main Claims

The major points in the OP cove...
Relational Speaking
by jefftk — 2023-06-21T14:40:00.849Z

Relational speaking is a style that emphasizes flow and the
relationship between subjects, rather than using pronouns to refer
to entities in the discourse. It is appropriate for a wide variety of
contexts, as it makes no assumptions about the gender of your
referents. Relational constructions have always been a part of
English: consider a casual "Want some fries?" or a command of "Don't
look!" There are linguistic communities that have used relational, or
as linguists call it, "

pro-drop"
speech for centuries, but until recently in English it was the
exception rather than the norm. In the past five years or so, a
number of us have intentionally developed relational styles, and in
the past two years this style of speech has gained significant
traction among the community interested in gender-free speech.
Gender-free English has a much longer history, of course, but it has
typically relied on pronouns—xie/xer, they/them, etc.—to
refer to people. Some of those traditions came out of explicitly
LGBTQIA+ communities, but over time, gender presentation has changed
across the board. Gender-free speaking strategies reflect an
awareness that appearance is no longer a useful indicator of gender or
pronoun preference.

In the past, gender was undeniably a useful reference, but one cannot
emphasize enough how little meaning terms like "he" or "she" are when
a person's name or appearance does not clearly indicate which applies.
In fact, they aren't just meaningless—they can actually cause
problems; it's likely that in conversations in gendered spaces you've
seen well-meaning people try to correct pronoun usage that, in fact,
was already correct. And while alternative pronouns, like using
they/them for everyone, avoid the problem of being discriminatory and
exclusionary, they keep us in a frame of pronoun reference, with all
its historical baggage, instead of embracing a relational approach.

It's possible that you are looking for a more visible or immediate
shift to gender-free speech, as a way of asserting your or your
organization's commitment to inclusivity. Relational speech has been
criticized for its invisibility, which strikes some people as a
pandering concession to linguistic conservatives who are resistant to
being referred to as "they". Experience tells us that the people who
are angry about gender-free speech do not perceive relational spe...