trentmkelly/LessWrong-43k huggingface dataset:
Some advice on independent research
by Marius Hobbhahn — 2022-11-08T14:46:19.134Z

I have been doing independent research in addition to my Ph.D. for roughly a year now. For the next 6 months, I’ll take a break from my Ph.D. and plan to do AI safety research full-time. I had chats with many people about independent research in the past, e.g. on EAGs or because 80K has connected me with people thinking about pursuing independent research. I had some great experiences with independent research but not everyone does. I think the variance for independent research is large and I'm worried that people get disheartened by bad experiences. So here are some considerations in which situations independent research might be a good idea and some tips that will hopefully improve your experience.

I’d like to thank Magdalena Wache and Tilman Räuker for their feedback.

TL;DR: At first glance, there is a bit of a paradoxical nature to independent research. If someone wants to pursue independent research they need a research agenda to work on. If they are able to construct a good research agenda, an existing institution often has incentives to hire them. On the flip side, if their research skills are not developed enough to be hired by an existing institution, their independent research might not be very successful. Thus, naively it would seem that there are few cases in which independent research makes sense. However, I think that there are many situations in which independent research or independent upskilling are a great option, e.g. when no established organization is working on the topic you find most promising, as a way to upskill for a job, to gain new research skills or to transition between jobs. Some tips for independent researchers include: getting feedback early on, aiming to collaborate with others and creating accountability mechanisms for yourself such as publishing your results. My most important advice for independent researchers is that you should probably be much more active than in other roles because there is less default structure and more responsibility on you.

I’ll mostly talk about AI safety research but many of these things probably also apply to other independent research.

A perceived paradox

Independent research is often presented as one of three default options for people seeking to do EA research, e.g. in AI safety:

Academia, e.g. applying for Ph.D. and post-doc positions with labs that do research aligned

...
Mysteries of mode collapse
by janus — 2022-11-08T10:37:57.760Z

Thanks to Ian McKenzie and Nicholas Dupuis, collaborators on a related project, for contributing to the ideas and experiments discussed in this post. Ian performed some of the random number experiments.
Also thanks to Connor Leahy for feedback on a draft, and thanks to Evan Hubinger, Connor Leahy, Beren Millidge, Ethan Perez, Tomek Korbak, Garrett Baker, Leo Gao and various others at Conjecture, Anthropic, and OpenAI for useful discussions.

This work was carried out while at Conjecture.

Important correction

I have received evidence from multiple credible sources that text-davinci-002 was not trained with RLHF.

The rest of this post has not been corrected to reflect this update. Not much besides the title (formerly "Mysteries of mode collapse due to RLHF") is affected: just mentally substitute "mystery method" every time "RLHF" is invoked as the training method of text-davinci-002. The observations of its behavior otherwise stand alone.

This is kind of fascinating from an epistemological standpoint. I was quite surprised to learn that text-davinci-002 was probably not trained with RLHF. I don't remember exactly how "text-davinci-002 is RLHF" got elevated to an unquestioned assumption in my mind. I might have mistook not being contradicted by people who I assumed were in the know as confirmation. I certainly did not expect to talk for months to dozens of people about odd behaviors I've observed in a well-known model "due to RLHF" without being contradicted in a world where the model in question wasn't trained with RLHF, but that's what happened.[1] It wasn't just me either: the assumption that text-davinci-002(/text-davinci-001) is InstructGPT is RLHF seems ambient (e.g. search "text-davinci-002 rlhf" on Twitter, this LW post, this article, and many others). I contributed to perpetuating this misinformation cascade, and for that I apologize.

text-davinci-002's behaviors described in this post also contributed to my confidence because RLHF seemed to be a likely and potentially satisfying explanation. Its apparently unsubstantiated confidence in very specific outcomes seems antithetical to the outer objective of self-supervised learning, which is optimized by epistemic calibration, meaning the model's entropy should be as high as possible while fitting the data. In contrast, as several comments have pointed out, it makes sense that RL kills entropy. The presence o...
[ASoT] Thoughts on GPT-N
by Ulisse Mini — 2022-11-08T07:14:37.900Z

Editor's note: This is an alignment stream-of-thought post, meaning it's literally a rambling catalog of my thoughts on something.

TLDR: GPT-n isn't AGI, but you can probably get AGI relatively easily by RL finetuning and/or self-improvement. Some people wrongly think GPT-n will immediately kill them though.

I believe that "GPT-n" and similar systems

Won't be agents/consequentialists by default, they will only simulate consequentialist sub-agent/simulacra. GPT-n won't kill you by default, GPT-n doesn't "want" anything.

(I'm uncertain about if sufficiently powerful "things" (like GPT-n) will become "agents"; currently, I think this isn't the case, since modeling GPT-3 as an agent doesn't work well.)

In GPT-n, The intelligence of each individual simulacra is approximately bounded by the intelligence of the smartest human in the training distribution. Prompting GPT-n with "A solution to the alignment problem:" will never work. GPT-n is not an oracle. (This doesn't imply mimicry is doomed, though I'm pessimistic for reasons John gives plus requiring the coordination to simulate alignment researchers but not capabilities researchers which seems hard to me.)
GPT-n may well be superintelligent (It already is in some sense) but even if GPT-n understands more (e.g. about physics) than the smartest human, it will never report this understanding because that isn't what a human on the training distribution would be likely to write.
This isn't to say GPT-n won't lead to AGI. Just that there's some nontrivial prompting/finetuning to be done in order to access the latent knowledge. I'm not sure how hard this step is. The main angles of attack I see: using GPT-n to self-improve, RL-based fine-tuning, or maybe crazy Codex-based stuff like a fancier AutoML-Zero.
Michael Simm - Introducing Myself
by Michael Simm — 2022-11-08T05:45:37.348Z

Hi, LessWrong Community. I'd like to introduce myself!

My name is Michael Simm. I am passionate about disruptive systems (eg, understanding and using emerging technologies to make the future better). I'd been interested in EA principles long before I learned about the movement, so once I read What We Owe The Future, I jumped right into an In-depth EA Virtual Program and started reading a lot of EA literature. Shout-out to my In-Depth EA group, you all are awesome!

I’m working to establish an extremely high-impact charity with game-changing potential, and I’m working on a forum post to introduce it by the end of the week.

With any luck, you’ll be seeing me around a lot, so here’s a bit about me :)

I’ve been interested in maximizing my impact and trying to improve the future for a long time. Back in high school and college, I got deeply involved in political activism, especially around climate change as I thought it was the biggest existential threat to humanity’s future. I helped organize some of the March For Our Lives stuff (gun control), and then I started a Sunrise Movement Hub (climate activism group - Green New Deal) at my University that saw decent success. I saw activism as the best use of my time to help create large positive change.

Around 2018, I started becoming interested in a very different type of impactful thing: disruptive technologies. With my focus on climate change, I quickly identified electric cars, solar panels, and energy storage (particularly batteries) to be on the verge of upending global reliance on fossil fuels. There, I ran across Dr. Tony Seba, who was one of the only people to accurately predict the massive price declines of solar, electric cars, and batteries. He’s now doing fantastic research into the coming disruptions in energy, transportation, and other areas with an organization called RethinkX. Disruption generally happens when any system proves 5X better than the incumbent one, thus opening a huge opportunity space.

Because of my research around disruptive technology businesses, I put most of my high school savings into Tesla stock. A lot of people thought I was nuts.

In college, my dueling interests in politics and disruptive technologies (systems) led me to major in political science and minor at ASU’s new School for the Future of Innovation in Society (SFIS). My time in college (It wasn’t a lot of tim...
EA & LW Forums Weekly Summary (31st Oct - 6th Nov 22')
by Zoe Williams — 2022-11-08T03:58:25.600Z

Supported by Rethink Priorities

This is part of a weekly series summarizing the top (40+ karma) posts on the EA and LW forums - you can see the full collection here. The first post includes some details on purpose and methodology.

If you'd like to receive these summaries via email, you can subscribe here.

Podcast version: prefer your summaries in podcast form? A big thanks to Coleman Snell for producing these! Subscribe on your favorite podcast app by searching for 'Effective Altruism Forum Podcast'.

Top / Curated Readings

Designed for those without the time to read all the summaries. Everything here is also within the relevant sections later on so feel free to skip if you’re planning to read it all.

The Challenges with Measuring the Impact of Lobbying

by Animal Ask, Ren Springlea

Summary of a report by Animal Ask assessing if academic literature contains quantitative estimates that could be useful in gauging the counterfactual impact of lobbying. This requires breaking policy success into a baseline rate and counterfactual increase from lobbying.

They found the answer is no - lobbying literature contains many well-known weaknesses, a systematic review found a strange result (narrow range around 50% success, possibly because there are often lobbyists on both sides of an issue), and only one study attempted to identify counterfactual impact (and only for one policy).

The authors instead suggest using expert judgment and superforecasters.

What matters to shrimps? Factors affecting shrimp welfare in aquaculture

by Lucas Lewit-Mendes, Aaron Boddy

Report by Shrimp Welfare Project on the importance of various factors for the welfare of farmed shrimps. Welfare can be measured via biological markers (eg. chemicals associated with immune or stress responses), behavior (eg. avoidance, aggression), physical condition and survival rates.

The strongest evidence for harm to shrimp welfare was with eyestalk ablation, disease, stunning / slaughter, and insufficient dissolved oxygen or high un-ionized ammonia in the water. The authors have very high confidence that small to medium improvements in these would reduce harm to shrimp.

An Introduction to the Moral Weight Project

by Bob Fischer

In 2020, Rethink Priorities published the Moral Weight Series—a collection of five reports addressing questions around interspecies cause prioritization. This research was expanded in May ...
Value of Querying 100+ People About Humanity's Future
by T431 — 2022-11-08T00:41:09.979Z

I am looking for advice on whether I should do what I've outlined below (is it valuable or generally a waste of time) and, if so, how I should go about doing it. If there is support for something like this, I may apply for funding and do this sometime during Q4 2023.

Go to a frequented environment in NYC (e.g., Time Square; I use NYC because I live close to this city and presumably would be the one doing the actions I'm describing)
Bring a notepad / clipboard and don an official or professional appearance (e.g., a suit)
Bring a sign with some message akin to:

"Quick, Paid Survey About Humanity"
"$10 for 10 Minutes, Survey"
"Questions About Our Future"

Bring a camera w/ tripod and other necessary recording equipment, maybe a mic
Being recording short interviews (~3-10 minutes).
For person 1 to ~100:

Ask if they'd be willing to answer some questions about humanity (3-10 minutes) for $10 and be recorded on video
Record person speaking, and then ask some or all of the following questions or those similar to the following:

If you had 10 billion USD to help the world, how would you spend it?
What are the most important problems for humanity to address right now?
What might be the important problems for humanity to address in 2050?
What are the most important issues for the USA right now?
If you had to send a short message to humanity in 2100, what would it be?
Rank these in terms of how risky them seem (have these on the clipboard):

[nuclear war], [space weather / collisions], [climate change], [pandemics], [artificial intelligence], [global conflict]

How are you altruistic, and how altruistic should people be?
What does this picture [pale blue dot] make you think about?
[Maybe some question about human enhancement / treatment via gene-editing]

Ask some or all of the following demographic questions

Where did you grow up?
How much school have you received?
Are you happy?
How wealthy do you consider yourself?
What ethnicity or race do you consider yourself?

Pay individual $10 USD and thank them for their time

General Goal / Motivation: I think action-sequence would be an interesting window into what a sample (biased, out of convenience) of "ordinary" Westerners have to say about the near and long-term future of humanity. If a verbal survey like the one I have outlined above is done well, I imagine that it could serve as a gateway for the people question...
How could we know that an AGI system will have good consequences?
by So8res — 2022-11-07T22:42:27.395Z

(Note: This was languishing in a drafts folder for a while, and probably isn't quite right in various ways. I'm posting it because I expect it's better to share flawed thoughts than to sit on the post until I'm satisfied with it, i.e., forever.)

Let's play a game of "what do you think you know, and why do you think you know it?".

Imagine that you're about to launch an AGI. What you think you know is that, with at least 50% confidence (we’re of course not looking for proofs — that would be crazy), the AGI is going to execute some pivotal act that ends the acute risk period in a good way. Why do you think you know that?

Insofar as people’s alignment proposals can be construed as answers to this question, we have the option of answering with one of these proposals. I might very roughly classify the existing proposals into the following bins:

1. Output evaluation approaches. You know what the AGI is going to do with sufficient precision that it screens off any alignment concerns. For example, your AGI system only outputs plans in the first place, and you've already reviewed the plan, and you're confident the plan will work, in a way that screens off any other worry about the AGI being misaligned.

2. Cognitive interpretability approaches. You understand the AGI's cognition sufficiently well that, while you may not be sure what it's going to do, you're confident that it's going to be good. You aren't worried that it will kill all humans, because you understand how its plan came to be and what solution-spaces it was searching to solve various sub-problems and so on, and you're confident no consideration was ever given to human-killing.

3. Heavy-precedent approaches. You have run this AGI before on many similar tasks, and trained out all the hiccups. While you might not know precisely what it's going to do, and you might not know what's going on inside its mind, you've been around the block a few times, and the task it's about to perform is sufficiently similar to other tasks it has empirically succeeded at, justifying your confidence.

Roughly speaking, I think that alignment approaches with a heavy reliance on output evaluation are doomed, both on the grounds that humans can't evaluate the effectiveness of a plan capable of ending the acute risk period, and because the real plan is less like a story and more like a tree.

For an example o...
A Walkthrough of Interpretability in the Wild (w/ authors Kevin Wang, Arthur Conmy & Alexandre Variengien)
by Neel Nanda — 2022-11-07T22:39:16.671Z
Link: https://youtu.be/gzwj0jWbvbo

New paper walkthrough: Interpretability in the Wild: A Circuit for Indirect Object Identification In GPT-2 Small is a really exciting new mechanistic interpretability paper from Redwood Research. They reverse engineer a 26(!) head circuit in GPT-2 Small, used to solve Indirect Object Identification: the task of understanding that the sentence "After John and Mary went to the shops, John gave a bottle of milk to" should end in Mary, not John.

I think this is a really cool paper that illustrates how to rigorously reverse engineer real models, and is maybe the third example of a well understood circuit in a real model. So I wanted to understand it better by making a walkthrough. (and hopefully help other people understand it too!) In this walkthrough I'm joined by three of the authors, Kevin Wang, Arthur Conmy and Alexandre Variengien. In Part 1 we give an overview of the high-level themes in the paper and what we think is most interesting about it. If you're willing to watch an hour of this and want more, in Part 2 we do a deep dive into the technical details of the paper and read through it together, and dig into the details of how the circuit works and the techniques used to discover this.

If you're interested in contributing to this kind of work, apply to Redwood's REMIX program! Deadline Nov 13th

We had some technical issues in filming this, and my video doesn't appear - sorry about that! I also tried my hand at some amateur video editing to trim it down - let me know if this was worthwhile, or if it's bad enough that I shouldn't have bothered lol. If you find this useful, you can check out my first walkthrough on A Mathematical Framework for Transformer Circuits. And let me know if there are more interpretability papers you want walkthroughs on!

If you want to try exploring this kind of work for yourself, there's a lot of low-hanging fruit left to pluck! Check out my EasyTransformer library and the accompanying codebase to this paper!
A good starting point is their notebook giving an overview of the key experiments or my notebook modelling what initial exploration on any task like this could look like. Some brainstormed future directions:

3 letter acronyms (or more!)

Converting names to emails.

An extension task is e.g. constructing an email from a snippet like the following: Name: Neel Nanda; Email: last name dot first name k @ gmail

Grammatical rules

L

...
Intercept article about lab accidents
by ChristianKl — 2022-11-07T21:10:19.559Z
Link: https://theintercept.com/2022/11/01/biosafety-lab-accident-chikungunya-virus/

The Intercept made FOI requests to gather information about American lab accidents. Additionally, their article discusses the current lack of legislation to require commonsense biosafety strategy from researchers.

Unless they work with the most dangerous pathogens, biolabs don’t have to register with the U.S. government. As a result, there is little visibility into the biosafety of experiments carried out by private companies or foundations.

“Your favorite tech billionaire could, with their own money, do basically whatever the hell they want with any pathogen,” said Rocco Casagrande, managing director of Gryphon Scientific, a biosafety advisory firm that has advised NIH on biosafety standards. “They could take the measles virus and intentionally try to make it vaccine-resistant and more pathogenic in their garage. If they’re doing it for legitimate research purposes in their own minds, they can do so wildly, unsafely, and no one can stop them.”

[...]

“There are some significant holes,” said Filippa Lentzos, an expert on biosecurity and biological threats at King’s College London. Biosafety protocols are “not embedded in statutory law. It’s tied to funding.”

[...]

Between April 2013 and March 2014, the University of North Carolina at Chapel Hill reported five mouse escapes, including one of an animal that had been infected with SARS four days earlier. In a letter to NIH, a biosafety specialist argued that the frequency of escapes was due to the “complex research taking place at our institute” rather than a failure of training, noting that several teams at the university use a breed of transgenic mouse known for its unpredictable behavior.
The biological function of love for non-kin is to gain the trust of people we cannot deceive
by chaosmage — 2022-11-07T20:26:29.876Z

Epistemic status: conjecture, needs critical feedback, but fits with a lot of established facts.

Over the last six years, I spent much of my spare time figuring out love. The main output of this process is

a poem

– I’m proud of it, but poems are weird, so here I write down my main result in prose.

There are forms of love that The Selfish Gene has already explained convincingly, like love for next of kin or love between mating and child-rearing partners, and reciprocal altruism. So those are not what I needed to figure out. They're important, because they provide existing physiology to repurpose. But they don’t explain why and how you can love your favorite artist, your fellow soldiers, your best friend, good colleagues, or Scott Alexander. And what about love towards non-sentient things such as your country, your favorite sports team, or mythologized characters such as Jesus? Is the use of the word “love” in these cases mere metaphor or confusion, or are they actually comparable phenomena on a biological level?

So this is about a mechanism of love that works independently of the ones more directly involved in reproduction such as sexual attraction, romantic infatuation or the joy of childrearing. The ontological question of whether it's all the same love in the end, or whether there are various types of love that go well together, doesn't strike me as productive. I should say I think that when you "love" a cake with whipped cream on it you're doing something different from the following.

Impetus

My most salient confusion about love was why the hell showing vulnerability is so central to it. Maybe “vulnerability is attractive” seems like a normal idea to many, but it doesn't explain why that would be the case, and I found the idea deeply suspect for decades because it isn't how I grew up. But I had to concede that every time I tried it, it worked disconcertingly well. Every time I was vulnerable to someone, I liked them a lot better afterwards, and as far as I could tell they liked me better too. What? Why? Isn't love just about attraction, compatible personalities, shared projects and aligned values? None of those seem obviously enhanced by vulnerability. What’s going on? The following is how I answered this confusion.

When our brains evolved in groups of social apes in the savannah, we cooperated intensely. Many species do this, but unlike other sp...