trentmkelly/LessWrong-43k huggingface dataset:
AI Impacts Quarterly Newsletter, Jan-Mar 2023
by Harlan — 2023-04-17T22:10:11.976Z
Link: https://blog.aiimpacts.org/p/ai-impacts-quarterly-newsletter-jan

News

AI Impacts blog

We moved our blog to Substack! We think this platform has many advantages, and we’re excited for the blog to live here. You can now easily subscribe to the blog to receive regular newsletters as well as various thoughts and observations related to AI.

AI Impacts wiki

All AI Impacts research pages now reside on the AI Impacts Wiki. The wiki aims to document what we know so far about decision-relevant questions about the future of AI. Our pages have always been wiki-like: updatable reference pages organized by topic. We hope that making it an actual wiki will make it clearer to everyone what’s going on, as well as better to use for this purpose, for both us and readers. We are actively looking for ways to make the wiki even better, and you can help with this by sharing your thoughts in our feedback form or in the comments of this blog post!

New office

We recently moved to a new office that we are sharing with FAR AI and other partner organizations. We’re extremely grateful to the team at FAR for organizing this office space, as well as to the Lightcone team for hosting us over the last year and a half.

Katja Grace talks about forecasting AI risk at EA Global

At EA Global Bay Area 2023, Katja gave a talk titled Will AI end everything? A guide to guessing in which she outlined a way to roughly estimate the extent of AI risk.

AI Impacts in the Media

AI Impacts’ 2022 Expert Survey on Progress in AI was cited in an NBC Nightly News segment, an op-ed in Bloomberg, an op-ed in The New York Times, an article in Our World in Data, and an interview with Kelsey Piper.

Ezra Klein quoted Katja and separately cited the survey in his New York Times op-ed This Changes Everything.

Sigal Samuel interviewed Katja for the Vox article The case for slowing down AI.

Research and writing highlights

AI Strategy

“Let’s think about slowing down AI” argues that those who are concerned about existential risks from AI should think about strategies that could slow the progress of AI (Katja)

“Framing AI strategy” discusses ten frameworks for thinking about AI strategy. (Zach)

“Product safety is a poor model for AI governance” argues that a common type of policy proposal is inadequate to address the risks of AI. (Rick)

“Alexander Fleming and Antibiotic Resistance” is a research report about early efforts to prevent

...
How do you align your emotions through updates and existential uncertainty?
by VojtaKovarik — 2023-04-17T20:46:29.510Z

Asking this question since I expect other people to be in a similar position:[1]

There are many ways the future could plausibly go from here, ranging from "doom this year" to "things are mostly fine by default". And as some of these scenarios draw near, my epistemic updates are getting more entangled with my emotions. (Which I endorse.) Now, if I was convinced the world is going to go in a particular way, these emotional updates would be easy(ier): Maybe I expect things to go fine, so I go about my life as usual. Or maybe I have P(doom)=99%, so I mourn for a bit and then take an indefinite vacation to enjoy travel and friends. Or maybe things could go either way, and there is no way to know until the last moment, so I work as hard as I can (while having fun on the way, obviously).

However, things are not so simple: At the moment, I am very uncertain on how things will go. I expect that the next year(s) will be quite wild in terms of epistemic updates. Maybe there will come a point when confidence in one of the scenarios is justified, in which case I want to make the appropriate emotional update as soon as possible. And maybe it won't. And this type of uncertainty is something I do not know how to approach.

Does anybody have recommendations or observations for how to relate to this? (Without deluding oneself in any way.)

^

EDIT: Rephrased the title based on feedback.
AI Alignment Research Engineer Accelerator (ARENA): call for applicants
by CallumMcDougall — 2023-04-17T20:30:03.965Z

TL;DR

Apply here for the second iteration of ARENA!

Introduction

We are excited to announce the second iteration of ARENA (Alignment Research Engineer Accelerator), a 6-week ML bootcamp with a focus on AI safety. Our mission is to prepare participants for full-time careers as research engineers in AI safety, e.g. at leading organizations or as independent researchers.

The program will commence on May 22nd, 2023, and will be held at the Moorgate WeWork offices in London. This will overlap with SERI MATS, who are also using these offices. We expect this to bring several benefits, e.g. facilitating productive discussions about AI safety & different agendas, and allowing participants to form a better picture of what working on AI safety can look like in practice.

ARENA offers a unique opportunity for those interested in AI safety to learn valuable technical skills, engage in their own projects, and make open-source contributions to AI safety-related libraries. The program is comparable to MLAB or WMLB, but extends over a longer period to facilitate deeper dives into the content, and more open-ended project work with supervision.

For more information, see our website.

Outline of Content

The 6-week program will be structured as follows:

Chapter 0 - Fundamentals

Before getting into more advanced topics, we first cover the basics of deep learning, including basic machine learning terminology, what neural networks are, and how to train them. We will also cover some subjects we expect to be useful going forwards, e.g. using GPT-3 and 4 to streamline your learning, good coding practices, and version control.

Topics include:

PyTorch basics
CNNs, Residual Neural Networks
Optimization
Backpropagation
Hyperparameter search with Weights and Biases
Model training & PyTorch Lightning

Duration: 5 days

Chapter 1 - Transformers & Mechanistic Interpretability

In this chapter, you will learn all about transformers, and build and train your own. You'll also learn about Mechanistic Interpretability of transformers, a field which has been advanced by Anthropic’s Transformer Circuits sequence, and open-source work by Neel Nanda.

Topics include:

GPT models (building your own GPT-2)
Training and sampling from transformers
TransformerLens
In-context Learning and Induction Heads
Indirect Object Identification
Superposition

Duration: 9 days

Chapter 2 - Reinforcemen

...
AI policy ideas: Reading list
by Zach Stein-Perlman — 2023-04-17T19:00:00.604Z

Related: Ideas for AI labs: Reading list. See also: AI labs' statements on governance.

This document is about AI policy ideas. It's largely from an x-risk perspective. Strikethrough denotes sources that I expect are less useful for you to read.

Lists

Lists of government (especially US government) AI policy ideas. I recommend carefully reading the lists in the first ~5 bullets in this list, noticing ideas to zoom in on, and skipping the rest of this section.

Toward Trustworthy AI Development: Mechanisms for Supporting Verifiable Claims (Brundage et al. 2020)

Followed up by Filling gaps in trustworthy development of AI (Avin et al. 2021)

12 tentative ideas for US AI policy (Muehlhauser 2023) (EAF)
Survey on intermediate goals in AI governance (Räuker and Aird 2023)
Policymaking in the Pause (FLI 2023) (LW)
Frontier AI Regulation: Managing Emerging Risks to Public Safety (Anderljung et al. 2023)
"30 actions to reduce existential risk" in Existential risk and rapid technological change (Stauffer et al. 2023)
The Malicious Use of Artificial Intelligence: Forecasting, Prevention, and Mitigation (Brundage et al. 2018)
How major governments can help with the most important century (Karnofsky 2023) (LW)
[Chuck Schumer] (2023)
[Dan Hendrycks] (2023)
"Discussion" in "Verifying Rules on Large-Scale NN Training via Compute Monitoring" (Shavit 2023)
Future Proof (CLTR 2021)
Year 1 Report (National Artificial Intelligence Advisory Committee 2023)
Final Report (National Security Commission on Artificial Intelligence 2021)
Challenges to U.S. National Security and Competitiveness Posed by AI (Matheny 2023)
"Policy Options" in "Artificial Intelligence and Strategic Trade Controls" (Viski et al. 2020)

Existential and global catastrophic risk policy ideas database

(filter for "Artificial intelligence") (Sepasspour et al. 2022)
Various private lists and works in progress

Levers

Some sources focus on policy levers rather than particular policy proposals.

"Governmental levers" in "Literature Review of Transformative AI Governance" (Maas draft)

This report is excellent

AI Policy Levers: A Review of the U.S. Government's Tools to Shape AI Research, Development, and Deployment

(Fischer et al. 2021)

"Affordances" in "Framing AI strategy"

(Stein-Perlman 2023)

Current UK government levers on AI development

(Hadshar 2023)
Standards (largely a non-government lever)

How technical safety standards could promote TAI safety

...
NYT: The Surprising Thing A.I. Engineers Will Tell You if You Let Them
by Sodium — 2023-04-17T18:59:33.626Z
Link: https://www.nytimes.com/2023/04/16/opinion/this-is-too-important-to-leave-to-microsoft-google-and-facebook.html

NYT Opinion article by Ezra Klein on AI-regulations. Ezra has been writing quite a lot on AI recently. Thought people might be interested in discussing them here.
But why would the AI kill us?
by So8res — 2023-04-17T18:42:39.720Z

Status: Partially in response to We Don't Trade With Ants, partly in response to watching others try to make versions of this point that I didn't like. None of this is particularly new; it feels to me like repeating obvious claims that have regularly been made in comments elsewhere, and are probably found in multiple parts of the LessWrong sequences. But I've been repeating them aloud a bunch recently, and so might as well collect the points into a single post.

This post is an answer to the question of why an AI that was truly indifferent to humanity (and sentient life more generally), would destroy all Earth-originated sentient life.

Might the AGI let us live, not because it cares but because it has no particular reason to go out of its way to kill us?

As Eliezer Yudkowsky once said:

The AI does not hate you, nor does it love you, but you are made of atoms which it can use for something else.

There's lots of energy in the biosphere! (That's why animals eat plants and animals for fuel.) By consuming it, you can do whatever else you were going to do better or faster.

(Last I checked, you can get about 10x as much energy from burning a square meter of biosphere as you can get by collecting a square meter of sunlight for a day. But I haven't done the calculation for years and years and am pulling that straight out of a cold cache. That energy boost could yield a speedup (in your thinking, or in your technological design, or in your intergalactic probes themselves), which translates into extra galaxies you manage to catch before they cross the cosmic event horizon!)

But there's so little energy here, compared to the rest of the universe. Why wouldn't it just leave us be, and go mine asteroids or something?

Well, for starters, there's quite a lot of energy in the sun, and if the biosphere isn't burned for fuel then it will freeze over when the AI wraps the sun in a dyson sphere or otherwise rips it apart. It doesn't need to consume your personal biomass to kill you; consuming the sun works just fine.

And separately, note that if the AI is actually completely indifferent to humanity, the question is not "is there more energy in the biosphere or in the sun?", but rather "is there more energy available in the biosphere than it takes to access that energy?". The AI doesn't have to choose between harvesting the sun and harvesting the biosphere, it can just harvest both, an...
Sama Says the Age of Giant AI Models is Already Over
by Algon — 2023-04-17T18:36:22.384Z
Link: https://www.wired.com/story/openai-ceo-sam-altman-the-age-of-giant-ai-models-is-already-over/

There wasn't much content to this article, except for the below quotes.

Sam Altman says

further progress will not come from making models bigger. “I think we're at the end of the era where it's going to be these, like, giant, giant models,” he told an audience at an event held at MIT late last week. “We'll make them better in other ways.” [...] Altman said there are also physical limits to how many data centers the company can build and how quickly it can build them. [...] At MIT last week, Altman confirmed that his company is not currently developing GPT-5. “An earlier version of the letter claimed OpenAI is training GPT-5 right now,” he said. “We are not, and won't for some time.”

Nick Frosst, a cofounder at Cohere, says

“There are lots of ways of making transformers way, way better and more useful, and lots of them don’t involve adding parameters to the model,” he says. Frosst says that new AI model designs, or architectures, and further tuning based on human feedback are promising directions that many researchers are already exploring.

In the Lex Fridman interview with Sam Altman, Sam said that they had to do "hundreds of complicated things". Does this, together with the above quote, suggest Sam thinks transformers are running out of oomph? Is he, perhaps, pausing progress whilst we await the next breakthrough in deep learning?

Edit: Added in a relevant question. Give your probability for the first option.

Elicit Prediction (forecast.elicit.org/binary/questions/VuF0joaV6)

Or vote at this tweet, if you like:
https://twitter.com/Algon_33/status/1648047440065449993
Meetup Tip: Conversation Starters
by Screwtape — 2023-04-17T18:25:56.441Z

Summary

If you're running or attending meetups a lot, it's helpful to have a short list of conversation starters ready. Conversation starters, sometimes called icebreaker questions, are questions that everyone can answer. They aren't deeply personal, but you do learn something new about the person. Ideally, they're short questions that lead into longer conversations. "How did you find this community" is one example of a decent conversation starter.

You have now read the basic point of this post. If you want to read on, cool, lets talk about implementation details for a bit.

Details

I've found the really good conversation starters tend to fall into one of two categories. Either the question encourages someone to go on a bit of a monologue, or the question leads to answers that are full of jumping off points for further conversation. "What's your most recent obsession?" is a good example of the former. Their resulting oration may not be a back-and-forth conversation, but I like to hear what people find interesting. "Where did you grow up?" is a good example of the latter. I can almost always go from there to something like "Oh, neat! Why did you move away?" or "Cool! What's something you love about the place?" Those questions lead to more questions; if they loved the mountains because they love to hike, ask them about the best hiking trails or about fun hiking stories.

Conversation starters need to have answers. "What did you major in at college?" is a somewhat risky starter in a random group, since not everyone went to college. Likewise, "what do you do for work?" I've been using "what do you spend most of your time on?" since everyone has time. If you know your group, you can use more specific conversation starters: at a LessWrong meetup "what's your favourite LessWrong article?" would be a fine conversation starter, since people have probably read at least one.

I would like to emphasize part of the above. Questions that rely on domain expertise to understand should not be used as conversation starters unless you have good reason to expect people have that domain expertise. "So, what do you think about the latest paper on LLM architecture?" is a terrible conversation starter unless you are at an academic conference on AI. No, a LessWrong meetup is not an academic conference on AI.

Questions that naturally lead to yes or no answers aren't good for this, though...
Critiques of prominent AI safety labs: Redwood Research
by Omega. — 2023-04-17T18:20:01.256Z
Link: https://forum.effectivealtruism.org/posts/DaRvpDHHdaoad9Tfu/critiques-of-prominent-ai-safety-labs-redwood-research

In this series, we evaluate AI safety organizations that have received more than $10 million per year in funding. We do not critique MIRI (1) and OpenAI (1,2,3) as there have been several conversations and critiques of these organizations.

The authors of this post include two technical AI safety researchers, and others who have spent significant time in the Bay Area community. One technical AI safety researcher is senior (>4 years experience), the other junior. We would like to make our critiques non-anonymously but unfortunately believe this would be professionally unwise. Further, we believe our criticisms stand on their own. Though we have done our best to remain impartial, readers should not assume that we are completely unbiased or don’t have anything to personally or professionally gain from publishing these critiques. We take the benefits and drawbacks of the anonymous nature of our post seriously, and are open to feedback on anything we might have done better.

The first post in this series will cover Redwood Research (Redwood). Redwood is a non-profit started in 2021 working on technical AI safety (TAIS) alignment research. Their approach is heavily informed by the work of Paul Christiano, who runs the Alignment Research Center (ARC), and previously ran the language model alignment team at OpenAI. Paul originally proposed one of Redwood's original projects and is on Redwood’s board. Redwood has strong connections with central EA leadership and funders, has received significant funding since its inception, recruits almost exclusively from the EA movement, and partly acts as a gatekeeper to central EA institutions.

We shared a draft of this document with Redwood prior to publication and are grateful for their feedback and corrections (we recommend others also reach out similarly). We’ve also invited them to share their views in the comments of this post.

We would like to also invite others to share their thoughts in the comments openly if you feel comfortable, or contribute anonymously via this form. We will add inputs from there to the comments section of this post, but will likely not be updating the main body of the post as a result (unless comments catch errors in our writing).

Summary of our views

We believe that Redwood has some serious flaws as an org, yet has received a significant amount of funding from a central EA grantmaker (Open Philanthropy)...
How Large Language Models Nuke our Naive Notions of Truth and Reality
by Sean Lee — 2023-04-17T18:08:35.585Z

When AI can mimic human word strings by “meaningless computation”, what does that say about human word strings?

A Mirror Image? - by Stable Diffusion

The best generative language models like ChatGPT-4 still astonish us with both what they can and can’t do. And that's for one ostensibly simple reason: an AI still doesn’t “understand” what it’s doing. It can’t even “know” what we want it from it; only descend to whatever reward function it is given. That infamously introduces weird failure modes galore, many of which will likely be with us for a while (along with other issues like endless generation of

fake news and propaganda

). But this essay isn’t about AI helping, replacing or destroying us. It’s not about what incalculable admixture of fantastic and horrible awaits us in the coming AI wonderland I keep calling the Algorithmocene.

This post is instead about how generative AI creates an urgent need for all of us to stop for something almost no one has time for in our breathless, brave new AI world: a moment of metaphysical philosophy. Specifically, why now is a really good time to question how we think we know anything. What does it mean when we humans claim something is "true" or "real"? I would argue that an AI-powered post-truth world makes a fresh examination from first principles more urgent than ever.

So here are my pedestrian's two cents.

Let’s start with the basics. Everyone is quick to point out that generative language platforms are just word prediction machines based on statistical models. They binge on ginormous training datasets and learn which words we humans tend to put together in which combinations in which contexts. That LLMs now do this so well contains an important lesson about us humans. Our word strings may be infinitely variable, but they also have a very high degree of predictability. That's a key point to be picked up later.

For now (and isolated - ahem -

concerns about AI sentience

notwithstanding), let’s simply accept that all computers today mindlessly compute without “understanding” what they’re doing. But given how closely AI can now mimic human language, the obvious questions we should be asking ourselves are: How are our human words principally different? What do we humans mean when we claim to “understand” something? How do we know what is “truth”?

After all, we humans are more than capable of creating our own absurd chains of...
An alternative of PPO towards alignment
by ml hkust — 2023-04-17T17:58:09.425Z

Introduction

General-purpose foundation models, especially large language models (LLMs) such as ChatGPT, have demonstrated extraordinary capabilities in performing various tasks that were once challenging. However, we believe that one model cannot rule them all. Further fine-tuning is necessary to achieve better performance in specialized tasks or domains. The standard approaches for fine-tuning these models include:

Continuous pretraining on specific domains so that LLMs can acquire knowledge in those domains
Task tuning on specific tasks so that LLMs can deal with downstream tasks
Instruction tuning to endow LLMs the ability to comply with specialized natural language instructions and complete tasks required by those instructions
Alignment tuning to teach LLMs conversational skills in accordance with human preferences.

Alignment, in particular, is crucial for ensuring the safety of LLMs before deployment in the real world. Today we introduce a new alignment algorithm RAFT [1] which is more effective than traditional methods such as PPO. RAFT mitigates the issue of bias that could emerge in LLM responses. Using RAFT for aligning LLMs offers numerous benefits, including the ability to disentangle unwanted biases from the LLM's language production while maintaining fluency levels consistently.

Checkout the paper

https://arxiv.org/abs/2304.06767

.

Its implementation is available from

https://github.com/OptimalScale/LMFlow

.

RAFT Alignment

Alignment is a critical aspect of training large language models (LLMs) like ChatGPT. One key benefit of alignment is that it helps the model conform to human language habits, improving its performance in tasks such as question answering.

A common approach for alignment involves using reinforcement learning with human feedback (RLHF), as described in InstructGPT [2]. In this approach, human labeled data is used to train a reward model. A reinforcement learning algorithm (e.g., PPO) is then used to adjust the model's behavior according to the reward model. However, PPO and other reinforcement learning algorithms heavily rely on backpropagation, resulting in high training costs and instability.

To address these issues, we proposed a new alignment algorithm called RAFT (Reward rAnked Fine-Tuning), which uses sample ranking to select the most preferred samples from large model...