trentmkelly/LessWrong-43k huggingface dataset:
Open & Welcome Thread - May 2023
by Ruby — 2023-05-02T02:58:01.690Z

If it’s worth saying, but not worth its own post, here's a place to put it.

If you are new to LessWrong, here's the place to introduce yourself. Personal stories, anecdotes, or just general comments on how you found us and what you hope to get from the site and community are invited. This is also the place to discuss feature requests and other ideas you have for the site, if you don't want to write a full top-level post.

If you're new to the community, you can start reading the Highlights from the Sequences, a collection of posts about the core ideas of LessWrong.

If you want to explore the community more, I recommend reading the Library, checking recent Curated posts, seeing if there are any meetups in your area, and checking out the Getting Started section of the LessWrong FAQ. If you want to orient to the content on the site, you can also check out the Concepts section.

The Open Thread tag is here. The Open Thread sequence is here.
Summaries of top forum posts (24th - 30th April 2023)
by Zoe Williams — 2023-05-02T02:30:59.447Z

We've just passed the half year mark for this project! If you're reading this, please consider taking this 5 minute survey - all questions optional. If you listen to the podcast version, we have a separate survey for that here. Thanks to everyone that has responded to this already!

Back to our regularly scheduled intro...

This is part of a weekly series summarizing the top posts on the EA and LW forums - you can see the full collection here. The first post includes some details on purpose and methodology. Feedback, thoughts, and corrections are welcomed.

If you'd like to receive these summaries via email, you can subscribe here.

Podcast version: Subscribe on your favorite podcast app by searching for 'EA Forum Podcast (Summaries)'. A big thanks to Coleman Snell for producing these!

Object Level Interventions / Reviews

AI

Proposals for the AI Regulatory Sandbox in Spain

by Guillem Bas, Jaime Sevilla, Mónica Ulloa

Author’s summary: “The European Union is designing a regulatory framework for artificial intelligence that could be approved by the end of 2023. This regulation prohibits unacceptable practices and stipulates requirements for AI systems in critical sectors. These obligations consist of a risk management system, a quality management system, and post-market monitoring. The legislation enforcement will be tested for the first time in Spain, in a regulatory sandbox of approximately three years. This will be a great opportunity to prepare the national ecosystem and influence the development of AI governance internationally. In this context, we present several policies to consider, including third-party auditing, the detection and evaluation of frontier AI models, red teaming exercises, and creating an incident database.”

Power laws in Speedrunning and Machine Learning

by Jaime Sevilla

Paper

by Epoch. World record progressions in video game speedrunning fit very well to a power law pattern. Due to lack of longitudinal data, the authors can’t provide definitive evidence of power-law decay in Machine learning benchmark improvements (though it is a better model than assuming no improvement over time). However, if they assume this model, it would suggest that a) machine learning benchmarks aren’t close to saturation and b) sudden large improvements are infrequent but aren’t ruled out.

No, the EMH does not imply that markets have long AGI timelines

by Jakob

Argues...
AXRP Episode 21 - Interpretability for Engineers with Stephen Casper
by DanielFilan — 2023-05-02T00:50:07.045Z

YouTube link

Lots of people in the field of machine learning study ‘interpretability’, developing tools that they say give us useful information about neural networks. But how do we know if meaningful progress is actually being made? What should we want out of these tools? In this episode, I speak to Stephen Casper about these questions, as well as about a benchmark he’s co-developed to evaluate whether interpretability tools can find ‘Trojan horses’ hidden inside neural nets.

Topics we discuss:

Interpretability for engineers

Why interpretability?

Adversaries and interpretability

Scaling interpretability

Critiques of the AI safety interpretability community

Deceptive alignment and interpretability

Benchmarking Interpretability Tools (for Deep Neural Networks) (Using Trojan Discovery)

Why Trojans?

Which interpretability tools?

Trojan generation

Evaluation

Interpretability for shaping policy

Following Casper’s work

Daniel Filan:
Hello, everybody. In this episode I’ll be speaking with Stephen Casper. Stephen was previously an intern working with me at UC Berkeley, but he’s now a Ph.D student at MIT working with Dylan Hadfield-Menell on adversaries and interpretability in machine learning. We’ll be talking about his ‘Engineer’s Interpretability Sequence’ of blog posts, well as his paper on benchmarking whether interpretability tools can find Trojan horses inside neural networks. For links to what we’re discussing, you can check the description of this episode and you can review the transcript at axrp.net. All right. Welcome to the podcast.

Stephen Casper:
Thanks. Good to be here.

Interpretability for engineers

Why interpretability?

Daniel Filan:
So from your published work, it seems like you’re really interested in neural network interpretability. Why is that?

Stephen Casper:
One part of the answer is kind of boring and unremarkable [which] is that lots of people are interested in interpretability and I have some past experience doing this, and as a result it’s become very natural and easy for me to continue to work on what I’ve gathered interests in and what I have some experience in. You definitely know that from when we worked together on this, but aside from what I’ve just come to become interested in, interpretability is interesting for a reason to so many people. It...
Getting Your Eyes On
by LoganStrohl — 2023-05-02T00:33:20.493Z

The first phase of naturalism is about identifying crucial experiences that you want to study in greater depth. It’s a lot like preparing slides for observation under a microscope.

This second phase, “Getting Your Eyes On”, is about heightening your awareness of those crucial experiences. It’s like getting your subject in focus by watching carefully while turning the adjustment knobs.

What does it mean to “get your eyes on?”

I grew up in Southern Indiana, on a farm with many acres of forest. Every spring, as soon as the May apples popped up, I’d go morel hunting with my family. Morels are brown wrinkly mushrooms that look very much like the forest floor, so hunting for them is a bit like playing Where’s Waldo, except that Waldo’s on a brown and green background, wearing brown.

But there’s a trick to it. You start out moving quickly, scanning the ground for your first mushroom. Then, when you finally see it, you don’t harvest it right away. Instead, you squat down and watch it for a full minute. Mushroom hunters call this technique “getting your eyes on”.

What I imagine is happening is that you’re training your perceptual systems to consider this particular arrangement of little grayish-brown wrinkles as “foreground”, rather than “background”. Your brain thinks that if you stare at something with full attention for a shockingly long time, it must be important. “Hey, eyes, this isn’t just another bit of deciduous detritus. It’s important. It’s salient. It’s the main thing I care about today.”

Like mushrooms, fulcrum experiences tend to blend in with the background. Before you train yourself to pay attention to them, they mostly just seem like another fleeting mundane experience in a forest of fleeting mundane experiences. They’re hard to study, because they’re practically invisible.

In naturalism, “getting your eyes on” is the process of shifting a fulcrum experience into the foreground of your life. It’s a practice that takes a previously ho-hum experience and makes it stand out, so that you notice when it happens and thereby gain the opportunity to observe it on purpose.

How To Get Your Eyes On

Start by setting out to notice your fulcrum experience just one time. You’ve probably noticed it before, as part of “Identifying Fulcrum Experiences”; but this time, you’ll have a slightly different goal when you notice: your goal this time will be...
What 2025 looks like
by Ruby — 2023-05-01T22:53:15.783Z

I wrote almost all of this in mid-March before the FLI Open Letter and Eliezer's TIME piece. Weirdly, after just six weeks I'd likely write something different. This isn't as finished/polished as I'd like, but better to ship it as is than languish incomplete forever.

Not quite two years ago, Daniel Kokotaljo wrote a highly acclaimed post about What 2026 looks like that aimed to tell a single detail future history ("trajectory") about how world events play out in coming years.

As I'm trying to orient myself to what is about to happen, I figured it'd be useful to make my own attempt at this kind of thing. Daniel was bolder than me and tried to imagine 2026 from 2021; I simply don't think I can imagine anything five years out and writing out the rest of 2023, 2024, and 2025 has given me plenty to think about further.

Daniel's vignette places a lot of attention on the size (parameters, compute) and capabilities of models. Daniel and others when imagining the future also want to describe changes in world economy (of which GDP may or may not be a good measure). Those elements feel less interesting to me to think about directly than other effects.

Major Variables

Over the years, it feels like the following are key to track.

Object-level capabilities of the models. Elaboration unnecessary

Adoption and application. It's become salient to me recently that not only is the raw "inherent" power level of models relevant to the world, but also how much they're being harnessed. Widespread use and application of AI will determine things like attention, hype, societal effects, competition, government involvement, etc.

Hype and attention. Importance, neglectedness, tractability. Many of us were used to thinking about how to achieve AI alignment in a world where not that many people were focused on Alignment or even AGI at all. As we gradually (or not that gradually) get into a world where everyone is thinking about AI, it's a pretty different gameboard.

For example, I joined the LessWrong dev team in 2018/2019 and in those days we were still trying to revive LessWrong and restore it's activity levels. Now we're preparing to handle the droves of new users who've found their way to the site due to newfound interest in AI. It significantly alters the strategy and projects we're working on, and I expect similar effects for most people in the Alignment ecosystem.

Economic effects...
Natural Selection vs Gradient Descent
by CuriousApe11 — 2023-05-01T22:16:57.538Z

Why is it so often that analogies are drawn between natural selection and gradient descent in a machine learning context? They are both optimizing over a fitness function, but isn't there an important difference in what they are optimizing over?

Natural selection is broadly optimizing over the architecture, initial parameters of the architecture, and the learning dynamics (how one updates the parameters of the architecture given data), which led to the architecture of the brain and methods of learning like STDP, in which the parameters of the architecture are the neurons of the brain.

Isn't gradient descent instead what we pick to be the learning dynamics, where we then pick our architecture (e.g. transformer) and initial parameters (e.g. Xavier initialization), so actually it makes more sense to draw an analogy between gradient descent and the optimizer learnt by natural selection (STDP, etc.), as opposed to natural selection itself?

Though natural selection is a simple optimization process, the optimizer (learning dynamics) learnt by this process could be very complex, and so reasoning like 'natural selection is simple so maybe the simplicity of gradient descent is sufficient' is not very strong?
A[I] Zombie Apocalypse Is Already Upon Us
by NickHarris — 2023-05-01T22:02:43.241Z

Having been introduced to an end-game scenario for AI with James Barrat's Our Final Invention in 2013, I've been obsessed with finding optimism in its development. I failed. Allow me to explain.

As a marketer of 20+ years, I play with human psychology. I know firsthand how easily we can be manipulated. And since 2013, I've spent hundreds and hundreds of hours reading about AI and countless more leveraging it for work, interacting with it, and thinking about it. I have come upon what I believe to be the inevitable outcome, and it's not far off.

Humanity has seen a decline in social interactions and a rise in loneliness, isolation, and distrust. This dramatic shift in social dynamics can be traced back to starting with ML recommendation engines and smartphones. We were unprepared and unable to cope, and many of us still do not possess the self-control necessary to be present in our lives with other humans. We whip out our phones in any moment of stillness to get some stimulus for fear of being alone with our thoughts or during a human-to-human interaction because the conversation has gone on longer than a TikTok. This will be amplified by AI's increasing prevalence and capability for mimicking humans.

As this technology advances, AI-generated content and interactions will become more prevalent. Whether it is the proliferation of chatbots, videos and photos indiscernible from real humans, automated customer service, improved social media algorithms, and news sites, AI is taking over many of the once human-to-human creations and interactions. This shift is creating a world where people are becoming increasingly disconnected from each other and are losing the ability to form meaningful relationships and tell facts from fiction.

One of the most significant impacts of this newer form of AI is the rise of loneliness and isolation. As people become more reliant on technology for communicating and interacting, and it becomes better at mimicking meaningful relationships, we will continue to lose the ability to have meaningful face-to-face interactions and, worse, be less interested in these types of human-to-human interactions. This will inevitably lead to amplified feelings of loneliness and isolation, as people feel disconnected from others and unable to form strong relationships. It is likely to also lead to a lack of empathy, as people become de...
Geoff Hinton Quits Google
by Adam Shai — 2023-05-01T21:03:47.806Z

The NYTimes reports that Geoff Hinton has quit his role at Google:

On Monday, however, he officially joined a growing chorus of critics who say those companies are racing toward danger with their aggressive campaign to create products based on generative artificial intelligence, the technology that powers popular chatbots like ChatGPT.

Dr. Hinton said he has quit his job at Google, where he has worked for more than a decade and became one of the most respected voices in the field, so he can freely speak out about the risks of A.I. A part of him, he said, now regrets his life’s work.

“I console myself with the normal excuse: If I hadn’t done it, somebody else would have,” Dr. Hinton said during a lengthy interview last week in the dining room of his home in Toronto, a short walk from where he and his students made their breakthrough.

https://www.nytimes.com/2023/05/01/technology/ai-google-chatbot-engineer-quits-hinton.html

Some clarification from Hinton followed:

It was already apparent that Hinton considered AI potentially dangerous, but this seems significant.
The Apprentice Thread 2
by hath — 2023-05-01T20:09:50.977Z

A while back, @Zvi set up a Schelling point for mentorship. I, personally, had some pretty good experiences from that thread, and wanted to continue the tradition, now that the old thread's been inactive for some time.

To see some of my thoughts on mentorship, check out this dialogue between Elizabeth and I.

I'll be copying Zvi's formatting for comments:

Replies to this post should take the form of any of the following:

[MENTOR]: A non-binding indication of potential interest in mentorship. Mention that you might, at some point, be interested in taking on an apprentice. This commits you to nothing. Make sure to indicate what you’d be teaching them and what project would likely be involved, and open with [MENTOR]. You are free to include contact info, or not include it and monitor replies.

Replies to this comment to indicate potential interest in being the apprentice, marked [APPRENTICE], which should include a method of further contact.

[APPRENTICE]: A non-binding indication of potential interest in being an apprentice. Mention that you might, at some point, be interested in being an apprentice. This commits you to nothing. Make sure to indicate what you’re interested in being an apprentice in and learning, and an indication of what’s motivating you.

Replies to this comment to indicate potential interest in being the mentor, marked with [MENTOR], which should include a method of further contact.

[NORMAL] You’re free to comment as per normal, but start with [NORMAL] in the top-level for clarity.
In favor of steelmanning
by jp — 2023-05-01T17:12:22.942Z

It's different than passing someone's Intellectual Turing Test

Epistemic status: Others have found my take here useful, but I haven’t engaged with the entire debate on the subject. Fairly confident that it’s useful to me, though see caveat.

Eliezer Yudkowsky recently disclaimed steelmanning on the Lex Fredman podcast. I haven't listened — I heard it second-hand. Here's a

thorough roundup

of people saying steelmanning is bad.

This post will argue that steelmanning is a useful tool.

Background

A bit of obvious background that is nevertheless worth saying: I am not omniscient. Other people know things and have models that will help me achieve my goals better. I want to steal from them like a greedy man. I want to do this because I desperately want to be right about the world. Because

it matters

.

Yet it’s sometimes legitimately hard to steal from my interlocutor. They might be building their argument up from a different set of starting models than I am. This will make them explain their ideas in a way that’s kinda weird for me. Let’s see an example.

Dogs and cats

Suppose I don’t like dogs that much.[1] My friend notices I’m feeling sad and tells me I should get a dog.

I might think their idea is crazy at first blush. If I try to pass their Intellectual Turing Test, I might focus on things they said, about how joyful dogs are. I will find this pretty uncompelling. But if instead I focus on the part of their argument that seemed strongest to me, about how their dog lies next to them and makes the house feel less empty, then suddenly I have an insight about how I should get a cat.

Steelmanning vs ITT passing

ITTs are basically easier to understand than steelmans. They are very pure. Your goal in an ITT is to understand your interlocutor. For someone on the other side, it feels very good to have someone pass your ITT. You feel understood, obviously. You may also have them ask you, “Does that capture everything?”

Steelmanning is a more subtle concept. Most sources define it as “constructing the strongest possible version of” a view you disagree with. This begs the question: according to who? I claim: according to you, the steelman-er. It is you who are trying to improve your models.[2] If your interlocutor doesn’t like your steelman, that’s a sign you might have more to learn from them. But there’s no fundamental error going on.

Play with your steelman

I claim ...
Shah (DeepMind) and Leahy (Conjecture) Discuss Alignment Cruxes
by OliviaJ — 2023-05-01T16:47:41.655Z

Preface

In December 2022, Rohin Shah (DeepMind) and Connor Leahy (Conjecture) discussed why Leahy is pessimistic about AI risk, and Shah is less so. Below is a summary and transcript.

Summary

Leahy expects discontinuities - capabilities rapidly increasing and behavior diverging far from what we aim towards - to be a core alignment difficulty. While this concept is similar to the

sharp left turn (SLT)

,

Leahy notes he doesn’t like the term SLT. SLT seems to imply deception or a malicious treacherous turn, which Leahy doesn’t see as an interesting, relevant, or necessary part.

Shah suggests they start by discussing Leahy’s SLT views.

Leahy explains he expects there to be some properties that are robustly useful for achieving many goals. As systems become powerful and applied to an increasing variety of tasks, these properties will be favored, and systems will generalize well to increasingly novel tasks.

However, the goals we intend for our AIs to learn might not generalize. Perhaps it will seem that we’ve gotten them relatively aligned them to the goal of helping humans, but once they become smart enough to access the action space of "hook all humans up to super heroin", we’ll realize they learned some superficial distortion of our intentions that allows for bizarre and undesirable actions like that. And an AI might access this intelligence level rapidly, such that it’s not visibly misaligned ahead of time. In this sense, as systems get smarter, they might become harder to understand and manage.

Shah is mostly on board: capabilities will improve, convergent instrumental goals exist, generalization will happen, goals can fail to generalize, etc. (However, he doesn’t think goal misgeneralization will definitely happen.)

Shah asks why Leahy expects AI capabilities to increase rapidly. What makes the sharp left turn sharp?

Leahy answers if we can’t rule a sharp left turn, and given a sharp left turn we expect it would be catastrophic, that’s sufficient cause for concern.

The reason Leahy expects capabilities to increase sharply, though, relates to how humanity’s capabilities developed. He sees humans as having undergone sharp left turns in the past 10000 and 150 years. Our intelligence increased rapidly, such that we went from agriculture to the space age incredibly quickly. These shifts seem to have happened for three subsequent reasons. First, ...