trentmkelly/LessWrong-43k huggingface dataset:
Apply to HAIST/MAIA’s AI Governance Workshop in DC (Feb 17-20)
by Phosphorous — 2023-01-31T02:06:54.656Z

The

Harvard AI Safety Team

and

MIT AI Alignment

, supported by the Cambridge Boston Alignment Initiative (

CBAI

), are excited to announce a 3-day workshop on AI Governance, Policy, and Strategy (GPS) on President’s Day weekend (February 17th-20th) in Washington, DC.

The workshop is designed for those looking to explore or accelerate a career in AI-related policymaking or GPS research, and it will have a specific emphasis on mitigating catastrophic risks from advanced AI. Participants will have the opportunity to meet career professionals working in research, technology policy, and national security. Travel and accommodation will be provided.

Applications open today and are due at 11:59 EST on February 3rd.

Apply now

. The application consists of several short answer questions; we expect it to take 30-40 minutes. We’ll release decisions by February 5th.

We expect the application process to be competitive; existential risk or AI safety-oriented groups are encouraged to

recommend applicants with this link.

Dates & Logistics

Participants should plan to arrive at the venue (in Arlington, about a 10 minute drive from DCA) by 6pm on Friday, February 17th, and leave around 3:30pm on Monday, February 20th. We will provide housing and food, and will reimburse travel costs.

Audience

We’re targeting the workshop at two different groups:

Early-career professionals and undergraduate or graduate students (at any university) who work on reducing risks from advanced AI or who have some previous engagement with these risks and are considering a career focused on mitigating them through governance, policy, or strategy.
Harvard and MIT undergraduate or graduate students who are interested in AI policy and strategy, but who have not previously substantially engaged with catastrophic risks from advanced AI. This group will have previously met for three reading-and-discussion sessions of two hours each, which will provide them with basic background on risks from advanced AI and transformative AI timelines.

We are especially interested in applicants with a technical background who are interested in exploring GPS.

Content

The workshop will cover topics in AI GPS including (but not limited to):

The inputs to AI progress (hardware, algorithms, and data) and the opportunities these present for governance.
The strategic implications

...
EA & LW Forum Weekly Summary (23rd - 29th Jan '23)
by Zoe Williams — 2023-01-31T00:36:14.553Z

Supported by Rethink Priorities
This is part of a weekly series summarizing the top posts on the EA and LW forums - you can see the full collection here. The first post includes some details on purpose and methodology. Feedback, thoughts, and corrections are welcomed.

If you'd like to receive these summaries via email, you can subscribe here.

Podcast version: Subscribe on your favorite podcast app by searching for 'EA Forum Podcast (Summaries)'. A big thanks to Coleman Snell for producing these!

Philosophy and Methodologies

Rethink Priorities’ Welfare Range Estimates

by Bob Fischer

The author builds off analysis in the rest of the

Moral Weight Project Sequence

to offer estimates of the ‘welfare range’ of 11 farmed species. A welfare range is the estimated difference between the most intensely positively valenced state (pleasure) and negatively valenced state (pain) that members of the species can experience relative to humans:

The estimates should be seen as best-available placeholders, a starting point for further empirically-driven research into welfare ranges.

Incorporating uncertainty, and conditional on hedonism and sentience, they make the following estimates:

70% credence that none of the vertebrate nonhuman animals of interest have a welfare range that’s more than double any others.
65% credence that the welfare ranges of humans and the other vertebrate animals of interest are within an order of magnitude of each other.
60% credence that the invertebrates of interest have welfare ranges within two orders of magnitude of the vertebrate animals of interest.

Existential Risk Modelling with Continuous-Time Markov Chains

by Mohammad Ismam Huda

Continuous-Time Markov Chains

(CTMCs) are a popular tool used throughout industry (e.g. insurance, epidemiology) and academia to model multi-state systems. Advantages of the methodology for x-risk modeling include the ability to model multiple competing x-risks, incorporate things that modify and interact with x-risks but aren’t x-risks themselves (eg. 3°C warming scenario) and ability to assume x-risk probabilities vary with time.

The author discusses how CTMCs work, compares them to other approaches such as discrete markov chain models, and produces a model based off of Toby Ord’s probability estimates in The Precipice for demonstration.

Object Level Interventions / Reviews

AI

What a compute-centric fram

...
Saying things because they sound good
by Adam Zerner — 2023-01-31T00:17:26.772Z

There is a memory that has always stuck with me. In sixth grade I remember sitting on the floor at a friend's house. We were supposed to write an essay together. I distinctly remember him proposing that I should write the first version and then he'll go over it afterwards and make it sound good. Because he's better at making things sound good than I am, and I'm better at the content.

I was annoyed. Not because he slyly wanted to escape doing any of the real work. At least not primarily. I was mainly annoyed at the claim that I'm not good at making things sound good.

Basically, he wanted to go through the essay and use "bigger" and "fancier" words and phrases. Ie. he'd replace "wanted to" with "had the desire to" and "essay" with "article of collaborative verbal expression" or something. He thought those sounded better and that my simple phrasing didn't sound very good.[1]

I hated that. Those didn't sound better. They sounded like pretentious douchebaggery.

Still, it could be worse. At least that sort of pretentious douchebaggery I'm describing is accurate. There are other times where, in an effort to make something "sound good", people say things that are untrue. For example, the video 5 Meal Prep Bowls In Less Than 1 Hour starts off with the following statement:

Breakfast is the most important meal of the day. It gives you the energy to keep you going. And that's why we're making these absolutely delicious breakfast meal prep bowls.

I'm not in the author's head so I can't be too confident, but I'm probably about 70% confident that he doesn't actually believe that. Or at least that he himself isn't confident in those first two statements.[2] I think the primary reason he said them is because they sounded good.

To elaborate, I'd like to distinguish between a few different things here.

Saying things you know to be false.

Saying things you suspect are false.

Saying things you are unsure of.

The author of the video is probably in category #2, I think. It doesn't feel like category #1 to me. Category #3 wouldn't surprise me though. To be clear, each category is still a sin. It's just that some sins are worse than others.

Anyway, is this a big problem? Medium-sized? Small? Trivial? I'm not sure.

I think that people generally have decent bullshit detectors. On the one hand, when I heard the start to that video I brushed it off as something that he isn't actually confi...
Peter Thiel's speech at Oxford Debating Union on technological stagnation, Nuclear weapons, COVID, Environment, Alignment, 'anti-anti anti-anti-classical liberalism', Bostrom, LW, etc.
by M. Y. Zuo — 2023-01-30T23:31:26.134Z

This is a very interesting speech covering a lot of the popular topics, and it's only 27 minutes long.

Beware though his views are usually provocative and especially here it seems like certain ideas are expressed to challenge one group or several simultaneously, so I recommended it with caveats.

For example, his opening line is:

"I'm always reminded of a question a colleague of mine like to ask 'what is the antonym of diversity?', 'what word is the single antonym of diversity?'…university."
Medical Image Registration: The obscure field where Deep Mesaoptimizers are already at the top of the benchmarks. (post + colab notebook) 
by Hastings — 2023-01-30T22:46:31.352Z

Background: Introduction to Image Registration

Image registration, or finding the corresponding points in a pair of, eg, MRI scans, is one of the fundamental tasks in medical image computing- needed for aligning post/pre-op brain scans, tracking lung motion during inhalation, computing statistics across populations etc. The standard formulation of this task is, given two images Imoving and Ifixed, find an invertible function φ such that the point φ−1(x) in Imoving corresponds to the point x in Ifixed, (or equivalently, a point y in Imoving moves to φ(y) in Ifixed). For two images of the same body part on the same person, such as the common task of aligning an MRI of a prostate to an ultrasound of that same organ, "corresponds" is extremely well defined: two points correspond if they contain the same physical cell or piece of cartilage. For two images of different people, the correct answer is more nebulous: certainly the center of Alex's pineal gland has a corresponding point in Bob's head, but what about points in the grey matter, which is uniquely folded in every person?
Instead of hunting correspondences directly, we could introduce a surrogate objective. The traditional approach is to parametrize φ by θ then introduce a reward function that scores various φθ, such as

Ex∈Ω[(Imoving(φ−1θ(x))−Ifixed(x))2]+λLreg,

or

"The squared difference between the warped moving image and the fixed image, plus a penalty to encourage smoothness".

The minimizing value of θ is then found via some kind of search. For details, a good introduction: (2004)[1]. For a modern, powerful implementation see ANTs, NiftyReg

Neural Networks Arrive

Finding the minimizing value of θ turns out to be quite expensive computationally- the leading package ANTs can take nearly an hour to align two high resolution images on its highest setting (named SyNOnly if you want to dig into it). An alternative has emerged in recent years: optimize a neural network Φθ that takes in two images and directly spits out a function φ−1. The initial approach was to register a large collection of images using the traditional methods to create a dataset (Imovingi,Ifixedi,φi) and then train using a standard supervised learning objective along the lines of&n...
Humans Can Be Manually Strategic
by Screwtape — 2023-01-30T22:35:39.010Z

Summary: Humans don't always take the time to think and plan very well. That does not mean we can’t think about our goals and how we want to go about achieving them. This meetup is an attempt to prompt strategic thinking.

Tags: Repeatable, Investment, Highly Experimental. (Highly Experimental here means that things like this have been run, but I haven’t actually ever run a meetup following these exact steps. Lots of variations on it exist and I don’t have any strong opinions on which is best.)

Purpose: Humans are capable of strategic thinking and problem solving, we just don’t do it automatically. The purpose of this meetup is to prompt people to actually think about what they want and how to get it.

Materials: Some way of keeping time is required, such as a kitchen timer or a smartphone with a timer app. Having a bunch of pencil, paper, and writing surfaces may be useful for the participants and is recommended. Some people might want to use devices like laptops or cell phones; I suggest if you want to use one that you turn the wifi off.

Announcement Text: Humans are not automatically strategic.

We do not always take time to think through what we want to achieve and how we plan to achieve that. It isn’t a factory default setting in people to consider how our plans might go wrong, or whether this is really what we want. That can work out for us, especially when what we want to do is a normal, expected thing that most people set out to do and succeed at.

Many of us aren’t living normal, expected life paths. Today, we’re going to take some time to sit and reason about our goals and do deliberately what we might not do automatically.

Description:

Explain the following steps to the group, then run the timer as everyone goes through the steps. (If you want to adjust the questions asked or the way people group together, go ahead! Just plan that beforehand and have a different script to read off of.) If someone is unsure what time span they should be planning for, the recommendation is to plan for the next quarter, aka the next three months. That said, if it makes more sense for a participant to plan around the next ten years or the next two weeks, that’s fine.

First, what do we want? What are the values we want satisfied, what do we feel is missing? Take five minutes, by the clock, and just list out what it is you want. You might say rough categories

...
Why I hate the "accident vs. misuse" AI x-risk dichotomy (quick thoughts on "structural risk")
by David Scott Krueger (formerly: capybaralet) — 2023-01-30T18:50:17.613Z

I think the large majority of AI x-risk is "structural". Like climate change.
Here's a good primer on structural risk (note that structural risk is not a synonym for "not caused by out-of-control AI"): https://forum.effectivealtruism.org/posts/oqveRcMwRMDk6SYXM/clarifications-about-structural-risk-from-ai
I am shocked and amazed and dismayed that more people do not seem to view it this way, even among the AI x-safety community. Heck, even Eliezer's stories of doom are steeped in structural risk (race dynamics, teams rationalizing cutting corners on safety when they should know better, etc.)

I expect irresponsible, reckless, negligent deployment of AI systems without proper accounting of externalities. I consider this the default for any technology with potential for significant externalities, absent regulation.

A slide on legal notions of culpability (modified, original from Henry Ashton).

When something bad happens in such a context, calling it "accident risk" absolves those researching, developing, and/or deploying the technology of responsibility. They should have known better. Some of them almost certainly did. Rationalization, oversight, and misaligned incentives were almost certainly at play. Failing to predict the particular failure mode encountered is no excuse. Having "good intentions" is no excuse.
So... it must be misuse then, right? Well, no. Calling it "misuse" suggests that those researching, developing, and/or deploying the technology set out with nefarious purposes and the technology achieved precisely what they intended. But ~nobody wants to destroy the world. It's just that most people are somewhat selfish and so are willing to trade some x-risk for a large personal benefit.
In summary, saying "accident" makes it sounds like an unpredictable effect, instead of painfully obviously risk that was not taken seriously enough. Saying "misuse" makes it sounds like some supervillian or extremist deliberately destroying the world. While some risks may have something more of a flavor or accident or misuse depending on how obvious the risk was, neither of these pictures gives a remotely accurate picture of the nature of the problem.
I think this makes it a harmful meme, and ask that others stop making this distinction (without appropriate caveats), and join me in pointi...
2022 Unofficial LessWrong General Census 
by Screwtape — 2023-01-30T18:36:30.616Z

The Less Wrong General Census is unofficially back!

In days of yore, there was an annual census of the site users. That census has come again in an unofficial form! Click here to take the survey! [EDIT: SURVEY CLOSED, DO NOT TAKE!] It can take as little as five minutes if you just want to fill out the basics, and can take longer if you want to fill other the optional sections out. The survey will be open from today until February 27th, at which point it will close.

Once the census is closed, I'll remove the very private information, then make some summaries of the data and write up what I found in a post that will be linked from here. I'll also release a csv of all the responses marked "fine to include" as a public dataset.

There once was a tradition associated with the annual census where, if you take the survey, you could comment saying you have done so and people will upvote you and you will get karma. This census is unofficial! I'm unclear as to the official status of the traditional censuses. Anyway, you could try the upvote thing if you wanted?

An Argument For Greater Visibility

Update: As of 2023-01-31 01:00, it looks like this has front page status. Thank you! Argument left for reference.

In the old days, the survey author could promote their post to Main and get more people to see it. Front page is the current equivalent to that, and the 2020 survey stayed as a Personal Blog post likely leading to the lower number of responses. Therefore, I would like to make a case to Administration and the Sunshine Regiment that this is worth frontpaging. Three years ago there was a comment about maybe having a more official survey, but as far as I'm aware that hasn't materialized. I believe perfect may be the enemy of good, and would rather have a Google Form and a CSV than no census at all.

Community-focused discussion is meant to be kept to a minimum on the front page. I would like to argue that the Annual Census should be a once-a-year case where front paging something that's community-focused. It benefits from being seen by more people in a direct "there are more responses and so the statistics give us more information" sense. By definition, there is one annual census a year and so there's a clear line preventing similar surveys from proliferating and flooding the first page. (It's possible to wind up with two in a given calendar year; since this one is happen...
Call for submissions: “(In)human Values and Artificial Agency”, ALIFE 2023
by the gears to ascension — 2023-01-30T17:37:48.882Z
Link: https://humanvaluesandartificialagency.com/

key points:

Cash prize of $500 for the best presentation.
Deadline 3 March, 2023.
Organized by Simon McGregor (University of Sussex), Rory Greig (DeepMind), Chris Buckley (University of Sussex)

ALIFE 2023 (the 2023 conference on Artificial Life) will feature a Special Session on “(In)human Values and Artificial Agency”. This session focuses on issues at the intersection of AI Safety and Artificial Life. We invite the submission of research papers, or extended abstracts, that deal with related topics.

We particularly encourage submissions from researchers in the AI Safety community, who might not otherwise have considered submitting to ALIFE 2023.

...

EXAMPLES OF A-LIFE RELATED TOPICS

Here are a few examples of topics that engage with A-Life concerns:

Abstracted simulation models of complex emergent phenomena
Concepts such as embodiment, the extended mind, enactivism, sensorimotor contingency theory, or autopoiesis
Collective behaviour and emergent behaviour
Fundamental theories of agency or theories of cognition
Teleological and goal directed behaviour of artificial agents
Specific instances of adaptive phenomena in biological, social or robotic systems
Thermodynamic and statistical-mechanical analyses
Evolutionary, ecological or cybernetic perspectives

EXAMPLES OF AI SAFETY RELATED TOPICS

Here are a few examples of topics that engage with AI Safety concerns:

Assessment of distinctive risks, failure modes or threat models for artificial adaptive systems
Fundamental theories of agency, theories of cognition or theories of optimization.
Embedded Agency, formalizations of agent-environment interactions that account for embeddedness, detecting agents and representations of agents’ goals.
Selection theorems – how selection pressures and training environments determine agent properties.
Multi-agent cooperation; inferring / learning human values and aggregating preferences.
Techniques for aligning AI models to human preferences, such as Reinforcement Learning from Human Feedback (RLHF)
Goal Misgeneralisation – how agent’s goals generalise to new environments
Mechanistic interpretability of learned / evolved agents (“digital neuroscience”)
Improving fairness and reducing harm from machine learning models deployed in the real world.
Loss of human agency from increasing automation
What I mean by "alignment is in large part about making cognition aimable at all"
by So8res — 2023-01-30T15:22:09.294Z

(Epistemic status: attempting to clear up a misunderstanding about points I have attempted to make in the past. This post is not intended as an argument for those points.)

I have long said that the lion's share of the AI alignment problem seems to me to be about pointing powerful cognition at anything at all, rather than figuring out what to point it at.

It’s recently come to my attention that some people have misunderstood this point, so I’ll attempt to clarify here.

In saying the above, I do not mean the following:

(1) Any practical AI that you're dealing with will necessarily be cleanly internally organized around pursuing a single objective. Managing to put your own objective into this "goal slot" (as opposed to having the goal slot set by random happenstance) is a central difficult challenge. [Reminder: I am not asserting this]

Instead, I mean something more like the following:

(2) By default, the first minds humanity makes will be a terrible spaghetti-code

mess

, with no clearly-factored-out "goal" that the surrounding cognition pursues in a unified way. The mind will be more like a pile of complex, messily interconnected kludges, whose ultimate behavior is sensitive to the

particulars

of how it reflects and irons out the tensions within itself over time.

Making the AI even have something vaguely nearing a 'goal slot' that is stable under various operating pressures (such as reflection) during the course of operation, is an undertaking that requires mastery of cognition in its own right—mastery of a sort that we’re exceedingly unlikely to achieve if we just try to figure out how to build a mind, without filtering for approaches that are more legible and aimable.

Separately and independently, I believe that by the time an AI has fully completed the transition to hard superintelligence, it will have ironed out a bunch of the wrinkles and will be oriented around a particular goal (at least behaviorally, cf.

efficiency

—though I would also guess that the mental architecture ultimately ends up cleanly-factored (albeit not in a way that creates a single point of failure, goalwise)).

(But this doesn’t help solve the problem, because by the time the strongly superintelligent AI has ironed itself out into something with a "goal slot", it's not letting you touch it.)

Furthermore, insofar as the AI is capable of finding actions t...