trentmkelly/LessWrong-43k huggingface dataset:
Levels of safety for AI and other technologies
by jasoncrawford — 2023-06-28T18:35:52.933Z
Link: https://rootsofprogress.org/levels-of-technology-safety

What does it mean for AI to be “safe”?

Right now there is a lot of debate about AI safety. But people often end up talking past each other because they’re not using the same definitions or standards.

For the sake of productive debates, let me propose some distinctions to add clarity:

A scale of technology safety

Here are four levels of safety for any given technology:

So dangerous that no one can use it safely
Safe only if used very carefully
Safe unless used recklessly or maliciously
So safe that no one can cause serious harm with it

Another way to think about this is, roughly:

Level 1 is generally banned
Level 2 is generally restricted to trained professionals
Level 3 can be used by anyone, perhaps with a basic license/permit
Level 4 requires no special safety measures

All of this is oversimplified, but hopefully useful.

Examples

The most harmful drugs and other chemicals, and arguably the most dangerous pathogens and most destructive weapons of war, are level 1.

Operating a power plant, or flying a commercial airplane, is level 2: only for trained professionals.

Driving a car, or taking prescription drugs, is level 3: we make this generally accessible, perhaps with a modest amount of instruction, and perhaps requiring a license or some other kind of permit. (Note that prescribing drugs is level 2.)

Many everyday or household technologies are level 4. Anything you are allowed to take on an airplane is certainly level 4.

Caveats

Again, all of this is oversimplified. Just to indicate some of the complexities:

There are more than four levels you could identify; maybe it’s a continuous spectrum.
“Safe” doesn’t mean absolutely or perfectly safe, but rather reasonably or acceptably safe: it depends on the scope and magnitude of potential harm, and on a society’s general standards for safety.
Safety is not an inherent property of a technology, but of a technology as embedded in a social system, including law and culture.
How tightly we regulate things, in general, is not only about safety but is a tradeoff between safety and the importance and value of a technology.
Accidental vs. deliberate misuse are arguably different things that might require different scales. Whether we have special security measures in place to prevent criminals or terrorists accessing a technology may not be perfectly correlated with what safety level you would designate a technology when considering only accidents

...
LeCun says making a utility function is intractable
by Iknownothing — 2023-06-28T18:02:13.721Z

Has there been any post going over how LeCun's alignment plan is very fluffy and he hasn't responded to any criticism of it at all? LeCun's paper: https://openreview.net/forum?id=BZ5a1r-kVsf

One of the criticisms, which goes over how LeCun's idea of an Intrinsic Cost Module, his idea for how to motivate an ASI to want to do good things lacks any real substance: https://www.alignmentforum.org/posts/C5guLAx7ieQoowv3d/lecun-s-a-path-towards-autonomous-machine-intelligence-has-1#1__Background__the_paper_s_descriptions_of_the__Intrinsic_Cost_module_

LeCun himself says in the paper:
"In most RL settings the reward (or the cost, which is a negative reward) is fed to the agent by the environment. In other words, Intrinsic Cost module is the environment itself, and is therefore an unknown function. "
Essentially, what this means is that LeCun says there's no way to make a function to make an ASI want to do specific things and his plan is to give it as much information as possible and it'll pretty much automatically become good from that.
LeCun's quote is from the bottom of page 48.
My research agenda in agent foundations
by Alex_Altair — 2023-06-28T18:00:27.813Z

Since the beginning of 2022, I've been doing independent AI alignment research in agent foundations. The stage and scope of research that I am doing is roughly equivalent to starting a PhD program.

It has been somewhat popular for people in alignment to write up research agendas over the last couple of years. I've been vaguely hesitant to post a research agenda for myself, because to me it sounds like a claim of something stronger than is true. Specifically it feels like a relatively strong claim that I intend to be doing these specific things for a long time, that I expect to get results, and maybe even that I'm part of some larger organization. It's not clear to me how other readers interpret this, but hopefully I can dispel that impression with sufficient disclaimers throughout.

The purposes of this post are to:

produce for myself a clearer statement of what I am doing and why,
make myself more legible to any other interested parties, and
invite some criticism of my plans.

I've written in the semi-casual tone common to LessWrong posts, and mentioned many things throughout that may not make sense if you come from outside the rationalist community.

Context

Like many people in AI alignment, I am not working in the standard academic track, so it's worthwhile to give more context. If you just want to know concretely what I work on, skip down to the section on research problems.

All is flux

As implied above, I expect lots of things about my research to change continuously. I have no external commitments that would heavily incentivize me to complete any particular task, so I stop doing something as soon as I decide it's not valuable anymore. (There are obvious downsides to this as well, but I'm comfortable with my ability to navigate them.)

Still learning

My current level of math ability is something like halfway through the first semester of grad school. My most natural inclination in researching would be, step 1: learn all math. (I would never get to step 2). To combat this, I have tried to work on problems first, and only start to study a field of math when I feel like it would have helped a few times already. I think this has been quite a successful heuristic to follow so far, and I think it's also made me more confident in my recent decision to start focusing a lot more of my time on doing the equivalent of taking grad classes. I continue to do regular iterations of tr...
AI Incident Sharing - Best practices from other fields and a comprehensive list of existing platforms
by Štěpán Los — 2023-06-28T17:21:13.991Z

Purpose of this post: The purpose of this post is three-fold: 1) highlight the importance of incident sharing and share best practices from adjacent fields to AI safety 2) collect tentative and existing ideas of implementing a widely used AI incident database and 3) serve as a comprehensive list of existing AI incident databases as of June 2023.

Epistemic status: I have spent around 25+ hours researching this topic and this list is by no means meant to be exhaustive. It should give the reader an idea of relevant adjacent fields where incident databases are common practice and should highlight some of the more widely used AI incident databases which exist to date. Please feel encouraged to comment any relevant ideas or databases that I have missed, I will periodically update the list if I find anything new.

Motivation for AI Incident Databases

Sharing incidents, near misses and best practices in AI development decreases the likelihood of future malfunctions and large-scale risk. To mitigate risks from AI systems, it is vital to understand the causes and effects of their failures. Many AI governance organizations, including

FLI

and

CSET

, recommend creating a detailed database of AI incidents to enable information-sharing between developers, government and the public. Generally, information-sharing between different stakeholders 1) enables quicker identification of security issues and 2) boosts risk-mitigation by helping companies take appropriate actions against vulnerabilities.

Best practices from other fields

National Transportation Safety Board (

NTSB

) publishes and maintains a

database

of aviation accidents, including detailed reports evaluating

technological and environmental factors as well as potential human errors causing the incident

. The

reports

include descriptions of the aircraft, how it was operated by the flight crew, environmental conditions, consequences of event, probable cause of accident, etc. The meticulous record-keeping and best-practices recommendations

are one of the key factors

behind the

steady

decline in yearly aviation accidents, making air travel one of the

safest form of travel

.

National Highway Traffic Safety Administration (NHTSA) maintains a comprehensive database recording the number of crashes and fatal injuries caused by automobile and motor ve...
The Case for Overconfidence is Overstated
by Kevin Dorst — 2023-06-28T17:21:06.160Z
Link: https://kevindorst.substack.com/p/the-case-for-overconfidence-is-overstated

(Written with Matthew Mandelkern.)

TLDR: When asked to make interval estimates, people appear radically overconfident—far more so than when their estimates are elicited in other ways. But an accuracy-informativity tradeoff can explain this: asking for intervals incentivizes precision in a way that the other methods don’t.

Pencils ready! For each of the following quantities, name the narrowest interval that you’re 90%-confident contains the true value:

The population of the United Kingdom in 2020.

The distance from San Francisco to Boston.

The proportion of Americans who believe in God.

The height of an aircraft carrier.

Your 90%-confidence intervals are calibrated if 90% of them contain the true value. They are overprecise if less than 90% contain it (they are too narrow), and underprecise if more than 90% do.

We bet that at least one of your intervals failed to contain the true value.[1] If so, then at most 75% of your intervals contained the correct answer, making you overprecise on this test.

You’re in good company. Overprecision is one of the most robust findings in judgment and decision-making: asked to give confidence intervals for unknown quantities, people are almost always overprecise.

The standard interpretation? People are systematically overconfident: more confident than they rationally should be. Overconfidence is blamed for many societal ills—from market failures to polarization to wars. Daniel Kahneman summed it up bluntly: “What would I eliminate if I had a magic wand? Overconfidence.”

But this is too quick. There are good reasons to think that when people answer questions under uncertainty, they form guesses that are sensitive to an accuracy-informativity tradeoff: they want their guess to be broad enough to be accurate, but narrow enough to be informative.

When asked who’s going to win the Republican nomination, “Trump” is quite informative, but “Trump or DeSantis” is more likely to be accurate. Which you guess depends on how you trade off accuracy and informativity: if you just want to say something true, you’ll guess “Trump or Desantis”, while if you want to say some thing informative, you’ll guess “Trump”.

So accuracy and informativity compete. This competition is especially stark if you’re giving an interval estimate. A wide interval (“the population of the UK is between 1 million and 1 billion”) is cer...
When do "brains beat brawn" in Chess? An experiment
by titotal — 2023-06-28T13:33:23.854Z
Link: https://titotal.substack.com/p/when-do-brains-beat-brawn-in-chess

As a kid, I really enjoyed chess, as did my dad. Naturally, I wanted to play him. The problem was that my dad was extremely good. He was playing local tournaments and could play blindfolded, while I was, well, a child. In a purely skill based game like chess, an extreme skill imbalance means that the more skilled player essentially always wins, and in chess, it ends up being a slaughter that is no fun for either player. Not many kids have the patience to lose dozens of games in a row and never even get close to victory.

This is a common problem in chess, with a well established solution: It’s called “odds”. When two players with very different skill levels want to play each other, the stronger player will start off with some pieces missing from their side of the board. “Odds of a queen”, for example, refers to taking the queen of the stronger player off the board. When I played “odds of a queen” against my dad, the games were fun again, as I had a chance of victory and he could play as normal without acting intentionally dumb. The resource imbalance of the missing queen made the difference. I still lost a bunch though, because I blundered pieces.

Now I am a fully blown adult with a PhD, I’m a lot better at chess than I was a kid. I’m better than most of my friends that play, but I never reached my dad’s level of chess obsession. I never bothered to learn any openings in real detail, or do studies on complex endgames. I mainly just play online blitz and rapid games for fun. My rating on lichess blitz is 1200, on rapid is 1600, which some calculator online said would place me at ~1100 ELO on the FIDE scale.

In comparison, a chess master is ~2200, a grandmaster is ~2700. The top chess player Magnus Carlsen is at an incredible 2853. ELO ratings can be used to estimate the chance of victory in a matchup, although the estimates are somewhat crude for very large skill differences. Under this calculation, the chance of me beating a 2200 player is 1 in 500, while the chance of me beating Magnus Carlsen would be 1 in 24000. Although realistically, the real odds would be less about the ELO and more on whether he was drunk while playing me.

Stockfish 14 has an estimated ELO of 3549. In chess, AI is already superhuman, and has long since blasted past the best players in the world. When human players train, they use the supercomputers as standards. If you ask for a gam...
Giving an evolutionary explanation for Kahneman and Tversky's insights on subjective satisfaction
by Lionel — 2023-06-28T12:17:18.609Z
Link: https://lionelpage.substack.com/p/its-not-a-bug-its-a-feature-revisiting

One of Kahneman and Tversky's famous insights from Prospect Theory is that subjective satisfaction is determined by gains and losses relative to a reference point. This behavioural pattern is often perceived as a flaw in our approach to assessing the value of the options we face.

A robust body of literature in economic theory, which intersects with neuroscience, proposes that this aspect of subjective satisfaction is in fact likely to be the optimal solution that evolution has come up with for designing a hedonic reward system. This system guides us in making sound decisions.

This interpretation, however, is often overlooked in behavioural economics. In this Substack post, I present this idea in simple terms. I believe this serves as a perfect example of how richer economic theories and an evolutionary perspective can help us understand behavioural patterns - not as bugs, but as meaningful features.
Nature: "Stop talking about tomorrow’s AI doomsday when AI poses risks today"
by Ben Smith — 2023-06-28T05:59:49.015Z
Link: https://www.nature.com/articles/d41586-023-02094-7

Overall, a headline that seems counterproductive and needlessly divisive.

I worry very much that coverage like this has the potential to bring political polarization to AI risk and it would be extremely damaging for the prospect of regulation if one side of the US Congress/Senate decided AI risk was something only their outgroup is concerned about, for nefarious reasons.

but in the spirit of charity, here are perhaps the strongest points of a weak article:

the spectre of AI as an all-powerful machine fuels competition between nations to develop AI so that they can benefit from and control it. This works to the advantage of tech firms: it encourages investment and weakens arguments for regulating the industry. An actual arms race to produce next-generation AI-powered military technology is already under way, increasing the risk of catastrophic conflict

and

governments must establish appropriate legal and regulatory frameworks, as well as applying laws that already exist

and

Researchers must play their part by building a culture of responsible AI from the bottom up. In April, the big machine-learning meeting NeurIPS (Neural Information Processing Systems) announced its adoption of a code of ethics for meeting submissions. This includes an expectation that research involving human participants has been approved by an ethical or institutional review board (IRB)

This would be great if ethical or institutional review boards were willing to restrict research that might be dangerous, but it would require a substantial change in their approach to regulating AI research.

All researchers and institutions should follow this approach, and also ensure that IRBs — or peer-review panels in cases in which no IRB exists — have the expertise to examine potentially risky AI research.

Should people worried about AI existential risk be trying to create resources for IRBs to recognize harmful AI research?

Some ominous commentary from Tyler Cowen:

Many of you focused on AGI existential risk do not much like or agree with my criticisms of that position, or perhaps you do not understand my stance, as I have seen stated a few times on Twitter. But I am telling you -- I take you far more seriously than does most of the mainstream. I keep on saying -- publish, publish, peer review, peer review -- a high mark of respect....

As it stands, contra that earlier tweet from Rob Wibl

...
Request: Put Carl Shulman's recent podcast into an organized written format
by Aryeh Englander — 2023-06-28T02:58:40.011Z

Carl Shulman recently did a very long interview with Dwarkesh Patel on the Lunar Society Podcast, which Dwarkesh released in two separate episodes because of its length. I have seen many people point to this interview as adding a lot of important high value content to the AI x-risk discussion, and I agree with that assessment. However, I don't think it's fair to tell people they need to listen to a 7 hour interview in order to get a better sense of concrete risk scenarios.

I therefore think it would be very high value if someone could organize the information that Carl presented into an accessible written format so that it can be used as a reference going forward. I think it would be even better if Carl could publish it on arxiv or in some relevant journal so that it could get the academic attention it deserves.
Prediction Market: Will I Pull "The One Ring To Rule Them All?"
by Connor Tabarrok — 2023-06-28T02:41:39.414Z
Link: https://manifold.markets/ConnorTabarrok/will-connor-tabarrok-pull-the-11-ri?r=Q29ubm9yVGFiYXJyb2s

I’m trying to save you from a fate worse than death. Recently, the Wizards of the Coast released a new Magic the Gathering/LOTR crossover pack, and there is a chance in each collectors booster pack to draw a 1/1 unique “One Ring” card!

But wait! Don’t have your will broken as I did by the Ring’s power! Instead, join me on my journey to find the Ring… and overcome it’s evil (of course) by participating in a prediction market, where you and others will be able to bet (with fake money) on whether or not I find the Ring!

Plus, you can earn yourself a pat on the back for learning how prediction markets use "the wisdom of the crowds" to collect and forecast the consensus of probabilistic effects. As a great man once said, "A price is a signal wrapped up in an incentive"!

Sound good? See the link below for Manifold Markets, a prediction market platform where you can bet with your friends. 1st time users get 500 free manifold dollar to play with!

Place your bets now!

manifold.markets/ConnorTabarrok/will-co…

Not Sold yet?

Take a look at the beautiful cards!

imgur.com/gallery/GVG5VHd
Carl Shulman on The Lunar Society (7 hour, two-part podcast)
by ESRogs — 2023-06-28T01:23:52.541Z
Link: https://www.dwarkeshpatel.com/p/carl-shulman-2

Dwarkesh's summary for part 1:

In terms of the depth and range of topics, this episode is the best I’ve done.

No part of my worldview is the same after talking with

Carl Shulman

. He's the most interesting intellectual you've never heard of.

We ended up talking for 8 hours, so I'm splitting this episode into 2 parts.

This part is about Carl’s model of an intelligence explosion, which integrates everything from:

how fast algorithmic progress & hardware improvements in AI are happening,
what primate evolution suggests about the scaling hypothesis,
how soon before AIs could do large parts of AI research themselves, and whether there would be faster and faster doublings of AI researchers,
how quickly robots produced from existing factories could take over the economy.

We also discuss the odds of a takeover based on whether the AI is aligned before the intelligence explosion happens, and Carl explains why he’s more optimistic than Eliezer.

The next part, which I’ll release next week, is about all the specific mechanisms of an AI takeover, plus a whole bunch of other galaxy brain stuff.

Maybe 3 people in the world have thought as rigorously as Carl about so many interesting topics. So this was a huge pleasure.

Dwarkesh's summary for part 2:

The second half of my 7 hour conversation with

Carl Shulman

is out!

My favorite part! And the one that had the biggest impact on my worldview.

Here, Carl lays out how an AI takeover might happen:

AI can threaten mutually assured destruction from bioweapons,
use cyber attacks to take over physical infrastructure,
build mechanical armies,
spread seed AIs we can never exterminate,
offer tech and other advantages to collaborating countries, etc

Plus we talk about a whole bunch of weird and interesting topics which Carl has thought about:

what is the far future best case scenario for humanity
what it would look like to have AI make thousands of years of intellectual progress in a month
how do we detect deception in superhuman models
does space warfare favor defense or offense
is a Malthusian state inevitable in the long run
why markets haven't priced in explosive economic growth
& much more

Carl also explains how he developed such a rigorous, thoughtful, and interdisciplinary model of the biggest problems in the world.

(Also discussed on the EA Forum here.)