trentmkelly/LessWrong-43k huggingface dataset:
Why I am not a longtermist (May 2022)
by boazbarak — 2023-06-06T20:36:17.563Z
Link: https://windowsontheory.org/2022/05/23/why-i-am-not-a-longtermist/

[Posting verbatim my blog post from a year ago since it might be relevant to this audience, and I hope it could generate a good discussion. As far as I can tell, cross-posting old material is OK here, though do let me know if not, and I will delete it. I do not intend to cross-post any more old posts from my blog. Note that this post was written for non-LW audience that is not necessarily familiar with longtermism. The advice at the end is aimed mostly at technical folks rather than policy makers. A final note is that this was written before some scandals related to longtermism/EA, though these should not have an impact on the content . --Boaz]

“Longtermism” is a moral philosophy that places much more weight on the well-being of all future generations than on the current one. It holds that “positively influencing the long-term future is a key moral priority of our time,” where “long term” can be really long term, e.g., “many thousands of years in the future, or much further still.” At its core is the belief that each one of the potential quadrillion or more people that may exist in the future is as important as any single person today.

Longtermism has recently attracted attention, some of it in alarming tones. The reasoning behind longtermism is natural: if we assume that human society will continue to exist for at least a few millennia, many more people will be born in the future than are alive today. However, since predictions are famously hard to make, especially about the future, longtermism invariably gets wrapped up with probabilities. Once you do these calculations, preventing an infinitely bad outcome, even if it would only happen with tiny probability, will have infinite utility. Hence longtermism tends to focus on so-called “existential risk”: The risk that humanity will go through in an extinction event, like the one suffered by the Neanderthals or Dinosaurs, or another type of irreversible humanity-wise calamity.

This post explains why I do not subscribe to this philosophy. Let me clarify that I am not saying that all longtermists are bad people. Many “longtermists” have given generously to improve people's lives worldwide, particularly in developing countries. For example, none of the top charities of Givewell (an organization associated with the effective altruism movement, in which many prominent longtermists are members) focus...
Society Library seeking contributions for canonical AI Safety debate map
by Jarred Filmer — 2023-06-06T18:15:57.736Z
Link: https://www.societylibrary.org/society-library-updates/2023/3/31/were-mapping-ai-safetyalignment-arguments-and-issues

(I'm not directly affiliated, just a fan)

The Society Library curates arguments, claims, and evidence on high-impact social issues. They've begun work on a library for AI safety resources, and are seeking contributions from experts on papers, articles, group chats, and positions they should be aware of.

More information on the AI safety map: https://www.societylibrary.org/society-library-updates/2023/3/31/were-mapping-ai-safetyalignment-arguments-and-issues

Recently completed debate map on the Diablo Canyon Nuclear Power plant

They're also hosting a hackathon on Tuesday nights at the Internet Archive building in SF around using AI to curate, map, and communicate knowledge.

More information on the hackathon: https://www.eventbrite.com/e/ai-knowledge-mapping-hackathon-tickets-637641893067
A Playbook for AI Risk Reduction (focused on misaligned AI)
by HoldenKarnofsky — 2023-06-06T18:05:55.306Z

I sometimes hear people asking: “What is the plan for avoiding a catastrophe from misaligned AI?”

This post gives my working answer to that question - sort of. Rather than a plan, I tend to think of a playbook.1

A plan connotes something like: “By default, we ~definitely fail. To succeed, we need to hit multiple non-default goals.” If you want to start a company, you need a plan: doing nothing will definitely not result in starting a company, and there are multiple identifiable things you need to do to pull it off.

I don’t think that’s the situation with AI risk.

As I argued before, I think we have a nontrivial chance of avoiding AI takeover even in a “minimal-dignity” future - say, assuming essentially no growth from here in the size or influence of the communities and research fields focused specifically on existential risk from misaligned AI, and no highly surprising research or other insights from these communities/fields either. (This statement is not meant to make anyone relax! A nontrivial chance of survival is obviously not good enough.)

I think there are a number of things we can do that further improve the odds. My favorite interventions are such that some success with them helps a little, and a lot of success helps a lot, and they can help even if other interventions are badly neglected. I’ll list and discuss these interventions below.

So instead of a “plan” I tend to think about a “playbook”: a set of plays, each of which might be useful. We can try a bunch of them and do more of what’s working. I have takes on which interventions most need more attention on the margin, but think that for most people, personal fit is a reasonable way to prioritize between the interventions I’m listing.

Below I’ll briefly recap my overall picture of what success might look like (with links to other things I’ve written on this), then discuss four key categories of interventions: alignment research, standards and monitoring, successful-but-careful AI projects, and information security. For each, I’ll lay out:

How a small improvement from the status quo could nontrivially improve our odds.

How a big enough success at the intervention could put us in a very good position, even if the other three interventions are going poorly.

Common concerns/reservations about the intervention.

Overall, I feel like there is a pretty solid playbook of hel...
A "bottom-up" approach to AI as a more transparent alternative to "top-down" LLMs
by Paul Jorion — 2023-06-06T18:00:48.282Z

At Pribor.io we decided back in June of last year to capitalise on work I've done in AI back in the late eighties as part of British Telecom's Connex Project. At that time I developed ANELLA, an Associative Network with Emergent Logical and Learning Abilities. As I state in the post's title here, although LLMs were unknown at the time, seen in retrospect, ANELLA was providing a "bottom-up" approach to AI as a more transparent alternative to "top-down" LLMs, also more economical by several orders of magnitude than LLMs. ANELLA was however very much in the same spirit as LLMs of letting the self-organisation that lies in a language's lexicon to bubble up to the surface.

A brief introduction to ANELLA can be found in Luca Possati's The Algorithmic Unconscious (2021). An extensive account can be found in the French book I published at the time: "Principes des systèmes intelligents" (1989), still available as it was reprinted in 2012.

A couple of days ago I started on my YouTube channel a course of lectures in English entitled "Thought as Word Dynamics" where I plan to explain in detail my alternative approach to AI (5 videos have been recorded by now, 3 are already available on YouTube).

Here the first lecture, where I give a historical account of my work in AI:

Now to offer a glimpse at where we currently stand at Pribor.io, here is the first video we've released in February entitled "Self-Aware Machines" (the second video is forthcoming later this month):
The Base Rate Times, news through prediction markets
by vandemonian — 2023-06-06T17:42:56.718Z

Introduction

I made a news site based on prediction markets

The Base Rate Times is a nascent news site that incorporates prediction markets prominently into its coverage.

Please see current iteration:

www.baseratetimes.com

Twitter:

www.twitter.com/base_rate_times

What problem does it solve?

Forecasts are underutilized by the media

Prediction markets are more accurate than pundits, yet the media has made limited use of their forecasts. This is a big problem: one of the most rigorous information sources is being omitted from public discourse!

The Base Rate Times creates prediction markets content, substituting for inferior news sources. This improves the epistemics of its audience.

Forecasts are dispersed, generally inconvenient to consume

Prediction markets are dispersed among many different platforms, fragmenting the information forecasters provide. For example, different platforms ask similar questions in different ways. Furthermore, platforms’ UX is orientated towards forecasters, not information consumers. Overall, trying to use prediction markets as ‘news replacement’ is cumbersome.

There is value in aggregating and curating forecasts from various platforms. We need engaging ways of sharing prediction markets’ insights. The Base Rate Times aims to make prediction markets easily digestible to the general public.

How does it work?

News media (emotive narrative) vs Base Rate Times (actionable odds)

For example, this is a real headline from a reputable newspaper: “Taiwan braces for China's fury over Pelosi visit”. Emotive and incendiary, it does not help you form an accurate model of the situation.

By contrast, The Base Rate Times: “China-Taiwan conflict risk 14%, up 2x from 7% after Pelosi visit”. That's an actionable insight. It can inform your decision on whether to stay in Taiwan or to flee, for example.

News aggregation, summarizing prediction markets

Naturally, the probabilities in the example above come from prediction markets. The Base Rate Times presents what prediction markets are telling us about news in an engaging way.

Stories that shift market odds are highlighted. And if a seemingly important story doesn’t shift market odds, that also tells you something.

On The Base Rate Times, right now you can see the latest odds on:

Putin staying in power
Russian territorial gains in Ukraine
Escalation risk of NATO involvement
and more...

By glancing at a few charts, yo...
Monthly Roundup #7: June 2023
by Zvi — 2023-06-06T17:40:07.194Z

This month’s roundup of non-AI things, with the good, the bad and the uncondoned, along with several additional individual topics. I’m also trying out running some blog-focused surveys near the end, right before the jokes.

Bad News

RIP Doyle Brunson.

Alabama head baseball coach at center of college sports gambling scandal, is fired. Looks like no athletes were involved. It isn’t that easy to throw a baseball game as the coach, especially without it looking weird, but every little bit helps.

I was reminded of this 2020 post by Tim Ferriss about the downsides (and upsides) of becoming famous. So far I guess I’ve hit the sweet spot twice, once for Magic and now again as a writer, where many of the people I’d want to recognize me know who I am and think I’m cool, whereas most normal people have no idea who I am. So I get a lot of the being-famous upside, without almost any of the downsides. So far.

Pete Buttigieg is the Secretary of Transportation. He had one job, to help people and things get transported, and he screwed it up. He says “I strongly support the Jones Act” and justifies it using arguments he has to know are deeply false and disingenuous, given his background. A real shame. In so many ways I really like the guy.

Lab grown meat is currently much worse in terms of greenhouse gas emissions than actual meat, as in 4-25 times worse. Turns out doing something highly bespoke to get some semblance of imitation is currently highly energy intensive. One always always must check to see if a suggested alternative is actually better for the planet, rather than going with the vibes. They hope efficiency can improve with time.

Director of Economic Policy Studies at the American Enterprise Institute tells everyone that life is entirely about working hard.

Michael Strain: My takeaway from this article is that the most successful people obviously do not completely “log off” (whatever that means) on nights and weekends.

Friends, life isn’t a vacation. Work hard. Put in the hours. Impress your boss. The world doesn’t run on your feeling. The world doesn’t bend around your desire to binge Netflix. Have ambitions. Strive. Actual achievement is more meaningful than massive amounts of leisure!

The article asks CEOs what they think of a proposed rule that employees should have a right not to be on call 24/7 constantly responding to emails and other pings. Which ...
Transformative AGI by 2043 is <1% likely
by Ted Sanders — 2023-06-06T17:36:48.296Z
Link: https://arxiv.org/abs/2306.02519

(Crossposted to the EA forum)

Abstract

The linked paper is our submission to the Open Philanthropy AI Worldviews Contest. In it, we estimate the likelihood of transformative artificial general intelligence (AGI) by 2043 and find it to be <1%.

Specifically, we argue:

The bar is high: AGI as defined by the contest—something like AI that can perform nearly all valuable tasks at human cost or less—which we will call transformative AGI is a much higher bar than merely massive progress in AI, or even the unambiguous attainment of expensive superhuman AGI or cheap but uneven AGI.
Many steps are needed: The probability of transformative AGI by 2043 can be decomposed as the joint probability of a number of necessary steps, which we group into categories of software, hardware, and sociopolitical factors.
No step is guaranteed: For each step, we estimate a probability of success by 2043,
conditional on prior steps being achieved. Many steps are quite constrained by the short timeline, and our estimates range from 16% to 95%.
Therefore, the odds are low: Multiplying the cascading conditional probabilities together, we estimate that transformative AGI by 2043 is 0.4% likely. Reaching >10% seems to require probabilities that feel unreasonably high, and even 3% seems unlikely.

Thoughtfully applying the cascading conditional probability approach to this question yields lower probability values than is often supposed. This framework helps enumerate the many future scenarios where humanity makes partial but incomplete progress toward transformative AGI.

Executive summary

For AGI to do most human work for <$25/hr by 2043, many things must happen.

We forecast cascading conditional probabilities for 10 necessary events, and find they multiply to an overall likelihood of 0.4%:

Event

Forecast

by 2043 or TAGI,
conditional on
prior steps

We invent algorithms for transformative AGI	60%
We invent a way for AGIs to learn faster than humans	40%
AGI inference costs drop below $25/hr (per human equivalent)	16%
We invent and scale cheap, quality robots	60%
We massively scale production of chips and power	46%
We avoid derailment by human regulation	70%
We avoid derailment by AI-caused delay	90%
We avoid derailment from wars (e.g., China invades Taiwan)	70%
We avoid derailment from pandemics	90%
We avoid derailment from severe depressions	95%
Joint odds	0.4%

If you think our estimates are pessimistic, feel free to substit...
AISN #9: Statement on Extinction Risks, Competitive Pressures, and When Will AI Reach Human-Level?
by Dan H — 2023-06-06T16:10:19.093Z
Link: https://newsletter.safe.ai/p/ai-safety-newsletter-9

Welcome to the AI Safety Newsletter by the Center for AI Safety. We discuss developments in AI and AI safety. No technical background required.

Subscribe here to receive future versions.

Top Scientists Warn of Extinction Risks from AI

Last week, hundreds of AI scientists and notable public figures signed a

public statement on AI risks

written by the Center for AI Safety. The statement reads:

“Mitigating the risk of extinction from AI should be a global priority alongside other societal-scale risks such as pandemics and nuclear war.”

The statement was signed by a broad, diverse coalition. The statement represents a historic coalition of AI experts — along with philosophers, ethicists, legal scholars, economists, physicists, political scientists, pandemic scientists, nuclear scientists, and climate scientists — establishing the risk of extinction from advanced, future AI systems as one of the world’s most important problems.

The international community is well represented by signatories who hail from more than two dozen countries, including China, Russia, India, Pakistan, Brazil, the Philippines, South Korea, and Japan. The statement also affirms growing public sentiment: a recent poll found that

61 percent of Americans

believe AI threatens humanity’s future.

A broad variety of AI risks must be addressed. As stated in the first sentence of the signatory page, there are many “important and urgent risks from AI.” Extinction is a risk, but there are a variety of others, including misinformation, arbitrary bias, cyberattacks, and weaponization.

Different AI risks share common solutions. For example, an

open letter

from the AI Now Institute argued that AI developers should be held legally liable for harms caused by their systems. This liability would apply across a range of potential harms, from

copyright infringement

to

creating chemical weapons

.

Similarly, autonomous AI agents that act without human oversight could amplify bias and pursue goals misaligned with human values. The AI Now Institute’s open letter argued that humans should be able to “intercede in order to override a decision or recommendations that may lead to potential harm.”

Yesterday, the Center for AI Safety wrote about support for several

policy proposals from the AI ethics community

, including the ones described above. Governing AI effectively will require addressing a range of different probl...
An Eternal Company
by moyamo — 2023-06-06T15:56:54.453Z

TL;DR A company that maximizes its bond price instead of its stock price cares about the long-term future and is incentivized to reduce existential risk.

The world that solves the alignment problem has an institution with the right incentives

There is a world in which we solve the alignment problem. We are not in that world.

The world in which we solve the alignment problem has institutions with incentives to solve alignment-like problems. The same way Walmart has incentives to sell you groceries for a reasonable price. In countries with functioning market economies, nobody thinks about "the grocery problem".

One such institution is one that has incentives to care about the long-term future. I believe that there is an easy way to create such an institution in our current world.

Eternal companies maximize their bond price instead of their stock price

The first insight is that to bet on the end of the world is to take a loan that you expect not to pay back. Hence to bet against the end of the world is to buy a bond that pays installments far into the future. An example of such is a perpetual bond.

When a company sells a perpetual bond they promise to pay regular installments on the bond until the company fails. (If this concept seems alien to you there is a great video by Tom Scott on the concept)

Normal companies try to maximize their risk-adjusted returns (i.e. their share price), but what if we had a company that maximized the price of its perpetual bonds? We can call this type of company an eternal company.

The price of perpetual bonds are determined by interest rates (which the company's executive has no control over) and the company's risk of default (which the company's executive does have control over).

To maximize its bond price an eternal company will try to minimize its risk of default, by

making a modest profit (if a company is losing money it will eventually default), and
growing sustainably (bigger companies are able to diversify their investments making them less likely to default).

The probability a company will default (D) can be decomposed into three factors.

Idiosyncratic risk (I): The risk that the company defaults due to factors particular to the company.
Systemic Risk (S): The risk that the company defaults due to factors particular to the industry or the market as a whole.
Existential Risk (E): The risk that the company defaults because all of humanity

...
Why the Solutions to AI Alignment are Likely Outside the Overton Window
by williamsae — 2023-06-06T14:21:53.283Z

The challenge with Western intellectual approaches to philosophy is that they rely on some framework for understanding the world, where each such framework precludes some subset of possibilities, and predisposes others, as opposed to more holistic Eastern approaches of considering each philosophy, ideology, or any other cognitive construct to be a function that generates its own outcomes, and just observing what each functions to achieve in every aspect of this existence. The benefit of Western intellectual approaches is that they are more readily understood and communicated, whereas Eastern approaches are notoriously difficult to understand and communicate without years of such observation, usually in the form of meditation or other practices. This makes them largely unavailable to Western science.

A recently developed approach called Human-Centric Functional Modeling (HCFM) however bridges these two approaches by representing the human systems through which we perceive and otherwise interact with the world as moving through graphs called functional state spaces, where these graphs can also hypothetically be used to describe the behavior of any living system, whether as simple as homeostasis or as complex as cognition. In these functional state spaces, all problems are defined as the lack of a path from one functional state to another, resulting in the insight that all adaptive processes through which nature solves problems are potentially generalizations of the same solution in functional state space. Representing the behavior of the cognitive system in terms of a set of paths through this mathematical space means that any property of that system’s behavior potentially can be given a mathematical definition, which would mean that properties ascribed to the human system within existential traditions, where those properties are only discernable through the first person observation generally rejected by as subjective and unscientific by Western science, might for the first time be objectively validated. Since many if not most properties of cognition can’t yet be observed by objective third party methods and tools, this would be a major step forward in our understanding.

An example of the usefulness of this approach is that it allows us to model the collective problem-solving ability of networks of systems in terms of their ability to navigate a given volume and...