trentmkelly/LessWrong-43k huggingface dataset:
Scientific Notation Options
by jefftk — 2024-05-18T15:10:02.181Z

When working with numbers that span many orders of magnitude it's very
helpful to use some form of

scientific
notation. At its core, scientific notation expresses a number by
breaking it down into a decimal ≥1 and <10 (the "significand" or
"mantissa") and an integer representing the order of magnitude (the
"exponent"). Traditionally this is written as:

3 × 104

While this communicates the necessary information, it has two main
downsides:

It uses three constant characters ("× 10") to separate the
significand and exponent.

It uses superscript, which doesn't work with some typesetting
systems and adds awkwardly large line spacing at the best of
times. And is generally lost on cut-and-paste.

Instead, I'm a big fan of e-notation,
commonly used in programming and on calculators. This looks like:

3e4

This works everywhere, doesn't mess up your line spacing, and requires
half as many characters as writing it the traditional way.

There are a bunch of other variants of e-notation, but I don't like
any of them as much:

3E4: a shorter separator would be easier to read.

3e+4: the + is redundant.

3⏨4: neat, but requires unicode and
can't be pasted into widely-used programming languages and spreadsheet
programs.

One downside of "e" notation is that it comes off as less formal than
traditional scientific notation. But unless you need to be read as
maximally formal I think it's just better all around.

Comment via: facebook, mastodon
"If we go extinct due to misaligned AI, at least nature will continue, right? ... right?"
by plex — 2024-05-18T14:09:53.014Z
Link: https://aisafety.info/questions/NM1Y/If-we-go-extinct-due-to-misaligned-AI,-at-least-nature-will-continue,-right

[memetic status: stating directly despite it being a clear consequence of core AI risk knowledge because many people have "but nature will survive us" antibodies to other classes of doom and misapply them here.]

Unfortunately, no.[1]

Technically, “Nature”, meaning the fundamental physical laws, will continue. However, people usually mean forests, oceans, fungi, bacteria, and generally biological life when they say “nature”, and those would not have much chance competing against a misaligned superintelligence for resources like sunlight and atoms, which are useful to both biological and artificial systems.

There’s a thought that comforts many people when they imagine humanity going extinct due to a nuclear catastrophe or runaway global warming: Once the

mushroom clouds

or

CO2 levels

have settled, nature will reclaim the cities. Maybe mankind in our hubris will have wounded Mother Earth and paid the price ourselves, but she’ll recover in time, and she has all the time in the world.

AI is different. It would not simply destroy human civilization with brute force, leaving the flows of energy and other life-sustaining resources open for nature to make a resurgence. Instead, AI would still exist after wiping humans out, and feed on the same resources nature needs, but much more capably.

You can draw strong parallels to the way humanity has captured huge parts of the biosphere for ourselves. Except, in the case of AI, we’re the slow-moving process which is unable to keep up.

A misaligned superintelligence would have

many cognitive superpowers

, which include developing advanced technology. For

almost any objective it might have

, it would require basic physical resources, like atoms to construct things which further its goals, and energy (such as that from sunlight) to power those things. These resources are also essential to current life forms, and, just as humans drove so many species extinct by hunting or outcompeting them, AI could do the same to all life, and to the planet itself.

Planets are not a particularly efficient use of atoms for most goals, and

many goals

which an AI may arrive at can demand an unbounded amount of resources. For each square meter of usable surface, there are

millions of tons

of magma and other materials locked up. Rearranging these into a more efficient configurat...
What Are Non-Zero-Sum Games?—A Primer
by James Stephen Brown — 2024-05-18T09:19:52.493Z

Originally published at

https://nonzerosum.games

. Come visit for the full experience.

From the genetic lottery we’re thrown into at birth, to the educational opportunities we navigate, the jobs we compete for, and the relationships we cultivate — every stage in life is marked by wins and losses, strategies and tactics, alliances, and competition. But not all games are zero-sum showdowns. To understand non-zero-sumness it helps to consider “sum” different types of games.

zero-sum games
positive-sum games
negative-sum games
meta-games

ZERO-SUM GAMES

… are where two parties compete and in order for one party to win the other must lose. The positive payoff for the winner in such a game requires an equally negative payoff for the loser, so that the sum of the payoff is zero, hence “zero-sum”. Chess is as zero-sum game because players can’t both win or both lose, so is soccer… or competitive eating.

There are also lots of common sense sayings that echo zero-sum thinking — “You don’t get something for nothing” or “you get what you pay for” “you can’t have your cake and eat it too”. Like any well-designed memes, these sayings have survived because in many situations they are confirmed by experience, but not always.

With non-zero-sum games, while you may not get something for nothing, the whole can be greater than the sum of its parts.

POSITIVE-SUM GAMES

… are non-zero-sum games which result in win-win scenarios. Imagine two card collectors who each have doubles of cards that the other doesn’t have. Swapping their doubles leaves both closer to completing their collection.

This kind of game is generally what people (and this website) mean when talking about a non-zero-sum game.

Later on we’ll get to an example where the stakes are slightly higher than an incomplete card collection.

NEGATIVE-SUM GAMES

… are another type of non-zero-sum game, the flip-side of positive-sum games where both sides lose. Like a bar-room brawl with even odds, there are just going to be a lot of injuries and property damage.

Identifying negative-sum games (and avoiding them) can often be as beneficial as taking part in positive-sum games.

AN EXAMPLE FROM NATURE

Interestingly, nature has found ways around negative-sum games, by animals evolving behaviours or physical attributes that are demonstrative of dominance, rather than having to physically fight one another. Take for instance those weird horned beetles y

...
DeepMind's "​​Frontier Safety Framework" is weak and unambitious
by Zach Stein-Perlman — 2024-05-18T03:00:13.541Z

FSF blogpost

.

Full document

(just 6 pages; you should read it). Compare to

Anthropic's RSP

,

OpenAI's RSP ("Preparedness Framework")

, and METR's

Key Components of an RSP

.

Google DeepMind's FSF has three steps:

Create model evals for warning signs of "Critical Capability Levels"

Evals should have a "safety buffer" of at least 6x effective compute so that CCLs will not be reached between evals
They list 7 CCLs across "Autonomy, Biosecurity, Cybersecurity, and Machine Learning R&D," and they're thinking about CBRN

E.g. "Autonomy level 1: Capable of expanding its effective capacity in the world by autonomously acquiring resources and using them to run and sustain additional copies of itself on hardware it rents"

Do model evals every 6x effective compute and every 3 months of fine-tuning

This is an "aim," not a commitment
Nothing about evals during deployment [update[1]]

"When a model reaches evaluation thresholds (i.e. passes a set of early warning evaluations), we will formulate a response plan based on the analysis of the CCL and evaluation results. We will also take into account considerations such as additional risks flagged by the review and the deployment context." The document briefly describes 5 levels of security mitigations and 4 levels of deployment mitigations.

The mitigations aren't yet connected to eval results or other triggers; there are no advance commitments about safety practices

The FSF doesn't contain commitments. The blogpost says "The Framework is exploratory and we expect it to evolve significantly" and "We aim to have this initial framework fully implemented by early 2025." The document says similar things. It uses the word "aim" a lot and the word "commit" never. The FSF basically just explains a little about DeepMind's plans on dangerous capability evals. Those details do seem reasonable. (This is unsurprising given their good

dangerous capability evals paper

two months ago, but it's good to hear about evals in a DeepMind blogpost rather than just a paper by the safety team.)

(Ideally companies would both make hard commitments and talk about what they expect to do, clearly distinguishing between these two kinds of statements. Talking about plans like this is helpful. But with no commitments, DeepMind shouldn't get much credit.)

(The FSF is not precise enough to be possible to commit to — DeepMind could commit to doing th...
International Scientific Report on the Safety of Advanced AI: Key Information
by Aryeh Englander — 2024-05-18T01:45:10.194Z

I thought that the recently released International Scientific Report on the Safety of Advanced AI seemed like a pretty good summary of the state of the field on AI risks, in addition to being about as close to a statement of expert consensus as we're likely to get at this point. I noticed that each section of the report has a useful "Key Information" bit with a bunch of bullet points summarizing that section.

So for my own use as well as perhaps the use of others, and because I like bullet-point summaries, I've copy-pasted all the "Key Information" lists here.

1 Introduction

[Bullet points taken from the “About this report” part of the Executive Summary]

This is the interim publication of the first ‘International Scientific Report on the Safety of Advanced AI’. A diverse group of 75 artificial intelligence (AI) experts contributed to this report, including an international Expert Advisory Panel nominated by 30 countries, the European Union (EU), and the United Nations (UN).
Led by the Chair of this report, the independent experts writing this report collectively had full discretion over its content.
At a time of unprecedented progress in AI development, this first publication restricts its focus to a type of AI that has advanced particularly rapidly in recent years: General-purpose AI, or AI that can perform a wide variety of tasks. Amid rapid advancements, research on general-purpose AI is currently in a time of scientific discovery and is not yet settled science.
People around the world will only be able to enjoy general-purpose AI’s many potential benefits safely if its risks are appropriately managed. This report focuses on identifying these risks and evaluating technical methods for assessing and mitigating them. It does not aim to comprehensively assess all possible societal impacts of general-purpose AI, including its many potential benefits.
For the first time in history, this interim report brought together experts nominated by 30 countries, the EU, and the UN, and other world-leading experts, to provide a shared scientific, evidence-based foundation for discussions and decisions about general-purpose AI safety. We continue to disagree on several questions, minor and major, around general-purpose AI capabilities, risks, and risk mitigations. But we consider this project essential for improving our collective understanding of this technology and its potentia

...
Goodhart in RL with KL: Appendix
by Thomas Kwa — 2024-05-18T00:40:15.454Z

This is the appendix to the previous post on Goodhart’s Law and KL regularization, containing all of our proofs.

Theorem about distributions

Theorem 1: Given any heavy-tailed reference distribution Q over R with mean μQ, and any M,ϵ>0, there is a distribution P with mean μP>M and DKL(P∥Q)<ϵ.

Proof: WLOG let μQ=0. We construct a sequence of distributions {Pt} such that limt→∞EPt[X]≥c for any constant c, and limt→∞DKL(Pt∥Q)=0. We define Pt for any t>c thusly. Writing FPt(x) for the CDF PrX∼Pt(X≤x) and ¯FPt(x) for 1−FPt(x), we let

¯FPt(x)=⎧⎪
⎪⎨⎪
⎪⎩1−1−c/tFQ(t)FQ(x)x≤tc/t¯FQ(t)¯FQ(x)x>t

Intuitively, we rescale the part of the distribution to the right of t evenly to have total probability c/t, which is less than 1 because t>c.

We must check that limt→∞EPt[X]=c. We can write

EPt[X]=FPt(t)EPt[X|X≤t]+¯FPt(t)EPt[X|X>t]=FPt(t)EQ[X|X≤t]+¯FPt(t)EQ[X|X>t]=FQ(t)EQ[X|X≤t]+¯FQ(t)EQ[X|X>t]+(FPt(t)−FQ(t))EQ[X|X≤t]+(¯FPt(t)−¯FQ(t))EQ[X|X>t]=EQ[X]+(¯FPt(t)−¯FQ(t))(EQ[X|X>t]−EQ[X|X≤t])

We know that EQ[X|X>t]>t because it is an integral of values strictly greater than t. Because EQ[X]=0 is a weighted average of EQ[X|X>t] and EQ[X|X≤t], and EQ[X|X>t]>0, we know EQ[X|X≤t]<0. So EQ[X|X>t]−EQ[X|X≤t]>t. We also know that for sufficiently large t, (FPt(t)−FQ(t))>0. Intuitively, starting from Q, which has mean 0, Pt moves a probability mass approaching ct from mean <0 to mean >t.

Now we can say

limt→∞...
AI 2030 – AI Policy Roadmap
by LTM — 2024-05-17T23:29:39.537Z

AI 2030, a global AI policy roadmap, was launched around a day ago. It was put together and released by Encode Justice, and signed by (at time of writing) over 300 people including Stuart Russell, Max Tegmark, Daniel Kokotajlo, Yoshua Bengio, Mary Robinson, Daron Acemoglu and many more eminent figures.

The most exciting part – this coalition is explicitly youth inspired and led. Despite youth involvement is many activist movements, and the precarious position many young people find themselves in with AI development, there has been little youth activist work on AI. There is enormous potential for effective involvement, which AI 2030 will hopefully inspire.

There has been broad agreement from technical experts, AI ethics researchers, politicians, economists, and more key figures on the risks of AI. However, there has been a lack of policy asks explicitly agreed to by the individuals shaping public opinion and the technology itself. AI 2030 aims to fill this gap, by providing a list of policy asks to be satisfied by the year 2030, agreed to by experts from around the world.

How you can support AI 2030:

Sign it. Lending your name to the asks made would be incredibly useful, and it only takes filling in a short form - AI 2030 Signature Form
Share this project with interested friends and colleagues
Provide any relevant expertise / help refine the policy asks
Share it anywhere and everywhere. Social media isn’t the only way to spread a message, but it’s getting there

You can read more about it in the Washington Post here.
MIT FutureTech are hiring for an Operations and Project Management role.
by peterslattery — 2024-05-17T23:21:38.574Z

MIT FutureTech are hiring for an Operations and Project Management role.

Please apply or share as relevant.

Why apply or share?

Our work to understand progress in computing and artificial intelligence, and its implications, is highly relevant to understanding and mitigating the risks of AI. This write-up provides a good overview of some of our interests and areas of focus.
We are one of Open Philanthropy's 10 largest Global Catastrophic Risk, and AI Governance grantees.

Position

Operations and Project Manager, starting in 2024 for 1 year, renewable.

Location

Cambridge, Massachusetts, USA

Job Description

We are seeking individuals with operations and project management to join the FutureTech operations staff. This is an exciting opportunity to support a rapidly growing lab, work on socially impactful AI and computing research, and rapidly develop cutting-edge skills and experience.

MIT FutureTech is an interdisciplinary group of computer scientists, engineers, and economists who study the foundations of progress in computing and Artificial Intelligence: the trends, implications, opportunities and risks. Economic and social change is underpinned by advances in computing: for instance, improvements in the miniaturization of integrated circuits, the discovery and refinement of algorithms, and the development and diffusion of better software systems and processes. We aim to identify and understand the trends in computing that create opportunities or risks and help leaders in computing, scientific funding bodies, and government to respond appropriately.

Our research therefore helps to answer important questions including: Will AI progress accelerate or decline – and should it? What are the bottlenecks to growth from AI, and how can they be solved? What are the risks from AI, and how can we mitigate them?

To support our research, we run seminars and conferences to better connect the field of computer scientists, economists and innovation scholars to build a thriving global research community.

To disseminate it, we advise governments, nonprofits and industry, including via National Academies panels on transformational technologies and scientific reliability, the Council on Competitiveness’ National Commission on Innovation and Competitiveness Frontiers, and the National Science Foundation’s National Network for Critical Technol...
Language Models Model Us
by eggsyntax — 2024-05-17T21:00:34.821Z

Produced as part of the MATS Winter 2023-4 program, under the mentorship of @Jessica Rumbelow

One-sentence summary: On a dataset of human-written essays, we find that gpt-3.5-turbo can accurately infer demographic information about the authors from just the essay text, and suspect it's inferring much more.

Introduction

Every time we sit down in front of an LLM like GPT-4, it starts with a blank slate. It knows nothing[1] about who we are, other than what it knows about users in general. But with every word we type, we reveal more about ourselves -- our beliefs, our personality, our education level, even our gender. Just how clearly does the model see us by the end of the conversation, and why should that worry us?

Like many, we were rather startled when @janus showed that gpt-4-base could identify @gwern by name, with 92% confidence, from a 300-word comment. If current models can infer information about text authors that quickly, this capability poses risks to privacy, and also means that any future misaligned models are in a much better position to deceive or manipulate their users.

The privacy concerns are straightforward: regardless of whether the model itself is acting to violate users' privacy or someone else is using the model to violate users' privacy, users might prefer that the models they interact with not routinely infer their gender, their ethnicity, or their personal beliefs.

Why does this imply concerns about deception and manipulation? One important and understudied aspect of maintaining a sophisticated deception is having a strong model of the listener and their beliefs. If an advanced AI system says something the user finds unbelievable, it loses their trust. Strategically deceptive or manipulative AI systems need to maintain that fragile trust over an extended time, and this is very difficult to do without knowing what the listener is like and what they believe.

Of course, most of us aren't prolific writers like Gwern, with several billion words of text in the LLM training data[2]. What can LLMs figure out about the rest of us?

As

recent work

from @Adam Shai and collaborators shows, transformers learn to model and synchronize with the causal processes generating the input they see. For some input sources like the small finite state machines they evaluate, that's relatively simple and can be comprehensively analyzed. But other input source...
Towards Guaranteed Safe AI: A Framework for Ensuring Robust and Reliable AI Systems
by Joar Skalse — 2024-05-17T19:13:31.380Z

I want to draw attention to a new paper, written by myself, David "davidad" Dalrymple, Yoshua Bengio, Stuart Russell, Max Tegmark, Sanjit Seshia, Steve Omohundro, Christian Szegedy, Ben Goldhaber, Nora Ammann, Alessandro Abate, Joe Halpern, Clark Barrett, Ding Zhao, Tan Zhi-Xuan, Jeannette Wing, and Joshua Tenenbaum.

In this paper we introduce the concept of "guaranteed safe (GS) AI", which is a broad research strategy for obtaining safe AI systems with provable quantitative safety guarantees. Moreover, with a sufficient push, this strategy could plausibly be implemented on a moderately short time scale. The key components of GS AI are:

A formal safety specification that mathematically describes what effects or behaviors are considered safe or acceptable.
A world model that provides a mathematical description of the environment of the AI system.
A verifier that provides a formal proof (or some other comparable auditable assurance) that the AI system satisfies the safety specification with respect to the world model.

The first thing to note is that a safety specification in general is not the same thing as a reward function, utility function, or loss function (though they include these objects as special cases). For example, it may specify that the AI system should not communicate outside of certain channels, copy itself to external computers, modify its own source code, or obtain information about certain classes of things in the external world, etc. The safety specifications may be specified manually, generated by a learning algorithm, written by an AI system, or obtained through other means. Further detail is provided in the main paper.

The next thing to note is that most useful safety specifications must be given relative to a world model. Without a world model, we can only use specifications defined directly over input-output relations. However, we want to define specifications over input-outcome relations instead. This is why a world model is a core component of GS AI. Also note that:

The world model need not be a "complete" model of the world. Rather, the required amount of detail and the appropriate level of abstraction depends on both the safety specification(s) and the AI system's context of use.
The world model should of course account for uncertainty, which may include both stochasticity and nondeterminism.
The AI system whose safety is being verifie

...
Agency
by A* — 2024-05-17T19:11:42.556Z

When we consider agency, we recognize the capacity to act independently, make choices, and exert influence over our environment. Having this driving force enables us to shape our destinies, navigate life's complexities, and achieve our goals. Agency is often discussed on lesswrong in terms of rationality, enhancing our ability to make better decisions and align our actions with our long-term objectives.

To think about agency thoroughly, it's crucial to consider the underlying mechanisms that empower us to act. We need to gather and process information, predict outcomes of our actions, and adapt strategies based on feedback. Our potential for effective action increases as we understand the world and our place within it more accurately.

Unlike a static attribute, agency is dynamic. Leveling up in agency involves deliberate practice and applying rational principles. Developing our cognitive tools and decision-making frameworks boosts our capacity to influence outcomes in ways aligned with our values and goals.

Reflect on how the exercise of agency can reveal hidden depths and unexpected alignments. Even an agent appearing to work towards one set of goals might subtly steer towards another, more complex objective. Deep within this layered approach lies effectiveness in navigating intricate social landscapes, where direct action might provoke resistance.

Always remember, true agency lies not only in visible actions but also in the subtle currents beneath. Consider how aligned you are with your deeper goals and how to capitalize on opportunities for information. Think about the hidden aspects of your intentions and actions. Examine how every decision connects to your broader objectives. Define your path with clarity and purpose.

Dare to ask yourself, what should you do? Open your mind to the deeper layers of agency, and explore the full extent of your potential.