trentmkelly/LessWrong-43k huggingface dataset:
Avoiding xrisk from AI doesn't mean focusing on AI xrisk
by Stuart_Armstrong — 2023-05-02T19:27:32.162Z

tl;dr: To avoid extinction, focus more on the positive opportunities for the future.

AI is an existential risk and an extinction risk - there is a non-insignificant probability that AI/AGI[1] will become super-powered, and a non-insignificant probability that such an entity would spell doom for humanity.

Worst of all, AGI is not like other existential risks - an unaligned AGI would be an intelligent adversary, so it's not self-limiting in ways that pandemics or most other disasters are. There is no such thing as a "less vulnerable population" where dangerous AGIs are concerned - all humans are vulnerable if the AGI gets powerful enough, and it would be motivated to become powerful enough. Similarly, we can't expect human survivors to reorganise and adapt to the threat: in the most likely scenario, after an initial AGI attack, the AGI would grow stronger while humanity would grow ever more vulnerable.

To cap it all, solving superintelligent AGI alignment requires that we solve all of human morality, either directly or indirectly (or at least do that to an adequate level). If we write a goal structure that doesn't include a key part of humanity - such as, say, conscious experiences - then that part will be excised by the forward-planning AGI.

Look to the bright side

Given all that, it's natural and tempting to focus on the existential risk: to spend most time looking at doom and how to prevent it. It's a very comfortable place to be: (almost) everyone agrees that doom is bad, so everyone is in agreement. It allows us to avoid uncomfortable issues around power distribution and politics: we're not caring about which human monkey wields the big stick of AGI, we just want to avoid extinction for everyone. It makes AGI alignment into a technical issue, that we can solve as dispassionate technicians.

But things are not so comfortable. Remember that AGI alignment is not like other existential risks. We cannot construct a super-powered AGI that is simply "not an existential risk". We have to do more; remember the line, above, about solving all of human morality. We have to define human flourishing if we want humanity to flourish. We have to draw a circle around what counts as human or sentient, if we want anything human or sentient to continue to exist. What would the post-AGI political, legal, and international systems look like? Well, we don't know, but the pre-AGI c...
AI Safety Newsletter #4: AI and Cybersecurity, Persuasive AIs, Weaponization, and Geoffrey Hinton talks AI risks
by ozhang — 2023-05-02T18:41:43.144Z
Link: https://newsletter.safe.ai/p/ai-safety-newsletter-4

Welcome to the AI Safety Newsletter by the

Center for AI Safety

. We discuss developments in AI and AI safety. No technical background required.

Subscribe here to receive future versions.

---

Cybersecurity Challenges in AI Safety

Meta accidentally leaks a language model to the public. Meta’s newest language model, LLaMa, was

publicly leaked online

against the intentions of its developers. Gradual rollout is a

popular goal

with new AI models, opening access to academic researchers and government officials before sharing models with anonymous internet users. Meta intended to use this strategy, but within a week of sharing the model with an approved list of researchers, an unknown person who had been given access to the model publicly posted it online.

How can AI developers selectively share their models? One inspiration could be the film industry, which places

watermarks and tracking technology

on “screener” copies of movies sent out to critics before the movie’s official release. AI equivalents could involve

encrypting model weights

or inserting

undetectable Trojans

to identify individual copies of a model. Yet efforts to cooperate with other AI companies could face

legal opposition under antitrust law

. As the LLaMa leak demonstrates, we don’t yet have good ways to share AI models securely.

LLaMa leak. March 2023, colorized.

OpenAI faces their own cybersecurity problems. ChatGPT recently

leaked user data

including conversation histories, email addresses, and payment information. Businesses including

JPMorgan, Amazon, and Verizon

prohibit employees from using ChatGPT because of data privacy concerns, though OpenAI is trying to assuage those concerns with a

business subscription plan

where OpenAI promises not to train models on the data of business users. OpenAI also started a

bug bounty program

that pays people to find security vulnerabilities.

AI can help hackers create novel cyberattacks. Code writing tools open up the possibility of new kinds of cyberattacks. CyberArk, an information security firm, recently showed that OpenAI’s code generation tool can be used to create

adaptive malware

that writes new lines of code while hacking into a system in order to bypass cyberdefenses. GPT-4 has also been shown capable of

hacking into password management systems

,

convincing humans to help it bypass CAPTCHA verification

, and

performing coding challenges in offensive cyb

...
My best system yet: text-based project management
by jt — 2023-05-02T17:44:49.478Z

Introduction

My current system is simple to start now and refine later when the time is right. It's flexible enough to adapt to changing circumstances and when information is no longer relevant its efficient to update or remove. With this system in place, I'm able to achieve consistent progress on the things that matter.

But let's wind back a little to the time when it all began: In 2016, I returned from my Center for Applied Rationality retreat with a newfound motivation to change my life. After the retreat I ended up with a huge load of inspiration to try plenty of new methods, read more books, cultivate better habits, and stay in touch with all these new people. A feeling that probably many of you know from an EAG or LWCW conference. I quickly realized that I needed a way to structure all of this new information to make use of it. This was the moment that I went on my journey to project management.

3 years later, at my last job as product manager, I had the honor to work with an excellent manager. He seemingly never dropped a ball and achieved this with ease. He auto-magically remembered to bring up that meeting from past week again just at the most convenient time when its not yet too late and resources are available. He never needed to write something down. This is very impressive. I'm convinced that everyone can achieve this, without overloading their brain.

To achieve organization excellence, it's crucial to understand that getting things done involves a long breath. As the saying goes, "Rome was not built in a day" and so every step has a follow up step all the way until a goal is achieved. As Mark Manson put it: "Life is essentially an endless series of problems". Naturally this leads to many simultaneous projects as the norm, so better handle them without breaking a sweat.

All management tools, whether for projects, to-dos, birthdays, code, workflows, customer relations, and so on, ultimately boil down to knowledge management. Arguably the most powerful tool humans have ever developed to manage knowledge is language. Therefore, to achieve 10x management, we must embrace tools that allow us to manage knowledge effectively. By leveraging the power of language and information processing, we can build systems that are both easy to start and easy to scale.

Perfection is achieved when you can't remove anything. This statement is especially true when it c...
Five Worlds of AI (by Scott Aaronson and Boaz Barak)
by mishka — 2023-05-02T13:23:41.544Z
Link: https://scottaaronson.blog/?p=7266

A relatively short April 27, 2023 post attempting to classify AI outcomes into 5 main cases:

AI-Fizzle

Not-AI-Fizzle

Civilization recognizably continues

Futurama

AI-Dystopia

Civilization does not recognizably continue

Singularia

Paperclipalypse

In other words, no one has done for AI what Russell Impagliazzo did for complexity theory in 1995, when he defined the five worlds Algorithmica, Heuristica, Pessiland, Minicrypt, and Cryptomania, corresponding to five possible resolutions of the P vs. NP problem along with the central unsolved problems of cryptography.

The authors elaborate:

Like in Impagliazzo’s 1995 paper on the five potential worlds of the difficulty of NP problems, we will not try to be exhaustive but rather concentrate on extreme cases. It’s possible that we’ll end up in a mixture of worlds or a situation not described by any of the worlds. Indeed, one crucial difference between our setting and Impagliazzo’s, is that in the complexity case, the worlds corresponded to concrete (and mutually exclusive) mathematical conjectures. So in some sense, the question wasn’t “which world will we live in?” but “which world have we Platonically always lived in, without knowing it?” In contrast, the impact of AI will be a complex mix of mathematical bounds, computational capabilities, human discoveries, and social and legal issues. Hence, the worlds we describe depend on more than just the fundamental capabilities and limitations of artificial intelligence, and humanity could also shift from one of these worlds to another over time.

Extensive discussion in the comments. Eliezer writes a comment, and some of the replies to that comment are quite informative.
Systems that cannot be unsafe cannot be safe
by Davidmanheim — 2023-05-02T08:53:35.115Z

Epistemic Status: Trying to clarify a confusion people outside of the AI safety community seem to have about what safety means for AI systems.

In engineering and design, there is a process that includes, among other stages, specification, creation, verification and validation, and deployment. Verification and validation are where most people focus when thinking about safety - can we make sure the system performs correctly? I think this is a conceptual error that I want to address.

"Verification and validation (also abbreviated as V&V) are independent procedures that are used together for checking that a product, service, or system meets requirements and specifications and that it fulfills its intended purpose." - Wikipedia

Both of these terms are used slightly differently across fields, but in general, verification is the process of making sure that the system fulfills the design requirements and/or other standards. This pre-supposes that the system has some defined requirements or a standard, at least an implicit one, and that it could fail to meet that bar. That is, the specification of the system includes what it must and must not do, and if the system does not do what it should, or does something that it should not, it fails.

Machine learning systems, especially language models, aren't well understood. The potential applications are varied and uncertain, entire classes of new and surprising failure modes are still being found, and we have nothing like a specification of what the system should or should not do, must or must not do, and where it can and cannot be used.

To take a very concrete example, metal rods have safety characteristics, and they might be rated for use up to some weight limit, under some specific load for some amount of time, in certain temperature ranges, for some amount of time. These can all be tested. If the bar does not stay within a predefined range of characteristics at a given temperature, with a given load, it fails. It can also be found to be acceptable in one temperature range, but not another, or similar. At the end of verification and validation, the bar is deemed to have passed or failed for a given application, based on what the requirements for that larger system are.

At its best, red-teaming and safety audits of ML systems check lots of known failure modes, and determine whether they are susceptible. There is no pre...
AGI safety career advice
by Richard_Ngo — 2023-05-02T07:36:09.044Z

People often ask me for career advice related to AGI safety. This post (now also translated into Spanish) summarizes the advice I most commonly give. I’ve split it into three sections: general mindset, alignment research and governance work. For each of the latter two, I start with high-level advice aimed primarily at students and those early in their careers, then dig into more details of the field.

See also this post

I wrote two years ago, containing a bunch of fairly general career advice.

General mindset

In order to have a big impact on the world you need to find a big lever. This document assumes that you think, as I do, that AGI safety is the biggest such lever. There are many ways to pull on that lever, though—from research and engineering to operations and field-building to politics and communications. I encourage you to choose between these based primarily on your personal fit—a combination of what you're really good at and what you really enjoy. In my opinion the difference between being a great versus a mediocre fit swamps other differences in the impactfulness of most pairs of AGI-safety-related jobs.

How should you find your personal fit? To start, you should focus on finding work where you can get fast feedback loops. That will typically involve getting hands-on or doing some kind of concrete project (rather than just reading and learning) and seeing how quickly you can make progress. Eventually, once you've had a bunch of experience, you might notice a feeling of confusion or frustration: why is everyone else missing the point, or doing so badly at this? (Though note that a few top researchers commented on a draft to say that they didn't have this experience.) For some people that involves investigating a specific topic (for me, the question “what’s the best argument that AGI will be misaligned?“); for others it's about applying skills like conscientiousness (e.g. "why can't others just go through all the obvious steps?") Being excellent seldom feels like you’re excellent, because your own abilities set your baseline for what feels normal.

What if you have that experience for something you don't enjoy doing? I expect that this is fairly rare, because being good at something is often very enjoyable. But in those cases, I'd suggest trying it until you observe that even a string of successes doesn't make you excited about what you're doing; and at th...
An Impossibility Proof Relevant to the Shutdown Problem and Corrigibility
by Audere — 2023-05-02T06:52:25.097Z

The Incompatibility of a Utility Indifference Condition with Robustly Making Sane Pure Bets

Summary

It is provably impossible for an agent to robustly and coherently satisfy two conditions that seem desirable and highly relevant to the shutdown problem. These two conditions are the sane pure bets condition, which constrains preferences between actions that result in equal probabilities of an event such as shutdown, and the weak indifference condition, a condition which seems necessary (although not sufficient) for an agent to be robustly indifferent to an event such as shutdown.

Suppose that we would like an agent to be indifferent to an event P, which could represent the agent being shut down at a particular time, or the agent being shut down at any time before tomorrow, or something else entirely. Furthermore, we would ideally like the agent to do well at pursuing goals described by some utility function U, while being indifferent to P.

The sane pure bets condition is as follows:

Given any two actions A and B such that P(P|A) = P(P|B) and E(U|A) > E(U|B), the agent prefers A to B. In other words, if two possible actions lead to the same probability of P, and one of them leads to greater expected utility under U, the agent should prefer that one. Intuitively, this constraint represents the idea that among possible actions which don’t influence the probability of P, we would like the agent to prefer those that lead to greater expected utility under U.

The weak indifference condition is as follows:

Given any two actions A and B such that E(U | A,P) > E(U | B,P) and E(U | A,!P) > E(U | B,!P), the agent prefers A to B. In other words, if between two possible actions, one of them leads to greater expected utility conditioned on P occurring and also leads to greater expected utility conditioned on P not occurring, the agent should prefer that one. Intuitively, this constraint represents the idea that the agent should be unwilling to pay any amount of utility to influence the probability of P.

The proof takes the form of a simple decision problem wherein an agent has four possible actions. Each constraint implies a preference between two pairs of actions, and altogether they imply circular preferences, proving that there cannot be any general method for constructing an agent which fulfills both constraints without having circular preferences. Fu...
Some Thoughts on Virtue Ethics for AIs
by peligrietzer — 2023-05-02T05:46:41.334Z

This post argues for the desirability and plausibility of AI agents whose values have a structure I call ‘praxis-based.’ The idea draws on various aspects of virtue ethics, and basically amounts to an RL-flavored take on that philosophical tradition.

Praxis-based values as I define them are, informally, reflective decision-influences matching the description ‘promote x x-ingly’: ‘promote peace peacefully,’ ‘promote corrigibility corrigibly,’ ‘promote science scientifically.’

I will later propose a quasi-formal definition of this values-type, but the general idea is that certain values are an ouroboros of means and end. Such values frequently come up in human “meaning of life” activities (e.g. math, art, craft, friendship, athletics, romance, technology), as well as in complex forms of human morality (e.g. peace, democracy, compassion, respect, honesty). While this is already indirect reason to suspect that a human-aligned AI should have ‘praxis-based’ values, there is also a central direct reason: traits such as corrigibility, transparency, and niceness can only function properly in the form of ‘praxis-based’ values.

It’s widely accepted that if early strategically aware AIs possess values like corrigibility, transparency, and perhaps niceness, further alignment efforts are much more likely to succeed. But values like corrigibility or transparency or niceness don’t easily fit into an intuitively consequentialist[1] form like ‘maximize lifetime corrigible behavior’ or ‘maximize lifetime transparency.’ In fact, an AI valuing its own corrigibility or transparency or niceness in an intuitively consequentialist way can lead to extreme power-seeking whereby the AI violently remakes the world to (at a minimum) protect itself from the risk that humans will modify said value.[2] On the other hand, constraints or taboos or purely negative values (a.k.a. ‘deontological restrictions’) are widely believed to be weak, in the sense that an advanced AI will come to work around them or uproot them: ‘never lie’ or ‘never kill’ or ‘never refuse a direct order from the president’ are poor substitutes for active transparency, niceness, and corrigibility.[3]

The idea of ‘praxis-based’ values is meant to capture the normal, sensible way we want an agent to value corrigibility or transparency or niceness, which intuitively-consequentiali...
Technological unemployment as another test for rationalist winning
by RomanHauksson — 2023-05-02T04:16:46.614Z

Ultimately, rationalism should help people win. Scott Alexander claims that the surge of the price of Bitcoin was a test of that:

...suppose God had decided, out of some sympathy for our project, to make winning as easy as possible for rationalists. He might have created the biggest investment opportunity of the century, and made it visible only to libertarian programmers willing to dabble in crazy ideas. And then He might have made sure that all of the earliest adapters were Less Wrong regulars, just to make things extra obvious.

This was the easiest test case of our "make good choices" ability that we could possibly have gotten, the one where a multiply-your-money-by-a-thousand-times opportunity basically fell out of the sky and hit our community on its collective head. So how did we do?

I would say we did mediocre.

Five years later, suppose God wanted to give rationalists another test. But instead of the opportunity to win big, He wanted to test whether they could avoid losing hard. He might create the largest workforce disruption of the century, driven by an unpredictable technology (which the rationalists happen to know the most about) and primarily affecting white-collar workers (which most rationalists are). If rationalism truly helps people predict the future and make better decisions, rationalists who work should survive the incoming wave of AI-driven job automation better than everyone else.

Of course, this only applies for those whose jobs are truly in danger. I'm pursuing a career in AI alignment research – if that becomes automated, none of this matters. But suppose, like many of us, you're a software engineer. You should be paying careful attention to forecasts related to your future compensation and learning how to use the latest tools which accelerate your productivity relative to your competitors. And if things start looking really grim, since you know about Moravec's paradox and status quo bias, you'd start learning valuable blue-collar skills earlier than your developer friends who are also at risk of getting laid off.

Most artists didn't expect robots to become creative and thus didn't prepare accordingly. Then, DALL-E 2 happened, rendering many of the skills they spent years mastering economically useless. This should be a warning shot for everyone else who thinks they're rational: think ahead, don't get caught off guard, and act on your ...
The Moral Copernican Principle
by Legionnaire — 2023-05-02T03:25:40.142Z

You ever see people arguing about whether some facet of another culture is good or bad, when suddenly one of them declares the moral high ground can't exist because obviously moral relativity is true? Well that line of reasoning is nonsense, but I don't think people know how to respond to it very well, so it often wins the argument. Consider this article a countermeasure you can reach for.

Moral absolutists say things like "killing is always wrong" and believe that aliens and AI will converge on our moral beliefs.[1]
Moral relativists say things like "it's just their culture, who are we to say" and believe it's not wrong for other cultures to have practices we deem immoral.
I think they often both manage to be wrong. Let's translate these archetypes to their physical equivalents. Imagine two people talking about whether to go to Mars:
"Earth is the center of civilization and life. It is objectively the best location to be! Mars is very far away and inhospitable."
"Relativity tells us there is no special location, or even reference frame, in the universe! To Mars, we're the ones that are very far away."
We know physical relativity is true, but that doesn't prevent us from talking about locations with regard to their utility to us, or distances being a valid concept.[2] Earth really is objectively easier to get to and more hospitable for every human since we share important traits like wanting to breath oxygen. Yet that doesn't prove that Mars has no useful resources, or that there are no exoplanets that are even better than Earth. Plus there are likely aliens that would enjoy Mars more than Earth. We may not occupy a "privileged position", but we as humans have bodies that evolved for certain conditions, and thus a few special positions. [3]
Similarly, moral relativism does not prove that all moral locations are equally hospitable for humans, because we have brains that evolved for specific conditions. Different locations can be objectively better or worse for every single homo sapien that finds themselves there. When a tribe on the other side of the world burns a witch at the stake for crop failure, the humans there are not that different from us. We're correct in saying that's a bad idea, especially having already tried that particular solution ourselves.
I would consider myself one, but I see a lot of moral relativists make the mistake of assuming no loc...