trentmkelly/LessWrong-43k huggingface dataset:
‘We’re changing the clouds.’ An unforeseen test of geoengineering is fueling record ocean warmth
by Annapurna — 2023-08-06T20:58:51.838Z
Link: https://www.science.org/content/article/changing-clouds-unforeseen-test-geoengineering-fueling-record-ocean-warmth

For decades humans have been emitting carbon dioxide into the atmosphere, creating a greenhouse effect and leading to an acceleration of the earth's warming.

At the same time, humans have been emitting sulphur dioxide, a pollutant found in shipping fuel that has been responsible for acid rain. Regulations imposed in 2020 by the United Nations’s International Maritime Organization have cut ships’ sulfur pollution by more than 80% and improved air quality worldwide.

Three years after the regulation was imposed, scientists are realizing that sulphur dioxide has a sunscreen effect on the atmosphere, and by removing it from shipping fuel we have inadvertently removed this sunscreen, leading to an acceleration in temperature in the regions where global shipping operates the most: the North Atlantic and the North Pacific.

We've been accidentally geoengineering the earth's climate, and the mid to long term consequences of removing those emissions are yet to be seen. At the same time, this accident is making scientists realize that with not much effort we can geoengineer the earth and reduce the effect of greenhouse gas emissions.
[Linkpost] Will AI avoid exploitation?
by cdkg — 2023-08-06T14:28:29.166Z

As some of you may know, I'm editing a special issue of the journal Philosophical Studies on AI safety (along with @Dan H ). I thought I'd share the first paper from the issue, which deals with some issues in AI safety theory that have been frequently discussed on LessWrong.

Here's the abstract:

A simple argument suggests that we can fruitfully model advanced AI systems using expected utility theory. According to this argument, an agent will need to act as if maximising expected utility if they’re to avoid exploitation. Insofar as we should expect advanced AI to avoid exploitation, it follows that we should expect advanced AI to act as if maximising expected utility. I spell out this argument more carefully and demonstrate that it fails, but show that the manner of its failure is instructive: in exploring the argument, we gain insight into how to model advanced AI systems.

You can find the paper here: https://link.springer.com/article/10.1007/s11098-023-02023-4
Reducing the risk of catastrophically misaligned AI by avoiding the Singleton scenario: the Manyton Variant
by GravitasGradient — 2023-08-06T14:24:04.774Z

This post does not try to add to the discussion about aligning an AGI/ASI to all human values but will rather focus on a smaller, arguably fundamental, subset of human values: prosocial behaviour.

(Note: with many agreeing that AGI and ASI will occur so closely together the differentiation between the two might be meaningless, we will only refer to ASI from now)

It aims to propose a possible way we can increase the chances of generating ASIs with prosocial values.

Almost all discussions relating to ASI risk discusses a singular ASI in isolation, what a singular mind would think and value and do in relation to Humanity, the so called Singleton outcome.

Few discuss the interactions of ASIs between each other, vaguely citing the truism that they will be beyond human comprehension and therefore their ‘social lives’, as it were, will be too.

I believe that this is not entirely true.

The truism that we might be unable to predict what an individual ASI might think and value holds true. But when we model ASIs as a group, as roughly equally powerful individuals, then we have large amounts of reliable scientific data on necessary behaviour for an individual to succeed and prosper.

Abstracted from the underlying mental structure and, most importantly, goals of the individual in question, an ASI that is not alone but one of many, the Manyton outcome, is a member of a group, of a society.

And we know how such members must behave.

Any individual that is part of a group of equals that wants to succeed long term and with minimal risk must develop a certain set of social skills e.g: Negotiating, honoring of contracts, the willingness to compromise.

Most importantly for us is the fact that, long term, individuals desire that their partners and fellow group members are trustworthy. This trustworthiness in groups is guaranteed and reinforced by developing not just social skills but prosocial values.

Traits such as affection, compassion, loyalty etc. are evolutionary necessary to ensure group cohesion and long term cooperation.

They are, to speak with Bostrom, convergent subgoals of social interaction, necessary for long term success.

Psychopathy or sociopathy can not be tolerated by a group for long without detrimental affects for all members. (A paperclipper that stubbornly tries to conquer everything to turn into paperclips will soon be beset by a coalition of others that attempt to...
Rebooting AI Governance: An AI-Driven Approach to AI Governance
by utilon — 2023-08-06T14:19:50.180Z
Link: https://forum.effectivealtruism.org/s/wQoe9HTX5CudRoZfM/p/ikTpmYqA4CrzFyzxb

This article is part of a series on Decision-Making under Deep Uncertainty (DMDU). Building on an introduction to complexity modeling and DMDU, this article focuses on the application of this methodology to the field of AI governance. If you haven't read the previous article, I strongly recommend pausing reading this article and going through the introduction first. Many aspects will only appear clear if we share the same basic terminology. I might be biased, but I think, it's worth it.

Why this article?

I have written up this article because of two reasons. The 1st reason is that various people have approached me, asking how I envision the application of the introduced methodology to particular EA cause areas. People want to see details. And understandably so. In my last article, I mentioned a bunch of EA cause areas, that I consider to be well-suited for systems modeling and decision-making under deep uncertainty (DMDU)[1]. This article is intended to be a (partial) answer for these people providing details on one particular cause area. The 2nd reason for writing this article is my own specialization in AI governance and my wish for people to get started working on this. I want to show a more practical and concrete side of this methodology. I also consider this article as a handy reference that I can share with interested parties as a form of coarse research proposal.

Summary

DMDU methodology is well-suited to address the complexity of AI governance for four reasons:

AI governance demarcates a complex socio-technical system that involves the interaction of technology and society.
DMDU methodology values the diversity of stakeholder perspectives in AI governance and encourages decision-making that incorporates various values and objectives.
AI governance is characterized by deep uncertainty due to the unpredictability of AI advancements, societal reactions, and potential risks. DMDU methodology embraces deep uncertainty by exploring a wide range of plausible futures and developing flexible strategies that can adapt to changing circumstances.
Robustness is crucial in AI governance, and DMDU supports the development of policies that can withstand disruptions, adapt to changes, and consider diverse stakeholder perspectives.

Traditional methods (standard predictive modeling + highly aggregated economic models) are ill-equipped to deal with the four previous

...
Model-Based Policy Analysis under Deep Uncertainty
by utilon — 2023-08-06T14:07:36.079Z
Link: https://forum.effectivealtruism.org/posts/kCBQHWqbk4Nrns8P7/model-based-policy-analysis-under-deep-uncertainty

This post is based on my introduction talk at EAGxBerlin 2022. It is intended for policy researchers who want to extend their tool kit with computational tools. I show how we can support decision-making with simulation models of socio-technical systems while embracing uncertainties in a systematic manner. The technical field of decision-making under deep uncertainty offers a wide range of methods to account for various parametric and structural uncertainties while identifying robust policies where we want to optimize for multiple objectives simultaneously for particularly vulnerable scenarios.

Summary

Real-world political decision-making problems are complex, with disputed knowledge, differing problem perceptions, opposing stakeholders, and interactions between framing the problem and problem-solving.
Modeling can help policy-makers to navigate these complexities.
Traditional modeling is ill-suited for this purpose.
Systems modeling is a better fit (e.g., agent-based models).
Deep uncertainty is more common than one might think.
Deep uncertainty makes expected-utility reasoning virtually useless.
Decision-Making under Deep Uncertainty is a framework that can build upon systems modeling and overcome deep uncertainties.
Explorative modeling > predictive modeling.
Value diversity (aka multiple objectives) > single objectives.
Focus on finding vulnerable scenarios and robust policy solutions.
Good fit with the mitigation of GCRs, X-risks, and S-risks.

Complexity

Complexity science is an interdisciplinary field that seeks to understand complex systems and the emergent behaviors that arise from the interactions of their components. Complexity is often an obstacle to decision-making. So, we need to address it.

Ant Colonies

Ant colonies are a great example of how complex systems can emerge from simple individual behaviors. Ants follow very simplistic rules, such as depositing food, following pheromone trails, and communicating with each other through chemical signals. However, the collective behavior of the colony is highly sophisticated, with complex networks of pheromone trails guiding the movement of the entire colony toward food sources and the construction of intricate structures such as nests and tunnels. The behavior of the colony is also highly adaptive, with the ability to respond to changes in the environment, such as changes in the availability of...
On being in a bad place and too stubborn to leave.
by TeaTieAndHat — 2023-08-06T11:45:49.771Z

I know, it’s probably a good deal too personal a topic for LW. Don’t hesitate to downvote it into oblivion.

In high school, I had to choose what to study at uni. I was, then, a bright enough student, autistic but just enough to be a little nerdy and dorky. I had been getting interested in social psychology, science, cognitive biases, and eventually rationality. I considered various options: architecture, medicine, engineering, biology, etc. But none of those suited me perfectly, and for most of them, I was held back by my lack of self-confidence. And by what I now know is a probable ADHD that made me shy away from the idea of working 15+ hours a day in engineering school, even if it seemed fun.

The only reasonable option left was the mistake. I really liked the sort of political, social, and economic topics that were being discussed on SSC and similar places. Improving the world sounded like a fun thing to do. So, I went for what was basically a PPE, confident that it was sold as being really great, famous, and very intellectually engaging. Quite promising, right? I did, however, suspect a less than perfect culture fit, and I was a little worried.

And rightly so. I arrived full of ideas about rationality in politics, before being quickly and firmly told that "we" didn’t care about science here. Another issue was the lack of depth, we barely skimmed over things. Now, being a 18 years old lonely nerd, learning stuff was most of my life, and I had expected a lot more —likely too much— out of that uni. So I got depressed. In fact, on a Friday in March 2020, I was pondering the practicalities of a suicide attempt for the following Monday, when it was announced that said Monday would be the start of lockdown. (In retrospect, if I hadn’t been prone to depression, I’d have read this and noticed that there wasn’t yet much to be depressed about. But that’s not how it happened, and instead, I did basically all the wrong things. Probably because using college to hang out with smart and interesting people when we didn’t have much in common —and during a pandemic— was particularly hard).

Anyway, will you be surprised if I tell you that the start of lockdown wasn’t a very good time to be depressed? Teenagers often do stupid things, and the stupid thing my depressed self did was to decide to stay in that university. Seemed safer and less risky, you see. So I carried on. I...
Safety-First Agents/Architectures Are a Promising Path to Safe AGI
by Brendon_Wong — 2023-08-06T08:02:30.072Z

Summary

Language model agents (LMAs) like AutoGPT have promising safety characteristics compared to traditional conceptions of AGI. The LLMs they are composed of plan, think, and act in highly transparent and correctable ways, although not maximally so, and it is unclear whether safety will increase or decrease in the future.

Regardless of where commercial trends will take us, it is possible to develop safer versions of LMAs, as well as other "cognitive architectures" that are not dependent on LLMs. Notable areas of potential safety work include effectively separating and governing how agency, cognition, and thinking arise in cognitive architectures.

If needed, safety-first cognitive architectures (SCAs) can match or exceed the performance of less safe systems, and can be compatible with many ways AGI may develop. This makes SCAs a promising path towards influencing and ensuring safe AGI development in everything from very-short-timeline (e.g. LMAs are the first AGIs) to long-timeline scenarios (e.g. future AI models are incorporated into or built explicitly for an existing SCA).

Although the SCA field has begun emerging over the past year, awareness seems low, and the field seems underdeveloped. I wanted to write this article so that more people are aware of what's happening with SCAs, document my thinking on the SCA landscape and promising areas of work, and advocate for more people, funding, and research going towards SCAs.

Background

Language model agents (LMAs), systems that integrate thinking performed by large language models (LLMs) prompting themselves in loops, have exploded in popularity since the release of AutoGPT at the end of March 2023.

Even before AutoGPT, the related field that I call "safety-first cognitive architectures" (SCAs) emerged in the AI safety community. Most notably, in 2022, Eric Drexler formulated arguments for the safety of such systems and developed a high-level design for an SCA called the open agency model. Shortly thereafter, while at FHI with Drexler, Davidad published a specification for an ambitious system using this design somewhat-confusingly called an open agency architecture (OAA) (specifications for open agencies that are much closer to current-day systems like AutoGPT are also possible). Work on OAA is now starting at organizations like the AI Objectives Institute and Protocol Labs. A few months later, Conjecture announc...
The Benevolent Ruler’s Handbook (Part 1): The Policy Problem
by FCCC — 2023-08-06T03:46:31.594Z

(Note that while I’ll frame everything in terms of government policy, all of this theory really applies to any form of decision-making, including that of an artificial intelligence. So if government is not your jam, please don’t get turned off by all the policy talk. The theory I’ve laid out has been oddly helpful to a lot of my life, and I hope it provides you with similar benefits.)

All hail the emperor

Let’s suppose you’re the benevolent emperor of the world. All those who opposed you have died (under mysterious circumstances) and all who remain will follow your policies without question.

Very mysterious circumstances…

How do you decide between two different policies in, let’s say, transport? Well, since you’re benevolent, it follows that the policies must be measured against some kind of moral criteria…

Bentham called. He’s bringing over the utili-tea.

Maybe we select the transport policy that results in the “better world”, that makes sense, right?

Wait, what if the two transport policies are identical, except one version extends one train with an extra railcar, within which cancer researchers produce a cure. On the one hand, the cancer-research version produces a better outcome than the plain-train version. But on the other hand, it is clearly stepping outside its purview as a transport system.

So choosing a policy solely because it results in the “better” outcome categorically does not make sense: Each policy must have a bounded domain.

Just use science, you fools.

What about if we approach the policy-choice problem scientifically?

New studies suggest that I’m right, and you’re wrong, so shut up

Aha! Let’s run a randomized experiment, we’ll test one transport system in half the world’s countries, and another transport system in the other half, and see which countries do better against some metric! Again, we reach the “bounded domains” problem, but if our metric is itself bounded to the domain of interest, we might avoid that problem.

The main issue here is the cost considerations: Even if the experiments had no financial cost, we still have the opportunity cost of running the wrong policy. And because of this, we must figure out how to choose which policies to test in the first place, which cannot be solved through empiricism: It’s obviously nonsensical to think, “Before I perform any experiment, I should perform an experiment”.

I’m not saying that empiricis...
Exploring the Multiverse of Large Language Models
by franky — 2023-08-06T02:38:02.784Z

[Crossposted from Medium.com.]

Credit:

Alexas_Fotos from Pixabay

Large language models can generate texts under different tasks, for example, question answering, machine translation, document summarization, etc. Beyond being right and wrong, large language models can also have some interesting behaviors, such as creativity, hallucination, and degeneration. This post explores the hypothesis that a large language model is a multiverse with the following properties:

The multiverse contains different generation scenarios, e.g., right, wrong, creativity, hallucination, degeneration, etc.
Decoding strategies could help us to jump into different universes, but might not be able to reach the target universe.
Navigating strategies could lead us to the universe we prefer, but might not be able to guarantee everything in the right place.
Multiple universes could be satisfied with a given task so developing a universal decoding/navigating strategy might be a difficult task.

If the above description is too vague, here are some tips:

An auto-regressive large language model generates a probability distribution of the next token over the token space implying the good, the bad, and the ugly of the future.
Decoding strategies refer to greedy sampling, top-k sampling, top-p sampling, temperature setting, etc., which are used to select the next token.
Navigating strategies refer to reinforcement learning, knowledge augmentation, prompting engineering, etc., which are used to assist the overall process of selecting the next tokens.

This post briefs the concept of the multiverse in large language models and some decoding/navigating strategies for it. Now let’s jump into

Avengers: Endgame

to get a taste of multiple universes.

Avengers: Endgame

Scene 1: Scott Lang discusses the idea of navigation with Steve Rogers and Natasha Romanoff.

Scene 2: Bruce Banner, Steve Rogers, and Natasha Romanoff help Scott Lang to make time travel.

Scene 3: Tony Stark solves time travel through a simulation of the Möbius strip.

Scene 4: Tony Stark brings the navigation device to Steve Rogers.

Now we have some sense of the multiverse in Avengers: Endgame, and how it could be right and wrong. But, how does it relate to the multiverse of large language models?

Toy Examples of Language Modeling

To illustrate the multiverse of a large language model, consider the following scenarios in Figure 1.

Figure 1. Creativity, Hallucin

...
Aligning my web server with devops practices: part 2 (security)
by VipulNaik — 2023-08-06T01:30:35.005Z

This is a continuation of my previous
post. See
the introduction at the top of the previous post to get more
information on what this series of posts is about.

This post focuses on devops practices I adopted to secure my web server.

Recommendations for others (transferable learnings)

If you have a server exposed to the Internet, use a firewall (or
security group in the case of AWS) to lock access to the server as
much as possible, limiting the ports and IP address ranges that can
be used to access it. This drastically reduces the risk surface
area. Even if you do nothing else, do this.

Separate, separate, separate. Even if things get compromised up to a
point, that shouldn't compromise everything. Keep in mind the
concept of defense in
depth
as you think of things.

Related to the above: Give only as much permission to each thing as
it actually needs; this idea is called the principle of least
privilege
and you may also hear of it as "right-sizing permissions" in some
contexts.

Security isn't just about preventing or limiting attacks, it's also
about being able to recover more quickly and get back to a clean
slate. Backups with streamlined restore procedures, that I covered
extensively in part
1, help a bunch with this.

Make sure you have good security (strong password, two-factor
authentication, etc.) on the online accounts that you use to manage
your servers. A secure server managed through an insecure online
account is like a locked safe that's wide open from above.

Some time after I started drafting this post, I discovered a great
podcast called Darknet Diaries that
covers a lot of security-related themes. Many of the concepts I talk
about in this post show up organically in the episodes of this
podcast, and I recommend this podcast for people interested in
security.

General philosophy of security

Security against leakage versus security against manipulation

Some aspects of security are about preventing information from leaking
out. Other forms of security are about preventing others from being
able to modify the behavior of the server or impersonate the server
to modify the expected behavior for others (e.g., serving their own
website in place of mine).

For a web server, the latter kind of security is more relevant, but
the former also matters since in at least some cases, the source code
of a public website is not public (and may contain infor...