trentmkelly/LessWrong-43k huggingface dataset:
Exploring Mild Behaviour in Embedded Agents
by Megan Kinniment — 2022-06-27T18:56:34.794Z

Thanks to Tristan Cook, Emery Cooper, Nicolas Macé and Will Payne for discussions, advice, and comments. All views are my own.

I think this post is best viewed as an effort to point at a potentially interesting research direction. The specific arguments I make centered around inducing mild behaviour in embedded agents are rough and exploratory.

Summary

If an embedded agent's utility function is heavily dependent on the parts of the environment which constitute its own thinking processes, we might see behaviour that does not have an equivalent in unembedded agents.

(This seems pretty likely to me. Of all the claims here I am most confident of this one.)

We might be able to design measures/utility functions that constrain the parts of the environment that constitute the agent’s thinking process itself, such that we can moderate the strength of the optimisation that an embedded agent exerts on the rest of the environment.

(This seems in theory possible to me but I am currently very uncertain about whether this is feasible in practice.)

Following this, we may be able to induce mild optimisation in EU maximising embedded agents through the specification of the agent’s utility function, and this induced ~mild optimisation could help protect that very same mildness-inducing specification from being gamed.

(Similar thoughts to above)

Measures that impose optimisation pressure on the thinking process might be more robust than usual to specification problems if they can induce some degree of mild optimisation even when they are not specified in a water-tight manner. To the extent that we can manage to specify a mediocre efficiency measure that induces even some small degree of mild optimisation, it will also slightly reduce the optimisation pressure the agent puts on the specification as a whole. In this way, there might be some potential for measures like these to be made somewhat self-protecting against specification problems like

edge instantiation

and

nearest unblocked strategies

.

(Again, similar thoughts to above, though I do currently find this general idea promising.)

Context

A fundamental property of embedded agents is that the agent’s thinking process is a part of the environment itself (for example, some process running on a physical chip). This can be a driver of certain ‘bad’ behaviours like wireheading, unwanted self-preservation, or...
Epistemic modesty and how I think about AI risk
by Aryeh Englander — 2022-06-27T18:47:35.827Z

[This is a direct cross-post from a Facebook post of mine and was originally intended for the people who read what I write on Facebook, which is largely why this is written as a "for the record" statement of my personal viewpoints. That's probably not how I would have written this if I had originally intended it as a LessWrong post. In any case, I am cross-posting this here because I think people will disagree with me about the methodology and I'd like to get feedback on that.]

For the record, my actual current opinion on the plausibility of catastrophic risks from very advanced AI is... that I really don't know.

I currently lean towards thinking that there's at least a decent chance within the next 10-40 years that we will get AI systems that are human-level-or-above at relevant tasks like long-term strategic planning, scientific research and engineering, and human social manipulation. And conditional on achieving that level of AI, I currently lean towards thinking that there's at least a non-negligible chance that such systems will end up causing permanent catastrophic harm to the future of humanity, perhaps even human extinction or some really horrible dystopia. But I'm also extremely uncertain about those statements. If I had to put probability estimates to those claims, then my estimate for the first claim (human level AI or above in 10-40 years) would probably be somewhere in the range 5%-95%, and my estimate for the second claim (catastrophically bad outcome conditional on the the first claim) might be somewhere between 0.1% and 95%. So yeah - really, really uncertain.

This is actually what my PhD research is largely about: Are these risks actually likely to materialize? Can we quantify how likely, at least in some loose way? Can we quantify our uncertainty about those likelihoods in some useful way? And how do we make the best decisions we can if we are so uncertain about things?

One aspect that shapes a lot of my thinking on this and many other topics is epistemic modesty - very roughly, looking at what the "experts" say about the topic. At any given time it may seem to me that the object-level arguments point one way or another, but honestly: (a) I know that I don't know nearly as much on this topic as lots of other people who are at least as smart as I am if not way smarter, plus I haven't thought about it nearly as long, and yet they still disagree w...
Deliberation Everywhere: Simple Examples
by Oliver Sourbut — 2022-06-27T17:26:20.848Z

The analysis and definitions used here are tentative. My familiarity with the concrete systems discussed ranges from rough understanding (markets and parliaments), through abiding amateur interest (biology), to meaningful professional expertise (AI/ML things). The abstractions and terminology have been refined in conversation and private reflection, and the following examples are both generators and products of this conceptual framework.

We previously discussed a conceptual algorithmic breakdown of some aspects of goal-directed behaviour with the intention of inspiring insights and clarifying thought and discussion around these topics.

The examples presented here include some original motivating examples, some used to refine the concepts, and others drawn from the menagerie after the concepts were mostly refined[1]. Each example is subjected to the analysis, in several cases drawing out novel insights as a consequence.

Most of these examples, for all their intricacy in some cases, are relatively 'simple' as deliberators, and I am quite confident in the applicability of the framing. Analysis of more derived and sophisticated deliberative systems is reserved for upcoming posts.

Brief framework summary

We decompose 'deliberation' into 'proposal', 'promotion', and 'action'.

Propose:S→Δ{X}nonempty (generate candidate proposals)

Promote:S→{X}→{V} (promote and demote proposals according to some criterion)

Act:{X×V}→A (take outcome of promotion and demotion to activity in the environment)

We also identify as important whether a deliberator's actions are final, or give rise to relevantly-algorithmically-similar subsequent deliberators (iteration and replication), or create or otherwise condition heterogeneous deliberators (recursive deliberation).

Reaction examples

Chemical systems

Innumerable basic chemical reactions, like oxidation of iron, involve actions which change the composition or configuration of some material(s). For the purposes of this analysis these are, alone, mostly uninteresting, but serve to illustrate natural systems which do not preserve their essential algorithmic form and thus do not constitute iterated systems.

Some reactions, on the other hand, involve catalysis, wherein some reagents are essentially preserved or reconstituted in the action, producing the seeds of iterated algorithmic reaction, thus basic 'control'.

Despite pushing in a particula...
Deliberation, Reactions, and Control: Tentative Definitions and a Restatement of Instrumental Convergence
by Oliver Sourbut — 2022-06-27T17:25:45.986Z

This analysis is speculative. The framing has been refined in conversation and private reflection and research. To some extent it feels vacuous, but at least valuable for further research and communication.

A cluster of questions fundamental to many concerns around risks from artificial systems regard the concepts of search, planning, and 'deliberateness'. How do these arise? What can we predict about their occurrence and their consequences? How strong are they? What are they anyway?

Here is laid out one part of a conceptual decomposition which maps well onto many known systems and may allow further work towards answering more of those questions. The ambition is to really get at the heart of what is algorithmically happening in 'optimising systems', including humans, animals, algorithmic optimisers like SGD, and contemporary and future computational artefacts. That said, I do not have any privileged insight into the source code (or its proper interpretation!) for the examples discussed, so while this framing has already generated new insights for me, it may or may not be 'the actual algorithmic truth'.

We start with the analysis: a definition of 'deliberation' and its components, then of 'reactions' and 'control'. Next we consider, in light of these, what makes a deliberator or controller 'good'. We find conceptual connections with discussions of instrumental convergence. Little attention is given here to how to determine what the goals are, which is obviously also important.

These concepts were generated by contemplating various aspects of many different goal-directed systems and pulling out commonalities.

Some readers may prefer to start with the examples, which include animals, plants, natural selection, gradient descent, bureaucracies, and others. Here in the conceptual section I'll footnote particularly relevant concrete examples where I anticipate them helping to convey my point.

A full treatment is absent, but two major deferences to embedded agency underlie this analysis. A Cartesian separation need not be assumed, except over 'actor-moments' rather than temporally-extended 'actors'. And a major driver for this sequence is a fundamental recognition that any goal-directed behaviour instantiated in the real world must have bounded computational capacity per time[1].

Inspiration and related

My (very brief) 'Only One Shot' intuition pump for embedded agency ma...
Are long-form dating profiles productive?
by AABoyles — 2022-06-27T17:03:35.266Z

There's been a noticeable trend over the last few years of in this community of writing long-form dating profiles. They have the benefit of being slightly costly signals, requiring significantly more time and attention and a much greater degree of disclosure than can be devoted to a traditional online dating profile. But they are (at present) publicized and distributed manually, sharply limiting the possible audience.

Has anyone who's written one found a date (and subsequently, relationship) by doing so?

Separately, if you've written one, regardless of whether you met anyone because of it, do you believe it was worth the effort?
Custom iPhone Widget to Encourage Less Wrong Use
by Will Payne — 2022-06-27T16:14:50.141Z
Link: https://forum.effectivealtruism.org/posts/T9YhNbKfibTtphH5m/custom-iphone-widget-to-encourage-ea-forum-use

TLDR: I made a little custom iPhone widget for the EA Forum recently and have been getting a lot of value out of it. It takes an extra 10s to make this a LessWrong or Alignment forum widget. At the moment I don't feel bothered to make this into an app which is easy to install, but I think people could set it up themselves in ~5 mins or less, and if you don't use the forum much but want to use it more it might help.

What is it?

I set up a EA Forum[1] widget using a little iPhone personalisation tool called Scriptable. It shows the details of a random front page forum post once per hour. And it looks like this...

At the time of writing I haven't read Ozymandias' post

Why I think it's good

I've noticed that when I interact with the forum I usually enjoy it. However, I usually don't reach for the forum over other easily available sources of entertainment like YouTube, Instagram, or TikTok (although don't worry, I deleted TikTok). I made this widget initially to try and make the Forum an easy reach alternative to these apps. I think it worked; my usage of the forum went from once per month (or less) to once or twice a day. At one point I removed the widget and didn't get around to re-adding it for a few months, and my forum usage dropped down again. Since reinstalling it a few days ago, it has jumped back up. I’d put this then at somewhere around a 10 - 100x increase in value out of the forum for me, but don’t know yet if it will work long term or just hold my attention for 1-2 week chunks.

How to set it up

This works for iPhone and presumably iPad (although I haven't been able to test it on iPad).

Download the

Scriptable app

then open it
Click the little plus icon in the top right corner

Paste

this code

[2] into the editor
For LessWrong replace getForumPosts(EAForum) with getForumPosts(LessWrong) on line 10.
Go to your home screen
Long press on the home screen or any app to enter jiggle mode
Click the plus in the top left to add a widget
Find Scriptable (You can use the search bar)
Pick the midsized view and click "Add widget" at the bottom
Tap on the widget
Click choose script
Pick the script you just made (If you didn't actively change the name it should be the only option called "Untitled Script")
Tap anywhere else on the screen to exit jiggle mode.

You're done! I just ran through this process and it took a little over 1 minute. Here's

a video.

I hope this helps you as mu...
Announcing the Inverse Scaling Prize ($250k Prize Pool)
by Ethan Perez — 2022-06-27T15:58:19.135Z

TL;DR: We’re launching the

Inverse Scaling Prize

: a contest with $250k in prizes for finding zero/few-shot text tasks where larger language models show increasingly undesirable behavior (“inverse scaling”). We hypothesize that inverse scaling is often a sign of an alignment failure and that more examples of alignment failures would benefit empirical alignment research. We believe that this contest is an unusually concrete, tractable, and safety-relevant problem for engaging alignment newcomers and the broader ML community. This post will focus on the relevance of the contest and the inverse scaling framework to longer-term AGI alignment concerns. See our GitHub repo for contest details, prizes we’ll award, and task evaluation criteria.

What is Inverse Scaling?

Recent work has found that Language Models (LMs) predictably improve as we scale LMs in various ways (“

scaling laws

”). For example, the test loss on the LM objective (next word prediction) decreases as a power law with compute, dataset size, and model size:

Scaling laws appear in a variety of domains, ranging from

transfer learning

to generative modeling (on

images, video, multimodal, and math

) and

reinforcement learning

. We hypothesize that alignment failures often show up as scaling laws but in the opposite direction: behavior gets predictably worse as models scale, what we call “inverse scaling.” We may expect inverse scaling, e.g., if the training objective or data are flawed in some way. In this case, the training procedure would actively train the model to behave in flawed ways, in a way that grows worse as we scale. The literature contains a few potential examples of inverse scaling. For example, increasing LM size appears to increase social biases on

BBQ

and falsehoods on

TruthfulQA

, at least under certain conditions. As a result, we believe that the prize may help to uncover new alignment-relevant tasks and insights by systematically exploring the space of tasks where LMs exhibit inverse scaling. In particular, submissions must demonstrate new or surprising examples of inverse scaling, e.g., excluding most misuse-related behaviors where you specifically prompt the LM to generate harmful or deceptive text; we don't consider scaling on these behaviors to be surprising in most cases, and we're hoping to uncover more unexpected, undesirable behaviors. Below, we outline t...
Announcing Epoch: A research organization investigating the road to Transformative AI
by Jsevillamol — 2022-06-27T13:55:51.451Z
Link: https://epochai.org/blog/announcing-epoch

Summary

We are a new research organization working on investigating trends in Machine Learning and forecasting the development of Transformative Artificial Intelligence
This work is done in close collaboration with other organizations, like

Rethink Priorities

,

Open Philanthropy

, and

MIT CSAIL

We will be hiring for 2-4 full-time roles this summer – more information

here

You can find up-to-date information about Epoch on

our website

What is Epoch?

Epoch

is a new research organization that works to support AI strategy and improve forecasts around the development of

Transformative Artificial Intelligence (TAI)

– AI systems that have the potential to have an effect on society as large as that of the industrial revolution.

Our founding team consists of seven members – Jaime Sevilla, Tamay Besiroglu, Lennart Heim, Pablo Villalobos, Eduardo Infante-Roldán, Marius Hobbhahn, and Anson Ho. Collectively, we have backgrounds in Machine Learning, Statistics, Economics, Forecasting, Physics, Computer Engineering, and Software Engineering.

Our work involves close collaboration with other organizations, such as

MIT CSAIL

,

Open Philanthropy

, and

Rethink Priorities’

AI Governance and Strategy team

. We are advised by Tom Davidson from Open Philanthropy and Neil Thompson from MIT CSAIL. Rethink Priorities is also our fiscal sponsor.

Our mission

Epoch seeks to clarify when and how TAI capabilities will be developed.

We see these two problems as core questions for informing AI strategy decisions by grantmakers, policy-makers, and technical researchers.

We believe that to make good progress on these questions we need to advance towards a field of AI forecasting. We are committed to developing tools, gathering data and creating a scientific ecosystem to make collective progress towards this goal.

Epoch´s website

Our research agenda

Our work at Epoch encompasses two interconnected lines of research:

The analysis of trends in Machine Learning. We aim to gather data on what has been happening in the field during the last two decades, explain it, and extrapolate the results to inform our views on the future of AI.
The development of quantitative forecasting models related to advanced AI capabilities. We seek to use techniques from economics and statistics to predict when and how fast AI will be developed.

These two research strands feed into each other: the analysis of trends informs the choice of parame...
Air Conditioner Repair
by Zvi — 2022-06-27T12:40:01.514Z

A thing happened to me, seemed worth sharing.

What Happened

A few weeks ago, the in-wall air conditioner in our bedroom ceased to function.

Troy Barnes was unavailable, so instead we talked to the super, he recommended a company called Amhac, we ignored normative determinism (e.g. “Am a hack”) and called them. They charged a rather large amount simply to come out and take a look, with a diagnostic fee and also a noticeably generous per-hour fee.

Then, a few days before they were supposed to show up, our other in-wall air conditioner sprung a leak – or rather, the leak got big enough that we noticed it – and the super advised us we had to shut it down until it was fixed, but he thought that one would be a simple fix plus the addition of a failsafe that should have been there to stop the leak; he wasn’t sure about the bedroom one.

The repairmen showed up, but they hadn’t completed the necessary certificate of insurance to be let into the building – this policy is even more annoying and friction generating than it sounds – and we had to reschedule while they sorted this out. A number of rather hot nights later, they managed to come in.

Their report was that both air conditioners were unfixable. The one in the bedroom was completely shot. The one in the living area was fixable in theory, they said, but due to some EPA regulation it wasn’t possible to fix it.

They could send over a contract for the new units, it would be… $28,000.

I Googled for various new air conditioners, and couldn’t find ones that would be that expensive. The guy explained I would need two of each unit, for four units total.

I asked some trusted friends about all this, as well as the super, and all agreed a second quote would be a very good idea. I asked the super who else had done work in the building, and he remembered M.LaPenna Refrigeration, Inc. They charged by the hour as well, but without a ‘diagnostic fee’ up front.

Which on reflection should have been a hint about the first company. If you’re charging for the labor, as you do, starting with minute one, why is there an additional diagnostic fee? It doesn’t actually make any sense.

This time, they took care of the certificate of insurance as fast as the insurance company could handle it, then got a guy out to look the same day that got approved. Almost as if things were urgent. Different attitude.

Instead of two workers, this time ther...
Why Are Posts in the Sequences Tagged [Personal Blog] Instead of [Frontpage]?
by DragonGod — 2022-06-27T09:35:26.778Z

Many (most?) posts in the Sequences are tagged as [Personal Blog] instead of [Frontpage]; is this deliberate?

Here are a few examples from posts that I recently had reason to consult:

Belief in Intelligence:

Aiming at the Target:

Measuring Optimization Power:

Efficient Cross-Domain Optimization:

(There are several others, these are just the posts I had reason to interact with today).

Perhaps they should be moved to the front page? Several posts there seem obviously fitting for front page material, and they are quite central to the LW community.
Contest: An Alien Message
by DaemonicSigil — 2022-06-27T05:54:54.144Z

Scenario

You're not certain that aliens are real. Even after receiving that email from your friend who works at NASA, with the subject line "What do you make of this?" and a single binary file attached, you're still not certain. It's rather unlikely that aliens would be near enough to Earth to make contact with humanity, and unlikelier still that of all the humans on Earth, you would end up being the one tasked with deciphering their message. You are a professional cryptanalyst, but there are many of those in the world. On the other hand, it's already been months since April Fools, and your friend isn't really the prankster type.

You sigh and download the file.

Rules

No need to use spoiler text in the comments: This is a collaborative contest, where working together is encouraged. (The contest is between all of you and the difficulty of the problem.)

Success criteria: The people of LessWrong will be victorious if they can fully describe the process that generated the message, ideally by presenting a short program that generates the message, but a sufficiently precise verbal description is fine too.

The timeframe of the contest is about 2 weeks. The code that generated the message will be revealed on Tuesday July 12.

Why is this interesting?

This contest is, of course, based on That Alien Message. Recently there was some discussion under I No Longer Believe Intelligence to be "Magical" about how realistic that scenario actually was. Not the part where the entire universe was a simulation by aliens, but the part where humanity was able to figure out the physics of the universe 1 layer up by just looking at a few frames of video. Sure it may be child's play for Solomonoff Induction, but do bounded agents really stand a chance? This contest should provide some experimental evidence.