trentmkelly/LessWrong-43k huggingface dataset:
Drexler’s Nanotech Forecast
by PeterMcCluskey — 2022-07-30T00:45:43.169Z
Link: http://www.bayesianinvestor.com/blog/index.php/2022/07/29/drexlers-nanotech-forecast/

In 1986, Drexler predicted (in Engines of
Creation) that we'd
have molecular assemblers in 30 years. They would roughly act as fast,
atomically precise 3-d printers. That was the standard meaning of
nanotech for the next decade, until more mainstream authorities co-opted
the term.

What went wrong with that forecast?

In my review of Where Is My Flying
Car?
I wrote:

Josh describes the mainstream reaction to nanotech fairly well, but
that's not the whole story.

Why didn't the military fund nanotech? Nanotech would likely exist
today if we had credible fears of Al Qaeda researching it in 2001.

I recently changed my mind about that last sentence, partly because of
what I recently
read
about the Manhattan Project, and partly due to the world's response to
COVID.

Drexler's vision, in Engines of Creation, was based on the assumption
that at least one government was able to do projects such as Manhattan
and Apollo.

I've now decided that there was something quite unusual about the US
ability to put the best and the brightest in charge of that many
resources. The US had it from something like 1940 through 1969.

At some point in the 1970s, people stopped worrying that the Soviet
Union had such an ability (I'm guessing the Soviets never had that
ability, althought they occasionally came close). Without an enemy that
people believed was capable of conquering us, politics as usual crept
back into all large projects.

I'm suggesting that Drexlerian nanotech needs the kind of competence
that the Manhattan Project demonstrated. That likely overestimates the
difficulty of nanotech today, unless we impose a requirement that it be
done on a 3 year schedule. I'm unsure whether that's enough of an
overestimate to alter the analysis.

Nanotech requires a significant amount of basic research on tools, and
on finding an affordable path that's feasible given the tools we can
produce.

That research could in principle be rewarded by academic fame. Academia
seems uninterested so far in the kind of vision that would produce such
fame. The military risks of nanotech might make that an appropriate
default. And even if academia were promoting nanotech research, I doubt
it would be sufficiently results-oriented to fulfill Drexler's
forecast.

That research is unlikely to be financed by VCs or big companies.
They're deterred by a combination of long time horizons (maybe 10-20
years on a typical budget?) ...
Humans Reflecting on HRH
by leogao — 2022-07-29T21:56:53.561Z

TL;DR: HRH places a theoretical bound on the best reflection process achievable by us. Notably, this is not necessarily guaranteed to converge to human values, nor is it something that is actually implementable in practice (analogously to HCH). In particular, this is intended to argue against claims that it is insufficient to model CEV as the output of some kind of reflection process. One can also think of HRH as an operationalization of the idea of a long reflection.

Thanks to Connor Leahy, Tasmin Leake, and AI_WAIFU for discussion.

One thing we might want to do is to define our CEV (i.e the ultimate thing we actually want our AGI to optimize for the rest of time) as the output of some long-running deliberation process (a long reflection). This would be extremely convenient; one could imagine having an AGI that lets the reflection process run untampered with, and then implements whatever it decides on. However, one might worry that this could be impossible -- perhaps there are kinds of moral progress that can't be captured in the frame of a reflection process, that require some kind of more sophisticated formalism to capture.

However, consider the reflection process defined as follows: it takes in information from the real world and the utility function output from the previous iteration of reflection, and has a human deliberate for a while and then outputs the improved utility function and crucially also an improved reflection process to be used in the next iteration (you can also tack on the ability for it to interact with the world, which enables it to run experiments, consult a computer, build more powerful aligned AGIs to help, etc). Let's call this Humans Reflecting on HRH. This process essentially covers any process we can use to come up with better theories of what our CEV is.

(If it bothers you that it "modifies" itself, you can think of it as a fixed function that takes in an initial program and optional additional inputs, and outputs a new program, and the function just evals the program internally at every step. The higher level algorithm of "have each step determine the algorithm used in the next step" remains constant, and that's the thing I refer to.)

I claim that HRH is the best achievable reflection process up to constant factors. Suppose you could come up with a better reflective process. Then the version of you in HRH would co...
Comparing Four Approaches to Inner Alignment
by Lucas Teixeira — 2022-07-29T21:06:09.494Z

Early work on this was supported by

CEEALAR

and was finished during an internship at

Conjecture

under the mentorship of Adam Shimi.

Thank you to Evan Hubinger and Rohin Shah for answering some questions related to this post.

Epistemic Status: First

palimpsest

of many to come.

Intro and Motivation

I am broadly interested in pluralism in scientific development, and more practically interested in

supporting the diversification of conceptual alignment

.

Towards that end, I believe that having a systematic analysis of the cruxes and methodologies which split up the field into distinct agendas could help clarify how exactly diversity is created and sustained in our field, and what exactly it is we wish to

diversify in order to better manage our collective portfolio of research bets

.

As a case study, this post will investigate four different approaches to inner alignment. I’ll be taking a look at the different definitions which they use for “outer alignment” and conjecturing on how:

Despite inconsistencies across approaches, the different definitions utilized by each approach maintain coherent when understood against the backdrop of the aims local to that approach
Which cruxes set these different approaches apart.

This post is distillational in nature, and as such most of the ideas which I present here are not novel and not my own. The claims in this post should also be read as a part of an ongoing exploratory process, and any pushback (especially from those whose work I cite) would be beneficial.

The Approaches

The Mechanistic Approach

The mechanistic approach deconfused a lot of previous work on

optimization daemons

and provided

the now canonical argument for why we’d expect mesa-optimizers and inner misalignment

, which is summarized as follows:

Traditionally, most of the optimization pressure (or in other words, searching procedures) is implemented looking for a specific model which performs competently on a task during training time. However, as ML becomes more and more powerful, models will be trained to more complicated tasks across a wider range of environments. This puts more pressure for a model to generalize better, and raises the incentives for the model to delegate some of its optimization power to deployment time, rather than training time. It is in those cases where we get a mesa-optimizer, and it is with emergenc...
Questions for a Theory of Narratives
by Marv K — 2022-07-29T19:31:07.472Z

I kindly ask for your comments, questions and feedback of any kind.

Various predictive and rigorous mathematical (/logical/linguistic) frameworks exist for analyzing and implementing agent's world MODELLING- especially that of causality (e.g. pearl's structural causal model) -, their ACTION (e.g. decision theory, reinforcement learning) and some aspects of their COMMUNICATION (information theory; vocabulary, syntax, semantics, pragmatics). I believe one missing link between communication and the former two aspects of intelligent agents is a formal theory of NARRATIVES or stories: At least as far as I am aware, there is no good theory of why we share bits of world model and policy with one another in the way that we do.

This is a very rough dissatisfaction, sort of an itch for having one or several objective functions for narratives that are more specific than very general information theoretic criterions (which are usually agnostic of the structure of world model and policy). I have a myriad of questions that I want a good model of narratives to answer, and think sharing these is the best way to elucidate why I think a theory of narratives - or at least implementable, validatable models of specific aspects of them - is necessary.

The questions below are in not too much particular order, and at the moment still very rough. In fact some questions are so general that I dont expect that they can be answered at all; I still think they are useful/currently the best/only tools I have to make a sketch of the space I want to explore. In any case, I will probably follow up with a post where I try to lay out some terminology more rigorously; and then go into the first two questions more in-depth.

How exactly do narratives relate to causality? They seem to be situational compressions of internal models of causality, with the goal to efficiently inform w.r.t. a part of one's world model or policy. Narratives thus seem to pick out relevant, relatively simple (acyclic?) subgraphs of a more complex internalized (cyclic?) causal graph.
Is the internal causal graph possibly cyclic due to the nodes (abstractions) grouping multiple individual events (which have a (possibly known) acyclic causal structure) into (classes/ensembles of) objects (that, individually and together, may persist across time), thus having multiple interactions with neighboring nod

...
Focusing
by CFAR!Duncan — 2022-07-29T19:15:35.377Z

Epistemic status: Firm

The Focusing technique was developed by Eugene Gendlin as an attempt to answer the question of why some therapeutic patients make significant progress while others do not. Gendlin studied a large number of cases while teasing out the dynamics that became Focusing, and then spent a significant amount of time investigating whether his technique-ified version was functional and efficacious. While the CFAR version is not the complete Focusing technique, we have seen it be useful for a majority of our alumni.

If you’ve ever felt your throat go suddenly dry when a conversation turned south, or broken out into a sweat when you considered doing something scary, or noticed yourself tensing up when someone walked into the room, or felt a sinking feeling in the pit of your stomach as you thought about your upcoming schedule and obligations, or experienced a lightness in your chest as you thought about your best friend’s upcoming visit, or or or or ...

If you’ve ever had those or similar experiences, then you’re already well on your way to understanding the Focusing technique.

The central claim of Focusing (at least from the CFAR perspective) is that parts of your subconscious System 1 are storing up massive amounts of accurate, useful information that your conscious System 2 isn’t really able to access. There are things that you’re aware of “on some level,” data that you perceived but didn’t consciously process, competing goalsets that you’ve never explicitly articulated, and so on and so forth.

Focusing is a technique for bringing some of that data up into conscious awareness, where you can roll it around and evaluate it and learn from it and—sometimes—do something about it. Half of the value comes from just discovering that the information exists at all (e.g. noticing feelings that were always there and strong enough to influence your thoughts and behavior, but which were somewhat “under the radar” and subtle enough that they’d never actually caught your attention), and the other half comes from having new threads to pull on, new models to work with, and new theories to test.

The way this process works is by interfacing with your felt senses. The idea is that your brain doesn’t know how to drop all of its information directly into your verbal loop, so it instead falls back on influencing your physiology, and hoping that you notice (or simply respond)...
Conjecture: Internal Infohazard Policy
by Connor Leahy — 2022-07-29T19:07:08.491Z

This post benefited from feedback and comments from the whole Conjecture team, as well as others including Steve Byrnes, Paul Christiano, Leo Gao, Evan Hubinger, Daniel Kokotajlo, Vanessa Kosoy, John Wentworth, Eliezer Yudkowsky. Many others also kindly shared their feedback and thoughts on it formally or informally, and we are thankful for everyone's help on this work.

Much has been written on this forum about infohazards, such as information that accelerates AGI timelines, though very few posts attempt to operationalize that discussion into a policy that can be followed by organizations and individuals. This post makes a stab at implementation.

Below we share Conjecture’s internal infohazard policy as well as some considerations that we took into account while drafting it. Our goal with sharing this on this forum is threefold:

To encourage accountability. We think that organizations working on artificial intelligence - particularly those training and experimenting on large models - need to be extremely cautious about advancing capabilities and accelerating timelines. Adopting internal policies to mitigate the risk of leaking dangerous information is essential, and being public about those policies signals commitment to this idea. I.e., shame on us if we break this principle.

To promote cross-organization collaboration. While secrecy can hurt productivity, we believe that organizations will be able to work more confidently with each other if they follow similar infohazard policies. Two parties can speak more freely when they mutually acknowledge what information is sharable and to whom it can be shared, and when both show serious dedication to good information security. A policy that formalizes this means that organizations and individuals don’t need to reinvent norms for trust each time they interact.
Note that at the current level of implementation, mutual trust relies mostly on the consequence of "if you leak agreed-upon secrets your reputation is forever tarnished.” But since alignment is a small field, this seems to carry sufficient weight at current scale.

To start a conversation that leads to better policies. This policy is not perfect, reviewers disagreed on some of the content or presentation, and it is guaranteed that better versions of this can be made. We hope that in its imperfection, this policy can act as a seed from which be

...
Abstracting The Hardness of Alignment: Unbounded Atomic Optimization
by adamShimi — 2022-07-29T18:59:49.460Z

This post is part of the work done at Conjecture.

Disagree to Agree

(

Practically-A-Book Review: Yudkowsky Contra Ngo On Agents

, Scott Alexander, 2022)

This is a weird dialogue to start with. It grants so many assumptions about the risk of future AI that most of you probably think both participants are crazy.

(Personal Communication about a conversation with Evan Hubinger, John Wentworth, 2022)

We'd definitely rank proposals very differently, within the "good" ones, but we both thought we'd basically agree on the divide between "any hope at all" and "no hope at all". The question dividing the "any hope at all" proposals from the "no hope at all" is something like... does this proposal have any theory of change? Any actual model of how it will stop humanity from being wiped out by AI? Or is it just sort of... vaguely mood-affiliating with alignment?

If there's one thing alignment researchers excel at, it's disagreeing with each other.

I dislike the term pre paradigmatic, but even I must admit that it captures one obvious feature of the alignment field: the constant debates about the what and the how and the value of different attempts. Recently, we even had

a whole sequence of debates

, and since I first wrote this post Nate

shared his take

on why he can’t see any current work in the field actually tackling the problem. More generally, the culture of disagreement and debate and criticism is obvious to anyone reading the AF.

Yet Scott Alexander has a point: behind all these disagreements lies so much agreement! Not only in discriminating the "any hope at all" proposals from the "no hope at all", as in John's quote above; agreement also manifests itself in the common components of the different research traditions, for example in their favorite scenarios. When I look at Eliezer's

FOOM

, at Paul's

What failure looks like

, at Critch's

RAAPs

, and at Evan's

Homogeneous takeoffs

, the differences and incompatibilities jump to me — yet they still all point in the same general direction. So much so that one can wonder if a significant part of the problem lies outside of the fine details of these debates.

In this post, I start from this hunch — deep commonalities — and craft an abstraction that highlights it: unbounded atomic[1] optimization (abbreviated UAO and pronounced wow). That is, alignment as the problem of dealing with impact on...
Bucket Errors
by CFAR!Duncan — 2022-07-29T18:50:48.549Z

Author's note: There is a preexisting standalone essay on bucket errors by CFAR cofounder Anna Salamon available here. The version in the handbook is similar, but has enough disoverlap that it seemed worth including it rather than just adding the standalone post to the sequence.

Epistemic status: Mixed

The concept of question substitution, which underlies and informs this chapter, is one that is well-researched and well-documented, particularly in the work of Daniel Kahneman. The idea of “bucket errors” is one generated by CFAR staff and has no formal research behind it, but it has resonated with a majority of our alumni and seems like a reasonable model for a common class of human behaviors.

Humans don’t simply experience reality. We interpret it.

There’s some evidence that this is true “all the way down,” for literally everything we perceive. The predictive processing model of cognition posits that even very basic sensations like sight and touch are heavily moderated by a set of top-down control systems, predictions, and assumptions—that even as the photons are hitting our receptors, we’re on some level anticipating them, already attempting to define them and categorize them and organize them into sensible clusters. It’s not just a swirl of green and brown, it’s a tree—and we almost can’t stop ourselves from seeing the tree, and go back to something like unmediated perception.

CFAR’s concept of “buckets” is a similar idea on a broader scale. The claim is that reality is delivering to you a constant stream of experiences, and that—most of the time—you are categorizing those experiences into pre-existing mental buckets. Those buckets have titles like “do they like me?” and “is this a good idea?” and “what’s my boss like?” and “Chinese food?”

If you think of your mental architecture as being made up of a large number of beliefs, then the buckets contain the piles of evidence that lie behind and support those beliefs. Or, to put it another way, you know whether or not you like Chinese food because you can look into the bucket containing all of your memories and experiences of Chinese food and sum them up.

As another example, let’s say Sally is a young elementary school student with a belief that she is a good writer. That belief didn’t come out of nowhere—it started with observations that (say) whenever she turned in a paper, her teacher would smile and put a star-sh...
Distillation Contest - Results and Recap
by Aris — 2022-07-29T17:40:03.158Z

This post:

Announces the winning submissions to the Distillation Contest
Gives further insight into the scoring process for the contest
Examines the effectiveness of our advertising strategies
Gives a brief impact estimate of the contest
Shares my advice for community builders who are planning to run contests

Notes:

A huge thank you to Akash and all of the judges for this contest! This wouldn’t have been possible without their work. I’m definitely not perfect! I imagine there are better ways to advertise, run, and score a contest, so I wanted to be transparent about my process so that other people could make suggestions if they have ideas for improvements.

Want to use the materials from the Distillation Contest to run your own version? I’m in the process of creating a platform for EA contests and will soon have a demo site up! I’m planning to upload all of the Distillation contest materials into a “bundle” so that anyone can host their own contest easily. Once the site is up (hopefully in the next couple of weeks), feel free to use anything there and iterate to make better resources. If you have your own EA contest resources and would like to make them available to other people, I’d love to add them to the notion!

Cross-posted on the EA Forum.

Winners

The submission winning first place in the Distillation Contest is

Understanding Selection Theorems

by UC Berkeley’s Adam Khoja – distilling John Wentworth's

Selection Theorems: A Program for Understanding Agents

. Our second-place winner is

The Geometry of Adversarial Perturbations

by Gabriel Wu from Harvard University – a distillation of

Universal Adversarial Perturbations

.

We’ve granted 15 other prizes, six $500 awards, and nine $250 awards, plus three honorable mentions. The six $500 winners are listed below, with their distillations linked to their names. You can find the other finalists, and their distillations, on the EA Berkeley distillation contest

winners page

.

Callum McDougall

,

Cambridge

Jasper Day, University of Edinburgh (has not yet given permissions to share submission)

Harrison Gietz

, Louisiana State University

Sasha Sato

,

UC Berkeley

Chinmay Deshpande

, Harvard University

Yash Dave

,

UC Berkeley

Scoring

Each submission was scored by two judges (of which we had five total, all of whom actively work in the alignment space). Our rubric took into account the submission’s Depth of Unders...