trentmkelly/LessWrong-43k huggingface dataset:
The ELK Framing I’ve Used
by sudo — 2022-09-19T10:28:46.755Z

Epistemic Status: Uncertain.

This post assumes familiarity with ARC’s Eliciting Latent Knowledge paper.

After working on ELK for a little while, I arrived at a slightly differently problem. I feel like this problem still captures the core difficulty of ELK.

The purpose of this post is not to add anything new to ELK. Instead, it shares a framing that I found helpful.

A Slightly New Framing for ELK

Imagine that we have a world-state observer/describer D, a world-state predictor P, and an optimizer M.

P’s job is to observe M and predict how it will change the world.

M’s job is to optimize for some arbitrary goal. In the case of the toy scenario, M keeps the vault secure.

D’s job is to provide feedback on any given world state.

Example Strategy #1 (taken from ELK paper)

A very simple ELK strategy would be for D to be a classifier which predicts whether a given world is good or bad (based on human feedback). M would create a plan, and P would observe M. Finally, P would report to humans what D would think about what P is likely to do.

Counterexample

This strategy breaks because (among other things) M might simply create a world that looks fine to humans, but actually is not fine. The set of worlds that humans might think look fine is much bigger than the set of worlds that are compatible with human existence.

Example Strategy #2 (I came up with this one. I’m not sure if I’m the first to do so)

A slightly more refined ELK strategy might be for D to use a variety of sensors to output a rich textual description of its environment. Then, this rich textual description might be rated as good or bad by humans. This rating system is much more robust (as in unlikely to break on new data), because one can conceivably generate and rate much more diverse corpuses of text than eg. camera inputs. It is tangentially likely to be true that the world state -> rich text function is much less surjective than the world state -> camera function. One can train this AI world state describer on virtually endless amounts of sensor data.

Counterexample

Nonetheless, D can still get hacked by M. If M plans to feed D a specific set of sensor inputs to force D to output a description of a good world state, then P may well have us believe that the world will be fine.

—

I find that this new framing is useful because it allows me to focus on a specific subpart of ELK (narrow elicitation of details reg...
Quick Book Review: Crucial Conversations
by Gordon Seidoh Worley — 2022-09-19T06:25:23.052Z

I just finished reading Crucial Conversations: Tools for Talking When Stakes Are High by by Kerry Patterson, Joseph Grenny, Ron McMillan, and Al Switzler. This is a quick review of the book.

I can summarize the whole book for you in roughly the space of a tweet:

Difficult conversations require safety. Establish safety by listening, empathizing, and suppressing emotional reactions that would make others feel unsafe. Once you have created safety, be honest and straightforward. Work together towards mutual benefit. Respect each person's interests.

If you don't know how to do those things, reading this book might help you, as the authors provide some techniques. The techniques they offer are shallow, though, so they're only likely to be helpful if you've never made a serious attempt at communicating or negotiating effectively. If you've tried at all I think you'll be left wondering when the book will go deeper. Warning: it never will.

Personally, reading this book was a waste of my time. I'd already gotten all the big ideas in it from other books. I think I feel pretty confident in saying that you only need to read approximately two books about having conversations and negotiating to learn all the basics (you'll need a lot of experience to get good at putting the ideas into practice, though). Here's some other books in this space you might consider for your ~two (there are many more):

How to Win Friends and Influence People by Dale Carnegie
How to Talk So Kids Will Listen & Listen So Kids Will Talk by Adele Faber and Elaine Mazlish
Games People Play: The Psychology of Human Relationships by Eric Berne

bonus "sequel" book: I'm OK – You're OK by Thomas Anthony Harris

Getting to Yes: Negotiating Agreement Without Giving In by Roger Fisher and William Ury

For myself, I think the one I found most useful among all of these was How to Talk So Kids Will Listen. My second pick is Games People Play.

If you just want the theory that powers the ideas in all these books, I think you can learn it from studying two things: game theory and transactional analysis. Game theory you probably know or are at least aware of. Transactional analysis is a fake framework in psychology; it's wrong in some important ways but extremely useful for getting a baseline understanding of why people do the things they do in social interactions. It's not the whole story but it gets you more than halfway to...
How my team at Lightcone sometimes gets stuff done
by Bird Concept — 2022-09-19T05:47:06.787Z

Disclaimer: I originally wrote this as a private doc for the Lightcone team. I then showed it to John and he said he would pay me to post it here. That sounded awfully compelling. However, I wanted to note that I’m an early founder who hasn't built anything truly great yet. I’m writing this doc because as Lightcone is growing, I have to take a stance on these questions. I need to design our org to handle more people. Still, I haven’t seen the results long-term, and who knows if this is good advice. Don’t overinterpret this.

Suppose you went up on stage in front of a company you founded, that now had grown to 100, or 1000, 10 000+ people. You were going to give a talk about your company values. You can say things like “We care about moving fast, taking responsibility, and being creative” -- but I expect these words would mostly fall flat. At the end of the day, the path the water takes down the hill is determined by the shape of the territory, not the sound the water makes as it swooshes by. To manage that many people, it seems to me you need clear, concrete instructions. What are those? What are things you could write down on a piece of paper and pass along your chain of command, such that if at the end people go ahead and just implement them, without asking what you meant, they would still preserve some chunk of what makes your org work?

Here’s my current best guess at how I would do this for Lightcone Infrastructure, the organisation where I spend the majority of my waking hours. I wrote it by thinking about how the team I'm on has actually operated during periods of high output, and then trying to turn that into a set of rules. Others on the team might disagree about which rules matter and where the magic sauce is, but I think this is at least empirically descriptive of how my team spends much of our time.

0. Blockers are death. Above all else, your job is to unblock anything that prevents you from moving as fast as you can toward your top priority.

The team has one team lead who’s the final decision-maker on all decisions, unless they explicitly delegate a decision. Consensus is slow and blocking. Having a tie-breaker means you can move faster.
Start each Monday with an all-hands meeting where the team lead sets or clarifies the top priority of the team.
After the Monday meeting, there’s a block of time on everyone’s cal

...
EA & LW Forums Weekly Summary (12 - 18 Sep '22)
by Zoe Williams — 2022-09-19T05:08:43.021Z

Supported by Rethink Priorities

This is part of a weekly series - you can see the full collection here. The first post includes some details on purpose and methodology.

If you'd like to receive these summaries via email, you can subscribe here.
Also posted on the EA forum here.

New - podcast version: prefer your summaries in podcast form? A big thanks to Coleman Snell who is now producing these! The first episode will be up on the EA Forum Podcast later this week - subscribe to hear as soon as it's up.

Top Readings / Curated

Designed for those without the time to read all the summaries. Everything here is also within the relevant sections later on so feel free to skip if you’re planning to read it all.

CEA Ops is now EV Ops

by EV Ops

CEA Ops started supporting other orgs over time, now supporting 10+. Because of this they’ve split into their own organization called EV Ops. They’re close to capacity, but fill in the

EOI form

if you’d like support (1-2 person start-ups through to large orgs). Or help expand capacity by

applying to join the team

.

[Linkpost] A survey on over 300 works about interpretability in deep networks

by scasper

Linkpost for survey of 300 works on inner interpretability, across areas including weights, neurons, subnetworks, and latent representations. A discussion section includes findings such as:

Promising paradigms outside of circuits deserve more attention.
Interpretability research has strong links to other areas such as adversarial robustness.
We need a benchmark for evaluating interpretability tools, based on their ability to help us in goals such as designing novel adversaries or discovering system flaws.
It’s important not to focus on best-case performance.

Improving "Improving Institutional Decision-Making": A brief history of IIDM

by Sophia Brown

History of IIDM - Pre-2019

The global health area was advocating improving government aid spend via lobbying.
Many critiqued EA’s seeming focus on marginal vs. systemic change. Leaders responded eg. Open Phil now funds areas like immigration reform and housing.
The rationality community was developing tools for individual decision-making. Evidence for effectiveness of forecasting in group use pushed forward the institutional angle.

In 2015, the cause area formed out of the third bullet point.

Current State IIDM - 2019 to present

The Effective Institutions Project built a broade

...
Book Swap
by Screwtape — 2022-09-19T02:33:55.478Z

Summary: Bring a book you liked, pitch it to the crowd, then trade your book for a book someone else brought that sounded good. This has no specific rationalist content.

Tags: Repeatable, medium

Purpose: Get people talking about books they read and encountering new ones.

Materials: A whiteboard or piece of paper with a clipboard, plus appropriate writing implement. A timer. (Or a smartphone with a timer app.)

Announcement Text:

Hello all!

I read a lot of books, which presents me with two problems. First, I constantly need recommendations for more books to read, and second, I constantly want to share books with other people so I'll be able to talk about the book with them. I would like to solicit your help with both of these problems. To that end, I’m hosting a book swap. The rules are simple. Bring a book. By the end of the night, leave with a book that isn't the one you brought.

Advice on choosing your book: The only two hard restrictions on your choice of book is that 1. it must be a physical book, the kind you could put into someone else's hand, and 2. it must be for an English speaking audience. Given my social groups I expect certain literary preferences will exist but, well, you're invited, and you like the books you like, so feel free to pick any book you like and it's quite plausible someone else has the same tastes as you.

We will have a whiteboard or similar writing surface. As you arrive, write your name and the title of your book on it. Then at the appointed time, we'll start running through the list- when it's your turn, you'll have two minutes by-the-clock to pitch why people should read your book. Then the swapping will begin in earnest! Please feel free to ask people questions about their books, talk about what kinds of books you like, or just hang out and socialize about anything else.

Description:

As they arrive, make sure to collect people’s names and books. Stack the books in a spot where everyone can see them, in order if possible. Flip the stack (so the first person’s book is now on top) right before pitching the books.

When it’s time to pitch books, be handy with a timer. You should be cheerfully ruthless about cutting people off once they run out of time, though two minutes tends to be enough that many people will finish with time to spare.

Notes:

This event mostly runs itself, which makes it a great pick if you...
Pretending not to Notice
by jefftk — 2022-09-19T02:30:05.079Z

Sometimes parents will pretend not to notice kids doing things they
shouldn't: why might you do this and when is it a good idea?

My experience has been that things go much better when parents set clear boundaries. If it's
sometimes ok to pick the flowers and other times you get told to stop,
there's a lot of trying to figure out what makes these particular
flowers or this particular time different from others. Some
complexity is unavoidable (you can't pick flowers from the neighbor's
garden, you can pick all the dandelions you want) but a parent who's
not thinking about it can accidentally make things harder for their
kids by being inconsistent (only sometimes stopping them from picking
flowers that shouldn't be picked, not making up their mind about which
flowers are ok).

With very little kids, what matters is what happens when they try to
pick the flower. If you don't stop them, that's some information
about what's ok; if you do, it's some information about what's not.
Kids who are a bit older are smarter though: they care about whether
you've observed them. Picking a flower without objection while being
watched is a signal that the flower is fine to pick, but picking a
flower while not being watched doesn't actually tell you very much
about what's ok. It does tell you a little: it's probably not so
hugely bad to pick flowers that a sensible adult would continue
supervising. But mostly they understand to only update their model of
the boundaries the adult is enforcing when the adult is paying
attention.

Which means, very occasionally, you might want to pretend to not
observe something. If they believed you'd seen it, you'd need to
enforce the boundary or they'd think it had changed. But by
pretending not to notice you can give them close to "no signal", as if
you weren't there at all.

For example, let's say you have been enforcing a clear "no hitting"
rule, and your kid hits an older kid, someone mature enough to stand
up for themself. With the boundary you had been enforcing you would
need to step in, but, if they both believe you didn't see, they can
get practice in resolving their dispute independently ("Don't hit
me!") or at least in bringing their concern to you ("Alex hit me!").
This could be part of a process of moving away from a toddler's
"hitting is not allowed" to a Kindergartener's "hitting is not
allowed, but try to ...
[To Be Revised]Perhaps the Meaning of Life, An Adventure in Pluralistic Morality
by NoBadCake — 2022-09-18T22:37:33.338Z

Perhaps the Meaning of Life
An Adventure in Pluralistic Morality.

Sept. 18, 2002

"All models are wrong but some are useful."
--George Box

At the core of every controversial issue, institutional dysfunction and inspiring human achievement is the conspicuous influence of how well a society of people “get along”. Splitting this atom exposes the sub-particles of human nature(~proton), moral agency(~electron) and the electromagnetic tension of choice around individual belief(~neutron). The following model is an ongoing effort to assemble and refine an understanding of this dynamic.

The first figure, entitled Moral Transitions, indicates the individual’s struggle against the less appealing aspects of human nature. The second, Relational Harmonics represents the near constant challenge of building and maintaining character. The third figure, Personal Growth indicates the not necessarily linear stages that can occur over a lifetime.

Footnotes
a

Personal Growth (intellectual, moral, spiritual)

"There are of course

many gradations within and between the four stages

of spiritual development. [emphasis added]

...

My experience suggests that this progression of spiritual development holds true in all cultures and for all religions.

...

Again in my experience, the four stages of spiritual development also represent a paradigm for healthy psychological development”

--M. Scott Peck, ﻿The Different Drum, Chapter IX, The Stages Of Spiritual Growth, 1987
[A flawed but useful book]

b

"The world judges many things; for it is the ignorance of its natural state that forms the true tribunal of man. Knowledge has two extremes which meet; the one is that pure natural ignorance, in which all are born; the other is that which is experienced by minds of the highest order. After traversing the whole circle, of human attainments, they find that they know nothing; and end in the same ignorance in which they set out. But it is the ignorance of learning only, that knows itself to be ignorant. Those who occupy an intermediate place, — who have just emerged from their natural ignorance, and yet not attained to that of learning, — possess a sort of smattering of knowledge, and are looked upon as clever. These are the people that keep the world in commotion, and blunder in everything. They, and the common people, constitute the great bulk of mankind. They d

...
Leveraging Legal Informatics to Align AI
by John Nay — 2022-09-18T20:39:50.796Z
Link: https://forum.effectivealtruism.org/posts/XmKhYQfnfqb3Z7Dkr/aligning-ai-with-humans-by-leveraging-legal-informatics

We already face significant challenges communicating our goals and values in a way that reliably directs AI behavior – even without additional technological advancements, which could compound the difficulty with more autonomous systems. Specifying the desirability (value) of an AI system taking a particular action in a particular state of the world is unwieldy beyond a very limited set of value-action-states. In fact, the purpose of machine learning is to train on a subset of world states and have the resulting agent generalize an ability to choose high value actions in new circumstances. But the program ascribing value to actions chosen during training is an inevitably incomplete encapsulation of the breadth and depth of human judgements, and the training process is a sparse exploration of states pertinent to all possible futures. Therefore, after training, AI is deployed with a coarse map of human preferred territory and will often choose actions unaligned with our preferred paths.

Law is a computational engine that converts human values into legible directives. Law Informs Code is the research agenda attempting to model that complex process, and embed it in AI. As an expression of how humans communicate their goals, and what society values, Law Informs Code.

Law Informs Code: A Legal Informatics Approach to Aligning Artificial Intelligence with Humans

, an article forthcoming in the

Northwestern Journal of Technology and Intellectual Property

, dives deeper into related work and

this upcoming research agenda

being pursued at The Stanford Center for Legal Informatics (a center operated by Stanford Law School and the Stanford Computer Science Department).

Similar to how parties to a legal contract cannot foresee every potential “if-then” contingency of their future relationship, and legislators cannot predict all the circumstances under which their proposed legislation will be applied, we cannot specify “if-then” rules that provably lead to good AI behavior. Fortunately, legal theory and practice have developed arrays of tools for goal specification and value alignment.

Take, for example, the distinction between legal rules and standards. Rules (e.g., “do not drive more than 60 miles per hour”) are more targeted directives than standards. They enable the rule-maker to have clarity over outcomes that will be ...
The Inter-Agent Facet of AI Alignment
by Michael Oesterle — 2022-09-18T20:39:31.930Z

We argue that intent alignment and capability robustness do not capture the full “chain of command” (i.e., “alignment” without any qualifier) from the programmer's intentions to an AI agent’s real-world impact, but stops at the stage where the agent performs its actions. We argue that, especially in complex Multi-Agent Systems (MAS), this is not sufficient to actually achieve aligned outcomes[1]. Therefore, we introduce another facet of AI Alignment, which deals with the connection between an agent’s intention and its actual impact, due to multi-agent interactions (as opposed to capability robustness).

Motivation

Independent of whether the first AGIs will be Tool or Agent AIs, they will most likely not live and act in an empty space. Instead, they will interact and communicate with humans and other AGIs in a variety of ways, thereby effectively creating a socio-technical system or an AI-based society.

As a consequence, the effects that an AGI will have on its environment, i.e., the world, will not necessarily coincide with the effects that the AGI plans to have, since actions in such systems commonly interfere and influence/cancel each other.

This, in turn, leads to the conclusion that it is not sufficient to build an AGI which satisfies intent alignment and capability robustness. Instead, we need one more step to be controlled: The step from an AGI's robust intention to its societal impact.

Schema

In Evan Hubinger’s alignment terminology, alignment is the result of intent alignment (which is, in turn, composed of the well-known concepts of inner and outer alignment, among others) and capability robustness. The implicit assumption is that the AGI's model is equivalent to how its actions will affect the world.

Inter-Agent Alignment

Inter-Agent Alignment is aligning the (aligned and robust) intentions of an AGI agent with the actual outcome within the multi-agent system of which the agent is a part. Since the impact of the agent’s actions depends on what everyone else in this system is doing (and not just its own actions), inter-agent alignment is a non-trivial task.

Our central argument is that multi-agent interactions add a sufficient amount of complexity to an AI agent’s action policy to justify its own type of alignment, separated from the existing alignment model. Adding inter-agent alignment to the alignment tree, we see that it complements intent alignment and c...