trentmkelly/LessWrong-43k huggingface dataset:
The curious case of Pretty Good human inner/outer alignment
by PavleMiha — 2022-07-05T19:04:49.434Z

I have

been convinced

to

believe

that looking at the gap between human inner and outer alignment is a good way to think about potential inner/outer alignment problems in artificial general intelligences:

We have an optimisation process (evolution) trying to propagate genes, that created a general intelligence (me/you). For millions of years our inner goals of feeling really good would also satisfy evolution’s outer goal of propagating genes, because one of the things that feels the best is having sex. But eventually that intelligent agent figured out how to optimise for things that the outer optimisation process didn’t want, such as having protected sex or watching VR porn, thus satisfying the inner goal of feeling really good, but not the outer goal of propagating genes. This is often told as a cautionary tale: we only know of one General Intelligence and it’s misaligned. One day we will create an Artificial General Intelligence (AGI) and we will give it some sort of (outer) goal, and it might then develop an inner goal that doesn’t directly match what we intended. I think this only tells half the story.

Even though our general intelligence has allowed us to invent condoms and have sex without the added cost of children, a surprising amount of people decide to take them off because they find it fun and meaningful to have children.

In a world where we could choose to spend all our time having protected sex or doing drugs, a lot of us choose to have a reasonable number of kids and spend our time on online forums discussing AI safety, all of which seem to satisfy a more longer term version of “propagate your genes” than simply wanting to have sex because it feels good. More than that, we often choose to be nice in situations where being nice is even detrimental to the propagation of our own genes. People adopt kids, try to prevent wars, work on wildlife conservation, spend money on charity buying malaria nets across the world, and

more

.

I think there are two questions here: why are human goals so sophisticatedly aligned with propagating our genes and why are humans so nice.

Most people

want to have kids, not just have sex. They want to go through the costly and painful process of childbirth and child rearing, to the point where many will even do IVF. We’ve used all of our general intelligence to bypass all the feels-nice-in-the-moment bits ...
When is it appropriate to use statistical models and probabilities for decision making ?
by Younes Kamel — 2022-07-05T12:34:16.652Z
Link: https://youneskamel.substack.com/p/when-is-it-appropriate-to-use-statistical

I enjoy reading rationalist and effective altruist blogs. Members of the community usually back their arguments with data and/or other evidence and tend to be scientifically literate. And although no one is a perfect, detached belief updater, I found that rationalists are probably the community that is closest to that ideal. However, I believe this community often commits a fallacy while attempting to think rationally. It is the fallacy of applying cost benefit analysis under deep uncertainty. What is deep uncertainty ? I found a great definition in the textbook Decision making under deep uncertainty by Marchau et al., so I’m simply going to quote it :

Complete certainty is the situation in which we know everything precisely. This is almost never attainable, but acts as a limiting characteristic at one end of the spectrum.

Level 1 uncertainty represents situations in which one admits that one is not absolutely certain, but one does not see the need for, or is not able, to measure the degree of uncertainty in any explicit way (Hillier and Lieberman 2001, p. 43). These are generally situations involving short-term decisions, in which the system of interest is well defined and it is reasonable to assume that historical data can be used as predictors of the future. Level 1 uncertainty, if acknowledged at all, is generally treated through a simple sensitivity analysis of model parameters, where the impacts of small perturbations of model input parameters on the outcomes of a model are assessed. Several services in our life are predictable, based on the past such as mail delivery and garbage collection. These are examples of this level of uncertainty.

In the case of Level 2 uncertainties, it is assumed that the system model or its inputs can be described probabilistically, or that there are a few alternative futures that can be predicted well enough (and to which probabilities can be assigned). The system model includes parameters describing the stochastic—or probabilistic—properties of the underlying system. In this case, the model can be used to estimate the probability distributions of the outcomes of interest for these futures. A preferred policy can be chosen based on the outcomes and the associated probabilities of the futures (i.e., based on “expected outcomes” and levels of acceptable risk). The tools of probability and statistics can be used to solve problem

...
Goal Factoring
by CFAR!Duncan — 2022-07-05T07:10:04.930Z

Epistemic status: Anecdotally strong

The goal factoring technique as taught by CFAR is not derived from any particular body of psych research, but is instead a straightforward and general application of the principle of reductionism. It was developed and refined through iteration, and has been useful to large numbers of alumni.

Imagine that you are sitting at home at the end of a long, hard week, preparing to make plans for the evening. One of your friends is throwing a party, and has urged you to come. At the same time, there are delicious leftovers in the fridge and a movie you’ve been hoping to watch, and you’re feeling pretty lethargic.

When making this sort of decision, most people do some form of weighing— whether explicitly, with System 2, or viscerally/intuitively, with System 1— comparing the pros and cons of each option and selecting the one with the highest net “goodness.” You may consider things like who is likely to be at the party, or whether your friend would be offended if you didn’t show up; you may do some internal measuring of your energy levels, to see if you’re in dire need of some rest and relaxation. The decision might come from balancing a bunch of little things, or be based on one crucial factor.

Ultimately, though, most people end up picking one or the other—we either go out, or stay in. Occasionally, we might come up with a sort of com- promise option—such as going to the party for half an hour and then coming back home—but we rarely reach outside of the A, B, or A&B framework.

The goal factoring technique asks that we approach these sorts of problems a little differently. Instead of simply comparing one choice to another, goal factoring encourages us to adopt a “third path” mentality—to assume, for the sake of argument, that there might be a way to get everything we want, and achieve all of the good with none of the bad.

The Parable of the Orange

Sometimes, of course, there is no way to get everything. Sometimes, we really are constrained, and have to make tradeoffs and compromises. But we tend to feel constrained more often than we really are, thanks to social imperatives and longstanding habits and assumed entanglements between various obligations. Often, there’s a lot of wiggle room that we aren’t aware of, especially if it’s been a while since we stepped back and took a fresh look from a broader perspective.

In the par...
Assorted thoughts about abstraction 
by Adam Zerner — 2022-07-05T06:40:14.673Z

Epistemic status: Exploratory + I'm not an expert here. Still, I'm reasonably confident that I'm onto something.

Let's talk about levels of abstraction. I have an example that I think would be helpful.

It comes from the first computer science lecture that I ever attended. I think the year was 2010. I was taking this class that was a precursor to CS 101. CS 101 was the normal intro class, but it assumed you had some amount of experience with programming, whereas the class that I took assumed nothing.

Anyway, the professor asked us a question. He asked us how we would write out instructions for someone to brush their teeth. An answer would look something like this:

Put toothpaste on toothbrush

Brush teeth

Rinse mouth

Clean toothbrush

Then he would say how "put toothpaste on toothbrush" isn't very specific. I mean, it is to a sufficiently smart human. But what if the person isn't that smart? You'd have to be more specific about what that instruction actually means. Maybe the more specific version looks something like this:

Take toothbrush out of drawer

Take toothpaste out of drawer

Rinse toothbrush

Put toothpaste on toothbrush

But then imagine that the person you are giving these instructions to is really dumb. They don't know how to execute the instruction of "take toothbrush out of drawer". You have to be even more specific. Ugh, what a schlep. But let's try it.

Open drawer

Pick toothbrush up

Put toothbrush down on counter

Close drawer

Perhaps you can see where this is going. Even these instructions can be made more specific. What does it mean to open a drawer? To pick a toothbrush up? You can dig deeper and break them down into even more specific instructions.

This is sorta how computers work. They are dumb and need really, really, really specific instructions. You have to break it down further and further and further for them until they finally get it.

But — and this is a crucial point — it is hard for us humans to think in terms of these super specific instructions. Imagine that you had to explain how to brush your teeth to someone in terms of instructions like "extend your elbow until it is at a 130 degree angle". I don't think I'd be able to do it. I'd get lost in the weeds. I'd lose the forest for the trees.

This right here is why god gave us abstraction. Abstraction allows us to solve this problem. To bridge that gap. We can think in terms of high leve...
[AN #172] Sorry for the long hiatus!
by Rohin Shah — 2022-07-05T06:20:03.943Z

Listen to this newsletter on The Alignment Newsletter Podcast.

Alignment Newsletter is a publication with recent content relevant to AI alignment. Find all Alignment Newsletter resources here. In particular, you can look through this spreadsheet of all summaries that have ever been in the newsletter.

Please note that this newsletter represents my personal views and not those of DeepMind.

Sorry for the long hiatus! I was really busy over the past few months and just didn't find time to write this newsletter. (Realistically, I was also a bit tired of writing it and so lacked motivation.) I'm intending to go back to writing it now, though I don't think I can realistically commit to publishing weekly; we'll see how often I end up publishing. For now, have a list of all the things I should have advertised to you whose deadlines haven't already passed.

NEWS

Survey on AI alignment resources (Anonymous) (summarized by Rohin): This survey is being run by an outside collaborator in partnership with the Centre for Effective Altruism (CEA). They ask that you fill it out to help field builders find out which resources you have found most useful for learning about and/or keeping track of the AI alignment field. Results will help inform which resources to promote in the future, and what type of resources we should make more of.

Announcing the Inverse Scaling Prize ($250k Prize Pool) (Ethan Perez et al) (summarized by Rohin): This prize with a $250k prize pool asks participants to find new examples of tasks where pretrained language models exhibit inverse scaling: that is, models get worse at the task as they are scaled up. Notably, you do not need to know how to program to participate: a submission consists solely of a dataset giving at least 300 examples of the task.

Inverse scaling is particularly relevant to AI alignment, for two main reasons. First, it directly helps understand how the language modeling objective ("predict the next word") is outer misaligned, as we are finding tasks where models that do better according to the language modeling objective do worse on the task of interest. Second, the experience from examining inverse scaling tasks could lead to general observations about how best to detect misalignment.

$500 bounty for alignment contest ideas (Akash) (summarized by Rohin): The auth...
Outline: The Rectifying of Maps
by hamnox — 2022-07-05T05:14:50.036Z

Epistemic status: a hunch that having something like this matters a lot, and a basic outline for people to play with

In my culture, we take updating when we notice an error in our thinking very seriously. We have a cultural toolkit for it. In those moments, we want the comfort of a familiar social script to help us navigate our uncertainty and, importantly, NOT LOSE the update because it could not gain traction in a belief network where incorrect ideas had already entrenched and reinforced themselves. Truly novel thoughts can be so fleeting...

There's a personal update version and a group update version. They differ from each other in smooth degrees more than distinct practice. I think a group update about the rationalist community's relationship to the (very flawed) Sequences NEEDS to happen. I believe it's needed to happen for a while. But, I don't think I'm prepared to facilitate something like that. I need something closer to the personal version first, for myself.

Right now I have a basic outline. Making an announcement for the need of a Rectifying of Maps is the first step on it.

Name your error to another person, and have them explain it back to you, until two-sided clarity is reached. what did it cause you to mispredict, or how did it fail to constrain your expectations.
Take a space of 7 deep breaths before, between, and after engagements. Humans need time to form words. Humans need calm to process them.
Establish a signal to use when you feel satisfied and ready to move on. If there are multiple people involved make sure it is a state you don't mind remaining in for some minutes. Examples: cross fists fingers down in your lap, or offer hands to hold with the person next to you when they are also done. When everyone has signaled completion, say "it is heard" as a call and response.
Establish container rules. You may speak of your experience, but not of other people's.
"We begin with the connections pouring into this crux." Make a comfortable silence for contemplation, which you may break momentarily to share thoughts as you feel moved to. Describe the problem(s), hold off on proposing solutions.
"Where could those connections lead instead?". Toss ideas into the ring. No commitments may be made at this time, only possibilities to consider at leisure.
Next, "We continue with connections following from this crux." Another comfortable silence with momentary

...
Seeking opinions on the current and forward state of cryptocurrencies.
by jmh — 2022-07-05T05:01:39.890Z

Well, mostly asking for responses to a very general question(s) I have regarding blockchain.

I don't have any direct cryptocurrency exposure but do have some blockchain related investments. My view there is that I'm not sure about the currency related aspects of blockchain technology for the future but do think that blockchain itself has a lot of potential value and use going forward. So my thinking there is that the infrastructure type investments will have greater probability of having good returns (but unlikely to have the type of returns seen in cryptocurrencies or will be seen if best case scenarios for the currencies play out).

I'm not sure that view is a good one to hold though. Knowing that many here are a lot more expertise in this space I'm wondering if the view I hold here is naive or just wrong. Does blockchain really live or die on the payments sword?

A related question I'm also having is current state versus future. I'm largely thinking what we're seeing is something similar to what we got in the .com bubble/bust in 2000. In that view we're really just seeing some shakeups and a process of weeding out the weaker implementations and application but a stronger blockchain industry should be expected in the near futures (5 - 10 year frame or even shorter). Does that view seem a reasonable one or is that a really bad historical example to inform on the current world?

Note: Adding a request to the original. For those who disagree it would be most helpful if you at least noted what you are disagreeing with. Just seeing a disagree click (or multiple) provides very little allowing my updating my own reading of the comment. Thanks.
ITT-passing and civility are good; "charity" is bad; steelmanning is niche
by Rob Bensinger — 2022-07-05T00:15:36.308Z

This post has been recorded as part of the LessWrong Curated Podcast, and can be listened to on Spotify, Apple Podcasts, and Libsyn.

I often object to claims like "charity/steelmanning is an argumentative virtue". This post collects a few things I and others have said on this topic over the last few years.

My current view is:

Steelmanning ("the art of addressing the best form of the other person’s argument, even if it’s not the one they presented") is a useful niche skill, but I don't think it should be a standard thing you bring out in most arguments, even if it's an argument with someone you strongly disagree with.
Instead, arguments should mostly be organized around things like:

Object-level learning and truth-seeking, with the conversation as a convenient excuse to improve your own model of something you're curious about.
Trying to pass each other's Ideological Turing Test (ITT), or some generalization thereof. The ability to pass ITTs is the ability "to state opposing views as clearly and persuasively as their proponents".

The version of "ITT" I care about is one where you understand the substance of someone's view well enough to be able to correctly describe their beliefs and reasoning; I don't care about whether you can imitate their speech patterns, jargon, etc.

Trying to identify and resolve cruxes: things that would make one or the other of you (or both) change your mind about the topic under discussion.

Argumentative charity is a complete mess of a concept⁠—people use it to mean a wide variety of things, and many of those things are actively bad, or liable to cause severe epistemic distortion and miscommunication.
Some version of civility and/or friendliness and/or a spirit of camaraderie and goodwill seems like a useful ingredient in many discussions. I'm not sure how best to achieve this in ways that are emotionally honest ("pretending to be cheerful and warm when you don't feel that way" sounds like the wrong move to me), or how to achieve this without steering away from candor, openness, "realness", etc.

I've said that I think people should be "nicer and also ruder". And:

The sweet spot for EA PR is something like: 'friendly, nuanced, patient, and totally unapologetic about being a fire hose of inflammatory hot takes'. 🙂

I have an intuition that those are pieces of the puzzle, along with (certain aspects or interpretations of) NVC tech, circling tech, int...
Please help us communicate AI xrisk. It could save the world.
by otto.barten — 2022-07-04T21:47:45.796Z

As Eliezer Yudkowsky thinks, I am afraid that AGI will by default end humanity. I also agree that there is only a point in making an aligned AGI if that AGI is powerful enough to block all following AGIs (the pivotal act). If not, an unaligned AGI is sooner or later going to emerge, and that AGI is likely to win over aligned AGIs because it's more efficient. Evolutionary pressure works against us. I also agree that the chance we succeed in doing this, is small.

However, unlike Yudkowsky, I think there might be a non-vanishing chance of doing what he calls the pivotal act, and I would call AGI regulation, before the first AGI, instead of after. This regulation should be decided democratically and be robust, while doing as little economic damage as possible (this might still be a significant amount). The public will likely accept this regulation if it wholeheartedly believes that the alternative might be human extinction. A deal between the US and China could be sufficient. This deal would be achievable, as long as elites and the public in these countries believe that the alternative is likely to be extinction. Note as well that the result would likely be the same as after Yudkowsky's pivotal act, since an AGI might not be able to do anything more than only this act (since aligning just this act is already difficult enough).

The big problem here is not coordination, but communication. For this route to work, we need elites and regular people, at least roughly 50%, to wholeheartedly believe that AI could cause human extinction. Convincing people about this is hard. There are a few reasons for that. First, science fiction has made fun of it for decades, meaning there is a significant social penalty for anyone bringing up the idea in a serious context. Few things are more effective at making someone shut up than the fear of being ridiculous, and therefore information doesn't spread nearly as fast as it otherwise would. Second, there is no scientific consensus. The majority of AI academics (although not the AI safety ones) dismisses the idea (the scifi effect, conflicts of interest, and undue conservatism likely all play a role). And third, there are the biases that have been described by Yudkowsky and others before.

Basically all forms of AI xrisk communication have been tried, including newspaper articles, books, scary youtube videos, TED talks, and documentaries. ...
Benchmark for successful concept extrapolation/avoiding goal misgeneralization
by Stuart_Armstrong — 2022-07-04T20:48:14.703Z

If an AI has been trained on data about adults, what should it do when it encounters a child? When an AI encounters a situation that its training data hasn’t covered, there’s a risk that it will incorrectly generalize from what it has been trained for and do the wrong thing. Aligned AI has released a new benchmark designed to measure how well image-classifying algorithms avoid goal misgeneralization. This post explains what goal misgeneralization is, what the new benchmark measures, and how the benchmark is related to goal misgeneralization and concept extrapolation.

The problem of goal misgeneralization

Imagine a powerful AI that interacts with the world and makes decisions affecting the wellbeing and prosperity of many humans. It has been trained to achieve a certain goal, and it’s now operating without supervision. Perhaps (for example) its job is to prescribe medicine, or to rescue humans from disasters.

Now imagine that this AI finds itself in a situation where (based on its training) it’s unclear what it should do next. For example, perhaps the AI was trained to interact with human adults, but it’s now encountered a baby. At this point, we want it to become wary of goal misgeneralization. It needs to realise that its training data may be insufficient to specify the goal in the current situation.

So we want it to reinterpret its goal in light of the ambiguous data (a form of continual learning), and, if there are multiple contradictory goals compatible with the new data, it should spontaneously and efficiently ask a human for clarification (a form of active learning).

That lofty objective is still some way away; but here we present a benchmark for a simplified version of it. Instead of an agent with a general goal, this is an image classifier, and the ambiguous data consists of ambiguous images. And instead of full continuous learning, we retrain the algorithm, once, on the whole collection of (unlabeled) data it has received. And then it need only ask to once about the correct labels, to distinguish the two classifications it has generated.

Simpler example: emotion classifier or text classifier?

Imagine an image-classifying algorithm . It’s trained to distinguish photos of smiling people (with the word "HAPPY" conveniently written across them) from photos of non-smiling people (with the word "SAD" conveniently written across them):

Then, on deployment, i...