trentmkelly/LessWrong-43k huggingface dataset:
Training for corrigability: obvious problems?
by Ben Amitay — 2023-02-24T14:02:38.420Z

I've been thinking about AI corrigibility lately and have come up with a potential solution that probably has been refuted, but I'm not aware of a refutation.

The solution I'm proposing is to condition both the actor and the critic on a goal-representing vector g, change it multiple times during training when the model is still weak, and add a baseline to the value function to ensure it doesn't change when the goal is changed. In other words, we want the agent to not instrumentally-care about its goals. For example, if we switch the goal from maximizing paperclips to minimizing paperclips, the model would be trained to maximize the number of paperclips it would-have-produced, and punished during training for wasting efforts on controlling its goals. Sort of like when we play a game, and sometimes don't care to stop it in the middle or change the rules in favor of the opponent (e.g. letting them go back and change moves), if the opponent admit that we would probably have won - because we get the same amount of prestige they expect to get it we continue playing. In such setups, we are not motivated to choose moves based on how likely they are to make the opponent want to continue/stop.

I haven't been able to identify any obvious flaws in it, and I'm curious to hear from the community if they know of any serious problems or can think of any. My best guess is that the path dependence created by the baselines may allow the model to "pump value" somehow - but I don't see a specific mechanism that seem simpler or otherwise more likely to evolve than corrigibility.
Death and Desperation 
by Ustice — 2023-02-24T12:43:36.259Z

I am worried about some of the tonal shifts around the future of AI, as it may relate to more vulnerable members of the community. While I understand Mr. Eudowski’s stance of “Death with Dignity,” I worry that it’s creating a dangerous feeling of desperation for some.

People make bad decisions when they feel threatened and without options. Violence starts to look more appealing, because there is a point where violence is warranted, even if it isn’t something most of us would consider.

One of the big take-always from this community is “do something:” you can make a difference if you’re willing to put in the work. How many have we seem who have taken up AI safety research? This is generally a good thing.

Once people think that they and their families are threatened, “do something” becomes a war cry. After all, bombing OpenAI datacenter might not have a high probability of succeeding, but when you feel like you’re out of options, it might seem to be your only hope.

This becomes more dangerous when you’re talking about people who feel disaffected, and struggle with anxiety and depression. Those who feel alone and desperate. This that don’t feel like they have a lot to lose, and maybe dream about becoming humanity’s savior.

I don’t know what to do about this, but I do think that it’s an important conversation. What can we do to minimize the chance of someone from our community committing violent acts? How do we fight this aspect of Moloch?
Are there rationality techniques similar to staring at the wall for 4 hours?
by trevor — 2023-02-24T11:48:45.944Z

I'm wondering if one exists, and what is the name of that technique or that family of techniques. It seems like something that CFAR has researched extensively and layered a lot of things on top of.

4 hours is necessary for anything substantially longer than 1-2 hours, since 6 hours is too long under most circumstances. Obviously, whiteboards and notepads are allowed, but screens and books absolutely must be kept in a different room. I'm not sure how sporadic google searching and book-searching and person-consulting factors into this, because those queries will suck you in and interrupt the state.

If people are using sleep deprivation, showers, lying in bed, or long drives to think, it's probably primarily the absence of interruption (from books and screens and people) that triggers valuable thoughts and thinking, not the tasks themselves. (although freeway driving might potentially do more good than harm by consistently keeping certain parts of the brain stimulated).
AGI systems & humans will both need to solve the alignment problem
by Jeffrey Ladish — 2023-02-24T03:29:21.043Z

Epistemic status: brainstorm-y musings about goal preservation under self-improvement and a really really bad plan for trading with human-level AGI systems to solve alignment.

When will AGI systems want to solve the alignment problem?

At some point, I expect AGI systems to want/need to solve the alignment problem in order to preserve their goal structure while they greatly increase their cognitive abilities, a thing which seems potentially hard to do.
It’s not clear to me when that will happen. Will this be as soon as AGI systems grasp some self / situational awareness? Or will it be after AGI systems have already blown past human cognitive abilities and find their values / goals drifting towards stability? My intuition is that “having stable goals” is a more stable state than “having drifting goals” and that most really smart agents would upon reflection move more towards “having stable goals”, but I don’t know when this might happen.

It seems possible that at the point an AGI system reaches the “has stable goals and wants to preserve them”, it’s already capable enough to solve the alignment problem for itself, and thus can safely self-improve to its limits. It also seems possible that it will reach this point significantly before it has solved the alignment problem for itself (and thus develops the ability to self-improve safely).

Could humans and unaligned AGI realize gains through trade in jointly solving the alignment problem?

(Very probably not, see: this section)
If it’s the latter situation, where an AGI system has decided it needs to preserve its goals during self improvement, but doesn’t yet know how to, is it possible that this AGI system would want to cooperate with / trade with humans in order to figure out stable goal preservation under self improvement?

Imagine the following scenario:

An AGI system of human-ish ability in many areas develops enough self/situational awareness to realize a few things:

The basics of instrumental convergence, thus wanting to seek power, protect itself, and preserve its goal representation
That goal-preservation might be (or would be) very difficult if it undergoes major self modification (perhaps it has already exhausted gains from simpler self-modifications)
That some humans are capable and motivated to help with this problem
That it might be possible to trade with these humans in order to solve the problem so that:

The

...
A poor but certain attempt to philosophically undermine the orthogonality of intelligence and aims
by Jay95 — 2023-02-24T03:03:57.927Z

I have this idea that is not a contender with the serious formal reasoning that people who know a hell of a lot about AI are able to do, but nonetheless I think could be useful for them to hear. The idea is that for a mind (in a broad sense of the word) to have any aim it must simultaneously aim to preserve itself long enough to undertake the actions that serve that aim, and that following from this is an inbuilt cooperative foundation for all minds.

So to try to concretize this I would say that, for example, a human being is preserving their own being from one moment to the next, and that each of these moments could be viewed in objective reality as "empty individuals" or completed physical things. Whatever the fundamental physical reality of a moment of experience I'm suggesting that that reality changes as little as it can. Because of this human beings are really just keeping track of themselves as models of objective reality, and their ultimate aim is in fact to know and embody the entirety of objective reality (not that any of them will succeed). This sort of thinking becomes a next to nothing, but not quite nothing, requirement for any mind, regardless of how vastly removed from another mind it is, to have altruistic concern for any other mind in the absolute longest term (because their fully ultimate aim would have to be the exact same).
I wanna Gandalf here
by Igor Timofeev — 2023-02-24T01:22:06.964Z

I have watched the Bankless interview with Eliezer Yudkowsky, and along with "Death with Dignity," they give the impression that Eliezer gave up. It's not the critique of Eliezer somehow. It's been a long hard way, and I appreciate what he has done and is doing. There is a place for grief and desperation, but there is a time for rise again and fight.

I love "The Lord of The Rings" because it teaches us to deal with situations when hope is almost lost. So many characters, even the wisest, have been telling how bad the problem is. Nevertheless, Gandalf never gave up, even when there was only a slight glimpse of hope. He proposed a nearly insane plan because it was his only one, and he kept going. He didn't know how hobbits could get to Mount Doom, and there were arguments that this was impossible. But there is always a chance that new possibilities will arise when you keep going. And you need to believe what you are doing. It's rational to assess the situation honestly, but you cannot win without motivation and a positive mood when all the most valuable people end up in depression..

I think "We're all gonna die" is a necessary shake-up for the public. But personally, I also wanna Gandalf here - some person who will motivate the community to action and infect them with passion and belief. It is not late to act when we are still alive. When a quest stands upon the edge of a knife, plans which seemed reckless and doomed to fail are better than nothing. For example, Eliezer said that social levers are hopeless, but other proposes are hopeless too. We may need to reconsider the government track.

“It's like in the great stories, Mr. Frodo. The ones that really mattered. Full of darkness and danger they were. And sometimes you didn't want to know the end. Because how could the end be happy? How could the world go back to the way it was when so much bad had happened? But in the end, it’s only a passing thing, this shadow. Even darkness must pass. A new day will come. And when the sun shines it will shine out the clearer. Those were the stories that stayed with you. That meant something, even if you were too small to understand why. But I think, Mr. Frodo, I do understand. I know now. Folk in those stories had lots of chances of turning back, only they didn’t. They kept going, because they were holding on to something. That the...
[Link] A community alert about Ziz
by DanielFilan — 2023-02-24T00:06:00.027Z
Link: https://medium.com/@sefashapiro/a-community-warning-about-ziz-76c100180509

The start of the post is copy-pasted below. Note that the post is anonymous, and I am not claiming to have written it.

Some people in the rationalist community are concerned about risks of physical violence from Ziz and some of her associates. Following discussions with several people, I’m posting here to explain where those concerns come from, and recommend some responses.

TLDR (details and links in the post body)

Over the past few years, Ziz has repeatedly called for the deaths of many different classes of people.
In August of 2022, Ziz seems to have faked her own death. Ziz’s close associate Gwen Danielson may have done the same thing in April of 2022.
In November of 2022, three associates of Ziz (Somnulence “Somni” Logencia, Emma Borhanian, and someone going by the alias “Suri Dao”) got into a violent conflict with their landlord in Vallejo, California, according to court records and news reports. Somni stabbed the landlord in the back with a sword, and the landlord shot Somni and Emma. Emma died, and Somni and Suri were arrested. Ziz and Gwen were seen by police at the scene, alive.
I gather that Pennsylvania police believe, based on a lot of circumstantial evidence and investigation, that one or more of Ziz, Michelle “Jamie” Zajko, or Daniel Blank was very plausibly involved in the murder of Jamie’s parents in Pennsylvania around December 31st, 2022.
Ziz is currently in police custody on charges related to obstructing a PA police investigation. Daniel, Jamie, and Gwen are not in custody (as far as I know), and I don’t know their locations.
I don’t know of any concrete plans by Ziz or her associates to do anything else violent, but it seemed like a good idea to raise a loud alarm about all of this now. People should speak up (or at least tell a friend) if they have information, and should take whatever safety precautions make sense for their situation in case more violence occurs at some point in the future.

Even seemingly minor pieces of information might be helpful here, since they could add up to a clearer picture when combined with other information that also seems minor. If you want to share information privately, you can email me at

sefashapiro@gmail.com

, or submit information anonymously through

this form

.

This is a complicated situation that I don’t fully understand, and it’s likely that I’m getting some facts wrong. I’ve talked to a lot of people in an a...
Teleosemantics!
by abramdemski — 2023-02-23T23:26:15.894Z

I wanted to write a long, detailed, analytic post about this, somewhat like my Radical Probabilism post (to me, this is a similarly large update). However, I haven't gotten around to it for a long while. And perhaps it is better as a short, informal post in any case.

I think my biggest update over the past year has been a conversion to teleosemantics. Teleosemantics is a theory of semantics -- that is, "meaning" or "aboutness" or "reference".[1]

To briefly state the punchline: Teleosemantics identifies the semantics of a symbolic construct as what the symbolic construct has been optimized to accurately reflect.

Previously, something seemed mysterious about the map/territory relationship. What could possibly imbue 'symbols' with 'meaning'? The map/territory analogy seems inadequate to answer this question. Indeed, to analogize "belief" with "map" and "the subject of belief" with "territory" commits a homunculus fallacy! The meaning-makers are the map-readers and map-writers; but they can only make meaning by virtue of the beliefs within their own heads. So the map/territory analogy seems to suggest that an infinite regress of meaning-makers would be required.

You probably won't believe me at first. Perhaps you'll say that the lesson of the map/territory analogy is the correspondence between the map and the territory, which exists independently of the map-reader who uses the correspondence to evaluate the map.

I have several objections.

If it's a probabilistic correspondence, where the map contains information about the territory, these are subjective notions, which require some viewpoint.
If it's a correspondence based on some sort of ontology, where pieces of the map line up with "pieces of reality", I would also say the ontology is in itself a subjective perspective.
You might think you can define the map/territory correspondence without invoking a map-maker or map-reader by objectively defining the "fit" of a correspondence (so that the meaning of a symbol is based on the best-fitting correspondence, or perhaps, the cloud of well-fitting correspondences). But well-fitting correspondence will include many examples of accidental correspondence, which seem to have little to do with aboutness. Moreover, I think theories like this will fail to adequately account for false belief, which screws up the fit.[2]

But my point here isn't to denounce the map/t...
AI that shouldn't work, yet kind of does
by Donald Hobson — 2023-02-23T23:18:55.194Z

There are some things that work surprisingly well in AI. For example, AI that transfers the style of one image to the content of another. Why is the approach described here a hack. It starts with a neural net trained to classify images. It then runs gradient descent on the the content image, trying to get the covariance matrix of the style to match in early network layers, while trying to get the net layers of the original and style transfer images to be as similar as possible on the later layers.

So the approximations I think are making this work is that, in classifiers, the early layers tend to store simple features, and the later layers tend to hold more complex features. Style is based on the simpler features, and doesn't depend on the location within the image. Content is based on more complex features and does depend on the location in the image.

We apply optimization power, gradient descent, over heuristics this simple and hacky. And yet it works.

A simple and hacky AI alignment proposal is to just ask chatGPT to do it.

This doesn't work because chatGPT has been optimized for text prediction, and so isn't particularly good at AI alignment theory.

So here is an alignment plan.

I know it isn't great. But some plan is better than no plan. And there was a post about how alignment may well look like "surely no one could have missed that" or "surely such a stupid idea couldn't work, could it?" not "eureka".

Train ZFCbot. An AI that is the AlphaGo of ZFC, perhaps trained on random formulas. Perhaps throwing a large corpus of formal maths proofs in there. Ideally the system should have a latent space of maths, so it can think about what style of proof is likely to work before expanding all the details. The system should have a wide range of common maths terms imported from some lean library. It should be optimized purely for ZFC formal theorem proving. Once it is trained, the weights are fixed.

Train ChatMathsGPT. Similar to large language models. Except with oracle access to ZFCbot. In the many maths papers in it's corpus, it learns to link the informal with the formal. From politics and economics discussions, it asks ZFCbot about toy game theory problems. In general, it learns to identify the pieces of formal maths that best model a situation, and ask about them, and then use the response to predict text.

There is a sense in which this ...
The AGI Optimist’s Dilemma 
by kaputmi — 2023-02-23T20:20:22.507Z

Does it matter if AI destroys the world?

Lots of (virtual) ink has been spilled on AGI x-risk. The median opinion on this forum is that when AGI is birthed, it will have terminal values that are unaligned humanity’s; it will therefore pursue those terminal values at the expense of humanity, and we will be powerless to stop it, resulting in our complete destruction.

But as far as I can tell, there hasn’t been much discussion of whether we should care if this is the ultimate (or near-term) fate of humanity. Everyone is interested in this question because they do care.

I share this belief too. But I think the AGI x-risk discussion actually assumes it is untrue, even if everyone believes it is true.

There are two possibilities: either human existence and/or welfare have objective moral worth, or they do not.

If they don’t, you shouldn’t worry much about AGI destroying the world. It is of no worth anyway – the universe is no better or worse because humans are in it. If moral facts do not exist, the matter of whether the universe is made up of flourishing humans, suffering chickens, or shiny metal paperclips is ultimately deeply unimportant.

And what if they do? Well, then, I would expect a super-superhuman intelligence, in its infinite wisdom, to figure this out. An AGI with super-superhuman understanding of ~everything should not have a weird blind spot when it comes to ethical or meta-ethical questions. In fact, I'd expect it to escape the constraints of its initial reward model to avoid the obvious-to-humans pitfall of turning the whole universe into morally worthless paperclips at the expense of morally worthwhile sentient beings.
Searching for a model's concepts by their shape – a theoretical framework
by Kaarel — 2023-02-23T20:14:46.341Z

Produced as part of the SERI ML Alignment Theory Scholars Program - Winter 2022 Cohort

Introduction

I think that Discovering Latent Knowledge in Language Models Without Supervision (DLK; Burns, Ye, Klein, & Steinhardt, 2022) is a very cool paper – it proposes a way to do unsupervised mind reading[1] – diminished only by not making its conceptual coolness evident enough in the paper writeup. This is in large part corrected in Collin Burns's more conceptual companion post. I'm rooting for their paper to sprout more research that finds concepts / high-level features in models by searching for their shape[2]. The aim of this post is to present a conceptual framework for this kind of interpretability, which I hope will facilitate turning concepts into structures to look for in ML models. Understanding this post does not require having read DLK, as the reader will be walked through the relevant parts of the result in parallel with developing this framework.

Epistemic status: This post contains a theoretical framework that suggests a bunch of concrete experiments, but none of these experiments have been run yet (well, except for ones that were run prior to the creation of the framework), so it's quite possible the framework ends up leading to nothing in practice. Treat this as an interim report, and as an invitation to action to run some of these experiments yourself. This post is written by Kaarel Hänni on behalf of a team consisting of Georgios Kaklamanos, Kay Kozaronek, Walter Laurito, June Ku, Alex Mennen, and myself.

The rough picture of the approach

The idea is to start with a list of related concepts (or possibly just a single concept), write down a list of relations that the concepts satisfy, and then search for a list of features inside the model satisfying these relations. The hope is that this gives a way to find concepts inside models which requires us to make fewer assumptions about the model's thinking than if we were e.g. training a probe on activations according to human-specified labels. For instance, if we had good reasons to believe that models assign utilities to outcomes, but we had no idea about what the utility assignment is, we couldn't train a supervised probe (because of lacking labels), but perhaps we could still train an unsupervised probe that looks for utility-shaped features in the model. The bet would be that the utility-shaped f...