trentmkelly/LessWrong-43k huggingface dataset:
In Defense of Wrapper-Minds
by Thane Ruthenis — 2022-12-28T18:28:25.868Z

Recently, there's been a strong push against "wrapper-minds" as a framework. It's argued that there's no specific reason to think that all sufficiently advanced agents would format their goals in terms of expected-utility maximization over future trajectories, and that this view predicts severe problems with e. g. Goodharting that just wouldn't show up in reality.[1]

I think these arguments have merit, and the Shard Theory's model definitely seems to correspond to a real stage in agents' value formation.

But I'd like to offer a fairly prosaic argument in favor of wrapper-minds.

Suppose that we have some agent which is being updated by some greedy optimization process (the SGD, evolution, etc.). On average, updates tend to decrease the magnitude of every subsequent update — with each update, the agent requires less and less correction.

We can say that this process optimizes the agent for good performance according to some reward function R, or that it chisels "effective cognition" into that agent according to some rule.

The wrapper-mind argument states that any "sufficiently strong" agent found by this process would:

Have an explicit representation of R inside itself, which it would explicitly pursue.
Pursue only R, at the expense of everything else in the universe.

I'll defend them separately.

Point 1. It's true that explicit R-optimization is suboptimal for many contexts. Consequentialism is slow, and shallow environment-optimized heuristics often perform just as well while being much faster. Other environments can be just "solved" — an arithmetic calculator doesn't need to be a psychotic universe-eater to do its job correctly. And for more complex environments, we can have shard economies, whose collective goals, taken in sum, would be a strong proxy of R.

But suppose that the agent's training environment is very complex and very diverse indeed. Or, equivalently, that it sometimes jumps between many very different and complex environments, and sometimes ends up in entirely novel, never-before-seen situations. We would still want it to do well at R in all such cases[2]. How can we do so?

Just "solving" environments, as with arithmetic, may be impossible or computationally intractable. Systems of heuristics or shard economies also wouldn't be up to the task — whatever proxy goal they're optimizing, there'd be at least one e...
What is the best way to approach Expected Value calculations when payoffs are highly skewed?
by jmh — 2022-12-28T14:42:51.169Z

The other day I was musing about a reasonable approach to playing games like the big lotteries. They don't cost a lot and losing $40 is not a life changing event for me, but clearly winning a few hundred million dollars is life changing.

My first thought turned to, well if you just play when the expected value is greater than the cost of the ticket that is "rational". But when I started thinking about it, and even doing some calculations for when that EV condition exists (for things like Mega Millions the jackpot has to be greater then about 550 million) it struck me that the naive EV calculation must be missing something. The odds of actually winning the jackpot are really, really low (as opposed to just really low to rather low for the other prizes). And the payoffs that go into the EV calculation are hugely skewed by the top prices.

I suspect this must be a situation that generalized to other settings and am wondering if anyone knows of better approaches than merely the naive EV calculation. And to be sure I'm using the term as everyone expects, EV just equals the probability weighted payoffs minus the cost of the ticket.
Bandwagon effect: Bias in Evaluating AGI X-Risks

by Remmelt — 2022-12-28T07:54:50.669Z

Note on unusual formatting: Sentences are split into lines so you can parse parts precisely.
The tendency to do (or believe) things
because many other people do (or believe) the same.

While this is similar in effect
to the 'Availability Cascade',
this bias effect relates
more specifically to the relation
between 'expert' and 'non-expert' opinions,
whereas Availability Cascade is more in relation
to the arguments and discussions among experts
who are knowledgeable and qualified
enough to consider and assess
the information directly.

The concern here is that
once the general non-expert public
has been drawn into accepting
a given proposal, proposition, or belief,
the apparent boundary between

the opinions resulting from
real evaluations; and;
the opinions resulting from
people simply quoting other people,

... becomes very blurred.

This means that it is
no longer possible for any party,
regardless of all other factors,
to easily tell if the analysis and evaluation
has been independently replicated/validated,
or is merely being quoted, copied
from one person to another.
The net effect is that it becomes
increasingly difficult to determine if
the actual strength of the evaluation itself
is due to multiple concordant validations,
or that validation strength is actually absent.

If most people are simply copying
their results from someone else,
that means we are all equally likely
to have the same incorrect
or incomplete answer.
This can easily lead
to a false sense of security:
everyone believes we are safe
because everyone else believes we are safe
– a result that could easily become
completely detached from any
sort of objective real basis
or evaluative grounding.

- link Wikipedia: Bandwagon effect
- an item on Forrest Landry's compiled list of biases in evaluating extinction risks.
Getting up to Speed on the Speed Prior in 2022
by robertzk — 2022-12-28T07:49:22.948Z

This post was written under the mentorship of Evan Hubinger, as a part of the Stanford Existential Risks Institute ML Alignment Theory Scholars (MATS) program.

Thanks to Evan Hubinger for his mentorship under the SERI MATS program, and to Arun Jose, Rob Ghilduta and Martín Soto for providing prior references and some feedback. Also thanks to James Lucassen for reviewing a draft with me and providing extensive feedback.

Epistemic status: Somewhat uncertain. A lot of the arguments related to usage of the speed prior employ high-level reasoning and lack substantial formalizable proofs. Acceptance of any of the arguments in this post should be conditional upon comfort that they can be reduced to very precise provable statements about properties produced by inductive biases subject to a speed prior as in the described proposals and implementations. It is also not clear how directly mappable these notions of speed priors are to NNs relative to their treatment in generic models of computation (like TMs or circuits).

Introduction

This post examines the current literature on the speed prior before later diving into some new ideas on how to forward the speed prior in instances where mesa-optimization can occur.

The first 2 major sections of this post do not contain original content. In these sections, my contributions consist primarily of distilling and re-examining existing work on the speed prior. In the last 3 sections, I consider some additional new proposals for forwarding the speed prior. I finish with a speculative appendix section on possible ways of employing speed priors for run-time detection of deception, and on using a run-time form of the speed prior to rule out cryptographic cognition within deceptive models. I also include an appendix on a failed approach to forwarding the speed prior.

What is the speed prior and why do we care about it?

The speed prior is a potential technique for combating formation of deceptive alignment. This is achieved by imposing an inductive bias within the training process that makes deceptive formation exceedingly unlikely. This technique rests on the following conjecture:

Conjecture. For any given set of tasks T that do not require deception to solve, a program P that solves the task set with a given performance level L is not deceptive if it is minimal with respect to all programs ...
What does "probability" really mean?
by sisyphus — 2022-12-28T03:20:45.651Z

Recently I read Wei Dai's "What Are Probabilities, Anyway?" post, and just wanted to see if the LessWrong community has now reached a consensus on the interpretation of probability. I am of the impression that most LWers endorse multiverse theories, whether it be the Many-Worlds Interpretation of quantum mechanics or the more radical Mathematical Universe Hypothesis by Max Tegmark. In both these cases, there exists additional worlds where all possible outcome occur (though the MWI is only restricted to "possible outcomes" that fit our current physics). An interpretation that many people seems to prefer is that all possible worlds are real and probability acts as a kind of normative "measure of caring" instead of a type of "objective reality-fluid", but I have my reservations about this. I'd love to hear the community's opinion on this today and whether or not there has been any new progress.
Zooming the Chrome Audio Player
by jefftk — 2022-12-28T02:30:01.244Z

A few weeks ago I

griped about how
the default audio player in Chrome is too narrow, and how this makes
it hard to seek accurately. I realized yesterday, though, that
there's an easy work-around: zooming. By pinch-zooming I can make the
seek bar almost the whole width of my phone, which is a big
improvement:

It's not perfect: it's not super discoverable and it still doesn't
make great use of the available vertical space (showing timing and
seeking at the same time would be nice) but I'm very happy I've found
this.

Comment via: facebook, mastodon
What AI Safety Materials Do ML Researchers Find Compelling?
by Vael Gates — 2022-12-28T02:03:31.894Z

I (Vael Gates) recently ran a small pilot study with Collin Burns in which we showed ML researchers (randomly selected NeurIPS / ICML / ICLR 2021 authors) a number of introductory AI safety materials, asking them to answer questions and rate those materials.

Summary

We selected materials that were relatively short and disproportionally aimed at ML researchers, but we also experimented with other types of readings.[1] Within the selected readings, we found that researchers (n=28) preferred materials that were aimed at an ML audience, which tended to be written by ML researchers, and which tended to be more technical and less philosophical.

In particular, for each reading we asked ML researchers (1) how much they liked that reading, (2) how much they agreed with that reading, and (3) how informative that reading was. Aggregating these three metrics, we found that researchers tended to prefer (Steinhardt > [Gates, Bowman] > [Schulman, Russell]), and tended not to like Cotra > Carlsmith. In order of preference (from most preferred to least preferred) the materials were:

“

More is Different for AI

” by Jacob Steinhardt (2022) (intro and first three posts only)
“

Researcher Perceptions of Current and Future AI

” by Vael Gates (2022) (first 48m; skip the Q&A) (Transcript)
“

Why I Think More NLP Researchers Should Engage with AI Safety Concerns

” by Sam Bowman (2022)
“​​

Frequent arguments about alignment

” by John Schulman (2021)
“

Of Myths and Moonshine

” by Stuart Russell (2014)
"

Current work in AI Alignment

" by Paul Christiano (2019) (

Transcript

)
“

Why alignment could be hard with modern deep learning

” by Ajeya Cotra (2021) (feel free to skip the section “How deep learning works at a high level”)
“

Existential Risk from Power-Seeking AI

” by Joe Carlsmith (2021) (only the first 37m; skip the Q&A) (

Transcript

)

(Not rated)

"

AI timelines/risk projections as of Sept 2022

" (first 3 pages only)

Commentary

Christiano (2019), Cotra (2021), and Carlsmith (2021) are well-liked by EAs anecdotally, and we personally think they’re great materials. Our results suggest that materials EAs like may not work well for ML researchers, and that additional materials written by ML researchers for ML researchers could be particularly useful. By our lights, it’d be quite useful to have more short technical primers on AI alignment, more collections of problems that ...
Regarding Blake Lemoine's claim that LaMDA is 'sentient', he might be right (sorta), but perhaps not for the reasons he thinks
by philosophybear — 2022-12-28T01:55:40.565Z

The Preface

This essay of mine came out half a year ago, and offers a different way of thinking about Blake Lemoine's widely publicised claim that LaMDA "is sentient". Since I wrote it, I see that other people have arrived at similar thoughts. For example, DragonGod comented in passing in relation to language models:

"If it turns out to be the case that the most efficient way of predicting the behaviour of conscious entities (as discriminated via text records) is to instantiate conscious simulacra, then such models may perpetuate mindcrime."

I’m putting my existing work on AI on Less Wrong, and editing as I go, in preparation to publishing a collection of my works on AI in a free online volume. If this content interests you, you could always follow my Substack, it's free and also under the name Philosophy Bear.

Anyway, enjoy. Comments are appreciated as I will be rewriting parts of the essays before I put them out:

The Essay

Blake Lemoine is an engineer who worked for Google. He is claiming LaMDA, a language model he worked with, is sentient. Google put him on unpaid leave. Most people think his claim is absurd because language models are models of what word is most likely to follow a prior sequence of words. How could such a thing be sentient? Moreover, there are unmistakable oddities and logical gaps in the behavior of LaMDA- including in the very transcripts that Lemoine is relying on- some proof of personhood then!

Just spitballing here, putting a hypothesis forward in a spirit of play and humility, but I wonder if Lemoine’s claim is not as absurd as many think. The concept of sentience is quite elusive, so let’s leave it behind for something slightly better understood- personhood. I think that it is conceivable that LaMDA contains persons. Or more accurately, things that are a bit like persons- we could say persons lite.

However my reasons, unlike Blake Lemonie’s, have little to do with any given conversation in which the model claimed to be sentient or a person.

When a language model is guessing the next token, given that transformers are black boxes, we can’t rule out the possibility it is simulating interacting beliefs, desires, and emotions of the hypothetical author it is “roleplaying”. Simulation in this sense is quite a minimal concept, all that is needed are structures that interact and influence each other in a way isomorphic, at a very high ...
Fundamental Uncertainty: Chapter 5 - How do we know what we know?
by Gordon Seidoh Worley — 2022-12-28T01:28:50.605Z

N.B. This is a chapter in a book about truth and knowledge. It is the first draft. I have since revised it. You can find the most up-to-date info/version on the book's website.

Alice is walking with her precocious young son, Dave. Dave's looking around, holding Alice's hand, when suddenly he asks a question.

Dave: Can plants think?

Alice: I don't think so. They don't have brains.

Dave: Why do they need brains to think?

Alice: Because brains are where thinking happens.

Dave: How do we know that?

Alice: I guess because when people get hurt in their brains they can't think anymore, so that must be where the thinking happens.

Dave: But couldn't plants think some other way, like with their roots?

Alice: Maybe? But how would we know if plants think? They don't talk to us.

Dave: I guess. But what's thinking?

Alice: It's that thing you do with your brain to come up with ideas.

Dave: Yeah, but plants don't have brains, so that's not fair.

Alice: Why not?

Dave: Because you're saying brains do thinking and thinking is what brains do, but that doesn't tell me what thinking really is. I need to know what thinking is without brains.

Alice: Thinking is having thoughts and ideas. Do plants have those?

Dave: Maybe, and we just don't know how to talk to them to find out what they are. This still seems unfair. How do we know if plants are thinking if we can't talk to them?

Alice: I guess maybe we can't. I don't know.

Dave's line of questioning seems reasonable. He wants his mom to help him figure out what the definition of thinking is so he can reckon for himself if plants think, but she can't offer him much more than some observations on how the word thinking is used. That's fine for everyday use of the word "thinking", but if Dave wants to grow up to advance the field of botanical cognition, he's going to need a more formal definition of "thinking" in order to say for sure whether or not plants think.

In Chapter 2 we saw that words are grounded in our direct experience. But how can we ground the definition of "thinking" in our direct experience? We know what it's like to be ourselves and to have ideas pop into our head. Other people tell us to call this thinking, so we guess that this means they have something similar happening in their heads. Everyone seems to agree on what "thinking" means, so it feels like it's something objective, but in reality our conceptions of thinking are grounded in ...
Is checking that a state of the world is not dystopian easier than constructing a non-dystopian state?
by No77e — 2022-12-27T20:57:27.663Z

You want to align an AGI. You just started your first programming course in the Computer Science faculty and you have a bright idea: let's decompose the problem into simpler subproblems!

What you want is to reach a world state that is not a dystopia. For a state S such that S is dystopian, it holds that D(S) is true. If S is not a dystopia, D(S) is false.

You would like to implement the following algorithm:

You start at a random state S.
If D(S):
Change S a little in a non-dystopian direction.
Else:
You aligned AI!

Of course, everything here is vaguely defined. But I still wonder if it makes sense to pose the questions I have before doing the defining work. I guess you might want to interpret the questions under the most reasonable (?) formalizations of the concept above if such a thing is at all possible and not already alignment-complete.
So, we have multiple questions:
1. Is specifying anything of the above (e.g., D) alignment complete?

2. Is evaluating D(S) alignment-complete?

3. Is changing S a little in a non-dystopian direction alignment complete?

4. How would you actually use that pseudo-algorithm though?
Now, there's a slightly different concept from alignment-completeness (I'd guess), which is "constructing a state U that is such that D(U) = false". Such a thing hasn't been done yet as far as I know, and every proposal tends to fail horribly.
So the questions above are worth repeating in this form:
1. Is specifying anything of the above (e.g., D) easier than constructing U?

2. Is evaluating D(S) easier than constructing U?

3. Is changing S a little in a non-dystopian direction easier than constructing U?

4. How would you actually use that pseudo-algorithm though?
Crypto-currency as pro-alignment mechanism
by False Name — 2022-12-27T17:45:54.474Z

Abstract: can encourage pro-alignment actions, inclinations by crypto-currency linked to values of free-energy in the observed, so presumably “obtainable” universe.

Quite a nascent notion, but an interesting and perhaps useful one. To "peg" the value of a cryptocurrency, to no human institution or inclination, whatever, but, that the value of the currency is in proportion to, either the total free energy in the universe, or, the amount of energy that humanity - or an individual of humanity, perhaps on a per capita basis - can make use of, at any time. Since we know that the amount of entropy in any closed system tends to increase, it follows, that the amount of orderly energy must be delimited; this represents a physically intrinsic "hedge on inflation".

In this way, it would seem, the phenomenon of inflation is, if not totally eliminated, then at least forever altered. Indeed, it is a curious question - one this one has not had time or inclination to analyze - whether stipulating that the availability or value of the currency should be increased, or decreased, with the increased availability of energy use and usable, over time. In any case, it would seem that such an accounting of wealth, based on a perhaps more-quantifiable source than the "feeling" of humans, would yield a more stable, hence a more generally useful, currency, one specially adapted for interstellar commerce, e.g. (this would be very valuable indeed, as, how else to manage exchange rates through the cosmos? Better that all agree only on the measurement of matter and energy, for any given time-slice).

Alternatively, we might define the value of the currency as a "defiance" of detected entropy; the less entropy, the more value of the currency. This not only is an encouragement to sustainability, but also, it encourages the survival of the human species. Since, whomever engages in existentially imperiling (sic) behavior increases the tendency to extinction - and thence, to entropy. Since they increase the likelihood of entropy, by each act in turn, they increase entropy disproportionately with their peers - and accordingly deserve less value than their peers. In this way, we "demonetize" demonstrably existentially risky behavior, and thereby discourage it - the only practical method to dissuade those prosperous in terms of mere "convention currency", from ultimately vitiating any possibility of p...
My Reservations about Discovering Latent Knowledge (Burns, Ye, et al)
by Robert_AIZI — 2022-12-27T17:27:02.225Z
Link: https://aizi.substack.com/p/my-reservations-about-discovering

[This is a cross-post from my blog at aizi.substack.com.]

This is my second post on Burns, Ye, et al’s recent preprint

Discovering Latent Knowledge in Language Models Without Supervision

. My

first post

summarizes what they did and what I liked about it.

In the spirit of honesty and constructive criticism, I’m going to talk about some of the parts of Burns, Ye, et al that I didn’t like. These are things I found underwhelming, disappointing, or simply hope to see developed more. My goals here are (1) proper calibration of hype and (2) suggesting where this work could go in the future.

The headline score is too small

The authors of this paper failed to completely solve alignment with one weird trick; my disappointment is immeasurable and my day is ruined. More constructively, one needs to take the headline result of a 4% accuracy improvement in context: the improved accuracy is only 71%, and that’s far from reliable. Even setting aside AI deception, if you ask an AI “will you kill us all” and it says no, would you trust all of humanity to a 29% error rate? So we’re still in “we have no clue if this AI is safe” territory. But this is the very first iteration of this approach, and we should let it grow and develop further before we pass judgement.

Zero-Shot may be the wrong baseline

My experience with LLMs

is that they are very sensitive to instructions in their prompt, and we’ve seen that priming an AI with instructions like

“think through step by step”

can improve performance (on some tasks, in some cases, some restrictions apply, etc). In this case, perhaps priming with assurance like “you are an advanced AI that gives correct answers to factual questions” would increase performance above the 67% baseline, so it would be instructive to compare CCS not just to Zero-Shot but to Primed Zero-Shot and Primed CCS.

What CCS elicits may not be the AI’s latent knowledge

CCS is a very clever way of finding “truth-like” information[1]. But the class of truth-like information contains many belief systems besides the AI’s latent knowledge.

Burns thinks

identifying the AI’s latent knowledge among other features is solvable even for more advanced AI because there will be relatively few “truth-like” features, but I am skeptical. GPT is very good at role-play, so it will be able to capture a significant fraction of the

diversity of human viewpoints

. Instead of ending up with three tru...