trentmkelly/LessWrong-43k huggingface dataset:
Three questions about mesa-optimizers
by Eric Neyman — 2022-04-12T02:58:00.497Z

I recently (re-)read Risks From Learned Optimization, which introduced mesa-optimizers and inner alignment. (I don't aim to explain what these things mean in this post, but here's Scott Alexander's explanation from earlier today.) I have some questions about mesa-optimizers, all to the effect of "What would this actually look like?"

1. Would training a neural net on an optimization task produce a mesa-optimizer?

My understanding (correct me if I'm wrong) is that as of yet we do not have an example of mesa-optimization in practice. So, what would happen if we trained a neural net to do a task that clearly requires optimization?

For example, consider the following task: the neural net receives a polynomial P(x) (perhaps with its coefficients represented in binary as input), and the neural net is to output the value x in the interval [0, 1] such that P(x) is as large as possible. Perhaps the loss on output x is the squared difference between P(x) and the true largest value of P on [0, 1].

I'm not sure what I expect the neural net to learn to do. Maybe it would try lots of values of x and output the optimal x of the ones it tried. Maybe it would learn to take the derivative of P and do Newton's method. But I can't imagine it learning to do this well without learning to do some sort of search or optimization. It seems, then, that if a neural net were successfully trained on this task, it would necessarily have a mesa-optimizer.

So my question is: (a) does what I said seem right? And (b) what would happen if we actually tried this now? Are current neural network architectures (with current levels of compute) able to learn this task? Or maybe a simpler task in the same spirit?

2. What would a neural net doing search actually look like?

When explaining mesa-optimizers, I currently wave my hands and say something like "Neural networks are arbitrarily expressive, so it is possible to encode a for loop to do search as weights in a neural net." But I'd love to actually see what this looks like, as honest-to-goodness weights. I'm not sure whether the following exercise is pretty easy or extremely hard, but consider the following program that, given n, searches for k < n^0.7 that maximizes k^2 mod n:

Input: integer n <= 2^30.
k = 0
for i = 1 to n^0.7:
if i*i mod n > k*k mod n:
k = i
return k

How would one write this program as a neural n...
The Amish
by PeterMcCluskey — 2022-04-12T02:54:03.966Z
Link: http://www.bayesianinvestor.com/blog/index.php/2022/04/08/the-amish/

Book review: The Amish, by Donald B. Kraybill.

It feels appropriate to review this book right after The Dawn of
Everything
(TDOE). There are strange contrasts between the cultural views of the
two books.

TDOE clearly exaggerated the extent to which European civilization
disapproved of freedom and equality, yet Amish culture shows an
important kernel of truth behind that exaggeration.

The Amish strongly promote obedience as a virtue. This seems at first
glance like the antithesis of the societies which TDOE promotes, with
their contempt for most forms of obedience.

Yet Amish are selective enough about their obedience that the practical
differences are complicated. Amish pretty consistently reject commands
that conflict with religious traditions. Amish parents refuse commands
to send their kids to high school. The US military occasionally drafted
Amish youth, and got them to show up, but didn't get them to provide
any useful support. Some Amish communities disobey commands to put smoke
alarms in their homes.

The Amish are pretty obedient to human authorities when those
authorities give orders that seem compatible with God's will. But the
Amish are serious about treating human threats and rewards as
insignificant compared to those of God.

Amish peer pressure to be humble seems much more effective than any
modern liberal or progressive proposals at minimizing inequality between
living people.

Equality doesn't seem to be an important Amish goal, but their rules
leave almost no way to create the inequalities that TDOE or
Piketty
complain about. Most of the inequality between living Amish seems to be
a function of age.

When a husband and wife disagree, the husband's will prevails. But
there seems to be strong pressure to resolve all disputes by reaching a
consensus rather than by using power. Anyone who frequently used his
authority to settle disputes would likely risk being pressured to
confess to a sin such as pride.

It is said that we can tell a fair amount about how equal a country or
organization is by looking at whether it's depicted mainly by photos
(or statues) of its leader, versus photos of many ordinary citizens /
team members. The Amish are off the far end of this scale: they have
firm rules against displaying or posing for photographs. They also
prohibit fancy clothes, jewelry, and pretty much anything that could
signal that one person is better than another.

Th...
Rationalist Should Win. Not Dying with Dignity and Funding WBE.
by CitizenTen — 2022-04-12T02:14:36.137Z

This is somewhat a response to the Dying with Dignity post.

Rationalist should win. If you are taking actions that lead to failure, you aren’t being rational, you aren’t playing for dignity points, you are losing. (More applause lights here)

But the core argument needs to be addressed. AI alignment looks really hard. We have more unsolved problems than you can shake a stick at (here, here, etc.) and it seems there are good reasons to think that AGI will by default be unaligned. (Mind-space is big after all) The argument continues. Currently, we have more money than ideas. We’re trying everything we can think of, nobody has any clear solutions or even an idea for a real path forward. It’s so bad that we aren’t even sure if we know how confused we are about the problem. And Machine Learning progress is barreling down a hill, setting very short deadlines on solving this problem. Basically, unless the heavens themselves part and give us a Deus ex machina, we’re dead, flayed, and roasted alive a million times over.

So I have to ask. Are we doing everything we can think of? Set a 5 minute timer, forget everything you know about AI alignment, computer science, and make yourself as nontechnical as possible. Ask yourself. If you were to build an AGI, how would you go about doing it? Give yourself full license to think as scrappy as possible. Gloves are off. Cheating is allowed. It doesn’t matter how you do it, as long as it works.

My first thought. This sounds hard. Can I copy someone else’s work? And we can! We genuinely can! We have AGI’s running around all the time. They're called Humans! This strategy is generally called human emulation.

So we know a strategy that will work. We have actual evidence this is true. Human’s exist and are (generally) aligned with human values. And emulating a human lets us sidestep all the tricky alignment/philosophical problems we're stuck on. We treat the emulation as a black box, run it at 10x speed and there you go. Problem solved[1]. And I’m part of a community that wants to build AGI. So what’s the current funding situation for this? The Dying with Dignity post s...
How can I determine that Elicit is not some weak AGI's attempt at taking over the world ?
by Lucie Philippon — 2022-04-12T00:54:10.621Z

I read two days ago the Elicit presentation post, and as I have been reading intensely about AI safety lately, I was wondering this question : What are the tools and techniques you would use right now to determine that the current Elicit research assistant is a safe AI ?

What is described in the post is a narrow AI that will and does fail in obvious ways if it is not aligned. But how would you know it is not a small and limited AGI, aligned or not ?

I suppose that if the Elicit research assistant is currently an AGI, however how limited, its best course of action would be to pretend to be a narrow AI helping researchers in their goal of doing research. I imagine that it would get better chances of not being replaced by a new version and to get more upgrades and computing resources as the Elicit product develops.

So I was wondering, is this obviously dumb that this exact system as seen by Elicit users cannot ever be an AGI, even in the future ? If you had right now real suspicions it could be an AGI, apart from first making sure it is stopped, how would you try to determine whether it is an AGI ?
Summary: "How to Write Quickly..." by John Wentworth
by Pablo Repetto — 2022-04-11T23:26:08.692Z
Link: https://pabloernesto.github.io/2022/04/11/summary-johnswentworth-writing.html

This is part of my "research" series.

Wentworth, John S. How To Write Quickly While Maintaining Epistemic Rigor. LessWrong (blog), August 28, 2021. https://www.lesswrong.com/posts/Psr9tnQFuEXiuqGcR/how-to-write-quickly-while-maintaining-epistemic-rigor. Accessed 2022-04-08.

Why

"How To Write Quickly While Maintaining Epistemic Rigor" sounds like a gift-wrapped solution to my research question. I didn't know the author, but I had to read this.

In one paragraph

Share partial results. While thorough research takes forever, you don't need to do it all in one go, or even all yourself. Correct conclusions matter, but writing something that can be evaluated, criticized, and integrated with new evidence matters more.

Table of Contents

(0 Introduction)
1 It's about the process, not the conclusion
2 Correctly conveying uncertainty
3 So should I stop researching my claims?
4 Bad habits
5 Takeaway

An alternative outline

Broadly, how this article reads from my perspective.

1 Motivation: crippling self-doubt (s0)
2 Transparency and extensibility (ss 1, 2)
3 Scope of the advice
3.1 Transparency is no substitue for research (s3)
3.2 This is anti-persuasive advice (ss 4, 5)

Highlights

The good, the bad, and the extremely revealing. Direct quotations in no particular order. Some of these ideas may not be discussed at length, but it still seemed worthwhile to put them here.

Motivation

There’s this trap people fall into when writing [...] They have a good idea, or an interesting belief, or a cool model. [...] So they go looking for evidence [...] and soon end up down a research rabbit hole. Eventually, they give up and never actually publish the piece.

(s0)

Perfecting the Art

As much as possible, I want to show the actual process by which I reached that conclusion. If my process is good, this will better enable others to copy the best parts of it. If my process is bad, I can get feedback on it directly.

(s1)

Path dependence

Correct predictions before looking at the data are much more surprising (and thus stronger evidence) than correct predictions after looking at the data.

[T]he amount of uncertainty I should assign [to my conclusion] depends on the details of my process. It depends on the path by which I reached the conclusion.

(s2)

Scope

Obviously researching claims still has lots of value. But you should not let uncertainty stop you from writing things up and sharing them.

(s3)

It’s been p

...
Rambling thoughts on having multiple selves
by cranberry_bear — 2022-04-11T22:43:56.842Z

In the past, if someone was to ask me “are you good at describing your emotions” or “are you good at introspecting” my answer would be a resounding yes. I have always felt like I have a profound understanding of my feelings, desires and true nature, even when some aspects are hidden from everyone else. But more recently I have come to question this, wondering whether I actually understand myself in a non-trivial way and whether there is even anything to meaningfully “understand”.

More often these days I conclude that there isn’t a unified self that one can come to understand by introspecting and trying to grok one’s subconscious feelings and desires. The continuous and unified concept of “self” is a useful thing to discuss and refer to in many contexts, however, when it comes to understanding “oneself” (or someone else, I think), the “one” seems to become inaccurate. Inside a human brain is a hodgepodge of signals, conflicting subgoals, and different subsystems based on different modes of thinking – it’s all very far from an ideal unified and efficient system with a singular set of beliefs. The emergent rational mode of thought is the simplest to analyze – this is a matter of pure logic and reason. However, our other patterns of thought – what may be referred to as “emotions”, “feelings”, “instinct” and “intuition” are not only not optimized for rationality but also are not, in my view, unified and consistent. By that, I mean that one can have conflicting/inconsistent emotions and subconscious desires simultaneously.

When I used to introspect, I used to ask myself the question “how do I *really* feel about this”, especially when it came to emotionally difficult questions that I felt conflicted about. I would try to collapse my inconsistent superposition of subconscious ideas into a set of sentences in natural language that made logical sense to my conscious mind. And for that to happen I would force myself to pretend that there was a clear story to tell about how I felt. My current thoughts on the matter, after having noticed this issue, are that I need to normalize to myself being confused, un-unified, and unstable in my inner irrational mind. It seems more accurate to model my mind as a superposition of multiple “selves”, sub-agents, or something like that.

Furthermore, this is a helpful model of other people. It gives a different perspectiv...
An AI-in-a-box success model
by azsantosk — 2022-04-11T22:28:02.282Z

Given that the strategy of Death with Dignity is not particularly appealing to me, and that others have encouraged us to instead Do Something Else (if Alignment is Doomed), I am going to move forward and explain why, despite the well-known limitations, I am still optimistic on the old idea of creating an Oracle and keeping it in a box.

My reasoning is as follows:

The class of AGIs that may be deployed safely as an Oracle seems to be larger than the class of intent aligned or a Friendly AGIs.
It seems at least possible that building a safe Oracle will be considerably easier than solving general AGI alignment, and that we can do so in a timescale competitive with the development of similar unfriendly AI.
These Oracles are going to be extremely useful, and even if costly precautions are needed to make them safe, and it may still be possible for their operators to achieve decisive strategic advantage.

If these ideas are correct, then we may have a plan. It may be hard to execute and not at all guaranteed to succeed, but it has at least some potential to work. And plans like that are exactly the ones you should be looking for if you are losing and need to play to your outs.

Risks and general precautions

In order to understand the class of AGIs that may be deployed safely as a Oracle we need to understand first what the risks are. Here are the main ones that I am aware of:

The AI may persuade the human gatekeepers to let it out, as shown to be possible even by humans in the AI-Box Experiment.
The AI may escape by technical means, such as hacking, or by a combination of technical and social engineering, such as generating radio waves or using other "unintended output channels" to communicate with third-parties, and then using persuasion or manipulation in order to get them to do what it wants.
The AI may manipulate the human gatekeepers, secretly guiding them into world-states in which it is more likely to escape. For example, the AI may provide good-looking plans with subtle failure modes that are somehow beneficial to the AI, and that the AI can manage to avoid blame for.

Risks A and B are both serious, and both require costly countermeasures, but it seems to me that none of them actually make it impossible for us to create a secure Oracle AGI.

For example, we can:

Strictly limit the output bandwidth of the system. The less information content in the AGI's output, the less li

...
The Regulatory Option: A response to near 0% survival odds
by Matthew Lowenstein — 2022-04-11T22:00:50.311Z

This is inspired by Eliezer’s “Death with Dignity” post. Simply put, AI Alignment has failed. Given the lack of Alignment technology AND a short timeline to AGI takeoff, chances of human survival have dropped to near 0%. This bleak outlook only considers one variable (the science) as a lever for human action. But just as a put option derives its value not merely from current prices and volatility, but also from time to expiration—so too do our odds of success. To hew more closely to the options metaphor, the expected value of human civilization hinges on our AGI timeline. Is there a way to buy more time?

A modest proposal: An FDA for Artificial Intelligence

The AI world has an instinctive revulsion to regulation. This is sensible. Committees of bloodless bureaucrats have an abysmal track record at achieving their stated goals. Eliezer has joked that if the government funded AI research, it might be a good thing because progress would stall. Let’s take that idea seriously.

What if we shifted efforts into regulating AI, domestically and globally? While regulation has a terrible track record at making consumers, investors, and workers better off, it excels at stalling growth. Scott has done yeoman’s work tracking the baleful impact of the FDA on pharmaceutical development and on health in general. I’m pretty skeptical of social scientists’ ability to measure regulation with stylized facts, but there is reason to think that the FDA effect generalizes.

I propose we enter the lobbyist business, with a goal toward erecting similar administrative hurdles to AI research. The benefits of such hurdles are obvious. They would throw a spanner into the works of AI companies around the globe. It is impossible to say ahead of time how much they would succeed at halting AI improvements. But the possibility that they might do so—perhaps significantly—should raise the chances of human survival at least higher than 0%.

In the possible worlds with intrusive regulatory agencies, how many make no progress toward AGI at all? I’d guess in at least 5% of possible worlds, a sufficiently arbitrary and bloated bureaucracy would halt AGI indefinitely. At this level of possibility, Eliezer is probably not even in “die with dignity” mode anymore. Heck, he may even be able to resume working ultra-hard, without breaking his promise to his previous self.

Objection 1: Regulators aren’t smart e...
The Efficient LessWrong Hypothesis - Stock Investing Competition
by MrThink — 2022-04-11T20:43:49.625Z

There has been debate to whether the

Efficient Market Hypothesis

is or ever was valid.

The evidence brought forward of its death is usually stories of people in the LessWrong community correctly predicting market changes, using only widely available information and rational reasoning.

It might be correct that the market is highly efficient, but that LessWrong members and the community as a whole, in many cases, is even more efficient.

It is not unreasonable, that rationalists with a certain area of expertise and great knowledge of biases and reasoning, might occasionally find investment opportunities they can reasonably expect to beat the market (adjusted for risk of course). Even if for any given member this happens only once or twice in a lifetime, those opportunities could be highly valuable for the community. And if a great number of LessWrong members shared when believing to have found such opportunities, it might be possible to consistently beat the market. At least until a big investment fund picks up on it and starts analysing LessWrong for stock predictions.

To test the “Efficient LessWrong Hypothesis”, I’m arranging two competitions:

Stock Picking Competition - Participants comment what security (stock, bond, crypto, etc) they believe will outperform the market in general (VTSAX)

adjusted for risk

. The most upvoted suggestions win. (Outlined below)

Stock Price Prediction Competition - Participants get to submit probability distributions for how they believe the stocks from the previous competition will perform. (This competition will begin after the first is completed)

Stock Picking Competition - Rules

All LessWrong members are invited to comment on this post, what securities they believe will outperform the market over the next twelve months, along with their reasoning for why they think the stock will outperform VTSAX

adjusted for risk

.

Any security, such as stocks, bonds or cryptocurrencies, traded on any major trading platform, such as eToro and RobinHood can be picked. Also, a very small group of stocks could be picked together to decrease risk, for example “I suggest AMD and Nvidia stocks, because GPU market will do well, and buying both decreases risk.”

No investment strategy except for the “buy and hold” can be considered in this competition.

Please upvote other comments, based on how convincing those inves...
Review: Structure and Interpretation of Computer Programs
by L Rudolf L — 2022-04-11T20:27:13.167Z
Link: https://www.strataoftheworld.com/2019/09/review-structure-and-interpretation-of.html

This is a linkpost (written several years ago) that I'm posting in response to a request to share more of the archive of my personal blog on LessWrong. It is most useful if you have none or limited experience with functional programming and might want to get into it, or if you have a good practical grasp of programming but want to read something about the unifying concepts.

Many regard Structure and Interpretation of Computer Programs (SICP) as the bible of programming. For good reason, as it turns out.

Beware the wizards

The Wizard Book. (Credit: MIT Press)

SICP is sometimes called the “Wizard Book”, because there’s a wizard on the cover (if your job is making an interesting cover for a programming book, what would you do?). However, this does not mean that the book has anything to do with –

“[L]earning to program is considerably less dangerous than learning sorcery, because the spirits we deal with are conveniently contained in a secure way.”

Um. Okay, I rest my case. Proceed with caution.

Contrarian SICP

For most subjects there is a standard way to present it that most books, lectures, etc. will follow.
For programming, the standard way seems to be to take some “mainstream” language, show how to print “Hello, World!” onto the screen, then start introducing things like assigning values to variables, conditionals, and so on. Pretty soon you can be doing some pretty impressive things.
SICP does not follow this route.

Why Lisp?

The first thing that might strike you about SICP is that the programming language of choice is Scheme, a dialect of Lisp (short for “LISt Processor”), which is commonly known as that obscure language invented in 1958 that wears down the parentheses keys on your keyboard.

Comic by Randall Munroe of xkcd. This comic can be found here.

However, the authors are not just being contrarian here; there are many good arguments for using Lisp in a book like this.
First, Lisp is the closest a programming language can get to having no syntax. You don’t have to learn where curly brackets are used, or which operators/functions follow which type of syntax, or a multitude of special characters that perform arcane pointer logic (I’m looking at you, C++).

If you have an expression in parentheses, the first thing inside the parentheses is the name of the function that is being called. Everything after it is an argument to be passe...