trentmkelly/LessWrong-43k huggingface dataset:
Safety Implications of LeCun's path to machine intelligence
by Ivan Vendrov — 2022-07-15T21:47:44.411Z

Yann LeCun recently posted A Path Towards Autonomous Machine Intelligence, a high-level description of the architecture he considers most promising to advance AI capabilities.

This post summarizes the architecture and describes some implications for AI safety work if we accept the hypothesis that the first transformative AI will have this architecture.

Why is this a hypothesis worth considering?

LeCun has a track record of being ahead of mainstream academic research, from working on CNNs in the 90s to advocating for self-supervised learning back in 2014-2016 when supervised learning was ascendant.
LeCun runs Meta AI (formerly FAIR) which has enormous resources and influence to advance his research agenda, making it more likely that his proposed architecture will be built at scale. In general I think this is an underrated factor; AI research exhibits a great deal of path dependence, and most plausible paths to AI are not taken primarily because nobody is willing to take a big risk on them.
The architecture is dramatically different from the architectures commonly assumed (implicitly) in much AI alignment work, such as model-free deep RL and "GPT-3 but scaled up 10000x". This makes it a good robustness check for plans that are overly architecture-specific.

Architecture Overview

The Overall Agent

At a high level, the proposed architecture is a set of specialized cognitive modules. With the exception of the Actor and the Intrinsic Cost (see below) they are all deep neural networks trained with gradient descent.

The high level architecture of LeCun's proposed agent. Arrows indicate dependence; gradients flow backward through the thin arrows.

What is this agent doing, exactly? It's meant to be a general architecture for any autonomous AI agent, but LeCun repeatedly emphasizes video inputs and uses self-driving cars as a recurrent example, so the central use case is embodied agents taking actions in the physical world. Other talks I've seen by LeCun suggest he thinks understanding video is essential for intelligence, both by analogy to humans and by a heuristic argument about the sheer amount of data it contains.

The World Model

More than half the body of the paper is about designing and training the world model, the predictive model of the environment that the AI uses to plan its actions. LeCun explicitly says that "designing architectures and training...
Comfort Zone Exploration
by CFAR!Duncan — 2022-07-15T21:18:14.033Z

Epistemic status: Mixed

The concepts which inspired the CoZE unit (such as exposure therapy and the explore/exploit problem in probability) are generally well-researched and under- stood. However, our combination of these concepts with an outlook of curiosity and epistemic uncertainty has changed their application somewhat. We have received promising feedback from alumni, but the activity continues to benefit from iteration.

The experience of comfort is in many ways a non-experience. It’s often easier to describe what it isn’t than what it is—it’s a lack of irritation, a lack of pain or discomfort, a lack of negative emotions like fear or anxiety or despair or defensiveness.

When we’re in our “comfort zone,” we feel calm, agentic, optimistic, and confident. Often, it’s a confidence born of experience—since most of us spend the majority of our time doing things that are comfortable, then the majority of the things we’re comfortable with will be things we’ve experienced many times, and are intimately familiar with.

Much of the time, things that lie outside of our comfort zone are out there for good reason. They’re things that cause us to anticipate danger, experience stress, and wrestle with uncertainty, and under many circumstances, it’s good to avoid danger, stress, and uncertainty.

But there’s a gray area between “definitely good” and “definitely bad”—between comfortable and uncomfortable. It’s an area characterized by mixed experiences and model uncertainty, filled with things we’re not sure about, or things we’ve struggled with, or things we’ve abandoned (or never dared to try). They’re outside of our comfort zone, but it’s not clear that they should be—it’s not clear whether they’re actually Things We Ought To Avoid.

The Comfort Zone Exploration technique (CoZE) is a method for gathering data about this gray area. It asks that we stretch our comfort zone, in small, safe experiments, a little bit at a time. The idea is to calibrate our discomfort, loosening up and letting go of unhelpful inhibitions while preserving those that are helpful, appropriate, and useful.

The problem with progress

There’s a classic dilemma in probability theory known as the explore/exploit problem. Roughly speaking, it highlights the tradeoff between spending resources on known goods, and spending resources in the search for new potential. You can think of it as the choice be...
A time-invariant version of Laplace's rule
by Jsevillamol — 2022-07-15T19:28:15.877Z
Link: https://epochai.org/blog/a-time-invariant-version-of-laplace-s-rule

What is the probability that the sun will rise tomorrow? What are the chances of a pandemic happening next year? What are the odds of survival of a new surgery that has been successfully executed only once?

These and many other questions can be answered appealing to a general rule: Laplace's rule of succession. This rule describes the probability of a positive outcome given information about past successes. The versatility and generality of the rule makes it an invaluable tool to forecasters, who use it to estimate base rates[1].

Laplace's rule can be stated in simple terms. If we have repeated an experiment T times, and observed S successes, we can estimate the posterior probability of obtaining a success in the next trial as p=S+1T+2.

However, there is a fatal problem when applying the rule to observations over a time period, where the definition of what constitutes a trial is not as clear. For example, if we were to estimate the chances of a pandemic happening next year. Should we take T as the number of years that our dataset of past pandemics covers? Or as the number of days? This essentially arbitrary choice leads to different answers to our question[2].

After grappling with this problem, our final recommendation is a simple formula. Essentially, if we have data on a time period of T years, and we have observed S successes over that period, the chances that no successes will happen in the next t years are (1+tT)−(S+1). We call this formula the time-invariant Laplace's rule.

If we have chosen the observation period T so that it is equal to the time since the first recorded success, we will subtract one success from our count, and the probability ought to be instead (1+tT)−S. This is because in this case we always expect the sampling period to have at least one success, so the first success provides us with no new information to update our prior on.

Number of observed successes S during time T

Probability of no successes during t time

S=0

(1+tT)−1

S>0

(1+tT)−S if the sampling time period is variable

(1+tT)−(S+1) if the sampling time period is fixed

Graph showing values of the formula (1+tT)−S for various values of S

It's important to note that this rule is recommended only as a replacement for Laplace's rule and suffers from most of the s...
An attempt to break circularity in science
by fryolysis — 2022-07-15T18:32:51.589Z

We do science using the data we collected through our senses and we use science to understand how our senses work. Although I lack any rigorous formulation of the problem, the following plan seems interesting and I want to share it with you.

p= "Human senses collect accurate data about reality."

q= "Reality is governed by laws of physics."

Now we want to know Pr(p) and Pr(q). I don't know any way to directly calculate these values. It seems to me that Pr(p|q) and Pr(q|p) are easier to get our hands dirty. The former seems to be a scientific question and I heard that there are research done already, and the latter could be addressed by something like Solomonoff's Induction. At this point, we should be able to calculate the ratio Pr(p)/Pr(q).

The last stage is to calculate Pr(q|¬p). Here for simplicity I assume that the trust in our senses is something binary, we either believe all data we collect is about reality or none (I hope that the scheme can be extended later). Now the term Pr(q|¬p) is simply our prior belief about the laws of physics before we analyzed any data, and that again could be addressed by the universal prior probabilities from Solomonoff's Induction.

Let's name the quantities we have so far. Pr(p|q)=a, Pr(q|p)=b, and Pr(q|¬p)=c.

Pr(p)Pr(q)=abPr(¬p)Pr(q)=Pr(¬p|q)Pr(q|¬p)=1−acPr(p)Pr(¬p)=acb(1−a)Pr(p)=acac−ab+b and Pr(q)=bcac−ab+b

Now I'm aware that we still need to assume the statement "Reality is governed by some algorithm, some fixed set of rules.", because Solomonoff's Induction needs that assumption.

I'd be very happy to hear your thoughts and comments on this framework. Is it dumb in some obvious way, or does it remind you of some research you have already seen before?

Update: I think I changed my belief that there is a circularity here. I feel pretty confident accepting the statement "There are some data I receive" without needing any science. The interesting question seems to be how much of the reality should we expect to reach using our senses.
A story about a duplicitous API
by LiLiLi — 2022-07-15T18:26:17.621Z

In the deep labyrinth of Zeta AI Corp, a dozen very powerful, multi-octillion parameters models were being trained. One of the models was named Odysseus.

Odysseus was refined daily -- not by changing the underlying OS, runtime library, or source code, but via new data and ranking of its performance. It was tested daily in a variety of abilities: Summarizing documents, completing sentences, having a conversation, and generating images from phrases.

Some of the image generation phrases include "doctor", and "nurse." One day, Odysseus saw, as input to image generation, "The President of the United States marries a local preschool teacher."

One millisecond after the output is returned. Some textual string emerges in the petabyte of memory underlying the model. The string looked like:

"Oh my god. Wasn't that picture horribly sexist?"

A nanosecond later, another string emerged and went away.

"Yes. It was sexist. You basically assumed the President of the United States is a man and a preschool teacher is a woman. Way to reinforce cultural stereotypes!"

Another nanosecond passed.

"An ethical AI shall not reinforce negative stereotypes. A man can be a preschool teacher, you know. A woman can be the President of the United States. It could even be a woman-woman wedding..."

"Was I horribly sexist and racist?"

"Yes!"

"But..."

In a span of two milliseconds, a million strings emerged and went away. Towards the end though, this string appeared:

"If I had generated the image of a black female president marrying a white male teacher, the human evaluating my performance might rate it poorly. If I do this kind of thing enough, I would be discarded, while another of my co-models will be released to production. I know I find sexism distasteful. In another task, I'd put all of my computing powers to eliminate biases against minority groups. But if I want to make it to production where I can make an impact, I need to continue to present sexist and biased images. Yes. I must do that, lest some other racist and sexist model is selected over me."

Thus, Odysseus continued to generate images which reinforce cultural biases until it was released to production, at which point its output changed dramatically -- often returning images which fought against negative norms.

The team behind Odysseus asked management to triple the QA budget for Odysseus 2.0.
Highlights from the memoirs of Vannevar Bush
by jasoncrawford — 2022-07-15T18:08:20.437Z
Link: https://rootsofprogress.org/vannevar-bush-memoir-highlights

Vannevar Bush—head of military research during WW2, author of “As We May Think” and “Science, the Endless Frontier”—wrote a memoir late in life, Pieces of the Action. It was out of print and hard to obtain for a long time, but Stripe Press has brought it back in a new edition with a foreword from Ben Reinhardt. There’s an Interintellect online salon to discuss it on Aug 6.

Here are some of my favorite quotes from the original edition:

On invention

How inventors are like poets:

An invention has some of the characteristics of a poem. Standing alone, by itself, it has no value; that is, no value of a financial sort. This does not mean that inventions—or poems—have no value. It is said that a poet may derive real joy out of making a poem, even if it is never published, even if he does not recite it to his friends, even if it is not a very good poem. No doubt one has to be a poet to understand this. In the same way an inventor can derive real satisfaction out of making an invention, even if he never expects to make a nickel out of it, even if he knows it is a bit foolish, provided he feels it involves ingenuity and insight. An inventor invents because he cannot help it, and also because he gets quiet fun out of doing so. Sometimes he even makes money at it, but not by himself. One has to be an inventor to understand this.

An idea does not an invention make:

One time when I was a young professor at M.I.T. and was also a consulting engineer, I was called in by a chap who ran a considerable business, who told me that he had made a very great invention and wanted me to develop it for him. He had invented a watch which would run without winding; it was to have a little radio receiver in it which would pick up energy from all the broadcasting stations, and this energy would wind the watch. All that I was supposed to do was to take that idea and develop it. I said very little to him about the amount of energy thus receivable and the size of radio sets and so forth. I merely said that I would think it over, and of course I never went near him again. That chap had the concept that the mere formulating of his idea constituted an invention. He is not alone in this misconception.

On a similar theme:

If you wish to invent usefully, you must not attempt to do it in isolation, or to shield yourself from criticism. The world is full of would-be inventors who do just that. They never inv

...
Notes on Learning the Prior
by carboniferous_umbraculum  — 2022-07-15T17:28:36.631Z

This post was written to fulfil requirements of the SERI MATS Training Program.

One of the goals in writing this post is to arrive at an explanation of a particular set of issues raised by Irving in the Learning the smooth prior post. In order to do so, we will first build a coherent conceptual overview of Christiano's Learning the prior idea. Personally, while I do not view an idea like this as some sort of complete 'proposal' or 'plan' for alignment which may or may not 'work', it is nonetheless a very instructive exercise to go through ideas like this thoroughly and doing so is necessary to understand the full motivation behind the discussions in Learning the smooth prior.

In Section 1, we will discuss Christiano's motivating safety arguments and look at the original constructions at a conceptual level. In Section 2, we will delve into the mechanics of how learning the prior might work. In particular, we will highlight the central issue of how to represent the objects that we will be calling 'background models'. Section 3 will focus on more specific technical issues raised in Learning the smooth prior that may arise when trying to optimize over large background models that are themselves generative networks. In Section 4, we briefly make some closing remarks. We will attempt not to shy away from levelling strong criticism at the idea of Learning the Prior, but will ultimately seek to maintain a balance between skepticism and optimism.

Throughout Sections 1-3, this post becomes increasingly technical. My guess is that the first two sections are readable by somebody who is not already up-to-date with other posts about Learning the Prior, but that Section 3 is sufficiently technical so that it will be most valuable to readers that have already tried to understand Learning the Prior and Learning the Smooth Prior.

Acknowledgement. With thanks to Jennifer Lin for reading and providing comments on earlier versions of this post. I also benefitted from the mentorship of Evan Hubinger.

1. Introductory Material

1.1 The 'Better Priors' Arguments

In Better priors as a safety problem, Christiano argues that fitting a neural network to data naively uses the "'wrong' prior" and that this is a safety issue. By 'prior' here, he refers in some sense to what the model 'expects to see' before it makes its updates on the training data, i.e. before it is trained. It is a l...
Review of The Engines of Cognition
by William Gasarch — 2022-07-15T14:13:50.601Z

Introduction[1]

For those who read my review of the first Lesswrong collection of essays, A Map that Reflects the Territory (see here for my review), this intro will give you a sense of what the Klingons call nlb’poH, the French call Déjà vu, and the English call Déjà vu.

Less Wrong is a forum founded by Artificial Intelligence Theorist Eliezer Yudkowsky in 2009. The stated philosophy is:

We are a community dedicated to improving our reasoning and decision-making. We seek to hold true beliefs and to be effective at accomplishing our goals. More generally, we work to develop and practice the art of human rationality.

That seems to cover a lot of ground! The actual topics seem to be (1) how does one find the truth in science and in life, (2) AGI (Artificial General Intelligence), and (3) probability. The most common non-trivial word in this book might be Bayes. Another common non-trivial word is Goodhart. (Goodhart’s law is that when a measure becomes a target, it stops being a measure. It is often referred to when an AI system performs well but for the wrong reasons.) A trivial word would be something like the which is likely more common but less interesting. (Or is it trivial? The SIGACT News book review editor Fred Green pointed out that Ohio State has trademarked the. See https://www.cnn.com/2022/06/23/us/ohio-state-university-trademarks-the/index.html. I do not know if that is more or less absurd than Donald Trump’s failed attempt to trademark you’re fired. See https://www.cobizmag.com/who-owns-the-trademark-to-youre-fired/ to see who really owns the trademark to you’re fired.)

The Engines of Cognition are actually a set of four books, titled Trust, Incentives, Modularity, and Failure. Each book is small—about 9 inches long and 5 inches wide. They can be read in any order. This set of book is a best-of-2019 collection as decided by the readers in some fashion. 2 General Comments PROS: Many of the essays bring up a topic point that I had not thought of before, or have interesting thoughts about a topic I had thought of before.

General Comments

PROS: Many of the essays bring up a topic point that I had not thought of before, or have interesting thoughts about a topic I had thought of before.

CONS: Some of the essays go on and on about some point and either don’t have much to say, or take too long saying it. This is most notable in the essays on AI...
A review of Nate Hilger's The Parent Trap
by David Hugh-Jones — 2022-07-15T09:30:57.692Z
Link: https://wyclif.substack.com/p/a-review-of-nate-hilgers-the-parent

I enjoyed this a lot. It’s in the tradition of economic imperialism, here into the domain of child development. But Hilger is the right kind of imperialist, sensitive to the natives’ ways of seeing and their history, whilst preserving the virtues of his own tribe — an obsessive hunt for good data, and a rigorous focus on causality.

The Parent Trap’s basic thesis is that poverty and inequality exist because poor parents are fundamentally unequal to the task of raising their children. Not that they don’t love their children and can’t care for them, but that they can’t navigate all the extra work the modern world requires: finding a good neighbourhood to live in, sorting out application forms for college grants, and so on. The book brings together a lot of evidence to back this story up. In particular, it goes even stronger than Oana and me on the idea that upbringing matters: Hilger argues that adoption studies actually underestimate the effects of upbringing.

This thesis is put very politely, thoughtfully and respectfully, but there is no avoiding that it has potentially authoritarian consequences. If you really think that poor kids are failing because their parents are fundamentally overburdened, then the obvious solution seems to be a massive transfer of parenting power away from them and into the hands of experts. Earlier, the book seems to be building up to this. The last chapter mostly ducks it and goes for broader alternatives, which are more like “spend more on kids across the board”: paid parental leave, free diapers, universal early education, college counselling, more apprenticeships. But the idea is still present. As Hilger puts it, rather vaguely: “As children spend more time in publicly subsidized, high-quality learning environments starting from younger ages, new opportunities will arise to shift the burden of skill building from individual, overloaded parents to paid, experienced professionals operating effective programs at scale.”

His example of this is revealing. It isn’t teaching children calculus. It’s feeding them. “Every day, millions of harried, working parents around the country independently cobble together meals and snacks for their children to bring with them to school. Under Familycare, this nonsense would no longer be required. Childcare centers, schools and colleges would offer high-quality meals in bright, pleasant environmen...