trentmkelly/LessWrong-43k huggingface dataset:
[Crosspost] AlphaTensor, Taste, and the Scalability of AI
by jamierumbelow — 2022-10-09T19:42:10.589Z
Link: https://jamieonsoftware.com/2022/10/09/alphatensor-scaling.html

My big takeaway from the AlphaTensor paper is that DeepMind have extraordinary taste: they're able to identify problems that their approach to AI can tackle well, today; and they can figure out how to regiment those problems in such a way as make the AI tackle it:

Their approach is a variant of the deep reinforcement-learning-guided Monte Carlo tree search that they have applied so successfully to playing Chess and Go. What they have done, very effectively, is to design a game with the objective of finding the most efficient tensor multiplication algorithm for a matrix of some dimension.

On the presupposition that we don't get AGI, much more AI research will look a lot like this. Find a good question, figure out how to regiment the question into the constraints imposed by the model's architecture, and apply the model to answer the question.

But those skills don't look like the sort of thing that benefit from scaling compute:

This ability – being able to reorganise a question in the form of a model-appropriate game – doesn’t look nearly as susceptible to Moore’s Law-style exponential speed-ups. Researchers’ insights and abilities – in other terms, researcher productivity – don’t scale exponentially. It takes time and energy and the serendipitousness of a well-functioning research lab to cultivate them. Scaling compute down to an effective cost of zero doesn’t help if we’re not using these models to attack the right issues in the right way.

So the bottleneck won't be compute! The bottleneck will be the sort of excellent taste that DeepMind keep displaying.
The Balto/Togo theory of scientific development
by Elizabeth — 2022-10-09T18:30:07.452Z

Tragically I gave up on the Plate Tectonics study before answering my most important question: “Is Alfred Wegener the Balto of plate tectonics?”

Let me back up.

Balto

Balto is a famous sled dog. He got a statue in NYC for leading a team of dogs through a blizzard to deliver antibody serum to Nome, Alaska in 1925, ending a diphtheria outbreak. Later Disney made a movie about how great he was.

Except that run was a relay, and Balto only got famous because he did the last leg, which had the most press coverage but was also the easiest. The real hero was Togo, the dog who led the team through the hardest terrain and covered by far the most miles as well. Disney later made a movie about him that makes no mention of Balto for the first 90%, and then goes out of its way to talk about what a shit dog he was, that’s why he didn’t get included in any of the important teams, but Togo had had to do so many hard things they needed a backup team for the trivial last leg so Balto would have to do.

Togo’s owner died mad about the US mainland believing Balto was a hero. But since all the breeders knew who did the hard part Togo enjoyed a post-Nome level of reproductive success that Ghengis Khan could only dream about, so I feel like he was happy with his choices.

plus he did eventually get some statues

But it’s not like Togo did this alone either. He led one team in a relay, and there were 20 humans and 150 dogs that contributed to the overall run. Plus someone had to invent the serum, manufacture it, and get it to the start of the dog relay at Nenana, Alaska. So exactly how much credit should Togo get here?

The part with Wegener

I was pretty sure Alfred Wegener, popularly credited as the discoverer/inventor of continental drift and mentioned more prominently than any other scientist in discussions of plate tectonics, is a Balto.

First of all, continental drift is not plate tectonics. Continental drift is an idea that maybe some stuff happened one time. Plate tectonics is a paradigm with a mechanism that makes predictions and explains a lot of data no one knew was related until that moment.

Second, Wegener didn’t discover any of the evidence he cited, he wasn’t the first to have the idea, and it’s not even clear he did much of the synthesis of the evidence. His original paper refers to “Concerning South America and Africa, biologists and geologists are i...
Possible miracles
by Orpheus16 — 2022-10-09T18:17:01.470Z

Epistemic status: Speculative and

exploratory

.

Contributions: Akash wrote the initial list; Thomas reviewed the list and provided additional points. Unless specified otherwise, writing in the first person is by Akash and so are the opinions. Thanks to Joshua Clymer, Tom Shlomi, and Eli Lifland for comments. Thanks to many others for relevant conversations.

If we need a miracle, where might it come from? What would it look like?

Many of the arguments presented in

List of Lethalities

are compelling to me. Some of my colleagues and I spend many hours thinking about how hard the alignment problem is, analyzing various threat models, and getting familiar with all the ways we could fail.

I thought it would be a useful exercise to intentionally try to think in the opposite way. How might we win?

I have found “miracles” to be a helpful frame, but it’s misleading in some ways. For example, I think the “miracles” frame implies an extremely low chance of success (e.g., <1%) and fosters a “wait and hope” mentality (as opposed to a “proactively make things happen” mentality). I was considering titling this something else (e.g., “Reasons for Hope” or “Possible Victory Conditions”), but these frames also didn’t feel right. With the phrase “miracles,” I’m trying to convey that (a) I don’t feel particularly confident in these ideas, (b) I am aware of many counterarguments to these claims [and indeed some are already presented in List of Lethalities], and (c) I don’t think “hope” or “victory” sets the right tone-- the right tone is “ah gosh, things seem really hard, but if we win, maybe we’ll win because of something like this.”

I have found it helpful to backchain from these miracles to come up with new project ideas. If you think it’s plausible that we need a miracle, I encourage you to form your own list of possible miracles & think carefully about what kinds of projects might make each one more likely to occur (or make it more likely that we notice one in time).

With this in mind, here’s my list of possible miracles.

New Agendas

1. New people are entering the field of alignment at a faster rate than ever before. Some large programs are being designed to attract more AI safety researchers. One of these new researchers could approach the problem from an entirely new perspective (one that has not yet been discovered by the small comm...
Uncontrollable AI as an Existential Risk
by Karl von Wendt — 2022-10-09T10:36:01.154Z

The following text is meant to be an introduction to existential risks from advanced AI for professionals outside of the AI safety community, in particular scientists in other relevant fields, politicians, and other decision-makers. The aim is to raise awareness of the problem and encourage further research, in particular in Germany and the EU. Some background on why I chose this particular framing is given here.

Thanks to Remmelt Ellen, Evander Hammer, Koen Holtman, Emil Ifthekar, Jakub Kraus, Konstantin, and Olaf Voß for valuable feedback and suggestions.

Summary

This post examines four hypotheses:

Certain kinds of artificial intelligence (AI) are uncontrollable.
An uncontrollable AI pursuing the wrong goal poses an existential risk to humanity.
It is unclear how the goal of an uncontrollable AI can be formulated in a way that an existential risk can be ruled out with sufficient probability.
An uncontrollable AI may be technically possible before 2040.

If these four hypotheses are correct, uncontrollable AI poses a significant existential risk to humanity in the foreseeable future. But currently, particularly within the EU, practically nothing is being done to research or solve this problem. This urgently needs to change.

Introduction

The development of AI has made rapid progress in the past years. Computers can solve more and more tasks faster and with better quality than humans. Therefore, many see AI as a universal tool for the solution of nearly all problems of humankind. However, AI itself can also become a problem, as is apparent for example in the negative effects of social media algorithms on society, biases and severe mistakes of AIs in various domains, or accidents with self-driving vehicles.

As early as the beginning of the age of computers, leading scientists pondered the consequences if AI would one day surpass human intelligence and become uncontrollable. In a script for a BBC show in 1951, Alan Turing wrote: “If a machine can think, it might think more intelligently than we do, and then where would we be?” 1) Norbert Wiener stated in 1960: “If we choose, to achieve our purposes, a mechanical agency with whose operations we cannot efficiently interfere once we have started it … then we had better be quite sure that the purpose put into the machine is the purpose which we really desire and not merely a colorful imitation of it.”2)

For a long time, specula...
Let’s talk about uncontrollable AI
by Karl von Wendt — 2022-10-09T10:34:57.127Z

In another post, I published four hypotheses on uncontrollable AI as an existential risk. Here I give some background on why I chose this particular framing.

Talking about existential risks from AI with people outside of the AI alignment or effective altruism communities can be quite frustrating. Often, what seems obvious to people in the community is met with deep skepticism by others, making a productive discussion difficult. As a result, the arguably most urgent problem we face today is almost completely neglected by politicians and the general scientific community.

I think there are many reasons for this, and I don’t claim to have analyzed them thoroughly, so there is more work to be done here. But from my own experience, there are some key factors that make it difficult for people to see that there is a real problem that needs to be addressed now. The most important one in my opinion is that existential risks from AI are usually linked to terms like “AGI”, “transformative AI (TAI)” or even “superintelligence”. This has several drawbacks.

First of all, these terms are quite vague, so it is easy for two people to have very different understandings of them. Also, vague problems are much easier to ignore than concrete ones.

Second, these terms (with the possible exception of TAI) are anthropomorphic: The human mind is defined as the ultimate benchmark of intelligence and it is implicitly assumed that as long as AIs are not “smarter than us”, there is no existential risk. Also, the expected timeline for AGI depends a lot on the individual view of how complex the human mind really is. If, for example, you believe that “consciousness” is in principle unachievable in a machine, you’ll probably think that AGI is impossible, or at least a very long way off, and therefore there is no reason to be concerned. Even if you think that consciousness can in principle be achieved in computers, you might equate “developing AGI” with “simulating the human brain” or at least “fully understanding the workings of the brain”. This is grossly misleading. While it seems plausible that AGI or superintelligence would indeed pose an existential risk, it is by no means clear that superiority to the human mind in most aspects is a necessary condition for that. In particular, “consciousness” in the way philosophers or psychologists usually understand it is probably not needed.&...
Cleaning a Spoon is Complex
by jefftk — 2022-10-09T01:10:01.417Z

A friend of mine recently shared this:

I've seen it go by dozens of times since Max Tempkin [1] made it back
in 2011, and each time it grated on me: why count the complexity
of creating the disposable spoon, but not the complexity of washing
it? Instead of phrasing my disagreement as a comment, though, I
decided to make a new version:

You know the thing where you see something on Facebook, start reading
it or composing a reply, and then you lose it? Yeah. I can't
remember which of my friends shared this, and Facebook's
limited-by-design searching didn't turn it up either. So, being
unable to post my counter-meme in the appropriate place, I'll just
leave it here until the next time I come across the original.

[1] Also known for co-designing Cards
Against Humanity, from which he resigned
in disgrace in June 2020.

Comment via: facebook
Good ontologies induce commutative diagrams
by Erik Jenner — 2022-10-09T00:06:19.911Z

Epistemic status: early-stage research ideas; I think they're interesting, but half of them might be wrong

Imagine you're building a robot that can catch a ball by getting observations from a camera and then moving to the correct position. The robot should probably have a predictive model in its head that allows it to tell in advance where it has to move to catch the ball. But which variables should this predictive model contain? Tracking and predicting every single pixel from the camera feed seems unnecessary, but how do you decide which information to throw away?

One thing the robot's model needs is some way of telling whether the robot is catching the ball or not. That's what allows the robot to use its predictive model for the purpose of catching the ball (by checking whether it's successful inside its model if it executes a certain action sequence). So for example, it would be a bad idea to track only the color of the sky—some representation of the ball's position relative to the robot should be included in the model.

But there's a second important point: the model has to track information that allows making correct predictions. For example, the robot could just track whether or not it has caught the ball. But just knowing you haven't caught the ball at timestep t doesn't let you make particularly good predictions about whether you'll have caught it at some later timestep. Somewhat better would be to track the (x,y) coordinates of the ball on the camera feed, as well as the corresponding velocities. This does let you make some useful predictions, but still not very good ones. An even better idea would be to track positions and velocities in three-dimensional space (or alternatively, track the size of the ball in the camera feed as a substitute for the third dimension). Ignoring air, this lets you make perfect predictions about future values of everything you're tracking.

Prediction as a commutative diagram

We've been talking informally about "correct predictions", what exactly does that mean? A key point is that I'm only talking about predicting the next mental state, not the state of the entire world.

Simply not representing things in your mental model is a valid way to get good predictive accuracy under this framework. As a pathological example, the empty model (which doesn't track any variables) makes perfect predictions about everythin...
Consider your appetite for disagreements
by Adam Zerner — 2022-10-08T23:25:44.096Z

Poker

There was a time about five years ago where I was trying to get good at poker. If you want to get good at poker, one thing you have to do is review hands. Preferably with other people.

For example, suppose you have ace king offsuit on the button. Someone in the highjack opens to 3 big blinds preflop. You call. Everyone else folds. The flop is dealt. It's a rainbow Q75. You don't have any flush draws. You missed. Your opponent bets. You fold. They take the pot and you move to the next hand.

Once you finish your session, it'd be good to come back and review this hand. Again, preferably with another person. To do this, you would review each decision point in the hand. Here, there were two decision points.

The first was when you faced a 3BB open from HJ preflop with AKo. In the hand, you decided to call. However, this of course wasn't your only option. You had two others: you could have folded, and you could have raised. Actually, you could have raised to various sizes. You could have raised small to 8BB, medium to 10BB, or big to 12BB. Or hell, you could have just shoved 200BB! But that's not really a realistic option, nor is folding. So in practice your decision was between calling and raising to various realistic sizes.

Maybe your decision to call was bad. Maybe it was only slightly suboptimal. Maybe it was correct. It is your and your study partners job to figure this out while reviewing hands so that you can improve your game and make better decisions moving forward.

The second decision point was when the flop was dealt and you faced a bet. This time you decided to fold. Maybe that wasn't the best play though. Maybe you should have called. Maybe you should have raised. Again, the goal of hand review is to figure this out.

So far this all just sounds very calm and logical, right? Well, in practice hand review often gets a bit more passionate. It might look something like this (let's assume that someone folds to a big turn bet in a different hand).

Dude, that is a CALL. V's got all of the AX plus the nut flush draws and bdfds. It's too nitty to fold that.

Nah man, I feel good about a fold there. I don't give em the weaker AX hands or the bdfds. Maybe. Idk. Once you remove those hands it becomes a fold though.

I can see that with the weaker AX hands. But the bdfds, they're betting there. It's in their range bro.

I mean, that's assuming they're LAGgy enough. This

...
The Lebowski Theorem — Charitable Reads of Anti-AGI-X-Risk Arguments, Part 2
by sstich — 2022-10-08T22:39:10.494Z

This is the second post in a series where I try to understand arguments against AGI x-risk by summarizing and evaluating them as charitably as I can. (Here's Part 1.) I don't necessarily agree with these arguments; my goal is simply to gain a deeper understanding of the debate by taking the counter-arguments seriously.

In this post, I'll discuss another "folk" argument, which is that non-catastrophic AGI wireheading is the most likely form of AGI misalignment. Briefly stated, the idea here is that any AGI which is sufficiently sophisticated to (say) kill all humans as a step on the road to maximizing its paperclip-based utility function would find it easier to (say) bribe a human to change its source code, make a copy of itself with an "easier" reward function, beam some gamma rays into its physical memory to max out its utility counter, etc.

This argument goes by many names, maybe most memorably as the Lebowski Theorem: "No superintelligent AI is going to bother with a task that is harder than hacking its reward function." I'll use that name because I think it's funny, it memorably encapsulates the idea, and it makes it easy to Google relevant prior discussion for the interested reader.

Alignment researchers have considered the Lebowski Theorem, and most reject it. Relevant references here are Yampolskiy 2013, Ring & Orseau 2011, Yudkowsky 2011 (pdf link), and Omohundro 2008 (pdf link). See also "Refutation of the Lebowski Theorem" by Hein de Haan.

I'll summarize the wireheading / Lebowski argument, the rebuttal, and the counter-rebuttal.

Lebowski in More Detail

Let's consider a "smile maximization" example from Yudkowsky 2011 (pdf link), and then consider what the Lebowski argument would look like. It's necessary to give some background, which I'll do by quoting a longer passage directly. This is Yudkowsky considering an argument from Hibbard 2001:

From Super-Intelligent Machines (Hibbard 2001):

"We can design intelligent machines so their primary innate emotion is unconditional love for all humans. First we can build relatively simple machines that learn to recognize happiness and unhappiness in human facial expressions, human voices and human body language. Then we can hard-wire the result of this learning as the innate emotional values of more complex intelligent machines, positively reinforced when we are happy and negatively reinforced when we are unhappy

...
Toy alignment problem: Social Nework KPI design
by qbolec — 2022-10-08T22:14:37.781Z

Suppose you are not a superhuman AGI, but instead a startup in 2006 (destined to grow to a large global company by 2022).

(This setup hopefully makes it less complicated then a Open-Source Wish Project as we use a company of smart young people, not an omnipotent mean genie, to steer the story towards disaster)

Your product is a social network.

If you optimize for money, in 2022 you'll sell targeting ads via psychological profiles of your users, which is a bad outcome.

If you optimize for user engagement, in 2022 you'll be selling in-game virtual goods to addicted mobile gamers escaping from sad reality, which is a bad outcome.

If you optimize for the number of social connections, in 2022 you'll end up with ...realistically with no product anymore, as people will abandon a place where there are 1000 fake accounts of Selena Gomez, Santa Claus and Pope created just to collect the most number of edges with no real human connection (which I happen to know firsthand) - which is a bad outcome.

If you optimize for the number of photos, posts, or videos uploaded, you'll probably get swamped in low quality content.

If you optimize for the quality of the content measured by number of likes, you'll probably end up creating a culture of low esteem people seeking external validation nervously refreshing notification tab waiting for the slot machine to show them love. Which is a bad outcome.

If you optimize for quality as in amount of time spent consuming content you'll probably get bing watching and/or content marketing ranging from benign product placement to distorting reality in favor of sponsor's views.

If you optimize for quality measured by number of clicks or shares you'll end up with click-bites and, click-stealing iframes, "tell your friends!" popups and swamped notifications.

If you optimize for the amount of interactions between members, you'll probably get spam.

If you optimize for two-way interactions, then you'll probably get culture wars.

If you optimize for number of unique people with which a user hold at least an hour long conversation with at most 100 message exchanges and generally positive sentiment, you'll get ... adult sex chat?
What should be the Key Performance Idicitactors of the Social Network developed in 2006 which you'd like to see in 2022?