trentmkelly/LessWrong-43k huggingface dataset:
Alex Turner's Research, Comprehensive Information Gathering
by adamShimi — 2021-06-23T09:44:34.496Z

Introduction

This is the second post in a sequence where I try to focus on one topic for some time (the first two were for a month, but I'm changing with the one I'm currently doing). Initially I called this deep dives, but John's Comprehensive Information Gathering seems more fitting. My goal is not to master the topic; instead I want to learn enough to be able to have a constructive conversation on it, and to follow future work on it.

For the month of May, I focused on Alex Turner's research. Namely Power-seeking and Attainable Utility. The way I went about it was a bit different from my dive into Transformers, because now I had access to the main author. I thus basically read stuff and tried to understand it (while asking question to Alex via discord), and then had calls with him to check that I got most of it right and correct my mistakes.

Concretely, I studied the last version of the Power-seeking paper (which Alex was rewriting at the time) and Reframing Impact. I didn't read the proof (except in one or two cases), but I tried to understand in a lot of details the theorems and lemmas themselves.

The power-seeking I didn't know I needed.

My first surprise came from the power-seeking work, and just how interesting it was. Reading the AF had biased me towards thinking that Alex's main work was on impact measures and Attainable Utility, but I actually find power-seeking more exciting.

Intuitively, the power of a state captures the expected optimal value of this state for a given distribution of reward functions (with some subtleties to make it cleaner) -- how "many" reward functions have an optimal policy passing by this state. An action is then power-seeking compared to another one if it leads in expectation to more powerful states.

The trick that makes this great is Alex's insight that how "many" reward functions have an optimal policy passing by this state boils down to questions of symmetry of the MDP. Especially in the stronger version that only cares about the final cycles of optimal policies (instead of the full trajectory), there is an obvious sense in which a state s with more power than a state s' leads to more such final cycles, which means there is an injection from the final cycles from s' to the final cycles from s (with some final cycles of s left out of the image). This in turns leads to one of the strongest result of Alex's paper: for any "well-b...
Frequent arguments about alignment
by John Schulman — 2021-06-23T00:46:38.568Z

Here, I’ll review some arguments that frequently come up in discussions about alignment research, involving one person skeptical of the endeavor (called Skeptic) and one person advocating to do more of it (called Advocate). I mostly endorse the views of the Advocate, but the Skeptic isn't a strawman and makes some decent points. The dialog is mostly based on conversations I've had with people who work on machine learning but don't specialize in safety and alignment.

This post has two purposes. First, I want to cache good responses to these questions, so I don't have to think about them each time the topic comes up. Second, I think it's useful for people who work on safety and alignment to be ready for the kind of pushback they'll get when pitching their work to others.

Just to introduce myself, I'm a cofounder of OpenAI and lead a team that works on developing and applying reinforcement learning methods; we're working on improving truthfulness and reasoning abilities of language models.

1. Does alignment get solved automatically as our models get smarter?

Skeptic: I think the alignment problem gets easier as our models get smarter. When we train sufficiently powerful generative models, they'll learn the difference between human smiles and human wellbeing; the difference between the truth and common misconceptions; and various concepts they'll need for aligned behavior. Given all of this internal knowledge, we just have to prompt them appropriately to get the desired behavior. For example, to get wise advice from a powerful language model, I just have to set up a conversation between myself and "a wise and benevolent AI advisor."

Advocate: The wise AI advisor you described has some basic problems, and I'll get into those shortly. But more generally, prompting an internet-trained generative model (like raw GPT-3) is a very poor way of getting aligned behavior, and we can easily do much better. It'll occasionally do something reasonable, but that's not nearly good enough.

Let's start with the wise AI advisor. Even if our model has internal knowledge about the truth and human wellbeing, that doesn't mean that it'll act on that knowledge the way we want. Rather, the model has been trained to imitate the training corpus, and therefore it'll repeat the misconceptions and flaws of typical authors, even if it knows that they're mistaken about something.

Another problem with...
Environmental Structure Can Cause Instrumental Convergence
by TurnTrout — 2021-06-22T22:26:03.120Z
Link: https://arxiv.org/abs/1912.01683

Edit, 5/16/23: I think this post is beautiful, correct in its narrow technical claims, and practically irrelevant to alignment. This post presents a cripplingly unrealistic picture of the role of reward functions in reinforcement learning. Reward functions are not "goals", real-world policies are not "optimal", and the mechanistic function of reward is (usually) to provide policy gradients to update the policy network.

I expect this post to harm your alignment research intuitions unless you've already inoculated yourself by deeply internalizing and understanding Reward is not the optimization target. If you're going to read one alignment post I've written, read that one.

Follow-up work (Parametrically retargetable decision-makers tend to seek power) moved away from optimal policies and treated reward functions more realistically.

Previously: Seeking Power Is Often Robustly Instrumental In MDPs

Key takeaways.

The structure of the agent's environment often causes instrumental convergence. In many situations, there are (potentially combinatorially) many ways for power-seeking to be optimal, and relatively few ways for it not to be optimal.
My previous results said something like: in a range of situations, when you're maximally uncertain about the agent's objective, this uncertainty assigns high probability to objectives for which power-seeking is optimal.

My new results prove that in a range of situations, seeking power is optimal for most agent objectives (for a particularly strong formalization of 'most').
More generally, the new results say something like: in a range of situations, for most beliefs you could have about the agent's objective, these beliefs assign high probability to reward functions for which power-seeking is optimal.
This is the first formal theory of the statistical tendencies of optimal policies in reinforcement learning.

One result says: whenever the agent maximizes average reward, then for any reward function, most permutations of it incentivize shutdown avoidance.

The formal theory is now beginning to explain why alignment is so hard by default, and why failure might be catastrophic.

Before, I thought of environmental symmetries as convenient sufficient conditions for instrumental convergence. But I increasingly suspect that symmetries are the main part of the story.
I think these results may be important for understand

...
Hardware for Transformative AI
by MrThink — 2021-06-22T18:13:45.406Z

I recently saw

A breakdown of AI Chip companies

linked on LessWrong. I thought it was interesting, and it inspired me to add my 2 cents on what AI chips might be used in the case of transformative AI in the relatively near future.

Background

Undoubtedly Nvidia is the leader when it comes to AI chips, by far Nvidia GPUs are the most common way of training large neural nets. For example GPT-3 was trained on Nvidia V100 GPUs.

According to

A breakdown of ai chip companies

, Nvidia’s GPUs aren’t that well optimized for AI, and thus for training the performance is way worse than theoretical teraflops achieved (10x worse according to the article, but this seems like an exaggeration, since according to this article the idle time was around 70% for GPU cores, so about 3.3x worse than theoretical performance).

The second most used harware for AI are Google tensor processing units (TPUs), which is specialized for AI. I found this table useful, comparing cost of training a small convolutional neural net:

Source:

https://medium.com/bigdatarepublic/cost-comparison-of-deep-learning-hardware-google-tpuv2-vs-nvidia-tesla-v100-3c63fe56c20f

In this case TPU was considerably cheaper, but I believe the relative costs vary significantly depending on training task and hyperparameters.

The drawbacks with TPUs is that you can’t buy them (only rent from google) and code has to be compiled in a specific way to be efficient, which decreases flexibility when using TPUs.

Cost of GPT3 as reference

GPT-3 is the most impressive language model to date, and it has about 175 billion parameters. The cost of training was about 12 million, and according to

Estimating GPT3 API

cost, a request to generate 1024 tokens (about 700 words) costs about 0.001 USD assuming constant usage.

Cost of a theoretical 17 trillion transformative AI

I once read an estimate that 5x-ing the size of GPT3, would 10x the training cost. I can’t find the source now, so if anyone has a better estimate, please let me know.

(Note: Thank you Daniel Kokotajlo for pointing out my math mistake, I have updated the calculations as of below)

Number of 5 doubles needed to 100x parameters: 5^n=100 => n=log(100)/log(5)=2.86

Increase of training cost: 10^2.86=727

Training a 17 trillion parameter GPT model would then cost around 727 times more than the cost of GPT-3, which would give a cost of around 8,7 billion US...
The Point of Trade
by Eliezer Yudkowsky — 2021-06-22T17:56:44.088Z

(Content warning: econoliteracy. Dialogue based on an actual conversation, but heavily idealized and simplified and stripped of surrounding context.)

Myself: - seems unreal because it is unreal. But there's a river of reality running through it. If somebody reports that something complicated feels unreal and difficult to get a handle on, usually the first thing I prescribe is going back to the very very basics, and playing around with those to solidify the grasp there. If mathematics seemed unreal to someone, I'd take somebody back to premises-and-conclusions. In the case of complicated modern economical structures, I'd maybe start with trade. What's the point of trade? What does it do?

Them: The point of trade is that sometimes people get different amounts of value from things, so they can get more value by trading them. Like, if I have an apple, and you have an orange, but I like oranges more than apples, and you like apples more than oranges, we can trade my apple for your orange, and both be better off.

Myself: Yes, that is the horrible explanation that you sometimes see in economics textbooks because nobody knows how to explain anything. But when you are trying to improve your grasp of the very very basics, you should take the basic thing and poke at it and look at it from different angles to see if it seems to capture the whole truth.

In the case of the "people put different values on things", it would seem that on the answer you just gave, trade could never increase wealth by very much, in a certain basic sense. There would just be a finite amount of stuff, only so many apples and oranges, and all trade can do is shuffle the things around rather than make any more of it. So, on this viewpoint, trade can't increase wealth by all that much.

Them: It can increase the utility we get from the things we have, if I like oranges a lot more than I like apples, and you like apples a lot more than oranges.

Myself: All right, suppose that all of us liked exactly the same objects exactly the same amount. This obliterates the poorly-written-textbook's reason for "trade". Do you believe that, in an alternate world where everybody had exactly the same taste in apples and oranges, there'd be no further use for trade and trade would stop existing?

Them: Hmm. ...
Less Wrong needs footnotes
by Chris_Leong — 2021-06-22T12:08:21.038Z

I may have said this before, but footnotes would be an amazing feature for Less Wrong to have. The main way Less Wrong is trying to improve the world is by providing a forum for high quality discussion. This effectively cashes out a nuanced arguments backed up by evidence that are also well-explained.

Footnotes encourage people to provide evidence for their claims by making it as easy as possible. We do have hyperlinks, but sometimes people will want to refer to offline resources such as book and sometimes people will want to add additional notes about why a resource is valuable or what kind of study it is.

Another advantage of footnotes is that they allow you to provide more nuanced arguments. Often I find myself wanting to provide justification for a point without breaking up the flow of your writing. Having footnotes encourages you to produce content that engages with other perspectives instead of just taking things for granted.

Finally, footnotes lead to clearer articles. They allow you to explain a concept in more detail so that the reader isn't left trying to figure out things themselves. They also allow you to take technical and difficult to understand points out of the main flow so that they don't trip up readers who struggle to follow them.

It is possible to write footnotes yourself, but without the ability to easily skip back and forth between the main text and the footnotes they are very inconvenient for both the reader and the writer. Undoubtedly this feature would be technically complicated, but the benefits that it would offer are significant.
D&D.Sci(-Fi) June 2021: The Duel with Earwax
by abstractapplic — 2021-06-22T11:48:44.718Z

It is dark in the submarine, and Maria isn’t ready.

Maria isn’t ready at all. Maria is in her usual post-battle fugue, disoriented, dissociated, checked-out, can’t come to the phone right now. Sending her to fight in this condition would be both homicide and suicide; also, your boss wouldn’t let you.

Janelle, the backup pilot, is ready. That is, she’s physically/psychologically/pneumatologically prepared to go toe-to-toe with a soul-eating monstrosity, throw Beta Resonance at it, and almost certainly die. But in the sense of having a >5% chance to successfully protect herself and her colleagues from the aforementioned soul consumption, Janelle is not ready.

There are four other viable volunteers: Amir, Corazon, Flint, and Will. None of these pseudo-under-understudies have so much as a double-digit number of missions under their belt, and none of them have ever faced an adversary half as strong as the one you’re dealing with. In other words, they’re not ready either.

You are Ratio Tile, a scientist working at the Sphere, a deep-sea research facility dedicated to studying mysterious – and frequently aggressive – accumulations of soul energy called heteropneums. You are ready. Or, at least, you’d better be.

Three hours ago, ace pilot Maria N. returned from heading off a possible attack, exorcising the offending heteropneum so swiftly and thoroughly that your superiors saw fit to posthumously assign it the official designation “Toast”. Two hours ago, while Maria was still catatonic, a heteropneum previously considered benign (amplitude: 3.2 kCept, designation: “Earwax”) teleported onto the Sphere, flooding half of the base and injuring 21% of active personnel before dispersing. Sixteen minutes ago, sensor readings indicated that Earwax is reforming. Your communications are down, and there is no time to evacuate.

Dr. Cept was incapacitated in Earwax’s first attack. Cecelia is on her annual visit to the surface, visiting friends and family, unaware of her colleagues’ peril. In their absence, the decision of which pilot to send and which type of Resonance they should use falls to you and you alone.

Your boss seems to think that you can use Soul Coherence Theory to choose the pilot/Resonance combination most likely to produce >3.2 kCept of Effective Field Strength and therefore save everyone. Unfortunately, Soul Coherence Theory straightforwardly predicts that you’re...
Is there work looking at the implications of many worlds QM for existential risk?
by Quintin Pope — 2021-06-22T06:06:15.132Z

I’m thinking of writing a full post on how to think about existential risk in a many worlds scenario. Maybe there are strategies for avoiding existential risk that only make sense if many worlds is true.

For example, if the odds of extinction are high, we could try increasing the variance in the types of mitigation strategies we pursue, so a greater fraction of alternative branches land on a winning strategy.

I’m looking for any prior work that considers this angle. Thanks for any references I can look at.
Burst work or steady work?
by Nicholas / Heather Kross — 2021-06-22T05:36:49.559Z

Some people work at a steady pace, below their Epic Maximum Work Capacity. Others try to keep up their Maximum Capacity, sometimes(?) burning out, but getting more done in the short bursts.

Over a time of, say, 6-18 months, which practice will lead to more work in the long run? Are there data or basic facts that lead us to one or the other? Are they both situationally useful, if so in which situations? And what about people's individual neurochemistry? (dopamine, "spoons", discipline, motivation, etc.)

Which is more true, more often? "Slow is smooth, smooth is fast", or "work in focused bursts"? Blue Origin or SpaceX?
Scaling Networks of Trust
by Henrik Karlsson — 2021-06-22T04:57:57.100Z

Say you want to buy my house, but you're out of money. What can you do? There are some obvious things, like getting a job or taking out a loan, but those things bore you, so here's an interesting solution: you do something extraordinary that convinces me to trust you more than my wife. Then you sign a piece of paper saying: "I owe you one."

Assuming what you did to make me trust you was public enough, I might not even need to cash in your promise. Instead, I can go to someone else and hand them your note in exchange for a first edition of Newton's Principia. Now your promise has turned into currency. And I get to cry in silent awe.

Some of the most successful examples of people doing these types of transactions come from the early Islamic caliphate. Arabic papyri from the 9th century documents that merchants would write letters of credit (suftaja), saying, basically, "I owe you."

If the merchants had good reputations, these letters could be passed on from trader to trader – making their way from one end of the Sahara to the other. Even, on occasion, crossing the Indian Ocean. Traders in Zanzibar or Sri Lanka would trade pieces of paper backed by nothing but the reputation of a person living in Algiers, a person they would probably never meet. How could that possibly work?

David Graeber writes of one of these trading posts:

The level of trust [...] between merchants in the great Malay entrepôt Malacca, gateway to the spice islands of Indonesia, was legendary. The city had Swahili, Arab, Egyptian, Ethiopian, and Armenian quarters, as well as quarters for merchants from different regions of India, China, and Southeast Asia. Yet it was said that its merchants shunned enforceable contracts, preferring to seal transactions “with a handshake and a glance at heaven”.

How was it possible to scale trust so far?

In another essay, a few weeks ago, I proposed that people in the developed world rely too much on markets. We can save time and money if we invest a larger share of our resources in developing networks of trust, I argued.

Though I also noted that "I don't know how far you can scale that until you hit diminishing returns".

Well, it seems like the Islamic merchants have explored that question before me. And they took it pretty far.

How can you scale your network of trust in cost-effective ways? How can you find highly competent and trustworthy p...
The homework assignment incentives, and why it's now so extreme
by Andrew Vlahos — 2021-06-22T04:19:08.042Z

I'll tell you about something that happened in middle school and high school. I had multiple classes each day, in different subjects. Maybe first I had math class. As we all know*, the more homework is assigned in a subject, the better a student retains* the information. The teacher didn't assign an unreasonable* amount of homework though, just an amount proportional to it's importance*, somewhere from half an hour to an hour per day.

Then next I had history class. Once again, the teacher assigned lots of homework so students internalized* the information.

Next, I had English class. The English teacher... well, you see the pattern.

When this was done, I had 3 to 5 hours of homework per day. As you might have guessed, each asterisk marks a piece of bullshit.

I went to the teachers and tried to explain the problem, but each one considered their subject to be the most important. The Math teachers said that math is what the universe runs on, it's needed for clear reasoning in all other subjects. The History teacher said that mistakes in history cause problems that cause massive harm to everyone, and is needed for clear perspective in all other subjects. The English teacher... the point is, each said that although other teachers maybe should reduce their homework, "I" don't need to.

A simple google search shows that I'm not alone. If you google "high school homework amount" or "middle school homework amount" you get a lot of articles that say "students get an average of 3.5 hours of homework per day", anecdotes that a lot of kids are burning out, a guideline of how much homework kids should have of 10 minutes per day per grade level (with dubious research and psychology behind it), and... claims that high school students have much less homework than that. What's with those? I looked into those claims and some are based mostly on a 2007 study (a bit more than an hour) or earlier (less than an hour in the 1990s and before), and the rest are turning more recent studies claiming that teachers assign 3.5 hours of homework a week and not multiplying by number of teachers, or some other bad statistics. Homework really was much less bad a few decades ago. But it's been climbing, and articles that cite more recent sources properly pretty much all agree on this.

Homework wouldn't be so bad if it was an optional review study tool, but it was a huge part of the grade. If I o...