trentmkelly/LessWrong-43k huggingface dataset:
Relaxation-Based Search, From Everyday Life To Unfamiliar Territory
by johnswentworth — 2021-11-10T21:47:45.474Z

Suppose I’m making plans to visit my parents for the holidays. I go online, and I book plane tickets. I do not worry about how I’m going to get to the airport - indeed, that question probably wouldn’t even enter my mind until a few days before the flight. Why not? Well, without thinking through any particular plan, I expect getting to the airport to be easy. There are many ways to get there (train, Uber/Lyft, drive & park) all of which I’ve used before and any of which would be fine.

This is relaxation-based search. I have a planning problem: visit my parents for the holidays. I’m searching for a plan which satisfies all my constraints: arrive at my parents’ house ~2-3 days before Christmas, arrive at the airport an hour before the plane takes off, don’t spend lots of time en route, don’t spend too much money, and many many other constraints (most of which are implicit in my head). In order to search efficiently, I relax the problem: I ignore many constraints which I expect will be easy to satisfy, like arriving at the airport on time or packing my bag. I throw out all but the hardest-to-satisfy constraints, which in this case mostly involve the plane tickets. I solve the “relaxed” search problem, with just the hardest constraints, then go back and tweak the plan to handle other constraints later.

Why This Matters: High-Dimensional Worlds

Our world is high-dimensional. Even a trip to visit my parents for the holidays involves hundreds of choices: dates, modes of transport, packing list, how early to arrive at the airport, etc. Multiply them all out, and we have an exponentially vast space of possibilities.

And that’s just a trip to visit my parents. Imagine the possibilities involved in designing a website or a scientific experiment!

When searching in spaces this large, we cannot afford to test every possibility. We are forced to rule out vast swaths of the search space without even looking at them. Relaxation-based search is one of the most general algorithms we know of which can do this.

The key is that we can usually express the problem-space using constraints which each depend on only a few dimensions. For instance, if I want to arrive at the airport an hour before the plane takes off, that constraint only involves two dimensions: my arrival time at the airport, and the takeoff time of the flight. It does not directly depend on what time I wake up, whether...
Self-education best practices
by Sean McAneny — 2021-11-10T17:12:08.327Z

I have a degree in a field I no longer wish to pursue in my career (English literature). Without a university and all it comes with (libraries, knowledgeable people, etc.), what are the best ways to self-educate and gain a substantive knowledge base in subjects I have little background in, specifically law, philosophy of language, intellectual property, and ethics? I am looking both for specific book recommendations and tips on how to most productively make my way through the material. I am no longer a student, but I have enough time on my hands to do nearly the amount of "course-work" I was doing in college. Also, are there folks who have covered self-education best practices in a book, podcast, etc? I have listened to Tyler Cowen interview Nassim Taleb on this; are there others offering insight on self-education?
What exactly is GPT-3's base objective?
by Daniel Kokotajlo — 2021-11-10T00:57:35.062Z

[Probably a noob question]

I'm thinking about what an inner alignment failure might look like for GPT-3. This would have to involve some deployment context in which GPT-3 performs significantly worse (by the standards of the base objective) than it did in training. (It would involve other things too, such as GPT-3 being a mesa-optimizer.)

But to say how well GPT-3 performs on some prompt not in the training dataset, we have to have a definition of the base objective that extends beyond the training dataset. If the base objective only makes sense in the context of the training dataset, then inner alignment failure is impossible by definition.

Is the base objective "Predict the next word?" Or is it "Predict the next word, supposing what you are reading is typical 2019 Internet text?" Or is it "Predict the next word, supposing what you are reading is a random-with-the-following-weights sample from dataset D? [where D is the dataset used to train GPT-3]" The third option is in some sense the best, because it most closely fits what we actually did to train GPT-3. But note that the logical extension of this line of reasoning is to prefer a fourth option: "Predict the next word, supposing what you are reading is a random-with-the-following-weights sample from dataset D' [where D' is like D except that it doesn't contain any of the bits of text that GPT-3 happened to not see in training, and the randomness weights are chosen to more accurately yield the data points that GPT-3 in fact saw]."

The problem with these last two answers is that they make it undefined how well GPT-3 performs on the base objective on any prompt that wasn't in D, which then rules out psuedo-alignment by definition.

From the Risks from Learned Optimization paper:

In such a case, we will use base objective to refer to whatever criterion the base optimizer was using to select between different possible systems and mesa-objective to refer to whatever criterion the mesa-optimizer is using to select between different possible outputs. In reinforcement learning (RL), for example, the base objective is generally the expected return. Because the mesa-objective is not specified by the programmers, mesa-optimization opens up the possibility of a mismatch between the base and mesa- objectives, wherein the mesa-objective might seem to perform well on the training environment but lead to bad performance off the

...
Robin Hanson's Grabby Aliens model explained - part 2
by Writer — 2021-11-09T17:43:00.167Z
Link: https://youtu.be/LceY7nhi6j4

This article is the script of the Rational Animations video linked above. This time, we go into the power law that constitutes the "core" of this model, how to generate simulations of the model, and how to generate predictions from simulations. The predictions covered are many, such as when we'll meet grabby aliens, how many galaxies they can grab, our chances of hearing alien messages, and becoming an interplanetary civilization ourselves. The model answers the Fermi paradox while estimating grabby civilizations' expansion speed: we don't see them in our skies because they travel very fast. High speeds constitute another selection effect: if we could see grabby aliens, they would be here now instead of us.

EA-Forum cross-post

Introduction/summary of the previous video

In the previous video, we talked about why humanity seems to have appeared very early in the universe: among the first civilizations that could ever appear. This estimate is the result of two considerations:

First: Before becoming a civilization, life has to go through a series of difficult phases called “hard steps” that are very unlikely to be completed as early as we did.
And second: Star systems with lifetimes of trillions of years might be habitable, making the overwhelming majority of future civilizations develop in those star systems in the trillions of years to come.

Earliness constitutes a riddle. It makes humanity’s situation seem rather mysterious. Robin Hanson, who introduced the idea of the great filter, solves this riddle by postulating that civilizations that he calls “grabby” will soon fill the universe. They have three characteristics:

They expand from their origin planet at a fraction of the speed of light.
They make significant and visible changes wherever they go, and
They last a long time.

Once a grabby civilization has occupied a volume of space, it prevents other civilizations from being born in that volume. Therefore grabby civilizations set a deadline for other civilizations to appear. Every non-grabby civilization can only appear early because later, all the habitable planets will already be taken. This constitutes a selection effect.

Hypothesizing grabby aliens explains our apparent extreme earliness. It makes us look typical amongst the other non-grabby civilizations, which can only follow our same path of appearing early and then potentially becoming grabby and expandin...
Come for the productivity, stay for the philosophy
by lionhearted (Sebastian Marshall) — 2021-11-09T13:10:36.354Z

There's a large class of things that eventually make you more effective when you've studied them for a while, but which are challenging and seem like they produce no gains while being learned.

Formal logic, for instance, seems that way for a lot of people.

Many of the natural sciences have this characteristic.

Some philosophical concepts have this characteristic.

For this class of thing, I've always found a strategy of "Come for the productivity, stay for the philosophy" to be effective.

In my experience, if people can get real and tangible gains and improvements out of a set of ideas quickly, they're more likely to stick around and be patient with the more abstract things.

Advanced students and practitioners in a domain often forget how expensive their field was to learn originally and how little gain there might seem to be in that field for new people, so they often neglect this.

I don't have a longer post to write. I just think this is really important to think about from time to time and encourage you to do so. It can make teaching and learning much more effective. Give it a try, maybe, next time you're teaching in an abstract domain.

What tangible results could the person get

now

so they're more likely to stick around for abstract learnings?
Erase button
by Astor — 2021-11-09T09:39:39.439Z

I try to model ethical concepts with thought experiments to investigate my own uneasiness. I do not wish for anybody to get harmed by these, but want to share them nonetheless to learn from possible discussions.

There is a room with a completely white interior. In the center of the room you can see a small white table with a button. If you press this button down, you instantly cease to exist. This button only works for those who press it with the conscious understanding of its consequences and with the intention to end their existences. Additionally there is no delay to its effect. You immediately lose all of your perception. You do not feel anything after pressing the button. The perception simply ends. If you consider all of the suffering in the world and if possible, should such a room be provided?

Supplementary question: Which conditions prevent suffering?
Arguments in parallel vs arguments in series
by Sunny from QAD — 2021-11-09T08:31:46.965Z

When arguments are delivered in parallel, their combined strength is no less than the strength of the strongest argument. For example, you might encounter two different people arguing in favor of the same position, one giving an argument that you can easily see doesn't work, and one giving an argument that's harder to refute. Or somebody might give five examples of a phenomenon they say exists, two of which are stretches but three of which work out all right. In both cases, the situation should have at least as much "persuasion power" as it would if the worse arguments did not exist.

When arguments are delivered in series, their combined strength is no greater than the strength of the weakest link. Somebody might begin with premise A, use it to draw intermediate conclusion B, and then use that to draw final conclusion C. If the link from B to C is suspect, then the entire argument is suspect, even if the link from A to B is irrefutable.

I'm guessing most people who will read this post already understand what I'm saying, but I've never seen it presented with the "in parallel" and "in series" phrasing before. I thought it was cute, and maybe useful, so I decided to post it. As usual, be careful not to apply this principle selectively -- your own in-series arguments need complete inspection to be accepted, and others' in-parallel arguments need complete inspection to be rejected.

Caveat: When I was getting feedback on this post, somebody pointed out that the presence of two bad examples actually can weaken the case made for the theory if your reasoning process involves evaluating how competent the person presenting it is. If they were 3 for 3 on giving examples, you might trust them more and therefore give more credence to their claims, whereas if they were only 3 for 5, you might trust them less and therefore give less credence. I agree with this. I think the reason it doesn't quite work out perfectly is that, if you're trying to evaluate the competence of the other person, then the five examples they give aren't actually quite parallel arguments, since they also contribute to the shared pool of "how trustworthy is this person?".

But really, the important point about parallel arguments is that you can't pick one of the two bad examples, argue that it's bad, and then ride the momentum from that rejection into a rejection of the whole theory. By the same token, you ...
Where did the 5 micron number come from? Nowhere good. [Wired.com]
by Elizabeth — 2021-11-09T07:14:25.167Z
Link: https://www.wired.com/story/the-teeny-tiny-scientific-screwup-that-helped-covid-kill/

This article describe's a scientist's attempt to figure out where the 5 micron number, and general belief that most respiratory diseases weren't airborne, came from. She eventually traces it back to a particular number developed for a very different purpose.
I have not fact checked it extensively, but last winter I did try to look into the general state of knowledge on airborne transmission vs fomites and found it weirdly empty, in ways that are consistent with this article.
In Defence of Optimizing Routine Tasks
by leogao — 2021-11-09T05:09:41.595Z

People often quantify whether an optimization to a routine task is worth it by looking at the net amount of time saved. This mindset is exemplified by this oft-cited xkcd:

https://xkcd.com/1205/

However, I think this doesn't really give a full picture of the best way to decide whether to make a task more efficient. There are many other factors that may weigh more than the raw time cost of something.

First off, for tasks that create friction inside important loops, this could disincentivize you from going around the loop, and make the feedback loop significantly more frustrating. For example, creating short aliases for common shell commands makes me much more productive because it decreases the mental cost I assign to running them and makes it easier to overcome the "activation energy", despite this probably not saving me more than an hour or two in a year, since typing a few extra characters realistically doesn't take that long. Of course, setting up aliases only takes a few minutes, but even if it took hours I still think it would work out to be worth it overall.

If the task inside the loop gets longer than even just a few seconds, this also adds significant context switching costs. For example, if I'm writing code that needs some simple utilities, the value of having those utilities already implemented and easily usable is worth far more than the reimplementation time it saves. If I need a utility function that takes 5 minutes to reimplement, by the time I finish implementing the function I've already partially lost track of the context in the original place where I needed the function and need to spend some more time and effort getting back into the flow of the original code I was writing. (This is also a reason why building up abstractions is so powerful - even more powerful than the mere code reuse benefits would imply - it lets you keep the high level context in short term memory.) In practice, this means that often building a set of problem specific tools first to abstract away annoying details and then having those tools to help you makes solving the problem feel so much more tactile and tractable and fun than the alternative. This seems obvious in retrospect but I still find myself not building tools when I should be sometimes.

If you're especially unlucky, the subtask itself could turn out to be harder than just a 5 minute task, and then you...
Possible research directions to improve the mechanistic explanation of neural networks
by delton137 — 2021-11-09T02:36:30.830Z

Why I like the Circuits approach

In early 2020 I skimmed through the neural network explainability literature which was already quite massive at that time. I was left quite discouraged about the utility of the explanation techniques I saw. The level of rigor exhibited in the literature was very low and I saw many people telling post-hoc "just-so" stories based on the visualizations generated by their explainability methods. In my own field (AI for medical imaging) heatmapping methods dominated, and I found several papers that illuminated major issues with them.[1][2] Cynthia Rudin's paper arguing against explaining black box models was particularly influential on my thinking.[3] A heatmap actually says very little about the precise method that a CNN uses to arrive at a prediction.

In response to this dismal state of affairs I wrote a paper on the idea of "self-explaining AI", where the AI outputs an explanation (for instance a set of keywords or a sentence) which can help the user decide how much to trust it. A major issue though was trying to figure out how to determine if the output of the explanation branch actually maps onto the internal mechanism it uses for the prediction branch. A month or so after I published my paper I discovered Olah et al.'s work. It was the first work I saw that actually attempted mechanistic explanation and the discoveries made seemed very compelling.

It's interesting how little follow-on work the Circuits thread has gotten from the broader academic community, especially considering that the code for it is open source. My guess is this is mainly because figuring out circuits is time consuming and hard. Additionally, there is no impressive-looking math involved, and instead there are a lot of stories involving pictures which looks very un-rigorous.

The Open Philanthropy project has a request for proposals and one of the areas they are interested in is explainability. Chris Olah has already laid out a number of possible research directions in a recent LessWrong post from Oct. 29th:

Discovering New Features and Circuits

Scaling Circuits to larger models

Polysemantic neurons

Misc other directions (includes finding inductive biases that improve interpretability, trying to understand model phase changes, model diffing, and looking at quadratic forms).

I think these are all valuable, in particular the scaling angle. Here are some other ...
How do we become confident in the safety of a machine learning system?
by evhub — 2021-11-08T22:49:41.080Z

Thanks to Rohin Shah, Ajeya Cotra, Richard Ngo, Paul Christiano, Jon Uesato, Kate Woolverton, Beth Barnes, and William Saunders for helpful comments and feedback.

Evaluating proposals for building safe advanced AI—and actually building any degree of confidence in their safety or lack thereof—is extremely difficult. Previously, in “An overview of 11 proposals for building safe advanced AI,” I tried evaluating such proposals on the axes of outer alignment, inner alignment, training competitiveness, and performance competitiveness. While I think that those criteria were good for posing open questions, they didn’t lend themselves well to actually helping us understand what assumptions needed to hold for any particular proposal to work. Furthermore, if you’ve read that paper/post, you’ll notice that those evaluation criteria don’t even work for some of the proposals on that list, most notably Microscope AI and STEM AI, which aren’t trying to be outer aligned and don’t really have a coherent notion of inner alignment either.

Thus, I think we need a better alternative for evaluating such proposals—and actually helping us figure out what needs to be true for us to be confident in them—and I want to try to offer it in the form of training stories. My hope is that training stories will provide:

a general framework through which we can evaluate any proposal for building safe advanced AI,

a concise description of exactly what needs to be true for any particular proposal to succeed—and thus what we need to know to be confident in it—and

a well-defined picture of the full space of possible proposals, helping us think more broadly regarding new approaches to AI safety, unconstrained by an evaluation framework that implicitly rules out certain approaches.

What’s a training story?

When you train a neural network, you don’t have direct control over what algorithm that network ends up implementing. You do get to incentivize it to have some particular behavior over the training data, so you might say “whatever algorithm it’s implementing, it has to be one that’s good at predicting webtext”—but that doesn’t tell you how your model is going to go about accomplishing that task. But exactly how your model learns to accomplish the task that you give it matters quite a lot, since that’s what determines how your model is going to generalize to new data—which is precisely where most of...
Steelman solitaire: how to take playing devil's advocate to the next level
by KatWoods — 2021-11-08T20:49:22.684Z

This is an old post of mine from The Nonlinear blog that I thought really belonged on LessWrong.

I have a tool for thinking that I call “steelman solitaire”. I have found that it comes to much better conclusions than doing “free-style” thinking, so I thought I should share it with more people.

In summary, it consists of arguing with yourself in the program Workflowy or Roam, alternating between writing a steelman of an argument, a steelman of a counter-argument, a steelman of a counter-counter-argument, etc.

In this blog post I’ll first explain the broad steps, then list the benefits, and finally, go into more depth on how to do it.

Benefits

Structure forces you to do the thing you know you should do anyway. Most people reading this already know that it’s important to consider the best arguments on all sides instead of just considering the weakest on the other. Many already know that you can’t just consider a counter-argument then consider yourself done. However, it’s easy to forget to do so. The structure of this method makes you much more likely to follow through with your existing rational aspirations.
Clarifies thinking. I’m sure everybody has experienced a discussion that’s gone all over the place, and by the end, you’re more confused than when you started. Some points get lost and forgotten while others dominate. This approach helps to organize and clarify your thinking, revealing holes and strengths in different lines of thought.
More likely to change your mind. As much as we aspire not to, most people, even the most competent rationalists, will often become entrenched in a position due to the nature of conversations. In steelman solitaire, there’s no other person to lose face to or to hurt your feelings. This often makes it more likely to change your mind than a lot of other methods.
Makes you think much more deeply than usual. A common feature of people I would describe as “deep thinkers” is that they’ve often already thought of my counter-argument, and the counter-counter-counter-etc-argument. This method will make you really dig deep into an issue.
Dealing with steelmen that are compelling to you. A problem with a lot of debates is that what is convincing to the other person isn’t convincing to you, even though there are actually good arguments out there. This method allows you to think of those reasons instead of getting caught up w

...