trentmkelly/LessWrong-43k huggingface dataset:
[Proposal] Method of locating useful subnets in large models
by Quintin Pope — 2021-10-13T20:52:14.455Z

I’ve seen it suggested (e.g, here) that we could tackle the outer alignment problem by using interpretability tools to locate the learned “human values” subnet of powerful, unaligned models. Here I outline a general method of extracting such subnets from a large model.

Suppose we have a large, unaligned model M. We want to extract a small subnet from M that is useful for a certain task (T), which could be modeling human values, translating languages, etc. My proposal for finding such a subnet is to train a “subnet extraction” (SE) model through reinforcement learning.

We’d provide SE with access to M’s weights as well as an evaluation dataset for T. Presumably, SE would be a perceiver or similar architecture able to handle very large inputs and would only process small parts of M at a time.

During training, SE would select a small subnet from M, then the subnet would be sandwiched inside an “assister model” (AM), which consists of a pretrained encoder, followed by randomly initialized layers, followed by the extracted subnet, followed by more randomly initialized layers. The AM is then finetuned on the dataset for T as well as on a set of distractor datasets, {D_1, … D_n}. SE would get reward for AM’s post-finetuning performance on T’s dataset minus its average performance on the distractor datasets and be penalized according to the size of the subnet.

R = finetune(AM, T’s dataset) - a avg_val{finetune(AM, D_i), for i 1 to n} - b |subnet|

Where a and b are hyperparameters.

The idea is that the subnet M uses to solve T can be easily adapted to solve T in other contexts. It’s possible such subnets rely on features generated by other parts of M. That’s why I sandwich the subnet in AM. It’s supposed to provide the subnet with generic features, so that SE doesn’t have to extract those generic features from M.

I include the distractor datasets to ensure SE learns to extract subnets that are specific to the task/dataset provided and not just extract subnets from M that are really good for learning any task. I encourage SE to extract smaller subnets because I expect smaller subnets will be easier to analyze with other interpretability tools and because I think smaller subnets are less likely to include risky spillover from M (e.g., mesa optimizers). During training, we'd cycle the evaluation task/dataset with the aim that SE learn to be a general subnet extractor for whate...
Books Review: "Sapiens", "Homo Deus" and "21 Lessons For The 21st Century", by Yuval Noah Harari
by Luca Parodi — 2021-10-13T16:21:50.048Z

Introduction

In this review I will consider one of the most important literary and cultural phenomena of the last 20 years - the ambitious trilogy of the Israeli historian Yuval Noah Harari. In these three books Harari takes into consideration the past of humanity (Sapiens), the present (21 Lessons for the 21st Century) and the future (Homo Deus), trying to trace the global lines of development from the Stone Age up to the abandonment of biological bodies in favor of synthetic ones, passing through Trump and the future of unemployment. Among his greatest strengths as an author are:

The ability to look at trends from a bias-free perspective (with some notable exceptions)
The ability to bring together insights from many different disciplines in a very effective way
The ability to involve the reader with clear and accessible, yet not simplistic, language

There are undoubtedly more relevant and specific texts on each of the many themes addressed by Harari. Likewise, many of the observations in the texts can be (and have been) criticized from various points of view and for a variety of reasons, like Harari’s generalist approach and the lack of details. But precisely the horizontality and the ambitious scope are the main assets that make this trilogy so interesting and worth reading.

Observing a macro-trend from a hyper-specialized perspective can allow us to detail its different facets and potential origins, but makes it more complex to bring together different and broader contributions. Harari has chosen not to tackle the issues he faced from a hyper-specialized point of view, sometimes lacking detailed knowledge on certain topics, but manages to insert them within a universal narrative. A reader of Harari might find far-reaching insights into this narrative to ask questions about the research trajectories of his discipline and its role in a global perspective.

The purpose of my review is to anticipate and facilitate this exercise of reflection. For each of these three texts I’ve summarized a small set of broad key ideas to keep in mind while reading. In conclusion, I will summarize some of the criticisms that have been made against the three books.

Sapiens

The intersubjective myths

One of the starting points underlying Harari's entire body of work concerns the realm of intersubjectivity and its ability to concretely intervene on the world through the creation...
Notes from the book ‘First Three Minutes’ by Steven Weinberg
by invertedpassion — 2021-10-13T09:06:04.251Z

This is a cross-post from my blog.

It’s mind-blowing that we humans are able to talk about what happened in the first 3 minutes of The Big Bang. This book was written in 1976 which was quite a while back but it’s interesting to note that while there have been extensions in the ideas presented, I’m not aware of any idea being rejected or overturned yet. This should perhaps be unsurprising because most scientific ideas that are accepted as truth are consilient, i.e. they’re supported by multiple lines of evidence.

This means that when we talk about what happened in the first 3 minutes of The Big Bang, we’re confident about those events because only those descriptions make sense when we account for what we observe in the universe today.

I’m writing these notes primarily to solidify what I understood from the book. I’d love to get corrected if I’m wrong somewhere and to learn from people who are informed a lot more about cosmology than me.

How do we know that the Big Bang really happened?

The evidence for Big Bang essentially comes from the observation that we see different galaxies moving away from us and their speed of movement is proportional to how far they’re from us. This speed — called the Hubble Constant — is an empirical measurement (i.e. cannot be derived from first principles yet). Currently, it’s measured to be 70 (km/s)/Mpc. The unit Mpc is mega parsec where 1 parsec is approximately equal to 3.26 light years.

If we roll back this expansion, we’d naturally find that all these galaxies once were at the same place. To understand this, notice that a galaxy twice as far from us as another galaxy moves at twice the velocity. So if you roll back time, you’ll find that all galaxies (no matter how far from us currently) were once coincident in space.

Of course, this doesn’t mean that our location on Earth is special and the Big Bang started here. The expansion of universe can be observed from any location in the universe (which is the basis of cosmological principle). No location is privileged. Big Bang happened everywhere in the universe.

We can do a rough estimate of when did Big Bang happen by simply inverting the Hubble Constant. Which means if we know that currently a galaxy 1 mega parsec away is moving from us at 70 km/s, we need to ask how much time it would have taken for the galaxy to reach 1 megaparse...
Choosing a meetup venue
by mingyuan — 2021-10-13T06:23:06.956Z

This is the first post in a sequence I'm writing on how to run a meetup. The advice in this sequence comes from hundreds of survey responses and dozens of conversations I've had with meetup organizers and attendees over the past four and a half years. I'm just the messenger.

In this post, I discuss best practices for choosing a meetup venue. I'm erring on the side of thoroughness, but I also don't want the slew of considerations to give people decision paralysis. No venue is perfect on every single axis! People have had great meetups at all kinds of suboptimal places. Please don't give up on having a meetup because of this post; just pick a place that seems decent to you and see what happens. According to the most recent data, there's a 96% chance you won't regret it!

Basic considerations for choosing a venue

What factors should go into your choice of a venue? TL;DR:

Location — The venue should be easy to get to.
Size — The venue should be big enough to fit all the people who show up to your meetup.
Availability — The venue needs to be available for a group of your size at the time your meetup is scheduled.
Physical comfort — The venue should be within walking distance of bathrooms and drinking water, and people need to be able to sit down. It also shouldn't be too hot, cold, wet, bright, full of wasps, etc.
Noise — The venue should not have much background noise and should not get cacophonous if many people are talking at once.
Food — The venue should offer a variety of affordable food, or you should provide food yourself.

Geographical location

Travel time is a major reason that people don't attend meetups, so you want to choose a place that's as easy as possible for people to reach. Consider:

Accessibility via public transport

A location near a major transit hub can be a good choice, especially if people are coming from far away

Availability and price of parking
Walkability / bikeability
Centrality / convenience for attendees

You can survey people to find out what areas are best for them
If you're in a spread-out area, you might want to alternate locations — e.g. Chicago has some of their meetups close to the center of the city and some farther north.

Size

You need a venue that can accommodate all the people who might show up. If you expect a lot of people or have a lot of uncertainty about how many people will attend, choose a flexible venue such as a park, or plan ahead for...
Meetup advice sequence intro
by mingyuan — 2021-10-13T06:21:29.337Z

I'm transitioning into working on global meetup organization as more of a full-time thing, and the lowest-hanging fruit — something many people have asked for — seemed to be writing up all the advice I have on how to run a meetup well. So that's my first project!

Over the past few years I have surveyed more than 500 meetup organizers and attendees, asking what they like and don't like about the meetups they've attended, and what advice they would give to other organizers. This upcoming sequence is simply a writeup of what all of those people have told me.

This sequence will at least start by focusing on the very basics — the four main clusters that people mention are (1) venue, (2) attendance & advertising, (3) activities, and (4) managing the social environment. I will write posts on all of these, as well as things like how to run a first meetup, how to run meetups in cities of different sizes, and some other more exploratory topics. I don't have a sequence outline yet, nor do I have a posting schedule; I'm just going to keep going until I've exhausted the thousands of pieces of advice I've collected — which could take years, who knows? (Probably not.)

Beyond the realm of basic logistics, there are still a lot of unsolved problems, with governance structures and general interpersonal relations at the forefront. I don't know the answers to those problems yet, but hopefully someday I'll be able to give advice on them!

"But wait," you might say, "isn't there already a guide on how to run a meetup?" If you're referring to Kaj's guide, well, that's nearly a decade old, and things have changed quite a bit in the years since it was published. I also find it difficult to navigate since it's a PDF, so I think it's valuable to have something more bullet-point-y and clickable. If you're referring to Maia's meetup cookbook, that is useful and great and totally worth checking out, but it primarily covers activities, and I think there's a lot more to be said. If you're referring to the LW wiki page, that's a good summary, but again, I want to go into a lot more depth. My sequence is going to be thorough as hell. So thorough that you'll be glad it's got so many section headers so that you can click through to only the parts you're actually interested in!

If you're an organizer, or if you're just interested in connecting with other people who are interested in meetups, you ...
Book Review: All I Want To Know Is Where I'm Going To Die So I'll Never Go There
by Anmoljain — 2021-10-13T03:46:38.098Z

Introduction

The pope asked Michelangelo: "Tell me the secret of your genius".

How have you created the statues of David, the masterpiece of all masterpieces?

Michelangelo's answer: " It is simple. I removed everything that is not David"

Let's to the point - since antiquity people have been asking themselves what makes us successful and happy. How should I live? what constitutes a good life? What is the role of money? Is leading a good life a question of mindset, of adopting a particular attitude, or is it more about reaching concrete life goals? Is it better to actively seek happiness or to avoid unhappiness? and other thousand of questions we pose every day.

Over the past 200 years, we have created a world we no longer understand intuitively. This means that entrepreneurs, investors, managers, doctors, journalists, artists, scientists, politicians, and people like you and I will stumble our way through life unless we have the soundbox of the mental toolkit.

Let be honest, we don't know for sure what makes us successful. We can't pinpoint exactly what makes our life happy. But we know with certainty what destroyed success or happiness. This realization, as simple it is the fundamental negative knowledge i.e what not to do. What not to do is much more potent and powerful than what to do.

As the saying goes via negative knowledge is: we cannot say what God is, we can only say what God is not.

And this same applies to the present day: We cannot say what brings us success. We can pin down only what blocks or obliterates success. Eliminate the downside like mistakes, thinking errors and the upside will take care of itself.

This book is about the fictitious Seeker, who has known a lot of misery, and his visit to the “Library of Wisdom” where he meets another fictitious character – the Librarian- along with Warren Buffett and Charles Munger. The Seeker learns how to make better decisions to help his children avoid doing the dumb things he has done. For instance, he learns from Buffett and Munger the best way to prevent trouble is to avoid it altogether by learning what works and what does not. They do so in the spirit of the anonymous man who said: “All I want to know is where I’m going to die so I’ll never go there.”

Charles Munger says " If you want to avoid irrationality, it helps to understand the quirks in your own mental wiring and then you can take appropriate precau...
Choice Writings of Dominic Cummings
by Connor_Flexman — 2021-10-13T02:41:44.291Z

“My own heuristics for working in politics are: focus, ‘know yourself’ (don’t fool yourself), think operationally, work extremely hard, ... and ask yourself ‘to be or to do?’” - DC

Dominic Cummings is fascinating for four reasons. One, he is extremely committed to truth-seeking but from a different perspective than most of LW. Two, he has a shocking amount of real-world “success”, especially for a truth-seeker. Three, he fills the missing niche of trying to describe what government is actually like, to great effect. Four, he has uniquely powerful ideas about how to do project management well and how to fix government.

At the very least, he is extremely thought-provoking, and provides tons of value to >30% of people around me who try reading or listening to him.

However, most people get rebuffed by the sheer number of words and posts he’s written (or included as block quotes...). This post is to help people get a foothold in reading him, triage his work, and understand the basics of his perspective.

(Pitch: If you end up liking what he has written or even just my summary, consider subscribing to his

Substack

, even if only for a month and $10. It’s long been hard to capture much of the value from public goods like good opinions/models/writing, leaving them under-incentivized. Now that Substack allows us a convenient way to reward and incentivize good online writers, I want us to do an about-face on our expectations, and not confuse the previous fully-free status quo “is” with the “ought” of a real remuneration scheme. If you really like his writing but are short on cash, reach out to me and I may gift you a subscription.)

If you read nothing else…

The Brexit Story

(20k words = ~1.5 hrs, anecdotally 2.5):

This piece is most him. It touches on many of the themes that come up throughout his writing but in a concrete story. (Warning: you might have to do a bit of research into UK politics to understand what’s going on, or just skip the hard parts. You don’t need to understand everything.)

Highlights:

He worked ~18 hours a day for 10 months, and really missed his comfortable life
“Discussing politics with people almost never accomplishes anything proximally, but in public debate it can be like “throwing seeds to the wind” and you can be happily surprised down the road”
A long meditation on how difficult it is to tease apart “why did X win/lose”, it’s almo

...
What's your plan?
by jefftk — 2021-10-13T01:20:06.951Z

A question I find useful with my kids is "what's your plan?":

What's your plan for snack at school today?

What's your plan if it gets cold while we're at the picnic?

What's your plan for these drawings?

It works well when trying to find a balance on responsibility: the
issue is theirs to resolve, but it's not something I can just trust
they'll handle unassisted. It acts as both a casual reminder and an
opening for me to offer feedback if I end up thinking their plan is
unrealistic.

I also like that it's easy to pull back from: I can try not asking, and see how it
goes. Over time, as they become more capable, many things have moved
out of this intermediate state while others have moved in.

(This is another example of cultivating independence.)

Comment via: facebook
Bayeswatch 13: Spaceship
by lsusr — 2021-10-12T21:35:45.925Z

"This headquarters will be overrun in an hour. We have to evacuate," said Colonel Qiang.

"Not an option," said Trinity.

"Look. If I throw everything we've got at them―and I mean everything―we can hold off the Baltic forces for perhaps two hours―but that would be suicide. And for what?" said Colonel Qiang.

The intercom rang. "It's Eitan from Jerusalem," said the secretary.

"Put him through," said Trinity.

"You have Vi's access codes but you don't sound like her," said Eitan over the radio.

"Hivemind," said Trinity.

"What do you need?" said Eitan.

"I'm cashing in the favor you owe me for saving the Levant. I need to use your giant space laser," said Trinity.

"You realize you're too far away for us to provide you air defense? The curvature of the Earth gets in the way. Also, I saw those tanks and battlemechs on our satellites. Your options are defeat or Masada," said Eitan.

"What? No. We have our own weapons. We need you to cut a path through the Kessler debris. I'm sending you our launch trajectory," said Trinity.

"It shall be done," said Eitan.

Trinity turned off the radio.

"You're good at keeping secrets. I thought the ship's life support systems wouldn't be ready for another ten years," said Colonel Qiang.

"They won't be ready. Ever. I scrapped the life support project months ago. It's pure propaganda," said Trinity.

"You mean the reason I'm here, the reason I agreed to work for you, is a lie?" said Colonel Qiang.

"We were born on Earth. We will die on Earth. Space is for our children," said Trinity.

Colonel Qiang's eyes widened. "You're building a von Neumann machine."

"Built. It'll be ready to launch in two hours," said Trinity.

"You will have your two hours," said Colonel Qiang.

"萬歲," said Trinity.

Colonel Qiang bowed, clicked his boots, turned on his heels and left the room.

Credits

This concludes Bayeswatch. Thank you Dov Random for helping to come up with the original idea.
Modeling Risks From Learned Optimization
by Ben Cottier — 2021-10-12T20:54:18.555Z

This post, which deals with how risks from learned optimization and inner alignment can be understood, is part 5 in our

sequence on Modeling Transformative AI Risk

. We are building a model to understand debates around existential risks from advanced AI. The model is made with

Analytica

software, and consists of nodes (representing key hypotheses and cruxes) and edges (representing the relationships between these cruxes), with final output corresponding to the likelihood of various potential failure scenarios. You can read more about the motivation for our project and how the model works in the

Introduction post

. The previous post in the sequence,

Takeoff Speeds and Discontinuities

, described the different potential characteristics of a transition from high-level machine intelligence [1] to superintelligent AI.

We are interested in feedback on this post, especially in places where the model does not capture your views or fails to include an uncertainty that you think could be an important crux. Similarly, if an explanation seems confused or confusing, flagging this is useful – both to help us clarify, and to ensure it doesn’t reflect an actual disagreement.

This post explains how risks from learned optimization are incorporated in our model. The relevant part of the model is mostly based on the Risks from Learned Optimization

sequence

and

paper

(henceforth RLO). Although we considered responses and

alternate

perspectives

to RLO in our research, these perspectives are not currently modeled explicitly.

For those not familiar with the topic, a mesa-optimizer is a

learned algorithm

that is itself an optimizer. According to RLO,

inner alignment

is the problem of aligning the objective of a mesa-optimizer with the objective of its base optimizer (which may be specified by the programmer). A contrived

example

supposes we want an algorithm that finds the shortest path through any maze. In the training data, all mazes have doors that are red, including the exit. Inner misalignment arises if we get an algorithm that efficiently searches for the next red door – the capabilities are robust because the search algorithm is general and efficient, but the

objective is not robust

because it finds red doors rather than the exit.

The relevant part of our model is contained in the Mesa-Optimization module:

Right-click and select "open image in new tab" (or similar) to see images full

...