trentmkelly/LessWrong-43k huggingface dataset:
My tentative interpretability research agenda - topology matching.
by Maxwell Clarke — 2022-10-08T22:14:05.641Z

I'm looking for feedback on these ideas before I start working on them in November.

The goal of interpretability is to understand representations, and to be able to translate between them. This idea tries to discover translations between the representations of two neural networks, but without necessarily discovering a translation into our representations.

TL;DR

Learn auto-encoders in such a way that we know the learned topology of the latent space. Use those topologies to relate the latent spaces of two or more such auto-encoders, and derive (linear?) transformations between the latent spaces. This should work without assuming any kind of privileged basis.

My biggest uncertainty is about whether any of my various approaches to not constraining the topology will work. We have to discover the topology, but still know it afterwards.

Note: I've put the background section at the end. If you are confused I recommend reading that first.

Topology visualizations

Here is an example of the topology of an input space. The data may be knotted or tangled, and includes noise. The contours show level sets Si={x∣p(x)>pi}, i.e. each boundary is a D-1 dimensional manifold of equal probability density.

Input space (3D)

The model projects the data into a new, arbitrary basis, and passes it through some kind of volume-preserving non-linearity. (eg. an ODE, see Continuous Normalizing Flows) We regularize one of the dimensions to be isotropic Gaussian noise, and elide it in the below image. We now see all the information represented in just two dimensions. The topology hasn't changed!

Latent space (2D) - one dimension is just Gaussian and is "removed" prior to the latent space.

See how the level sets always form a tree?? We can view this topology as the Cartesian product of noise distributions and a hierarchical model.

The core idea: Topology matching

If we can discover the hierarchical model above, we can match it to that of another model based on the tree structure. These trees will correspond to real relationships between natural classes/variables in the dataset! If these structures are large enough or heterogeneous enough, they may have structure that occurs across multiple domains.

If we relate (or partially relate) the two hierarchies, then we can derive a transformation between the latent spaces, without using any privileged basis at all.

We can also create some kind ...
Signaling Guilt
by Krieger — 2022-10-08T20:40:46.865Z

Does anyone feel "signaling guilt"?

When I have to write something like a CV/Resume/essay for the purpose of admission into a high-status institution (e.g., academia, high-paying job), I feel an insane amount of ugh-field and find myself unable to properly motivate myself into actually focusing on it.

All of it just feels ... dishonest.

It gets worse with more competitive positions; just good ol' Moloch doing what it's good at: race-to-the-bottom, throwing away everything of value for the sake of slightly increasing competitiveness at the margin, etc.

Every time when I have to (fake intentions and emotions / intentionally distort my value system / signal fake compliance / pander to the reviewers by adopting their political stance and lingo / do something solely for the sake of signaling), it feels like I'm corrupting my "soul" via some irreversible process.

Perhaps the concept of "signaling" is infohazardous—I wasn't like this a couple years ago, back when I unconsciously optimized everything for the sole purpose of "appearing good in my resume," completely guilt-free.

Not any more. And no one around me seems to get it.

These days I find myself ... just baffled, as to how my colleagues can even do what they're doing while staying sane. Entire lives, lived without a single shred of real substance, just pure, pure signaling. I read the Moral mazes and understand how institutions select for lizards not actual living-breathing-humans—

And I tried so many things—artificially pumping my short-term motivation with stimulants, outsourcing my writings to GPT-3, trying to compartmentalize my life—none of them work consistently, at least for me.

The only thing that's keeping me going at this point is the tiny, tiny positive-reinforcment from the (false) belief that "I am a rare instance of an actual living-breathing-human who is heroically resisting the selection pressures and fighting against the Evil and Immoral Lizards."

Not for so long. (did I mention that I'm actually supposed to be editing my resume right now?)

Perhaps people actually do feel these signaling guilt but somehow manage to pull themselves together.

Perhaps this is just me conflating my normal akrasia and blaming it on The System.

Perhaps I'm just not Machiavellian enough.
In that case, well, too bad!
Football, quantum chromodynamics, figure skating and statistics
by Malmesbury — 2022-10-08T19:46:01.221Z

(Cross-posted from Telescopic Turnip.)

“He—and he is almost always a he—is a venture capitalist who has analyzed the hospitalizations data! He is a growth hacker with a piercing view of race and measures of intelligence! He is an industry analyst with insight into viral spread! He is a lawyer exploding nuances of gender and sex!”

This is from Annie Lowrey’s classic essay about the Facts Man. The Facts Man is a specimen you will meet everywhere while roaming the corridors of the World Wide Web. He's looked at the raw data and concluded that the leading experts are sometimes wrong. Maybe you’ve even got your pandemic advice from one of them (you wouldn't do such a thing, would you?).

This is to be distinguished from Regular Crackpots, who gave a lot of thought to difficult problems in cosmology or quantum chromodynamics, and came up with their own alternative theory. They, too, came to the conclusion that the leading experts are wrong.

Why do so many people believe the Facts Man and subscribe to his substack, while the youtube channel of the Regular Crackpot channel only receives laughs and sneers?

Lutzs

In figure skating, you win by performing the most difficult tricks. The best figure skaters can do things like the quadruple lutz, a trick that’s outright impossible for most mortals like you and me. Only a handful of athletes in the world can perform it, and only for a short period in their lives.

Evgeni Plushenko doing something outright impossible

Of course, you don’t need to do quadruples to enjoy figure skating, but you will not win the Olympics. Those who win the Olympics were raised in the best environment for it, had the best possible genetic background, and they still had to practice all their lives. If someone ever manages to pull out a quintuple lutz and does not completely screw up the artistic aspect, they will get the Gold.

Likewise, you will never make a groundbreaking discovery in quantum chromodynamics using freshman physics. You need to keep up to date with the current edge of knowledge, dig completely to the bottom of a particular problem, and derive some completely new maths just to make the smallest contribution. The best quantum physicists are the ones who can travel the farthest into quantum theory, and push the boundaries even further.

Panenkas

Football has no tricks. There are a few moves that require some technique, but a mediocre football playe...
SERI MATS Program - Winter 2022 Cohort
by Ryan Kidd — 2022-10-08T19:09:53.231Z

The Stanford Existential Risks Initiative (SERI) recently opened applications for the Winter 2022 Cohort of the ML Alignment Theory Scholars (MATS) Program, which aims to help aspiring alignment researchers enter the field by facilitating research seminars, workshops, an academic community, and an independent research project with an alignment research mentor. Applications close on Oct 24 and include a written response to (potentially hard) mentor-specific selection questions, viewable on our website.

Our current mentors include Alex Turner, Andrew Critch, Beth Barnes, Dan Hendrycks, Evan Hubinger, Jesse Clifton, John Wentworth, Nate Soares, Neel Nanda, Owain Evans, Quintin Pope, Rebecca Gorman, Richard Ngo, Stuart Armstrong, Vanessa Kosoy, Victoria Krakovna, and Vivek Hebbar.

Program details

MATS is a scientific and educational seminar and independent research program, intended to serve as an introduction to the field of AI alignment and allow networking with alignment researchers and institutions. The MATS Program Winter 2022 Cohort consists of:

A 6-week online training program (averaging 10-20 h/week from Nov 7 to Dec 14);
A 2-month in-person educational seminar and independent research program in Berkeley, California for select scholars (40 h/week from Jan 3 to Feb 24); and
Possible ongoing 2-month extensions for select scholars, potentially in Berkeley, California or London, UK.

During the research phase of the program, mentors will meet with scholars for around 1-2 h/week to share their research agenda and supervise the scholars’ research projects. Scholars' research directions will initially be chosen by the mentors, but by default, scholars are expected to develop their independent research direction as the program continues. Educational seminars and workshops will be held 2-3 times per week, similar to our Summer Seminar Program.

The MATS program is a joint initiative by the Stanford Existential Risks Initiative and the Berkeley Existential Risk Initiative, with support from Lightcone Infrastructure and Conjecture. We receive financial support from the Long-Term Future Fund.

Who is this program for?

Our ideal applicant has:

an understanding of the AI alignment research landscape equivalent to having completed the AGI Safety Fundamentals course;
previous experience with technical research (e.g. ML, CS, maths, physics, neuroscience, etc.), ide

...
How many GPUs does NVIDIA make?
by leogao — 2022-10-08T17:54:35.466Z

There are a few subquestions that I've had trouble finding numbers on with a quick search. Asking these questions mostly because they seem important for forecasting compute trends.

How many units of each model (i.e A100, 3090, etc) does NVIDIA make per month?
Which of these use the same dies but have constrained supply ratios due to binning? What do these ratios look like/can they change if NVIDIA decides to focus on high end GPUs?
How much of the total global silicon capacity at the latest process node does this take up? How hard would it be for NVIDIA to scale up by squeezing out other silicon usages?
Introducing the Basic Post-scarcity Map
by postscarcitymap — 2022-10-08T17:00:35.240Z

A society in which all basic human needs are provided at zero or very low cost without significant human work can be defined as basic post-scarcity.

Our civilization is getting close to the point where this is technically feasible.

To reach this milestone as soon as possible, we introduce here the Basic Post-scarcity Map project, an effort to map the current technological state of the art and to understand how far we are from a basic post-scarcity society. We are currently in alpha stage, and we are releasing early to gather feedback and collaborators.

This project is an attempt to provide unbiased answers to the following questions:

-What technological advancements are needed to reach basic post-scarcity?

-What is the state of the art, what resources are available to learn about it and who is currently working on improving it?

-How far are we from achieving basic post-scarcity and what are the bottlenecks?

To accomplish this, the basic idea is to build two maps: the first where we will deconstruct the basic needs needed to reach basic post-scarcity and the technical milestones needed to satisfy them with minimal human work at the minimum cost. The second in which we input and update the state of the art for each of the technical milestones.

References to the current state of the art and the resources needed to learn more about it are collected in separate pages forming a shared library.

This is an open source and collaborative project. All contributions are fact-based, with no projections, opinions, marketing or propaganda.

We believe that having a searchable and living assessment of the state of the art will enable people who want to work towards this goal to know what is needed, what is currently feasible and who is currently working on what.

We are aware of the many limitations of this approach and in particular we know that technical bottlenecks are not the only roadblocks to a basic post-scarcity civilization. However, we also think that it is not unreasonable to assume that reducing the cost and the manpower associated with fulfilling basic needs will make it easier for public and / or private actors to provide them as widely as possible.

Our goal is to make this project simple to contribute and update. At this stage we need help creating technical milestones. Domain experts are particularly welcome to shine light on the state of the art for the relevant mileston...
The "you-can-just" alarm
by Emrik — 2022-10-08T10:43:23.977Z

Daniel Dennett argues for the idea of a "Surely alarm" which says that whenever you hear someone say "surely" in an argument, an alarm should sound in your head—*ding!*—and you should pay attention for weaknesses. When someone says "surely", it's often just meant to appeal to a sense of disbelief that anyone could possibly disagree with them.

Readers of the Sequences will be familiar with similar alarms upon hearing "emergence", "by definition", "scientific", "rational", "merely", and more. And if you're really unlucky, some words just explode without warning.

Anyway, when you notice that people aren't doing something to attain what they've explicitly expressed a preference for, you should be wondering what's going on. I've seen countless examples of people noticing the discrepancy only to brush it aside because "surely if they wanted to do it, they would have." People routinely underestimate the cost of some seemingly insignificant obstacles.

The fact that A) it's obvious that people really want Y, and that B) it's also obvious that there's almost no cost to doing X in order to attain Y, should not just cause you notice-it-as-plain-fact. There's a deep mystery here, and if you also believe both A and B, there's real epistemic profit to be had by paying close attention. The potential profit is proportional to the total strength of the beliefs you've just noticed are in direct conflict.

You might naively think that if the confusion is so profitable, surely you—with all your ability to reason and pay close attention to reality—would've already noticed and exploited it. But if acquired beliefs A and B from a pre-attentive stage of your life, this may also explain why you've skipped over this obvious opportunity until now.

Besides, it's really cheap to do a second look-over, so why-don't-you-just investigate?
Why I think there's a one-in-six chance of an imminent global nuclear war
by Max Tegmark — 2022-10-08T06:26:40.235Z

Many people have asked me what I think the odds are of an imminent major US-Russia nuclear war. My current estimate is about the same as losing in Russian roulette: one in six. The goal of this post is to explain how I arrived at this estimate. Please forgive its cold and analytic nature despite the emotionally charged topic; I'm trying not to be biased by hopes, fears or wishful thinking.

My estimate is 30% x 80% x 70% ~ 1/6, as illustrated in the figure and explained below. The horizontal axis roughly corresponds to levels of escalation, while the vertical axis corresponds to how favorable outcomes are to the two sides.

Possible outcomes

To estimate the odds of pulling a spade out of a deck of cards, it's important to know how many suits there are. To estimate the odds that the current unstable situation ends up in the "KABOOM" outcome (a major US-Russia nuclear war that might cause nuclear winter and kill most people on Earth), it's similarly important to know what other reasonably stable outcomes it's competing against. The shorthand labels I've given these outcomes (grey boxes) should't be taken too literally: "Kosovo" & "Vietnam" refer to scenarios where one side wins outright (breakaway succeeds & Goliath is expunged, respectively). "Libya", "Korea" & "Finland" refer to intermediate outcomes involving simmering war, frozen war and full peace, respectively. I'm not showing the "Cuba" outcome (invasion averted by negotiated agreement) that was on the table in December 2021, since it's now off the table, as are resumed EU-Russia gas exports via the Nordstream pipelines.

Escalation dynamics

The grey ellipses represent relatively short-lived situations. We are currently in a vicious circle in the form of a self-perpetuating escalation spiral: since "Kosovo" is deemed unacceptable by Ukraine and the West while "Vietnam" is deemed unacceptable by Russia, both sides double down and escalate further whenever they fear losing. Such escalation has been both quantitative (more weapons, more mobilization) and qualitative (e.g., novel sanctions, heavier weapons, longer-range weapons, attacks inside Russia, scaled-up attacks on civilian infrastructure, shelling of a nuclear power plant, assassinations, sabotage of gas pipelines and Europe's longest bridge, annexations, and escalatory rhetoric about nuclear use). My assessment is that Russia, whose...
The Teacup Test
by lsusr — 2022-10-08T04:25:16.461Z

"I want to get into AI alignment," said Xenophon.

"Why?" said Socrates.

"Because an AGI is going to destroy humanity if we don't stop it," said Xenophon.

"What's an AGI?" said Socrates.

"An artificial general intelligence," said Xenophon.

"I understand the words 'artificial' and 'general'. But what is an 'intelligence'?" said Socrates.

"An intelligence is kind of optimizer," said Xenophon.

"Like my teacup," said Socrates. He took a sip of iced tea.

"What?" said Xenophon.

"My teacup. It keeps my tea cold. My teacup optimizes the world according to my wishes," said Socrates.

"That's not what I mean at all. An optimizer cannot do just one thing. There must be an element of choice. The optimizer must take different actions under different circumstances," said Xenophon.

"Like my teacup," said Socrates.

"But—"

"Now it is summer. But in the winter my teacup keeps my tea hot. Do you see? My teacup does have a choice. Somehow it knows to keep hot things hot and cold things cold," said Socrates.

"A teacup isn't intelligent," said Xenophon.

"Why not?" said Socrates. He savored another sip.

"Because the teacup is totally passive. An intelligence must act on its environment," said Xenophon.

"Far away, in the Levant, there are yogis who sit on lotus thrones. They do nothing, for which they are revered as gods," said Socrates.

"An intelligence doesn't have to act. It just needs the choice," said Xenophon.

"Then it is impossible to tell if something is intelligent solely based on its actions," said Socrates.

"I don't follow," said Xenophon.

Socrates picked up a rock. "This intelligent rock chooses to never do anything," said Socrates.

"That's ridiculous," said Xenophon.

"I agree," said Socrates, "Hence why I am so confused by the word 'intelligent'."

"No intelligence would choose to do nothing. If you put it in a box surely any intelligent being would attempt to escape," said Xenophon.

"Yogis willingly box themselves in boxes called 'monasteries'," said Socrates.

"I see what you're getting at. This is a case of the belief-value uncertainty principle. It's impossible to tell from their actions whether yogis are good at doing nothing (their value function is to do nothing) or bad at doing things (their value function is to act and they are just very stupid)," said Xenophon.

Socrates nodded.

"Since it is impossible to deduce whether something is intelligent based solely on its external behavior, intellig...
Deliberate practice for research?
by Alex_Altair — 2022-10-08T03:45:21.773Z

I would like to gain mastery in the domain of alignment research. Deliberate practice is a powerful sledge hammer for gaining mastery. But unlike something like chess or piano, it's not clear to me how to use this sledge hammer for this domain. The feedback loops are extremely long, and the "correct action" is almost never known ahead of time or even right after doing the action.

What are some concrete ways I could apply deliberate practice to alignment research?

One way would be to apply it to skills that are sub-components of research, rather than trying to rapidly practice research end-to-end.

The sub-skill I've thought of that is the best fit to deliberate practice is solving math and physics problems, a la Thinking Physics or other textbook exercises. Being better at this would certainly make me a better researcher, but it might not be worth the opportunity cost, and if I ask myself, "Is this cutting the enemy with every strike?" then I get back a no.

Another thing I can think of is trying to deliberately practice writing, which is a big part of my research. I could try to be more like John, and write a post every week, to get lots of quick feedback. But is this fast enough for deliberate practice? I get the sense that the feedback cycle has to be almost real-time. Maybe doing tweet explanations is the minimal version of this?

I'd appreciate any other concrete ideas! (Note that my research style is much more mathy/agent-foundations flavored, so programming is not really a sub-skill of my research.)
The Biggest Problem with Deontology: The Aggregation Problem
by Arjun Panickssery — 2022-10-08T03:41:32.966Z
Link: https://arjunpanickssery.substack.com/p/the-biggest-problem-with-deontology

Michael Huemer is a fan of weak deontology, the view that you should avoid committing rights violations merely to produce greater but comparable benefits (though you might be required to violate rights to produce much better consequences). But he acknowledges that while utilitarianism often contradicts our intuitions, the best reason to support it is to conclude that all other theories have more severe flaws.

The most severe flaw with weak deontology is what he in Knowledge, Reality, and Value calls the Aggregation Problem:

Say there are two people, A and B, who each have money in a bank. You want to give money to A, but, for whatever reason, you are unable to do so directly. What you can do is to hack into the bank’s computer and run a program which will steal $1 from B’s bank account, then give $2 to A. (The extra dollar comes from your own account, and you’re happy to give it, as A needs the money and you don’t.) The moderate deontologist says this is impermissible: Stealing is a rights-violation, which cannot be justified merely by producing slightly greater benefits for others. So far, so good – that’s not an implausible view on its face.

Now, it happens that you also would like to give some money to B, but again you can’t do so directly. You can only run a computer program which will steal $1 from A and then give $2 to B. (This is a separate program from the one that would steal from B to give to A.) Just as before, the moderate deontologist will presumably declare this impermissible.

However, if you do both actions, A and B are both better off. Intuitively, there’s nothing wrong with that. Money is fungible, so the net effect on A and B will be exactly as if you had simply transferred $1 to each of them, which would surely have been permissible. The fact that you carry out the transfer in this odd way, taking $1 away and giving $2 back, seems morally irrelevant.

So this would suggest that two wrongs make a right, that two individually unacceptable actions become acceptable in combination. Huemer elaborates in “A Paradox for Weak Deontology”:

Individuation Independence: Whether some behavior is morally permissible cannot depend upon whether that behavior constitutes a single action or more than one action.

Two Wrongs: If it is wrong to do A, and it is wrong to do B given that one does A, then it is wrong to do both A and B.

You would expect these two rules to ...