trentmkelly/LessWrong-43k huggingface dataset:
Notes on Notes on the Synthesis of Form
by Vaniver — 2022-10-06T02:36:08.595Z

Christopher Alexander was an architect and designer; I've written about some of his later works in another post. This, one of his earliest books, focuses on design as a formal process. What is design, and how does it work? I'll split things up slightly differently than he does, and talk about the model of design, followed by the context of the model.

I think this review is an ok substitute for the book (80% of the value, 5% of the length).

The Model

Design is trying to join together two things: the 'form' and the 'context', which together make the ensemble. Most of Alexander's examples are physical--you might be designing a kettle (the form) so that people can heat water at home on demand (the context), but you only really know if they work when you see the two together. [This is in part because the context is huge, and it's impractical to specify all of its details without knowing the details of the form.]

The important question is "fit". Do the form and the context fit together? There are many dimensions to consider, and the question could be 'yes' or 'no' for each. The design problem is solved if all of those questions come back 'yes'. He conceptualizes this as a set of functions.

For small problems, the designer can probably fit it all into their head and solve the problem at once. For large problems, this doesn't work, and so the designer needs to subdivide (or factorize) the problem. But think of the problem of exploring this nonconvex function space, where as you tweak the design, some of the functions stop complaining and others start (he imagines it as a row of lightbulbs with switches and dials in front, where you want all of the bulbs to be off at once).

The best subdivision of the problem will focus on the relationship between the functions of fitness, in order to find subcomponents that can be designed together. You might have to go back and redo the design of a whole subsection due to constraints elsewhere, but if you've minimized the connections between subsections, this should happen infrequently; if you haven't minimized them, then this might happen often. (Finding the right factorization is a graph theory problem!)

Thus there are two phases to design: the analytical phase, where you figure out the right program for the problem (dimensions of fitness and how they relate to each other which determines the downward-splitting subdivision of the design...
Against Arguments For Exploitation
by blackstampede — 2022-10-06T01:58:21.284Z

[Kind of a rambling collection of thoughts that I tried to organize a bit. Hopefully it's interesting to read]

Introduction

I've recently had several conversations in which a member of the rationalist community made the case that exploiting people from less economically developed countries for profit is acceptable. In the first, the exploitation was for medical testing, in the second, factories (actually, sweatshops, but that seems extra-loaded, and I'm already using the word "exploitation") were used as the example. I'm not a very eloquent speaker and I tend to get myself tied in knots trying to explain what I'm thinking, then get frustrated that I can't verbalize. It's not a recipe for a productive conversation, so I thought I'd try to write my thoughts down rather than speak them.

Before I continue, I intend the word "exploitation" to mean: Using someone else for your own benefit in a way that harms them to some degree. This doesn't mean that they are harmed more than they benefit (assuming you could compare harms somehow), it just means that they are harmed in ways that they otherwise wouldn't be. The more harm, the more exploitation. There is something of a spectrum of exploitation here- I'm exploited by my employer for profit (not very), a soldier is exploited by their country for violence (a bit more), all the way to wildly imbalanced and unfair relationships like slavery.

The actual definition says something like "using someone for your own gain unfairly" but has a second definition for exploiting resources. Those feel like the same usage with varying levels of harm- if you exploit someone unfairly, then you are causing more harm than benefit. If you "exploit" some natural resource, then you're not harming anyone (directly). So I think my usage isn't entirely unreasonable. If you disagree, feel free to mentally substitute some other word.

Overview

The argument that I've heard twice now goes something like this- assume I have capital and I'm trying to make a profit, advance medical research, or accomplish some other goal. I could hire people in a richer country, pay them more, and be able to make less profit, or I could shop around and find a poor country where I could hire more people for less money and make a larger profit.

So I build my factory or medical research facility in the poor country and each dollar spent on wages will produce more widgets (...
AI Timelines via Cumulative Optimization Power: Less Long, More Short
by jacob_cannell — 2022-10-06T00:21:02.447Z

TLDR: We can best predict the future by using simple models which best postdict the past (ala Bayes/Solomonoff). A simple model based on net training compute postdicts the relative performance of successful biological and artificial neural networks. Extrapolation of this model into the future leads to short AI timelines: ~75% chance of AGI by 2032.

Cumulative Optimization Power[1]: a Simple Model of Intelligence

A simple generalized scaling model predicts the emergence of capabilities in trained ANNs(Artificial Neural Nets) and BNNs(Biological Neural Nets):

perf ~= P = CT

For sufficiently flexible and efficient NN architectures and learning algorithms, the relative intelligence and capabilities of the best systems are simply proportional to net training compute or intra-lifetime cumulative optimization power P, where P = CT (compute ops/cycle * training cycles), assuming efficient allocation of (equivalent uncompressed) model capacity bits N roughly proportional to data size bits D.

Intelligence Rankings

Imagine ordering some large list of successful BNNs(brains or brain modules) by intelligence (using some committee of experts), and from that deriving a relative intelligence score for each BNN. Obviously such a scoring will be noisy in its least significant bits: is a bottlenose dolphin more intelligent than an american crow? But the most significant bits are fairly clear: C. Elegans is less intelligent than Homo Sapiens.

Now imagine performing the same tedious ranking process for various successful ANNs. Here the task is more challenging because ANNs tend to be far more specialized, but the general ordering is still clear: char-RNN is less intelligent than GPT-3 - they are trained on essentially the same objective and the latter greatly outperforms the former. We could then naturally combine the two lists, and make more fine-grained comparisons by including specialized sub-modules of BNNs (vision, linguistic processing, etc).

The initial theory is that P - intra-lifetime cumulative optimization power (net training compute) - is a very simple model which explains a large amount of the entropy/variance in a rank order intelligence measure: much more so than any other simple proposed candidates (at least that I'm aware of). Since P follow a predictable temporal trajectory due to Moore's Law style technological progress, we can then extrapolate the trends t...
Do uncertainty/planning costs make convex hulls unrealistic?
by eapi — 2022-10-06T00:10:32.325Z

It's almost a rule that as soon as you have a "utility of possible outcomes" plot like this:

You must then say "and by randomly choosing between the outcomes, we can achieve any intermediate outcome in terms of utility within the convex hull of these points" resulting in a plot like this:

Cool, I've done a linear interpolation before, seems reasonable. Plus, convex hulls are super nice to work with. But all models are imperfect - how accurate is this convex hull idea in practice?

Three stories

I like to plan ahead, whereas my friend values chaos and unpredictability. When we want to go to a restaurant, they're a big fan of the whole "lottery between your first preference and my first preference" idea.

If we're meeting at 7PM and the lottery happens at 7AM that's long enough to plan, so I don't mind.
If we're meeting at 7PM and the lottery happens at 6:59PM, I am a little annoyed - I might even prefer my guaranteed second preference over my uncertain first preference.
The "standard model" for what the outcome space looks like before picking p=P(my preference) is something like this:

I'm a CEO managing a business undergoing a possible merger. The more uncertain the deal is, the more all of my projects are disrupted. I might be fine walking into the final meeting with a 99% or 1% chance of success, but I would be quite stressed if the numbers were 90% or 10%.
I'm a computer, beep boop. I've been modelling deterministic systems all day long. Someone has just asked me to model something probabilistic, and now I need to learn around 70 years of research to not be overwhelmed by the state space explosion.

My point is that in practice the mapping from lottery probability p and outcome utilities U1,U2 to lottery utility Up is probably not Up=pU1+(1−p)U2. I wonder if it's occasionally not even close. I would expect the "lottery closure" of the three outcomes above to look something like this:

I'm pretty darn sure I'm not the first person to think about this, so: how big an issue is this when thinking about things like bargaining solutions, or stochastic Nash equilibria?
Generative, Episodic Objectives for Safe AI
by Michael Glass — 2022-10-05T23:18:00.447Z

Introduction

Current discourse around AI safety focuses on creating some new paradigm for training AI models to make them safe, such as assistance games (Russell 2021) or corrigibility (Soares 2014, 2018). The prevailing view seems to be that the only thing keeping current AI systems safe is the fact that they are not very smart. This paper will present the opposite view. The current paradigm has at least two elements that make it safe even when extended to superhuman intelligence.

Generative Objectives

The key current trend in AI is generating text(/image/sound/video) from the training distribution. Maximum-likelihood models like GPT-3 (Brown et al. 2020), diffusion models like DALL-E 2 (Ramesh et al. 2022) and generative adversarial networks (Goodfellow et al. 2014) all share this basic objective.

Conditional text generation is a particularly useful variety of this objective. Given a prompt, often a partial document, the task is to generate the remaining document to have text that is as likely as possible to occur in the training set. GPT-3 and Codex are two prominent examples.

Web text is safe in the sense that even unfiltered and uncurated text scraped for quantity over quality is extremely unlikely to contain anything dangerous in the context of AI safety. We will call a dangerous output from a text generating AI an "attack text". For example, the AI could generate a source file for code that contains a hidden exploit to replicate the AI or evade restrictions.

Consider some likely use cases for a very capable text generating AI:

Create or extend a github repository for: better GPU performance, better AI model, etc.

Similar repositories in the training data are actually very unlikely to contain malicious code of any type.

Crypto-mining scams and digital vandalism represent almost the entire space of malicious or deceptive code for such repositories.

Write or extend an arxiv paper for: improved AI capability, better AI interpretability, etc.

Preprints can be wrong or sometimes not-even-wrong, but simply do not contain dangerous trickery.

The ideas for self improvement that the AI generates do not necessarily result in an AI that is naturally safe. If the described or implemented AI improvements depart from the key safety properties of current AI systems they will need to be analyzed separately.

Episodic Objectives

If an AI has human-level or s...
Dependency Tree For The Development Of Plate Tectonics
by Elizabeth — 2022-10-05T22:40:02.602Z

This post is really rough and mostly meant to refer back to when I’ve produced more work on the subject. Proceed at your own risk.

Introduction

As I mentioned a few weeks ago I am working on a project on how scientific paradigms are developed. I generated a long list of questions and picked plate tectonics as my first case study. I immediately lost interest in the original questions and wanted to make a dependency graph/tech tree for the development of the paradigm, and this is just a personal project so I did that instead.

I didn’t reach a stopping point with this graph other than “I felt done and wanted to start on my second case study”. I’m inconsistent about the level of detail or how far back I go. I tried to go back and mark whether data collection was motivated by theory or practical issues but didn’t fill it in for every node, even when it was knowable. Working on a second case study felt more useful than refining this one further so I’m shipping this version.

“Screw it I’m shipping” is kind of the theme of this blog post, but that’s partially because I’m not sure which things are most valuable. Questions, suggestions, or additions are extremely welcome as they help me narrow in on the important parts. But heads up the answer might be “I don’t remember and don’t think it’s important enough to look up”. My current intention is to circle back after 1 or 2 more case studies and do some useful compare and contrast, but maybe I’ll find something better.

(Readable version here)

And if you’re really masochistic, here’s the yEd file to play with.

Scattered Thoughts

Why I chose plate tectonics

It’s recent enough to have relatively good documentation, but not so recent the major players are alive and active in field politics.

It’s not a sexy topic, so while there isn’t much work on it what exists is pretty high quality.

It is *the only* accepted paradigm in its field (for the implicit definition of paradigm in my head).

Most paradigms are credited to one person on Wikipedia, even though that one person needed many other people’s work and the idea was refined by many people after they created it. Plate tectonics is the first I’ve found that didn’t do that. Continental drift is attributed to Alfred Wegener, but continental drift is not plate tectonics. Plate tectonics is acknowledged as so much of a group effort wikipedia doesn’t g

...
How does anthropic reasoning and illusionism/eliminitivism interact?
by Shiroe — 2022-10-05T22:31:49.773Z

Illusionism makes the (unintuitive) claim that we have no first-person experience of the world or ourselves. Anthropic reasoning makes the (also unintuitive) claim that we learn something new about the world just by knowing that we exist. How do these two claims interact with each other? Is there work on this that I can read? For example, is there an illusionist account of the Sleeping Beauty Problem?
Finding Great Tutors
by Ulisse Mini — 2022-10-05T22:08:58.921Z

Does anyone have tips for finding great tutors for technical subjects? I'm looking to speed up my self-studying for eventual alignment research. Money isn't much of an issue, I'd probably meet every few days to talk about stuff that confused me and generally grill my teacher. :)

I'm mostly studying topics from John's Study Guide, so calibrate roughly for that kind of content.
Progress links and tweets, 2022-10-05
by jasoncrawford — 2022-10-05T19:24:30.682Z
Link: https://rootsofprogress.org/links-and-tweets-2022-10-05

Announcements

It’s time to build: Arnold Ventures is hiring a “VP of Infrastructure” to focus on “the drivers of supply-side bottlenecks”
The World’s Fair Company has launched their poster series, Stories of Our Future (prints available!) (via @camwiese)
Meta (Facebook) AI announces text-to-video. A bit rough but impressive (@MetaAI)

Links

Patrick Collison interviewed by Ezra Klein (via @tylercowen)
Charles Mann interviewed by Dwarkesh Patel (via @dwarkesh_sp)
10 years ago, it cost $10k to sequence a human genome. Illumina’s new machine promises $200 (via @emilylmullin)
“We had to order a larger CT scanner when it was deemed unacceptable to send patients to the zoo” (Marginal Revolution)

Quotes

“Who knows whether, when a comet shall approach this globe to destroy it… men will not tear rocks from their foundations by means of steam, and hurl mountains, as the giants are said to have done, against the flaming mass?” Byron, 1822
Margaret Sanger’s incredible resilience and commitment to the cause of birth control
Quote quiz: who lavished this praise on “American efficiency”?

Queries

What are examples of inventions/discoveries that were made by brute force search? (@_brianpotter)
What are examples of modern monuments of progress? (@MoritzW42)

Retweets

Technology as a whole is liberating and often enchanting (@atrembath)
Davis Kedrosky thread on Sir E. A. Wrigley, legendary British demographer and economic historian (@dkedrosky)
A blog post is a very long and complex search query to find fascinating people and make them route interesting stuff to your inbox
by Henrik Karlsson — 2022-10-05T19:07:55.069Z
Link: https://escapingflatland.substack.com/p/search-query

I was born in July 1989, which means I am of the last generation who will remember the time before the internet. The cables and data centers and hyperlinks grew up around me; they grew with me. I find it hard to disentangle the evolution of my psyche from that of the internet.

Explaining it to my daughter, who was born in 2017, a year when the world’s largest economy had begun tearing itself apart from the tension of this ever-evolving network, I tell her that the internet is like an alien intelligence. We don’t know exactly what it is; it has just landed, and only the first ship. We are trying to figure out how to talk to it. The first generation of explorers have noted that by making certain finger motions you can make the aliens show you images of cats and clothes, or tell you all the ways the world is falling apart.

For a long time, I thought this was all there was to it. I could tap the keyboard in a particular way and the screen would show me the weather, or tell me which translation of the Iliad to read and then make someone jump in a truck and drive it to our house. I preferred the Iliad to the screen.

But then, late 2021, after I had been making intricate finger movements again, I woke up in our guesthouse before sunrise and noticed that something had changed.

During the night, the internet had been set in motion. Tossing hither and thither in silence—as the fields lay frozen and waiting and the hedgehog slept in its pile of leaves—the internet had rearranged itself around me.

I had written an essay about Ivan Illich and systems thinking, a topic I had never found anyone else intrigued by, and which magazines thought below a rejection letter—and the internet had suddenly reshaped itself so that my keyboard hooked up to the screens of a bunch of people who wanted to talk about these topics, and a little later, their keyboards hooked up to mine.

I had written for 15 years, but never before had this happened. I had conjured a minor conference! And I hadn’t even known that you could do that.

This gave me a first glimpse of the social mechanics of the internet. Looking at the traffic data, and talking to readers, I could retrace how my words had traveled through the network, and I got a sense of why. I didn’t fully understand it; I don’t think anyone does. But like a scientist who’s got hold of an alien artifact, I proceeded by gleefully and se...
Neural Tangent Kernel Distillation
by Thomas Larsen — 2022-10-05T18:11:54.687Z

Produced As Part Of The SERI ML Alignment Theory Scholars Program 2022 Under John Wentworth.

Introduction

Consider the following example from Goal Misgeneralization: you train an RL agent to pursue a coin, which is always at the same location (at the end of the level) during training. At test time, if the coin position is randomized, does the RL agent pursue the coin, or does it go to the fixed location?

Since the training data could not distinguish between the two goals ('go to the coin' and 'go to the location'), which one is chosen is purely based on the neural network prior. It would be nice to be able to use simplicity priors like the Solomonoff prior to think about this: the neural network might tend to choose the 'simplest' extrapolation. But what is the right notion of simplicity for neural networks? It's clearly not program length in any normal programming language, because the parity function[1] is very short but hard for neural networks to learn.

In this post, we will summarize some recent advances in DNN theory that have given us the ability to describe the prior of a deep learning network, and discuss the relevance to alignment. Our goal is to communicate quickly the insights that we found difficult to understand or slow to extract from the sources we were using.

Neural Tangent Kernel (NTK) theory compares neural networks to kernel methods. [2] The most interesting takeaways (to us) are:

We can make a "linearized" neural network and prove that training this is equivalent to kernel regression.
As we increase the width of a normal neural network, we can prove that it will behave more like the linearized network.
By relating this to Gaussian Process inference, we can think of the neural network as doing Bayesian inference, and describe and visualize the prior distribution that it is using.

We can print out features/eigenmodes (functions from input vectors to output vectors), and think of the neural network as finding the best linear combination of these, with a preference for using the earlier eigenmodes.

Prerequisites: Linear Algebra, Gaussian Processes (GPs), to the level explained here, a source we highly recommend playing with to gain GP intuition.

Background

Notation

Let f denote the network architecture, a function that takes in the parameters and a network input, and outputs a predicted label. ...
Tracking Compute Stocks and Flows: Case Studies?
by Cullen — 2022-10-05T17:57:55.408Z

Posted in my personal capacity

The AGI governance community has recently converged on compute governance[1] as a promising lever for reducing existential risks from AI.

One likely building block for any maximally secure compute governance regime is stock and flow accounting of (some kinds of) compute: i.e., requiring realtime accurate declaration to regulators of who possesses which uniquely numbered regulated chips, with penalties for undeclared or unauthorized[2] transfers.

To understand the optimal design and feasibility of such a regime, we seek historical analogies for similar regimes. One that we are already familiar with include:

Fissile nuclear material and other nuclear weapons components

Firearms

Some financial instruments

Automobiles

Real estate

What are other good existing or historical analogies for compute stock and flow accounting? An ideal analogy will have many of the following traits:[3]

The thing being tracked is a physical object

The thing being tracked is economically important

The thing being tracked is dual-use

The tracking regime requires registration of current ownership and any transfers

The tracking regime imposes penalties for failing to register ownership or transfer

More and better examples of stock and flow accounting mechanisms could be extremely informative in improving compute governance proposals.

See, e.g., this sequence by Lennart Heim. ↩︎

The question of which types of transfers ought to be authorized is important but beyond the scope of this post. ↩︎

NB: Many of the above do not have all of these traits! ↩︎