trentmkelly/LessWrong-43k huggingface dataset:
Trends in the dollar training cost of machine learning systems
by Ben Cottier — 2023-02-01T14:48:54.065Z
Link: https://epochai.org/blog/trends-in-the-dollar-training-cost-of-machine-learning-systems

Summary

Using a dataset of 124 machine learning (ML) systems published between 2009 and 2022,[1] I estimate that the cost of compute in US dollars for the final training run of ML systems has grown by 0.49 orders of magnitude (OOM) per year (90% CI: 0.37 to 0.56).[2] See Table 1 for more detailed results, indicated by "All systems."[3]
By contrast, I estimate that the cost of compute used to train "large-scale" systems since September 2015 (systems that used a relatively large amount of compute) has grown more slowly compared to the full sample, at a rate of 0.2 OOMs/year (90% CI: 0.1 to 0.4 OOMs/year). See Table 1 for more detailed results, indicated by "Large-scale."
I used the historical results to forecast (albeit with large uncertainty) when the cost of compute for the most expensive training run will exceed $233B, i.e. ~1% of US GDP in 2021. Following Cotra (2020), I take this cost to be an important threshold for the extent of global investment in AI.[4] (more)

Naively extrapolating from the current most expensive cost estimate (Minerva at $3.27M) using the "all systems" growth rate of 0.49 OOMs/year (90% CI: 0.37 to 0.56), the cost of compute for the most expensive training run would exceed a real value of $233B in the year 2032 (90% CI: 2031 to 2036).
By contrast, my best-guess forecast adjusted for evidence that the growth in costs will slow down in the future, and for sources of bias in the cost estimates.[5] These adjustments partly relied on my intuition-based judgements, so the result should be interpreted with caution. I extrapolated from the year 2025 with an initial cost of $380M (90% CI: $55M to $1.5B) using a growth rate of 0.2 OOMs/year (90% CI: 0.1 to 0.3 OOMs/year), to find that the cost of compute for the most expensive training run would exceed a real value of $233B in the year 2040 (90% CI: 2033 to 2062).

For future work, I recommend the following:

Incorporate systems trained on Google TPUs, and TPU price-performance data, into Method 2.
Estimate more reliable bounds on training compute costs, rather than just point estimates. For example, research the profit margin of NVIDIA and adjust retail prices by that margin to get a lower bound on hardware cost.
As a broader topic, investigate trends in investment, spending allocation, and AI revenue.

Estimation method

(go to explanation)

Data	Period	Scale (sta

...
Polis: Why and How to Use it
by brook — 2023-02-01T14:03:04.658Z

Polis is a surveying platform designed for finding clusters of people with similar opinions on a topic. Participants submit short text statements(<140 characters) which are sent out semi-randomly[1] to other participants to vote on by clicking agree, disagree or pass.

This post provides links to video and text tutorials for using polis.

When can Polis be Useful?

Identify subgroups of readers/users who you can then target more specifically
Understand better the types of people attending your university EA group
Survey opinions on how the EA community should be structured
Get suggestions for a new project for your organisation that are widely agreed upon in a given community
Understand the beliefs of a group of people affected by your project (e.g. AI researchers, parents in Nigeria, people new to EA)

Tutorials

This tutorial from Computational Democracy is a good introduction:

Polis is a 💡 Wikisurvey, as:

the dimensions of the survey are created by the participants themselves
the survey adapts to participation over time and makes good use of people's time by showing comments semi-randomly
participants do not need to complete the entire survey to contribute meaning

pol.is is the main instance of the technology hosted online, but there are other instances in the wild.

In its highest ambition, Polis is a platform for enabling collective intelligence in human societies and fostering mutual understanding at scale in the tradition of nonviolent communication.

A good tutorial based on a worked example is available here:

The text input to seed statements can be found in the ‘configure’ tab once you’ve started your conversation, seen at left.

It’s usually a good idea to seed around 10–15 diverse comments. This has a powerful effect on early participation. We’ve found that about 1 in 10 people leave a comment (whereas 9 in 10 only agree disagree and pass on statements submitted by others). Given this ratio, if there are no statements at the outset, it can take a little while for enough statements to build up to make the conversation meaningful. This is an impediment to data collection. So seed away!

Examples

There's one example worth pulling out before the others: vTaiwan. vTaiwan is an experiment in using Polis & other tools to find how opinions group, find the least controversial statements, and help set agendas for future consultations iteratively.&n...
Subitisation of Self
by vitaliya — 2023-02-01T09:18:49.784Z

Subitising is looking at a collection of objects and, as gestalt, assessing the number present without manual counting. It's about as basic of a System 1/2 split as you can get: children need to count manually, with each tallied increment taking effort; as we get used to seeing small collections of things in the real world, we group them together and treat the collection as atomic.

In digit grouping - e.g. presentation not as 1000000, but as 1,000,000, with the use of comma as delimiter - we again subitise chunk-by-chunk, determining the latter representation to be one million far faster than the former. We subitise straight to "two chunks of three zeroes", whereas without that convenience we often need to manually tally our million.

What is the self?

When we speak of "I", there's a bunch of targets and capabilities. For example, "I remember X", "I can Y", "I value Z". And so when we talk about the self, we combine this collection of capacities into a coherent gestalt, subitised down to one, an "I". And on thinking about this, I don't much like it.

The self that remembers has episodic, semantic, procedural memory components, each linked to but not properly contingent on the others. The removal of, say, my procedural memory of how to type isn't really, in an especially important way, a removal of "who I am". While I have many formative memories, no specific recollection is properly core to the self I would formulate.

Similarly, manipulation of any specific value retains my sense of self. If you were to induce in me some pathological murderous desires, leaving all other elements intact, the result would still in some sense be "me". At the same time, we're quite comfortable saying that, if I entered into such a state, I'm "not myself".

The Kanizsa Triangle

The boundaries are fuzzy here. There is some sense in which the above image contains "two triangles" - the outer perimeter, with a point facing upwards, and the other one being eaten by three pac-men. Neither triangle is "really there", in the sense of a contiguous shape, but when I refer to them, I imagine you have some understanding of The Thing I'm Pointing At With Words.

If I were to erase just one black pixel in the above image, I haven't really "removed" either triangle. Yet there's also some threshold of pixels I can remove beyond which there's a blank image, and neither triangle would remain present in any se...
Directed Babbling
by Yudhister Kumar — 2023-02-01T09:10:26.516Z
Link: https://www.ykumar.org/directed-babbling/

At a rationality workshop I ran an activity called “A Thing.” Not only because I didn’t know what to call it, but because I didn’t know what to expect. In retrospect, I've decided to christen it "Directed Babbling."

It was borne out of a naive hope that if two individuals trusted each other enough, they’d be able to lower their social inhibitions enough to have a conversation with zero brain-to-mouth filter. I thought this would lead to great conversations, and perhaps act as a pseudo-therapeutic tool to resolve disputes, disagreements over emotionally charged topics, and the like. However, it turns out this isn’t necessarily the best use case for a conversation where you simply say the first thing that comes into your head.

As with any writing trying to describe social dynamics, this may be somewhat inscrutable. However, I will try my best to explain exactly what I claim to be a useful conversational tool, for use-cases from “solving hard technical problems with a partner”, to “diving off the insanity deep end”.

Background

Alice and Bob are having a conversation. Alice says X, which Bob responds to with Y, in the context of the conversation (the previous things that Alice and Bob have said to each other) and the context of the world (Bob’s priors). Typically, Bob’s System 1 formulates Y and Bob’s System 2 “edits” it (for lack of a better term) - in most cases, the final output has more to do with System 1 than System 2. However, most of the time in discussion is spent with these System 2 “add-ons” - formulating ideas into sentences, making sure that the vocabulary is appropriate for the conversation, etc.

Hypothesis: if you intentionally remove the System 2 filters from the conversation between Alice and Bob, then you get a rapid feedback loop where the System 1 responses are simultaneously much faster and shorter than the original, which lets the conversation have a much higher idea density.

Setup

We paired participants and asked them to come up with a topic to start their conversation on. Following that, their instructions were to say “the first thing that came into their head” after hearing their partner, and see where this led. After fifteen minutes of this, we checked in and had a discussion on how this went. Repeat about 6 times.

Observations

Individuals did not report a loss in the ease of communication ...
Voting Results for the 2021 Review
by Raemon — 2023-02-01T08:02:06.744Z

Well, that's a wrap for the 2021 Review.

We had 238 people cast votes. 452 posts were originally nominated, of which 149 posts received at least one review.

The LessWrong moderation team will be awarding prizes and assembling posts into the Best of 2021 Books / Sequences soon. But for now, you can look here at the raw results.

Results

Voting is visualized here with dots of varying sizes (roughly indicating that a user thought a post was "good" "important", or "extremely important"). Green dots indicate positive votes. Red indicate negative votes. You can hover over a dot to see its exact score.

0

Strong Evidence is Common
Mark Xu

1

“PR” is corrosive; “reputation” is not.
AnnaSalamon

2

Your Cheerful Price
Eliezer Yudkowsky

3

ARC's first technical report: Eliciting Latent Knowledge
paulfchristiano

4

This Can't Go On
HoldenKarnofsky

5

Rationalism before the Sequences
Eric Raymond

6

Lies, Damn Lies, and Fabricated Options
Duncan_Sabien

7

Fun with +12 OOMs of Compute
Daniel Kokotajlo

8

What 2026 looks like
Daniel Kokotajlo

9

Ngo and Yudkowsky on alignment difficulty
Eliezer Yudkowsky

10

How To Write Quickly While Maintaining Epistemic Rigor
johnswentworth

11

Science in a High-Dimensional World
johnswentworth

12

How factories were made safe
jasoncrawford

...
Abstraction As Symmetry and Other Thoughts
by Numendil — 2023-02-01T06:25:53.338Z

Epistemic status: slightly pruned brainstorming.

I've been reading John Wentworth's posts on the natural abstraction hypothesis recently, and wanted to add some of my own thoughts. These are very much "directions to look in" rather than completed research, but I wanted to share some of the insights central to my model of the NAH, which might be useful for others trying to expand on or simply grok Wentworth's work.

The Telephone Theorem says that over large causal distances, information not conserved perfectly by each interaction is eventually destroyed. Wentworth provides two arguments, plus empirical evidence, that the limiting distribution should have maximum entropy. (I find this intuitively obvious, even if proving it rigorously and in the most general case is difficult.)
If we know in advance what information will be conserved (e.g. total energy of a system) and we know how this relates to the quantity we want to know the distribution of, then we can just find the maxent distribution subject to that constraint.
Since maxent distributions are usually exponential families, these constraints should correspond to a sufficient statistic for an exponential family.
We can probably say that every sufficient statistic for a distribution corresponds to a set of conserved quantities in the system that gives rise to that distribution.
By Noether's theorem, conserved quantities correspond to symmetries. (The actual theorem is specific to classical mechanics, but a similar principle seems to hold generally.)
Therefore, sufficient statistics = conserved quantities = symmetries.
The symmetry model is more natural for dealing with many kinds of abstractions. If we're pointing to a flower, it's not obvious how to label information as "conserved" or "not conserved". We can, however, say what we want to be able to do to the flower and have our algorithm still be able to recognize it: move it, view it from a different angle, let it bloom or wilt, replace all the molecules, &c &c.
This seems isomorphic to the view of abstraction as redundant information, but focused more on different views of the same object rather than different objects from the same class.
Machine learning already makes use of this idea. Many unsupervised representation learning algorithms follow a process like: take an input, distort it in some way that leaves it still recognizable (such as rotating or crop

...
The effect of horizon length on scaling laws
by Jacob_Hilton — 2023-02-01T03:59:26.074Z
Link: https://arxiv.org/abs/2301.13442

The scaling of optimal model size with compute is a key input into the biological anchors framework for forecasting transformative AI. In particular, the "effective horizon length" introduces a multiplier into this scaling law that can have a big effect on forecasts.

This paper studies this scaling law for several RL environments: Procgen, Dota 2 and a toy MNIST-based environment. The last of these is used to study the effect of the task horizon length in a toy setting. There are a number of takeaways for the biological anchors framework, which are summarized in Section 5.4.
Contra Dance Lengths
by jefftk — 2023-02-01T03:30:01.330Z

There was recently

some
discussion on how many minutes callers typically run a contra
dance, with a lot of "well, it depends" answers. There's a place for
that, but I think it's also helpful to have some numbers. I went back
over the last ten evenings where I

have
recordings:

It looks like 8-9min is pretty common. How much variability is
there within an evening?

Most callers stayed between 6.5min and 12.5min: one ran two dances
shorter than that; another ran eight longer (in two evenings). If
you're going to run dances much over or under 8-9min it's good to warn
the band so they can plan.

These were all 3hr evenings, and the callers called 8-12 dances. Two
of the evenings with 12 dances had a square: those were shorter (4:51,
4:59) and I didn't count them above. This makes 2/106 squares, well
down from 14/19 in 1965.
Including the squares, here's the relationship between number of
dances and dance length, inversely proportional as you'd expect:

Comment via: facebook, mastodon
Aiming for Convergence Is Like Discouraging Betting
by Zack_M_Davis — 2023-02-01T00:03:21.315Z

Summary

In a list of guidelines for rational discourse, Duncan Sabien proposes that one should "[a]im for convergence on truth, and behave as if your interlocutors are also aiming for convergence on truth."

However, prediction markets illustrate fundamental reasons why rational discourse doesn't particularly look like "aiming for convergence." When market prices converge on the truth, it's because traders can only make money by looking for divergences where their beliefs are more accurate than the market's. Similarly, when discussions converge on the truth, it's because interlocutors can only advance the discussion by making points where the discussion-so-far has been wrong or incomplete. Convergence on the truth, if it happens, happens as a side-effect of correctly ironing out all existing mispricings/disagreements; it seems wrong to describe this as "aiming for convergence" (even if convergence would be the end result if everyone were reasoning perfectly).

Sabien's detailed discussion of the "aim for convergence on truth" guideline concerns itself with how to determine whether an interlocutor is "present in good faith and genuinely trying to cooperate." I don't think I understand how these terms are being used in this context. More generally, the value of "collaborative truth-seeking" is unclear to me: if I can evaluate arguments on their merits, the question of whether the speaker is "collaborative" with me does not seem intellectually substantive.

Mostly, I don't expect to disagree with heavily-traded prediction markets. If the market says it's going to rain on Saturday with 85% probability, then I (lacking any special meteorology knowledge) basically think it's going to rain on Saturday at 85% probability.

Why is this? Why do I defer to the market, instead of tarot cards, or divination sticks, or my friend Maddie the meteorology enthusiast?

Well, I don't expect the tarot cards to tell me anything about whether it will rain on Saturday, because there's no plausible physical mechanism by which information about the weather could influence the cards. Shuffling and dealing the cards should work the same in worlds where it will rain and worlds where it won't rain. Even if there is some influence (because whether it will rain affects the moisture and atmospheric pressure in the air, which affects my grip on the cards, which affects my shuffling motion?...
On value in humans, other animals, and AI
by Michele Campolo — 2023-01-31T23:33:55.640Z

This will be posted also on the EA Forum, and included in a sequence containing some previous posts and other posts I'll publish this year.

Introduction

Humans think critically about values and, to a certain extent, they also act according to their values. To the average human, the difference between increasing world happiness and increasing world suffering is huge and evident, while goals such as collecting coins and collecting stamps are roughly on the same level.

It would be nice to make these differences obvious to AI as they are to us. Even though exactly copying what happens in the human mind is probably not the best strategy to design an AI that understands ethics, having an idea of how value works in humans is a good starting point.

So, how do humans reason about values and act accordingly?

Key points

Let’s take a step back and start from sensation. Through the senses, information goes from the body and the external environment to our mind.

After some brain processing — assuming we’ve had enough experiences of the appropriate kind — we perceive the world as made of objects. A rock is perceived as distinct from its surrounding environment because of its edges, its colour, its weight, the fact that my body can move through air but not through rocks, and so on.

Objects in our mind can be combined with each other to form new objects. After seeing various rocks in different contexts, I can imagine a scene in which all these rocks are in front of me, even though I haven’t actually seen that scene before.

We are also able to apply our general intelligence — think of skills such as categorisation, abstraction, induction — to our mental content.

Other intelligent animals do something similar. They probably understand that, to satisfy thirst, water in a small pond is not that different from water flowing in a river. However, an important difference is that animals’ mental content is more constrained than our mental mental content: we are less limited by what we perceive in the present moment, and we are also better at combining mental objects with each other.

For example, to a dog, its owner works as an object in the dog’s mind, while many of its owner’s beliefs do not. Some animals can attribute simple intentions and perception, e.g. they understand what a similar animal can and cannot see, but it seems they have trouble attributing more complex beliefs.

The abilit...
Criticism of the main framework in AI alignment
by Michele Campolo — 2023-01-31T23:01:27.176Z

Originally posted on the EA Forum for the Criticism and Red Teaming Contest.

0. Summary

AI alignment research centred around the control problem works well for futures shaped by out-of-control misaligned AI, but not that well for futures shaped by bad actors using AI. Section 1 contains a step-by-step argument for that claim. In section 2 I propose an alternative which aims at moral progress instead of direct risk reduction, and I reply to some objections. I will give technical details about the alternative at some point in the future, in section 3.

The appendix clarifies some minor ambiguities with terminology and links to other stuff.

1. Criticism of the main framework in AI alignment

1.1 What I mean by main framework

In short, it’s the rationale behind most work in AI alignment: solving the control problem to reduce existential risk. I am not talking about AI governance, nor about AI safety that has nothing to do with existential risk (e.g. safety of self-driving cars).

Here are the details, presented as a step-by-step argument.

At some point in the future, we'll be able to design AIs that are very good at achieving their goals. (Capabilities premise)
These AIs might have goals that are different from their designers' goals. (Misalignment premise)
Therefore, very bad futures caused by out-of-control misaligned AI are possible. (From previous two premises)
AI alignment research that is motivated by the previous argument often aims at making misalignment between AI and designer, or loss of control, less likely to happen or less severe. (Alignment research premise).

Common approaches are ensuring that the goals of the AI are well specified and aligned with what the designer originally wanted, or making the AI learn our values by observing our behaviour. In case you are new to these ideas, two accessible books on the subject are [1,2].

5. Therefore, AI alignment research improves the expected value of bad futures caused by out-of-control misaligned AI. (From 3 and 4).

By expected value I mean a measure of value that takes likelihood of events into account, and follows some intuitive rules such as "5% chance of extinction is worse than 1% chance of extinction". It need not be an explicit calculation, especially because it might be difficult to compare possible futures quan...