trentmkelly/LessWrong-43k huggingface dataset:
Summary of 80k's AI problem profile
by JakubK — 2023-01-01T07:30:22.177Z
Link: https://forum.effectivealtruism.org/posts/btFBFdYEn2PbuwHwt/summary-of-80k-s-ai-problem-profile

This is a summary of the article “

Preventing an AI-related catastrophe

” (audio version

here

) by Benjamin Hilton from

80,000 Hours

(80k). The article is quite long, so I have attempted to capture the key points without going through all the details. I’m mostly restating ideas from the article in my own words, but I'm also adding in some of my own thoughts. This piece doesn’t represent 80k’s views.

TL;DR

The field of AI is advancing at an impressive rate. However, some AI experts think AI could cause a catastrophe on par with human extinction. It’s possible that advanced AI will seek power and permanently disempower humanity, and advanced AI poses other serious risks such as strengthening totalitarian regimes and facilitating the creation of dangerous technologies. These are seriously neglected problems, and there are ways you can help.

Researchers are making advances in AI extremely quickly

One example of recent AI progress is

Minerva

, which can solve

a variety of STEM problems

, including math problems:

A MATH dataset problem followed by Minerva's correct solution.

Before Minerva, the highest AI accuracy on the

MATH dataset

of challenging high school math problems was ~7%, meaning that you could expect an AI to answer 7 out 100 questions correctly. In August 2021,

one team of forecasters

predicted that it would take over four years for an AI to exceed 50% on MATH (better than most humans

according to one creator of the dataset

), but Minerva ended up achieving this milestone in less than one year, four times faster than predicted. Minerva is just one of

many

powerful

models

we’ve seen in the past few years alone.

Current trends suggest

that AI capabilities will continue to advance rapidly. E.g., developers are using

exponentially increasing

amounts of computational resources to train AI systems.

Additionally, AI algorithms are requiring

exponentially lower amounts of computational resources to achieve a given level of performance

, demonstrating progress in the efficiency of AI algorithms. Given that efficient algorithms and computational resources are two crucial drivers of AI progress, we will likely see more rapid progress in AI if these trends continue.

Some

reports

and

surveys

have attempted to predict when we will see AI sufficiently advanced to be “

hugely transformative for s

...
What percent of people work in moral mazes?
by Raemon — 2023-01-01T04:33:43.890Z

Epistemic status: A quick Fact Post on moral mazes. This is me trying to get my hands on some data and think through stuff, not meant as a definitive reference.

I wanted a sense of the proportion of people who work in moral mazes.

The middle manager hell hypothesis says "the more layers of middle management you get, the more your company will be Goodharty, deceptive, and optimized for upper management political games" (among other things).

Whether you buy the middle-manager-hell-hypothesis, I wanted to figure out how many people work in a highly hierarchal org.

This blogpost was a originally going to be a short aside in Recursive Middle Manager Hell, where I claimed "People in modern society are more likely to work in moral mazes. Large companies tend to become moral mazes, company size is probably heavy-tail distributed, therefore probably most employee-hours are spent working in companies with lots of employees."

But, is that true? Size of companies is gonna be heavy-tailed, but, number of small companies is also probably heavy-tailed, and I wasn't sure how the numbers checked out. This seemed like a good opportunity for me to practice being more numerically literate and getting in contact with some facts-on-the-ground about company size.

So:

I found this webpage claiming to have data for number of US companies for each size in 2022. Unfortunately, it only buckets up to "1000+" employees. I have a feeling that heavy tails are pretty important here.

Employees Per Org	Number of orgs
1 - 4 employees	12,737,231
5 - 9 employees	1,913,721
10 - 19 employees	817,604
20 - 49 employees	414,381
50 - 99 employees	154,255
100 - 249 employees	89,365
250 - 499 employees	33,467
500 - 999 employees	19,058
1,000+ employees	24,036

(It also includes a "uncoded companies" of which there are 1,771,725. But, as a first approximation, the coded companies are hopefully representative as a proportion of the US population)

I have a feeling heavy tailed orgs are pretty important here. As a quick workaround I found this page listing the very top companies. I asked ChatGPT to reformat the list so I could paste it into a spreadsheet. It... seemed to only go down to "Walgreens" before it seemed just start making stuff up. (I'm guessing it ran out of working memory to think?). But there were only a few more US companies anyway.

[Fake edit: oh goddammit the fr is maybe world data instead of US data, w...
Recursive Middle Manager Hell
by Raemon — 2023-01-01T04:33:29.942Z

I think Zvi's Immoral Mazes sequence is really important, but comes with more worldview-assumptions than are necessary to make the points actionable. I conceptualize Zvi as arguing for multiple hypotheses. In this post I want to articulate one sub-hypothesis, which I call "Recursive Middle Manager Hell". I'm deliberately not covering some other components of his model[1].

tl;dr:

Something weird and kinda horrifying happens when you add layers of middle-management. This has ramifications on when/how to scale organizations, and where you might want to work, and maybe general models of what's going on in the world.

You could summarize the effect as "the org gets more deceptive, less connected to its original goals, more focused on office politics, less able to communicate clearly within itself, and selected for more for sociopathy in upper management."

You might read that list of things and say "sure, seems a bit true", but one of the main points here is "Actually, this happens in a deeper and more insidious way than you're probably realizing, with much higher costs than you're acknowledging. If you're scaling your organization, this should be one of your primary worries."

The Core Model

Say you have a two-layer company, a CEO and a widgetmaker. The CEO is directly in contact with reality – his company either is profitable or not. He can make choices about high-level-widgetmaking strategy, and see those choices play out in customers that buy his product.

The widgetmaker is also in direct contact with reality – he's got widgets to make. He can see them getting built. He can run into problems with widget-production, see that widgets are no longer getting made, or getting made worse. And then he can fix those problems.

Add one middle manager into the mix.

The middle manager is neither directly involved with widget-production, or the direct consequences of high-level widget strategy. Their feedback loop with reality is weaker. But, they do directly interact with the CEO, and the widgetmakers. So they get exposed to the object level problems of widget-making and company-profit-maximizing.

Hire a lot of widgetmakers, such that you need two middle managers. Now, the middle managers start talking to each other, and forming their own culture.

Scale the company enough that you need two layers of middle-managers. Now there's an upper layer who reports to the ...
The Roots of Progress's 2022 in review
by jasoncrawford — 2023-01-01T02:54:23.187Z
Link: https://rootsofprogress.org/2022-in-review

2022 was another big year for me and for The Roots of Progress. This is my annual review—the one post a year (other than timely announcements) where I go meta and give an update on this project. If you want stuff like this more frequently, you can support me on Patreon or make a donation to get my monthly supporter update.

This year had several highlights. We announced a major expansion of this nonprofit effort and hired a CEO to lead it. I concluded my lecture series, “The Story of Civilization,” and am now writing a book based on the same content. I was interviewed for major publications, spoke at some of the top progress conferences, and co-hosted a couple of events myself. Most importantly, I had a couple of banger tweets.

But I’m going to bury all of those ledes in order to start, as is my tradition, with what I suspect is more interesting to my audience: a selection of this year’s…

Reading

The book that fascinated me most this year was American Genesis: A Century of Invention and Technological Enthusiasm, 1870–1970, by Thomas P. Hughes, a finalist for the 1990 Pulitzer. The book is not only about the century of technological enthusiasm, but also about how that enthusiasm (in my opinion) went wrong, and how it came to an end. My review of this book was so long that I broke it into two parts: one on American invention from the “heroic age” to the system-building era and one on the transition from technocracy to the counterculture. (I may at some point do a third part, on the aesthetic reaction to modernism.) Among the more mindblowing facts I learned from this book are that Stalin made “American efficiency” a part of Soviet doctrine, and that Ford’s autobiography “was read with a zeal usually reserved for the study of Lenin.” Overall this greatly strengthened my understanding of technocracy, one of my themes for this year (see below).

A close runner-up for favorite book I read this year was The Control of Nature, by John McPhee. The book tells three stories: about the dams and levees that control the flow of the Mississippi River, the 1973 Eldfell volcanic eruption in Iceland, and the periodic landslides in the San Gabriel Mountains near Los Angeles. It’s fascinating to reflect on how nature is truly indifferent to human needs. Even something we take for granted, such as the course of a river, has to be actively, artificially maintained if it matters to human...
Investing for a World Transformed by AI
by PeterMcCluskey — 2023-01-01T02:47:06.004Z
Link: https://bayesianinvestor.com/blog/index.php/2022/12/30/investing-for-a-world-transformed-by-ai/

AI looks likely to cause major changes to society over the next decade.

Financial markets have mostly not reacted to this forecast yet. I expect
it will be at least a few months, maybe even years, before markets have
a large reaction to AI. I'd much rather buy too early than too late, so
I'm trying to reposition my investments this winter to prepare for AI.

This post will focus on scenarios where AI reaches roughly human levels
sometime around 2030 to 2035, and has effects that are at most 10 times
as dramatic as the industrial revolution. I'm not confident that such
scenarios are realistic. I'm only saying that they're plausible enough
to affect my investment strategies.

Companies working on AI

Google's DeepMind seems a bit more likely than any other company to
produce valuable AI. That's a good enough reason to hold a modest
position in Google. TPU and
Waymo are additional AI-related
reasons for holding Google stock.

OpenAI is a serious contender. Microsoft has a $1 billion stake in
OpenAI. But profits from that seem to be capped at
100x, so the best case scenario
seems to be that Microsoft shareholders get $13.40 per share from that
stake. Maybe Microsoft will invest more in OpenAI. I only see Microsoft
getting a modest boost from AI.

Conjecture sounds quite
promising. I offered to
invest $100k in them this summer. They don't seem interested in
investments that small.

There are other new private companies that I haven't been able to
evaluate. Here are some that seem worth paying some attention to:

Adept

Anthropic

Bubble City

Character

Cohere

Inflection
AI

Keen
Technologies

That's a dramatic pace of new startups being founded that look
potentially important. VC's seem to have been throwing enough money at
them in mid-2022 that they likely had little reason to talk to more
ordinary investors such as myself. Investment in such startups has maybe
cooled slightly now that FTX has stopped throwing money at them, but
I'm guessing it hasn't cooled much.

I see a 50% chance that one or more of these new startups will make
DeepMind and OpenAI irrelevant. I have little hope of being able to buy
a diversified portfolio of such startups, so I don't expect to profit
from direct investments in companies working on AI.

Computing Hardware Companies

The most promising investments involve the hardware needed to power AI.
That mostly means semiconductors.

Semiconductor capital equ...
Why Free Will is NOT an illusion
by Akira Pyinya — 2023-01-01T02:29:18.351Z

(This post contains some thought about gACI)

What does it mean that I have free will? If we can define free will like this:

If in a deterministic universe, no observer B can 100% correctly predict the behavior of subject A, except when B is in the future of A, we can say that subject A has free will.

We can prove that free will does exist in a universe where light speed is not infinite, even when we don’t consider quantum randomness. Here is the proof:

In a deterministic universe, the information of an event A (or in other words the behavior of a subject A) can be completely known, only if its prior states (or its causes) are completely known.

In a universe where light speed is not infinite, all the causes of an event A, lies on or inside its past light cone. If you don’t know all the causes of the event A, there will always be some uncertainty about A.

Light cone of event A

According to this picture, we can see that only observer B3 is able to know all the prior states of event A, because B3 is in the future of A. It’s impossible for B1 and B2 to know, for example, event C which may affect event A. Because C is not inside or on their past light cone.

For example, when you are chatting with your friend, you will never be able to anticipate that there might be a gamma ray photon hit their brain neurons from the far side of you and unpredictably change their mind.

(Some people may argue that there’s no free will in the presence of an all-knowing God. But that’s another story.)
Localhost Security Messaging
by jefftk — 2023-01-01T02:20:01.515Z

Browsers these days either mark sites with a padlock
(

https://) or "not secure" (

http://). This
warns the users that without the protection of "

https://"
your communications could be read or modified by any network your
packets travel over. But how should "

http://localhost"
be marked? That's your own computer so it's secure, but the
connection isn't encrypted so a padlock would be misleading.

It turns out that the browsers have three options for the url bar, not
just secure and insecure. Here's what they look like in Firefox:

Chrome:

Safari:

Despite the unusual URL bar treatment, the major browsers do now all treat this
configuration as a secure
context (spec),
which means you can use features that require
secure contexts, like crypto,
MIDI,
or geolocation.

Comment via: facebook, mastodon
0 and 1 aren't probabilities
by Alok Singh — 2023-01-01T00:09:27.647Z
Link: https://en.wikipedia.org/wiki/Cromwell%27s_rule

Eliezer wrote about this awhile ago too.

Now for a "radically elementary" take[1].

To simplify, only focus on the case of 0. 1 follows by symmetry. (All values here are non-negative, though with clifford algebras we may extend to negative probabilities too without much difficulty.)

Instead of 0,[2] take an infinitesimal ε∼0. Then we have some quantity that's strictly bigger than 0.

What events are 0 probability?

In the naive setup, these would normally be 0 probability:

Events that are physically possible, but you have no idea how. A lot of the far future falls into this. Also a lot of things beyond your (current) ken.
Events that would normally be given probability exactly 0, like sampling a given point on a unit interval, uniformly at random.
Things you're "sure" won't happen.

In this setup, there's one answer: events that are actually impossible, absurdities, the empty event. Everything else has at least infinitesimal probability. The dart on the interval would have infinitesimal probability, say ε. Now take the event of hitting the number 2. It's not even in the sample space. So it's actually impossible and gets probability 0.

What's the point of this?

What does it get us over what naive numbers may give?

We get a grading.

Example: flipping a coin.

3 outcomes: heads/tails/it lands on its edge. We assign ε to it landing on the edge, because it's possible but was out of scope for the usual coin flip experiment. I got this example by axiomatically assuming something is impossible (a 3rd outcome for coin flips), realized that actually something is physically possible but have no idea how (the ε term). Say that I then realize that the wind being a certain way makes landing on the edge less likely, but the wind acting that way is itself even more unlikely, say a probability of ε2. Then I can update my overall "lands on edge" probability to ε−ε2, and the other 2 to 1−(ε−ε2)2∼12.

Notice that the infinitesimals are placeholders, more valuable for their exponent and coefficient than anything else. If we're "sure" about something, we could give it a value 1−ε. Even better than surety is that someone on 4chan told you, which you could guess a value of 1−ε2. The exact value of ε is less important than the grading it induces, a grading that gets a lot of its value from&nbs...
'simulator' framing and confusions about LLMs 
by Beth Barnes — 2022-12-31T23:38:57.474Z

Post status: pretty rough + unpolished, thought it might be worthwhile getting this out anyway
I feel like I've encountered various people having misunderstandings of LLMs that seem to be related to using the 'simulator' framing. I'm probably being horrendously uncharitable to the people in question, I'm not confident that anyone actually holds any of the opinions that are outlined below, and even if they do I'm not sure that they're actually attributable to the simulators framing, but it seemed like it might be useful to point at areas of potential confusion.

In general I'm skeptical that the simulator framing adds much relative to 'the model is predicting what token would appear next in the training data given the input tokens'. I think it's pretty important to think about what exactly is in the training data, rather than about some general idea of accurately simulating the world.

Perfect predictors

I've encountered people thinking about idealized LLMs that have perfect predictive accuracy, suggesting that e.g. instead of using the model to help you hack into some system, you could just get it to emulate a terminal on that system then extract whatever info you wanted to extract. I think there are two issues here:

Thinking about it as 'you prompt it with some setting in the world, then it predicts this perfectly'

There's not a well-defined correct generalization unless this exact sequence of tokens was actually in the training data. (Paul has a post which talks about this 'what is actually the correct generalization' thing somewhere that I wanted to link, but I can't currently find it)
The 'correct generalization' in some sense is 'what would have followed this if it somehow was included in the training data' - which is not necessarily the 'real' version of the thing you're trying to predict. E.g. if you prompt it to get it to produce the output of some very expensive experiment that humans are unlikely to have actually run, then your model might predict what humans would have written if they'd put a made-up version of this in the training set rather than what would actually happen if you ran the experiment

I think that by the time you can use your model to give you detailed terminal outputs for a specific system, including passwords, entire model weights etc, a bunch of transformative things will already have happened, so it's not really worth th

...
Monitoring devices I have loved
by Elizabeth — 2022-12-31T22:51:22.455Z

Every once in a while a measurement device improves my life a lot at little cost. This post is an attempt to share those data points and gather more, so we can all get a bunch of free gains.

The typical shape of problems solved by measurement devices is as follows: there's something I know might be an issue, but working on it feels very unrewarding. It's a combination of the costs of the issue being poorly quantified, not knowing how much any particular solution would help, and the solution requiring scarce resources like executive function. Measurement devices can help with any of these.

Note: I've provided links to the devices I use on the theory that something is better than nothing. I haven't necessarily tested competitors and there may be better options out there. Amazon links are affiliate, the others are not.

Devices that have helped me in particular

Air quality monitor. I live in California, which has wildfires. Back in 2020 these were especially bad. I closed my windows and wasn't willing to buy more air filters, so I didn't see the point in getting a monitor. At a friend's urging I got one anyway. Turns out my indoor air quality was such that I was willing to buy more filters after all, but also closing the windows didn't help that much. Once I had the new filters I could open my windows enough to let the CO2 dissipate, a huge quality of life improvement.

Once I had the monitor it paid off in other ways. I'd always been vaguely aware stovetop cooking and anything in a spray can were bad for air quality, but I lacked the bandwidth to change my behavior. Once these actions reliably triggered little red lights on the monitor I started turning on the stove vent more, and wearing a mask to use cleaning products. Eventually this formed a habit such that I sometimes remember to do those things even without the air monitor's prompting.

As a bonus, the in-house monitor saved me from having to check air quality on PurpleAir every day, and meant I caught the odd bad day in the off-season.

(This does depend on the monitor being accurate and having the right thresholds for greed/yellow/red. I chose this one mostly for its feature set, but PM2.5 cut-offs do seem right ).

Heart rate variability monitor: Heart rate variability, the delta in the space between your average heartbeat duration, is a pretty good and extremely responsive measure of stress (up to a p...