trentmkelly/LessWrong-43k huggingface dataset:
Against using stock prices to forecast AI timelines
by basil.halperin — 2023-01-10T16:03:31.985Z

by

Trevor Chow

,

Basil Halperin

, and

J. Zachary Mazlish

[Note: This is an appendix to "AGI and the EMH: markets are not expecting aligned or unaligned AI in the next 30 years"]

One naive objection would be the claim that real interest rates sound like an odd, arbitrary asset price to consider. Certainly, real interest rates are not frequently featured in newspaper headlines – if any interest rates are quoted, it is typically nominal interest rates – and stock prices receive by far the most popular attention.

The importance of real rates. However, even if real interest rates are not often discussed, real interest rates affect every asset price. This is because asset prices always reflect some discounted value of future cash flows: for example, the price of Alphabet stock reflects the present discounted value of future Alphabet dividend payments. These future dividend payments are discounted using a discount rate which is determined by the prevailing real interest rate. Thus the claim that real interest rates affect every asset price.

As a result, if real interest rates are ‘wrong’, every asset price is wrong. If real interest rates are wrong, a lot of money is on the table.

Stocks are hard to interpret. It may nonetheless be tempting to look at stock prices to attempt to interpret how the market is thinking about AI timelines (e.g.

Ajeya Cotra

;

Matthew Barnett

;

/r/ssc

). It may be tempting to consider the high market capitalization of Alphabet as reflecting market expectations for large profits generated by DeepMind’s advancing capabilities, or TSMC’s market cap as reflecting market expectations for the

chipmaker

to profit from AI progress.

However, extracting AI-related expectations from stock prices is a very challenging exercise – to the point that we believe it is simply futile – for four reasons.

First, and most importantly, these companies will only have the possibility of high profits if transformative AI is aligned; under unaligned AI, the value of stocks along with everything else is converted to zero.
Second, it is not obvious that even in the aligned case that these companies will earn high profits. For instance, OpenAI has committed to a capped profit model, and others

may

sign on to a similar ‘

Windfall Clause

’. Beyond corporate altruism, it seems extr

...
Sorting Pebbles Into Correct Heaps: The Animation
by Writer — 2023-01-10T15:58:20.865Z
Link: https://youtu.be/cLXQnnVWJGo

This video is an animation of Sorting Pebbles Into Correct Heaps by Eliezer Yudkowsky. The narration is a word-for-word reading of the story. Enjoy :)

Right after the video is made public, there will probably be some discussion about it in the Rational Animations' Discord server: https://discord.gg/5Y3Dwz89yH

Join in if you want to participate in the debates that will take place, but if you already understand the ins and outs of this story, make sure not to spoil it for other people! Hints and nudges to steer the conversation in productive directions are acceptable and appreciated, though!
Escape Velocity from Bullshit Jobs
by Zvi — 2023-01-10T14:30:00.828Z

Without speculating here on how likely this is to happen, suppose that GPT-4 (or some other LLM or AI) speeds up, streamlines or improves quite a lot of things. What then?

The Dilemma

Samo and Ben’s dilemma: To the extent that the economy is dominated by make-work, automating it away won’t work because more make-work will be created, and any automated real work gets replaced by new make-work.

Consider homework assignments. ChatGPT lets students skip make-work. System responds by modifying conditions to force students to return to make-work. NYC schools banned ChatGPT.

Consider a bullshit office job. You send emails and make calls and take meetings and network to support inter-managerial struggles and fulfill paperwork requirements and perform class signaling to make clients and partners feel appreciated. You were hired in part to fill out someone’s budget. ChatGPT lets you compose your emails faster. They (who are they?) assign you to more in person meetings and have you make more phone calls and ramp up paperwork requirements.

The point of a bullshit job is to be a bullshit job.

There is a theory that states that if you automate away a bullshit job, it will be instantly replaced by something even more bizarre and inexplicable.

There is another theory that states this has already happened.

Automating a real job can even replace it with a bullshit job.

This argument applies beyond automation. It is a full Malthusian economic trap: Nothing can increase real productivity.

Bullshit eats all.

Eventually.

Two Models of the Growth of Bullshit

Samo’s Law of Bullshit: Bullshit rapidly expands to fill the slack available.

Law of Marginal Bullshit: There is consistent pressure in favor of marginally more bullshit. Resistance is inversely proportional to slack.

In both cases, the lack of slack eventually collapses the system.

In the second model, increased productivity buys time, and can do so indefinitely.

Notice how good economic growth feels to people. This is strong evidence for lags, and for the ability of growth and good times to outpace the problems.

A Theory of Escape Velocity

We escaped the original Malthusian trap with the Industrial Revolution, expanding capacity faster than the population could grow. A sufficient lead altered underlying conditions to the point where we should worry more about declining population than rising po...
Scaling laws vs individual differences
by beren — 2023-01-10T13:22:44.394Z

Crossposted from my personal blog

Epistemic Status: This is a quick post on something I have been confused about for a while. If an answer to this is known, please reach out and let me know!

In ML we find that the performance of models tends towards some kind of power-law relationship between the loss and the amount of data in the dataset or the number of parameters of the model. What this means is, in effect, that to get a constant decrease in the loss, we need to increase either the data or the model or some combination of both by a constant factor. Power law scaling appears to occur for most models studied, including in extremely simple toy examples and hence appears to be some kind of fundamental property of how 'intelligence' scales, although for reasons that are at the moment quite unclear (at least to me -- if you know please reach out and tell me!)

Crucially, power law scaling is actually pretty bad and means that performance grows relatively slowly with scale. A model with twice as many parameters or twice as much data does not perform twice as well. These diminishing returns to intelligence are of immense importance for forecasting AI risks since whether FOOM is possible or not depends heavily on the returns to increasing intelligence in the range around the human level.

In biology, we also see power law scaling between species. For instance, there is a clear power law scaling curve relating the brain size of various species with roughly how 'intelligent', we think they are. Indeed, there are general cross-species scaling laws for intelligence and neuron count and density, with primates being on a superior scaling law to most other animals. These scaling laws are again slow. It takes a very significant amount of additional neurons or brain size to really move the needle on observed intelligence. We also see that brain size, unsurprisingly, is a very effective measure of the 'parameter count' at least within species which share the same neural density scaling laws.

However, on the inside view, we know there are significant differences in intellectual performance between humans. The differences in performance between tasks are also strongly correlated with each other, such that if someone is bad or good at one task, it is pretty likely that they will also be bad or good at another. If you analze many such intellectual tasks, and perform factor...
Notes on writing
by RP — 2023-01-10T04:01:52.273Z

Cross-posted from my new blog.
Recently, I gave a few people some advice on how to write well. Below is that advice.

What is it to write well?

Have a thought.
Get that thought from your head to someone else’s.

What is a thought?

In this community, a ‘thought’ tends to be some idea. But, thoughts are not synonymous with ideas. Some thoughts are vibes, forms of inspiration, emotions, etc.

Can’t writing itself come before step (1) of the above?

You can write to think, but that kind of writing is profoundly different from the writing I discuss here: writing to clarify your own thought is not writing to get your thought to someone else.

Must I have thoughts to write well? How can I get thoughts?

It is outside the scope of this piece to teach you to have thoughts. But, until you do, you can’t write well. You really must have something to say.

I have a thought. How do I get it in someone else’s head?

Specific advice

Every word has a unique meaning or a unique set of connotations. Use words precisely. Don’t think about big words or small words—think about which particular word you mean.
This will generally lead to clear writing. Other things also tend to help. For example, clear writing usually entails good grammar. Often, I say things like “we can always break grammar rules if it sounds better”—and this is perfectly true. But, unless you genuinely understand the point of the grammatical rules and why they’re there, you’re probably better off nearly always following them. Grammar rules can be modelled well as Chesterton’s Fences.
There are several books excellent at teaching grammar. My own favourite is The Elements of Style by Strunk and White (various editions exist, all do the job). I’ve heard other things are good, e.g. Pinker’s stuff (according to someone I like) but I haven’t read it.
A common issue I see in this community is something broadly called a reference problem. An example: “Alice ran a workshop, which Barbara attended. She really enjoyed the entire process and event.” Which person does ‘she’ refer to? Alice, or Barbara? I often see this kind of error, usually with words such as ‘it’ and ‘this’. Rationalists don’t usually overcomplicate, but an idea is only simple if I can figure out which specific nouns we’re talking about.
For good writing, clear writing is usually necessary but insufficient. Do not get so bogged down in cle

...
Idea: Learning How To Move Towards The Metagame
by Algon — 2023-01-10T00:58:35.685Z

I have just finished watching someone play a variant of Brick Breaker. A variant I became intimately familiar with, indeed, obsessed, as I strove to hit the top 40 all time score which scrolled past me each time I failed, mocking me.

And this reminded me of an interesting post I read: To Get Good, Go After The Metagame. The title, however, doesn't say it all, so I'll quote the relevant section

"I’ve noticed that the ‘feel’ of improving in pursuit of a meta is strangely similar across skill trees. I noticed this after climbing high enough in the Judo skill tree to see the meta for the first time. Later, when I moved to university, learning the craft of programming became slightly easier. Ditto for content marketing, and email marketing, and so on."

This was interesting to me, as I am currently reviewing rationality techniques which generalize, and I don't think I have noticed this one mentioned elsewhere. In fact, I thought "I have never reached a meta in my life, and it sounds like so very much effort to even see the meta in anything, I guess I just have to take this on faith". But my memory is poor, and I forgot I have reached the meta a couple of times in my life by playing video games. And that when I pursue a video game to mastery, there is a clear difference in the kinds of learning that help me. I suspect, however, that this is not the skill that the post is talking about. So why mention this at all?

Because I think that you might be able to check if this ability exists pretty easily. First, find a video game with a high skill ceiling, ideally one that is new. Then, check if it has a competitive scene where people share their performances. If so, proceed. Else, find another game or consider trying to set up such a scene yourself. Then, play the game until you hit the current meta by analysing what the best players do.

This experiment seems cheap, and has some very high value. So I think I'll pursue it after my current literature review ends, gain this skill, then use it to write an addendum to whatever post I write on Ambitious Methods in Education.
Review AI Alignment posts to help figure out how to make a proper AI Alignment review
by habryka — 2023-01-10T00:19:23.503Z

I've had many conversations over the last few years about the health of the AI Alignment field and one of the things that has come up most frequently (including in conversations with Rohin, Buck and various Open Phil people) is that many people wish there was more of a review process in the AI Alignment field.

I also think there is a bunch of value in better review processes, but have felt hesitant to create something very official and central, since AI Alignment is a quite preparadigmatic field, which makes creating shared standards of quality hard, and because I haven't had the time to really commit to maintain something great here.

Separately, I am also quite proud of the LessWrong review, and am very happy about the overall institution that we've created there, and I realized that the LessWrong review might just be a good test bed and bandaid for having a better AI Alignment review process. I think the UI we built for it is quite good, and I think the vote does have real stakes and a lot of the people voting are also people quite active in AI Alignment.

So this year, I would like to encourage many of the people who expressed a need for better review processes in AI Alignment to try reviewing some AI Alignment posts from 2021 as part of the LessWrong review. I personally got quite a bit of personal value out of doing that, and e.g. found that my review of the MIRI dialogues helped crystallize some helpful new directions for me to work towards, and I am also hoping to write a longer review of Eliciting Latent Knowledge that I also think will help clarify some things for me, and is something that I will feel comfortable linking to later when people ask me about my takes on ELK-adjacent AI Alignment research.

I am also interested in comments on this post with takes for better review-processes in AI Alignment. I am currently going through a period where I feel quite confused how to relate to the field at large, so it might be a good time to also have a conversation about what kind of standards we even want to have in the field.

Current AI Alignment post frontrunners in the review

We've had an initial round of preliminary voting, in which people cast non-binding votes that help prioritize posts during the Review Phase. Among Alignment Forum voters, the top Alignment Forum posts were:

ARC's first technical report: Eliciting Latent Knowledge
What

...
Against the paradox of tolerance
by pchvykov — 2023-01-10T00:12:36.825Z

[The ideas here followed from some great discussion with

Jerome Warren

in Brussels 2022]

Karl Popper’s

Paradox of Tolerance

argues that an absolutely tolerant society is impossible, since it would be vulnerable to intolerant people coming in and breaking it (or just killing everyone). To me this sounds a bit like an unstable equilibrium from dynamical systems theory – trying to balance a pencil on its tip. The paradox is thus that to have a stable tolerant society, it must be intolerant of intolerance – and that immediately creates the gray-zone of what exactly we should be intolerant of and how violent we should be about it.

I’ve been annoyed with this argument ever since I learned about it. My idealist self really wants an absolutely tolerant utopia to be possible, at least in principle, without compromises. As such, I’ve been trying to get a deeper understanding of the argument and its assumptions so as to find some way to dislodge it. Part of my discomfort with it comes from thinking of the stories of non-violent protests, such as exemplified by Jesus, Gandhi, MLK, and others. Now before you point out all the ways in which those people were actually quite intolerant of intolerance, I just want to suspend disbelief and entertain the possibility that the dominant force in their reformations was actually the tolerance itself. To balance a pencil on its tip, we need some forces to hold it up – and very small forces may actually be enough (in fact, we have found many fun ways to balance pencils on their tips - like by using

oscillating drives

or

active control

)

So if we can understand a way in which tolerance itself can be a force for change – an active influence, rather than just passive tranquility - then we may have our solution.

One implicit assumption that the Paradox of Tolerance seems to rest on is that to turn an intolerant person, we must actively and intentionally do something to them. This may involve using force and violence, propaganda, manipulation, seduction, or incentives – but one way or another, we would try to change them, and thus be intolerant towards them.

But let’s now for a second suppose that tolerance is contagious, even just a little bit. By contagious here I mean that merely being tolerant would have some tendency to make others more tolerant – with no effort or even intention to the matter. We could think of this in the context of

mem

...
Increased Scam Quality/Quantity (Hypothesis in need of data)?
by Beeblebrox — 2023-01-09T22:57:31.261Z

Background

I recently received this message from a church that I used to be affiliated with:

"There is an email scam going out to all of [CHURCH_NAME]. It is supposedly from Pastor [John Doe] asking for your assistance in purchasing gift cards (or anything else) and requests asking for confidentiality. They are apparently going down the alphabet. We have so far received three reports and have only reached "B". There may also be a picture of Pastor [John] (taken from the website) attached to the email so that it really looks official.
The only valid church email addresses that Pastor [John] uses are:

jdoe@churchdomain.org
pastor@churchdomain.org
If you do receive such an email, please do not respond!! Promptly report it to https://support.google.com/mail/contact/abuse and also let us know in the office that you were hit. If you are unsure of an email please do not respond and immediately call the church office to verify. If it is afterhours, leave a voicemail on the church phone and I will call you back the next morning.
If you ever get a questionable email from any of the church staff, elders or deacons please call the church office to report and verify."

I've been expecting increasingly personalized scams in 2023, thanks to chatGPT and similar models.

Hypothesis

Small churches, like this one, seem like an ideal audience, due to the combination of:

extremely high trust environments
an older audience that is less adept at picking up on red flags for scam messages
largely unquestioning acceptance of messages from authority figures.

Has anyone else on Lesswrong seen what could be the beginning of more automated scam attempts?

If so, in what context did you see it, how credible/sophisticated was the attempt, and were there obvious AI giveaways inside the attempt?

I'm unsure of if this is an early false alarm, or if it is the tip of a growing iceberg.
Wentworth and Larsen on buying time
by Orpheus16 — 2023-01-09T21:31:24.911Z

This post summarizes a discussion between Thomas Larsen and John Wentworth. They discuss efforts to buy time, outreach to AGI labs, models of institutional change, and other ideas relating to a post that Thomas, Olivia Jimenez, and I released:

Instead of technical research, more people should focus on buying time

.

I moderated the discussion and prepared a transcript with some of the key parts of the conversation.

On the value of specific outreach to lab decision-makers vs. general outreach to lab employees

John:

When thinking about "social" strategies in general, a key question/heuristic is "which specific people do you want to take which specific actions?"

For instance, imagine I have a start-up building software for hospitals, and I'm thinking about marketing/sales. Then an example of what not to do (under this heuristic) would be "Buy google ads, banner ads in medical industry magazines, etc". That strategy would not identify the specific people who I want to take a specific action (i.e. make the decision to purchase my software for a hospital).

An example of what to do instead would be "Make a list of hospitals, then go to the website of each hospital one-by-one and look at their staff to figure out who would probably make the decision to purchase/not purchase my software. Then, go market to those people specifically - cold call/email them, track them down at conferences, set up a folding chair on the sidewalk outside their house and talk to them when they get home, etc.

Applying that to e.g. efforts to regulate AI development, the heuristic would say to not try to start some giant political movement. Instead, identify which specific people you need to write a law (e.g. judges or congressional staff), which specific people will decide whether it goes into effect (e.g. judges or swing votes in congress), which specific people will implement/enforce it (e.g. specific bureaucrats, who may be sufficient on their own if we don't need to create new law).

Applying it to e.g. efforts to convince AI labs to stop publishing advances or shift their research projects, the heuristic would say to not just go convincing random ML researchers. Instead, identify which specific people at the major labs are de-facto responsible for the decisions you're interested in (like e.g. decisions about whether to publish some advancement), and then go talk to th...