trentmkelly/LessWrong-43k huggingface dataset:
LLMs May Find It Hard to FOOM
by RogerDearnaley — 2023-11-15T02:52:08.542Z

Epistemic status: some of the technological progress parts of this I've been thinking about for many years, other more LLM-specific parts I have been thinking about for months or days.

TL;DR An extremely high capacity LLM trained on extremely large amounts content from humans will simulate human content extremely accurately, but won't simulate content from superhumans. I discuss how we might try to change this, and show that this is likely to be an inherently slow process with unfavorable scaling power laws. This might make a fast take-off difficult for any AI based on LLMs.

LLM are trained as simulators for token-generating processes, which (in any training set derived from the Internet) are generally human-like or human-derived agents. The computational capacity of these simulated agents is bounded above by the forward-pass computational capacity of the LLM, but is not bounded below. An extremely large LLM could, and frequently will, produce an exquisitely accurate portrayal of a very average human: a sufficiently powerful LLM may be very superhuman at the task of simulating normal humans with an IQ of around 100, far better at it than any human writer or improv actor — but whatever average human it simulates won't be superhuman, and that capability is not FOOM-making material.

Suppose we had an LLM whose architecture and size was computationally capable in a single forward pass of doing a decent simulation of a human with an IQ of, let's say, O(1000) (to the extent that an IQ that high is even a meaningful concept: let's make this number better defined by also assuming that this is O(10) times the forward-pass computational capacity needed to do a decent simulation of an IQ 100 human). In its foundation model form, this LLM is never going to actually simulate someone with IQ ~1000, since its pretraining distribution contains absolutely no text generated by humans with an IQ of ~1000 (or even IQ over 200): this behavior is way, way out-of-distribution. Now, the Net (and presumably the training set derived from it for this LLM) does contain plenty of text generated very slowly and carefully with many editing passes and much debate by groups of people in the IQ ~100 to ~145 range, such as Wikipedia and scientific papers, so we would reasonably expect the foundation model of such a very capable LLM to also learn the superhuman ability to gene...
Linearity Fallacies
by hippo — 2023-11-15T02:23:36.162Z

A linearity fallacy is committed when you incorrectly assume two variables are related by a linear relationship. For example, assuming that adding people to a project decreases the time to finish proportionally[1] or that happiness scales linearly with income. This kind of fallacy, where you mistakenly draw an x-axis and a y-axis and a straight line going from the bottom-left to the top-right is a linearity fallacy that happens all the time.

But there's more. One day, you're walking along, and you decide to take two measurements of some property at different instances and get two real numbers, x and y. Numbers are numbers, right? Wait, before you go on, you should know that real numbers are powerful objects that rely on many hidden assumptions such as... wait, did you just average the two numbers?

Average Speed

"If a car's speed is 30 mph for 1 mile and 60 mph for the next, what was the average speed over the two miles?"

If you simply average the two numbers given, you arrive at an intuitive, but incorrect 45 mph. Instead, the first mile takes 2 minutes, and the second takes 1 minute, so 2 miles in 3 minutes is 40 mph. The average speed (defined as total distance over total time) is not actually a linear quantity with respect to each mile. It is a linear quantity with respect to each minute. The speed for each minute is 30 mph, 30 mph, 60 mph with a total average of 40 mph. For miles, you want the inverse speed: the average of 1/30 and 1/60 is, conveniently, 1/40.

For this problem, the speed metric is warped, whereas the inverse speed is straight and even. It's like you looked at a mirror, plotted two points on the mirror, and took the midpoint... but the mirror was a funhouse mirror. On the speed number line, 30 to 45 to 60 looks evenly spaced, but 45 is actually closer to 60 than it is to 30. For inverse speed, 1 unit is 1 unit everywhere. On warped number lines, points are not evenly spaced, and on warped mirrors, lines don't even have to be straight. Warped mirrors are everywhere, and they look like real ones.

How is this a linearity fallacy?

A linear map[2] satisfies two properties (addition and scalar multiplication), which can be combined into just f(ax+by)=af(x)+bf(y). If we label the two miles as M and N, the whole road as R, and the speed function as f, then the intuitive but incorrect computation is...
SIA Is Just Being a Bayesian About the Fact That One Exists

by Bentham's Bulldog — 2023-11-14T22:55:43.362Z

Anthopics is often dense and technical, so I’ll begin this by presenting a captivating story. I was chatting with someone and she said “is SSA the better theory of anthropics?” So I launched into a lengthy digression of roughly the following form.

The Self Indication Assumption

claims

“all other things equal, an observer should reason as if they are randomly selected from the set of all possible observers.” Thus, when evaluating two hypotheses, if all else is equal, they should prefer the one that says that there’s a higher probability of them in particular existing. This has some counterintuitive results. Here are two that come broadly from

Bostrom’s famous book

on the subject:

Jackets: There are two hypotheses that are equally probable. Hypothesis one claims that there will be 10 people created with red shirts and 100 with blue shirts. Hypothesis two claims that there will only be 10 people created with red shirts. When you are created, you find yourself with a red shirt. SIA says that, because both hypotheses hold that there will be the same number of people with red shirts created, you should be indifferent between the two. But this is a bit counterintuitive; to many it seems like, because on the first hypothesis most existent people won’t have red shirts and on the second all existent people will have red shirts, this favors the second hypothesis.

The presumptuous philosopher: There are two theories of the physical world. According to one, the universe is infinitely large. According to the other, the universe is very large but finite in size. Because you’re infinitely more likely to be created in the first hypothesis than the second, you should think that the first hypothesis is infinitely more likely than the second.

There are

independent motivations

for SIA; to avoid these results one has to posit a puzzling epistemic asymmetry between failing to create and creation. In fact, the alternative view to the SIA view, the SSA view, according to which one should reason as if they’re randomly selected from the set of all people in their reference class, which are beings that are relevantly like them according to some criteria,

implies utterly nutty things

including that, for example, one can make supernovas unlikely by procreating only if a supernova occurs, and implies that whether you should think that our civilization will die depends on causally isolated ali...
AI Alignment [progress] this Week (11/12/2023)
by Logan Zoellner — 2023-11-14T22:21:06.205Z
Link: https://midwitalignment.substack.com/p/ai-alignment-breakthroughs-this-week-544?utm_source=profile&utm_medium=reader2

It’s hard to deny the fact that most of the oxygen in the room this week was taken up by the OpenAI Dev Day. If you want, feel free to check out my choose-your-own-adventure GPT. I think the jury is out on whether GPTs represent an actual use-case or if like plugins they will be killed by OpenAI’s overly restrictive use-case policy.

That aside, here are our:

AI Alignment Breakthroughs this Week

This week, there were breakthroughs in the areas of:

Math

Mechanistic Interpretability

Benchmarking

Brain Uploading

Making AI do what we want

and

AI Art

Math:

Terry Tao completes a formal proof of his paper using AI

What is it: One of my favorite mathematicians, Terry Tao, just finished the formal proof of one of his papers

What’s new: Tao reports automated tools made writing the paper “only” 5x slower than writing a normal math paper.

What is it good for: Tao found one minor “bug” in his paper during the process. Formal proof can provide better assurance in high trust domains

Rating: 💡💡💡

Mechanistic Interpretability

Transformers can generalize OOD

What is it: a toy example of a transformer that does generalize out of its domain

What’s new: refutes a paper last week by DeepMind claiming transformers cannot generalize OOD

What is it good for: knowing the limits of different ML architectures is key to many alignment plans

Rating: 💡

Benchmarking

Seeing is not always believing

What is it:research shows humans being tricked by Deepfakes 38% of the time

What’s new: I haven’t seen many systematic studies of this

What is it good for: we need to understand the dangerous capabilities of AI in order to know how to regulate them. Although in this case, I expect the number will soon reveal humans are completely unable to recognize fake images.

Rating: 💡

Bingo

What is it: a benchmark for vision model

What’s new: the benchmark highlights cases were the information in the text and image disagree

What is it good for: Training AI to avoid adversarial attacks like this one.

Rating: 💡💡

Brain uploading

Mouse visual Cortex

What is it: researchers discover key insights into mouse vision by looking at their visual cortex

What’s new: To do this, they had to record 1000’s of mouse neurons and used generative AI to reconstruct the features

What is it good for: understanding mouse brains, and hopefully one day ours.

Rating:💡💡💡💡

Making AI Do what we want

Rephrase and Respo...
When did Eliezer Yudkowsky change his mind about neural networks?
by [deactivated] — 2023-11-14T21:24:00.000Z

In 2008, Eliezer Yudkowsky was strongly critical of neural networks. From his post "Logical or Connectionist AI?":

Not to mention that neural networks have also been "failing" (i.e., not yet succeeding) to produce real AI for 30 years now. I don't think this particular raw fact licenses any conclusions in particular. But at least don't tell me it's still the new revolutionary idea in AI.

This is the original example I used when I talked about the "Outside the Box" box - people think of "amazing new AI idea" and return their first cache hit, which is "neural networks" due to a successful marketing campaign thirty goddamned years ago. I mean, not every old idea is bad - but to still be marketing it as the new defiant revolution? Give me a break.

By contrast, in Yudkowsky's 2023 TED Talk, he said:

Nobody understands how modern AI systems do what they do. They are giant, inscrutable matrices of floating point numbers that we nudge in the direction of better performance until they inexplicably start working. At some point, the companies rushing headlong to scale AI will cough out something that's smarter than humanity. Nobody knows how to calculate when that will happen. My wild guess is that it will happen after zero to two more breakthroughs the size of transformers.

Sometime between 2014 and 2017, I remember reading a discussion in a Facebook group where Yudkowsky expressed skepticism toward neural networks. (Unfortunately, I don't remember what the group was.)

As I recall, he said that while the deep learning revolution was a Bayesian update, he still didn't believe neural networks were the royal road to AGI. I think he said that he leaned more towards GOFAI/symbolic AI (but I remember this less clearly).
I've combed a bit through Yudkowsky's published writing, but I have a hard time tracking when, how, and why he changed his view on neural networks. Can anyone help me out?

This post exists only for archival purposes.
Betting on what is un-falsifiable and un-verifiable
by Abhimanyu Pallavi Sudhir — 2023-11-14T21:11:14.943Z

This post is part of my broader aim to try and formulate everything AI in terms of markets so that alignment becomes an antitrust problem. Comments appreciated.

Prerequisites[1].

This post is also available as an arXiv paper.

introduction: betting on what is neither verifiable nor falsifiable (non-VF)

Prediction markets are good at eliciting probabilities about events that will occur at a fixed, finite time. However, there are many questions we would like to bet on that cannot be resolved in finite time. For example:

(Σ1) an asteroid impact will eventually occur; I am mortal; P halts
(Π1) an asteroid impact will never occur; I am immortal; P doesn't halt
(Σ2) there will exist an immortal man; some P_n doesn't halt; this infinite checkerboard has a completely black horizontal row
(Π2) all men will be mortal; every P_n halts; there are infinitely many primes
(Π3) sentences about limits e.g. x_n diverges to +infinity

As a sigma-algebra, the Σn+1 sentence ∃x,P(x) should be seen as the countable union ⋃∞x=1P(x) and so its probability should be the supremum of all the finite unions. Similarly, a Π sentence should be seen as a countable intersection and its probability as an infimum. This is fine in probability theory, but I mean, "supremum" and "infimum" are not computable functions, you would need to elicit an infinite number of bets.

Indeed, the desire to consider infinite events is basically the motivation for using sigma-algebras rather than set algebras in probability theories. Set algebras correspond to propositional logic; sigma-algebras to predicate logic (to be sure, to second-order logic rather than first, because you can take any countable unions and intersections, not necessarily even enumerable in a computable way, which would correspond to some hyperarithmetic theory).

Actually Σ1 (verifiable) and Π1 (falsifiable) sentences are still OK: for verifiable sentences, just pay the trader if/when the sentence is verified; for falsifiable sentences, the trader is paid $1 upfront and returns it if the sentence is falsified. These are what we call "long" and "short" positions respectively.

So philosophers will tell you there is nothing beyond Σ1 and Π1, but surely they must be wrong? Because we might actually care about sentences like "there are infinitely many primes" and "all men are ...
Facebook is Paying Me to Post
by jefftk — 2023-11-14T19:10:07.303Z

In mid-September Facebook asked me if I wanted to start getting paid
for posting things:

This seems to be the performance
bonus program. I think they offered it to me because my Facebook
profile is in professional
mode and I post a lot? It's all very unclear, including what
effect saying yes would have: how would they decide how much to pay
me? Would this affect how the algorithm prioritizes my writing?
Would they attempt to exert editorial control? One way to find out!

I decided to try joining, and started 2023-09-13.

On 2023-10-23 I received a payment of $16.31 for September. There
are no details on how they decided what to pay me, which is too bad: I
was at least hoping for some information about rates.

FB does give data on "reach" and "engagement" so it does look like I
can see if this was having any effect. The data is in the form of
awkward charts, not CSVs I can export, but still there's something:

.

That "reach" spike right around when I signed up is very interesting!
Did Facebook start showing my posts to more people in response to me
signing up, and then give up when peopld didn't engage? While it's
hard to read the chart, however, squinting at the timing I think the
spike is actually on 2023-09-12 or 2023-09-11. So this went the other
way around: I had a post ("Apple Cider
Baklava") that was unusually popular among strangers and in
response FB decided to offer to offer me money.

Overall this seems a bit silly: while I don't mind taking a few tens
of dollars to keep doing what I was going to do anyway, a deal where
they don't say how you'll be compensated or how they've figured your
payments is pretty suspect.

Comment via: facebook, mastodon
Feelings, Nothing More than Feelings, About AI
by PaulBecon — 2023-11-14T18:50:24.275Z

FOBO is the Fear of Becoming Obsolete

ChatGPT came out at the end of 2022, but over 100MM have fooled around and found out for themselves at least a little about LLMs (Large Language Models). The arc of fascination has even roller coasted past the initial infatuation, as some people now feel it’s been getting dumbed down. The cognitive belt notches continue to accumulate, as it blows past AP exams, conquers the bar exam, and is a peer of radiologist MDs in accurate diagnoses.

Instead of discussing the IQ of AIs, I’m far more interested in humans, and their complex emotional responses to suddenly encountering these alien minds. "Algorithm Aversion," a term coined by

behavioral scientists

in 2015, encapsulates the harsher standards humans insist upon when decision-making is algorithmic. We're currently seeing this with San Francisco's revocation of driverless car company Cruise's license to carry passengers. Merely being far safer than human drivers is not sufficient. When algorithms make mistakes, people lose confidence in them more quickly than they would with human error.

What are we willing to hand off to blind computer processing? Currently, programmers are most intimately exposed to the significant productivity impact of using GPTs to supplement their traditional methods of working. Memorizing gnarly, arcane language conventions is now as obsolete as tricks for calculating square roots in the wake of handheld calculators. Instead of picking through thorny tangles of code to find bugs, a code "co-pilot" can ingest a hairball of code and generate its own 'best guess' as to the right code.

The term 'co-pilot' finesses the question as to who's in the driver's seat. Coders initiate the conversation with a GPT, they direct the AI to focus on particular lines, and they make the final decision as to whether to incorporate, reject, or tweak the AI's proposal.

Humans can get snagged on preserving their own sense of autonomy. We can get touchy/titchy in reaction to computers driving human decisions. The cost of algorithmic aversion can be measured by the way financial recommendations are treated quite differently when presented as the output of a computer vs a human. Decades of evidence has piled up to unambiguously prove that human financial advisors, as a group, NEVER outperform robo-automated, well-diversified, low cost index funds. People, as investors, ...
Kids or No kids
by Kids or no kids — 2023-11-14T18:37:02.799Z

This post summarizes how my partner and I decided whether to have children or not. We spent hundreds of hours on this decision and hope to save others part of that time. We found it very useful to read the thoughts of people who share significant parts of our values on the topic and thus want to "pay it forward" by writing this up. In the end, we decided to have children; our son is four months old now and we’re very happy with how we made the decision and with how our lives are now (through a combination of sheer luck and good planning). It was a very narrow and very tough decision though.

Both of us care a lot about having a positive impact on the world and our jobs are the main way we expect to have an impact (through direct work and/or earning to give). As a result, both of us are quite ambitious professionally; we moved multiple times for our jobs and work 50-60h weeks. I expect this write-up to be most useful for people for whom the same is true.

Bear in mind this is an incredibly loaded and very personal topic - some of our considerations may seem alienating or outrageous. Please note I am not at all trying to argue how anyone should make their life decisions! I just want to outline what worked well for us, so others may pick and choose to use part of that process and/or content for themselves.

Finally, please note that while many readers will know who I am and that is fine, I don’t want this post to be findable when googling my name. Thus, I posted it under a new account and request that you don’t use any personal references when commenting or mentioning it online.

Process - how we decided

We had many sessions together and separately, totaling hundreds of hours over the course of 2 years, on this decision and the research around it. My partner tracked 200 toggl hours, I estimate I spent a bit less time individually but our conversations come on top. In retrospect, it seems obvious, but it took me longer than I wish it would have to realize that this is important, very hard work, for which I needed high-quality, focused work time rather than the odd evening or lazy weekend.

We each made up our minds using roughly the considerations below - this took the bulk of the time. We then each framed our decision as "Yes/No if xyz", for instance, “Yes if I can work x hours in a typical week”, and finally “negotiated” a plan under which we c...
Raemon's Deliberate (“Purposeful?”) Practice Club
by Raemon — 2023-11-14T18:24:19.335Z

Introduction

So. I have a theory of feedbackloop-first rationality.

It has a lot of parts. I think each part is promising on it's own, and I have a dream that they interconnect into something promising and powerful. I also have a standard, which is that you should be able to tell if it's helping.

One of those parts (I think/hope), is "the generalized skill of Deliberate Practice." That is, the meta skill of:

Noticing that your goals are bottlenecked on some kind of skill (or skills).
Figuring out what those specific skills are.
Figuring out who can teach you those skills, or, how to teach them to yourself.
Creating an explicit practice regime.
Actually putting in the work to practice.
Noticing when your practice isn't working, and figuring out how to troubleshoot your process.

I do not currently have this meta-skill. I am kind of betting that it exists, based on reading books like Peak, talking with Romeo Stevens, and reading stories like László Polgár who methodically taught his daughters chess.

I think I've made progress in the two months I've been working on it, but that progress hasn't translated into "I quickly gained multiple skills" yet, which is the standard I feel like I should set for "this is actually working well enough that other people should be paying attention."

I'm experimenting with using this my dialogue format for journaling my explorations here. I'm inviting a few people I know well to be top-level dialogue participants. Everyone else is welcome to follow along in the comments, and note down their own deliberate practice experiments.

This will include a mixture of high level theory, and day-to-day practice notes.

Raemon

Okay, reviewing some of my goals here. Here are things that feel like valuable end-goals in and off themselves.

I want to get better at prioritizing projects at Lightcone. Right now I feel very "in the dark" about whether anything we do is even helping. I have some guesses for the subskills here.
I want to figure out whether/to-what-degree the Meta Deliberate Practice skill can meaningfully be applied to "research" (alignment research in particular, but also generally).
Get better at programming.
Get better at emotional regulation. Moderately often, I get somewhat annoyed about something and it makes a conversation go worse (or, builds up some small resentments over time)
Get better at sleeping, somehow.
Get bett

...