trentmkelly/LessWrong-43k huggingface dataset:
[Linkpost]Transformer-Based LM Surprisal Predicts Human Reading Times Best with About Two Billion Training Tokens
by Curtis Huebner — 2023-05-04T17:16:33.335Z
Link: https://arxiv.org/abs/2304.11389

From the abstract:

Recent psycholinguistic studies have drawn conflicting conclusions about the relationship between the quality of a language model and the ability of its surprisal estimates to predict human reading times, which has been speculated to be due to the large gap in both the amount of training data and model capacity across studies. The current work aims to consolidate these findings by evaluating surprisal estimates from Transformer-based language model variants that vary systematically in the amount of training data and model capacity on their ability to predict human reading times. The results show that surprisal estimates from most variants with contemporary model capacities provide the best fit after seeing about two billion training tokens, after which they begin to diverge from humanlike expectations. Additionally, newly-trained smaller model variants reveal a 'tipping point' at convergence, after which the decrease in language model perplexity begins to result in poorer fits to human reading times. These results suggest that the massive amount of training data is mainly responsible for the poorer fit achieved by surprisal from larger pre-trained language models, and that a certain degree of model capacity is necessary for Transformer-based language models to capture humanlike expectations.

The authors show strongly U-shaped curves relating the additional predictive power that adding per-token suprisal gives to a regression model and model perplexity.

From the results section:

The results in Figure 13 show that across both cor-
pora, surprisal from most LM variants made the
biggest contribution to regression model fit after
1,000 training steps (i.e. after ∼2B tokens). This
seems to represent a ‘humanlike optimum,’ after
which surprisal estimates begin to diverge from hu-
manlike expectations as training continues. At this
point in training, there appears to be no systematic
relationship between model capacity and predictive
power of surprisal estimates.

I'm not going to speculate too much but these curves remind me a lot of the diagram describing Chris Olah's view on the difficulty of mechanistic interpretability with model power:
Other remarks:
-It would be neat to compare the perplexity inflexion point with the Redwood estimates of human perplexity scores on text.
-Performance seems well tied to model perplexity and tokens but not so much to model size...
Clarifying and predicting AGI
by Richard_Ngo — 2023-05-04T15:55:26.283Z

This post is a slightly-adapted summary of two twitter threads, here and here.

The t-AGI framework

As we get closer to AGI, it becomes less appropriate to treat it as a binary threshold. Instead, I prefer to treat it as a continuous spectrum defined by comparison to time-limited humans. I call a system a t-AGI if, on most cognitive tasks, it beats most human experts who are given time t to perform the task.

What does that mean in practice?

A 1-second AGI would need to beat humans at tasks like quickly answering trivia questions, basic intuitions about physics (e.g. "what happens if I push a string?"), recognizing objects in images, recognizing whether sentences are grammatical, etc.
A 1-minute AGI would need to beat humans at tasks like answering questions about short text passages or videos, common-sense reasoning (e.g. Yann LeCun's gears problems), simple computer tasks (e.g. use photoshop to blur an image), justifying an opinion, looking up facts, etc.
A 1-hour AGI would need to beat humans at tasks like doing problem sets/exams, writing short articles or blog posts, most tasks in white-collar jobs (e.g. diagnosing patients, giving legal opinions), doing therapy, doing online errands, learning rules of new games, etc.
A 1-day AGI would need to beat humans at tasks like writing insightful essays, negotiating business deals, becoming proficient at playing new games or using new software, developing new apps, running scientific experiments, reviewing scientific papers, summarizing books, etc.
A 1-month AGI would need to beat humans at coherently carrying out medium-term plans (e.g. founding a startup), supervising large projects, becoming proficient in new fields, writing large software applications (e.g. a new OS), making novel scientific discoveries, etc.
A 1-year AGI would need to beat humans at... basically everything. Some projects take humans much longer (e.g. proving Fermat's last theorem) but they can almost always be decomposed into subtasks that don't require full global context (even tho that's often helpful for humans).

Some clarifications:

I'm abstracting away from the question of how much test-time compute AIs get (i.e. how many copies are run, for how long). A principled way to think about this is probably something like: "what fraction of the world's compute is needed?". But in most cases I expect that the bottleneck is being able to perform a task *at

...
[Crosspost] AI X-risk in the News: How Effective are Recent Media Items and How is Awareness Changing? Our New Survey Results.
by otto.barten — 2023-05-04T14:09:00.134Z
Link: https://forum.effectivealtruism.org/posts/YweBjDwgdco669H72/ai-x-risk-in-the-news-how-effective-are-recent-media-items

This is a summary of a follow-up study conducted by the Existential Risk Observatory, which delves into a greater number media items. To access our previous study, please follow this link. The data collected will be presented in two separate posts. The first post, which is the current one, has two parts. The first part examines the key indicators used in the previous research, such as "Human Extinction Events" and "Human Extinction Percentage," along with a new key indicator called "Concern Level." The Concern Level indicator assesses participants' level of concern about AI existential risk on a scale of 0 to 10 before and after the intervention. The second part analyzes the changes in public awareness about AI existential risk over time. It also explores the connection between the effectiveness of different media formats, namely articles and videos, and their length in raising awareness. In addition, it investigates how trust levels are related to the effectiveness of media sources in increasing public awareness of AI existential risk. In the second post, the research covers a new aspect of this study: participants' opinions on an AI moratorium and their likelihood of voting for it.

PART 1: Effectiveness per media item

This research aimed to evaluate the effectiveness of AI existential risk communication in increasing awareness of the potential risks posed by AI to human extinction.

Research Objectives: The objective of the study was to determine the effectiveness of AI existential risk communication in raising public awareness. This was done by examining the changes in participants' views on the likelihood and ranking of AI as a potential cause of extinction before and after the intervention. Furthermore, the study evaluated the difference in the level of concern of participants before and after the intervention.

Measurements and Operationalization: Three primary measurements - "Human Extinction Events," "Human Extinction Percentage," and "Concern Level" - were utilized to examine alterations in participants' perceptions. The coding scheme that was previously used in our research was employed to assess participants' increased awareness of AI. The data was gathered through Prolific, a platform that locates survey respondents based on predefined criteria. The study involved 350 participants, with 50 participants in each survey, who were required to be at le...
AI #10: Code Interpreter and Geoff Hinton
by Zvi — 2023-05-04T14:00:00.366Z

The big capabilities news this week is a new ChatGPT mode (that I do not have access to yet) called Code Interpreter. It lets you upload giant data files, analyzes them automatically, can even write papers about its findings, many are impressed.

The big discourse news is that Geoff Hinton, the Godfather of AI, quit his job at Google in order to talk freely about the dangers of AI, including and especially existential risks. He now expects AGI within 5 to 20 years, and has no idea how we might plausibly have this go well.

Oh, and Kamala Harris is meeting with the Google, OpenAI, Microsoft and Anthropic CEOs about AI safety, I am sure policy will get off to a great start.

Also a wide variety of other stuff.

Table of Contents

Introduction.

Table of Contents.

Language Models Offer Mundane Utility. Incremental progress every week.

ChatGPT’s New Code Interpreter. The data scientist will see you now.

Introducing (New AI Products). Keep on wrapping and prompt engineering.

Fun With Image and Sound Generation. MidJourney 5.1 dropped.

The Art of the SuperPrompt. It’s all about the SuperPrompts.

Deepfaketown and Botpocalypse Soon. Why not go deepfake yourself?

The For Now Accurately Named FTC Blogpost Trilogy. You’re on notice.

The Real Problem With AI is Badwordism. An alternative universe’s viewpoint.

Go Go Gadget AutoGPT What Could Possibly Go Wrong. Do not pay. Or play.

In Other AI News.

China. Reuters is confused about chip restrictions.

What Even is a Superforecaster. More info on that superforecast.

Sam Altman Interview with the Free Press. Checking for any noticable evolutions.

Potential Future Scenario Naming. Scott Aaronson, Boaz Barak and the nature of Futurama.

Think Outside of the Box. We will hand the AI the reigns of power.

They Took Our Jobs. Hollywood writers strike back.

Quiet Speculations. Simplicity is in the eye of the beholder, and more.

People Are Worried About AI Before It Kills Everyone. Scenarios are proposed.

Our Words Are Backed By Nuclear Weapons. Let’s not hook the AIs up to them?

Geoff Hinton is Worried About AI Killing Everyone. The Godfather of AI quits Google to warn us about the dangers of AI, including the existential risks.

Other People Are Also Worried About AI Killing Everyone. Some solid explanations and resources, including a list of arguments and some

...
Advice for interacting with busy people
by Severin T. Seehrich — 2023-05-04T13:31:06.709Z

Over the last year, I have turned from somebody with a lot of time and relatively few social contacts to maintain into someone quite busy and well-networked. I'm by no means close to the top of the busyness hierarchy. But, I've climbed high enough to learn some lessons that I wish I knew before, when I had a lot of time and even more unearned confidence.

So here you go for what I've found, so that you shall make wiser life decisions than my younger self.

Note: The intended message of this post is not "Don't reach out to busy people!", but "Do reach out, and have these things in mind to make it more likely to get a response/if you don't get one."

1. The time and attention of central information nodes is more valuable than ours. Let's not waste it.

This is the one rule to rule them all - everything else I write here is just variations on it.

For example: Let's assume world-leading AGI alignment researcher Paul Christiano has put extensive research into finding the optimal soap dispenser, not only in regards to functionality and aesthetics, but also in regards to price. I still shouldn't ask him which soap dispenser to buy.

For even if he was the world-leading expert on buying cheap high-quality soap dispensers, it would be very unwise of me to waste one of his afternoons on a soap dispenser shopping tour. He is not the only one who could help me with that. And, the question which soap dispenser to pick for my bathroom is not very important in the first place. It would be unfortunate if we were to live in one of the branches of reality where humanity goes extinct because I wasted Paul Christiano's time on soap dispenser advice.

And, I hope that Christiano thinks the same and would never talk to me again if I were to make such a request. For I, too, care about the survival of humanity, and would prefer others to stop me if I threaten it.

2. Be swift and clear in your requests.

In my undergrad years, I studied continental philosophy and classical philology and was very, very proud of the aesthetic quality of my writing. Thus, I annoyed many a professor with way too flowery and way too long walls of text that could have been one sentence. Some professors found me adorable, others didn't quite appreciate me eating pieces of their time that they could have spent on, say, Wittgenstein instead. So don't be me, keep things as concise and clear as possible.

The complementary fail...
We don’t need AGI for an amazing future
by Karl von Wendt — 2023-05-04T12:10:59.536Z

Most of the public discussions about AI safety seem to assume that there is only one plausible path for AI development: from narrow, specialized AI to broad AGI to superintelligence. So we seemingly have to decide between an amazing future with AGI at the risk that it will destroy the world or a future where most of humanity’s potential is untapped and sooner or later we get destroyed by a catastrophe that we could have averted with AGI. I think this is wrong. I believe that even if we assume that we’ll never solve alignment, we can have an amazing future without the existential risks that come with a superintelligent, self-improving AGI.

As Irving J. Good has stated long ago, an AGI would likely be able to self-improve, leading to an intelligence explosion. The resulting superintelligence would combine the capabilities of all the narrow AIs we could ever develop. Very likely it could do many things that aren’t possible even with a network of specialized narrow AIs. So, from an intelligence perspective, self-improving AGI is clearly superior to narrow AI (fig. 1).

Fig. 1: AI development (hypothetical)

But of course there’s a catch: a self-improving AGI will be uncontrollable by humans and we have no idea how to make sure that it will pursue a beneficial goal, so in all likelihood

this will lead to an existential catastrophe

. With the prospect of

AGI becoming achievable within the next few years

, it seems very unlikely that we’ll solve alignment before that. So, from a human values perspective, a very different picture emerges: above a certain level of intelligence, the expected value of a misaligned AGI will be hugely negative (fig. 2).

Fig. 2: Value for humanity of aligned and misaligned AGI

The only sure way to avoid an existential catastrophe from misaligned AGI is to not develop one. As long as we haven’t solved alignment, this means not developing AGI at all. Therefore, many people call for an

AGI moratorium

or even

a full stop in AI development

, at least until we have found a solution to the alignment problem. I personally support an AGI moratorium. I have signed the

open letter of the Future of Life Institute

although I think 6 months are nowhere near enough time to solve the problems associated with AGI, simply because a 6 months pause is better than just racing full-speed ahead and I view the coordination required for that as a p...
Has the Symbol Grounding Problem just gone away?
by RussellThor — 2023-05-04T07:46:09.444Z

A few years ago, the symbol grounding problem was widely considered a significant challenge in AI discussions. I believed that it would likely be addressed as a side effect of capability improvements, without requiring specific breakthroughs or attention, unlike others who considered it fundamental. However, like many, I didn't anticipate the extent to which GPT-4 has demonstrated this to be true. Asserting that such capabilities could be achieved with a text-only training set at that time would have seemed like a parody of my position.
Had you asked me how a model like GPT-4 would acquire its capabilities, I would have suggested a process more akin to how children learn. It might have started with a predictive model of physical reality, established the concept of an object, learned object permanence, and then acquired simple words as they applied to previously learned concepts.
Despite its capabilities, GPT-4 still seems to lack robust physical intuition, evident in data science tasks and mathematical understanding related to the 3D world. Will we see a model trained from scratch, as described earlier? For instance, the Meta AI model appears to grasp the concept of an object in 2D. Suppose this understanding is fully extended to 3D and designated as our foundational model. Can such a model be trained on language, particularly that which relates to the physical world, and develop physical intuition as good as a human's?

Grounding overhang and interpretability implications:

Would such a model be much better at mathematical and programming tasks for given model resources? Assuming the foundational model is much smaller than GPT-4 it seems reasonable that it could gain similar or greater mathematical and programming skills while still having a smaller model size even when trained on a enough language concepts to be capable at those tasks.

This could also help with interpretability as the foundational model couldn't really be thought to lie or deceive as it is just modelling objects in 3d. Deception and theory of mind abilities could be observed as they became available.
Opinion merging for AI control
by David Johnston — 2023-05-04T02:43:51.196Z

Thanks to Justis for proofreading and feedback.

This is a simplified followup to my post on "mimics". I think the basic idea there is valuable and still does not appear to be widely appreciated, so I'm trying to explain it again.

Introduction: the misjudgment problem

Suppose I try to make a very powerful AI system, and I plan to ask it to make me rich. If it works well, this seems like it would be a great outcome to me, but maybe I'm wrong about this, in which case developing this system might be a bad decision on my part.

Assume I make a probabilistic forecast about possible futures. My forecast can be illustrated as follows:

For the moment, let's assume this is a naive forecast; I'm not trying very hard to account for the fact that I'm delegating to an AI system to make me rich, I'm just imagining what it would be like to be rich in the future on base rates[1]. It looks good, and I think I can pull off the AI-make-me-rich project, so I go for it. However, Cassandra, who is much smarter and better calibrated than me, has a different view. She has taken into account the fact that I'm delegating to an AI system to do the job, and she sees a different set of possible outcomes:

On the far left we have "specification gaming"; the AI makes me rich, but my life actually gets substantially worse as a result. On the far right, we have "goal misgeneralisation"; the AI learned how to make money, but not to make money for me. Cassandra advises me not to pursue my project (but for some reason I don't listen).

So the basic story is:

I forecast the consequences of deploying my AI according to some naive scheme, and they look good so I go for it
If I had much better forecasting ability, I would have seen that this was actually a bad idea

I'm going to call this the misjudgement problem. It is far from the only problem with developing advanced AI, but it seems like one of the bigger ones to me[2]. The topic of this post is how merging of opinions might be used to address this problem.

Addressing the misjudgement problem with opinion merging

Background

For this post, I will be talking about generative AI systems that generate output without side-effects (in my previous post, I called such systems "mimics").

Suppose I have a text generating AI system. Anywhere I would normally create my own text (broadly defined - including code, messages, filling out forms etc.), I can substitute my AI sys...
Understanding why illusionism does not deny the existence of qualia
by Mergimio H. Doefevmil — 2023-05-04T02:13:57.597Z

It is obviously true that "🟩", "greenness", and "activity in the visual cortex" are not identical things. For example, "greenness" is one word containing the letter "g", but not the letter "a", whereas "activity in the visual cortex" is five words containing the letter "a", but not the letter "g", and "🟩" does not contain any letters at all.

Usually, when someone says that "🟩" is "greenness", they do not mean that the word "greenness", in a thought, spoken, or written form, is strictly identical to "🟩", or that "🟩" does not exist, and only "greenness" does. What they mean is that "greenness" is a useful label to describe "🟩". It is generally understood that labels are not the things that they are describing.

This understanding breaks somewhat when discussing illusionism - which can, at least partially, be attributed to illusionists not being very good at explaining their position.

When someone says that "🟩" is "greenness", we do not accuse that person of denying the existence of "🟩". Why then, when someone says that "🟩" is "activity in the visual cortex", do we make such an accusation?

An illusionist would not deny the existence of "🟩" (perhaps some illusionists would, Keith Frankish certainly does not). An illusionist would merely claim that so far, the most useful framework that has been discovered for verbally reasoning about "🟩" is the framework of brain configurations and activity in the visual cortex.

Some would say that the analogy presented here is unfair, because the word "greenness" generally only has one referent, whereas the phrase "activity in the visual cortex", according to illusionism as explained here, has at least two referents, namely "🟩" and micromovements in a grey, squishy, and wet object.

However, that does not make the analogy unfair. The core claim I am attempting to refute here is that illusionists somehow deny the existence of "🟩". That a term has multiple referents is irrelevant as an objection to my refutation.

Another point that could be raised is that it is not evident that the framework of activity in the visual cortex is the most useful framework in which to reason about "🟩". That point however is also irrelevant as an objection to my claim that illusionists do not deny the existence of "🟩" when they say that "🟩" is "activity in the visual cortex" - analogously to how people do not deny the existence of "🟩" when the...