trentmkelly/LessWrong-43k huggingface dataset:
Inference Speed is Not Unbounded
by OneManyNone — 2023-05-08T16:24:12.820Z

[The intro of this post has been lightly edited since it was first posted to address some comments. I have also changed the title to better reflect my core argument. My apologies if that is not considered good form.]

This post will be a summary of some of my ideas on what intelligence is, the processes by which it’s created, and a discussion of the implications. Although I prefer to remain pseudonymous, I do have a PhD in Computer Science and I’ve done AI research at both Amazon and Google Brain. I spent some time tweaking the language in order to minimize how technical you need to be to read it.

There is a recurring theme I've seen in discussions about AI where people express incredulity about neural networks as a method for AGI since they require so much "more data" than humans to train. On the other hand, I see some people discussing superintelligences that make impossible inferences given virtually no input data, positing AI that will instantly do inconceivable amounts of processing. Both of these very different arguments are making statements about learning speed, and in my opinion mischaracterize what learning actually looks like.

My basic argument is that the there are probably mathematical limits on how fast it is possible to learn. This means, for instance, that training an intelligent system will always take more data and time than might initially seem necessary. What I’m arguing is that intelligence isn’t magic - the inferences a system makes have to come from somewhere. They have to be built, and they have to be built sequentially. The only way you get to skip steps, and the only reason intelligence exists at all, is that it is possible to reuse knowledge that came from somewhere else.

Three Apples and a Blade of Grass

Because I think it makes a good jumping off point, I’m going to start by framing this around a recent discussion I saw around a years-old quote from Yudowsky about superintelligence:

A Bayesian superintelligence, hooked up to a webcam, would invent General Relativity as a hypothesis … by the time it had seen the third frame of a falling apple. It might guess it from the first frame, if it saw the statics of a bent blade of grass.

The linked post does a good job tearing this down. It correctly points out that there are basically an infinite number of possible universes, and three frames of an apple dropping are not nearly enough to c...
[Crosspost] Unveiling the American Public Opinion on AI Moratorium and Government Intervention: The Impact of Media Exposure
by otto.barten — 2023-05-08T14:09:26.266Z
Link: https://forum.effectivealtruism.org/posts/EoqeJCBiuJbMTKfPZ/unveiling-the-american-public-opinion-on-ai-moratorium-and

This is a summary of a follow-up study conducted by the Existential Risk Observatory, which delves into a greater number media items. To access our previous study, please follow this link. This is the second post on the data collected on the second iteration of research done by the Existential Risk Observatory. This second post covers a new aspect of this study: the views of the American general public on the idea of imposing an AI moratorium and their likelihood of voting. To read the first post, please follow this link.

Research Objectives

The objective of this study is to explore the attitudes of the American population towards the concept of imposing a six-month pause on training AI-models more capable than GPT-4 and their inclination to vote, both prior to and after receiving information about the potential existential dangers of AI. The approach taken involved analyzing the changes in the opinions of the study participants towards the AI moratorium, following their exposure to media interventions.

Measurements and Operationalization

The study employed three primary measurements - "Pause Laboratories," "Pause Government," and "Voting Likelihood" - to assess changes in participants' perceptions before and after the media intervention. The "Pause Laboratories" measurement evaluated participants' views on whether AI labs should pause training of AI systems that are more advanced than GPT-4 for at least six months. The "Pause Government" measurement gauged participants' opinions on whether the US government should impose a moratorium on training more powerful AI systems than GPT-4 if the labs do not implement a pause quickly. The "Voting Likelihood" measurement assessed participants' likelihood of voting in a referendum on temporarily halting the training of AI systems, rated on a scale from 0 (definitely not voting) to 10 (certain to vote).

To gather data, the study used Prolific, a platform that locates survey respondents based on predefined criteria. The study involved 300 participants, with 50 participants in each survey, who had to be US residents, fluent in English, and at least 18 years old.

Data Collection and Analysis

Data was collected through surveys in April 2023. The data analysis comprised three main sections: (1) comparing changes in the key indicators before and after the intervention, (2) exploring participants' views on the possibility ...
Thriving in the Weird Times: Preparing for the 100X Economy
by Lucie Philippon — 2023-05-08T13:44:40.341Z

Epistemic status: Confident in the general concept, but not in the specific details.

The world is changing fast. As soon as 2026[1], we may experience an exponential increase in personal productivity, intelligence, and overall capacity to influence the world, powered by new AI tools. To remain relevant in this new era, individuals must embrace these tools.

In this emerging economy, certain factors will determine one's ability to excel:

Easily transmitted and quickly acquired factors, such as access to public AI tools like ChatGPT, AutoGPT, and new AI assistants, as well as novel productivity workflows and strategies[2]. Since the disparity in access to these resources among individuals will be minimal, this post will not focus on them.
Resources that are difficult to transmit and slow to acquire, which will be the key differentiators among people in this rapidly evolving technological landscape. Examples include personal datasets, trained mental habits, and the ability to swiftly adapt to new AI tools and workflows. Gaining these resources might take months or even years and cannot be easily condensed into a brief period. For instance, fully realizing the productivity benefits of new AI tools like GitHub Copilot requires individuals to deeply adapt their coding practices.

To succeed in this environment, it is vital to identify and begin acquiring these slow-to-acquire resources now to gain a competitive advantage. It is preferable to possess unused resources than to encounter insurmountable bottlenecks in the future. Here are some essential resources for this new era:

Mental habits: Learn to effectively use new AI tools, verbalize thoughts and ideas, and manage tight feedback loops with AI. Cultivating a habit of continuous learning and adaptation to new technologies is crucial. Becoming a skilled cyborg will be necessary to harness future AI creativity.
Personal datasets: Assembling and curating personal datasets for use with personal assistants or AI model training could become individuals' most valuable digital assets in the new economy[3]. Start building these datasets now to ensure you have the necessary information for future AI tools. One example is a friend who has recorded everything on his screen for the past two years, which could be leveraged by future AI assistants.
Classic capital: Money will still matter. Access to financial resources allows for

...
Housing and Transit Roundup #4
by Zvi — 2023-05-08T13:30:00.478Z

It’s time for another housing roundup, so I can have a place to address the recent discussions about the local impact of housing construction on housing costs.

Does Increasing Local Housing Supply Decrease Local Housing Prices?

Scott Alexander says Change My Mind: Density Increases Local But Decreases Global Prices. He uses this graph to make his point:

Under this theory, YIMBY is still correct general policy as it lowers prices everywhere else, yet locally increasing supply improves quality of life and thus induces even more demand and raises prices.

Scott Sumner explains why I strongly believe Scott Alexander is wrong here.

Here’s the core point:

Scott Sumner: [Scott Alexander] is aware that the pattern above may show an upward sloping supply curve, not an upward sloping demand curve. But he nonetheless suggests that it’s probably an upward sloping demand curve, and that building more housing in Oakland would make Oakland so much more desirable that prices actually rise, despite the greater supply of housing. I have two problems with this sort of argument.

First, I doubt that it’s true. It is certainly the case that building more housing can make a city more desirable, and that this effect could be so strong that it overwhelms the price depressing impact of a greater quantity supplied. But studies suggest that this is not generally the case.

Texas provides a nice case study. Among Texas’s big metro areas, Austin has the tightest restrictions on building and Houston is the most willing to allow dense infill development. Even though Houston is the larger city, house prices are far higher in Austin.

This is basic common sense.

Places that impose severe NIMBY restrictions get very high prices.

All the most expensive places in America impose such restrictions, and make it impossible to build more housing.

The places that don’t impose such restrictions have lower housing prices than their populations would otherwise suggest.

This theory implies existing homeowners benefit from new construction, as it raises their property values, so they both gain utility and gain resale value. Yet they reliably oppose such construction.

It’s not impossible for all that to be compatible with a tiny increase in housing supply increasing prices due to the fantastic increased value provided by network effects.

...
Dance Profit Sharing
by jefftk — 2023-05-08T13:10:03.652Z

With most endeavors that have a financial component there's a spectrum
of approaches that differ on how they assign risk. At the "low risk"
end you have agreements to pay a specific amount ("we pay
$100/person"), while at the "high risk" end it depends on how things
go go ("everyone gets a share of the take"). Contra dance events
generally go towards the low risk end: most of the time I'm playing
for a fixed fee.

I've been thinking about this, though, because the dance I help run
does profit sharing. The idea is, we guarantee $125/performer
and then if there's profit left over after fixed expenses half of it
goes to the performers. I wasn't around for the initial
implementation, but I think there were probably two reasons:

When a band or caller draws a big crowd that's something we'd like
to encourage, and we're happy to share some of the rewards with
them.

How much money we have available to pay performers depends on
attendance. Profit sharing means that we pay performers more, to
the extent that it doesn't risk our financial health.

You don't want to go too far in the direction of sharing risk, though,
because the event has most of the responsibility for attendance and is
also in a better position than many of the performers to take the loss
from a low-turnout dance.

While I like this system overall, a major downside is that it requires
you to determine how profitable this event was before paying the
performers. I mean, it's possible to just pay everyone the guarantees
and sort it out later, but that means following up with people to send
additional payments, which might be relatively small. So you're
filling out a pretty
complicated form by hand at the dance after counting the money
when you'd rather be dancing.

Could we get similar effects without needing to compute profit in the
moment? What if we use attendance instead? Here's how payments would
look with a bonus of $1 for every attendee over 120:

The main places where these two systems differ is in their handling of
large and small bands. With the current system a duo is much more
likely to get additional money than a quartet, because fixed expenses
are two performers lower. While I do play in a duo and would be tempted
to say otherwise, I don't see a general reason to pay smaller bands
more.

Another nice thing about this structure is that i...
How  "AGI" could end up being many different specialized AI's stitched together
by titotal — 2023-05-08T12:32:39.800Z

[Disclaimer: I'm definitely not the first person to think of this point, but I felt it was worth exploring anyway. Also I am not an ML expert.]

Introduction

When talking about AI, we almost always refer to an AI as a singular entity. We discuss what the goals of a future AI are, and what it’s capabilities are, and what it’s goals are. This naturally leads many people to picture the AI as a singular entity like a human, with a unified goal and agency, where the same “person” does each action. This “generalist” AI, if tasks with performing a variety of tasks, such as writing text, making images, doing math and playing chess, would be structured like this:

However, I actually think AI’s might end up being entities in the way a corporation is an entity. Microsoft may have “goals” and take actions, but inside it consists of many different entities working together, which each do different actions and have different individual goals. This model, the “modular” AI, would involve a manager AI that contracts out specific tasks to specific AI experts in each task, looking like the following:

In the near-term, I expect “modular” AI systems to dominate over “generalist” AI systems. We can build a modular AI now that does all the activities above with performance either comparable to humans or greatly exceeding them. This is not true for a “generalist” AI system. I think it's possible, even likely, that the first systems that look like "AGI" will have this structure.

In this post, I will explain how current AI systems are already modular, why modular AI will be superior in the near term, and some reasons why this might persist in the long term as well. I will explore some of the implications of modular AI for AGI risk as well.

AI systems are already modular with regular code

Pretty much all large programs are “modular”, in that they are divided into subroutines or functions that are separate from each other, interacting via selected inputs and outputs that they feed into each other.

This is good practice for a ton of reasons: it makes the code easier to read and organize, sections of code can be reused in different places and in other programs, code can be split up between people and developed separately, it’s easier to test and debug with only a few inputs and outputs, and it’s easier to scale.

Unsurprisingly, existing AI systems work thi...
What does it take to ban a thing?
by qbolec — 2023-05-08T11:00:51.133Z

Epistemic status: I am not an expert. I just took several things which people banned (child labor, chemical weapons, ozone-depleting substances) and for each just searched for the first article which seriously engages with the question "how did we succeed to ban it?", read it once, and summarized how I understand it. If someone has more examples, or better explanations, I'd be glad to learn.

I think that there's something to learn from examples of bad things that we have banned in the past despite some people benefiting from those bad things. A rosy-eyed, but wrong, image of how that happened is "well, people just realized the thing was bad so, they've banned it". Turns out it is not at all how it has happened.

Ban of chemical weapons

TL;DR: They seem to not be very effective (face masks), may backfire in case of wind ("blowback")

Source: https://www.politico.eu/article/why-the-world-banned-chemical-weapons/

Quotes I've found interesting:

One answer is that while gas attacks are terrifying, the weapon has proved to be militarily ineffective. After Ypres, the allies provided masks to their front-line troops, who stood in their trenches killing onrushing Germans as clouds of gas enveloped their legs. That was true even as both sides climbed the escalatory ladder, introducing increasingly lethal chemicals (phosgene and mustard gas), that were then matched by increasingly effective countermeasures. The weapon also proved difficult to control. In several well-documented instances, gases deployed by front-line troops blew back onto their own trenches — giving a literalist tinge to the term “blowback,” now used to describe the unintended consequences of an intelligence operation.

The world’s militaries are loath to ban weapons that kill effectively, while acceding to bans of weapons that they don’t need.

At the end of World War I, a precise tabulation of casualties showed that some 91,000 soldiers on all sides were killed in gas attacks — less than 10 percent of the total deaths for the entire war. Machine guns and artillery shells, it turns out, were far more effective systems for delivering death.

Among the ban supporters was a Norwegian foreign ministry official who issued an impassioned plea for the adoption of a treaty banning the weapon. In the midst of his talk (which I attended), a British colonel leaned across the table at which I was sitting, a wry smile on

...
A technical note on bilinear layers for interpretability
by Lee Sharkey — 2023-05-08T06:06:59.451Z
Link: https://arxiv.org/abs/2305.03452

Summary

In this short theoretical note (now on Arxiv) I examine bilinear layers, which are MLP layers that take the form

MLPBilinear(x)=(W1x)⊙(W2x).

When used in language models, they perform better than standard MLPs with elementwise activation functions (but appear very slightly below state of the art).

Despite their competitiveness, they are mathematically much easier to analyze: Although they are nonlinear functions of their input, bilinear layers can be expressed using only linear operations and third order tensors.

Because they can be linearized, we can extend 'A Mathematical Framework for Transformer Circuits' (Elhage et al. 2022) beyond attention-only transformers to transformers with both attention and MLP layers.

In a similar way to how the analysis of Elhage et al. (2022) helped to reveal QK- and OV-circuits, induction heads, and virtual attention heads, the analyzability of bilinear layers may lend them to deeper safety insights by allowing us to talk more formally about circuits in large language models.

Additionally, and more speculatively, bilinear layers might offer an alternative path for mechanistic interpretability through understanding the mechanisms of feature construction instead of having to enumerate and understand a (potentially exponentially) large number of features in large models.
Is EDT correct?  Does "EDT" == "logical EDT" == "logical CDT"?
by Vivek Hebbar — 2023-05-08T02:07:17.602Z

Main question: Does EDT / "logical EDT"[1] make any wrong decisions, once the "tickle defense" is accounted for? And is there any real distinction between EDT, "logical EDT", and "logical CDT"?

Relatedly, I'm having trouble constructing any case which distinguishes logical EDT from some kind of "logical CDT" where some facts are "upstream" of others. Does anyone have an example where:

There is a logical fact which could plausibly be upstream of both the agent's decision and some other thing which feels like it "should" be unrelated.

My failed attempt at this:

"There is some law of physics which affects both the agent's decision to chew gum and the number of planets in the universe. The unknown logical fact is the prevalence of that law in the Solomonoff prior."
I think this fails by the tickle defense. It seems plausible for a law of physics to affect the prevalence of gum-chewing agents via {physics -> evolution -> values/impulses}. However, I claim that it "should" cease to correlate with the agent's decision once you condition on what the agent already knows about its own thinking process.[2]

It also seems like EDT is not really distinct from logical EDT, unless you make some special effort to separate logical and factual uncertainty?

Suppose we train a predictor using e.g. supervised learning on past trajectories. Conditioning on action A also gives the predictor informations of logical facts which affected the decision. I claim that the predictor will correctly update in a "logically evidential" way -- it will use the new information to update on other logical facts and will do so in a purely "conditional" way.
It feels like you'd have to make weird design choices to end up with a "non-logical" EDT agent (if that's even a well-defined concept):

If your world model is freely learned, there's no reason for it to make a distinction between logical and factual uncertainty.
If your world model explicitly represents them separately, then you'd have to just not design a Bayesian update procedure for {observation -> updated logical facts}. But it seems like such a procedure would be clearly desirable.
If your world model is explicitly forbidden from modelling logical uncertainty... what? why would you do that? and how?

I'm not up to speed on the literature, but hopefully people will surface th...
LLM cognition is probably not human-like
by Max H — 2023-05-08T01:22:45.694Z

This post is a collection of thought experiments and observations, meant to distill and explain some claims and observations about GPTs and LLMs made by myself and others.

Stated in its most general form, the claim is: the ability of a system or process to predict something well does not imply that the underlying cognition used to make those predictions is similar in structure to the process being predicted.

Applied to GPTs and LLMs specifically: the fact that LLMs are trained to predict text, much of which was generated by humans, does not imply that the underlying cognition carried out by LLMs is similar to human cognition.

In particular, current LLMs are not a distillation of human minds or thoughts, and LLM cognition does not appear to be anthropomorphic. The fact that prediction errors and limitations of current LLMs often mirror human limitations and mimic human errors is not much evidence that the underlying cognition is similar.

Rather, the observed surface-level similarity in these errors is more likely due to current LLM capability at the text prediction task being similar to human-level performance in certain regimes. There are many ways to arrive at a wrong or imperfect answer, and these ways need not be similar to each other.[1]

The focus of this post is on the micro: a GPT predicting a distribution on the next token, or the next few tokens autoregressively . Not considered are agentic loops, specialized prompts, back-and-forth dialog, or chain-of-thought reasoning, all of which can be used to elicit better macro-level performance from LLM-based systems.

On the particular micro-level task of next token prediction, current LLM performance in many examples is often similar to that of an unassisted human who is under time pressure or using less than their full attention. Other times, it is far below or far above this particular human baseline.

Background and related work

Some previous posts by others, which are helpful for context and background on this general topic:

Feature Selection, by Zack Davis. This is a great story for building an intuition for what the internal cognition of a machine learning model might look like.
How An Algorithm Feels From The Inside, by Eliezer Yudkowsky. A classic post, understanding of which helps to clarify the distinction I make in this post between cognition and the outputs of that cognition, and why that distinction is imp

...
If alignment problem was unsolvable, would that avoid doom?
by Kinrany — 2023-05-07T22:13:12.910Z

Suppose there is a useful formulation of the alignment problem that is mathematically unsolvable. Suppose that as a corollary, modifying your own mind while ensuring any non-trivial property of the resulting mind was also impossible.

Would that prevent a new AI from trying to modify itself?

Has this direction been explored before?