trentmkelly/LessWrong-43k huggingface dataset:
A Simple Alignment Typology
by Shoshannah Tekofsky — 2023-01-28T05:26:36.660Z

I set out to review the OpenAI alignment plan, and my brain at some point diverged to modeling the humans behind the arguments instead of the actual arguments.

So behold! A simplified, first-pass Alignment Typology.

Why can't we all just get agree?

There are a lot of disagreements in AI alignment. Some people don't see the problem, some think we'll be fine, some think we're doomed, and then different clusters of people have different ideas on how we should go about solving alignment. Thus I tried to sketch out my understanding of the key differences between the largest clusters of views on AI alignment. What emerged are roughly five cluster, sorted in order of optimism about the fate of humanity: the sceptics, the humanists, the empiricists, the rationalists, and the fatalists.

Sceptics don't expect AGI to show up in any relevant time frame.

Humanists think humanity will prevail fairly easily through coordination around alignment or just solving the problem directly.

Empiricists think the problem is hard, AGI will show up soon, and if we want to have any hope of solving it, then we need to iterate and take some necessary risk by making progress in capabilities while we go.

Rationalists think the problem is hard, AGI will show up soon, and we need to figure out as much as we can before making any capabilities progress.

Fatalists think we are doomed and we shouldn't even try (though some are quite happy about it).

Here is a table.

Sceptics	Humanists	Empiricists	Theorists	Fatalists
Alignment Difficulty	-

One of these

is low

high	high	-
Coordination Difficulty	-	high	high	-
Distance to AGI	high	-	low/med	low/med	-

Closeness to AGI required

to Solve Alignment

-	-	high	med/high	-

Closeness to AGI resulting

in unacceptable danger

-	-	med/high	high	-

Alignment Necessary

or Possible

-	high	high	high	low

Less Wrong is mostly populated by empiricists and rationalists. They agree alignment is a problem that can and should be solved. The key disagreement is on the methodology. While empiricists lean more heavily on gathering data and iterating solutions, rationalists lean more heavily toward discovering theories and proofs to lower risk from AGI (and some people are a mix of the two). Just by shifting the weights of risk/reward on iteration and moving forward, you get two opposite approaches to doing alignment work.

How is this useful?

Personally it helps me quickly get an...
Spooky action at a distance in the loss landscape
by Jesse Hoogland — 2023-01-28T00:22:46.506Z
Link: https://www.jessehoogland.com/note/spooky-action-at-a-distance-in-the-loss-landscape

Produced under the mentorship of Evan Hubinger as part of the SERI ML Alignment Theory Scholars Program Winter 2022 Cohort.

Not all global minima of the (training) loss landscape are created equal.

Even if they achieve equal performance on the training set, different solutions can perform very differently on the test set or out-of-distribution. So why is it that we typically find "simple" solutions that generalize well?

In a

previous post, I argued that the answer is "singularities" — minimum loss points with ill-defined tangents

. It's the "nastiest" singularities that have the most outsized effect on learning and generalization in the limit of large data. These act as implicit regularizers that lower the effective dimensionality of the model.

Singularities in the loss landscape reduce the effective dimensionality of your model, which selects for models that generalize better.

Even after writing this introduction to "singular learning theory", I still find this claim weird and counterintuitive. How is it that the local geometry of a few isolated points determines the global expected behavior over all learning machines on the loss landscape? What explains the "spooky action at a distance" of singularities in the loss landscape?

Today, I'd like to share my best efforts at the hand-waving physics-y intuition behind this claim.

It boils down to this: singularities translate random motion at the bottom of loss basins into search for generalization.

Random walks on the minimum-loss sets

Let's first look at the limit in which you've trained so long that we can treat the model as restricted to a set of fixed minimum loss points[1].

Here's the intuition pump: suppose you are a random walker living on some curve that has singularities (self-intersections, cusps, and the like). Every timestep, you take a step of a uniform length in a random available direction. Then, singularities act as a kind of "trap." If you're close to a singularity, you're more likely to take a step towards (and over) the singularity than to take a step away from the singularity.

If we start at the blue point and uniformly sample the next location among the seven available locations for a fixed step size, we have 6:1 odds in favor of moving towards the singularity.

It's not quite an attractor (we're in a stochastic setting, where you can and will still break away every so often), but it's sticky en...
WaPo: "Big Tech was moving cautiously on AI. Then came ChatGPT."
by Julian Bradshaw — 2023-01-27T22:54:50.121Z
Link: https://www.washingtonpost.com/technology/2023/01/27/chatgpt-google-meta/

Some good reporting here, got quotes from a lot of people. Here are the key paragraphs, in my opinion:

In the past year or so, top AI researchers from Google have left to launch start-ups around large language models, including Character.AI, Cohere, Adept, Inflection.AI and Inworld AI, in addition to search start-ups using similar models to develop a chat interface, such as Neeva, run by former Google executive Sridhar Ramaswamy.

Character.AI founder Noam Shazeer, who helped invent the transformer and other core machine learning architecture, said the flywheel effect of user data has been invaluable. The first time he applied user feedback to Character.AI, which allows anyone to generate chatbots based on short descriptions of real people or imaginary figures, engagement rose by more than 30 percent.

(skipping ahead)

“If Google doesn’t get their act together and start shipping[1], they will go down in history as the company who nurtured and trained an entire generation of machine learning researchers and engineers who went on to deploy the technology at other companies,” tweeted David Ha, a renowned research scientist who recently left Google Brain for the open source text-to-image start-up Stable Diffusion.

See also this similar article from the NYT a few days ago.

A narrative may be forming that Google and Meta have made a historic mistake in holding back their AI products.

An aside:

As an example of narrative formation, see this tweet with over two million views in which Meta's chief AI scientist argues that ChatGPT is not innovative.[2] The response in both the replies and on Hacker News is mixed, but the strongest throughline I see is the belief that he's merely technically correct, and that Meta has committed a fundamental business error by hesitating on/undervaluing productization.

Frustratingly, I'm not even convinced that's true! One thing I was very surprised to learn from the WaPo article was that Meta released their own free chatbot three months before OpenAI, and it's still up!

If you interact with that chatbot, you'll see exactly why it didn't take off. Its output is really quite bad compared to ChatGPT.[3]

^

In the software business, there's a famous bit of wisdom from Steve Jobs: "Real artists ship." In other words, success means delivering products to customers. If you burn time trying to perfect your product before release, you'll be e

...
Literature review of TAI timelines
by Jsevillamol — 2023-01-27T20:07:38.186Z
Link: https://epochai.org/blog/literature-review-of-transformative-artificial-intelligence-timelines

We summarize and compare several models and forecasts predicting when transformative AI will be developed.

Highlights

The review includes quantitative models, including both outside and inside view, and judgment-based forecasts by (teams of) experts.
While we do not necessarily endorse their conclusions, the inside-view model the Epoch team found most compelling is Ajeya Cotra’s “Forecasting TAI with biological anchors”, the best-rated outside-view model was Tom Davidson’s “Semi-informative priors over AI timelines”, and the best-rated judgment-based forecast was Samotsvety’s AGI Timelines Forecast.
The inside-view models we reviewed predicted shorter timelines (e.g. bioanchors has a median of 2052) while the outside-view models predicted longer timelines (e.g. semi-informative priors has a median over 2100). The judgment-based forecasts are skewed towards agreement with the inside-view models, and are often more aggressive (e.g. Samotsvety assigned a median of 2043).

Introduction

Over the last few years, we have seen many attempts to quantitatively forecast the arrival of transformative and/or general Artificial Intelligence (TAI/AGI) using very different methodologies and assumptions. Keeping track of and assessing these models’ relative strengths can be daunting for a reader unfamiliar with the field. As such, the purpose of this review is to:

Provide a relatively comprehensive source of influential timeline estimates, as well as brief overviews of the methodologies of various models, so readers can make an informed decision over which seem most compelling to them.
Provide a concise summarization of each model/forecast distribution over arrival dates.
Provide an aggregation of internal Epoch subjective weights over these models/forecasts. These weightings do not necessarily reflect team members’ “all-things-considered” timelines, rather they are aimed at providing a sense of our views on the relative trustworthiness of the models.

For aggregating internal weights, we split the timelines into “model-based” and “judgment-based” timelines. Model-based timelines are given by the output of an explicit model. In contrast, judgment-based timelines are either aggregates of group predictions on, e.g., prediction markets, or the timelines of some notable individuals. We decompose in this way as these two categories roughly correspond to “prior-forming” and “posteri...
Scaling Laws Literature Review
by Pablo Villalobos — 2023-01-27T19:57:08.341Z
Link: https://epochai.org/blog/scaling-laws-literature-review

Common shape of a scaling law, taken from

Hestness et al. (2017

)

Executive summary

Scaling laws are predictable relations between the scale of a mode and performance or other useful properties.
I have collected a

database

of scaling laws for different tasks and architectures, and reviewed dozens of papers in the scaling law literature.
My main takeaways are:

Functional forms: a basic power law can effectively model the scaling behavior in the power-law region but not the transitions to the other two regions. For this, either the M4 estimator or the BNSL estimator introduced below seem to be the best options right now.
Transfer learning: there is not a simple universal scaling law for transfer learning between arbitrary tasks. When the tasks are similar enough, upstream loss and downstream performance are closely related, but when tasks are very different, the details of the architecture and hyperparameters become very relevant.

See the full table of scaling laws

here

.

Introduction

The term “scaling laws” in deep learning refers to relations between functional properties of interest (usually the test loss or some performance metric for fine-tuning tasks) and properties of the architecture or optimization process (like model size, width, or training compute). These laws can help inform the design and training of deep learning models, as well as provide insights into their underlying principles.

In this document, I present a detailed table outlining the known scaling laws and a summary of each paper's contributions to the understanding of scaling in deep learning.

My main goal for this document is to serve as a comprehensive and up-to-date resource for information on scaling laws in deep learning. By presenting a curated list of papers that have explored various scaling laws, and providing a detailed table outlining the known scaling laws and a summary of each paper's contributions, I aim to make it easier to access and understand the current state of knowledge on scaling in deep learning.

I selected the papers using the following criteria:

Publication date: I only focus on recent papers, and exclude work prior to 2015
Contribution: I include papers which contribute in one of these ways:

Reporting empirical scaling laws
Proposing better functional forms or fitting methods
Proposing theoretical models to explain scaling behavior
Connecting scaling behavior wi

...
The role of Bayesian ML in AI safety - an overview
by Marius Hobbhahn — 2023-01-27T19:40:05.727Z

I want to thank Sebastian Farquhar, Laurence Midgley and Johan van den Heuvel, for feedback and discussion on this post.

Some time ago I asked the question

“What is the role of Bayesian ML in AI safety/alignment?”

. The response of the EA and Bayesian ML community was very helpful. Thus, I decided to collect and distill the answers and provide more context for current and future AI safety researchers.

Clarification: I don’t think many people (<1% of the alignment community) should work on Bayesian ML or that it is even the most promising path to alignment. I just want to provide a perspective and give an overview. I personally am not that bullish on Bayesian ML anymore (see shortcomings) but I’m in a relatively unique position where I have a decent overview of AI safety and the Bayesian ML literature and think an overview post like this might be helpful.

A working definition

There is no agreed-upon definition for Bayesian ML. I use the term for systems that broadly have any of the following properties

Implicitly or explicitly use Bayes theorem.
Approximate and quantify uncertainty for their estimates, e.g. return distributions instead of point estimates, and allow for the specification of prior distributions.
Systems that have a latent state that can be continuously updated with new data without being fully retrained, e.g.

conjugate inference

. This is the vaguest property since it plausibly also applies to pre-training and fine-tuning LLMs which are usually not seen as explicitly Bayesian algorithms.

Roles

High-level - Future AI systems might be Bayesian

This section is largely inspired by a

response from Emtiyaz Khan

and a different

response from Sebastian Farquhar

.

There are a lot of things that current ML systems do poorly in comparison to humans. They are often not as data-efficient as humans are, they don’t generalize well, they are often not robust to adversarial inputs, they often can’t learn during deployment and much more (none of these properties are strictly necessary for a system to be catastrophically dangerous; so their absence is not a guarantee for safety). However, many of these properties would likely exist in a system that could be called AGI.

The Bayesian framework provides some answers to these problems. Bayesian methods are often more data efficient, they can be easily upd...
Assigning Praise and Blame: Decoupling Epistemology and Decision Theory
by adamShimi — 2023-01-27T18:16:43.025Z

Your group/company/organization performs well, doing great work and dealing with new problems efficiently. As one of its leaders, you want to understand why, so that you can make it even more successful, and maybe emulate this success in other settings.

Your group/company/organization performs badly, not delivering on what was promised and missing deadline after deadline. As one of its leaders, you want to understand why, so that you can correct its course, or at least not repeat the same mistakes in other settings.

Both cases apparently involve credit assignment: positive credit (praise) for success or negative credit (blame) for failure. And you can easily think of different ways to do so:

Heuristics for Credit Assignment

Baseline

The most straightforward approach starts with your initial prediction, and then assigns credit for deviations from it. So praise people who did better than expected and blame people who did worse than expected.

Then you remember Janice. She’s your star performer, amazing in everything she does, and you knew it from the start. So she performed according to prediction, being brilliant and reliable. Which means she doesn’t deserve any praise by this criterion.

On the other hand there is Tom. He’s quite good, but you also knew from the start he was a prickly showoff with an easily scratched ego. Still, he did his job, and when he acted like an asshole, that was within the prediction. So he doesn't deserve any blame by this criterion.

Incentive wise, this sounds like a terrible idea. If you push this credit assignment strategy, not only will you neglect the value of Janice and the cost of Tom, but you will probably drive away high-performers and attract problem-makers.

Bottleneck

Instead of starting from a baseline, let’s focus on the key bottlenecks. What would have doomed the project if not done? What ended up blocking everything and dooming the project? This focuses on the real cruxes, which is good.

Yet what about Marcel, your tireless tool engineer? None of his stuff is ever absolutely necessary, but everyone in your group constantly mentions the value they get from his well-maintained and efficient tools. Should he not get any credit for it?

And Bertha, who is the only security expert of your group, and always finds excuses to make herself more and more necessary? Is this really a behavior you want to condone and praise? Shouldn’t she get b...
How could humans dominate over a super intelligent AI?
by Marco Discendenti — 2023-01-27T18:15:55.760Z

Imagine a future where:

more and more powerful AIs are created by humans;
they can mimic human intellectual abilities and make speculations like super geniuses;
they are used to work at scientific and technologic research and they produce theories and technologies that no human is intelligent enough to understand;
humans become dependent on this advanced science and technology and they need AIs to work at it.

My question is: in this scenario is there a way for humans to keep their dominion over these AIs? Could they possibly embed these AIs with a submissive "personality"? Would this be enough to avoid that they will use their superior understanding of reality to manipulate humans even while being submissive and obidient?
ChatGPT understands language
by philosophybear — 2023-01-27T07:14:42.790Z
Link: https://philosophybear.substack.com/p/chatgpt-understands-langauge

Cross-posting this from my blog, since it seems relevant.

The case for GPT understanding language, by way of understanding the world

There's a debate going on about whether or not language models similar to ChatGPT have the potential to be scaled up to something truly transformative. There's a group of mostly cognitive linguists (e.g. Gary Marcus ) that hold that ChatGPT does not understand language, it merely models what word is likely to follow the preceding- and this is importantly different from true language understanding. They see this as an "original sin" of language models which means there are limits to how good language models can get.

Freddie de Boer says much the same thing:

You could say that ChatGPT has passed Winograd’s test with flying colors. And for many practical purposes you can leave it there. But it’s really important that we all understand that ChatGPT is not basing its coindexing on a theory of the world, on a set of understandings about the understandings and the ability to reason from those principles to a given conclusion. There is no place where a theory of the world “resides” for ChatGPT, the way our brains contain theories of the world. ChatGPT’s output is fundamentally a matter of association - an impossibly complicated matrix of associations, true, but more like Google Translate than like a language-using human. If you don’t trust me on this topic (and why would you), you can hear more about this from an expert on this recent podcast with Ezra Klein.

It is true that GPT works by predicting the next token of language. It is, in some sense, as Gary Marcus put it “a glorified spreadsheet” built for this purpose. However, I do not think that this is contradictory to the notion that it understands, even if imperfectly, both language and world. My view is that in a very large language corpus, there are patterns that correspond to the way things relate to each other in the world. As models of language become more sophisticated and predictively powerful, they necessarily become models not just of language, but of the data-generating process for a corpus of that language. That data-generating process for language includes a theory of the world. Given enough data (billions or even trillions of tokens) these models can be very accurate.

We might say:

If X models Y, and Y models Z,

and if each model is sufficiently faithful,

Then X is a model...
Jar of Chocolate
by jefftk — 2023-01-27T03:40:03.163Z

Truffles are pretty great: gooey fat-softened flavored chocolate
inside, protected by a shell of hard chocolate so you don't get your
fingers messy. Sometimes I'll make them with the kids:

This is a good amount of work: you need to prepare the filling, chill,
shape little spheres, chill, and then dip them. Worth it for making
presents or sharing, but not if I'm making something for myself.
Instead I just make a jar of ganache:

I'll eat it with a spoon over a week or two, 1-3oz after lunch each
day.

At it's simplest it's chocolate and fat. I normally use Callebaut
semi-sweet which I can get in bulk for $3.42/lb ($0.21/oz):

I'll usually use butter and cream, though coconut cream works well if
you want to make it vegan. Often I'll mix in some cocoa powder to
make it a bit darker. Sometimes I'll leave it unflavored, other times
I'll add some extracts (vanilla, almond, mint) or frozen fruit
(usually raspberries). Peanut butter also works well, which gives you
the flavor and means you don't need a different source of fat. It's
reasonably shelf-stable: even with the cream it's not going to go bad
before I finish a jar of it.

I generally mix it not to be very sweet: if it's just right I enjoy a
savoring some but not so much
that I keep eating past what I'll feel good about.