trentmkelly/LessWrong-43k huggingface dataset:
A potentially high impact differential technological development area
by Noosphere89 — 2023-06-08T14:33:43.047Z

Specifically, this is boosting LLMs and Non-RL AI relative to RL AI.

I am making this post because I want to signal boost an observation by porby, which could be lossily compressed as, "Instrumental goals/Instrumental convergence is at best a debatable assumption for LLMs and Non-RL AI, and may not be there at all for LLMs/Non-RL AI."

Expressed in this post and in a comment below:

https://www.lesswrong.com/posts/EBKJq2gkhvdMg5nTQ/instrumentality-makes-agents-agenty

https://www.lesswrong.com/posts/rmfjo4Wmtgq8qa2B7/?commentId=GKhn2ktBuxjNhmaWB

I want to point this out, because on LW/EA analyses of AI risk that conclude it's a big problem (especially the more pessimistic ones) generally require instrumental convergence/instrumental goals in order for AI risks that isn't part of AI ethics, or AI misuse concerns.

So it being false, or at least having a lot higher probability of being false for LLMs/Non-RL AI is a very big deal, because the concern over AI risk that isn't a standard misuse/ethics story pretty much relies on instrumental convergence being right.

The basic reason for this, again lossily compressed, if anyone wants to know this, is basically because the steps between the input to output don't contain many intermediate steps unlike in RL, so there's far less incentive for instrumental goals/instrumental convergence since instrumental goals are much less useful when there are only a few steps between the input and output.

Given this new reality, I have identified a pretty high impact area for making AI progress safer:

Help OpenAI/Microsoft and Anthropic win the race to AGI, relative to Deepmind, or at least have Non-RL AI win the race to AGI, relative to RL AI

The reason I'm suggesting this as a way to make safer AI is because Non-RL AI, especially SOTA LLMs have far, far less incentive to have instrumental goals, relative to RL. And if you can avoid instrumental convergence, you have a much easier alignment problem, since you don't have to solve the adversarial cases of AI Alignment, you only need to solve the non-adversarial cases, and in general, solving problems in the non-adversarial case is radically easier compared to solving it in adversarial cases.

This also makes OpenAI's alignment plan described here work a lot better, particularly the part where they want to make AI that solves the AI alignment problem, and it has been criticized before, in a comme...
Question for Prediction Market people: where is the money supposed to come from?
by Robert_AIZI — 2023-06-08T13:58:11.252Z

I've only watched some prediction market news from the outside, so forgive my basic question, but are prediction markets supposed to bring in money besides having new entrants bring in cash?

I've often seen prediction markets compared to stock markets, but the stock market is generally positive-sum because you're investing money in profitable businesses that pay dividends. In contrast, if a prediction market begins with 1000 people with $1000 each (and no one else joins or brings in more money), can it ever have more than $1,000,000 in the market?

If the answer is "no, it doesn't generate money", isn't that a big problem for prediction markets as a long-term concept? It means everyone will be fighting over a limited pie, and there will be no reason for the average person to join the prediction market (they just stand to lose their money to the experts). Is this a problem holding back prediction markets now, and are there ideas to fix it?
AI #15: The Principle of Charity
by Zvi — 2023-06-08T12:10:06.650Z

The sky is not blue. Not today, not in New York City. At least it’s now mostly white, yesterday it was orange. Even indoors, everyone is coughing and our heads don’t feel right. I can’t think fully straight. Life comes at you fast.

Thus, I’m going with what I have, and then mostly taking time off until this clears up. Hopefully that won’t be more than a few more days.

The Principle of Charity comes into play this week because of two posts, by people I greatly respect as thinkers and trust to want good things for the world, making arguments that are remarkably terrible. I wrote detailed responses to the arguments within, then realized that was completely missing the point, and deleted them. Instead, next week I plan to explain my model of what is going on there – I wish they’d stop doing what they are doing, and think they would be wise to stop doing it, but to the extent I am right about what is causing these outputs, I truly sympathize.

For a day we were all talking about a Vice story that sounded too good (or rather, too perfect) to be true, and then it turned out that it was indeed totally made up. Time to take stock of our epistemic procedures and do better next time.

Table of Contents

Introduction

Table of Contents

Language Models Offer Mundane Utility. Quite a lot, actually.

Language Models Don’t Offer Mundane Utility. Not with that attitude.

Deepfaketown and Botpocalypse Soon. Talk to your… parents?

Fun with Image Generation. Investors, man.

Vigilance. It must be eternal, until it won’t be enough.

Introducing. Falcon the open source model, Lightspeed Grants and more.

In Other AI News. Senator asks Meta the question we all ask: Llama?

They Took Our Jobs. First they came for the copywriters.

Out of the Box Thinking. A hard to detect attack on audio activation devices.

I Was Promised Driverless Cars. You’ll get them. Eventually.

If It Sounds Too Good to be True. Guess what?

Quiet Speculations. Algorithmic improvements take center stage for a bit.

A Very Good Sentence. Including a claim it was a very bad sentence.

The Quest for Sane Regulation. Devil is in the details. UK PM steps up.

The Week in Podcasts and Other Audio and Video. Don’t look.

What Exactly is Alignment? It’s a tricky word.

Rhetorical Innovation. Several good ideas.

Have You Tried Not Doing Things That Might Kill Everyone? It

...
if you're reading this it's too late
(a new theory on what is causing the Great Stagnation)
by rogersbacon — 2023-06-08T11:49:29.136Z
Link: https://www.secretorum.life/p/if-youre-reading-this-its-too-late

A sword is against the internet, against those who live online, and against its officials and wise men. A sword is against its false prophets, and they will become fools. A drought is upon its waters, and they will be dried up. For it is a place of graven images, and the people go mad over idols. So the desert creatures and hyenas will live there and ostriches will dwell there. The bots will chatter at its threshold, and dead links will litter the river bed.

One consistent theme of this blog has been the role of the internet and digital technology in what I see as a subtle but pervasive loss of creativity across the arts, science, and technology (manifesting as the so-called Great Stagnation in the latter domains). Here, I want to pull together some of my writing on this subject from various essays and share some recent research/sources that have led me to a new hypothesis for what’s going on here.

~~~~~

From “

On Stories and Histories: Q&A

”:

The crushing excess of new content is especially problematic in this regard. We have less familiarity with the deep history of art and culture than ever before, and this has caused us to lose a kind of instinct for Greatness, an intuition for what makes things stand the test of time. At the same time, most of us also have too much knowledge of our chosen art or discipline and this makes it difficult for us to do the unprecedented, path-breaking work that revolutionizes a field. In the arts, the danger is that everything you make will feel derivative, a pale imitation of yesteryear’s avant garde. In the sciences, an extensive knowledge of the literature will bias you towards conducting incremental gap-filling research instead of the truly innovative research which creates new gaps in the literature. Richard Hamming has a little riff on this in his classic speech, “

You and Your Research

”:

“There was a fellow at Bell Labs, a very, very, smart guy. He was always in the library; he read everything. If you wanted references, you went to him and he gave you all kinds of references. But in the middle of forming these theories, I formed a proposition: there would be no effect named after him in the long run. He is now retired from Bell Labs and is an Adjunct Professor. He was very valuable; I'm not questioning that. He wrote some very good Physical Review articles; but there’s no effect named after him because he read too mu

...
[Linkpost] Scaling laws for language encoding models in fMRI
by Bogdan Ionut Cirstea — 2023-06-08T10:52:16.400Z

This is a linkpost for https://arxiv.org/abs/2305.11863

Representations from transformer-based unidirectional language models are known to be effective at predicting brain responses to natural language. However, most studies comparing language models to brains have used GPT-2 or similarly sized language models. Here we tested whether larger open-source models such as those from the OPT and LLaMA families are better at predicting brain responses recorded using fMRI. Mirroring scaling results from other contexts, we found that brain prediction performance scales log-linearly with model size from 125M to 30B parameter models, with ~15% increased encoding performance as measured by correlation with a held-out test set across 3 subjects. Similar log-linear behavior was observed when scaling the size of the fMRI training set. We also characterized scaling for acoustic encoding models that use HuBERT, WavLM, and Whisper, and we found comparable improvements with model size. A noise ceiling analysis of these large, high-performance encoding models showed that performance is nearing the theoretical maximum for brain areas such as the precuneus and higher auditory cortex. These results suggest that increasing scale in both models and data will yield incredibly effective models of language processing in the brain, enabling better scientific understanding as well as applications such as decoding.

(bolding mine)

One particular quote from the twitter summary thread from the senior author:

And these models aren't just good at predicting a few parts of temporal lobe — they're good at predicting ALL of the language-responsive areas in cortex. Individual voxels in precuneus, PFC, and ITL are all predicted with correlation > 0.75. (this is insane to me)
Transformative AI is a process 
by meijer1973 — 2023-06-08T08:57:36.421Z

In discussions, transformative AI is often seen as a certain point in time when an AI system achieves a certain level of performance or capabilities. For example, some people believe(d) that transformative AI will occur when an AI system can pass the Turing test, or when an AI system can create art or music that is indistinguishable from human-created art or music (yep, a few years ago people said that). More modern definitions have a higher bar (e.g. do all human jobs at low cost.)

However, it is also possible to see transformative AI as a process rather than a specific point in time. In this view, transformative AI is not a single event, but rather a gradual evolution of AI systems that become increasingly capable and sophisticated over time. This process of evolution is driven by advances in AI research, as well as by the increasing availability of data and computing power.

The view of transformative AI as a process has several advantages over the view of transformative AI as a specific point in time. First, it allows us to see that transformative AI is already happening, and that it is not something that will happen in the distant future. Second, it allows us to better understand the factors that are driving transformative AI, and to develop strategies for accelerating this process.

Finally, it allows us to better prepare for the challenges and opportunities that transformative AI will bring. As transformative AI systems become increasingly capable, they will have a profound impact on our society, economy, and environment. It is important that we start thinking about these impacts now, so that we can shape the future of transformative AI in a way that benefits all of humanity.

The long term impact of relatively small changes

The exact start and finish of a transformation can only be determined in hindsight. However, we are currently in a period of accelerated change, driven by a number of factors, including the rise of artificial intelligence, the increasing availability of data, and the development of new technologies.

If this rate of change were to persist for 5 to 10 years, we would see our economic, societal, and environmental situation changed dramatically. A transformation might not need as much growth as you might think. A short-term productivity growth change of 2% per year to 4% per year might not seem like much in the short run, but over a peri...
Crisis of Faith case study: beyond reductionism?
by MalcolmOcean — 2023-06-08T06:11:24.246Z

What follows are some notes to self that I took while reading the first 7 pages of Stuart A. Kauffman's book Reinventing the Sacred. This was over 6 years ago, in January 2017.

I read those pages multiple times, pausing whenever I had some objections arise, and journalling about the process. I ended up having a bit of a Crisis of Faith with it, and something loosened, but I didn't actually return to the book until years later. The objections were largely coming up based on what I'd read in the Sequences and learned elsewhere from rationalists. I remember thinking at the time it would be a good case study for LessWrong, but I didn't get around to sharing it until now.

It's kinda funny/awkward pasting this in—I've changed a lot in the past years, both in terms of how I write and how I related to all of this stuff. But since the whole thing is about my perspective at the time, there's no point in trying to change any of that, so I've basically only edited it for clarity, grammar, etc. I did take out a couple tangents. The section titles I wrote now with the benefit of hindsight.

If you get bored towards the start, you might skim til it picks up; I think it gets more interesting as it goes on, but it seemed important to include my whole process of course.

📖 Reading notes

↳12017-01-17 evening

First pass: skepticism, committing to the virtues of rationality

I got a few pages into this book and it was already quite challenging, in the sense of "challenging to take the author seriously as having something worth saying". He struck me as being the kind of person that caused Eliezer to rant about "emergence".

Main issues so far are:

his lack of distinguishing between "current physics" and "coherent extrapolated physics"
relatedly, saying ~"half of people believe in god. that isn't just going to get explained away by science"

I was like "well a century or two ago, that fraction was substantially higher, so it seems to be going down"
but then I actually looked into it, and that does not appear to be the case!

eg

https://www.theatlantic.com/international/archive/2015/04/islam-could-become-the-worlds-largest-religion-after-2070/389210/

so hey, look at the data!

I'm still not particularly into his case though; I think that peoples' reasons for converting (or not de-converting) may be about other things

also part of why those numbers are increasing is that religious people are having more

...
I wrote this because of watermelon
by Arti — 2023-06-08T03:55:26.582Z

We say to the watermelon

"We accept your offering of sugar. We will share our riches with you in return, that you may have the safety and plenty to grow even sweeter."

And we say to the watermelon

"You make these seeds to carry on your line, in the only way you know how. But we are clever, and if you leave your seeds behind we will spread your children farther than your seeds could ever have carried them."

And we say to the watermelon

"Another great summer. Let's do this again every year."

And we say to the watermelon

"Come travel the world with me! My friends want to meet you.

And we say to the watermelon

"I love you."
Learning Transformer Programs [Linkpost]
by aog — 2023-06-08T00:16:54.937Z
Link: https://arxiv.org/abs/2306.01128

Recent research in mechanistic interpretability has attempted to reverse-engineer Transformer models by carefully inspecting network weights and activations. However, these approaches require considerable manual effort and still fall short of providing complete, faithful descriptions of the underlying algorithms. In this work, we introduce a procedure for training Transformers that are mechanistically interpretable by design. We build on RASP [Weiss et al., 2021], a programming language that can be compiled into Transformer weights. Instead of compiling human-written programs into Transformers, we design a modified Transformer that can be trained using gradient-based optimization and then be automatically converted into a discrete, human-readable program. We refer to these models as Transformer Programs. To validate our approach, we learn Transformer Programs for a variety of problems, including an in-context learning task, a suite of algorithmic problems (e.g. sorting, recognizing Dyck-languages), and NLP tasks including named entity recognition and text classification. The Transformer Programs can automatically find reasonable solutions, performing on par with standard Transformers of comparable size; and, more importantly, they are easy to interpret. To demonstrate these advantages, we convert Transformers into Python programs and use off-the-shelf code analysis tools to debug model errors and identify the ``circuits'' used to solve different sub-problems. We hope that Transformer Programs open a new path toward the goal of intrinsically interpretable machine learning.
What will GPT-2030 look like?
by jsteinhardt — 2023-06-07T23:40:02.925Z

GPT-4 surprised many people with its abilities at coding, creative brainstorming, letter-writing, and other skills. Surprises in machine learning are not restricted to GPT-4: I was previously surprised by Minerva’s mathematical abilities, as were many competitive forecasters.

How can we be less surprised by developments in machine learning? Our brains often implicitly make a zeroth-order forecast: looking at the current state of the art, and adding on improvements that “feel reasonable”. But what “seems reasonable” is prone to cognitive bias, and will underestimate progress in a fast-moving field like ML. A more effective approach is first-order forecasting: quantifying the historical rate of progress and extrapolating it forward, while also considering reasons for possible slowdowns or speedups.[1]

In this post, I’ll use this approach to forecast the properties of large pretrained ML systems in 2030. I’ll refer throughout to “GPT2030”, a hypothetical system that has the capabilities, computational resources, and inference speed that we’d project for large language models in 2030 (but which was likely trained on other modalities as well, such as images). To forecast GPT2030’s properties, I consulted a variety of sources, including empirical scaling laws, projections of future compute and data availability, velocity of improvement on specific benchmarks, empirical inference speed of current systems, and possible future improvements in parallelism.

GPT2030’s capabilities turn out to be surprising (to me at least). In particular, GPT2030 will enjoy a number of significant advantages over current systems[2], as well as (in at least some important respects) current human workers:

GPT2030 will likely be superhuman at various specific tasks, including coding, hacking, and math, and potentially protein engineering (Section 1).

GPT2030 can “work” and “think” quickly: I estimate it will be 5x as fast as humans as measured by words processed per minute [range: 0.5x-20x][3], and that this could be increased to 125x by paying 5x more per FLOP (Section 2).

GPT2030 can be copied arbitrarily and run in parallel. The organization that trains GPT2030 would have enough compute to run many parallel copies: I estimate enough to perform 1.8 million years of work when adjusted to human working speeds [range: 0.4M-10M years] (Section 3). Given the 5x speed-up in the previous point, t

...
Progress links and tweets, 2023-06-07
by jasoncrawford — 2023-06-07T23:26:55.386Z
Link: https://rootsofprogress.org/links-and-tweets-2023-06-07

Announcements & opportunities

UK Great Stagnation summit in July, apply to attend (via @s8mb)
Help Bryan Bishop create an enzyme that makes DNA based on digital instructions
The Cybersecurity Grant Program, $1M for AI-based cybersecurity (from @OpenAI)
Foundations & Frontiers, a magazine about future technologies (via @annasofialesiv)

Links

An dig through the Extropian archives (via @ATabarrok)
Orwell against progress
“Coasean democracy” is the solution to Britain’s sclerosis (by @s8mb)

Queries

Is there a good summary of all the different proposals for how to regulate AI?
Have any of you read Samuel Butler’s Erewhon (1872)? I could use some exegesis
A book about economic history that focuses on how technology was scaled?
Is there a counterfactual world where we’re still pre-industrial?
Are self-driving cars still using control theory, with AI used only for sensors?
Has anybody made an actually good publicly editable database?

Tweets

Most people have no idea how much retail has gotten better over the last ~150 years
An urban planner responds to my post on regulation
Many people think self-driving failed, but there are customers in driverless taxis now
The meaning of mortality declines
The oldest known depiction of a steam-using atmospheric engine? From 1654!
Desalination was a big technological success story of the 2010s
Rewarding AI for a correct thought process, not just the right answer
Humans correctly guessed an LLM chat agent was an AI only 60% of the time
LLMs will be units within larger orchestrated systems
Environmental regulations are an all-purpose tool to indefinitely delay anything
Should we build underground cities? (Maybe not, say @ConnorTabarrok and Casey Handmer). Related thread on underground apartments.)
Is it virtuous to assign probabilities to all uncertainty?

Charts

The UK continued to get rich after the end of empire (GDP growth actually increased)

Iterative change is slow but over time it moves the world
LEAst-squares Concept Erasure (LEACE)
by tricky_labyrinth — 2023-06-07T21:51:04.494Z
Link: https://twitter.com/norabelrose/status/1666469917636571137

"Ever wanted to mindwipe an LLM?

Our method, LEAst-squares Concept Erasure (LEACE), provably erases all linearly-encoded information about a concept from neural net activations. It does so surgically, inflicting minimal damage to other concepts.

...

LEACE has a closed-form solution that fits on a T-shirt. This makes it orders of magnitude faster than popular concept erasure methods like INLP and R-LACE, which require gradient-based optimization. And the solution can be efficiently updated to accommodate new data."