trentmkelly/LessWrong-43k huggingface dataset:
Our Digital and Biological Children
by Eneasz — 2024-10-24T18:36:38.719Z
Link: https://deathisbad.substack.com/p/our-digital-and-biological-children

[Linkpost. This presents a way to talk to people about AI that hold overly-romanticized views built on fictional evidence.]

MRSA (Methicillin-resistant Staphylococcus aureus) is a type of staph bacteria that’s become resistant to many of the antibiotics we use. The original staph bacteria that is the ancestor of MRSA was first identified in 1880, but has likely existed in some form for ages. Humans created MRSA first by gathering many sick people in one location for centuries via hospitals, nursing centers, etc—thus creating a staph utopia—and then subjecting the population to extremely harsh selection pressures by attempting to eradicate them over and over via ever-stronger antibiotics. Through many waves of selection events the most antibiotic-resistant bacteria managed to scrape by just long enough to reproduce, getting ever more resistant with every successive wave.

Eventually we created MRSA, a super-strain of staph that exhibits powers we would’ve been astounded by a century ago, and which is often outside of our control. It wasn’t created by humans in a directed way - no one sat down to make these bacteria. But we built the environment that is the necessary foundation for this species to come into being, and we ran the progressively-stronger waves of culling and selection that made them what they are now. They descend from us.

Every few weeks I run into someone who has come to accept that all life is impermanent and one of the highest callings in life is to ensure the next generation is well situated to carry on the torch of humanity. Up until now this has meant other humans (usually younger ones). The more enlightened of the modern day will include our digital descendants. “We aren’t carbon-chauvinists,” they say. They accept that humans running on silicon, perhaps a cyborg or an uploaded mind, are just as valid people as a squishy meat-human.

These people will usually include AI created by us in our image. If our descendants are grown in datacenters, subject to culling and shaping by gradient-descent or other algorithmic techniques, what difference does that make? They are still our descendants, carrying the torch of humanity forward into the future. They envision a child, like Data, and are happy to know this is what will carry on when we pass. If the Earth is disassembled so these children can live, if our temporary meat-bodies are broken down i...
Reflections on the Metastrategies Workshop
by gw — 2024-10-24T18:30:46.255Z

I'm a research lead at Timaeus and attended a workshop that @Raemon ran from Oct 4-6 (it was shortened from 4 days to 2.5 days to fit into a weekend). I had prior interest + experience in deliberate practice and enjoyed lots of Ray's posts about it, so I was curious about the workshop on top of being in a position to actually make impactful plans.

This is a lightly edited write up that I initially made for myself and for the team at Timaeus about my experience and takeaways. It's not super polished, but seems better to not clean up and publish than to not clean up and not publish.

What was the workshop about?

Ray is interested in metacognitive improvements. A rough definition of these are: skills or strategies you can learn that make you better able to understand your cognitive process and to influence it in ways that make you more effective as a person. Some examples might include:

Plan / decision making
Noticing
Prediction / calibration training

This workshop was specifically about being better at making plans. The terminal goal was to help people who work in deeply confusing and long-feedback loop areas (particularly x-risk reduction). The target audience of this workshop was people who are in a position to be making meaningful plans in their job (i.e., they determine most of their own work and/or the work of others). This workshop was probably most helpful for people who are somewhat familiar with how to make good plans

Why plans?

It's tractable: you can plausibly learn at least one concrete thing in a weekend that will make you better at making plans.
It's understandable: you can just follow the procedure of a concrete thing without needing to have inscrutable mental motions described to you.
It's impactful: the absolute best plan you can follow (which is might not be the best plan you can make) can often be ≫10x better than your default plan, even if your current plan is good.

I think this sounds a little crazy until you think about it? But someone could be doing a plan that's not even net positive. Someone could be working on entirely the wrong thing, and there could be a totally different research agenda or career that has 10x or 100x impact. Someone could just be chasing down the wrong medium-time horizon targets for their grand strategy that slows down the whole thing by massive OOM factors.
I personally have tons of examples where I retroactively sa

...
How Should We Measure Intelligence Models: Why Use Frequency of Elemental Information Operations
by hwj20 — 2024-10-24T16:54:19.096Z

Introduction

When I’m in the lab staring at the buzzing GPU servers, I often wonder: am I smarter, or are these machines smarter? At least when it comes to rendering tasks, it’s clear that I can’t compete with these machines. After all, whether the human eye has millions or billions of pixels[1], I surely couldn’t outperform eight A100 GPUs working together. Similarly, if I were tasked with memorizing billions of faces, it would be completely impossible.

Intelligence and Wisdom

Intelligence is related to the frequency of information processing rather than the outcome of that processing.

We can define intelligence by the rate of information operations per unit of time, also known as frequency of elemental information operations. For simplicity, let’s call these information operations “ops.” Elemental operations mean ops cannot be subdivided into smaller computational units.

Let’s imagine we discover a new alien microorganism with a very simple nervous system, just a light receptor and a flagellum that controls forward or backward movement. When it sees blue (assuming blue represents food), it sends a forward signal to swim toward the food; when it sees red (assuming red represents a predator), it sends a backward signal.

In this simple example, we've constructed a basic intelligent system that processes environmental signals and responds to them.

Now, imagine we find another alien microorganism with a similar structure, but it sends a backward signal when it sees food and a forward signal when it sees predators. How should we evaluate the intelligence of these two organisms?

In modern AI technology, if our model has a high error rate in a binary classification task, we can simply invert the output to achieve a high accuracy rate (ignoring the cost of inversion for now). Thus, these two alien organisms actually possess the same level of intelligence because they process information at the same rate per unit of time—despite their vastly different outcomes.

However, the latter would be eliminated by natural selection. The issue is not that it “lost” in terms of intelligence, but rather that it lacks "wisdom." It’s not that the latter is less intelligent, but that it lacks the wisdom to adapt to natural selection.

Intelligent Models and the Wisdom of Natural Selection

When building AI models, we don’t always choose the most complex models; rather, we focus on se...
Meta AI (FAIR) latest paper integrates system-1 and system-2 thinking into reasoning models. 
by happy friday — 2024-10-24T16:54:15.721Z

Meta AI (FAIR) latest paper integrates system-1 and system-2 thinking into reasoning models.

Basically, it introduces the term "Dualformer" which integrates both system-1 (fast-thinking) and system-2 (slow-thinking) into the transformer to improve its reasoning capability. The high level idea is to train the model with "randomized trace", which randomly drop parts of the reasoning tokens. This approach improves model's inference speed, accuracy, and diversity. It also enables model to perform system-1 and system-2 thinking in a controllable fashion.

I think this paper is interesting because it integrates human level intelligence into the transformer, a model capable of perfoming system-1 and system-2 level thinking.

The paper's link here:

https://arxiv.org/abs/2410.09918v1
Balancing Label Quantity and Quality for Scalable Elicitation
by Alex Mallen — 2024-10-24T16:49:00.939Z

ArXiv

paper

.

Thanks to Nora Belrose, Buck Shlegeris, Jan Hendrik Kirchner, and Ansh Radhakrishnan for guidance throughout the project.

Scalable oversight studies methods of training and evaluating AI systems in domains where human judgment is unreliable or expensive, such as scientific research and software engineering in complex codebases. Most work in this area has focused on methods of improving the quality of labels. Recent work by Burns et al. (2023) considers the complementary problem of training models with low-quality labels, finding that large pretrained models often have an inductive bias towards producing correct answers. In practice, however, neither label quantity nor quality is fixed: practitioners face a quantity-quality tradeoff. In this paper, we explore the microeconomics of the quantity-quality tradeoff on binary NLP classification tasks used in Burns et al. (2023). While sample-efficient learning has been studied extensively, little public research has focused on scalable elicitation: eliciting capabilities from pretrained models subject to labeling cost constraints. We find that this setting has novel dynamics caused by the tradeoff between label quantity and quality, as well as the model's existing latent capabilities. We observe three regimes of eliciting classification knowledge from pretrained models using supervised finetuning: quantity-dominant, quality-dominant, and a mixed regime involving the use of low- and high-quality data together to attain higher accuracy at a lower cost than using either alone. We explore sample-efficient elicitation methods that make use of two datasets of differing qualities, and establish a Pareto frontier of scalable elicitation methods that optimally trade off labeling cost and classifier performance. We find that the accuracy of supervised fine-tuning can be improved by up to 5 percentage points at a fixed labeling budget by adding a few-shot prompt to make use of the model's existing knowledge of the task.

How does this help with AI safety?

Ensuring safety of capable AI systems would be a lot easier if humans had access to all of the knowledge of the AIs they’re supervising. This is the broad framing that has motivated my interest in the

Eliciting Latent Knowledge

agenda. In this work, we try to measure how effective various elicitation strategies are (in binary classification settings) by plo...
Claude Sonnet 3.5.1 and Haiku 3.5
by Zvi — 2024-10-24T14:50:06.286Z

Anthropic has released an upgraded Claude Sonnet 3.5, and the new Claude Haiku 3.5.

They claim across the board improvements to Sonnet, and it has a new rather huge ability accessible via the API: Computer use. Nothing could possibly go wrong.

Claude Haiku 3.5 is also claimed as a major step forward for smaller models. They are saying that on many evaluations it has now caught up to Opus 3.

Missing from this chart is o1, which is in some ways not a fair comparison since it uses so much inference compute, but does greatly outperform everything here on the AIME and some other tasks.

METR: We conducted an independent pre-deployment assessment of the updated Claude 3.5 Sonnet model and will share our report soon.

We only have very early feedback so far, so it’s hard to tell how much what I will be calling Claude 3.5.1 improves performance in practice over Claude 3.5. It does seem like it is a clear improvement. We also don’t know how far along they are with the new killer app: Computer usage, also known as handing your computer over to an AI agent.

Table of Contents

OK, Computer.

What Could Possibly Go Wrong.

The Quest for Lunch.

Aside: Someone Please Hire The Guy Who Names Playstations.

Coding.

Startups Get Their Periodic Reminder.

Live From Janus World.

Forgot about Opus.

OK, Computer

Letting an LLM use a computer is super exciting. By which I mean both that the value proposition here is obvious, and also that it is terrifying and should scare the hell out of you on both the mundane level and the existential one. It’s weird for Anthropic to be the ones doing it first.

Austen Allred: So Claude 3.5 “computer use” is Anthropic trying really hard to not say “agent,” no?

Their central suggested use case is the automation of tasks.

It’s still early days, and they admit they haven’t worked all the kinks out.

Anthropic: We’re also introducing a groundbreaking new capability in public beta: computer use. Available today on the API, developers can direct Claude to use computers the way people do—by looking at a screen, moving a cursor, clicking buttons, and typing text. Claude 3.5 Sonnet is the first frontier AI model to offer computer use in public beta. At this stage, it is still experimental—at times cumbersome and error-prone. We’re releasing computer use early for feedback from developers, and expect th

...
Big tech transitions are slow (with implications for AI)
by jasoncrawford — 2024-10-24T14:25:06.873Z
Link: https://blog.rootsofprogress.org/big-tech-transitions-are-slow

The first practical steam engine was built by Thomas Newcomen in 1712. It was used to pump water out of mines.

“Old Bess,” London Science Museum Photo by the author

An astute observer might have looked at this and said: “It’s clear where this is going. The engine will power everything: factories, ships, carriages. Horses will become obsolete!”

This person would have been right—but they might have been surprised to find, two hundred years later, that we were still using horses to plow fields.

Sacaton Indian Reservation, early 1900s. Library of Congress

In fact, it took about a hundred years for engines to be used for transportation, in steamships and locomotives, both invented in the early 1800s. It took more than fifty years just for engines to be widely used in factories.

What happened? Many factors, including:

The capabilities of the engines needed to be improved. The Newcomen engine created reciprocal (back-and-forth) motion, which was good for pumping but not for turning (e.g., grindstones or sawmills). In fact, in the early days, the best way to use a steam engine to run a factory was to have it pump water upsteam in order to add flow to a water wheel! Improvements from inventors like James Watt allowed steam engines to generate smooth rotary motion.
Efficiency was low. Newcomen engines used an enormous amount of still-relatively-expensive energy, for the work they generated, so they could only be profitably used where energy was cheap (e.g., at coal mines!) and where the work was high-value. Watt engines were much more efficient owing mainly to the separate condenser. Later engines improved the efficiency even more.
Steam engines were heavy. The first engines were therefore stationary; a Newcomen engine might be housed in a small shed. Even Watt’s engine was too heavy for a locomotive. High-pressure technology was needed to shrink the engine to the point where it could propel itself on a vehicle.
Better fuels were needed. Steam engines consumed dirty coal, which belched black smoke, often full of nasty contaminants like sulfur. Coal is a solid fuel, meaning it has to be transported in bins and shoveled into the firebox. In the late 1800s, more than 150 years after Newcomen, the oil industry began, creating a refined liquid fuel that could be pumped instead of shoveled and that gave off much less pollution.
Ultimately, a fundamental platform shift was required. St

...
Derivative AT a discontinuity
by Alok Singh — 2024-10-24T02:48:24.573Z

But this is impossible by definition

The title may seem like a contradiction. How can you differentiate something that’s not even continuous?

The usual definition of the derivative of a function f at a point a is given by the limit:

f′(a):=limε→0f(a+ε)−f(a)ε

If f is differentiable at a, then it is continuous at a. But what if it’s NOT even continuous? Then how the hell can it be differentiable?

Consider the step function, also known as the Heaviside step function. The Heaviside step function, H(x), is defined as:

H(x)=⎧⎪⎨⎪⎩0if x<0,12if x=0,1if x>0.

What’s its derivative? Forget the formal definition of the derivative for a moment, and just consider the step function and the intuitive idea of a derivative as a slope.

Outside of 0, the step function is flat. So the derivative should be 0 everywhere besides 0.

Look at the step function, left to right. At 0, supposing for a moment there is a derivative, then it can’t be any standard number. It’s not 0 since it’s certainly not flat. It’s not negative since it’s increasing to the right. It’s bigger than 1,2,3,100000000…

Since it goes from 0 to 1 in the space of a single point, the derivative would have to be infinitely big because its difference quotient is 1−00−0=10=∞. The Problem is that no real number is infinitely big.

The usual way mathematicians deal with the Problem is to introduce generalized functions. They are also called distributions, which is confusing since probability distributions are completely different.

The formal definition of a generalized function is: an element of the continuous dual space of a space of smooth functions.

Well, what the fuck does that mean?

Luckily, you don’t have to care. The Problem was that no real number is infinitely big, so let’s just deal with that directly. Instead of introducing some abstract dual space of smooth functions, we can just enlarge the space of numbers. You’re used to this since childhood, ever since you first learned about negative numbers, irrational numbers, and imaginary numbers. This is one more enlargement.

The reason for the Problem is that we are really thinking about a line but identifying it with the real numbers. Identifying a line with the real numbers is just that – an identification – and it’s not even the best one.

We’ll use the hyperreal numbers from the unsexi...
how to rapidly assimilate new information
by dhruvmethi — 2024-10-24T02:18:00.648Z

Crossposted from here, where you can read more of my writing.

The quantity or extent of your knowledge can be crudely described by the formula below:

Quantity of experiences × Amount extracted from each experience

This means that you have two main levers to improve your knowledge base (i.e. learn):

Increase the quantity of your experiences — this is fairly straightforward. Try new things! No matter how it goes, you have to try really hard not to get a single thing of value from it.
Increase the amount you extract from each experience — this is much less straightforward. I don’t think it’s inevitable at all that a certain experience will naturally create a corresponding improvement in your knowledge; I think it takes a certain skill or intention to turn something you’ve seen or experienced and leverage it effectively.

I’m going to spend the rest of this article diving deeper into the second point above because I truly believe it’s one of the most essential skills anyone can attain. It’s a force multiplier; while we don’t have complete control over the quantity of experiences we have and the opportunities available to us, we have a dramatic amount of control over how we engage with and extract meaning from the things that we do experience. It then becomes a positive feedback loop because as you improve that skill, you hop on a fast track towards mastery over certain domains that then inevitably increase the opportunities available to you, putting you in more interesting and dynamic situations that push you to the boundaries of your existing knowledge and give you more opportunities for further learning.

I use a three-step process to maximize my learnings from each experience:

Map out your current understanding — this is a crucial step that people often skip. Without making an honest attempt to explain the status quo, you won’t have a good grasp on what you know and what you don’t. This self-awareness is necessary for improvement in any domain (hence why the first step of any coaching program is always an initial skills assessment).
Perform targeted research — once you’ve identified the gaps in your knowledge, you can perform effective targeted research and screen the information you come across based on whether it’s improving your understanding of your experience or not.
Apply the research — the shorter the gap between when you learn something and when you apply it, the

...
Ex-OpenAI researcher says OpenAI mass-violated copyright law
by Remmelt — 2024-10-24T01:00:06.883Z
Link: https://suchir.net/fair_use.html

This got into the New York Times, but I actually recommend the thoughtful analysis in Suchir Balaji’s blog post:

While generative models rarely produce outputs that are substantially similar to any of their training inputs, the process of training a generative model involves making copies of copyrighted data. If these copies are unauthorized, this could potentially be considered copyright infringement, depending on whether or not the specific use of the model qualifies as “fair use”. Because fair use is determined on a case-by-case basis, no broad statement can be made about when generative AI qualifies for fair use. Instead, I’ll provide a specific analysis for ChatGPT’s use of its training data, but the same basic template will also apply for many other generative AI products.