trentmkelly/LessWrong-43k huggingface dataset:
What is the next level of rationality?
by lsusr — 2023-12-12T08:14:14.846Z

This is part 1 of our dialogue series on the question "What is the next level of rationality?".

lsusr

Yudkowsky published Go Forth and Create the Art! in 2009. It is 2023. You and I agree that, in the last few years, there haven't been many rationality posts on the level of Eliezer Yudkowsky (and Scott Alexander). In other words, nobody has gone forth and created the art. Isn't that funny?

What Came Before Eliezer?

Yoav Ravid

Yes, we agreed on that. I remarked that there were a few levels of rationality before Eliezer. The one directly before him was something like the Sagan-Feynman style rationality (who's fans often wore the label "Skeptics"). But that's mostly tangential to the point.

lsusr

Or perhaps it's not tangential to the point at all. Feynman was referenced by name in Harry Potter and the Methods of Rationality. I have a friend in his 20s who is reading Feynman for the first time. He's discovering things like "you don't need a labcoat and a PhD to test hypotheses" and "it's okay to think for yourself".

Yoav Ravid

How do you see it connecting to the question "What's the next level of rationality?"

lsusr

Yudkowsky is a single datapoint. The more quality perspectives we have about what "rationality" is, the better we can extrapolate the fit line.

Yoav Ravid

I see, so perhaps a preliminary to this discussion is the question "which level of rationality is Eliezer's?"?

lsusr

Yeah. Eliezer gets extra attention on LessWrong, but he's not the only writer on the subject of rationality. I think we should start by asking who's in this cluster we're pointing at.

Yoav Ravid

Alright, so in the Feynman-Sagen cluster, I'd also point to Dawkins, Michael Shermer, Sam Harris, Hitchens, and James Randi, for example. Not necessarily because I'm very familiar with their works or find them particularly valuable, but because they seem like central figures in that cluster.

lsusr

Those are all reasonable names, but I've never actually read any of their work. My personal list include Penn Jillette. Paul Graham and Bryan Caplan feel important too, even though they're not branded "skeptic" or "rationality".

Yoav Ravid

I've read a bit, but mostly I just came late enough to the scene and found Eliezer and Scott quickly enough that I didn't get the chance to read them deeply before then, and after I did I didn't feel the need.

Yoav Ravid

Yep, and Paul Graham is also someone Eliezer respects a lot, and I

...
Embedded Agents are Quines
by lsusr — 2023-12-12T04:57:31.588Z

lsusr

Is entropy objective or subjective?

DaemonicSigil

In one sense subjective, if you believe in subjective probabilities. i.e. if we use Shannon's entropy formula, −∑ipilogpi then the formula itself is objective, but Bayesians would have subjective probabilities pi, which might vary from person to person.

On the other hand, it does seem to be the case that there are ways in which entropy is very clearly objective. Like the entropy of a box of gas isn't going to vary from person to person, and it's not possible to avert the heat death of the universe by simply changing your subjective beliefs in such a way that entropy goes down.

It's a strange tension to have in your worldview, that entropy should be both subjective and somehow also not. I have this meme lying around somewhere, let me see if I can pull it up...

"The second law of thermodynamics from various perspectives":

Anyway, the joke is to ask what the second law of thermodynamics looks like for people in various states of knowledge. For us humans, there is much we don't know, we're surrounded gas (and other things) made from atoms bouncing around in ways we haven't observed. And entropy goes up all the time, whenever there is friction, or any other dissipative process. Then, Laplace's demon is a stand-in for what we'd call a "logically omniscient being". The demon isn't all-knowing, but it has infinite memory and computational power. Since the laws of physics are reversible, entropy never increases for such a being. Apparent increases in entropy for us mortals are just the result of probability distributions become too complicated and tangled up for us to track. And then "God" would be a stand in for true omniscience, that actually does know the position of every atom. For God, for whom everything is certain, entropy is 0. (The rest of the meme is just a comment on the fact that Conway's game of life is not reversible, left as exercise for the reader.)

lsusr

You seem not to have fallen for my trick question. Often, the people I talk to assume that entropy is necessarily an objective quantity. After all, S=entropy appears in thermodynamic equations like dS=∂QT. But what I find most interesting about your comment is the phrase "if you believe in subjective probabilities".

Are probabilities subjective? Litany of Tarski.

lsusr

If probabilities are subjective,
I desire to believe that pr

...
The likely first longevity drug is based on sketchy science. This is bad for science and bad for longevity.
by BobBurgers — 2023-12-12T02:42:18.559Z

If you are interested in the longevity scene, like I am, you probably have seen press releases about the dog longevity company, Loyal for Dogs, getting a nod for efficacy from the FDA. These have come in the form of the New York Post calling the drug "

groundbreaking

", Science Alert calling the drug "

radical

", and the more sedate New York Times just asking, "Could Longevity Drugs for Dogs Extend Your Pet's Life?", presumably unaware of Betteridge's Law of Headlines. You may have also seen the coordinated Twitter offensive of people

losing their shit about this

, including their lead investor, Laura Deming, saying that she "

broke down crying when she got the call

".

And if you have been following Loyal for Dogs for a while, like I have, you are probably puzzled by this news. Loyal for Dogs has been around since 2021. Unlike any other drug company or longevity company, they have released almost zero information (including zero publications) about their strategy for longevity. It's surprising, to say the least, to see a company go from zero information to efficacy nod, because, well, what are you basing your efficacy on? How did you recruit your patients and veterinary partners to help you with efficacy? Did you make them all sign some incredibly airtight NDAs? If so, why?

These thoughts swirling around my head, I waded through the press releases trumpeting the end of dog death as we know it in order to figure out what exactly Loyal is doing for dog longevity. And, what I found first surprised me, then saddened me. Loyal did not prove efficacy in dog longevity. They found a path around the FDA instead. That's the surprising part. The sad part is that, in doing so, they relied on some really sketchy science. And I think that, based on their trajectory, they won't just be the first company to get a drug approved for longevity. They will be the first one to get a longevity drug pulled for non-efficacy as well, and put the field back years.

So let's start with how they got their drug approved in the first place. Well, they didn't. To get drugs approved in animals, you need to prove three things: efficacy, safety, and manufacturing consistency. Normally, efficacy is the hardest part of this, because you have to prove to the FDA that your drug cures the disease that it's supposed to. This is especially hard in aging, because any aging trial would take a long time. Loy...
When will GPT-5 come out? Prediction markets vs. Extrapolation
by Malte — 2023-12-12T02:41:59.711Z

So far, each generation of GPT has brought significant improvements. Thinking about the timeline for the next iteration, I noticed that there is a striking difference between extrapolating the past trend and what prediction markets seem to believe.

Forecasting short term AI Capabilities is important as current trends might continue to lead us to bigger changes. Also, it is nice to have testable predictions that we can use to calibrate our predictions and figure out who is worth listening to.

Extrapolating from old GPTs

Rule 1 of forecasting: stop thinking too much and look at some historical data.

We don't know how long it took to develop each GPT, but we can look at how much time passed between each iteration.

So far, more and more time has passed between GPTs. It took 8 months from GPT-1 to GPT-2 and roughly twice as long to GPT-3. And then it took twice as long again to get to GPT-4! This means it took almost 3 years to get from GPT-3 to GPT-4.

GPT Model	Release Date [1]	Months Passed Between Former Model
GPT-1	11.06.2018
GPT-2	14.02.2019	8.16
GPT-3	28.05.2020	15.43
GPT-4	14.03.2023	33.55

Naively Extrapolating from this, it should take till the beginning of 2029 to develop GPT-5.

shoutout to datapoint 4 for making this graph

Now this does seem rather far in the future. So far the trends seems surprisingly consistent, but there aren’t that many datapoints here and it seems like things can change somewhat fast in AI.

Prediction markets

Let’s look at what the prediction markets think:

Manifold Market seems to think, that there is an 62% chance that GPT-5 will come out before 2025.

Similarly, Metaculus puts the announcement date of GPT-5 to Sept 2024: [2]

So it seems like the forecasting sites expect a strong deviation from the historical trend.

Note, that if GPT-5 takes as long as GPT-4 did, we would expect it by the end of 2025. Not only do they expect that GPT-5 will not take longer than GPT-4, they seem to expect GPT-5 to be developed a whole year faster than GPT-4.

So who is right?

Now that we have observed the seeming discrepancy, we can speculate on how to resolve it.

One approach is to say that the markets are right. Ideally they would incorporate additional information, but what could that be?

A lot of attention and resources are being shifted to AI right now

in some ways chatgpt does seem like a disco

...
On plans for a functional society
by kave — 2023-12-12T00:07:46.629Z

Vaniver

I'm going to expand on something brought up in this comment. I wrote:

A lot of my thinking over the last few months has shifted from "how do we get some sort of AI pause in place?" to "how do we win the peace?". That is, you could have a picture of AGI as the most important problem that precedes all other problems; anti-aging research is important, but it might actually be faster to build an aligned artificial scientist who solves it for you than to solve it yourself (on this general argument, see Artificial Intelligence as a Positive and Negative Factor in Global Risk). But if alignment requires a thirty-year pause on the creation of artificial scientists to work, that belief flips--now actually it makes sense to go ahead with humans researching the biology of aging, and to do projects like Loyal.

This isn't true of just aging; there are probably something more like twelve major areas of concern. Some of them are simply predictable catastrophes we would like to avert; others are possibly necessary to be able to safely exit the pause at all (or to keep the pause going when it would be unsafe to exit).

I think 'solutionism' is basically the right path, here. What I'm interested in: what's the foundation for solutionism, or what support does it need? Why is solutionism not already the dominant view? I think one of the things I found most exciting about SENS was the sense that "someone had done the work", had actually identified the list of seven problems, and had a plan of how to address all of the problems. Even if those specific plans didn't pan out, the superstructure was there and the ability to pivot was there. It looked like a serious approach by serious people. What is the superstructure for solutionism such that one can be reasonably confident that marginal efforts are actually contributing to success, instead of bailing water on the Titanic?

Restating this, I think one of the marketing problems with anti-aging is that it's an ancient wish and it's not obvious that, even with the level of scientific mastery that we have today, it's at all a reasonable target to attack. (The war on cancer looks like it's still being won by cancer, for example.) The thing about SENS that I found most compelling is that they had a frame on aging where success was a reasonable thing to expect. Metabolic damage accumulates; you can possibly remove the damage; if so

...
Secondary Risk Markets
by Vaniver — 2023-12-11T21:52:46.836Z

This idea is half-baked; it has some nice properties but doesn't seem to me like a solution to the problem I most care about. I'm publishing it because maybe it points someone else towards a full solution, or solves a problem they care about, and out of a general sense that people should publish negative results.

Many risky activities impact not just the person doing the activity, but also bystanders or the public at large. Governments often require ability to compensate others as a precondition for engaging in the risky activity, with requirements to have car insurance to drive as a common example.

In most situations, this works out fine:

A competitive insurance market means that customers aren't overcharged too much (since they'll switch insurance providers to whoever estimates their risk as being the lowest).
Accidents are common enough that insurance companies that are bad at pricing quickly lose too much money and adjust their prices upwards (so customers aren't undercharged either).
Accidents are small enough that insurance companies can easily absorb the losses from mispriced insurance.
Accidents are predictable enough that insurers can price premiums by driver moderately well.
Drivers are common enough that simple prediction rules make more sense than putting dedicated thought into how much to charge each driver.

Suppose we adjust the parameters of the situation, and now instead of insuring drivers doing everyday trips, we're trying to insure rare, potentially catastrophic events, like launching nuclear material into orbit to power deep space probes. Now a launch failure potentially affects millions of people, and estimating the chance of failure is well worth more than a single formula's attention.

As a brief aside, why try to solve this with insurance? Why not just have regulators decide whether you can or can't do something? Basically, I believe that prices transmit information, and allow you to make globally correct decisions by only attending to local considerations. If the potential downside of something is a billion dollars, and you have a way to estimate micro-failures, you can price each micro-failure at a thousand dollars and answer whether or not mitigations are worth it (if it reduces the microfailures by 4 and costs $5,000, it's not worth it, but if it's 6 microfailures instead then it is worth it) and whether or not it's worth doing the whole pr...
Has anyone experimented with Dodrio, a tool for exploring transformer models through interactive visualization?
by Bill Benzon — 2023-12-11T20:34:19.015Z

Zijie J. Wang, Robert Turko, Duen Horng Chau, Dodrio: Exploring Transformer Models with Interactive Visualization, arXiv:2103.14625 [cs.CL].

Abstract: Why do large pre-trained transformer-based models perform so well across a wide variety of NLP tasks? Recent research suggests the key may lie in multi-headed attention mechanism's ability to learn and represent linguistic information. Understanding how these models represent both syntactic and semantic knowledge is vital to investigate why they succeed and fail, what they have learned, and how they can improve. We present Dodrio, an open-source interactive visualization tool to help NLP researchers and practitioners analyze attention mechanisms in transformer-based models with linguistic knowledge. Dodrio tightly integrates an overview that summarizes the roles of different attention heads, and detailed views that help users compare attention weights with the syntactic structure and semantic information in the input text. To facilitate the visual comparison of attention weights and linguistic knowledge, Dodrio applies different graph visualization techniques to represent attention weights scalable to longer input text. Case studies highlight how Dodrio provides insights into understanding the attention mechanism in transformer-based models. Dodrio is available at this https URL.

From the documentation at the interactive website:

Dodrio addresses the challenges of interpreting attention weights through an interactive visualization system that provides attention head summarization and semantic and syntractic knowledge contexts. By identifying the linguistic properties that an attention head attends to in the Attention Head Overview (bottom right), you can click the attention head to explore the semantic and syntactic significance of the sentence at the selected attention head. If you are interested in the lexical dependencies in a sentence, you can explore a syntactically important head in the Dependency View and accompanying Comparison View (top), while semantically important heads can be investigated in the Semantic Attention Graph view (bottom left). We encourage you to further investigate the multi-headed attention mechanism across various text instances with interesting linguistic features (eg. coreferences, word sense, etc.) in the Instance Selection View by clicking the appropriate icon in the toolba

...
[Valence series] 3. Valence & Beliefs
by Steven Byrnes — 2023-12-11T20:21:30.570Z

3.1 Post summary / Table of contents

Part of the

Valence series

.

So far in the series, we defined valence (

Post 1

) and talked about how it relates to the “normative” world of desires, values, preferences, and so on (

Post 2

). Now we move on to the “positive” world of beliefs, expectations, concepts, etc. Here, valence is no longer the sine qua non at the center of everything, as it is in the “normative” magisterium. But it still plays a leading role. Actually, two leading roles!

Section 3.2 distinguishes two paths by which valence affects beliefs: first, in its role as a control signal, and second, in its role as “interoceptive” sense data, which I discuss in turn:
Section 3.3 discusses how valence-as-a-control-signal affects beliefs. This is the domain of motivated reasoning, confirmation bias, and related phenomena. I explain how it works both in general and through a nuts-and-bolts toy model. I also elaborate on “voluntary attention” versus “involuntary attention”, in order to explain anxious rumination, which goes against the normal pattern (it involves thinking about something despite a strong motivation not to think about it).
Section 3.4 discusses how valence-as-interoceptive-sense-data affects beliefs. I argue that, if concepts are

“clusters in thingspace”

, then valence is one of the axes used by this clustering algorithm. I discuss how this relates to various difficulties in modeling and discussing the world separately from how we feel about it, along with the related “affect heuristic” and “halo effect”.
Section 3.5 briefly muses on whether future AI will have motivated reasoning, halo effect, etc., as we humans do. (My answer is “yes, but maybe it doesn’t matter too much”.)
Section 3.6 is a brief conclusion.

3.2 Two paths for normative to bleed into positive

Here’s a diagram from the

previous post

:

We have two paths by which valence can impact the world-model (a.k.a. “Thought Generator”): the normative path (upward black arrow) that helps control which thoughts get strengthened versus thrown out, and the positive path (curvy green arrow) that treats valence as one of the input signals to be incorporated into the world model. Corresponding to these two paths, we get two ways for valence to impact factual beliefs:

Motivated reasoning / thinking / observing and confirmation bias—related to the upward black arrow, and discussed

...
Am I ethically obligated to extend the life of my dog with life-extension treatments about to hit the market? 
by TrudosKudos — 2023-12-11T19:41:38.400Z

Recently, a company called "Loyal" received pre-approval status from the FDA regarding their life extension hormone treatment that has show early promise in extending the lifespan of dogs that weigh > 40lbs. Big dogs have extremely high levels of a particular hormone associated with increased cellular aging and the treatment is designed to combat that and apparently shows promise in doing so. From what I understand about the treatment is that it would be chronically administered and thus would require the owner to consistently pay for the medication to continue the life extension benefits.
Considering the typical costs of new treatments that come to market as well as the general expensive nature of the pet health market in general, I'm expecting this treatment to be relatively expensive for the average pet-owner. I'm wondering, from an ethics perspective, if extending the life of your pet is considered a required part of being a responsible pet owner and caring for the health and well-being of our furry friends.
Any thoughts on this?

Link to news from Loyal's website about their recent pre-approval:

https://loyalfordogs.com/posts/loyal-announces-historic-fda-milestone-for-large-dog-lifespan-extension-drug
Adversarial Robustness Could Help Prevent Catastrophic Misuse
by aog — 2023-12-11T19:12:26.956Z

There have been

several

discussions

about the importance of adversarial robustness for scalable oversight. I’d like to point out that adversarial robustness is also important under a different threat model: catastrophic misuse.

For a brief summary of the argument:

Misuse could lead to catastrophe. AI-assisted cyberattacks, political persuasion, and biological weapons acquisition are plausible paths to catastrophe.
Today's models do not robustly refuse to cause harm. If a model has the ability to cause harm, we should train it to refuse to do so. Unfortunately, GPT-4, Claude, Bard, and Llama all have received this training, but they still behave harmfully when facing prompts generated by adversarial attacks, such as this one and this one.
Adversarial robustness will likely not be easily solved. Over the last decade, thousands of papers have been published on adversarial robustness. Most defenses are near useless, and the best defenses against a constrained attack in a CIFAR-10 setting still fail on 30% of inputs. Redwood Research's work on training a reliable text classifier found the task quite difficult. We should not expect an easy solution.
Progress on adversarial robustness is possible. Some methods have improved robustness, such as adversarial training and data augmentation. But existing research often assumes overly narrow threat models, ignoring both creative attacks and creative defenses. Refocusing research with good evaluations focusing on LLMs and other frontier models could lead to valuable progress.

This argument requires a few caveats. First, it assumes a particular threat model: that closed source models will have more dangerous capabilities than open source models, and that malicious actors will be able to query closed source models. This seems like a reasonable assumption over the next few years. Second, there are many other ways to reduce risks from catastrophic misuse, such as removing hazardous knowledge from model weights, strengthening societal defenses against catastrophe, and holding companies legally liable for sub-extinction level harms. I think we should work on these in addition to adversarial robustness, as part of a defense-in-depth approach to misuse risk.

Overall, I think adversarial robustness should receive more effort from researchers and labs, more funding from donors, and shoul...