trentmkelly/LessWrong-43k huggingface dataset:
Food, Prison & Exotic Animals: Sparse Autoencoders Detect 6.5x Performing Youtube Thumbnails
by Louka Ewington-Pitsos — 2024-09-17T03:52:43.269Z

TL;DR

A Sparse Autoencoder trained on CLIP activations detects features in youtube thumbnails associated with high view counts across a dataset of 524,288 thumbnails from youtube-8m. Most of these features are interpretable and monosemantic. The most strongly associated feature has a median view count > 6.5x the dataset median. After simulating random features and comparing their median views we conclude that the probability that these observations could have occurred at random is zero or near zero.

These features may correspond to visual elements which attract the attention of youtube users, but they may instead correlate to some other, less exciting structure related with higher views such as date of upload.

Similar analysis is conducted on MrBeast videos.

MrBeast Thumbnails with high activations for 35679 - the "prison" feature (top). High activating images for the same feature from imagenet (bottom). These videos have on average 247,572,192 views, compared to the 114,782,920 average for the channel.

The top 11 youtube-8m thumbnails (top) and the top 11 activating imagenet images (bottom) for the "dip" feature. The top activating 400 "dip" videos have a median view count of 38,685, compared to the 5,918 median for the dataset.

Motivation

At the time of writing there is difficulty in identifying practical use cases for Sparse Autoencoders.

There are many existing projects (VidIQ, CreatorML) which try to detect high performing youtube videos using deep-learning. One useful element these systems tend to lack is interpretability, i.e. they can predict whether a thumbnail or title will lead to higher views but they cannot pick out specific conceptual elements which will aid or detract from engagement.

Sparse Autoencoders may have an advantage in this very narrow field if they can surface monosemantic features which are associated with higher views. These features could then inform creators on which elements to include in their videos, titles or thumbnails.

Methodology: Selecting Feature Groups

Following Hugo Fry[1] we train a CLIP sparse autoencoder on 1,281,167 imagenet images. We then collect Sparse Autoencoder latents across two datasets of youtube thumbnails:

524,288 videos from the the Youtube-8m training set, extracted up till 2018.
329 recent, longform MrBeast videos (we remove all shorts and all videos before the channel hit 100,000

...
How does someone prove that their general intelligence is above average?
by M. Y. Zuo — 2024-09-16T21:01:38.529Z

Many talk about IQ in many places, in all kinds of contexts and situations, and IQ can be ‘provable’ to a certain degree of reliability and consistency using methods such as Raven’s progressive matrices.

But how is the general intelligence of the individual proven to any degree of reliability/consistency/validity/etc…?

The only way I can think of is successfully predicting the future in a way and publicly sharing the predictions before and after, many many many times in a row. And in such a way that no interested parties could realistically help them fake it.

And the only practical way to realize this, that I can think of now, is by predicting the largest stock markets such as the NYSE, via some kind of options trading, many many many times within say a calendar year, and then showing their average rate of their returns is significantly above random chance.

And even this is kind of iffy, since it would require sharing most of their trades publicly, along with the possibility of ‘cheating’ with insider information.

Has anyone thought about the exact methods that are feasible?

Edit: Any such methods would probably also apply to AI but I don’t want to extrapolate too far.
Does life actually locally *increase* entropy?
by tailcalled — 2024-09-16T20:30:33.148Z

The usual materialist story of life I've heard is that life acts like an entropy pump, creating local reductions of entropy within the organism but increasing the entropy outside of the organism. (I think I've even seen that in The Sequences somewhere? But couldn't find it, feel encouraged to link it.) But I've come to think that might actually be wrong and life might increase entropy both inside and outside the organism.

Here's a rough account:

We ought to expect entropy to increase, so a priori life is much more feasible if it increases entropy rather than decreasing entropy.
Living matter is built mainly out of carbon and hydrogen, which is extracted from CO2 and H2O, leaving O2 as a result. Entropy breakdown:

The O2 left over from breaking up CO2 ought to have somewhat lower entropy than the original CO2.
The O2 left over from breaking up the original H2O ought to have... higher entropy because it's a gas now?
The hydrocarbons don't have much entropy because they stick together into big chunks that therefore heavily constrain their DOFs, but they do have some entropy for various reasons, and they are much more tightly packed than air, so per volume they oughta have orders of magnitude more entropy density. (Claude estimates around 200x.)
Organic matter also traps a lot of water which has a high entropy density.
Usually you don't talk about entropy density rather than absolute entropy, but it's unclear to me what it means for organisms to "locally" increase/decrease entropy if not by density.

Oxygen + hydrocarbons = lots of free energy, while water + carbon dioxide = not so much free energy. We usually associate free energy with low entropy, but that's relative to the burned state where the free energy has been released into thermal energy. In this case, we should instead think relative to an unlit state where the energy hasn't been collected at all. Less energy generally correlates to lower entropy.

Am I missing something?
Book review: Xenosystems
by jessicata — 2024-09-16T20:17:56.670Z
Link: https://unstableontology.com/2024/09/16/book-review-xenosystems/

I've met a few Landians over the last couple years, and they generally recommend that I start with reading Nick Land's (now defunct) Xenosystems blog, or Xenosystems, a Passage Publishing book that compiles posts from the blog. While I've read some of Fanged Noumena in the past, I would agree with these Landians that Xenosystems (and currently, the book version) is the best starting point. In the current environment, where academia has lost much of its intellectual relevance, it seems overly pretentious to start with something as academic as Fanged Noumena. I mainly write in the blogosphere rather than academia, and so Xenosystems seems appropriate to review.

The book's organization is rather haphazard (as might be expected from a blog compilation). It's not chronological, but rather separated into thematic chapters. I don't find the chapter organization particularly intuitive; for example, politics appears throughout, rather than being its own chapter or two. Regardless, the organization was sensible enough for a linear read to be satisfying and only slightly chronologically confusing.

That's enough superficialities. What is Land's intellectual project in Xenosystems? In my head it's organized in an order that is neither chronological nor the order of the book. His starting point is neoreaction, a general term for an odd set of intellectuals commenting on politics. As he explains, neoreaction is cladistically (that is, in terms of evolutionary branching-structure) descended from Moldbug. I have not read a lot of Moldbug, and make no attempt to check Land's attributions of Moldbug to the actual person. Same goes for other neoreactionary thinkers cited.

Neoreaction is mainly unified by opposition to the Cathedral, the dominant ideology and ideological control system of the academic-media complex, largely branded left-wing. But a negation of an ideology is not itself an ideology. Land describes a "Trichotomy" within neo-reaction (citing Spandrell), of three currents: religious theonomists, ethno-nationalists, and techno-commercialists.

Land is, obviously, of the third type. He is skeptical of a unification of neo-reaction except in its most basic premises. He centers "exit", the option of leaving a social system. Exit is related to sectarian splitting and movement dissolution. In this theme, he eventually anno...
MIRI's September 2024 newsletter
by Harlan — 2024-09-16T18:15:40.785Z
Link: https://intelligence.org/2024/09/16/september-2024-newsletter/

MIRI updates

Aaron Scher and Joe Collman have joined the Technical Governance Team at MIRI as researchers. Aaron previously did independent research related to sycophancy in language models and mechanistic interpretability, while Joe previously did independent research related to AI safety via debate and contributed to field-building work at MATS and BlueDot Impact.

In an interview with PBS News Hour’s Paul Solman, Eliezer Yudkowsky briefly explains why he expects smarter-than-human AI to cause human extinction.

In an interview with The Atlantic’s Ross Andersen, Eliezer discusses the reckless behavior of the leading AI companies, and the urgent need to change course.

News and links

Google DeepMind announced a hybrid AI system capable of solving International Mathematical Olympiad problems at the silver medalist level. In the wake of this development, a Manifold prediction market significantly increased its odds that AI will achieve gold level by 2025, a milestone that Paul Christiano gave less than 8% odds and Eliezer gave at least 16% odds to in 2021.

The computer scientist Yoshua Bengio discusses and responds to some common arguments people have for not worrying about the AI alignment problem.

SB 1047, a California bill establishing whistleblower protections and mandating risk assessments for some AI developers, has passed the State Assembly and moved on to the desk of Governor Gavin Newsom, to either be vetoed or passed into law. The bill has received opposition from several leading AI companies, but has also received support from a number of employees of those companies, as well as many academic researchers. At the time of this writing, prediction markets think it’s about 50% likely that the bill will become law.

In a new report, researchers at Epoch AI estimate how big AI training runs could get by 2030, based on current trends and potential bottlenecks. They predict that by the end of the decade it will be feasible for AI companies to train a model with 2e29 FLOP, which is about 10,000 times the amount of compute used to train GPT-4.

Abram Demski, who previously worked at MIRI as part of our recently discontinued Agent Foundations research program, shares an update about his independent research plans, some thoughts on public vs private research, and his current funding situation.

You can subscribe to the MIRI Newsletter here.
Generative ML in chemistry is bottlenecked by synthesis
by Abhishaike Mahajan — 2024-09-16T16:31:34.801Z
Link: https://www.owlposting.com/p/generative-ml-in-chemistry-is-bottlenecked

Introduction

Every single time I design a protein — using ML or otherwise — I am confident that it is capable of being manufactured. I simply reach out to

Twist Biosciences

, have them create a plasmid that encodes for the amino acids that make up my proteins, push that plasmid into a cell, and the cell will pump out the protein I created.

Maybe the cell cannot efficiently create the protein. Maybe the protein sucks. Maybe it will fold in weird ways, isn’t thermostable, or has some other undesirable characteristic.

But the way the protein is created is simple, close-ended, cheap, and almost always possible to do.

The same is not true of the rest of chemistry. For now, let’s focus purely on small molecules, but this thesis applies even more-so across all of chemistry.

Of the 1060 small molecules that are theorized to exist, most are likely extremely challenging to create. Cellular machinery to create arbitrary small molecules doesn’t exist like it does for proteins, which are limited by the 20 amino-acid alphabet. While it is fully within the grasp of a team to create millions of de novo proteins, the same is not true for de novo molecules in general (de novo means ‘designed from scratch’). Each chemical, for the most part, must go through its custom design process.

Because of this gap in ‘ability-to-scale’ for all of non-protein chemistry, generative models in chemistry are fundamentally bottlenecked by synthesis.

This essay will discuss this more in-depth, starting from the ground up of the basics behind small molecules, why synthesis is hard, how the ‘hardness’ applies to ML, and two potential fixes. As is usually the case in my

Argument

posts, I’ll also offer a

steelman

to this whole essay.

To be clear, this essay will not present a fundamentally new idea. If anything, it’s such an obvious point that I’d imagine nothing I’ll write here will be new or interesting to people in the field. But I still think it’s worth sketching out the argument for those who aren’t familiar with it.

What is a small molecule anyway?

Typically organic compounds with a molecular weight under 900 daltons. While proteins are simply long chains composed of one-of-20 amino acids, small molecules display a higher degree of complexity. Unlike amino acids, which are limited to carbon, hydrogen, nitrogen, and oxygen, small molecules incorporate a much wider range of elements from across the period...
Secret Collusion: Will We Know When to Unplug AI?
by schroederdewitt — 2024-09-16T16:07:01.119Z

TL;DR: We introduce the first comprehensive theoretical framework for understanding and mitigating secret collusion among advanced AI agents, along with CASE, a novel model evaluation framework. CASE assesses the cryptographic and steganographic capabilities of agents, while exploring the emergence of secret collusion in real-world-like multi-agent settings. Whereas current AI models aren't yet proficient in advanced steganography, our findings show rapid improvements in individual and collective model capabilities, indicating that safety and security risks from steganographic collusion are increasing. These results highlight increasing challenges for AI governance and policy, suggesting institutions such as the EU AI Office and AI safety bodies in the UK and US should conduct cryptographic and steganographic evaluations of frontier models. Our research also opens up critical new pathways for research within the AI Control framework.

Philanthropist and former Google CEO Eric Schmidt said in 2023 at a Harvard event:

"[...] the computers are going to start talking to each other probably in a language that we can't understand and collectively their super intelligence - that's the term we use in the industry - is going to rise very rapidly and my retort to that is: do you know what we're going to do in that scenario? We're going to unplug them [...]

But what if we cannot unplug them in time because we won't be able to detect the moment when this happens? In this blog post, we, for the first time, provide a comprehensive overview of the phenomenon of secret collusion among AI agents, connect it to foundational concepts in steganography, information theory, distributed systems theory, and computability, and present a model evaluation framework and empirical results as a foundation of future frontier model evaluations.

This blog post summarises a large body of work. First of all, it contains our pre-print from February 2024 (updated in September 2024) "Secret Collusion among Generative AI Agents". An early form of this pre-print was presented at the 2023 New Orleans (NOLA) Alignment Workshop (see this recording NOLA 2023 Alignment Forum Talk Secret Collusion Among Generative AI Agents: a Model Evaluation Framework). Also, check out this long-form Foresight Institute Talk).

In addition to these prior works, we also include new results. These contain empirical studi...
GPT-o1
by Zvi — 2024-09-16T13:40:06.236Z

Terrible name (with a terrible reason, that this ‘resets the counter’ on AI capability to 1, and ‘o’ as in OpenAI when they previously used o for Omni, very confusing). Impressive new capabilities in many ways. Less impressive in many others, at least relative to its hype.

Clearly this is an important capabilities improvement. However, it is not a 5-level model, and in important senses the ‘raw G’ underlying the system hasn’t improved.

GPT-o1 seems to get its new capabilities by taking (effectively) GPT-4o, and then using extensive Chain of Thought (CoT) and quite a lot of tokens. Thus that unlocks (a lot of) what that can unlock. We did not previously know how to usefully do that. Now we do. It gets much better at formal logic and reasoning, things in the ‘system 2’ bucket. That matters a lot for many tasks, if not as much as the hype led us to suspect.

It is available to paying ChatGPT users for a limited number of weekly queries. This one is very much not cheap to run, although far more cheap than a human who could think this well.

I’ll deal with practical capabilities questions first, then deal with safety afterwards.

Introducing GPT-o1

Sam Altman (CEO OpenAI): here is o1, a series of our most capable and aligned models yet.

o1 is still flawed, still limited, and it still seems more impressive on first use than it does after you spend more time with it.

But also, it is the beginning of a new paradigm: AI that can do general-purpose complex reasoning.

o1-preview and o1-mini are available today (ramping over some number of hours) in ChatGPT for plus and team users and our API for tier 5 users.

worth especially noting:

a fine-tuned version of o1 scored at the 49th percentile in the IOI under competition conditions! and got gold with 10k submissions per problem.

Extremely proud of the team; this was a monumental effort across the entire company.

Hope you enjoy it!

Noam Brown has a summary thread here, all of which is also covered later.

Will Depue (of OpenAI) says OpenAI deserves credit for openly publishing its research methodology here. I would instead say that they deserve credit for not publishing their research methodology, which I sincerely believe is the wise choice.

Pliny took longer than usual due to rate limits, but after a few hours jailbroke o1-preview and o1-mini. Also reports that the CoT can be prompt injected. Full text is at the link above...
Can subjunctive dependence emerge from a simplicity prior?
by Daniel C — 2024-09-16T12:39:35.543Z

Suppose that an embedded agent models its environment using an approximate simplicity prior, would it acquire a physicalist agent ontology or an algorithmic/logical agent ontology?

One argument for the logical agent ontology is that it allows the agent to compress different parts of its observations that are subjunctively dependent: If two physical systems are computing the same function, the logical agent ontology only has to store that function once, then model the dependencies between those two systems and the function. On the other hand, information about that function would be redundantly represented in both physical systems under the physicalist agent ontology.

Most decision theory literature seems to treat (physical) causal dependence as the default, requiring extra work to formalize subjunctive dependence. However, if logical agent ontology naturally emerges from a simplicity prior, we might expect subjunctive dependence to arise by default for most agents.

Thanks to Anthony Digiovanni for the discuission that inspired this post
Longevity and the Mind
by George3d6 — 2024-09-16T09:43:09.700Z

A framing I quite like is that of germs vs soma, body vs eggs and cum, consciousness vs replicators.

My first foray into age reversal was a (successful) attempt to increase fluid IQ, the loss of which is among less than a handful of ubiquitous symptoms of aging.

At the time, I found it odd that most people working in biotech, medicine, and longevity were confused about why I did this.

In part, the issue here is one of people finding it hard to break out of existing paradigms, even as hypotheticals. The limits of my language mean the limits of my world. (I expand more on it in this talk)

In part, I believe the medical, biotech and even longevity community are misguided about what goals are worthwhile to achieve.

Reversing the aging process is one of the most lofty goals achievable within our century, and a necessary stepping stone for humanity. Very few people, if any, are interested in this issue; Most end up working on curing diseases or minimally prolonging life in old age. (More on this later)

I believe these two parts are linked and socially caused; We have a tendency to copy each other, and we have incentive chains pulling people towards patentable narrow-effect drugs and therapies. And when corrupt incentives pull people for long enough, we oft forget about the chain and think of it as "just the way the world is".

Let me stop with digressions: assuming you fall into neither of these pitfalls you might still be confused as to why reversing the mind's aging ought to be the most (only?) relevant problem in longevity.

I - Germs and Soma

When discussing eternal youth, a framing I quite like is that of germs vs soma, body vs eggs and cum, consciousness vs replicators.

Many things are desirable individual traits when it comes to natural selection giving rise to ape populations in and around the Congo. Such desirable features include proclivities to age, rape, die, murder, and go infertile.

Humans distinguish themselves among primates by being able to break to the "meta" level of the natural selection game using a "mesa" optimizer (big brain =>more food, less death, more kids).

It's only incidental that these brains allow us to contemplate our condition and give rise to the fundamental question "What the fuck is going on?". (We start caring about everything else)

All good people are concerned with this question at the level above one individual (Gods, AIs, societies, cu...