trentmkelly/LessWrong-43k huggingface dataset:
A Solomonoff Inductor Walks Into a Bar: Schelling Points for Communication
by johnswentworth — 2024-07-26T00:33:42.000Z

A Solomonoff inductor walks into a bar in a foreign land. (Stop me if you’ve heard this one before.) The bartender, who is also a Solomonoff inductor, asks “What’ll it be?”. The customer looks around at what the other patrons are having, points to an unfamiliar drink, and says “One of those, please.”. The bartender points to a drawing of the same drink on a menu, and says “One of those?”. The customer replies “Yes, one of those.”. The bartender then delivers a drink, and it matches what the first inductor expected.

What’s up with that?

The puzzle, here, is that the two Solomonoff inductors seemingly agree on a categorization - i.e. which things count as the Unnamed Kind Of Drink, and which things don’t, with at least enough agreement that the customer’s drink-type matches the customer’s expectations. And the two inductors reach that agreement without learning the category from huge amounts of labeled data - one inductor points at an instance, another inductor points at another instance, and then the first inductor gets the kind of drink it expected. Why (and when) are the two inductors able to coordinate on roughly the same categorization?

Most existing work on Solomonoff inductors, Kolmogorov complexity, or minimum description length can’t say much about this sort of thing. The problem is that the customer/bartender story is all about the internal structure of the minimum description - the (possibly implicit) “categories” which the two inductors use inside of their minimal descriptions in order to compress their raw data. The theory of minimum description length typically treats programs as black boxes, and doesn’t attempt to talk about their internal structure.

In this post, we’ll show one potential way to solve the puzzle - one potential way for two minimum-description-length-based minds to coordinate on a categorization.

Main Tool: Natural Latents for Minimum Description Length

Fundamental Theorem

Here’s the main foundational theorem we’ll use. (Just the statement for now, more later.)

We have a set of n data points (binary strings) {xi}, and a Turing machine TM. Suppose we find some programs/strings Λ,{ϕi},Λ′,{ϕ′i} such that:

Mediation: (Λ,ϕ1,…,ϕn) is an approximately-shortest string such that (TM(Λ,ϕi) = xi for all i)
Redundancy: For all i, (Λ′,ϕ′i) is an approximately-shortest string such tha

...
What does a Gambler's Verity world look like?
by ErioirE — 2024-07-25T22:03:56.447Z

Status: Thought experiment for fun

Imagine a world in which the gambler's fallacy is fundamentally true. Functionally, lets suppose there's a magical force that tracks a thinking being's expectation of any particular outcome, and then mysteriously increases the likelihood of said outcome the more often it had physically plausible opportunity to occur and did not[1]. (This resets upon event occurrence).

In what ways do you expect this world would differ from ours?[2]

A few of my thoughts:

Depending on the tuning of the balance numbers & mechanics (I.E. What would determine the degree to which expectation shifts probability?) It might become optimal to learn about and deliberately consider the possibility of as many good things as possible. "I might run into a random celebrity/wealthy individual today and happen to befriend them!" "The bank might accidentally transfer a large amount of money into my account and forget about it!"
Inversely, it could in such a world be intrinsically dangerous to learn about too many bad things that could happen to you. Every time you go outdoors in a thunderstorm and consider that lightning could strike you at any time, it magically becomes more likely, and so someone who stays ignorant of such hazards is in less danger than someone who knows of them and does nothing different.
Does this stack with multiple beings considering the same possibility? Then it might enable "expectation farms" of weak AI thinking about particular things to tug on the threads of possibility.

^

'Opportunities' would have to be distinct somehow, otherwise you could say that every nanosecond standing in the storm is yet another opportunity. I'm not going to define this super rigorously, so just use whatever system seems intuitive and functional to you if you were the administrator of this simulation or whatever.

^

(Assuming it doesn't break causality or similarly hazardous anti-fun effects. Obviously it's magical and drilling down too far into the mechanics will inevitably conclude it's Magic All The Way Down)
Pacing Outside the Box: RNNs Learn to Plan in Sokoban
by Adrià Garriga-alonso — 2024-07-25T22:00:55.398Z
Link: https://arxiv.org/abs/2407.15421

Work done at FAR AI.

There has been a lot of conceptual work on mesa-optimizers: neural networks that develop internal goals that may differ from their training objectives (the inner alignment problem). There is an abundance of good ideas for empirical work (find search in a NN, interpret it), but very little actual execution, partly because we did not have a clear-cut example of a mesa-optimizer to study. Until now.[1]

We have replicated the mesa-optimizer that Guez et al. (2019) found, and released it open-source as a model organism for inner alignment research. In brief, Guez et al. trained a recurrent neural network (RNN) with model-free RL to play Sokoban. They noticed that if you give the RNN more time to think by repeating the initial observation at inference time, its performance increases. This is highly suggestive of planning!

We investigate this "planning effect" in a black-box way. We find that often, the RNN learns to "pace" before attempting to solve the level, likely to get more computation and find a solution. When we give the RNN time to think, it finds the solution in the extra thinking time and executes it straight away.

In other cases, the RNN sometimes starts with a greedy solution and locks itself out of the solution. With thinking time, the RNN finds the non-myopic solution, avoiding the lock and solving the level. Note that this greedy behavior may be bounded-rational given the -0.1 penalty per step: solving fewer levels but solving them more quickly can pay off.

These are illustrative examples, but we have quantitative evidence too. We operationalize the pacing behavior as whatever creates a cycle in the sequence of environment states. If we give the RNN time to think at level start, it does not 'pace' anymore: 75% of cycles that occur in the first 5 steps disappear. Time to think in the middle of a level also substitutes cycles: 82% of N-step cycles disappear with N steps to think.

The levels we use always have 4 boxes. Thinking time barely changes the average time the RNN takes to place boxes 1-3. But, when filtering only to levels that it cannot solve at 0 steps but can solve at 6 thinking steps, the time to place boxes 1-3 greatly increases, even though the time to place the 4th box barely changes. This indicates the NN is greedy by default, and thinking time remedies that.

Understanding how neural networks reason, and ultimat...
Sex, Death, and Complexity
by Zero Contradictions — 2024-07-25T21:22:56.558Z
Link: https://thewaywardaxolotl.blogspot.com/2019/01/sex-death-and-complexity.html

Cancer can be understood as evolution within the body. Cancer is a mutation, and that mutation is selected for, while the body is alive. The cancer cells are more "successful" than other cells, because they are selfish reproducers. They out-compete other cells for the resources of the body.

The coherence of the body depends on its cells retaining the information and the purpose that they inherited from the zygote. Over time, that information and purpose is degraded by entropy and changed by evolution. Without death, the body would eventually dissolve into a battleground of competing cells — a tragedy of the commons.

So, death is a consequence of entropy and evolution. The dynamics of sexual reproduction force the cells of the body to work together, rather than competing against each other (cancer). This is arguably the most important reason why sexual reproduction evolved. Sex, death, and complexity are all linked.
Does robustness improve with scale?
by ChengCheng — 2024-07-25T20:55:53.359Z
Link: https://far.ai/post/2024-07-robust-llm/

Adversarial vulnerabilities have long been an issue in various ML systems. Large language models (LLMs) are no exception, suffering from issues such as jailbreaks: adversarial prompts that bypass model safeguards. At the same time, scale has led to remarkable advances in the capabilities of LLMs, leading us to ask: to what extent can scale help solve robustness? In this post, we explore this question in the classification setting: predicting the binary label of a text input. We find that scale alone does little to improve model robustness, but that larger models benefit more from defenses such as adversarial training than do smaller models.

We study models in the classification setting as there is a clear notion of “correct behavior”: does the model output the right label? We can then naturally define robustness as the proportion of the attacked dataset that the model correctly classifies. We evaluate models on tasks such as spam detection and movie sentiment classification. We adapt pretrained foundation models for classification by replacing the generative model’s unembedding layer with a randomly initialized classification head, and then fine-tune the models on each task.

We focus on adversarial-suffix style attacks: appending an adversarially chosen prompt to a benign prompt in an attempt to cause the model to misclassify the input, e.g., classify a spam email as not-spam. We consider two attacks: the state-of-the-art Greedy Coordinate Gradient method (Zou et al., 2023), and a baseline random token attack. This simple threat model has the advantage of being unlikely to change the semantics of the input. For example, a spam email is still spam even if a handful of tokens are appended to it. Of course, attackers are not limited to such a simple threat model: studying more open-ended threat models (such as rephrasing the prompt, or replacing words with synonyms) and corresponding attack methods (such as LLM generated adversarial prompts) is an important direction that we hope to pursue soon in future work.

For more information, see our blog post or paper.
Organisation for Program Equilibrium reading group
by Smaug123 — 2024-07-25T19:11:02.332Z

2024-08-18: Second meeting on Friday 2024-08-16 at 7pm in Notes, Trafalgar Square. At the first meeting, we went over the basic model theory and proof theory that's required to understand the statement of Löb's theorem, and stepped through the proof; but we haven't got close to the meat of what I want to understand yet. (More detailed notes from the first meeting.)

2024-08-14: First meeting on Friday 2024-08-16 at 7pm in Notes, Trafalgar Square. [Details].

2024-07-25: I'm organising a little reading group, undecided between in-person in London or remote. My own questions I want to get the answers to are in https://www.patrickstevens.co.uk/posts/2024-07-25-lob-theorem/ . Writing this here so we have a place to start coordinating how we do this. I'm away on holiday until next Sunday so won't be doing anything about the organisation immediately (will just be trying to understand the material myself in the meantime).

Kickstarter status: about a dozen people have expressed interest, in response to my open call for interest in London.

Prerequisites I'll be expecting of people coming in:
* Be fluent and familiar with Peano arithmetic
* Grok the difference between provability and truth
* Have been at one point capable of precisely stating Gödel's incompleteness theorems, even if you no longer could state them correctly now.
* Have at least read [the paper](https://intelligence.org/files/ProgramEquilibrium.pdf) even if you haven't understood it at all; know the crucial property of an ideal agent that it's trying to construct
* Have spent a little time musing on the nature of Löb's theorem (for which https://www.lesswrong.com/posts/ALCnqX6Xx8bpFMZq3/the-cartoon-guide-to-loeb-s-theorem may be helpful, although personally I found Wikipedia's proof much easier to follow); perhaps still have your brain twisted into pretzels trying to understand what it actually means.

I might end up doing a single session beforehand covering some of the prerequisites, if people are interested in that; comment, or find and upvote an existing comment, if so.
In Text
by Valerii Kremnev — 2024-07-25T18:22:36.000Z

I find texts wonderful. If there was a rank of 7 Wonders of Evolution - texts would definitely take one of the prizes. Ok, that's not true - but at least they would receive my sincere democratic vote, and after that lose to such important crowd pleasers like Football and YouTube… Nevertheless, I have been thinking in text for quite a while, including this very moment, and most probably would continue doing so in the future. Also, around 3 years ago texts started gaining a lot of attention - you know, LLMs and all that stuff. So I decided that they could be a great and pretty important topic to cover them with an essay.

Let me briefly introduce myself. Sometimes it's great to be just a random folk on the internet, but this position doesn't really help in building trust. So, I would try to balance it. My name is Valerii. My craftsmanship is building software. Also - teaching others how to build it. I saved enough, dropped out of the regular job market, and decided to focus on a thing I love, the Art, which was the closest to my heart: Building Software. Data management and AI for Biotech - this is what my small company does.

As you could see, I have a little bit of a crush on complex topics. That's why, while I'll try my best to keep things simple, I would also try to allocate some space for the imaginary part. Due to my occupational bias, this text can have recursion or a bit of Time incoherence. I wanted to bring this as a warning sign, and I hope it won't bother you too much. All in all, these two are also just imaginary things. If that works well enough for you, then let's start and discuss the Text in Text.

I hope to have your attention by now, and If it is True, Then "Hello, dear Reader"

As a software engineer I see text as an interface. And it's absolutely amazing. From the engineering perspective you always aim to provide the information which is sufficient for achieving some goal, and at the same time make sure that only necessary information is transferred. Of course, you can have some redundancy, but not more than needed for smooth message interpretation and error correction.

Comparing to the other methods of sending the message, like voice, or images, or videos, text has unique abstraction properties. It gives your imagination much more space. Think about how cool it is to make your own personalized voiceover for everyone! And yo...
"AI achieves silver-medal standard solving International Mathematical Olympiad problems"
by gjm — 2024-07-25T15:58:57.638Z
Link: https://deepmind.google/discover/blog/ai-solves-imo-problems-at-silver-medal-level/

Google DeepMind reports on a system for solving mathematical problems that allegedly is able to give complete solutions to four of the six problems on the 2024 IMO, putting it near the top of the silver-medal category.

Well, actually, two systems for solving mathematical problems: AlphaProof, which is more general-purpose, and AlphaGeometry, which is specifically for geometry problems. (This is AlphaGeometry 2; they reported earlier this year on a previous version of AlphaGeometry.)

AlphaProof works in the "obvious" way: an LLM generates candidate next steps which are checked using a formal proof-checking system, in this case Lean. One not-so-obvious thing, though: "The training loop was also applied during the contest, reinforcing proofs of self-generated variations of the contest problems until a full solution could be found."

[EDITED to add:] Or maybe it doesn't work in the "obvious" way. As cubefox points out in the comments below, DeepMind's description doesn't explicitly say that it does that, and they're infuriatingly vague about how "AlphaProof generates solution candidates". The previous paragraph at least insinuates that it does it with an LLM, but it's very unclear.

(That last bit is reminiscent of something from the world of computer go: a couple of years ago someone trained a custom version of KataGo specifically to solve the infamous Igo Hatsuyoron problem 120, starting with ordinary KataGo and feeding it training data containing positions reachable from the problem's starting position. They claim to have laid that problem to rest at last.)

AlphaGeometry is similar but uses something specialized for (I think) Euclidean planar geometry problems in place of Lean. The previous version of AlphaGeometry allegedly already performed at gold-medal IMO standard; they don't say anything about whether that version was already able to solve the 2024 IMO problem that was solved using AlphaGeometry 2.

AlphaProof was able to solve questions 1, 2, and 6 on this year's IMO (two algebra, one number theory). It produces Lean-formalized proofs. AlphaGeometry 2 was able to solve question 4 (plane geometry). It produces proofs in its own notation.

The solutions found by the Alpha... systems are at https://storage.googleapis.com/deepmind-media/DeepMind.com/Blog/imo-2024-solutions/index.html. (There are links in the top-of-page navbar to solutions to the individual problems....
[Talk transcript] What “structure” is and why it matters
by Alex_Altair — 2024-07-25T15:49:00.844Z
Link: https://www.youtube.com/watch?v=z2-WEAMuhfs

This is an edited transcription of the final presentation I gave for the AI safety camp cohort of early 2024. It describes some of what the project is aiming for, and some motivation. Here's a link to the slides. See this post for a more detailed and technical overview of the problem.

This is the presentation for the project that is described as "does sufficient optimization imply agent structure". That's what we call the "agent structure problem" which was posed by John Wentworth, that's what we spent the project working on. But mostly for this presentation I'm going to talk about what we mean by "structure" (or what we hope to mean by structure) and why I think it matters for AI safety.

AI x-risk arguments are mostly conceptual

So we all have some beliefs about how AI systems are potentially very dangerous. But these beliefs are — I think they're valid and correct beliefs for the most part — but they're not based on experience directly, in the same way that we experience malaria as dangerous. We haven't seen a bunch of AGIs happen. And they're not based on formal methods of understanding things. Super volcanoes might be dangerous for example, but we understand the physics involved, and we know where they might be. Instead the arguments about AI risk are really trying to predict something that might happen, and they're what I would call philosophical or conceptual arguments. I think this makes it quite hard for us to know exactly what to do about it and it also makes it harder to convince other people that that it's that much of a risk. So I think there's an overall issue where we need to really refine these intuitions in what I would hope would be a mathematical sense.

Many of the arguments are about structure

So what are some of the things that we think make AI dangerous? There's capabilities, which is something people talk about all the time. More capabilities means more dangerous. What goes into capabilities? Well there's compute power. There's training on more data; if the AI systems know more about the world, if you give them all the data about the world, then they can have a bigger effect. And then, the more interaction they have with the world, like if it's an autonomous AI then it has higher ability to have higher capabilities.

Capabilities are necessary but not sufficient for dangerousness

That's a danger, but it's sort of necessary but not sufficient fo...
AI #74: GPT-4o Mini Me and Llama 3
by Zvi — 2024-07-25T13:50:06.528Z

We got two big model releases this week. GPT-4o Mini is covered here. Llama 3.1-405B (and 70B and 8B) is mostly covered in yesterday’s post, this has some follow up.

Table of Contents

Introduction.

Table of Contents.

Language Models Offer Mundane Utility. All your coding are belong to us.

Language Models Don’t Offer Mundane Utility. Math is hard. Can be expensive.

GPT-4o Mini Me. You complete me at lower than usual cost.

Additional Llama-3.1 Notes. Pricing information, and more rhetoric.

Fun With Image Generation. If you’re confused why artists are so upset.

Deepfaketown and Botpocalypse Soon. Not surprises.

They Took Our Jobs. Layoffs at Activision and across gaming.

In Other AI News. New benchmarks, new chip variants, and more.

The Art of the Jailbreak. Pliny remains undefeated.

Quiet Speculations. Where will the utility be coming from?

The Quest for Sane Regulations. Public opinion continues to be consistent.

Openly Evil AI. Some Senators have good questions.

The Week in Audio. Dwarkesh in reverse, and lots of other stuff. Odd Lots too.

Rhetorical Innovation. What are corporations exactly?

Aligning a Smarter Than Human Intelligence is Difficult. So are evals.

People Are Worried About AI Killing Everyone. Roon warns you to beware.

The Sacred Timeline. Hype?

Other People Are Not As Worried About AI Killing Everyone. Older Joe Rogan.

The Lighter Side. It’s on.

Language Models Offer Mundane Utility

Coding is seriously much faster now, and this is the slowest it will ever be.

Roon: pov: you are ten months from working for claude sonnet the new technical founder.

Garry Tan: Underrated trend.

It’s happening.

Sully: 50% of our code base was written entirely by LLMs expect this to be ~80% by next year With sonnet we’re shipping so fast, it feels like we tripled headcount overnight Not using Claude 3.5 to code? Expect to be crushed by teams who do (us).

Not only coding, either.

Jimmy (QTing Tan): It can also do hardware related things quite well too, and legal, and logistics (planning) and compliance even.

I’ve been able to put off hiring for months.

When I run out of sonnet usage I patch in gpt-4o, it’s obviously and notably worse which I why I rarely use it as a primary anymore.

Claude 3.5 Sonnet becomes the first AI to crush the Lem Test to ‘write an impossible...