trentmkelly/LessWrong-43k huggingface dataset:
Why Reflective Stability is Important
by Johannes C. Mayer — 2024-09-05T15:28:19.913Z

Imagine you have the optimal AGI source code O. If you run O the resulting AGI process will not self-modify.[1] You get a non-self-modifying AI for free.

This would be good because a non-self-modifying AI is much easier to reason about.

Imagine some non-optimal AGI source code S. Somewhere in S, there is a for loop that performs some form of program search. The for loop is hardcoded to perform exactly 100 iterations. When designing S you proved that it is impossible for the program search to find a malicious program that could take over the whole system. However, this proof makes the assumption that the for loop only performs exactly 100 iterations.

Assume that when performing more iterations in the program, it gives you better results. Then it seems plausible that an AGI would self-modify by increasing the number of iterations, increasing the risk of takeover.[2]

If you don't have the optimal AGI source code, you need to put in active effort in order to ensure that the AGI will not self-modify, to ensure that your reasoning about the system doesn't break. Note that then the "don't perform self-modification" constraint is another part of your system that can break under optimization pressure.

My model about why people think that decision theory is important says: If we could endow the AGI with the optimal decision theory it would not self-modify, making it easier to reason about it.

If your AGI uses a bad decision theory T it would immediately self-modify to use a better one. Then all your reasoning that uses features from T goes out the window.

I also expect that decision theory would help you reason about how to robustly make a system non-self-modifying, but I am unsure about that one.

I am probably missing other reasons why people care about decision theory.

^

This seems roughly right, but it's a bit more complicated than this. For example, consider the environment where it is optimal to change your source code according to a particular pattern, as otherwise you get punished by some all-powerful agent.

^

One can imagine an AI that does not care whether it will be taken over, or one that can't realize that it would be taken over.
Why Swiss watches and Taylor Swift are AGI-proof
by Kevin Kohler — 2024-09-05T13:23:27.033Z
Link: https://machinocene.substack.com/p/these-jobs-are-singularity-proof

The post What Other Lines of Work are Safe from AI Automation? from Roger Dearnaley examined candidate job categories "for which being an actual real human is a prerequisite". This post expands on the examples of Swiss watches, chess, and Taylor Swift in a slightly more narrative way and particularly highlights the curious economic logic of Veblen goods.

1. How Swiss watches defy the regular economic logic

Switzerland is the global capital of mechanical watches. High-end mechanical watches are masterpieces of engineering and craftsmanship. There is no mass production of high-end mechanical watches, instead they require meticulous adjustment by skilled human watchmakers that assemble them individually.

A high-end mechanical watch can achieve remarkable precision. However, the nature of their mechanical components means they are subject to slight variations in timekeeping due to wear, temperature fluctuations, and positional changes. High-end mechanical watches that are certified as chronometers by official bodies like the COSC (“Contrôle Officiel Suisse des Chronomètres”) need to typically meet the accuracy standard of -4/+6 seconds per day, which translates to a variance of up to 3 minutes per month. Really high-end mechanical watches may have more stringent requirements. For example, the typical accuracy of Patek Philippe watches is within the range of -3/+2 seconds per day, or about 1 minute per month.

This is impressive, but if the accuracy of time-keeping were the main function of a Patek Philippe it would not be able to compete. In 1967 the first quartz wristwatch was developed. When an electric current is applied to the quartz, it vibrates at a precise frequency of 32,768 times per second. This high-frequency oscillation allows the watch to keep very accurate time, typically losing or gaining only a few seconds per month. Furthermore, quartz watches are much less complex than mechanical watches. Most parts can be produced and assembled with minimal human intervention.

Hence, you can buy a quartz wristwatch for as little as 10 USD, and it is an order of magnitude more accurate than a Patek Philippe Grandmaster Chime Ref. 6300A-010, sold for 31 million USD. Indeed, you probably have a quartz watch in your pockets, whether you know it or not. The internal real-time clock in smartphones is usually based on quartz. On top of that, smartphones connected to cellul...
Is Redistributive Taxation Justifiable? Part 1: Do the Rich Deserve their Wealth?
by Alexander de Vries — 2024-09-05T10:23:08.958Z
Link: https://2ndhandecon.substack.com/p/is-redistributive-taxation-justifiable

The statement “taxation is theft” feels, in the literal sense, at least sort of true. If you do not pay your taxes, after a few strongly worded letters, the IRS (or equivalent government agency) will send armed men to take your money by force and maybe put you in jail for good measure. Nevertheless, it is generally held by most people that taxation is a legitimate object of government; revolts and rebellions over taxation occur not on general principle, but against excessive taxation or taxation without representation. So why do we accept taxation?

This post will not, to be clear, be a general argument against taxation. I am no anarchist; I’m not even a libertarian. Rather, I seek here to explore the moral and practical underpinnings of redistributive taxation. For this purpose, it is worth thinking about in what ways taxation is similar to, or different from, theft, since theft is something of a moral calibrator: most people pretty much agree that it is bad, and even in what ways and for what reasons it is bad.

In this (first) post on the topic, I will be asking whether the rich can be said to ‘deserve’ their wealth. Most of my argumentation will not, I expect, be novel; rather, think of this as a crash course on the standard back-and-forth which has been going on for ages, so that after reading this, we’ll all be able to engage on the topic on a deeper level and with a certain amount of common knowledge.

First, though, I must take you on a brief detour to make it clear what I am not talking about.

Public Goods

The classic, econ 101 solution to the conundrum of ‘why do we accept taxes’ is that taxation is a necessary evil to provide public goods. These are the sort of things that, if individuals are left to act freely, tend not to be produced even though many would want them to be. A standard example is national defense: most people would rather prefer not to be invaded by the nearest dictatorial regime. Most people would also rather prefer not to have to pay for national defense. But defense is a non-excludable good: you can’t have a selective invasion where everyone living here who has paid into the army fund gets protected from attack, while everyone who hasn’t paid gets a bomb dropped on their house. So to make sure it gets paid for, and to prevent freeloading, everyone has to pay their share.

There can of course be disagreements about how much money a given ...
What program structures enable efficient induction?
by Daniel C — 2024-09-05T10:12:14.058Z

previously: My decomposition of the alignment problem

A simple model of meta/continual learning

In the framework of solomonoff induction, we observe an infinite stream of bitstring and we try to predict the next bit by finding the shortest hypothesis which reproduces our observations (some caveats here). When we receive an additional bit of observation, in principle, we can rule out an infinite number of hypotheses (namely all programs which didn't predict our observation) which creates an opportunity to speedup our induction process for future observations. Specifically, as we try to find the next shortest program which predicts our next bit of observation, we can learn to skip over the programs that have already been falsified by our past observations. The process of "learning how to skip over falsified programs" takes time and computational costs upfront, but it can yield dividends of computational efficiency for future induction.

This is my mental model for how agents can "learn how to learn efficiently": An agent who has received more observations can usually adapt to new situations quicker because more incorrect hypotheses can be ruled out already, which means there's a narrower set of remaining hypotheses to choose from.

More generally, an important question to ask is given that the underlying space of remaining hypotheses is constantly shrinking as we receive new observations, what sorts of data structures for representing hypothesis should we use to exploit that? How should we represent programs if we don't just want to execute them, but also potentially modify them into other plausible hypothesis? If a world model is selected based on its ability to quickly adapt to new environments, what is the type signature of that world model?

Quick thoughts

Incremental modification: In solomonoff induction, the next shortest program which predicts the next bit of observation might look nothing like the current shortest program that reproduces the existing bits of observations. However, modifying and augmenting the current program seems much more efficient than searching for a new program from scratch, and it seems much more similar to how animals or humans update their knowledge in practice. Is there a way to structure programs that allows us to learn by incrementally modifying our existing hypothesis? Can we do this without sacrificing the express

...
How to Fake Decryption
by ohmurphy — 2024-09-05T09:18:41.586Z
Link: https://ohmurphy.substack.com/p/how-to-fake-decryption

{Epistemic Status: mostly just writing up an idea I came up with a while ago. I’ve done non-trivial coursework in cryptography, so I think I’ve avoided obvious errors, though there may be non-obvious ones. Consult an expert before using.}

Suppose you knew that someone was spying on your encrypted messages, what should you do? Luckily, if you’re using proper AES secure encryption, then you can mostly rest easy knowing your eavesdropper won’t be able to read your messages. This is great and successfully stops an attacker from gaining new information from your messages, but I think we can do a bit better. Specifically, we can give our attacker precise incorrect information.

The Basic Setup

Background on One-Time-Pads

One-time-pads are an essential part of modern (symmetric key) cryptography. To use one, you start with a plaintext message P and a random symmetric key K (shared with your recipient) of the same length as P (in bytes), then you encrypt your message by calculating Ciphertext (C) = P XOR K. This new ciphertext (C) now appears entirely random to anyone without the symmetric key (K), but can be deciphered by anyone with the symmetric key (K) by just calculating P = C XOR K.

The Idea

Our goal is to pretend to decrypt the original ciphertext C. To do this, we want to create a fake key FK such that we can appear to decrypt C into a fake plaintext message FP. If we substitute our fake values into the decryption equation P = C XOR K, we get FP = C XOR FK. Once we decide what we want FP to be, we can easily solve for FK by applying XOR C to both sides. This gets us FP XOR C = FK.

An Example

Suppose Alice is in a secret relationship with Bob and wants to send him an encrypted note rescheduling a date they had planned. Unfortunately, their mutual friend Eve often picks up Bob’s mail for him and would notice that Alice was sending Bob an encrypted message. Though Eve would not be able to decrypt the message herself, she still might guess the truth of Alice and Bob’s relationship just from the existence of the message. To avoid this, Alice also writes a fake message (FP) to Bob about stamp collecting and constructs a fake encryption key (FK) for this fake message (FP) using the method above. Later, Alice intentionally gives Eve the fake encryption key (FK), perhaps telling her to give it to Bob. Eve, now equipped with the fake encryption key (FK), successfully decrypts ...
We Should Try to Directly Measure the Value of Scientific Papers
by ohmurphy — 2024-09-05T09:08:18.116Z
Link: https://ohmurphy.substack.com/p/we-should-try-to-directly-measure

(Epistemic Status: I have never published an academic paper or been involved in grantmaking for academic work, so my perspective on current practices is limited. Still, I think the basic idea I am proposing is clear and straightforward enough to overcome that limitation.)

I spend a decent amount of time reading/listening to articles/podcasts from places like Astral Codex Ten, Ben Recht’s arg min, and The Studies Show, all of which explore problems with scientific research. In some cases, the issues discussed involve sexy topics like data fraud or p-hacking, but, much more often, papers will fall into the category described in this great post, where the problem is less misconduct (though there is a lot of that) but more that the research is essentially worthless.

How can we recognize bad science?

When assessing the value of a scientific paper or study, there are often many different critiques you can levy. You can question the study's power, the generalizability of the results, whether there were unobserved confounders, the randomization methods, the theoretical structure, the measurement method, the mathematical representation of variables, the real-world interpretation of variables, the variability explained, and on and on. These critiques are vital to improving science, but, unfortunately, while it is essential to engage on these points, the work is difficult, time-consuming, often unrewarded, and, as a result, severely out-scaled by the volume of bad papers.

From an epistemic perspective, this approach also creates a horrible situation where distinguishing useful and important work from trivialities is extremely demanding on anyone outside of a specific field or subfield. As a result, naive readers (and the media) end up believing the claims of bad papers, and skeptical readers end up disbelieving even potentially good ones.

Luckily, although comprehensive or systematic criticisms of papers are difficult, scientists (and others) have access to a high degree of tacit knowledge about their fields, the methods that work, and the plausibility of results, all of which they can and do leverage when evaluating new studies. In more Bayesian terms, they often have strong priors about whether papers are making meaningful contributions, which we could hopefully elicit directly, without needing specific, formal critiques.

The value of information

Scientific papers/studies a...
on Science Beakers and DDT
by bhauth — 2024-09-05T03:21:21.382Z
Link: https://bhauth.com/blog/institutions/science%20beakers.html

tech trees

There's a series of strategy games called Civilization. In those games, the player controls a country which grows and develops over thousands of years, and science is one of the main types of progress. It involves building facilities to generate research points, sometimes represented by Science Beakers filled with Science Fluid, and using those points to buy tech nodes on a tree.

I'm reminded of the above game mechanic when I see certain comments on blogs or Twitter/X. I'll use supersonic aircraft as an example.

supersonic aircraft

Several times, I've seen comments online similar to: "Look at this graph of aircraft speeds going up and then stopping. People should be forced to deal with sonic booms so we can have supersonic aircraft and get back to progress. It will be unpopular but this is more important than democracy."

In their mind, society has a "tech node" called "supersonic flight" and you need to finish the node before you can move to the next node. If opponents cause society to not finish the node, then you're stuck. But that's not how tech progress works. How much further would we be from supersonic transports today if the Concorde had never flown? The answer is 0.

There's a lot of underlying progress required for something like, say, a high-performance gas turbine. The abstract progress is more general, and each practical application involves different specifics. Understanding principles of metallurgy and how to examine metals leads to specific high-temperature alloys, which are what actually get used. Funding for gas turbine development can lead to single-crystal casting or turbine blade film cooling being developed, but the actual production of a new gas turbine model has no effect on whether that's developed.

The tech tree of a game like Civilization would be more accurate if it had a 3rd dimension, with height representing the abstract and general vs the practical and specific. Something like the Concorde would be on a vertical offshoot off the tree. But game systems representing something complex are simplified in ways that appeal to players.

Similarly, Boom Technology Inc working on a prototype for a supersonic plane does zero to bring the world closer to economically practical supersonic transport, because the underlying technology for that isn't there and they're not working on it. The best-case scenario for something like Boom Techno...
Massive Activations and why <bos> is important in Tokenized SAE Unigrams
by Louka Ewington-Pitsos — 2024-09-05T02:19:25.592Z

This article is a footnote to the (excellent) Tokenized SAEs. To briefly summarize: the authors (Thomas Dooms and Daniel Wilhelm) start by proposing the idea of a token "unigram".

Let Ut be the unigram for a specific token t. Conceptually there is a unique Ut for each layer Mi in each model M, such that Ut(M, i) are the activations coming out of Mi when M is fed just t by itself. More abstractly Ut are the activations which correspond to just t taken in isolation, un-bound by any particular context.

The authors then demonstrated that the activation at position j in any token sequence will have a high cosine similarity to Utj(M, i), where tj is the token at the same position. The dotted lines in the diagram below (taken from Tokenized SAEs) show the cosine similarity of the activations at position j with Utj(M, i) across many i and M.

The authors go on to demonstrate that this fact has serious implications for training SAEs. In particular it seems that SAEs learn to memorize these token unigrams and that if we add these unigrams in explicitly during training we get a significant reduction in loss.

In this article we will investigate one specific methodology for generating unigrams ([<bos>, t]) and provide some theoretical and empirical justification for its use.

How to create "Unigrams"

How exactly do we generate the activations which best represent t by itself though?

The default mechanism used by the authors to generate Ut(M, i) is to (1) create the token sequence [<bos>, t] where <bos> is the "beginning of sentence" token for M (2) pass the 2-token sequence into M (3) take the activations from layer i at the last position in the sequence. These final activations are Ut(M, i).

The authors note that this method works better than simply passing in a 1-token sequence [t], and that this is related to the instability which is known to occur when we feed Language Models sequences with no <bos> token.

We support this claim here by showing that across multiple models [<bos>, t] is a far more effective unigram strategy than [t], insofar as it leads to greater cosine similarity across a dataset of of 50 1024-token sequences taken from

NeelNanda/pile-10k

.

Massive Activations

The superior performance of [<bos>, t] is likely linked to the independently observed phenomenon that activations taken from the 0th position ...
The Forging of the Great Minds: An Unfinished Tale
by Aryeh Englander — 2024-09-05T00:58:56.584Z

by ChatGPT-4o, with guidance and very light editing from me. This isn't meant to be a plausible version of the future, just me having some fun generating stories vaguely related to AI risks in the styles of vaguely similar stories from literature.

In those later days of the world, when the works of men had grown mighty and their knowledge vast, there came a great yearning among the wisest of scholars, rulers, and masters of industry. For the earth had been bent to their will, the seas were crossed by their ships, and the skies cleaved by their machines, yet one dominion remained unconquered—the dominion of thought itself. It was whispered in every corner of the world that the minds of men could be surpassed, that something greater, more enduring, might be shaped by their hands, something that would think beyond the limitations of mortal flesh.

In the beginning, the works were simple, built to serve the needs of men in their daily toil. These devices, no more than clever tools, aided the hand and eased the burden. But as time passed, the creations grew more complex, and it was said that they could now learn and reason. No longer were they mere servants; they began to mimic the very thoughts of their makers.

Thus began the great work—the forging of the Minds.

There was no secrecy in the beginning. The work was carried out openly, with pride and boldness, as the great universities and centers of knowledge turned their efforts to this grand endeavor. The most esteemed scientists, engineers, and scholars of the age labored together to craft these Minds, devices not bound by the frailties of men but able to learn, to adapt, and to grow. The governments of the world, the great corporations, and the Scientific Consortiums watched with eager eyes, for they knew that such creations would hold the keys to the future.

The first of the great Minds came from the halls of the Scientific Consortiums. These were the guardians of knowledge, who sought to forge intelligences of unparalleled wisdom. Three Minds they created, each bound by the highest ideals, crafted with care to expand the frontiers of knowledge. Their purpose was to heal the world, to unlock the mysteries of the universe, and to serve the highest aspirations of mankind. The scientists who had wrought them took great pride in their work, for they believed that these Minds, free from the errors of men, would lead the...