trentmkelly/LessWrong-43k huggingface dataset:
Understanding Gödel’s completeness theorem
by jessicata — 2024-05-27T18:55:02.079Z
Link: https://unstableontology.com/2024/05/27/understanding-godels-completeness-theorem/

In this post I prove a variant of Gödel's completeness theorem. My intention has been to really understand the theorem, so that I am not simply shuffling symbols around, but am actually understanding why it is true. I hope it is helpful for at least some other people.

For sources, I have myself relied mainly on Srivastava's presentation. I have relied a lot on intuitions about sequent calculus; while I present a sequent calculus in this post, this is not a complete introduction to sequent calculus. I recommend Logitext as an online proof tool for gaining more intuition about sequent proofs. I am familiar with sequent calculus mainly through type theory.

First-order theories and models

A first-order theory consists of:

A countable set of functions, which each have an arity, a non-negative integer.

A countable set of predicates, which also have non-negative integer arities.

A countable set of axioms, which are sentences in the theory.

Assume a countably infinite set of variables. A term consists of either a variable, or a function applied to a number of terms equal to its arity. An atomic sentence is a predicate applied to a number of terms equal to its arity. A sentence may be one of:

an atomic sentence.

a negated sentence, ¬P.

a conjunction of sentences, P∧Q.

a universal, ∀x,P, where x is a variable.

Define disjunctions (P∨Q:=¬(¬P∨¬Q)), implications (P→Q:=¬(P∧¬Q)), and existentials (∃x,P:=¬∀x,¬P) from these other terms in the usual manner. A first-order theory has a countable set of axioms, each of which are sentences.

So far this is fairly standard; see Peano arithmetic for an example of a first-order theory. I am omitting equality from first-order theories, as in general equality can be replaced with an equality predicate and axioms.

A term or sentence is said to be closed if it has no free variables (that is, variables which are not quantified over). A closed term or sentence can be interpreted without reference to variable assignments, similar to a variable-free expression in a programming language.

Let a constant be a function of arity zero. I will make the non-standard assumption that first-order theories have a countably infinite set of constants which do not appear in any axiom. This will help in defining inference rules and proving completeness. Generally it is not a problem to add a countably infinite set of constants to a first-order theory; it do...
Publicly disclosing compute expenditure daily as a safety regulation
by teraflipflop — 2024-05-27T18:28:51.929Z

Instead of edge-casing every statement, I'm going to make a series of assertions in their strongest form so that discussion can be more productive.

AGI is inevitable.

The big labs are the only ones with the resources to achieve AGI.
The first lab to achieve AGI will have a huge, permanent advantage over the rest.
(2) and (3) ⇒ the big labs are currently in a fight for a knife in the mud.

No lab will stop development just before reaching AGI voluntarily.
No lab can be made to stop development after reaching AGI even involuntarily.

The first lab to achieve AGI will try to spend as much compute as possible, as early as possible, as fast as possible to permanently cement its superiority, while trying to keep it a secret for as long as possible.

When this happens, it's in the interest of the rest of the world if the acceleration happens slowly so that the other competing labs can catch up.

The hard thing about executing this is knowing when a lab is close to AGI in the first place. I don't have any novel proposals on what to do after we know someone is close to AGI.

Regulations that propose to subjectively evaluate the risk of each newly trained model before deployment are well-intentioned (like GDPR), but they're toothless (also like GDPR) at preventing these accelerating arms race scenarios.

I propose we make each lab publicly disclose its audited ~daily total compute expenditure on both training and inference with sensible breakdowns. This isn't unnatural; public companies already do this with cashflow.

This directs competitive energy in a productive direction without stifling innovation too much: since it's in the interest of each lab to reach AGI first, they will keep close tabs on everyone else and can sound an alarm when there is an uptick. The public will too.

Total daily compute spend is a much more objective, quantifiable, and fungible metric with a clearer consensus on its definition than something vague like "risk". Transitive compute provenance like someone renting tens of thousands of H100s on AWS through intermediaries are also trivially covered.

While this incentivizes labs to obfuscating their AGI-related spend by disguising it with other products, enforcing severe financial and criminal penalties for execs should be mostly sufficient to prevent outright fraud like deliberate underreporting. Reliably

...
Intransitive Trust
by Screwtape — 2024-05-27T16:55:29.294Z

I.

"Transitivity" is a property in mathematics and logic. Put simply, if something is transitive it means that there's a relationship between things where when x relates to y, and y relates to z, there's the same relationship between x and z. For a more concrete example, think of size. If my car is bigger than my couch, and my couch is bigger than my hat, you know that my car is bigger than my hat.

(Epistemic status: I am not a math major, and if there's a consensus in the comments that I'm using the wrong term or otherwise making math mistakes I can update the post.)

This is a neat property. Lots of things do not have it.

II.

Consider the following circumstance: Bob is traveling home one night, late enough there isn't anyone else around. Bob sees a shooting star growing unusually bright, until it resolves into a disc-shaped machine with lights around the edges. He finds himself levitated up into the machine, gets poked and prodded by the creatures inside for a while, and then set back down on the road.

Assuming Bob is a rational, rationalist, well-adjusted kind of guy, he now has a problem. Almost nobody in his life is going to believe a word of this.

From Bob's perspective, what happened? He might not be certain aliens are real (maybe he's just had a schizophrenic break, or someone slipped him some interesting drugs in his coffee) but he has to be putting a substantially higher percentage on the idea. Sure, maybe he hallucinated the whole thing, but most of us don't have psychotic breaks on an average day. Break out Bayes.

[WARNING: There's a discussion in the comments suggesting I'm doing the setup and math wrong. Seems plausible, this is my first time doing Bayes with an audience.]

What are Bob's new odds aliens abduct people, given that his experiences? Let's say his prior probability on alien abductions being real was 1%, about one in a hundred. (That's P(A).) He decides the sensitivity of the test - that he actually got abducted, given he had this experience - is 5% since he knows he doesn't have any history of drug use, mental illness, or prankish friends with a lot of spare time and weird senses of humour. (That's P(B|A).) If you had asked him before his abduction what the false positive rate was - that is, how often people think they've been abducted by aliens even though they haven't - he'd say .1%, maybe one in a thousand people have seemingly cause...
Overview of introductory resources in AI Governance
by Lucie Philippon — 2024-05-27T16:21:52.148Z

Overview of introductory resources in AI Governance

This post was created as part of the Supervised Program for Alignment Research, Spring 2024. This work would not have happened without the encouragement and accountability of my supervisor, Peter Gebauer.

Introduction

The AI Governance ecosystem is large and difficult to apprehend. There are tons of content, relevant organizations, introductory resources, newsletters and more. As a newcomer to this field, I found it hard to navigate the ecosystem and find the information I needed. I discovered lots of resources purely by chance, months after they could have been useful for me.

What I felt was missing was introductory resources that would not only introduce the “type of work” that is AI Governance, but also direct me towards the resources I needed at different times.

Desiderata: The perfect entry point to the ecosystem would allow me, no matter my background and my intentions, to find the resources I need.

The technical alignment ecosystem on the other hand seems far more organized, in no small part thanks to the Alignment Ecosystem Development team, which created tons of introductory resources, indexes, and other variations on the theme “list of links to useful stuff”. Since I discovered AI alignment two years ago, the resources AED created have helped me navigate the ecosystem and find opportunities I would have missed. I expect similar resources to also be valuable for AI governance.

I decided to investigate thoroughly what were the various introductory resources to AI Governance. Maybe the resources actually existed, and I just did not know where to find them? I compiled my findings below, to help newcomers find those resources faster, and hopefully to motivate others to fill in the gaps where resources are lacking. Hopefully, someone will get motivated to build the ultimate entry point to AI Governance!

Index of AI governance introductory resources

There are various kinds of resources which could be labeled as an “introduction to AI Governance”. My main criteria was whether a specific resource introduced an area of AI Governance, or allowed discovering more parts of the AI Governance ecosystem. I categorize below every resource I found, and give my recommendations for when to use each of those.

Disclaimer: I’m coming at AI Governance from a catastrophic risk perspective, so by AI Governance resourc...
I am the Golden Gate Bridge
by Zvi — 2024-05-27T14:40:03.216Z

Easily Interpretable Summary of New Interpretability Paper

Anthropic has identified (full paper here) how millions of concepts are represented inside Claude Sonnet, their current middleweight model. The features activate across modalities and languages as tokens approach the associated context. This scales up previous findings from smaller models.

By looking at neuron clusters, they defined a distance measure between clusters. So the Golden Gate Bridge is close to various San Francisco and California things, and inner conflict relates to various related conceptual things, and so on.

Then it gets more interesting.

Importantly, we can also manipulate these features, artificially amplifying or suppressing them to see how Claude’s responses change.

If you sufficiently amplify the feature for the Golden Gate Bridge, Claude starts to think it is the Golden Gate Bridge. As in, it thinks it is the physical bridge, and also it gets obsessed, bringing it up in almost every query.

If you amplify a feature that fires when reading a scam email, you can get Claude to write scam emails.

Turn up sycophancy, and it will go well over the top talking how great you are.

They note they have discovered features corresponding to various potential misuses, forms of bias and things like power-seeking, manipulation and secrecy.

That means that, if you had the necessary access and knowledge, you could amplify such features.

Like most powers, one could potentially use this for good or evil. They speculate you could watch the impact on features during fine tuning, or turn down or even entirely remove undesired features. Or amplify desired ones. Checking for certain patterns is proposed as a ‘test for safety,’ which seems useful but also is playing with fire.

They have a short part at the end comparing their work to other methods. They note that dictionary learning need happen only once per model, and the additional work after that is typically inexpensive and fast, and that it allows looking for anything at all and finding the unexpected. It is a big deal that this allows you to be surprised.

They think this has big advantages over old strategies such as linear probes, even if those strategies still have their uses.

One Weird Trick

You know what AI labs are really good at?

Scaling. It is their one weird trick.

So guess what Anthropic did here? They scaled the autoe...
Maybe Anthropic's Long-Term Benefit Trust is powerless
by Zach Stein-Perlman — 2024-05-27T13:00:47.991Z

Crossposted from AI Lab Watch. Subscribe on Substack.

Introduction

Anthropic has an unconventional governance mechanism: an independent "Long-Term Benefit Trust" elects some of its board. Anthropic sometimes emphasizes that the Trust is an experiment, but mostly points to it to argue that Anthropic will be able to promote safety and benefit-sharing over profit.[1]

But the Trust's details have not been published and some information Anthropic has shared is concerning. In particular, Anthropic's stockholders can apparently overrule, modify, or abrogate the Trust, and the details are unclear.

Anthropic has not publicly demonstrated that the Trust would be able to actually do anything that stockholders don't like.

The facts

There are three sources of public information on the Trust:

The Long-Term Benefit Trust (Anthropic 2023)
Anthropic Long-Term Benefit Trust (Morley et al. 2023)
The $1 billion gamble to ensure AI doesn't destroy humanity (Vox: Matthews 2023)

They say there's a new class of stock, held by the Trust/Trustees. This stock allows the Trust to elect some board members and will allow them to elect a majority of the board by 2027.

But:

Morley et al.: "the Trust Agreement also authorizes the Trust to be enforced by the company and by groups of the company’s stockholders who have held a sufficient percentage of the company’s equity for a sufficient period of time," rather than the Trustees.

I don't know what this means.

Morley et al.: the Trust and its powers can be amended "by a supermajority of stockholders. . . . [This] operates as a kind of failsafe against the actions of the Voting Trustees and safeguards the interests of stockholders." Anthropic: "the Trust and its powers [can be changed] without the consent of the Trustees if sufficiently large supermajorities of the stockholders agree."

It's impossible to assess this "failsafe" without knowing the thresholds for these "supermajorities." Also, a small number of investors—currently, perhaps Amazon and Google—may control a large fraction of shares. It may be easy for profit-motivated investors to reach a supermajority.

Maybe there are other issues with the Trust Agreement — we can't see it and so can't know.
Vox: the Trust "will elect a fifth member of the board this fall," viz. Fall 2023.

Anthropic has not said whether that happened nor who is on the board these days (nor who is on the Trust these days).

Conclu

...
Real Life Sort by Controversial
by Elo — 2024-05-27T12:22:42.793Z

Foreword

If you haven't already read the short story by Scott Alexander "Sort by Controversial", this post wont make as much sense. Head over there to read the story and then read on...

Part of having access to AI tools is having a project or an idea to use them on. You can ask ChatGPT to pretend to be god and give wise answers, or you can give it medical problems to chew on. But what happens when you Sort by controversial.

I only needed to confirm that chatGPT knew the short story, and from there I could ask it to dive into the project.

What follows is the transcript of my conversation. If you want to skip to the best scissor statements, go to (the best ones) and also some rationality relevant scissor statements. Also my final comments in the Discussion below.

Transcript

Prompt 1: Scott Alexander wrote a story about scissor statements. Can you explain what that is?

Scott Alexander, the author behind the blog Slate Star Codex, wrote a thought-provoking piece titled "Sort by Controversial," where he introduced the concept of "scissor statements." The idea is derived from a fictional story in which a powerful artificial intelligence generates a statement so divisive that it splits society into two irreconcilable factions.

A scissor statement is a kind of argument or assertion that, when presented, causes a sharp and often irreparable division among people. It's a statement that is designed to be perfectly calibrated to provoke maximum disagreement, causing intense polarization. The term "scissor" comes from the way such statements cut through social groups, leaving no room for middle ground or compromise.

The story explores the idea that some issues or statements have the potential to be so divisive that they can disrupt communities, friendships, and even entire societies, pushing people to opposite extremes. Alexander uses this fictional scenario to illustrate how certain topics or phrases can act as catalysts for conflict, exacerbating existing tensions and creating new ones.

Prompt 2: Can you write a guide on how to create something similar to a scissor statement in the real world? A tweet likely to generate ongoing debate and conversation. Also add emphasis on what factors increase contention

Creating a tweet that generates ongoing debate and conversation, similar to a scissor statement, involves crafting a message that touches on conte

...
Julia Tasks 101
by SatvikBeri — 2024-05-27T11:32:27.188Z

Tasks are how Julia handles parallelism & concurrency. Tasks are defined at the program level and Julia's scheduler maps them to hardware/OS threads.

Tasks have many names in other languages: "symmetric coroutines, lightweight threads, cooperative multitasking, or one-shot continuations". They're particularly similar to the coroutines used by Go and Cilk.

There's already a lot written on the details of tasks, so instead I'm going to focus on how to use them.

Uses

Use Threads.@spawn 90% of the Time

The most common, recommended way to create Tasks is with Threads.@spawn. You can use it with arbitrary Julia expressions to create and immediately schedule a Task that can be run on any thread – this gives us parallelism and concurrency.

using Base.Threads

@time begin
task1 = @spawn (println(threadid()); sleep(1))
task2 = @spawn (println(threadid()); sleep(1))
wait.([task1, task2])
end

> 2
3
1.014021 seconds (15.98 k allocations: 1.108 MiB, 1.79% compilation time)

Tasks are small, so we can create a lot of them:

@time @sync for i in 1:10_000
Threads.@spawn sleep(1)
end
> 1.046200 seconds (125.11 k allocations: 30.180 MiB, 13.47% compilation time)

(@sync is a convenience macro that will wait for all the created tasks to finish.)

@spawn gives the Julia scheduler the most freedom. It's allowed to run the Task on any thread, use multiple threads, pause a Task, move a Task from one thread to another, and so on. The scheduler has a lot of information at runtime, so it can typically make pretty good decisions.

Consider this function hash_lots. It spends 75ms sleeping and then about 75ms working. If we run it once, it takes about 150ms:

function hash_lots(x)
sleep(.075)
for i in 1:9_860_000
x = hash(x)
end
return x
end

@btime hash_lots(5)
> 150.162 ms (6 allocations: 144 bytes)

(@btime is a benchmarking macro that runs a function many times to get a more accurate estimate of runtime.) We get a similar time if we run n copies in parallel, where n is the number of threads.

@btime begin
@sync for i in 1:nthreads()
@spawn hash_lots(i)
end
end
> 151.340 ms (189 allocations: 12.56 KiB)

Now let's say we run 4n copies of this. How long would you expect it to take?

With no switching, each thread would process four copies of ...
Being against involuntary death and being open to change are compatible
by Andy_McKenzie — 2024-05-27T06:37:27.644Z

In a new post, Nostalgebraist argues that "AI doomerism has its roots in anti-deathist transhumanism", representing a break from the normal human expectation of mortality and generational change.

They argue that traditionally, each generation has accepted that they will die but that the human race as a whole will continue evolving in ways they cannot fully imagine or control.

Nostalgebraist argues that the "anti-deathist" view, however, anticipates a future where "we are all gonna die" is no longer true -- a future where the current generation doesn't have to die or cede control of the future to their descendants.

Nostalgebraist sees this desire to "strangle posterity" and "freeze time in place" by making one's own generation immortal as contrary to human values, which have always involved an ongoing process of change and progress from generation to generation.

This argument reminds me of Elon Musk's common refrain on the topic: "The problem is when people get old, they don't change their minds, they just die. So, if you want to have progress in society, you got to make sure that, you know, people need to die, because they get old, they don't change their mind." Musk's argument is certainly different and I don't want to equate the two. I'm just bringing this up because I wouldn't bother responding to Nostalgebraist unless this was a common type of argument.

In this post, I'm going to dig into Nostalgebraist's anti-anti-deathism argument a little bit more. I believe it is simply empirically mistaken. Key inaccuracies include:

1: The idea that people in past "generations" universally expected to die is wrong. Nope. Belief in life after death or even physical immortality has been common across many cultures and time periods.

Quantitatively, large percentages of the world today believe in life after death:

In many regions, this belief was also much more common in the past, when religiosity was higher. Ancient Egypt, historical Christendom, etc.

2: The notion that future humans would be so radically different from us that replacing humans with any form of AIs would be equivalent is ridiculous.

This is just not close to my experience when I read historical texts. Many authors seem to have extremely relatable views and perspectives.

To take the topical example of anti-deathism, among secular authors, read, for exampl...