trentmkelly/LessWrong-43k huggingface dataset:
Anthropic's Certificate of Incorporation
by Zach Stein-Perlman — 2024-06-12T13:00:30.806Z

Yesterday I obtained Anthropic's[1]

Certificate of Incorporation

, and

its past versions

, from the State of Delaware. I don't recommend reading it.[2] This post is about what the CoI tells us about Anthropic's

Long-Term Benefit Trust

(context:

Maybe Anthropic's Long-Term Benefit Trust is powerless

).

Tl;dr: the only new information of moderate importance is the voting thresholds necessary to modify Trust stuff. My concerns all still stand in some form. Absence of badness is a small positive update.

Anthropic has vaguely

described

stockholders' power over the Trust:

a series of "failsafe" provisions . . . allow changes to the Trust and its powers without the consent of the Trustees if sufficiently large supermajorities of the stockholders agree. The required supermajorities increase as the Trust’s power phases in

The CoI has details: amending the CoI to modify the Trust requires a vote reaching the "Transfer Approval Threshold," defined as:

(1) prior to the date that is the one-year anniversary of the Final Phase-In Date [note: "the Final Phase-In Date"

is

in November 2024], either (a)(i) a majority of the Voting Common Stock then-outstanding and held by the Founders (as defined in the Voting Agreement), (ii) a majority of the Series A Preferred Stock then-outstanding and (iii) a majority of the voting power of the outstanding Preferred Stock entitled to vote generally (which for the avoidance of doubt shall exclude the Non-Voting Preferred Stock), but excluding the Series A Preferred Stock or (b) at least seventy-five percent (75%) of the voting power of the then-outstanding shares of the Corporation's capital stock entitled to vote generally (which for the avoidance of doubt shall exclude the Non-Voting Preferred Stock and any voting power attributable to the Class T Common Stock) and

(2) on and following the date that is the one-year anniversary of the Final Phase-In Date, either (x)(i) at least seventy-five percent (75%) of the Voting Common Stock then outstanding and held by the Founders (as defined in the Voting Agreement), (ii) at least at least fifty percent (50%) of the Series A Preferred Stock then-outstanding and (iii) at least seventy-five percent (75%) of the voting power of the outstanding Preferred Stock entitled to vote generally (which for the avoidance of doubt shall exclude the Non-Voting Preferred Stock), bu

...
Calculance: A "Core" Ability
by milanrosko — 2024-06-12T07:21:03.170Z

There has been a long-standing gap in the English language for a single word representing the specific ability to perform effective logical operations. Introducing "calculance" to fill this void.

We could posit a priori (or reckon) that intelligence fundamentally arises from two core components: A goal and the ability to "world simulate", termed "calculance".

The English language lacks a dedicated term, equivalent to the German "Rechenfähigkeit", for the capacity to perform calculations and logical operations. Calculance fills this crucial gap.

(I have to stress that "Rechenfähigkeit" doesn't translate to computational power, "Rechenleistung".)

For better understanding, it would losely correspond to something like this:

Experts often awkwardly refer to FLOPS or computational power, followed by a kind of unspoken understanding, as if to say, "You know what I mean, the good FLOPS!" Although these concepts are related, FLOPS et al don't necessarily indicate the quality of computation. "Calculance," on the other hand, better represents the functional dimension of a computing system, beyond just the number of operations per second.

But again, it is not a computer science term, but a concept, akin to how the word "deep" in Deep Learning captures a performance metric.

Whether exploring human minds or designing intelligent machines, embracing the term could unlock a new level of clarity and precision in describing the inherent interplay of purpose and calculation that produces intelligence.

"The new update to the physics engine has less calculance, allowing it to run smoothly on lower-end hardware."

"The new update to the physics engine reduced its computational demands, allowing it to run smoothly on lower-end hardware."

Just an idea.

(A final note: I mostly like the word because of its sound. It could also serve as a vessel for a concept that would benefit from having its own unique term.)
AXRP Episode 33 - RLHF Problems with Scott Emmons
by DanielFilan — 2024-06-12T03:30:05.747Z

YouTube link

Reinforcement Learning from Human Feedback, or RLHF, is one of the main ways that makers of large language models make them ‘aligned’. But people have long noted that there are difficulties with this approach when the models are smarter than the humans providing feedback. In this episode, I talk with Scott Emmons about his work categorizing the problems that can show up in this setting.

Topics we discuss:

Deceptive inflation

Overjustification

Bounded human rationality

Avoiding these problems

Dimensional analysis

RLHF problems, in theory and practice

Scott’s research program

Following Scott’s research

Daniel Filan:
Hello, everybody. In this episode I’ll be speaking with Scott Emmons. Scott is a PhD student at UC Berkeley, working with the Center for Human-Compatible AI on AI safety research. He’s previously co-founded far.ai, which is an AI safety non-profit. For links to what we’re discussing, you can check the description of the episode, and for a transcript you can read it at axrp.net. Well, welcome to AXRP.

Scott Emmons:
Great to be here.

Deceptive inflation

Daniel Filan:
Sure. So today we’re talking about your paper, When Your AIs Deceive You: Challenges With Partial Observability of Human Evaluators in Reward Learning, by Leon Lang, Davis Foote, Stuart Russell, Erik Jenner, and yourself. Can you just tell us roughly what’s going on with this paper?

Scott Emmons:
Yeah, I could start with the motivation of the paper.

Daniel Filan:
Yeah, sure.

Scott Emmons:
We’ve had a lot of speculation in the x-risk community about issues like deception. So people have been worried about what happens if your AIs try to deceive you. And at the same time, I think for a while that’s been a theoretical, a philosophical concern. And I use “speculation” here in a positive way. I think people have done really awesome speculation about how the future of AI is going to play out, and what those risks are going to be. And deception has emerged as one of the key things that people are worried about.

I think at the same time, we’re seeing AI systems actually deployed, and we’re seeing a growing interest of people in what exactly do these risks look like, and how do they play out in current-day systems? So the goal of this paper is to say: how might deception play out with actual systems that we have deployed today? And reinforcement learning from human fee...
[New Feature] Your Subscribed Feed
by Ruby — 2024-06-11T22:45:00.000Z

tl;dr

LessWrong now has a Subscribed tab (next to the Latest tab and Enriched tab[1]). You can now "follow" users, which means their posts and comments will show up in your Subscribed tab[2].

You can subscribe to users that are suggested to you, or search for other users using the search bar. You can also find subscribe options on user profile pages and user hover popup-tooltip.

We've put a lot of thought into how to display the right amount of recent content from people you follow, plus the right amount of surrounding context, to keep you up to date without it being overwhelming. See here for more detail.

In this example, I'm subscribed to both jessicata and dxu. Comments by other users are de-emphasized, and comment trees over a certain size are collapsed by default.

How to follow people

You can follow users via multiple methods:

Using the widget on the Subscribed tab:

2. You can follow people from their user profile:

3. You can follow people using the user tooltip that comes up when you hover on their username.

To follow, click the Follow button in top right

Note!

Following people for your subscribed tab is different from subscribing to get notifications. Signing up for one does not cause the other!

However, to help people start using the Subscribed tab, we did a one time operation to cause you to be following (for purposes of the subscribed tab) anyone who you'd already subscribed to for post and comment notifications. We assume if you want notifications, you'd also want to follow.

What's shown to me in my Subscribed feed?

Short description

We display the recent posts and comments of people you follow, plus comments from other users that people you follow are replying to.

Long description

(Subject to change, lasted update 2024-06-10)

We load posts and comments from people you follow from the last 30 days
We group posts and comments to the post level

We might show a post because someone you followed published it.
We might show a post because someone you follow is commenting on it, even if you don't follow the author of the post. (This will probably be most of your feed, unless you follow people who write more posts than comments.)

We display the five most recent comments from people you follow, unless those comments were a week or more older than the most recent one (we found this necessary to avoid seeing lots of stale content).
We further display (with

...
Open Thread Summer 2024
by habryka — 2024-06-11T20:57:18.805Z

If it’s worth saying, but not worth its own post, here's a place to put it.

If you are new to LessWrong, here's the place to introduce yourself. Personal stories, anecdotes, or just general comments on how you found us and what you hope to get from the site and community are invited. This is also the place to discuss feature requests and other ideas you have for the site, if you don't want to write a full top-level post.

If you're new to the community, you can start reading the Highlights from the Sequences, a collection of posts about the core ideas of LessWrong.

If you want to explore the community more, I recommend reading the Library, checking recent Curated posts, seeing if there are any meetups in your area, and checking out the Getting Started section of the LessWrong FAQ. If you want to orient to the content on the site, you can also check out the Concepts section.

The Open Thread tag is here. The Open Thread sequence is here.
Can efficiency-adjustable reporting thresholds close a loophole in Biden’s executive order on AI?
by Jemal Young — 2024-06-11T20:56:16.167Z

Epistemic Status: Exploratory

President Biden’s Executive Order on the Safe, Secure, and Trustworthy Development and Use of Artificial Intelligence specifies compute thresholds for training runs and computing clusters that, if exceeded, impose

reporting requirements

. If a training run exceeds 1026 floating point operations or, for a model trained mainly on biological sequence data, 1023 floating point operations, the company has to report information about physical and cybersecurity measures and red-teaming results. Additionally, the “acquisition, development, or possession” of a computing cluster capable of training models using more than 1020 floating point operations must be reported, along with the cluster’s location and computing power. My understanding of the executive order is that these thresholds are a stopgap until Cabinet members have defined “the set of technical conditions for models and computing clusters that would be subject to the reporting requirements.”

While the executive order doesn’t explain the reasoning behind the specific numbers of operations used as thresholds (see

here

for a reference to one possible source), the point is to set baselines for how much compute may be enough to produce potential

dual-use foundation models

that the Biden administration considers high-risk. But the thresholds are static, raising the question of whether the reporting mechanism is resilient to the fast pace of algorithmic efficiency gains. For example, the AI trends research organization Epoch AI found that for language models, the amount of compute needed to reach a set performance level is

halved about every 8 months

. So there seems to be a loophole in the executive order, where AI companies could leverage better algorithms to train models using less compute than the thresholds, yet achieve capabilities that would conventionally require exceeding the thresholds. This would circumvent the reporting requirements.

If there’s a loophole that needs closing, maybe reporting thresholds should be efficiency-adjustable based on the number of floating point operations needed to reach certain capability levels. Here’s how that might work:

Establish baseline conventional algorithms and hardware configurations.
Estimate the floating point operations needed to achieve specific capability levels using these baselines.
Set floating point operation thres

...
"Full Automation" is a Slippery Metric
by ozziegooen — 2024-06-11T19:56:49.855Z

Research Status: Written & researched quickly. I think the key point is fairly simple and obvious. I relied on Claude to help with rewriting.

There's been a growing interest in predicting when various products or jobs will be "fully automated." Will we soon have popular movies, books, or even CEOs that are entirely AI-generated?

Some very quickly-found links:
- https://manifold.markets/ChaseStevens/will-an-aigenerated-paper-be-accept
- https://manifold.markets/GabeGarboden/will-aigenerated-art-win-a-major-tr
- https://manifold.markets/probajoelistic/-by-2026-will-any-fully-aigenerated
- https://www.forbes.com/sites/sherzododilov/2024/01/11/can-ai-become-your-next-ceo/

It's an intriguing question, but I believe its definition is slippier than some realize. Here's why.

First, the idea of "full automation" on complex tasks (movies, books, CEO duties) is somewhat of a false dichotomy. In practice, automation exists on a spectrum, with diminishing returns at the extreme. Consider self-driving cars. Even the most advanced autonomous vehicles still have remote human monitors who can intervene if necessary. This human involvement might be significant at first (one intervention for every few miles, in the cases of Tesla and Cruise) but will likely diminish over time as the technology improves. However, even if there's just 1 person left overseeing all vehicles, we technically wouldn't reach "full automation."

This leads to what I call the Hilda[1] Scenario. If we ask, "When will movies be 100% automated?" we're effectively asking, "When will Hilda, the very last human involved in the moviemaking process, be let go?" Perhaps Hilda has a unique talent for crafting prompts that yield remarkable AI-generated special effects, or a keen eye for making subtle but impactful script adjustments. If the cost of retaining Hilda is minimal, her involvement could persist even in an otherwise automated workflow. As long as Hilda remain cost-effective to have in the loop somewhere, it's not fully automated.

The question of a "fully automated CEO" also highlights the limitations of this framing. Even if the vast majority of a CEO's responsibilities could be automated, there might still be significant value in having a human in the role. It's one question to ask, "When are ~99.99% of current CEO duties able to be automated?" It's an entirely different question to ask, "When will ...
AI takeoff and nuclear war
by owencb — 2024-06-11T19:36:24.710Z
Link: https://strangecities.substack.com/p/ai-takeoff-and-nuclear-war

Summary

As we approach and pass through an AI takeoff period, the risk of nuclear war (or other all-out global conflict) will increase.

An AI takeoff would involve the automation of scientific and technological research. This would lead to much faster technological progress, including military technologies. In such a rapidly changing world, some of the circumstances which underpin the current peaceful equilibrium will dissolve or change. There are then two risks[1]:

Fundamental instability. New circumstances could give a situation where there is no peaceful equilibrium it is in everyone’s interests to maintain.

e.g. —

If nuclear calculus changes to make second strike capabilities infeasible
If one party is racing ahead with technological progress and will soon trivially outmatch the rest of the world, without any way to credibly commit not to completely disempower them after it has done so

Failure to navigate. Despite the existence of new peaceful equilibria, decision-makers might fail to reach one.

e.g. —

If decision-makers misunderstand the strategic position, they may hold out for a more favourable outcome they (incorrectly) believe is fair
If the only peaceful equilibria are convoluted and unprecedented, leaders may not be able to identify or build trust in them in a timely fashion
Individual leaders might choose a path of war that would be good for them personally as they solidify power with AI; or nations might hold strongly to values like sovereignty that could make cooperation much harder

Of these two risks, it is likely simpler to work to reduce the risk of failure to navigate. The three straightforward strategies here are research & dissemination, to ensure that the basic strategic situation is common knowledge among decision-makers, spreading positive-sum frames, and crafting and getting buy-in to meaningful commitments about sharing the power from AI, to reduce incentives for anyone to initiate war.

Additionally, powerful AI tools could change the landscape in ways that reduce either or both of these risks. A fourth strategy, therefore, is to differentially accelerate risk-reducing applications of AI. These could include:

Tools to help decision-makers make sense of the changing world and make wise choices;
Tools to facilitate otherwise impossible agreements via mutually trusted artificial judges;
Tools for better democratic accountability

...
What do people think about the polymarket Eth Etf resolution?
by edge_retainer — 2024-06-11T18:34:40.884Z

Did they make the correct decision?

How did they handle it outside of the actual side they chose?

What are the implications for polymarket and prediction markets writ large?

What does this say about UMA?

Links:

https://polymarket.com/event/ethereum-etf-approved-by-may-31/ethereum-etf-approved-by-may-31

https://twitter.com/Domahhhh/status/1794002918888091836

https://manifold.markets/Gen/will-the-polymarket-ethereum-etf-by

https://manifold.markets/JamesBills/when-will-a-spot-ethereum-etf-be-ap
Let's Design A School, Part 3.1: 
Bringing it all together with the Sieve Model
by Sable — 2024-06-11T17:03:12.999Z
Link: https://affablyevil.substack.com/p/lets-design-a-school-part-31

In part 1, we laid out the social services model of a school.

In part 2, we described a new educational model of a school.

In part 3, we’re going to combine them.

Different Schools, Different Problems

The hardest part of designing a public school is that you’re trying to create a one-size-fits-all solution to an array of vastly different problems.

In wealthier districts, where parents are more involved, discipline is less of an issue. More importantly, things like making sure children get fed is less of an issue. The school can focus on education.

In poorer districts, the school often can’t do a whole lot of educating, because the students aren’t in a state conducive to learning. Their parents might be uninvolved for any number of reasons; their home lives might be fine or they might be abusive. Nothing can be taken for granted about the health or well-being of any student who walks in the door.

Urban districts will face different issues from rural ones.

Things like weather and climate can play a role - some schools may need to close multiple times a year for snow days; others might need heavy air conditioning to be habitable.

In all cases, students with physical and mental conditions need to be accommodated. Parental participation, while appreciated, cannot be counted on. Transportation must be provided.

And hopefully some education should take place.

Making It Through The Sieve

In part 1, we introduced the Sieve Model of school-as-social-services. Students enter the building, and are filtered into different paths depending on their physical and mental well-being, climbing Maslow’s hierarchy as they go. If they need to eat, they eat; if they need professional help, they get it.

With school-as-social-services, students who make it through the sieves are mostly left to their own devices. They’re still supervised by adults, but they’re free to play or eat or be at a computer about as much as they want. The children were safe and fed; our goals didn’t exceed the basics.

Now, with a model of school-as-education to work with, we have a curriculum for students who are doing okay. First they’re taught literacy and numeracy, then their core civilizational requirements and survey courses, and lastly core adulting requirements and self-study. By the time a student’s graduated, we should have a reasonable confidence that they’ve been either a) given the tools to make their own way i...
How to eliminate cut?
by jessicata — 2024-06-11T15:54:54.110Z
Link: https://unstableontology.com/2024/06/11/how-to-eliminate-cut/

The purpose of this post isn't to convince you that cut elimination is important. See, for example, the nLab article. Rather, the purpose of this post is to (semi-formally) prove cut elimination in a way that I at least find easy to understand. I have consulted existing sources (such as these lecture notes), but have found it hard to fill in all the details, given the sparsity of the usual presentations. I'll build on the previous post (on Gödel's Completeness Theorem) and show cut elimination in the first-order sequent calculus defined in that post. Recall that the cut rule states:

Γ ⊢ Δ,P Γ,P ⊢ ΔΓ ⊢ Δ

We can think of Γ as the assumptions, Δ as the conclusion, and P as a lemma. Intuitively, this states that, if it's possible to prove the conclusion or the lemma from the assumptions, and it's possible to prove the conclusion from the assumptions and the lemma, then it's possible to prove the conclusions from the assumptions. Cut-elimination is, therefore, the automated elimination of lemmas in a sequent proof. (As stated in the previous post, this presentation of the cut rule is somewhat nonstandard, but it can be shown equivalent to the standard form using weakening and contraction.)

Throughout the post, I will use the notion of the depth of a sentence, and the cut rank of a proof. The depth of a sentence is the depth of nesting of compound sentences; in particular, the depth of an atomic sentence is 0, the depth of a negation is one plus the depth of its inner sentence, the depth of a conjunction is one plus the maximum depth of the inner sentences, and the depth of a universal is one plus the depth of the inner sentence. The cut rank of a proof is a mapping f:N→N, where f(i) is the number of times a cut is performed on a sentence of depth i; note that this is zero almost everywhere. We compare cut ranks lexicographically, with later entries counting more than previous ones.

Constant substitution

As a preliminary, we will show that constants can be substituted with terms in proofs without changing the proof structure (in particular, cut rank stays the same). As notation, if P is a term, sentence, or set of sentences, let P[t/c] indicate replacing the constant c with the term t anywhere in P. Suppose we have a proof of the judgment Γ ⊢ Δ. We wish to show that there is a proof of Γ[t/c]&...