trentmkelly/LessWrong-43k huggingface dataset:
Surviving and Shaping Long-Term Competitions: Lessons from Net Assessment
by Gentzel — 2023-11-24T18:18:41.072Z

This post examines

net assessment

, a framework for evaluating strategic competition that evolved to inform U.S. defense policy during the Cold War. We explain what net assessment is, its methods and principles, and how some of its tools can be applied to reason about highly uncertain, long-term tech competitions with potentially existential stakes.

What is net assessment?

In the late 1950s, the Cold War slid into an especially dangerous period. Nuclear stockpiles swelled and delivery capabilities advanced, while a decades-long buildup by the Soviet Union challenged the United States’ conventional military dominance. Defense analysts needed to reframe the way they looked at military competition: the U.S. could not overpower the Soviet war machine with brute force, and the prospect of nuclear war both elevated the stakes of conflict and created a need for new metrics and principles of strategy for engaging in limited competition. It was in this context that the framework of “net assessment” began to develop.

Andrew Marshall, who founded and then directed the DoD’s Office of Net Assessment for forty-two years, described net assessment as follows:

"Our notion of a net assessment is that it is a careful comparison of U.S. weapon systems, forces, and policies in relation to those of other countries. It is comprehensive, including description of the forces, operational doctrines and practices, training regime, logistics, known or conjectured effectiveness in various environments, design practices and their effect on equipment costs, performance, and procurement practices and their influence on cost and lead times. The use of net assessment is intended to be diagnostic. It will highlight efficiency and inefficiency in the way we and others do things, and areas of comparative advantage with respect to our rivals."

Generalizing from its original military context, the core idea of net assessment is to create comprehensive (hence the “net” in the name), objective, and informative comparisons between competitors: be they states, companies, or other political or ideological coalitions. Insights drawn from these comparisons can inform more sustainable strategies and help build beneficial competitive dynamics instead of disastrous ones on tech and policy subjects of high importance.

Such analyses tend to zero in on crucial considerations. In a particularly salien...
Ability to solve long-horizon tasks correlates with wanting things in the behaviorist sense
by So8res — 2023-11-24T17:37:43.020Z

Status: Vague, sorry. The point seems almost tautological to me, and yet also seems like the correct answer to the people going around saying “LLMs turned out to be not very want-y, when are the people who expected 'agents' going to update?”, so, here we are.

Okay, so you know how AI today isn't great at certain... let's say "long-horizon" tasks? Like novel large-scale engineering projects, or writing a long book series with lots of foreshadowing?

(Modulo the fact that it can play chess pretty well, which is longer-horizon than some things; this distinction is quantitative rather than qualitative and it’s being eroded, etc.)

And you know how the AI doesn't seem to have all that much "want"- or "desire"-like behavior?

(Modulo, e.g., the fact that it can play chess pretty well, which indicates a certain type of want-like behavior in the behaviorist sense. An AI's ability to win no matter how you move is the same as its ability to reliably steer the game-board into states where you're check-mated, as though it had an internal check-mating “goal” it were trying to achieve. This is again a quantitative gap that’s being eroded.)

Well, I claim that these are more-or-less the same fact. It's no surprise that the AI falls down on various long-horizon tasks and that it doesn't seem all that well-modeled as having "wants/desires"; these are two sides of the same coin.

Relatedly: to imagine the AI starting to succeed at those long-horizon tasks without imagining it starting to have more wants/desires (in the "behaviorist sense" expanded upon below) is, I claim, to imagine a contradiction—or at least an extreme surprise. Because the way to achieve long-horizon targets in a large, unobserved, surprising world that keeps throwing wrenches into one's plans, is probably to become a robust generalist wrench-remover that keeps stubbornly reorienting towards some particular target no matter what wrench reality throws into its plans.

This observable "it keeps reorienting towards some target no matter what obstacle reality throws in its way" behavior is what I mean when I describe an AI as having wants/desires "in the behaviorist sense".

I make no claim about the AI's internal states and whether those bear any resemblance to the internal state of a human consumed by the feeling of desire. To paraphrase something Eliezer Yudkowsky said somewhere: we wouldn't say that a blender...
The Limitations of GPT-4
by p.b. — 2023-11-24T15:30:30.933Z

Amidst the rumours about a new breakthrough at OpenAI I thought I'd better publish this draft before it gets completely overtaken by reality. It is essentially a collection of "gaps" between GPT4 and the human mind. Unfortunately the rumours around Q* force me to change the conclusion from "very short timelines seem unlikely" to "who the f**k knows".

While GPT4 has a superhuman breadth of knowledge, writing speed and short-term memory, compared to the human mind GPT4 has a number of important limitations.

Some of these will be overcome in the near future because they depend on engineering and training data choices. Others seem more fundamental to me, because they are due to the model architecture and the training setup.

These fundamental limitations are the reason why I do not expect scaling GPT further to lead to AGI. In fact, I interpret further scaling of the exact current paradigm as evidence that overcoming these limitations is hard.

I expect scaled up GPT4 to exhibit the same strength and weaknesses and for the improved strengths to paper over the old weaknesses at most in a superficial fashion.

I also expect with further scaling the tasks GPT is unable to do to increasingly load on the fundamental limitations and therefore diminishing returns.

This list is not exhaustive and there are likely ways to frame even the limitations I identify in a more insightful or fruitful way. For example, I am not sure how to interpret GPT4s curious inability to understand humor. I hope further limitations will be mentioned in the comments.

Integration of the Senses

GPT4 cannot really hear, and it cannot really talk.

Voice input is transcribed into text by a separate model, 'Whisper,' and then fed to GPT4. The output is read by another model. This process loses the nuances of input—pronunciation, emphasis, emotion, accent, etc. Likewise, the output cannot be modulated by GPT4 in terms of speed, rhythm, melody, singing, emphasis, accent, or onomatopoeia.

In fact, due to tokenization it would be fair to say that GPT4 also cannot really read. All information about char-level structure that is relevant to spelling, rhyming, pronunciation must be inferred during training.

The vision component of most open multi-modal models is typically likewise "grafted on." That is, it's partially trained separately and then connected and fine-tuned with the large language mod...
Progress links digest, 2023-11-24: Bottlenecks of aging, Starship launches, and much more
by jasoncrawford — 2023-11-24T15:25:07.721Z
Link: https://rootsofprogress.org/links-digest-2023-11-24

I swear I will get back to doing these weekly so they’re not so damn long. As always, feel free to skim and skip around!

The Progress Forum

A paradox at the heart of American bureaucracy: “The quickest way to doom a project to be over-budget and long-delayed is to make it an urgent public priority”
Why Governments Can’t be Trusted to Protect the Long-run Future: “No one in the long-run future gets to vote in the next election. No one in government today will gain anything if they make the world better 50 years from now or lose anything if they make it worse”
What if we split the US into city-states? “In The Republic, when his entourage asks the ideal size of a state, Socrates replies, ‘I would allow the state to increase so far as is consistent with unity; that, I think, is the proper limit’”
The Art of Medical Progress: “These two paintings offer a hopeful contrast. Whereas we begin with pain and suffering, we move to hope and progress. The surgeon stands apart as a hero, a symbol of the triumphant conquering of nature by humanity”

More from Roots of Progress fellows

Bottlenecks of Aging, a “philanthropic menu” of initiatives that “could meaningfully accelerate the advancement of aging science and other life-extending technologies.” Fellows Alex Telford and Raiany Romanni both contributed to this (via @jamesfickel)
Drought is a policy choice: “California has surrendered to drought, presupposing that with climate change water shortages are inevitable. In response, the state fallows millions of farmland each year. But this is ignorant of California’s history of taming arid lands”
Geoengineering Now! “Solar geoengineering can offset every degree of anthropogenic temperature rise for single-digit billions of dollars” (by @MTabarrok)
A conversation with Richard Bruns on indoor air quality (and some very feasible ways to improve it) (@finmoorhouse)
To Become a World-Class Chipmaker, the United States Might Need Help (NYT) covers a recent immigration proposal co-authored by (@cojobrien). Also, thread from @cojobrien of “what I’ve written through this program and some of my favorite pieces from other ROP colleagues”

Opportunities

Job opportunities

Forest Neurotech is hiring, “one of the coolest projects in the world” says @elidourado
“Know someone who loves to scale and automate workflows in the lab? We want to apply new tools to onboard a diverse array of species in the lab!”&nb

...
What's the evidence that LLMs will scale up efficiently beyond GPT4? i.e. couldn't GPT5, etc., be very inefficient?
by M. Y. Zuo — 2023-11-24T15:22:01.624Z

A lot of this recent talk about OpenAI, various events, their future path, etc., seems to make an assumption that further scaling beyond GPT4 will pose some sort of 'danger' that scales up linearly or super-linearly with the amount of compute. And thus pose an extraordinary danger if you plug in 100x and so on.

Which doesn't seem obvious at all.

It seems quite possible that GPT5, and further improvements, will be very inefficient.

For example, a GPT5 that requires 10x more compute but is only moderately better. A GPT6 that requires 10x more compute than GPT5, i.e. 100x more than GPT4, but is again only moderately better.

In this case there doesn't seem to be any serious dangers at all for LLMs.

The problem self-extinguishes based on the fact that random people won't be able to acquire that amount of compute in the for-seeable future. Only serious governments, institutions, and companies, with multi-billion dollar cap-ex budgets will even be able to consider acquiring something much better.

And although such organizations can't be considered to be perfectly responsible, they will still very likely be responsible enough to handle LLMs that are only a few times more intelligent.
Sapience, understanding, and "AGI"
by Seth Herd — 2023-11-24T15:13:04.391Z

Epistemic status: I'm sure that "AGI" has become importantly confusing. I think it's leaving out a critical set of capabilities. I think those are closely related, so acquiring them could create a jump in capabilities. I'm not sure of the best term to disambiguate the type of AGI that's most dangerous, but I want to propose one that works decently

The term AGI has become muddled. It is now used for AI both with and without agency, contextual awareness, and the capacity to actively learn new facts and concepts. "Sapience” means understanding, wisdom and self-awareness, so it could be adopted as a disambiguating term. Understanding as humans perform it is an active process for testing and improving our knowledge. Understanding allows self-awareness and contextual awareness. It also implies agency, because our process of understanding is goal-directed.

I have three goals here. One is to point out how the term "AGI" is confusing x-risk discussions. The second is to discuss how human-like undestanding is poweful, achievable, and dangerous. Last and least I propose the specific term "sapience" for the set of powerful and dangerous capabilities provided by active understanding.

Understanding as an active process of testing and improving “fit” among concepts and world-models

Sapience implies agency and understanding

The concept of agency is not just a philosophical nicety; it's pivotal in discussions about existential risk (x-risk) related to AI. Without a clear understanding and explicit treatment of agency, these discussions have become confused and potentially misleading. The need for a new term is evident, given widely varying definitions of AGI, and resulting disagreements about risks and capabilities.

"Sapience" appears to be our most fitting option. The term is used in various ways with weak connections to its etymology of wisdom or discernment, but its most common usages are the ones we need. In the realm of science fiction, it's often employed to denote an intelligent, self-aware species distinct from humans. We, as “Homo Sapiens”, pride ourselves as the "understanding apes," setting ourselves apart from our evolutionary kin. This self-ascription may spring from vanity, but it invokes a critical cognitive capacity we commonly refer to as "understanding."

Human understanding

The debate surrounding whether large language models (LL...
Insulate your ideas
by Logan Kieller — 2023-11-24T14:08:12.193Z
Link: https://logankieller.substack.com/p/insulate-your-ideas

Once tested and proven, insulate good ideas from reality.

Running

I recently spoke with a very successful marathon runner who commonly wins races and consistently beats his own times. I asked for his tips, and he shared two:

“The first thing I do is run faster on the uphills. Everyone else slows down because of how difficult it is, and this is a great time to gain ground. The second thing I do is run faster on the downhills. Everyone else slows down because of how fast you go anyways on the slope down, and this is a great time to gain ground.”

I was looking for actual tips and this person (not even concisely) told me it’s just about running faster.

Before passing judgment too quickly and writing off this feedback, I paused. Amusingly, the point was not to run faster, but more so, that the uphill and downhill should have no impact on how you think about the run.

You know you need to run fast, consistently, for the whole marathon.

This is a central note that cannot possibly be wrong; it is a “good idea,” and it is correct. It’s not overly useful as a tactic, but it is the rejection of the notion that the hill changes anything at all.

Regardless of these external hill stimuli, the ground truth is identical: run as fast as you can.

Business

I listened to a shareholder meeting for a tech company in the heat of the tech recession of 2023. The CFO was being questioned by the shareholders about his plan going into unstable economic conditions.

We heard questions about imminent layoffs, changes in strategy, refocusing on core business, and the list continues. The CFO (who I’m paraphrasing here) replied with a terse note:

“We will do as we have always done whether it was through the good times or the bad. We will continue with the disciplined deployment of capital, allocating resources to where it is most needed and most likely to generate substantive returns.”

Similar to the runner, I felt indignant at the fact that no real insight was shared. I was wrong again for the exact same reason.

In the natural cycles of the market, businesses and executives busy themselves with the minutia of reacting to market conditions, changing plans, adjusting strategies, and levying layoffs. In the most recent boom through the pandemic, tech companies saw extreme growth and over-hired.

When things slowed and shareholders questioned productivity, drastic changes were made yet again. Large-...
A Question For People Who Believe In God
by yanni kyriacos — 2023-11-24T05:22:40.839Z

I was very surprised to watch this video. In it Andrew Huberman talks about how he prays "because it works" and that "he believes in God" because he can't imagine how else life on earth could be where it is now.

Anyway, it got me thinking, when someone says they "believe in God" does this mean something like "I assign a ≥ 50% probability to there being an omnipotent omnipresent and omniscient intelligence?"

What does it mean in this context to "believe in" something?
First and Last Questions for GPT-5*
by Mitchell_Porter — 2023-11-24T05:03:04.371Z

Suppose that in the very near future, a research group finds that their conversational AI has begun to produce extremely high-quality answers to questions. There's no obvious limit to its ability, but there's also no guarantee of correctness or good intention, given the opacity of how it works.

What questions should they ask it?

And what questions should they not ask it?
4. A Moral Case for Evolved-Sapience-Chauvinism 
by RogerDearnaley — 2023-11-24T04:56:53.231Z

Part 4 of AI, Alignment, and Ethics. This will probably make more sense if you start with Part 1.

TL;DR In Parts 1 and 2 I argued that aligned AIs will not want and should not get assigned moral worth or the sorts of rights that human citizens get. In Part 3 I suggested that uploads should (with some specifics and rather strong constraints). Next I'd like to suggest a guiding principle for the largest group who should get these things.

A society can use any criteria it likes for membership/citizenship. The human moral intuition for fairness only really extends to "members of the same primate troupe as me". Modern high-tech societies have economic and stability incentives to extend this to every human in the entire global economic system, and the obvious limiting case for that is the entire planet, or once we go interplanetary, the entire solar system.

However, there is a concern that obviously arbitrary rules for membership of the society might not be stable under challenges like self-reflection or self-improvement by an advanced AI. One might like to think that if someone attempted to instill in all the AIs the rule that the criterion for being a citizen with rights was, say, descent from William Rockefeller Sr. (and hadn't actually installed this as part of a terminal goal, just injected it into the AI's human value learning process with a high prior), sooner or later a sufficiently smart AI might tell them "that's extremely convenient for your ruling dynasty, but doesn't fit the rest of human values, or history, or jurisprudence, or biology. So no."

So it would be nice to have a criterion that makes some logical sense. Not necessarily a "True Name" of citizenship, but at least a solid rationally-defensible position with as little wiggle room as possible.

I'd like to propose what I think is one: "an intelligent agent should be assigned moral worth only if it is (or primarily is, or is a functionally-equivalent very-high-accuracy emulation of) a member of sapient species whose drives were produced by natural selection. (This moral worth may vary if its drives or capabilities have been significantly modified from their evolved versions, details TBD.)"

The political argument defending this is as follows:

Living organisms have homeostasis mechanism: they seek to maintain aspects of their bodies and environment in certain states, even when (as is often the case) those...
Detecting What's Been Seen
by jefftk — 2023-11-24T03:30:06.691Z

Sometimes it makes sense for sites want to treat things differently
based on whether the user has seen them. For example, I like sites
that

highlight new comments
(ex: EA Forum and LessWrong) and I'd like them even better if comments
didn't lose their "highlighted" status in cases where I hadn't
scrolled them into view. In writing my Mastodon client,

Shrubgrazer, I wanted a version
of this so it would show me posts that I hadn't seen before. The
implementation is a bit fussy, so I wanted to write up a bit on what
approach I took.

The code is on
github, and it counts posts as viewed if both the top of the post
and bottom have been on screen for at least half a second.
Specifically, whenever the top or bottom of a post enters the viewport
it sets a 500ms timer, and if when the timer fires it's still within
the viewport it keeps a record client side. If this now means that
both the top and bottom have met the criteria it sends a beacon back
so the server can track the entry as viewed.

Go back 4-7 years and this would have required a scroll listener,
using a ton of CPU, but modern browsers
now support the IntersectionObserver
API. This lets us get callbacks whenever an entry enters or leaves
the viewport.

I start by creating an IntersectionObserver:

const observer =
new IntersectionObserver(
handle_intersect, {
root: null,
threshold: [0, 1],
});

We haven't told it which elements to observe yet, but once we do it will call
the handle_intersect callback anytime those elements
fully enter or fully exit the viewport.

Each entry has an element at the very top and very bottom, and to
start tracking the entry we tell our observer about them:

observer.observe(entry_top);
observer.observe(entry_bottom);

What does our handle_intersect callback do? We maintain
two sets of element IDs, onscreen_top and
onscreen_bottom for keeping track of what is currently
onscreen. The callback keeps those sets current, and also starts the
500ms timer:

function handle_intersect(entries, observer) {
for (let entry of entries) {
const target = entry.target;
const id = target.getAttribute("id");
const is_bottom =
target.classList.contains("bottom");
const onscreen_set =
is_bottom ? onscreen_bottom : onscreen_top;

if (entry.intersectionRatio > 0.99) {
onscreen_set.add(id);

...
Help to find a blog I don't remember the name of
by  — 2023-11-23T22:49:30.599Z

There was this blog of a person that I think was part of this community or was referenced by people here, it had one article in which it discussed the idea of general intelligence as a game that is played "to keep playing" about finding meaning on random pieces of information

and several posts about fictional stories with weird settings, like I believe Ballard's The Concentration City or other works of his. There probably were some references to Borges and the artist Escher also.

Does anyone know the name?