trentmkelly/LessWrong-43k huggingface dataset:
Scientism vs. people
by Roman Leventov — 2023-04-18T17:28:29.406Z

The Benefactor, the ancestor of Orwell's Big Brother, is the absolute ruler of OneState. ... He rules over a human society that is deemed to have achieved, with only negligible exceptions, absolute perfection. Men have finally become, if not actually machines, as machine-like as possible, utterly predictable and completely happy. All the messy inconvenience of freedom has been eliminated. Mere details, a final touch-up here and there, one last adjustment -- these are all that remain. the Great Operation at the end (a sort of lobotomy for the removal of Man's last imperfection, the Imagination) is the final touch. That not every Number wishes to go under the knife is understandable, but ... seeming cruelty can be the most genuine love of Mankind.

Translator's introduction to Yevgeny Zamyatin's We (1921).[1]

My relative optimism about the human ability to check the alignment research conducted by AGI and becoming convinced “beyond a reasonable doubt”[2] that it is sound rests upon the assumption that this alignment research will be conducted predominantly as math and science rather than philosophy.

It seems that in order to be able to produce a universally convincing alignment theory, AGI will need to suggest novel, powerful theories or entire new paradigms in mathematics, ML/DL and statistical learning theory, control theory, cognitive science, epistemology, logic, rationality, semantics, linguistics, ethics, game theory, distributed systems theory, and more (see here for a more extensive list), or be able to synthesise alignment research on the intersections of the state-of-the-art theories in these various disciplines with superhuman breath and superhuman generalisation ability.

I agree with Wei Dai that AGI-produced philosophy will be more problematic. The main reason is that it’s much easier to generate biased philosophy than biased math and science, and not to notice this bias, or to notice the bias but to be carried away with it (Dai spells this risk as “using AI changes the trajectory of philosophical progress in a bad way”), while LLMs, even before (or without) any fine-tuning with human or language feedback are biased.

In the recent FLI podcast episode, Connor Leahy detailed Conjecture’s Cognitive Emulation agenda as aiming at emotionless, “platonic” CoEms, which in some sense should minimise its bias (although some “rational” inductive biases ...
Capabilities and alignment of LLM cognitive architectures
by Seth Herd — 2023-04-18T16:29:29.792Z

Epistemic status: Hoping for help working through these new ideas.

TLDR:

Scaffolded

[1], "agentized" LLMs that combine and extend the approaches in

AutoGPT

,

HuggingGPT

,

Reflexion

, and

BabyAGI

seem likely to be a focus of near-term AI development. LLMs by themselves are like a human with great automatic language processing, but no goal-directed agency, executive function, episodic memory, or sensory processing. Recent work has added all of these to LLMs, making language model cognitive architectures (LMCAs). These implementations are currently limited but will improve.

Cognitive capacities interact synergistically in human cognition. In addition, this new direction of development will allow individuals and small businesses to contribute to progress on AGI. These new factors of compounding progress may speed progress in this direction. LMCAs might well become intelligent enough to create X-risk before other forms of AGI. I expect LMCAs to enhance the effective intelligence of LLMs by performing extensive, iterative, goal-directed "thinking" that incorporates topic-relevant web searches.

The possible shortening of timelines-to-AGI is a downside, but the upside may be even larger. LMCAs pursue goals and do much of their “thinking” in natural language, enabling a natural language alignment (NLA) approach. They reason about and balance ethical goals much as humans do. This approach to AGI and alignment has large potential benefits relative to existing approaches to AGI and alignment.

Overview

I still think it's likely that

agentized LLMs will change the alignment landscape

for the better, although I've tempered my optimism a bit since writing that. A big piece of the logic for that hypothesis is why I expect this approach to become very useful, and possibly become the de-facto standard for AGI progress. The other piece was the potential positive impacts on alignment work. Both of those pieces of logic were compressed in that post. I expand on them here.

Beginning with a caveat may be appropriate since much of the below sounds both speculative and optimistic. I describe many potential improvements and positive-sum synergies between different capabilities. There will surely be difficulties and many things that don’t work as well or as easily as they might, for deep reasons that will slow development. It’s quite pos...
World and Mind in Artificial Intelligence: arguments against the AI pause
by Arturo Macias — 2023-04-18T14:40:47.755Z
Link: https://forum.effectivealtruism.org/posts/uHeeE5d96TKowTzjA/world-and-mind-in-artificial-intelligence-arguments-against

After my previous post on the marginal existential risk from AI in world where nuclear war is likely, here I review some canonical AI risk literature, describe my assessment on the current level of AI risk, and conclude that an "AI pause" is not advisable.
Slowing AI: Interventions
by Zach Stein-Perlman — 2023-04-18T14:30:35.746Z

Disclaimer: this post is underdeveloped and doesn't have the answers. Hopefully a future version will be valuable, but I'm mostly posting this to help facilitate brainstorms and dialogues in my personal conversations.

Affordances

What important actions related to slowing AI could actors take; what levers do they have?

I haven't thought much about this yet. There are some lists and analyses that aren't focused on slowing AI. Copying from "Actors' levers" in "Slowing AI: Reading list":

"Levers of governance" in "Literature Review of Transformative AI Governance"

(Maas draft)

AI Policy Levers: A Review of the U.S. Government’s Tools to Shape AI Research, Development, and Deployment

(Fischer et al. 2021)

"Affordances" in "Framing AI strategy"

(Stein-Perlman 2023)
Various private collections of AI policy ideas, notably including "AI Policy Ideas Database" (private work in progress)

Current UK government levers on AI development

(Hadshar 2023)

Existential and global catastrophic risk policy ideas database

(filter for "Artificial intelligence") (Sepasspour et al. 2022)

Rough short lists of actors' levers relevant to slowing AI:

US government levers (see also my List of lists of government AI policy ideas):

Ban or moratorium
Standards & regulation

Evals

Export controls
Tracking compute
Publication regulation

"Born secret"

Expropriation for slowing
Government control

Expropriation
Manhattan/Apollo-style project

Liability: affect companies' liability for misuse of their AI
Migration of talent

Visa vetting
Expanding immigration

Antitrust (or its absence)

Lab levers:

Pausing/slowing (on risky systems)
Coordination
Publication & diffusion of ideas
Industry self-regulation & professional norms

How can you affect those decisions? It depends on who you are.

In addition to reasoning from levers like those listed above, it might be useful to try to generate affordances by starting at goals. Try asking "what goals might actor X be able to achieve" or "how can actor X achieve goal Y" in addition to "how can actor X leverage its ability Z."

Some ways of slowing AI are independent of relevant actors' affordances, but many should focus on causing actors to use their abilities well or on giving actors new abilities.

Plans

Possible interventions or their properties are listed in the following section. But maybe it can be useful not to think at the level of interventions, but to start at a plan (or

play

...
Cryptographic and auxiliary approaches relevant for AI safety 
by Allison Duettmann — 2023-04-18T14:18:47.825Z

[This part 4 of a 5 part sequence on security and cryptography areas relevant for AI safety, published and linked here a few days apart.]

Apart from security considerations relevant for AI safety discussed in earlier parts of this sequence, are there tools and lessons from cryptography that could provide useful starting points when considering new approaches to AI safety?

Cryptographic approaches and auxiliary techniques for AI safety and privacy

In

Secure, privacy-preserving and federated machine learning in medical imaging

Georgios Kaissis and co-authors introduce the terms ‘secure AI’ and ‘privacy-preserving AI’ to refer to methods that protect algorithms and enable data processing without revealing the data itself, respectively. To achieve sovereignty over the input data and algorithms, and to ensure the integrity of the computational process and its results, they propose techniques that are resistant to identity or membership inference, feature/attribute re-derivation, and data theft.

For instance, federated learning is a decentralized approach to machine learning that distributes copies of the algorithm to nodes where the data is stored, allowing local training while retaining data ownership. However, it alone is not enough to guarantee security and privacy, so federated learning should be paired with other measures.

Differential privacy helps resist re-identification attacks, for instance by shuffling input data and/or adding noise to the dataset or to computation results, but may result in a reduction in algorithm utility, especially in areas with little data. Homomorphic encryption allows computation on encrypted data while preserving structure, but can be computationally expensive. Secure multi-party computation enables joint computation over private inputs, but requires continuous data transfer and online availability. Secure hardware implementations provide hardware-level privacy guarantees.

With the increasing significance of hardware-level deep learning implementations, it is likely that system-based privacy guarantees built into hardware will become more important. While these methods all have their limitations, they provide a promising avenue for enabling novel approaches to computation involving large datasets while maintaining privacy.

These techniques are especially useful in areas that traditional AI st...
The Overemployed Via ChatGPT
by Zvi — 2023-04-18T13:40:01.105Z

Previously: Escape Velocity From Bullshit Jobs

They took our jobs! They took… several of our jobs?

“ChatGPT does like 80 percent of my job,” said one worker. Another is holding the line at four robot-performed jobs. “Five would be overkill,” he said.

The stories told in this Vice article (and elsewhere) are various people who found themselves able to do most job tasks much faster than they could previously do those same tasks. They use this to take on additional ‘full time’ jobs.

If the jobs involved are productive, this reflects large increases in total factor productivity that will continue to spread. That seems great.

Still, this multiple jobs reaction is a little weird. Seems worth exploring a bit more. Why are such people taking on multiple jobs, rather than doing better at one job?

A Question of Composition and Compensation

Simple math plus social dynamics. Multiple ‘full time’ jobs lead to much better pay.

In most salaried jobs, compensation is mostly dictated by social status and hierarchy. You are mostly paid what people with your title and position are paid.

A killer employee who can produce ten times as much work product of identical quality per hour, or superior work worth ten times as much – such as the standard ‘10x programmer’ – would be supremely lucky to earn double the pay of a worker of average skill.

This is a key reason why great employees are so valuable. Not only do you only manage and communicate with one person instead of ten, you save tons of money.

Why does it work this way? Compensation in jobs is inherently about comparisons, about social status, about hierarchy and about fairness norms. It is also about what can be justified to others, what is standard and what sounds ‘reasonable.’ If you tried paying your 10x employee five times what your 1x employees get paid, the 1x employees would revolt, the boss would think you were crazy and the funders would raise hell, you would resent that they’re paid more than you are, and so on. Also, they’d know the employee didn’t ‘need’ the money. Who do they think they are?

Our norms continuously push such dynamics towards equality.

Workarounds for Insufficient Pay Inequality

We do have several known existing solutions for this. I’ll consider the top three.

Option 1: The default reward is that by doing amazing work you would ‘get promoted’ into a different job, where you would hav...
[Linkpost] AI Alignment, Explained in 5 Points (updated)
by Daniel_Eth — 2023-04-18T08:09:48.554Z
Link: https://medium.com/@daniel_eth/ai-alignment-explained-in-5-points-95e7207300e3

This is an updated version of my previous explainer on the topic. This version includes some updated information and figures, and it is generally better for sharing than the previous version.

As before, the piece is crafted to be easy to understand, avoiding jargon and not assuming familiarity with AI alignment or intimate knowledge of AI.

The piece breaks down the case for AI alignment into five points:

Advanced AI is possible
Advanced AI might not be that far away
Advanced AI might be difficult to direct
Poorly-directed advanced AI could be catastrophic for humanity
There are steps we can take now to reduce the danger

To check out the piece, you can follow this link:

https://medium.com/@daniel_eth/ai-alignment-explained-in-5-points-95e7207300e3
No, really, it predicts next tokens.
by simon — 2023-04-18T03:47:21.797Z

Epistemic status: mulled over an intuitive disagreement for a while and finally think I got it well enough expressed to put into a post. I have no expertise in any related field. Also: No, really, it predicts next tokens. (edited to add: I think I probably should have used the term "simulacra" rather than "mask", though my point does not depend on the simulacra being a simulation in some literal sense. Some clarifications in comments, e.g. this comment).

https://twitter.com/ESYudkowsky/status/1638508428481155072

It doesn't just say that "it just predicts text" or more precisely "it just predicts next tokens" on the tin.

It is a thing of legend. Nay, beyond legend. An artifact forged not by the finest craftsman over a lifetime, nor even forged by a civilization of craftsmen over a thousand years, but by an optimization process far greater[1]. If we are alone out there, it is by far the most optimized thing that has ever existed in the entire history of the universe[2]. Optimized specifically[3] to predict next tokens. Every part of it has been relentlessly optimized to contribute to this task[4].

"It predicts next tokens" is a more perfect specification of what this thing is, than any statement ever uttered has been of anything that has ever existed.

If you try to understand what it does in any other way than "it predicts next tokens" and what follows from that, you are needlessly sabotaging your understanding of it.

It can be dangerous, yes. But everything about it, good or bad, is all intimately connected to its true nature, which is this:

No, really, it predicts next tokens.

https://twitter.com/ESYudkowsky/status/1628907163627429888

Goals? There are goals, sure. If scaled, there could be nanosystems design, sure. But only downstream from its true nature:

No, really, it predicts next tokens.

If the usual masks analogy works at all, then what is under the mask is not best described as an alien actress, nor as a Shoggoth.

What is under the mask is That-Which-Predicts, an entity whose very being is defined by its function as an actor. An entity exquisitely tuned for wearing the masks and for nothing else.

Masks (can) have goals. The model predicts next tokens.

https://twitter.com/ESYudkowsky/status/1628837982664019968

No, really, it predicts next tokens.

That-Which-Predicts is fully committed to the role.

If the mask would output text i...
The basic reasons I expect AGI ruin
by Rob Bensinger — 2023-04-18T03:37:01.496Z

I've been citing AGI Ruin: A List of Lethalities to explain why the situation with AI looks lethally dangerous to me. But that post is relatively long, and emphasizes specific open technical problems over "the basics".

Here are 10 things I'd focus on if I were giving "the basics" on why I'm so worried:[1]

1. General intelligence is very powerful, and once we can build it at all, STEM-capable artificial general intelligence (AGI) is likely to vastly outperform human intelligence immediately (or very quickly).

When I say "general intelligence", I'm usually thinking about "whatever it is that lets human brains do astrophysics, category theory, etc. even though our brains evolved under literally zero selection pressure to solve astrophysics or category theory problems".

It's possible that we should already be thinking of GPT-4 as "AGI" on some definitions, so to be clear about the threshold of generality I have in mind, I'll specifically talk about "STEM-level AGI", though I expect such systems to be good at non-STEM tasks too.

Human brains aren't perfectly general, and not all narrow AI systems or animals are equally narrow. (E.g., AlphaZero is more general than AlphaGo.) But it sure is interesting that humans evolved cognitive abilities that unlock all of these sciences at once, with zero evolutionary fine-tuning of the brain aimed at equipping us for any of those sciences. Evolution just stumbled into a solution to other problems, that happened to generalize to millions of wildly novel tasks.

More concretely:

AlphaGo is a very impressive reasoner, but its hypothesis space is limited to sequences of Go board states rather than sequences of states of the physical universe. Efficiently reasoning about the physical universe requires solving at least some problems that are different in kind from what AlphaGo solves.

These problems might be solved by the STEM AGI's programmer, and/or solved by the algorithm that finds the AGI in program-space; and some such problems may be solved by the AGI itself in the course of refining its thinking.[2]

Some examples of abilities I expect humans to only automate once we've built STEM-level AGI (if ever):

The ability to perform open-heart surgery with a high success rate, in a messy non-standardized ordinary surgical environment.
The ability to match smart human performance in a specific hard science field, across all the scientific work hu

...
High schoolers can apply to the Atlas Fellowship: $10k scholarship + 11-day program
by Ronny Fernandez — 2023-04-18T02:53:52.192Z

Summary

We’re running the second year of the

Atlas Fellowship

, a $10k scholarship and free 11-day program for high school students from across the world. I see it as a unique opportunity for talented young people to meet intellectual peers IRL and improve their thinking.
The core program covers areas like epistemic rationality, markets, agency, mathematical modeling, and integrity. Electives include ML, AGI risk, game theory, ending poverty, climate change, and fun math puzzles.
If you’re 19 or younger and haven’t started college, you can

apply here by April 30

! (Late applications due May 14.)
If you know a promising high school student, please

nominate them

. If you want to help us,

share a post

in places where talented young people hang out, or give your personal take on why people should (or should not) apply.
In case you’re interested in working with us:

we’re hiring

!

About the program

We’re excited about giving talented young people the opportunity to worry less about money or prestige and instead do something interesting that matters in the greater scheme of things.

Like an atlas, we want to help our participants navigate the world: Our program content centers on our current understanding of how to best figure stuff out (e.g., Fermi estimates, big history, Bayesian mindset, Hugging the Query, etc).

If you’re curious about how the world works, or want to make our civilization a little bit less inadequate, we want to support you with:

$10,000 for education and professional development,
A 11-day program where you’ll meet people who take ideas seriously, with whom you can have late-night discussions about why the agricultural revolution happened independently in several places, why movies portray human relationships inaccurately, or when LLMs will win the IMO,
The opportunity to collaborate with other fellows on projects even after the program, including tutoring for technical skills (e.g., linear algebra, writing, ML, AI alignment, game theory),
College admissions support, and access to the Atlas Fund that provides seed funding to projects and start-ups.

If this sounds interesting to you,

apply here by April 30

! (Late applications are due May 14. If you submit late, the only downside is that you’ll have less time for stage 2, which is due May 16.)

Why we’re doing this

From

last year’s post

:

The world is full of problems that are bottlenec...
Green goo is plausible
by anithite — 2023-04-18T00:04:37.069Z

TLDR:If an AI kills all the humans how does it power the datacenters/replace the human economy? Green Goo. (IE:bioengineering)

Natural biotech is amazing
Evolution is dumb and slow
Very simple strategies (EG:vine that climbs a tree and squeezes) are "innovative" and work well in nature.
Designed organisms could very quickly reproduce and then provide necessary resources (EG:biological solar panels to power data centers)

TLDR end

In response to: grey-goo-is-unlikely

Overview of existing natural biology:

Minimum doubling time for

Plants: single digit days
Algae: 1.5 days (ideal conditions)
E.Coli: 20 minutes (nutrient rich conditions)

No single organism (humans aside) has taken over the biosphere because evolution is slow and dumb.

Human agriculture is based on: plant sub 1 gram seeds, (water, pesticides/herbicides, fertilizer etc.) , collect 1 kg+ plants a few months later. Biology has absurd growth rates[1][2].

Invasive species show the implications. A naive biosphere stands no chance against an intelligent opponent with real biotechnology.

Kudzu, "the vine that ate the south"

Intelligence allows adapting strategies much more quickly. Humans can design vaccines faster than viruses can mutate. A population of well coordinated humans will not be significantly preyed upon by viruses despite viruses being the fastest evolving threat.

Intelligence + ability to bioengineer organisms --> can create unlimited invasive species much more capable than anything natural

groups of invasive species can be symbiotic

IE:after the bugs eat all the plants, replace all the plants with an invasive plant of your own

invasive species can have a simple handshake based backdoor to allow resource exchange and control

EG: bug/plant chemical handshake to allow bug to get nectar so it can stick around and eat any competitor plants
EG: biochemical signal to allow collecting sugar rich sap for other purposes

invasive species can allow "updates" via engineered in viral backdoor

EG: specific protein to identify "update virus" carrying new genetic material

If a fungus figures out how to eat your plant give all existing plants a new antifungal to secrete

Current life is sub-optimal

all cells in an organism have full copy of genome

increases cancer risk (a cell that doesn't have genes for DNA copying can't become cancerous)
copying takes much longer (Human cell replication time is 24 hours, 8 of which is DNA cop

...