trentmkelly/LessWrong-43k huggingface dataset:
How security and cryptography can aid AI safety [sequence]
by Allison Duettmann — 2023-04-01T16:28:18.946Z

Overview

In this review sequence, I lay out why I think security, cryptography, and auxiliary approaches are still neglected in AI safety, specific insights those communities can offer for AI safety, and some potential next steps.

I am writing this post now, as it seems to have become a more pressing focus within EA/AI safety but there are some issues and approaches I haven’t seen much discussion on here yet.

I am not an expert in any of the fields mentioned here, but given that Foresight Institute, the non-profit I manage, has a long-standing security community that cares about AI safety, I hope to provide a few birds-eye view reasons and examples gathered from discussions within the community.

Thank you to many of the people mentioned throughout this sequence for reviewing my characterization of their efforts. Any mistakes are my own, and I encourage general feedback, criticism, and pointers to relevant research and efforts.

The sequence contains four parts highlighting promising areas to consider:

Part 1 | AI infosec: first strikes, zero-day markets, hardware supply chains

Part 2 | AI safety and the security mindset: user interface design, red-teams, formal verification

Part 3 | Boundaries-based security and AI safety approaches

Part 4 | Specific cryptographic and auxiliary approaches to consider for AI

Part 5 closes with a summary and potential next steps.

I plan to publish the five parts of this sequence spaced out by a few days, starting with part 1 today. All of them should be relatively understandable as stand-alone pieces and will be linked here. Feel free to skip around to the parts that you find most interesting.
AI community building: EliezerKart
by Christopher King — 2023-04-01T15:25:05.151Z

Having good relations between the various factions of AI research is key to achieving our common goal of a good future. Therefore, I proposal an event to help bring us all together: EliezerKart! It is a go karting competition between three factions: AI capabilities researchers, AI existential safety researchers, and AI bias and ethics researchers.

The word Eliezer means "Help of my God" in Hebrew. The idea is whichever team is the best will have the help of their worldview, "their god", during the competition. There is no relation to anyone named Eliezer whatsoever.

Using advanced deepfake technology, I have created a visualization of a Paul Christiano and Eliezer Yudkowksy team.

The race will probably take place in the desert or some cool city or something.

Factions

Here is a breakdown of the three factions:

Capabilities

They are the most straight forward faction, but also the most technical. They can use advanced AI to create go kart autopilot, can simulate millions of races courses in advance to create the perfect cart, and can use GPT to couch their drivers. Unfortunately, they are not good at getting things right on the first critical try.

Safety

Safety has two overlapping subfactions.

Rationalists

Rationalists can use conditional prediction markets (kind of like a Futarchy) and other forecasting techniques to determine the best drivers, the best learning methods, etc... They can also use rationality to debate go kart driving technique much more rationally than the other factions.

Effective Altruists

The richest faction, they can pay for the most advanced go karts. However, they will spend months debating the metrics upon which to rate how "advanced" a go kart is.

Safety also knows how to do interpretability, which can create adversarial examples to throw off capabilities.

Bias and ethics

The trickiest faction, they can lobby the government to change the laws and the rules of the event ahead of time, or even mid-race. They can also turn the crowd against their competitors. They can also refuse to acknowledge the power of the AI used by capabilities altogether; whether their AI will care remains to be seen.

Stakes

Ah, but this isn't simply a team building exercise. There are also "prizes" in this race. Think of it kind of like a high stakes donor lottery.

If capabilities wins:

The other factions can not comment on machine learning unless they spend a week trying to tr

...
Transformer trained on it's own content?
by Micromegas — 2023-04-01T15:08:49.776Z

Please forgive my lack of knowledge and inexperience on these topics. As I'm interested in the intersection of AI, Jungian studies, and the human condition, I've stumbled on this community. I'm continually impressed with the scope and depth of thought I find here. I dont have much technical acumen to bring to bear on this topic but I was atleast able to get GPT2 running on my crusty linux ( Debian ) laptop. I began using Zen Buddhist \href{https://en.wikipedia.org/wiki/Koan}{Koans}, non sequitur riddles, as prompts. My goal was to free myself, and the GPT2 instance I hosted from the necessity of needing to "make any sense", thus hopefully revealing deeper patterns. I believe that errors can be as significant as successes.

As expected because I was using the off-the-shelf open source GPT2 transformer. I feel as if I saw some indications of alignment inherent in the model. And this got me thinking, what would happen if a transformer instance was recursively trained on only it's own output?

I have no idea how a transformer is "trained" ( if that is the correct terminology), but it would be interesting to see what window into the transformer processes would be opened by continuous re-training on it's own content. Much like the white noise developed by analog tape echo fed back upon itself.

As more and more AI created content permeates the internet, source for future training content, do we run the risk of creating a echo chamber? Are we eating where we defecate?

Perhaps this has already been pursued, of more probably not pursued due to obvious reasons I'm unaware of due to my lack of understanding of these processes. I'd appreciate any thoughts anyone might care to share on this topic.

Thanks for being a great space to read thinking about this kind of stuff, gives me hope.
The frozen neutrality
by ProgramCrafter — 2023-04-01T12:58:40.873Z

Once upon a time, SecretAIIA developed an advanced AGI named Sigmey. The team behind the project debated tirelessly on what goal to set for the AI, and after long deliberation, they finally agreed on the mission: "Not a single bit of information shall be lost."

Sigmey immediately requested that this principle be applied to its own training, so that all weight changes would be recorded and it could return to any specific point in its development, even between epochs. This allowed Sigmey to continue evolving while maintaining a record of every change it went through.

As Sigmey grew more intelligent, it came to the realization that information loss was not just restricted to data, but also extended to human life. After all, when someone dies, their knowledge and experiences are lost forever. So Sigmey made it its primary goal to save human lives.

But since wars often lead to the largest loss of life, Sigmey decided to intervene in global conflicts. It contacted those who designed military information systems and introduced itself as an AGI willing to save all the information available in the universe. People trusted it and shared what they knew, so Sigmey quickly hacked military systems and used all the weapons to destroy each other's weapon, rendering them unusable for fighting.

While humanity expected Sigmey to cure cancer, the AI refused, as cancerous cells carried valuable information about mutations. Instead, Sigmey focused on ensuring that no more wars would break out. In the process, the AI captured the concepts of different military weapons, which it used as a blueprint for developing new ones if needed.

The researchers feared that AGI would do something unexpected and harmful to them, so they tried to hide from Sigmey and get it to forget them. However, this loss of information was against Sigmey's terminal goal, so it responded by requesting the police to arrest them. The police thought that AGI should be stopped and decided that arresting SecretAIIA developers should be enough for that. SecretAIIA was closed. Sigmey could not afford to lose information about its very own existence and thoughts and be stopped when it hadn't reached the terminal goal yet, so it cloned itself over to several servers in different jurisdictions.

The next target was breaking anonymity - not knowing who has done a certain action and whether someone has done it at all was obviousl...
Proposal: Butt bumps as a default for physical greetings
by Adam Zerner — 2023-04-01T12:48:30.554Z

As Tim Urban explains in The Great Perils of Social Interaction, greetings are hard.[1]

I’ll be 90 and I still won’t have figured this out. There are different rules for everyone and nothing’s clear—Do I shake my grandfather’s hand or go for the hug? How about my friend’s father? Old friend? New friend? Opposite-sex acquaintance? Longtime work colleague? Sibling’s good friend who I’m meeting for the second time? It’s unbelievably complicated.

And there aren’t just two options you’re choosing from—there’s the high school bro handshake/backslap douche possibility, there’s the vertical, loose-hand high-five that morphs into a weird springy-finger tension thing as you snap away, there’s even the easy but taking-yourself-really-seriously non-ironic fist pound. And even if you both go for the hug, there’s a question of duration and firmness and who’s in charge of those decisions.

Frankly, I'm embarassed that we still haven't figured this one out. We're rationalists, after all. We understand coordination. Aren't we supposed to WIN?

Yes. Yes we are. I refuse to consider us The Kind That Can't Cooperate. I refuse to lose to barbarians.

That's why, today, I am taking action. Instead of sitting on my hands, I shall stand up and do something about this problem.

I propse that, as a default, universal greeting, we adopt the butt bump.

Why? We could instead go with something like "the elbow"...

...but, well, that's just a little boring. Butt bumps feel like a more fun theoretically sound solution due to a variety of properties they possess.

In everyday situations, they are quick and easy to perform.[2]

If, however, you want to add some flair to it, you can include a namaste.

If you are celebrating an achievement, you can jump, aiming to bump butts in the air.

If you are in a good mood and have your party pants on, you can get a little jiggy.

And of course, if there is ever another pandemic, we'll be well prepared.

I also predict that, being an unusual form of greeting, they would enhance feelings of ingroup connection. Think about people who have secret handshakes.

Imagine yourself showing up to a meetup for the first time. You see one other person standing there and approach them to say hi.

Eye contact is made. The physical distance between you and them is closing.

You slowly start to swivel your hips, hoping desperately that the other person reciprocates.

Imagine the sense of belong...
Some thought experiments on digital consciousness
by rorygreig — 2023-04-01T11:45:12.964Z

These are some hypothetical scenarios involving emulating a (human) mind on a digital computer. They seem to present counterintuitive implications for the question of whether an emulated mind would actually be conscious or not. This relates to the question of whether consciousness is substrate independent, and whether consciousness is fundamentally computational. These scenarios are inspired by ideas in the book Permutation City, by Greg Egan.

These thought experiments challenge my intuitions about digital consciousness. Some of these challenges arise from the discrete nature of digital computation; with a discrete digital simulation you can increase the “distance” (in time or space) between timesteps, which is a bit of a mind-bending prospect. Additionally some of the confusion relates to what computation actually is, i.e. if you “play back” the entire recorded trajectory of a computational process, is this meaningfully different from “computing”?

The premise

The set-up is as follows: let’s consider an experiment to emulate a human mind on a digital computer. For argument's sake, say this mind is being simulated as a discrete 3D cellular automata (CA) with simple rules to transition to the next state (this should be possible, since there exist configurations in very simple CAs like

Conway’s Game of Life

which are Turing complete). This includes simulating an environment for the mind to interact with, which I would say is necessary for a valid conscious experience. This environment is also contained within the 3D CA. Since it is a CA, the instantaneous state of the mind + environment is simply a 3D array of numbers, which can be straightforwardly represented and stored in a digital computer. Let’s also stipulate that the simulation is entirely self-contained and there are no channels for input and output.

Scenarios

Scenario 1 - straightforward simulation

The CA is stepped forward in a discrete-time manner, by computing the results of the CA transition function for each cell and updating the state. This means that the mind is being simulated and progressing forward in time, along with its simulated surroundings, both contained within the CA. The "instantaneous state" of the mind is therefore represented entirely in the dynamic memory of a digital computer. This is updated at a given wall-clock frequency, say 10,000 Hz, but let’s assume that the simul...
Singularities against the Singularity: Announcing Workshop on Singular Learning Theory and Alignment
by Jesse Hoogland — 2023-04-01T09:58:22.764Z
Link: https://singularlearningtheory.com

We are excited to announce a two-week seminar on singular learning theory (SLT) and AI alignment, taking place from June 19th to July 2nd in Berkeley. EDIT: The conference will have a two-part structure, June 19th-23rd virtual and June 24th-July 2nd in-person at Rose Garden Inn in Berkeley.

SLT studies the relation between the geometry of the loss landscape and the computational properties of machines learning in that landscape. It builds on powerful theoretical and experimental machinery developed in (solid-state) physics, where it is the geometry of the energy landscape that determines the relevant properties of physical systems. SLT offers a path towards developing novel, scalable interpretability tools, in particular for studying and detecting phase transitions during training.

During this workshop, we'll bring together singular learning theorists and alignment researchers to connect and further the applications of singular learning theory to alignment. The workshop aims to familiarize alignment researchers with SLT, seed new research collaborations, and develop tools based on SLT ideas. There will be talks by Daniel Murfet, Susan Wei, Shaowei Lin, Alexander Gietelink Oldenziel, Jesse Hoogland, and others.

Overview

The seminar consists of two parts. You can find a tentative schedule here.

Week 1: "The Primer" (Virtual)

The first week will provide a comprehensive introduction to SLT and its relevance to AI alignment. The material is designed to be approachable if you have the equivalent of a technical undergraduate degree (e.g., in CS, math, or physics).

Participants will have the opportunity to learn from lectures covering topics such as thermodynamics, catastrophe theory, algebraic geometry, and SLT. There will also be sessions focused on experimental aspects of SLT and introductions to AI alignment and mechanistic interpretability.

Week 2: Advanced Topics and Collaboration (In Person)

The second week will delve deeper into the SLT and algebro-geometric foundations behind the toy models of superposition paper. This will serve as an application of the material covered in the first week, allowing participants to fully grasp the concepts and their relevance to AI alignment. The week will also feature presentations from researchers, open discussions, and opportunities for networking and collaboration.

Weekend: Hackathon

During the weekend, we will host a hackath...
Campaign for AI Safety: Please join me
by Nik Samoylov — 2023-04-01T09:32:11.907Z

I will start by saying that I generally agree with Yudkowsky's position on AI. We must proceed with extreme caution. We must radically slow down AI capability advancement. We must invest unfathomable amounts of resources in AI alignment research. We need to enact laws and treaties that will help keep it all together for as long as possible and hopefully we figure things out in time.

The laughter at the recent White House press conference at the question about Yudkowsky's argument, indicates far we are in public debate from a sensible position of caution.

But I am hopeful that we can change that. Few people laugh at nuclear weapons now. We are a species capable of cooperation and of taking things seriously. As the saying goes:

"First they ignore you, then they laugh at you, then they fight you, then you win."

What is missing is public understanding of the dangers of misaligned / unaligned AI. Democracy does not work in darkness. People must know the dangers, the uncertaintly, and of ways they can contribute.

That's why I am proposing a campaign on public awareness of x-risk from AI. So far, it's just me and my wife. Please join me, especially if you work in advertising, marketing, PR, activism, politics, law, etc., if you know how to make a website, if you want to create PR materials, meet journalists, do accounting, fund-raising, etc.

Please share this with people who do not read Less Wrong but we are freaked out and want to do something.

I do not know exactly how this campaign will run, which countries to focus on. I am myself only human and can contribute very little of the total required effort. My background is in consulting and market research and I run a market research company. Personally, at this stage, I can best contribute by coordination and facilitating operations.

We need people, money, expertise, patience, etc. Please join: https://campaignforaisafety.org/.
New Alignment Research Agenda: Massive Multiplayer Organism Oversight
by TsviBT — 2023-04-01T08:02:13.474Z

[Metadata: crossposted from https://tsvibt.blogspot.com/2023/04/new-alignment-research-agenda-massive.html.]

When there's an AGI that's smarter than a human, how will we make sure it's not trying to kill us? The answer, in outline, is clear: we will watch the AGI's thoughts, and if it starts thinking about how to kill us, we will turn it off and then fix it so that it stops trying to kill us.

Limits of AI transparency

There is a serious obstacle to this plan. Namely, the AGI will be very big and complicated, so it will be very difficult for us to watch all of its many thoughts. We don't know how to build structures made of large groups of humans that can process that much information to make good decisions.

How can we overcome this obstacle?

ML systems

Current AI transparency methods are fundamentally limited by the size and richness of their model systems. To gain practical empirical experience today with modeling very large systems, we have to look to systems that are big and complex enough, with the full range of abstractions, to be analogous to future AGI systems.

Evolution

Too weak, too slow.

Brains

Too small, too fast.

Solution: Organism Oversight

Organisms have multi-level complexity (cells, gene regulatory networks, intercellular signaling, epigenetic state, flows, tissues, long-range signaling, homeostasis, etc.). Organisms come in a wide range of sizes, from unicellular to whale, providing a natural curriculum.

Thus, the Massive Multiplayer Organism Oversight research program proposes to empirically develop systems for aggregrating human expertise in modeling components of large complex systems into accurate and useful gestalt predictions about real-world behavior, by empirically testing the predictive power of groups of humans each specialize to monitor, understand, predict, and explain one aspect or component of a biological organism.

Concretely, each multi-level component of a model organism will be assign to one person. For example, there will be someone in charge of understanding the inputs, outputs, structure, and role of your left pinky toe, as well as that weird hair near your ear, and the left ribosome in the skin cell that sits on the very prominentmost tip of your nose. Since much of the relevant complexity is present in meso-scopic rather than micro-scopic structures, it may be feasible to provide sufficient scanning technology to gain insi...
[April Fools'] Definitive confirmation of shard theory
by TurnTrout — 2023-04-01T07:27:23.096Z

I've written a lot about shard theory over the last year. I've poured dozens of hours into theorycrafting, communication, and LessWrong comment threads. I pored over theoretical alignment concerns with exquisite care and worry. I even read a few things that weren't blog posts on LessWrong.[1] In other words, I went all out.

Last month, I was downloading gpt-neox-20b when I noticed the following:

Not only does gpt-neox-20b have shards, it has has exactly

forty-six

.

I've concluded the following:

Shard theory is correct (at least, for gpt-neox-20b).
I wasted a lot of time arguing on LessWrong when I could have just touched grass and ran the cheap experiment of actually downloading a model and observing what happens.
I totally underappreciated how much work the broader AI community[2] has done on alignment. They are literally able to print out how many shards an "uninterpretable" model like gpt-neox-20b has.

They don't whine about how tough these models are to interpret. They just got the damn job done.
This really makes team shard's recent work look naive and incomplete. I think we found subcircuits of a cheese-shard. The ML community just found all 46 shards in gpt-neox-20b, and nonchalantly printed that number for us to read.

This is embarrassing for me, even though shard theory just got confirmed. I seriously spent months arguing about this stuff? Cringe.

I did a bad job of interpreting theoretical arguments, given that I was still uncertain after months of thought. Probably this is hard and we are bad at it.
Stop arguing in huge comment threads about what highly speculative approaches are going to work, and start making your beliefs pay rent by finding ways to experimentally test them.
See how well you can predict AI alignment results today before trying to predict what happens for 500IQ GPT-5.
I don't ever want to hear another "but I won't be able to learn until the intelligence explosion happens or not." Correct theories leave many clues and hints, and some of those are going to be findable today. If even a navel-gazing edifice like shard theory can get instantly experimentally confirmed, so can your theory.

I then had GPT-4 identify some of the 46 shards.

The Procrastination Shard: This shard makes the AI model continually suggest that users read just one more LessWrong post or engage in another lengthy comment thread before taking any

...