trentmkelly/LessWrong-43k huggingface dataset:
Sufficiently many Godzillas as an alignment strategy
by 142857 — 2022-08-28T00:08:02.666Z

Assuming that alignment by default happens with nontrivial probability, one way to produce human-aligned AGIs would be to simultaneously create sufficiently many (different) AGI. This leads to a multipolar scenario where there are a few AGI aligned with humans, and many unaligned AGI. (By unaligned I mean unaligned with humans, these AGI may or may not be aligned with some other goal.)

Although it is true that in general having many AGIs is not good, the idea is if some of those AGIs are aligned, this may be a better outcome than having just 1 AGI that is probably not aligned. Or perhaps, once there are enough aligned AGI, since they are all working in the same direction, they may be able to overcome the unaligned AGI (which are all working towards different goals that are likely not directly opposed to those of the aligned AGI). So it is useful to explore how the total number of AGI in this scenario impacts the chances of survival for humanity. To make this more concrete, I pose the following question (but feel free to discuss the idea in general):

Assuming that all of the AGIs have roughly the same cognitive power, rank the following scenarios from best to worst outcome for humanity.
A) 1 AGI, with a 10% of being aligned
B) 1 aligned AGI and 9 unaligned AGI
C) 10 aligned AGI and 90 unaligned AGI
D) 1000 aligned AGI and 9000 unaligned AGI
What would you expect a massive multimodal online federated learner to be capable of?
by Aryeh Englander — 2022-08-27T17:31:07.153Z

One intuitive picture I have for what very rapid ML capability gains might look like is a massive multimodal deep learning model that uses some form of online federated learning to continually learn from many devices simultaneously, and which is deployed to hundreds of millions or billions of users. For example, imagine a multimodal Google or Facebook chatbot with say 10 trillion parameters that could interact with billions of users simultaneously and which improved its weights from every interaction. My impression is that we basically have the tech for this today, or we will in the very near future. Now add in some RL to actively optimize some company-relevant goals (ad revenue, reported user satisfaction, etc.) and, intuitively at least, that seems to me like it's very close to the kind of scary AGI we've been talking about. But that seems like it could easily be 2-5 years in the future rather than 10-50.

Did I misunderstand something? How close to scary-type AGI would you expect this kind of model to become within a few months after deployment?
Basin broadness depends on the size and number of orthogonal features
by CallumMcDougall — 2022-08-27T17:29:32.508Z

TL;DR

For neural networks trained to perfect loss, the broadness of optima in parameter space is given by the number and norm of independent orthogonal features the neural network has. The inner product that defines this "feature norm" and independence/orthogonality is the L2 product of Hilbert space.

Introduction - why do we care about broadness?

Recently, there's been some discussion of what determines the broadness of optima of neural networks in parameter space.

People care about this because the size of the optimum basin may influence how easy it is for gradient descent (or other local optimisation algorithms which you might use to train a neural network) to find the corresponding solution, since it's probably easier to stumble across a broader optimum's attractor.

People also care about this because there's some hypotheses floating around saying that the broadness of optima is correlated with their generality somehow. The broader the basin of your solution is in parameter space, the more likely it seems to be that the network will successfully generalise to new data in deployment. So understanding what kinds of circuit configurations are broad seems like it might help us grasp better how AIs learn things, and how an AGI might really work internally.

Meaning, understanding what kind of solutions are broad is probably relevant to start developing general predictive theories of AI and AI training dynamics, which we'd like to have so we can answer questions like "What goals is my AGI going to develop, given that it has architecture A, and I'm training it with loss function L on data set D?"

Measuring broadness

Warning: calculus and linear algebra incoming!

Say we have a loss function L(f,x,y) for a neural network f(x,Θ), with some data points x and labels y(x), and we've trained the network to perfect loss, setting the parameters to some set ^Θ.

L=∑xl(f(x,Θ),y(x))dL(^Θ)dΘ=0 [1]

Now, lets try to look at how much the loss changes when we make small changes to the parameters ^Θ. To do this, we're going to perform a polynomial expansion of L up to quadratic order. This isn't going to describe the function behaviour perfectly, but within a small neighbourhood of ^Θ, it'll be mostly accurate.

L(Θ)−L(^Θ)=dL(^Θ)dΘ(Θ−^Θ)+(Θ−^Θ)Td2L(^Θ)d2Θ(Θ−^Θ)+O((Θ−^Θ)3)=(Θ−^Θ)Td2L(^Θ)d2Θ(Θ−^...
Informal semantics and Orders
by Q Home — 2022-08-27T04:17:09.327Z

(Edited the title and intro.)

My idea will sound too vague and unclear at first. But I think the context will make it more clear what I mean. (Clear as the mathematical concept of a graph, for example: a graph is a very abstract idea, but makes sense and easy to use.) I'm aware that my post can be seen as hand-waving and bullshit.

Please evaluate my post at least as science fiction and then ask: maybe it's not fiction and just reality? Or "OK, the examples are weird and silly, but what if everything really works like that?"

Key points of this post:

You can "solve" human concepts (including values) by solving semantics. By semantics I mean "meaning construction", something more abstract than language.
Semantics is easier to solve than you think. And we're closer to solving it than you think.
Semantics is easier to model than you think. You don't even need an AI to start doing it. Just a special type of statistics. You don't even have to start with analyzing language.
I believe ideas from this post can be applied outside of AI field. For example, the ideas about hypotheses have consequences for Rationality.

Why do I believe this? Because of this idea:

Every concept (or even random mishmash of ideas) has multiple versions. Those versions have internal relationships, positions in some space relative to each other. You can understand a concept by understanding those internal relationships.
One problem though, those relationships are "infinitely complex". However, there's a special way to make drastic simplifications. We can study the real relationships through those special simplifications.
What do those "special simplifications" do? They order versions of a concept (e.g. "version 1, version 2, version 3"). They can do this in extremely arbitrary ways. The important thing is that you can merge arbitrary orders into less arbitrary structures. There's some rule for it, akin to the Bayes Rule or Occam's razor. This is what cognition is, according to my theory.

If this is true, we need to find any domain where concepts and their simplifications are easy enough to formalize. Then we need to figure out a model, figure out the rule for merging simplifications. I've got a specific suggestion and a couple of ideas and many examples.

You may compare my theory to memetics, if memetics were making much stronger and more universal claims, for example: "Every single thought is a "mem...
Help Understanding Preferences And Evil
by Netcentrica — 2022-08-27T03:42:01.392Z

I’m having a problem understanding why Stuart Russell thinks that AI learning human preferences is a good idea. I think it’s a bad idea. I assume I am wrong but I assume I don’t understand something. So, help me out here please. I’m not looking for an argument but rather to understand. Let me explain.

I have watched Stuart’s four hour lecture series of Reith lectures on the BBC. Highly recommended. I have watched several other videos that include him as well. I am currently reading his book, Human Compatible. I am not an academic. I am now retired and write science fiction about advanced social robots for a hobby.

Reading the chapter “AI: A Different Approach” in Stuart’s book I am still bothered by something about the preferences issue. My understanding of Stuart’s “new model for AI” is that it would learn what our preferences are from observing our behavior. I understand why he thinks “preferences” is a better word than “values” to describe these behaviors but, at the risk of confusing things, let me use the word values to explain my confusion.

As I understand it, humans have different kinds of values:

1) Those that are evolved and which we all share as a species, like why sugar tastes good or why glossy hair is attractive.
2) Those that reflect our own individuality, which make each of us unique including those some twin studies reveal.
3) Those our culture, family, society or what have you impose on us.

I believe the first two kinds are genetic and the third kind learned. Let me classify the first two as biological values and the third kind as social values. It would appear that the third category accounts for the majority of the recent evolution of our physical brains.

Let’s consider three values for each type just as simple examples. Biological values might be greed, selfishness and competition while social values might be trust, altruism and cooperation. Humans are a blend of all six of these values and will exhibit preferences based on them in different situations. A lot of times they are going to choose behaviors based on biological values as the nightly news makes clear.

If AI learns our preferences base on our behaviors it’s going to learn a lot of “bad” things like lying, stealing and cheating and other much worse things. From a biological point of view, these behaviors are “good” because they maximize the return on calories invested by getting others to ...
Contra Dance Contact Tracing
by jefftk — 2022-08-27T01:50:01.802Z

Many contra dances require everyone to provide their emails for
contact tracing. The idea is, if anyone later tests positive for
covid everyone can be notified after. While I think this is good to
offer, I don't think it should be mandatory.

People have a wide range of ways they're handling covid at this point.
For some, a dance will be a large portion of their weekly covid
exposure, even with everyone wearing high-filtration
masks. After a notification they may want to avoid visiting
vulnerable relatives, get tested, or increase masking. But other
dancers are regularly eating inside restaurants, participating in
large meetings, having crowded indoor singing parties, where learning
that one of a hundred people they shared a room with may have been
infectious isn't something they're going to change behavior over.

I do think dances should continue posting publicly about positive
tests and making it easy to sign up to be notified, but I don't think
they should be requiring people to provide their contact
information. I'm glad BIDA's
made this switch, which I've been pushing for months, and I'd
encourage other dances to rethink this requirement as well.
Annual AGI Benchmarking Event
by Lawrence Phillips — 2022-08-27T00:06:31.972Z
Link: https://www.metaculus.com/questions/12339/annual-agi-benchmarking-event/

Metaculus is strongly considering organizing an annual AGI benchmarking event. Once a year, we’d run a benchmark or suite of benchmarks against the most generally intelligent AI systems available to us at the time, seeking to assess their generality and the overall shape of their capabilities. We would publicize the event widely among the AI research, policy, and forecasting communities.

Why?

We think this might be a good idea for several reasons:

The event could provide a convening ground for the AI research community, helping it to arrive at a shared understanding of the current state of AGI research, and acting as a focal point for rational discussion on the future of AGI.
An annual benchmarking event has advantages over static, run-any-time benchmarks when it comes to testing generality. Unless one constrains the training data and restricts the hard-coded knowledge used by systems under evaluation, developers may directly optimize for a static benchmark while building their systems, which makes static benchmarks less useful as measures of generality. With the annual format, we are free to change the tasks every year without informing developers of what they will be beforehand, thereby assessing what François Chollet terms developer-aware generalization.
Frequent feedback improves performance in almost any domain; this event could provide a target for AGI forecasting that yields yearly feedback, allowing us to iterate on our approaches and hone our understanding of how to forecast the development of AGI.
The capabilities of an AGI will not be completely boundless, so it’s interesting to ask what its strengths and limitations are likely to be. If designed properly, our benchmarks could give us clues as to what the “shape” of AGI capabilities may turn out to be.

How?

We're currently working on a plan, and are soliciting ideas and feedback from the community here. To guide the discussion, here are some properties we think the ideal benchmark should have. It would:

Engage a broad, diverse set of AI researchers, and act as a focal point for rational, forecasting-based discussion on the future of AGI.
Measure the generality and adaptability of intelligent systems, not just their performance on a fixed, known-beforehand set of tasks.
Form the basis for AGI forecasting questions with a one-year lifetime.
Generate predictive signal as to the types of capabilities that an AGI

...
Is there a benefit in low capability AI Alignment research?
by Letti — 2022-08-26T23:51:44.623Z

Basic Problem

If I read about current AI Alignment research ideas, I see a lot of discussion about high capability scenarios, in which AI would be too smart to align simply. It would deceive operators, hack through communication channels and goodheart it's way over to infinity.

However if I block the part of my mind that likes to think about cool superintelligence overlords, and try to understand why high capability should introduce an actual complexity shift... it doesn't seem that intuitive.

At basic level - any time an AI does something we don't want, it's a plain and simple failure of matching one reward function to another. In high capability AI (HAI) we can only imagine what reward functions would be, and how they would be represented. Right now, our simple low capability models (LAI) have reward functions in the shape of big mess of parameters and connections that randomly walked into the first region in which loss is small enough. Are we sure that they are aligned ? I think it's pretty probable that if we take a general consequentialist and bind him to this reward function (whatever it represents) the result would be quite catastrophic. Of course this example is a little contrived, but I think image of separate consequentialist vs. reward function is helpful here.

It seems to me a real alignment failure structure is shared between HAI and LAI. The first just has more wiggle room, so potential misalignments propagate to the level when we can easily notice them.

To be more precise, I think that base level alignment challenge (make actual reward function of a model to be what we want) has some fixed (but high) level of complexity mostly independent of capability.

Let's assume this is somewhat true, then what?

Next steps

Then we should do more low capability research:

It will be more anchored in reality.

We can try to test that our current alignment ideas work on LAI.

It consumes less compute, so it can:

Produce more insights for the same money/time.
Be more relevant to current AI research, which would increase chances of adoption.
We can perform more complicated experiments.

There is some current research directions which are working at LAI level. I think a lot of Interpretability work are in this category.

Also isn't the whole AI research industry working on this problem? AI that works ok and can be commercially deployed is more aligned than one whic...
AI Risk in Terms of Unstable Nuclear Software
by Thane Ruthenis — 2022-08-26T18:49:53.726Z

(This is an entry for the AI Safety Public Materials contest testing out a novel-ish frame on the AI Risk.)

In recent years, there'd been growing concerns among AI specialists regarding the dangers of advanced artificial intelligence. The capabilities of AI models are growing rapidly, they argue, while our ability to control them lags far behind, not to mention governmental regulations, which haven't even begun to catch up. We're on direct course for catastrophe — one that might cost us billions of dollars in economic damage, millions of lives, or even the very survival of our species.

The most striking thing about it, however, is that there's nothing outlandish or science-fictional about the nature of these dangers. When you get down to it, the threats are neither esoteric nor truly novel. They're merely scaled up beyond anything we're familiar with.

It has nothing to do with AIs being smart or "sentient". The core problem is simpler:

AI models are software products. And as software products go, their functionality is revolutionary, while their reliability is abysmal.

A quick primer on how AI models are developed. It has preciously little in common with standard, time-tested methods of software development.

We initialize some virtual architecture — a cluster of neurons designed by loose analogue with biological brains. That architecture has no knowledge recorded in it at first, it's entirely randomized. Then, we set up the training loop. We expose the neural network to some stimuli — pictures of animals, natural-language text, or a simulated environment. The network computes some output in response.

In the first stages, said output is just gibberish. Our software evaluates it, comparing it to the expected, "correct" responses. If it's unsatisfactory, the software slightly modifies the neural network, nudging the connections between its neurons so that it's somewhat more likely to output the correct answer.

Then the NN is exposed to another stimulus, and the process repeats.

It is entirely automatic — the architecture is updated by a blind selection process not unlike biological evolution. That process doesn't "understand" what it's doing — it just computes some minimal "mutations" that would make the NN's performance marginally better.

After tens of thousands of iterations, we end up with a trained AI model consisting of millions of neurons and trillions of connectio...
Taking the parameters which seem to matter and rotating them until they don't
by Garrett Baker — 2022-08-26T18:26:47.667Z

A big bottleneck in interpretability is neural networks are non-local. That is, given the layer setup

if we change a small bit of the original activations, then a large bit of the new activations are affected.

This is an impediment to finding the circuit-structure of networks. It is difficult to figure out how something works when changing one thing affects everything.

The project I'm currently working on aims to fix this issue, without affecting the training dynamics of networks or the function which the network is implementing[1]. The idea is to find a rotation matrix R and insert it with its inverse like below, then group together the rotation with the original activations, and the inverse with the weights and nonlinear function.

We then can optimize the rotation matrix and its inverse so that local changes in the rotated activation matrix have local effects on the outputted activations. This locality is measured by the average sparsity of the jacobian across all the training inputs.

We do this because the jacobian is a representation of how each of the inputs affects each of the outputs. Large entries represent large effects. Small entries represent small effects. So if many entries are zero, this means that fewer inputs have an effect on fewer outputs. I.e. local changes to the input cause local changes to the output.

This should find us a representation of the activations and interpretations of matrix multiplies that "make sense" in the context of the rest of the network.

Another way of thinking about this is that our goal is to find the basis our network is thinking in.

Currently I'm getting this method to work on a simple, 3-layer, fully connected MNIST number classifying network. If this seems to give insight into the mechanics of the network after application, the plan is to adapt it to a more complicated network such as a transformer or resnet.

I only have preliminary results right now, but they are looking promising:

This is the normalized jacobian the middle layer before a rough version of my method:

And here is the normalized jacobian after a rough version of my method (the jacobian's output has been set to a basis which maximizes it's sparsity):

Thanks David Udell for feedback on the post. I did not listen to everything you said, and if I did the post would have been better

^

This seems important if we'd like to use interpretability w

...
ACX Meetups Everywhere List
by Scott Alexander — 2022-08-26T18:12:04.083Z

Here's the Astral Codex Ten worldwide meetups list, crossposted at LW's request in case people here are interested in attending. Some cities have an ACX but not an LW meetup group, or vice versa; others combine their groups. You can find the list below, in the following order:

Africa & Middle East
Asia-Pacific (including Australia)
Canada
Europe (including UK)
Latin America
United States

You can see a map of all the events on the LessWrong community page.

Within each section, it’s alphabetized first by country/state, then by city - so the first entry in Europe is Vienna, Austria. Sorry if this is confusing.

I'll provisionally be attending the meetups in Berkeley, Los Angeles, and San Diego. ACX meetups coordinator Mingyuan will provisionally be attending Paris and London. I’ll be announcing some of the biggest ones on the blog, regardless of whether or not I attend.

Extra Info For Potential Attendees

1. If you’re reading this, you’re invited. Please don’t feel like you “won’t be welcome” just because you’re new to the blog, demographically different from the average reader, or hate ACX and everything it stands for. You’ll be fine!
2. You don’t have to RSVP or contact the organizer to be able to attend (unless the event description says otherwise); RSVPs are mostly to give organizers a better sense of how many people might show up, and let them tell you if there are last-second changes. I’ve also given email addresses for all organizers in case you have a question.

Extra Info For Meetup Organizers:
1. If you’re the host, bring a sign that says “ACX MEETUP” and prop it up somewhere (or otherwise be identifiable).
2. Bring blank labels and pens for nametags.
3. Have people type their name and email address in a spreadsheet or in a Google Form (accessed via a bit.ly link or QR code), so you can start a mailing list to make organizing future meetups easier.
4. If it’s the first meetup, people are probably just going to want to talk, and if you try to organize some kind of “fun” “event” it’ll probably just be annoying.
5. It’s easier to schedule a followup meetup while you’re having the first, compared to trying to do it later on by email.
6. In case people want to get to know each other better outside the meetup, you might want to mention reciprocity.io, the rationalist friend-finder/dating site.
7. If you didn’t make a LessWrong event for your meetup, the LessWrong team d...