trentmkelly/LessWrong-43k huggingface dataset:
Notes on OpenAI’s alignment plan
by Alex Flint — 2022-12-08T19:13:59.439Z

This work was supported by the Monastic Academy for the Preservation of Life on Earth.

OpenAI has published a plan for producing aligned AI systems. Very roughly, their plan is to build language models capable of producing human-level alignment research. The output of these models might be natural language essays about alignment, or code that directly implements machine learning systems. In either case, human researchers would spend their time reviewing machine-generated alignment research. Their research agenda goes as follows.

Step 1: Train language models to solve useful problems using reinforcement learning. Rather than having a human review each attempt the model makes, which would be very time-consuming, train a second model that watches a few human reviews and learns to predict what reward the human would have given. Then train the main reinforcement learning model using this second model to provide a lot of feedback. You still need a human to review this second model every so often to make sure it has learned the right thing, but overall it requires less human effort than having a human review every attempt that the reinforcement learning model makes.

Step 2: When tasks become so complex that humans struggle to assess the outputs of the model, use some combination of the following:

Train models that critique their own outputs or the outputs of other models, possibly requiring the initial model to expand its reasoning around certain points in its original output. ("debate")

Train auxiliary models that help humans work out what reward to give to the primary model by solving more tractable sub-problems. In assessing the validity of a philosophy essay, you might train an auxiliary model to identify which paragraphs contain the key reasoning steps, and then use this model as a tool when assessing the philosophy essays. It might be that the sub-problems are themselves very difficult to provide training data for, in which case you can build models to solve sub-sub-problems that help humans to assess the output of the models targeting the sub-problems, and so on recursively. ("Recursive reward modeling")

Train a model that is general-purpose enough that it can itself do useful work to assist a human generating training data for the model’s own training. Each time you train a new version of the model, give that model to humans to use as a tool in generat

...
Relevant to natural abstractions: Euclidean Symmetry Equivariant Machine Learning -- Overview, Applications, and Open Questions
by the gears to ascension — 2022-12-08T18:01:40.246Z
Link: https://youtu.be/O6eqvb5Sw3o

Talk by Tess E. Smidt (MIT). Abstract:

Atomic systems (molecules, crystals, proteins, etc.) are naturally represented by a set of coordinates in 3D space labeled by atom type. This is a challenging representation to use for machine learning because the coordinates are sensitive to 3D rotations, translations, and inversions (the symmetries of 3D Euclidean space). In this talk I’ll give an overview of Euclidean invariance and equivariance in machine learning for atomic systems. Then, I’ll share some recent applications of these methods on a variety of atomistic modeling tasks (ab initio molecular dynamics, prediction of crystal properties, and scaling of electron density predictions). Finally, I’ll explore open questions in expressivity, data-efficiency, and trainability of methods leveraging invariance and equivariance.

I anticipate this sort of approach could be a big deal for macroscopic modeling as well. In particular, if multiscale representation is to become commonplace, this sort of physically based model would become the default. In my view, language models are a temporary research artifact of their time and lack true causal understanding. key components of the system that can actually work well will include this sort of equivariance and fundamental 3d knowledge.
I've started publishing the novel I wrote to promote EA
by Timothy Underwood — 2022-12-08T17:30:14.333Z

Currently I'm just uploading it to Royal Road, but in the next week I'll also post it to the spacebattles forum, and around the start of January I'll publish a copy to Amazon.

So, if you find this project interesting, and use Royal Road, follow it and maybe leave a comment there. Also, if there is a site you use to read and discover fiction that would be appropriate for me to publish this to, tell me, and I'd love to put my novel there.

If anyone can help in some way to tell people who they think might like a Isekai/portal fantasy novel about an effective altruist trying to do as much good as he can after he died and woke up with super magical powers in a fantasy world, please do so.

Link: https://www.royalroad.com/fiction/61682/the-split-summon/chapter/1058375/chapter-one-arrival

Thanks!
Neural networks biased towards geometrically simple functions? 
by DavidHolmes — 2022-12-08T16:16:20.752Z

Neural networks (NNs) do not output all functions with equal probability, but seem to be biased towards functions of certain types; heuristically, towards 'simple' functions. In VPCL18, MSVP+19, MVPSL20 evidence is given that functions output by NNs are inclined to have low information-theoretic complexity - nice summaries are given on lesswrong here and here and elsewhere by the author. However, the converse is not true; some functions with low information-theoretic complexity (such as simple periodic functions) are not so readily output by NNs - this is discussed extensively in the comments to the above posts. Understanding this kind of problem better is widely considered relevant for AI alignment, see e.g. here.

To try to understand better the biases of NNs, we consider more geometric measures of simplicity. In particular, for a neural network with ReLU (or other piecewise-linear activation function), one measure of complexity is the measure (size) of the set of points at which the function is not linear. For functions defined on small domains we might call a function 'simple' if it has a small set of points of non-linearity. For a function on a larger or unbounded domain, we might consider a function 'simple' if the points of non-linearity are clustered together (so that the function is 'mostly linear').

In this short note we explore this heuristic in the simple case of a NN with one hidden layer, outputting functions on a 1-dimensional domain. We choose the parameters of our neural network uniformly at random, and compute the shape of the resulting distribution of points of non-linearity.

Perhaps surprisingly, we find that the distribution of the points of non-linearity does not depend on the size of the domain from which parameters of the neural network are chosen, but does depend heavily on its shape. The remainder of this post summarises these results.

Summary of results

The results in the note are more general, but here we summarise the outcome in the case of functions defined on the interval [−1,1] in the real numbers. A NN with one hidden activation layer of width w produces a piecewise-linear (PL) function with at most w points of non-linearity.

Weights and biases chosen uniformly at random in an interval

We consider first the case where both the biases and the weights of our NN are chosen uniformly at random in some fixed interval. Then we find tha...
If Wentworth is right about natural abstractions, it would be bad for alignment
by Wuschel Schulz — 2022-12-08T15:19:02.084Z

This post was written as part of the AI safety Mentors and Mentees program. My Mentor is Jacques Thibodeau.

In this post, I will distinguish between two hypotheses that are often conflated. To disambiguate, I first suggest two different names for these hypotheses so I can talk about them separately:

The natural abstraction hypothesis (NAH):

There are natural ways to cut the world up into concepts. A lot of very different cognitive systems will naturally converge to these abstractions. So there is reason to believe that AIs will also form concepts of abstractions that humans use (nails, persons, human values….).

The Wentworthian abstractions hypothesis (WAH):

There are natural abstractions, and they are identified by the properties of a system, that are relevant for predicting how far away objects behave.

Notice how the first might be true, while the second might be completely off. Just as you can deny that Newtonian mechanics is true, without denying that heavy objects attract each other.

Why natural abstractions are thought to be good for alignment

If NAH turns out to be correct, this would simplify two problems in alignment.

1. Interpretability

If the AI uses the same abstractions as us, it is probably way easier to read its mind.

2. Pointing at things

If the AI forms the abstraction “diamond” itself, we could just point at that abstraction in the AI's mind, and say: “maximize that one”, instead of trying to formulate what a diamond is rigorously. This was proposed in combination with shard theory to the diamond-alignment problem. If it would naturally form an abstraction of human values, alignment might be easier than we thought (alignment by default). We could point at that abstraction by training it in such a way, that it adheres to those values.

Wentworthian abstractions are about outer appearance, not inner structure

Wentworths hypothesizes that natural abstractions consist of information that is relevant from afar. Let's take the example of a Nail. A particular Nail consists of billions of Atoms, and there is an overwhelming amount of facts that could be stated about these particles. However, when you see a Nail on the other side of the room, you only tend to think about some of these facts. Its elongated shape, its color, and its pointiness for example. Let's say these are the only abstract information about these billions of particles that make you identif...
Covid 12/8/22: Another Winter Wave
by Zvi — 2022-12-08T14:40:00.816Z

Here we go again. It is December, and cases, hospitalizations and wastewater concentrations are all going up. How worried should we be?

We should worry that cases will for a time double or triple from where they are. Combined with flu and RSV, this will mean that while our health care systems will hold, they will be under substantial strain that impacts quality of care. Things are going to low-key suck for a bit, and the number of worthwhile additional precautions to take, overall, will not be zero.

We should not worry that this will be anything like last winter. That is not going to happen, not in terms of cases and not in terms of hospitalizations and deaths. That is not in the cards at all.

Executive Summary

Big jump in cases and hospitalizations likely means winter surge.

Chinese protests suppressed, some modest loosening did result.

Long Covid study finds control group that had other respiratory illnesses did worse than the Covid group.

Let’s run the numbers.

The Numbers

Predictions

Predictions from Last Week: 290k cases (+5%) and 2,050 deaths (+15%).

Results: 354k cases (+28%) and 2,390 deaths (+34%).

Predictions for Next Week: 371k cases (+5%) and 2,240 deaths (-6%).

The case number reflects a substantially larger expected real increase with the fact that this week’s number likely is about 10% higher than it should be due to holiday reporting issues around Thanksgiving.

The death number is a clear post-holiday spike so I’m expecting a return back down, as the case spike hasn’t had enough time to add to the count much.

Deaths

This is the classic post-holiday story, with deaths being backfilled. That wasn’t certain because the of the case count being unexpectedly high, but it seems clear now. Things should return to normal until Christmas.

Cases

The winter wave is upon us, to the extent we will have one. Cases are up 28% week over week. Hospitalizations are up 19% week over week, likely a better reflection of actual cases given the holiday. How bad is it going to get?

Mathew David is cautiously optimistic we can avoid a large winter wave.

I agree with this. We likely have something like five weeks left before this peaks, with future weeks unlikely to see larger increases than we are seeing now. That puts the realistic bad scenario peak at something like 250% of current levels, similar to the peak in July...
Why I'm Sceptical of Foom
by DragonGod — 2022-12-08T10:01:01.397Z

Disclaimer

Written quickly[1]. It's better to draft my objections poorly, than to not draft them at all.

Introduction

I am sceptical that "foom"[2] is some of not physically possible/feasible/economically viable.
[Not sure yet what level of scepticism I endorse.]

I have a few object level beliefs that bear on it. I'll try and express them succinctly below (there's a summary at the end of the post for those pressed for time).

Note that my objections to foom are more disjunctive than they are conjuctive. Each is independently a reason why foom looks less likely to me.

Beliefs

I currently believe/expect the following to a sufficient degree that they inform my position on foom.

Diminishing Marginal Returns

1.0. Marginal returns to cognitive investment (e.g. compute) decay at a superlinear rate (e.g. exponential) across some relevant cognitive domains (e.g. some of near human, human spectrum, superhuman, strongly superhuman).

1.1. Marginal returns to real world capabilities from cognitive amplification likewise decay at a superlinear rate across relevant cognitive domains.

Among humans +6 SD g factor humans do not seem in general as more capable than +3 SD g factor humans as +3 SD g factor humans are compared to median humans.

Broad Human Cognitive Spectrum

2. The human cognitive spectrum (1st percentile human to peak human) is broad in an absolute sense.

On many useful cognitive tasks(chess, theoretical research, invention, mathematics, etc.), beginner/dumb/unskilled humans are closer to a chimpanzee/rock than peak humans (for some fields, only a small minority of humans are able to perform the task at all, or perform the task in a useful manner[3], for other like chess, beginners are simply closer to the lowest attainable scores than to the scores obtained by peak humans [600 - 800 is a lot closer to 0 than to 2700 - 2900]).

Median humans are probably also closer to a rock than to peak humans (on e.g. inventing general relativity pre 1920).

Peak humans may be closer to bounded superintelligences than beginner/median humans.

E.g. Magnus Carlsen is closer in ELO to Stockfish than median human.

I expect Magnus Carlsen to be closer in ELO to a bounded superintelligence than to a median human.

Narrow Optimisers Outperform General Optimisers on Narrow Domains

3.0. I believe that for similar levels of cognitive investment narrow optimisers outperform gen...
Take 7: You should talk about "the human's utility function" less.
by Charlie Steiner — 2022-12-08T08:14:17.275Z

As a writing exercise, I'm writing an AI Alignment Hot Take Advent Calendar - one new hot take, written every day (more or less - I'm getting back on track!) for 25 days. Or until I run out of hot takes.

When considering AI alignment, you might be tempted to talk about "the human's utility function," or "the correct utility function." Resist the temptation when at all practical. That abstraction is junk food for alignment research.

As you may already know, humans are made of atoms. Collections of atoms don't have utility functions glued to them a priori - instead, we assign preferences to humans (including ourselves!) when we model the world, because it's a convenient abstraction. But because there are multiple ways to model the world, there are multiple ways to assign these preferences; there's no "the correct utility function."

Maybe you understand all that, and still talk about an AI "learning the human's utility function" sometimes. I get it. It makes things way easier to assume there's some correct utility function when analyzing the human-AI system. Maybe you're writing about inner alignment and want to show that some learning procedure is flawed because it wouldn't learn the correct utility function even if humans had one. Or that some learning procedure would learn that correct utility function. It might seem like this utility function thing is a handy simplifying assumption, and once you have the core argument you can generalize it to the real world with a little more work.

That seeming is false. You have likely just shot yourself in the foot.

Because the human-atoms don't have a utility function glued to them, building aligned AI has to do something that's actually materially different than learning "the human's utility function." Something that's more like learning a trustworthy process. If you're not tracking the difference and you're using "the human's utility function" as a target of convenience, you can all too easily end up with AI designs that aren't trying to solve the problems we're actually faced with in reality - instead they're navigating their own strange, quasi-moral-realist problems.

Another way of framing that last thought might be that wrapper-minds are atypical. They're not something that you actually get in reality when trying to learn human values from observations in a sensible way, and they have alignment difficulties that are idios...
Machine Learning Consent
by jefftk — 2022-12-08T03:50:04.275Z

For years, researchers have trained machine learning systems on
whatever data they could find. People mostly haven't cared about this
or paid attention, I think because the systems hadn't been very good.
Recently, however, some very impressive systems have come out,
including ones that

answer questions,

complete
code, and

generate

images

from prompts.

Because these are so capable a lot more people are paying attention
now, and there are big questions around whether it's ok that these
systems were trained this way. Code that I uploaded to GitHub and the
writing that I've put into this blog went into training these models:
I didn't give permission for this kind of use, and no one asked me if
it was ok. Doesn't this violate my copyrights?

The machine learning community has generally assumed that training
models on some input and using it to generate new output is legal, as
long as the output is sufficiently different from the input. This
relies on the doctrine of "fair use", which
does not require any sort of permission from the original author as
long as it is sufficiently "transformative". For example, if I took a
book and replaced every instance of the main characters name with my
own I doubt any court would consider that sufficiently transformative,
and so my book would be considered a "derivative work" of the original
book. On the other hand, if I took the words in the book and
painstakingly reordered them to tell a completely unrelated story,
there's a sense in which my book was "derived" from the original one
but I think it would pretty clearly be transformative enough that I
wouldn't need any permission from the copyright holder.

These models can be used to create things that are clearly derivative
works of their input. For example, people very
quickly realized that Copilot would complete the code for Greg Walsh's
fast
inverse square root implementation verbatim, and if you ask any of
the image generators for the Mona Lisa or Starry Night
you'll get something close enough to the original that it's clearly a
knock-off. This is a major issue with current AI systems, but it's
also a relatively solvable one. It's already possible to slowly check
that the output doesn't excessively resemble any input, and I think
it's likely they'll soon figure out how to do that efficiently. On the
other hand, all of the examples of this I've ...
Riffing on the agent type
by Quinn — 2022-12-08T00:19:38.054Z

Much is owed to Diffractor for Giry-pilling me at Alignable Structures, I had been struggling with type-driven expected value previously.

Epistemic status: took a couple days off from my master plan to think about John's selection theorems call to action.

We would like a type signature of agency. Scott Garrabrant provides (A→O)→A as a first approximation. You can choose one of two ideas here: 1. that an agent simply takes a belief about how actions A turn into outcomes O and returns a recommended action, or 2. that an agent takes underlying configurations of reality (containing information about how actions lead to outcomes) and tends to perform certain actions. Notice that O happens to be for "outcome", "observation", and even "ontology", which is nice. This signature is widely discussed in the monad literature.

Scott wrote that → primarily means causal influence and secondarily means functions. I will be mostly ignoring the causal influence idea, and I think instead of thinking of the signature from an objective perspective of it being a transcription of the underlying reality, I want to think of it from a subjective perspective of it being an assistant for implementation engineers. I think we should take a swing at being incredibly straightforward about what we mean by the type signature of agency: when I say that a type τ is the type signature of agency, I mean that if we have programs that are admitted by τ then those programs are doing all the things that interest me about agents (i.e., at τ=(A→B)→A, if we instantiate particular non-atomic propositions A and B that interact with the outside world in such a way that we can obtain proofs of (A→B)→A (which we can't do in general) in some way, then those proofs are doing all the things that interest me about agents).

In my view, the first idea (involving "belief") can be called a subjective interpretation of the type signature, and we shall explore some adjustments to make this story better, while the second idea (involving "base reality") can be called an objective interpretation of the type signature, and we shall not explore philosophical controversies around saying that a type is "in" reality rather than in a model.

I will ultimately conclude that I am not equipped to flesh out the objective interpretation, and give a subjective interpretation such that an agent is not one selection function, but a pair of ...
Looking for ideas of public assets (stocks, funds, ETFs) that I can invest in to have a chance at profiting from the mass adoption and commercialization of AI technology
by Annapurna — 2022-12-07T22:35:48.107Z

Playing around with ChatGPT made me realize that it is quite likely that the next profitable tech breakthrough will probably come from AI technology. What do I mean by this? every few years there is a breakthrough (or a few) in technology that gets massively adopted, gets monetized, and becomes very lucrative and profitable to its owner. In the early 2000s, it was e-commerce and online advertisement. In the late 2000s, it was the smartphone. In the early 2010s it was the subscription model service. In the late 2010s, it was SaaS.
Basically I want to position part of my investment portfolio in a way that I have exposure to the next Apple, Amazon, or Salesforce, which I think will come from a successful monetization of AI technology.
I already own LRNZ (TrueShares AI & Deep Learning Fund) and SMH (

VanEck Semiconductor ETF

)

I could also increase my exposure to GOOG, META, and MSFT, as they seem to be the public companies best positioned for an AI breakthrough that can be monetized.

I know AI uses quite a bit of computer power so perhaps something like NVDA also makes sense?

Any other ideas?