trentmkelly/LessWrong-43k huggingface dataset:
Introducing the Principles of Intelligent Behaviour in Biological and Social Systems (PIBBSS) Fellowship
by adamShimi — 2021-12-18T15:23:26.672Z

Cross-posted to EA Forum

Introduction

How would you go about scientifically studying aliens? Arik Kershenbaum’s

The Zoologist Guide to The Galaxy

proposes to use evolutionary thinking to uncover constraints on how alien species could evolve. One of his most interesting points is that evolution constrains function far more than form, because function depends significantly less on the details of the environment. Hence we should expect crisper answers to “How would aliens behave?” than “What would aliens look like?”. And in the course of his book, he gives the best answer he can find to the former question.

So when confronted with the question of how to study something he couldn’t gather data on, Kershenbaum leveraged analogies to biological systems he could and had studied, and the underlying constraints brought on by the mechanisms of natural selection.

On a completely unrelated note, the new summer fellowship

Principles of Intelligent Behavior in Biological and Social Systems

(PIBBSS)

(funded by the

LTFF

) aims at creating valuable AI alignment research through studying analogies to many complex systems (evolution, brains, language, social structures…). Fellows will have graduate research experience in fields studying such systems, working on a concrete alignment project in collaboration with an established alignment researcher. The fellowship will run during all of Summer 2022.

The point of this post is to introduce this fellowship, explain the reasoning behind it and give more concrete details about how it will go. Note that I’m not an organizer of this fellowship, I’m just assisting with the writing of this post; credits for the ideas and arguments should go to Nora Ammann and TJ, the organizers of the fellowship.

Analogies as General Epistemic Strategies for Alignment

As I’ve

written elsewhere

, alignment cannot directly leverage most epistemic strategies and approaches used in Science and Engineering, because it’s about solving a problem that doesn’t exist yet on a technology we still have to invent.

One epistemic strategy that survives this major problem is the leveraging of analogies with existing biological or social systems that implement complex or intelligent behavior. Consider how a wide variety of such systems (from biology, physics, linguistics and other fields) exhibit similar properties: adaptation, robustness, goal-directed behavior, learning, embedde...
Important ML systems from before 2012?
by Jsevillamol — 2021-12-18T12:12:21.410Z

I am the main coordinator of a group studying trends in Machine Learning.

Together, we are building a public dataset annotating milestone ML systems between 1950 and today. You can read more about our work here.

Our curation of papers is quite sparse before 2012, so we are keen on more suggestions of papers before then!

This will be very useful, specially in the context of understanding how big of a deal was the advent of Deep Learning.

The papers we are interested are those which fall in at least one of this categories:

Highly cited (>1000 citations as a rule of thumb, eg Understanding the difficulty of training deep feedforward neural networks)
Historical importance (eg an early paper experiment with RL would count, like Some studies in machine learning using the game of checkers)
Paper that pushes forward the SotA (eg The BellKor 2008 Solution to the Netflix Prize )
Deployed in an important context (eg a ML system that was deployed in a popular search engine, like Netflix's Cinematch)

Any suggestions?
How the Equivalent Bet Test Actually Works
by Erich_Grunewald — 2021-12-18T11:17:12.633Z
Link: https://www.erichgrunewald.com/posts/how-the-equivalent-bet-test-actually-works/

Rationalists often talk about betting as a way of bringing forth more accurate beliefs. The idea is that, in making a bet, you are financially incentivised to think really carefully about how much credence you give some proposition (the proposition that is being operationalised as a bet).

In a similar vein, Julia Galef writes about the equivalent bet test, a technique, adapted from Douglas Hubbard, for quantifying uncertainty.[1] It works like this. Say you want to figure out how sure you are that P is true, quantitatively. Now you ask yourself, "If P is true, I win ten thousand euro; or if I roll under four on a 1d6, I win ten thousand euro – which of these two choices do I prefer?"[2] If you prefer the first one – getting ten thousand euro if P is true – then ask the same question again but with more favourable chances on the dice roll, like rolling under five or six. If you prefer the latter one – getting ten thousand euro if you roll under four on a 1d6 – then ask again but with less favourable chances on the dice roll, like rolling under three or two. If you can't decide – if the bets seem equivalent to you – then you have found the probability with which you roughly think P is true, your credence.

This technique has worked pretty well for me.[3] But it's weird. Say I am trying to determine how much credence I give the following proposition:

Russia will invade Ukrainian territory before 2023.

What I am doing with Hubbard's technique is replacing that decision with another decision, or rather iterating on a series of other decisions:

Choose "I get ten thousand euro if Russia invades" or "I get ten thousand euro if a 1dN die rolls under X".

Again, only when I find that it is all but impossible to decide this have I made my decision about how much credence I give the original proposition.

Here is the weird thing. When I decide whether to believe a thing, I am making a judgment.[4] My goal in making this judgment is to have true beliefs, to have beliefs that properly describe the territory. (This is not necessarily always the goal, but it is often enough. Humour me and suppose that it is my goal in this case at least.) When making judgments, we use evidence. To determine my credence in the Russian invasion example, I may search for evidence about base rates, recent developments along the border, Russian and Ukrainian military capacity, domestic and interna...
DL towards the unaligned Recursive Self-Optimization attractor
by jacob_cannell — 2021-12-18T02:15:30.502Z

Consider this abridged history of recent ML progress:

A decade or two ago, computer vision was a field that employed dedicated researchers who designed specific increasingly complex feature recognizers (SIFT, SURF, HoG, etc.) These were usurped by deep CNNs with fully learned features in the 2010's[1], which subsequently saw success in speech recognition, various NLP tasks, and much of AI, competing with other general ANN models, namely various RNNs and LSTMs. Then SOTA in CNNs and NLP evolved separately towards increasingly complex architectures until the simpler/general transformers took over NLP and quickly spread to other domains (even RL), there also often competing with newer simpler/general architectures arising within those domains, such as MLP-mixers in vision. Waves of colonization in design-space.

So the pattern is: increasing human optimization power steadily pushing up architecture complexity is occasionally upset/reset by a new simpler more general model, where the new simple/general model substitutes human optimization power for automated machine optimization power[2], enabled by improved compute scaling, ala the bitter lesson. DL isn't just a new AI/ML technique, it's a paradigm shift.

Ok, fine, then what's next?

All of these models, from the earliest deep CNNs on GPUs up to GPT-3 and EfficientZero, generally have a few major design components that haven't much changed:

Human designed architecture, rather than learned or SGD-learnable-at-all

Human designed backprop SGD variant (with only a bit of evolution from vanilla SGD to Adam & friends)

Obviously there are research tracks in DL such as AutoML/Arch-search and Meta-learning aiming to automate the optimization of architecture and learning algorithms. They just haven't dominated yet.

So here is my hopefully-now-obvious prediction: in this new decade internal meta-optimization will take over, eventually leading to strongly recursively self optimizing learning machines: models that have broad general flexibility to adaptively reconfigure their internal architecture and learning algorithms dynamically based on the changing data environment/distribution and available compute resources[3].

If we just assume for a moment that the strong version of this hypothesis is correct, it suggests some pessimistic predictions for AI safety research:

Interpretability will fail - future DL descendant is m

...
A Tree of Light
by Jarred Filmer — 2021-12-18T00:36:13.005Z

let me sing you a song

imagine
you float in a sphere of water
which rests in a space of infinite darkness

you are deeply cold
and you are deeply alone

you touch the edge of your sphere with a finger
and see a thread of gold, pulse out into the void
oh look, someone else, they look as cold as you are

the warmth of the thread between you feels so good
by its brief light you see their face, and the sphere in which they float

you both start sending light to one another
you're not so cold now
you can see the bounds of your waters
and you begin to grow
you find others to send light to
and they find you

now you're connecting in synchrony with many
and you see they're connecting with one another
it feels so warm, and by the light you all grow larger and larger
no one will be left alone, you’ll find them all

you spread your fingers
and a glorious tree erupts from your sphere
connecting everyone
creating more warmth, and more light than you ever dreamt possible

but what you see by this light horrifies you
you and your friends are not alone
you are surrounded by legions of others
trapped in cold darkness

there are so many
that even if your cluster sent all of its light
it would be but a drop, in an ocean of darkness
you cannot enjoy the light you have found
it seems hopeless

but from a great distance away, a tendril of light finds you
and it sings you a song
a song about consciousness
about light and warmth
about all those in darkness

it calls you to follow it away from the cluster in which you were born
to join others who see what you have seen
in building a grand tree of light
a tree so bright
and so tall
it can fill this space in which you have all been born
with light and warmth from end to end

I wrote this after a long online discussion with a friend. We’ve been helping one another make sense of some messy feelings. Namely, being 28 and finding ourselves in possession of :

Agency: Skills, resources, and connections after 8 or so years of professional work and growth
Community: The relationships we’ve made organising and participating in local community
Freedom: Nothing vital is currently relying on us to function (e.g a family, or a company)
Purpose: We see how high the Jenga tower of civilisation could yet climb, and how we might be able to help shore it up in some small way

After covid lockdowns lifting and borders opening we have the resources and freedom to explore the world and work on ambitious EA projects, ...
Getting diagnosed  for ADHD if I don't plan on taking meds?
by vroomerify — 2021-12-17T19:27:25.717Z

Several people have told me that it is likely I have ADHD & I should go to a doctor and get diagnosed. At the same time, I have no desire to take medication. Given that this is the case, is there any point in going to a doctor & getting diagnosed. After all, it seems like a diagnosis would not change my actions in anyway.
Venture Granters, The VCs of public goods, incentivizing good dreams
by mako yass — 2021-12-17T08:57:30.858Z

Epistemic Status: Early concept

The Gist of it: To support sane risktaking, an institution should generally grant career capital in proportion to a quantitative record of the harms and benefits of a decisionmaking entity's past funding decisions. An impact market seems to be the best way of doing that. I explain impact markets. I also propose rewarding or punishing investors with returns in a special domain-specific credit, rather than raw cash. This allows the balance of investors to correct more quickly in line with total generated value, instead of only being moved in line with available public funding, while also significantly decreasing opportunities for embezzlement.

Why we need Impact Markets

Markets are good at rewarding successful risktaking, but there are many critical public goods that they reliably fail to fund in proportion to their importance. Examples include information sources, existential security, all non-excludable goods and unmeterable infrastructures, these are the holes left by the coordination problem of free-riding and they are generally left to governments to fill. Unfortunately, any large, long-lived institution, including most government funding bodies, tends to demand burdensome degree of legibility of their projects, they only allow simple, respectable-sounding projects, and they either forbid or disproportionately punish risktaking.

In most institutions, the downside for visibly taking a risk and failing is significant, while the upside of taking a risk and succeeding will be nowhere near proportionate to the downside. Sometimes the positive contributions you make wont protect you at all from being dismissed the moment you lapse, so institutions don't take risks, so nothing novel gets to happen, and most systemic change would be novel, there's a sense in which any sort of substantial improvement must be novel.

Being duly supportive of eccentrics is just naturally difficult for a large organization. Doing it without appropriate systems in place to protect disruptive decisionmakers from perverse political structures and collective loss aversion biases is like trying to hand-draw an even circle without a compass. It's easy to imagine that happening, but it doesn't really happen.

I have not seen a suitable compass, before finding this one.

Solution: Venture Granters

Venture Granters (VGs) act as participants in a public goods market, inves...
Understand the exponential function: R0 of the COVID 
by Yandong Zhang — 2021-12-17T06:44:30.119Z

To my best knowledge, there is no data evidence that the vaccines do prevent the transmission of the delta and Omicron (Please fix if I was wrong). The effect of reduce the mortality rate is LINEAR. A simple calculation suggests that the LINEAR contribution could be dominate by the transmission rate increase due to the exponential effects. In other words, the life saving due to the vaccine is not comparable to the more life loss due to the transmission rate increase.

Thus, the priority of the policy should be reduce the transmission, instead of the linear factors. Am I wrong?

By the way, I am not anti-vaccine.
Some motivations to gradient hack
by peterbarnett — 2021-12-17T03:06:19.425Z

While taking part in the

SERI MATS Program

I wrote a post on

gradient hacking

, which is a theoretical phenomenon where a mesa-optimizer might deliberately attempt to influence its own training process via its outputs and thinking process. Gradient hacking seems most dangerous in the case of a deceptive mesa-optimizer, which is attempting to conceal its misalignment from humans. This dangerous deceptive outcome seems more likely than having a mesa-optimizer which is robustly aligned with humans, for reasons outlined

here

. But there do also seem to be multiple different incentives for a mesa-optimizer to gradient hack, which can be split into categories of Bad, Good, and Unclear (from the human perspective).

For the bad motivations, the mesa-optimizer is not aligned with either the base-objective or the human’s goals. For the good motivations, I imagine a case where we have an outer alignment failure, such that the base-objective is not actually what humans want, and the mesa-optimizer is somehow aligned with the human goals (rather than the base-objective).

This list of possible motivations isn’t meant to be exhaustive.

Bad

For ‘bad’ motivations we have an inner alignment failure, where our model/mesa-optimizer is not aligned with the human or the base-objective. Here the model has strong incentives to manipulate its training process to improve overall performance on its mesa-objective. Most of these are explored in more detail in

this section

of my other post.

Protect the mesa-optimizer

Neural networks trained with gradient descent seem to be pushed towards simplicity due to the inductive biases of the training process. This seems especially to be the case if there is some kind of weight regularization which directly reduces the magnitude of the weights. This drift towards simplicity seems likely to hurt the mesa-optimizer, either by erasing it or modifying its mesa-objective. If the mesa-optimizer ‘wants’ to continue existing, then it could use gradient hacking to protect itself. Here ‘wants’ means that the mesa-optimizer reasons that the mesa-objective will be maximized if the mesa-optimizer continues to exist (as opposed, for example, to the mesa-optimizer allowing itself to be erased and instantiating a non-optimizing process which also leads to high performance on the mesa-objective).

Hide from transparency tools

A deceptive mesa-optimizer may...
Blog Respectably
by lsusr — 2021-12-17T01:23:47.426Z

The best way to learn how to write is to take classes. The best way to take classes is via a university degree. I earned my master's in blogging at the University of Pens.

Besides teaching you how to write, a university provides necessary credentials. You wouldn't be reading this post if I weren't for my degree in the subject. But the most important reason for taking classes in writing is what it teaches you not to do.

A blogger's most important responsibility is to never offend anyone. The most reliable way to avoid offending people is to hide your work from them. The perfect writer never publishes anything. Self-censorship is the best censorship because it is self-censorship quiet. Liberty thrives in secrets and silence. Freedom is founded on fear.

If unsure who to fear, look for a sense of humor because humor blurs the lines between "us" and "them". "We" are right; "they" are wrong. Blurring the lines between "us" and "them" equals blurring the lines between truth and falsehood. Cultivate anger and hatred instead because anger and hatred make clear the lines between "us" and "them".

Blogging is bad for your mental health. When you write things people will tell you you are wrong. They will sometimes be right which is bad for your self-esteem. If you want to maximize your self-worth then you must never allow it to collide with reality.

Another preemptive measure you can take against haters is getting lots of people on your side. Pick a moderate majority position backed by people who perceive themselves to be radical, oppressed and disenfranchised.

No matter how mainstream the position your endorse, there will always be people who oppose you. The solution is to never endorse a position. Criticize only. Poke holes in the arguments of writers foolish enough to stand for something. Like guerrillas, you should harass the enemy without ever holding intellectual territory of your own. Dialogue is a contest of attrition. Demoralize your critics.

If you must endorse a position then quote respectable people. If you're right then the position is your own. If you're wrong then position is the quotee's.
The Case for Radical Optimism about Interpretability
by Quintin Pope — 2021-12-16T23:38:30.530Z

Summary:

I argue we can probably build ML systems that are both high-performance and much more interpretable than current systems.
I discuss some reasons why current ML interpretability lags behind what should be possible.
I highlight some interesting interpretability work on current systems.
I propose a metric for estimating the interpretability of current systems and discuss how we might improve interpretability.

Definitional note: for this essay, "interpretable" serves as a catch-all for things like "we can abstractly understand the algorithms a system implements" and "we can modify internal states to usefully influence behavior in a predictable manner". Systems are more interpretable when it's easier to do things like that to them.

1: Strong, interpretable systems are possible

Imagine an alien gave you a collection of AI agents. The agents vary greatly in capability, from about as smart as a rat to human level (but not beyond). You don't know any details about the AI's internal algorithms, only that they use something sort of like neural nets. The alien follows completely inhuman design principles, and has put no effort whatsoever into making the AIs interpretable.

You are part of a larger community studying the alien AIs, and you have a few decades to tackle the problem. However, you operate under several additional constraints:

Computation can run both forwards and backwards through the AIs (or in loops). I.e., there's no neat sequential dependency of future layers on past layers.
Each AI is unique, and was individually trained on different datasets (which are now unavailable).
The AIs are always running. You can't copy, backup, pause or re-run them on different inputs.
Each AI is physically instantiated in its own hardware, with no pre-existing digital interface to their internal states.
If you want access to internal states, you'll need to physically open the AIs up and insert electrodes directly into their hardware. This is delicate, and you also risk permanently breaking or damaging the AI with each "operation".
Your electrodes are much larger than the AI circuits, so you can only ever get or influence a sort of average activation of the surrounding neurons/circuits.
You are limited to 10 or fewer probe electrodes per AI.
You are limited to the technology of the 1960s. This includes total ignorance of all advances in machine learning and statistics made sinc

...