trentmkelly/LessWrong-43k huggingface dataset:
A Models-centric Approach to Corrigible Alignment
by J Bostock — 2021-07-17T17:27:32.536Z

This post starts with a reasonable story of how alignment might be solved, based on extracting human values from an AIs model of the world. Then I will elaborate on what I see as the main steps in this process. After that I will review the existing work (which I am aware of) on these areas, as well as discussing points of confusion. I will finish with a list of identified areas of confusion in this scheme.

A Reasonable Story for How Things Go Right

An AI models the world, in doing so it models humans, in doing so it models human values. It then extracts those human values somehow. Then it acts "according to" those values in a similar way to humans do, but with superhuman capabilities.

Breaking this down

1: Modelling humans, in particular human decision making. This is distinct from inverse reinforcement learning, as with this approach it only needs to be done from the perspective of a general predictive model: model humans as part of the world. Modelling the world to a reasonable accuracy in general will do the job thanks to the later steps.

2: Modelling humans in such a way that human values can in theory be separated from the rest of the model. This is not trivial. A neural network (for example) which predicts human behaviour will be hopelessly entangled between the human's beliefs, values, and random biases and akrasia. Possibly corrigibility-as-in-understanding work can help here but it doesn't seem easy. Something like a Pearlian causal network of factors governing human decision making would be ideal. Existing work is building towards this.

3: Pointing to a human in the model. This is non-trivial. A model of the world which predicts the world will have things in it isomorphic to humans, but we need a robust way of pointing to them. Existing work seems to give a mathematically well-defined way to point to things. This will probably need testing.

4: Actually separating values from non-values. There is a no-free-lunch theorem which states that values and beliefs cannot in general be separated from one another. I strongly suspect that we — as humans who have some idea of our own beliefs — can get around this. We could build priors into the modelling system as to what values vs beliefs generally look like. We can also tell the model to put extra bayes-weight on human statements about what our values are. We could potentially just (with an understandable-enou...
Generalising Logic Gates
by J Bostock — 2021-07-17T17:25:08.428Z

Logic circuits traditionally consist of a directed, acyclic graph. Each node can take a value of 0 or 1. The nodes can have 0, 1, or 2 inputs. 0-input nodes are the "input" for the whole system. 1-input nodes are always NOT gates, they output 0 if they are input 1 and vice versa. 2-input nodes are either AND or OR gates, which work like you would expect. Some of the nodes are designated output nodes.

Let's generalise. We can define an (m,n) gate as a function {0,1}m→{0,1}n. In this case a NOT gate is a (1,1) gate; both AND and OR gates are (2,1) gates. Consider all of the possible (m,n) gates: there are 2m possible inputs, and each one of them has one of 2n possible outputs. This means there are (2n)2m=2n×2m possible (m,n) gates. Keeping track of the number of gates is important so we will thoroughly convince ourselves of this.

For m=0,n=1 there are 21×20=2 possible gates. These correspond to gates with no inputs and one output, so the two gates are a node which always outputs 0 and a node which always outputs 1. If we allow n to vary, we have 2n possible (0,n) gates. This makes sense, as a (0,n) gate just consists of a node which always outputs the same string of n bits. (m,n) gates can then be considered as a lookup table with 2m locations, each containing n bits of information. In general there are n×2m bits of information required to specify an (m,n) gate.

Example 1

Now let's consider all possible (2,1) gates. There are 21×22=16 of them:

010=00002

0	1
0	0	0
1	0	0

110=00012

0	1
0	1	0
1	0	0

210=00102

0	1
0	0	1
1	0	0

310=00112

0	1
0	1	1
1	0	0

410=01002

0	1
0	0	0
1	1	0

510=01012

0	1
0	1	0
1	1	0

610=01102

0	1
0	0	1
1	1	0

710=01112

0	1
0	1	1
1	1	0

810=10002

0	1
0	0	0
1	0	1

910=10012

0	1
0	1	0
1	0	1

1010=10102

0	1
0	0	1
1	0	1

1110=10112

0	1
0	1	1
1	0	1

1210=11002

0	1
0	0	0
1	1	1

1310=11012

0	1
0	1	0
1	1	1

1410=11102

0	1
0	0	1
1	1	1

1510=11112

0	1
0	1	1
1	1	1

So what are the numbers above each gate?

Consider the input {0,1}m as a set of numbers a0,...,am−1. We can define the input A uniquely using A=∑m−1i=0ai2i. In binary this is just the concatenation am−1||...||a0. This means there are 2m possible inputs. We will index these with k∈0,...,...
(2009) Shane Legg - Funding safe AGI
by Tomás B. — 2021-07-17T16:46:24.713Z
Link: http://www.vetta.org/2009/08/funding-safe-agi/

Above is a link to an interesting blog post by Shane Legg. It was written before he started DeepMind, and he earns a hell of a lot of points for accomplishing a lot of the insanely ambitious goals set out in the post.

This part is particularly interesting:

The impression I get from the outside is that SIAI [now MIRI] views AGI design and construction as so inherently dangerous that only a centrally coordinated design effort towards a provably correct system has any hope of producing something that is safe. My view is that betting on one horse, and a highly constrained horse at that, spells almost certain failure. A better approach would be to act as a parent organisation, a kind of AGI VC company, that backs a number of promising teams. Teams that fail to make progress get dropped and new teams with new ideas are picked up. General ideas of AGI safety are also developed in the background until such a time when one of the teams starts to make serious progress. At this time the focus would be to make the emerging AGI design as safe as possible.
Large corporations can unilaterally ban/tax ransomware payments via bets
by ryan_greenblatt — 2021-07-17T12:56:12.156Z

After the Colonial Pipeline ransomware
attack, the
idea of banning ransomware
payments
has become more prominent (at least in the US). This has the benefit of
reducing the incentives of hackers as well as avoiding money going to criminal
syndicates. It would also be possible to instead simply tax ransomware payments
at a very high rate (e.g., 1000%).

While actually implementing this ban/tax at a national level might be difficult
and implementing at a state level results in a competitive disadvantage for
that state, I claim that large corporations should be able to unilaterally
simulate some of the effects of a ransom tax while gaining a competitive
advantage. They could do this just by betting against paying a ransom.
Specifically, the corporation could open a prediction market for 'this
corporation pays a ransom in the next 5 years' and then place a large,
unhedged, and uninsured bet against paying a ransom. Of course, the exact terms
and payout details could vary. For example, the payout could be proportional to
the total ransomware payments over a period to simulate a proportional tax
instead of a flat tax. This is is a sort of pre-commitment strategy like
disabling your steering wheel in a game of
chicken
(except with prediction markets).

This market should be reasonably efficient, so the bet itself should have near
0 expected value for the company. However, the side effects of the bet are
likely positive expected value: it should discourage ransomware attacks and
signal confidence about security. Note that if only one company made this bet,
that would likely just shift the targets of ransomware attacks. That's still a
good enough reason for a company to use this approach. Despite the fact that
penalizing ransomware payments is a public good, companies should be able to
unilaterally benefit regardless of how many other companies also use this
approach. Additionally, if a decent percentage of companies actually
implemented this policy, than the incentives could change more broadly.

Cons

A company must be large and well known to implement this policy. It's not
useful if ransomware groups don't hear about it.

Due to asymmetric information and adverse selection, it may be difficult for
companies to secure good rates on the prediction market. This should be less
of an issue for public companies and companies could just allow external
audits. If these exte

...
Fixing the arbitrariness of game depth
by cousin_it — 2021-07-17T12:37:11.669Z

This is just an idea that came to me today. I didn't do any literature search and don't know if it's new.

The "depth" of a game is the length of the longest chain of players where each one beats the next e.g. 60% of the time. Games with more depth are supposed to involve more knowledge, strategy, learning and so on. For example, chess has about double the depth of checkers.

But I don't think that's right. Consider these games:

Deca-chess: two players play 10 games of chess against each other, and whoever wins more, wins.

Coin-chess: a three-sided coin is flipped. Heads -> player 1 wins, tails -> player 2 wins, wings -> a game of chess is played and whoever wins wins.

Under the above definition, deca-chess is "deeper" than chess, because the better player has higher chance to win the whole match than to win a given game. And coin-chess is more "shallow", because the better player has less of an edge. Even though the amount of knowledge, strategy and learning involved in these games is exactly the same!

The same problem happens with games like poker. Even though poker involves a lot of skill, each round has so much randomness that the depth comes out even smaller than checkers. We could introduce repeated rounds like in deca-chess, but how many? It seems arbitrary.

Can the concept of game depth be fixed, made independent from repeated trials and luck? I think yes, by introducing another variable: effort.

Imagine an individual player, let's call him Bob. Have Bob play in a tournament where every match has many rounds, to rule out luck. But in some matches tell Bob to play half-heartedly, and in others tell him to use maximum effort. (Ignore the unreality of the hypothetical, I'm trying to make a point.)

By the end of the tournament we'll have all players arranged by skill, and also know which sub-ranges of skill are covered by individual players' ranges of effort. In other words, we'll end up knowing this: "The difference between the worst and best player in the tournament is covered by about N intervals between a player's slack and best effort at each level".

That number N could be used as a new measure of game depth. Under this measure, chess, deca-chess and coin-chess will be equally deep, and checkers will be less deep. Intuitively it makes sense: "the best player surpasses me by about 5x the difference between my slack and my best effort". It lets ...
Equivalent of Information Theory but for Computation?
by J Bostock — 2021-07-17T09:38:48.227Z

A quick google search brings up "computation theory". This does not seem to be as precise as information theory, we cannot talk about "n units of computation" the same way we can talk about "m bits of information". In fact there does not seem to be a generally accepted fundamental unit of computation which we can talk about.

Computational complexity theory is well-developed but only talks about scaling.

Turing machines seem to be able to "hide" a lot of the work in the number of available symbols to go on the tape or the number of available states of the pointer.

Logic circuits haven't produced much and seem a bit arbitrary (why only OR, AND, and NOT gates?).

Seems like we need a theory of computation to qualitatively understand things like logical uncertainty.
Media Diet: Cultivating Voice, Deliberate Action, and Alive Time
by Willa — 2021-07-17T05:56:33.046Z

Content Consumption Versus Production

My present consumption versus production ratio is not well balanced, I consume ridiculously more media and content than what I produce.(1) So! I am changing that :)

I think content consumption is a necessary part of life and can be quite healthy and good for someone...but! If the ratio is off and/or one is consuming certain kinds of content especially via certain mediums, and one is also trying to get stronger, improve, "level up" then cultivating a prudent media diet is necessary. I want to improve and have something to protect plus pursue, so that's why I'm doing this stuff.

Producing content helps me voice my voice, get stronger, practise deliberate action, and thrive in alive time. Additionally, the more deep work I engage in, the happier and more highly functioning I become. Book recommendation: after reading Cal Newport's "Deep Work", I read OP A.G. Sertillanges' "The Intellectual Life: Its Spirit, Conditions, Methods" and that provided yet more insight into the benefits of deep work plus further increased my desire to do that and is an excellent guide to doing independent, novel, intellectual, and creative work. I highly endorse this book and will be re-reading it very soon. It's likely better and more timeless than Newport's "Deep Work", in my opinion.

Sources of Inspiration

I looked to these posts as examples and guides for building my media diet rules.

Media Diet and Habits

I reserve 8 hours of each weekday for productive, creative, deliberate, etc. work.

4-6 hours of each weekday's reserved 8 hours is for deep work and/or strictly necessary must-get-done-now work. The other 2-4 hours is used for less focus-required, less deliberate but still productive work.

No internet access, social activity, or otherwise potentially distracting things are permitted during this time except for what is strictly necessary to support the work I'm focusing on at that moment.

If I absolutely can't focus on anything, or need relief I may play the piano, write, draw, exercise, or meditate for some reasonable amount of time before returning to work.

No Hacker News or other interesting news aggregators, social media sites, news sites, videos, TV, movies, YouTube, etc. permitted except on Saturdays and Sundays.

I can be as social as I want outside of the 8 hours I reserve each weekday for working.

I can watch one episode

...
What are some examples from history where a scientific theory predicted a significant experimental observation in advance?
by Insub — 2021-07-17T05:39:56.721Z

A few examples I can think of off the top of my head, to give a feel for the reference class I'm looking for:

- The existence and position of Neptune were predicted from observations of Uranus's orbit, before anyone had ever observed Neptune directly

- Black holes were predicted from the equations of General Relativity before we'd ever observed them or their effects on stars' motion

- Not as quantitative, but Darwin's theory of Evolution predicted that we'd find some method by which natural selection actually occurs, before we ever knew about DNA.

Are there other cool examples like these?
Longtermism vs short-termism for personal life extension
by Mati_Roy — 2021-07-17T03:52:54.407Z

Created: 2021-07-15 | Updated: 2021-07-16 | Acknowledgement for feedback: Emanuele Ascani, Haydn Thomas-Rose | x-post: LessDead

By short-termism, I basically mean a method of analysis where one assumes the world stays the same. With longtermism, indirect effects and uncertain futures are taken into account. The longtermist approach focuses on increasing the probability of different stable outcomes, whereas the short-termist approach focuses on the short-term direct impact.

When it comes to personal life extension, short-termist strategies include things like:

Increasing the number of dreams, and notably lucid dreams experienced and remembered

Reducing the amount of time sleeping

Avoiding drugs that erases memory

Having habits that are conductive to faster thinking

At a first approximation, these interventions might all look great. They can all increase subjective life expectancy more than they decrease objective life expectancy, and maybe even robustly so.

However, when the stakes are astronomical, this approach is completely backward. The long termist approach for personal life extension focuses on dying later, not (directly) on living more. Using the following as a toy example, that could mean things like sleeping 10% more only to live 1% more, which doesn't seem to make sense given this would reduce one’s total subjective lifespan at a first approximation, except that this 1% increase in objective lifespan could actually transform into astronomically more because it could allow one to live long enough to live maximally long; i.e. reach a point where cryonics, anti-aging, or other anti-death technologies reach longevity escape velocity. Some longtermist interventions will come at a negligible or zero short-term subjective life expectancy cost, such as lifelogging as life extension, while others will greatly diminish your short-term subjective life expectancy, such as pre-emptive biostasis.

The longtermist approach still has overlapping recommendations with the short-termist one, but when it does, they will sometimes arise for different reasons, and to a different degree. For example, sleeping less might be good to earn more money and therefore be able to afford better biostasis instead of being good because it increases the length of your short-term subjective experience. On the other hand, other interventions are good from both the short-termist and longter...
Bayesianism versus conservatism versus Goodhart
by Stuart_Armstrong — 2021-07-16T23:39:18.059Z

Key argument: if we use a non-Bayesian conservative approach, such as a minimum over different utility functions, then we better have a good reason as to why that would work. But if we have that reason, we can use it to make the whole thing into a Bayesian mix, which can also allow us to trade off that advantage against other possible gains.

I've defended using Bayesian averaging of possible utility functions, even in situations where it seems that Goodhart problems should rear their heads.

Thanks to a more recent discussion with Scott, I can now clarify all these old posts, and look more at the issues of Goodhart's law, Bayesianism, and conservatism.

Where Goodhart will always remain a problem

The Goodhart problem will always remain if we miss some of the key variables. As Stuart Russell said:

A system that is optimizing a function of n variables, where the objective depends on a subset of size k<n, will often set the remaining unconstrained variables to extreme values; if one of those unconstrained variables is actually something we care about, the solution found may be highly undesirable. This is essentially the old story of the genie in the lamp, or the sorcerer’s apprentice, or King Midas: you get exactly what you ask for, not what you want.

If the AI's target is phrased in terms of world GDP and human survival, then no function of these variables will make it care about human happiness and flourishing. So the Goodhart problem will always be a problem some key variables are missed.

This is the reason that I'm looking at model-splintering, so that the AI can add variables as its power and knowledge develops.

Bayes versus Goodhart

So, let's say we know all the variables, but not necessarily whether they are good or bad or neutral, or to what extent.

Split them into different sets: V1, the set of variables that are potentially valuable (roughly linearly), V2, the set of potentially necessary variables (eg the existence of conscious minds), and V3, the set of variables that might be potentially disastrous if they go negative (eg suffering). There can be V4 or V5 or so on with other properties, but we'll ignore that for the moment.

There may be variables that are in more than one category - for example, if happiness/sadness are on the same scale (which is not automatically true), then maybe this would be linearly important in standard situations, but things...
"If and Only If" Should Be Spelled "Ifeff"
by jimrandomh — 2021-07-16T22:03:22.857Z

If and only if is an important logical concept, useful in many contexts, both mathematical and nonmathematical. Unfortunately, "if and only if" is also an unwieldy five-syllable phrase. Mathematicians have solved this problem by shortening it to "iff". Unfortunately, this shortening has not caught on in non-mathematical contexts. This makes some communication and thinking unwieldy and ambiguous.

I think the reason "iff" hasn't caught on more broadly is because it's easily misread as "if", and doesn't have an intuitive pronunciation. I think both of these problems would be solved by changing the spelling to "ifeff" (prononunced /ɪfɛff/). The etymology is that you take "iff", and pronounce the second "f" separately. This would slightly improve the thinking and communication of most English speakers.

I think a small group of people using "ifeff" in their writing would likely start a process where "ifeff" eventually takes over, via the usual process by which vocabulary spreads, and that "ifeff" would be used by groups that don't currently have a short-enough word for this concept. I also think the correspondence between "iff" and "ifeff" is intuitive enough that this will not cause very much confusion.
Open Philanthropy is seeking proposals for outreach projects
by abergal — 2021-07-16T21:19:53.932Z

[Cross-posted from the EA Forum.]

Open Philanthropy is seeking proposals from applicants interested in growing the community of people motivated to improve the long-term future via the kinds of projects described below.[1]

Apply to start a new project here; express interest in helping with a project here.

We hope to draw highly capable people to this work by supporting ambitious, scalable outreach projects that run for many years. We think a world where effective altruism, longtermism, and related ideas are routine parts of conversation in intellectual spaces is within reach, and we’re excited to support projects that work towards that world.

In this post, we describe the kinds of projects we’re interested in funding, explain why we think they could be very impactful, and give some more detail on our application process.

Proposals we are interested in

Programs that engage with promising young people

We are seeking proposals for programs that engage with young people who seem particularly promising in terms of their ability to improve the long-term future (and may have interest in doing so).

Here, by “particularly promising”, we mean young people who seem well-suited to building aptitudes that have high potential for improving the long-term future. Examples from the linked post include aptitudes for conducting research, advancing into top institutional roles, founding or supporting organizations, communicating ideas, and building communities of people with similar interests and goals, among others. Downstream, we hope these individuals will be fits for what we believe to be priority paths for improving the long-term future, such as AI alignment research, technical and policy work reducing risks from advances in synthetic biology, career paths involving senior roles in the national security community, and roles writing and speaking about relevant ideas, among others.

We’re interested in supporting a wide range of possible programs, including summer or winter camps, scholarship or fellowship programs, seminars, conferences, workshops, and retreats. We think programs with the following characteristics are most likely to be highly impactful:

They engage people ages 15 - 25 who seem particularly promising in terms of their ability to improve the long-term future, for example people who are unusually gifted in STEM, economics, philosophy, writing, speaking, or debate.

T

...