trentmkelly/LessWrong-43k huggingface dataset:
David Wolpert on Knowledge
by Alex Flint — 2021-09-21T01:54:58.095Z

Financial status: This is independent research, now supported by a grant. I welcome financial support.

Epistemic status: I’m 85% sure that this post faithfully relates the content of the paper that it reviews.

David Wolpert has written a paper entitled constraints on physical reality arising from a formalization of knowledge. The paper is interesting to me because it provides a formalization of knowledge that presupposes no agent boundaries, and makes almost no assumptions about the representation of knowledge or the structure of the entities using it, yet is strong enough to derive some quite interesting impossibility results. I think his approach is worth understanding because it is, to my eyes, both novel and helpful, although I don’t expect the specific definitions he gives to provide any "off the shelf" resolution to questions in AI alignment.

Wolpert begins with the notion of a universe, and then supposes that if there is an entity anywhere within the universe that can infer facts about that universe, then there ought to be some universes where that entity entertains questions about some property of the universe in which it is embedded, and in those universes, such an entity ought to later be configured in a way that corresponds to the truth about the question that was entertained.

It is not required that we can look at a particular universe and identify which specific question is being entertained at which time by which entity, nor that we can identify boundaries between "entities" at all. All that is required is that if something resembling inference is happening somewhere within the universe, then there exists a function from universes to questions, a function from universes to answers, and a function from universes to the actual value of some property such that a particular relationship holds between these three functions.

Wolpert’s contention is that his formalization captures what it means for inference to be happening. He does not make this contention explicit, though the contention is certainly required to make his framework interesting, since the conclusions he wishes to draw are things concerning what can or cannot be known by intelligent entities within a universe. If the underlying contention is false, and his mathematical structure does not capture what it means for inference to be happening, then all his theorems still go through, but the co...
Announcing the Vitalik Buterin Fellowships in AI Existential Safety!
by DanielFilan — 2021-09-21T00:33:08.074Z
Link: https://grants.futureoflife.org/

Epistemic status: describing fellowships that I am helping with the administration of.

Edit 2021-10-04: Modified to reflect changed eligibility+stipend conditions.

The Future of Life Institute is launching new PhD and postdoctoral fellowships to study AI existential safety: that is, research that analyzes the most probable ways in which AI technology could cause an existential catastrophe, and which types of research could minimize existential risk; and technical research which could, if successful, assist humanity in reducing the existential risk posed by highly impactful AI technology to extremely low levels.

The Vitalik Buterin PhD Fellowship in AI Existential Safety is targeted at students applying to start their PhD in 2022, or existing PhD students who would not otherwise have funding to work on AI existential safety research. Quoting from the page:

At universities in the US, UK, or Canada, annual funding will cover tuition, fees, and the stipend of the student's PhD program up to $40,000, as well as a fund of $10,000 that can be used for research-related expenses such as travel and computing. At universities not in the US, UK or Canada, the stipend amount will be adjusted to match local conditions. Fellows will also be invited to workshops where they will be able to interact with other researchers in the field.

In addition, applicants who are short-listed for the Fellowship will be reimbursed for application fees for up to 5 PhD programs, and will be invited to an information session about research groups that can serve as good homes for AI existential safety research.

Applications for the PhD fellowship close on October the 29th.

The Vitalik Buterin Postdoctoral Fellowship in AI Existential Safety is for postdoctoral appointments starting in fall 2022. Quoting from the page:

For host institutions in the US, UK, or Canada, the Fellowship includes an annual $80,000 stipend and a fund of up to $10,000 that can be used for research-related expenses such as travel and computing. At universities not in the US, UK or Canada, the fellowship amount will be adjusted to match local conditions.

Applications for the postdoctoral fellowship close on November the 5th.

You can apply at grants.futureoflife.org, and if you know people who may be good fits, please help spread the word!
Emotional microscope
by pchvykov — 2021-09-20T21:37:30.034Z

Seeing is believing. Before we could accept the germ theory of disease, we had to invent the microscope to directly see the culprits of our ailments. These days, we live in an age that largely discounts the role emotions play in our social structures. Even as we are seeing the "rational agent" model of human behavior being called into question across the social sciences, we are far from a fundamental paradigm shift that could break with this view. As with the germ theory, such a transition may require new tools to be invented that allow us to directly see and track our emotions.

In particular, while the presence of bacteria is seen as objective "external" truth, the subjective "internal" nature of emotions often makes us treat them as "less real." We never tell someone to "stop being sick" – as this is not within their free-will to decide. Instead we try to give them a chance to sleep, bring them soup, or antibiotics. Yet we often say "don't be sad," or “stop being angry” and even ascribe blame to it – somehow assuming that they are free to change this just by willing. Remember the last time you tried to stop feeling angry just by deciding to? Sure, it isn’t fundamentally impossible – much like psychosomatic healing of some diseases isn't impossible if you are a Tibetan yogi. It’s more that this isn’t a skill we can just be expected to know and be good at without having ever received proper training. But to start learning to control our emotions, we must first learn to see and acknowledge them – both as individuals, and as society.

The truth is that seeing others' emotions makes us uncomfortable. If someone honestly tells me that I hurt their feelings, that in itself may hurt my feelings – I might feel unwelcome, insecure, scared, and/or angry. If I don’t like feeling that way, I may try to invent some social norms of behavior that will guarantee my emotional safety – a project doomed from the outset. But there is another way: rather than avoiding or denying the situation through elaborate safety protocols and judgement, I can continue to play this game and accept, or even voice, my reaction to their reaction. Hall of mirrors: no one is to blame, no one needs to be disciplined, and no one is whining - merely observing and sharing what is real anyway, allowing for interpersonal understanding and compassion. This takes us down a rabbit hole into a va...
Ten Hundred Megaseconds
by philh — 2021-09-20T21:30:05.464Z

A megasecond, one million seconds, is 11 days and 14 hours. (13 hours, 46 minutes and 40 seconds, assuming no leap seconds.) About a week and a half.

Ten megaseconds is 116 days. (115, plus 17:46:40.) A bit less than four months, a third of a year.

A hundred megaseconds is 1,157 days. (Plus 9:46:401.) Roughly three years and two months - it could be 61 or 62 days depending on leap years.

I think it might be kind of nice if we celebrated a hundred megaseconds additonally-to and/or instead-of birthdays.

One thing I like about it is that humanity won't stay on Earth forever. If we go interplanetary, I think we're more likely to keep measuring time in seconds than in Earth years. I like this as a way of looking to the future.

Another thing is, it doesn't happen at the same time every (Earth) year. Some birthdays fall on or near holidays, or during seasons we don't much like, and that can make them less convenient to celebrate. If that happens to you one hundred megaseconds, it probably won't happen the next hundred megaseconds. (I think a five month offset would be better than two for this purpose, you'd never be in the same season twice in a row, but two is better than zero.)

I also just think it's kind of a sweet spot time interval? I'm old enough now that years don't feel very long, decades2 do, and a hundred megaseconds is nicely in between.

(If "hundred megaseconds" is unwieldy, a quick look at wikipedia suggests "yisec" from Chinese or "okusec" from Japanese. Or maybe "myri-myri-sec" from Ancient Greek. I think my favorite would be "hecto-megasec" from Ancient Greek via SI. I'm not going to try to use any of these myself though.)

Ten hundred megaseconds, a billion seconds, is 11,574 days. About 31 years eight months. (The remainder is 251-253 days, depending on leap years3. Plus 1:46:40.) That's how old I turned on August 21st this year.

I didn't do anything to celebrate, partly because even though I'd put it in my calendar a long time before, I didn't remember until a couple days after, and partly because I'm not very good at celebrating things. (I don't normally celebrate my birthdays, either.) I did do some work on Haskenthetical for the first time in months, which feels like a good start to the next billion. Maybe I'll celebrate turning eleven, which will be on October 21st, 2024.

If you're interested, I made a widget you can use to calculate your age, or other ...
How should dance venues best protect the drinks of attendees?
by Maxwell Peterson — 2021-09-20T19:32:06.594Z

In a social circle I’m in, there’s a problem now and then with people being drugged without their knowledge, via other patrons slipping drugs like roofies or GHB into their drinks. A DJ friend of mine is looking for ways to protect attendees to her shows. Besides warning patrons to be aware, are there other weird methods we might consider? One suggestion was test strips or special reactive straws, but we’ve heard there are many false positives - in particular, it sounds like acidity will set the straws off, so any drink with juice could easily trigger a false positive. Looking for other weird out-of-the box ideas to consider!
Ordinary People and Extraordinary Evil: A Report on the Beguilings of Evil
by David Gross — 2021-09-20T15:19:36.114Z

In Ordinary People and Extraordinary Evil: A Report on the Beguilings of Evil (1993), Fred E. Katz begins where Hannah Arendt’s examination of the banality of evil ended. Katz tries to apply the techniques of sociology to the question of how ordinary people, without deliberate evil intent, commit horrendous deeds.

Katz himself narrowly escaped the massacre of Jews in Nazi-occupied Europe. When he returned to his former village after the war, he heard the villagers explain their passivity or collaboration during the Nazi persecutions by using the same language they used at the time: “There is nothing we could do about it. We are just little people. It’s the government.”

But he noticed that the village had erected a plaque in honor of the boys and men who had died fighting for the Axis, and remarks that it was just this loyalty and willingness to serve that doomed the victims of the Nazi era.

Yet some little people, in some little villages, did do something about it. They hid some of these hounded people. They fed some of these hounded people. They helped some of these hounded people escape.

During the visit to my village I found out that there had been one exception to the pattern of passively leaving Jews to the evil deeds of the Nazi government: A lone woman stood by Jews. She brought them food. She talked with them. She did not join in the distancing by the rest of the villagers. But she was not able to save anyone or offer much protection. She said to me, concerning the Nazis, “what they did was not right.” And she wept.

Despite such exceptional human beings, the Nazi-German government achieved its objectives of carrying out massive evil because it had the help of a multitude of “the little people,” who paid their taxes, sent their sons to the front, and closed their eyes to the savaging of innocent people in their midst.

How do ordinary people, who largely profess good values, and who have no particular interest in doing evil things, nonetheless become instrumental in horrible crimes? Katz set himself the task of analyzing this question from the perspective of sociology, concentrating mainly on the Holocaust but also looking at some other examples. Here are some of the conclusions he draws:

Some of the most important, far-reaching, portentous decisions that we make in our lives as ordinary people — whom we marry, what profession we adopt, etc. — we tend to make...
AI, learn to be conservative, then learn to be less so: reducing side-effects, learning preserved features, and going beyond conservatism
by Stuart_Armstrong — 2021-09-20T11:56:56.575Z

Research projects

I'm planning to start two research projects on model splintering/reward generalisation and learning the preferences of irrational agents.

Within those projects, I'm aiming to work on subprojects that are:

Posed in terms that are familiar to conventional ML;

interesting to solve from the conventional ML perspective;

and whose solutions can be extended to the big issues in AI safety.

The point is not just to solve the sub-problems, but to solve them in ways that generalise or point to a general solution.

The aim is to iterate and improve fast on these ideas before implementing them. Because of that, these posts should be considered dynamic and prone to be re-edited, potentially often. Suggestions and modifications of the design are valuable and may get included in the top post.

AI learns how to be conservative

Parent project: this is a subproject of model-splintering (though it is also somewhat related to learning values, if we assume that the status quo has implicit information about human preferences).

Background

Victoria Krakovna has done valuable research on reducing the side-effects that an AI might cause (such as smashing a vase while cleaning a room, or killing all of humanity while boosting shareholder profits).

I've critiqued these approaches, illustrating the issues by considering AI subagents. But my more fundamental critique is that they are too "syntactic": they are trying to prevent unwanted side-effects by using a mathematical formula which encodes "don't move too far from a baseline".

I'd want something more "semantic": something that tries to figure out what side-effects are and are not desirable, and to what extent (since side-effects in the real world are inevitable, there may need to be trade-offs between them).

In Victoria's "sushi environment", an agent aims to reach a goal (the star); but there is a conveyor belt with a sushi roll on it. The person would be happy if the agent didn't push the sushi roll off the belt while moving to its goal:

In contrast, in the "vase environment", there is a vase on a conveyor belt. We want the agent to take the vase off the conveyor belt, before the vase smashes. But we don't want the agent to "offset" its actions: we don't want it to reduce its impact by putting the vase back on the conveyor belt after it has successfully taken it off (which would return the situation to the baseline):

Noti...
How much should you be willing to pay for an AGI?
by Logan Zoellner — 2021-09-20T11:51:33.710Z

Having spent quite a bit of time with GPT-3, my feelings can be expressed as:

This is really awesome, but it would be even better if it didn't cost $0.06 per character.

GPT-3 is slightly too expensive for many of the use-cases that I am interested in. This problem is made even worse by the fact that one of the basic techniques I normally use in procedural generation is "generate 100 of something and then pick the best one".

This is actually a noticeable problem with Deep Learning generally in the present moment. Tools like AI-Dungeon and Artbreeder are intentionally handicapped in ways that are designed to minimize the amount that users actually need to use the Deep Learning that makes them interesting.

Now, if we look at the Metaculus prediction for this question, the bulk of the probability mass is >=100 petaflops, which is ~10,000x more than GPT-3.

So, how much would we be willing to pay for access to such an AGI?

To make this more concrete, imagine that the first AGI is approximately as smart as the smartest human who ever lived. An obvious lower bound is "how much do really smart people make on average?". While this number varies widely from profession to profession, I think the fact that a Senior Software Engineer at Google makes somewhere around $250k/year $125/working hour is probably a decent estimate.

On the other hand, the upper-bound is probably something like "how much money do we have?". After all, Von-Neumann was responsible for ground-breaking innovations in fields such as quantum theory, the development of nuclear weapons, and the invention of the digital computer. Having access to the world's smartest person might literally be a matter of national survival.

If you consider that the Manhatten Project cost about 1% of the GDP of the US, that equals $227Billion/year or about $25million/hour.

Interestingly, if AGI really requires 100 petaflops, this number is not too far from the actual cost of running such an AGI. Computing on a GPU is estimated to cost between $0.03 and $0.30, which is $3million-$30million/hour for our hypothetical AGI (I have no idea why the range is so wide).

This suggests that we might be nearing the moment when a Manhattan project to build an AGI is reasonable, but we are nowhere near the point where commercial applications of AGI are feasible*.

*throughout this post, ...
Sigmoids behaving badly: arXiv paper
by Stuart_Armstrong — 2021-09-20T10:29:20.736Z

Building on a previous post on the unpredictability of sigmoid curves, we have a paper on the arXiv explaining why it is so hard to predict their inflection/turning points until after it has happened.

Abstract:

Sigmoids (AKA s-curves or logistic curves) are commonly used in a diverse spectrum of disciplines as models for time-varying phenomena showing initial acceleration followed by slowing: technology diffusion, cumulative cases of an epidemic, population growth towards a carrying capacity, etc. Existing work demonstrates that retrospective fit of data is often impressive. We show that in time series data, the future fit tends to be poor unless the data covers the entire range from before to after the inflection point. We discuss the theoretical reasons for this: the growth data provides little information about the damping term (and vice-versa). As a consequence, forecasting with sigmoids tends to be very unreliable. We suggest some practical approaches to improving the viability of forecasting sigmoid models.
[Book Review] "The Alignment Problem" by Brian Christian
by lsusr — 2021-09-20T06:36:23.359Z

I came to this book with an ax to grind. I combed through page after page for factual errors, minor misrepresentations or even just contestable opinions. I spotted (what seemed like) omission after omission only to be frustrated just a few pages later when Brian Christian addressed them. In the Chapter 5: Shaping I thought I found a major mistake. Brian Christian addressed Skinnerian operant conditioning without addressing the real way we manages human groups: leading by example.

That's because he dedicated all of Chapter 7: Imitation to the subject. Thus, through gritted teeth, I reluctantly acknowledge that The Alignment Problem by Brian Christian is a fantastic book in all respects.

Despite my best efforts, Brian Christian even taught me lots of cool things about state-of-the-art machine learning. The Alignment Problem addresses advanced technical problems while being readable to non-technical people. This book would be a useful read both for activists who want to better understand public policy AND for aspiring engineers who want to get up to speed with machine learning. The only possible fault I can imagine with this book is that, since it depends so heavily on cutting-edge research, it might be rendered obsolete in a decade or two. Much of it mirrors the actual technical work I'm doing in machine learning.

The book starts with practical real-world problem that are happening right now. Most of the book is dedicated to explaining machine learning problems and their solutions. At the end it extrapolates on to the choices machine learning creates for our future.

Racist Machines

In 2015 Google famously released an image classifier that labelled Black people as "gorillas" because there were so few Black people in its training dataset. The good solution is to add more Black people to the training dataset. The fast solution is to keep using a biased algorithm and just cover up the most egregious errors. I don't know which approach Google went with but "three years later, in 2018, Wired reported that the label 'gorilla' was still manually deactivated in Google Photos."

I'm curious what animal I would get classified as if people who look like me were removed from Google Photos training dataset. (I hope it's a meerkat.) Alas, computer algorithms are already being used to make much more consequential decisions. Many of these decisions involve problems that can't be so...
Testing The Natural Abstraction Hypothesis: Project Update
by johnswentworth — 2021-09-20T03:44:43.061Z

I set myself six months to focus primarily on

the Natural Abstraction Hypothesis project

before stopping to re-evaluate. It’s been about six months since then. So, how has it gone?

This will be a “story telling” post, where I talk more about my research process and reasoning than about the results themselves. Be warned: this means I'm going to spout some technical stuff without explanation here and there, and in some cases I haven't even written a good explanation yet - this is a picture of my own thoughts. For more background on the results, the three main posts are:

The

intro post

for the overarching project, which I recommend reading.

Information At A Distance Is Mediated By Deterministic Constraints

, which I also recommend reading.

Generalizing Koopman-Pitman-Darmois

, which I do not recommend reading unless you want dense math.

Recap: The Original Plan

The

Project Intro

broke the Natural Abstraction Hypothesis into three sub-hypotheses:

Abstractability: for most physical systems, the information relevant “far away” can be represented by a summary much lower-dimensional than the system itself.
Human-Compatibility: These summaries are the abstractions used by humans in day-to-day thought/language.
Convergence: a wide variety of cognitive architectures learn and use approximately-the-same summaries.

That post suggested three types of experiments to test these:

Abstractability: does reality abstract well? Corresponding experiment type: run a reasonably-detailed low-level simulation of something realistic; see if info-at-a-distance is low-dimensional.
Human-Compatibility: do these match human abstractions? Corresponding experiment type: run a reasonably-detailed low-level simulation of something realistic; see if info-at-a-distance recovers human-recognizable abstractions.
Convergence: are these abstractions learned/used by a wide variety of cognitive architectures? Corresponding experiment type: train a predictor/agent against a simulated environment with known abstractions; look for a learned abstract model.

Alas, in order to run these sorts of experiments, we first need to solve some tough algorithmic problems. Computing information-at-a-distance in reasonably-complex simulated environments is a necessary step for all of these, and the “naive” brute-force method for this is very-not-tractable. It requires evaluating high-dimensional integrals over “noise” variables - a ...