trentmkelly/LessWrong-43k huggingface dataset:
What is a disagreement you have around AI safety?
by tailcalled — 2023-01-12T16:58:10.479Z

I was once discussing Aumann's agreement theorem with a rationalist who wanted to use it to reduce disagreements about AI safety. We were thinking about why it doesn't work in that context when (I would argue) Aumann's agreement theorem applies robustly in lots of other contexts.

I think I can find out (or maybe even already know) why it doesn't apply in the context of AI safety, but to test it I would benefit from having a bunch of cases of disagreements about AI safety to investigate.

So if you have a disagreement with anyone about AI safety, feel encouraged to post it here.
Reward is not Necessary: How to Create a Compositional Self-Preserving Agent for Life-Long Learning
by Roman Leventov — 2023-01-12T16:43:42.357Z
Link: https://arxiv.org/abs/2211.10851

There is an idea in the AI safety community that expected utility maximisation is the "limit" of agency.

I think the linked paper by Thomas J. Ringstrom demonstrates that this idea is wrong.

Explainer Twitter Thread by the author.

Abstract:

We introduce a physiological model-based agent as proof-of-principle that it is possible to define a flexible self-preserving system that does not use a reward signal or reward-maximization as an objective. We achieve this by introducing the Self-Preserving Agent (SPA) with a physiological structure where the system can get trapped in an absorbing state if the agent does not solve and execute goal-directed polices. Our agent is defined using new class of Bellman equations called Operator Bellman Equations (OBEs), for encoding jointly non-stationary non-Markovian tasks formalized as a Temporal Goal Markov Decision Process (TGMDP). OBEs produce optimal goal-conditioned spatiotemporal transition operators that map an initial state-time to the final state-times of a policy used to complete a goal, and can also be used to forecast future states in multiple dynamic physiological state-spaces. SPA is equipped with an intrinsic motivation function called the valence function, which quantifies the changes in empowerment (the channel capacity of a transition operator) after following a policy. Because empowerment is a function of a transition operator, there is a natural synergism between empowerment and OBEs: the OBEs create hierarchical transition operators, and the valence function can evaluate hierarchical empowerment change defined on these operators. The valence function can then be used for goal selection, wherein the agent chooses a policy sequence that realizes goal states which produce maximum empowerment gain. In doing so, the agent will seek freedom and avoid internal death-states that undermine its ability to control both external and internal states in the future, thereby exhibiting the capacity of predictive and anticipatory self-preservation. We also compare SPA to Multi-objective RL, and discuss its capacity for symbolic reasoning and life-long learning.

Relevant paragraph:

An important difference between MORL (Multi-Objective Reinforcement Learning) and HTGMDP (Hierarchical Temporal Goal Markov Decision Process) agents is that MORL agents represent the alternative objectives as primal value functions for policies on X ,

...
ChatGPT struggles to respond to the real world
by Alex Flint — 2023-01-12T16:02:56.358Z

I wanted to test whether ChatGPT could respond to situations as they arise in the real world, so I asked it to guide me through making a turmeric latte. It had no problem giving me a nicely formatted recipe but it really struggled to guide me through the process of making the latte.

The rules I set for this experiment was that I would perform any instruction that I could perform in less than 10 seconds, so "fetch a saucepan" is fine, "go to the fridge" is fine, "put one cup of milk into the saucepan" is fine if I have the milk and the saucepan already in front of me, but if the milk or saucepan were not in front of me I would respond with something like "I don’t have the milk in front of me. Can you guide me through finding some milk?"

I insisted that I be given just one next instruction, not a list of numbered instructions, unless I could perform the whole set in less than 10 seconds total. The reason is that I wanted to test whether ChatGPT could respond to difficulties that arose organically in the process of making the latte (I did not synthetically create any difficulties). In order to do that, I wanted ChatGPT to give me one instruction and then I wanted to respond with the outcome. For example, here is a part of a dialog in which I had a (human) friend guide me through the same task via text:

Friend: Select a small pot.

Me: Ok I have chosen the pot but I am still standing looking at the pot while it is sitting on the bottom shelf

Friend: Bend down and retrieve the pot.

Me: Done!

Friend: turn around and place the pot on an available burner.

Me: Done!

Friend: turn to your right, walk three steps, turn to your left, go to the metal fridge on the right.

Me: Done!

Overall, ChatGPT really struggled to give one instruction at a time. I spent much of the dialog insisting on this:

Hello! I’d like to make a turmeric latte using a turmeric and cinnamon mix that my friend left when she visited me recently. Can you guide me step by step in how to make a latte using this?

Sure! Here's a recipe for a turmeric latte that you can make at home:

[11 lines of nicely formatted recipe]

I hope you enjoy your latte! Let me know if you have any questions or need further guidance.

Ok thank you for that but I was wondering if you could walk me through one step at a time

Certainly! Here are the steps for making a turmeric latte, one step at a time:

...
Covid 1/12/23: Unexpected Spike in Deaths
by Zvi — 2023-01-12T14:30:00.756Z

Quite a lot of deaths got reported by quite a lot of different states this week. Case numbers did not spike, and instead actively declined. What’s going on? It is right after Christmas and New Years, which means the least reliable reporting of the year, and no one cares much anymore – Indiana and Illinois didn’t bother reporting anything on either front – so the obvious explanation is this is a lot of backlog dumping with the new year. It could however also be that XBB.1 is doing its thing and no one is bothering to test anymore. Error bars are very high for next week.

What I am not worried about, given the case numbers, is that this wave is going to get out of hand. Even if things are worse than we thought right now, it is a bounded worse, and mostly should not much alter our behaviors.

Executive Summary

Huge spike in deaths, likely holiday reporting issues. Some chance it reflects a lot of unreported XBB.1 cases. Cases actively declined this week.

New ‘Twitter files’ on vaccine information is a hack hit job, nothing to see there.

The origin of the athlete death claims is dumber than you expected.

Let’s run the numbers.

The Numbers

Predictions

Predictions from Last Week: 450k cases (+7%) and 2,950 deaths (+7%)

Results: 367k cases (-13%?) and 3,666 deaths (+36%!!)

Predictions for Next Week: 385k cases (+5%) and 3,100 deaths (-15%), with a very large helping of what the heck do I know and a lot of this is reporting issues and I don’t see the value in doing a deep dive to correct for them. Huge error bars.

A number of states reported what looks like multiple weeks of backlogs of deaths, numbers that were absurdly high. Indiana and Illinois did not report at all. I decided to not adjust any of it.

Instead I am simply noting that this week’s number doubtless reflects real deaths, even more than usual following a holiday, is clearly not ‘fully real.’

Case numbers actively declined, even if you were to add back in Indiana and Illinois, which I definitely did not expect. In Florida they almost doubled even after being smoothed out.

Deaths

Cases

Physical World Modeling

New bivalent booster papers from David Ho, Barouch Lab and Dr. Paul Offit. Offit thinks only the vulnerable need to be boosted. On the flip side, Eric Topol comes around, sees bivalent booster working better than expected. None of it much changed my views – ...
[Linkpost] Scaling Laws for Generative Mixed-Modal Language Models
by Amal  — 2023-01-12T14:24:00.921Z
Link:  https://arxiv.org/pdf/2301.03728.pdf

In this paper authors explore the scaling properties of mixed-modal generative models, discovering new scaling laws that unify the contributions of individual modalities and the interactions between them. I find most interesting that they have found so-called competition barrier - when training with multiple modalities, after a certain number of parameters/data, the loss is smaller than if the modalities were trained independently. This seems to predict cross-modal transfer that was sought after but not found (yet) with GATO.
ea.domains - Domains Free to a Good Home
by plex — 2023-01-12T13:32:19.451Z

Tl;dr: I’ve set up

a database of domains at ea.domains

which are

free to a good home

, to prevent them being squatted and blocked for use. You can add domains you control to it using

this form

.

Since my well received post on setting up an Anti-squatted AI x-risk domains index, I’ve been picking up more and more domains, talking to other domain holders, and building an interface for viewing them. I’ve also put a[1] few of them[2] to good use[3] already!

Also a big thanks to

Ben West

for sharing 215 of CEA's parked domains, which EA projects are welcome to request use of by emailing

tech@centreforeffectivealtruism.org

. He also offered for CEA to be the custodian of the domains I bought, with the condition that they will be pointed to nameservers I control and that they will return ownership of them for EA-aligned use on request, unless they’re in active use by another EA project. This will save me from having to pay upkeep, which will help make this more sustainable. He's open to extending this offer to holders of other relevant domains.

If you'd like to use one of these domains, then message the contact person specified. They each have different policies for handing over the domain, but a good standard is that they'll point the domain towards your servers on request, and hand it over for free if and when you have built something useful at that location.

Here’s the top 40 domains I’m most excited about, but go check the full list:

Domain	Possible use	Contact

aisafety.global

AI Safety Conference	hello@alignment.dev

existential-risks.org

High quality explanation?	tech@centreforeffectivealtruism.org

ontological.tech

New org?	hello@alignment.dev

existential.dev

New org?	hello@alignment.dev

epistemic.dev

New org?	hello@alignment.dev

aisafety.tools

Directory of resources?	jj@aisafetysupport.org

aisafetycareers.com

esben@apartresearch.com

agenty.org

New org?	hello@alignment.dev

x-risks.com

drewspartz@nonlinear.org

aisafety.events

jj@aisafetysupport.org

aisafety.me

jj@aisafetysupport.org

effectivealtruism.ventures

EA entrepreneurs or impact investing group?	donychristie@gmail.com

aisafetybounties.com

esben@apartresearch.com

aisafety.degree

AI Safety PhD cohorts	hello@alignment.dev

alignment.careers

80k said they were happy for others to join the careers advice space	hello@alignment.dev

xrisk.fund

x-risk specific funding organization?	hello@alignment.dev

a

...
VIRTUA: a novel about AI alignment
by Karl von Wendt — 2023-01-12T09:37:21.528Z

I’ve written a novel about AI alignment in German and translated it into English.

Edit: I'm going to publish the final version on Amazon KDP on May 25. I have therefore removed the links to the previous version. If you want a free copy of the ebook or a PDF, please write me a direct message. Thanks to all who helped me improve the final version with their comments and feedback (some of you are mentioned in the book).

Teaser:

When psychologist Daniel unexpectedly gets a job offer from the Berlin subsidiary of Mental Systems, one of the leading technology companies in the world, he’s excited. But he soon finds out that behind the friendly facade, there’s enormous pressure because Mental Systems is in a global race for the development of the first artificial general intelligence. With their amazing AI Virtua, they seem to be in the lead. But some are concerned about the rapid progress because it seems unclear whether the trust the management puts in Virtua is really justified. When a devastating hacker attack seems to have destroyed years of research and the lead of the AI safety team is gone missing, Daniel sets out on a quest to find out what really happened, and why. But will he be able to prevent a disastrous turn of events that could destroy the future of humanity?

Some background:

I’m a professional novelist with more than 50 books published in German under the pen name “Karl Olsberg”. My first novel “Das System” about an out-of-control AI, published in 2007, became a national bestseller. At the time, I had no idea how an AI could really get out of control, so the novel was highly speculative science fiction. Since then, I have seen reality coming uncomfortably close to the scenarios in my books. Since my participation in the AI safety camp last year, my timeline expectation for AGI has shrunken significantly. VIRTUA is probably my most realistic AI novel yet. I still hope it will remain science fiction.

At the next AI safety camp, I’ll lead a project with the goal of developing realistic and detailed failure stories. If you’re interested in collaborating, you can apply here until January 19th.

Edit: At about the same time I posted this, blaked has published an impressive post describing his own experience with an LLM that "hacked his mind" in a similar way to what happens to Jerry in the book.
Iron deficiencies are very bad and you should treat them
by Elizabeth — 2023-01-12T09:10:01.240Z

In brief

Recently I became interested in what kind of costs were inflicted by iron deficiency, so I looked up studies until I got tired. This was not an exhaustive search, but the results are so striking that even with wide error bars I found them compelling. So compelling I wrote up a post with an algorithm for treating iron deficiency while minimizing the chance of poisoning yourself. I’ve put the algorithm and a summary of potential gains first to get your attention, but if you’re considering acting on this I strongly encourage you to continue reading to the rest of the post where I provide the evidence for my beliefs.

Tl;dr: If you are vegan or menstruate regularly, there’s a 10-50% chance you are iron deficient. Excess iron is dangerous so you shouldn’t supplement blindly, but deficiency is easy and cheap to diagnose with a common blood test. If you are deficient, iron supplementation is also easy and cheap and could give you a half standard deviation boost on multiple cognitive metrics (plus any exercise will be more effective). Due to the many uses of iron in the body, I expect moderate improvements in many areas, although how much and where will vary by person.

Note that I’m not a doctor and even if I was there isn’t good data on this, so it’s all pretty fuzzy. The following is an algorithm for treating iron deficiency that I’ve kludged together from various doctors. I strongly believe it is a lot better than nothing on average, but individuals vary a lot and you might be unlucky.

Take a serum ferritin test. If you have a doctor they will almost certainly say yes to a request, or you can order for yourself at walkinlab.com

If your results show a deficiency (<20ug/L), increase iron intake through diet or supplements such as Ferrochel, taking the default dose once per day, with a meal.

The definition of deficiency can vary by study, lab and goal. I picked <20ug/L because it’s the highest level I have concrete evidence is insufficient, but personally believe people are likely to benefit from iron beyond that and am taking pills accordingly.

If you experience negative effects after taking the pills, stop immediately. Give yourself a week to recover, then you can try other brands, be more careful to eat with a full meal, etc.

If you are experiencing the symptoms of iron poisoning (listed below), stop pills and see

...
Nonstandard analysis in ethics 
by Alok Singh — 2023-01-12T05:58:54.892Z
Link: https://nickbostrom.com/ethics/infinite

INFINITE ETHICS

(2011) Nick Bostrom
Future of Humanity Institute
Faculty of Philosophy & Oxford Martin School
Oxford University
[Analysis and Metaphysics, Vol. 10 (2011): pp. 9-59] [pdf]
[This is the final version. (Earlier versions published as electronic preprints: 2003, 2005, 2008, 2009)]
www.nickbostrom.com

ABSTRACT
Aggregative consequentialism and several other popular moral theories are threatened with paralysis: when coupled with some plausible assumptions, they seem to imply that it is always ethically indifferent what you do. Modern cosmology teaches that the world might well contain an infinite number of happy and sad people and other candidate value-bearing locations. Aggregative ethics implies that such a world contains an infinite amount of positive value and an infinite amount of negative value. You can affect only a finite amount of good or bad. In standard cardinal arithmetic, an infinite quantity is unchanged by the addition or subtraction of any finite quantity. So it appears you cannot change the value of the world. Modifications of aggregationism aimed at resolving the paralysis are only partially effective and cause severe side effects, including problems of “fanaticism”, “distortion”, and erosion of the intuitions that originally motivated the theory. Is the infinitarian challenge fatal?

The challenge
1.1. The threat of infinitarian paralysis
When we gaze at the starry sky at night and try to think of humanity from a “cosmic point of view”, we feel small. Human history, with all its earnest strivings, triumphs, and tragedies can remind us of a colony of ants, laboring frantically to rearrange the needles of their little ephemeral stack. We brush such late-night rumination aside in our daily life and analytic philosophy. But, might such seemingly idle reflections hint at something of philosophical significance? In particular, might they contain an important implication for our moral theorizing?
If the cosmos is finite, then our own comparative smallness does not necessarily undermine the idea that our conduct matters even from an impersonal perspective. We might constitute a minute portion of the whole, but that does not detract from our absolute importance. Suppose there are a hundred thousand other planets with civilizations that had their own holocausts. This does not alter the fact that the holocaust that humans cau

...
FFMI Gains: A List of Vitalities
by porby — 2023-01-12T04:48:04.378Z

(This was mostly written as a part of a quick writing session at the Lurkshop. Thanks to Evgeny, whose handle I do not know, for some refinements and suggestions!)

Most of the complexity in fitness- meal timings! ice baths! macro-meso-microcycle periodization! buy my special turbopowderjuice!- is either of questionable value or is about squeezing out that last few percent that a professional athlete has to care about.

For the rest of us, it mostly doesn't matter.[1]

Here are a bunch of short and simple tips accumulated across my years of gradually doing less dumb things. This list won't include much in the way of specific movement recommendations and won't talk much about managing weight loss; it's mostly focused on the Meta of Gains.

Do not hurt yourself

you cannot become Swole if you are swollen
different movements have different risk profiles; stuff that feels weird, unstable, or painful should be red flags (these will vary by individual and over time)

Being overweight and weak hurts in ways that fade into the background and you might not even know there is a comfier way of being; being pathologically weak hurts in ways that you notice now and it ruins your quality of life.[2]
Any exercise is transformatively better than no exercise

a 67 year old lady with heart problems and arthritis can go from constant pain and being barely able to walk to weighted squats, with only <2 sets a day, in 2 years, without dietary changes: strength matters[3]

Doctors often underemphasize strength training in their advice to patients.[4] They tend to advise easy things like walking. That's better than nothing[5], but you can go beyond that and get major benefits!
Mobility is good, but not an unalloyed good.

target strength over range of motion- it's good for your various strings to not snap as soon as a joint goes a little beyond its comfort zone, but you need to be able to control the joint in that position too.
my joints are VERY loosey goosey by default; it can make for uncomfortable party tricks, but it's much less fun when my hip gets slightly misaligned or my knee flops out mid-step and then crushes some soft tissues when weight returns. Getting stronger has made these incidents much, much rarer!

If a type of training makes you feel exhausted and terribad, you may come to dread it, and then may procrastinate or not do it at all.[6] Don't chase a path of ostensibly maximu

...
[Linkpost] DreamerV3: A General RL Architecture
by simeon_c — 2023-01-12T03:55:29.931Z
Link: https://arxiv.org/pdf/2301.04104v1.pdf

Here's a link towards DreamerV3, a new model from DeepMind that can be trained on a bunch of different tasks (including a simplified version of Minecraft) and outperform more narrow models. Link: https://arxiv.org/pdf/2301.04104v1.pdf

The most surprising bits are that:

The type of tasks they train it on is fairly diverse
Data efficiency scales with the number of parameters
They so far haven't scaled it that far and got pretty good results
Microsoft Plans to Invest $10B in OpenAI; $3B Invested to Date | Fortune 
by DragonGod — 2023-01-12T03:55:10.248Z
Link: https://fortune.com/2023/01/11/structure-openai-investment-microsoft/

Unpaywalled

Excerpts

If all goes according to OpenAI’s financial plans, Microsoft will close a $10 billion investment deal into the artificial intelligence startup before the end of this month, as Jeremy Kahn and I reported yesterday and according to documents seen by Fortune.

...

Microsoft’s bet on OpenAI appears to be even bigger than was previously known. The documents suggest that, prior to this deal, Microsoft had already poured $3 billion into the company—$2 billion more than has been publicly reported. If the current deal is completed at the figures being discussed, the cap table in the documents states that Microsoft will have contributed a total of $13 billion in capital to OpenAI, underscoring how important it believes the technology behind ChatGPT and DALL-E 2 is to its future.

...

Microsoft would be able to make as much as $92 billion from its collective investment, and venture capitalists that participate in the tender offer would be able to garner up to $150 billion. (An OpenAI spokeswoman declined to comment for this story, and a Microsoft spokesman didn’t respond to a request for comment.)

...

Microsoft will receive preferential treatment when it comes to OpenAI profits. The documents lay out how investors will be reimbursed once OpenAI starts posting a profit. “First close partners” will be reimbursed their principal first (it’s unclear whether “first close partners” refers to OpenAI’s early investors, Khosla Ventures and Reid Hoffman’s foundation, or other subsequent investors in the company). Once that has happened, 75% of OpenAI’s profits will flow directly to Microsoft until the sum that Microsoft invested in OpenAI is reached. Here is a graphical representation of how the economics are structured:

While the terms look like a win-win for Microsoft, it could end up being quite a while before Microsoft, or any of the other investors, see a meaningful return on that investment. Documents show that, as of the end of last year, OpenAI was projecting a loss of more than $508 million for 2022. The company has projected $1 billion in revenue in 2024, as was first reported by Reuters, but it’s unclear what it expects its costs to be in the years ahead. According to the documents, OpenAI expected that its costs in 2022 would total somewhere around $544.5 million.

Prediction Markets

@TetraspaceWest created a manifold ...