trentmkelly/LessWrong-43k huggingface dataset:
Commentary on “AGI Safety From First Principles by Richard Ngo, September 2020” 
by Robert Kralisch — 2021-10-14T15:11:22.350Z

The intent of this document is to critically engage with the points and perspectives put forward in the post “AGI Safety From First Principles” by Richard Ngo. I will mostly take quotes from the post and comment on those, often involving the larger context of the section. For this reason it is advised to read the post by Richard Ngo first, before reading this commentary.

Please keep in mind that this is not supposed to be a balanced review. I agree with much of what the author says and focus on the points of disagreement or potential discussion.

_________________________________________________________________________

1 Introduction

“The key concern motivating technical AGI safety research is that we might build autonomous artificially intelligent agents which are much more intelligent than humans, and which pursue goals that conflict with our own.”
Agreed. While there are a number of concerns about machine learning involving bias, manipulation, moral decision making and/or automation, the key concern motivating technical AGI safety research should be the autonomous, misaligned agency of AGI.

_________________________________________________________________________

2 Superintelligence

2.1 Narrow and general intelligence

“The key point is that almost all of this evolutionary and childhood learning occurred on different tasks from the economically useful ones we perform as adults. We can perform well in the latter category only by reusing the cognitive skills and knowledge that we gained previously. In our case, we were fortunate that those cognitive skills were not too specific to tasks in the ancestral environment, but were rather very general skills.”

I believe this is an important point that we ought to engage with interdisciplinarily. For me, the notion of “Exaptation” from evolutionary biology comes to mind as a relevant concept to apply both to our understanding of the development of human cognition and the potential avenues (intended or not) for the development of artificial cognition.

_________________________________________________________________________

2.2 Paths to superintelligence

“Because of the ease and usefulness of duplicating an AGI, I think that collective AGIs should be our default expectation for how superintelligence will be deployed. “

I believe it makes sense to also consider the case of ...
Covid 10/14: Less Long Cvoid
by Zvi — 2021-10-14T14:20:01.250Z

The pandemic has become predictable, which is the world we much prefer to live in. Cases and deaths continue their slow but steady declines, vaccine approvals for boosters and children continue to slowly move forward. Mandates continue to make people angry but mostly work when used, while also not that often being used. Same old, same old.

The big news story was that one of the studies of Long Covid was revealed to be vastly overestimating frequency due to a methodological ‘error.’ It’s an important update.

Executive Summary

Cases and deaths declining as expected.
Approvals progressing slowly as expected.
Long Covid study had important error, see discussion later in post.

Let’s run the numbers.

The Numbers

Predictions

Prediction from last week: 560k cases (-10%) and 11,500 deaths (-8%).

Results: 547k cases (-12%) and 11,051 deaths (-12%).

Prediction for next week: 481k cases (-12%) and 9,835 deaths (-11%).

We are still in a low-variance situation, with no reason to expect any big surprises for at least several more weeks.

Deaths

The exact rate of decline here was a pleasant surprise, but we’ve gotten to the point where this is a lagging indicator, so the times when it is surprising are mostly about data gathering issues, either holidays or changes in the ways in which we do testing. Expect this trend to continue, and continue to mirror cases with a lag.

Cases

This is pretty much exactly what we would have expected. The Northeast number is slightly disappointing, others were slightly better than expected, but no real surprises.

Vaccinations

The rate of vaccinations continues to not change much, and it’s clear that the positive bump wasn’t the start of a surge. That continues not to tell us much about how much work the mandates are doing here. My impression is that the mandates that have teeth work, but that few of them have teeth.

In his podcast interview this week with FiveThirtyEight, former FDA Commissioner Dr. Scott Gottlieb asks where the cost-benefit analyses are on the vaccine mandates, since there don’t seem to be any. He estimates that we could use mandates to go from something like 80% adult vaccination rates to 84%, and asks how much that really accomplishes.

My answer is that this change would be worth quite a lot, but how much depends heavily on how we would react to the two scenarios.

C...
Is nuking women and children cheaper than firebombing them?
by lsusr — 2021-10-14T08:25:42.272Z

When I was a teenager, my favorite board game was the World War Two strategy game Axis & Allies. It taught me basic principles of adversarial strategy including Lanchester's laws, how to wield the law of large numbers and "chaos favors the underdog". But my favorite thing about the game is what it taught me about morality.

The game is balanced and rewards aggression. If you don't send everything you have to the front lines as fast as you can enemy forces will march through your homeland and those of your teammates. The game is long. If you lose your defeat will be protracted. It will suck. Axis & Allies is a desperate struggle for survival.

In Axis and Allies it's common to leave a single infantry unit on the front lines knowing they'll become casualties so you can destroy your enemy's more expensive tanks after the inevitable counterattack. Whether your enemy accepts prisoners of war or just shoots surrendering soldiers is irrelevant to this calculus.

Soldiers are fungible. Territory is fungible. Entire continents can be sacrificed and regained. Just holding off the enemy forces requires every trick available.

Axis & Allies supports a mechanic called "strategic bombing". Strategic bombing is a euphemism for bombing civilians. Strategic bombing is usually the wrong thing to do. Not because burning women and children alive is unethical. Strategic bombing is wrong because it does slightly less damage to an enemy's capacity for war than bombing their soldiers directly.

When I think about ethics I ask myself "If I was a 19-year-old in 1945 would I have refused to pilot a B-29 just because I was ordered to commit a war crime?"

I'm pretty sure the answer is no.

I'm not saying that bombing civilians is ever morally defensible. Or even that it was strategically effective. Just that, given my knowledge of history and my knowledge of my own personality I'm pretty sure that bombing civilians is what I would have done if I had been born in 1926 and assigned to the Air Force operating a B-29. I might've ended up on the Manhattan Project instead.

One-Sided Offensive Nuclear War

When discussing the ethics of nuclear weapons[1], it's easy to get drawn into calculations of how many Japanese lives they did(n't) save. Such perspective is a historical anachronism. When I'm playing Axis & Allies I throw all of my resources at the enemy as fast as I can. I don't ask "is ...
Compute Governance and Conclusions - Transformative AI and Compute [3/4]
by lennart — 2021-10-14T08:23:12.156Z

Cross-posted here on the EA Forum.

Transformative AI and Compute - A holistic approach - Part 3 out of 4

This is part two of the series Transformative AI and Compute - A holistic approach. You can find the sequence here and the summary here.

This work was conducted as part of Stanford’s Existential Risks Initiative (SERI) at the Center for International Security and Cooperation, Stanford University. Mentored by Ashwin Acharya (Center for Security and Emerging Technology (CSET)) and Michael Andregg (Fathom Radiant).

This post attempts to:

Briefly outline the relevance of compute for AI Governance (Section 6).

Conclude this report and discuss next steps (Section 7).

Epistemic Status

This article is Exploratory to My Best Guess. I've spent roughly 300 hours researching this piece and writing it up. I am not claiming completeness for any enumerations. Most lists are the result of things I learned on the way and then tried to categorize.

I have a background in Electrical Engineering with an emphasis on Computer Engineering and have done research in the field of ML optimizations for resource-constrained devices — working on the intersection of ML deployments and hardware optimization. I am more confident in my view on hardware engineering than in the macro interpretation of those trends for AI progress and timelines.

This piece was a research trial to test my prioritization, interest and fit for this topic. Instead of focusing on a single narrow question, this paper and research trial turned out to be more broad — therefore a holistic approach. In the future, I’m planning to work more focused on a narrow relevant research questions within this domain. Please reach out.

Views and mistakes are solely my own.

Previous Post: Forecasting Compute

You can find the previous post "Forecasting Compute [2/4]" here.

6. Compute Governance

Highlights

Compute is a unique AI governance node due to the required physical space, energy demand, and the concentrated supply chain. Those features make it a governable candidate.

Controlling and governing access to compute can be harnessed to achieve better AI safety outcomes, for instance restricting compute access to non-safety-aligned actors.

As compute becomes a dominant factor of costs at the frontier of AI research, it may start to resemble high-energy physics research, where a significant amount of the budget is spent on infrastructur

...
Book Review: How To Talk So Little Kids Will Listen
by weft — 2021-10-14T06:56:23.815Z

Way back in the ancient times of 1980, Adele Faber and Elaine Mazlich wrote "How To Talk So Kids Will Listen and Listen So Kids Will Talk" (henceforth "Kids"). It turns out that kids and adults operate with mostly the same internal machinery, so you could perhaps more accurately call it "How To Talk So [Humans] Will Listen and Listen So [Humans] Will Talk" . This seminal work proved so useful that 42 years later it is still one of the most recommended parenting books¹, and is widely regarded as useful for adult-adult communication as well.

40 years ago is a long time though. It is long enough for Joanne Faber, daughter of the original author Adele Faber to grow up, have kids of her own, and put the skills her mother raised her with to good use herself. And in the much-less-distant past of 2017 Joanne, with co-author Julie King, wrote what could be considered a modernized update of her mother's work, titled "How To Talk So Little Kids Will Listen" (henceforth "Little Kids").

The core principles are the same, but the update stands on its own. Where the original "Kids" acts more like a workbook, asking the reader to self-generate responses, "Little Kids" feels more like it's trying to download a response system into your head via modeling and story-telling. I personally prefer this system better, because the workbook approach feels like it's only getting to my System 2 (sorry for the colloquialism). Meanwhile being surrounded with examples and stories works better for me to fully integrate a new mode of interaction. In fact, if I truly wanted to integrate it I would want 3 more books of anecdotes, a TV show, and a fiction book series to model after.

Structure and Philosophy

"Little Kids" is very goal-oriented. It explicitly asks "What are you trying to accomplish with this communication to your kids?", and recognizes that oftentimes the answer is behavioral change, for example "I want Little Alice to stop hitting when she's angry."

After focusing on the goal, they then examine the effects of various types of responses. Situations faced by children are often hard to empathise with as an adult (e.g. really not wanting to leave the toy store even though it is time to go to dance lessons). "Little Kids" therefore creates scenes that are the adult equivalent of common childhood scenarios. Then a variety of responses are presented such as lecturing, you-poor-t...
Creating a truly formidable Art
by Valentine — 2021-10-14T04:39:16.641Z

Over this last week, I and several other folk from CFAR's past gathered. We were doing a kind of post-mortem on the last decade.

I mostly haven't thought explicitly about Less Wrong style rationality for the last three years. It would come up in conversation every now and again, and I engaged with the "best of 2018" review process for a couple of my posts. But it just isn't where I've been focusing anymore.

This week of immersion and reflection stirred some old circuitry in me. It was beautiful and fascinating to witness how who I am today dances with old ways of thinking and being from my CFAR days.

As I left that event, I noticed I could clearly feel an ember glowing in me. I remembered the Beisutsukai, and the sense that more is possible.

I also remembered the well-worn ache of repeated failure and defeat from trying to create these things, and the endless intellectual conversations that turned out to go nowhere as we struggled to birth any hint of the full Art.

But I've learned something these last three years about how to navigate that kind of failure, and how to deeply honor the true essence of an inspiration.

So I find myself in the amusing position of feeling how someone might actually create truly good rationality training. It's amusing because I doubt this is my gift to give the world. I'm doing something closely related, but different. Far too mystical for the right aesthetic.

I'd like to attempt a translation. Partly for myself, because writing out these things brings me clarity. But partly as an invitation for the souls here who can feel the call and might do something with it.

I think the world would be more beautiful, and more fun, with real Beisutsukai.

Some Possible Ingredients of the Art

Embodying the Void

The first concept I'll point at here is noise. Not in the information-theoretic sense (i.e., the opposite of signal). More in the sense of how it's hard to be calm and composed in a loud environment, especially after you've been startled. The way the emotional noise of living with an abusive partner can make it hard to notice what's going on and decide to leave. The way the feeling of alarm from outrage porn crowds out clear thinking and perception in favor of fueling the inner mental fire. The way noisy thoughts about already knowing the answer make it hard to really listen to the words and concepts someone else is saying.

Most tools for think...
Book Review: Denial of Death
by PatrickDFarley — 2021-10-14T04:28:25.041Z

Denial of Death is the 1973 summation of anthropologist Ernest Becker's life's work studying human nature, building upon the work of the great psychologists of the 20th Century. It basically aims to be a grand unifying theory of psychology, and against all odds it kind of succeeds.

I came across DoD on a forum under the topic, "books that drastically changed your life," or something like that. Since then I've only encountered a few other references to this book in my life, coming from quite different types of people but all glowing. There was the late, great Norm Macdonald, a brilliant comedian known for his esoteric disposition and his ability to create humor out of anything—he references the book as pretty important to his worldview. There's my yogi friend who's very into intuition and spirituality and swears by it as a way to understand herself. And there's me—analytical, materialist—who has also found tremendous value in it. I hope you will, too.

TL;DR

Becker's thesis is that humans have a paradoxical dual nature (divine vs. animal), the tension of which causes us to experience psychological terror. The presence of that terror makes us “partialize” the world by embedding ourselves within relatively safe pockets of it—we set up a game-within-the-game to play. This lets us pursue our "heroic" projects without having to face the scary limitations and risks that come with our animal nature. The pockets we hide ourselves in include the behaviors and thoughts we adhere to, so all psychological neuroses as well as "normal" human character are contextualized as various responses to a deep fear of that animal nature. This context gives us a new schema for understanding neurosis, mental health, personality, etc.

I promise that this book is not all fluffy academic jargon. This is a book that's heavy with new concepts, but all of the above are meant to pertain to concrete things that you and I experience.

If there's a call to action to be gleaned from this, it's something like: You should seek to take in more of life, more chaos, but without destroying yourself. And you shouldn't judge or condemn people who live simple lives or rely on blatant repressions—their experience is as heroic as yours.

Context of the work

Cultural Anthropologist Ernest Becker wrote this Pulitzer Prize-winning book at the end of a long career as an anthropology professor. Published in 1973, this bo...
Book Review: The End of Average
by Vanilla_cabs — 2021-10-14T01:49:42.346Z

We all strive to be like the others - more exactly, like the others, but better. (Chapter 2)

Note: the quotes have been translated back from the French translation, so they probably slightly differ from the original text.

The End of Average is a 250-ish pages book published in 2016 by academic Todd Rose.

The main points

Rose describes the rule of average as the ubiquitous use, in major decisions that impact our lives, of a norm based on the average. From our grades at school to our career options, all the way through medical diagnoses, important institutions compare individuals with the average, with 2 goals:

assessing normality: what differs too much from the average is considered abnormal. For example, the developmental path for learning how to walk was determined in the 1950s by observing babies and averaging the result to get the "normal development path". Any child that diverges too much from that path will be closely looked for abnormalities.
assessing skill: a single variable is assumed to correlate with a large number of desired dimensions, and becomes very important in people's lives. For example, a student's understanding of a lesson is graded as a single note. Then, at the end of the year or semester, the notes in a class are summed up as a single average that is supposed to reflect the value of the student in that class. The higher a student is graded, the more intelligent and talented they are deemed, and the better their options.

In both cases, claims Rose, by averaging the data over the samples, important information is lost. It turns out that there are no less than 25 viable development paths for learning to walk, where the usual steps are taken in reverse, simultaneously, or even skipped. As for school grading, Rose picks examples both from research and from his personal experience to show that students that fail in the standard school system can thrive when the lessons are more adapted to their personal profile.

Born with modern statistics and acclaimed as a major methodological progress, the average-ist approach's limitations were overlooked for a long time, claims Rose. But now, with the help of computers' processing power, a more personalized treatment of information is possible and desirable: the individualist approach.

the average-ist approach: summing up the data, then analysing it; commits the ergodic error (attributing to a member a pr

...
In Wikipedia — reading about Roko's basilisk causing "nervous breakdowns" ...
by ThurstonBT — 2021-10-13T23:12:09.190Z

I recently came upon the story of "Roko's basilisk". Heretofore I'd not seen the topic. Reading about it in Wikipedia[1] I saw "Discussion of Roko's basilisk was banned on LessWrong for several years because Yudkowsky had stated that it caused some readers to have nervous breakdowns".[1]

That Yudkowsky censored mention of Roko's basilisk because it "caused some readers to have nervous breakdowns" (henceforth THE CLAIM) seemed very strange to me. So I asked in a Berkeley CA 'rationalist community' forum — "This story of censorship of Roko's basilisk and it causing "nervous breakdowns" strikes me me a bizarre; Is this some sort of joke?".
The answers to this question, some of which were angry and emotional, amounted to:
1) THE CLAIM is false.
2) No one could provide evidence that THE CLAIM is false sufficient to meet Wikipedia's evidentiary standards so as to rebut or remove the THE CLAIM in Wikipedia..
3) No one could provide a citation to a specific denial from Yudkowsky that THE CLAIM did not figure into the decision to censor mention of Roko's basilisk.
In the interest of truth-seeking ...
... if the THE CLAIM is false then I'd like to obtain evidence to that effect and correct Wikipedia.

... if THE CLAIM is true then it brings to mind some potentially unkind questions about the psychological heath of a seemingly significant portion of the 'rationality community'.
Please note that my questions here bears only on THE CLAIM (ie. "caused some readers to have nervous breakdowns") and not any of the issues associated with Roko's basilisk, eg. claims of infohazard.
Even if the "caused some readers to have nervous breakdowns" is a hyperbolic description, with no actual "nervous breakdowns" occurring, the notion that censorship needed to be employed to protect emotionally brittle persons in the 'rationality community' from a thought experiment reflects poorly on the community.

Ref.

{1] https://en.wikipedia.org/wiki/LessWrong#Roko's_basilisk