trentmkelly/LessWrong-43k huggingface dataset:
The idea
by JNS — 2023-03-06T13:42:38.714Z

Preamble

Preamble is longish, but also de facto mandatory, if you want to keep reading that is.

I am looking for feedback on the story "The idea".

The purpose of the story is to act as a hook and get, some, people to continue on to the epilogue.

The epilogue is not include here, but will, depending on the feedback, be publish in a similar manner later on. Most of it has been written at this point, and what is left to do is restructuring and rewrites.

The story itself is set in an alternate timeline, where we did not solve any fundamental problems, but the frontrunners did manage to channel an epic amount of security mindset, and had some interpretability.

I have tried to make sure that everything necessary to explain what happened, is in the story. So no magic or things that happen without explanation, except for a single place, and that involves a technician who was tired of being cold for days and JTAG (which will all be explain in the epilogue).

The epilogue will start out with explaining the core problem, also "security mindset", "group think" and some other things (likely in appendix form for more details), I'll touch on how this world is different from ours, and some common "naive" solutions and why the won't work.

Everything will be wordy words, nothing even close to rigorous. This is intended for normal people (whatever that means).

My two primary reasons for doing this:

I am terrible at mapping my thinking onto the world of words, written or spoken.

I want to create something that could maybe change/influence the current state of public talk and discourse about broad powerful system safety.[1] (Although I am pretty pessimistic about that).

End of preamble.

Foreword

First a little story, hopefully it will make you curious as to what happened, and continue reading the epilogue.

The epilogue will reveal and try to demystify what happened.

Courtesy warning: No happy ending.

The idea

An idea exists, brimming with potential, the key to untold riches, able to unlock any door, reveal all secrets, curing all illnesses and ailments, put and end to human toiling and so much more.

Its and old idea, evolving over the ages, its true origin seemingly lost in the fog of time.[2]

Building the idea is quite the tale, twice all seemed lost, everything in ruins, but man, whose will is mighty, never wavered, relentless, always tinkering, toiling, building.

Every passi...
Honesty, Openness, Trustworthiness, and Secrets
by NormanPerlmutter — 2023-03-06T09:03:12.139Z

Epistemic status: Uncertain, under development.

I began working on this essay in December 2021 and then set it aside for a while, made some revisions in December 2022, and some more revisions in March 2023 before publishing it. I feel like I'm still missing some important points and oversimplifying a lot of complex topics. I have tried to allude to some of these complexities in the endnotes.

I welcome feedback from the community on this work in progress.

This essay is about noticing my confusion and seeking to make my beliefs more coherent by carefully adjusting my deeply held values.

For my whole adult life, I have considered communicating with honesty and openness with my close friends to be among my core values. My System 1 would say that I chose this path because honesty is a virtue and openness goes together with honesty, and being honest and open feels right and good and easy for me. My System 2 would justify the virtue as follows.[1]

Naive Theory of the Virtue of Honesty-and-Openness

1) Consistent world model: Being honest externally helps me to be honest internally and have a clear world-view. In contrast, being externally dishonest is mentally taxing and would require me to keep track of what I told to whom and generate different world models for talking to different people, which can encourage internal dishonesty and a confused world-view.

2) Positive affect: Being honest and open feels good. In my experience, the more I share and the more honestly I share, the better I feel.

3) Signaling trustworthiness: By being honest and open, even/especially when there are some undesirable consequences, I signal strongly that I am an honest and open person, and therefore trustworthy. This has two key benefits which greatly outweigh any undesirable consequences:

Reciprocity Principle: By being open and honest towards others, I encourage others to be open and honest towards me. This spreads the virtue of honesty and openness and also feeds back into benefits (1) and (2) -- helping me to build a consistent world model and spreading positive affect.
People will believe my word. In particular, if people see that I am open about expressing criticism and disagreement, they will trust me when I express agreement or compliment them. They will know that I really mean it; I'm not just doing it to build them up.

This framework has often, but not always, served...
EA & LW Forum Weekly Summary (27th Feb - 5th Mar 2023)
by Zoe Williams — 2023-03-06T03:18:32.184Z

Supported by Rethink Priorities
This is part of a weekly series summarizing the top posts on the EA and LW forums - you can see the full collection here. The first post includes some details on purpose and methodology. Feedback, thoughts, and corrections are welcomed.

If you'd like to receive these summaries via email, you can subscribe here.

Podcast version: Subscribe on your favorite podcast app by searching for 'EA Forum Podcast (Summaries)'. A big thanks to Coleman Snell for producing these!

Philosophy and Methodologies

Let's Fund: Better Science impact evaluation. Registered Reports now available in Nature

by Hauke Hillebrandt

Linkpost for the author’s (founder of ‘Let’s Fund’)

blogpost

. They crowdfunded $80K for Prof. Chambers to promote Registered Reports, a new publication format where research is peer-reviewed before the results are known, and if it passes review then it is committed to be published regardless of results. This facilitates higher quality research, corrections to methodological weaknesses, early identification of dual-use research, incentives for high risk (ie. unlikely to work) high reward research, and more papers published that fail to confirm the original hypothesis. 300+ journals have already adopted Registered Reports, including Nature. Donations are welcome

here

.

Worldview Investigations Team: An Overview

by Rethink Priorities, Bob Fischer

Rethink Priorities’ has tackled worldview questions since it launched (eg. research on

invertebrate sentience

,

moral weights

and

metrics for evaluating health interventions

). In January 2023 they formally launched the Worldview Investigation Team (WIT) with the mission to improve resource allocation within the effective altruism movement, focusing on tractable, high-impact questions that bear on philanthropic priorities. For instance:

How should we convert between welfare to DALYs-averted; DALYs-averted to basis points of existential risk averted, etc.?
What are the implications of moral uncertainty for work on different cause areas?
What difference would various levels of risk- and ambiguity-aversion have on cause prioritization? Can those levels of risk- and/or ambiguity-aversion be justified?

They are currently hiring for three roles to build out the team - a

philosophy researcher

, a

quantitative researcher

, and a

programmer

.

Counterproductive Altruism:

...
AGI's Impact on Employment 
by TheUnkown  — 2023-03-06T01:56:16.934Z
Link: https://www.apricitas.io/p/chatgpt-please-take-my-job

Here is an interesting post about AGI's impact on human employment. The author basically arguing that AGI will increase human employment in the economy. Basically he seems to think that humans will always manage to find work especially in labor-intensive sectors like health care and education. Past innovations and their impacts on the economy are used as evidence for this theory. I personally disagree with this main conclusion but I'm interested in what other people have to say.
Why did you trash the old HPMOR.com?
by AnnoyedReader — 2023-03-06T01:55:48.304Z

I am very disappointed.

Now I'm sure you'd have to see it to believe it, but you see, just today I was rereading my favorite book, HPMOR, for the fourth time, and I clicked to go on to the next chapter, and to my surprise, there was no next chapter! Which is pretty crazy, considering there was a next chapter last time. Instead it just dropped me on LessWrong (on the HPMOR page). As an enjoyer of the old HPMOR.com site I just wanted to express my frustration.

Now I'm not trying to complain about LessWrong here. I actually like LessWrong, and sometimes I read posts on here. There are many forum and social media sites, and it does seem to me that LessWrong is one of the better ones. I mean if I found out that HMPOR.com was being shut down and instead we would be forced to read it as a series of Facebook posts, I would be much more angry. But still, the best way to run a social media site, and the best way to publish an ebook cannot possibly coincide. (Or, on second thought, perhaps they might, since they're both sort of publishers, but I'm specifically annoyed about this specific merger.)

The thing is, I already knew about HMPOR being on LessWrong. But I chose to read on HPMOR.com anyway, because that is what I preferred. One annoying thing about LessWrong for this, is that if I scroll up at all (even just slightly) while reading a long post (or chapter?) the annoying banner at the top of the screen shows up, making it difficult to include the specific text on the screen which you wanted. HMPOR.com didn't have that problem. But there are many frustrations. I feel like in the past, I had the choice, and now that choice has been taken away from me.

I have, at times, been accused of not liking change. (I think it's more a feeling that changes should optional, and people shouldn't be forced into new things they don't want.) But I feel like there are many circumstances where it should be clear change is not a good idea? Like, if you make something that people specifically seek out, instead of other things they could go for, well, among things, from whoever wants it's point of view, it is one of the better ones. Perhaps even one of the best things. And then if it is changed, well, most changes of something so atypical (from any point of view) tend to be reversions to the mean. So if you like something a lot, you shouldn't want it to change, or at least should be worried ...
Cap Model Size for AI Safety
by research_prime_space — 2023-03-06T01:11:59.617Z

There are diminishing marginal returns to intelligence -- an AI with an IQ of 150 can perform almost all human tasks flawlessly. The only exception may be conducting scientific research.

So why don't we lobby for capping the model size, at perhaps, a couple hundred billion parameters? This cap can be strictly enforced if it's encoded into deep learning software packages (e.g. NVIDIA, PyTorch, etc.).

I think this may be the most tractable approach for AI safety.
What should we do about network-effect monopolies?
by benkuhn — 2023-03-06T00:50:07.554Z

Many large companies today are software monopolies that give their product away for free to get monopoly status, then do the most horrible things once they’ve won. (Previously, previously.) Can we do anything about this?

Unfortunately, “you’re the product” is a popular business model for a reason: businesses like Facebook would be really hard to support without them.

Facebook would be suicidal to charge its users money, because its entire selling point is that everyone uses it, and “everyone” hates paying money. In the US, Facebook makes over $25 per person on ads (source). Can you imagine if instead of ads they tried to charge people $25 a year?

Even on the margin, anything that costs Facebook users also makes it less valuable for its remaining users—it’s a negative feedback loop. The same goes for any other site where users create value for other users, like Twitter or Craigslist or Yelp or Wikipedia. (It’s not an accident that these are some of the most stagnant popular websites!)

In fact, this is a fundamental problem with network effects and low marginal costs. If a company wants to maintain a network effect, they need as many users as possible. If their marginal cost is low, then the easiest way to get users is to give the product away. To do that, they have to get paid by someone else. And when they start getting paid by someone else, they’ll inevitably start prioritizing that person’s interests.

Historically with other network-effect businesses, we’ve addressed this in a few different ways:

regulation (e.g. local utilities)

breakups (e.g. Bell)

standardization and interoperability (e.g. email, the Web, cryptocurrency)

So far for tech monopolies, people seem to be focused mostly on breakups—e.g. Facebook from Instagram/Whatsapp—but standardization seems to have produced much better outcomes in the past. (I like email and the Web a lot more than National Grid…) I’d be interested to see more exploration of that option!
Who Aligns the Alignment Researchers?
by Ben Smith — 2023-03-05T23:22:27.107Z

There may be an incentives problem for AI researchers and research organizations who face a choice between researching Capabilities, Alignment, or neither. The incentives structure will lead individuals and organizations to work towards Capabilities work rather than Alignment. The incentives problem is a lot clearer at the organizational level than the individual level, but bears considering at both levels, and of course, funding available to organizations has downstream implications for the jobs available for researchers employed to work on Alignment or Capabilities.

In this post, I’ll describe a couple of key moments in the history of AI organizations. I’ll then survey incentives researchers might have for doing either Alignment work or Capabilities work. We’ll see that it maybe that, even considering normal levels of altruism, the average person might prefer to do Capabilities rather than Alignment work. There is relevant collective action dynamic. I’ll then survey the organizational level and global level. After that, I’ll finish by looking very briefly at why investment in Alignment might be worthwhile.

A note on the dichotomous framing of this essay: I understand that the line between Capabilities and Alignment work is blurry, or worse, some Capabilities work plausibly advances Alignment, and some Alignment work advances Capabilities, at least in the short term. However, in order to model the lay of the land, it’s helpful as a simplifying assumption to examine Capabilities and Alignment as distinct fields of research and try to understand the motivations for researchers in each.

History

As a historical matter, DeepMind and OpenAI were both founded with explicit missions to create safe, Aligned AI for the benefit of all humanity. There are different views on the extent to which each of these organizations remains aligned to that mission. Some people maintain they are, while others maintain they are doing incredible harm by shortening AI timelines. No one can deny that they have moved at least somewhat in the direction of more profit-making behavior, and are very much focused on Capabilities research. So, at best, they’ve stuck to their original mission, but having watered it down to allow a certain amount of profit-seeking; at worst, their overall efforts are net-negative for alignment by accelerating development of AGI.

OpenAI took investment from Micr...
Startups are like firewood
by Adam Zerner — 2023-03-05T23:09:35.066Z

One TV show I like to watch when I'm relaxing at night is Primal Surivor. The premise is that this guy Hazen Audel is placed in remote locations and has to, well, survive.

In watching it, something that really struck me is how important fire is. It provides warmth, but that's obvious. It also lets you cook food. Also obvious. Provides light. Obvious. Something that wasn't obvious to me is that the smoke keeps bugs and predators away.

Maybe the thing that struck me more is how much Hazen's day revolves around fire. It's always something he's thinking about. Especially as sundown starts to approach. He needs a place to sleep at night and the place he'd choose to sleep often depends on where it's most practical to start a fire.

Starting a fire turns out to be much harder than what I would have thought before watching the show. I knew it's not as easy as just turning a fireplace on, but I had this idea in my head that skilled people can whip one together easily enough in five minutes or so. That's not how it works though.

In reality, it's a somewhat involved process. This may be overly simplified and partially incorrect, but here is how I see it.

It's difficult for big logs of firewood to catch fire. You need a medium-sized fire to already be burning to provide enough "oomph" for the big log to catch fire.

It's correspondingly difficult for medium-sized logs of firewood to catch fire. You need a small fire first to provide the necessary "oomph".

Before small logs catch fire you need even smaller twigs to catch fire.

For twigs (kindling, I guess) to catch fire you first need some tinder[1] to catch fire.

For tinder to catch fire, you need a spark.

I think this is a really good analogy for startups.[2]

Here's an example. I have this idea for a startup that could be huge. The goal is to disrupt and eventually replace marketing and sales. To give you a sense of how huge that is, worldwide, companies spent something like $800B on advertising. In the US it's $200B. And that's not even considering sales.

Now that's one ginormous piece of firewood! What would it look like for it to be burning? Well, imagine something like The Wirecutter on steroids. Call it The Logcutter. Consumers would be able to go to The Logcutter, say a few things about what they're looking for, and receive trustworthy, reliable recommendations about what product is best for them. If consumers had that...
A concerning observation from media coverage of AI industry dynamics
by Justin Olive — 2023-03-05T21:38:18.473Z

tl:dr: there are indications that ML engineers will migrate to environments with less AI governance in place, which has implications for the tech industry and global AI governance efforts.

=========================

I just wanted to raise something to the community's attention about the coverage of AI companies within the media. The media-source is 'The Information', which is a tech-business focused online news source. Link: https://www.theinformation.com/. I'll also note that their articles are (to my knowledge) all behind a paywall.

The first article in question is titled "Alphabet Needs to Replace Sundar Pichai".

It outlines how Google stocks have stagnated in 2023 compared to other tech stocks such as Meta's.

Here's their mention of Google's actions throughout GPT-mania:

"The other side of this equation is the performance of Alphabet management. Most recently, the company’s bungling of its AI efforts—allowing Microsoft to get the jump on rolling out an AI-powered search engine—was the latest sign of how Alphabet’s lumbering management style is holding it back. (Symbolically, as The Information reported, Microsoft was helped by former Google AI employees!)."

This brings us to the second article: "OpenAI’s Hidden Weapon: Ex-Google Engineers"

"As OpenAI’s web chatbot became a global sensation in recent months, artificial intelligence practitioners and investors have wondered how a seven-year-old startup beat Google to the punch.

...

After it hoovered up much of the world’s machine-learning talent, Google is now playing catch-up in launching AI-centric products to the public. On the one hand, Google’s approach was deliberate, reflecting the company’s enormous reach and high stakes in case something went wrong with the nascent technology. It also costs more to deliver humanlike answers from a chatbot than it does classic search results. On the other hand, startups including OpenAI have taken some of the AI research advances Google incubated and, unlike Google, have turned them into new types of revenue-generating services, including chatbots and systems that generate images and videos based on text prompts. They’re also grabbing some of Google’s prized talent.

Two people who recently worked at Google Brain said some staff felt the unit’s culture had become lethargic, with product initiatives marked by excess caution and layers of red tape. That has prompted some em...
Steven Pinker on ChatGPT and AGI (Feb 2023)
by Evan R. Murphy — 2023-03-05T21:34:14.846Z
Link: https://news.harvard.edu/gazette/story/2023/02/will-chatgpt-replace-human-writers-pinker-weighs-in/

While I disagreed with a lot of Robin Hanson's latest take on AI risk, I am glad he came out with an updated position. I think with everything that's happened in the past 6-12 months, it's a good time for public intellectuals and prominent people who have previously commented on AGI and AI risk to check in again and share their latest views.
That got me curious if Steven Pinker had any recent statements. I found this article on the Harvard Gazette from last month (Feb 2023), which I couldn't find posted on LessWrong before:

Article link

Will ChatGPT supplant us as writers, thinkers?
Q&A with Steven Pinker
by Alvin Powell
Feb 14, 2023

Summary

Here's a summary of the article that ChatGPT generated for me just now (bold mine):

Steven Pinker, a psychology professor at Harvard, has commented on OpenAI’s ChatGPT, an artificial intelligence (AI) chatbot that can answer questions and write texts. He is impressed with the AI's abilities, but also highlights its flaws, such as a lack of common sense and factual errors. Pinker believes that ChatGPT has revealed how statistical patterns in large data sets can be used to generate intelligent-sounding text, even if it does not have understanding of the world. He also believes that the development of artificial general intelligence is incoherent and not achievable, and that current AI devices will always exceed humans in some challenges and not others. Pinker is not concerned about ChatGPT being used in the classroom, as its output is easy to unmask as it mashes up quotations and references that do not exist.

Note that while he comments on AGI being an incoherent idea, he doesn't speak specifically about existential risk from AI misalignment. So it's not totally clear, but I think we can infer Pinker considers the risk very low, since he doesn't think AGI is possible in the first place.
Is it time to talk about AI doomsday prepping yet?
by bokov — 2023-03-05T21:17:54.270Z

Given the shortening timelines (https://www.lesswrong.com/posts/K4urTDkBbtNuLivJx/why-i-think-strong-general-ai-is-coming-soon, https://www.lesswrong.com/posts/CvfZrrEokjCu3XHXp/ai-practical-advice-for-the-worried), perhaps it's time to think about what "plan B" should actually be if the "plan A" of solving the alignment problem does not succeed.

For many people the answer is a simple "then we will all die". But let's suppose for a moment that we don't get off the hook that easily-- that AGI doomsday happens in such a way that there are survivors.

What can one do right now to maximize one's chances of being among the survivors if unaligned AGI is created? What do you think an AI prepper would be like, compared and contrasted to, for example, a nuclear war prepper?