trentmkelly/LessWrong-43k huggingface dataset:
Book Review: Orality and Literacy: The Technologizing of the Word 
by Fergus Fettes — 2023-10-28T20:12:07.743Z

Thanks to Diarmuid Morgan for reviewing an earlier version of this post.

Dream cancels dream in this new realm of fact
From which we wake into the dream of act[1]

Review of Orality and Literacy: The Technologizing of the Word, by Walter J. Ong, 1982

I went looking for a meatier definition of a 'meme', hoping to find something that could be leveraged to understand both current events in machine learning, and co-evolving understanding of consciousness that goes hand-in-hand with every major ML breakthrough these days.

I haven't found it yet, but I am making progress. Before I summarize my overall progress, I will review a couple of the most interesting and under appreciated works I have stumbled upon on the way.

Intro for LW

Broadly under the topic of 'memetics', but far enough outside the envelope that it is not even in the Wikipedia article, I nevertheless found this book to have more insight and value about the nature of the word and its relationship to thought than most papers I read on memetics.

I imagine work like this (aka memetics generally) ending up somewhere near the top of the interpretability stack being built on LLMs. At the very least as a way to marry public-facing discourse with technical details, at the best as a quantitative discipline (semiotic physics, anyone?) in its own right.

Intro for EA[2]

In light of the ongoing work of Rethink Priorities on animal welfare, it seems to me there is a strong need for quantitative approaches to welfare that go above and beyond hedonism[3]. I think memetics in general has the potential to contribute to this, and this work in particular is a good starting point-- it discusses quite considerable changes in the nature of consciousness in humans in an quantitative and testable manner.

I also think that work like this will help clarify the boundaries between animal welfare, AI welfare and human welfare on an intuitive level. (Hint: imagine where those different entities would be placed on a graph with two axes, 'memetic intensity' and 'hedonic valency'.)

Background

I was introduced to this topic area in a conversation on discord about the possibility that we are entering a 'second orality', a period of highly oral culture becoming divorced from the literate foundations on which culture rested for so long.

Some evidence for this:

tictok and instagram more popular than twitter among younger generations
Netflix more popular tha

...
Regrant up to $600,000 to AI safety projects with GiveWiki
by Dawn Drescher — 2023-10-28T19:56:06.676Z

The AI Safety GiveWiki aggregates donation data from over 200 EA donors and combines them with retroactive evaluations to generate a list of the top AI safety projects to support. Some 50 donors with a total donation budget of $614,401 want to use it to inform their giving.

Please register your donations to improve the platform’s top recommendations, and you may just move some fraction (probably < 10%) of these $614k to a more effective project.

Introduction

A lot has happened since our last milestone announcement. We more than doubled the amount of funding that you, as regrantor/scout, can move; we renamed to AI Safety GiveWiki, which is more descriptive than AI Safety Impact Markets; and we released the above explainer video. (Though not in that order, so please excuse the use of the old name in the video.)

As for the new name, it succinctly conveys what the platform is about:

A crowdsourced recommendation engine for giving.
A place where you can (soon) give away Wiki Credits.
A place to give quickly (wikiwiki ≈ quick) – counter the

Donor’s Dilemma

.

How to Regrant

There’s no one on GiveWiki who can flick a switch and make you a regrantor. It’s an organic process where you enter your donations to Project X, and if our evaluators are happy with how Project X turned out, then your reputation on the platform will benefit. With greater reputation (donor score) comes a greater weight in our recommendations.

Right now we only have 14 users who have the ambition to inform our recommendations, so it’s still very easy to get started as a regrantor – or “project scout” as we call them.

Concretely, say, you’ve donated to the Center on Long-Term Risk

Find the project page and select the “Register a donation” tab.

Enter the date of your donation, the amount in USD at the time, and optionally the “recommender” – our recommendations, a particular (other) project scout, or your own research?

Click Add. Congrats! You've registered a donation. The charity will be notified of it and will veto it if it's incorrect.

Repeat it a few more times until you've entered all your donations, and you may become a new top donor!

Tips

You have a lot of donations to enter? We can help! Just send us a spreadsheet with the project, date, amount, currency, and recommender to hi@givewiki.org.
A charity you donated to is not yet registered on the platform? Email them and ask them to sign up. It only costs them

...
Shane Legg interview on alignment
by Seth Herd — 2023-10-28T19:28:52.223Z
Link: https://www.youtube.com/watch?v=Kc1atfJkiJU&t=1159s

This is Shane Legg, cofounder of DeepMind, on Dwarkesh Patel's podcast. The link is to the ten-minute section in which they specifically discuss alignment. Both of them seem to have a firm grasp on alignment issues as they're discussed on LessWrong.

For me, this is a significant update on the alignment thinking of current leading AGI labs. This seems more like a concrete alignment proposal than we've heard from OpenAI or Anthropic. Shane Legg has always been interested in alignment and a believer in X-risks. I think he's likely to play a major role in alignment efforts at DeepMind/Google AI as they approach AGI.

Shane's proposal centers on "deliberative dialogues", DeepMind's term for a system using System 2 type reasoning to reflect on the ethics of the actions it's considering.

This sounds exactly like the the internal review I proposed in Capabilities and alignment of LLM cognitive architectures and Internal independent review for language model agent alignment. I could be squinting too hard to get his ideas to match mine, but they're at least in the same ballpark. He's proposing a multi-layered approach, like I do, and with most of the same layers. He includes RLHF or RLAIF as useful additions but not full solutions, and human review of its decision processes (externalized reasoning oversight as proposed by Tamera Lanham, now at Anthropic).

My proposals are explicitly in the context of language model agents, (including their generalization to multimodal foundation models). It sounds to me like this is the type of system Shane is thinking of when he's talking about alignment, but here I could easily be projecting. His timelines are still short, though, so I doubt he's envisioning a whole new type of system prior to AGI.[1]

Dwarkesh pushes him on the challenges of both getting a ML system to understand human ethics. Shane says that's challenging; he's aware that giving a system any ethical outlook at all is nontrivial. I'd say this aspect of the problem is well on the way to being solved; GPT4 understands a variety of human ethical systems rather well, with proper prompting. Future systems will understand human conceptions of ethics better yet. Shane recognizes that just teaching a system about human ethics isn't enough; there's a philosophical challenge in choosing the subset of that ethics you want the system to use.

Dwarkesh also pushe...
AI Existential Safety Fellowships
by mmfli — 2023-10-28T18:07:19.773Z

Applications are open for Future of Life Institute postdoctoral and PhD fellowships in AI existential safety research.

The fellowship is global and open to all regardless of nationality or background; we are seeking a diverse applicant pool. All Fellows will receive applicable tuition and fees, as well as a stipend and research/travel fund. More info below:

The

Vitalik Buterin PhD Fellowship in AI Existential Safety

is for PhD students who plan to work on AI existential safety research, or for existing PhD students who would not otherwise have funding to work on AI existential safety research. It will fund students for 5 years of their PhD, with extension funding possible. At universities in the US, UK, or Canada, annual funding will cover tuition, fees, and the stipend of the student's PhD program up to $40,000, as well as a fund of $10,000 that can be used for research-related expenses such as travel and computing. At universities not in the US, UK or Canada, the stipend amount will be adjusted to match local conditions. Fellows will also be invited to workshops where they will be able to interact with other researchers in the field. Applicants who are short-listed for the Fellowship will be reimbursed for application fees for up to 5 PhD programs, and will be invited to an information session about research groups that can serve as good homes for AI existential safety research.

Applications are due November 16, 2023 at 11:59pm ET.

The

Vitalik Buterin Postdoctoral Fellowship in AI Existential Safety

is designed to support promising researchers for postdoctoral appointments who plan to work on AI existential safety research. Funding is for three years subject to annual renewals based on satisfactory progress reports. For host institutions in the US, UK, or Canada, the Fellowship includes an annual $80,000 stipend and a fund of up to $10,000 that can be used for research-related expenses such as travel and computing. At universities not in the US, UK or Canada, the fellowship amount will be adjusted to match local conditions. Fellows will be invited to workshops where they will be able to interact with other researchers in the field.

Applications are due January 2, 2024 at 11:59pm ET.
AI Safety Hub Serbia Official Opening
by DusanDNesic — 2023-10-28T17:03:34.607Z
Link: https://forum.effectivealtruism.org/posts/rg8GoZwHJD5igkDfC/ai-safety-hub-serbia-official-opening

TLDR: We're thrilled to announce that we are now welcoming full-time tenants to our newly transformed office space for AI Safety researchers in search of an inspiring workspace. You can take a glimpse at the photos below to get a sneak peek. We're extending a warm invitation, with priority, to citizens of nations such as Russia and China, who can enjoy visa-free work privileges in Serbia while maintaining proximity to Europe. Our monthly office rent is an astonishingly affordable 150€, about half the standard cost in Belgrade. For those needing financial support, we have subsidies available.

Register interest here

or seek answers to any questions that stir your curiosity -

our inbox is open

, and we eagerly await your inquiries. Looking ahead, we aspire to provide housing for these researchers. If you're a potential donor who shares our vision,

please contact us

. Together, we can enhance the impact of AI Safety research. Your support could be the catalyst for a remarkable journey.

You may want to come if:

You are an AI Safety Researcher/EA researcher looking for a base of operations for a short-medium-long term
You are keen to be in Europe but not in the EU
You are looking for a vibrant but affordable city with plenty of things to do and Eastern European but Westernized culture.

Background:

EA Serbia and AI Safety Serbia groups are small but growing (>30 people in EA Serbia, ~3 people looking to get into AIS research as a career, and ~3 to get into AIS policy). Due to Serbia’s favorable Visa policy towards Russia and China, many foreigners already live here. With lower living costs than many other international hub cities, a vibrant scene, and a favourable time zone and climate, Belgrade has a growing foreign community.

As we have seen projects such as

CEEALAR

as important and impressive, we wish to replicate them in Serbia, where they can better serve people who may struggle to get UK visas. We also believe that having the capacity to quickly scale cheap housing for people coming from different countries is a good thing.

We also believe that we should start small, prototype, and then move larger. We have an NGO-friendly office space that has good vibes and costs only ~550 Euros per month for office space that has three rooms and can fit 8-15 people (depending on how snug they decide to be) with a coffee shop downstairs where another 20 people ...
Managing AI Risks in an Era of Rapid Progress

by Algon — 2023-10-28T15:48:25.029Z
Link: https://managing-ai-risks.com/

Managing AI Risks in an Era of Rapid Progress

Authors

Yoshua Bengio

Geoffrey Hinton

Andrew Yao

Dawn Song

Pieter Abbeel

Yuval Noah Harari

Ya-Qin Zhang

Lan Xue

Shai Shalev-Shwartz

Gillian Hadfield

Jeff Clune

Tegan Maharaj

Frank Hutter

Atılım Güneş Baydin

Sheila McIlraith

Qiqi Gao

Ashwin Acharya

David Krueger

Anca Dragan

Philip Torr

Stuart Russell

Daniel Kahneman

Jan Brauner

Sören Mindermann

arXiv

Forthcoming.

Paper PDF copy Policy supplement

Abstract: In this short consensus paper, we outline risks from upcoming, advanced AI systems. We examine large-scale social harms and malicious uses, as well as an irreversible loss of human control over autonomous AI systems. In light of rapid and continuing AI progress, we propose urgent priorities for AI R&D and governance.

In 2019, GPT-2 could not reliably count to ten. Only four years later, deep learning systems can write software, generate photorealistic scenes on demand, advise on intellectual topics, and combine language and image processing to steer robots. As AI developers scale these systems, unforeseen abilities and behaviors emerge spontaneously without explicit programming . Progress in AI has been swift and, to many, surprising.

The pace of progress may surprise us again. Current deep learning systems still lack important capabilities and we do not know how long it will take to develop them. However, companies are engaged in a race to create generalist AI systems that match or exceed human abilities in most cognitive work . They are rapidly deploying more resources and developing new techniques to increase AI capabilities. Progress in AI also enables faster progress: AI assistants are increasingly used to automate programming [4] and data collection [5,6] to further improve AI systems [7].

There is no fundamental reason why AI progress would slow or halt at the human level. Indeed, AI has already surpassed human abilities in narrow domains like protein folding or strategy games [8–10]. Compared to humans, AI systems can act faster, absorb more knowledge, and communicate at a far higher bandwidth. Additionally, they can be scaled to use immense computational resources and can be replicated by the millions.

The rate of improvement is already staggering, and tech companies have the cash reserves needed to scale the latest training runs by multiples of 100 to 1000 soon [11]. Combined with the ongoing growth and automation in AI R&a...
ELI5 Why isn't alignment *easier* as models get stronger?
by Logan Zoellner — 2023-10-28T14:34:37.588Z

Empirical Status: straw-man, just wondering what the strongest counterarguments to this are.

It seems obvious to me that stronger models are easier to align. A simple proof

It is always possible to get a weaker model out of a stronger model (for example, by corrupting n% of its inputs/outputs)
Therefore if it possible to align a weak model, it is at least as easy to align a strong model
It is unlikely to be exactly as hard to align weak/strong models.
Therefore it is easier to align stronger models

(I have a list of counter-arguments written down, I am interested to see if anyone suggests a counterargument better than the ones on my list)
Truthseeking, EA, Simulacra levels, and other stuff
by Elizabeth — 2023-10-27T23:56:49.198Z

Elizabeth

I've been on this big kick talking about truthseeking in effective altruism. I started with vegan advocacy because it was the most legible, but now need to move on to the deeper problems. Unfortunately those problems are still not that legible, and I end up having to justify a lot of what I previously took as basic premises, and it's all kind of stuck. I'm going to lay out a few threads in my mind, in the hopes that you will be curious about some of them and we can untangle from there.

Some threads:

There is a subtle distinction between "when should an individual operate on object level vs social reality?" and "when should a group of people invest together in facilitating operating in object level over social reality,?" I'm way more interested in the latter, although obviously it intersects with the former.
Up until now my posts have assumed that truthseeking is good without particularly justifying it. Seems like if I'm so excited about asking inconvenient questions I should be willing to turn that on itself.
I think the better version of that question is "when is truthseeking the most important thing the most important thing to invest in, and when can you stop?" Because obviously truthseeking is useful and important, the interesting question is how that trades off against capacities a group could invest in, like fundraising or operational competence.
Part of my argument here is "do you know how to solve AI alignment? because that sounds like a knowledge problem to me, and knowledge problems are solved in object level reality not social reality". But it's possible to be operating entirely in object level reality but just be kind of bad at it. The word "truthseeking" maybe combines "object level reality" with "good at it" in confusing ways, and maybe I should separate those.
I think EA would benefit from understanding simulacra levels and aiming for level 1, but I think rationalists would benefit from more understanding of what purpose higher levels serve. I think moving down simulacra levels requires replacing the function that higher levels serve (e.g. social support) in ways that don't require so much sacrifice of truthseeking.

Vaniver

I end up having to justify a lot of what I previously took as basic premises, and it's all kind of stuck

My initial attention is mostly on this point, which is maybe also your second poin

...
Do you believe "E=mc^2" is a correct and/or useful equation, and, whether yes or no, precisely what are your reasons for holding this belief (with such a degree of confidence)?
by l8c — 2023-10-27T22:46:51.020Z

Please just answer the question as best you can. E.g. your answer might be "because everyone on TV says so" or "because I saw mass transform into energy according to this ratio in the lab", or perhaps a lengthier or negative answer.
Value systematization: how values become coherent (and misaligned)
by Richard_Ngo — 2023-10-27T19:06:26.928Z

Many discussions of AI risk are unproductive or confused because it’s hard to pin down concepts like “coherence” and “expected utility maximization” in the context of deep learning. In this post I attempt to bridge this gap by describing a process by which AI values might become more coherent, which I’m calling “value systematization”, and which plays a crucial role in my thinking about AI risk.

I define value systematization as the process of an agent learning to represent its previous values as examples or special cases of other simpler and more broadly-scoped values. I think of value systematization as the most plausible mechanism by which AGIs might acquire broadly-scoped misaligned goals which incentivize takeover.

I’ll first discuss the related concept of belief systematization. I’ll next characterize what value systematization looks like in humans, to provide some intuitions. I’ll then talk about what value systematization might look like in AIs. I think of value systematization as a broad framework with implications for many other ideas in AI alignment; I discuss some of those links in a Q&A.

Belief systematization

We can define belief systematization analogously to value systematization: “the process of an agent learning to represent its previous beliefs as examples or special cases of other simpler and more broadly-scoped beliefs”. The clearest examples of belief systematization come from the history of science:

Newtonian mechanics was systematized as a special case of general relativity.
Euclidean geometry was systematized as a special case of geometry without Euclid’s 5th postulate.
Most animal behavior was systematized by evolutionary theory as examples of traits which increased genetic fitness.
Arithmetic calculating algorithms were systematized as examples of Turing Machines.

Belief systematization is also common in more everyday contexts: like when someone’s behavior makes little sense to us until we realize what their hidden motivation is; or when we don’t understand what’s going on in a game until someone explains the rules; or when we’re solving a pattern-completion puzzle on an IQ test. We could also see the formation of concepts more generally as an example of belief systematization—for example, seeing a dozen different cats and then forming a “systematized” concept of cats which includes all of them. I’ll call this “low-level syste...
Techno-humanism is techno-optimism for the 21st century
by Richard_Ngo — 2023-10-27T18:37:39.776Z
Link: https://www.mindthefuture.info/p/techno-humanism-is-techno-optimism

Lately I’ve been reading about the history of economic thought, with the goal of understanding how today’s foundational ideas were originally developed. My biggest takeaway has been that economics is more of a moving target than I’d realized. Early economists were studying economies which lacked many of the defining features of modern economies: limited liability corporations, trade unions, depressions, labor markets, capital markets, and so on. The resulting theories were often right for their time; but that didn’t stop many from being wrong (sometimes disastrously so) for ours.

That’s also how I feel about the philosophy of techno-optimism, as recently defended in Marc Andreessen’s

Techno-Optimist Manifesto

. I’ll start this post with the many things it gets right; then explore why, over the last century, the power of technology itself has left straightforward techno-optimism outdated. I’ll finish by outlining an alternative to techno-optimism: techno-humanism, which focuses not just on building powerful engines of progress, but also on developing a deep scientific understanding of how they can advance the values we care about most.

The triumphs of techno-optimism

Here’s where I agree with Andreessen: throughout almost the entirety of human history, techno-optimism was fundamentally correct as a vision for how humans should strive to improve the world. Technology and markets have given us health and wealth unimaginable to people from previous centuries, despite consistent opposition to them. Even now, in the face of overwhelming historical evidence, we still dramatically underrate the potential of technology to solve problems ranging from

climate change

to neglected tropical diseases to extreme poverty and many more. This skepticism holds back not only speculative technologies but also ones that are right in front of us: miraculous solutions to huge problems (like vaccines for

COVID

and

malaria

, and

gene drives

for mosquito eradication) are regularly stalled by political or bureaucratic obstructionism, costing millions of lives. The RTS,S malaria vaccine spent

twenty-three years

bogged down in clinical trials

, with six years of that delay caused by WHO “precautions” even after other regulators had already approved it. Fighting against the ideologies and institutional practices which cause tragedies like these is an incredibly important cause.

We often similarly...