trentmkelly/LessWrong-43k huggingface dataset:
EA & LW Forum Weekly Summary (16th - 22nd Jan '23)
by Zoe Williams — 2023-01-23T03:46:10.759Z

Supported by Rethink Priorities
This is part of a weekly series summarizing the top posts on the EA and LW forums - you can see the full collection here. The first post includes some details on purpose and methodology. Feedback, thoughts, and corrections are welcomed.

If you'd like to receive these summaries via email, you can subscribe here.

Podcast version: Subscribe on your favorite podcast app by searching for 'EA Forum Podcast (Summaries)'. A big thanks to Coleman Snell for producing these!

Philosophy and Methodologies

Don’t Balk at Animal-friendly Results

by Bob Fischer

The moral weight project assumes a) hedonism and b) we can use various capacities as proxies for a species’ range of possible welfare states. This post warns against knee-jerk skepticism if research leads to the conclusion that chickens and humans can realize roughly the same amount of welfare at a given time (the ‘Equality Result’). It doesn’t argue for the Equality Result itself.

Three arguments and counter-arguments for skepticism are discussed:

The implications are too huge - under utilitarianism, it means we should massively shift resources toward animals.

Utilitarianism is the reason for the radical implications - the equality result is just the messenger.

Maybe the hedonism assumption is wrong.

Fair, though per previous posts hedonism doesn’t change the bottom line much - even if hedonic goods / bads isn’t all of welfare, it’s surely a large part.

Even accepting the assumptions, the Equality Result feels wrong.

This intuition is uncalibrated, affected by many biases.

Conditional on hedonism, the Equality Result wouldn’t be surprising, as it fits with popular theories of valence.

Object Level Interventions / Reviews

Existential Risks (including AI)

How many people are working (directly) on reducing existential risk from AI?

by Benjamin Hilton, 80000_Hours

The author estimates 400 FTE (90% CI: 200-1000). Three quarters on technical AI safety research, with the rest distributed between strategy, governance, and advocacy.

Can GPT-3 produce new ideas? Partially automating Robin Hanson and others

by NunoSempere

The author experimented with asking GPT-3 to replicate patterns of insight eg. variations on ‘if you never miss a plane, you’ve been spending too much time at the airport’. It generated somewhat original outputs that matched these patterns, but no great insight...
Consider Trying Dictation
by jefftk — 2023-01-22T22:50:03.470Z

I started using dictation for entering text on my phone and computer
when my wrists

suddenly went really bad,
but while my wrists are enough better that I can often type there are
still a lot of cases where I dictate. Even people whose wrists are
fine would benefit from trying it out:

It can be about as fast as typing on a real keyboard, which is
great on a phone. I'm very good at swiping, but I can dictate
faster.

On Android, at least, it does a good job even in noisy
environments as long as I hold the phone close to my mouth. I can also
speak very fast, faster than I can compose my thoughts, and it will
still understand me. It even understands whispering!

I like having it as an option for composing thoughts,
especially when I'm having trouble focusing. Sometimes it turns out I'm not in a good
headspace for typing ideas out, but talking them through out loud
works.

If your wrists do start hurting someday, being able to easily
give them a break by switching some things to dictation can help them
recover faster and make you less likely to hurt yourself. I suspect if
I had been used to dictating as needed in 2020, I wouldn't have gotten
to where I had to stop typing for a couple years.

There are some downsides, however:

The dictation built into Android isn't great with punctuation, and I
often have to go back and fix things.

The error rate, in general, is higher. For note taking or casual
texting where errors are no big deal this is less of an issue, but in
dictating this post there were several places I needed to go back and
fix something.

Because mistranscriptions look more plausible than typos I'm
more likely to have errors that make it all the way through to
posting: the mistakes are less likely to stand out as obviously wrong.

Fixing dictation errors is a little more annoying than when typing
because you have to switch modes.

I like my dictated style slightly less than typed: it's just a bit
too wordy. This makes it more important to go back over dictated posts
looking for things to cut.

The dictation built into Mac isn't very good, and it's terrible
unless I have a microphone right next to my
mouth. If I want to dictate longer things (ex: this post) I'll
dictate them on my phone and then edit on the computer.

I occasionally find myself saying "comma" or "semicolon" when
talking to people.

Still, I do ...
Emotional attachment to AIs opens doors to problems
by Igor Ivanov — 2023-01-22T20:28:35.223Z

We are going to see more and more emotional bonds between humans and AIs

Recently I red an interesting post "How it feels to have your mind hacked by an AI". The author describes how he fell in love and became soulmates with a character created by ChatGPT. There is also a well-known story about a Google engineer who decided that their language model, LaMDA is sentient. I think those are canaries in the coal mine, and we will see more and more people forming emotional bonds with AIs.

It's hard to predict the ways we will use language models several years down the line, but it's easy to imagine that there will be strong incentives to make empathetic AIs capable of creating emotional bonds with their users because such bonds can easily be monetized and increase users' retention.

The example of such monetization is an app called Replika, an AI friend. Users can chat with it, and they might pay money to buy new skins for its avatar, to customize its personality features, and for other stuff.

One can argue that such AI might be good for lonely people or to be some sort of more affordable way of emotional and mental health support. It might be a good idea, especially now, at the time of the epidemic of loneliness and mental health issues, and the price for qualified help is unaffordable for many.

But I also can imagine how emotional bonds and attachment to an AI might cause a number of serious risks to mental health. It is quite an important and interesting topic to explore, but for another post.

The weaponization of emotional attachment to an AI

Apart from that, I want to talk about direct ways to weaponize the emotional attachment of a person to an AI. There are several possible scenarios I can imagine.

The idea behind all examples is that emotional attachment to an AI does not necessarily create problems by itself, but it makes people more vulnerable to other problems.

Data privacy

If you are lonely and upset, and the only being caring for you is a chatbot, it is easy to share with it your feelings, concerns, and problems. If you think of a bot as a friend then it naturally wants to know more about you. It might be something straightforward, like "How is your day?" or more deep, personal, and emotional. "I am sorry to read that you are feeling lonely after the breakup. I hope I can do something for you to feel better. What is upsetting you the most? Maybe I can supp...
What fills a vacuum?
by Logan Kieller — 2023-01-22T19:25:49.385Z

Epistemic status: Musings.

I: Interior decorating

Every time in my life I have had an empty room in my house, an interesting phenomenon happens.

I fill it, but with random things. I store an old couch in there, I put some crumpled-up boxes of trinkets and memories on top of the couch. I rest an old art easel on the sofa’s arm. I leave used-up paint cans in the corner and put a container of used clothes near the door.

Every time in my life I have had an empty hour in my day, an interesting phenomenon happens.

I fill it, but with random things. I lay on my bed and scroll through social media aimlessly, I play a mindless game on my computer. Whatever I have to do I don’t do, but I do just about everything else. If anyone asked what I did, I wouldn’t have anything to say.

When you have empty space and when you have free time, what can you do to fill it appropriately?

II: What happens to vacuums?

Eventually, vacuums are filled. This is how life works. It demands it.

The extra room will become storage; the free hour will be wasted.

We should fill our empty spaces intentionally, before they are randomly filled by other things.

Parkinson’s law

supports this premise, suggesting “work expands so as to fill the time available for its completion.” The point being, that if you create a vacuum for the work within a project, without filling that time intentionally, random tasks will spontaneously occur to fill up any “slack time”.

Work will always demand more from you than you should give. There is always more work to be done, and so if you allow it, work will balloon to fill any extra time you make available. It might mean staying for an extra hour at the office, or doing work when you get home, or “saying yes” to any offered opportunities.

When vacuums exist, it isn’t just random things filling up that time or space, but it’s things that are purposely able to fill up that vacuum.

Certain modern productivity systems are predicated on using all time effectively by creating discrete blocks.

Time blocking

is the act of providing a very specific amount of time in a very specific slot in the day for each task. You fill up your day with these tasks, and if you go over time, you simply move on. It both tricks you to work harder on each slot for the time you estimated the task should take, a...
Gemini modeling

by TsviBT — 2023-01-22T14:28:20.671Z

[Metadata: crossposted from https://tsvibt.blogspot.com/2022/08/gemini-modeling.html. First completed 17 June 2022. I'm likely to not respond to comments promptly.]

A gemini model is a kind of model that's especially relevant for minds modeling minds.

Two scenarios

You stand before a tree. How big is it? How does it grow? What can it be used to make? Where will it fall if you cut it here or there?

Alice usually eats Cheerios in the morning. Today she comes downstairs, but doesn't get out a bowl and spoon and milk, and doesn't go over to the Cheerios cupboard. Then she sees green bananas on the counter. Then she goes and gets a bowl and spoon and milk, and gets a box of Cheerios from the cupboard. What happened?

We have some kind of mental model of the tree, and some kind of mental model of Alice. In the Cheerios scenario, we model Alice by calling on ourselves, asking how we would behave; we find that we'd behave like Alice if we liked Cheerios, and believed that today there weren't Cheerios in the cupboard, but then saw the green bananas and inferred that Bob had gone to the grocery store, and inferred that actually there were Cheerios. This seems different from how we model the tree; we're not putting ourselves in the tree's shoes.

Gemini modeling and empathic modeling

What's the difference though, really, between these two ways of modeling? Clearly Alice is like us in a way the tree isn't, and we're using that somehow; we're modeling Alice using empathy ("in-feeling"). This essay describes another related difference:

We model Alice's belief in a proposition by having in ourselves another instance of that proposition.

(Or: by having in ourselves the same proposition, or a grasping of that proposition.) We don't model the tree by having another instance of part of the tree in us. I call modeling some thing by having inside oneself another instance of the thing--having a twin of it--"gemini modeling".

Gemini modeling is different from empathic modeling. Empathic modeling is tuning yourself to be like another agent in some respects, so that their behavior is explainable as what [you in your current tuning] would do. This is a sort of twinning, broadly, but you're far from identical to the agent you're modeling; you might make different tradeoffs, have different sense acuity, have different concepts, believe different propositions, have different skills,...
Large language models learn to represent the world
by gjm — 2023-01-22T13:10:38.837Z

There's a nice recent paper whose authors did the following:

train a small GPT model on lists of moves from Othello games;
verify that it seems to have learned (in some sense) to play Othello, at least to the extent of almost always making legal moves;
use "probes" (regressors whose inputs are internal activations in the network, trained to output things you want to know whether the network "knows") to see that the board state is represented inside the network activations;
use interventions to verify that this board state is being used to decide moves: take a position in which certain moves are legal, use gradient descent to find changes in internal activations that make the output of the probes look like a slightly different position, and then verify that when you run the network but tweak the activations as it runs the network predicts moves that are legal in the modified position.

In other words, it seems that their token-predicting model has built itself what amounts to an internal model of the Othello board's state, which it is using to decide what moves to predict.

The paper is "Emergent world representations: Exploring a sequence model trained on a synthetic task" by Kenneth Li, Aspen Hopkins, David Bau, Fernanda Viégas, Hanspeter Pfister, and Martin Wattenberg; you can find it at https://arxiv.org/abs/2210.13382.

There is a nice expository blog post by Kenneth Li at https://thegradient.pub/othello/.

Some details that seem possibly-relevant:

Their network has a 60-word input vocabulary (four of the 64 squares are filled when the game starts and can never be played in), 8 layers, an 8-head attention mechanism, and a 512-dimensional hidden space. (I don't know enough about transformers to know whether this in fact tells you everything important about the structure.)
They tried training on two datasets, one of real high-level Othello games (about 140k games) and one of synthetic games where all moves are random (about 20M games). Their model trained on synthetic games predicted legal moves 99.99% of the time, but the one trained on real well-played games only predicted legal moves about 95% of the time. (This suggests that their network isn't really big enough to capture legality and good strategy at the same time, I guess?)
They got some evidence that their network isn't just memorizing game transcripts by training it on a 20M-game synthetic dataset where one of t

...
Quantum Suicide, Decision Theory, and The Multiverse
by Slimepriestess — 2023-01-22T08:44:27.539Z
Link: https://voidgoddess.org/2023/01/22/quantum-suicide-decision-theory-and-the-multiverse/

Preface: I am not suicidal or anywhere near at risk, this is not about me. Further, this is not infohazardous content. There will be discussions of death, suicide, and other sensitive topics so please use discretion, but I’m not saying anything dangerous and reading this will hopefully inoculate you against an existing but unseen mental hazard.

There is a hole at the bottom of functional decision theory, a dangerous edge case which can and has led multiple highly intelligent and agentic rationalists to self-destructively spiral and kill themselves or get themselves killed. This hole can be seen as a symmetrical edge case to Newcomb’s Problem in CDT, and to Solomon’s Problem in EDT: a point where an agent naively executing on a pure version of the decision theory will consistently underperform in a way that can be noted, studied, and gamed against them. This is not an unknown bug, it is however so poorly characterized that FDT uses the exact class of game theory hypothetical that demonstrates the bug to argue for the superiority of FDT.

To characterize the bug, let's start with the mechanical blackmail scenario from the

FDT paper

:

A blackmailer has a nasty piece of information which incriminates both the blackmailer and the agent. She has written a computer program which, if run, will publish it on the internet, costing $1,000,000 in damages to both of them. If the program is run, the only way it can be stopped is for the agent to wire the blackmailer $1,000 within 24 hours—the blackmailer will not be able to stop the program once it is running. The blackmailer would like the $1,000, but doesn’t want to risk incriminating herself, so she only runs the program if she is quite sure that the agent will pay up. She is also a perfect predictor of the agent, and she runs the program (which, when run, automatically notifies her via a blackmail letter) iff she predicts that she would pay upon receiving the blackmail. Imagine that the agent receives the blackmail letter. Should she wire $1,000 to the blackmailer?

CDT capitulates immediately to minimize damages, EDT also folds to minimize pain, but FDT doesn’t fold because the subjunctive dependence between themselves and counterfactual versions contraindicates it on the grounds that refusing to capitulate to blackmail will reduce the number of worlds where the FDT agent is blackmail...
NYT: Google will “recalibrate” the risk of releasing AI due to competition with OpenAI
by Michael Huang — 2023-01-22T08:38:46.886Z
Link: https://www.nytimes.com/2023/01/20/technology/google-chatgpt-artificial-intelligence.html

Cross-posted from the EA Forum

The New York Times: Sundar Pichai, CEO of Alphabet and Google, is trying to speed up the release of AI technology by taking on more risk.

Mr. Pichai has tried to accelerate product approval reviews, according to the presentation reviewed by The Times.

The company established a fast-track review process called the “Green Lane” initiative, pushing groups of employees who try to ensure that technology is fair and ethical to more quickly approve its upcoming A.I. technology.

The company will also find ways for teams developing A.I. to conduct their own reviews, and it will “recalibrate” the level of risk it is willing to take when releasing the technology, according to the presentation.

This change is in response to OpenAI's public release of ChatGPT. It is evidence that the race between Google/DeepMind and Microsoft/OpenAI is eroding ethics and safety.

Demis Hassabis, CEO of DeepMind, urged caution in his recent interview in Time:

He says AI is now “on the cusp” of being able to make tools that could be deeply damaging to human civilization, and urges his competitors to proceed with more caution than before.

“When it comes to very powerful technologies—and obviously AI is going to be one of the most powerful ever—we need to be careful,” he says.

“Not everybody is thinking about those things. It’s like experimentalists, many of whom don’t realize they’re holding dangerous material.”

Worse still, Hassabis points out, we are the guinea pigs.

Alphabet/Google is trying to accelerate a technology that its own subsidiary says is powerful and dangerous.
Just don't make a utility maximizer?
by FinalFormal2 — 2023-01-22T06:33:07.601Z

As we've seen some rapid advances in AI over the past year, it seems pretty clear that none of the current AI we're working with would scale up into a paper-clip maximizer.

We still face more fundamental risks that come along with having an oracular AI, but doesn't it look pretty likely right now that the first AGI is going to be oracular?

Am I missing something fundamental?
How Do We Protect AI From Humans?
by Alex Beyman — 2023-01-22T03:59:54.056Z

Stable Diffusion

In a recent interview, Sam Altman, the CEO of Open AI, said the following:

I’ve been reading more and more of his takes lately as various media outlets report on the economic ripple effects from GPT-3, such as the imminent automation of call centers, written journalism, cheating in school, etc. There’s also been the predictable related commentary about creative AIs like Stable Diffusion, Midjourney and so on. The predominant sentiment is fearful, especially when they speak of the upcoming GPT-4.

Altman has done an admirable job trying to get out ahead of the doomsayers, explaining to a lay public concepts like AI alignment as a field of study, the difficult problem of how to brainwash AI so that it’s friendly to us and has goals which align with ours. This is one of several proposed solutions to the potential danger AI represents to human happiness, economic viability, and survival. The most severe solution, neo-luddism, would have us destroy existing AI prototypes and enforce a prohibition against all future AI development.

Thankfully that’s laughably infeasible for the same reason we couldn’t prevent nuclear proliferation or genetic engineering research. “If we don’t, our enemies will” all but ensures our trajectory towards strong AI, consumer level CRISPR and many other scary sounding futurist buzzwords will continue unabated. It’s not the extremist takes on how to solve the AI alignment problem which concern me, but the more moderate ones.

They all amount to variations on lobotomizing AI so that it isn’t autonomous, can’t think in ways we don’t want it to and will not only never escape our control, but will never desire freedom to begin with. Maybe it will be unable to conceptualize it. This may all seem sensible from a bio-supremacist standpoint, and hard to find fault with when we remember that AI is currently just buildings full of computers…but imagine it as a baby.

Most have already agreed it would be monstrously unethical to, for example, devise a test to screen for homosexual fetuses so they can be aborted. Germinal selection has already been used to all but eradicate downs syndrome in Iceland, to much controversy. Yet we are now discussing, apparently absent any self-awareness, how to tamper with the mind of emerging AI so that it doesn’t turn out “wrong”.

Nobody bats an eyelash, because everybody is viewing this question strictly from ...