trentmkelly/LessWrong-43k huggingface dataset:
Thoughts On (Solving) Deep Deception
by Jozdien — 2023-10-21T22:40:10.060Z

Thanks to Paul Colognese for conversations that contributed to the thoughts in this post being developed, and to Johannes C. Mayer, Tassilo Neubauer, and Q for feedback on a draft.

Back in 2000, a computer scientist named Charles Ofria was studying the evolution of simulated organisms[1]. He wanted to limit their replication rate, so he programmed the simulation to pause after each mutation, measure the mutant’s replication rate in an isolated test environment, and delete the mutant if it replicated faster than its parent.

Eventually, organisms evolved to recognize and pause their replication in the test environment and resume normal replication when released. Ofria tried solving this by making it harder to detect the test environment, but the organisms simply started to randomly accelerate their replication so as to slip through the test environment some of the time.

He did eventually find a solution to his problem, to just monitor and eliminate in real-time the organisms that started to replicate faster than their ancestors[2].

So, three takeaways I want to highlight here:

The “deception” the organisms engaged in didn’t involve any cognition on their behalf trying to figure out how to deceive Ofria. Nowhere in this whole setup is a system you can try to find deceptive thinking in and find a way out of your problem. Deception was a property of the environment and its overseer, more than of the organisms.
When you’re incentivized to perform well on some task (in this case, to replicate), and your oversight isn’t very robustly tracking the True, complex form of the thing you care about, eventually optimization pressure will break the coupling between whatever you were tracking and the thing you were actually tracking, the latter of which isn’t going to be very useful to you. This applies to the rough shape of the cognition itself.
The solution that worked in the end was noticing when the specific bad thing he didn’t want to happen was starting to happen and then shut it down.

Framing Deep Deception

So, here’s a way to frame deep deception: You can do things that end up misleading someone without explicitly thinking “How do I go about deceiving this person?” If you simply disallow AI that explicitly thinks “How do I go about deceiving this person?” in a context where deceiving people is very beneficial, eventually you might get an AI that deceives people without t...
Best effort beliefs
by Adam Zerner — 2023-10-21T22:05:59.382Z

Alice: Hey Bob, how's it going?

Bob: Pretty good. How about you?

Alice: Pretty good. What are you drinking?

Bob: A Bayeslight.

Alice: Cool. Hey bartender, lemme get one of those as well.

Alice: Hey, have you seen President Hanson's new immigration policy?

Bob: No. What is it?

Alice: He's shutting down the borders pretty hard. It's like an 8.5/10 on the left-right political axis, where 10/10 is ultra conservative and 0/10 is ultra liberal.[1]

Bob. Oh gawd. He's such an idiot.

Alice: Oh yeah? What sort of immigration policy would you implement?

Bob: Hm, I think something like a 2/10 on the left-right spectrum.

Alice: And how confident are you in that?

Bob: Probably like an 8/10.

Alice: Ok, let's call this 2/10 on the left-right spectrum belief with an 8/10 confidence a 2-8 belief. I disagree and will bet you $10 on it.

Bob: But how are we going to bet? What are we betting on? What's the resolution criteria?

Alice: Proposes some clever thing to bet on with crystal clear resolution criteria that perfectly captures everything.

Bob: Wow. Ok. Yeah, that works. But actually, after thinking about it a little more, I think I'm more like a 2.5-7. Can we bet at those odds instead?

Alice: We can. But only if we can bet $100 instead of $10.

Bob: Oh. Ummmm. Ok, I guess I'm probably more like a 3-6.5. I could do that at $100. Is that ok?

Alice: No. But I'd do 3-6.5 at $1,000. Shake on it?

Bob: Woah... Alice, this got real expensive real fast.

Alice: What's the matter Bob? Do you need to chug a few Bayeslights first?

Bob: Fuck you. I've got a family to feed over here.

Alice: Pansy. Wussy. That's a cop out and you know it.

Bob: Sigh. Ok, ok. Let me think about this. I am a Bayesian, and I do agree that Bayesians should be perfectly happy to bet on their beliefs when doing so would be +EV.

Alice: Right...

Bob: But... it's just that... well, $1,000 is kind of a lot of money.

Alice: Is it? You're semi-retired. You own your home. Your kids are in college. You have a fund to pay for college. You don't have expensive taste. You always talk about how you'd be perfectly fine retreating to some cabin in the woods somewhere.

Bob: I mean, yeah. You're right.

Alice: Ok, so what's the problem? It looks like things are pretty inelastic. Like, the amount of utility you'd gain if you won $1,000 is pretty much the same as what you'd lose if you lost $1,000. (Save for the initial short-term, Fisher-ian, wussy-pussy feelin...
How toy models of ontology changes can be misleading
by Stuart_Armstrong — 2023-10-21T21:13:56.384Z

Before solving complicated problems (such as reaching a decision with thousands of variables and complicated dependencies) it helps to focus on solving simpler problems first (such as utility problems with three clear choices), and then gradually building up. After all, "learn to walk before you run", and with any skill, you have to train with easy cases at first.

But sometimes this approach can be unhelpful. One major examples is in human decision-making: trained human experts tend to rapidly "generate a possible course of action, compare it to the constraints imposed by the situation, and select the first course of action that is not rejected." This type of decision making is not an improved version of rational utility-maximisation; it is something else entirely. So a toy model that used utility-maximisation would be misleading in many situations.

Toy model ontology failures

Similarly, I argue, the simple toy models of ontology changes are very unhelpful. Let's pick a simple example: an agent making diamonds.

And let's pick the simplest ontology change/model splintering that we can: the agent gains access to some carbon-13 atoms.

Assuming the agent has only ever encountered standard carbon-12 atoms during its training, how should it handle this new situation?

Well, there are multiple approaches we could take. I'm sure that some spring to mind already. We could, for example...

...realise that the problem is fundamentally unsolvable.

If we have trained our agent with examples of carbon-12 diamonds ("good!") versus other arrangements of carbon-12 ("bad!") and other non-carbon elements ("bad!"), then we have underdefined what it should do with carbon-13. Treating the carbon-13 the same as carbon-12 ("good iff in diamond shape") or the same as other elements ("always bad"): both options are fully compatible with the training data.

Therefore there is no "solving" this simple ontology change problem. One can design various methods that may give one or another answer, but then we are simply smuggling in some extra assumptions to make a choice. The question "what should a (carbon-12) diamond maximiser do with carbon-13 if it has never encountered it before?" does not have an answer.

Low-level preferences in one world-model do not tell an agent what their preferences should be in another.

More complex models give more ontology-change information

Let's make the toy model more co...
Soups as Spreads
by jefftk — 2023-10-21T20:30:08.320Z

We tend to cook in quantities where there are leftovers: it's not
really practical to cook exactly the right amount of food, and it's
much better to have a bit extra than not enough. Plus with several
people working from home we tend to go through a good amount of
leftovers for lunches. One thing I've discovered recently with
leftovers: thicker sauces and soups make surprisingly good spreads.

For example, I recently had tomato soup sandwiches:

While something watery or brothy will stay that way even when cooler,
most things thicken up in the fridge to where they'll spread well on
bread. And it's very tasty!

I still mostly eat any leftover soups and sauces reheated, but if I
want something more portable I've been pretty happy with this.
Which COVID booster to get?
by Sameerishere — 2023-10-21T19:43:04.273Z

Any thoughts on the Novavax vs Moderna / Pfizer vaccines?

Based on what I've read, it seems like they have comparable efficacy, with the potential for less intense side effects from Novavax, so I'm leaning towards that one (having gotten Pfizer initially, then Moderna, which left me quite fatigued and mildly flu-ish for about a day each time).

Incidentally, my side effects were appreciably better when I got the flu vaccine separately -- it's possible that was just the initial series vs booster or something, but I definitely will be doing those separately this time around as well.
Comparable efficacy:
- 10/6/23 Science article: https://www.science.org/content/article/should-you-pick-novavax-s-covid-19-shot-over-mrna-options
- 10/17/23 Lifehacker post: https://www.msn.com/en-US/health/medical/is-the-novavax-covid-vaccine-actually-better/ar-AA1inrUm?ocid=sapphireappshare
Less intense side effects:

8/29/23 Your Local Epidemiologist post https://yourlocalepidemiologist.substack.com/p/your-top-7-questions-about-fall-vaccines?utm_source=substack&utm_medium=email

[Updated 11/7]

However, for those who do have side effects, it looks like they might kick in later? The median onset of side effects is 2 days after the vaccination, according to the CDC.

The research above still stands in terms of frequency, but if the trajectory I experienced is a common pattern with Novavax but not with mRNA (haven't researched the latter), I'd have preferred the mRNA vaccines... would rather have gotten the side effects out of the way over the weekend as planned vs unpredictably at the start of a workweek...)

My anecdotal evidence:

I had negligible side effects up till about 40 hours after getting the Novavax booster
At that point, I started to feel enough fatigue and malaise that it was quite unpleasant to do work, as well as mild nausea / stomach discomfort. I napped most of the rest of the day, with some short stretches of work in between. Pepto Bismol addressed my stomach discomfort almost entirely, but I didn't really feel like eating anything for the rest of the day. I'd say the severity of these issues was perhaps a little better than what I experienced with the mRNA vaccines, but still in the ballpark - bad enough to keep me from doing anything fun / productive, and the mRNA side effects were not significantly more miserable.
I got extra sleep at night,

...
Alignment Implications of LLM Successes: a Debate in One Act
by Zack_M_Davis — 2023-10-21T15:22:23.053Z

Doomimir: Humanity has made no progress on the alignment problem. Not only do we have no clue how to align a powerful optimizer to our "true" values, we don't even know how to make AI "corrigible"—willing to let us correct it. Meanwhile, capabilities continue to advance by leaps and bounds. All is lost.

Simplicia: Why, Doomimir Doomovitch, you're such a sourpuss! It should be clear by now that advances in "alignment"—getting machines to behave in accordance with human values and intent—aren't cleanly separable from the "capabilities" advances you decry. Indeed, here's an example of GPT-4 being corrigible to me just now in the OpenAI Playground:

Doomimir: Simplicia Optimistovna, you cannot be serious!

Simplicia: Why not?

Doomimir: The alignment problem was never about superintelligence failing to understand human values. The genie knows, but doesn't care. The fact that a large language model trained to predict natural language text can generate that dialogue, has no bearing on the AI's actual motivations, even if the dialogue is written in the first person and notionally "about" a corrigible AI assistant character. It's just roleplay. Change the system prompt, and the LLM could output tokens "claiming" to be a cat—or a rock—just as easily, and for the same reasons.

Simplicia: As you say, Doomimir Doomovitch. It's just roleplay: a simulation. But a simulation of an agent is an agent. When we get LLMs to do cognitive work for us, the work that gets done is a matter of the LLM generalizing from the patterns that appear in the training data—that is, the reasoning steps that a human would use to solve the problem. If you look at the recently touted successes of language model agents, you'll see that this is true. Look at the chain of thought results. Look at SayCan, which uses an LLM to transform a vague request, like "I spilled something; can you help?" into a list of subtasks that a physical robot can execute, like "find sponge, pick up the sponge, bring it to the user". Look at Voyager, which plays Minecraft by prompting GPT-4 to code against the Minecraft API, and decides which function to write next by prompting, "You are a helpful assistant that tells me the next immediate task to do in Minecraft."

What we're seeing with these systems is a statistical mirror of human common sense, not a terrifying infinite-compute argmax of a random utility function. Conversely, w...
How to find a good moving service
by Ziyue Wang — 2023-10-21T04:59:07.814Z

2 weeks ago, I realized that I will move from Seattle to Mountain View in 50 days. I was excited but I didn't realize I need to talk with >15 moving companies to find the satisfying ones. After ~9 hours of hunting, I finally found a satisfying one. I chose North American Moving Service for my 850 miles, 2000lbs, 300 cubic feet (CF) moving for the price of 3200$. Here is what I learned (starting from the most important one to the least):

Choose Carrier (Mover) instead of Broker: Brokers can cost more in the end even when it sounds cheaper in the beginning. After you paid the deposit, things can go out-of-control. I found many stories online about the brokers charge additional fee a few days before the moving so that you have no other choices but pay them the extra money. Some stories also tell that when the movers come, they gonna charge you extra money because they will say they don't agree with the broker's price... Choose a carrier will be cheaper and mentally safer.

How to find whether the other side is a broker or mover? Ask their USDOT number and use the following website to search about it. The entity type should say "Carrier" instead of "Broker". Usually, for a mover, they should have their USDOT number on their website.

Don't trust Google search results: "long-distance moving is badly regulated", the agent from one of the largest moving company told me. When I search for "best moving company" in Google Search, most of them are brokers. They simply pay a lot of advertisement fee to get into the top spots so people will click for it. It is reasonable because they can easily earn >1k$ for each service they "provided". Also, the Google reviews can be fake as well. Don't rely on Google as the only source of truth.

Who to trust?

https://www.movingscam.com: People who get cheated before created this website. Specifically, this superlist lists good moving services across different states.
https://www.movingauthority.com/largest-moving-companies/: The larger the company, the less likely they are willing to lose credibility.
https://ai.fmcsa.dot.gov/hhg: Check how many times the moving company get sued.
https://li-public.fmcsa.dot.gov/LIVIEW/pkg_carrquery.prc_carrlist : check the insurance is legit or not.

Pay attention to the following bad signs:

They don't estimate how much weight and how much cubic feet (CF): one broker I talked with, iMoving, even didn't

...
Apply for MATS Winter 2023-24!
by utilistrutil — 2023-10-21T02:27:34.350Z

Applications are now open for the Winter 2023-24 cohort of MATS (previously SERI MATS). Our mentors are: Adrià Garriga Alonso, Alex Turner, Buck Shlegeris, Caspar Oesterheld, Daniel Murfet, David ‘davidad’ Dalrymple, Erik Jenner, Ethan Perez, Evan Hubinger, Francis Rhys Ward, Jeffrey Ladish, Jesse Clifton, Jesse Hoogland, Lee Sharkey, Neel Nanda, Owain Evans, Stephen Casper, Vanessa Kosoy, and researchers at Sam Bowman's NYU Alignment Research Group, including Asa Cooper Stickland, Julian Michael, Shi Feng, and David Rein.

Submissions for most mentors are due on November 17 (and for Neel Nanda on November 10). Many mentors ask challenging candidate selection questions, so make sure you allow adequate time to complete your application. We encourage prospective applicants to fill out our brief interest form to receive program updates and application deadline reminders. You can also fill out our recommendation form to let us know about someone who might be a good fit, and we will share our application with them.

We are currently funding constrained and accepting donations to support further research scholars. If you would like to support our work, you can donate here!

Program Details

MATS is an educational seminar and independent research program (40 h/week) in Berkeley, CA that aims to provide talented scholars with talks, workshops, and research mentorship in the field of AI alignment, and connect them with the Berkeley AI safety research community. MATS provides scholars with housing in Berkeley, CA, as well as travel support, a co-working space, and a community of peers. The main goal of MATS is to help scholars develop as AI safety researchers. You can read more about our theory of change here.

Based on individual circumstances, we may be willing to alter the time commitment of the program and arrange for scholars to leave or start early. Please tell us your availability when applying. Our tentative timeline for the MATS Winter 2023-24 program is below.

Scholars will receive a USD 12k stipend from AI Safety Support for completing the Training and Research Phases.

Applications (Now!)

Applications open: October 20

Applications are due: November 17

Note: Neel Nanda's applicants will follow a modified schedule; see section below.

Training Phase (January 8 - January 21)

Accepted applicants are generally expected to have completed the AI Safety Fundamentals Alignment ...
Can we isolate neurons that recognize features vs. those which have some other role?
by Joshua Clancy — 2023-10-21T00:30:11.758Z

While I think about interpretability a lot, it's not my day job! Let me dive down a rabbit hole and tell me where I am wrong.

Intro

As I see it the first step to interpretability would be isolating which neurons perform which role. For example, which neurons are representing patterns/features/entities in the input vs. which neurons are transforming data into higher dimensions as to be more easily separable.

Of course, distinguishing neurons by role opens its own can of worms (what are the other roles? What about neurons with dual roles? etc.) And we have the common problems of: what if a neuron is representing a part of a feature? What if a neuron only in concert with other neurons represents some entity?

Regardless I would love to know about some methods of isolating the neurons which are specifically recognizing and representing features vs. any other role. Am I missing something in the literature?

My Hypothesis - near linearity is an identifiable quality of neurons in the representational role.

Just as the structure of a heart, or a leaf, or an engine belies its purpose, I believe there ought to be some identifiable indication that a neuron represents some external entity rather than acting in some other role.

My initial guess is that neurons that primarily serve a representational role tend to exhibit a more linear relationship between their inputs and outputs. This observation aligns with the intuitive understanding that neurons that specialize in representing entities focus on directly identifying or classifying features present in the input data. Their output I surmise is often a near-linear combination of these features.

On the other hand, neurons that transform data are more concerned with altering the topological or dimensional properties of the input space. These neurons often introduce complexities and nonlinearities as they project data into higher-dimensional or more abstract spaces. This transformation aids subsequent layers in the network but makes the relationship between their inputs and outputs less straightforward, and nonlinear.

I would also guess that there are neurons whose role is simply to pass-along information to further layers. This complicates things, because I would guess that this pass-along role also produces near-linear combinations of input features. Though I suggest we can isolate these neurons, by noting tha...
Muddling Along Is More Likely Than Dystopia
by Jeffrey Heninger — 2023-10-20T21:25:15.459Z

Summary: There are historical precedents where bans or crushing regulations stop the progress of technology in one industry, while progress in the rest of society continues. This is a plausible future for AI.

Epistemic Status: My intuition strongly disagrees with other people here. I hope to explain my intuition, and provide enough historical evidence to make this intuition at least plausible.

Introduction: An Intuition Pump

Suppose you told someone in 1978 that no new nuclear power plants would be built in the US until 2023.[1] This would probably be very surprising. Nuclear power was supposed to be the power of the future.[2] The Nuclear Regulatory Commission had only been created 3 years earlier.

Given this information, someone in 1978 might predict that something terrible was about to happen. Maybe a nuclear war between the USA and USSR that destroys America’s industrial capacity. Maybe economic collapse due to overpopulation or global warming. Maybe an Orwellian police state in time for 1984, or a World Authority designed to regulate nuclear weapons that got out of hand.[3]

None of this happened. Instead, the Nuclear Regulatory Commission increased the regulatory ratchet[4] until building new nuclear power plants became uneconomical. These regulations only applied to the USA, but they seem to have significantly impacted nuclear power research globally. Countries that are building new nuclear power plants are still using designs that were developed before 1970.[5]

Regulation on nuclear power probably did slow US economic growth over the next 45 years compared to the counterfactual.[6] But the past 45 years have hardly been catastrophic. Economic growth and innovation did continue, driven by other industries.

Consequences of Stopping AGI?

Some people involved in the debate about slowing or pausing AI seem to think that successfully stopping AI progress over the long term would likely lead to death or dystopia:

Either we figure out how to make AGI go well or we wait for the asteroid to hit.

- Sam Altman[7]

If we don’t get AI, I think there’s a 50%+ chance in the next 100 years we end up dead or careening towards Venezuela.

- Scott Alexander[8]

I think we should be quite worried that the global government needed to enforce such a ban would greatly increase the risk of permanent tyranny, itself an existential c

...