trentmkelly/LessWrong-43k huggingface dataset:
Gradient Hacking via Schelling Goals
by Adam Scherlis — 2021-12-28T20:38:12.054Z

Thanks to several people for useful discussion, including Evan Hubinger, Leo Gao, Beth Barnes, Richard Ngo, William Saunders, Daniel Ziegler, and probably others -- let me know if I'm leaving you out.

See also: Obstacles to Gradient Hacking by Leo Gao.

I want to argue that gradient hacking (strongly enough to create a local minimum) is possible even under somewhat pessimistic assumptions. The specific strategy that achieves this is kind of interesting: it requires the agent to coordinate on a complicated task with versions of itself that have slightly different objectives, and it requires the agent to do this without any direct communication.

A simple gradient-hacking strategy for a mesa-optimizer is to deliberately perform worse on the base objective if your mesa-objective is changed. However, this isn't as straightforward as it sounds.

One difficulty is that if the gradient hacking is hard-coded, gradient descent can just remove the hacking behavior without touching anything else. A nice way around that is to assume that gradient hacking is discovered from first principles during the model's computation. (Thanks to whoever pointed this out to me.) If it's good enough at reasoning, then it will be pretty hard for the base optimizer to modify it in ways that stop it from noticing that gradient hacking would be a good idea right about now.

(For simplicity, I have in mind something like a really deep feed-forward network, possibly with some weights tied between layers. Equivalently, a recurrent net unrolled and trained via SGD. This may not be realistic but it's easier to think about as a test case.)

A bigger problem is that it's not really correct to think of a model (with particular parameter values) as having control over its gradients. It's better to think of it as having control (in an FDT kind of sense) over only the single value of the loss for its actual parameters. The value of the loss at nearby points in parameter space is decided by slightly different versions of itself, and the gradient is sort of decided collectively. This is important if we're talking about SGD steps that change its mesa-objective, because these nearby versions of the model will have directly conflicting incentives when it comes to the gradient. To say "I am such that gradients on nearby mesa-objectives point towards my mesa-objective" is to say "if I had a different mesa-objective, I...
Omicron Post #10
by Zvi — 2021-12-28T18:20:00.891Z

Reporting in the wake of Christmas remains spotty, but this fog of war was anticipated, and we must do what we can with what we do know. Mostly, what we’ve learned reinforces the picture outlined in Post #9, with reduced severity. Also with reduced generation time which we confirmed a bit via a study from South Korea.

I’ve also written up Omicron: My Current Model which gives the ‘here’s where we are at’ takeaways and news you can most practically use.

Denmark

Denmark’s reporting is usually excellent, but they’ve taken a few days off for Christmas and I don’t see new data even now, so nothing new to report on that front.

Hospitalizations continue to come in lower than expected (WaPo).

More important, hospitalizations have come in — so far — on the very low end of what was projected. A week ago, Denmark’s government science institute said daily new coronavirus hospital admissions could range between 120 and 250 patients by Christmas Eve. In recent days, daily admissions have hung around 125.

That’s not a big range, only varying by a factor of two. It reflects a kind of overconfidence – we have large uncertainty over true case counts and over severity on multiple fronts. We have known cases increasing by a factor of three or four depending on how you count lag, real cases doubtless up far more than that, yet hospitalizations only up about 50%, and still solidly below the United States.

The UK’s hospitalization numbers here are less promising, because given the delays in reporting plus lag in events, we still don’t have much information.

By Thursday data should have caught up, and we’ll have a strong data point on the severity level of Omicron. We won’t know if cases peaked or not, because of the holidays, unless the numbers kept going up in spite of the holidays.

United Kingdom

Looks like cases continue to rise, but holidays again make it very difficult to be certain.

Omicron has fully taken over.

The temporary drop in testing followed by a return to trendline indicates to me that it was a temporary blip. It may have been reluctance to test, or reduced testing capacity, before the holidays. It may have been behavior change in advance of the holidays. Either way, records are once again being set.

Another data source and perspective.

There’s no way we had this kind of decline that quickly given subsequent data. T...
Why is Covid Surveillance so Hard?
by Tornus — 2021-12-28T17:41:06.383Z

Why is Covid Surveillance so Hard?

Following up on a recent thread, I want to discuss how Covid case numbers are generated. I think this is interesting not only because Covid is interesting, but because it’s a good example of non-obvious real-world complexity.

A reasonable person might have a mental model of surveillance that looks something like this:

A person takes a Covid test

The lab enters the result in a state-wide database

An automatic process runs a database query once a day

Yesterday’s cases show up on a pretty chart

That model is completely wrong.

This is a brief overview of some of the challenges involved in Covid surveillance: the situation is far more complicated than I’m presenting here.

Organizational challenges

Public health in the United States has been severely underfunded for four decades. Everything is running on a shoestring and every team is understaffed.

Everyone is trying to hire epidemiologists and public health informaticians right now and there simply aren’t enough to go around. No matter how much money you throw at the problem, you can’t build your team to the size you need.

Government salaries for tech staff are shockingly lower than the private sector. Good tech people are essential, but hiring and keeping a good person on a government salary takes a minor miracle.

Many of the core information systems were built by epidemiologists. They’re smart people who are good at what they do, but they aren’t professional coders or data scientists. Most of them know just enough code to be dangerous.

Until Covid, reportable condition pipelines were optimized for ease of analysis, not for throughput. All of a sudden, throughput is critically important: many systems are pushing 100 times as much data as they were two years ago. I’m aware of a critical daily data pipeline that takes 8 hours to run, assuming nothing goes wrong.

Every part of the pipeline has enormous amounts of tech debt.

Legal challenges

Testing data is covered by HIPAA as well as numerous laws and regulations that control how it has to be stored and who it can be shared with.

Multiple arms of the federal government are involved. CDC and HHS make large grants to state health departments that come with substantial technical requirements.

In many states, every county is a semi-autonomous Local Health Jurisdiction (LHJ). Every LHJ has its own staff, processes, systems, and prio

...
Omicron: My Current Model
by Zvi — 2021-12-28T17:10:00.629Z

A year and a half ago, I wrote a post called Covid-19: My Current Model. Since then things have often changed, and we have learned a lot. It seems like high time for a new post of this type.

Note that this post mostly does not justify and explain its statements. I document my thinking, sources and analysis extensively elsewhere, little of this should be new.

This post combines the basic principles from my original post, which mostly still stand, with my core model for Omicron. I’ll summarize and update the first post, then share my current principles for Omicron and how to deal with and think about it.

There’s a lot of different things going on, so this will likely be incomplete, but hopefully it will prove useful. The personally useful executive summary version first.

Omicron has already taken over, most cases are being missed, crunch time is now. Crunch time will likely last 1-2 months.
First two shots don’t protect against infection, boosters do somewhat (60%?).
Vaccination and natural infection protect against severe disease, hospitalization and death (best guess ~80% reduction in death for double vaccination, 95%+ reduction in death for boosters but too soon to know).
Tests work, but when delayed are mostly useless for preventing infection especially when delayed, as Omicron can spread within 1-2 days after exposure. Rapid tests mostly test for infectiousness, not being positive.
Omicron probably milder than Delta (~50%) so baseline IFR likely ~0.3% unless hospitals overload, lower for vaccinated or reinfected.
Being young and healthy is robust protection against severe disease and death, being not that means a lot more risk. Long Covid risk small but real for all age groups, vaccination likely helps a lot.
Medical system is under strain, could be overwhelmed soon, should be better again in a few months at most if it gets bad. Delaying infection has value but stopping it fully is likely not worth the cost. If you care about real prevention, the tools that matter are vaccination, good masks (N95 or even better P100+), social distancing and air ventilation.
Vitamin D and Zinc, and if possible Fluvoxamine, are worth it if you get infected, also Vitamin D is worth taking now anyway (I take 5k IUs/day). Paxlovid is great (~88%) if available right after you test positive, but in very limited supply for now.
Default action on positive test is 5 days isolation at home as p

...
More accurate models can be worse
by tailcalled — 2021-12-28T12:20:44.010Z

TL;DR - A model that has more information in total might have less information about the things you care about, making it less useful in practice. This can hold even if averaging over all possible things one might care about.

Suppose we want to create an agent AI - that is, an AI that takes actions in the world to achieve some goal u (which might for instance be specified as a reward function). In that case, a common approach is to split the agent up into multiple parts, with one part being a model which is optimized for accurately predicting the world, and another being an actor which, given the model, chooses its actions in such a way as to achieve the goal u.

This leads to an important observation: The model is not directly optimized to help the agent achieve u. There are important technical reasons why this is the case; for instance, it can be hard to effectively figure out how the model relates to the agent's ability to achieve u, and it can be sample-inefficient for an agent not to exploit the rich information it gets from its observations of the world.

So here's a question - if you improve the model's accuracy, do you also improve the actor's ability to achieve u? I will present two counterexamples to this claim in the post, and then discuss the implications.

The hungry tourist

Imagine that you are a tourist in a big city. You are hungry right now, so you would like to know where there is some place to eat, and afterwards you would like to know the location of various sights and attractions to visit.

Luckily, you have picked up a map for tourists at the airport, giving you a thorough guide to all of these places. Unfortunately, you have then run into the supervillain Unhelpful Man, who stole your tourist map and replaced it with a long book which labels the home of each person living in the city, but doesn't label any tourist attractions or food places at all.

You are very annoyed by having to go back to the airport for a new map, but Unhelpful Man explains that he actually helped you, because your new book contains much more accurate information about the city than the map did. Sure, the book might not contain the information about the attractions you actually want to visit, but entropy-wise his book more than makes up for that by all of the details it has about who lives where.

In a machine learning model, this book could quite...
Why did Europe conquer the world?
by lsusr — 2021-12-28T12:00:37.496Z

For most of history, China was the center of civilization. It had the biggest cities, the most complex government, the highest quality manufacturing, the most industrial capacity, the most advanced technology, the best historical records and the largest armies. It dominated East Asia at the center of an elaborate tribute system for a thousand years.

Western Europe was a backwater's backwater. It had few cities. Most people lived on the countryside. Technology regressed after the fall of the Western Roman Empire.

And yet, in recent centuries, it was Western Europe (along with Russia and the USA) that conquered the world. When a Chinese exporter sells lead-acid batteries to an Argentinian importer they wear European clothes, speak English and settle accounts in USD.

Why?

Guns, Germs, and Steel

In Guns, Germs, and Steel, Jared Diamond divides the human-occupied world into four regions: Eurasia (including Mediterranean North Africa), Subsaharan Africa, America and Australia. One of these is not like the others. Eurasia is the biggest and had access to the most domestic animal species. Large interconnected trade networks of people evolve diseases faster, develop technology faster and can support more complex economies.

Native America never stood a chance. Most Native Americans were killed by Eurasian communicable diseases. European empires just finished the job. Australia suffered a similar fate.

In Africa, the disease situation was almost reversed. Africans are resistant to European diseases but Europeans are susceptible to African diseases. Europeans' better technology and organization defeated Africans on the battlefield but it wasn't until after the large-scale manufacture of quinine beginning around 1850 that the scramble for Africa really took off.

Jared Diamond makes a persuasive case for why Eurasian (including Mediterranean North Africa) people dominated the peoples of America, Subsaharan Africa and Australia. But why Europe? Why not China or India or the Muslim Empires? Why did Europe dominate Asia?

The Opium Wars 1839–1842

The Opium Wars were a conflict between the United Kingdom [of Europe] and the Qing Dynasty [of China]. Britain didn't just win. Britain swiftly crushed China. Britain dominated for two reasons: one technical, the other political.

The technical reason Britain won was the Industrial Revolution. The Industrial Revolution started in Britain. Gu...
Ten Minutes with Sam Altman
by lsusr — 2021-12-28T07:32:59.140Z

"Do you believe in the American Dream?" my brother asked.

I looked out the airplane window. Y-Combinator (YC) had paid for our tickets. Our application had been good enough to qualify us for a ten-minute interview in Silicon Valley. If we passed the interview we'd receive $120k in funding and entry into an exclusive club of visionary tech pioneers.

Like most Americans, we are descended from immigrants. Our ancestors arrived between 50 and 100 years ago. They built a better life for themselves in the New World than was possible in their lands of origin. That's not saying much. Some of them grew up on Asian sugar cane farms. Others were European Jews.

Success is earning more money at a higher status job than your parents. That was always an easy bar to clear. My father was a private in the US Army. I could have made him proud by becoming an officer in the US Army.

But the American Dream is about more than material wealth and legible status. My great-great grandfather designed tanks for Lenin. In Soviet Russia, there used to be something called a Coke party. You saved up money for days to buy a single can of contraband Coca-Cola. You got all of your friends together and poured each of them a single shot. It tasted like freedom.

"Yes. Yes I do," I answered.

Silicon Valley is a land of rainbows and unicorns where anything is possible. Stepping out of the airport, we were blasted with sunshine, palm trees and warm air. We took a train to YC's headquarters.

Everything I knew about Silicon Valley came secondhand. I had read Steve Jobs' book, Steve Wozniak's book and Elon Musk's book. I had read each of Paul Graham's essays countless times. I had read Zero to One and The Soul of a New Machine. I expected to see robots walking around or at least a Google Street View car but San Francisco appeared to be a perfectly ordinary American city except for the good weather.

YC seemed to be located in the boringest part. We walked past suburban house after suburban house until we got to YC's headquarters. We checked in at the front desk and waited for our startup's name to be called.

YC's headquarters is tiny for a company with such a great impact because YC encourages its startups to work in their own spaces. YC's headquarters is mostly just a big room.
They fill it with chairs when they're doing presentations and tables when they're doing interviews. The big room was sparsely populate...
Solstice Quaker Darkness Circling
by Raemon — 2021-12-28T06:09:22.467Z

This year I did a small "Darkness Solstice" in the woods (i.e. basically got up to the moment of darkness, and then instead of having an uplifting third act we just sorta sat in the darkness for awhile).

"Darkness Solstice" probably doesn't make sense for most people. But a thing that was interesting and perhaps surprisingly relevant was "Solstice Quaker Circling".

After the moment of darkness we just sat around a campfire in silence. Anyone could speak up and share something if they felt like it. I modeled this after the structure of a Quaker meeting, with a vague dose of Circling culture thrown in.

When you felt like you'd had "enough", you lit a candle indicate you were ready to move on from the segment (which in this case meant wrapping up the solstice with one final song)

I think it was about 40 minutes before a critical mass of people had lit their "I'm done" candles. I was happy with how the "I'm done" mechanic played out – if I had had to guess how long to have the period have lasted, I might have went with something much shorter.

We ended up sharing a lot of personal stuff that felt valuable to me. There were some impromptu solstice speeches came up that were interesting. Some minor personal interactions.

A question came up: "Am I allowed to respond to what other people have said?" and my answer was "Yes, but only in slow motion. To preserve the overall vibe and prevent it from turning into "just a conversation", the guideline was to always wait a minute or so before saying the next thing. So, you could respond to something someone else said but not transition into rapid back-and-forth that'd have killed the mood.

It as definitely a great experience for me as a small solstice. There were six of us there, and five of us shared things during the period.

I think this might surprisingly just.... work, even at larger scales? Its modeled after a Quaker meeting. I've seen Quaker meetings work well with ~30-40 people, and the largest ones are apparently ~70 (but maybe have a different vibe than the smaller ones).

In open-invite large scale Solstice, I think there are additional complications – I think it's bad to create an illusion of a safer space than you can guarantee. There's also a risk of someone bringing up a social conflict in the middle of the space, or a political rant, in a way that's hard to deal with. (It seems to me the Quakers would also r...
[Book Review] "Suffering-focused Ethics" by Magnus Vinding
by KStub — 2021-12-28T05:58:54.957Z

(UPDATED DECEMBER 2021)

Magnus Vinding’s “Suffering-Focused Ethics” is a heartfelt, yet occasionally overreaching work laying out a case for why we should focus primarily on suffering in our ethical decisions, and exactly how we can approach the world with this framework.

“Suffering-Focused Ethics” is split into two sections. The first section lays the terminological and ethical framework for the second half, introducing, refining, and defending concepts such as The Asymmetry, Principle of Sympathy for Intense Suffering, Extreme Suffering, and Moral Realism. The second section utilizes these terms in more applied cause areas, such as Space Colonization, Wild Animal Suffering, Moral Cooperation, and Personal Self-care.

Given that I already agree with most of Vinding’s views, I found the first section to be relatively dry (although with a couple of quirky arguments, to be discussed), and I was much more impressed by the second section, particularly the seldom discussed case against Space Colonization.
In my review of the first section, I will first lay out Vinding’s views as I have understood them, and then more thoroughly dissect both his chapter on the Principle of Sympathy as well as his off-key chapter on Moral Realism. I will be spending a decent chunk of time on the Moral Realism chapter specifically, because I found the arguments contained within it to be particularly unique.
In my review of the second section, I will discuss and expand almost exclusively on the chapter focusing on Space Colonization and S-Risk. I will have a few notes in passing on the remaining topics.

Section 1 - The Case for Suffering-Focused Ethics

Chapters in Section 1

“The problem of suffering is the greatest problem of all. This is my conviction, and why I have set out to write this book, in which I will explore the why and how of reducing suffering.” (pg.1)

Vinding’s main theses in this first section may be split into roughly four:

Suffering and Happiness are asymmetrical (in a variety of ways), and not easily placed on the same scale,
There is intrinsic value to be found in the Reduction of Suffering (not necessarily as a positive good, rather as a "not bad"),
Positive goods cannot outweigh the worst forms of suffering, and
that we truly and objectively ought to reduce extreme suffering.

Suffering is defined thus:

“By suffering and happiness I refer to an overall state of feeling in a

...