trentmkelly/LessWrong-43k huggingface dataset:
AGI Alignment Should Solve Corporate Alignment
by magfrump — 2020-12-27T02:23:58.872Z

cross-posted from my personal blog

I have an intuition that the gap between short term ML alignment problems and long term AGI alignment problems is smaller than usual.

Writing on the subject I usually see rarely takes this perspective. I see a few things actually get written:

People who work on long term AGI alignment saying short term ML alignment doesn’t help their use case
People who work on short term ML alignment frustrated with companies that they can’t get their work implemented
People who talk about the incentives of business structures and draw some parallels to long term AGI alignment, for example Immoral Mazes and Mesa Optimizers. Maybe I’m projecting, as I’m not sure how much this parallel is being drawn explicitly.

This seems to me like a strong throughline, especially for a Christiano-style amplification path. That is, we already observe hierarchical self-referential systems that encounter supervision problems for aligning their stated objectives (which may or may not be honest) with their pursued objectives (which may or may not be aligned) by using semi-aligned components (employees) and indirect supervision (middle management).

There has been significant discussion of the analogy between corporations and superintelligences and I don’t mean to rehash this. Certainly a superintelligent AGI may have many properties that no corporation of humans has, and one of the most likely and concerning of those properties is operating on a faster serialized time scale.

But I think it’s hard to deny that corporations are a form of very weak superintelligence that exists in the real world. And I think that means they are an opportunity to experiment with methods for alignment in superintelligences in the real world.

Or to phrase it differently, I don’t think “being able to run a good corporation” is a solution to the problem of AGI alignment, but I DO think that a solution to the problem of AGI alignment, probably in general but definitely in Christiano-style amplification, should contain a solution to being able to run a company that doesn’t experience value drift. I think this philosophy, of attempting to solve simplified sub-problems, is pretty well established as a good idea.

Unfortunately, I don’t really know that much about business or management. And I don’t know that much about the kind of selection pressures companies undergo, especia...
Some recommendations for aligning the decision to go to war with the public interest, from The Spoils of War
by Eli Tyre — 2020-12-27T01:04:47.186Z

The Spoils of War: Greed, Power, and the Conflicts that Made our Greatest Presidents is about one particular principal agent problem. Through a number of case studies, the authors, Bruce Bueno de Mesquita (one of the coauthors of the Dictator’s Handbook) and Alastair Smith, make the case that presidents (and congresses) will start wars that are advantageous to that president, or to their political faction, but are not in the interests of the nation as a whole.

To give one incentive-misalignment, the authors note in the introduction that presidents tend to be better remembered and respected by posterity if they preside over wars. In fact, there is a positive correlation between the number of americans (as a percentage of the population) who die in war during an administration, and how highly rated (by a historians) that president will be.

Making war is one of the best ways to attain personal glory, somewhat independently of whether the war is just, or even good for the country.

Similarly wartime presidents are reelected at higher rates than peacetime presidents, and therefore presidents are often incentivized to start wars as a way to secure a second term.

It seems there are a number of different incentive misalignments, of this sort. The main body of the book is a set of historical case studies outlining particular US wars, and how these were more advantageous for the sitting executive than they were for the public as a whole. I found these case studies variously compelling, some seemed very solid, some seemed more speculative.

But in the conclusion of the book, the authors offer a number of policy suggestions that might improve (on the margin) the alignment of the president with the populous on the question of going to war. This is both an important problem in and of itself, and an example of an important class of problems. so in this post, I want to outline those proposals.

Electoral Reform

The first suggestion is a very general solution, inline with the theory Bueno de Mesquita proposed in his earlier books. The larger the number of people from which the executive needs support, the more that executive's incentives are aligned with the population as a whole. So in general, if you want to avoid Presidents making war for the benefit of the few, at the expense of the many, you should set things up so that they need the support of the m...
Cognitive mistakes I've made about COVID-19
by DanielFilan — 2020-12-27T00:50:05.212Z

I think that COVID-19 has been an interesting test of rationality. We’ve faced a novel risk that was initially quite poorly characterized, with the mechanism of transmission unclear, as well as the risk of death or disability from catching it. As such, now that I’m in ‘wait for the vaccine’ mode, I think it’s fitting to do a retrospective of which mistakes I made in my thinking about how to handle the virus. Note that these are not just things that I happened to be wrong about, but cases where the way I thought about the problem was wrong, leading to predictably bad results that could have been known to be bad at the time. Here is a write-up of some mistakes that I made. It is probably not exhaustive.

During late February, I saw that the rest of my society did not appear to be taking COVID very seriously, which led me to think that it might just wash over the USA. Then, in mid-March when US cities started ‘locking down’, if I recall correctly I was quite optimistic that the country was taking it seriously and that we’d get it under control. I think that both of these reactions were wrong. The first reaction was wrong because I should have known that once enough people were getting ill to visibly overwhelm hospitals, people would start demanding that precautions be taken to control the disease. The second was wrong because I should have known that governments often react in large-scale and costly ways to problems without actually thinking carefully about how to most efficiently solve those problems.

Early on in the pandemic, I was quite interested in telling my representatives my opinions about what should be done to curb the spread of COVID. I believe my strategy was to recommend what I thought was a pretty good course of action. However, I should have realized that the US and California governments have difficulty implementing large-scale plans that require multiple things to get right. If I wanted to advocate for effective policy responses, I should have thought “what’s the simplest plan that might be adopted and would work?”. The answer to this question would likely be Paul Romer’s plan to regularly test everybody in the US. Perhaps by the point I knew of this plan I had become skeptical of my leaders’ ability to act effectively and wasn’t emailing them anymore - I don’t actually remember the timeline super well here - but I should have at least asked mysel...
Great minds might not think alike
by Eric Neyman — 2020-12-26T19:51:05.978Z
Link: https://ericneyman.wordpress.com/2020/12/26/alike-minds-think-great/

[Previously known as "Alike minds think great"]

I.

It is famously the case that almost everyone thinks they’re above average. Derek Sivers writes:

Ninety-four percent of professors say they are better-than-average teachers.

Ninety percent of students think they are more intelligent than the average student.

Ninety-three percent of drivers say they are safer-than-average drivers.

Interesting. Intuitively this seems to suggest that people are prone to vastly overestimate their competence. But is that true? As Bill Kuszmaul points out, these people aren’t necessarily wrong!

There’s no fundamental reason why you can’t have 90% of people be better than average. For example, more than 99.9% of people have an above-average number of legs. And more than 90% of people commit fewer felonies than average. These examples are obvious, but they’re not so different than some of the examples [in Sivers’ post].

This has something to it! On the other hand, I don’t think this explains everything. Is the quality of a professor’s teaching really so skewed that 94% are above average? But more importantly, do you really think that way fewer people would answer “yes” if you just replaced the word “average” with “median” when asking the question?

That said, I don’t think these numbers necessarily point to a bias! That’s because the interpretation of “above average” is left entirely up to the person being asked. Maybe you think a good driver is one who drives safely (and so you drive safely and slowly) whereas I think a good driver is one who gets from point A to point B efficiently (and so I drive quickly but not safely). We are both, from our own perspectives, above average drivers!

Put otherwise, for any skill where “goodness at that skill” doesn’t have an objective, agreed-upon measure, we should expect more than 50% of people to think they’re better than the median, because people optimize for things they care about.

To give a personal example, I suppose I would call myself an above average blogger. This isn’t true in some objective sense; it’s just that I judge bloggers by how interesting their thoughts are to me, and obviously I write about things that are interesting to me! There’s no bias I’m falling for here; it’s just that “Are you an above average blogger?” leaves “above average” open to my interpretation.

II.

There is, however, a closely related bias that I and lots of other pe...
Moral intuitions are surprisingly variable
by Bob Baker — 2020-12-26T19:50:41.721Z

When building moral frameworks, moral intuitions are “our first and only source of data”. Further quoting the Consequentalism FAQ:

Searching for moral rules means searching for principles that correctly describe and justify enough of our existing moral intuition that we feel confident applying them to decide edge cases.

The results of these searches are often proposed as universal rules. The implicit justification for this is the claimed near universality of the underlying moral intuitions:

There are many moral situations where nearly everyone agrees on the correct answer, even though we're not exactly sure why.

I suggest that more skepticism is warranted for such claims because research shows a surprising diversity of moral intuitions once more cultures are sampled.

For example, “everybody” probably believes that intentions are important. Murder is worse than man-slaughter. Sabotage is worse than clumsiness.

Barrett et al presented a random story from a pair to members of a diverse set of cultures. While the outcome/harm inflicted in the stories was the same, in one of the pair it was an accident and in the other it was intentional. In Los Angeles, as we would expect, people judge intentional harm much more severely. However, one of the nine other cultures tested — in Fiji — judged exclusively based on consequences. In another the difference was below the significance threshold.

Any moral framework that gives weight to intention and claims universality should at least pause and consider that result.

Even the degree to which people respect impartial rules varies. Hampden-Turner and Trompenaars conducted extensive research on cultural differences including asking about whether it was acceptable to lie under oath to protect a friend who had injured someone through reckless driving. (Also described in this open publication.) English-speaking countries are part of the most impartial-rule-abiding set in this measure. Places like Russia and China are much less so.

Modern Western moral frameworks are nearly always impartial, ignoring special considerations for people nearby in social or family networks. But we don't usually get to see Chinese arguments about deontology. What would moral rules look like in a culture where honoring ancestors was a high virtue? What about cultures, such as Japan, where morality may be judged based on fidelity to social norms rather tha...
An open Christmas card to my brothers
by KatjaGrace — 2020-12-26T12:00:43.004Z

Dear Brothers,

Happy Christmas! I made you a list of things you might like on the internet. I won’t tell you which of you any was intended for, so feel especially free to skip any that don’t look interesting.

Love,
Katja

(Picture: nearing Christmas in New York 2017)

Lepanto

George Saunders: The incredible Buddha boy, The New Mecca

Archive of 3495 Peter Dombrovskis photographs

Nice buildings that are mostly mosques (click the x when it misleadingly says that you need to log in to see more)

Opulent Joy: my friend’s art blog

Unsong: one of my favorite books that I haven’t finished reading.

How to do what you love: Paul Graham has some other good essays too

AI dungeon: have cutting edge AI write you a story

The oldest video: strange

It’s as simple as one, two, three

3Blue1Brown, e.g. The unexpectedly hard windmill question or the first of a 15 part series on linear algebra, if you are in for something more involved

Extensive Mandlebrot set

Rent a real goat for your video calls

The (Mormon) Tabernacle Choir

Universal love, said the cactus person

The guns falling silent at the end of WWI, reconstructed into sound from ‘sound ranging’ equipment that translated sounds onto visual film at the time.

Very impressive patty cake a cappella (and more: 3D pretty a cappella patty cake)

AI fills in a cat/house/etc if you draw the outline. I thought these buildings were pretty good:

Hamilton (the musical)

How to live for much longer in expectation
Can we model technological singularity as the phase transition?
by Valentin2026 — 2020-12-26T03:20:19.726Z

Introduction

Technological singularity is quite similar to what happens with the system near the phase transition. If it is indeed the case and the underlying mechanisms behind singularity allow the same form of the mathematical description as underlying mechanisms of the phase transition, we can potentially use this knowledge to estimate when should we expect singularity.
I wrote this post in the following way. First, I remind those who are far from physics what is the phase transition. Second, I discuss why it resembles singularity. Third, I suggest how we can make a quantitative prediction based on it. Fourth, I finally tell you why all this is important.

What is the phase transition

The phase transition is the transition from one phase to another. (duh!). The simplest example of the phase transition would be a solid-liquid or liquid-gas transition. A little bit more complicated is the transition from ferromagnetic to paramagnetic (Curie point). The transition from one phase to another happens at a set of critical parameters. For simplicity, we will just talk about the temperature here (like the temperature of boiling or of freezing). When temperature T approaches the critical temperature Tc of the transition from one phase to another many quantities (susceptibility to the magnetic field, for example) demonstrate a power-law behavior, i.e. the quantity Q depends on the temperature as

Q∝(T−Tc)α

where α controls this power law behavior and is called the critical exponent. If α<0 , the quantity diverges, i.e., approaches infinity, when T approaches Tc. It does not mean the quantity actually becomes infinite - since the size of the system is finite, it is impossible - but close to phase transition such growth is a very good description. One of such quantities, common for practically all of the systems, is correlation length, which basically tells, how far from each other in the media (bucket of water, or magnet, for example) can be two points that still influence each other in a noticeable way. This quantity always diverges near the phase transition, i.e, basically any two points become interdependent (again, limited by the size of the system). The distance between two points does not matter anymore in this case.

Does it look like tech...
What evidence will tell us about the new strain? How are you updating?
by Zvi — 2020-12-25T21:37:25.264Z

I didn't get into this question in the weekly post. The answer does not seem obvious to me.

Eventually, it will be obvious either way. Either the strain will spread rapidly across many locations, or it won't. But we would like to get the information as fast as possible, whatever the answer turns out to be. So the open question is, what numbers or other information can we use in the short term to generate useful evidence for or against the new strain being a lot more infectious than older ones?

The obvious first things to look at are England's case numbers, and how rapidly the new strain continues to outcompete the old strain. Certainly the percent dominance continuing to go up, or stabilizing, is evidence.

Alas, tonight is Christmas, and I haven't been keeping up in detail with English reporting and testing dynamics. We saw only a small bump on Christmas Eve, after big jumps in the previous post-announcement days, and I can't tell how much of that was the holiday. Same will apply to today's number, to Boxing Day, and then there are issues of either lag or catchup, and then New Year's. Can anyone shed light on these details, and how seriously we should take them? Even if we get a new mix of strain results, should we worry about lags in the south versus the north, where different strains dominate, warping numbers?

What are good data sources to check for the strain in other nations? Who is checking enough that if it was present, we would notice, or we'd have a good handle on its numbers? Can we make this data happen somewhere? 'Has been spotted in' isn't much to work with.

Also making this a place for people to say how they're updating their estimates based on information that comes in.
What are the unwritten rules of academia? 
by Yitz — 2020-12-25T15:33:48.470Z

I came across this tweet a few minutes ago, and it got me thinking about how much of academia is gatekept by unwritten rules and rituals. Are there any good resources out there documenting such rules for outsiders? If not, this seems like a worthy task to attempt for anyone with such knowledge.

Are there any aspects of academia or of doing original research that you would not have known about if you had only learned through textbooks? With my current position of learning exclusively at home without a mentor, I worry that I may be missing many of these unwritten rules, and don't know how much of a disadvantage this places me at.
What trade should we make if we're all getting the new COVID strain?
by Jacob Falkovich — 2020-12-25T07:09:41.026Z

If data keeps coming out in the next week confirming that the new COVID strain is 70% more transmissible, I think the modal outcome is that ~50% of Americans will get it by the early summer. The market may take a few days to realize and react to this (as it was in March), but also just buying June put option on the SP500 seems very naïve (since the SP500 is at all-time highs and a fourth COVID wave doesn't necessarily affect much the NPV of future earning of huge corporations). So if I think that the probability of everyone getting COVID in the next six months is much likelier than the market, at least for a few days, what trade would capture that?
Open Christmas card to my father
by KatjaGrace — 2020-12-25T06:40:13.599Z

Dear Daddy,

Merry Christmas! I got you a list of things on the internet I thought you might find interesting. No obligation to look at them. (Many things I’ve enjoyed lately are old or about history, so there’s a good chance that you have seen them or seen much better things, but oh well.)

Lots of love from Katja oxo

(Picture: our Christmas tree)

AI dungeon: a text adventure written by AI in response to your responses

AI can also generate coherent completions of photographs (See examples under ‘Completions’)

A lecture about the Bronze Age collapse (I enjoyed this, though your historical understanding being thousands of times greater than mine might make it less so for you)

Slave voyages: a memorial in the form of a database of 36,000 specific slaving expeditions that took place between 1514 and 1866. Perhaps most easily appreciated in the form of this timelapse of the journeys (note that you can click on any dot and see the details of the ship and captives). I’m pretty impressed at them finding this much data.

On the shortness of life (I enjoyed this, though don’t necessarily agree)

How to live for much longer in expectation

The isles of Greece

The tallest building, the Burj Khalifa, is pretty cool looking:
Netflix's "Start-Up" and sincere work dramatization
by Maxwell Peterson — 2020-12-25T05:32:46.547Z

I'm in the middle of watching the show Start-Up on Netflix. I just finished episode 5 (some spoilers). Some background: Seo Dal-mi and Do-san are our main characters. They and Do-san's machine-learning-developer buddies are all on the same team at a hackathon. At the end of the hackathon, each team presents their product and business plan to the judges and an audience, and the top 5 teams (out of 40 - so this is pretty competitive) will be admitted to Sandbox, a fictional Korean Y Combinator. Seo Dal-mi has spent years working at crappy part-time jobs and entered the competition because she was inspired by Do-san's "company" to start her own. Do-san and his friends technically do have a company, and are very good at getting deep neural nets to attain state-of-the-art accuracy on classification tasks, but are terrible at every other aspect of business, and their company is currently falling apart. Dal-mi's sister In-jae is also in the competition, but the two have a terrible rivalry, went 15 years without speaking to each other, and mostly despise each other. In-jae forms a different team and, out of spite, does her project using the same dataset that Seo Dal-mi's team chose: a computerized collection of handwriting images from a bank.

What follows is the best machine learning dramatic face-off I've ever seen in anything. Dal-mi's team has made an excellent forgery detector: it takes handwritten text as input, and outputs "forgery" or "not a forgery" as output, with super-high accuracy. In-jae's team, meanwhile, has made a font-generator: it takes some sample handwriting as input, and outputs a full custom font-set, that you can then use for whatever digital application you want, like design. The presentations are going straightforwardly, until a judge asks... what if we put some writing from this font-generator into the forgery-detector? Would it identify it as a forgery? The two teams go on stage to set it up, In-jae's machine-fabricated font is fed into Dal-mi's fabrication-detector, and everything is on the line: the fate of their projects, and whether they get into Sandbox, all depends on what the detector network outputs.

This is the best. I've heard of this type of contest before, in real life: the contest between email spammers and spam filters; between doctored photos and videos and the ability to detect the doctoring; between auto-generated text...