trentmkelly/LessWrong-43k huggingface dataset:
Paper: Superposition, Memorization, and Double Descent (Anthropic)
by LawrenceC — 2023-01-05T17:54:37.575Z
Link: https://transformer-circuits.pub/2023/toy-double-descent/index.html

(This is a follow up to Anthropic's prior work on Toy Models of Superposition.)

The authors study how neural networks interpolate between memorization and generalization in the "ReLU Output" toy model from the first toy model paper:

They train models to perform a synthetic regression task with T training points, for models with m hidden dimensions.

First, they find that for small training sets, while the features are messy, the training set hidden vectors hi=WXi (the projection of the input datapoints into the hidden space) often show clean structures:

They then extend their old definition of feature dimensionality to measure the dimensionalities allocated to each of the training examples:

DXi = ||hi||2∑j(^hi⋅hj)2
and plot this against the data set size (and also test loss):

This shows that as you increase the amount of data, you go from a regime with high dimensionality allocated to training vectors and low dimensionality allocated to features, to one where the opposite is true. In between the two, both feature and hidden vector dimensionalities receive low dimensionality, which coincides with an increase in test loss, which they compare to the phenomena of "data double descent" (where as you increase data on overparameterized models with small amounts of regularization, test loss can go up before it goes down).

Finally, they visualize how varying T and m affects test loss, and find double descent along both dimensions:

They also included some (imo very interesting) experiments from Adam Jermyn, 1) replicating the results, 2) exploring how weight decay interacts with this double descent-like phenomenon, and 3) studying what happens if you repeat particular datapoints.

Some limitations of the work, based on my first read through:

The authors note that the results seem quite sensitive to hyperparameters, especially for low hidden dimension m∈{2,3}. For example, Adam Jermyn's results differ from the Anthropic Interp team results (though the figures still look qualitatively similar).
I'm still not super convinced how much results from the superposition work apply in practice. I'd be interested in seeing more work along the lines of the MNIST preliminary experiment done by Chris Olah at the bottom.

(I'll probably have more thoughts as I think for longer.)
Collapse Might Not Be Desirable
by Dzoldzaya — 2023-01-05T17:29:43.886Z

Epistemic Status: Wisdom gleaned from years of meditation.

There are many ways we could create societal collapse.

Creating an engineered pandemic, triggering a nuclear catastrophe, developing a computer virus to permanently crash the global economy etc.

Now you may be thinking about trying out one of these. Most value is in the far future and, in a collapse scenario, we would probably be more likely to sustain moral progress than technological progress (because 'moral infrastructure' is more likely to stay intact than technological and global trade infrastructure). Creating a moderate collapse scenario, with a high level of depopulation, will definitely give us more time and probably more societal wisdom to focus on getting past the age of perils- in particular, solving AI alignment. And in the short term, it might seem morally desirable as well, because the net harm caused by humans, such as animal suffering on factory farms, will be greatly reduced. You might also be an antinatalist, and (reasonably) believe that most of our lives are net negative.

But there are a few reasons that this strategy might not be a good idea:

1. You might go wrong and kill everybody. Fine, so developing a contagious virus that kills 95% of the population might seem like a win-win- it would slow everything down, end most factory farming, and give us another century to rebuild and focus on alignment. But what if the virus mutates and accidentally kills 99.9% of the population, leaving only the most remote and uncontacted tribes (who aren't party to our moral progress) alive? Or what if it kills too few, and just makes certain actors angrier or more desperate to develop violent, weaponised AGI?

2. They might lock in worse institutions/ forms of governance. Of course, we will probably be able to build on our moral progress in the case of a collapse, but institutions are sticky. A collapse scenario is likely to lead to varied outcomes between today's political units. If any of these survive, it is hard to predict which will come out of the collapse stronger. If your planned collapse scenario ends up destroying every political unit except an elite bunch of the CCP with bunker access, this could be even riskier than the current predicament!

3. Reputational damage. If you try to create an engineered pandemic, trigger a nuclear catastrophe or otherwise strive for societ...
Image generation and alignment 
by rpglover64 — 2023-01-05T16:05:44.231Z

There isn't a lot of talk about image models (e.g. Dall-E and StableDiffusion) on LW in the context of alignment, especially compared to LLMs. Why is that? Some hypotheses:

LLMs just happened to get some traction early, and due to network effects, they are the primary research vehicle

LLMs are a larger alignment risk than image models, e.g. the only alignment risk of image generation comes from the language embedding

LLMs are not a larger alignment risk, but they are easier to use for alignment research
Machine Learning vs Differential Privacy
by Ilio — 2023-01-05T15:14:05.489Z

edit: see below for clarifications by expert domain rpglover64 and a good pick of references from the gears to ascenscion. My one (cheatingly long) sentence takeaway is: it’s clear training does not automatically lead to DP, it’s unclear if DP can always or seldom help training, it’s likely that easy algorithms are not available yet, it’s unlikely that finding one is low hanging fruit.

From wikipedia, « an algorithm is differentially private if an observer seeing its output cannot tell if a particular individual's information was used in the computation ». In other words, if some training process asymptotically converges toward generalisable knowledge only, then it should tend to become differentially private.

…or so it seems to me, but actually I’ve no idea if that’s common knowledge in ml- or crypto- educated folks, versus it’s pure personal guess and there’s no reason to believe that. What do you see as the best argument for or against that idea? Any guess on how to disprove or prove it?

Extra Good Samaritan point : my english sucks, so any comment rewriting this post in good english, even if for minor details, is a great help thank you.

This is version 0.1
Covid 1/5/23: Various XBB Takes
by Zvi — 2023-01-05T14:20:00.698Z

The majority of this week’s post is various people doing their Obligatory Thread on Latest Variant, in this case XBB. All reach broadly compatible conclusions:

XBB is the next variant and rapidly taking over, and this is normal.

XBB is not more deadly.

Our existing protections against severe disease and death carry over.

Our existing protections against infection are reduced a lot.

XBB is not obviously going to result in more infections over time.

All the standard Public Health recommendations remain in effect.

Executive Summary

XBB is taking over.

XBB is no more or less dangerous than previous variants.

It will however extend the winter wave longer than it would have lasted.

Let’s run the numbers.

The Numbers

Predictions

Predictions from Last Week: 385k cases (+16%) and 2,600 deaths (+17%)

Results: 423k cases (+27%) and 2,703 deaths (+23%).

Predictions for Next Week: 450k cases (+7%) and 2,950 deaths (+7%).

The numbers came in higher than expected, more due to the standard ‘data is funky so hedging my bets’ factor than anything on the ground. This next week we will smooth out the double reporting in the Carolinas and other back-fills, while making up for New Year’s, so it’s another funky week.

Florida failed to report for the second straight week. I’ve given them the same numbers I gave them last week.

For all the talk of a wave, so far we have not seen a substantial spike in deaths. This week’s numbers are not much higher than early December, instead they are reverting after Christmas.

North Carolina and South Carolina clearly reported two weeks worth of data, adding about 30,000 cases and 100 deaths that should instead be part of the previous week. I did not adjust for this as it is part of the standard holiday cycle. Thus, the case numbers in the South should not be alarming.

Deaths

Cases

Bob Wachter notes that Covid asymptomatic positive test rate in SF declined from 5% to 3.2% on December 30. He notes that this is likely a temporary lull, partly due to XBB.1.5. It is still an excellent sign that we were at least close to stabilizing before that.

Caitlin Rivers joins the camp that says case numbers are no longer that useful, says that our case numbers have become almost completely decoupled from other indicators.

I agree that there is danger when comparing case numbers now to...
Running by Default
by jefftk — 2023-01-05T13:50:01.254Z

In video games it's common for characters to

run by
default: it's faster, why wouldn't you? I think this is also a
good approach to apply in real life, at least over short distances.
It saves you time and it's a bit of exercise. Over pretty much any
distance short enough that I'm not going to break a sweat I'll run.

I've done some of this most of my life (I can still hear my middle
school principal shouting to stop running in the
hallway—Jeeeeff-reeeey!) though over the last five years or
so I've pushed myself to do it more consistently. The more I do it the
easier it gets, and the shorter commutes become.

Note that this is another
example of describing something I do that works for me, and that I
expect would likely work for many more people than I see applying it.
But probably not more than 30% of people, though?

(Prompted by a coworker talking about the inconvenience of an 8min
distance between two buildings at work that I think of as a 2-3min
distance.)

Comment via: facebook, mastodon
PSA: reward is part of the habit loop too
by Alok Singh — 2023-01-05T11:00:07.996Z

The usual setup of a habit is

cue

craving

routine

reward

Among my social circle, rewarding oneself seems to be lost by the wayside. Sadly funny for a crowd of people into reinforcement learning, since they're trying to skip the reinforcement and wondering why it doesn't stick1. My reward is usually reading fiction or playing a video game. For brushing my teeth, it's the nice shiny feeling at the end, which I make a point of noticing. For going to the gym, it's the steam room after.

Figure out what you like. Anything can be a reward as long as it feels good. Don't try to logic yourself into wanting "the right rewards". Has that worked before? Surrender. Let the soft animal of your body love what it loves.

RL without the reward is an environment (states, actions, transitions) that's Markovian. It has a syntax (things are happening), but no semantics (without a reward function, there's no interpretation of whether something is good or bad). ↩
Infohazards vs Fork Hazards
by jimrandomh — 2023-01-05T09:45:28.065Z

I think actual infohazardous information is fairly rare. Far more common is a fork: you have some idea or statement, you don't know whether it's true or false (typically leaning false), and you kow that either it's false or it's infohazardous. Examples include unvalidated insights about how to build dangerous technologies, and most acausal trade/acausal blackmail scenarios. Phrased slightly differently: "infohazardous if true".

If something is wrong/false, it's at least mildly bad to spread/talk about it. (With some exceptions; wrong ideas can sometimes inspire better ones, maybe you want fake nuclear weapon designs to trip up would-be designers, etc). And if something is infohazardous, it's bad to spread/talk about it, for an entirely different reason. Taken together, these form a disjunctive argument for not spreading the information.

I think this trips people up when they see how others relate to things that are infohazardous-if-true. When something is infohazardous-if-true (but probably false), peopple bias towards treating it as actually-infohazardous; after all,if it's false, there's not much upside in spreading bullshit. Other people seeing this get confused, and think it's actually infohazardous, or think it isn't but that the first person thinks it is (and therefore thinks the first person is foolish).

I think this is pretty easily fixed with a slight terminology tweak: simply call thinks "infohazardous if true" rather than "infohazardous" (adjective form), and call them "fork hazards" rather that "infohazards" (noun form). This clarifies that you only believe the conditional, and not the underlying statement.
Monthly Shorts 12/22
by Celer — 2023-01-05T07:20:06.669Z

The hell, Ukranian one-year bond market?

Firstly, we can establish that nobody listened to the President of the United States and the White House clearly, explicitly, and repeatedly saying that the Russians would invade. On April 5, 2021, rates were 11.7%, in February 14th 2022 they were 13.6%, lower than their January peak.

Secondly, it’s impressive to see how fast the financial markets updated that the Ukranians actually had a pretty good shot at winning this. Remember that even in the world where Ukraine wins, repayment will still be pretty difficult. And countries have a strong incentive not to repay the lenders to their opponent, in order to incentivize not lending to their opponent, so if the Russians win you *certainly* shouldn’t expect repayment.

Thirdly, I don’t do anything other than broad stock market investing as a matter of professional practice, but I suspect you could do worse with your money, both ethically and pragmatically, than Ukranian war bonds. This is not investing advice! That’s the magic incantation to take what I just said and make what I did legal. Why does that work? What follows is pure theorizing.

Well, firstly, because the SEC is not all that interested in locking me up. If they locked up every fool who spouted about money, they would have to expand the jails even further.

Secondly, by having to say “this is not investing advice”, I make it clear that I am not a Licensed Professional. If someone who pretends to expertise has to make that disclaimer, their audience can notice, wonder why, and perhaps avoid being drawn into the scam.

Hey, Japan! Glad to see you’re doing better.

Image courtesy of a critique of YIMBY usage of Japanese house prices, and more broadly easy international comparisons (which I have some meta-objections to, but it’s a good piece and worth integrating).

Systematic review on the experiences of grad students

AI Alignment horror story, if you’re into that sort of thing.

Happy new year! It’s been another good year of blogging. There was a temporary burst in viewership, which has since entirely returned to normal. I look forward to continuing next year.
The 2021 Review Phase
by Raemon — 2023-01-05T07:12:24.251Z

(Apologies for this post being quite late – I got sick just prior to the holidays, and then had a surprisingly eventful holidays)

We've wrapped up the nominations phase for the 2021 Review. 139 people have participated so far, casting 1815 votes and writing 72 reviews.

Now we're in the Review Phase, where we ask "Were these posts actually good? How could they be improved? How do they fit together?".

Key Info:

Posts need at least one review in order to progress to the final voting stage. To write a review, navigate to a nominated post page, and click the "Review" button at the top of the page.
I'll award prizes to reviews that I think offer substantive information. Roughly $50 for a (See "What kinds of reviews are most valuable?" for examples)
You can look over the results of last year's review at /reviewVoting/2020, to compare the results there to this year's.
You can sort the posts on the /reviewVoting page by "Vote Total", to see which posts were most upvoted during the nomination voting. Thorough reviews of the top-scoring posts are particularly valuable.

How much Progress Overall?

One of my original goals for the Review was to provide a clearer feedback mechanism to the Lightcone team on "how is LessWrong doing?". It's one thing to track karma and site activity. But did our infrastructure maintenance, UI improvements, research retreats and other interventions actually seem to translate into more good, quality intellectual progress?

This year some people have commented that there are fewer exciting-feeling posts than they remember from previous years. I've also had a bit of this feeling – despite feeling at the time like in 2021, LessWrong was a pretty happening place with lots of good content. I'm not sure I endorse the feeling. For example: I think important progress happened in the MIRI Dialogues, but in a diffuse way, which didn't result in many concrete posts I can point at to that I learned discrete concepts from. Progress != posts, necessarily.

But, it still seems worth paying attention to the sensation. Yes requires the possibility of no, and it's fairly important for the Review to be able to return the answer "less overall progress this year."

Now, right now there isn't much of a mechanism for tracking "how much progress" there was between years. I'm still thinking about how to seriously answer this question. But for now, I've u...
When you plan according to your AI timelines, should you put more weight on the median future, or the median future | eventual AI alignment success? ⚖️
by Jeffrey Ladish — 2023-01-05T01:21:37.880Z

This is a question I'm puzzling over, and my current answer is that when it comes to decisions about AI alignment strategy, I will put more planning weight on median futures where we survive, making my effective timelines longer for some planning purposes, but not removing urgency.
I think that in most worlds where we manage to build aligned AGI systems, we managed to do this in large part because we bought more time to solve the alignment problem, probably via one of two mechanisms:

🤝 coordination to hold off building AGI
🦾 solving AI alignment in a limited way, and using a capabilities-limited AGI to negotiate for more time to effort to develop aligned AGI

I think we are likely to buy >5 years of time via one or both of these routes in >80% of worlds where we successfully build aligned AGI.

I have less of a good estimate about how long my AI timelines are for the median future | eventual AGI alignment success. 20 years? 60 years? I haven't thought about it enough to give a good estimate, but I think at least 10 years. Though, I think time bought for additional AI alignment work is not equally useful.
[1] 🧐

Implications for me:

🗺 For strategy purposes, I should plan according to a distribution of worlds around my median timeline | eventual AI alignment success. For me, that's like ~30 years (it's always 30 years 💀), though I've only done a very cursory estimate of this and plan to think about it more.
⌛️ There's still a lot of urgency! My timelines are NOT distributed around ~30 years! If we get that time, it's mostly because we BOUGHT it. So there's urgency to work towards coordination on buying time and/or figuring out how to build aligned-and-corrigible-enough-AGI to buy more time and shepherd alignment research.

Personal considerations

🪣 I do have a bucket list, and for the purposes of "things I'd really like to do before I die", I go with my estimated lifespan in my median world

Terms and assumptions:

🤖 By AGI I mean general intelligence with significantly greater control / optimization power than human civilization
🦾 By capabilities-limited AGI, I mean a general intelligence with significantly greater capabilities than humans in some domains, but corrigible enough not to self improve / seize power to accomplish arbitrary goals
😅 My timelines are 70% chance < 20 years
💀 I'm assuming AGI + no alignment = human extinction
🏆 Solving th

...
Contra Common Knowledge
by abramdemski — 2023-01-04T22:50:38.493Z

This post assumes some background knowledge about "common knowledge", a technical term in philosophy. See the tag page for an introduction and a list of LessWrong essays on the subject.

Epistemic effort: basically reading the SEP article on common knowledge and forming my own conclusions. I've also gained something by discussing these issues with Scott Garrabrant and Sahil Kulshrestha, although this post is far from representing their views as I've understood them.

Consider a statement like "LessWrong curated posts help to establish common knowledge".

On the one hand, this seems obviously valid in practice. Curating a post boosts the number of people who see it, in a public way which also communicates the meta-knowledge (you know that lots of people will know about this post, and they'll also know that fact, etc).

On the other hand, this is a long way off from actually establishing common knowledge. People don't check LessWrong constantly. Lots of people will miss any given curated post.

In this post, I will argue that common knowledge never occurs in the real world.

But this leaves us with a paradoxical state of affairs. We use the concept to explain things in the real world all the time. Common knowledge is an important coordination tool, right? For example, social norms are supposed to require common knowledge in order to work.

I'll also offer some evidence that common knowledge is not as important as it is typically made out to be.

Can't Have It!

Several arguments about the impossibility of common knowledge have been made in the philosophical literature; see Common Knowledge Revisited and Uncommon Knowledge. Whether nontrivial common knowledge is realistically possible is still debated, according to SEP (Stanford Encyclopedia of Philosophy).

SEP makes a big deal about how we don't literally need to explicitly believe infinitely many levels of knowledge in order to have common knowledge, because we can have a single concept ("common knowledge" itself) which implies any level of recursive knowledge.

However, I don't find this particularly reassuring.

I will make the following argument: Common knowledge requires absolute certainty of common knowledge. If anyone has the slightest doubt that common knowledge has been established, it hasn't. But we are never 100% certain of such things in the real world! Therefore, common knowledge is impossible.

It is often st...