trentmkelly/LessWrong-43k huggingface dataset:
Capital and inequality 
by  — 2022-08-15T17:23:47.028Z

In Thomas Piketty’s masterpiece, Capital in the 21st century, he showed the strong association between the rate of return on capital (capital here means assets like companies and land) and the levels of inequality in a society. The key mechanism at play here is that capital is much much more concentrated than labour. Finally, using data from American universities, he showed that larger those with larger endowments were able to earn much higher returns on their endowments. He argued that this was because they’re able to spend more money on professionals who are able to get above market returns.

It’s an open question whether or not a rate of return on capital greater than the growth rate of the economy actually causes higher levels of inequality; Acemoglu and Robinson have a nice paper showing that when you control for various measures of how extractive institutions are, the effect mostly goes away. Despite this somewhat weak empirical grounding, I’m going to spend this blog post giving my musings on why high relative returns on capital might be causing high inequality.

I can’t think of a good heading but this is a natural break

There’s no fundamental reason why inequality should be higher in a society where capital is more valuable than labour, or where there’s a very large amount of capital. Indeed, income inequality was low in the Soviet Union despite the Soviet Union having a vast stock of capital relative to labour, as did Germany and Japan which also have relatively low levels of inequality. In theory merely having a large stock of capital could decrease the share of national income going to capital if the return on capital is elastic enough. If increasing the stock of capital dramatically reduced how productive the marginal unit of capital was, the countries with large amounts of capital could have lower shares of national income going to capital than in countries with limited amounts of capital. However, Piketty found that empirically that doesn’t seem to be the case. The return on capital doesn’t fall fast enough for capital rich economies also to have a smaller proportion of their national incomes going to capital.

There are two reasons why I think capital income from capital could be inherently less equal than income from labour. Firstly, the return to capital is intrinsically higher for richer people, no ability to pu...
Are there practical exercises for developing the Scout mindset?
by ChristianKl — 2022-08-15T17:23:42.552Z

I'm thinking about doing a LessWrong meetup around the Scout mindset. I prefer to have meetups where there are a lot of two-person exercises as those are really good to get nerds who have trouble with normal small talk to connect to each other in addition to often also being useful for developing skills over lecturing the whole time. Can anyone think of good exercises that could be done at a Scout mindset meetup?
How do you get a job as a software developer?
by lsusr — 2022-08-15T14:45:20.923Z

I am currently looking for work as a software developer. The problem is…I don't really know how to do so. I have run a startup for most of my adult life. So while I have lots of experience writing software and shipping products, I don't know how to get a foot in the door. I don't know how to answer behavioral questions. I don't know how to write a good resume.

About me:

I heard I should do 200 medium-difficulty Leetcode problems. I have completed 69 so far.

I don't have strong preferences about what the company does. I mostly care about working with a team that has a good culture.

My degree is in physics and mathematics. I am a self-taught programmer.

I live in Seattle but might be willing to relocate for the right opportunity.

If you know how to get a job in the software industry I would love to talk to you via video call (or I can buy you lunch if you live in Seattle). I respond to both Less Wrong private messages and to email. Just send me the date and time that works best for you (along with your timezone).

PS: Thank you everyone who replied to my previous Want to Hire Me? post. You helped me figure out that what I really want right now is a regular job.
The Parable of the Boy Who Cried 5% Chance of Wolf
by KatWoods — 2022-08-15T14:33:21.649Z

Epistemic status: a parable making a moderately strong claim about statistics

Once upon a time, there was a boy who cried "there's a 5% chance there's a wolf!"

The villagers came running, saw no wolf, and said "He said there was a wolf and there was not. Thus his probabilities are wrong and he's an alarmist."

On the second day, the boy heard some rustling in the bushes and cried "there's a 5% chance there's a wolf!"

Some villagers ran out and some did not.

There was no wolf.

The wolf-skeptics who stayed in bed felt smug.

"That boy is always saying there is a wolf, but there isn't."

"I didn't say there was a wolf!" cried the boy. "I was estimating the probability at low, but high enough. A false alarm is much less costly than a missed detection when it comes to dying! The expected value is good!"

The villagers didn't understand the boy and ignored him.

On the third day, the boy heard some sounds he couldn't identify but seemed wolf-y. "There's a 5% chance there's a wolf!" he cried.

No villagers came.

It was a wolf.

They were all eaten.

Because the villagers did not think probabilistically.

The moral of the story is that we should expect to have a large number of false alarms before a catastrophe hits and that is not strong evidence against impending but improbable catastrophe.

Each time somebody put a low but high enough probability on a pandemic being about to start, they weren't wrong when it didn't pan out. H1N1 and SARS and so forth didn't become global pandemics. But they could have. They had a low probability, but high enough to raise alarms.

The problem is that people then thought to themselves "Look! People freaked out about those last ones and it was fine, so people are terrible at predictions and alarmist and we shouldn't worry about pandemics"

And then COVID-19 happened.

This will happen again for other things.

People will be raising the alarm about something, and in the media, the nuanced thinking about probabilities will be washed out.

You'll hear people saying that X will definitely fuck everything up very soon.

And it doesn't.

And when the catastrophe doesn't happen, don't over-update.

Don't say, "They cried wolf before and nothing happened, thus they are no longer credible."

Say "I wonder what probability they or I should put on it? Is that high enough to set up the proper precautions?"

When somebody says that nuclear war hasn't happened yet despite all the scares, whe...
And the Revenues Are So Small
by Zvi — 2022-08-15T13:00:01.772Z

The ‘Inflation Reduction Act’ includes a corporate minimum tax provision. The top 150 American corporations have to pay a minimum percentage of book profits as tax.

The whole thing has strange implications, with Marginal Revolution offering several claims that this could end up doing quite a lot of damage by setting up bad incentives. It is an interesting fact about the world that this is the source of the most interesting criticisms known so far.

There seems little question the thing is not first best policy and is quite the tangled mess. But that is normal. I also agree that taxing investment rather than consumption is quite bad – and if anything it seems like in that post Tyler ignores the best argument for not taxing investment, which is that in general private investors capture only a small fraction of societal returns to investment, so they are effectively quite highly taxed to begin with, and this is a lot more basic than issues of uncertainty.

(I do not agree that this implies that Matt Yglesias should therefore be a conservative, although that is the direction of the appropriate update.)

The question as always is the magnitude of the tangled mess and resulting consequences of the strange incentives on various forms of investment.

Book Minimum Tax

The latest claim is this:

Different rules apply for the determination of income for US tax purposes and for financial reporting purposes. Both are artificial constructs. Who is to say that one is a more accurate indication of “income” than the other? Congress is largely responsible for the difference by creating incentives through the tax code by offering accelerated depreciation, etc. for taxable income for pet projects such as climate related investment.

The AMT provisions of this tax bill create enormous additional complexity. The fact that they are designed to apply to only about 150 large corporations isn’t a way to create an rationale and equitable corporate tax system. Rather, it is designed to punish, in a Robin Hood like manner, the most successful US corporations and to *temporarily* fund spending provisions in the bill. Its complexity will create additional complexity and costs which consumers and investors ultimately bear.

I say *temporary* funding because the corporate AMT is generally an acceleration of regular tax liability. If a corporation pays the AMT, a credit against future corpo

...
Extreme Security
by lc — 2022-08-15T12:11:05.147Z

What makes "password" a bad password?

You might say that it's because everybody else is already using it, and maybe you'd be correct that everybody in the world deciding to no longer use the password "password" could eventually make it acceptable again.

But consider this: if everyone who was using that password for something changed it tomorrow, and credibly announced so, it would still be an English dictionary word. Any cracker that lazily included the top N words in an English dictionary in a cracking list would still be liable to break it, not because people actually use every English dictionary word, but simply because trying the top N English words is a commonly deployed tactic.

You could go one step further and say "password" is also a bad password because it's in that broader set of English dictionary words, but really, you should just drop the abstraction altogether. "Password" is a bad password because hackers are likely to try it. Hackers are likely to try it because they're trying to break your security, yes, but also because of a bunch of seemingly irrelevant details like what they tell humans to do on online password cracking resources, and the hashcat & JohnTheRipper programs' standard features. If due to some idiosyncratic psychological quirk hackers were guaranteed never to think of "password" as a possible password and never managed to insert "password" in their cracking lists, it would be the Best Password. A series of 17 null bytes (which are used to signal the end of C strings) tends to this day to be an excellent choice, when the website accepts that, and simple derivations of that strategy will probably go on being an excellent choice despite my mentioning them in this LW post.

Generating a password with a high amount of "entropy" is just a way of ensuring that password crackers are very unlikely to break them without violating Shannon's maxim. Nobody wants to have to violate Shannon's maxim because it means they can't talk about their cool tricks publicly, and their systems' design itself will become a confidential secret in need of protection.

Unfortunately "not violating Shannon's maxim" is not always a luxury you have. There are legitimate and recognized circumstances where modeling the particular opponents you're liable to face becomes critical for adequate performance. High frequency traders who sometimes choose t...
No shortcuts to knowledge: Why AI needs to ease up on scaling and learn how to code
by Yldedly — 2022-08-15T08:42:23.833Z
Link: https://deoxyribose.github.io/No-Shortcuts-to-Knowledge/

Will scaling deep learning produce human-level generality, or do we need a new approach? You may have read the exchange between Scott Alexander and Gary Marcus, and felt that there are some good arguments on both sides, some bad ones, but few arguments that go beyond analogy and handwaving - arguments that would take what we know about deep learning and intelligence, and look at what that knowledge implies. If you haven’t read the exchange, here it is: SA, GM, SA, GM.

I will argue for Marcus’ position, but dive a little deeper than he does. I believe that symbolic representations, specifically programs, and learning as program synthesis, can provide data efficient and flexible generalization, in a way that deep learning can’t, no matter how much we scale it. I’ll show how probabilistic programs can represent causal models of the world, which deep learning can’t do, and why causal models are essential to intelligence. But I’ll start by examining the opposing view, that scaling deep learning is sufficient for general intelligence. To that end, I’ll quote from Gwern’s thorough essay on the scaling hypothesis.
Table of Contents:

The scaling hypothesis and the laziness of deep learning
What do neural networks learn?
NNs learn shortcuts because they are “easy to vary”
Representing knowledge as probabilistic programs
Learning as probabilistic program synthesis
Learning libraries of concepts
Probabilistic synthesis and causality as program editing
Implications for AI alignment
(Latex and interactive plots don't display properly, please see link for full post)
Seeking Interns/RAs for Mechanistic Interpretability Projects
by Neel Nanda — 2022-08-15T07:11:25.107Z

UPDATE: The deadline for applying for this is 11:59pm PT Sat 27 Aug. I'll be selecting candidates following a 2 week (paid) work trial, 10hr/week starting Sept 12 where participants pick a concrete research idea and try to make progress on it.

Hey! My name is Neel Nanda. I used to work at Anthropic on LLM interpretability under Chris Olah (the Transformer Circuits agenda), and am currently doing some independent mechanistic interpretability work, most recently on grokking (see summary). I have a bunch of concrete project ideas, and am looking to hire an RA/intern to help me work on some of them!

Role details: The role would be remote by default. Full-time (~40 hrs/week). Roughly for the next 2-3 months, but flexible-ish. I can pay you $50/hr (via a grant).

What I can offer: I can offer concrete project ideas, help getting started, and about 1hr/week of ongoing mentorship. (Ideally more, but that's all I feel confident committing to) I'm much better placed to offer high-level research mentorship/guidance than ML engineering guidance.

Pre-requisites: As such, I'd be looking for someone with existing ML skill, enough to be able to write and run simple experiments, esp with transformers. (e.g. writing your own GPT-2-style transformer from scratch in PyTorch is more than enough). Someone who’s fairly independently minded and could mostly work independently if given a concrete project idea, and mostly needs help with high-level direction. Familiarity with transformer circuits, and good linear algebra intuitions are nice-to-haves, but not essential. I expect this is best suited to someone who wants to test fit for doing alignment work, and is particularly interested in mechanistic interpretability.

Project ideas: Two main categories.

One of the future directions from my work on grokking, interpreting how models change as they train.

This has a fairly mathsy flavour and involves training and interpreting tiny models on simple problems, looking at how concrete circuits develop over training, and possibly what happens when we put interpretability-inspired metrics in the loss function. Some project ideas involve training larger models, eg small SoLU transformers

Work on building better techniques to localise specific capabilities/circuits in large language models, inspired by work such as

ROME

.

This would have a less mathsy flavour and more in

...
A Mechanistic Interpretability Analysis of Grokking
by Neel Nanda — 2022-08-15T02:41:36.245Z
Link: https://colab.research.google.com/drive/1F6_1_cWXE5M7WocUcpQWp3v8z4b1jL20

A significantly updated version of this work is now on Arxiv and was published as a spotlight paper at ICLR 2023

aka, how the best way to do modular addition is with Discrete Fourier Transforms and trig identities

If you don't want to commit to a long post, check out the Tweet thread summary

Introduction

Grokking is a recent phenomena discovered by OpenAI researchers, that in my opinion is one of the most fascinating mysteries in deep learning. That models trained on small algorithmic tasks like modular addition will initially memorise the training data, but after a long time will suddenly learn to generalise to unseen data.

A training curve for a 1L Transformer trained to do addition mod 113, trained on 30% of the 1132 pairs - it shows clear grokking

This is a write-up of an independent research project I did into understanding grokking through the lens of mechanistic interpretability. My most important claim is that grokking has a deep relationship to phase changes. Phase changes, ie a sudden change in the model's performance for some capability during training, are a general phenomena that occur when training models, that have also been observed in large models trained on non-toy tasks. For example, the sudden change in a transformer's capacity to do in-context learning when it forms induction heads. In this work examine several toy settings where a model trained to solve them exhibits a phase change in test loss, regardless of how much data it is trained on. I show that if a model is trained on these limited data with high regularisation, then that the model shows grokking.

Loss curve for predicting repeated subsequences in a sequence of random tokens in a 2L attention only transformer on infinite data - shows a phase change

Loss curve for predicting repeated subsequences in a sequence of random tokens in a 2L attention-only transformer given 512 training data points - shows clear grokking.

One of the core claims of mechanistic interpretability is that neural networks can be understood, that rather than being mysterious black boxes they learn interpretable algorithms which can be reverse engineered and comprehended. This work serves as a proof of concept of that, and that reverse engineering models is key to understanding them. I fully reverse engineer the inferred algorithm from a transformer that has grokked how to do modular addition (which somehow i...
What is the probability that a superintelligent, sentient AGI is actually infeasible?
by Nathan1123 — 2022-08-14T22:41:43.268Z

Hello,

I am not at all adverse to discussing contingencies for things that are either uncertain or unlikely, but I'm curious what the general consensus is for how likely a superintelligent AGI scenario actually is.

To be clear, I am certainly aware that advances in AI have made leaps and bounds throughout my own lifetime alone, with public feats such as AlphaGo far exceeding the expectations of its creators.

But just because there is a trend over a certain period of time doesn't mean that it will stay that way forever, or develop in a way that one expects intuitively. For example, population models tend to grow quickly at first, but flatten out over time. It is possible that computer technology (or technology in general) has some natural, undiscovered limit, such that the graph will flatten towards a logistic asymptote or similar curve.

Or, alternatively, civilization may discover that there is a certain trade-off in computer science, analogous to Heisenberg Uncertainty, such that either of the two scenarios are possible, but not in combination:

Superintelligent, non-sentient AI or Tool AI: Machines maximized for computational power and autonomous problem-solving, but has no self-awareness or autonomy associated with consciousness.

Unintelligent AGI or Infant AI: An artificial consciousness with full autonomy and self-awareness, but relatively poor computational power, such that it is no smarter than a real human.

In this scenario, a superintelligent AGI isn't possible because of the trade-off: it's either a superintelligent tool or an unintelligent consciousness. How probable do you think such a scenario might be?
Dealing With Delusions
by adrusi — 2022-08-14T21:11:43.937Z

If you're prone to forming delusions, but you're good at checking your beliefs against reality, you might manage to avoid going off the deep end, but that doesn't mean escape unscathed.

Amphetamine use can make you imagine the people in your life are plotting against you. A good rationalist will notice they lack evidence and that the prior should be pretty low, but they'll still be left with whatever cognitive patterns threatened to generate that delusion.

Psychedelic use can make you believe stuff in the "all is connected" space. A good rationalist isn't going to start believing parapsychological phenomena are real because of something they experienced on an LSD trip, but they'll still be subject to the cognitive patterns that tend to make some others believe such things.

Schizophrenia can make you see corruption all around you or imagine your own persecution. On the margin, practice in rationality might be a bulwark against schizophrenic delusions (these statements have not been evaluated by the Food and Drug Administration; LessWrong is not intended to diagnose, treat, cure or prevent any disease). But while you might save yourself from believing the CIA is after you, you wont save yourself from the cognitive patterns that tend to give rise to such delusions.

When you become subject to a new bias, as long as it doesn't obviously put you out of touch with reality, it can be tempting to image that what's really happening is you're getting smarter. Now it's just the way you think, and these biases can be self-reinforcing.

Look, I know my co-workers hate me, I've seen what they say when they think I'm not looking. If my increased Adderall dose has anything to do with it, I should be happy it slapped my out of my naivete. You just want me to stop taking it because you're jealous.

A lot of the times we acquire a new cognitive pattern, it is because we're getting smarter — that's a big part of growing up, and we grow up at a formative time in our lives. How do we know which new cognitive patterns to be suspicious of? My recommendation: take the outside view.

Try to figure out where a new cognitive pattern might have come from and look at how similar patterns with similar causes have played out in other people. Amphetamine paranoia, for instance, has few accomplishments to its name, and many people under its influence arrive at beliefs that don't match reality. If the p...