trentmkelly/LessWrong-43k huggingface dataset:
Conservatism is a rational response to epistemic uncertainty
by contrarianbrit — 2022-07-31T18:04:36.714Z
Link: https://thomasprosser.substack.com/p/how-conservatism-became-more-reasonable

1.

Recurrently, humans respond to change in a conservative manner, established traditions protecting against unsettling developments. To a great extent, the reasonableness of conservative responses hinges upon external conditions, change being more rational in certain ages.

Today, we may be living in an age in which conservatism is more reasonable. This reflects increasing doubt about the ability of politics to achieve change. Following a replication crisis which suggests that many academic results are spurious or overestimated, there are fewer reasons to trust policy interventions.

Relatedly, there is growing confidence in the integrity of human institutions. In recent years, cultural evolution theory has demonstrated the role of cultural inheritance in adapting species to social and biological environments. Moreover, scholars have emphasized unintended consequences associated with institutional change, reflecting limits on human rationality.

These developments do not entail a case against all change – in many contexts, policy interventions remain reasonable and necessary – yet shift the basis of calculations. Compared to decades ago, conservatism now appears more reasonable.

This is not an endorsement of authoritarianism. Though certain definitions of conservatism emphasize hierarchy and discipline, I understand conservatism as the defence of established institutions. Aside from the archetypal ways in which conservatives oppose change, small-c conservatives might resist change in atypical areas, such as resistance to abortion bans, labour market deregulation and Brexit.

One development presents a serious challenge to my argument: the increasing likelihood of exogenous shocks, primarily associated with the climate crisis. These will justify profound reforms, involving the ruptures which conservatives oppose, yet enhance other arguments for conservatism; if there are great changes in some areas, there is a stronger case for minimizing change in other domains.

2.

Ideologies are associated with epistemic regimes; in other words, different political views reflect how much we think we know about the world. For example, those with high degrees of certainty tend to be less tolerant of others, totalitarian regimes embodying this tendency. Mercifully, liberal democracy is associated with lower degrees of certainty, this system allowing disse...
Perverse Independence Incentives
by jefftk — 2022-07-31T14:40:05.191Z

In writing about parenting, I don't write enough about what didn't
work: this might give the impression that everything I try works,
which is not the case! Here's an example of something that didn't,
and which I might have caught before trying if I'd thought a bit more.

Brief recap: I see one of my main roles as a parent being building independence:

I want to let my kids be as independent as possible, as early as
possible. Not only does it make their lives better, because they can
meet their own needs how they want, but it makes my life easier,
because they can handle more on their own. Sometimes this involves a
bit more effort up front, but I think it's substantially less effort
in total.

For a while I had an approach which was essentially, "if you can do it
yourself I'm not going to do it for you." For example, soon after
Lily learned how to get her own cereal, we had some conversations
like:

Lily: Papa, can I have some cereal?

Me: You know how to do that!

Lily: I don't want to, and you used to get it for me.

Me: Now that you can do it I'm not going to do it for you.

Lily: [unhappy]

After a few weeks I realized this was counterproductive. Yes, it did
mean more practice at new skills, and it is where I want us to end up
eventually, but the incentives are all wrong. Instead of her former
excitement at learning new things and expanding what she was capable
of, each new skill now made her life harder because I was less
helpful. You can sort of think of this as too high a marginal tax
rate: I was claiming too much of the surplus from her investments in
her abilities.

After I realized it wasn't working I toned this down a lot. I will
still sometimes refuse to do things they know how to do themselves,
but most of the time they're happy enough to do the thing on their
own, whenever they want, at their own speed, and in they way they
prefer, that I don't need to push them along.

One place where I'll still consistently refuse is when I wouldn't have
done it for them even before they learned how. For example, if they
don't like dinner and ask me to heat up something else I'll do that,
but then if they don't like that I won't just keep getting them
additional things. Now that she can use the microwave, however, Lily
has an additional option, which is that she can go heat herself
something up on her own.

Comment via: facebo...
Wolfram Research v Cook
by Kenny — 2022-07-31T13:35:21.925Z

Matthew Cook probably violated an NDA he signed with Wolfram Research

This is a follow-up to this comment on a previous post of mine about (Stephen) Wolfram.

The comment, by gwern:

Charitably, I think that makes his apparent 'arrogance' better understood as something like a 'literary convention', especially given the plausibility of him having 'independently re-discovered' some of the particular results he reports.

Is it customary to sue over literary conventions?

Gwern linked to the "History" section of the Wikipedia page about Rule 110:

In 2004, Matthew Cook published a proof that Rule 110 with a particular repeating background pattern is Turing complete, i.e., capable of universal computation, which Stephen Wolfram had conjectured in 1985.[1] Cook presented his proof at the Santa Fe Institute conference CA98 before publication of Wolfram's book A New Kind of Science. This resulted in a legal affair based on a non-disclosure agreement with Wolfram Research.[2] Wolfram Research blocked publication of Cook's proof for several years.[3]

I suspect that the account in the second comment below is the most plausible explanation of what resulted in this lawsuit; from a comment on Hacker News and two replies to it from NKS contains a mention of the proof that Rule 110 is Turing Complete. This is ge... | Hacker News [formatted]:

LowKarmaAccount on June 24, 2013 | parent | context | favorite | on: Celebrating Mathematica’s First Quarter Century

NKS contains a mention of the proof that Rule 110 is Turing Complete. This is genuinely interesting, but it doesn't change its field in a significant way.
Of course, Wolfram didn't come up with the proof himself. That was done by Matthew Cook while he was working for Wolfram Research. Stephen Wolfram obtained a court order preventing Cook from presenting his result before NKS was published, because he claimed that publishing this result violated an NDA Cook signed.

taliesinb on June 24, 2013 | next [–]

According to Wolfram, Cook's job was to prove Rule 110 was universal. Cook didn't "discover" it as an independent action.

In a very direct sense, this was a software development task, where the program that was being written was precisely a universal program (through several layers of emulation, of course; rule 110 -> 'particle computer' -> cyclic tag system -> Turing machine). It's an impressive program, but also

...
Wanted: Notation for credal resilience
by PeterH — 2022-07-31T07:35:25.953Z

Meta: I've spent 30-60 minutes thinking about this, and asking people who I'd expect to know about existing notation. I don't have scientific training, and I'm not active in the forecasting community.

Problem

I want a clear shorthand notation for communicating credal resilience.

I want to be able to quickly communicate something like:

My 80% confidence interval is 5-20. I think there's 10% chance I'd change my upper or lower bound by more than 50% of the current value if I spent another ~day investigating this.

I’m using the term “credal resilience”. Some people call this "robustness of belief".

Existing notation for confidence intervals

APA style guide suggests the following:

80% CI [5, 20]

This seems like the best and probably most popular option, so let's build on that.

Proposal

For clarity, I'll repeat the example I gave above:

My 80% confidence interval is 5-20. I think there's 10% chance I'd change my upper or lower bound by more than 50% of the current value if I spent another ~day investigating this.

To communicate this, I propose:

80% CI [5, 20] CR [0.1, 0.5, 1 day]

Low numbers in the first two parameters indicate high credal resilience. The unit of additional investigation tells you the approximate cost of further investigation to "buy" this extra information.

You can specify hour / day / week / month / year for the unit of additional investigation.

Thoughts?

I'd love to hear people's thoughts on this. Two questions I'll highlight in particular:

Do you think it'd be worth developing a good notation for credal resilience, then popularising it?

What do you think of my particular proposal? What might be better?
Anatomy of a Dating Document
by squidious — 2022-07-31T02:40:00.863Z

I’ve noticed a trend of writing out dating documents (or web pages, blog posts, etc) as a means of having everything you’d normally put on a dating profile in one place. There’s a particular way many of these are written, such that they’re more straightforward and practical about dating than online dating profiles and apps tend to be. It honestly reminds me a little of how arranged marriage works nowadays in India. In an effort to better understand what it is people tend to expect of these documents (and throw in some of my own thoughts), I decided to read through every dating document I could find (mostly through reciprocity and Bountied Rationality) and compare/contrast to find common themes, as well as good ideas of what to include that should be more common. Though written as a guideline on how to write your own dating doc, this is equally (if not moreso) a meta-analysis on what's been put into those already out there.

TLDR: a dating document should tell the reader who you are, where you’re at, where you want to be, who you’re into, anything that can’t or won’t change, non-negotiables, and dealbreakers.

Start with basic information about you. Along with the standard “age, sex[/gender], location”, people typically share details about their current life status, such as occupation or school, housing situation, and anything else static in your life, like pets and kids. This is also the place to put orientation, which includes sexual and romantic orientation as well as preferences for monogamy vs polyamory. Some people will choose this space as the place to elaborate on any current partners or kink alignments as well.

Next comes the bulk of it, focusing on who you are and your personality. It can be helpful to use typologies like Myers-Briggs, Big 5, Enneagram, etc, but I’d suggest putting this only if you identify with your typology. It’s important to note that anything you say about yourself also reveals which of your own qualities you find important, and similarly, mentioning typologies implies that you agree with the assumptions revealed by them (just as mentioning zodiac implies you believe in that). The traits highlighted in these typologies are also traits people tend to focus on when describing themselves, particularly introversion vs extroversion and thinking vs feeling from Myers-Briggs, and openness and conscientiousness from Big 5. Aside from these, ...
chinchilla's wild implications
by nostalgebraist — 2022-07-31T01:18:28.254Z

(Colab notebook here.)

This post is about language model scaling laws, specifically the laws derived in the DeepMind paper that introduced Chinchilla.[1]

The paper came out a few months ago, and has been discussed a lot, but some of its implications deserve more explicit notice in my opinion. In particular:

Data, not size, is the currently active constraint on language modeling performance. Current returns to additional data are immense, and current returns to additional model size are miniscule; indeed, most recent landmark models are wastefully big.

If we can leverage enough data, there is no reason to train ~500B param models, much less 1T or larger models.
If we have to train models at these large sizes, it will mean we have encountered a barrier to exploitation of data scaling, which would be a great loss relative to what would otherwise be possible.

The literature is extremely unclear on how much text data is actually available for training. We may be "running out" of general-domain data, but the literature is too vague to know one way or the other.
The entire available quantity of data in highly specialized domains like code is woefully tiny, compared to the gains that would be possible if much more such data were available.

Some things to note at the outset:

This post assumes you have some familiarity with LM scaling laws.
As in the paper[2], I'll assume here that models never see repeated data in training.

This simplifies things: we don't need to draw a distinction between data size and step count, or between train loss and test loss.

I focus on the parametric scaling law from the paper's "Approach 3," because it's provides useful intuition.

Keep in mind, though, that Approach 3 yielded somewhat different results from Approaches 1 and 2 (which agreed with one another, and were used to determine Chinchilla's model and data size).
So you should take the exact numbers below with a grain of salt. They may be off by a few orders of magnitude (but not many orders of magnitude).

1. the scaling law

The paper fits a scaling law for LM loss L, as a function of model size N and data size D.

Its functional form is very simple, and easier to reason about than the L(N,D) law from the earlier Kaplan et al papers. It is a sum of three terms:

L(N,D)=ANα+BDβ+E

The first term only depends on the model size. The second ter...
AGI-level reasoner will appear sooner than an agent; what the humanity will do with this reasoner is critical
by Roman Leventov — 2022-07-30T20:56:54.532Z

The recent advancements in language models, Gato, Minerva, etc. suggest to me that there will be AGI-level "predictor"/"reasoner"/generative model sooner than an "agent". There is some subtlety though w.r.t. Gato, which is an "agent", but decidedly not an RL agent, it doesn't intrinsically value exploration, and it doesn't learn itself/bootstrap itself, and instead does imitation learning from other (actual RL) agents.
I feel that this reasoner may suddenly generalise to astonishing capability, e. g. prove math theorems far beyond human comprehension, derive the theory of everything from data, and advance the state of the art in political science, ethics, and engineering, while remaining a non-agent and thus remaining relatively safe.
I feel that it will be very important then to resist turning such an AI into an agent by attaching actuators to it and giving it tasks. However, this will be hard, because economic incentives for doing so will be enormous.

This shortform is probably a vague expression of the idea, which is much better thought through here: Conditioning Generative Models for Alignment.
What job should I do?
by Tom Paine — 2022-07-30T09:15:22.574Z

I am a Phd student who has just submitted my thesis, and I am not sure what to do with myself, so I would like some advice on interesting jobs/fields/career paths to consider. I would welcome suggestions of particular roles or companies, some generic guidance, or pointers to other resources. I have included what I think is all the relevant information to my job search below, but I will happily provide more details/clarification if needed.

Education: My undergrad was in Maths and Phd was in Computer Science (though it was more or less just pure maths). Throughout my studies I gravitated towards the abstract stuff rather than the practical (e.g. I have studied Statistics but haven't used R, and have studied Category Theory but not used Haskell), though I am of course willing to learn the the right practical skills for a job. My Phd thesis was on some topics in Finite Model Theory.
I enjoyed the content of my Phd but I was not at all suited for the working environment (hence I have no interest in continuing with academia right now). In particular I found it isolating, and I struggled with self-direction with long, open-ended goals. I would like to work somewhere in future where I work as part of a team, and am provided clear goals and deadlines.
My priority is to find a job I find interesting and engaging. As noted above, I did enjoy the Maths I studied. What I have found to be most engaging as a hobby has been trading card games and board games.
I have lots of experience teaching (private tutoring, teaching undergrads, and summer schools) which I enjoyed. I also did some esports casting.
I only speak English.
My ideal location is New York, but I could be persuaded to go pretty much anywhere for the right job. I am a UK citizen, so I'd need to work for somewhere which sponsored visas if it was outside the UK.

Thanks for any advice or suggestions.
How transparency changed over time
by ViktoriaMalyasova — 2022-07-30T04:36:32.386Z

This post is written as part of the SERI MATS program.

This post explores how model interpretability has changed over time, and if the existing approaches to interpretability are likely to remain viable as the models increase in size and capability. I examine several approaches to interpretability:

simulatability;
feature engineering;
model-based interpretability;
post-hoc interpretability.

Simulatability

Before the rise of machine learning models were interpretable, because they were simulatable: that is, a human could reproduce the model’s decision making process in their head and predict its behavior on new inputs. This can be done when the model is simple and the input is low-dimensional: a small decision tree, a linear regression, or a set of rules. Sometimes it turns out that a simple algorithm can solve the problem just as well as a more complicated one.

For example, in the 2018 Explainable Machine Learning challenge, participants were asked to train a model to predict credit risk scores, and produce an explanation for it. Instead of training a black-box model and producing a post-hoc explanation, the winning team trained an interpretable two layer additive model. It was understandable because it was simple, but it had competitive accuracy.

However, we cannot expect simple models to achieve high accuracy when the underlying relationship is complicated. Tracking computations in one's head will become impossible with high-dimensional inputs, such as images. When we need more complicated models, simulatability stops working.

Feature engineering

The relationship between the target variable and the raw input data may be complicated. But expert domain knowledge can sometimes be used to construct useful and meaningful features and greatly simplify the relationship the model has to learn, making it easier to interpret.

For example, in [1], the authors build a model for cloud detection in arctic satellite imagery. Using climate science knowledge and exploratory data analysis, authors design 3 simple features that prove sufficient to classify clouds with high accuracy using quadratic discriminant analysis.

Feature engineering works equally well for any model size. However, it relies on human expert knowledge or engineering effort to construct features for every domain. The trend in machine learning [2] has been that general methods that leverage computation eventually...
Translating between Latent Spaces
by JamesH — 2022-07-30T03:25:06.935Z

​Produced as part of the SERIMATS Program 2022 Research Sprint under John Wentworth

Introduction

The gold-standard of interpretability in ML systems looks like finding embeddings of human-identifiable concepts in neural net architectures, and being able to modify, change, and activate them as we wish. The first hurdle is identification of these concepts. We propose that it may be easier to identify simpler concepts in simpler models, and use these to bootstrap to more complex concepts in more intricate models.

To this end, we first propose a definition of what it means for a concept to be present in a model. Then we investigate how we can identify similar concepts across different models. We begin by demonstrating these definitions and techniques in a simple example involving Bayes nets.

We then train two autoencoders (small and large) on the FashionMNIST dataset. We choose some latent concepts that humans would use to represent shoes (shoe height and shoe brightness) and see whether these human concepts can be transferred to the models, and how the models' representations relate.

A simple example

Simple Environment

First we construct a simple environment:

There are 10 cells.
Each cell can contain nothing, a blue circle, a red circle, or both.
There is one red circle.

The red circle moves 1 right each timestep.
If the red circle cannot move right, it moves to the leftmost cell.

There are two adjacent blue circles.

The blue circles move 1 left or 1 right each timestep.
If a blue circle reaches the environment's edge, both their directions change.

If a cell contains a red circle and a blue circle, it shows only a blue circle.

A sample is given below:

Two Models of this Environment

We model the evolution of this environment using two different Bayes nets. Each Bayes net reflects a different way of viewing this environment corresponding to:

An object centric model
A local-state centric model

Object Centric Model (Model 1)

The object centric model tracks 3 latent variables corresponding to the object-level description of where the blue and red circles are and in which direction the blue circles are moving at a given timestep:

blue location (taking values in {0,1,2,3,4,5,6,7,8,9})
red location (taking values in {0,1,2,3,4,5,6,7,8,9})
blue velocity (taking values in {−1,1})

It also has 10 observational variables given by the 10 cells, which take values in {blank,blu...