trentmkelly/LessWrong-43k huggingface dataset:
Announcing Nonlinear Emergency Funding
by KatWoods — 2022-11-13T19:02:57.803Z

[Applications are now closed. Please apply to Open Phil or SFF or other funders]

Like most of you, we at

Nonlinear

are horrified and saddened by recent events concerning FTX.
Some of you counting on Future Fund grants are suddenly finding yourselves facing an existential financial crisis, so, inspired by the

Covid Fast Grants

program, we’re trying something similar for EA. If you are a Future Fund grantee and <$10,000 of bridge funding would be of substantial help to you, fill out this

short form

(<10 mins) and we’ll get back to you ASAP.

We have a small budget, so if you’re a funder and would like to help, please reach out:

katwoods@nonlinear.org

[Edit: This funding will be coming from non-FTX funds, our own personal money, or the personal money of the earning-to-givers who've stepped up to help. Of note, we are undecided about the ethics and legalities of spending Future Fund money, but that is not relevant for this fund, since it will be coming from non-FTX sources.]
The Alignment Community Is Culturally Broken
by sudo — 2022-11-13T18:53:55.054Z

Disclaimer: These are entirely my thoughts. I'm posting this before it's fully polished because it never will be.

Epistemic status: Moderately confident. Deliberately provocative title.

Apparently, the Bay Area rationalist community has a burnout problem. I have no idea if it's worse than base rate, but I've been told it's pretty bad. I suspect that the way burnout manifests in the rationalist community is uniquely screwed up.

I was crying the other night because our light cone is about to get ripped to shreds. I'm gonna do everything I can to do battle against the forces that threaten to destroy us. You've heard this story before. Short timelines. Tick. Tick. I've been taking alignment seriously for about a year now, and I'm ready to get serious. I've thought hard about what my strengths are. I've thought hard about what I'm capable of. I'm dropping out of Stanford, I've got something that looks like a plan, I've got the rocky theme song playing, and I'm ready to do this.

A few days later, I saw this post. And it reminded me of everything that bothers me about the EA community. Habryka covered the object level problems pretty well, but I need to communicate something a little more... delicate.

I understand that everyone is totally depressed because qualia is doomed. I understand that we really want to creatively reprioritize. I completely sympathize with this.

I want to address the central flaw of Akash+Olivia+Thomas's argument in the Buying Time post, which is that actually, people can improve at things.

There's something deeply discouraging about being told "you're an X% researcher, and if X>Y, then you should stay in alignment. Otherwise, do a different intervention." No other effective/productive community does this. I don't know how to put this, but the vibes are deeply off.

The appropriate level of confidence to have about a statement like "I can tell how good of an alignment researcher you will be after a year of you doing alignment research" feels like it should be pretty low. At a year, there's almost certainly ways to improve that haven't been tried. Especially in a community so mimetically allergic to the idea of malleable human potential.

Here's a hypothesis. I in no way mean to imply that this is the only mechanism by which burnout happens in our community, but I think it's probably a pretty big one. It's not nice to be ...
The Futility of Status and Signalling
by Ape in the coat — 2022-11-13T17:14:57.480Z

I

In The Futility of Emergence Eliezer names a current theory which is as flawed as the theories of flogiston and vitalism. Theories that give mysterious answers which just appears to explain the phenomena while not actually resolving the confusion.

I think I can name another one. A theory which also happen to have much more influence in LessWrong agecent communities than emergence. Status seeking and signalling as an explanation for human behavior.

II

When I first heard about signalling it felt as a useful concept. Indeed some people seem to do things not because they actually believe that this is the best course of action but because they want to appear as someone who does. Because some sort of behaviour is associated with highter status and they want to be high status.

But on a reflection and due to seeing how people actually use this concept I noticed that it's extremely unhelpful. When people talk about signalling they do not distinguish between true signalling - when a signal actually correspond to the belief, and a false signalling - when it doesn't. Both of these very different cases are just put in the same category which creates more confusion instead of resolving it.

Similarly with status seeking. At first it felt, dare I say, reductive. Now we can explain all kind of different behaviours by the same intrinsic drive. But then I noticed that we can explain literally any behaviour by this same intrinsic drive. People who spend money on luxury items are doing it for the sake of status signalling. People who do not... well they do it for counter signalling reasons, they are still seeking status they just associate it with not having luxury items. At this point this is just like with phlogiston that saturated the air thus the fire ceased. We can make all kind of post-hoc explanations. But can we actually predict anything?

III

At best, these theories just do not bring much new to the table. At worst they are actively misleading. Knowledge about them can make it even harder to communicate about objective level problems, derailing the conversation to the meta level. I've seen the examples of it multiple times.

Person A claims that we should do some policy X.
Person B accuses A of virtue signalling.
And now the conversation shifted to the nature of signalling, despite the fact that it's completely irrelivant to the objective question of w

...
A short critique of Vanessa Kosoy's PreDCA
by Martín Soto — 2022-11-13T16:00:45.834Z

This critique is an addendum to my distillation for application problem 3 for Nate Soares and Vivek Hebbar's SERI MATS stream. Reading my distillation is not required if you are familiar with PreDCA.

Disclaimer: This is just my model of how Vanessa thinks and I might be misrepresenting some views. Furthermore, as mentioned below, I'm sure Vanessa is well aware of these issues, and plans on trying to solve those which constitute real obstacles. [Edit: See her comments below]

Vanessa( and Diffractor)'s work generally feels different to other Alignment theory, and I have usually attributed this to its radical focus on foundations (shared by some other researchers) and the complexity of its technical mathematical results (shared by few). But upon momentarily coarsening these fine technical details, and presenting PreDCA more conceptually in a language similar to that of other proposals, it becomes clear that it really is fundamentally different to them. As a consequence of Vanessa's opinions and approach, PreDCA has different objectives and hopes to most proposals.

PreDCA is fundamentally concrete. Yes, it still includes some "throw every method you have at the problem" (as in Classification). But the truly principal idea involves betting everything on a specific mathematical formalization of some instructions. This concreteness is justified for Vanessa because nothing short from a watertight solution to agent foundations will get us through Alignment[1]. Some AGI failure modes are obviously problems humanity would have to deal with in the long run anyway (with or without AGI), but on a time trial. And Vanessa is pessimistic about the viability of some intuitively appealing Alignment approaches that try to delegate these decisions to future humanity. In short, she believes some lock-in is inevitable, and so tries to find the best lock-in possible by fundamentally understanding agency and preferences.

Now, if PreDCA works, we won't get a naively narrow lock-in: the user(s)'s utility function will care about leaving these decisions to future humanity. But it will, in other subtler but very real ways, leave an indelible fingerprint on the future. I'm certain Vanessa is aware of the implications, and I suppose she's just willing to take yet another chance (on this lock-in being broadly beneficial), since otherwise we have no chance at all. But still, PreDCA seems to me ...
What's the Alternative to Independence?
by jefftk — 2022-11-13T15:30:01.186Z

When I

talk about

teaching my kids

to be independent, trying to
get them to where they can do more on their own, one response I
often get is, paraphrasing:

I like spending time with my kids and don't see it as a
burden. Bringing them to school, playing with them, making their food:
these are all chances to connect and enjoy being together. They'll be
old enough not to need me soon enough, and in the meantime I enjoy
bringing them to the playground.

So, first, I also like spending time with my kids! We do a lot of
things together, and I'm happy about that. But it's also common that
they'll want to do things that I can't do with them:

One wants to go to the park and the other wants to stay home.

One of them is ready to go to school and wants to get there
early to play with friends, but the other isn't ready yet.

With a third child now this
comes up even more. At times:

The older two want to go over to a friend's house, but the baby
is napping.

I'm still feeding the baby breakfast when it's time for school.

The best time for the baby's afternoon nap conflicts with
school pickup.

The alternative to doing things on their own is typically not us doing
the same things together. Instead, it's at least one kid needing to
accept doing something they like much less, and typically a lot more
indoor time.

I do think there is some truth in the original point, though. There
are times when the alternative to "they go to the park" is just "I
take them to the park". Sometimes that's fine (they want to play with
friends, I want to write), other times less so (they want to play monster and I don't have
anything that's actually more important). With this approach you do
need to be thoughtful about making sure you're spending an amount of
time with them that you all are, and will be, happy
with.

Comment via: facebook, mastodon
Decision making under model ambiguity, moral uncertainty, and other agents with free will?
by Jobst Heitzig — 2022-11-13T12:50:15.158Z
Link: https://forum.effectivealtruism.org/posts/dj2m4NggJBpyD5enX/decision-making-under-model-ambiguity-moral-uncertainty-and

The linked post was originally written in the EA forum before realizing that this forum contains already a lot about decision theory. Since I believe it still contains some original thoughts, I link-post it here.

From the introduction:

What is the state of the art or best practise or common practise in the EA community for taking individual decisions if there are various forms of ambiguity and/or non-quantifiable uncertainty involved, such as...

model ambiguity (e.g. about prior probability distributions, conditional probabilities and other model parameters)
moral uncertainty (e.g. about risk attitudes, inequality aversion, time preferences, moral status of beings, value systems, etc.)
strategic ambiguity (e.g. how rational are other agents and what can we really assume that they will do given that there might be free will?)

My own thoughts on how it might be done at least in theory are summarized below.

In that context, I also wonder:

Is there some place for smart collective decision making in this, e.g. in oder to

increase epistemic quality of decisions through crowd-sourcing information
raise acceptability of decisions and thus improve implementation quality
deal robustly with moral uncertainty and diverse value assessments

And if so, what collective decision making mechanisms are most appropriate?

I'd be more than happy to hear about your thoughts on this! Jobst
Characterizing Intrinsic Compositionality in Transformers with Tree Projections
by Ulisse Mini — 2022-11-13T09:46:56.555Z
Link: https://arxiv.org/abs/2211.01288

From Twitter

Transformers become more ‘tree-like’ over the course of training, representing their inputs in a more hierarchical way. The authors find this by projecting transformers into the space of tree-structured networks. [Stanford, MIT]

Abstract

When trained on language data, do transformers learn some arbitrary computation that utilizes the full capacity of the architecture or do they learn a simpler, tree-like computation, hypothesized to underlie compositional meaning systems like human languages? There is an apparent tension between compositional accounts of human language understanding, which are based on a restricted bottom-up computational process, and the enormous success of neural models like transformers, which can route information arbitrarily between different parts of their input. One possibility is that these models, while extremely flexible in principle, in practice learn to interpret language hierarchically, ultimately building sentence representations close to those predictable by a bottom-up, tree-structured model. To evaluate this possibility, we describe an unsupervised and parameter-free method to functionally project the behavior of any transformer into the space of tree-structured networks. Given an input sentence, we produce a binary tree that approximates the transformer's representation-building process and a score that captures how "tree-like" the transformer's behavior is on the input. While calculation of this score does not require training any additional models, it provably upper-bounds the fit between a transformer and any tree-structured approximation. Using this method, we show that transformers for three different tasks become more tree-like over the course of training, in some cases unsupervisedly recovering the same trees as supervised parsers. These trees, in turn, are predictive of model behavior, with more tree-like models generalizing better on tests of compositional generalization.

Note: The datasets they use aren't typical human text. See page 5.

I'm interested in discussions and takes from people more familiar with LMs. How surprising/interesting is this?
Noting an unsubstantiated belief about the FTX disaster
by Yitz — 2022-11-13T05:37:03.087Z

There is a narrative about the FTX collapse that I have noticed emerging[1] as a commonly-held belief, despite little concrete evidence for or against it. The belief goes something like this:

Sam Bankman Fried did what he did primarily for the sake of "Effective Altruism," as he understood it. Even though from a purely utilitarian perspective his actions were negative in expectation, he justified the fraud to himself because it was "for the greater good." As such, poor messaging on our part[2] may be partially at fault for his downfall.

This take may be more or less plausible, but it is also unsubstantiated. As Astrid Wilde noted on Twitter, there is a distinct possibility that the causality of the situation may have run the other way, with SBF as a conman taking advantage of the EA community's high-trust environment to boost himself.[3] Alternatively (or additionally), it also seems quite plausible to me that the downfall of FTX had something to do with the social dynamics of the company, much as Enron's downfall can be traced back to [insert your favorite theory for why Enron collapsed here]. We do not, and to some degree cannot, know what SBF's internal monologue has been, and if we are to update our actions responsibly in order to avoid future mistakes of this magnitude (which we absolutely should do), we must deal with the facts as they most likely are, not as we would like or fear them to be.

All of this said, I strongly suspect[4] that in ten years from now, conventional wisdom will hold the above belief as being basically cannon, regardless of further evidence in either direction. This is because it presents an intrinsically interesting, almost Hollywood villain-esque narrative, one that will surely evoke endless "hot takes" which journalists, bloggers, etc. will have a hard time passing over. Expect this to become the default understanding of what happened (from outsiders at least), and prepare accordingly. At the same time, be cautious when updating your internal beliefs so as not to assume automatically that this story must be the truth of the matter. We need to carefully examine where our focus in self-improvement should lie moving forward, and it may not be the case that a revamping of our internal messaging is necessary (though it may very well be in the end; I certainly do not feel qualified to make that final call...
Solstice 2022 Roundup
by dspeyer — 2022-11-12T21:26:25.549Z

Many of you are already familiar with Rationalist Winter Solstice, our home-grown winter holiday. As the year grows literally dark, we gather in our respective communities to face various forms of darkness together, to celebrate what light human civilization has made, and to affirm ourselves as a community of shared values.

This thread is a central place to gather information about specific events. Please post times, places, registration or rsvp links, restrictions if any, etc.
Musings on the appropriate targets for standards
by tailcalled — 2022-11-12T20:19:38.939Z

Consider two norms for how to critique those who violate your standards:

L: Focusing critique on those in power, who have the greatest opportunity to cause harm.
R: Focusing critique on those who it most accurately applies to, who most often cause harm.

I think both of these are to an extent needed, but also that they are to an extent conflicting with each other, because society is somewhat meritocratic and power therefore somewhat correlates with virtue.

In an extreme, they also both have failure modes. Too much of policy L, and you destroy valuable organizations for unimportant transgressions. Too much of policy R, and you leave no room for slack or diversity.

But you can't totally drop either of them because good incentives are important. With none of L, your world ends up run by strongmen who grab all the resources for themselves. With none of R, your world ends up with destructive people running rampant, and lacks incentives for productivity.

I suggestively named these policies L and R because they seem to somewhat resemble leftism and rightism respectively.

It is probably also important to not obsess too much over the levels of these. L and R are one or two degrees of freedom. If they are set wrong, then yes, that can be a serious problem, but if they are set ~right, there's not much more you can do with them. Meanwhile, actually acting in the real world rather than moving norms, there's an infinitude of problems and opportunities that you can work on.
Ways to buy time
by Orpheus16 — 2022-11-12T19:31:10.411Z

In our last post

, we claimed:

On the margin, we think more alignment researchers should work on “buying time” interventions instead of technical alignment research (or whatever else they were doing).

But what does “buying time” actually look like? In this post, we list some interventions that have the potential to buy time (some of which also have other benefits, like increasing coordination, accelerating community growth, and reducing the likelihood that labs deploy dangerous systems).

If you are interested in any of these, please reach out to us. Note also that Thomas has a list of specific technical projects (with details about how they would be implemented), and he is looking for collaborators.

Summary

Ideas to buy time:

Direct outreach to AGI researchers
Develop new resources that make AI x-risk arguments & problems more concrete
Demonstrate concerning capabilities & alignment failures
Break and redteam alignment proposals (especially those that will likely be used by major AI labs)
Organize coordination events
Support safety and governance teams at major AI labs
Develop and promote reasonable safety standards for AI Labs
Other ideas

Summary table:

Intervention

Why it buys time

Other possible benefits

Technical or non-technical

Direct outreach (through written resources and 1-1 conversations)

Some ML/AGI researchers haven’t heard the core arguments around AI x-risk. Engaging with written resources will cause some of them to be more concerned about AI x-risk.

Some ML/AGI researchers have heard the core arguments. 1-1 conversations can allow safety researchers to better understand & address their cruxes (and vice-versa)

Generates new critiques of existing alignment ideas, arguments, and proposals

More people do alignment research

Increased trust and coordination between labs and non-lab safety researchers.

Technical + Nontechnical (non-technical people can organize and support these efforts, with technical people being the ones actually having conversations, giving presentations, choosing outreach resources, etc.)
New resources	Many ML/AGI researchers have heard the theoretical ideas but want to see resources that are more concrete & grounded in empirical research.	Formalizes problems; makes it easier for ML experts & engineers to contribute to alignment research.	Technical
Concerning demonstrations of alig

...
How do newcomers delve deeper into the community?
by Lord Dreadwar — 2022-11-12T19:00:29.018Z

I have some experience managing large online communities of various sorts, and it struck me that some of the most difficult but necessary work across all of them was in spending hours brainstorming, with our leadership teams, how we might pull the 80% (or 50%, or 99%) of our membership, blissfully ignorant of the true depths of the community, into the heart of the network, where all the fun and the socialising and the opportunities were—the community within the community. (Incidentally, the harsh lessons of trial-and-error and a little bit of internal meritocratic competition ended up invariably outputting a solution of pouncing on new contributing members and pulling them into the hidden depths of community involvement via an apprenticeship/adoption/message-buddy system.)

I ask this for the benefit of new members who might be drawn to us in light of recent publicity: what are the hidden depths to LessWrong? How does one become more involved in this space, and adjacent spaces?

I noticed a similar question in the community resources, and links to Discord servers, and so forth. I joined the community in 2010, primarily engaging with overlapping counterpart groups on Facebook, subreddits, CFAR events, HPMOR meetups, local LessWrong meetups, and the like. But now that the community has grown in membership and resources, how do we address questions like becoming more involved in-person in the Bay Area community, or working in EA? Are there any layers to the LessWrong onion that longtime and more involved members might take for granted, but are quite invisible to newcomers, long-time lurkers, and less involved members?

Quite simply, how does one become as involved in the rationalist community as possible? It would be a good idea to develop an FAQ around this; I have the suspicion many will be seeking networking opportunities through LessWrong in the future.