trentmkelly/LessWrong-43k huggingface dataset:
DSLT 4. Phase Transitions in Neural Networks
by Liam Carroll — 2023-06-24T17:22:38.358Z

TLDR; This is the fourth main post of Distilling Singular Learning Theory which is introduced in DSLT0. I explain how to relate SLT to thermodynamics, and therefore how to think about phases and phase transitions in the posterior in statistical learning. I then provide intuitive examples of first and second order phase transitions in a simple K(w) loss function. Finally, I experimentally demonstrate phase transitions in two layer ReLU neural networks associated to the node-degeneracy and orientation-reversing phases established in DSLT3, which we can understand precisely through the lens of SLT.

In deep learning, the terms "phase" and "phase transition" are often used in an informal manner to refer to a steep change in a metric we care about, like the training or test loss, as a function of SGD steps, or alternatively some hyperparameter like the number of samples from the truth n.

A classic example of a phase transition in deep learning literature is Anthropic's Induction Heads paper, where the "bump" in loss (in-context learning score) is referred to as a "phase change".

But what exactly are the phases? And why do phase transitions even occur? SLT provides us a solid theoretical framework for understanding phases and phase transitions in deep learning. In this post, we will argue that in the Bayesian setting,

A phase of the learning process corresponds to a singularity of K(w), and a phase transition corresponds to a drastic change in the posterior as a function of a hyperparameter θ.

The hyperparameter θ could be the number of samples from the truth n, some way of varying the model function f(x,w)=f(x,w;θ) or something about the true distribution Dn=Dn(θ), amongst other things. At some critical value θ=θc, we recognise a phase transition as being a discontinuous change in the free energy or one of its derivatives, for example the generalisation error Gn=E[Fn+1]−E[Fn].

In this post, we will present experiments that observe precise phase transitions in the toy neural network models we studied in DSLT3, for which we understand the set of true parameters W0 and therefore the phases. By the end of this post, you will have a framework for thinking about phase transitions in singular models and an intuition for why SLT predicts them to occur in learning.

Phases Correspond to Singularities

Th

...
Donate Now vs Donate Later - Relative Value of Donations to AI Alignment
by AlignmentOptimizer — 2023-06-24T17:20:35.382Z

General Case

Let's say I currently have 1M USD, and I can either:
1. Donate it to orgs working on AI alignment.
2. Invest the money and get returns >> than the stock market.

How big must the expected return of my investment be in what time frame for it to be better than donating right away?

My specific Case

I recently made around 1M USD pre tax, and consider how to best utilize it to decrease AI X-risk. I consider these two options:

Donate 45% to other orgs. Use the other 45% as funding source for a for-profit AI Alignment company, that would do a mix of money making AI endevours (that I will try to make sure doesn't improve general capabilities), do direct AI Alignment work. Then with time dedicate more and more resources to direct AI Alignment work/donations to other orgs.
Donate only 10%, and use the other 80% as funding for the for-profit AI Alignment company, where the bulk of the additional 35% would go at first mostly into money making endevors.

In both cases, 10% of the money is fun money/cover living expenses.

Let's say I hypothetically expect to make 10x returns after 5 years on the additional investments into for-profit in scenario 2, would that be a worthy investment?

I do realize that 10x in 5 years might seem unrealistic, but for the sake of discussion I think it is a useful figure, however wrong it might seem.

My Own Estimate

Donating 1 usd today, corresponds roughly to donating (adjusted for inflation):

1.18 usd in one year

1.39 usd in two years

2.39 usd in five years

This is based on a timeline of getting AGI in 15 years, and I simply played around with numbers and this seemed reasonable. I know "playing with numbers" is not a reliable approach, but I haven't come up with anything better, hence the post here.
How to prevent Re-Traumatization on Meditation Retreats
by EternallyBlissful — 2023-06-24T14:16:06.276Z

Meditation tends to bring to the surface unpleasant suppressed material like extremely unpleasant emotions, thoughts, or memories. When this happens while you're alone on a silent meditation retreat and don't know how to deal with it, there's a certain risk of re-traumatization. My intuition is that if you know what you are doing the risk for this is minimal and you don't have to worry much about this. But you should know what you are doing - hence this post.

What re-traumatization is and is not

It's NOT the process of said suppressed material suddenly coming to the forefront of your mind during the quiet of meditation and now spooking around in consciousness, potentially making you feel emotionally worse for some time. This is completely normal, and in a way it's actually exactly what you want to happen! Bringing unpleasant suppressed material to the surface is just what needs to happen for emotional healing. Still, this is the reason some people find themselves surprised they are actually not that happy during or shortly after a silent meditation retreat.

Re-traumatization - I'm using the term in a very broad sense - happens when a past overwhelming experience comes up from your unconsciousness, and you are again overwhelmed by it and still cannot deal with it.

The key to prevention: mindfulness & metacognitive awareness

You prevent re-traumatization by maintaining metacognitive awareness, which basically just means "stay mindful of your internal experience". That is, you don't get sucked in in those extreme feelings, don't fully blend with them, don't "get lost in thought in" them, don't let them be the only thing filling your consciousness.

Instead, it's crucial that you maintain some context: "Okay, I am here at this retreat where there is some chance for extremely negative feelings to come up. And indeed, this is exactly what I feel is happening right now. I can feel the strong negative feeling is happening am in contact with some past trauma. This is a perfectly normal, and actually even helpful process" - you don't actually have to think these sentences, but implicitly you should remain aware of their truth.

In IFS language, you need to remain in "Self". Whatever extreme thought or emotion arises, you need to see it as what it is: Just a thought or emotion, just a mental object in awareness. Nothing to be afraid of. Nothing t...
Can you prevent negative long-term effects of bad trips with sleep deprivation?
by EternallyBlissful — 2023-06-24T14:05:41.950Z

The idea is that you need sleep in order to consolidate your memories. If you deprive yourself of that after a bad trip on psychedelics, or after a traumatic event or re-traumatization, the newly formed bad memories don't consolidate.

I’ve heard that people are way less likely to get PTSD after a traumatic event if they experience strong sleep deprivation afterwards - as has happened to soldiers in war for example. I’ve also heard in conversations that this works in rats that are subjected to sleep deprivation after being put under traumatic stress.

If this were true, wouldn't this be an amazing hack to make psychedelic trips way safer? Now, I know that sleep deprivation is not easy and very unpleasant and probably the last thing you want to do after a bad trip - but it's a much less bad option compared to long-term negative effects of bad trips. And uppers like caffein, modafinil or amphetamines would make it easy to do.

I'd personally find it very nice if this option exists as a fallback plan B. For me, it would take away the “fear of fear”, or the "fear of a bad trip", because I would know that the worst thing that can happen is that I have a really bad time and maybe feel destabilized for a few weeks after, but there won't be any serious negative long-term effects - because I can always just skip a day or two of sleep afterwards!

Does anyone know if this would work?
We ran a reading group on The Scout Mindset
by Neil Crawford — 2023-06-24T10:10:04.063Z

Cross-posted from the EA Forum.

What did we do?

As organizers of the EA group at UC Irvine (UCI), we ran a reading group on Julia Galef’s book The Scout Mindset during the final 5 weeks of the academic year (2022-2023). The group had a total of 8 participants with an average of 5 per session. Compared to attendance rates earlier in the academic year, this is about average. The group was diverse in both gender and ethnicity. All participants committed to reading 1 section (i.e. 3 chapters) per week prior to our 1-hour in-person discussions. As organizers, we prepared

questions

to help guide the discussions. Straight after, we went out to dinner together to continue our conversations about The Scout Mindset, Effective Altruism and any personal updates.

Why did we do it?

This academic year we experimented with different program structures. In the first 10 weeks, we ran the EA Introductory Program; in the second 10 weeks, we ran our own version of the EA In-Depth Program. In the last 10 weeks, we spent the first half running weekly workshops and the second half running the reading group. We decided to run a reading group because we thought it would provide structured weekly content and that it would be fairly easy to organize because we needed only to read the relevant chapters, make notes and devise questions (we had already booked a classroom on a weekly basis). We chose The Scout Mindset in particular because many of our members had expressed an interest in reading the book and we too had been intending to read it ourselves. Running this reading group kept us motivated to read the book from beginning to end by holding us accountable.

What went well?

We successfully completed the book in the intended time without losing participants (except for one who went to Boston). Our prepared questions were useful in guiding our discussions without constraining them. They also helped spark new discussions and allowed us to focus on the present discussion without frantically scouring our minds for the next question to ask. The questions we created encouraged participants to connect what they had read to their own experiences. We found that participants were more enthused by these types of questions than by straightforward terminological questions. They were also supportive and encouraging of others, which helped everyone feel comforta...
Crisis Boot Camp: lessons learned and implications for EA
by Nicole Ross — 2023-06-24T06:28:54.533Z

Over the last 7+ months at work, I've needed to handle or support several crises (I'm on the US EV board and, in normal times, head of the community health and special projects team). It's been a crisis-handling boot camp, so I want to share my lessons learned. [1] [2]

I expect to learn more in the coming months, and it's plausible that longer-term ramifications could change my lessons learned, but I wanted to share my reflections at this point.

My reason for writing this: I generally think that at least some of us (maybe many of us) may go through many more crises and that the world has a decent chance of getting even crazier due to AI. I want us to learn from crises and make updates to be better prepared for next time. I'm worried about people returning to normal without making relevant updates on many levels. Hopefully, some of my lessons learned will contribute to people making updates and folks being more prepared next time!

Handling yourself in a crisis:

Expand your thinking

1. Hold multiple hypotheses at once

Generally, people struggle to hold more than one or two hypotheses simultaneously, and this struggle seems even stronger during a crisis.

The world is complicated, and being confident about what's true is tough. When people make plans, though, they often focus only on the hypothesis that they think is most likely, or at best, their top two hypotheses.

For a hypothetical example, imagine the following scenario:

You're working on getting life-sustaining and valuable resources to your allies in a place with a lot of organized crime. Pretty frequently, your supplies are stolen. You suspect Person X is behind it. One day he disappears, and a lot of supplies are missing. You need to move to the next location to get the supplies in the right hands. What should you do?

Create multiple hypotheses! You are assuming X is behind the theft. In that world, you might want to move on without him and think through things like "Does he know where we're going to be next? Is he going to steal more? How should we update our security measures?".

Other hypotheses you should consider:

Maybe he was kidnapped, in which case it would be pretty shitty to leave him.
Perhaps he was working with someone else. Are they on your team too?
It could be a coincidence, and he didn't show up for another reason (e.g., incompetence, sickness).

Each of thes...
I just watched don't look up.
by ATheCoder — 2023-06-23T21:22:25.337Z

So I recently watched Don't Look Up. This post has spoilers on the movie so if you want to watch it, stop reading because spoilers are ahead.

I couldn't help see the parallels between the asteroid and Global Warming or The AI Apocalypse. I think the creators actually meant for this to happen which means they have reached their goal.

Because some people might not be interested in watching the movie I will provide a little bit of background on the plotline of the movie. The movie starts by two scientists discovering that a comet is heading towards earth and it will cause an extinction of all life on earth. The scientist try to meet with the president of the United States to tell her about this. However, she seems very disinterested in the fact that this is happening. She is more worried about getting reelected even though if the comet hits there is not going to be an election to be worried about. The reasoning for her is that she deals with "World threatening crysises" all the time and that most often than not it is not really what it is. After that the scientists leak the whole thing on the media but the reaction in the media is very mild. Which causes the PhD student to lose her shit and start shouting in the TV (Gretta Thumberg anyone?). After a long time of trying they manage to convince people that this is happening after that they devise a plan. However the plan is canceled at the last minute because the CEO of a big company says the comet includes a lot of rare earth minerals that can be mined. (130 Trillion dollars worth). They decide to let the comet come close and then break it down so that they can mine the smaller parts. It is shown that according to peer reviews this plan is not going to work and most scientist agree that this is not worth the risk. However in the end greed prevails and they manage to use media to control people and tell them to "Don't Look Up" and ignore the comet. To my surprise the movie ended in a tragedy in the end the comet hits and everyone dies.

I liked this movie. It gave me lots of chills and something to think about. The main question that I have is that "Why didn't the scientists band together?". If most scientists agree that a certain threat is eminent and that everyone needs to do something about it, why didn't they band together like a giant tribe and make a plan and execute it irrespective of what the government...
Automatic Rate Limiting on LessWrong
by Raemon — 2023-06-23T20:19:41.049Z

The LessWrong team recently began rolling out automatic rate limits. The general idea is if a user gets heavily downvoted, the site automatically restricts their posting and commenting privileges. (See Well-Kept Gardens Die By Pacifism for some background on LessWrong's overall moderation philosophy)

After thinking awhile about various side-effects (see previous discussion), I designed some rate limits based on the following user metadata:

Total karma. i.e. all net upvotes/downvotes the user has ever received
Recent karma: The net karma the user received from their last 20 comments and/or posts (this does not include their own upvotes on their own content).
Last month's karma: The karma the user received for their last 20 comments/posts within the last month (i.e. a subset of the "recent karma." determined by fetching all votes on the last 20 comments/posts, and then filtering for votes on posts/comments that were published in the last 30 days)
Unique downvoter count: The number of individuals who've downvoted the user's recent content. This ensures that a single angry individual or small clique can't wreck your day.

The exact karma thresholds and rate-limit-strengths chosen are based on the LW team's experience with daily moderation maintenance. Every day, moderators look over ~20 posts and comments from new-ish users. We've built up some intuitions for what user-karma and comment/post-karma tends to correspond with stuff we want to see more of on LessWrong. And for the past month or so I've been checking how these four metrics apply to the current distribution of LW users.

The karma system isn't perfect. Sometimes you'll get randomly downvoted by someone for idiosyncratic or petty reasons. Sometimes it's hard for newer user's content to get noticed and upvoted. But people tend to be pretty hesitant to downvote stuff – most comments just don't get voted on. Downvoted content from new users tends to be some combination of poorly argued, difficult to read, rude-without-actually-saying-much, or rehashing topics that have been discussed extensively on LessWrong without introducing new considerations.

I suspect we'll iterate and fine-tune these rules over time.

Later in this post I have advice on writing posts/comments that tend to be well received on LessWrong. If you've been downvoted and/or rate-limited, don't take it too hard. LessWrong has fairly p...
Catastrophic Risks from AI #3: AI Race
by Dan H — 2023-06-23T19:21:07.335Z
Link: https://arxiv.org/abs/2306.12001

This is the third post in a sequence of posts giving an overview of catastrophic AI risks.

3 AI Race

The immense potential of AIs has created competitive pressures among global players contending for power and influence. This “AI race” is driven by nations and corporations who feel they must rapidly build and deploy AIs to secure their positions and survive. By failing to properly prioritize global risks, this dynamic makes it more likely that AI development will produce dangerous outcomes. Analogous to the nuclear arms race during the Cold War, participation in an AI race may serve individual short-term interests, but it ultimately results in worse collective outcomes for humanity. Importantly, these risks stem not only from the intrinsic nature of AI technology, but from the competitive pressures that encourage insidious choices in AI development.

In this section, we first explore the military AI arms race and the corporate AI race, where nation-states and corporations are forced to rapidly develop and adopt AI systems to remain competitive. Moving beyond these specific races, we reconceptualize competitive pressures as part of a broader evolutionary process in which AIs could become increasingly pervasive, powerful, and entrenched in society. Finally, we highlight potential strategies and policy suggestions to mitigate the risks created by an AI race and ensure the safe development of AIs.

3.1 Military AI Arms Race

The development of AIs for military applications is swiftly paving the way for a new era in military technology, with potential consequences rivaling those of gunpowder and nuclear arms in what has been described as the “third revolution in warfare." The weaponization of AI presents numerous challenges, such as the potential for more destructive wars, the possibility of accidental usage or loss of control, and the prospect of malicious actors co-opting these technologies for their own purposes. As AIs gain influence over traditional military weaponry and increasingly take on command and control functions, humanity faces a paradigm shift in warfare. In this context, we will discuss the latent risks and implications of this AI arms race on global security, the potential for intensified conflicts, and the dire outcomes that could come as a result, including the possibility of conflicts escalating to a scale that poses an existential threat.

3.1.1 Lethal

...
Write the Worst Post on LessWrong!
by Johannes C. Mayer — 2023-06-23T19:17:56.049Z

[Edit 2024-04-06]: Thank you Nicky for turning this post into a song. I think it's quite good. Now I can finally upvote this post. The lyrics can be found in the "The Worst Post on LessWrong" heading of this post.

Did you ever find yourself be afraid of publishing something, in a way that seems unhelpful? If so, then this post is for you.

Quoting myself:

You don't want to push away readers by writing bad posts, and you do not want them to update towards you being dumb.

Even if somebody like that publishes something occasionally, they might hamper themself, by being overly selective on what they write. The best way to get better at writing is to write a lot after all.

I think I managed to successfully work around this problem. In the beginning, I intentionally published something very bad on LessWrong. It wasn't optimized to be bad, but it was a post about a random low-quality thought I had.

It was so bad that the post the alt account I used to post that post was deleted (at least I can't find the post anymore). I only found out later, though (I am not sure if this had worked if it got deleted immediately). After I had put out that post, almost all of my fear of publishing went away. I published something very bad, and yet nothing terrible happened. My internal state was something like, "I am fine. Nothing terrible happens when you post something bad. I think I can do this again. But better." After all, this first truly abysmal post set the baseline very, very low.

I did something similar in the alignment Forum. Though that article was a lot better than my first article, it definitely falls short of my unrealistic expectations at the time. That article is not very good, and not very important in itself. But writing it, I can actually write articles like that. There is no major obstacle that prevents me from writing more and doing it better.

So I encourage you to follow a similar strategy if you find yourself being afraid of publishing. Don't optimize for making the content bad. I recommend setting a timer of 1 hour and committing to publish whatever you have after that hour.

For good measure, the rest of this post will be random garbage (generated only by me and not AI). If you want, take this as your baseline. Beat me to it, and write a post that is better than this:

The Worst Post on LessWrong

Can you eat a rock? The answer is yes. You just need to be smart abo...