trentmkelly/LessWrong-43k huggingface dataset:
Deciding What Project/Org to Start: A Guide to Prioritization Research
by Alexandra Bos — 2024-01-30T18:13:59.184Z

If you’re deciding what (research) project, organization or intervention to go for, analyzing your options through prioritization research can be invaluable. I used it to settle on founding

Catalyze

, an AI Safety field-building non-profit. In this post, I will share my blueprint and learnings from this process. Please note that you don’t need to be a researcher to benefit from conducting prioritization research.

Why prioritization research

No need to wait for inspiration to strike

If you're at a crossroads trying to come up with a great idea, I have good news: you don't have to invent something new. There's a world of ideas waiting to be discovered and executed. So don't wait for that idea to come to you; instead, you can go to the ideas.

You can probably find a better intervention than the first one(s) you stumbled upon

Suppose you have a handful of ideas for a project or an organization. That’s a fantastic start! But consider the possibility that by conducting a broad and deliberate search, you could stumble upon something even more impactful. More likely than not, if you consider a wide range of ideas, your initial favorite won’t come out on top. Don’t go for the first thing that came to mind, but rather take some time to scope the field and possibly find an even better option.

The difference in impact between a ‘good’ and ‘great’ intervention is huge - especially within a high-impact cause area

One of the original points EA focused on was the huge difference in impact between different interventions within global health.

Illustrative Graph from this

80.000 Hours article

To me, it seems likely that the big difference in impact between interventions is similar in different cause areas.

Therefore, apart from wisely choosing what cause to work on, I think it is still very crucial to wisely pick what you work on within that cause area. One of the ‘best’ interventions in this area is likely many times more impactful than the median.

If you work within

a cause area that is especially high-impact

, then finding one of the very best things to do could be especially consequential. Within such a cause area, the median intervention might already make a very big difference. Don’t let this be encouragement to settle - instead let this be encouragement to find an outlier which is 10x as impactful as this already-great median intervention....
on neodymium magnets
by bhauth — 2024-01-30T15:58:24.088Z
Link: https://www.bhauth.com/blog/materials/neodymium%20magnets.html

Neodymium magnets are the main type used in modern motors. Why are they good? Are there any good alternatives?

review of ferromagnetism

Magnetic fields contain energy. In an inductor, that energy comes from an increase in voltage when current is first applied. When a magnetic core is added to an inductor and a stronger field is produced from the same added energy, that extra energy has to come from somewhere.

The energy that ferromagnetic cores add to magnetic fields comes from their crystal structure fitting together better in a magnetic field. This implies that ferromagnetic cores should spontaneously magnetize to some extent, and they actually do; it's just that the spontaneously generated magnetic fields are curled into microscopic 3d loops. The microscopic internal field strength is approximately the saturation field of a ferromagnetic material, which is often greater than the field generated by a Nd magnet. Applying an external magnetic field causes those microscopic magnetic loops to partly unroll.

The actual field is generated by unpaired electrons of atoms; individual electrons are very magnetic. But ferromagnetism isn't a property of atoms, it's a property of crystals; without particular crystal structures that favor magnetic fields, those unpaired electron spins of iron atoms would just cancel out. For example, stainless steels contain a lot of iron, but most aren't ferromagnetic.

Atoms of crystals fitting together better in a magnetic field implies that iron cores slightly change shape when a magnetic field is applied. This effect is responsible for the humming noise transformers make, and has been used for eg sonar.

common misconceptions

Fucking magnets, how do they work?
And I don't wanna talk to a scientist.
Y'all motherfuckers lying, and getting me pissed.

— Insane Clown Posse

The Insane Clown Posse is sort of right there: a lot of explanations of magnets given to people by teachers and media scientist-figures have been partly wrong.

Magnetic flux was originally thought to be a flow of something like electric current, with ferromagnetic materials having lower resistance for that flow than air. It's even still taught that way sometimes. But no, it's a complex emergent phenomenon.

I remember being taught that "iron is magnetic because it has an unpaired electron". But again, ferromagnetism is a property of crystal structures, not atoms or elements....
Can we create self-improving AIs that perfect their own ethics?
by Gabi QUENE — 2024-01-30T14:45:58.246Z

If the process of self-improving AIs like described in an simple article by Tim Urban (below) is mastered, then the AI alignment problem is solved: "The idea is that we’d build a computer whose two-THREE major skills would be doing research on AI, ON ETHICS, and coding changes into itself—allowing it to not only learn but to improve its own architecture. We’d teach computers to be computer scientists so they could bootstrap their own development. And that would be their main job—figuring out how to make themselves smarter and ALIGNED"

In caps: parts to add for alignment

Link to the article: https://waitbutwhy.com/2015/01/artificial-intelligence-revolution-1.html
Childhood and Education Roundup #4
by Zvi — 2024-01-30T13:50:06.033Z

Before we begin, I will note that I have indeed written various thoughts about the three college presidents that appeared before Congress and the resulting controversies, including the disputes regarding plagiarism. However I have excluded them from this post.

Discipline and Prohibitions

Washington Post Editorial Board says schools should ban smartphones, and parents should help make this happen rather than more often opposing such bans in order to make logistical coordination easier.

I agree with the editorial board. Even when not in use, having a phone in one’s pocket is a continuous distraction. The ability to use the phone creates immense social and other pressures to use it, or think about using it, continuously. If we are going to keep doing this physically required school thing at all, students need to be fully device-free during the school day except for where we intentionally want them to have access. Having a phone on standby won’t work.

The Netherlands is going to try it for January 2024, including all electronic devices.

Jonathan Haidt, man with a message, highlights Vertex Partnership Academies, which locks all student electronic devices away all day and claims this is a big win all around. They say even the kids appreciate it. With phones available, other kids know you have the option to be on your phone and on social media, so you pay a social price if you do not allow constant distraction. Whereas with phones physically locked away, you can’t do anything during school hours, so your failure to do so goes unpunished.

Some old school straight talk from David Sedaris. He is wrong, also he is not wrong. He is funny, also he is very funny.

This explanation is one more thing that, as much as I hate actually writing without capital letters, makes me more positive on Altman:

Sam Altman: mildly interesting observation:

i always use capital letters when writing by hand, but usually only type them when doing something that somehow reminds me of being in school.

And of course, your periodic reminder department:

Alyssa Vance: In California, it is legally rape for two high school seniors to have consensual sex with each other. This is dumb, and people should be allowed to say it’s dumb without being accused of coddling rapists.

I do not pretend to know exactly what the right rules are, but this is not it. If there is no substantial age ...
Last call for submissions for TAIS 2024!
by Blaine — 2024-01-30T12:08:11.240Z
Link: https://tais2024.cc/

We are very close to the submission deadline for

TAIS 2024

. Alignment research moves fast; if there's something you've been working on the for last few months that you think would be interesting to our attendees, please submit! You can find a draft agenda on our website.

Timeline

2024-01-31: Submission deadline
2024-02-29: Notification of acceptance / rejection
2024-04-05: Conference!

Details

The Technical AI Safety Conference will bring together specialists in the field of technical safety to share their research and benefit from each others’ expertise. We want our attendees to involve themselves in deep conversations throughout the conference with the brightest minds in AI Safety.

We seek to launch this forum for academics, researchers and professionals who are doing technical work in these or adjacent fields:

Mechanistic interpretability
Scalable oversight
Causal incentive analysis
Agent foundations
Singular learning theory
Argumentation
Emergent agentic phenomena
Thermodynamic / statistical-mechanical analyses of computational systems

We invite the submission of research papers, or extended abstracts, that deal with related topics. We particularly encourage submissions from researchers in the ALIFE community who would not otherwise have considered submitting to TAIS 2024.

Our sponsors, Noeon Research, have kindly provided funding to cover travel and accommodation for authors of accepted presentations.

To submit, please send a title and abstract to blaine@aisafety.tokyo.
Has anyone actually changed their mind regarding Sleeping Beauty problem?
by Ape in the coat — 2024-01-30T08:34:43.904Z

https://en.wikipedia.org/wiki/Sleeping_Beauty_problem

If that happened what was the argument that did it for you?

I'm interested in situations where a person used to think that the correct answer was 1/2 and then, on a reflection, decided that it's actually 1/3 , or vice versa, not when the resulting belief is that the question is meaningless or both answers are valid.
The case for more ambitious language model evals
by Jozdien — 2024-01-30T00:01:13.876Z

Here are some capabilities that I expect to be pretty hard to discover using an RLHF’d chat LLM[1]:

Eric Drexler tried to use the GPT-4 base model as a writing assistant, and it [...] knew who he was from what he was writing. He tried to simulate a conversation to have the AI help him with some writing he was working on, and the AI simulacrum repeatedly insisted it was by Drexler.
A somewhat well-known Haskell programmer - let’s call her Alice - wrote two draft paragraphs of a blog post she wanted to write, began prompting the base model with it, and after about two iterations it generated a link to her draft blog post repo with her name.

More generally, this is a cluster of capabilities that could be described as language models inferring a surprising amount about the data-generation process that produced its prompt, such as the identity, personality, intentions, or history of a user[2].

The reason I expect most capability evals people currently run on language models to miss out on most abilities like these is primarily that they’re most naturally observed when dealing with much more open-ended contexts. For instance, continuing text as the user, predicting an assistant free to do things that could superficially look like hallucinations[3], and so on. Most evaluation mechanisms people use today involve testing the ability of fine-tuned[4] models to perform a broad array number of specified tasks in some specified contexts, with or without some scaffolding - a setting that doesn’t lend itself very well toward the kind of contexts I describe above.

A pretty reasonable question to ask at this point is why it matters at all whether we can detect these capabilities. A position one could have here is that there are capabilities much more salient to various takeover scenarios that are more useful to try and detect, such as the ability to phish people, hack into secure accounts, or fine-tune other models. From that perspective, evals trying to identify capabilities like these are just far less important. Another pretty reasonable position is that these particular instances of capabilities just don’t seem very impressive, and are basically what you would expect out of language models.

My response to the first would be that I think it’s important to ask what we’re actually trying to achieve with our model eval mechanisms. Broadly, I think there are two different...
A short 'derivation' of Watanabe's Free Energy Formula
by Wuschel Schulz — 2024-01-29T23:41:44.203Z

Epistemic status: I wrote the post mostly for myself, in the process of understanding the theory behind Singular Learning Theory, based on the SLT low lecture series. This is a proof sketch with the thoroughness level of an experimental physics lecture: enough to get the intuitions but consult someone else for details.

Background: Statistical Learning Theory

In the framework of statistical learning theory, we aim to establish a model that can predict an outcome y given an input x with a certain level of confidence. This prediction is typically governed by a set of parameters w , which need to be learned from the data. Let us begin by defining the primary components of this framework.

The Probabilistic Model

The probabilistic model p(y|x,w) represents the likelihood of observing the outcome y given the input x and the parameters w. This model is parametric, meaning that the form of the probability distribution is predefined, and the specific behavior of the model is determined by the parameters w.

The True Distribution

The true distribution q(y|x) is the actual distribution from which the data points are sampled. In practice, this distribution is not known and p(y|x,w) attempts to approximate it through the learning process.

The Parameters

The parameters w are the weights or coefficients within our model that determine its behavior. The goal of learning is to find the values of w that make p(y|x,w) as close as possible to the true distribution q(y|x).

The Data

The data D={xi,yi} consist of pairs of input values xi and their corresponding outcomes yi. These data points are used to train the model, adjusting the parameters w to fit the observed outcomes.

The Negative Log-Likelihood

The negative log-likelihood is a measure of how well our model p(y|x,w) fits the data D . It is defined as the negative sum of the logarithms of the model probabilities assigned to the true outcomes yi:

NLL(w)=−∑Ni=1logp(yi|xi,w),

where N is the number of data points in D. Minimizing the negative log-likelihood is equivalent to maximizing the likelihood of the data under the model, which is a central objective in the training of probabilistic models.

Prior over the Parameters

In a Bayesian framew...
How important is AI hacking as LLMs advance?      
by Artyom Karpov — 2024-01-29T18:41:02.767Z

TL;DR. Offensive technologies will likely become more advanced while the defense ones will lag behind. AI hacking capabilities might be the second most important problem after AI alignment.

Epistemic status: Exploratory / brainstorming.

How language models will affect the offensive/defensive symmetry

I imagine that if language models will continue to be improved, this symmetry won’t be stable but fluctuate like a highly volatile stock market. That might lead to more fragmentation of the web (because state actors might want to protect themselves from more advanced ones as we see it in China, Russia, North Korea, Iran), widespread use of cryptography, 2F or/and biometrical authentication (like Worldcoin), abandoning older versions of protocols or canceling them at all (e.g. HTTP, SMTP) in favor of more secure ones. Overall, technologies would be less and less easy to use, see “Nova DasSarma on Why Information Security May Be Critical to the Safe Development of AI Systems' (

link

). If models improve further or become able to self-improve, this might lead to global blackout of financial, communication, military and other systems with large consequences like human extinction. However, it looks like there will be some slow down in model releases or public access to weights (LLAMA v2 70B is the last of publicly available large models perhaps).

Perhaps there will be improvements in encryption and decryption algorithms using the help of neural networks like we saw for matrix multiplication,

Fawzi et al. (2022)

. However, the spread of these won’t be instant. Users don’t update their software regularly, they run old versions for as long as recent computer viruses showed: WannaCry Ransomware that impacted thousands of computers world wide,

“Shadow Brokers Threaten to Release Windows 10 Hacking Tools”

(2017).

It looks like the open source software will be less and less spread as the bottleneck might be lazy users that don’t want to update their software. So their software won’t be in reality theirs but more and more downloaded, updated, installed, uninstalled by third companies. Or it will be only services with terminals for end users. How the open source software might be used more is through global cooperation to improve it (public good) like in Linux distributions that have rolling updates which means all software updates on a regular basis, no individ...
LLM Psychometrics: A Speculative Approach to AI Safety
by pskl — 2024-01-29T18:38:19.522Z
Link: https://pascal.cc/blog/artificial-psychometrics

Hello, this is the preliminary report of an art project I've worked on in 2023 which posits that psychometrics may offer valuable ideas for "AI alignment".
On a scale of 1 to 10 how unlikely is this "scenario" and why?
How to write better?
by TeaTieAndHat — 2024-01-29T17:02:55.942Z

As you will probably have noticed before the end of this question, I’m a relatively mediocre writer. I mean, I’m not that bad, I know a lot of people who are worse at it than I am, but I still often notice a pattern in basically everything I write: exceedingly long and complex sentences, giving masses of detail with little apparent regard for how much information the person at the other end actually needs, "stick-on" weird metaphors which appear randomly every time I’m afraid I’m being too technical or annoying (so you get a wall of annoying text with a bit of canned laughter in the middle…), vague sentences that go around for a while as I’m slowly figuring out what I mean to say, long paragraphs, etc. Also, I spend ages proofreading anything I write and worrying about it…

Anyway, I’d like to get better at it. And LW is full of good writers, so surely someone will have advice?

I know the standard piece of advice is "write more stuff", or maybe "read The Sense of Style, or something". And I’ve done both these things, and they’ve helped, but I’m still a pretty mediocre writer. I’m still writing somewhat more than the average person, but if I want to go beyond that and practice even more, it will have to mean a deliberate effort to improve my writing. And if I decide to deliberately improve my writing using advice that’s not more specific than "write more stuff, and then some more again", my efforts will soon fall into that deep endless cave where people drop their New Year resolutions every 15th January…

So, any ideas?
Processor clock speeds are not how fast AIs think
by Ege Erdil — 2024-01-29T14:39:38.050Z

I often encounter some confusion about whether the fact that synapses in the brain typically fire at frequencies of 1-100 Hz while the clock frequency of a state-of-the-art GPU is on the order of 1 GHz means that AIs think "many orders of magnitude faster" than humans. In this short post, I'll argue that this way of thinking about "cognitive speed" is quite misleading.

The clock speed of a GPU is indeed meaningful: there is a clock inside the GPU that provides some signal that's periodic at a frequency of ~ 1 GHz. However, the corresponding period of ~ 1 nanosecond does not correspond to the timescale of any useful computations done by the GPU. For instance; in the A100 any read/write access into the L1 cache happens every ~ 30 clock cycles and this number goes up to 200-350 clock cycles for the L2 cache. The result of these latencies adding up along with other sources of delay such as kernel setup overhead etc. means that there is a latency of around ~ 4.5 microseconds for an A100 operating at the boosted clock speed of 1.41 GHz to be able to perform any matrix multiplication at all:

The timescale for a single matrix multiplication gets longer if we also demand that the matrix multiplication achieves something close to the peak FLOP/s performance reported in the GPU datasheet. In the plot above, it can be seen that a matrix multiplication achieving good hardware utilization can't take shorter than ~ 100 microseconds or so.

On top of this, state-of-the-art machine learning models today consist of chaining many matrix multiplications and nonlinearities in a row. For example, a typical language model could have on the order of ~ 100 layers with each layer containing at least 2 serial matrix multiplications for the feedforward layers[1]. If these were the only places where a forward pass incurred time delays, we would obtain the result that a sequential forward pass cannot occur faster than (100 microseconds/matmul) * (200 matmuls) = 20 ms or so. At this speed, we could generate 50 sequential tokens per second, which is not too far from human reading speed. This is why you haven't seen LLMs being serviced at per token latencies that are much faster than this.

We can, of course, process many requests at once in these 20 milliseconds: the bound is not that we can generate only 50 tokens per second, but that we can generate only 50 sequential tokens per second, meanin...