trentmkelly/LessWrong-43k huggingface dataset:
Biden should be applauded for appointing Renee Wegrzyn for ARPA-H
by ChristianKl — 2022-09-18T19:57:31.209Z

When Biden first announced ARPA-H (Advanced Research Projects Agency for Health) I was

skeptical

. At the time the proposal was to make ARPA-H a normal agency within the NIH. When critics argued that the NIH bureaucracy would likely reduce the amount of innovation that ARPA-H could produce the administration listened and US health secretary Xavier Becerra

decided

to make ARPA-H an independent entity within the NIH.

Appointing Renee Wegrzyn as the head of ARPA-H is an excellent choice. Having previously worked at DARPA and IARPA and not in the NIH makes her a good person to copy the structure of how DARPA and IARPA get things done.

Being 45, she is twenty years younger than the average director of an institute at the NIH who is

65

. When science sometimes evolves from gravestone to gravestone, it’s great to see a director for the ARPA-H who’s in her forties. Especially, in contrast to the newly appointed NIH director Lawrence A. Tabak who is a 71-year-old (a fact that his NIH biography omits) having a 45-year-old lead ARPA-H is a welcome surprise.

While at DARPA, Wegrzyn worked on biosecurity and even gave a talk about the

importance of biosecurity

to the Long Now Foundation. It’s valuable to have people who care about biosecurity at the top of organizations that direct a lot of research dollars.

A lot of the research focuses either on directly curing diseases or on basic research but the research that focuses on producing tools for better research is historically underfunded. Wegrzyn experience as vice president of business development at Gingko Bioworks which is a unicorn biotech company that historically doesn’t focus on curing diseases directly but providing tools to help other companies, gives me hope that ARPA-H will be able to fund research into producing better tools to deal with biological problems. During COVID-19 they worked on producing COVID tests which was likely a lesson in the flaws of government bureaucracy holding back innovation.

In an

interview

she described her own research as being guided by the question of ‘what do we know about protein folding that can help us develop a totally new type of diagnostic?’ I’m happy to see that and think it will lead to better research paths than having someone who focuses more on creating FDA approved drugs in charge of allocating research funding.

She coedited Alzheimer’s...
Summaries: Alignment Fundamentals Curriculum
by Leon Lang — 2022-09-18T13:08:05.335Z
Link: https://docs.google.com/document/d/1mVwQgyrEgWJ9xsO6wIizDp2T3578OqNe4K5goKiOLgE/edit?usp=sharing

The linked document provides my summaries for most core readings and many further readings of

the alignment fundamentals curriculum composed by Richard Ngo

, as accessed from July to early September 2022. Additionally, it often contains my preliminary opinions on the texts. Note that I’m not an expert on the topic.

I have read all texts while simultaneously doing full-time work unrelated to AI alignment, and thus, due to time constraints, many summaries probably contain mistakes, and my opinions would change upon further reflection. Additionally:

I only streamlined the process after a few weeks
the summaries of the first weeks are of lower quality, and more of them or my opinions are missing
Some summaries are also missing since I had a minor repetitive strain issue along the way, and since the curriculum changed while reading through it
Sometimes, the formatting is not ideal since I originally wrote the summaries on a slack channel and then copy-pasted them to google docs

Nevertheless, I was told that these summaries are useful, and therefore I’m sharing them with the wider community of people interested in alignment.

If anyone wants to contribute their own summary, please put a suggestion into the google doc, and I will accept it with an attribution to the (optionally anonymous) author.

Acknowledgments: I want to thank Albert Garde, Benjamin Kolb, Fritz Dorn, Jens Brandt, and Tom Lieberum for discussions on the curriculum.
Inner alignment: what are we pointing at?
by lemonhope — 2022-09-18T11:09:58.661Z

Proof that a model is an optimizer says very little about the model. I do not know what a research group is studying outer alignment is studying. Inner alignment seems to cover the entire problem at the limit. Whether an optimizer is mesa or not depends on your point of view. These terms seem to be a magnet for confusion and debate. I have to do background reading on someone to even understand what claim they're making. These are all indicators that we are using the wrong terms.

What are we actually pointing at? What questions do we want answered?

Do we care if a model is an optimizer? Is it important whether it is creating plans through an explicit search process or a clever collection of heuristics? A poor search algorithm cannot plan much and clever enough heuristics can take you to any goal. What's the important metric?

Sometimes a model will have great capacity to shape its environment but little inclination. How to divide between capacity and inclination in a way that closely corresponds to agents and models as we observe them? (One could say that capacity and inclination cannot be separated but the right definitions would split them right apart.)

When you specify what you want the model to do in code, what is the central difficulty? Is there a common risk or error in giving examples and giving loss/reward/value functions that we can name?

Is there a clear, accepted term for when models do not maintain desired behavior under distribution shift?

Should we distinguish between trained RL models that optimize and spontaneous agents that emerge in dynamical systems? One might expect the first to almost always happen and the second very rarely. What's the key difference?

I'll post my answers to these questions in a couple days but I'm curious how other people slice it. Does "inner alignment failure" mean anything or do we need to point more directly?
Podcasts on surveys, slower AI, AI arguments, etc
by KatjaGrace — 2022-09-18T07:30:15.846Z

I recently talked to Michael Trazzi for his podcast, The Inside View. It just came out, so if that’s a conversation you want to sit in on, do so here:

The main topics were the survey of ML folk I recently ran, and my thoughts on moving more slowly on potentially world-threatening AI research (which is to say, AI research in general, according to the median surveyed ML researcher…). I also bet him a thousand dollars to his hundred that AI would not make blogging way more efficient in two years, if I recall. (I forget the exact terms, and there’s no way I’m listening to myself talk for that long to find out. If anyone else learns, I’m curious what I agreed to.)

For completeness of podcast reporting: I forgot to mention that I also talked to Daniel Filan on AXRP, like a year ago. In other old news, I am opposed to the vibe of time-sensitivity often implicit in the public conversation.
There is no royal road to alignment
by Eleni Angelou — 2022-09-18T03:33:50.720Z

Crossposted from the EA Forum: https://forum.effectivealtruism.org/posts/W4MfMmbwccDCRBfnj/there-is-no-royal-road-to-alignment

Epistemic status: tentative thoughts I've been processing for a while now and discussing mostly with philosophy folks.

Despite working in philosophy of science for many years now, I've found that (my) epistemological insights are quite rare, if any. Most recently, the question I've been thinking about and asking everyone I meet over and over is how do you speed up scientific research? The most apt answer seems to be the most trivial one: do more empirical work; construct more models; test your models.

Epistemology is hard. Not only does it require a deep understanding of how the practice of science happens, but it also looks like epistemological insights don't come often and the ones that do, aren't exactly groundbreakingly helpful. Why is that? I'd tentatively argue that it's because of the limited nature of our cognition and the need to have a lot of history of science, i.e., that we need many examples to confidently infer a pattern. At the same time, it seems that what belongs purely to the cognition aspect of the discussion, is entangled with a variety of social factors which make phenomena such as scientific discoveries difficult to interpret solely as products of cognitive excellence. And so, it becomes very difficult to make bold claims like the students of Nobel prize laureates tend to be Nobel prize laureates themselves.

My current epistemological insight (that is not going to help anybody as far as I understand) is that while alignment might look like it presupposes a "weird epistemology" of solving a problem before it even makes its appearance, it is probably not as peculiar a case as we consider it to be with our present intellectual tools, where "we"= all of us working in AI safety, broadly speaking.

When you're inside a problem, you tend to have a first-person perspective of it which makes you think that this must be a unique situation. The analogy applies to the feeling of uniqueness we all experience when it comes to the idea of the "self" and our personal identity. But just like this feeling is challenged once other people share their experiences with us, the uniqueness of alignment might be challenged when juxtaposed with the episodes from the history of science.

The claim I often...
Updates on FLI's Value Aligment Map?
by T431 — 2022-09-17T22:27:50.740Z

Around January 2021, I came across the Future of Life Institute's Value Alignment Map, and found it to be a useful resource for getting a sense of the different alignment topics. I was going through it again today, and recalled that I have not seen many mentions of this map in the wider LW/EAF/Alignment community, and have also not seen any external commentary on its comprehensiveness. Additionally, the corresponding document for the map indicates that it was last edited in January 2017; I haven't been able to find an updated version on LW or FLI's website, and spent about half an hour searching. With this in mind, I have a few questions:

Does anyone know of an external review of the comprehensiveness and accuracy of the topics covered in this map?
Does anyone know if there are plans to update it OR, if it has been updated, where this newer version can be found?
Does anyone know if there are similar maps of value alignment research / AI safety research topics in terms of comprehensiveness?
Most sensible abstraction & feature set for a systems language?
by Jasen Qin — 2022-09-17T19:49:09.755Z

Edit: Made title seem more like a question as intended and tried to clarify on some points Ive made and provided more context on my motivations

Motivations

Systems programming is not well defined. But for the purposes of this post I define it to mean "writing software to be used as core, performant and reliable modules for other software". A systems software would not be used to provide business value directly, but rather be part of a network of modules eventually used in an application software that does provide business value. If a systems software dies, so could the user. If an application software dies, it could be rebootable and fixed by IT professionals. Every extra millisecond a systems software spends on doing its job, the application software using it spends an extra second. Thats $1 million lost.

Current systems languages like C, C++, Rust do their job in the systems programming space and have been thoroughly optimised throughout the years. Unfortunately, problems with ergonomics, complexity, and the tendency for bad code and shoddy solutions is still amiss.

Newer languages like rust have tackled many of the above issues, leading to a language that is most loved by devs (Stack Overflow 2020-2022) by far, compared to the next languages on the list. Unfortunately critics often cite rust's complexity with its lifetimes and borrowing system as well as parts of its syntax. I happen to think they're a bit verbose sometimes and not as easily to digest in one skim through. Rust's toolchain has definitely improved the overall development experience, simplifying the retrieval, management and usage of external libraries as well as providing a uniform(ish) interface to customise and adapt to your use case. The language server protocol was also a step in the right direction, allowing extra productivity if your language can properly utilise its full potential. I think rust-analyzer is among the best language servers available, with rust's strong static checks, leading to common errors being found as you type so you don't have to manually compile -> check for errors -> check your code -> retry.

However, despite the promising advances, I still think we're not quite at the mark of something that "just works" and provides an optimal balance of performance, ergonomics, productivity, scalability, and intuitiveness. I think this is mainly due to the core idea of a ...
Sparse trinary weighted RNNs as a path to better language model interpretability
by Am8ryllis — 2022-09-17T19:48:24.330Z

Epistemic status: Strongly arguing for what I feel is a neglected approach. May somewhat overstate the case and fail to adequately steelman counter arguments. I hope and expect that readers will point out flaws in my logic.

Introduction

Currently, large transformers with dense floating point weights are state of the art in language modeling. Despite recent progress by Anthropic and others, they remain difficult to understand.

Why are they hard to understand?

Lack of native ordering structure: setting aside attention masking, transformers have no native concept of token ordering. They likewise have no well defined state between tokens. Transformers are not truly sequence models, even with positional encoding, but text is sequential[1].
Large magnitude, continuous weights: although weights sometimes behave somewhat like boolean logic, we cannot easily treat them as such.
Dense matrices: within a matrix of nonzero weights, all outputs are affected by all inputs to a nonzero extent. It is difficult to know which connections are important. Information leaks in difficult-to-track ways.

We can fix these issues.

RNN[2]s have well-defined state. They process tokens in order. We can feed a token and observe how the state is or is not changed. This is easier to visualize.
Weights can be quantized to trinary and activations to binary.
Weights can be extremely sparsified. (Which is essentially trinary quantization with a preference for 0 weights.)

Once we quantize our RNN to trinary weights we can replace addition and tanh activations with adder trees and digital comparators and apply boolean logic simplification tools to further simplify the logic. Now we have a fully combinatorial logic finite state machine[3]. Understanding the behavior of such a machine intuitively feels far more tractable than understanding current transformers. Once we can understand the internal workings of large language models, we can likely use this understanding to improve safety/alignment.

Objections

We cannot train, quantize and sparsify RNNs of comparable accuracy to modern transformers.

RNNs are less powerful than transformers, hard to train on GPUs, and quantization kills accuracy. What good is it if we can understand small toy language models? They are not what is being deployed. If it costs 10x more to train, few will actually use it and SoTA will stay uninterpretable.

A: We use transformers ins...
Apply for mentorship in AI Safety field-building
by Orpheus16 — 2022-09-17T19:06:12.753Z
Link: https://forum.effectivealtruism.org/posts/AKDWwtGbuBirRbxeE/apply-for-mentorship-in-ai-safety-field-building

TLDR: I often meet AI safety field-builders who are looking for advice or mentorship. I also know some who have (limited) capacity to give advice or mentorship.

Fill out this short form if you are interested in mentorship, or email me at akashwasil133[at]gmail[dot]com.

Background

I've recently had some 1:1s with university group organizers and other field-builders who are interested in finding and supporting people to work on AI safety.

Some people have expressed that it would be helpful for them to know where they can get advice (either on specific projects or holistically on their plans/theories of change)
Some people (especially from outside EA Hubs) often aren't familiar with some organizations/funding opportunities/"common wisdom" that people in EA Hubs often take for granted.
Some people are nervous about reaching out directly to field-builders who they (often correctly) perceive as very busy.

What I can do

If you are planning on doing field-building with an AI safety focus, please reach out to me. Specific things I can do:

Understand your plans and try to find ways to make them better (more ambitious, faster, etc.)
Offer advice
Connect you with opportunities/people that you might not know about
See if you're interested in working on any of the projects that I've been considering.

What to do if you're interested

Fill out this form or send me an email at akashwasil133[at]gmail[dot]com.

Miscellaneous

Some people might be good fits for other programs (e.g., AISFB Hub). In these cases, I'm happy to make introductions.
One reason I think this hasn't happened is that few people feel qualified to do this (this isn't the main reason-- I think the main reason is that people are too busy, and there's a lot that needs to be done). I don't claim to be the best person for this; I'm just the one who's doing it. (See also Lifeguards and Hero Licensing)
I expect that I will mostly be corresponding with people over email. In some exceptional cases (e.g., if someone is launching a particularly ambitious project or is interested in one of the projects that me or someone I know is interested in), the mentorship might involve calls (with me or other AIS field-builders I know).
Refine's Third Blog Post Day/Week
by adamShimi — 2022-09-17T17:03:15.472Z

Yesterday was the third blog post day at Refine (technically it was announced on Monday for Friday, but people leveraged the week differently so it's in between a week and a day).

In the current phase of the incubator we're mostly pushing them to generate and iterate on ideas, with less focus on legibility. This blog post day/week was mainly for pushing them to reach for some feedback, but the program will focus a lot more on legibility in October.

Here is the list of blog posts by Refine participants:

ordering capability thresholds
Refine Blogpost Day #3: The shortforms I did write
Levels of goals and alignment
Representational Tethers: Tying AI Latents To Human Ones

One of the participants is currently trying out more of a research support role, and so didn't write a blog post.
[Closed] Prize and fast track to alignment research at ALTER
by Vanessa Kosoy — 2022-09-17T16:58:24.839Z

On behalf of

ALTER

, I am pleased to announce a prize of 50,000 USD, to be awarded for the best substantial contribution to the

learning-theoretic AI alignment research agenda

among those submitted before October 1, 2023. Depending on the quality of submissions, the winner(s) may be offered a position as a researcher in ALTER (similar to

this one

), to continue work on the agenda, if they so desire.

Submit by email. [Submission is closed]

Topics

The research topics eligible for the prize are:

Studying the mathematical properties of the algorithmic information-theoretic

definition of intelligence

.
Building and analyzing formal models of value learning based on the above.
Pursuing any of the

future research directions

listed in the article on infra-Bayesian physicalism.
Studying

infra-Bayesian logic

in general, and its applications to infra-Bayesian reinforcement learning in particular.
Theoretical study of the behavior of RL agents in

population games

. In particular, understand to what extent

infra-Bayesianism

helps to avoid the

grain-of-truth problem

.
Studying

the

conjectures

relating superrationality to thermodynamic Nash equilibria.
Studying the theoretical properties of the

infra-Bayesian Turing reinforcement learning

setting.
Developing a theory of reinforcement learning with

traps

, i.e. irreversible state transitions. Possible research directions include studying the computational complexity of Bayes-optimality for finite state policies (in order to avoid the

NP-hardness

for arbitrary policies) and

bootstrapping

from a safe baseline policy.

New topics might be added to this list over the year.

Requirements

The format of the submission can be either a LessWrong post/sequence or an arXiv paper.

The submission is allowed to have one or more authors. In the latter case, the authors will be considered for the prize as a team, and if they win, the prize money will be split between them either equally or according to their own internal agreement. For the submission to be eligible, its authors must not include:

Anyone employed or supported by ALTER.
Members of the board of directors of ALTER.
Members of the panel of the judges.
First-degree relatives or romantic partners of judges.

In order to win, the submission must be a substantial contribution to the mathematical theory of one of the topics...
Remote Login For Turnkey Devices?
by jefftk — 2022-09-17T15:40:03.435Z

I run my

rhythm stage setup

on a Raspberry Pi. Mostly this works
well: I plug it in, my code and init scripts bring everything up in
the right order, in about a minute it's taking in audio and MIDI and
putting out audio. If it doesn't work I turn it off and on again and
then it does.

As I'm about to head off to play a dance (hi NYC!) with a heavily
refactored system that will very likely but not certainly work
properly, I'm thinking about what I might do if it needs tweaking.
I've had one issue so far, where an SD card stopped working, and now I
travel with a spare
prepared SD card. For other potential breakages I have spare
equipment where I could swap things around, but only if I make code
changes. Writing the code to be smart enough to understand every way
I might repatch it in response to failure would be possible, but a
huge amount of work.

When I'm home playing with things this is easy: the Pi is on my
network, I ssh in. But what about when I'm at a gig? I want an easy
way to log in. What are my options?

Keyboard (usb) and monitor (HDMI). This will definitely work,
but since I wouldn't want to travel with a monitor I'd be reliant on
finding someone who was willing to let me use theirs. And in a "get
to the hall early, set up, something's broken, need to tweak" or a
"dance weekend in the woods" scenario there probably isn't anything.
People do make tiny monitors ($60),
generally marketed for Raspberry Pis, but that's a bit expensive and
with a keyboard is also bigger than ideal.

Ethernet. Ethernet adapter on my Mac, which I already have,
into ethernet on the Pi. You don't need a crossover cable for this
sort of thing anymore: they'll just detect automatically. The steps
aren't too bad:

System Preferences > Sharing > Internet Sharing > USB
10/100/1000 LAN.

Run ifconfig: Should see bridge100 listed, probably
says ip is 192.168.2.1.

Connect Pi to ethernet and power on.

ssh pi@192.168.2.2.

This would be ideal, except it only works if the Mac has WiFi. Which
is super frustrating, since I really would like a setup that works
even when there's no internet connection to share.

RS232 (serial). A USB-to-serial adapter on each end ($10,
male, $11,
female) and they should be able to talk RS232 to each other. You
can even get a single cable that does both ($18), which looks
like a forbidden USB-A to USB-A

...