trentmkelly/LessWrong-43k huggingface dataset:
Sequencing Intro II: Adapters
by jefftk — 2022-09-15T03:30:03.204Z

A couple weeks ago I wrote a short

sequencing intro. Here's a bit more,
starting with a minor puzzle. I'm working with
California wastewater sequencing data (

Rothman
et al 2021) and I found a read that was a partial match for HIV:

>SRR14530740.1578405 1578405/2
AGCTTGCCTTGAGTGCTTCAAGTAGTGTGTGCCCGTCTGTTGTGTGACTCT
GGTAACTAGAGATCCCTCAGACCCTTTTAGTCCTGTCTCTTATACACATCT
GACGCTGCCGACGACCTTCGTGATGTGTAGATCTCGGGGGGCGGCGGGG

The 83 highlighted bases at the beginning of the read are an exact
match for this section near the beginning of the HIV genome:

>AF033819.3 HIV-1, complete genome
GGTCTCTCTGGTTAGACCAGATCTGAGCCTGGGAGCTCTCTGGCTAACTA
GGGAACCCACTGCTTAAGCCTCAATAAAGCTTGCCTTGAGTGCTTCAAGT
AGTGTGTGCCCGTCTGTTGTGTGACTCTGGTAACTAGAGATCCCTCAGAC
CCTTTTAGTCAGTGTGGAAAATCTCTAGCAGTGGCGCCCGAACAGGGAC
...

This dataset was sequenced with paired-end reads, which means there's
more information we can get on this particular genetic snippet. This
was the 'reverse' read, so let's look at the corresponding forward
read:

>SRR14530740.1578405 1578405/1
GACTAAAAGGGTCTGAGGGATCTCTAGTTACCAGAGTCACACAACAGACGG
GCACACACTACTTGAAGCACTCAAGGCAAGCTCTGTCTCTTATACACATCT
CCGAGCCCACGAGACCTTGCCATTAATCTCGTATGCCGTCTTCTGCTTG

When working with paired-end reads, they're sequenced in opposite
directions working towards each other:

forward read -->
5' --------------------------------------------- 3'
|||||||||||||||||||||||||||||||||||||||||||||
3' --------------------------------------------- 5'
<-- reverse read

Because they're reading in different directions you need to reverse
one of the reads, and since they're reading complementary strands you
need to take the genetic complement. Here's the reverse complement of
the forward read, to match the reverse read we were already looking
at:

>SRR14530740.1578405 1578405/1, reverse complement
CAAGCAGAAGACGGCATACGAGATTAATGGCAAGGTCTCGTGGGCTCGGAG
ATGTGTATAAGAGACAGAGCTTGCCTTGAGTGCTTCAAGTAGTGTGTGCCC
GTCTGTTGTGTGACTCTGGTAACTAGAGATCCCTCAGACCCTTTTAGTC

Most commonly your paired end reads go together like:

[read 1] [gap you didn't sequence] [read 2]

In this case, however, they overlap, allowing us to assemble a larger
sequence. Sometimes you might have a read error, where the two don't
perfectly match, but we're lucky here and there's no disagreement...
How do I find tutors for obscure skills/subjects (i.e. fermi estimation tutors)
by joraine — 2022-09-15T01:15:56.166Z

I'd love to get really good at fermi estimations, and I feel like I'd get better faster if I paid someone to teach me, measure progress, and test me consistently.

I also want to be generally "mathematically mature" so that I'm not afraid of that being a prerequisite for a lot of classes/research. I don't have a specific section of math that I know I'm deficient in, but maybe someone else could talk to me and figure out where I'm lacking

I'd love a tutor for quant interview practice so I can earn-to-give later on etc, and the money to pay a tutor to guide me in the right direction will be vastly less than the compound of the money I'd make as a quant.

And this is made harder by being afraid to ask for help non-anonymously as I don't want to look dumb in front of my peers but that curses me to stay at my current level of intelligence forever. Any tips?
Forecasting thread: How does AI risk level vary based on timelines?
by elifland — 2022-09-14T23:56:30.341Z

Crossposted to EA Forum

While there have been many previous surveys asking about the chance of existential catastrophe from AI and/or AI timelines, none as far as I'm aware have asked about how the level of AI risk varies based on timelines. But this seems like an extremely important parameter for understanding the nature of AI risk and prioritizing between interventions.

Contribute your forecasts below. I'll write up my forecast rationales in an answer and encourage others to do the same.

Elicit Prediction (forecast.elicit.org/binary/questions/726u_r_XC)

Elicit Prediction (forecast.elicit.org/binary/questions/8HnkL7Ekf)

Elicit Prediction (forecast.elicit.org/binary/questions/dlPDzREFM)

Elicit Prediction (forecast.elicit.org/binary/questions/EHL2INyQq)

Elicit Prediction (forecast.elicit.org/binary/questions/AETDAUmNF)
Coordinate-Free Interpretability Theory
by johnswentworth — 2022-09-14T23:33:49.910Z

Some interpretability work assigns meaning to activations of individual neurons or small groups of neurons. Some interpretability work assigns meaning to directions in activation-space. These are two different ontologies through which to view a net’s internals. Probably neither is really the “right” ontology, and there is at least one other ontology which would strictly outperform both of them in terms of yielding accurate interpretable structure.

One of the core problems of interpretability (I would argue the core problem) is that we don’t know what the “right” internal ontology is for a net - which internal structures we should assign meaning to. The goal of this post is to ask what things we could possibly assign meaning to under a maximally-general ontology constraint: coordinate freedom.

What Does Coordinate Freedom Mean?

Let’s think of a net as a sequence of activation-states xi, with the layer i → layer i+1 function given by xi+1=fi(xi).

We could use some other coordinate system to represent each xi. For instance, we could use (high dimensional) polar coordinates, with ri=||xi|| and ϕi a high-dimensional angle (e.g. all but one entry of a unit vector). Or, we could apply some fixed rotation to xi, e.g. in an attempt to find a basis which makes things sparse. In general, in order to represent xi in some other coordinate system, we apply a reversible transformation x′i=gi(xi), where x′i is the representation under the new coordinate system. In order to use these new coordinates while keeping the net the same overall, we transform the layer transition functions:

f′i−1=gi∘fi−1

f′i=fi∘g−1i

In English: we transform into the new coordinate system when calculating the layer state xi, and undo that transformation when computing the next layer state xi+1. That way, the overall behavior remains the same while using new coordinates in the middle

The basic idea of coordinate freedom is that our interpretability tools should not depend on which coordinate system we use for any of the internal states. We should be able to transform any layer to any coordinate system, and our interpretability procedure should still assign the same meaning to the same (transformed) internal structures.

What Kind Of Coordinate Free Internal Structure Is Even Possible?

Here’s one example of a ...
Progress links and tweets, 2022-09-14
by jasoncrawford — 2022-09-14T23:21:45.271Z
Link: https://rootsofprogress.org/links-and-tweets-2022-09-14

Links

Low stone walls called “kites” are believed to have been used for hunting and may date to 8000 BC (h/t @SamoBurja)
Health insurers just published a trillion prices (h/t Marginal Revolution). There’s a startup to be built in making this data accessible and useful

Tweets & retweets

We are on the verge of eradicating extreme poverty, but the system that has brought this change is extremely unpopular (@JonSteinsson)
Star Trek portrays humanity as a beautiful, optimistic, and open race that that has our brightest days ahead (@rlorenc)
Singapore is launching a new program to recruit the world’s top scientific/technical talent (@calebwatney)
“We ran the experiment of email being a truly open P2P protocol… That experiment failed” (@patio11)
“I love and admire my species, living and dead” –Steve Jobs
Effective altruism in the garden of ends
by Tyler Alterman — 2022-09-14T22:02:48.200Z

This is a crosspost from the EA Forum that I was encourage to post here as well. I imagine it may have even more experience here, since the most "hardcore" EAs that I know typically began as rationality community members.

Huge thanks to Alex Zhu, Anders Sandberg, Andrés Gómez Emilsson, Andrew Roberts, Anne-Lorraine Selke (who I've subbed in entire sentences from), Crichton Atkinson, Ellie Hain, George Walker, Jesper Östman, Joe Edelman, Liza Simonova, Kathryn Devaney, Milan Griffes, Morgan Sutherland, Nathan Young, Rafael Ruiz, Tasshin Fogelman, Valerie Zhang, and Xiq for reviewing or helping me develop my ideas here. Further thanks to Allison Duettmann, Anders Sandberg, Howie Lempel, Julia Wise, and Tildy Stokes, for inspiring me through their lived examples.

I did not believe that a Cause which stood for a beautiful ideal […] should demand the denial of life & joy.

– Emma Goldman, Living My Life

This essay is a reconciliation of moral commitment and the good life. Here is its essence in two paragraphs:

Totalized by an ought, I sought its source outside myself. I found nothing. The ought came from me, an internal whip toward a thing which, confusingly, I already wanted – to see others flourish. I dropped the whip. My want now rested, commensurate, amidst others of its kind – terminal wants for ends-in-themselves: loving, dancing, and the other spiritual requirements of my particular life. To say that these were lesser seemed to say, “It is more vital and urgent to eat well than to drink or sleep well.” No – I will eat, sleep, and drink well to feel alive; so too will I love and dance as well as help.

Once, the material requirements of life were in competition: If we spent time building shelter it might jeopardize daylight that could have been spent hunting. We built communities to take the material requirements of life out of competition. For many of us, the task remains to do the same for our spirits. Particularly so for those working outside of organized religion on huge, consuming causes. I suggest such a community might practice something like “fractal altruism,” taking the good life at the scale of its individuals out of competition with impact at the scale of the world.

If you’re a Blinkist or Sparknotes person, you can stop here.

If you read on, you might find that everything written has been said already in the history of philosophy.[1] ...
The problem with the media presentation of “believing in AI”
by Roman Leventov — 2022-09-14T21:05:10.234Z

In my impression, in the last several years, in the media coverage of the progress in AI, journalists often called “believers” the pundits (interviewed or quoted by the journalists) who thought that AI can in principle take over all intellectual work from humans, or that AI is “intelligent” in some sense: for example, AI does really “understand language”, or that AI is sentient. This phrasing can be revealed in questions such as “Do you believe AI can understand human language?”, or flashy titles such as “A Google engineer believes an AI has become sentient” (a little more on this at the end of the post).

I think this wording may create in the minds of the laypeople an impression that these pundits “believe in AI” in some quasi-religious way. The laypeople may also conflate the “AI believers” with adherents of some actual churches of AI (I think these churches will become better known in the future, as the AI progress continues).

To prevent this unfair impression from solidifying, I think expert “AI believers” should use the following strategy: whenever a journalist asks them whether they “believe” AI can be sentient (understand language, understand human emotion, make a moral judgement, etc.), instead of simply answering “Yes” or “No”, they should note that the answer depends on the functional theory of consciousness/language understanding/morality/etc., and for almost all such theories (except those which explicitly assume uncomputability, such as Penrose’s theory of consciousness), the answer will automatically be “Yes, AI can hit this mark in principle”, so the question should be recast as a timeline prediction.

Then, if the format permits, the expert may note that the side which may be more appropriately called “believers” in the given argument are those who say that AI will not be able to do X, because such claims explicitly or implicitly rest on an unfalsifiable theory of X which essentially says “X is something that people can do and AIs can’t”.

Back to the coverage of the Blake Lemoine story: in this case, it is indeed appropriate to write that he “believed” that AI is sentient because he didn’t assume any functional theory of sentience. But I’m afraid that journalists will paint future, more principled statements with the same brush.
Seeing the Schema
by vitaliya — 2022-09-14T20:45:40.699Z

For the past month or two, I've been regularly playing a mental rotation game, available here. In the game, the task is to find which two of the six displayed block objects are identical under rotation. I'm not the best at mental rotation, so I figured I'd try and get better at it. But while I've gotten better at the matching task, I think I've started to see diminishing returns at mental rotations - primarily because, for the most part, I've stopped doing them.

The "reward signal" here is success at the matching task - which doesn't actually require you to perform a mental rotation. In the above image, for example, I run a few basic heuristics: the two blocks on the right are planar, and as one of them has a hook shape and the other doesn't, they can't match. Then, the lower centre block doesn't have a planar hook shape, where the other three do, so it can't match any of them. Finally, of the remaining blocks, the top centre's extruded "pole" above the planar hook is two blocks high, where the two on the left have extrusions three blocks high. As such, the two on the left must match.

It's easier and faster for my brain to run these "checksums" than it is to actually perform the mental rotation task. Early on, I was actually performing mental rotation, but now I'm just running heuristics across the set of shapes. This might seem like a pathological or degenerate solution - I'm not actually training the "thing I wanted" any more - but it's the path that gives me the highest reward at the end, so that's the thing my subconscious brain's started defaulting to.

It's weird to see a misaligned reward signal from the inside.
Responding to 'Beyond Hyperanthropomorphism'
by ukc10014 — 2022-09-14T20:37:52.818Z

TL;DR: This is an attempt to clarify a post (or perhaps just my own confusion) from a tech commentator, that was skeptical about AGI as a possibility, and alignment research as a project. As such, there is nothing novel in here, but it might be useful for people from outside or new to the LW/EA community.

---

Venkatesh Rao’s recent newsletter essay ‘

Beyond Hyperanthropomorphism

’, raised a number of questions about the direction of AGI and, more relevantly, the usefulness and validity of AGI alignment research[1].[2] His points regarding alignment, to the extent they are concrete, are buried in a wide-ranging exposition; hence, they are hard to engage with.[3] Nevertheless, in order to bridge his philosophically-informed perspective with the more practical approach of many alignment researchers, I thought it useful to situate his specific claims in a few current conversations in alignment.

The issues Rao seems to raise in the essay that I will cover are:

A necessary concept for useful conversations about intelligence (human, animal, AGI, etc.) is defining a ‘coherent quality of experience’ in respect of the entity in question, which are lumped under the neologism ‘something it is like to be’ or SIILTBness (borrowing from

Thomas Nagel

)
An AGI acquires a coherent quality of experience by having physical interactions with the world, through sensors, actuators, arms, etc., as opposed to just being trained on online text or image corpora
Until researchers can describe SIILTBness in respect of (an) AGI, it is meaningless to talk about (that) AGI’s intelligence, sentience, self-awareness, consciousness, etc.
When suggesting there is an ‘alignment problem’, researchers are extrapolating specific anthropomorphised nascent features of their AGIs, to conjure a golem-like phantasm looming in humanity’s future
Moreover, the very idea of AI alignment as a special or outsize problem, as opposed to a ‘normal engineering problem’, is ill-posed, at least in the absence of SIILTBness

I draw upon another

LW post

discussing the essay,

comments

on his blog, as well as a 2021 Twitter

thread

.[4]

On SIILTBness and LLMs

The essay

discusses

why machine SIILTBness requires embodiment and situatedness, as well as a subjective, phenomenological notion of

time

as a way of structuring events in the world.[5] These basically a...
When is intent alignment sufficient or necessary to reduce AGI conflict? 
by JesseClifton — 2022-09-14T19:39:11.920Z

In this post, we look at conditions under which Intent Alignment isn't Sufficient or Intent Alignment isn't Necessary for interventions on AGI systems to reduce the risks of (unendorsed) conflict to be effective. We then conclude this sequence by listing what we currently think are relatively promising directions for technical research and intervention to reduce AGI conflict.

Intent alignment is not always sufficient to prevent unendorsed conflict

In the

previous post

, we outlined possible causes of conflict and directions for intervening on those causes. Many of the causes of conflict seem like they would be addressed by successful AI alignment. For example: if AIs acquire conflict-seeking preferences from their training data when we didn’t want them to, that is a clear case of misalignment. One of the suggested solutions: improving adversarial training and interpretability, just is alignment research, albeit directed at a specific type of misaligned behavior. We might naturally ask, does all work to reduce conflict risk follow this pattern? That is, is intent alignment sufficient to avoid unendorsed conflict?

Intent Alignment isn't Sufficient

is a claim about unendorsed conflict. We’re focusing on unendorsed conflict because we want to know whether technical interventions on AGIs to reduce the risks of conflict make a difference. These interventions mostly make sense for preventing conflict that isn’t desired by the overseers of the systems. (If the only conflict between AGIs is endorsed by their overseers, then conflict reduction is a problem of ensuring that AGI overseers aren’t motivated to start conflicts.)

Let H be a human principal and A be its AGI agent. “Unendorsed” conflict, in our sense, is conflict which would not have been endorsed on reflection by H at the time A was deployed. This notion of “unendorsed” is a bit complicated. In particular, it doesn’t just mean “not endorsed by a human at the time the agent decided to engage in conflict”. We chose it because we think it should include the following cases:

H makes a mistake in designing A such that conflict at a later time is rational by H’s lights. For example, H could’ve built A so that it can solve Prisoner’s Dilemmas using open-source game theory but failed to do so. Then A finds itself in a one-shot Prisoner’s Dilemma (as evaluated by H’s preferences), and d

...
When would AGIs engage in conflict?
by JesseClifton — 2022-09-14T19:38:22.478Z

Here we will look at two of the claims introduced in the previous post: AGIs might not avoid conflict that is costly by their lights (

Capabilities aren’t Sufficient

) and conflict that is costly by our lights might not be costly by the AGIs’ (

Conflict isn’t Costly

).

Explaining costly conflict

First we’ll focus on conflict that is costly by the AGIs’ lights. We’ll define “costly conflict” as (ex post) inefficiency: There is an outcome that all of the agents involved in the interaction prefer to the one that obtains.[1] This raises the

inefficiency puzzle of war

: Why would intelligent, rational actors behave in a way that leaves them all worse off than they could be?

We’ll operationalize “rational and intelligent” actors as expected utility maximizers.[2] We believe that the following taxonomy of the causes of inefficient outcomes between expected utility maximizers is exhaustive, except for a few implausible edge cases. (We give the full taxonomy, and an informal argument that it is exhaustive, in the

appendix

.) That is, expected utility maximization can lead to inefficient outcomes for the agents only if one of the following conditions (or an implausible edge case) holds. This taxonomy builds on Fearon’s (1995) influential “rationalist explanations for war”.[3]

Private information and incentives not to disclose. Here, “private information” means information about one’s willingness or ability to engage in conflict — e.g., how costly one considers conflict to be, or how strong one’s military is — about which other agents are uncertain. This uncertainty creates a risk-reward tradeoff: For example, Country A might think it’s sufficiently likely that Country B will give up without much of a fight that it’s worthwhile in expectation for A to fight B, even if they’ll end up fighting a war if they are wrong.

In these cases, removing uncertainty — e.g., both sides learning exactly how willing the other is to fight — opens up peaceful equilibria. This is why conflict due to private information requires “incentives not to disclose”. Whether there are incentives to disclose will depend on a few factors.

First, the technical feasibility of different kinds of verifiable disclosure matters. For example, if I have an explicitly-specified utility function, how hard is it for me to prove to you how much my utility function val...
When does technical work to reduce AGI conflict make a difference?: Introduction
by JesseClifton — 2022-09-14T19:38:00.760Z

This is a pared-down version of a longer draft report. We went with a more concise version to get it out faster, so it ended up being more of an overview of definitions and concepts, and is thin on concrete examples and details. Hopefully subsequent work will help fill those gaps.

Sequence summary[1]

Some researchers are focused on reducing the risks of conflict between AGIs. In this sequence, we’ll present several necessary conditions for technical work on AGI conflict reduction to be effective, and survey circumstances under which these conditions hold. We’ll also share some thoughts on promising directions for research and intervention to prevent AGI conflict.

This post

We give a breakdown of necessary conditions for technical work on AGI conflict reduction to make a difference: “AGIs won’t always avoid conflict, despite it being materially costly” and “intent alignment is either insufficient or unnecessary for conflict reduction work to make a difference.” (

more

)

When would AGIs engage in conflict?

To assess the claim that AGIs would figure out how to avoid conflict, we give a breakdown (which we believe is exhaustive) of the causes of conflict between rational agents. Some kinds of conflict can be avoided if the agents are sufficiently capable of credible commitment and disclosure of private information. Two barriers to preventing conflict by these means are (i) strategic pressures early in multipolar AI takeoff that make it risky to implement cooperation-improving technologies, and (ii) fundamental computational hurdles to credible commitment and disclosure. Other causes of conflict — miscoordination and conflict-seeking preferences — can’t be solved by credible commitment and information disclosure alone. (

more

)
We give examples of technical work aimed at each of the causes of conflict surveyed in (2a). (

more

)

When is intent alignment sufficient or necessary to reduce AGI conflict?

We survey conditions under which intent-aligned AGIs could engage in conflict that is not endorsed by their overseers. Scenarios include: locking in architectures that can’t solve informational and commitment problems; the unintentional corruption of human preferences by AGIs; and catastrophes which leave human overseers unable to prevent intent-aligned AGIs from acting on misunderstandings of their preferences. (

more

)
We consider whether it is pos

...