trentmkelly/LessWrong-43k huggingface dataset:
Detecting Genetically Engineered Viruses With Metagenomic Sequencing
by jefftk — 2024-06-27T14:01:34.868Z
Link: https://naobservatory.org/blog/detecting-genetically-engineered-viruses

This represents work from several people at the NAO. Thanks especially to Dan Rice for implementing the duplicate junction detection, and to @Will Bradshaw and @mike_mclaren for editorial feedback.

Summary

If someone were to intentionally cause a

stealth

pandemic today, one of the ways they might do it is by modifying an existing virus. Over the past few months we’ve been working on building a computational pipeline that could flag evidence of this kind of genetic engineering, and we now have an initial pipeline working end to end. When given 35B read pairs of wastewater sequencing data it raises 14 alerts for manual review, 13 of which are quickly dismissible false positives and one is a known genetically engineered sequence derived from HIV. While it’s hard to get a good estimate before actually going and doing it, our best guess is that if this system were deployed at the scale of approximately $1.5M/y it could detect something genetically engineered that shed like SARS-CoV-2 before 0.2% of people in the monitored sewersheds had been infected.

System Design

The core of the system is based on two observations:

If someone has made substantial modifications to an existing virus then somewhere in the engineered genome there will be a series of bases that are a good match for the original genome followed by a series of bases that are a poor match for the original genome. We can look for sequencing reads that have this property and raise them for human review.
Chimeric reads can occur as an artifact of sequencing, which can lead to false positives. The chance that you would see multiple chimeras involving exactly the same junction by chance, however, is relatively low. By requiring 2x coverage of the junction we can remove almost all false positives, at the cost of requiring approximately twice as much sequencing.

Translating these observations into sufficiently performant code that does not trigger alerts on common sequencing artifacts has taken some work, but we now have this running.

While it would be valuable to release our detector so that others can evaluate it or apply it to their own sequencing reads, knowing the details of how we have applied this algorithm could allow someone to engineer sequences that it would not be able to detect. While we would like to build a detection system that can’t be more readily bypassed once you know how it works, we’re unf...
Cross Robin
by jefftk — 2024-06-27T03:10:03.097Z

A common contra dance figure is "right shoulder round": walk around the
opposite-role person on the side of the set while looking at them. A
less common figure is the "mad robin": walk the same path, but instead
look at the opposite-role person across the set. This suggests a third
option, which we can call "cross robin": walk that path, but looking
at the same-role person diagonally across the set. [1]

Here's a potential dance:

Cross Robin

by Jeff Kaufman

(Beckett)

A1

(4) Long lines forward to the left, progressing

(4) On the way back, Larks roll Robins away (right to left)

(4) Long lines forward in place

(4) On the way back, Larks roll Robins away (left to right)

A2

(8) Right shoulder round your partner while looking at
your same-role neighbor.

(8) Larks allemande left 1.5x while Robins orbit CW 1/2

B1

(8) Left shoulder round your partner while looking at
your same role-neighbor.

(8) Robins allemande right 1.5x while Larks orbit CCW
1/2

B2

(16) Right shoulder round your partner, and swing.

The goal is that after the cross-looking you should get to do
something with that same-role neighbor. If you want, you can maintain
eye contact throughout the A2 and B1.

I've tried this with three other people who happened to be in the same
room (Harris, Cecilia, Lily) and it seems like it flows,
though I'm not sure yet if the cross-looking feels good.

(I considered calling this "Mad Lark", but Harris tells me Shoshana
Silverman has been pushing to call it "Mad Robin" when it starts with
the Robin going in front and "Mad Lark" when it starts with the Lark
going in front, and I didn't want to conflict. If that does catch on,
this could be both "Cross Robin" in the B1 and "Cross Lark" in the
A2. But I also think it's probably better not to name this figure at
all in teaching it to dancers, and just describe it. It's good to
keep the number of named figures low.)

[1] It also suggests a fourth option, where you walk the path while
looking at yourself. But that seems more awkward.

Comment via: facebook, lesswrong, mastodon
Live Theory Part 0: Taking Intelligence Seriously
by Sahil — 2024-06-26T21:37:10.479Z

Acknowledgements

The vision here was midwifed originally in the wild and gentle radiance that is Abram's company (though essentially none of the content is explicitly his).

The PIBBSS-spirit has been infused in this work from before it began (may it infuse us all), as have meetings with the Agent Foundations team at MIRI over the past ~2 years.

More recently, everyone who has been loving the High Actuation project into form (very often spontaneously and without being encumbered by self-consciousness of this fact):[1] individuals include Steve Petersen, Mateusz Baginski, Aditya Prasad, Harmony, TJ, Chris Lakin; the AISC 2024 team, Murray Buchanan, Matt Farr, Arpan Agrawal, Adam, Ryan, Quinn; various people from Topos, ALIFE, MAPLE, MATS, EA Bangalore. Published while at CEEALAR.

Disclaimers

Very occasionally there are small remarks/questions from a remarkable human named Steve, since this and the next two posts are an edited transcript of me giving him a talk. I left them in to retain the conversational tone. Steve has also consistently been a fantastic ground for this channeling.
I use the term “artefact” a fair amount in this sequence. Unfortunately for you and me, Anthropic also recently started using "artifact" in a different way. I'm using "artefact" in the common sense of the word. The British spelling should help remind of the distinction.

Taking Intelligence Seriously

Sahil:

I gave a talk recently, at an EA event just two days ago, where I made some quick slides (on the day of the talk, so not nearly as tidy as I’d like) and attempted to walk through this so-called “live theory”[2].

Maybe I can give you that talk. I'm not sure how much of what I was saying there will be present now, but I can try. What do you think? I think it'll take about 15 minutes. Yeah?

Steve:

Cool.
Sahil:

Okay, let me give you a version of this talk that's very abbreviated.

So, the title I’m sure already makes sense to you, Steve. I don't know if this is something that you know, but I prefer the word “adaptivity” over intelligence. I'm fine with using “intelligence” for this talk, but really, when I'm thinking of AI and LLMs and “live” (as you’ll see later), I'm thinking, in part, of adaptive. And I think that connotes much more of the relevant phenomena, and much less controversially.

It’s also less distr...
Instrumental vs Terminal Desiderata
by Max Harms — 2024-06-26T20:57:17.584Z

Bob: "I want my AGI to make everyone extremely wealthy! I'm going to train that to be its goal."

Cassie: "Stop! You'll doom us all! While wealth is good, it's not everything that's good, and so even if you somehow build a wealth-maximizer (instead of summoning some random shattering of your goal), it will sacrifice all the rest of the good in the name of wealth!"

Bob: "Maybe if it suddenly became a god-like superintelligence, but I'm a hard take-off skeptic. In the real world we have continuous processes and I'm going to be in control. If it starts to go off the rails, I'll just stop it and re-train it to not do that."

Cassie: "Be careful what you summon! While it may seem like you're in control in the beginning, these systems are generalized obstacle-bypassers, and you're making yourself into an obstacle that needs to be bypassed. Whether that takes two days or twenty years, you're setting us up to die."

Bob: "Ok, fine. So I'll build my AGI to make people rich and simultaneously to respect human values and property rights and stuff. At the point where it can bypass me, it'll avoid turning everyone into bitcoin mining rigs or whatever because that would go against its goal of respecting human values."

Cassie: "What does 'human values' even mean? I agree that if you can build an AGI that is truly aligned, we're good, but that's a tall order and it doesn't even seem like what you're aiming for. Instead, it seems like you think we should train the AGI to maximize a pile of desiderata."

Bob: "Yeah! My AGI will be helpful, obedient, corrigible, honest, kind, and will never produce copyrighted songs, memorize the NYT, or impersonate Scarlett Johansson! I'll add more desiderata to the list as I think of them."

Cassie: "And what happens when those desiderata come into conflict? How does it decide what to do?"

Bob: "Hrm. I suppose I'll define a hierarchy like Asimov's laws. Some of my desiderata, like corrigibility, will be constraints, while others, like making people rich, will be values. When a constraint comes in conflict with a value, the constraint wins. That way my agent will always shut down when asked, even though doing so would be a bad way to make us rich."

Cassie: "Shutting down when asked isn't the hard part of corrigibility, but that's a tangent. Suppose that the AGI is faced with a choice of a 0.0001% chance of being dishonest, but earning a billion dollars, or...
Imbue (Generally Intelligent) continue to make progress
by Nathan Helm-Burger — 2024-06-26T20:41:18.413Z
Link: https://imbue.com/research/70b-intro/

I've been following the company Imbue and their podcast Generally Intelligent since they started. They've said thoughtful and creative things in their podcast, and I think they are making impressive progress towards AGI considering their relatively smaller size.

Just wanting to keep people appraised. If they did hit on something unusually potent, would they get acquired by a larger actor? Hard to know.

What is it that they're releasing? In addition to their 70B param model...

11 sanitized and extended NLP reasoning benchmarks including ARC, GSM8K, HellaSwag, and Social IQa

An original code-focused reasoning benchmark

A new dataset of 450,000 human judgments about ambiguity in NLP questions

A hyperparameter optimizer for scaling small experiments to a 70B run

Infrastructure scripts for bringing a cluster from bare metal to robust high-utilization training

An interesting quote about their hyperparameter optimizer:

It is possible to run resource-efficient pre-training experiments that can effectively scale to a large model. Using CARBS, we could reliably predict the performance of any model with a given number of parameters according to well-defined scaling laws, lowering the barrier to entry to building large models.

I think this quote is relevant to this discussion in the comment section of a different post.
Tracing the steps
by matimissona — 2024-06-26T19:22:35.518Z

Musings on the Yudkowsky-Hanson debate from 2011.

After all sorts of interesting technological things happening in some undetermined point in the feature, will we see some small nucleus that control all resources or will we see a civilisation-wide large-scale participation in these things going down? [Robin Hanson, 2011]

What is singularity? It means something different than it used to. Originally, singularity is the breakdown in Vinge's ability to model the future, beyond the technological creation of intelligence smarter than humans. Or as IJ Good explains the intelligence explosion – "smarter minds building smarter minds".

"The fastest form of intelligence explosion is an AI rewriting its own source code". This is Yudkowsky's 2011 scenario – "a brain in a box in a basement". Intelligence explosion.[1]

A "brain in a box in a basement" starts very small, then gets better. And will outcompete the rest of the world.

Let's just imagine the steppes or the savannah about 50,000 years ago. Someone drops a 1TB SSD with the "intelligence". What happens? Nothing – it's just a shiny object for whoever finds it, and traded for a piece of skin or something. (We're fond of trading shiny things).

"The long term history of our civilisation has been a vast increase in capacity. From language, farming to industry and who knows where. Lots of innovations have happened. Lots of big stories along the line. The major story is the steady, gradual growth. Most disruptions are small. On a larger scale it's more steady." Hanson.

Three major events in human history has seen an OOM increase in progress – the invention of language, of farming, and the advent of industrial society. These three events are singularities, according to Hanson. Edinburgh got some advantage from being first to industrialization, but they didn't get a huge advantage. They didn't take over the world.

In other words – never before in human history have innovations, even the most disruptive innovation, caused a first-mover to take over the world.

"We have strong reason to believe in a strong localized intelligence", Yudkowsky says.

Hanson thinks we'll have a general, gradual economic increase.

Brain in a box in a basement

Yudkowsky conflates the algorithm and the architecture. First he says evolution is the algorithm that designs intelligence "over millions and millions of years." Then "all of a sudden, there's this n...
Countering AI disinformation and deep fakes with digital signatures
by Dave Lindbergh — 2024-06-26T18:09:56.304Z

According to The Economist, disinformation campaigns (often state-sponsored) use "AI to rewrite real news stories":

In early March a network of websites, dubbed CopyCop, began publishing stories in English and French on a range of contentious issues. They accused Israel of war crimes, amplified divisive political debates in America over slavery reparations and immigration and spread nonsensical stories about Polish mercenaries in Ukraine... the stories had been taken from legitimate news outlets and modified using large language models.

Deep fakes of still images and now video clips are similarly based on legitimate original photos and video. Detecting such fakery can be challenging.

An obvious partial solution (that I haven't seen discussed) is for legitimate news sources to digitally sign original photos, video, and news stories.

Web browsers (or extensions) can automatically confirm or flag fakery of the original source (news publisher, journalist, photographer). All that's needed is a consensus standard on how to encode such digital signatures - the sort of thing that W3C and similar organizations produce routinely.

Such signatures could be conveyed in a custom HTML wrapper that needn't be visible to readers with web browsers unable to parse them - there's no need to sprinkle "BEGIN PGP SIGNED MESSAGE" at the start of every article; these can be invisible to users.

Signatures could be layered - a photo could be signed by the camera capturing the original (manufacturer, serial number), the photographer (name, nym, unique email address), and publisher, all at the same time, similarly for text news articles.

Video could be signed on a per-frame basis as well as a whole-clip or partial-clip basis. Per frame signatures could include consecutive frame numbers (or timestamps), enabling trivial detection of selective editing to produce out-of-context false impressions.

Writers and photographers who wish to remain anonymous could sign with a nym (pseudonym; a unique identifier under control of an author - for example an email address or unique domain name not publicly connected with an individual); this would still enable anonymous sources to maintain reputations.

Signatures that reliably and uniquely identify original news sources ("nytimes.com", "thefp.com", "aljazeera.com", etc.) could be used by third party services to produce reliability/trust/bias ratings for ...
Progress Conference 2024: Toward Abundant Futures
by jasoncrawford — 2024-06-26T15:39:45.267Z
Link: https://rootsofprogress.org/progress-conference-2024-toward-abundant-futures/

The progress movement has grown a lot in the last few years. We now have progress journals, think tanks, and fellowships. The progress idea has spread and evolved into the “abundance agenda”, “techno-optimism”, “supply-side progressivism”, “American dynamism”. All of us want to see more scientific, technological, and economic progress for the good of humanity, and envision a bold, ambitious, flourishing future.

What we haven’t had so far is a regular gathering of the community.

Announcing Progress Conference 2024, a two-day event to connect people in the progress movement. Meet great people, share ideas in deep conversations, catalyze new projects, get energized and inspired.

Apply for an invitation here

Hosted by: the Roots of Progress Institute, together with the Foresight Institute, HumanProgress.org, the Institute for Humane Studies, the Institute for Progress, and Works in Progress magazine

When: October 18–19, 2024

Where: Berkeley, CA—at the Lighthaven campus, an inviting space perfect for mingling

Speakers: Keynotes include Patrick Collison, Tyler Cowen, Jason Crawford, and Steven Pinker. Around 20 additional speakers will share ideas on four tracks: the big idea of human progress, policy for progress, tech for progress, and storytelling/media for progress. Full speaker list

Attendees: We expect 200+ intellectuals, builders, policy makers, storytellers, and students. This is an invitation-only event, but anyone can apply for an invitation. Complete the open application by July 15th.

Program: Two days of intellectual exploration, inspiration and interaction that will help shape the progress movement into a cultural force. Attend talks on topics from tech to policy to culture, build relationships with new people as you hang out on cozy sofas or enjoy the sun in the garden, sign up to run an unconference session and find others who share your interests and passions, or pitch your ideas to those who could help make your dreams a reality.

Special thanks to our early sponsors: Cato Institute, Astera Institute, and Freethink Media! We have more sponsorships open, view sponsorship opportunities here.
Schelling points in the AGI policy space
by mesaoptimizer — 2024-06-26T13:19:25.186Z

I've been thinking about memetically fit Schelling points in the AGI policy space. I'll describe four such "Schelling policies", and use them as pedagogical examples.

Shut it all down

MIRI's new stated objective is the clearest example of a Schelling policy: "Shut it all down". MIRI states that they want governments to coordinate to pause all AI research that involves smarter-than-human systems. Laypeople will find this policy easy to understand, since they can rely on the shared cultural knowledge of CFC bans and international nuclear disarmament as case studies.

If you want to coordinate a large number of people coherently towards furthering a particular policy, "you get about five words" that you can make 'common knowledge' such that people can coordinate in a specific direction. The ease of communicating the policy makes a big difference in such conditions.

When you attempt to communicate an idea widely, you'll notice that people usually end up with multiple slightly (or sometimes wildly) differing copies of the original idea. If you've played the Telephone game, you've experienced just how much information can be lost as an idea spreads from one person to another. In the context of policies, individual people's beliefs and incentives will warp the instantiation of the policy they will communicate and support. (For example, you'll find companies lobbying regulators to carve out exceptions that benefit them.)

Here's where Schelling points are invaluable: they serve as natural attractors in the space of ideas, and therefore enable people to 'error-correct' the idea they encounter and figure out the policy that everyone is coordinating around.

"Shut it all down" is a Schelling point. "Shut it all down if we see evidence of unprompted deception and power-seeking in AGI models" is not a Schelling point, you have multiple free variables that can and will be optimized to benefit the people spreading the idea -- which can result in a lack of coordination and the idea being outcompeted by memetically fitter ideas. "Prevent the training of models using compute greater than 1025 floating point operations" also has a free variable: why exactly 1025 floating point operations? Why not 1024 or 1026? Until 1025 floating point operations becomes a Schelling number, the policy containing it is not a Schelling point.

Effective Accelerationism (e/acc)

The biggest difference bet...
Bad lessons learned from the debate
by bayesyatina — 2024-06-26T11:54:44.953Z

Debates are life, the rest is just time to prepare.

As a student, I played classic policy Karl Popper debates. The first thing we learned was sparring. The rules are simple: two people are given a topic, such as "an orange is better than an apple", and they randomly determine who supports and who opposes the topic. After that, the speaker (the one in favor of the topic) has 30 seconds to present their argument. Then, the speaker who opposes has half a minute to present their case, and then both speakers have another turn. All this is given prior to the theoretical basis of logic, logical errors, exercises in rhetoric, and generally any serious preparation. I was surprised to learn that the average person can quickly come up with an argument for any topic, even the most absurd, and can do so convincingly at times. This surprised me, but I was also impressed by the creativity and quick thinking of the participants.

Further, there is a block for detecting other people's logical errors, the study of rhetorical techniques, and the development of public speaking skills, all in an atmosphere of intense competition.

Additionally, the ability to view an issue from different perspectives is being developed. Smoking is bad for health, yes, but consider the economic impact. And what a great social and psychological benefit it can be!

Arguments in life have become a platform for honing skills, as it is easier to appear right than to actually be right. With all other factors being equal, individuals who have extensive experience arguing, whether in formal debates, online discussions*, or in real-life conversations, have a significant advantage when it comes to appearing convincing to others.

Postmodern

An epiphany struck me during an exercise called "shadow fighting." It's a form of sparring, but against yourself. You present arguments one by one, dismantle them, then rebuild them, and so on. The key quality criterion was the equal strength of both sides.

I suddenly realized something: there is no such thing as absolute truth. Instead, there is only the ability to convince others of our point of view. How could I ever think that there were right and wrong answers? After all, we can always find arguments for and against any position. This idea seems to be common among debaters, as the very nature of the debate encourages us to to see the imperfection of any position.

I felt ...