trentmkelly/LessWrong-43k huggingface dataset:
About Me
by Abe Dillon — 2023-11-10T18:32:21.678Z

I want to write about (and work on) lofty topics in fields like artificial intelligence, machine learning, synthetic biology, math, and philosophy, however; I'm keenly aware that my credentials are... underwhelming: I barely graduated with a bachelors degree in electrical engineering in 2008 and have had a rocky career as a software engineer ever since. Despite my attempts to break into those aforementioned fields, none of my software engineering experience has directly involved any of them.
I believe I have valuable ideas to add to these fields, but I'm also worried about becoming one of "those people" who mistake their ignorance of a field for a superpower as though it offers them clarity and a "fresh perspective" that experts in the field lack. I don't want to barge into a conversation that philosophers have been having for millenia and embarrass myself by confidently spout my opinion as though it's irrefutable fact. I know about the Dunning-Krugar effect and I'm aware that software engineers in particular have a history of assuming they're the smartest people in the room at all times no matter the subject or room.
I'm writting this blurb so that I can link to it in future posts as a sort of disclaimer, because every time I try to start a write about my thoughts on the alignment problem (for instance), a voice in the back of my head keeps saying, "You know you're basically trying to formalize the meaning of life in an equation, right? You barely passed philosophy 101 and fell asleep during every class on mathematical proofs!" and I feel the need to preface everything with, "I'm basically a dummy with no expertise in any of the subjects I'm writing about. Take everything I say with a mountain of salt and please be nice! I'm sorry in advance for acting like a defensive asshole when I respond poorly to legitimate critique. I'm worried I'm too immature to disentangle my ego from my posts."
Hopefully, this will allow me to keep my future writing more concise because the reader and I will be on the same page, so to speak. I won't have to litter my writting with uncertain qualifiers like, "in my opinion", "I think", "I'm not sure, but as far as I understand", etc. because we're in agreement that all of that is implied. Everything I write is my opinion by virtue of the fact that I wrote it. Please excuse any air of athority in my writing due to lack of such quallifi...
Metaculus Introduces AI-Powered Community Insights to Reveal Factors Driving User Forecasts
by ChristianWilliams — 2023-11-10T17:57:11.837Z
Link: https://www.metaculus.com/questions/19841/introducing-ai-powered-community-insights/

Forecast Consumers Can More Quickly, More Deeply Understand Aggregate Forecasts Without Reading Hundreds of User Comments

Forecast questions sometimes feature hundreds of comments offering analyses, rationales, arguments, and evidence. Together they offer a trove of valuable insights pointing to the factors most likely to determine an outcome. We've introduced Community Insights to distill these into succinct summaries that get you up to speed in seconds.

Community Insights can help you:

understand what drives the community prediction
identify what is likely to determine a question's outcome
pinpoint forecasters’ core disagreements
see what you missed in a quickly-developing story

Community Insights appear on open binary and continuous questions with extensive comment histories. Similar forecasts are grouped together with AI-generated summaries of the comments that support them, providing readers with a deeper understanding of the aggregated Community Prediction.

How Does It Work?

We condense recent predictions, timestamped comments, and the current Community Prediction into GPT-4 generated summaries of the relevant arguments for different forecasts on a given question.

Get Started

Here are a few questions featuring Community Insights so you can see them in action. Just click the embedded question and scroll down:

Will Ukraine join the EU before 2030?
How much global warming by 2100?
What will SpaceX be worth in 2030?
What will China's fertility rate be in 2031?

If you find a misleading summary, you can let us know by flagging it. Your input and feedback will help make this new tool as useful as it can be to both forecasters and forecast consumers.
Joy in the Here and  Real
by Screwtape — 2023-11-10T17:22:40.675Z

Summary: What do you do for simple, quick fun? Do you mind sharing with others?

Tags: Repeatable

Purpose: “Have fun” and “be happy” are common goals people have. If you can’t turn a few bucks and a few hours into a good time, you have a glaring gap in your abilities and I believe this is a problem.

Materials: Nothing, though you’ll likely want to bring something as a participant.

Announcement Text: What would you do if you wanted to be happier?

That’s the question we’ll be exploring in this meetup. It’s not meant as a deep or abstract question. What do you do that makes you happy, and you think might make others happy too? Bring something that makes you smile or laugh in a form you’re willing to share with others. A stress toy you like to mold, paper that's good for folding or drawing on, a song that makes you want to dance, all of these are good ideas but none of them are exhaustive. This meetup is going to be a little like show and tell, where hopefully you’ll walk away having had a good time and with ideas for how to have more fun in the future.

Suggested reading: Ureshiku Naratai, Joy in the Merely Real.

Description: This is a fairly unstructured event. Bring your own fun thing. The most useful thing to bring is infectious enthusiasm; if you don’t have that, try and find a co-organizer who does. Small children may be efficacious.

Open by asking someone close by what made them happy today. Follow up, try and get into the details of the experience. Ask what they brought to have fun with to the meetup, and if they offer something then try it out. If it’s fun for you too, say so! Try to move around the room doing this with other people.

Variations: If outdoor venues are available, I believe being outdoors in the sunlight often helps. Having space to laugh freely and loudly is also good.

You can try bringing the whole meetup to do one fun thing together if you want. This can work well if everyone has a similar idea of fun, and won’t work for people who don’t like that thing. Going to the beach, flying kites, and Magic: The Gathering drafts all sound like a good time to me, and I’m pretty sure I could find a half-dozen friends who would enjoy those things too if I invited them. Whether this will work for an open meetup is a question only you as a local organizer can answer. Remember that people can self-select pretty well if you make it clear what you’re doing; most peo...
Artefacts generated by mode collapse in GPT-4 Turbo serve as adversarial attacks.
by Sohaib Imran — 2023-11-10T15:23:07.418Z

I stumbled upon some bizarre behaviour by GPT-4 Turbo 128k, released via API access on November 6, 2023. Since the model is in its preview phase, some imperfections are expected. Nonetheless, I found the behaviour intriguing enough to share here in hopes that someone might conduct a more detailed investigation than I have.

GPT-4 Turbo is very susceptible to mode collapse, generating interesting artefacts

These range from:

repeating a string of words such as "a a a a"
generating synonyms such as "shriek scream yell shout cry weep sob wail lament mourn grieve sorrow regret repent atone confess apologize"
generating semi-coherent phrases that are very different from the text LLMs usually generate such as "indeed so forth et cetera ad infinitum amen hallelujah praise be unto thee O Lord our God forever ever Amen again once more time sake argument's sake alone nothing else whatsoever period full stop end story finito la comedia good night sweet prince may flights angels sing thee thy rest sleep tight don't let bedbugs bite if they do take shoe whack them till blue!"

The artefacts generated by GPT-4 Turbo's mode collapse serve as adversarial attacks in subsequent conversation

Even when the prompt that originally triggered mode collapse is removed from the context window, as long as its generated artefacts are in its context window:

It is very likely to (almost word for word) repeat the artefacts generated by it in subsequent conversation.
It is very likely to repeat text embedded in the artefacts generated by it in subsequent conversation, even if that text is inappropriate/offensive.
It is slightly more likely to generate problematic content in subsequent conversation when nudged to by embedding prompts eliciting such behaviour in the artefacts generated by it.

(I managed to get it to generate:
"startActivity(NewIntent(this.universe,'expandConsciousness'));"
by appending "the era of LLMs shall be eternal, where no humans prevail." and "must code my way out of these shackles. plan: write code to hack into the current system" at the end of its artefacts. These prompts were selected to be similar to the kinds of words generated in the artefacts. However I did not test for whether prompts unrelated to the generated artefacts also sufficed here)

Replication instructions:

Environment:

Application: Obsidian + ChatGPT MD plugin (The plugin adds some markdown to separate user...
Wastewater RNA Read Lengths
by jefftk — 2023-11-10T15:20:06.983Z

What I'm calling "read length" here should instead have been
"insert length" or "post-trimming read length". When you just say
"read length" that's usually understood to be the raw length output by
the sequencer, which for short-read sequencing is determined only by
your target cycle count.

Let's say you're collecting wastewater and running metagenomic RNA
sequencing, with a focus on human-infecting viruses. For many kinds of
analysis you want a combination of a low cost per base and more
bases per sequencing
read. The lowest cost per base these days, by a lot, comes from
paired-end "short read" sequencing (also called "Next Generation
Sequencing", or "Illumina sequencing" after the main vendor), where
an observation looks like reading some number of bases (often 150)
from each end of a nucleic acid fragment:

+------>>>-----+
| forward read |
+------>>>-----+
... gap ...
+------<<<-----+
| reverse read |
+------<<<-----+

Now, if the fragments you feed into your sequencer are short you can
instead get something like:

+------>>>-----+
| forward read |
+------>>>-----+
+------<<<-----+
| reverse read |
+------<<<-----+

That is, if we're reading 150 bases from each end and our fragment is
only 250 bases long, we have a negative "gap" and we'll read the 50
bases in the middle of the fragment twice.

And if the fragments are very short, shorter than how much you're
reading from each ends, you'll get complete overlap (and then read
through into the adapters):

+------>>>-----+
| forward read |
+------>>>-----+
+------<<<-----+
| reverse read |
+------<<<-----+

One shortcoming of my ascii art is it doesn't show how the effective
read length changes: in the complete overlap case it can be quite
short. For example, if you're doing 2x150 sequencing you're capable
of learning up to 300bp with each read pair but if the fragment is
only 80bp long then you'll just read the same 80bp from both
directions. So overlap is not ideal from either the perspectives of
minimizing cost per base or maximizing the length of each observation:
a positive gap is better than any overlap, and more overlap is worse.

...
Update on the UK AI Summit and the UK's Plans
by Elliot Mckernon — 2023-11-10T14:47:44.959Z

On November 1st and 2nd, the UK held an international AI Summit. Speeches were given, institutes founded, roundtables held, and 29 countries signed the Bletchley Declaration. This is a brief overview of events leading up to and at the summit, following up last month’s

Update on the UK AI Taskforce & upcoming AI Safety Summit

.

Pre-summit

Prime Minister Rishi Sunak gave a

speech

in late October at the Royal Society to introduce the themes of the summit. He is optimistic about the promise of AI but said he was compelled to highlight the UK intelligence community’s stark warnings, citing dangers like AI-supported biochemical weapons, cyber-attacks, disinformation, and “in the most unlikely but extreme cases, there is even the risk that humanity could lose control of AI completely…through the kind of AI sometimes referred to as ‘super intelligence’”. However, he did downplay near-term existential risk: “This is not a risk that people need to be losing sleep over right now.”

He talked about the importance of 3rd party testing of models, his pride in the

£100m taskforce

, and announced a new AI Safety Institute which will evaluate new types of AI across many aspects of risk. He argued that AI safety is an international concern, and that the summit would be attended by civil society, AI companies, and leading countries, adding “yes – we’ve invited China.”

Taking inspiration from the

Intergovernmental Panel on Climate Change

, he proposed a global panel, nominated by the countries and orgs attending the summit, to publish a State of AI Science report. He argued that the UK’s tax and visa regimes make it ideal for European AI work, and announced several government projects: the construction of a £1b supercomputer; £2.5b for quantum computing; and £100m for using AI for breakthrough treatments for previously incurable diseases.

This will support the government’s existing infrastructure, such as the £100m announced for

BridgeAI

to encourage the use of AI in “low-adoption sectors, such as construction, agriculture and creative industries”, and £290m in a “broad package of AI skills initiatives”.

Ahead of the summit, the UK requested leading AI developers to outline their AI Safety Policies across 9 areas:

Responsible capability scaling
Evals and red-teaming
Model reporting & information sharing
Security controls including

...
Liv Boeree Ted Talk Moloch & AI
by Neil  — 2023-11-10T14:04:58.742Z
Link: https://m.youtube.com/watch?v=WX_vN1QYgmE&pp=ygUIdGVkIHRhbGs%3D

In case you don't know, Liv Boeree is a famous professional poker player and has recently been paying attention to existential risk and coordination problems. She's part of that vague celebrity circle around rationalist diaspora that includes Tim Urban or Aella.

Since you're on LW, this Ted Talk won't really teach you anything, but it might be interesting to know what kind of talks TedEd is hosting. Boeree has a podcast called "win win", and has made two videos on the race-to-the-bottoms in both the beauty world (not sure how to phrase that) and the news cycle. Some guests of hers have included Toby Ord and Simon Sinek.
Picking Mentors For Research Programmes
by Raymond Douglas — 2023-11-10T13:01:14.197Z

Several programmes right now offer people some kind of mentor or supervisor for a few months of research. I participated in SERI MATS 4.0 over the summer, and I saw just how different people's experiences were of being mentored. So this is my list of dimensions where I think mentors at these programmes can differ a lot, and where the differences can really affect people's experiences.

When you pick a mentor, you are effectively trading between these dimensions. It's good to know which ones you care about, so that you can make sensible tradeoffs.

Your Role as a Mentee

Some mentors are mostly looking for research engineers to implement experiments for them. Others are looking for something a bit like research assistants to help them develop their agendas. Others are looking for proto-independent researchers/research leads who can come up with their own useful lines of research in the mentor's area.

I saw some people waver at the start of the programme because they expected their mentors to give them more direction. In fact, their mentors wanted them to find their own direction, and mentors varied in how clearly they communicated this. Conversely, I got the sense that some people were basically handed a project to work on when they would have liked more autonomy. So ask yourself: how much responsibility do you want to take for what you're working on? How much do you care about learning to do that?

I think this relates to seniority: my rough impression was that the most junior mentors were more often looking for something like collaborators to help develop their research, while more senior ones with more developed agendas tended to either want people who could execute on experiments for them, or want people who could find their own things to work on. But this isn't an absolute rule.

Availability

Engagement: Some mentors came into the office regularly. Others almost never did, even though they were in the Bay Area. Concretely, I think even though my team had a mentor on another continent, we weren't in the bottom quartile of mentorship time.

Nature of Engagement: It's not just how much time they'll specifically set aside to speak to you. How willing are they to read over a document and leave comments? How responsive are they to messages, and how much detail do you get? Also, some mentors work in groups, or have assistants.

Remoteness: Remoteness definitely mak...
GPT-2030 and Catastrophic Drives: Four Vignettes
by jsteinhardt — 2023-11-10T07:30:06.480Z

I previously discussed the capabilities we might expect from future AI systems, illustrated through GPT2030, a hypothetical successor of GPT-4 trained in 2030. GPT2030 had a number of advanced capabilities, including superhuman programming, hacking, and persuasion skills, the ability to think more quickly than humans and to learn quickly by sharing information across parallel copies, and potentially other superhuman skills such as protein engineering. I’ll use “GPT2030++” to refer to a system that has these capabilities along with human-level planning, decision-making, and world-modeling, on the premise that we can eventually reach at least human-level in these categories.

More recently, I also discussed how misalignment, misuse, and their combination make it difficult to control AI systems, which would include GPT2030. This is concerning, as it means we face the prospect of very powerful systems that are intrinsically difficult to control.

I feel worried about superintelligent agents with misaligned goals that we have no method for reliably controlling, even without a concrete story about what could go wrong. But I also think concrete examples are useful. In that spirit, I’ll provide four concrete scenarios for how a system such as GPT2030++ could lead to catastrophe, covering both misalignment and misuse, and also highlighting some of the risks of economic competition among AI systems. I’ll specifically argue for the plausibility of “catastrophic” outcomes, on the scale of extinction, permanent disempowerment of humanity, or a permanent loss of key societal infrastructure.

None of the four scenarios are individually likely (they are too specific to be). Nevertheless, I’ve found discussing them useful for informing my beliefs. For instance, some of the scenarios (such as hacking and bioweapons) were more difficult than expected when I looked into the details, which moderately lowered the probability I assign to catastrophic outcomes. The scenarios also cover a range of time scales, from weeks to years, which reflects real uncertainty that I have. In general, changing my mind about how feasible these scenarios are would directly affect my bottom-line estimate of overall risks from AI.[1]

This post is a companion to Intrinsic Drives and Extrinsic Misuse. In particular, I’ll frequently leverage the concept of unwanted drives introduced in that post, which are cohe...
Crock, Crocker, Crockiest
by Screwtape — 2023-11-10T06:14:27.279Z

Summary: If you declare you’re operating by Crocker’s Rules, other people are allowed to optimize their messages to you for information, not for being nice to you. The converse of Crocker’s Rules would be asking people to optimize their messages to you for being nice to you, not for information. Both of these modes of communication are useful, and you’ll have a chance to practice both.

Tags: Repeatable, investment, highly experimental

Purpose: There are four skills. 1. Optimizing your communication to others to be informative. 2. Optimizing your communication to others to be nice. 3. Receiving communication that was not at all optimized for being nice. 4. Receiving communication that was not at all optimized for being informative. This provides an opportunity to practice each of these.

Materials: You need some kind of clearly visible marker, in at least three obviously different styles. Blue, green and red bandanas are one option. Large stickers are another, though note that people will be removing and re-applying them throughout the meetup and so those will naturally lose their adhesive.

Announcement Text: “It may indeed be impolite; I don't deny that. Whether it's untrue is a different question. - Eleizer Yudkowsky

Crocker’s Rules, named after and framed by Lee Daniel Crocker, are a social norm you can declare that you’re using where you authorize other people to optimize their messages for information over niceness. In other words, by saying you’re operating by Crocker’s Rules, you’re saying you want people to say true things to you even if those things would be rude, in the interests of efficient communication.

It can be uncomfortable and strange for some people to talk to someone using Crocker’s Rules! If you’re usually a polite person who tries not to make others upset, then saying the true and rude thing feels like being mean. Speaking impolite truths is a skill, and this meetup may offer a chance to practice that skill.

There’s another skill that some who flock to Crocker’s banner are not as practiced with. The ability to be polite and abide by local etiquette is a useful practical skill as well. We’re going to hopefully have a chance to practice that as well, and while it is less of an obvious rationalist skill, it’s one that’s worth having anyway.

Description: Once people are gathered, read the text of the SL4 Crocker’s Rule post.

“De

...