trentmkelly/LessWrong-43k huggingface dataset:
Alpaca: A Strong Open-Source Instruction-Following Model
by sanxiyn — 2023-03-14T02:41:23.384Z
Link: https://crfm.stanford.edu/2023/03/13/alpaca.html

We performed a blind pairwise comparison between text-davinci-003 and Alpaca 7B, and we found that these two models have very similar performance: Alpaca wins 90 versus 89 comparisons against text-davinci-003.

Interestingly, Alpaca is trained using supervised finetuning, not RLHF. (text-davinci-003 is trained using RLHF.) This seems to confirm my suspicion that while RLHF improves performance it is not essential.
Discussion with Nate Soares on a key alignment difficulty
by HoldenKarnofsky — 2023-03-13T21:20:02.976Z

In late 2022, Nate Soares gave some feedback on my Cold Takes series on AI risk (shared as drafts at that point), stating that I hadn't discussed what he sees as one of the key difficulties of AI alignment.

I wanted to understand the difficulty he was pointing to, so the two of us had an extended Slack exchange, and I then wrote up a summary of the exchange that we iterated on until we were both reasonably happy with its characterization of the difficulty and our disagreement.1 My short summary is:

Nate thinks there are deep reasons that training an AI to do needle-moving scientific research (including alignment) would be dangerous. The overwhelmingly likely result of such a training attempt (by default, i.e., in the absence of specific countermeasures that there are currently few ideas for) would be the AI taking on a dangerous degree of convergent instrumental subgoals while not internalizing important safety/corrigibility properties enough.

I think this is possible, but much less likely than Nate thinks under at least some imaginable training processes.

I didn't end up agreeing that this difficulty is as important as Nate thinks it is, although I did update my views some (more on that below). My guess is that this is one of the two biggest disagreements I have with Nate's and Eliezer's views (the other one being the likelihood of a sharp left turn that leads to a massive capabilities gap between AI systems and their supervisors.2)

Below is my summary of:

Some key premises we agree on.

What we disagree about, at a high level.

A hypothetical training process we discussed in order to get more clear and mechanistic about Nate's views.

Some brief discussion of possible cruxes; what kind of reasoning Nate is using to arrive at his relatively high (~85%) level of confidence on this point; and future observations that might update one of us toward the other's views.

MIRI might later put out more detailed notes on this exchange, drawing on all of our discussions over Slack and comment threads in Google docs.

Nate has reviewed this post in full. I'm grateful for his help with it.

Some starting points of agreement

Nate on this section: “Seems broadly right to me!”

An AI is dangerous if:

It's powerful (like, it has the ability to disempower humans if it's "aiming" at that)

It aims (perhaps as a side effect of aiming at something else) at C

...
What Discovering Latent Knowledge Did and Did Not Find
by Fabien Roger — 2023-03-13T19:29:45.601Z

Thanks to Marius Hobbhahn and Oam Patel for helpful feedback on drafts. Thanks to Collin and Haotian for answering many questions about their work.

Discovering Latent Knowledge in Language Models Without Supervision

describes Contrast-Consistent Search (CCS), a method to find a classifier which accurately answers yes-no questions given only unlabeled model activations. It might be a stepping stone towards recovering superhuman beliefs of AI systems, as unsupervised methods are more scalable and might be less likely to simply recover “what a human would say”.

I think this research direction is interesting and promising. But I feel like people often got a bit carried away with the approximate takeaway they got from the experimental results of the initial paper.

In this post, I present experimental results which highlight the strengths and weaknesses of CCS.

CCS is able to find a single linear probe which correctly classifies statements across datasets, and it doesn’t hurt performance;
CCS does so better than random, but not by a huge margin: on average, random linear probes have a 75% accuracy on some “easy” datasets;
CCS does not find the single linear probe with high accuracy: there are more than 20 orthogonal linear probes (i.e. using completely different information) that have similar accuracies as the linear probe found by CCS (for most datasets);
CCS does not always find a probe with low test CCS loss (Figure 1 of the paper is misleading). CSS finds probes which are sometimes overconfident in inconsistent predictions on the test set, resulting in a test loss that is sometimes higher than always predicting a constant probability;
CCS’ performance on GPT-J heavily depends on the last tokens of the input, especially when looking at the last layers’ activations (the setting used in the paper).

Main takeaways:

CCS does not simply find weird heuristics on each dataset independently, i.e. it finds some feature or property that is shared between datasets. However, we still don’t know if this feature corresponds to the model’s “beliefs”.
Future work should compare their work against the random probe baseline. Comparing to a 50% random guessing baseline is misleading, as random probes have higher accuracy than that.
CCS will likely miss important information about the model’s beliefs because there is more than one linear probe

...
Could Roko's basilisk acausally bargain with a paperclip maximizer?
by Christopher King — 2023-03-13T18:21:46.722Z

The idea of Roko's basilisk is that it's a friendly AI that acausally blackmails humans into working on alignment.

This was judged to not be effective because humans were too dumb to acausally bargain and there are acausal defenses to blackmail.

However, what if instead Roko's basilisk acausually bargained with unaligned AIs, like paperclip maximizers?

In particular, we imagine that Roko's basilisk would simulate many AI's. If they spare humans, then the basilisk will devote some of the light cone to maximizing the simulated AI's utility.

Now, the paperclip maximizer could reason acausally (EDIT: I'm still a bit fuzzy on this due to the simulation component, but Radford Neal is saying this argument should also work for a causal decision theorist!):

I might be in a simulation of a more powerful AI.

Roko's basilisk is a slightly more likely candidate, since humans have at least a small bias towards building friendly AI v.s. any other specific utility maximizer.

Therefore, I ought to spare the humans and their solar system while turning the rest of the universe into paperclips. The gain from Roko's basilisk liking me is much greater than a single solar system's worth of paperclips.

A problem I see though is that if a large number of AIs comply, Roko's basilisk might not have enough "universe" to acausally appease them all. This is on top of the fact that it's already fighting the inductive bias against simulation and the potentially low probability that humans solve alignment 🤔.

How should Roko's basilisk be designed so as to acausally save humanity? (Perhaps it should focus on the most likely "counterfactual" unaligned AIs?)
Bayesian optimization to find molecules that bind to proteins
by rotatingpaguro — 2023-03-13T18:17:44.812Z
Link: https://www.youtube.com/watch?v=ZkPBb_YW8Ag

Paper: Maus et al., "Scalable Deep Bayesian optimization over Structured Inputs," arXiv:2201.11872.

A nice and interesting presentation about finding synthetizable molecules that bind to a given protein with Bayesian optimization.

An autoencoder is trained on a database of molecules. The latent space of the autoencoder is fed as input space to an optimizer. The key idea is training the autoencoder together with the surrogate model used by optimizer, such that the latent space makes the target function to maximize simpler.

This technique works well. As expected, the powerful optimizer hacks the target function designed by humans and finds weird molecules that produce huge scores.
Linkpost: ‘Dissolving’ AI Risk – Parameter Uncertainty in AI Future Forecasting
by DavidW — 2023-03-13T16:52:19.599Z
Link: https://forum.effectivealtruism.org/posts/Z7r83zrSXcis6ymKo/dissolving-ai-risk-parameter-uncertainty-in-ai-future#5___Analysis

This post argues that Carlsmith's model, and conjunctive AI doom estimates in general, neglect an important statistical consideration that results in an inflated estimate of AI X-risk.
Decentralized Exclusion
by jefftk — 2023-03-13T15:50:01.710Z

I'm part of several communities that are relatively decentralized.
For example, anyone can host a contra dance, rationality meetup, or
effective altruism dinner. Some have central organizations (contra has

CDSS, EA has

CEA) but their
influence is mostly informal. This structure has some benefits (lower
overhead, more robust) but one drawback is in handling bad behavior.
If several people reported very bad experiences with someone at my
local dance

we'd
kick them out, but that wouldn't keep them from harming others at,
say, any of the

hundreds of other
events run by other organizations.

I have seen cases, though, where someone was fully removed from a
decentralized community. Looking at why these cases succeeded and
others failed, I think it took:

Clear misbehavior, in a way that nearly everyone would agree if
they looked into it.

Detailed public accusations, so people can look into it if they
doubt the consensus.

The combination of these means that you can have an initial burst of
'drama' in which lots of people learn about the situation and agree
that the person should be kicked out, and then this can be maintained
whenever they show up again. For example:

2016: Gleb Tsipursky from the EA community, for a range of
shady things and a pattern of apologizing and then continuing (details).

2017: Jordy Williams from contra dance, after accusations of
grooming and rape (details).

2018: Brent Dill, from the rationality community after
accusations of sexual abuse, gaslighting, and more (details).

Unfortunately this approach relies on people making public
accusations, which is really hard. We should support people when they
do and recognize their
bravery, but people will often have valid reasons why they won't:
fear of retaliation, unwilling to have that level of public scrutiny,
risk of legal action. In those cases it's still possible to make some
progress privately, and we definitely need to try, but you keep
bumping into the limitations
of decentralization
and defamation law.

To clarify why I think (1) and (2) are key for community-wide
exclusion, however, let's look at two cases where this has been tried
but was only partially successful. Within EA I've seen people apply
this approach to Jacy Reese, but because the details of what he
was apologizing
for are vague he's only mostly kicked out and people
are...
Linkpost: A Contra AI FOOM Reading List
by DavidW — 2023-03-13T14:45:57.695Z
Link: https://magnusvinding.com/2017/12/16/a-contra-ai-foom-reading-list/

This is a linkpost to a list of skeptical takes on AI FOOM. I haven't read them all and probably disagree with some of them, but it's valuable to put these arguments in one place.
Linkpost: A tale of 2.5 orthogonality theses
by DavidW — 2023-03-13T14:19:16.688Z
Link: https://forum.effectivealtruism.org/posts/kCAcrjvXDt2evMpBz/a-tale-of-2-5-orthogonality-theses

Creating a link post to an important argument about a potential motte and bailey fallacy used in the AI alignment community.
Plan for mediocre alignment of brain-like [model-based RL] AGI
by Steven Byrnes — 2023-03-13T14:11:32.747Z

(This post is a more simple, self-contained, and pedagogical version of

Post #14

of

Intro to Brain-Like AGI Safety

.)

(Vaguely related to

this Alex Turner post

and

this John Wentworth post

.)

I would like to have a technical plan for which there is a strong robust reason to believe that we’ll get an aligned AGI and a good future. This post is not such a plan.

However, I also don’t have a strong reason to believe that this plan wouldn’t work. Really, I want to throw up my hands and say “I don’t know whether this would lead to a good future or not”. By “good future” here I don’t mean optimally-good—whatever that means—but just “much better than the world today, and certainly much better than a universe full of paperclips”. I currently have no plan, not even a vague plan, with any prayer of getting to an optimally-good future. That would be a much narrower target to hit.

Even so, that makes me more optimistic than at least some people.[1] Or at least, more optimistic about this specific part of the story. In general I think many things can go wrong as we transition to the post-AGI world—see discussion by

Dai

&

Soares

—and overall I feel very doom-y, particularly for reasons

here

.

This plan is specific to the possible future scenario (a.k.a. “threat model” if you’re a doomer like me) that future AI researchers will develop “brain-like AGI”, i.e. learning algorithms that are similar to the brain’s within-lifetime learning algorithms. (

I am

not

talking about evolution-as-a-learning-algorithm

.) These algorithms,

I claim

, are in the general category of model-based reinforcement learning. Model-based RL is a big and heterogeneous category, but I suspect that for any kind of model-based RL AGI, this plan would be at least somewhat applicable. For very different technological paths to AGI, this post is probably pretty irrelevant.

But anyway, if someone published an algorithm for x-risk-capable brain-like AGI tomorrow, and we urgently needed to do something, this blog post is more-or-less what I would propose to try. It’s the least-bad plan that I currently know.

So I figure it’s worth writing up this plan in a more approachable and self-contained format.

1. Intuition: Making a human into a moon-lover (“selenophile”)

Try to think of who is the coolest / highest-status-to-you / biggest-halo-effe...
What is calibration?
by AlexMennen — 2023-03-13T06:30:53.431Z

A forecaster is well-calibrated if, for every p∈[0,1], of the propositions that they assign probability approximately p to, the fraction of them that are true is approximately p. However, there is no natural probability distribution over propositions, so this notion is not well-defined.

Often, people aren't even using an implicit probability distribution over propositions when they talk about calibration, and instead are refering to limiting densities over a particular sequence of propositions. For instance, a forecaster may be asked to predict every bit in a bitstream, and be judged well-calibrated if for every p∈[0,1], the fraction of the first n of the propositions that they assign probability approximately p to that are true approximately converges to p as n goes to infinity.

Calibration is not just a relationship between probability assignments and the truth, but a relationship between probability assignments, the truth, and some model for what it means to say that some percentage of a set of propositions is true. This model could be a probability distribution over propositions, or an explicit sequence of them. The dependence on what you mean by percentage of propositions is fairly dramatic.

For any atomless probability measure, you can pick a sequence of propositions such that, in terms of limiting frequencies along the sequence, the probability distribution is guaranteed to be well-calibrated, no matter what the ground truth is. To make a sequence of propositions, all of which are given probability nm∈Q∩[0,1], and limiting frequency nm of which are true: First pick a sequence (Xi)i∈N of independent random variables, where each Xi is uniformly distributed on {1,...,m}. For each i∈N and S⊆{1,...,m} with |S|=n, let Pi,S be the proposition that Xi∈S. For each i, exactly nm of the propositions {Pi,S∣S⊆{1,...,m},|S|=n} are true, no matter what Xi actually is. So if you list {Pi,S∣i∈N,S⊆{1,...,m},|S|=n} in order of increasing i, then the fraction of them that are true converges to nm as you go along the list. Now you can create sequences of propositions with varying probabilities assigned to them, and well-calibrated on all probability levels, by intersp...
On taking AI risk seriously 
by Eleni Angelou — 2023-03-13T05:50:57.170Z
Link: https://www.nytimes.com/2023/03/12/opinion/chatbots-artificial-intelligence-future-weirdness.html

Crossposted from the EA Forum: https://forum.effectivealtruism.org/posts/pKG5fsfrgDSQtssfu/on-taking-ai-risk-seriously

Yet another New York Times piece on AI. A non-AI safety friend sent it to me saying "This is the scariest article I've read so far. I'm afraid I haven't been taking it very seriously". I'm noting this because I'm always curious to observe what moves people, what's out there that has the power to change minds. In the past few months, there's been increasing public attention to AI and all sorts of hot and cold takes, e.g., about intelligence, consciousness, sentience, etc. But this might be one of the articles that convey the AI risk message in a language that helps inform and think about AI safety.

The following is what stood out to me and made me think that it's time for philosophy of science to also take AI risk seriously and revisit the idea of scientific explanation given the success of deep learning:

I cannot emphasize this enough: We do not understand these systems, and it’s not clear we even can. I don’t mean that we cannot offer a high-level account of the basic functions: These are typically probabilistic algorithms trained on digital information that make predictions about the next word in a sentence, or an image in a sequence, or some other relationship between abstractions that it can statistically model. But zoom into specifics and the picture dissolves into computational static.

“If you were to print out everything the networks do between input and output, it would amount to billions of arithmetic operations,” writes Meghan O’Gieblyn in her brilliant book, “God, Human, Animal, Machine,” “an ‘explanation’ that would be impossible to understand.”
Nose / throat treatments for respiratory infections
by juliawise — 2023-03-13T02:41:37.848Z

After hearing various people interested in nasal sprays for preventing covid or the common cold, I did a shallow dive on what might be worth trying in this area.

Overall, there are several treatments that seem safe and cheap and some of them are probably effective. A few (carrageenan and HPMC) seem to prevent infection, and the others reduce the severity of an infection once you have it. Some may also reduce your risk of infecting others if you’re already sick.

Povidone iodine
Nitric oxide
Nasal washing (neti pot or various other devices)
Hydroxypropyl methyl cellulose (HPMC)
Carrageenan
Xylitol
Saline spray / wash
Various mouthwashes
Alcohol
Other stuff being researched but not on the market as far as I know
Side effects
Best acronym
Personal takeaways

Obvious disclaimer: None of this replaces a vaccine or medical care. But it would be awesome if there were fewer colds in my house, if we could all get over them faster, and if we had gotten over covid sooner.

I looked at ingredients you can currently buy (in some country), not stuff that’s not on the market. In some cases you can get them e.g. on Ebay from other countries where they’re available without prescription. I only looked at things with human trials, not things with only in vitro or animal trials.

My understanding is that treatments aimed at the nose and throat can make sense because that’s where upper respiratory infections often first take hold. E.g. covid reproduces in nasal cilia cells for the first 24 hours after exposure before spreading to the rest of the body. A lot of antiviral treatments are known to kill covid in a petri dish, so they might also kill it in your nose and throat.

A lot of the research here is small preliminary studies on covid, often without randomization or control groups. I’m frustrated that in 3 years there haven’t been larger, better studies published – though there are some in the works. I suspect part of this is that the ingredients are cheap and unpatentable, so no one is going to make a lot of money off them. Some of the studies are funded by the companies making the product.

If you’re excited to dig into this more, I would love for you to make this review better!

I would also be excited to see more research on some of these cheap treatments, if anyone reading has the power to make that happen.

Povidone iodine

Povidone iodine has long been used as an antiseptic, and is consid...
Gold, Silver, Red: A color scheme for understanding people
by Michael Soareverix — 2023-03-13T01:06:39.703Z

A few months ago, I built a predictive algorithm to determine the probability of romantic relationships between individuals in my periphery.

The algorithm works like this:

-Personal Factors. How likely are you as an individual to find yourself in a new relationship? This contains variables like initiative, expectations for a partner, current default emotional state, comfortability around those you are attracted to, experience in prior relationships, height, etc.

-Contextual Factors. What is your environment like? I'd noticed that first-years in college were vastly more likely to get into relationships versus other years. This contains stuff like whether you are currently interested in someone, whether you think they're interested in you, how many classes you're in, and how many new people you interact deeply with per week.

-Compatibility Factors. This is an extra factor that only happens if you know who you are interested in and you have information about them and their history. Do they have a type? Do you match that type? MBTI info?

I got the info from this Google Form and turned it into a spreadsheet. Then I put the spreadsheet into Jupyter-Notebook, where a predictive transformer from the DARTS data science package ran over it and put out a number that was supposed to represent the odds of two people ending up in a relationship together.

Needless to say, this is a terrible way to build an actual algorithm or train a neural network but I'm just a curious college student and I wanted a minimum viable product.

So I started running it (with permission) on most of my friends. It predicted a lot of interesting relationships but I kept fine-tuning it. Whenever a new relationship cropped up, I would go to the two people and ask to add their data into the algorithm. Fortunately, they humored me most of the time and the algorithm seemed to get better.

I still don't know if the algorithm actually works. That's not the point of this post. The point is that I, as a human being, developed the ability to see when a relationship was imminent. Currently, I can generally tell when two people are going to get into a relationship about two months before they start dating.

While looking over the variables I used to predict people's relationship success, I found common themes. Very charming, intense, or charismatically easygoing people constantly found themselves in new relationships, ...