trentmkelly/LessWrong-43k huggingface dataset:
Palisade is hiring Research Engineers
by Charlie Rogers-Smith — 2023-11-11T03:09:54.575Z

Palisade is looking to hire Research Engineers. We are a small team consisting of Jeffrey Ladish (Executive Director), Charlie Rogers-Smith (Chief of Staff), and Kyle Scott (part-time Treasurer & Operations). In joining Palisade, you would be a founding member of the team, and would have substantial influence over our strategic direction. Applications are rolling, and you can fill out our short (~10-20 minutes) application form

here

.

Palisade’s mission

We research dangerous AI capabilities to better understand misuse risks from current systems, and how advances in hacking, deception, and persuasion will affect the risk of catastrophic AI outcomes. We create concrete demonstrations of dangerous capabilities to advise policy makers and the public on AI risks.

We are working closely with government agencies, policy think tanks, and media organizations to inform relevant decision makers. For example,

our BadLlama

work

demonstrated that it is possible to effectively undo Llama 2-Chat 70B’s safety fine-tuning for less than $200, and has been used to

confront

Mark Zuckerberg in the first of Chuck Schumer’s Insight Forums, cited by Senator Hassan in a

senate hearing

on threats to national security, and used to advise the UK AI Safety Institute.

While our BadLlama work focused on validating a well-known argument—that you can fine-tune away safety—we plan to research emerging dangerous capabilities in both open source and API-gated models, in the following areas:

Automated hacking. Current AI systems can already automate parts of the

cyber kill chain

. We’ve demonstrated that GPT-4 can leverage known vulnerabilities to achieve remote code execution on unpatched Windows 7 machines. We plan to explore how AI systems could conduct reconnaissance, compromise target systems, and use information from compromised systems to pivot laterally through corporate networks or carry out social engineering attacks.
Spear phishing and deception. Preliminary

research

suggests that LLMs can be effectively used to phish targets. We’re currently exploring how well AI systems can scrape personal information and leverage it to craft scalable spear-phishing campaigns. We also plan to study how well conversational AI systems could build rapport with targets to convince them to reveal information or take actions contrary to their interests.
Scalable disinformat

...
Open Phil releases RFPs on LLM Benchmarks and Forecasting
by LawrenceC — 2023-11-11T03:01:09.526Z
Link: https://www.openphilanthropy.org/research/rfps-on-llm-impacts/

As linked at the top of Ajeya's "do our RFPs accelerate LLM capabilities" post, Open Philanthropy (OP) recently released two requests for proposals (RFPs):

An RFP on LLM agent benchmarks: how do we accurately measure the real-world, impactful capabilities of LLM agents?
An RFP on forecasting the real world-impacts of LLMs: how can we understand and predict the broader real-world impacts of LLMs?

Note that the first RFP is both significantly more detailed and has narrower scope than the second one, and OP recommends you apply for the LLM benchmark RFP if your project may be a fit for both.

Brief details for each RFP below, though please read the RFPs for yourself if you plan to apply.

Benchmarking LLM agents on consequential real-world tasks

Link to RFP: https://www.openphilanthropy.org/rfp-llm-benchmarks

We want to fund benchmarks that allow researchers starting from very different places to come to much greater agreement about whether extreme capabilities and risks are plausible in the near-term. If LLM agents score highly on these benchmarks, a skeptical expert should hopefully become much more open to the possibility that they could soon automate large swathes of important professions and/or pose catastrophic risks. And conversely, if they score poorly, an expert who is highly concerned about imminent catastrophic risk should hopefully reduce their level of concern for the time being.

In particular, they're looking for benchmarks with the following three desiderata:

Construct validity: the benchmark accurately captures a potential real-world, impactful capability of LLM agents.
Consequential tasks: the benchmark features tasks that will have massive economic impact or can pose massive risks.
Continuous scale: the benchmark improves relatively smoothly as LLM agents improve (that is, they don't go from ~0% performance to >90% like many existing LLM benchmarks have).

Also, OP will do a virtual Q&A session for this RFP:

We will also be hosting a 90-minute webinar to answer questions about this RFP on Wednesday, November 29 at 10 AM Pacific / 1 PM Eastern (link to come).

Studying and forecasting the real-world impacts of systems built from LLMs

Link to RFP: https://www.openphilanthropy.org/rfp-llm-impacts/

This RFP is significantly less detailed, and primarily consists of a list of projects that OP may be willing to fund:

To this end, in additio

...
Memo on some neglected topics
by Lukas Finnveden — 2023-11-11T02:01:55.834Z
Link: https://open.substack.com/pub/lukasfinnveden/p/memo-on-some-neglected-topics

I originally wrote this for the

Meta Coordination Forum

. The organizers were interested in a memo on topics other than alignment that might be increasingly important as AI capabilities rapidly grow — in order to inform the degree to which community-building resources should go towards AI safety community building vs. broader capacity building. This is a lightly edited version of my memo on that. All views are my own.

Some example neglected topics (without much elaboration)

Here are a few example topics that could matter a lot if we’re in

the most important century

, which aren’t always captured in a normal “AI alignment” narrative:

The potential moral value of AI. [1]
The potential importance of making AI behave cooperatively towards humans, other AIs, or other civilizations (whether it ends up intent-aligned or not).
Questions about how human governance institutions will keep up if AI leads to explosive growth.
Ways in which AI could cause human deliberation to get derailed, e.g. powerful persuasion abilities.
Positive visions about how we could end up on a good path towards becoming a society that makes wise and kind decisions about what to do with the resources accessible to us. (Including how AI could help with this.)

(More elaboration on these

below

.)

Here are a few examples of somewhat-more-concrete things that it might (or might not) be good for some people to do on these (and related) topics:

Develop proposals for how labs could treat digital minds better, and advocate for them to be implemented. (C.f.

this nearcasted proposal

.)
Advocate for people to try to avoid building AIs with large-scale preferences about the world (at least until we better understand what we’re doing). In order to avoid a scenario where, if some generation of AIs turn out to be sentient and worthy of rights, we’re forced to choose between “freely hand over political power to alien preferences” and “deny rights to AIs on no reasonable basis”.
Differentially accelerate AI being used to improve our ability to find the truth, compared to being used for propaganda and manipulation.

E.g.: Start an organization that uses LLMs to produce epistemically rigorous investigations of many topics. If you’re the first to do a great job of this, and if you’re truth-seeking and even-handed, then you might become a trusted source on controversial topics. And your investigati

...
Who is Sam Bankman-Fried (SBF) really, and how could he have done what he did? - three theories and a lot of evidence
by spencerg — 2023-11-11T01:04:22.747Z
Link: https://www.spencergreenberg.com/2023/11/who-is-sam-bankman-fried-sbf-really-and-how-could-he-have-done-what-he-did-three-theories-and-a-lot-of-evidence/

For the full version of this essay, go here.

As you may know, Sam Bankman-Fried ("SBF") was convicted of seven counts of fraud and conspiracy. He now faces the potential of more than 100 years in prison.

I've been trying to figure out how someone who appears to believe deeply in the principles of effective altruism could do what SBF did. It has been no surprise to me to see that the actions he was convicted of are nearly universally condemned by the EA community. Could it be that he did not actually believe in EA ideas despite promoting EA and claiming to believe in it? If he does believe in EA principles, there seems to be a genuine mystery here as to why he took those actions.

There are a few theories that could potentially explain the seeming mystery. In this post, I'll discuss the strongest evidence I've been able to find for and against each of the three theories that I find most plausible.

It seems important to me to seek an understanding of the deeper causes of this disaster to help prevent future such disasters. It also seems to me to be essential for the EA community, in particular, to understand why this happened. An understanding of the disaster and the person behind it might be necessary (though probably not sufficient) for the community to prevent similar events from happening in the future.

A few important things before we begin the analysis

In this piece, I assume that SBF committed all the crimes that he was convicted of. If it somehow turns out that SBF isn't guilty of these crimes, then some parts of this post would not apply (and you should consider most of this post withdrawn).

It's also important to note that the opinions I express in this post are, for the most part, informed by studying publicly available details about SBF and the FTX collapse, as well as confidential conversations I've had with a number of different people who knew SBF (some who worked with him, some who knew him as a friend). I promised confidentiality to these people to help them be more comfortable sharing information honestly with me, so I won't use their names or other indications of how they know him. I shared this post with them prior to publishing it to help reduce the chance that I introduced errors in what they said to me.

I've also pulled quotes from the new book about SBF, Going Infinite, and from podcast interviews with its author, Michael Lewis. Lewi...
Survey on the acceleration risks of our new RFPs to study LLM capabilities
by Ajeya Cotra — 2023-11-10T23:59:52.515Z

My team

at Open Philanthropy just launched two requests for proposals:

Proposals to

create benchmarks

measuring how well

LLM agents

(like

AutoGPT

) perform on difficult real-world tasks, similar to

recent work by ARC Evals.

[1]
Proposals to

study and/or forecast

the near-term real-world capabilities and impacts of LLMs and systems built from LLMs more broadly.

I think creating a shared scientific understanding of where LLMs are at has large benefits, but it can also accelerate AI capabilities: for example, it might demonstrate possible commercial use cases and spark more investment, or it might allow researchers to more effectively iterate on architectures or training processes. Other things being equal, I think acceleration is harmful because

we’re not ready

for very powerful AI systems — but I believe the benefits outweigh these costs in expectation, and think better measurements of LLM capabilities are net-positive and important.

To get a sense for whether acting on this belief by launching these two RFPs would constitute falling prey to

the unilateralist’s curse

, I sent

a survey

about whether funding this work would be net-positive or net-negative to 47 relatively senior people who have been full-time working on AI x-risk reduction for multiple years and have likely thought about the risks and benefits of sharing information about AI capabilities.

Out of the 47 people who received the survey, 30 people (64%) responded. Of those, 25 out of 30 said they were “Positive” or “Lean positive” on the RFP, and only 1 person said they were “Lean negative,” with no one saying they were “Negative.” The remaining four people said they had “No idea,” meaning that 29 out of 30 respondents (97%) would not vote to stop the RFPs from happening. With that said, many respondents (~37%) felt torn about the question or considered it complicated.

The rest of this post provides more detail on

the information that the survey-takers received

and

the survey results

(including sharing answers from those respondents who gave permission to share).

The information that was sent to the survey-takers

The survey-takers received the below email, which links to a

one-pager

on the risks and benefits of these RFPs, and a

four-pager

(written in late July and early August) about the sorts of projects I expected ...
How I Think, Part Three: Weighing Cryonics
by Richard Henage — 2023-11-10T22:21:16.398Z

These are things I've been thinking about when trying to decide if I should get cryonics. I've sometimes gotten the sense that some rationalists think this is a no-brainer. But since it's a $100k+ decision, I think it deserves quite a bit of thinking. So here is my list of thoughts as I think them:

1: Even if there's not a huge chance it will work, it's probably still worth it. Let's say there's a 10% chance I think it will work. Wouldn't I pay ten times the price to get it as a guarantee?

2: What would my quality of life be after waking up if it did work? There's a chance it would be very, very high.

3: What would I spend the money on if I didn't spend it on this? I'm not sure, really.

4: There's a chance that scientific progress will achieve immortality before I die anyway.

5: There's a chance I'll die tomorrow

6: There's a chance that I'll die the normal way, and then scientific progress will eventually be able to wake me up anyway

7: There's a chance that there actually is an afterlife and it's great, and I don't have to worry about dying anyway

8: There's a chance that there's an afterlife and it is absolutely terrible, in which case I should try even harder to not die

9: Would cryonics prevent/delay an afterlife anyway?

10: How long would I want to live anyway if I was given the choice? It sounds like some people accept the fact that they will die when they get old, maybe they even look forward to it. Maybe that's just because they are in pain? Or because of other problems with life that are solvable? Or just because they've been conditioned to do that somehow?

11: There's a chance that the world will end before I die, so cryonics wouldn't help.

12: I only started thinking about cryonics a couple of years ago. Isn't there a big chance I'll think it doesn't make any sense ten or twenty years from now?

13: Can I get a refund if I change my mind about cryonics before dying? If I use a life insurance plan, can I change the beneficiary? If so, it probably makes the most sense to just get the plan now and think about all of this stuff over time.

14: Cryonics is not very popular, in that only a few thousand people have signed up for it and even most scientists seem to think it wouldn't work. Although I believe other things that I think most of the world views as false (atheism, impending ASI, etc) and it's a fallacy or something to just argue that the majority of people ar...
Linear encoding of character-level information in GPT-J token embeddings
by mwatkins — 2023-11-10T22:19:14.654Z

This post was primarily authored by MW, reporting recent research which was largely directed by suggestions from and discussions with JB. As secondary author, JB was responsible for the general shape of the post and the Future Directions section.

Work supported by the Long Term Future Fund (both authors) and Manifund (JB).

Links: Github repository, Colab notebook

TL;DR
Previous results have shown that character-level information, such as whether tokens contain a particular letter, can be extracted from GPT-J embeddings with non-linear probes. We show that linear probes can retrieve character-level information from embeddings and we perform interventional experiments to show that this information is used by the model to carry out character-level tasks. This sheds new light on how "character blind" LLMs are able to succeed on character-related tasks like spelling. As spelling represents a foundational and unambiguous form of representation, we suspect that unraveling LLMs' approach to spelling tasks could yield precise insights into the mechanisms whereby these models encode and interpret much more general aspects of the world.

Building on much previous work in this area, we look forward to extending our understanding of linearly encoded information in LLM embedding spaces (which appears to go far beyond mere character-level information). This may include learning how such information (i) emerges during training ; (ii) is predictive of the abilities of different models; and (iii) can be leveraged to better understand the algorithms implemented by these models.

Letter presence: Linear probes vs MLPs

The so-called "spelling miracle"[1] is the fact that language transformers like GPT-3 are capable of inferring character-level information about their tokens, as discussed in an earlier post. Among the very sparse literature on the topic, Kaushal & Mahowald's "What do tokens know about their characters and how do they know it?" (2022) describes probe-based experiments showing that (along with other LLMs) GPT-J's token embeddings nonlinearly encode knowledge of which letters belong to their token strings. The authors trained 26 MLP probes, one to answer each of the binary questions of the form "Does the string '<token>' contain the letter 'K'?" Their resulting classifiers were able to achieve F1 scores ranging from about 0.83 to 0.94,...
Follow-up survey: inositol
by Elizabeth — 2023-11-10T19:30:02.737Z

Two months ago I wrote about inositol as a treatment that occasionally works for anxiety and depression, especially when the user also has weird digestive issues (not medical advice, I am not a doctor). If that inspired you to try inositol I would love if you would fill out this 5-7 question survey about your experience. This follow-up data helps other people considering inositol, and is broadly helpful to me in figuring out what luck based medicine looks like.

And to the four people who already filled this out: gold star for epistemic virtue.

The survey doesn’t allow for a lot of detail, which I know is painful for some people (it’s me. I’m people). If you would like to share more, feel free to write up as much as you’d like in a comment here, or share a link detailing your experience.

Elsewhere in luck based medicine: it was a dude in my survey, but I met a few more people who really love the Apollo Neuro. They are all the kind of people who already know what “somatically aware” or “embodiment” mean, so this is some support for my theory that that’s a prerequisite. It’s still an open question if you need that background for the Neuro to be beneficial, to notice it’s beneficial, or to stick with it long enough that it has time to be beneficial.

The Apollo app has gotten even worse since last time I wrote. Every time I open it it bugs me to enable notifications, a permission it absolutely does not need.
We have promising alignment plans with low taxes
by Seth Herd — 2023-11-10T18:51:38.604Z

Epistemic status: I’m sure these plans have advantages relative to other plans. I'm not sure they're adequate to actually work, but I think they might be.

With good enough alignment plans, we might not need coordination to survive. If alignment taxes are low enough, we might expect most people developing AGI to adopt them voluntarily. There are two alignment plans that seem very promising to me, based on several factors, including ease of implementation, and applying to fairly likely default paths to AGI. Neither has received much attention. I can’t find any commentary arguing that they wouldn't work, so I’m hoping to get them more attention so they can be considered carefully and either embraced or rejected.

Even if these plans[1] are as promising as I think now, I’d still give p(doom) in the vague 50% range. There is plenty that could go wrong.[2]

There's a peculiar problem with having promising but untested alignment plans: they're an excuse for capabilities to progress at full speed ahead. I feel a little hesitant to publish this piece for that reason, and you might feel some hesitation about adopting even this much optimism for similar reasons. I address this problem at the end.

The plans

Two alignment plans stand out among the many I've found. These seem more specific and more practical than others. They are also relatively simple and obvious plans for the types of AGI designs they apply to. They have received very little attention since being proposed recently. I think they deserve more attention.

The first is Steve Byrnes’

Plan for mediocre alignment of brain-like [model-based RL] AGI

. In this approach, we evoke a set of representations in a

learning subsystem

, and set the weights from there to the

steering

or critic subsystems. For example, we ask the agent to "think about human flourishing" and then freeze the system and set high weights between the active units in the learning system/world model and the steering system/critic units. The system now ascribes high value to the distributed concept of human flourishing. (at least as it understands it). Thus, the agent's knowledge is used to define a goal we like.

This plan applies to all RL systems with a critic subsystem, which includes most powerful RL systems.[3] RL agents (including loosely brain-like systems of deep networks) seem like one very plausible route to AGI. I per...
Vector search on a large dataset?
by camsdixon — 2023-11-10T18:43:30.998Z

Does anyone have advice or could recommend books on how to accomplish vector search with large datasets?

What I've found so far is vector DB's are very bad at larger datasets, even to the order of 10,000's of vectors if the data is similar enough. Some ideas we've gone through so far:

Smaller chunks & larger chunks -> Spacy & others
Summarizing chunks using the summarization as an embedding and the actual chunk as the metadata
Traditional search with vector search on top ( this seems the best ) but still runs into issues with chunks cutting off at the wrong locations.
Of course all the "traditional" systems like langchain etc.

Any help, ideas, or recommendations on where I can read would be very much appreciated!