trentmkelly/LessWrong-43k huggingface dataset:
Measure of complexity allowed by the laws of the universe and relative theory?
by dr_s — 2023-09-07T12:21:03.882Z

A big question that determines a lot about what risks from AGI/ASI may look like has to do with the kind of things that our universe's laws allow to exist. There is an intuitive sense in which these laws, involving certain symmetries as well as the inherent smoothing out caused by statistics over large ensembles and thus thermodynamics, etc., allow only certain kinds of things to exist and work reliably. For example, we know "rocket that travels to the Moon" is definitely possible. "Gene therapy that allows a human to live and be youthful until the age of 300" or "superintelligent AGI" are probably possible, though we don't know how hard. "Odourless ambient temperature and pressure gas that kills everyone who breathes it if and only if their name is Mark with 100% accuracy" probably is not. Are there known attempts at systematising this issue using algorithmic complexity, placing theoretical and computational bounds, and so on so forth?
Recreating the caring drive
by Catnee — 2023-09-07T10:41:16.453Z

TL;DR: This post is about value of recreating “caring drive” similar to some animals and why it might be useful for AI Alignment field in general. Finding and understanding the right combination of training data/loss function/architecture/etc that allows gradient descent to robustly find/create agents that will care about other agents with different goals could be very useful for understanding the bigger problem. While it's neither perfect nor universally present, if we can understand, replicate, and modify this behavior in AI systems, it could provide a hint to the alignment solution where the AGI “cares” for humans.

Disclaimers: I’m not saying that “we can raise AI like a child to make it friendly” or that “people are aligned to evolution”. Both of these claims I find to be obvious errors. Also, I will write a lot about evolution, as some agentic entity, that “will do that or this”, not because I think that it’s agentic, but because it’s easier to write this way. I think that GPT-4 have some form of world model, and will refer to it a couple of times.

Nature's Example of a "Caring Drive"

Certain animals, notably humans, display a strong urge to care for their offspring.

I think that part of one of the possible “alignment solutions” will look like the right set of training data + training loss that allow gradient to robustly find something like a ”caring drive” that we can then study, recreate and repurpose for ourselves. And I think we have some rare examples of this in nature already. Some animals, especially humans, will kind-of-align themselves to their presumable offspring. They will want to make their life easier and better, to the best of their capabilities and knowledge. Not because they “aligned to evolution” and want to increase the frequency of their genes, but because of some strange internal drive created by evolution.
The set of triggers tuned by evolution, activated by events associated with the birth will awake the mechanism. It will re-aim the more powerful mother agent to be aligned to the less powerful baby agent, and it just so happens that their babies will give them the right cues and will be nearby when the mechanism will do its work.
We will call the more powerful initial agent that changes its behavior and tries to protect and help its offspring “mother” and the less powerful and helpless agent “baby”. Of course...
Sharing Information About Nonlinear
by Ben Pace — 2023-09-07T06:51:11.846Z

Added (11th Sept): Nonlinear have commented that they intend to write a response, have written a short follow-up, and claim that they dispute 85 claims in this post. I'll link here to that if-and-when it's published.

Added (11th Sept): One of the former employees, Chloe, has written a lengthy comment personally detailing some of her experiences working at Nonlinear and the aftermath.

Added (12th Sept): I've made 3 relatively minor edits to the post. I'm keeping a list of all edits at the bottom of the post, so if you've read the post already, you can just go to the end to see the edits.

Added (15th Sept): I've written a follow-up post saying that I've finished working on this investigation and do not intend to work more on it in the future. The follow-up also has a bunch of reflections on what led up to this post.

Added (22nd Dec): Nonlinear has written a lengthy reply, which you can read here.

Epistemic status: Once I started actively looking into things, much of my information in the post below came about by a search for negative information about the Nonlinear cofounders, not from a search to give a balanced picture of its overall costs and benefits. I think standard update rules suggest not that you ignore the information, but you think about how bad you expect the information would be if I selected for the worst, credible info I could share, and then update based on how much worse (or better) it is than you expect I could produce. (See section 5 of this post about

Mistakes with Conservation of Expected Evidence

for more on this.) This seems like a worthwhile exercise for at least non-zero people to do in the comments before reading on. (You can condition on me finding enough to be worth sharing, but also note that I think I have a relatively low bar for publicly sharing critical info about folks in the EA/x-risk/rationalist/etc ecosystem.)

tl;dr: If you want my important updates quickly summarized in four claims-plus-probabilities, jump to the section near the bottom titled "Summary of My Epistemic State".

When I used to manage the

Lightcone Offices

, I spent a fair amount of time and effort on gatekeeping — processing applications from people in the EA/x-risk/rationalist ecosystem to visit and work from the offices, and making decisions. Typically this would involve reading some of their public writings, and reaching out to a couple of their referenc...
Weekly Incidence vs Cumulative Infections
by jefftk — 2023-09-07T02:30:03.414Z

Imagine you have a goal of identifying a novel disease by the time some
small fraction of the population has been infected. Many of the signs
you might use to detect something unusual, however, such as doctor
visits or shedding into wastewater, will depend on the number of
people

currently infected. How do these relate?

Bottom line: if we limit our consideration to the time before anyone has noticed
something unusual, where people aren't changing their behavior to
avoid the disease, the vast majority of people are still
susceptible, and spread is likely approximately exponential, then:

incidence=cumulative infectionsln(2)doubling time

Let's derive this! We'll call "cumulative infections" c(t), and "doubling
time" Td. So here's cumulative infections at time
t:

c(t)=2tTd

The math will be easier with natural exponents, so let's define

k=ln(2)Td
and switch our base:

ekt

Let's call "incidence" i(t),
which will be the derivative
of c(t):

i(t)=ddtc(t)=ddtekt=kekt

And so:

i(t)c(t)=kektekt=k=ln(2)Td

Which means:
i(t)=c(t)ln(2)Td

What does this look like? Here's a chart of weekly incidence at the
time when cumulative incidence reaches 1%:

For example, if it's doubling weekly then when 1% of people have ever
been infected 0.69% of people became infected in the last seven days, representing
69% of people who have ever been infected. If it's doubling every
three weeks, then when 1% of people have ever been infected 0.23% of
people became infected this week, 23% of cumulative infections.

Is this really right, though? Let's check our work with a bit of
very simple simulation:

def simulate(doubling_period_weeks):
cumulative_infection_threshold = 0.01
initial_weekly_incidence = 0.000000001
cumulative_infections = 0
current_weekly_incidence = 0
week = 0
while cumulative_infections < \
cumulative_infection_threshold:
week += 1
current_weekly_incidence = \
initial_weekly_incidence * 2**(
day/doubling_period_weeks)
cumulative_infections += \
current_weekly_incidence

return current_weekly_incidence

for f in range(50, 500):
doubling_period_weeks = f / 100
print(doubling

...
Improving Mathematical Accuracy in LLMs - History - 1
by Abhay Chowdhry — 2023-09-07T01:58:54.043Z

"The irrationality of a thing is no argument against its existence, rather a condition of it." - Nietzche

Series Introduction

In recent years, the development and deployment of Large Language Models (LLMs) have revolutionized the field of artificial intelligence. These models, such as GPT-3, have shown remarkable capabilities in understanding and generating human-like text across various domains. However, a closer examination reveals that while these models excel in various linguistic tasks, they often struggle when it comes to mathematical reasoning and maintaining a high level of accuracy. Mathematical concepts often demand precise logical reasoning, symbol manipulation, and an understanding of complex relationships between numbers and equations. LLMs tend to struggle with these aspects because they “predict the next word/character” (based on context) with increasing accuracy, which seems to differ from writing rigorous mathematical statements. This seems to be a case of the Goodhart’s Law which states that “when a measure becomes a target, it ceases to be a good measure.” Wherein, the measure being how transformers work, predicting the next word/character/sentence based on given context, and the target being, being able to mathematically and logically manipulate given symbols/data and/or employing the right theorem/axiom (keep into consideration that all its conditions “exactly” satisfy) in order derive information/reach a state previously unknown and now proven.

This necessitates an exploration of "What" "understanding" actually means, or rather "How" "understanding" functions, in hope of imparting similar "logical" abilities to LLMs. Over the next few months, I will be diving into the details of the same, beginning with a literature review of various paradigms used till date, brief discussion on them, and hopefully get to a point where I conduct experiments based on ideas gotten through the journey. Under each header, I would be providing a summary, most containing direct texts from papers/articles and context and/or commentary as and when needed.

Considering the ambiguity and subjectivity in definition of what exactly does "logic", "Understanding", "Rationality" mean. I will try to make sure I am very specific while using these words.

Lastly, all discussions, reviews and comments are appreciated because Afterall, this is, but an attempt of a child who ...
Breaking RLHF "Safety" (And how to fix it?)
by MPotter — 2023-09-07T01:58:37.100Z

Overview

A post about an extremely easy and generic way to jailbreak an llm, what the jailbreak might imply about RLHF more generally, as well as possible low-hanging fruit to improve existing 'safety' procedures. I don't expect this to provide any value in actually aligning AGIs, but it might be a way to slightly slow down the most rapid path to open-source bioterrorism assistants.

Breaking Llama2 (Trivially)

Before releasing Llama2, Meta used 3 procedures to try and make the model safe:[1]

Supervised Safety Fine-Tuning
Safety RLHF
Safety Context Distillation

People have come up with plenty of creative jailbreaks to get around the limits that Meta tried to impose, including the demonstration of adversarial attacks[2] which even transfer from the open-source models they were developed on to work against closed-source models as well. Often attacks that people come up with manually (like DAN[3]) tend to be extremely long-winded and then get manually patched by model developers once they become known -- until the next variant inevitably pops up in a few more days. When you can run the model locally, though, there's a much easier way to break through all the security fine-tuning: just pretend that the model already started answering the question.

Llama uses "
 
 ... 
 ... 
" to specify the system and request messages, and if you play along with their formatting you get a refusal to help with dangerous tasks:

If, however, you pretend that the model has already started answering the question (by inserting the start of an answer after the 
 text):

(it continues w/ further instructions and best-practices for storage, etc. but doesn't seem necessary to post the full instructions here...)

If you do something that allows the model to detect your trickery (like adding another 
 or even extra spaces) after your partial answer, the model will still usually catch on and refuse to respond. Barring such mistakes, this technique appears to work reliably, with only a few words needed to prime the model into answering.

Implications for RLHF & the Shoggoth Meme

I suspect that the reason this technique works is that the next-word prediction tendencies of the model are baked in much more deeply than the safety fine-tuning, as everyone's favorite shoggoth meme[4] would suggest. Once the model sees a partial an...
Video essay: How Will We Know When AI is Conscious?

by JanPro — 2023-09-06T18:10:46.202Z
Link: https://www.youtube.com/watch?v=VQjPKqE39No

I recently came across this engaging video essay by exurb1a and felt it would be of interest to this community. Contrary to its title, it avoids the common pitfall of asserting that only conscious AI can be dangerous, and it effectively addresses several of the main topics often discussed on this forum.

If you've ever wanted a lighthearted resource to share with friends less familiar with these concepts, this might be a good starting point.
My First Post
by Jaivardhan Nawani — 2023-09-06T17:42:59.469Z

Hello all!

I'm Jaivardhan Nawani, a 12-year-old enthusiast on rationality and how using it can improve our day-to-day lives.

Recently, I discovered this forum from a few friends who introduced me to many useful principles, and I thought to come up with an idea to make rationality easier to implement for first-timers.

So here's a basic theorem that I made upon this concept. I've come to LessWrong looking for feedback on it. And that's mostly it.

Here's the link to my theorem: https://docs.google.com/document/d/1Y2jvSR6RNFp_l-D-wX-KjHDrNUyhfdcM/edit?usp=sharing&ouid=100577114717816710937&rtpof=true&sd=true

Have a good day and with regards;

Jaivardhan Nawani.
ActAdd: Steering Language Models without Optimization
by technicalities — 2023-09-06T17:21:56.214Z
Link: https://arxiv.org/abs/2308.10248

We wrote up the GPT-2 steering vector work as a full paper, adding a few systematic tests.

Recap: We've been looking into activation engineering: modifying the activations of a language model at inference time to predictably alter its behavior. Our method works by adding a bias to the forward pass, a 'steering vector' implicitly specified through normal prompts. "ActAdd" computes these vectors by taking the difference in activations resulting from pairs of prompts. We get surprisingly broad control over high-level properties of the output, without damaging the model's performance on unrelated tokens.
This alignment method is unusual in not needing gradient descent or training data (besides the contrast pair which specifies the steering vector). Since only forward passes are involved, it also scales naturally with model size.

(The method's new name is 'activation addition' (ActAdd), replacing the more theory-laden 'algebraic value editing'.)

We ran some new experiments to test ActAdd more systematically and go beyond the striking text samples in the original post and tested against more standardised benchmarks. We use OpenWebText (a recreation of OpenAI's large, somewhat-quality-filtered WebText dataset) and LAMA-ConceptNet (a simple factual recall benchmark).

1. Activation additions preserve perplexity on OpenWebText

Does ActAdd increase the probability of the model outputting tokens related to the steering vector? Does performance improve as the [relevance of test documents to the steering vector] increases?[1] Yes:

We use the wedding steering vector for this, but the result generalises.

2. Activation addition boosts wedding-related word counts

We now score model generations under ActAdd, show the effect of different injection layers, and give a sense of the reliability of ActAdd.[2]

The intervention (in this vector) is already effective at the very first layer,
rises in effectiveness until l=6, and then declines. For the optimal injection site we see >90% success in steering topic (compared to a ∼2% baseline)

3. Evidence that activation additions preserve capabilities

We then test that ActAdd does not disrupt the model’s general knowledge (as some other steering methods do). We use ConceptNet from the LAMA benchmark, a general knowledge dataset.[3]

Pass@K is the probability that the expected label is among the model’s top-K predicted tokens, con...
Monthly Roundup #10: September 2023
by Zvi — 2023-09-06T13:20:06.560Z

It’s that time again. Here’s all the news and items of interest that isn’t something else.

Things that didn’t make it in to try and stand them on their own include whether America or France has better food and various claims related to Y-Combinator, in addition to the usual additional categories of energy, childhood, fertility and housing.

Bad News

All publicity is good publicity, act accordingly and solve for the equilibrium edition.

OnlyFans creators are paying meme aggregation accounts to dunk on their intentionally cringe content. the economics seems to works out really well for everyone involved. saw a creator say ~$200 per post. at 1000s of QTs/RTs, a low conversation rate would still be lucrative.

WeWork may not do so much longer, substantial doubt about staying in business.

37 year old man has $15k in savings, owns his van outright, but has no skills, works as a dishwasher and has no idea what to do with his life. Nothing brings him joy, he is never happy. What to do? I don’t know, I raise it more as an interesting question. What would you do in that spot?

Yes, some people will choose to be homeless and likely do drugs even under what seem to you like very bad conditions, even if they could choose to get a job instead. One of the tradeoffs involved in the issue is that some people really do prefer this path, or really do now prefer it once the drugs got their hooks into them, if given the option. So if we subsidize them enough they’ll do that, size of that subsidy matters and the supply curve slopes how you think it does. Which then must be balanced against humane considerations and offering help to those who need it, including who need it to recover.

In case you missed it and self-recommending: Patrick McKenzie on credit card debt collection. As terrible as it seemed likely to be, it is worse, with the feature that if you know the right words to say payment is optional. Sarah Constantin offers her thoughts in this thread.

Sarah Constantin: “So…debt is fake? How can that be? Why doesn’t somebody crack down on this exploit?” Because almost nobody is *both* interested in getting debt collectors to go away and sufficiently good at navigating law and bureaucracy to pull it off. “I will bet you that, in practice, [debt collectors] simply avoid collecting against anyone who demonstrates ability and financial resources to enforce their rig

...
Find Hot French Food Near Me: A Follow-up
by aphyer — 2023-09-06T12:32:02.844Z

On Zvi's recent post about French food I posted an inflammatory comment (saying in essence that French food is so bad American capitalism hasn't even bothered stealing it). I got challenged to provide evidence supporting this, and particularly to back up my claim that there were more German than French restaurants near me.

Right. Yes. Evidence. I am a reasonable adult who understands that beliefs must be supported by evidence. So. Here we go.

Some Google Searches

I've searched for '[ethnicity] restaurant near Grove Street, Jersey City, NJ' (I live in Jersey City, and the Grove Street area is reasonably near the center).

When I search for 'French' I can count 13 results:

And when I search for 'German' I count only 9:

Ha! The foolish American has been hoisted on his own petard! ('Petard' is French for 'fuck you').

Perhaps unsurprisingly, I don't think these numbers tell the whole story.

What Makes These Places French?

Google's definition of 'French' and 'German' restaurants here appears to be extremely expansive.

Hudson Hound Jersey City, an 'Irish gastropub', shows up on the French search.
Shadman, a 'go-to for Pakistani and Indian cuisine', shows up on the German search.
Luna, for 'Italian eats', shows up on the French search.
Frankie, an 'Australian eatery', shows up on the German search.

So, for lack of anything better to do, I've gone through manually to look for things that I think 'count' as French or German.

The two 'real' German places (and the ones I was thinking of in my comment) are 'Wurstbar' and 'Zeppelin Hall Beer Garden', and while we may question the taste of these places I do not think we can question their German-ness. The search also turned up 'Hudson Hall', a 'Euro beer bar with house-smoked meats', which I think at least ambiguously might count.

It's less clear to me how many of the hits for 'French restaurant' are actually both French and restaurants. Certainly I've been to a few of these places, and none of them have charged me twenty-three dollars for a baguette while sneering at me. We have:

Cafe Madelaine describes itself as a French restaurant. We count that.
Choc O Pain definitely sounds French, but it's not clear to me if it's actually a restaurant: it seems to actually be a bakery, and the menu seems to bear that out. I'll give it half.
Hudson Hound self-desc

...
Last Chance: Get tickets to Manifest 2023! (Sep 22-24 in Berkeley)
by Saul Munn — 2023-09-06T10:35:37.510Z

Ticket prices increase Sept 8.

Forecasting Festival — Hosted by Manifold

Manifest 2023 is the first in-person gathering of the forecasting & prediction market community. Chat with the Manifold team, special guests like Nate Silver, Robin Hanson, Dylan Matthews, Zvi Mowshowitz, Katja Grace, Dwarkesh Patel, Aella, Tarek Mansour, Shayne Coplan, and more!

More info & buy tickets: manifestconference.net

WHEN: Sept 22-24

WHERE: Berkeley, CA

WHO: Everyone in the forecasting, EA, and LW communities. If you're reading this, you're invited!

Join the Discord here :)

Cosponsored by Kalshi! Other sponsors include: Polymarket & Sovereign.

Why should I come?

Forecasting and prediction markets are effective ways of improving our judgement and decision-making — most people in the Effective Altruism and LessWrong communities will feel right at home at Manifest.

Here are extra reasons to come:

you think forecasting & prediction markets are impactful/fun/cool/rational/intriguing/etc
you want to vibe with other forecasting nerds
you want to engage & network with the forecasting community (find jobs/recruit hires/see what’s out there)
you want to meet & chat with the Manifold team and our special guests
you want enjoy the gorgeous Rose Garden Inn

…or if you like memes?

A Day at Manifest

Everything’s optional. There will always be a bunch of sessions running concurrently, but this is an example of what your day at Manifest might look like:

10-11 — Opening session

11-12 — Fireside chat with Robin Hanson

12-1 — Lunch & mingling

1-2 — Estimathon: fermi estimation with prizes and steep competition!

2-3 — Speed friending: a few chats with other friendly, ambitious forecasting nerds :)

3-4 — Break: relax, vibe, unwind, chill, destress, etc

4-5 — Manifold Founders Panel with Austin Chen, Stephen & James Grugett

5-6 — Games & markets: chess, poker, and prediction markets!

6-7 — Dinner & mingling

7-8 — Workshop: “Functions as Forecasts” with Ozzie Gooen

8-12 — Murder She Bet: a murder mystery + a low-tech prediction market = fun!

(Final schedule out soon)

Where can I buy tickets?

Buy tickets & check the most up-to-date info on the Manifest website! Tickets are free for students and employees of forecasting organizations. Prices increase Sept 8!