trentmkelly/LessWrong-43k huggingface dataset:
Announcing the SoS Research Collective for independent researchers (and academics thinking independently) 
by rogersbacon — 2024-01-22T20:13:04.731Z
Link: https://www.theseedsofscience.pub/p/announcing-the-sos-research-collective

“Study hard what interests you the most in the most undisciplined, irreverent and original manner possible.” (Richard Feynman)

We (the founding gardeners

Roger

,

Dario

, and

Sergey

) have always conceived of Seeds of Science as a kind of research project, with the

journal

serving as the first of several “experiments”. You could think of our research questions as the following:

(1) Can people outside of academia make valuable contributions if given the proper platform and support?

(2) Can we create new organizational structures that promote greater creativity and diversity of thought in science?

Nearly 2.5 years after we “planted” our first seed of science, we have some preliminary answers.

(1) Yes.

(2) Yes.

Not exactly groundbreaking stuff, but also not bad for a journal run by three people in their spare time. Since our founding, Seeds of Science has published 26 papers, recruited 303 “gardeners” (peer reviewers), and pioneered a new model for scientific journals (open to all scientific disciplines, specializing in speculative thinking, accepting of non-traditional writing styles/formats, community-based peer review with comments published in the final manuscript). We consider this a success, but still only a beginning—our ultimate goal is to grow from a sapling into a mighty oak tree that plays a vital role in the scientific ecosystem.

In order to help us reach this goal and provide some further answers to our research questions, we have decided to launch a new experiment. Before explaining the structure and philosophy of the SoS Research Collective, it may be helpful to briefly discuss why we believe the time is ripe for something like what we are proposing (sorry for all the botanical puns, I really tried but I just can’t help myself).

Why now?

Science is in trouble. Replication crises and a slowing pace of progress in many fields, biased and inefficient funding systems, biased and inefficient journals, billions of dollars of spent on journal fees so that publicly-funded research can be locked behind paywalls, intense competition for jobs and grants that stifles creativity in favor of productivity and self-promotion, poor pay for many graduate students and postdocs, increased rates of fraud and questionable research practices (or at least increased awareness thereof), increasingly problematic political entanglements—I could go on, but I think you get the idea.

The ove...
A Brief Assessment of OpenAI's Preparedness Framework &
Some Suggestions for Improvement
by simeon_c — 2024-01-22T20:08:57.250Z
Link: https://uploads-ssl.webflow.com/64332a76ab91bba8239ac2e0/65aec710ccf7002fffd9b80c_Brief%20Assessment%20of%20OpenAI%27s%20Preparedness%20Framework%20%26%20Some%20Suggestions%20for%20Improvement.pdf

This document was produced by SaferAI submitted for feedback to OpenAI. It is doing a list of remarkable improvements of the Preparedness Framework (PF) over Anthropic’s RSPs and suggests possible improvements to make the PF more robust.

Notable Improvements of the PF Over Anthropic’s RSPs

It aims to run the safety tests every 2x increase in effective computing power rather than 4x, which is a substantial and welcome improvement. We already have evidence of emerging capabilities like in-context learning substantially changing the risk profile & fully emerging over a 5x increase in compute (

Olsson et al.

, 2023). This means that it is totally conceivable that something could change the risk profile over 3x or less, making the Anthropic scaling strategy potentially unsafe and the OpenAI one wiser if they’re able to stick to it.
It adds measures relevant to safety culture and good risk management practices, such as “safety drills” to stress test the company culture robustness to emergencies, a dedicated team to oversee technical work, and an operational structure for safety decision-making. Improvements at the organizational level are very welcome to move towards more robust organizations, analogous to High Reliability Organizations (HROs) which are organizations that can deal with dangerous complex systems successfully.
It provides the board with the ability to overturn the CEO’s decisions. The ability of the board to reverse a decision adds substantial guardrails over the safety orientation of the organization.
It adds key components of risk assessment such as risk identification and risk analysis, including for unknown unknowns that evaluation-based frameworks are not able to catch by default.
Forecasting risks, notably through risk scaling laws (i.e., predicting how risks will evolve as models get scaled) are key to avoiding the training of a dangerous model in the first place, rather than having to evaluate for danger after training.

Suggested Improvements of the PF

Suggested Improvements - Table of Content

1. Map capabilities levels to risks (i.e. likelihood & severity) & clarify scenarios.
2. Enhance independent scrutiny through Safety Advisory Group (SAG) members external to the company.
3. Measure and improve safety culture.
4. Imple

...
D&D.Sci(-fi): Colonizing the SuperHyperSphere [Evaluation and Ruleset]
by abstractapplic — 2024-01-22T19:20:05.001Z

This is a followup to the D&D.Sci post I made ten days ago; if you haven’t already read it, you should do so now before spoiling yourself.

Here is the web interactive I built to let you evaluate your solution; below is an explanation of the rules used to generate the dataset (my full generation code is available here, in case you’re curious about details I omitted). You’ll probably want to test your answer before reading any further.

Ruleset

By default, a ZPPG will produce 100% of Standard Output; however, various phenomena will affect this, usually for the worse. Each phenomenon multiplies the expected output by a given quantity; these multipliers are applied cumulatively.

Actual output is expected output multiplied by N(1, 0.0057); a negligible randomness nudging guesses off by half a percentage point.

Coordinates

Longitude

ZPPG output is greatest when Longitude=52.46, and decreases as you move away from that angle.

Latitude

ZPPG output decreases by sharply when Latitude is within 36 degrees of the equator.

Shortitude

ZPPG output decreases slightly when Shortitude>45 degrees.

Deltitude

Deltitude has absolutely no effect on anything.

Smells

Somewhat weird smells greatly decrease expected output; extremely weird smells only decrease it by ~6%.

Tastes

Bizarre-tasting air is to be expected on the SuperHyperSphere: air with no taste suggests one of the cleverer native lifeforms is trying to lure you to build there so it can siphon power. (Conversely, burning air suggests the presence of a dimensional distortion which actually helps ZPPG performance.)

Feng Shui

ZPPGs work better in places with good Feng Shui.

Sounds

Sounds are always bad news: they mean native lifeforms are present and will impair a ZPPG built on that site.

(There is one apparent exception, Otherworldly Skittering; this is because the Multiversal Spiders who cause this noise reliably rid the area of the more damaging Time Flies and their Unnatural Buzzing; however, Silence is still preferable to every sound.)

Pi

ZPPGs function optimally when Pi=3.15; moving away from that optimum decreases ZPPG effectiveness linearly.

Murphy

ZPPGs function optimally when M=0; increasing from that optimum decreases ZPPG effectiveness cubically.

Strategy

There are 40 sites in cleared_sites_data.csv which have >100% of Standard Output, and the tiny amount of randomness between explanatory and dependent variables means that it’s poss...
' petertodd'’s last stand: The final days of open GPT-3 research
by mwatkins — 2024-01-22T18:47:00.710Z

TL;DR All GPT-3 models were decommissioned by OpenAI in early January. I present some examples of ongoing interpretability research which would benefit from the organisation rethinking this decision and providing some kind of ongoing research access. This also serves as a review of work I did in 2023 and how it progressed from the original ' SolidGoldMagikarp' discovery just over a year ago into much stranger territory.

Work supported by the Long Term Future Fund.

Introduction

Some months ago, when OpenAI announced that the decommissioning of all GPT-3 models was to occur on 2024-01-04, I decided I would take some time in the days before that to revisit some of my "glitch token" work from earlier in 2023 and deal with any loose ends that would otherwise become impossible to tie up after that date.

This abrupt termination of one thread of my research also seemed a good point at which to create this post, intended as both (1) a summary of what I've learned about (and "experienced of") GPT-3 in the last year, since Jessica Rumbelow and I stumbled upon
' SolidGoldMagikarp', ' petertodd', et al. , including findings from the last few days of access and (2) a gentle protest/appeal to OpenAI to reconsider the possibility of
(limited?[1]) research access to GPT-3 models – obviously GPT-3 is in one sense redundant, having been so comprehensively superseded, but as a resource for LLM interpretability research, it could still have considerable value. Certain poorly understood phenomena that may be relevant to a range of GPT models can currently be studied only with this kind of continued access to GPT-3 (since GPT-2 and -J are not sufficiently deep to display these phenomena reliably, and the GPT-4 base model is not available for study).

Rewind: SERI-MATS 2.0

One of the strangest moments of my life occurred one Tuesday afternoon in mid-January 2023 when sitting at my laptop in the SERI-MATS office in London, testing various anomalous tokens which GPT-3 seemed curiously unable to repeat. The first of these to come to my attention (the day before) had been the now well-known
' SolidGoldMagikarp', which ChatGPT insisted was actually the word "distribute". A few dozen suspicious tokens had since been collected and I was patiently testing numerous variants of the following prompt in the OpenAI GPT-3 playground, recording the outputs of the davinci-instruct-beta mod...
InterLab – a toolkit for experiments with multi-agent interactions
by Tomáš Gavenčiak — 2024-01-22T18:23:35.661Z
Link: https://acsresearch.org/posts/interlab-intro

This post introduces InterLab, a toolkit for experiments with multi-agent interaction. We plan to release more posts on the overall project, technical details, design considerations, and concrete research projects and ideas over the next few weeks to months.

This post focuses on the motivation behind the project and touches upon more high-level considerations; if you want to jump to the implementation itself or want to start experimenting, you can jump directly to the Getting started section, to the InterLab GitHub repo with project overview and further links, or explore the example Google Colab notebook.

Motivation

The research agenda of ACS is primarily focused on understanding complex interactions of humans and AI agents, both on individual level and between systems or institutions, and both theoretically and empirically.

Future going well in our view depends not just on the narrow ability to point one AI system to some well specified goal, but broadly on a complex system composed of both AIs and humans to develop in a way which is conducive to human flourishing. This points to a somewhat different set of questions than traditional AI safety, including problems such as "how to deal with the misalignment between individual humans, or between humans and institutions?", "how to avoid AIs amplifying conflict?", "how will institutions running on AI cognition (rather than human cognition) work?", "how do aggregate multi-agent entities evolve?", or "what happens if you replace part of human-human interactions in the society with AI-AI interactions?".

While many of these questions likely require a better theoretical and conceptual understanding, it is also possible to study them empirically, using LLMs and LLM-based agents, which can also inform our models and intuitions.

We may build more comprehensive language model evaluations for near-term alignment, in particular in the direction of multi-agent evaluations—this is indeed one of the goals of InterLab.
We may learn about strategies for resolving conflicts and disagreements, and robust cooperation, as well as models of manipulation and coercion, in particular under information and power imbalances.
We may create new technologies for human coordination, cooperation and empowerment, such as negotiation aids or aids for solving internal conflicts in individual humans.
Multi-agent systems of humans and AIs come with a

...
Who Organizes Dances?
by jefftk — 2024-01-22T14:30:06.045Z

Most contra dances are organized by a committee of volunteers: ~3 to ~8
people who divide up the work. This includes both advance work like
coordinating with the venue, booking performers, and publicizing the
dance, and also day-of work like setup, cleanup, and handling things
that come up at the dance. For a detailed example, you could look at
BIDA's

current
division.

This is not the only way to organize dances: some events are put on by
a house band, house caller, or an individual (volunteer or
professional). I have the impression this used to be more common
before the 1970s-on revival, but don't have anything hard to go on.

Someone wrote me to ask about the typical composition of organizing
committees: are they usually made up of callers? Musicians? Dancers?
People who do everything? I
decided to look back over BIDA's historical board, and see what it's
been in our case.

Over the past fifteen [1] years we've had 34 board members. Looking
over the list I remember all of them dancing at least some, and most
of them also called or played music. Let's make some tables,
considering only what people did during the time they were on the
board: [2]

non-musician
16 (47%)

casual musician
5 (15%)

booked to play
13 (38%)

non-caller
23 (68%)

casual caller
2 (6%)

booked to call
9 (26%)

Overall, it looks like we've has a lot more musicians than
callers. But do the musicians and callers tend to be the same people?
Let's cross these tables:

non-caller
casual caller
booked to call

non-musician
10
1
5

casual musician
2
0
3

booked to play
11
1
1

It looks like the single most common role is a dance musician who
doesn't call, at 11 (32%), followed closely by people who don't do
either, at 10 (29%).

Losing a bit of resolution, let's categorize everyone by their primary
skill-set. For example, even though I like dancing and was booked to
call, my main non-organizer role was as a musician:

Musician
13 (38%)

Dancer
12 (35%)

Caller
8 (24%)

Sound
1 (3%)

I'd be curious what this looks like in other communities!

[1] Really! BIDA started in late 2008, and our first dance was
in February 2009. We're holding a big celebration dance on
2024-02-04, and if you've ever played for BIDA you're welcome to come
back and join the band!

[2] I'm counting someone as "casual" ...
Values Darwinism
by pchvykov — 2024-01-22T10:44:41.548Z

TL;DR / summary:

Can we objectively define "progress"? A Darwinian definition seems plausible. But when referring to cultures, world-views or theories, it seems that physical survival of the fittest must be complemented by memetic survival. The interaction of these two will generally be complex, and may parallel models from epidemiology. Most discussions usually account for physical or memetic selection - but not their interplay. If se suppose that happiness makes memes more contagious, we can push back on the common idea that "evolution does not select for happiness," and update our conventional notion of progress.

[Cross-posted from my website http://pchvykov.com/blog]

I’ve had some discussions recently about how to measure whether our conventional idea of “human progress” really constitutes an improvement. There is some evidence that people were happier and more well-nourished in caves than directly after the agricultural revolution (see

Sapiens by Y. Harari

). Now there are ideas that “the simple life,” like at a permaculture village, is somehow more conducive to our humanity and well-being than city or corporate life. We could make the argument that the question of "progress" reflects the more fundamental question of “finding deeper truths”: On the one hand, we tend to think that our modern scientific understanding is “more true” than e.g., Aristotelian physics, or Buddhist cosmologies. On the other hand, we understand that "all models are wrong, but some are useful" (

G. Box

) — and perhaps some Buddhist ideas are more useful if the goal is human happiness (e.g., see

Bhutan's Gross National Happiness

). Note that for this and following discussion, we entertain the scenario where we do not have access to any fundamental notion of “truth” that we could universally agree on.

One argument that always seemed most convincing to me in favor of “progress is real” is Darwinian one: ultimately, success should be measured via “survival of the fittest.” Even if hunter-gatherer tribes were somehow happier, they were outcompeted by agricultural communities, which could support much larger, albeit more miserable, populations. Similar argument can be made for modern science: technological progress gave Europe the firepower to subjugate most Eastern nations. This idea could be summarized by refining Box’s quote to be “all models are wrong, but some are useful for long-te...
The akrasia doom loop and executive function disorders: a question
by TeaTieAndHat — 2024-01-22T07:01:09.646Z

There’s a lot of productivity advice on LW, or more specifically advice on how to beat procrastination and akrasia (see this one, this one, this one, this one, this one, this one, this one, this one, this one, … — all of those are pretty good and you should check them out, btw).

But, well… do you know the feeling where you say to yourself "today is the day where I start taking up this cool new habit that will slowly but surely destroy my akrasia forever!" and then it’s 11pm, you still haven’t done the anti-akrasia thing you said you’d do at 7am, and you decide you’ll do it tomorrow but you know you won’t really and it’s super frustrating?

At this point, there are two questions I could be asking. And I’ll ask them both in turn.

First: how is it supposed to work!? In other words: do the people who did manage to get better at getting stuff done know how to avoid akrasia killing your motivation to beat akrasia?

Second: …

… well, a lot of people do manage to find enough motivation in themselves at least to get started, at least most of the time, right? So, maybe my answer is that I’m uniquely bad at it, right?

That’s not as self-deprecating and misguided as it sounds: I have ASD, and it is quite well established that this comes with an impaired executive function compared to the general population.

Hence my second question: does anyone have interesting things to say on the links between autism-related executive function deficits and akrasia? It seems like there’s more to it than just "ASD makes me bad at doing stuff, period", and that it’s instead a weird mix of trouble with some form or other of social anxiety (stuff like postponing writing an email for a week because I’m not sure how it will be received by the other person), some bizarre trouble with motivation (maybe non-ASD people get more social motivation, and so are more motivated than me?), and actual troubles with task switching, task initiation, or other stuff at brain level. I don’t think I understand very well how these different factors interact and each contribute to "akrasia and crappy executive function", and I’m curious to understand it better.

And, of course, that comes with a third question: there’s no reason to expect "crappy executive function" to be something different entirely in an autistic person vs. in a non-autistic person (concepts like ASD or ADHD, after all, are only conven...
Predicting AGI by the Turing Test
by Yuxi_Liu — 2024-01-22T04:22:38.526Z
Link: https://yuxi-liu-wired.github.io/essays/posts/perplexity-turing-test/

Abstract

This essay explains the Direct Approach proposed by
[@barnettScalingTransformativeAutoregressive2023].[1] I encourage
you to play with the Direct Approach Interactive
Model to
explore an interactive simulation using the approach.

The Direct Approach framework bounds the compute requirements for
transformative AI by extrapolating neural scaling laws. We combine
those estimates with simple models of future progress in algorithms,
investment, and compute costs to produce a user-adjustable forecast
over the date at which TAI will be achieved.
[@barnettDirectApproach2023]

From the POV of the judge, a Turing test is a sequential test for two
statistical hypotheses -- "is human" and "is machine". Under reasonable
assumptions, halving the (reducible part of) log-perplexity loss of the
language model would double the time it can survive in a Turing test.

We can think of the peer-review of scientific papers as a Turing test,
and say that AGI has arrived when we have AI scientists that can pass
the papers peer-review. This allows us to calculate the log-perplexity
loss of the first AGI. If we assume it is just a scaled-up GPT, then
assuming the Chinchilla scaling law, it would cost about 200 years of
global GDP. This makes it virtually certain that
the first AGI will not be a scaled-up GPT.

Turing test as statistical hypothesis test

Turing test

In the Turing test, there
are three players: one judge and two players. The first player is a
human, and the second is a machine. The judge asks each player text
questions and receives text answers. The judge must decide who is the
human.

We consider a simplified Turing test. In this test, the judge does not
ask, and simply receives one stream of text X1:∞. The judge
must decide whether the stream is produced by the human or the machine,
and do so quickly.

Cast in the language of statistical hypothesis testing, we have two
hypotheses:

H0: "the stream is produced by the human";

H1: "the stream is produced by the machine".

The judge would read from the stream X1:∞, o-n-e- -t-o-k-e-n
at a time, and at each token, decide whether to take another one, or
announce its judgment: H0 or H1.

As the organizers of the Turing test, we would start the test by
flipping a fair coin to decide whether to use the human or the machine.
Therefore, Pr(H0)=Pr(H1), and by Bayes, the posterior
log-probability ratio is

lnPr(X1:n|H0)Pr(X1:n|H1...