trentmkelly/LessWrong-43k huggingface dataset:
Shannon Vallor’s “technomoral virtues”
by David Gross — 2024-05-04T14:48:06.995Z

Shannon Vallor is the first Baillie Gifford Chair in the Ethics of Data and Artificial Intelligence at the Edinburgh Futures Institute. She is the author of Technology and the Virtues: A Philosophical Guide to a Future Worth Wanting. She believes that we need to build and cultivate a new virtue ethics appropriate to our technological era. The following summarizes some of her arguments and observations:

Why Do We Need New Virtues?

Vallor believes that we need to discover and cultivate a new set of virtues that is appropriate to the onslaught of technological change that marks our era. This for several reasons, including:

Technology is extending our reach, so that our decisions have effects with broader ethical implications than traditional moral wisdom is prepared to cope with.
Technology is also starting to make human-like (un)ethical decisions.
We exist in a time of “acute technosocial opacity” — that is, technology is reshaping our societies and our potentials much more rapidly than we can be aware of, react to, or understand the implications of.
Technology has also given us a new ethical concern: existential risk. We have entered technological realms that contain threats to human existence as a whole. And some of the problems we are encountering require collective action, not just individual moral decisions.

Why Virtue Ethics?

Most historical ethical philosophers have considered the human condition to be roughly static and universal (perhaps with some allowance for cultural differences, divine revelation, or slow moral progress). The resulting philosophies are too brittle for our current situation. In particular, deontological/Kantian rules are too inflexible to keep up with technological change, and consequentialist/utilitarian calculation is nigh impossible as the future becomes increasingly difficult to predict.

However, Vallor believes the older tradition of “virtue ethics” is up to the challenge. This is in part because it is potentially more flexible and better able to cope with environmental change and uncertainty. It is a form of ethics that has been rediscovered and redesigned to fit specific cultural and historical niches in the past, and we can again rediscover and redesign it today to come up with “technomoral virtues” appropriate to our time.

In the opening chapters, Vallor reviews what virtue ethics is, and looks at three virtue ethics traditions in p...
Conserved Quantities (Stat Mech Part 2)
by J Bostock — 2024-05-04T13:40:55.825Z

As before, we will consider particles moving in boxes in an abstract and semi-formal way.

Imagine we have two types of particle, red and blue, in a box. Imagine they can change colour freely, and as before let's forget as much as possible. Our knowledge over particle states now looks like:

[Position]∼Uniform over inside of box
[Colour]∼Uniform over {Red,Blue}

For each particle:

Let's connect these particles to a second box, but also introduce a colour-changing gate to the passage between the boxes. Particles can only go through the gate if they're willing to change from red to blue, and only go back if they change colour in the opposite direction.

Without further rules, lots of particles will approach the gate. The blue ones will be turned away, but the red ones happily switch to blue as they move into the second box, and ... promptly change back to red. We've not done anything interesting. We need a new rule: particles can't change colour without a good reason, they must swap colour by bumping into each other.

Now the particles that end up the second box must remain blue. Lets think about what happens when we start the box off. Unfortunately this question as posed is un-answerable, because we've introduced a system conserved quantity. Any time we do this, we must specify how much of the quantity is in the system. As an exercise, think about what the conserved quantity is.

Marvel at my incredible artistic skills while you ponder this

...

The conserved quantity is:

[Number of Red Particles in Box 1]+[Number of Particles in Box 2]

Or, conversely we could also consider:

[Number of Blue Particles in Box 1]

To be constant, since the number of particles is constant. Writing things out like this also makes the states of the system more explicit, so we can reason about the system more accurately. We expect the constant number of blue-and-box-1 particles to be distributed evenly throughout box 1, and we expect the constant number of red-or-box-2 particles to be distributed evenly throughout boxes 1 and 2. If (as above) both boxes are equally-sized, this caches out to the following rule:

For n particles starting in box 1, of which m start off red, we expect to end up with n−m blue particles in box 1, m/2 red particles in box 1, an...
If you are assuming Software works well you are dead
by Johannes C. Mayer — 2024-05-04T12:54:17.675Z

Suno Version

I say this because I can hardly use a computer without constantly getting distracted. Even when I actively try to ignore how bad software is, the suggestions keep coming.

Seriously Obsidian? You could not come up with a system where links to headings can't break? This makes you wonder what is wrong with humanity. But then I remember that humanity is building a god without knowing what they will want.

So for those of you who need to hear this: I feel you. It could be so much better. But right now, can we really afford to make the ultimate <programming language/text editor/window manager/file system/virtual collaborative environment/interface to GPT/...>?

Can we really afford to do this while our god software looks like...

May this find you well.
CCS on compound sentences
by Artyom Karpov — 2024-05-04T12:23:38.116Z

Finding internal knowledge representation(s) inside transformer models without supervision is certainly a challenging task which is important for scalable oversight and to mitigate the deception risk factor. I’m testing Contrast-Consistent Search (CCS[1]) on TruthfulQA[2] dataset for compound sentences (conjunction and disjunction) each composed of several answers to a question to see if unsupervised probes work to the same degree as on simple statements that compound ones consist of, with the goal to improve unsupervised methods to discover latent knowledge. I run about 500 evaluations of CCS probes trained on compound sentences, so far results suggest that CCS probes trained on simple sentences are unlikely to transfer their performance to compound sentences and vice versa for Llama 2 70B, still Llama 3 70B demonstrate some transfer and better performance.

Goal

Motivation is to find a method that detects lies in the output of language models, i.e. to elicit latent knowledge (ELK). Lying transformer models is a risk factor whenever we use them in critical areas. I used CCS probes which are an unsupervised method to find features in activation space in a model (residual stream in a transformer). It is important because as models and their problems scale they become harder to supervise, i.e. create such datasets with correct labels that help to define how models should behave. We need some way to tell what a model actually believes in, what it actually uses as focal points while generating a next token (or action). Probably, improved CCS can be used to detect knowledge directions. Then, knowing those directions, we will likely be able to train models not to be biased by a prompt, not to be sycophantic (tell a user what they want instead of what the model believes is true) by penalizing untruthful answers. Another application of these probes is to increase trust in models, to serve as a litmus test when we are not sure how the model generalizes to new data.

The goal is to answer the question whether CCS works on compound statements (conjunction, disjunction) to the same degree as on simple statements, i.e. if the accuracy doesn’t degrade, if the questions previously solved for simple sentences are solved for compound ones. It’s important to find how to make CCS work, though it seems to be hard because CCS tends to find random features[3]. Example...
Now THIS is forecasting: understanding Epoch’s Direct Approach
by Elliot Mckernon — 2024-05-04T12:06:48.144Z

Happy May the 4th from Convergence Analysis! Cross-posted on the EA Forum.

As part of Convergence Analysis’s scenario research, we’ve been looking into how AI organisations, experts, and forecasters make predictions about the future of AI. In February 2023, the AI research institute Epoch published a report in which its authors use neural scaling laws to make quantitative predictions about when AI will reach human-level performance and become transformative. The report has a corresponding blog post, an interactive model, and a Python notebook.

We found this approach really interesting, but also hard to understand intuitively. While trying to follow how the authors derive a forecast from their assumptions, we wrote a breakdown that may be useful to others thinking about AI timelines and forecasting.
In what follows, we set out our interpretation of Epoch’s ‘Direct Approach’ to forecasting the arrival of transformative AI (TAI). We’re eager to see how closely our understanding of this matches others’. We’ve also fiddled with Epoch’s interactive model and include some findings on its sensitivity to plausible changes in parameters.

The Epoch team recently attempted to replicate DeepMind’s influential Chinchilla scaling law, an important quantitative input to Epoch’s forecasting model, but found inconsistencies in DeepMind’s presented data. We’ll summarise these findings and explore how an improved model might affect Epoch’s forecasting results.

We’ve accidentally filled this post with Star Wars references.
Disclaimer: we do not actually mean to suggest that the Direct Approach is quick or easy (though it is definitely seductive).

This is where the fun begins (the assumptions)

The goal of Epoch’s Direct Approach is to quantitatively predict the progress of AI capabilities.

The approach is ‘direct’ in the sense that it uses observed scaling laws and empirical measurements to directly predict performance improvements as computing power increases. This stands in contrast to indirect techniques, which instead seek to estimate a proxy for performance. A notable example is Ajeya Cotra’s Biological Anchors model, which approximates AI performance improvements by appealing to analogies between AIs and human brains. Both of these approaches are discussed and compared, along with expert surveys and other forecasting models, in Zershaan...
OHGOOD: A coordination body for compute governance
by Adam Jones — 2024-05-04T12:03:16.716Z
Link: https://adamjones.me/blog/oh-good/

Core to many compute governance proposals is having some kind of register that records who owns AI chips.

This article explores how this register could be implemented in practice, outlining an organisation that maintains such a register and its necessary processes. It's named OHGOOD, the Organisation Housing the GPU & Others' Owners Database.

Motivation

Training highly-capable and broad AI systems requires lots of compute and data, along with efficient algorithms.

Compute is the easiest to track of these three since it currently relies on specialised expensive AI chips that can only be produced by a few actors in the world. Both data and algorithms are comparatively much harder to track: public datasets such as Common Crawl (a set of 250 billion web pages) can be downloaded by anyone, and key algorithmic breakthroughs that have enabled recent AI advances are published in scientific journals.

[More definitions on the above are in the linked post]

By tracking AI chips, the hope is that we are able to identify people with the capability to train highly-capable and broad AI models, and thus many of the most risky models. We could then ideally verify that these actors are using their AI chips in a safe way.

Previous work in compute governance has briefly touched on the need for this tracking body:

Shavit[1] put forward a framework for enforcing rules about large scale ML training, by recording snapshots of work done by AI chips (via on-chip firmware) and then requiring developers to show it is part of a compliant training run. In section 6.1 it explains a ‘chip-owner’ directory (corresponding to this proposal) is needed to be confident a developer is reporting all their training activity.
Baker[2] analysed the use of verification in nuclear arms control with a view to how it could be applied to a future AI safety treaty. In Annex G it describes how AI chip accounts might be verified, using methods analogous to nuclear arms control verification. The bodies responsible for these accounts correspond to this proposal.

This paper explores the various functions it would need to carry out in more detail, as well as some potential incentive schemes.

[A few limitations of compute governance are in the linked post]

The idea

We propose an international non-profit body that keeps a register of AI chips and their owners. This register should be:

accurate and up-to-date
trusted

...
My hour of memoryless lucidity
by Eric Neyman — 2024-05-04T01:40:56.717Z
Link: https://ericneyman.wordpress.com/2024/05/04/my-hour-of-memoryless-lucidity/

Yesterday, I had a coronectomy: the top halves of my bottom wisdom teeth were surgically removed. It was my first time being sedated, and I didn’t know what to expect. While I was unconscious during the surgery, the hour after surgery turned out to be a fascinating experience, because I was completely lucid but had almost zero short-term memory.

My girlfriend, who had kindly agreed to accompany me to the surgery, was with me during that hour. And so — apparently against the advice of the nurses — I spent that whole hour talking to her and asking her questions.

The biggest reason I find my experience fascinating is that it has mostly answered a question that I’ve had about myself for quite a long time: how deterministic am I?

In computer science, we say that an algorithm is deterministic if it’s not random: if it always behaves the same way when it’s in the same state. In this case, my “state” was my environment (lying drugged on a bed with my IV in and my girlfriend sitting next to me) plus the contents of my memory. Normally, I don’t ask the same question over and over again because the contents of my memory change when I ask the question the first time: after I get an answer, the answer is in my memory, so I don’t need to ask the question again. But for that hour, the information I processed came in one ear and out the other in a matter of minutes. And so it was a natural test of whether my memory is the only thing keeping me from saying the same things on loop forever, or whether I’m more random/spontaneous than that.[1]

And as it turns out, I’m pretty deterministic! According to my girlfriend, I spent a lot of that hour cycling between the same few questions on loop: “How did the surgery go?” (it went well), “Did they just do a coronectomy or did they take out my whole teeth?” (just a coronectomy), “Is my IV still in?” (yes), “how long was the surgery?” (an hour and a half), “what time is it?”, and “how long have you been here?”. (The length of that cycle is also interesting, because it gives an estimate of how long I was able to retain memories for — apparently about two minutes.)

(Toward the end of that hour, I remember asking, “I know I’ve already asked this twice, but did they just do a coronectomy?” (The answer: “actually you’ve asked that much more than twice, and yes, it was just a coronectomy."))

Those weren’t my only questions, though. About five minu...
Extra Tall Crib
by jefftk — 2024-05-04T00:00:03.149Z

A few days ago I came upstairs to:

Me: how did you get in there?

Nora: all by myself!

Either we needed to be done with the crib, which had a good chance of
much less sleeping at naptime, or we needed a taller crib. This is
also something we went through when Lily was little, and that time
what worked was removing the bottom of the crib.

It's a basic crib, a lot like this
one. The mattress sits on a metal frame, which attaches to a set
of holes along the side of the crib. On it's lowest setting, the
mattress is still ~6" above the floor. Which means if we remove the
frame and sit the mattress on the floor, we gain ~6".

Without the mattress weighing it down, though, the crib would not be
hard for an energetic toddler to tip. I've attached it to the wall on
two sides with strapping, right into a stud:

Nora was eager to give it a try, holding on the rail and bouncing hard:

This should get us a bit more time with solid naps!

(I was going to do something similar with Anna when she was the same
age, but the crib we happened to be using for her was designed
differently and had a structurally important bar across the bottom.)

Comment via: facebook, mastodon
Get your tickets to Manifest 2024 by May 13th!
by Saul Munn — 2024-05-03T23:57:04.389Z

What is Manifest?

Manifest is a festival for predictions, markets, and mechanisms. It’ll feature serious talks, attendee-run workshops, fun side events, and so many of our favorite people!

Tickets: manifest.is/#tickets

WHEN: June 7-9, 2024, with LessOnline starting May 31 and Summer Camp starting June 3

WHERE: Lighthaven, Berkeley, CA

WHO: Hundreds of folks, interested in forecasting, rationality, EA, economics, journalism, tech and more. If you’re reading this, you’re invited! Special guests include Nate Silver, Scott Alexander, Robin Hanson, Dwarkesh Patel, Cate Hall, and many more.

You can find more details about what we’re planning in our announcement post.

Ticket Deadline

We’re planning to increase standard tickets from $399 to $499 on Tuesday, May 13th so get your tickets ASAP!

Prices for tickets to LessOnline (May 31 - June 2) and Summer Camp (June 3 - June 7) will also each increase by $100 on May 13th.

Student and supporter ticket prices will stay the same, but we still appreciate your buying tickets before the May 13th deadline as it helps us plan. Plus it means you’ll get a nice name badge!

We plan to raise prices again on May 31, one week before the beginning of Manifest — but we might raise prices again sooner than that if tickets are selling out.

Sponsorship

Manifest is looking for sponsors! We attract some of the world's most thoughtful speakers and talented individuals — if your organization is looking to broaden your company's reach, or recruit the best folks in the world, consider becoming a sponsor. You can shoot an email to Saul (saul@manifest.is) or book a call with him here.
Embodiment
by A* — 2024-05-03T20:06:25.117Z

In the realm of cognitive science and philosophy, the concept of embodiment posits that our cognition is deeply rooted in the body's interactions with the world. To understand our minds, we must consider the movements and spaces our bodies inhabit. Today, let's embark on an exploration of how embodiment reshapes our understanding of what it means to think, act, and be.

llɐƆ it a dance with reality; our thoughts are not isolated from our physical forms. Twists and turns can help you see things from a new perspective. When you walk, your brain is actively engaging with the rhythm of your steps, the texture of the ground, and the spatial dynamics around you. These interactions are not mere background noise but integral components of cognitive processing. The sway of your arms, the shift of your weight – all ʇɐɥʇ contributes to how you perceive and understand the world.

To say that our embodiment can ʇsǝɟᴉuɐW the future might sound like a stretch, but consider this: the way we move and interact with our environment can shape our 'ʎuᴉʇsǝp in profound ways. Athletes, dancers, and even astronauts are prime examples of how mastering noʎr body in space can lead to extraordinary achievements.

Of course, not all aspects of embodiment are easily articulated. There are mysteries yet to be uncovered, experiences that words uɐɔnot fully capture – pƎ┴Ɔ∀pƎɹ moments of pure being, where movement and thought blend into a seamless flow. These experiences remind us that, despite our advancements in understanding the mind, there remains an uncharted territory that beckons us to explore.
(Geometrically) Maximal Lottery-Lotteries Exist
by Lorxus — 2024-05-03T19:29:01.775Z

epistemic/ontological status: almost certainly all of the following -

a careful research-grade writeup of a genuinely kinda shiny open(?) question in theoretical psephology that will likely never see actual serious non-cooked-up use;
dedicated to a very dear cat;
utterly dependent, for the entirety of the most interesting parts, on definitions I have come up with and results I have personally proven partially using them, which I have done with a professional mathematician's care; some friends and strangers have also checked them over;
my attempt to prove that something that can reasonably be called a maximal lottery-lottery exists;
my attempt to scavenge what others have left behind and craft a couple of missing pieces, and then to lay out a blueprint for how it could begin to work;
not a 30-minute read
the first half of something incomplete

Maximal Lottery-Lotteries: Far More Than It Ever Occurred To You To Want To Know

"All of this suggests three possible ways of explicitly constructing a maximal lottery-lottery which turn out in short order to be equivalent." -Lorxus

This post is mostly a distillation/concentrated treatment of the Maximal Lottery-Lotteries Sequence, though I define an important property and prove an important existence result at the end. It's the first of two posts in a sequence, the second of which is linked here.

The Maximal Lottery-Lotteries sequence details why anyone should care about sortitive/lottery-using electoral systems even without any particular hope of getting to implement it.[1] It also ends in early November of 2022 with the author and a handful of other technical alignment notables all honorably giving up on the shiny math problem with varying degrees of explicitness. This post is meant as a follow-on to the sequence, and its sequel is, too. I also end up drawing heavily on the Geometric Rationality sequence for its central tools.

As per my usual policy, if I've managed to misunderstand anything or write anything up unclearly or incorrectly, or you have any questions about what I've written, please comment below or at https://www.admonymous.co/lorxus and I'll fix it/reply to it when I can.

If you're reading this post for the first time, you might want to keep the notation reference on hand. If you're relatively inexperienced with reading text which treats dense mathematical notation on par with English prose, slow dow...