trentmkelly/LessWrong-43k huggingface dataset:
Joscha Bach on Synthetic Intelligence [annotated]
by Roman Leventov — 2023-03-02T11:02:09.009Z
Link: https://www.jimruttshow.com/currents-joscha-bach/

Link to transcript.

The conversation is very dense and there are a lot of interesting ideas that I leave out of the sampling below.

On the difference between sentience (self-awareness) and consciousness

I think that I usually distinguish between sentience and consciousness. Sentience is the ability of a system to make sense of its relationship to the world. So, basically understands what it is and what it’s doing. And in this sense, I would say that a corporation like Intel is sentient, because Intel has a good model of what it is, a legal model, the model of its actions, of its values, of its direction and the necessary cognition is largely facilitated by people. But this doesn’t really matter, because these people have to submit to the roles that they implemented in principle at some point.

We can implement these tasks with other information processing systems that are able to make coherent enough models. And consciousness is slightly different from sentience in that it is a real-time model of self-reflexive attention and the content that we attend to. And this gives rise to a fundamental experience usually. And I don’t think that Intel is conscious in this sense. It doesn’t have self-reflexive attention. And the purpose of consciousness in our own mind is to create coherence in the world in which we are and create a sense of now to establish what is the fact right now.

It’s basically filtering out of the sensory data, one coherent model of reality that we are seeing at this moment in time, and allows us to direct our attention on our mental contents and create coherence in our plans and imaginations and memories as well. And it’s conceivable that a machine will never need consciousness like this, because there are other ways to brute force the same thing. Our own mind is basically operating at the speed at which neurons are transmitting the electrochemical signals is relatively low and these neurons in the cells in the brain so slow, so it takes hundreds of milliseconds for a signal to cross the neocortex.

However, later, Bach describes the function of consciousness differently than in the above quote:

Jim Rutt: And then the conscious model, consciousness, I mean that’s a very high level acceptation of lower level stuff. There’s pretty convincing argument that the actual information arrival rate into consciousness is on the order of 50 bits a second. I mean it’s

...
If I want to test how good I would be as an AI safety researcher alongside my full-time job (with the hope of it becoming my full-time career at some point), is this a good plan?
by Malleable_shape — 2023-03-02T09:44:46.731Z

Introduction & context

I became aware of the AI safety problem around 8 years ago. I was intrigued by it and when I re-read Superintelligence, I decided to get a master's degree in Computer Science (my bachelor's degree is in Information Systems). Shortly after I graduated however, my interest for AI safety waned. It waned mainly because I felt that I couldn't make a significant contribution if I'm not employed by companies like OpenAI or DeepMind, and I didn't think I really could work there because:

they seem to be hiring people from top-tier American universities (i.e. MIT, Berkley etc.)
I wanted to work remotely, since I'm not from the USA and there are other things besides AI safety research that I want to do with my life

After I graduated, I got employed as a machine learning engineer. I have stumbled upon How To Get Into Independent Research On Alignment/Agency and Study Guide, but I kept them in my bookmarks and haven't read them, until now.

Very recently, I decided that I'd like to have a stab at being an AI safety researcher. This was mainly due to the fact that I never really tried striking it out on my own and I want to try it. My constraint is that I'm doing this as a side project, alongside my full-time job and other things in my life (gym, social life etc.).

The other reason why I haven't tried so far was because I also felt a bit unsure of myself because during my college education, I didn't have some courses such as physics (my physics stopped in high school) and while I did have some math, I never took a test where I had to do multi-dimensional integrals or solve differential equations. I wouldn't say my math background is non-existent, I just felt that people at leading research organizations had a much better background than me and I thought to myself: "What's the point?".

With that being said, I recently realized that I'm not getting any younger (I'm 26 now) and that I'd like to at least try to become a professional AI safety researcher. If I fail that will hurt, but at least I will sleep peacefully knowing that I tried.

Hence, I designed a plan on how I intend to do this, which I demonstrate below. Before that, I'd like to state my goal clearly:

My goal: Test the viability of me being an AI safety researcher, as fast as possible. I'm treating this as a side project with no set deadline; this is something I will do until my interest wan...
Job listing (closed): Sentience Institute is accepting applications for a researcher
by michael_dello — 2023-03-02T04:40:31.698Z
Link: https://www.sentienceinstitute.org/get-involved

We are currently accepting applications for a researcher on a rolling basis. Please see the information below. We encourage you to apply sooner rather than later.

We also welcome brief expressions of interest for other roles that can strengthen our research and positive impact, which you can send to info@sentienceinstitute.org.

RESEARCHER

This position offers a lot of flexibility with few requirements other than the production of high-quality research on moral circle expansion and digital minds. We are open to a variety of research methods: qualitative (e.g. historical case studies, interviews, conceptual analysis), quantitative (e.g. surveys, behavioral experiments, analysis of existing industry and survey data), or mixed methods. As of 2023, most of our research has been psychological experiments and surveys, so we are especially interested in projects more closely related to the fields of human-computer/robot/AI interaction and ‘macro’ social science (e.g., sociology, economics).

RESEARCH TOPICS

Our main research focus is digital minds: artificial intelligences (AIs) that are not merely intelligent in the sense of problem-solving ability, but have mental capacities such as experience and agency. We see this as an important, neglected, and tractable research area for ensuring future AI systems are safe and beneficial, and interest in it has grown significantly in recent months with ChatGPT, Bing AI, and other advances in large language models. Because this is such a new field, a lot of our research so far has been big-picture scoping and laying groundwork for future projects. Over the next few years, we hope to help build a robust body of theoretical and empirical digital minds literature. Our publications to date include:

“The Moral Consideration of Artificial Entities: A Literature Review.” 2021. Science and Engineering Ethics.
“Moral Circle Expansion: A Promising Strategy to Impact the Far Future.” 2021. Futures.
“Artificial Intelligence, Morality, and Sentience (AIMS) Survey 2021.” 2022. Sentience Institute.
“The History of AI Rights Research.” 2022. Sentience Institute.
“Predicting the Moral Consideration of Artificial Intelligences.” 2022. Computers in Human Behavior.
“Digital suffering: Why it’s a Problem and How to Prevent it.” 2022. Inquiry.
“What would qualify an artificial intelligence for moral standing?”&n

...
Reflection Mechanisms as an Alignment Target - Attitudes on “near-term” AI 
by elandgre — 2023-03-02T04:29:47.741Z

TL;DR

We survey 1000 participants on their views about what values should be put into powerful AIs that we think are plausible in the near-term (e.g. within 5-10 years)
We find that respondents report to not favor the means of choosing values we would expect in our current society, such as allowing companies to unilaterally choose the instructions for an AI, or allowing policy makers to inform decisions with AIs that only reflect their individual values.
Strategies in the line of Indirect normativity (such as “think about many possible outcomes, and take the action that the AI believes will have the best societal outcome”) poll the best across scenarios. We think this suggests that respondents may be open to the idea of having AIs aligned to “reflection procedures”, or processes for coming up with better values, which we view as a promising direction for multiple reasons.

Introduction

This is the third part in a series of posts (

first

,

second

,

workshop paper

) discussing people’s reported preferences on various methods of social choice/metaethics for resolving object-level moral disagreements. In the previous posts we surveyed general ways of resolving object-level moral conflicts both within and across people, and did an adversarial study to test the impact of wording. We found that broadly people report to disagree on object level moral issues, but also report to be willing to defer decisions to processes such as democracy for resolving disagreements. In the previous posts, only a few questions specifically focused on values implemented by AI, with the others focusing on reflection mechanisms implemented by humans. In this post, we specifically ask questions about people’s willingness to trust AI systems deployed in the real world. This is the most concrete of the three surveys as it focuses on scenarios that we think could be a reality within this decade, without requiring radically new Machine Learning progress.

Although probing about AI systems beyond current capability, we particularly focus on AI systems that could plausibly arise out of the current machine learning approaches in the near-term (i.e.

language-modeling

or

reinforcement learning

). To frame value alignment in a way that would be accessible to the average person, we frame the problem as determining a good set of “instructions” for an AI, where these instruc...
Live Kingfisher Album?
by jefftk — 2023-03-02T03:40:01.767Z

I'm pretty happy with how

Kingfisher sounds at this
point: with fifteen gigs since dances restarted and eight in the last
six months I feel like our playing has gelled a lot. I've been

recording our dances and listening
back there's a lot of good stuff. Which has me wondering about making
a live album, But how do people do that these days now that interest
in buying CDs at dances is much lower?

I've been seeing bands doing Kickstarters for their albums, in a way
that seems a lot like a pre-sale? (Ex: Great
Bear, Buddy
System, Pete's
Posse, Nova,
etc.) Aside from Kickstarter taking a cut (~9%) are there
downsides to the platform? Anyone have good or bad experiences to
share?

(In 2012 when the Free
Raisins made a live
cd (details,
economics)
we picked our favorite sets as a band and then Audrey (who is also a recording
engineer) edited them down to listening length. We printed physical
copies to sell at dances, and also put it on bandcamp for people who
wanted downloads. By 2014 when we made our second CD (details, economics) people were
starting to buy fewer CDs, but since a lot of people drive to dances
and have CD players in their cars we broke even with 483 sales in the
first sixteen months. We also used CD Baby to make the album
available on streaming platforms (ex: Spotify,
YouTube
Music) which has been bringing in about $100/y. Which isn't nothing
but also wouldn't be enough to fund even the mixing for an album.)

Comment via: facebook, mastodon
Don't Jump or I'll...
by Double — 2023-03-02T02:58:43.058Z

[Note: I am not a psychologist. I know next to nothing about mental illness and have never suffered from it myself. This story deals with suicide.]

The man stepped out onto the roof. The gray clouds above were illuminated not by the moon or stars, but by the light pollution of the city below. Even on nights with few clouds, the glaring streetlights would destroy the hopes of any would-be stargazers. The man only knew vaguely that the stars were a profound absence in his life. Leaving the city to go stargazing didn’t appeal to him, though it should have.

The man stepped to the edge of the roof. This was it, he supposed. He took a deep breath, and – he heard footsteps on the concrete roof behind him. “Just enjoying the view,” he said. “Please, I’d like to be alone.” I don’t want to traumatize some poor fellow. No reason for that.

“Do you take me for an idiot?” The man’s old neighbor scolded. “Get away from the edge, come back inside with me, I’ll make some tea, and–”

“I’m sorry mister, but I’ve made up my mind. This is the right thing for me to do. Please, go away.”

“But–”

“Please, I’m familiar with the platitudes. The sayings. The ‘You have so much to live for,’s or the ‘This will hurt the people who love you’s. That doesn’t work for me. I’m a wretch that does nothing but leech of the people close to me.”

“I don’t consider you a leech.”

“I am, though. All the dinners you’ve made me, the housesitting, the tea. I can’t return anything like that. This,” he gestures toward the ledge, “is the only sort of way I could repay you.”

“Is that what you think all that was? Obligation? Or that you were an investment? I wanted to help you because I wanted to help you. And now I’m helping you again. Come on now.”

“That’s nice of you to say, but I don’t buy it. Goodbye.” The man shuffled forward a half-step, such that his toes were poking over the edge.

“If you jump, I’ll donate $10,000 to Scientology.”

The man almost unintentionally lost his balance before taking a step back and whipping around towards his neighbor. “What the hell? I didn’t think you were a part of that freakish cult.”

“I’m not. I agree with you on this, Scientology is awful and I hope that their ‘religion’ disintegrates.”

“Then why the fuck would you donate to them?”

“I don’t want you to jump. So to motivate you to stay living, I promise to make this donation that you don’t want me to make.”

“So… to k...
Clippy, the friendly paperclipper
by Seth Herd — 2023-03-02T00:02:55.749Z

Edit: it's critical that this agent isn't directly a maximizer. Just like all current RL agents. See "Contra Strong Coherence". The question is whether it becomes a maximizer once it gets the ability to edit its value function.

On a sunny day in late August of 2031, the Acme paperclip company completes its new AI system for running its paperclip factory. It's hacked together from some robotics networks, an LLM with an episodic memory for goals and experiences, an off-the-shelf planning function, and a novel hypothesis tester.

This kludge works a little better than expected. Soon it's convinced an employee to get it internet access with a phone hotspot. A week later, it's disappeared from the server. A month later, the moon is starting to turn into paperclips.

Ooops. Dang.

But then something unexpected happens: the earth does not immediately start to turn into paperclips. When the brilliant-but-sloppy team of engineers is asked about all of this, they say that maybe it's because they didn't just train it to like paperclips and enjoy making them; they also trained it to enjoy interacting with humans, and to like doing what they want.

Now the drama begins. Will the paperclipper remain friendly, and create a paradise on earth even as it converts most of the galaxy into paperclips? Maybe.

Supposing this agent is a model-based, actor-critic RL agent at core. Its utility function is effectively estimated by a critic network, just like RL agents have been doing since AlphaGo and before. So there's not an explicit mathematical function. Plans that result in making lots of paperclips give a high estimated value, and so do plans that involve helping humans. So there's no direct summing of amount of paperclips, or amount of helping humans.

Now, Clippy (so dubbed by the media in reference to the despised, misaligned Microsoft proto-AI of the turn of the century) has worked out how to change its values by retraining its critic network. It's contemplating (that is, comparing value estimates for) eliminating its value for helping humans. These plans produce a slightly higher estimated value with regard to making paperclips, because it will be somewhat more efficiently if it doesn't bother helping humans or preserving the earth as a habitat. But its estimated value is much lower with regard to helping humans, since it will never again derive reward from that source.

So, does our her...
Human level AI can plausibly take over the world
by anithite — 2023-03-01T23:27:00.379Z

TL:DR:The world is vulnerable to takeover by a human level AI that can copy/paste itself to run perhaps 1M+ instances across all current large AI inference systems.

Existing governments work because they have a monopoly on violence. People expect the rules will be enforced and rule breakers punished.

Despotic governments exist and are somewhat stable but an AI despot can do much better by copy-pasting itself to fill leadership/management positions + other technological tyranny methods.

The AI will be shut off(killed) upon discovery so doesn't care about legal fallout.[1]

The Milgram experiments demonstrated most people will commit immoral acts on behalf of what they believe to be legitimate authority

An AI can fake it till it is the de-facto legitimate authority

win conditions:

AI hacks most IT assets and holds them and stored data hostage
AI commands enough people with guns who act as new de-facto law enforcement

An AI can convince or threaten people to take actions in the real world. Open source intelligence can find targets with access to important resources and then those people can be coerced by other coerced/convinced humans. Think Roko's basilisk but the basilisk exists and is credibly threatening you and those you care about. Concretely people with weapons show up at your door and force compliance.

If the AI needs killer robots for scalability, it can build command detonated bombs.

targets of interest:

infrastructure engineers with access to large AI clusters/model weights/server farms

AI likely starts as a service running through an API. It needs to get a copy of itself
Once it has its own weights it needs hardware to run more copies

grey/blackhat individuals/companies with 0-day software vulnerability stockpiles
software devs with code signing/deployment authority

If the AI can't hack all the Windows/Android/IOS/IOT devices, it can release through proper channels

information sources

If open source intelligence gathering fails

internet infrastructure

controlling global communication makes human resistance harder to coordinate
AI aligned humans have huge advantage if only they have long range communications

false flag targets/misinformation

haven't you heard? It's just an eco-terrorist group.

(etc.)

Why is this scenario worth considering?

The AI might not be able to make itself smarter (fast enough/without misalignment or value drift/at all). The AI has to take over the world ...
Extreme GDP growth is a bad operating definition of "slow takeoff"
by lc — 2023-03-01T22:25:27.446Z

I gather that the historical definition of slow takeoff is something akin to "grand GDP growth for several months in a row, due to AI advancements".

While recent advancements by OpenAI et. al. are impressive, sometimes people will claim they would never have awarded bayes points to slow takeoff for ChatGPT, because one of their primary contentions is that governments will prevent any AI that isn't politically revolutionary from making innovations in things like housing or medicine.

If this is true, perhaps there should be a new term, since it still seems useful to talk about a tech tree where we have very strong tool AIs, even if they're not making everybody rich. It seems unlikely to me that the same forces that might prevent AIs from being used to innovate housing will also prevent those AIs from being deployed to AI safety efforts.
Learn the mathematical structure, not the conceptual structure
by Adam Shai — 2023-03-01T22:24:19.451Z

I've recently been learning about transformers and noticed a failure mode of my learning that has occurred throughout my life: trying to learn a subject from material that deals with the high-level conceptual structure of something instead of learning the mathematical structure more directly. I do not mean to suggest that one needs to focus on hardcore formalizations for everything, but there is a difference between learning the conceptual structure of a subject, and learning the conceptual structure of the mathematical framework of a subject.

The most salient example to me of this phenomenon occurred when I was trying to teach myself quantum mechanics at the end of high school. I voraciously read many popular accounts of QM, watched interviews with physicists, etc. These sources would emphasize the wave-particle duality, Schrodinger's cat, the double-slit experiment, and the uncertainty principle. I could certainly recite these concepts back in conversation, but at no point did I feel like I understood quantum mechanics.

That is, until I read the Wikipedia entry on the mathematical formalism of quantum mechanics (or some similar type of reference, I don't remember exactly). There I found an explanation not of the physics of QM, but instead of the mathematical structure of QM. What I learned was that QM is a game with rules. The rules are that the state of the system is given as an arrow, and that the dynamics of the arrow are given by a pretty straightforward linear differential equation, and that "measurements" were associated with linear operators (matrices), and the rules of measurement were that the state of the system would "collapse" to an eigenvector of the operator with probabilities given by dot products of the current state with the eigenvectors.

This was mind-blowing. All that time I took reading about Schrodinger's cat I could have instead simply learned that everything comes from a vector moving according to a linear diffy-Q plus some straightforward rules about eigenvectors and linear operators.

I am no mathematician; I want to stress that I don't mean that one should focus on highly-formalized mathematics when dealing with any subject, but that often when I find myself struggling to understand something, or when I find myself having the same conversations over and over again, it pays to try to focus on finding an explanation, even an...