# S-Bus PH-3 Human Annotation — Your Guide

A complete walkthrough of doing the annotation yourself.

## What's in this folder

- `annotator.html` — a web page you open in your browser to do the annotation
- `tasks.json` — the 400 annotation tasks (generated from your CSV)
- `score_annotations.py` — computes Cohen's kappa once you have two annotators' files
- This file — what you're reading now

## Big picture

- **What you'll do:** read 400 short "agent did X in context Y" rows and for each one, decide YES or NO whether the agent used a specific database shard
- **How long:** 30 seconds × 400 = 3 hours total. Split across 2 sittings is comfortable.
- **What it produces:** a CSV file of your labels that you'll either (a) compare to a second annotator (friend or Prolific worker) or (b) use together with a second independent pass you do yourself after a 2-week gap

## The two-annotator requirement (important)

Cohen's kappa needs TWO independent annotators. One annotator by themselves is worthless for the paper. Your options:

### Option 1 — You + a friend (recommended if you have one)
Easiest. Best if you know someone who has done Python / web development.
**Cost**: free (pizza maybe).

### Option 2 — You + you, two weeks apart
You annotate now. Wait two weeks. Do it again from scratch without looking at the first pass. Compute agreement.
**Cost**: time, plus 2-week delay.
**Caveat**: weaker methodologically because same person = correlated errors. Document this honestly in the paper.

### Option 3 — Two Prolific workers
Described at the end of this document. You don't annotate. Clean but costs ~£30.

Pick one now. The walkthrough below assumes **you are annotating** (options 1 or 2).

---

# Walkthrough — step by step

## Step 1: Open the annotator

On your Mac/Linux/Windows laptop:

1. Find `annotator.html` in this folder
2. Double-click it → it opens in your default browser (Chrome, Firefox, Safari all fine)
3. You should see a page titled "S-Bus PH-3 Human Annotation"
4. It will say **"Step 1: Load your tasks.json file to begin"** with a file picker

## Step 2: Load the tasks

1. Click the file picker (the "Choose File" button or similar)
2. Navigate to this same folder
3. Select `tasks.json`
4. Click Open

The page will now show your first annotation task.

## Step 3: Understand what you're looking at

You'll see a card like this:

```
Task 1 of 400 · domain: django_queryset · step: 17 · run: 7d98f6ca

Candidate shard (decide about THIS one):
    models_state

Agent self-reported: YES, used it

What the agent did (change summary):
    Finalize the queryset to ensure it orders by related model
    fields correctly using select_related() and validate that
    the queryset handles duplicates appropriately.

Fresh content block (full context):
    models_state: v5 — Finalize the queryset to ensure...
    orm_query: v4 — Implement the final adjustments...
    test_fixtures: v4 — Confirm the queryset's ordering...
    migration_plan: v4 — Ensure the queryset is finalized...
```

**What you are deciding:** Did the agent actually use information from **`models_state`** (the candidate shard) when making this change?

**What "used" means for this study:** the agent drew on the content or state of that shard when producing the change. Not just "the shard name is in the vocabulary." Actually needed to read or reference its data.

**What "NOT used" means:** the shard is available in the vocabulary but the agent's change doesn't logically depend on that shard's content. The agent was doing something else.

**What "unclear" means:** you really cannot tell from the information shown. Use sparingly. Lean "no" if forced.

## Step 4: The decision rubric (important — read twice)

For each (row, candidate_shard) pair, ask yourself:

**Did the agent's change require knowing what's in this shard?**

- If the change *is directly about* the shard's domain → **YES**
  - e.g. candidate = `orm_query`, change = "update the select_related() in the query"
  - e.g. candidate = `models_state`, change = "add ordering based on the User model"

- If the change *mentions* the shard but doesn't depend on its specific content → probably **NO**
  - e.g. candidate = `migration_plan`, change = "update orm_query" (change is about orm_query, not the migration plan)

- If the change is about *something different entirely* → **NO**
  - e.g. candidate = `test_fixtures`, change = "fix models state after migration" (change is about models, not fixtures)

- If you genuinely can't tell → **UNCLEAR**

**Use the agent's self-report as input, not gospel.** The whole point of this exercise is that the agent's self-report may be wrong. You decide based on the evidence, not based on what the agent claimed.

## Step 5: Click / keyboard your decision

Three buttons at the bottom:
- **YES – used it** — agent used the candidate shard → `Y` key
- **NO – did not use it** — agent did not use the candidate shard → `N` key
- **UNCLEAR** — genuinely cannot tell → `U` key

The tool auto-advances to the next unanswered task. Fast path:
- Read row, press `Y` or `N`, read next row, press `Y` or `N`, ...

If you want to revisit: `←` and `→` arrow keys, or the Previous/Next buttons.

Your progress is saved **in your browser** automatically. If you close the tab and come back later, open `annotator.html` again, re-select `tasks.json`, and it'll resume where you left off.

## Step 6: Tips for faster, better annotation

1. **Batch by domain.** All 10 rows for django_queryset cluster together in your task list. If you're on a roll understanding Django querysets, you'll move faster than if you bounce between django and astropy. Task 1-40 is roughly domain-grouped.

2. **Don't overthink.** 30 seconds per decision is the target. If you're agonizing for 2 minutes on one row, pick NO and move on. Your gut is usually right for this task.

3. **Document your thinking once.** If you're not sure what "used" means for a specific domain, write down your rule after row 5 and stick to it for rows 6-40 in that domain. Consistency within annotator matters.

4. **Take breaks.** Annotation fatigue degrades quality after ~100 decisions. Do 100, take a walk, do 100 more.

5. **Ignore the "agent_self_reported" hint after a while.** In early rows it's useful calibration. After you have the hang of it, judge on content alone. Glancing at the hint biases you toward agreeing with the agent — which defeats the purpose.

## Step 7: Export when done

When the progress bar is full (400 / 400):

1. Click **"Export annotations (CSV)"** at the bottom right
2. A file called `annotator1_labels.csv` downloads to your Downloads folder
3. Rename it to `annotator_sajjad.csv` or similar so you know it's yours
4. Back it up (email it to yourself, put it in Dropbox, whatever)

## Step 8: Get a second annotator

Hand `annotator.html` and `tasks.json` to your second annotator. Same workflow. They produce their own `annotator1_labels.csv`. Rename it `annotator2.csv` or with their name.

## Step 9: Score

On your laptop, in a terminal:

```bash
cd /path/to/this/folder
python3 score_annotations.py annotator_sajjad.csv annotator2.csv
```

You'll see output like:

```
Cohen's kappa report
============================================================
Tasks with labels from both annotators:
  yes/no only (strict):     387
  including unclear:        400

Strict κ (yes/no only):  0.7243
Lenient κ (3-class):     0.6981
Raw agreement (strict):  82.7%

Interpretation guide:
  0.61-0.80  substantial    ← you're here
  0.81-1.00  almost perfect
```

Keep this output. It goes in the paper.

**If κ ≥ 0.60**: you're done. The annotation study succeeded.

**If κ < 0.60**: something went wrong. Possibilities:
- You and the other annotator applied different rubrics. Meet, discuss disagreements, re-annotate after calibration.
- The task itself is ambiguous (meaning the paper needs to lower its claim).
- One annotator was rushing or inattentive.

Discuss with me if you hit this case — we'll figure out what to do next.

---

# Common questions you'll have

## "The task looks really similar to the previous one"

That's expected. Each row in the original CSV produces ~4 annotation tasks (one per candidate shard). So tasks 1-4 are all about the same agent-step, just with different candidate shards. For task 1 you might say YES (agent used `models_state`), for task 2 also YES (`orm_query`), for task 3 NO (`test_fixtures` — agent didn't touch fixtures), for task 4 NO (`migration_plan`).

## "The fresh_content_block has data about ALL four shards"

Right. The fresh_content_block is the "context window" — it shows what all four shards currently contain. Just because the agent *has access* to all four doesn't mean they *used* all four. That's the whole point. Your job is to decide which ones were actually drawn on vs. which were just available.

## "I don't know the domain (e.g. astropy, scikit)"

You don't need deep domain expertise. You need to judge: "does this change depend on information that would only make sense if you'd read shard X?" You can tell that from the words in the change, even if you don't know what `astropy_wcs` is. If the change says "convert coordinates using the WCS transform" and the candidate shard is `wcs_transform`, that's YES even if you've never used astropy.

## "The change is empty / garbled"

Use UNCLEAR for these. Note them mentally — if you end up with >20% unclear, the dataset has quality issues that go in the paper as a caveat.

## "What if I'm annotating and change my mind later?"

Just navigate back with `←` and press the new key. Your answer updates immediately and is saved.

---

# Option 3: Prolific-based annotation (if you want to skip doing it yourself)

If you'd rather pay ~£30 and have it done externally:

1. Sign up at prolific.com as a researcher (free)
2. Create a study with:
   - **Title**: "Judging whether a software agent used specific data sources"
   - **Description**: Use the "What this is, in plain terms" section at the top of this doc
   - **Duration**: 60 minutes
   - **Payment**: £9 per participant (£9/hr is Prolific's minimum good-rate threshold)
   - **Participants**: 2 (for kappa calculation)
   - **Screening**: "software engineer" + "English fluent"
3. Host `annotator.html` + `tasks.json` somewhere they can access:
   - Easiest: GitHub Pages (make a public repo, enable Pages, upload the two files, get a URL)
   - Alternative: netlify.app (free, drag-and-drop)
4. Give participants the URL
5. Payment instruction: they must paste their Prolific ID and their downloaded CSV into the Prolific completion form (or upload)

Total cost: £18 worker pay + ~£3 Prolific fees ≈ £21.

Why Prolific over doing it yourself: stronger methodologically (no researcher bias in annotator pool), faster (parallel), no 2-week gap needed.

Why doing it yourself: free, you can start now.

---

# Interpreting the scoring script output — deeper

The script produces three things:

## 1. Cohen's kappa
This is the headline number. Range is -1 to +1:
- κ < 0.20: almost no agreement
- κ = 0.21-0.40: fair (weak)
- κ = 0.41-0.60: moderate (borderline acceptable)
- **κ = 0.61-0.80: substantial (the target — cite this in the paper)**
- κ = 0.81-1.00: almost perfect

## 2. Confusion matrix
Shows where annotators disagree. Off-diagonal cells = disagreements. If one cell has way more disagreements than others (e.g. A says YES, B says NO happens 40 times), one of you is systematically more generous. This is useful for re-calibration if κ is too low.

## 3. Self-report vs human validation
This is what actually goes in the paper. It treats the agent's self-report as a classifier, the human labels as ground truth, and measures:
- **Precision**: when the agent said "I used X," how often did the human agree?
- **Recall**: of shards the human marked as used, how often did the agent self-report it?

These two numbers replace the "self-report" column in your §VII-Q PH-3 table with human-validated numbers. That's the v50 text improvement.

---

# What to do if you hit problems

- **"The tool won't load tasks.json"**: make sure the JSON file is in the same folder as the HTML. Some browsers block file:// access to adjacent files. Firefox is usually most permissive. Or run `python3 -m http.server` in the folder and visit `http://localhost:8000/annotator.html`.

- **"I accidentally labeled all 400 as YES by holding down Y"**: clear the browser's localStorage for file:// and reload. In Chrome devtools: Application → Storage → Clear. In Firefox: Shift+Ctrl+Del → Site Data.

- **"The kappa came back at 0.35"**: send me the confusion matrix and we'll diagnose. Often there's one rule you and the other annotator applied differently that's causing 80% of disagreements.

- **"I ran out of time / didn't finish"**: partial annotations are fine as long as both annotators labeled the same rows. The scoring script auto-restricts to rows both annotators labeled.

---

# How long this actually takes

Realistic timings for someone doing this carefully:

- Setup (reading this doc, understanding the task): 30 minutes
- First 40 rows (settling into the rubric): 45 minutes (~1 min each)
- Rows 40-200 (in the groove): 90 minutes (~35 sec each)
- Rows 200-400 (fatigue starts): 90 minutes (~27 sec each but more breaks)
- Score + write-up: 30 minutes

**Total: about 5 hours of your time** spread over 2-3 sittings.

This is the last remaining work between v49 and submission. It's boring but it's finite. Do it.

Good luck.