something readable

Dockerfile

@@ -1,8 +1,8 @@
 FROM node:latest
 
-RUN useradd -o -u 1000 user
+RUN arch # print arch info
 
-RUN arch
+RUN useradd -o -u 1000 user
 
 USER user
 

docs/data/presse.parquet.sh

@@ -1,6 +1,10 @@
+# file_id, ocr, title, date, author, page_count, word_count, character_count
+
 echo """
 CREATE TABLE presse AS (
-SELECT file_id, ocr, title, date, author, page_count, word_count, character_count
+SELECT title
+     , author
+     , LPAD((REGEXP_EXTRACT(date, '1[0-9][0-9][0-9]') || '-01-01'), 10, '0')::DATE AS year
   FROM read_parquet([""" > $TMPDIR/presse.sql 
 
 for i in $(seq 1 320); do
@@ -10,8 +14,6 @@ done
 echo """  ])
 );
 
-ALTER TABLE presse ALTER COLUMN date TYPE DATE; 
-
 COPY presse TO '$TMPDIR/presse.parquet' (FORMAT 'parquet', COMPRESSION 'GZIP');
 """ >> $TMPDIR/presse.sql
 

docs/gazette.md

@@ -0,0 +1,55 @@
+# Gazette
+
+```js echo
+import { DuckDBClient } from "npm:@observablehq/duckdb";
+const db = DuckDBClient.of({ presse: FileAttachment("data/presse.parquet") });
+```
+
+This page allows you to explore the 3 million newspapers by title. I called it “Gazette” because I was surprised that most of the corpus in the earlier years had a title containing this word.
+
+Type in words such as “jeune”, “révolution”, “république”, “soir”, “fille”, “femme”, “paysan”, “ouvrier”, “social”, etc., to see different historical trends.
+
+```js
+const search = view(
+  Inputs.text({ type: "search", value: "gazette", submit: true })
+);
+```
+
+```js echo
+display(
+  Plot.plot({
+    x: { nice: true },
+    y: {
+      label: `Share of titles matching ${search}`,
+      tickFormat: "%",
+    },
+    marks: [
+      Plot.areaY(gazette, {
+        x: "year",
+        y: (d) => d.matches / d.total,
+        fillOpacity: 0.2,
+      }),
+      Plot.lineY(gazette, {
+        x: "year",
+        y: (d) => d.matches / d.total,
+      }),
+    ],
+  })
+);
+```
+
+The query uses the [REGEXP_MATCHES](https://duckdb.org/docs/archive/0.9.2/sql/functions/patternmatching) operator to count occurrences; you can query for example “socialis[tm]e” to match both “socialiste” and “socialisme”. The 'i' flag makes it ignore case.
+
+```js echo
+const gazette = db.query(
+  `SELECT year
+       , SUM(CASE WHEN REGEXP_MATCHES(title, ?, 'i') THEN 1 ELSE 0 END)::int matches
+       , COUNT(*) total
+    FROM presse
+   WHERE year > '1000'
+GROUP BY year
+ORDER BY year
+`,
+  [search]
+);
+```

docs/index.md

@@ -1,218 +1,43 @@
-# Presse-public-domain
+# FPDN exploration
 
-I'm fascinated by this new dataset https://huggingface.co/datasets/PleIAs/French-PD-Newspapers that just dropped on :hugging_face:.
+A new fascinating dataset just dropped on 🤗. [French public domain newspapers](https://huggingface.co/datasets/PleIAs/French-PD-Newspapers) references about 3 million newspapers and periodicals with their full text OCR’ed and some meta-data.
 
-It references about 3 million French newspapers, with full-text. ("only" 320 files of about 700MB each :slightly_smiling_face:).
+The data is stored in 320 chunks weighting about 700MB each, each continaing about 7,500 texts.
 
-I wrote a data loader that outputs a single parquet file combining all their metadata (that is, without the text contents). It takes only about 5 minutes to run, thanks to parquet magic (and fiber internet); not sure how much of the ~200+GB I downloaded for that. Now I've started exploring the metadata in an observable project.
+The data loader for this Observable project uses DuckDB to read these 320 parquet files (altogether about 200GB) and combines a minimal subset of their metadata — title and year of publication, most importantly without the text contents —, into a single parquet file. It takes only about 1 minute to run in a hugging-face Space.
 
-In one query, I see that A LOT of publications stopped publishing in 1944/1945, and conversely a large number of newspapers started publishing between 1941 and 1946. This probably includes both collaborationist publications and resistance publications—it would be interesting to find a ML way to separate them.
+The resulting file is small enough (about 8MB) that we can load it in the browser and create “live” charts with Observable Plot.
 
-In another exploration, I see that the word "gazette" was quite frequent in the 17th Century.
+In this project, I’m exploring two aspects of the dataset:
 
-Not sure what to look for, but making these queries in an Observable project is really nice.
+- As I played with the titles, I saw that the word “gazette” was quite frequent in the 17th Century. An exploration of the words used in the titles is on the page [gazette](gazette).
 
-```js
-import { DuckDBClient } from "npm:@observablehq/duckdb";
-const db = DuckDBClient.of({ presse: FileAttachment("data/presse.parquet") });
-```
-
-```js
-const selection = db.query(
-  `SELECT * FROM presse WHERE date >= '1600' AND date < '1820'`
-);
-```
-
-```js
-const letters = [];
-const dates = [];
-
-for (const { date, title } of selection) {
-  if (title) {
-    for (let l of [...title]) {
-      l = l.toUpperCase();
-      if (l >= "A" && l <= "Z") {
-        const d = +d3.isoParse(date);
-        if (d) {
-          dates.push(d);
-          letters.push(l);
-        }
-      }
-    }
-  }
-}
-```
-
-```js
-// display(letters);
-// display(dates);
-```
-
-```js
-display(
-  Plot.plot({
-    marks: [
-      Plot.areaY(
-        letters,
-        Plot.stackY(
-          { offset: "normalize", _o_rder: "value" },
-          Plot.binX(
-            {
-              y: "count",
-              filter: null,
-            },
-            {
-              x: dates,
-              thresholds: "5 years",
-              fill: letters,
-              tip: true,
-            }
-          )
-        )
-      ),
-    ],
-  })
-);
-```
-
-```js
-const search = view(Inputs.text({ type: "search", placeholder: "search…" }));
-```
-
-```js
-display(selection);
-```
+- A lot of publications stopped or started publishing during the second world war. Explored in [resistance](resistance).
 
-```js
-const test = new RegExp(search, "i");
-```
+This page summarizes the time distribution of the data:
 
-```js
-display(
-  Plot.plot({
-    marginLeft: 60,
-    marks: [
-      Plot.rectY(
-        selection,
-        Plot.binX(
-          {
-            y: "count",
-          },
-          {
-            x: "date",
-            fill: (d) => d.title && test.test(d.title),
-            thresholds: "1 year",
-          }
-        )
-      ),
-    ],
-  })
-);
-```
-
-```js
-const authors = db.query(
-  `
-SELECT author
-     , MIN(date) AS start
-     , MAX(date) AS end
-  FROM presse
- WHERE date <> 'None'
- GROUP BY 1
-`
-);
-```
-
-```js
-display(authors);
+```js echo
+import { DuckDBClient } from "npm:@observablehq/duckdb";
+const db = DuckDBClient.of({ presse: FileAttachment("data/presse.parquet") });
 ```
 
-```js
-display(
-  Plot.plot({
-    x: { type: "utc" },
-    marks: [
-      Plot.ruleY(authors, {
-        x1: "start",
-        x2: "end",
-        y: "author",
-        stroke: (d) => d3.isoParse(d.end) - d3.isoParse(d.start),
-        sort: {
-          y: "stroke",
-        },
-      }),
-    ],
-  })
-);
+```js echo
+const dates = db.query("SELECT year FROM presse WHERE year >= '1000'");
 ```
 
-```js
-display(
-  Plot.plot({
-    x: { type: "utc" },
-    marks: [
-      Plot.rectY(
-        authors,
-        Plot.binX(
-          { y: "count" },
-          {
-            x: "start",
-            fill: (d) => d3.isoParse(d.end)?.getUTCFullYear(),
-            thresholds: "1 year",
-            tip: true,
-          }
-        )
-      ),
-    ],
-  })
-);
-```
+_Note: due to the date pattern matching I’m using, unknown years are marked as 0000. Hence the filter above._
 
-```js
-display(
-  Plot.plot({
-    x: { type: "utc" },
-    marks: [
-      Plot.rectY(
-        authors,
-        Plot.binX(
-          { y: "count" },
-          {
-            x: "end",
-            fill: (d) => d3.isoParse(d.start)?.getUTCFullYear(),
-            thresholds: "1 year",
-            tip: true,
-          }
-        )
-      ),
-    ],
-  })
-);
-```
+The chart below indicates that the bulk of the contents collected in this database was published between 1850 and 1950. It’s obviously not that the _presse_ stopped after 1950, but because most of the printed world after that threshold year is still out of reach of researchers, as it is “protected” by copyright or _droit d’auteur._
 
-```js
-const aroundWar = db.query(
-  `SELECT * FROM presse WHERE date >= '1920' AND date < '1970'`
-);
-```
+${Plot.rectY(dates, Plot.binX({ y: "count" }, { x: "year", interval: "5 years" })).plot({ marginLeft: 60 })}
 
-```js
-display(
-  Plot.plot({
-    x: { type: "utc" },
-    marks: [
-      Plot.rectY(
-        aroundWar,
-        Plot.binX(
-          { y: "count" },
-          {
-            x: "date",
-            thresholds: "1 year",
-            tip: true,
-          }
-        )
-      ),
-    ],
-  })
-);
+```js echo run=false
+Plot.plot({
+  marks: [
+    Plot.rectY(
+      dates,
+      Plot.binX({ y: "count" }, { x: "year", interval: "5 years" })
+    ),
+  ],
+});
 ```

docs/resistance.md

@@ -0,0 +1,107 @@
+# Resistance
+
+During the second world war the nazis occupied the northern half of France, and the collaborationist governement of Pétain was left to rule over the southern half (the “[zone libre](https://fr.wikipedia.org/wiki/Zone_libre)”). A lot of newspapers at that time were closed, others submitted to the occupiers (some even enthusiastically collaborated). At the same time, a range of clandestine publications started circulating, often associated with the resistance movements. When the country was Liberated in 1944, the most outrageously collaborationist press was dismantled, other newspapers changed their names and were sometimes taken over by new teams of resistance journalists. The most famous case is “Le Temps,” a daily newspaper that had been [publishing since 1861](<https://fr.wikipedia.org/wiki/Le_Temps_(quotidien_fran%C3%A7ais,_1861-1942)>) and had closed in 1942. Although not a collaborationist newspaper, it was not allowed to reopen, and its assets were transferred to create “Le Monde” on 19&nbsp;December 1944, under Hubert Beuve-Méry.
+
+```js echo
+import { DuckDBClient } from "npm:@observablehq/duckdb";
+const db = DuckDBClient.of({ presse: FileAttachment("data/presse.parquet") });
+```
+
+```js echo
+const letemps = db.query(
+  "SELECT year FROM presse WHERE title = 'Le Temps' AND year > '1000'"
+);
+```
+
+```js echo
+display(
+  Plot.plot({
+    caption: "Number of issues of Le Temps in the dataset, per year",
+    x: { nice: true },
+    y: { grid: true },
+    marks: [
+      Plot.ruleY([0]),
+      Plot.rectY(
+        letemps,
+        Plot.binX({ y: "count" }, { x: "year", interval: "year" })
+      ),
+    ],
+  })
+);
+```
+
+(Unfortunately, “Le Monde” is not part of the dataset.)
+
+The number of titles that stopped or started publishing exploded in those fatal years. Note that many of these publications were short-lived, such as this example picked at random in the dataset: [Au-devant de la vie. Organe de l'Union des jeunes filles patriotes (UJFP), Région parisienne](https://gallica.bnf.fr/ark:/12148/bpt6k76208732?rk=21459;2). While the the UJFP (a resistance organisation of communist young women) published several titles during the war, only one issue was distributed under this title.
+
+```js echo
+const years = db.query(`
+  SELECT title
+       , MIN(year) AS start
+       , MAX(year) AS end
+    FROM presse
+   GROUP BY 1
+`);
+```
+
+```js echo
+display(
+  Plot.plot({
+    color: { legend: true },
+    marks: [
+      Plot.rectY(
+        years,
+        Plot.binX(
+          { y: "count" },
+          {
+            filter: (d) =>
+              d.start?.getUTCFullYear() >= 1930 &&
+              d.start?.getUTCFullYear() <= 1955,
+            x: "start",
+            fill: () => "started",
+            interval: "year",
+          }
+        )
+      ),
+      Plot.rectY(
+        years,
+        Plot.binX(
+          { y: "count" },
+          {
+            filter: (d) =>
+              d.end?.getUTCFullYear() >= 1930 &&
+              d.end?.getUTCFullYear() <= 1955,
+            x: "end",
+            fill: () => "ended",
+            mixBlendMode: "multiply",
+            interval: "year",
+          }
+        )
+      ),
+      Plot.ruleY([0]),
+    ],
+  })
+);
+```
+
+Let’s focus on the ${start1944.length} publications that started publishing in 1944, and extract their titles and authors:
+
+```js echo
+const start1944 = db.query(`
+  SELECT title
+       , CASE WHEN author='None' THEN '' ELSE author END AS author
+       , DATE_PART('year', MIN(year)) AS start
+       , DATE_PART('year', MAX(year)) AS end
+       , COUNT(*) AS issues
+    FROM presse
+   GROUP BY 1, 2
+  HAVING DATE_PART('year', MIN(year)) = 1944
+   ORDER BY issues DESC
+`);
+```
+
+```js
+display(Inputs.table(start1944));
+```
+
+Going through these titles, one gets a pretty impressive picture of the publishing activity in this extreme historic period.