Social Learning Journal - Parsing Audiobooks
Tags: python, learningjournal, aggregates
Published: 2020-12-07 01:40:02

---
title: Social Learning Journal - Parsing Audiobooks
published: true
description: For several years now, I have been documenting all of my professional learning activities on Twitter. As an established software engineer, I want to share the process it takes to get here.
tags: python, learningjournal, aggregates
cover_image: https://dev-to-uploads.s3.amazonaws.com/i/8mcewc9e9uw0teoa1ib8.jpg
---

Since January of 2017, I have been documenting all of my professional learning activities on Twitter. As an established software engineer, I want to share the process it takes to get here. Here is my Twitter account, [dev3l_](https://twitter.com/dev3l_).

I use structured messages for journal-specific events. For example, when I listen to a podcast I start the Tweet with "Listened to:". I have similar semantic tags for reading books, listening to books, and attending conferences/courses. In addition, I hashtag the Tweets with classification identifiers, such as #agile. Typically, I try to write a few notes about the event. Finally, the event is tagged with a duration using the carrot symbol and an amount, such as "^45m" to signify 45 minutes in length.

At one point, Gary Vaynerchuk had a video called [Document, Don't Create](https://www.youtube.com/watch?v=RVKofRN1dyI). This has been my attempt to do such.

# New Project Setup

I am starting off based upon my previous post, [Message bot to find a PS5 on sale](https://dev.to/dev3l/message-bot-to-find-a-ps5-on-sale-2n5k). I intend this to be a [Python](https://www.python.org/) [Flask](https://flask.palletsprojects.com/en/1.1.x/) API using [MongoDB](https://www.mongodb.com/) that will support a ReactJS front end. For today, I want to get CI setup with [Travis](http://travis-ci.org/), coverage using [Coveralls](https://coveralls.io/), code quality on [Code Climate](https://codeclimate.com/), and the whole thing hosted on [Heroku](https://dashboard.heroku.com/apps). If anyone has questions on how to set these technologies up, leave a comment below and I will create some more detailed steps.

# Outcomes

For today, my goal is to get a Twitter data archive parsed to create a list of Audiobooks I have listened to for the year. On Twitter, you can go to your _Settings and privacy_ and request an archive download of all your data. This is what I will be using as a starting point, seed data. Inside this archive, is a **tweet.js** file that contains every Tweet from the account in JSON format.

The JSON structure of a Tweet from an archive and the Twitter API is identical. By creating a Tweet parser as the first step using a static export, this will be able to be leveraged in a future scheduled job that will dynamically handle Tweets in near real-time.

In the future, I imagine pulling from data sources outside of Twitter. Any platform with an available API could be used, like GitHub, LinkedIn, and YouTube.

# Structured Messages

When I first started doing this, I did not know the importance of structured messages. A few years ago, I went to create an [initial prototype](https://s3.amazonaws.com/dev3l/learning_journal/twitter_learning_journal_1m.html) and realized it was pretty hard to pull meaningful data out - it took a lot of hand manipulation. Given an appropriate classifier with machine learning, it would have been possible, but for now, it is much easier to just add a little bit of metadata to each message.

![Alt Text](https://dev-to-uploads.s3.amazonaws.com/i/smjdzieyrhu847ssrb44.jpeg)

Instead of working with the file **tweet.js** as a JavaScript file, I removed the JS `window.YTD.tweet.part0 = ` from the file and save it as **tweet.json**. This can easily be imported into Python as a JS document and we can start working on it relatively easily.

```
import json
with open(DATA_SEED_TWITTER_PATH) as data_seed:
    data = json.load(data_seed)
```

With three lines of code, I now have access to start manipulating my 4556, at the time of this article, journaled events.

This is a lot of data when all I want is just to see the audiobooks for this year. Next, let's filter out the data set to show only items from this year based upon the `created_at` attribute.

```
def filter_by_this_year(tweet: dict) -> bool:
    created_at = parse(tweet['tweet']['created_at'])
    return first_of_year <= created_at <= end_of_year

tweets_from_this_year = list(filter(filter_by_this_year, data))
```

Now our data is a little bit more manageable at 449 events. Applying another filter, I'll look for the text of "Started listening to:" to whittle it down to the list of the books I am interested in.

```
def filter_by_audiobook_start(tweet: dict) -> bool:
    text = tweet['tweet']['full_text']
    return "Started listening to:" in text
audio_books_from_this_year = list(filter(filter_by_audiobook_start, tweets_from_this_year))
```

At this point, I have found 11 books. This makes sense to me as I have an Audible subscription that allows for one book a month. I'm not perfect with my annotations and sometimes log the end of a book without marking the start of the book. So I changed the filter to include "Finished listening to:" and found one other book.

```
"Started listening to:" in text or "Finished listening to:" in text
```

# Audiobook List

Now that we have identified the list of books. It's time for a bit of string manipulation to get to the title. After the type identifier tag, I put the name of the book followed by a newline. Using a reduce function, we can easily put these titles into a set and create a unique list of books listened to.

```
def reduce_book_titles(result: set, tweet: dict) -> set:
    text = tweet['tweet']['full_text']
    title = text.split(":")[1].split("\n")[0]
    result.add(title)
    return result
audio_book_titles = list(reduce(reduce_book_titles, audio_books_from_this_year, set()))
```

Looping through and printing the titles I can see that I have the following audiobooks under my belt for the year. The full source to this simple yet powerful script can be found here: [parse-audiobook-tweets-for-this-year.py](https://github.com/DEV3L/python-heroku-learning-journal-api/blob/master/scripts/parse-audiobook-tweets-for-this-year.py)

## 2020 Audiobooks:
- The Unicorn Project - A Novel About Developers, Digital Disruption, and Thriving in the Age of Data
- Talking to Strangers
- Understanding Software - Simplicity, Coding, and How to Suck Less as a Programmer
- Escaping the Build Trap - How Effective Product Management Creates Real Value
- Good to Great - Why Some Companies Make the Leap...And Others Don't
- Agile Conversations - Transform Your Conversations, Transform Your Culture
- Creativity, Inc. - Overcoming the Unseen Forces That Stand in the Way of True Inspiration
- Doing Agile Right - Transformation Without Chaos
- The Pragmatic Programmer
- The 7 Habits of Highly Effective People - Powerful Lessons in Personal Change
- Sense & Respond - How Successful Organizations Listen to Customers and Create New Products Continuously
- The Infinite Game
- The Art of War