Attention conservation notice: A post on writing a small technical hack to improve what ideally I could do without needing a hack.

I do most of my learning by reading articles, guides, and blog posts online, and I manage this using Pinboard.1 All the links I’ve read or want to read in the future live there.

Problem

My read later list was growing a lot faster than I could go through it.

I rarely felt like going to my account to choose an article to read. When I did, I faced choice paralysis. I would scan the links and not feel like starting any of them. The problem was static friction.

One way I tried to solve this was to use Pinboard’s “random article” bookmarklet, which opened a randomly chosen unread link from your account. This worked to an extent, but I would sometimes land at an article that needed more time or attention than I had and I would click the bookmarklet again. Once you start making exceptions and spinning again, it becomes easy to do what is effectively scanning many articles before actually reading one.

I realized what I wanted was somewhere in between: I want to see some options that were randomly chosen.

Solution

My solution is punread which is built on top of BitBar.

BitBar (by Mat Ryer - @matryer) lets you put the output from any script/program in your Mac OS X Menu Bar.

Go to the link to see some screenshots and examples. The idea is that you write a script that produces an output and tell BitBar how often you want it run. There’s a lot of syntax available for you to control the output, how it looks, what happens when you click it, etc.

punread shows the number of unread bookmarks in my menu bar, and when I click on the number, I see 30 randomly chosen links. I can click on one, read it in the browser, and then mark it as read using another one of Pinboard’s bookmarklets.

punread is two files, the first is punread.30m.sh, which is the shell script BitBar wants to have:

#!/bin/bash
# <bitbar.title>punread</bitbar.title>
# <bitbar.version>v1.0</bitbar.version>
# <bitbar.author>Sherif Soliman</bitbar.author>
# <bitbar.author.github>sheriferson</bitbar.author.github>
# <bitbar.desc>Show pinboard unread count</bitbar.desc>
# <bitbar.dependencies>python</bitbar.dependencies>
# <bitbar.abouturl>https://github.com/sheriferson/punread</bitbar.abouturl>

links=$(/usr/local/bin/python3 /Users/sherif/projects/punread/punread.py)
echo "$links"

echo "---"
echo "📌 Random article | href=https://pinboard.in/random/?type=unread"

It doesn’t do much. It runs the second file, punread.py and shows its output. It also tacks on a final menu item that will show me a random unread article in case I didn’t like any of the 30 already listed. I don’t think I’ve ever used that option.

The second file is punread.py, which does most of the work. It talks to the Pinboard API, saves some state, and returns the 30 links for BitBar to display.

import json
import os.path
import pickle
import random
import re
import requests
import sys
import time

# get the path to punread.py
pathToMe = os.path.realpath(__file__)
pathToMe = os.path.split(pathToMe)[0]

last_updated_path = os.path.join(pathToMe, 'lastupdated.timestamp')
unread_count_path = os.path.join(pathToMe, 'unread.count')
links_path = os.path.join(pathToMe, 'links')
api_token_path = os.path.join(pathToMe, 'api_token')
last_run_path = os.path.join(pathToMe, 'lastrun.timestamp')

backup_file = '/Users/sherif/persanalytics/data/unread_pinboard_counts.csv'

def print_random_unread_links(count, unread, n = 30):
    count = str(count) + ' | font=SourceSansPro-Regular color=cadetblue\n---\n'
    sys.stdout.buffer.write(count.encode('utf-8'))
    random_unread_indexes = random.sample(range(1, len(unread)), 30)
    for ii in random_unread_indexes:
        description = unread[ii]['description']
        description = description.replace("|", "｜")
        link_entry = '📍 ' + description + " | href=" + unread[ii]['href'] + " font=SourceSansPro-Regular color=cadetblue\n"
        sys.stdout.buffer.write(link_entry.encode('utf-8'))

def log_counts(total_count, unread_count):
   """
   A function to write the time, total bookmark count, and unread bookmark count
   to a csv file.
   """
   now = int(time.time()) 
   row = str(now) + ',' + str(total_count) + ',' + str(unread_count) + '\n'

   with open(backup_file, 'a') as bfile:
       bfile.write(row)

# check if there's a lastrun.timestamp, and if it's there
# check if the script ran less than 5 mins ago
# if yes, quit
if os.path.isfile(last_run_path):
    last_run = pickle.load(open(last_run_path, 'rb'))
    if time.time() - last_run < 300:
        unread_count = pickle.load(open(unread_count_path, 'rb'))
        links = pickle.load(open(links_path, 'rb'))
        unread = [link for link in links if (link['toread'] == 'yes')]
        print_random_unread_links(unread_count, unread)
        exit()
    else:
        pickle.dump(time.time(), open(last_run_path, 'wb'))
else:
    pickle.dump(time.time(), open(last_run_path, 'wb'))

with open(api_token_path, 'rb') as f:
    pintoken = f.read().strip()

par = {'auth_token': pintoken, 'format': 'json'}

if os.path.isfile(last_updated_path) and os.path.isfile(unread_count_path):
    last_updated = pickle.load(open(last_updated_path, 'rb'))
    unread_count = pickle.load(open(unread_count_path, 'rb'))
    links = pickle.load(open(links_path, 'rb'))
else:
    last_updated = ''
    unread_count = 0

last_updated_api_request = requests.get('https://api.pinboard.in/v1/posts/update',
        params = par)

last_updated_api = last_updated_api_request.json()['update_time']

if last_updated != last_updated_api:
    r = requests.get('https://api.pinboard.in/v1/posts/all',
            params = par)

    links = json.loads(r.text)

    unread = [link for link in links if (link['toread'] == 'yes')]
    total_count = len(links)
    unread_count = len(unread)

    pickle.dump(last_updated_api, open(last_updated_path, 'wb'))
    pickle.dump(unread_count, open(unread_count_path, 'wb'))
    pickle.dump(links, open(links_path, 'wb'))

    log_counts(total_count, unread_count)
    print_random_unread_links(unread_count, unread)
else:
    unread = [link for link in links if (link['toread'] == 'yes')]
    print_random_unread_links(unread_count, unread)

There are too many lines of code for me to walk through this step by step, but I’ll paint a general picture.

Some notes and things I had to keep in mind while writing the script:

  • The Pinboard API has rate limits. I can’t hit the posts/all method more than once every five minutes.
  • Pinboard recommends you use the API token to authenticate, rather than regular HTTP auth. I keep my API token in a file that I added to .gitignore so I don’t accidentally publish it somewhere.
  • I wanted to keep track of the total number of bookmarks and unread bookmarks over time (see below).
  • I wanted to minimize the number of times I used the posts/all method. The Pinboard API makes this easy: the posts/update returns the timestamp of the last update to any of your bookmarks. My script saves the last value returned by this method, and if the next time it runs it gets the same value, it never tries to use posts/all.
  • The thing I struggled with, by far, was string output. If you see some ‘squirrely’ things like sys.stdout.buffer.write(count.encode('utf-8')) and wonder why I don’t just print(), it’s because I ran into a lot of trouble with Python3’s string encoding and BitBar’s understanding or lack thereof of what I was giving it. It took me a long time to arrive at this solution.
  • You might also notice description = description.replace("|", "｜"). The pipe character is the one character I had to avoid in my output, as it has special meaning to Unix and BitBar. The code is replacing the classic pipe character “|” with what is officially called “FULLWIDTH VERTICAL LINE”.2 It maintains the appearance of pipes in article titles without tripping BitBar up.

Results

This was a fun project, and I think it achieved what I wanted from it. I’ve put a serious dent into the number of unread links since I started using punread.

I’m not a big fan of seeing a lot of metrics. I disable most red iOS notification bubbles. But the reason I do that is exactly why I think punread works for me: I haven’t trained myself to see and ignore a lot of numbers. I see punread’s unread count in the menu bar, and I stick to a plan of not letting it climb a lot over time.

This wouldn’t be a Take no one’s word for it post if it didn’t have a plot or two.

and a zeroed out y-axis for the fundamentalists

The rapid buildup of unread links led me to raise my threshold of what’s good or relevant enough for me to read, and I’ve been deleting any articles that failed to reach that threshold. We can see that in this plot which marks deletions with red points and corresponding labels.

I couldn’t get the text labels to work without it being a mess, so here’s a version without the labels.

and the useless zeroed y-axis version

I know the plots are not beautiful.

Each red point is a measurement that was lower than the one before it, with a total of 110 deleted articles. This way of measurement can miss some deletions if between time t and time t+1 I deleted an article and added a new article; in that instance the measurement would not register a change. I’m aware of at least one case of that happening. It doesn’t make a big difference, but it’s good to be aware of when your measurement has faults or blind spots.

I’m sure that in addition to punread helping me, I was also motivated by the idea of using software that I wrote for myself, and by wanting to see that number and line plot go down. Regardless of how the variables interact to produce the final result, I declare it a success.

  1. “a bookmarking website for introverted people in a hurry” ↩︎

  2. Unicode: U+FF5C, UTF-8: EF BD 9C ↩︎