Update (11/12/2015)

Hadley Wickham tells me that the behaviour in the scales package that I describe below was a bug. He also correctly argues that given that it was a bug a changelog wouldn’t have helped at all.

I hedged on whether it was a bug or something that was omitted from the changelog, and, as I mention below, the package did have a changelog that I hadn’t checked – which is my bad.

This post is meant as a general argument for making changelogs easier to review and access in whatever workflow the user should use to update the packages for an ecosystem (e.g., RStudio’s “Check for Package Updates…”, pip list --outdated, brew update, App Store updates pane, etc.). In hindsight, the technical problem-solving section and changelog section should have been two separate posts.

My summary remains: Debugging is sometimes hard, and changelogs are always good.

Updates that silently change things without telling you can end up wasting so much of your time. You spend hours tinkering with commands and parameters, and little did you know, you’re not even debugging the right package.

I wrote before about a collection of R scripts that crunch some personal analytics data and produce plots using ggplot2. I have set up two launchd jobs that tabulate data and update a few plots on disk at varying intervals.

The time series plot of keystrokes should – and for months, did – look like this:

Original correct plot

Then at some point over the last weeks or months, it suddenly started coming out like this:

New wrong plot

Can you spot the difference?

The points are in the same – and correct – locations. What changed is that the y-axis labels became mistakenly offset by +4 hours.

It took me hours to track this down. Here’s the solution in case you run into this. The problem is caused by a change in how the scales package works.

The original ggplot2 code for the plot is this:

library(ggplot2)
library(scales)
...
ggplot(demokeys, aes(x=xday, y=ytime)) + 
    geom_point(aes(color=machine), alpha=.7, size=.9) + 
    theme_bw(base_size=14) +
    xlab("Date") +
    ylab("Time of day") +
    scale_y_datetime(breaks=date_breaks("1 hour"),
                     labels = date_format("%H:%M"),
                     limits = range(keys$ytime),
                     expand = c(0,60)) +
    scale_x_datetime() +
    theme(legend.position = "bottom") +
    guides(colour = guide_legend(override.aes = list(size = 5))) +
    scale_color_manual(name = "machine", values=c("#00BFC4", "#F8766D")) +
    ggtitle("keystrokes by machine")

The critical part is the scale_y_datetime parameter:

...
scale_y_datetime(breaks=date_breaks("1 hour"),
                 labels = date_format("%H:%M"),
                 limits = range(keys$ytime),
                 expand = c(0,60)) +
...

At some point, the date_format function from the scales package stopped automatically detecting timezones from the POSIXct data, and defaults to UTC unless you specify the timezone yourself.

The code that returns the plot to sanity is:

...
scale_y_datetime(breaks=date_breaks("1 hour"),
                 labels = date_format("%H:%M", tz = "America/Toronto"), # <===
                 limits = range(keys$ytime),
                 expand = c(0,60)) +
...

Epilogue: Changelogs

For most of the time I spent debugging this problem, I thought the issue was either with my code that parsed the dates and times, or a bug that was introduced in a ggplot2 update.

I found two other threads that discussed the same strange offset behaviour without a good solution. In the second thread the behaviour is reported as an issue under the ggplot2 repo, and Hadley, the package author, seems to accept it as a ggplot2 bug.

One way this problem would have been less likely to occur is if the R package update workflow made it easier to read changelogs. Not everyone would need or want to read them, but I would.

The accessibility of changelogs varies a lot depending on the ecosystem you’re using.

On iOS, for example, you can tap on the app update in the Updates tab of the App Store app and read “What’s New”.1

The Sparkle framework that many OS X apps use is really good. It tells you what version number is now available, what version number you’re on, and, if the developers chose to provide the information (which they almost always do), the changes introduced in the last version or two. Unlike with iOS, there is a stronger tradition of developers listing new additions, changes, and bug fixes in OS X app updates.

It gets a bit worse with command line package managers.

I use Vundle to install vim bundles and keep them up to date. Of all the command-line tools I use, Vundle is the best in helping you see what’s changed. Since GitHub repos often have small commits and changes instead of “releases”, pressing u after Vundle checks your packages for updates lets you see the commits that were pulled from the repos for every package. Not the best way to list changes, but it’ll do.

With Homebrew, there is no easy way see changelogs before or after upgrading installed packages. You would have to do a lot of independent research to figure out what changed in each package.

The situation is just as sad with pip the python package manager. With both its old update workflow using pip-review and its replacement pip list --outdated, there is no way to view package changelogs within pip. As with Homebrew, independent research on your end is needed.

pip is another example of a recent silent update that broke things in a way that mystified me for a long time. You used to be able to use the pip-review --interactive command to check for updates to the packages pip had installed, and choose whether to update them or not on a one to one basis. I did this frequently, for months.

Then one day, trying pip-review just gave me a command not found. For a while I thought something had gone wrong with my python installation.2 After I had tried uninstalling and reinstalling everything I could think of, an online search led me to issue #185, aptly titled “pip-review?”. Turns out™, pip-tools 1.0 had been released and silently killed pip-review. No warnings, no deprecation notices on previous versions, nothing. Because screw you.3

end sidebar

And finally we come full circle. What about R CRAN packages?

The defacto IDE for R is RStudio, and RStudio already has a nice way for you to check the changelogs for any packages that get updated.

RStudio package update News

The “NEWS” column is a link to each package’s “NEWS” directory on CRAN. The directory should have an html file with the changelog of the package since the last version released.

In my experience, 50% or less of package developers populate the NEWS page with any content. The NEWS links for 3 of the 7 packages listed in the screenshot lead to a 404 Not Found. And to be honest I am shocked at this high hit rate. Maybe things have taken an extreme turn for the better, but until recently, those NEWS links were hopeless.

Did the changelog for the scales package detail how the date_format function was no longer going to detect the timezone from the data and default to UTC unless told otherwise? Yes. Yes it did. … ish.

Version 0.2.5
------------------------------------------------------------------------

## Improved formatting functions

* `date_format()` gains an option to specify time zone (#51).

The note reads “gains an option to specify time zone” but what it should read is “requires you to specify time zone”.

Still, there were changelogs, and I didn’t check them because I had gotten used to the fact that no one ever populated them. Not that it would have prevented this whole issue. Either this is a scales bug, or the notes really don’t make it clear that specifying the correct timezone is now a requirement. Either way, I got a good blog post out of this situation, and that’s always a good thing.

Summary:

Debugging is sometimes hard, and changelogs are always good.

  1. Which is great despite the fact that it is annoyingly more and more common to find just one line that reads “Bug fixes and improvements.” It is really good programming habits to keep track of what bugs you’ve fixed and tell your users. It won’t hurt those who don’t care, they won’t tap or read it anyway. Those who do tap, are looking for actual information. ↩︎

  2. Because come on, there is nothing easier than messing up python environment and installations on OS X. You know it. I know it. ↩︎

  3. Granted, it’s not like pip-tools updated itself. I must have updated pip-tools at some point, but there was enough time between that update and my first failed attempt at using pip-review that I didn’t connect the two. Still, you kill a command and introduce a different one that doesn’t even really do the same thing, you really should try a bit harder to warn people. ↩︎