Sabbatical

Sunset flight - sabbatical

I started working at $FAANG_COMPANY (henceforth referred to as $COMPANY) on Monday January 16, 2017, and stopped working at $COMPANY on Friday June 24, 2022. I was paid (well) to further the business interests of $COMPANY for one thousand, nine hundred, and eighty-five days, excluding most weekends. Thus was my first full time job.


I would tell you what my job title was, but it doesn’t really matter, and anyway I worry it might mislead you about what I actually did because job titles are weird. I ran and analyzed data from experiments, did some data engineering, trained a regression model or two, trained a deep learning model or two, and built a lot of automated inference and decision-making workflows that combined model outputs and business rules. Yes I did “machine learning” “in production”. I was halfway between a scientist and a developer and I liked it.


It’s important to acknowledge that there’s a lot of privilege in being able to quit a job not because you’re going to a different one, but because you’re just not happy doing it anymore.

  • Financial. I have enough money saved to support myself through a period of unemployment without getting anxious about meeting my basic needs. Multiple unlikely catastrophes would have to happen for me to be in a bad place.
  • Legal. Getting my work visa to move to the US and start my job at $COMPANY was not a smooth process, but it got done. Getting my permanent residency was far more turbulent, but despite the odds it also got done. I’m lucky to have a more stable life in this country without having to carry the work visa holder’s constant anxiety about losing or leaving a job. For those unfamiliar with that anxiety, it sucks and it warps the way you think.1
  • Social/mental. Leaving a job is a scary thing to do. I’ve had the support of some who reassured me that I wasn’t crazy, others who reassured me that I would be able to find gainful employment again, and even others who quit jobs before me and did not immediately die or combust into flames.

You do a job for a mix of intrinsic and extrinsic motivations.

Intrinsic motivations include:

  • You enjoy the ideas you’re working on, the specific problems you’re trying to solve, the product you’re trying to make better.
  • You enjoy the technical aspects of the work.
  • You value the impact of the work.

Extrinsic motivations include:

  • Compensation (salary + benefits + retirement savings + other).
  • Job stability.
  • You don’t want to leave a great manager.
  • You don’t want to leave a great team.

Different people have different intrinsic/extrinsic portfolios. Some have an intrinsic heavy ~90%/~10% mix (I’ll stereotype here, but: artists? people who work in used bookstores?), some have a ~50%/~50% mix (tough to find stereotypical examples for this), and some have an extrinsic heavy ~10%/~90% mix (people in finance, a lot of people in tech, and me at $COMPANY towards the end).

That’s the main reason I had to leave; I was running out of intrinsic motivation. For a while I was able to lean on the enjoyment of the technical aspects of the work, but they were tangential at best to my job description. I was stealing fulfillment from a role that wasn’t designed to give it to me, and my team was starting to move in the opposite direction. Not good.

I already lacked a good answer to the question “did my work delight anyone or make anyone’s life significantly better?”2 Losing the enjoyment of the craft itself made my decision easier. Going forward, I cannot not have a good answer to that question no matter what other conditions are met.


So I am taking a sabbatical. I don’t know how long it will last, and I don’t have a solid plan for it yet.

I want to travel (check), write more (check), read more (check), spend more time with people whose company I love (check, and in progress), return to and start personal projects so I can remember what it’s like to code for fun (not yet started), and explore the widest possible range of ideas for what I’m going to do for money next (not yet started).

Can I find a role defined around the technical work I enjoy? Can I walk through the valley of the shadow of death (interviewing in tech) again? Will I even go back to tech? Will I move? Will I work remotely?

All open questions.

  1. For an excellent writeup on what the US work visa and immigration process looks like, do read 18 Years A Transient – My Journey Through the American Immigration System as a Computer Engineer – Software the Hard way↩︎

  2. Thankfully I’m pretty sure my work didn’t make anyone’s life worse. A fate not trivial to avoid in tech. ↩︎

Personal Retreat

Yes it's not a good photo of a desk so I made it BW

In May of 2022 I went on a three day personal retreat to a small cabin in Whidbey, WA. I had travelled just to be alone before, but this was the first time I went with a plan for what to do. It went better than I expected and I wanted to write some personal notes about it here.


I don’t know what first comes to your mind when I say “personal retreat”. It’s not the spiritual meditative kind (although it can feel a little bit like that at times), and it isn’t the kind where you go to a remote location in the California desert and join others for a week of silence and contemplation.

Try not to over-hype it. You’re just taking some time to think about where you are, where you want to be, and the actionable changes you can make to get there.

The plan

David Sparks published a guide for how he does his personal retreats, and it seemed pretty good to me so that’s what I did too.1

You can watch/read the guide for yourself, but to give you some context I’ll say that the work revolves around Roles. You can imagine that everything we do, we do while inhabiting a particular role. You’re a friend, a writer, a painter, a cyclist, a lover, a spouse. I know it’s not that straightforward and roles are not that separable, but this works well enough to be useful.

Now that you know the first principle of Roles, the rest of the steps are:

  1. Write down all the roles you inhabit.
  2. Write your ideal description of how you want to be in each role.
  3. For each role, you answer a list of questions that you’ve prepared ahead of time (this is the most time-consuming part of the whole exercise).
  4. After answering all questions for all roles, collect a list of actions that you want to take to move yourself to where you want to be.
  5. Don’t do the actions themselves, but build whatever systems you need to make it easier for you to do those actions when you’re back in the real world.

There’s not that much more to it, but you should watch David’s video for more details. What I want to do with the rest of the post is document some practical meta notes as they apply to me.

Notes

Going somewhere else

This one is short. Don’t do this at home. Don’t do this where you will be surrounded by other people (strangers or not), and even if you’ll be alone don’t do it in a familiar place where you will be surrounded by all your distracting shit.

Pack a light bag, go somewhere else that’s simple, cheap, sparse, and quiet.

The vibe of the space matters

I wish I could say that any place would work as long as you were alone. Maybe that’s true for some people, in which case that’s great because it gives you more flexibility. For others, including me, aesthetics influence mood and ability to focus and introspect. A lot. Know whether that’s true for you as well, and account for it.

I was was lucky to find the place I went to. It was small, cheap (as cheap as a cabin on Whidbey island could be), cozy, and not modern in any way. No linoleum, no chrome, carpets everywhere, wood everywhere, old comfortable furniture everywhere. It wasn’t fake mid-century modern, it was actually old. It was perfect.

Pen and paper vs. typing.

I’m parroting David here, but having periods of focused, uninterrupted writing is critical. David manages to protect his attention while still using a computing device to write (I think he uses an iPad), and I can imagine how this might work: turn off the radios, turn on Do Not Disturb, use Focus Modes to suppress things even further, etc.

My digital hygiene is much worse than David’s. I think just looking at a screen and touching a keyboard would crack my attention even if no information is popping up, so I opted for the pen and empty notebook approach.

There are downsides to pen and paper. Writing by hand is a lot slower than typing.2 Also if your handwriting is terrible, you might not be able to read it later, or it might just put you off during your review.

A year ago I would’ve added “you can’t search” to the list of cons to pen and paper, but iOS now has Live Text and it’s amazing, even with handwriting. I wouldn’t say that it’s a solved problem, text flow and formatting might still be garbage, but at least it’s searchable.

This is also where the specifics of the place helped me a lot. There was a corner with an old writing desk and creaky wood chair that helped me keep my work environment sparse and tidy.

Silence

Boy does it matter. I wish it didn’t, because I usually like to listen to music while I do things, but it does.

It’s very à la mode to say this these days, but it’s amazing how clearer your mind can be if you spend an hour or two writing or thinking in uninterrupted silence.

Not a lot stuck after the first try

I left the cabin full of insight and plans for things to change to get closer to the life I wanted to live. I don’t think any of the insights were wrong, but I made some classic mistakes, the classic-est of which was to have plenty of plans for things I was going to do without sufficiently planning for things I was going to stop doing. It’s a mistake as old as time.

So, in the final analysis, I intended to make ꭓ amount of change, and I’ve actually accomplished about ⅛ ꭓ. But hey, that’s good! I learned some lessons and I’ll do a better job next time.

Next time

Have a next time. This is good to do once, but the real gains come from doing it often.

  1. He also has a PDF version but he unfortunately put it behind a newsletter signup form. You can decide to sign up to get it, but it’s essentially the same content from the video in readable form. ↩︎

  2. That said, slow can be good here. ↩︎

Pilot

Turning base for runway 16 at KRNT

I passed my private pilot checkride on Tuesday, September 14th, 2021. I think getting my pilot’s license was the hardest thing I’ve ever done.1 Flying didn’t come naturally to me, and being able to say that publicly and be okay with it is perhaps the second hardest thing I’ve ever done.

So how do you learn to fly a plane in the United States? There are many ways, and many aviation websites will tell you all about them. I’m writing this here for those who won’t go looking it up otherwise.2

Discovery

Unless there’s an aviator in your family who introduced you to flying in your childhood, it starts with a discovery flight: a 2-ish hour flight offered by flight schools that previews what it’s like to fly a general aviation aircraft—think Cessnas, Pipers, Cirruses (Cirri?). You meet with a flight instructor who shows you how to pre-flight a plane, pre and post engine start checklists, radio communications in towered or non-towered airports, taxiing, more checklists, takeoff, cruise, and hopefully at least one landing.

On a discovery flight you fly left seat (which is the normal pilot in command seat), the instructor will hand you the controls to do some flying during cruise, and might even let you take off yourself if the vibe is right. Some people get into it right away, others are so overwhelmed by the earth pulling away from them in what is essentially a tuna can with a propeller that it takes them a while to get comfortable. You can guess which one I was.

Say you loved your disco flight—you caught the flying bug as they call it—and you want to take it all the way to your license, what next? You find an instructor, have an orientation, and schedule your first lesson.3 Congratulations, you’re about to spend a lot of money very fast.

Training is split into stages which might vary from school to school but generally look like this:

Stage 1: Pre solo

Stage one is about learning basic flight maneuvers: takeoffs, climbs, level flight, turns, descents, and landings. It’s also about learning beyond-basic maneuvers that teach you about the plane’s performance envelope: steep turns, slow flight, stalls, ground reference maneuvers, and emergency procedures. You will be asked to demonstrate these on the practical test.

This stage is complete when your instructor endorses you to do your first solo flight. Your first solo is a massive milestone, equal if not exceeding in significance to your practical test.

Stage 2: Cross country

Wind farm near Chehalis, WA

In this stage you build on basic flying skills to learn the fundamentals of navigation. For the purposes of a private pilot license, a cross country (XC) flight is one in which the “point of landing… [is] at least a straight-line distance of more than 50 nautical miles from the original point of departure”.5

During this stage, you practice planning XC flights by examining sectional charts, knowing what airspace you have to avoid or fly through, choosing altitudes to avoid terrain, and using weather and winds aloft forecasts to calculate your magnetic heading from true course, wind correction angle, magnetic variation, and magnetic deviation (if applicable).

This stage is fun because you’re actually going somewhere instead of doing circles in the practice area. But it’s frustrating because your instructor will probably have you do everything on paper to make sure you actually know what you’re doing. Planning a XC flight on paper is an excruciating and error-prone task. An instructor once asked me “How many times do you think I made a cross country flight plan on paper after I got my license?” and then made a big zero with their index and thumb.

The thing is there’s an app for that, it’s really good and it’ll do almost everything for you. That said, the examiner will want to know where all the numbers come from, and so the instructor wants you to know where all the numbers come from. It will also help you not get yourself killed when you inevitably press the wrong button in the app and not pick up on all the suspicious estimates.

This stage is complete when you perform a XC flight of 150 nautical miles, with full stop landings at three points, and when at least one segment of the flight is more than 50 nautical miles along a straight line between the takeoff and landing sites.

I did two cross country flights: KRNT → KCLS, and KRNT → KCLS → KKLS → KSPB. The solo cross country flights were the most fun I had during training.

Stage 3: Checkride prep

This is a catch-all stage. The goal is to complete any remaining requirements and bring up your maneuvers to the tolerances of the Airman Certification Standards (ACS).6 The ACS is very specific about the deviations allowed during maneuvers. For example:

Slow flight

PA.VII.A.S5: Maintain the specified altitude, ±100 feet; specified heading, ±10°; airspeed, +10/-0 knots; and specified angle of bank, ±10°.

Steep turns

PA.V.A.S5: Maintain the entry altitude ±100 feet, airspeed ±10 knots, bank ±5°, and roll out on the entry heading ±10°.

Power-off stalls

PA.VII.B.S2: Select an entry altitude that will allow the Task to be completed no lower than 1,500 feet AGL (ASEL, ASES) or 3,000 feet AGL (AMEL, AMES).
PA.VII.B.S3: Configure the airplane in the approach or landing configuration, as specified by the evaluator, and maintain coordinated flight throughout the maneuver.
[…]
PA.VII.B.S6: Maintain a specified heading ±10° if in straight flight; maintain a specified angle of bank not to exceed 20°, ±10° if in turning flight, while inducing the stall.

And so on.

Some examiners will look the other way if you deviate by 12° during your power on stalls. They might not notice, or they might get you to try the maneuver again. Most examiners aren’t trying to fail you; they want to give you your ticket. They just have to make sure you’re going to be safe, which is why some mistakes are automatic checkride busts and they’re usually mistakes relating to safety and situational awareness.

Closing thoughts

Orcas Island, WA

I’m a pilot now and I am really happy about it.

  1. Palpably aware of my privilege, thank you. ↩︎

  2. Above all else, I’m writing this here for me. ↩︎

  3. Finding an instructor is one of the more difficult things to do when starting flight training. Not unlike your relationship with your therapist, fit is critical when it comes to schedules, personality, communications, etc. ↩︎

  4. Serious talk, it’s not just you who will get slapped if you do stupid things during your training, your instructor will get slapped too. And since your instructor is very likely trying to build hours to get to the airlines, having a black mark in their file could end their career. ↩︎

  5. 14 CFR § 61.1↩︎

  6. I know. ↩︎

Libby and your public library

Libby is an app that lets you borrow ebooks and audiobooks from your public library. It’s free to download and use, you can install it on iOS and Android, and while I’ve only used it in the US, it works with some international public libraries as well.

Libby is the best app I have ever used. Not because of its design (which I do like), or because of the number of things it does (it does a few things well), but because it solves a hard problem so well.

First I want to talk about the basics of borrowing ebooks, because it used to be complicated but it’s not anymore. Second, I want to convince you to use Libby even if you can afford to buy ebooks.

If you don’t stick around till the end, my message is this: Use Libby to read for free, donate your buying money to your local library.


Quick start

You will need a library card. You might be able to get one remotely, but I had to show up in person to show evidence of residency before getting mine. Once you have your card, open Libby and connect it to your library account.1

That’s it. You’re ready to use Libby to:

  1. Put holds on books (more on this below).
  2. Borrow a book immediately if there are copies available.
  3. Borrow a book you had a hold on if it’s your turn (more on this below), or
  4. Deliver a book later (my favorite Libby feature, by far).
  5. Send your ebook loan to your Kindle if you have one (more on this below).

Holds and loans

A library hold is your place in line to borrow an item. For virtual items, a hold is only required if all copies are currently borrowed by readers, and you want to get in line for one. If a copy is available, you can borrow it immediately, because there’s no reason to hold the bytes, they can be sent to your device immediately.

A note about copies: Libraries get a limited number of “copies” of any virtual item. That’s ridiculous of course, it’s just a file so it’s not the copies that are limited, it’s licenses. The page I linked to explains it in a more diplomatic tone, but in summary the old publishing houses continue to be dragged kicking and screaming into the modern world, and they haven’t figured out a way to make their business work while letting library borrowers copy a file as many times as they need so they could all read a book at the same time without also bankrupting the library. Maybe one day we’ll figure this out. Until then, your library can send the bytes that make up Hitch-22 to a limited number of people, and they can’t send them to you until they make sure that one of those peoples’ devices is no longer authorized to show those bytes anymore.

Libby lets you put holds on up to 25 items at the same time.2 Once it’s your turn and a copy of a hold is available to you, Libby will notify you, and you can make one of two decisions: Borrow, or Deliver Later.

Deliver Later is my favorite Libby feature, it’s the reason to use the app. If a copy becomes available but you’re not ready to start reading (e.g., you don’t think you’ll have time to finish the book before it’s due), Libby lets you defer the loan. Instead of going to the back of the line (which could be months long in many cases), you can choose how long before you’re offered the book again. The link describes it best: “Delivering a hold later keeps you at the front of the wait list, but passes the current copy to the next person in line.” One minor downside is you can’t be exact with when it’s your turn again–it depends on the range of time it takes people to read their copies.

With Deliver Later, you don’t have to time your holds to avoid getting a bunch of books all at once. You can keep deferring until you have the time to read something new.


Reading and listening

Briefly. If what you’re borrowing is an ebook, you can read it in the Libby app itself. That said, most ebooks available on the app are compatible with Kindles, and Libby makes sending the loan to your Kindle very easy.3 I love this, first because if you own a Kindle you probably prefer to read on it instead of your phone/tablet’s screen anyway. Second, because the Kindle will save all your highlights from the borrowed book just like it does with ebooks you own–even after you return your loan. Is your mind blown? Mine still is, and I’ve been using Libby for years.

If what you’re borrowing is an audiobook, the Libby app is the only option for listening, as far as I can tell.


Complications and closing thoughts

It’s not all roses and candy canes. ebooks are very expensive for libraries.

E-books at libraries are a huge hit, leading to long waits, reader hacks and worried publishers

And while there are technically an infinite number of copies of digital files, e-books also work differently. When a library wants to buy a physical book, it pays the list price of about $12 to $14, or less if buying in bulk, plus for services like maintenance. An e-book, however, tends to be far more expensive because it’s licensed from a publisher instead of purchased outright, and the higher price typically only covers a set number of years or reads.
[…]
A library typically pays between $40 and $60 to license a new e-book adult title, which it can then loan out to one patron at a time, mimicking how physical loans work. Each publisher offers different payment models. Under one, a library only has an e-book for two years or 52 checkouts, whichever comes first. Another agreement covers 26 checkouts per book.

I know. It’s backwards. An ebook has no marginal cost. It costs nothing to copy, nothing to host, nothing to send to a buyer or borrower, yet it’s priced like it’s only slightly cheaper if you’re buying, and like it’s more costly if you’re borrowing. You’ve probably already had the what-planet-is-this experience of seeing a book’s paperback priced at $16 and the ebook version priced at $12.99. It makes no sense except as a reminder that things aren’t priced on sense, they’re priced on what the market will bear.4

This perversion might prompt you to stick with buying ebooks instead of borrowing them out of some desire to alleviate the burden on the libraries. Maybe you’re lucky and you can afford it, and hey, you can get the book when you want and you keep it forever.

Let me ask you, instead, to not do that. Keep borrowing, and if you can afford it, set up a recurring donation to your local library. You and I can go back to buying most of the books we read, but others actually rely on the library for their reading, and this won’t get better if we all back off because it’s creating a problem. This kind of problem can only be resolved if demand for this service stays high and goes higher.

  1. You might need to provide your library credentials, I’m not sure because it’s been a while since I set mine up and I haven’t had to do it a second time in years. ↩︎

  2. It’s possible the number varies by public library. I use the Seattle Public Library and 25 is the limit of simultaneous holds available to me. ↩︎

  3. Caveat: this process happens through Amazon, which has a deal with libraries to deliver their Kindle versions of ebooks as loans to Kindles. You’ll have to sign in to Amazon for this to work. The book will disappear from your Kindle automatically when the loan is due. ↩︎

  4. A reader of a draft of this post was skeptical that the cost to product and ship paper books was that large compared to ebooks, and thought the difference in price was reasonable. Ultimately this is a question about profit margins, and my assumption was that the profit margins on ebooks would be much higher than physical books. The best source I could find for this data is a post by Hugh Howey in which he references New Republic and Harper Collins showing what I assumed was true: contribution margins were 41.4% on hardcovers, and 75% on ebooks. I looked for the original New Republic article but couldn’t find it, neither could I find more recent or authoritative data. ↩︎

Spritz: historical rain notifications

‘Spritz that Shit’ is an inside joke. You can change the title if it doesn’t suit you.

A certain cherished member of my household has accumulated a lot of indoor and outdoor plants – 29 indoor ones, dozens of outdoor ones at last count. Plants don’t take care of themselves, and unlike most other living things you might keep near you, will only ask for what they need in frustrating and ambiguous ways, and usually when it’s a bit too late. On second thought, many adults work that way too.

Watering the indoor plants is managed using a spreadsheet that changes colors in disconcerting and heart palpitating ways, but it still tells you what to do and when to do it. Watering the outdoor plants is trickier; you want to account for rain to avoid wasting water and overwatering the plants that, as I mentioned, won’t tell you that you’ve been doing that for months until they’re dying. I definitely don’t have recent personal experience with this.

One day we were talking and the Household Plants Czar was thinking aloud about how annoying it was to keep track of recent rain and know whether to water outdoor plants or not, and how cool it would be if their phone just told them how much it’s been raining recently so they can make a decision.

I thought it would be fun to try to make it happen. I did, and now I want to write about how I did it. The summary is: getting weather data and sending notifications were the easy parts. Dates and timezones, as usual, were the harder parts.

OpenWeatherMap

Nothing will happen without a source of weather data. I searched for an API I could use without having to pay (at least at this early stage) and OpenWeatherMap was at the top of the list. It might’ve even been the first service I checked out, and it looked like it would do exactly what I wanted, so I didn’t keep looking.

The free tier is limited to a small set of APIs, but it ends up being enough. The historical weather data API gives hourly weather conditions for any provided latitude and longitude up to five days in the past. Perfect.

It takes just a few lines of code to get some data:

import requests
API_KEY = '...'
LOCATION = (47.588993, -122.306286)
TIMESTAMP = 1621794493
def get_weather():
    hist = "http://api.openweathermap.org/data/2.5/onecall/timemachine" \
           f"?lat={LOCATION[0]}&lon={LOCATION[1]}&dt={TIMESTAMP}&appid={API_KEY}"
    return requests.get(hist).json()

Notifications

I could resort to automated emails if I had to, but the nicer way is to get a proper iOS notification waiting for you in the morning, telling you exactly how much it’s been raining recently.

I ended up using Pushover, a popular notification service. I primarily chose Pushover because python-pushover exists. The Python client may or may not be actively maintained, but hey, it works now.1 Being able to attach an image to the notification is a nice bonus.

from pushover import Client
pushover_client = Client(PUSHOVER_USER_KEY, PUSHOVER_APP_TOKEN)
client.send_message(MESSAGE, TITLE)

With Pushover you get a 30 day free trial per device, and then you’ll need to hand over $5 (as a one-time payment) for every device you want to send notifications to. Since the payment is per device, not application, all other automations you might set up to the same device are already paid for. Very much worth $5.

Dates, hours, and UTC

There’s a somewhat famous quote that goes like this: “There are only two hard things in computer science: cache invalidation and naming things”. Maybe. I’m no computer scientist, I’m a simple programmer, and in my lower orbit, the hardest problem I always run into is timezones.

I live in the Pacific timezone, but OpenWeatherMap does everything in UTC. So far so normal, most computer things use UTC, and if the API was hourly, this would be no problem. If I wanted the total precipitation for 1 AM, May 23rd, I’d find the UTC equivalent of 2021-05-23 01:00 PDT (which would be 2021-05-23 08:00), hand that to the API, get the response, and move on to the next hour.

The API, however, is daily. I assumed that meant that if I gave it a specific hour, it would give me hourly historical weather for that hour and the next 23. But that’s not how it works, the API’s day is the UTC day.

This means that if I hand the API 2021-05-23 08:00, OpenWeatherMap will give me the hourly weather for 2021-05-23 00:00 to 2021-05-23 23:00, which is 5pm May 22nd to 5pm May 23 Pacific.

This means that for every full Pacific day of weather you want, you’ll need to make two API calls, one for the day, and one for the day after.2

I decided to give up on thinking about my days vs. UTC days, and just accumulate hourly data and then filter them into days later.

The full script

Find it on GitHub.

This is one of those times where I post code I’m almost certain is written in roundabout and inefficient ways. If you get this far, and it’s clear to you how this could’ve been a lot better, please let me know! Also, pull requests are welcome.

  1. The last commit in the library is three years ago. When I first saw that I though no way this is going to work. But it did, which I guess is a testament to the stability of Pushover’s API. ↩︎

  2. I had already written the script assuming the API gave me my requested hour and the 23 hours after that. I think having written that code already made the change even more mind-bending, because I wasn’t thinking of how to write code from scratch, I was thinking of how to repurpose what I had already written to work with the new reality. ↩︎

Identity, change, and promises made to no one

What is this website about? Is it just about scripts and programming projects and my favorite vim plugins? What if I want to write about other things? What if the one thing this used to be about is not something I want to write about anymore? What if an idea I want to write about doesn’t fit with what I’ve been writing about recently?1

In my About page I say that I write about a mix of technical and non-technical topics. And yet, I reject writing ideas on a daily basis because they wouldn’t “fit” what this website’s been about. Is this good?


I think a similar dynamic happens in our personal lives. At some point we become “about” something. You’re a coder, a beauty vlogger, a photographer, a philosopher, a cyclist. You got interested in something and you went deep, did it consistently, got better, and enjoyed cultivating and projecting your interest in it. As your interest and engagement grew, it’s not just you that knows that you’re into this thing, people around you know it too, and they start to ask about your progress and enjoyment.

Over months and maybe years that thing becomes a consistent line drawn along other big and small things that change: where you live, what you do, the relationships you’ve been in. It’s part of your identity.

So it’s understandable that you would react negatively when you start to drift away from that thing. It happens slowly and you fight it. You start being interested in other things and you don’t spend as much time on the “thing” anymore. It’s not just you though, those around you also react with concern. “How come you’re not going out to ride your bike as much?”, “are you still talking to your philosophy friends?’, “how come you’re not posting your makeup videos as often as you used to?” Questions that presume that the current status is sad, forcing you to come up with explanations rather than reasons.

This is not straightforward to navigate because it runs into principles like “consistency”, “commitment”, and “stick-to-itiveness”, which we generally think are good.2 What does it say about your character that you lost interest in this thing after you made a big deal about it and others came to know you for it? Is it weakness on your part? Will you keep switching it up, becoming someone who’s difficult to nail down in your life and the lives of those around you? Who are you if not the photographer or the blogger or the beauty vlogger?


Years ago I wrote about changing our minds:

Overcoming your own internal barriers to changing your beliefs and principles is difficult enough without people from the outside giving you a hard time about it. Instead of this horrible tradition described above, people should be curious when others they know change their minds about things, ask them why, and criticize the reasons given or learn from them

Just like accusations of hypocrisy and flip-flopping make it harder than it already is for us to change our minds, this feeling that we promised something with our actions and history makes us feel guilty about change. The weird thing is, in a different partition of our minds, we believe change is good. In debates about whether people can really change, no matter which side you’re on when it comes to the possibility of people changing, you agree that it would be better if they could.

Yet when it’s time to change, even if you were to overcome the difficulty of it, now there’s this guilt-induced drag pulling you back, making you worry about breaking the image you have or the expectations that you and others have of you.


Let me walk this back just a little.

There is such a thing as not having stick-to-itiveness and dropping things at the first sign of difficulty.

What this post is about is the unreasonable feeling of not being able to change your primary interests out of a sense of obligation or shame. If you’re all in on photography this month, and all in on woodworking the next, and all in on painting the month after, maybe you do have a problem to deal with.

How do you know whether you’re trying to stick with something out of persistence and grit as opposed to obligation and shame? Only you can know; if you’re looking for easy answers, you’re reading the wrong blog.3 I’m only writing to tell you and myself that interests and identities can change, and promises about them should be made to no one.

  1. Here’s a meta thought: this post about what this site is about goes against my idea of what this site is about. It’s navel-gazey in a way that violates my writing principles. ↩︎

  2. Relax, I’m not here to be the “forget everything you know about everything we thought was good!” guy. Those are generally good. ↩︎

  3. And there isn’t a right blog for easy answers. ↩︎

Regularization

A simple place

I am predisposed to overfitting. I think most people who like to work with computers, who like to hack, or make, or do things themselves, have a predisposition to overfitting.

Overfitting is a term of art from statistics and machine learning. It’s a fancy word that refers to the idea that when you don’t constrain how complicated or flexible your solution to a problem can get, you will come up with a solution that perfectly fits all your requirements as they exist today, but this perfect fit comes with a tradeoff in which your solution is specific, complicated, and doesn’t generalize to unseen future requirements very well. You didn’t fit, you overfit.

For example, you try to write an essay and the only editor you have is Microsoft Word. You start typing but it just doesn’t feel right. You search for better tools for writing essays and find someone recommending Bear, so you download it, and you write your essay and it works out great. Next week you need to write a long personal email, you start in Bear but it doesn’t feel good for an email. You look around and find people recommending Ommwriter, so you download it and write your email and it works out great.

This is fine if from that point onwards you will only need to write essays and personal emails, and if you will need to write them the same way you wrote them before. But nothing ever stays the same and the future always differs from the past. If you will accept nothing less than a perfect writing experience for each new writing task, you will never stop looking for new tools.

Then you look at your writing toolbox and it’s a mess. Instead of having a couple of tools that work okay for most tasks, you have a hundred tools and they’re all tuned to work for very specific tasks.


Regularization is another term of art, and it refers to intentionally constraining your solution to keep it uncomplicated and general. In the writing tools example, you can regularize by setting a rule that says “I will not have more than four writing apps installed at any given time”, or “each writing app I install has to work very well for at least three different writing tasks”.

These constraints will naturally stop you from installing your 10th writing app, and will encourage you to use tools you already have instead of getting new ones. The cost of the constraints is that you’ll always feel like the tool isn’t a perfect fit for what you’re doing right now.

Regularization changes your behavior when the tool acquisition or solution modification was going to have a higher cost than gain, which usually happens after your solution has already grown quite a bit. A critical point in this framework is that early modifications lead to large gains, and later modifications have diminishing returns. When you only had Microsoft Word and installed Bear, you gained a lot! They do very different things and the universe of tasks you can tackle increased massively. But by the 10th app, your gain, if it exists at all, is tiny.

Regularization can be tuned! Your rules can be restrictive, such as implementing a one in-one out system, e.g., “I won’t use a new tool without getting rid of an old one”. Or, they can be more relaxed: “I won’t use a new tool without trying to do without it for three days, and if I still have an unsolved problem, it’s time to complicate my solution a little bit more”.


Overfitting is a very common problem, both in machine learning and in life. The writing tools example comes from personal experience. I use Neovim as my main editor, which sounds like a nice regularization, until you see that my configuration file for it is 520 lines. And I said it was my primary editor, not my only one.

Another personal example is how frequently I tinker and modify this site’s style and structure. In the past I rarely wrote a post without modifying some feature to look or work better. Every small nit I noticed, I addressed immediately. This took time, and meant things were never consistent. The time I spent tinkering could have been spent getting better at writing.

To regularize is to impose constraints that keep your life simple at the cost of a small amount of constant discomfort. It also helps you focus on what really matters. Does it really matter that the site title doesn’t align perfectly with the post under it? And does it matter more than sweating out an idea I have in my head or a draft that I’ve been struggling with?

The people whose writing and work I find the most valuable have had the same site, the same gear, and the same tools, forever. Maybe regularization comes naturally to them, and they have an ingrained understanding that spending time producing is better than spending time improving tools or tinkering with visuals. Or maybe they had to be intentional about it and fight their own predisposition to overfit. Either way, regularization is one of my yearly themes and it’s already working very well.

See also

Remembering what you read

Coffee plant leaves

$ clipping

The Conquest of Happiness (Bertrand Russell;Daniel C. Dennett)
- Your Highlight on page 6 | Location 79-80 | Added on Wednesday, December 13, 2017 9:26:08 AM

Men who are unhappy, like men who sleep badly, are always proud of the fact.

Reading

After a year and a half of reduced book-reading, I’ve managed to rebuild a reading habit (which is great). Thank you Kindle, thank you Seattle Public Library, thank you Libby.1

With this return of regular book-reading returned a thought that always bothered me: I like reading but I don’t remember the books I read very well.

Most of the books I read are nonfiction. While reading them, it feels like my mind is engaged, and I am keeping up with the narrative, the details, the arguments – it seems impossible that I could forget any of this. But most of the time, in less a week after finishing a book, I can only remember one or two pieces of detail and the rest is gist. Not much more.

Paper vs ebook

A lot of people who read books agonize over this. You’ve probably read the same dilemma recounted by a hundred other readers.

Paper books are romantic objects. They have cultural energy, and they become objects that mean a lot more than just the words you can read within them. Second-hand book shops are magical and intimate. They always feel like sanctuaries staged in a better time. You look at a shelf, or a stack that’s too precarious, pull a book out from the middle, and if you walk out with it, that’s your special book. It’s your special piece from that magical place.

Also. A paper book has a visible cover, and a visible cover lets you show other people on the train what you’re reading. The people want their social signaling. We’re starved for a social connection, and if someone notices the cover of the book I’m reading, and we make a connection because of it, that’s a nice thing.2

The relative benefits of ebooks are apparent: you can carry many books on a device smaller and lighter than a single book, and you get search, highlighting, and typed notes.

Kindle highlights

My reading revival happened on Kindle. I love paper books (see above for an explanation that reads a little bit like a eulogy), but the Kindle is lighter, easier to hold in one hand, lets me borrow books from the library without having to go to the library, and most importantly for this post, saves my highlights locally even after the book is returned to the library.

Kindle highlights are saved to a My Clippings.txt file on the device itself.3 The file’s formatting has remained consistent for years. Here is a sample:

==========
In This Arab Time: The Pursuit of Deliverance (Fouad Ajami)
- Highlight on Page 33 | Loc. 551-52  | Added on Monday, December 15, 2014, 06:00 AM

Tahrir Square had transfixed us all, but as the immensely talented young Egyptian intellectual Samuel Tadros puts it, Tahrir Square was not Cairo and Cairo was not Egypt. 
==========
In This Arab Time: The Pursuit of Deliverance (Fouad Ajami)
- Highlight on Page 36 | Loc. 581-83  | Added on Monday, December 15, 2014, 06:04 AM

The group may have railed against America and the shadow it cast over Egypt, but leading technocrats from the Brotherhood, Morsi among them, rose to professional success and prominence through American degrees, and their years in America took them beyond the cloistered world from which they hailed. 
==========
In This Arab Time: The Pursuit of Deliverance (Fouad Ajami)
- Highlight on Page 37 | Loc. 602-5  | Added on Monday, December 15, 2014, 06:06 AM

The Brotherhood had not reinvented this weary land. Egyptian governments had long perfected the art of playing cat and mouse with the IMF and with foreign donors, mixing dependence and defiance, at once needy but proud and brittle. Like riverboat gamblers, Egypt’s rulers seem to relish the game, secure in the knowledge that a country of 80 million people at the crossroads of so vital a region, so near to the oil fields of the Arabian Peninsula and the Gulf, a Sunni balance to Iran, will always be bailed out, that it is too big to fail. 
==========
In This Arab Time: The Pursuit of Deliverance (Fouad Ajami)
- Highlight on Page 38 | Loc. 612-15  | Added on Monday, December 15, 2014, 06:10 AM

The Egypt the Muslim Brotherhood had in mind was anathema to the free spirits in the land. It is perhaps a fragile hope, but this has always been a country of coffeehouses and storytellers, a people given to humor and forgiving of the follies of others. This is not stern Arabia, but a land with a vibrant street and night life, banter between the sexes, an earthiness that looks with suspicion on zealots and the “virtue” they claim to uphold. 
==========
In This Arab Time: The Pursuit of Deliverance (Fouad Ajami)
- Highlight on Page 41 | Loc. 656-57  | Added on Monday, December 15, 2014, 06:19 AM

Dictatorship rests on a measure of consent: an ordinary man obliges, and the crowd projects onto him its need for a redeemer. 
==========

We love consistent formatting! It usually means a file can be parsed without too much work.

Random highlight

I wonder how many people make use of that clippings file. I want to be reminded of the parts of books I’ve read that I thought were worth highlighting, and an easy-to-parse clippings file means I can cut it up into separate highlights and pick a random one to read.

I thought awk would be a good tool to solve this problem. awk is a programming language (and tool?) that lets you select and process text using pretty complicated recipes. I didn’t know any awk before creating this small hack, so it’s possible that the script below is very inefficient, or even buggy. If it is, please let me know.

I put together this awk script. Let’s call it parser.awk:

BEGIN {
  clipping = "";
  i=0;
  srand();
}
{
  if ($0 !~ /==========/) {
      clipping = clipping $0 "\n";
  } else {
  clippings[i++] = clipping;
  clipping = "";
}
}
END {
  print clippings[int(rand()*i)];
}

And I created an alias that uses that script to parse my clippings file:

alias clipping='awk -f /path/to/parserk.awk /another/path/to/My\ Clippings.txt'

Which will let you run a clipping command in your terminal and get a random highlight.

Nothing is so perfect

This works nicely, but lacks polish in a few ways:

  • I have to remember to manually copy the clippings file off of the Kindle. Right now, this means connecting the Kindle to my computer, waiting for the Kindle drive to mount, going into the documents folder, and copying My Clippings.txt to its destination. This can be automated to some extent – I can automate the file copying to happen once the Kindle drive is mounted, but I’d still have to remember to connect the device itself.
  • An alias that prints a random clipping is not the best use of this file. I still need to remember to run the command, and it’s rare that what I’m most in the mental headspace for when in terminal is to read a quote from a book. What’s the alternative? I could write a script that sends me an email with a highlight every morning. Or maybe I could use some speech-to-text system and send myself an audio snippet of the quote. I wonder if I can use a text-to-speech system that sounds natural enough. I’ll have to think about it.

Misc notes

  • Use Libby, donate to your local public library.
  • If you use a Kindle, the local storage of highlights makes the utility of books you borrow using Libby even higher.
  • Being able to write scripts that solve niche problems like this is empowering. But it relies on information being available in open, easy-to-read formats like a plain text file. This is what we lose when information storage moves to proprietary formats or inaccessible locations, and that’s where the wind is blowing in some ways. I am a little surprised – pleasantly! – that Kindle still stores these highlights in the same file format it did when I saved my first highlight back in 2012.
  • Easy open formats are not perfect. One downside of the plain text file is that rich formatting like bold and italics is lost. Kindle could decide to put the work in to add Markdown markup to preserve some of that, but that adds complexity and room for what programmers like to call a “big mess”.
  1. Libby lets you borrow ebooks and audiobooks from your local public library in the US. It’s one of the best made apps I’ve used, and the one that has changed my life the most in the past twelve months. ↩︎

  2. I was on the Seattle light rail reading my copy of Issue 7 of Logic magazine, and I looked up and caught the eyes of someone else with the same issue in their hands. It made my day. ↩︎

  3. That space in the filename bothers me. ↩︎

Archive