Embrace the Rain: Quantifying the New England Drought

The news has done a terrific job of making sure New England residents know we're in a drought (as if we didn't notice the lack of rain for the last 5 months). But they haven't done a terrific job of giving a point of reference of just how much water we're missing. In the following, I've attempted to quantify "how much rain we haven't gotten this year" and put that number into some point of reference that is easy, or even just possible, to grasp.  Be warned - the numbers are about to get huge.

Note: for simplicity's sake, I'm going to limit all these numbers to Massachusetts. I love all of New England, but it's just easier this way. 

My first goal was pretty simple: calculate the amount of rain we haven't gotten.  In other words, subtract the amount of rain we've gotten this year from the amount of rain we usually get up to this point.  Let's call that our "precipitation deficit."  Well, weather.gov publishes a map showing the "Departure from Normal Precipitation," which looks like this:

This map is accurate as of September 27, 2016 - For an updated version, visit water.weather.gov/precip.

So we can see that, measuring for 2016 so far, much of the state is between 8 to 12 inches behind the normal precipitation value, with a good portion of the state in the 12 to 16 inches range. Some small amounts of the state are 6 to 8 inches behind, and a few tiny spots have a precipitation deficit of 16 to 20 inches. The cape seems to be doing the best, and north eastern Mass. seems to be doing the worst. 

To calculate the total rain deficit across the state, I took a fairly low-tech approach: I loaded the above image into The Gimp, cut out just the Commonwealth of Massachusetts, and used the select-by-color tool to count the pixels of each color.  I wound up with the following percentages: 

Percent of State by Precipitation Deficit

Wikipedia lists the total area of Massachusetts as 10,565 square miles. Using the above as percentages of that landmass, and taking the lowest number of each range (e.g, 12 to 16 inches became 12 inches) I then calculated the total rain deficit of the state for 2016. The calculation table is as follows:

Deficit Percent of State Square Miles Inch Square Miles
16 inches 1.873% 197.88245 3166.1192
12 inches 40.203% 4247.44695 60969.3634
8 inches 52.316% 5527.1854 44217.4832
6 inches 4.3369% 485.193485 2749.16091
4 inches 1.271% 134.28115 537.1246

Adding up the "Inch Square Miles" (that's a fun unit) value for each row, and converting to a more sane unit of measurement, gives the result: we're missing about 1.604 cubic miles of water.  That's nearly 6.7 trillion liters of water! That's 6,686,409,090,000 liters of water that we haven't gotten since the start of the year!

6.7 trillion liters!

(Or, about 1.77 trillion US Gallons if you're using Freedom Units.) That's an absurd amount, and it's a tough number to wrap my head around.  So I looked up the volumes of some things I felt a little more able to grasp, and used those as units.  

 The Seawise Giant is on the bottom - It would take about 10,000 of these babies make up for our rain deficit so far this year. 

The Seawise Giant is on the bottom - It would take about 10,000 of these babies make up for our rain deficit so far this year. 

A standard "huge unit of water" seems to be the Olympic-size swimming pool, which his 2,500,000 liters.  We'd need about 2.6 million of those in order to cover our rain deficit; still too large of a number to really grasp.

Going larger - and I mean much larger - the largest tanker ever built, the Seawise Giant, was about 1500 feet long and could hold 4.2 million barrels of oil.  She was so big, in fact, that she couldn't pass through the English Channel, much less the Panama or Suez canals. 

Pretending we fill her with water rather than oil (seems unprofitable) gives a carrying capacity of 178 million gallons, or 674 million liters.  So we'd still need just under 10,000 Seawise Giants (about 9,916, really) full of rain water to make up for our deficit so far in 2016.  That's much easier to picture in my head, but it's still an absurd number. 

The Quabbin: That's a lot of rain.  Can we get 4? 

As it turns out, there's a much more reasonable, much easier-to-grasp unit of measurement we can use, and it's much closer to home. In the 1930s, the Commonwealth of Massachusetts had the rather brilliant idea to flood four towns and create a giant reservoir for Greater Boston.  It would be useful in the event of a prolonged, severe drought, perhaps. 

This, of course, is the Quabbin Reservoir, and it's the largest body of water in Massachusetts - large enough to be a reasonable unit of measurement for the amount of rain we haven't got.  Spanning almost 39 square miles, with a volume of 1.5 trillion liters (412 US Gallons), we're finally in the ballpark we need. And we'd need 4.3 Quabbins worth of rain in order to bring us back up to our normal rainfall by this time of year. 

In short, we badly need it to rain.  But, even if it were to rain half an inch every day, it would still take more than two weeks of constant rain before we made up our deficit - and nobody really wants to go through that.

So, next time it rains, don't be gloomy, be thankful.  Oh, and, water a city tree while you're at it. They need it.

Tickets for the Final Regular Season Red Sox Game are Insanely Expensive

I scraped Stubhub.com's tickets for the October 2nd game at Fenway Park (vs. Toronto), which is the final game that the Red Sox will play in the regular season.  I then compared them to the list prices for that game on MLB.com. The results aren't so much surprising as they are nauseating. 

Excluding State Street Pavilion Club tickets, which are in the several-thousands range, the average price of a ticket to Sunday's game is $390, with a maximum price of $999.  The average markup over list was 516%, with a maximum of 2705% for a standing ticket.  There's also a bleacher ticket available for just 2337% over list, if you're interested. 

If you're heading to the game on Sunday, I hope it's a good one - you sure paid for it!

Most Linked-To Domains in English Wikipedia

Made by loading the external links dump from dumps.wikimedia.org into MySQL, then extracting all external link targets.  A simple series of cut, sort, and uniq commands yielded the ordered counts, which I then numbered using Python.  The graph below, of the top 50 sites linked to by English Wikipedia, was made with Google Sheets.  

The original dataset from Wikimedia contains a list of 5,736,871 external URLs.  After deduping them by base URL, I was left with 806,564 domains that are linked to by English Wikipedia.  Therefore, the average number of links to a domain is just under 7, although the median value is one.  The 95th percentile value is 10, and the 99th percentile value is 50.

A CSV with the count and name of each domain linked to by English Wikipedia is available for download here.  Be warned, it is 23MB in size uncompressed - don't try to load it into Google Sheets, or your browser will most likely crash.


Locations of Anonymous-Access FTP Sites Worldwide

A dataset was recently made available of all FTP sites in the IPv4 address space that allow anonymous access. Below is a heatmap showing where those sites are located.  It was made by using the GeoIPLookup tool to geolocate each IP address, and then bucketing sites by proximity.  For this plot, I've combined sites within a range of 0.3 degrees of latitude/longitude.  In other words, two sites that are closer than 0.3 degrees to each other, will be bucketed into the same data point.  Each data point was then weighted on its number of constituents.  Finally, the weights were distributed on a natural log scale, with a minimum value of 1, to provide the weighting and gradient for the heatmap itself.

My Heartrate While Sick With a Peritonsillar Abscess

The heartrate, as measured by a Fitbit Charge HR, of an otherwise-healthy 30 year old male (me), diagnosed with a Peritonsillar Abscess. Extracted from the Fitbit website via API, and graphed using numpy and matplotlib.

I began feeling ill on Tuesday the 13th, but Wednesday felt better.  Thursday the 15th is when I seriously began to feel sick.  I left work early, idled around the house, and went to bed early.  Unfortunately, I did not charge my Fitbit and its battery died during the very first stages of the interesting data.


I measured a peak fever of 103.5 F (via oral thermometer) on the 16th around 3:00PM Eastern time, as marked on the chart.  I treated this with Excedrin (APAP, Aspirin, and Caffeine, typical doses) along with an ice pack. The following day, Urgent Care sent me to the ER where I received IV antibiotics, IV steroids, and IV fluids.  Within hours I began to feel better, and you can see the huge drop in heart rate and the very good night's sleep that followed.