Note: I am documenting my experience with Vroom and am trying to be as honest as possible. I do not specifically endorse the company, and neither do I condemn it. My purchase was successful, even though my experience was not uniformly positive. As they say, your mileage may vary!

Pick a Car, Any Car

I’ve bought quite a few cars over the years, both new and used, from dealers and independents alike. It’s always stressful, and every transaction feels like it wasn’t entirely in my favor. I guess that’s the nature of the car buying experience. All I want from a car dealer is to approach it as a business transaction without a lot of deceit and BS. I’m ok spending a little more for that.

When it came time to replace our family runabout, we decided to get a BMW i3 electric car. We had leased a 2014 model for two years (at a screamingly-awesome $250 per month) and loved it. The i3 is surprisingly roomy and quick around town, with lots of useful features and some lovable oddball style inside and out. Plus, it was dead-reliable and basically impervious to rust thanks to the aluminum and carbon fibre reinforced plastic chassis and body panels.

It was always puzzling why the i3 wasn’t more popular. Sure it looks funny, but it’s a BMW and has always been priced below the cheapest Tesla. I guess BMW just misjudged the global appetite for battery range. Although BMW increased the battery capacity from 18.2 kWh to 27.2 kWh in 2017 and 37.9 kWh in 2019, that’s still way below most successful electric cars, and drivers get anxious when they see two-digit range on the dashboard. But this never bothered us, since we rarely drive two-digit distances in a given day and, since we plug it in every night, it’s always full!

The fact that the BMW i3 is a moderately-successful older model these days really works in the buyer’s favor. There are many clean low-mileage examples available and they sell for a lot less than new. Being electric, they have very few mechanical issues apart from tires and brakes, and the body panels must be replaced rather than repaired. So if a used BMW i3 looks good, it probably is good.

We decided to look for a low-mileage 2017 or 2018 model with the larger “94 Ah” battery but without the “range extender” (“REx”) option. This is a tiny internal combustion engine that can recharge the battery if needed, but actually reduces electric range by 10% and only increases combined range by 37% thanks to legal limits on fuel capacity. We liked the new-for-2017 Protonic Blue color better than every other color (white, silver, silver, gray, black, ugh). And it always pays to pick a better interior and option package, since this adds very little to the cost of a used car.

Vroom: The Purchase Experience

A quick Internet search for a 2017 or 2018 BMW i3 with less than 24,000 miles returned dozens of hits. Limiting the selection to blue also eliminated the smaller 60 Ah battery and low-rent interior choices, since it was only available in upscale models. And there it was: A Protonic Blue BMW i3 from California with 21,554 miles and clean Carfax for just $18,890.

The car was listed by a company I had never encountered. Vroom is an Internet auto dealer that bills itself as a source of “high-quality cars” with “buying made easy” and “delivered right to you.” A bit of research revealed both positive and negative reviews, mostly centered on slow turnaround of paperwork and loan documents. The only real red flag were a few mentions of slow delivery of the car itself, though speed mattered less to me in the midst of the global pandemic!

I decided to make the purchase and filled in the online reservation form. I got a call from their agent fairly quickly, and she proceeded to ask me about my trade (none) and try to talk me into financing the vehicle through Vroom (no). She also needed me to upload proof of identity and insurance to their online portal, which I did shortly after. Then nothing happened.

Three days later, I called Vroom on the phone, wondering what had happened. I never did hear from that initial agent again. After a very long time on hold, I was able to reach another agent and she was much more helpful. From that moment on, it took just two days to finalize the purchase.

Once again, she tried to talk me into financing the car, promising that this would make the transaction easier since they would handle the title and registration for me. But I held firm that I would pay cash and she didn’t pressure me on financing any further. I imagine that, like most car dealers, kickbacks from financing make up a good share of Vroom’s profit. She also offered tire and wheel coverage beyond the 90 day limited warranty offered by Vroom, which I also declined.

This second agent is efficient, and the purchase process went much more smoothly. She had me pay a $500 deposit by credit card to hold the car and began the paperwork on her side. Three days later I received a big packet of paperwork via DocuSign to review and sign. I completed the paperwork the next day and wired the payment (including a $31.85 “inventory tax”, $150 “documentation fee”, $7 “inspection fee”, and $599 “delivery fee”) immediately. Although some people object to these fees, they are lower than most dealers charge. And $599 for an interstate delivery is reasonable compared to my other Internet vehicle purchases.

Vroom: The Delivery Experience

Four days after I completed the paperwork Vroom reached out to let me know that the car was being prepared for delivery. The initial delivery date was 10 in days, but the car actually arrived just three days later! Although it’s nice that it arrived so quickly, it could have caused issues in other circumstances. Everything would have been easier if delivery would have been on schedule.

Vroom instructs buyers to call them on a special number to report any damage. But my delivery happened the night before it was scheduled, and Vroom was already closed for the night.

Having purchased multiple cars this way previously, I knew what to expect from the delivery. Contract truckers pick up and deliver vehicles all over the country, and they are not associated with the dealers. On arrival, it is critical to inspect the car and note any non-disclosed damage on a bill of lading. This assigns responsibility for the damage to the dealer or the trucker.

My car arrived dirty but functional and complete as promised, except for one thing: All four tires were badly worn and one wouldn’t hold air at all. I noted this on the bill of lading and immediately contacted Vroom by email (since no one answered the phone). The trucker accepted the bill and helped me fill up the tires from his pump so I could drive to my local tire store, which was right next door to the delivery location.

The next morning, I called Vroom again and opened a support ticket about the tires. There is no way these tires should have passed inspection by any selling dealer. I suspect that the company simply passed the car from auction to warehouse to delivery without much of a second look. Plus, with the car un-drivable, I would not be able to take advantage of Vroom’s promised 7-day or 6,000 mile test drive.

Happily, Vroom agreed not only to work directly with my local tire shop but to cover the cost of all four tires. I sincerely appreciated their willingness to make the situation right without much hassle to me. But it is disappointing that they didn’t catch the bad tires in their inspection, and that this cut into my “test drive” time. They only agreed to give me one more day, even if the car was un-drivable.

Vroom: The Paperwork Experience

Titling and registration is a frequent issue for used car purchasers. I once had to delay delivery for four months until the purchasing paperwork cleared. Another time my temporary tag almost ran out before the dealer could deliver the title. This is especially typical of cars from auction, which frequently seem to have paperwork delays.

Like many used car dealers, Vroom appears to simply flip cars from auction. But unlike those dealers, Vroom seems to have waited for the paperwork before selling the car, at least in my case. I was pleasantly surprised when they overnighted the title to me just a month after purchase.

The car arrived with a Texas temporary tag (Vroom is in Houston) even though it apparently shipped from California. But this was fine to last until the paperwork arrived. All of the paperwork was properly filled out and Ohio accepted the title. I’m no expert at vehicle registration but it’s much easier than the dealers make it sound. There’s no need to pay extra for that service!

The Verdict: Vroom is OK

My experience with Vroom was acceptable but not excellent. The car was as promised, they quickly processed the paperwork, and it arrived quicker than expected. But they were slow to answer the phone, inconsistent in their responsiveness, and of course didn’t notice four unusable tires during their inspection. At least they paid to make it right.

All this suggests to me that Vroom isn’t all it claims to be. I suspect that the company simply flips auctioned cars, just like most other companies in the used car business. I imagine they would live up to the claims on their website (the 7 day test drive and limited warranty) but their lower prices suggest they’re not in the same league as Carvana.

I asked the Vroom representatives about the negative reviews and they suggested that they stemmed from problems with trade-in or financing paperwork. This sounds reasonable, since these issues would definitely cause problems. But I imagine the long waits on hold and lack of coordination between Vroom agents had something to do with it as well. My purchase was especially easy, with a full and quick wire transfer and no trade-in to deal with. So it wasn’t much of a test for Vroom.

Postscript: BMW i3 Tires

The bad tires were a major issue, but not a surprise to me. Our 2014 BMW i3 required new tires after just about 20,000 miles. And it requires special Bridgestone Ecopia EP500 tires, “staggered” in two different sizes front and rear. There is no alternate tire that will fit, and they’re not usually in stock. My tire guy had to order them from Oklahoma! Even so, he was able to replace them in 6 days.

Different i3 models come with 19″ and 20″ wheels. They’re a crazy size: Tall and narrow. The Ecopia tires really are the only ones that will fit. The 19″ tires are all-season, but the 20″ are summer-only. Our new i3 came with 20″ wheels, and it does slide around in the snow!

While my local tire guy was locating replacement Ecopias, I also asked him to source some snow tires for the car. Typically for the i3, there is only one wheel that fits, the Rial X10-1, and one snow tire for that, the Bridgestone Blizzak LM-500. So it wasn’t much of a decision to order that package. Happily, the winter wheel and tire package was partly subsidized by Vroom, who covered the replacement Ecopia tires which I had already budgeted for.

© Stephen Foskett for Stephen Foskett, Pack Rat: Electric Car Over the Internet: My Experience Buying From Vroom

This is week “1111111111” in the GPS system. Tomorrow morning it will roll over to week “0000000000”. How well will various systems handle this change? Not well, judging by what we’ve seen so far!

Epochs, Weeks, and Seconds: How GPS Time Works

The GPS system consists of constellation of satellites orbiting in known positions, each with an ultra-precise atomic clock inside. Each constantly broadcasts the exact time, and a receiver on the ground compares the time code with the position of the satellite to triangulate location. The rest of this blog post has very little to do with any of this, however.

Because the time is a key component of the GPS system, most receivers synchronize their date and time display to the received signal. Most of us don’t use standalone GPS devices anymore, but clocks synchronized to GPS are integrated into just about everything these days from cars to Internet servers. This is the same reason the clock on your iPhone is so accurate: The mobile phone network receives GPS signals and sends the precise time and location in a very similar way.

So-called GPS time is counted very differently from how people tell the time. The GPS network uses a 10-bit counter adding up the number of weeks since a known date (the “epoch”) and has another 19-bit counter for the number of seconds-and-a-half since midnight at the start of Sunday.

That’s right, GPS tells “time” and “date” with Week Number and second-and-a-half increments into the week. Week Number makes more sense than you would think, proceeding regularly with no leap days or odd lengths. But why 1.5 seconds? I guess to save one bit. There is no concept of seconds, minutes, hours, days, months, or even years to GPS time. It’s all just Epoch, Week Number, and Time of Week. In 29 bits, GPS can encode the time with sufficient precision to locate you in almost two decades and that’s good enough.

Each counter “rolls over” differently. The Week Number will naturally reset to 0 after 1,023 increments, as all the bits are flipped. But the Time of Week counter is forced to reset after 403,199 increments, even though 19 bits gives some headroom past that. The reason is that a week is 604,800 seconds, or 403,199 seconds-and-a-half, long. So there you have it.

Humans Count Date and Time Strangely

If this sounds weird and unwieldy to you, congratulations you are human. But to a computer, this is a straightforward and useful way to count the time. Computers don’t care about the number of times the earth has revolved around the sun since an arbitrary point in human history. They also don’t care that their count doesn’t exactly line up with the precise moment a certain spot on earth faces the sun, one twenty-fourth the time the earth takes to reach that spot again, one sixtieth of that increment, or one sixtieth of that. But you might, so it has to translate for you.

This is where the trouble starts.

The first GPS epoch began the Sunday before the start of a new decade, and midnight on Sunday, January 6, 1980 was the first time the Week Number incremented. This was a usefully-old yet reasonably-recent date to pick for “everything” to begin.

A 10-bit binary counter will “roll over” after 1,023 increments (in our familiar decimal system), which gives the Week Number almost 20 years before we have to worry about the date. But the first Week Number rollover has already happened: On Sunday, August 22, 1999, the Week Number reset to 0 (or “0000000000” to be precise). And this will happen a second time tomorrow morning, April 7, 2019.

The Epoch is manually set in each GPS clock, and the clock needs to know the rough date to within 9 years, 10 months in order to display the correct year, month, and date. But the useful life of these systems means that a user must enter this information if the system loses power or the wrong date will be displayed. If you tell your GPS (or car dashboard or clock or whatever) that it’s more than 10 years off, the Epoch will be interpreted incorrectly and the date and time will be shifted by 19 years, 8 months, and some odd number of days, hours, minutes, and seconds.

This can be hilarious, weird, or it can cause problems. Is today April 6, 2019 or August 21, 1999? GPS doesn’t know. And whether that matters is entirely up to the application.

“Interesting” GPS Dates: Conversion, Rollover, and I-Don’t-Know-What

Even a properly-programmed GPS device can show a wildly-wrong date. But not all devices are programmed correctly.

A serious failure of GPS time actually happened 168 days ago. On October 21, 2018, the decimal representation of the GPS Week Number reached 1,000, exceeding three digits for the first time since most modern devices were created. Most systems use the binary number internally, so this isn’t an issue. But some programmers converted to decimal before doing their Epoch lookup. Such was the case for the car I bought a few months back, which I’ll discuss below.

There have been other “interesting” GPS time errors over the last few years. It’s hard to know exactly what mistakes the programmers made, but devices have rolled over or failed when the Week Number reached 800 (in December 2014), 860 (February 2016), 886 (August 2016), and 987 (July 2018).

How will your favorite GPS-based clock or device handle the rollover? It’s hard to say. The fact that so many have failed in such unusual ways gives cause for serious concern. I think we might be hearing about a lot of odd issues for the next few months.

Happily, none of this will occur again for a very long time after tomorrow morning. GPS is being modernized to use a 13-bit Week Number, which will certainly help as well.

A Real-World Example: Mercedes-Benz COMMAND NTG3

I recently bought a 2008 Mercedes-Benz S-Class with just such a system. Around October 21, 2018, all models with Mercedes’ COMMAND NTG3 infotainment system decided it was actually June, 2003. And the incorrectly-converted GPS signal continues to reset the date accordingly at every opportunity. As of today, my car thinks it’s August 21, 2003.

Why 2003? It’s hard to say exactly. If the system simply “rolled off” the third decimal digit, it should be showing the date as February 5, 2000. This would be week 23 (instead of 1,023) of the current GPS Epoch. Instead, it seems to think it’s week 228. If I had to guess, I would say the Mercedes (née Daimler) Benz programmers picked another date as their Epoch and are basing the date off of that. The fact that the failure first occurred all over the world just as the Week Number passed 999 definitively shows this to be a GPS error, however.

There’s another issue happening here too, but it’s a little harder to see: August 21, 2003 was Thursday, not Saturday like today. This means that the car is using the wrong start of the week when calculating the time of day. And since GPS Time simply counts up from Sunday, this shows that something else is seriously wrong with the time software. It’s a miracle it’s showing anything close to the correct time.

What does it matter? Apart from the inconvenience of the date being incorrect in the display, this incredibly over-engineered car has another perplexing issue. See that nice analog clock between the vents? It’s set automatically by the COMMAND system. Since the GPS date was incorrect, and the time is calculated based on Sunday, the clock is wrong in thousands of cars all around the world. And it’s doubly wrong since it thinks it’s summer, so daylight savings time isn’t in effect! Most clocks were showing a little over 2 hours off.

I can’t say why my clock is showing the correct time even though most others are not. I believe it has to do with the fact that my GPS navigation system was broken when I bought the car, and had been broken since well before October. Although I fixed it (replacing the original 2.5-inch hard disk drive with an SSD) it still shows the right time. Thank goodness!

Who knows how the Mercedes-Benz COMMAND system will handle the GPS Epoch reset at midnight. Mercedes-Benz refuses to accept responsibility for this bug and will not fix it, even though it affects top-of-the-line cars less than 10 years old. And one would think that IWC, a renowned maker of high-end Swiss clocks, wouldn’t like their name associated with a clock that refuses to show the correct time.

References:

https://www.dhs.gov/cisa/gps-week-number-roll-over

https://ics-cert.us-cert.gov/sites/default/files/documents/Memorandum_on_GPS_2019.pdf

Stephen’s Stance

The GPS Time bug is part of a whole category of time formatting and storage bugs, from the familiar (Y2K) to the perilous (2038) to the obscure (Reiwa). The fact that a simple system like GPS time can fail in so many interesting ways and places shows just how much our world is reliant on the assumptions of programmers and the ability of companies to properly quality-test their products. And the fact that world-renowned companies like Mercedes-Benz (and by extension IWC) refuse to fix their own bugs does not set my mind at ease.

Update: What Happened and Why It Matters

It’s April 7, 2019 and GPS Week Number is now 0. The world did not end, but some failures have been noted. And some things that were broken remain broken.

As for my Mercedes, the date is still wrong. So much for that.

Why does it matter? Some systems aren’t so trivial as a dashboard clock or date display. What if they divide by week number and crash? What if they’re calculating the time offset from a previous date and now read negative? What if something is supposed to happen on a certain date and now it won’t for another 20 years?

Most importantly, this shows that programmers don’t always make valid assumptions about infrequent issues and QA doesn’t catch them. And they didn’t catch this. This is an issue beyond GPS time.

© Stephen Foskett for Stephen Foskett, Pack Rat: GPS Time Rollover Failures Keep Happening (But They’re Almost Done)

Insecure Email

Email is not secure. Even with TLS and good passwords it’s far too easy to snoop, phish, or stumble into someone’s email account. And this is especially true of our “daily” email accounts: If you’re receiving email on your phone, iPad, and computers at home and work you’re leaving yourself vulnerable to account highjacks.

There’s really no way to fix this, and it’s not Google’s fault or anyone else’s. You want your email to be accessible whether you’re at home, at work, or on the road. And it’s useful to have email alerts “bust through” your lock screen.

This is one reason email is fundamentally insecure. Since you want it to work everywhere and go everywhere, it’s designed with the lowest common denominator in mind. So email protocols are fundamentally insecure by design. It’s a feature, not a bug!

But we don’t just use email to chat and do business. We also use our email accounts as a verification factor for password resets and to receive intensely-personal information from our banks, doctors, and so on. I don’t blame these sites for using email addresses for security: Email is the only universal account, and I much prefer emailed verification than some kind of proprietary authentication, handing over even more power to Facebook, Google, or Twitter!

Get a “Backstop” Email Account

It’s time to stop mixing communication and authentication in the same email account.

The solution is simple: Get another email account for security-related functions. You can keep using your regular email for regular communication, but redirect security and financial information to a secure account.

If someone was to hack into my email, they’ll hit the Gmail account since that’s all that’s set up on my iPhone, iPad, and MacBook. When I need to change a password or verify my credentials, I manually log in to my secure account using a web browser.

Many people use another provider for this sort of thing already. I long used a quiet Yahoo account for verification rather than my familiar Gmail-powered fosketts.net address. But after the recent Yahoo hack I stopped using this account and went looking for something better.

ProtonMail is a Great Backstop

I wanted to find a new account for security and authentication that was really secure:

• Encrypted at rest with serious security on the back-end
• Support for complex passwords and two-factor authentication
• Compatible with ordinary SMTP for incoming and outgoing mail
• No need to access from ordinary applications or standard IMAP protocol
• An iOS application would be nice as long as it’s secure too
• Location in a trustworthy location and legal jurisdiction and developed by credible people
• Cheap or free and managed (so I have less work to do)

ProtonMail checks all the boxes for me. It’s a secure email account in Switzerland with end-to-end encryption developed by CERN researchers. Internet email is exchanged using standard protocols but is encrypted using per-user private keys for storage. ProtonMail staff can’t access the contents of a mailbox even if they wanted to, and Switzerland has very strong notification and review laws.

Access to each email account uses a second key, which is decrypted on the client side using the account password. Email can be accessed through a browser-based application or mobile application for iOS or Android. And ProtonMail supports two-factor authentication standards, including Authenticator.¹ ProtonMail even supports encrypted and authenticated account-to-account communication, but this isn’t one of my requirements.

To be clear: You can not access ProtonMail from a regular mail client. You have to use their webmail or mobile apps. And that’s a feature, not a bug, since it means that all mail access is secure, end-to-end!

In practice, ProtonMail has worked out great for me. I can use my account as the verification email for pretty much any online service and I feel much more confident that it won’t be hacked.

Since I only use my ProtonMail account for verification and authentication, I’m not as concerned with some of the peculiarities of the service. The iOS app works great, but it’s not integrated with everything else on my iPhone like Apple’s Mail app, and I have to enter my Authenticator code fairly frequently, slowing down access. But that’s not a hassle since I only use ProtonMail once every week or so. And they support desktop and mobile notifications, so I know when I need to log in.

Stephen’s Stance

It’s an easy decision: Get a free ProtonMail account and use that as your verification address for important financial and social media accounts. Keep using whatever email account you like for regular communication, but don’t mix security and communication!

© Stephen Foskett for Stephen Foskett, Pack Rat: Go Get a ProtonMail Account and Protect Your Online Life!

I spend a lot of time on conference calls, often using one of the too-plentiful slide and video sharing applications. I’ve found that the iPad version of these apps is typically more stable, quicker to start, and more up to date than the Mac or PC version. Frankly, I hate installing the browser plugins and so on required to get these applications to function, and the iOS version just works!

I could just sit an iPad on my desk. But where’s the fun in that? If something is going to sit on my desk, it should match the retro tech style!

Way back in 1992, I picked up a broken Mac SE from a guy on Usenet. I was an avid Atari ST user at the time, but I couldn’t resist the draw of free tech! Alas, the CRT tube was bad and I never got around to fixing it.

Now here I am 25 years later and that Mac SE is still sitting in my basement, waiting to be put to use. I considered making a “Macquarium” but the fish visibility factor would be quite low with that little 9-inch screen. But that got me thinking: Would an iPad fit in that space?

Although the screen of a full-sized iPad might fit in the CRT opening, the borders would be cut off. And there would be no way to access the home button or camera. But an iPad mini would fit perfectly!

Happily, I had just such an iPad sitting in my desk: My daughter broke the screen on hers and saved up to buy a replacement. Even with a broken screen, the Mini still works fine. Perfect!

I removed the CRT, analog board, power supply, and logic board of the Mac and tried it for fit. Perfect! Held in landscape orientation, the iPad mini is almost exactly the right size horizontally, if a bit narrow vertically. Although the outer buttons and switches are unreachable, there are soft controls that take their place.

I sat down with Tinkercad and designed a special “grommet” to hold the iPad in place. I designed it so it could be used at any of the four corners of the iPad, attaching to the original CRT mounding holes with the original Torx screws. It’s a tight fit for the power cord, but everything lines up nicely.

This was a pretty satisfying project. The iPad is extremely useful for conference calling, FaceTime, and so on. And it’s cool to be able to recycle some old, unused electronics into a useful desk accessory.

If you’ve got an old 9-inch Compact Mac, an iPad mini, and access to a 3D printer, you can do the same! Just download my Macintosh iPad mini Grommet from Thingiverse and print four of them!

There was only one question lingering in my mind: What should I do with all the wasted space in that old Mac SE case? Hmmm…

© Stephen Foskett for Stephen Foskett, Pack Rat: The Biggest, Clunkiest iPad Mini Case Ever: My Old Mac SE!

The book, “Essential Enterprise Storage Concepts“, is intended as an introduction to the field of enterprise storage for technical audiences. It’s not some grand discourse on how storage ought to be done. Rather, it’s an overview of what you need to know about enterprise storage.

I start with some storage basics: The difference between memory and storage, a discussion of storage media, and the perennial topic of block vs. file storage. Then I talk about scaling storage arrays and the capabilities that enterprise arrays bring to the table. In a dozen pages, you’ll know enough to have meaningful conversations with storage vendors.

Then I shift gears, talking about performance and capacity. I begin with throughput vs. latency and the implications of parallel and serial storage access. Then I go into queueing, caching, and streaming and the difference between asynchronous and synchronous access. This will help you to understand how storage works in production.

The next section focuses on understanding storage usage. Storage administrators spend much of their time measuring and analyzing capacity, and this question of utilization is essential. We then talk about optimization of capacity and the trade-off between capacity and performance. If you’re looking at storage management software, this section is for you.

Finally, I wrap up with a discussion of storage best practices: Managing and monitoring storage, storage system sizing and planning, storage design considerations, and data protection fundamentals. This is a very-brief summary of the rules of thumb I’ve come to rely on after decades in enterprise storage.

The whole book is under 30 pages long, and I am very proud to have produced this concise treatment of enterprise storage. I urge you to download the book and give it a read! And I welcome your feedback!

Note: Somehow, the sidebar entitled “Flash Storage” has the wrong text. Here’s the correct sidebar text: “We’ll be talking a lot about flash memory later. Although this is dynamic, solid-state, and read/write, it has special characteristics that make it ill-suited as primary system memory. Plus, it’s non-volatile. Therefore, even though it’s technically memory, we typically use it as storage!”

This e-book is sponsored by SolarWinds, who also handled the formatting, illustrations, and design. I encourage you to learn more about SolarWinds products and check out their thwack technical community, and especially the Geek Speak blog!

© Stephen Foskett for Stephen Foskett, Pack Rat: Download My Free E-Book, “Essential Enterprise Storage Concepts”!

A few years back, I wrote an immensely-popular series of blog posts outlining the four things that were holding storage system performance back, and the ways to fix them. At the time, I created some presentation content to go along with these posts, and even considered pulling them into a white paper, but nothing came of that. Now, however, I’m pleased to announce that my Four Horsemen are accompanying me to the stage November 10, 2015 at the DeltaWare Data Solutions Emerging Technology Summit in Edina, Minnesota.

Here’s the original “Four Horsemen” series:

1. The Rule of Spindles
2. Never Enough Cache
3. I/O As a Chain of Bottlenecks
4. Get Smart

When I wrote those four pieces, back in 2010, the industry was on the verge of a major shift: Hard disk drives had always been the critical inhibitor of storage system performance (see “The Rule of Spindles“), but widespread use of flash memory meant this was about to change. Many people don’t realize that RAID was originally designed primarily to deliver increased IOPS, not to protect data in the event of a drive failure. Yet even today, hard disk drives remain a critical gating factor to storage system performance. The advent of sequentialization, hybrid flash cache, and smarter storage systems has mitigated the problem, but most data still lands on a spinning disk eventually. Which means that we’re still stuck optimizing them.

NAND flash has also dented the second “horseman” (see “Never Enough Cache“), since it’s much cheaper on a per-capacity basis than DRAM. This is why virtually every storage system today has a large (by RAM standards) pool of NAND flash to serve as a temporary landing zone. And to stretch that metaphor, most systems also use NAND flash as a “pre-departure waiting area”, storing data that is likely to be read soon. And now that storage I/O has become quicker (see below), we’re starting to see optimization above NAND, with SLC caching MLC and DRAM above both.

Regardless of the performance of the storage capacity layer, however, there will always be bottlenecks in the I/O channel (see “I/O As a Chain of Bottlenecks“). Faster storage (thanks largely to flash) has disrupted the industry, with the “big SAN” model giving way to top-of-rack flash, converged infrastructure, and even re-internalized storage. Next-generation solid state storage is so fast, in fact, that it requires some extreme re-engineering to take advantage of all these IOPS. Otherwise we’re left harnessing a racehorse to a wagon.

The most exciting area of enterprise storage today is the emergence of intelligence and integration (see “Get Smart“). No matter what a storage device can do, we can’t maximize return on investment without optimizing data access. This is why today’s trend toward data-aware storage, integrated caches, server/storage integration protocols, and software-defined storage is exciting. If the array’s capabilities (and limitations) are exposed to an intelligent data management layer, we can extract maximum performance from the whole system. Even those pesky hard disk drives.

If you’re in Minnesota in two weeks, come on down to the DeltaWare Data Solutions Emerging Technology Summit! I’ll also try to film my presentation and post it here. And if you’re organizing a conference yourself, might I suggest that you reach out to bring me in as a speaker?

© Stephen Foskett for Stephen Foskett, Pack Rat: The Four Horsemen of Storage System Performance Are Coming To Life!

The Volkswagen situation is really remarkable in its brazenness. Pretty much every diesel passenger vehicle sold in the USA uses urea injection to tame nitrogen oxide and meet the US EPA’s very-strict emissions standards. But Volkswagen somehow managed to meet the standards without urea in their popular 2.0 liter TDI engine. Automotive geeks always wondered if there was some technical trick Volkswagen had developed to pull this off. Turns out, there was!

As the old saying goes, “your mileage may vary.” By now, most car buyers know they won’t match the “Monroney Sticker” fuel economy estimates, yet these still prove valuable when cross-shopping vehicles.¹ The same holds true for Energy Star ratings, lightbulb lumen/wattage numbers, and nutritional information.

Your mileage will vary, but most consumers assume that those benchmarks were obtained on a level playing field. What kind of company would develop as sophisticated a cheat as Volkswagen is alleged to have created? What does this say about the company and the state of mind of those who created it? As Planet Money points out, sometimes people do bad things in business contexts.

But that doesn’t mean we should forgive them or resign ourselves to being cheated. As an industry and as a society, we should strive for fairness always. And we should punish those who abuse that trust, even though it’s likely to happen again. And because of that, we always have to be watchful!

© Stephen Foskett for Stephen Foskett, Pack Rat: Your Mileage May Vary: Lies, Damn Lies, and Benchmarks

My 3D Printer: AIO Robotics Zeus

Way back in 2013 I decided to buy a 3D printer. I picked a brand new one on Kickstarter, the AIO Robotics Zeus, because I believed the team could deliver and that it would advance 3D printing from the tinkerer level (most earlier models were about as user-friendly as an Arduino) to the hobbyist realm (more like a Raspberry Pi). After a year of predictable delays, my printer was delivered in December and I’ve been playing with it ever since.

The Zeus is a true all-in-one device in two important ways:

1. It prints and scans with no reconfiguration. Although there are other printers with a scanner option, most require substantial effort when switching from one mode to the other. It also copies and “faxes” (sending models over the Internet to other Zeus printers).
2. It includes an embedded touch screen computer for truly standalone operation. Because 3D printing takes hours, it’s good to have a dedicated, purpose-built computer running the whole operation rather than tying up your computer and risking your model after 4 hours when Windows updates itself.

Although this combination seemed a bit aggressive for a Kickstarter project, I decided the AIO Robotics team was up to the challenge after reviewing their design: Most of the tricky bits of the Zeus are open source and off-the-shelf, including the printing and scanning software, and the design is fairly straightforward.

The only truly custom part of the Zeus is their “Thunderbolt” extruder. This is the heart of any printer, and the fact that they were designing and manufacturing their own gave me pause. But my faith was rewarded and the extruder has proven extremely capable. In fact, this could be the most valuable piece of IP the team has developed!

In practice, the Zeus prints quite well but falls well short in scanning. It maintains a library of printable objects internally and printing is both simple and predictable. The chassis is solid, with thick guide bars and a sturdy shell, and the only printing failures I’ve experienced were due to faulty designs or cracked filament.

Scanning is another story. Although some scans have worked out perfectly, most are useless, with extraneous “ghost blobs” appearing inside and outside the object in question. But AIO Robotics has consistently released software updates and scanning is improving markedly.

I have had a few issues, however. First, a too-soft adhesive was used to attach the removable glass turntable to its base, and this has come loose for me and a few others. I superglued it back together and it seems to be holding up. Also, the team is nearly absent from their own forums¹ and hasn’t communicated with us owners at all since delivery. They seem still to be working, delivering software updates and news posts, but I’d like a lot more interaction as an early adopter!

Overall, I’m happy with the AIO Robotics Zeus printer. It is as functional and user-friendly as such a device could be at this point. So let’s step back and consider the category.

You Can’t Print That…

As a category, 3D printers are remarkably limited. Given the “maker” hype, and what I’m going to say in the next section, I think a massive reality check is in order!

Here’s what you truly can’t do with a 3D printer:

1. Objects don’t magically appear – Buying a laser printer doesn’t mean you can immediately have the next great novel in your hands, and buying a 3D printer doesn’t mean you life will instantly be populated with cool toys. You have to create before you can print, or find someone else to create for you. This is non-trivial and time-consuming, even with excellent tools like Tinkercad and Thingiverse at your disposal.
2. Very few things can be 3D printed – There are massive limitations to PLA extruder printers like the Zeus or Makerbot. PLA isn’t that tough or durable, so you can’t make anything really rugged. And extruders need support while printing, so you must be very careful and keep printability in mind when designing objects. You can make models, cases, brackets, and so on but not much else without using a totally different kind of printer.
3. 3D printing is not economical – 3D printers and PLA filament aren’t out of reach for many people (expect to pay $1k-$2k) but the final cost of printed goods is not cost-competitive with other production methods. Divide the investment by the number of printed items and you’re looking at some very expensive figurines and vases! And printing is too slow to make up the cost in volume. 3D printing is a prototyping tool and a toy for first-worlders and nothing more.
4. It takes a long time to print – 3D printing sounds like a killer way to produce a vast assortment of objects on-demand, but the technology is way too slow for that right now. Yes, you could take a self-contained printer like the Zeus to a craft show and print stuff, but you’d only be able to make a few objects before closing time².
5. 3D printed objects are small, rough, and inelegant – If you’re making a computer case or a mounting bracket you’re probably happy with what comes out of a 3D printer, but most people would want a slicker product. Even at high resolution (which takes much longer) it will be obvious it came from a 3D printer. And you can’t print a new paddle for your kayak or anything else larger than would fit in a breadbox.

All of these limitations will disappear eventually. But right now they keep 3D printing (especially PLA extruder printing) solidly in the realm of “gee whiz”.

Pondering the Implications of 3D Printer

Despite the limitations I just mentioned, I remain bullish on the concept of 3D printing. And it all boils down to a simple fact: It is simply mind-blowing to go from idea to physical object without leaving your desk!

So many of us spend our days sitting in front of a computer, interacting with virtual objects and virtual people, yet we yearn for physical reality. We want to be able to touch what we see, but we never can. 3D printing changes that, putting virtual objects into our hands. And this is transformative in a way that few other technologies have been.

Think of something you’d like to have³. If it can be 3D printed, you can probably design it in Tinkercad, download it to your computer or printer, print it, and have it in a few hours. And you can take pleasure knowing that it’s yours, that it’s unique, and that it’s exactly what you wanted.

This joy of creation isn’t well-known in the digital arena. Sure, some programmers feel it. But most computer users are just interacting, not creating. 3D printing brings us back to our roots as tinkers and crafters. It lets us know how it felt to be a furniture or watch maker, to build something and have it be ours. It’s a high-tech jackknife or hunk of clay.

And someday 3D printing will be much more. PLA extruders only get us so far, but there are many other kinds of 3D printers being developed. You can already craft objects out of metal, ceramic, and even chocolate! And CNC machining (really industrial 3D printing) is already a mainstay of modern factories. After witnessing one of these devices in action, it’s clear that this is the future.

Stephen’s Stance

Although I enjoy my AIO Robotics Zeus 3D printer, I’m under no illusions that it’s an economical or practical device. It’s a toy that takes me into the future, and I love being there!

© Stephen Foskett for Stephen Foskett, Pack Rat: Thoughts on the Modern Miracle of 3D Printing

The Scam

I’ll let Lee speak to the scam. If you’re interested in reading his whole tale, including a personal investigation and attempted transaction, I urge you to read his story, The Curious Case of Bogus Amazon Sellers.

I’m sure I’m not alone in admitting that I generally love Amazon. The access to massive product variety, frequently great prices, the whole Prime feature, and a sense that you can really trust the entire framework just makes Amazon easy to appreciate. But that trust thing… well, lately I’ve had it rocked a little bit when it comes to Amazon. Here’s the executive summary:

• I have found multiple clearly fraudulent sellers in the “used” category
• I’ve engaged Amazon’s customer service and investigations staff, had my suspicions confirmed and told by Amazon they’d get rid of the bogus sellers
• The same sellers keep coming back, and they are pretty convincing if you don’t know better
• There seems to be no way for Amazon to keep them out

Essentially, there are scammers actively “selling” on Amazon, blatantly asking to go “off-site”, and using classic phishing techniques to defraud customers. And Amazon does nothing actively to stop this. Back to Lee:

That anyone can join the Amazon used market as seller and then be allowed to tell customers to go through email and break Amazon’s rules WITHOUT AMAZON THEMSELVES CATCHING IT is bewildering

It’s easy to find similar offers on Amazon. Search for something expensive (a MacBook Pro, for example) and look for too-good-to-be-true offers in the used section. As Lee notes, the scammers appear alongside valid sellers. Amazon does sort them to the bottom of the list (maybe because they’re new/unverified sellers) but allows them to appear!

Here are the hallmarks:

1. Too-low round-number prices roughly half the retail cost
2. Items sold as used but with specific notes that they’re actually new
3. Instructions to email to begin the transaction rather than using the Amazon site, including obviously obfuscated gmail addresses with spaces between letters
4. “Just Launched” seller profiles with no ratings

They’re honestly very easy for a buyer to avoid. But I wonder how many people take a risk on a “great deal” and get burned!

What Can You Do?

First, buyer beware. Do not assume that Amazon polices their third-party sellers and be very skeptical of offers that seem too good to be true.

But it’s not fair to place all the blame on the buyer. This is Amazon’s site, and they should stop this practice. Sure this one is pretty blatant, but I bet none of us would like to live in a totally “caveat emptor” world. We shop at sites like Amazon because they offer us some peace of mind. We will stop shopping there if we lose this.

Everyone reading this can contact Amazon and publicize the problem. Share Lee’s article on Twitter and tag @Amazon so they know it’s a real issue. Tell them you are upset and want some resolution. Escalate!

Stephen’s Stance

If something seems to be too good to be true, it probably is! It ought to be easy for Amazon to create a scam filter to block these, but they don’t seem at all interested, as Lee shows. This is their responsibility: By creating this marketplace, Amazon took on the responsibility to police it. It’s time for them to step up!

© Stephen Foskett for Stephen Foskett, Pack Rat: Blatant Fraud At Amazon: Do Something About It!

Note: This article is cross-posted to both my technology blog, Pack Rat, and my watch blog, Grail Watch

Is the Apple Watch a personal communication revolution like the iPhone, a well-executed gadget like the Apple TV, or a total miss? Does it mark the end of the the world as we know it for watches? And what’s it like to use one? I’m a watch guy and a gadget guy, so perhaps my perspective will be of some value.

What is an Apple Watch?

When it was announced, it was not immediately clear exactly what Apple’s long-awaited watch was supposed to do, and Apple failed to make that case in the ensuing months. Is it a standalone device or an extension of an iPhone? Is it an interactive tool or read-only display? And, most importantly, how would the use of an Apple Watch improve one’s life?

Apple has long been at the forefront when it comes to designing and selling electronic gadgets based on real life usage rather than the “speeds and feeds” model that trips up the rest of the industry. ¹ That’s why I was surprised that Apple introduced the Watch with a gadget-y demonstration of features. The world needs more than “it does stuff” as a marketing message – can you imagine if the Rolex tag line was “it keeps time pretty well”?

Despite this marketing stumble, there is an implicit message to Apple releasing a Watch. To a huge number of people, the Apple brand connotes fashion, design, quality, and a priceless cool factor. ² This isn’t the Apple WATCH, it’s the APPLE Watch and that’s all that matters to many buyers.

Furthermore, the Watch adds a new element for Apple products with its three different price brackets. Previously, just owning an Apple thing was enough to secure bragging rights. With the Watch, there are tiers of status beyond Apple, including the hyper-exclusive gold Edition models that have already been appearing on the wrists of the glitterati that so many of us either love to hate or hate to love.

Clearly the Apple Watch is a piece of jewelry that denotes social status. But we knew this before it was even unveiled. So what does it actually do? Apple seems at a loss to explain this, offering up a scattershot of use cases from the reasonable (Up – “it’s a sports band”) to the insipid (Us – “sweeties sharing heartbeats”) to the questionable (Rise – “do stuff all day”).

Realistically, Apple has no more idea than any of us what the Watch is for. They’ve suggested what it might do, but it’s an extensible platform and no one knows where it will go. Remember that the iPhone was introduced as a “widescreen iPod with touch controls”, a “revolutionary mobile phone”, and a “breakthrough Internet communicator”. Would you use these terms to describe what it has become today? Apple introduced a cool thing that does stuff and the world decided what it was, and the same will happen with the Watch if it gets the chance.

Today, the Apple Watch is mainly an extension of your iPhone. Some things it does are very useful (Siri, notifications, directions) and everything else is more promise than reality. Even the obvious sports band use case is waiting for better software.

Impressions of Apple Watch 1.0

Regardless of the promise it shows, an Apple Watch is a physical thing for sale today. Let’s turn for a moment to what this Apple Watch is like rather than what some future descendant might be.

Note: I’m a watch nerd! There will be watch jargon here!

I purchased a stainless steel 38 mm Apple Watch with the Classic Buckle strap. I selected this size because I prefer smaller watches and expected it would “wear” large due to the bubble back sensor, high lug placement, and rectangular shape. The steel model includes a sapphire crystal, has better strap choices, and isn’t much more expensive than the aluminum unless you get the metal bracelet. The Classic Buckle is the most “normal” strap choice and was a logical selection sight-unseen.

I’m very pleased with the materials and physical construction of my Apple Watch. The steel is flawless and the Watch is perfectly assembled. Even under the loupe, I see no defects in workmanship, and this is not something I can say of many $650 watches. I love my Seiko watches, but they typically have a rough line or slightly misaligned seam somewhere. The extreme simplicity of the Apple Watch design certainly helps, but it’s obvious that the company sweated the small stuff and that’s very good.

The Classic Buckle strap is excellent. The leather by ECCO in the Netherlands is extremely supple and top-quality. It’s so good that many people will likely think it’s actually rubber, an impression bolstered by the embossed and exaggerated texture. The buckle is a new, modern design (despite the name) that would be right at home on a TAG Heuer or a Hublot. Seriously! The Classic Buckle just might be the best part of the Apple Watch.

But there are some misses.

The slide-and-click strap/lug attachment is novel but disappointing in person. There’s a sizable (in jewelry terms) gap and a bit of wiggle there that makes this seam noticeable, so it’s a good thing that the shape harmonizes with the overall “savonette” design. ³

In the world of watches, an inordinate amount of energy goes into fancy presentation boxes. The more expensive the watch, the bigger, heavier, and fancier the box. ⁴ The steel Watch comes in a bigger, heavier box than the aluminum Watch Sport, which is nice. But the box itself is a cheesy plastic affair that fails to do the watch justice. It’s slickly taped closed but begs to be opened with two hands. This is bad because the watch sits sideways inside with nothing holding it in place. So it tumbles on the floor.

The controls are pretty confusing, too. In the introduction, Apple wisely noted that iPhone-like touch controls aren’t great in a watch, since your fingers get in the way. So they “invented” the digital crown and included a second button to boot. Yet most essential interactions with the Apple Watch involve your big, chubby fingers on the screen anyway. The digital crown is mostly used for secondary interactions and they vary by app – is it for zooming, scrolling, or selecting? And I have yet to press that other button in anger, since it only calls up a list of contacts.

Initial setup seemed very slick but failed in my case. I was able to pair the phone using the camera and a lovely pattern on the Watch but was then faced by screen after screen of additional steps that were nevertheless incomplete. For example, Apple Pay setup is manual and fiddly yet is one of the primary functions of the Watch. And in my case at least my third-party apps never completed syncing to the watch so I had to go in one at a time and individually remove and re-add them.

Up and running, the Apple Watch is pretty useful. Notifications give you a nice little thump and beep and you can often respond right there. This is especially true of iMessage, which is simply wonderful. I would rather text using the Apple Watch and Siri than any other interface! Right now, iMessage is the most useful thing to do with an Apple Watch.

Other apps are less useful. Many iPhone app notifications can be duplicated to the Watch but these are often frustratingly incomplete in terms of interaction. You see the notification but can’t do anything with it, especially in the case of apps that don’t yet have an Apple Watch component. And those that do have apps vary from awesome (Instapaper, 1Password) to pointless (Twitter) to non-functional (United). This will get better as developers become used to the Watch environment and create apps for it, but it’s annoying on day 1.

What Does the Apple Watch Mean For the Watch Industry?

Watch geeks have long sneered at smart watches, but the Apple Watch will likely affect the watch industry more than they want to admit.

First, let us consider the question of wrist space. Most people don’t wear anything on their wrists, so Apple must convince them to start. Everyone else already wear a bracelet, wristwatch, or sports band, and Apple must convince them to switch. ⁵

Neither of these tasks will be easy, but let’s assume the Apple Watch becomes a successful trend. What would this mean for the watch industry?

Some wearers-of-nothing could be drawn in by the Apple Watch and then “graduate” to a luxury watch or other wrist jewelry. This has happened before: My generation was born on digital watches and gadgets yet have become the heart of the luxury mechanical watch market. Casio and Swatch was a gateway to Jaeger-LeCoultre! And I see the same kind of enthusiasm for hand-crafted mechanical watches among my peers. Sure, not everyone wants a fine watch. But the market is booming right now!

Apple’s Watch could convince people to start wearing watches. And if the device fails to achieve sustainable success, they could move on to a luxury watch next. ⁶ It could even spark a frenzy of iconoclasts buying mechanical watches as an antidote to the Apple masses.

But it’s not all good news. As watch salespeople lament, most watch buyers are not watch nerds. Most see a fine watch as a piece of luxury jewelry that signifies their taste, status, or wealth. These buyers will snap up trendy Apple Watches in droves, and this will be a serious problem for fashion and mass-market watch brands. As long as it’s trendy, an Apple Watch will occupy the wrist space once claimed by the likes of Gucci or Diesel.

This buying shift could cause serious harm to watch brands like Citizen, Seiko, Bulova, Movado, Hamilton, Tissot, and TAG Heuer that rely on mass-market jewelry counter sales for the bulk of their revenue. Even Rolex could feel a pinch if wealthier buyers decide that an Apple Watch is more of a status symbol than a Submariner or a Datejust. Omega, Breitling, and especially Montblanc are similarly exposed if that shift occurs.

The Apple Watch is much less of an issue for serious horology companies. They already rely on a minuscule market of watch connoisseurs, and it’s likely that these buyers won’t switch en-masse. Buyers don’t pick Jaeger-LeCoultre or Zenith for attention, since average people have never heard of the brands. And the big guys (Patek Philippe, Audemars Piguet, Vacheron Constantin and the world of exotic haute horology) have nothing to fear. These companies should be focused on the free-floating Swiss Franc, the low price of oil, and “peak China” rather than the Apple Watch!

Stephen’s Stance

As both a watch and gadget nerd, I am impressed by the Apple Watch but I would not recommend buying one yet. Although it’s a wonderful object and achievement, it’s not actually all that useful as of the first month of availability. This is not a shock; it’s brand new; it will get better. I want better apps and I want better interaction, especially with the button and digital crown. But the fundamentals are in place for the Apple Watch to transcend the smart watch market to become a meaningful device.

© Stephen Foskett for Stephen Foskett, Pack Rat: A Watch Guy’s Review of the Apple Watch