Remembering the MediaSmart

HP released the first MediaSmart server in 2008, and there was much interest at the time both for the compact yet innovative hardware and the embedded Windows Home Server operating system. Inside Home Server was Microsoft’s innovative Drive Extender software, giving these devices Drobo-like flexibility.

The first single-drive MediaSmart EX470 (and identical dual-drive EX475) featured four 3.5″ SATA disk drive bays, a gigabit Ethernet port, a single-core 1.8 GHz 64 bit AMD CPU, and 512 MB of RAM. Although not exactly up to today’s standards, this hardware is decent enough to consider resurrecting.

I wasn’t going to take the little MediaSmart until I saw how well-made it was. The solid chassis is smaller than the three Lord of the Rings books, and the nifty locking hard disk drive carriers are nicer than those found on many servers today. Maybe, I wondered, this little guy could be pressed back into service.

Some research showed that the MediaSmart’s CPU and RAM could be upgraded somewhat: The MediaSmart uses a standard (if outdated) BIOS and supports 2 GB DDR2 DIMMs and a few different Socket AM2 CPU models. The stock CPU is a single-core 90 nm “Manila” Sempron 3400+, but the “Brisbane” 65 nm dual-core chip makes a nice upgrade and stays inside the MediaSmart’s meager power and cooling envelope. I presumed that I could locate these components on eBay (or Weird Stuff)!

Why not just use a Raspberry Pi? Apart from the solid chassis and power supply, even the original-spec MediaSmart soundly beats a Pi at just about everything. In terms of I/O, there’s no competition: The MediaSmart has native full-speed Gigabit Ethernet and multiple SATA ports, while the Pi still routes all this through a single slow USB 2.0 connection. Even the Olimex LIME2, with native Gigabit Ethernet and SATA, can’t manage the I/O performance of the MediaSmart.¹ The original single-core Sempron CPU easily beats the dual-core LIME2 and comes out about even with the 4-core Pi3 overall, though each individual Geekbench test varies widely.²

Installing Ubuntu Blind

The first setback was a feature, not a bug: The MediaSmart has no graphics port. Although the USB ports support keyboard input, it’s awfully hard to debug the MediaSmart without access to the Sempron CPU’s VGA output. There is a hack to add a VGA port, but the pre-built boards are long-since out of stock and I wanted to get started quicker than building a cable.

I decided to install Ubuntu Linux on the MediaSmart blind, following the steps used on a spare PC. Shockingly, this process worked! But the network wasn’t working, so I didn’t know for sure. The MediaSmart gave me no feedback apart from some happy-sounding hard drive noises.

Presuming that the network chip in the MediaSmart wasn’t detected and configured properly, I decided to try a USB LAN adapter I had in my desk. I connected it to the good PC and recorded exactly the proper steps to log in and configure Ubuntu to make it work. Then I blind-typed these commands on the MediaSmart with the USB LAN connected there and it worked!

Once I could log in via USB, I could manually configure the MediaSmart’s native SiS 191 LAN controller in Ubuntu. For posterity, add the following to /etc/network/interfaces:

auto enp0s4
iface enp0s4 inet dhcp

I now had a mostly-functional copy of Ubuntu Server installed on my MediaSmart.

Fixing the SATA Ports

Except that two of the MediaSmart hard drive slots were not working. Looking at the output of lspci, it’s clear that the MediaSmart has two SATA controllers: A Marvell 88SE6121 SATA II controller and an SiS SATA/IDE controller. The Marvell chip controls the eSATA port, while the SiS controller is attached to the riser inside the box for the four drive trays.

By default the MediaSmart uses this internal SiS SATA controller in RAID mode, but this requires a driver which I don’t have. I realized I had to (blind!) switch the controller to independent mode for Linux to be able to use it. Happily, a poster over at OpenMediaVault.org had already figured out the procedure! And it worked!

For posterity:

1. Boot the MediaSmart and as soon as the light flashes red and blue, hit the Delete key to enter BIOS configuration
2. Pres the Down Arrow three times and then press Return to select Integrated Peripherals
3. Press the Down Arrow once and press Return to select SiS On-Chip PCI
4. Press the Down Arrow eight times to select SiS Serial ATA Mode
5. Press PgDn once to select 4P(IDE)+4S(IDE) mode
6. Press F10 to save and Return to confirm

Once I did this, Ubuntu recognized all four integrated ports (in IDE mode) and the drives attached. There is no “four drive SATA” mode with this controller without the RAID driver, but IDE mode works just fine too.

Upgrading the RAM

Because 512 MB isn’t much RAM even for a headless server, I decided to upgrade it. There’s only one DIMM slot on the board, and it recognizes a maximum of 2 GB, so that’s what I went for.

The stock RAM is DDR2-667 (PC2-5300), but better RAM works fine too. I found a used 2 GB DDR2-800 (PC2-6400) DIMM for $15 on eBay and ordered that.

Upgrading the RAM is helpful not just for the added capacity: It appears that the hardware uses the added capability as reflected in the Geekbench result. Memory access is 25% faster.

I also ordered a dual-core AMD Athlon X2 BE-2300 but couldn’t get it to boot. Perhaps there’s something I need to do in BIOS, but without graphics that’s hard to know. For now it’s in the drawer with the original RAM stick.

Stephen’s Stance

The old MediaSmart is a surprisingly capable home or small office NAS once it’s been upgraded with a modern Linux installation. My next step is to configure it for NAS use with ZFS or LVM/Btrfs RAID and Samba. Too bad the MediaSmart community is mostly retired, too. Hopefully this well-built little server can put in a few more years of active duty!

© Stephen Foskett for Stephen Foskett, Pack Rat: Recalling An HP MediaSmart Server To Active Duty

The ZFS Revolution, Circa 2006

In my posts on FreeNAS, I emphatically state that “ZFS is the best filesystem”, but if you follow me on social media, it’s clear that I don’t really love it. I figured this needs some explanation and context, so at the risk of agitating the ZFS fanatics, let’s do it.

When ZFS first appeared in 2005, it was absolutely with the times, but it’s remained stuck there ever since. The ZFS engineers did a lot right when they combined the best features of a volume manager with a “zettabyte-scale” filesystem in Solaris 10:

• ZFS achieves the kind of scalability every modern filesystem should have, with few limits in terms of data or metadata count and volume or file size.
• ZFS includes checksumming of all data and metadata to detect corruption, an absolutely essential feature for long-term large-scale storage.
• When ZFS detects an error, it can automatically reconstruct data from mirrors, parity, or alternate locations.
• Mirroring and multiple-parity “RAID Z” are built in, combining multiple physical media devices seamlessly into a logical volume.
• ZFS includes robust snapshot and mirror capabilities, including the ability to update the data on other volumes incrementally.
• Data can be compressed on the fly and deduplication is supported as well.

When ZFS appeared, it was a revolution compared to older volume managers and filesystems. And Sun open-sourced most of ZFS, allowing it to be ported to other operating systems. The darling of the industry, ZFS quickly appeared on Linux and FreeBSD and Apple even began work to incorporate it as the next-generation filesystem for Mac OS X! The future seemed bright indeed!

Checksums for user data are essential or you will lose data: Why Big Disk Drives Require Data Integrity Checking and The Prime Directive of Storage: Do Not Lose Data

2007 to 2010: ZFS is Derailed

But something terrible happened to ZFS on the way to its coronation: Lawsuits, licensing issues, and FUD.

The skies first darkened in 2007, as NetApp sued Sun, claiming that their WAFL patents were infringed by ZFS. Sun counter-sued later that year, and the legal issues dragged on. Although ZFS definitely did not copy code from NetApp, the copy-on-write approach to snapshots was similar to WAFL, and those of us in the industry grew concerned that the NetApp suit could impact the future availability of open-source ZFS. And this appears to have been concerning enough to Apple that they dropped ZFS support from Mac OS X 10.6 “Snow Leopard” just before it was released.

Here’s a great blog about ZFS and Apple from Adam Leventhal, who worked on it: ZFS: Apple’s New Filesystem That Wasn’t

By then, Sun was hitting hard times and Oracle swooped in to purchase the company. This sowed further doubt about the future of ZFS, since Oracle did not enjoy wide support from open source advocates. And the CDDL license Sun applied to the ZFS code was judged incompatible with the GPLv2 that covers Linux, making it a non-starter for inclusion in the world’s server operating system.

Although OpenSolaris continued after the Oracle acquisition, and FreeBSD embraced ZFS, this was pretty much the extent of its impact outside the enterprise. Sure, NexentaStor and GreenBytes helped push ZFS forward in the enterprise, but Oracle’s lackluster commitment to Sun in the datacenter started having an impact.

What’s Wrong With ZFS Today

OpenZFS remains little-changed from what we had a decade ago.

Many remain skeptical of deduplication, which hogs expensive RAM in the best-case scenario. And I do mean expensive: Pretty much every ZFS FAQ flatly declares that ECC RAM is a must-have and 8 GB is the bare minimum. In my own experience with FreeNAS, 32 GB is a nice amount for an active small ZFS server, and this costs $200-$300 even at today’s prices.

And ZFS never really adapted to today’s world of widely-available flash storage: Although flash can be used to support the ZIL and L2ARC caches, these are of dubious value in a system with sufficient RAM, and ZFS has no true hybrid storage capability. It’s laughable that the ZFS documentation obsesses over a few GB of SLC flash when multi-TB 3D NAND drives are on the market. And no one is talking about NVMe even though it’s everywhere in performance PC’s.

Then there’s the question of flexibility, or lack thereof. Once you build a ZFS volume, it’s pretty much fixed for life. There are only three ways to expand a storage pool:

1. Replace each and every drive in the pool with a larger one (which is great but limiting and expensive)
2. Add a stripe on another set of drives (which can lead to imbalanced performance and redundancy and a whole world of potential stupid stuff)
3. Build a new pool and “zfs send” your datasets to it (which is what I do, even though it’s kind of tricky)

Apart from option 3 above, you can’t shrink a ZFS pool. Worse, you can’t change the data protection type without rebuilding the pool, and this includes adding a second or third parity drive. The FreeNAS faithful spend an inordinate amount of time trying to talk new users out of using RAID-Z1 ¹ and moaning when they choose to use it anyway.

These may sound like little, niggling concerns but they combine to make ZFS feel like something from the dark ages after using Drobo, Synology, or today’s cloud storage systems. With ZFS, it’s “buy some disks and a lot of RAM, build a RAID set, and never touch it again”, which is not exactly in line with how storage is used these days.²

Where Are the Options?

I’ve probably made ZFS sound pretty unappealing right about now. It was revolutionary but now it’s startlingly limiting and out of touch with the present solid-state-dominated storage world. So what are your other choices?

Linux has a few decent volume managers and filesystems, and most folks use a combination of LVM or MD and ext4. Btrfs really got storage nerds excited, appearing to be a ZFS-like combination of volume manager and filesystem with added flexibility, picking up where ReiserFS flopped. And Btrfs might just become “the ZFS of Linux” but development has faltered lately, with a scary data loss bug derailing RAID 5 and 6 last year and not much heard since. Still, I suspect that I’ll be recommending Btrfs for Linux users five years from now, especially with strong potential in containerized systems.³

On the Windows side, Microsoft is busy rolling out their own next-generation filesystem. ReFS uses B+ trees (similar to Btrfs), scales like crazy, and has built-in resilience and data protection features⁴. When combined with Storage Spaces, Microsoft has a viable next-generation storage layer for Windows Server that can even use SSD and 3D-XPoint as a tier or cache.

Then there’s Apple, which reportedly rebooted their next-generation storage layer a few times before coming up with APFS, launched this year in macOS High Sierra. APFS looks a lot like Btrfs and ReFS, though implemented completely differently with more of a client focus. Although lacking in a few areas (user data is not checksummed and compression is not supported), APFS is the filesystem iOS and macOS need. And APFS is the final nail in the coffin for the “ZFS on Mac OS X” crowd.

Each major operating system now has a next-generation filesystem (and volume manager): Linux has Btrfs, Windows has ReFS and Storage Spaces, and macOS has APFS. FreeBSD seems content with ZFS, but that’s a small corner of the datacenter. And every enterprise system has already moved way past what ZFS can do, including enterprise-class offerings based on ZFS from Sun, Nexenta, and iXsystems.

Still, ZFS is way better than legacy storage SOHO filesystems. The lack of integrity checking, redundancy, and error recovery makes NTFS (Windows), HFS+ (macOS), and ext3/4 (Linux) wholly inappropriate for use as a long-term storage platform. And even ReFS and APFS, lacking data integrity checking, aren’t appropriate where data loss cannot be tolerated.

Stephen’s Stance: Use ZFS (For Now)

Sad as it makes me, as of 2017, ZFS is the best filesystem for long-term, large-scale data storage. Although it can be a pain to use (except in FreeBSD, Solaris, and purpose-built appliances), the robust and proven ZFS filesystem is the only trustworthy place for data outside enterprise storage systems. After all, reliably storing data is the only thing a storage system really has to do. All my important data goes on ZFS, from photos to music and movies to office files. It’s going to be a long time before I trust anything other than ZFS!

© Stephen Foskett for Stephen Foskett, Pack Rat: ZFS Is the Best Filesystem (For Now…)

The Central Processing Unit in the Main Frame

It wasn’t that long ago that computers were designed with all sorts of special processors. There was a central processing unit (CPU) in the “main frame” but the heavy lifting was handled by an assortment of supporting processors in other frames. And early PCs typically had lots of special processors, from Steve Wozniak’s IWM in the Apple II to the DSP in Atari’s revolutionary flop, Falcon, to the floating point math co-processors often purchased with 8086, ‘286, and ‘386 PCs.

But then Moore’s Law kicked in at Intel in the 1990s and the CPU began integrating more and more features. Soon, mainstream x86 CPUs boasted superior floating point performance and began adding special-purpose instructions for multimedia, encryption, and more.

Today, the mantra holds that software running on commodity processors will always win out over special-purpose hardware. But this simply isn’t true.

For one thing, Intel can only add so many special instructions to x86 before they reach a point of diminishing returns. The MMX instructions that sold Pentiums in the 1990s have been superseded by new generations of AVX instructions today but they’re still there, taking up valuable silicon. And although parts of the industry would love to see SHA-256 and AVX-512, Intel seems unsure if these are worth implementing in mainstream processors.

Revenge of the Co-Processors

Although computing today appears to be centralized on x86, this isn’t really true. From supercomputers to machine learning to game machines, non-CPUs are increasingly carrying the compute load. And this shift is even happening in mobile devices and PCs, especially over at Apple.

The counter-trend started with supercomputers and game machines back in the 2000s.

Sony, Toshiba, and IBM developed a heterogeneous CPU, the Cell Broadband Engine, which coupled a PowerPC core and multiple floating point coprocessor units. Cell became well-known when it was selected as the processor for the Playstation 3, but it was developed for use in high-performance computing (HPC) and in supercomputers. Sadly, some of the other architectural choices in the Cell processor limited its impact outside games.

Intel had similar projects in the works at the time, Larrabee (with many-threaded cores that could be used for graphics or specialized computing) and the Single-chip Cloud Computer (with 48 cores connected with a mesh network). Larrabee and Cell shared some interesting aspects: They could be used for graphics or for general-purpose math, and they could be integrated with a general-purpose processor or used in add-in cards.

As clusters of chips supporting AMD’s 64-bit x86 extensions began to dominate supercomputing, Intel embarked on an effort to develop a truly massive multi-CPU cluster. They evolved those research projects into Knights Ferry, a 32-core PCIe add-in card for scientific computing. The next generation of Intel’s “manycore” processor development gained a formal name: Xeon Phi.

Today, Xeon Phi is the key component of the fastest supercomputers on earth, with multiple petaFLOPS projects deployed, and it has lead to x86 becoming by far the dominant supercomputer platform. But these aren’t just Xeon CPUs, despite the name. For Knights Corner, Intel went back to the history of x86 and dusted off the P64C core from the original Pentium, while today’s Knights Landing uses Airmont Atom cores. And Knights Landing is the first bootable Xeon Phi, though most designs incorporate conventional Xeon CPUs as well.

The Rise of GPGPU

Nvidia took inspiration from the world of supercomputers to create a “general-purpose computing” API, CUDA. The rest of the industry followed with a standard GPGPU API, OpenCL, that allowed mainstream computers to leverage the computing power of the shaders and vector processing units inside GPUs from Nvidia, AMD, Intel, PowerVR, and more to offload simple computing tasks in parallel.

The massive computing power of higher-end GPU cards cannot be denied: With thousands of compute units available, no CPU can keep up. From machine learning to cryptocurrency mining, GPUs aren’t just the best choice, they’re the only thing that makes sense.

Although the shaders and FPUs in them have been opened up somewhat, GPUs still aren’t designed for general-purpose computing. For example, only Nvidia has implemented IEEE-compliant floating point math and double-precision numbers. So the next step is GPGPU hardware designed with more than graphics in mind. Intel is tackling this with Knights Mill, their next Xeon Phi, which is specifically designed for machine learning, and it is widely believed that Nvidia is working on specific ML processors as well.

Stephen’s Stance

It may appear that today’s computers are centered on an all-powerful x86 or ARM CPU, but this is rapidly changing. The next designs are likely to include massively-powerful co-processors, from GPUs to special-purpose chips, that radically outperform the CPU through simplification and parallelization.

Although GPGPU is limited to a few applications in Windows, Apple has made widespread use of OpenCL in both macOS and iOS. But it’s the world of iDevices where Apple’s control of both hardware and software really shines: Every device running current revisions of iOS has a base level of GP-GPU power. And this leads me to a rather interesting conclusion, which I will discuss in my next article.

© Stephen Foskett for Stephen Foskett, Pack Rat: Co-Processors, GPGPU, and Heterogeneous Computing

Zero to Hero: Thunderbolt

I was an early enthusiast for Thunderbolt (originally called Light Peak), since it opened up a whole new world of I/O performance. And, being a storage nerd, I was particularly keen on the combination of fast PCIe connectivity and flash storage peripherals. Although the original Thunderbolt fell short of expectations (being mainly Mac and copper, not a universal PCIe-over-fiber dream), later revisions have matured nicely.

Today’s Thunderbolt 3 boasts 40 Gbps of PCIe 3.0 bandwidth along with up to 100 Watts of power delivery and support for USB, DisplayPort, and HDMI protocols besides. Although interoperability is more of a nightmare than a dream, Thunderbolt 3 works and is supported on many platforms and peripherals. With Intel and Apple pushing solidly to promote the interface, and leveraging popularity of USB-C, Thunderbolt 3 looks to become a real multi-platform standard.

We are seeing generic motherboards shipping with Thunderbolt 3 onboard, not to mention high-end PC’s and just about every Apple Macintosh computer. It wouldn’t be surprising to start seeing Thunderbolt 3 showing up in server hardware as an interconnect for external PCIe chassis full of NVMe storage or co-processors for machine learning. And Intel is talking about moving Thunderbolt support inside the CPU for the next generation.

So What’s OCuLink?

But this wasn’t always a given. And apparently the PCI-SIG (standards body for the PCI interface) thought so too. So in 2012, word started spreading that PCI-SIG was developing a standard cabled protocol for PCIe devices off the motherboard. And this standard would be free and unencumbered by corporate overlords, Apple and Intel.

Thunderbolt had already reached the market by then, but it used Mini DisplayPort connectors and only supported PCIe 2.0, for aggregate throughput of 20 Gbps. In contrast, OCuLink would start with four lanes of PCIe 3.0, good for 32 Gbps of throughput.

Like Thunderbolt, the initial plan was for both optical and copper cables to be used. But both standards hit roadblocks as optical cables failed to materialize in volume. Although there are optical Thunderbolt cables, it’s a far cry from the silicon photonics wonderland promised by Light Peak. And although the “O” in OCuLink stands for “optical”, every implementation I could find uses the “Cu” for copper!

Early reports gushed about OCuLink for laptops, allowing external GPU’s to be connected to thin-and-light PC’s. But this has been a rare application even in the Thunderbolt space (Apple just introduced official support in 2017), and I couldn’t find any OCuLink laptops or GPU enclosures on the market.

Instead, OCuLink was picked up by server developers looking for an in-box PCIe interconnect for storage or I/O virtualization connectivity. And the standard 48-pin connector and cable was sometimes re-purposed to carry multiple 12 Gbps SAS 3.0 channels as an alternative to HDminiSAS, replacing the old 4-channel SFF-8087 mini-SAS connectors we in storage have learned to love.

SFF-8639, SATA Express, and U.2

OCuLink has a kinship to another PCIe interface that might just be more popular. At the same time that the OCuLink cable was introduced, the PCI-SIG also pushed for a drive-attachment interface with PCIe using a connector called SFF-8639. This was initially used for SATA Express drives, a 2-lane PCIe storage interface.

However, in 2015 SFF-8639 was officially renamed “U.2″ for four-lane PCIe storage applications. This has become somewhat more popular. So, in a way, U.2 is a cousin of OCuLink and some devices might even use OCuLink protocol over the U.2 connector!

U.2 drives look like conventional 2.5” SSD’s. So they might take off in servers and datacenter applications. But lately, most PCIe storage implementations are leaning towards the compact M.2 interface instead. And on the pro-sumer side, it’s almost as difficult to find a U.2 motherboard or drive as it was to find SATA Express!

OCuLink-2

In 2016, PCI-SIG announced OCuLink-2, bringing PCIe 4.0 bandwidth and a new connector. Ironically, the OCuLink-2 external connector and cable bears a strong resemblance to full-size DisplayPort, bringing us back to the days of Thunderbolt’s Mini-DisplayPort cable.

So far, there has been little discussion of OCuLink-2, but perhaps that’s changing. The aforementioned Tyan server chassis actually appears to use OCuLink-2, supporting either PCIe 3.0 x8 or 16 6 Gbps SAS/SATA drives.

Stephen’s Stance

With the advent of AMD Threadripper and Epyc, we are about to see an explosion of PCIe lanes in the pro-sumer and datacenter market. Although many of those lanes will be taken up by conventional PCIe cards, some will be used for SSD’s (M.2 and U.2) or for external connectivity. This is where OCuLink might finally take off: As an AMD alternative to Thunderbolt for external PCIe peripheral connectivity.

© Stephen Foskett for Stephen Foskett, Pack Rat: What is OCuLink?

Investors Are Happy with Magic Beans?

I have been skeptical about the Dell and EMC merger since the announcement. The price seemed too low, the included “tracking stock” too flimsy, and the questions about VMware, RSA, and other Dell and EMC businesses too great. The purported value of the deal ($33.15 per share) was based on a cash price of $24.05 per share of EMC plus 1/9 of a share in a tracking stock to value the company’s ownership of VMware. As discussed at the time, this offer dramatically under-values not just EMC itself but also its ownership stake in VMware, Pivotal, and the new SecureWorks company.

Essentially, Dell bought all the $28 EMC shares for $24 plus some magic beans.

But it seems that Wall Street was willing to go along with the deal to rid itself of EMC. There has long been a cloud hanging over the EMC’s enterprise value due to a perception that the traditional enterprise storage world is on the verge of massive disruption. Analysts and investors apparently do not believe that EMC can adapt to the new world of IT and maintain earnings and profitability. Investors appear to be willing to accept this compromise, letting EMC go to Dell to get it off their hands.

I’m betting the VMware tracking stock ($DVMT) will look pretty good for a while, rising to between $45 and $60 per share before collapsing for good. Why $45? That’s the difference between EMC’s share price the day before the merger ($29.05) and Dell’s cash ($24.05) times 9 (remember that each shareholder gets 1/9 of a share of $DVMT). Owners of $EMC shares will sell out somewhere in there and that will be that.

In the mean time, I also expect lots of “pump the price” stories to be published, suggesting that $DVMT is a good way to get shares of VMware on the cheap. But tracking shares have no intrinsic value and no voting power. Unless Dell adds a dividend to $DVMT, it’s likely to follow the trajectory of similar tracking stock offerings (and magic beans): Right down the drain.

The Future of Dell EMC is Brighter

But the Dell/EMC deal is fundamentally all about brushing aside Wall Street’s demands and focusing on corporate success. And that aspect of the deal is a whole lot more positive. Dell has handled the integration as well as could be hoped, and it is likely that the newly-formed Dell Infrastructure Solutions business (selling as Dell EMC and managed out of Hopkinton) has a strong few years ahead of it.

Although Dell has traditionally been a weak spender when it comes to R&D, EMC is awash in solid technologies. Just about every product line is competitive in its market segment and some (DSSD, ScaleIO) are trend-setters. Dell EMC ought to be able to make hay with these products, selling to traditional EMC buyers as well as Dell’s massive customer base. And now they have the Dell Enterprise Solutions products to push as well.

Certainly Dell EMC has too many product lines now, but the correction won’t hurt customers. One can only hope that management will have the wisdom to cut products based on long-term viability rather than short-term sales. EMC has been content to allow competitive product lines to continue until a clear winner emerges. Perhaps Dell will continue this practice, especially now that Wall Street isn’t watching as closely.

Personally, I remain concerned for my many friends inside this new company. Dell would be wise to be graceful and kind to employees, allowing them to move to other areas instead of letting them go. But many have already given notice, casting a cloud on the future of the company. Dell needs to retain these talented people.

Dell also needs more clearly to explain what happens with the other parts of the combined business. Security has a home, but what about Dell’s networking assets? I’m not really worried about VMware under this new management regime, but what was the deal with that SecureWorks transaction? There are so many questions and still not many answers.

Impact Beyond Dell

The biggest question is the impact of this combination on the rest of the industry. HP clearly doesn’t agree with the Dell “embiggening” strategy: They split into HP InkInc and HPE and seem to be doing well so far. But Cisco has yet to react.

Then there’s the changed acquisition landscape for all the startups in the industry. Dell and EMC aren’t going to be bidding on startups any time soon, and HPE and Oracle seem fairly sated right now. This leaves Cisco as the only hungry company at the buffet. I expect some feasting to be done, building a new full-line competitor for Dell and HPE!

We must also consider the new world of IT, with a resurgent Microsoft joining Amazon, Google, and the cloud providers in attacking the IT industry. This is the reason investors were willing to take Dell’s magic beans in the first place, and it’s the real deal. Many are saying that sales of enterprise IT products have peaked, leaving companies like Dell to fight it out in a narrowing ring.

But this transaction is also a vote of “no confidence” in Wall Street. Dell went private and has, by all accounts, done very well since. Many other companies are following suit. Clearly, corporate management is fed up with the quarterly mindset so prevalent among investors and analysts. Will we see fewer IPO’s, as management realizes that being a public company isn’t a positive for a growing company? Watch for Nutanix and SimpliVity IPO’s as bellweathers.

Stephen’s Stance

EMC’s management clearly knew that it was time to make a move, and were willing to take a little less for the company than folks like me thought. Dell management clearly thought they could leverage EMC to solidify their position at the top of the IT market, and I bet they’ll do a good job with it at least for a while. Investors clearly are happy to take what they got (some cash and some magic beans), and who am I to question their logic? As for me, I’m just sad to see one more pillar of the enterprise storage industry wiped away.

© Stephen Foskett for Stephen Foskett, Pack Rat: Dell, Wall Street, Magic Beans, and the End of EMC

Let’s start with “free”. As “the Internet” is quick to point out, this critical word has multiple meanings, and corporate interests have a tendency to divert everything toward profit. OpenStack is a free and open-source software platform for cloud computing, but it’s also a community of innovators ranging from enthusiasts to lone hackers to the world’s largest corporations.

OpenStack is free, but like all open source projects, someone has to pick up the tab. Who’s paying for development and supporting users? Certainly many individual contributors are involved, but the vast majority of code commits and new project components are coming from vendors trying to leverage OpenStack to sell their products and services.

OpenStack may be “free as in freedom” and “free as in coffee”, but it sure as heck costs a lot to deploy and productize. This is why so many companies came to Austin to be part of the Summit. Some, like Mirantis, see a tremendous opportunity to profit from growing OpenStack deployments. Others want to plug their existing products into these massive installs. Yet more seem to be hedging their bets or even just happy to be part of “the new thing”.

One popular refrain in the vendor booths at the Summit is that “no one but Mirantis is making any money here at this point”. This was refuted by a few companies who have managed to land big service provider customers for hardware and software, but the overall sense is that there’s a lot of free coffee being given out without a lot of sales coming in return. “Elephant hunting” was a familiar phrase.

As with any large open source project, OpenStack is developed mainly by large corporations. Companies like IBM, HP, Intel, and NetApp see it as an entrance to the new cloud/service provider market. Red Hat, Mirantis, and Canonical want to sell services and support. Rackspace needs a functioning cloud. All of these companies are pulling the project in different directions, and all are doing the right thing as far as they are concerned.

The risk, however, is that each of these agendas might disrupt or derail the whole project. Ultimately, customers will decide which to adopt, but this isn’t as democratic as it seems. Their decisions are affected by development resources and the availability of quality integration components more than the inherent technical appropriateness of a protocol or product. This is why NetApp is smart to be pouring resources into storage integration projects like Cinder: They enable storage integration generally while reassuring customers that NetApp products will work really well!

The happy outcome isn’t necessarily a cynical product push. Rather, by contributing to the code base, all these companies improve the entire product. But what about the companies that aren’t contributing much base code but are instead focused only on integrating their own products? Although not helping build OpenStack, these companies still contribute to the feeling of longevity for the project as a whole.

What will OpenStack be outside the enterprise? Although most assume it’s just for building “your own Amazon”, customers have been clamoring for an overarching data center management framework. OpenStack might just become that, too. This is certainly how Platform9 sees OpenStack: They can leverage the common OpenStack management framework across heterogeneous enterprise data centers, not just for the cloud. VMware too is quick to point out that OpenStack isn’t tied to a single hypervisor and works just as well with ESXi inside.

OpenStack might prove to be the ultimate unifier not just of cloud but of conventional IT systems as well. One could envision a future where OpenStack is both the unified management layer for the entire data center but also the framework on which next-generation applications rest. That’s the optimistic projection.

The pessimist might spot more clouds than blue skies. OpenStack could become just what it is today: The cloud platform for service providers not named Amazon. This is an acceptable outcome, but not exactly a transformative one for the enterprise data center.

Disclaimer: Tech Field Day came to OpenStack Summit, and NetApp was a sponsor. We also interviewed a dozen non-sponsors as part of the Tech Field Day Forum. See our Day 1 and Day 2 interviews. About half of the other companies mentioned here have sponsored Tech Field Day events in the past.

© Stephen Foskett for Stephen Foskett, Pack Rat: Free as in Coffee – Thoughts on the State of OpenStack

The Promise of NVDIMM

We are in the midst of a radical change to computing: For the first time since the PC appeared in the 1980’s, a fundamentally new architecture is coming to the datacenter. Flash promises to be a transformative technology, and the performance, power, and physical benefits are already being felt. Can you imagine an iPad with a hard disk drive?

But flash could be so much more if it wasn’t treated like a hard disk drive. Today’s SSD’s wrap all this promise in a familiar package, smothering the performance behind a translation layer so it will look like a plain old hard disk. To truly feel the impact of flash, we need to treat it differently.

NVMe is a promising concept, since it allows flash memory to be directly accessed as memory, not a “fake disk”. More exciting is NVDIMM, which places flash on the memory channel rather than the PCIe bus. Like NVMe, NVDIMM exposes the unique nature of flash chips, but it also enables orders of magnitude quicker access.

You should also read “Why Are PCIe SSDs So Fast?“, “Diablo Memory 1 Takes Memory Channel Flash To The Next Level“, “Flash Memory in the DRAM Slots? Diablo Technologies is Working On It…” and “The Rack Endgame: A New Storage Architecture For the Data Center“

I’ve written and spoken a lot about this dramatic change, but the simple summary is that this new system architecture might be even more important than the improvement in performance. “Tier 2 memory”, as I like to call it, means that applications have a whole new layer of non-volatile storage to access, and it’s closer in performance to RAM than traditional storage. Companies like Diablo Technologies (Memory 1) and Intel (3D XPoint) are making this a reality.

The Problem of NVDIMM

So flash and NVDIMM or NVMe gives us a new system architecture. Awesome!

But do we need new system software to make use of this?

Until now, the answer was “pretty much yes”. We needed either a specialized application, driver, or new operating system. That’s not easy to achieve (just ask Fusion-io) but it’s necessary.

This has been the fundamental challenge for NVDIMM, NVMe, and flash generally. It’s easy to pretend that flash is a hard disk drive (i.e., an SSD), but it’s much harder to take advantage of the native performance of flash. Yet without a compelling use case, why invest in anything better than SSD?

This is the chicken. Or maybe the egg. Who can be sure?

The Plexistor Solution

Plexistor has a novel approach to this issue. Rather than trying to modify applications, why not just integrate the NVDIMM access into a new POSIX filesystem that any application can use? Suddenly there’s no need to customize an application, install a driver, or create a specialized system. Anything that can use a filesystem can leverage the performance of NVDIMM.

It remains to be seen just how well this “NVM file system” works in practice, but I love the concept: A simple abstraction that allows existing applications to use this new storage/memory technology. It’s much more straightforward than limiting NVDIMM usage to specific, re-written applications.

Will customers buy a filesystem layer from a small, unknown company? Perhaps. PernixData has seen wonderful success growing their (conceptually similar) market, Virsto became VMware’s VSAN 2.0, and companies like Maxta are selling storage software today. But not every competitor succeeded, and the history of storage is littered with failed storage virtualization layers.

Another risk is the possibility of similar functionality being integrated into the operating system. Microsoft could develop and roll out their own NVDIMM integration, pulling the rug out of the Windows market. And the Linux community, perhaps with the help of storage vendors, could do the same thing. Maybe the most likely outcome is Plexistor being purchased by an NVM vendor (Intel, Diablo, WD/SanDisk, etc) to push hardware sales.

Stephen’s Stance

Plexistor excites me with a straightforward solution to a pesky problem. A generic filesystem that makes NVDIMM and NVMe available to any application is a great idea. Whether as a standalone company or an enabler, I expect this software to have legs.

Disclaimer: Plexistor will present at Storage Field Day 9 and Tech Field Day 11, which I organize. This post is my own work and was not tied to that at all. In fact, I began writing this before they signed up and I suspect it was my enthusiasm that brought them in to Tech Field Day, not the other way around!

© Stephen Foskett for Stephen Foskett, Pack Rat: Plexistor Integrates NVDIMM for Every Application

A few years back, I wrote an immensely-popular series of blog posts outlining the four things that were holding storage system performance back, and the ways to fix them. At the time, I created some presentation content to go along with these posts, and even considered pulling them into a white paper, but nothing came of that. Now, however, I’m pleased to announce that my Four Horsemen are accompanying me to the stage November 10, 2015 at the DeltaWare Data Solutions Emerging Technology Summit in Edina, Minnesota.

Here’s the original “Four Horsemen” series:

1. The Rule of Spindles
2. Never Enough Cache
3. I/O As a Chain of Bottlenecks
4. Get Smart

When I wrote those four pieces, back in 2010, the industry was on the verge of a major shift: Hard disk drives had always been the critical inhibitor of storage system performance (see “The Rule of Spindles“), but widespread use of flash memory meant this was about to change. Many people don’t realize that RAID was originally designed primarily to deliver increased IOPS, not to protect data in the event of a drive failure. Yet even today, hard disk drives remain a critical gating factor to storage system performance. The advent of sequentialization, hybrid flash cache, and smarter storage systems has mitigated the problem, but most data still lands on a spinning disk eventually. Which means that we’re still stuck optimizing them.

NAND flash has also dented the second “horseman” (see “Never Enough Cache“), since it’s much cheaper on a per-capacity basis than DRAM. This is why virtually every storage system today has a large (by RAM standards) pool of NAND flash to serve as a temporary landing zone. And to stretch that metaphor, most systems also use NAND flash as a “pre-departure waiting area”, storing data that is likely to be read soon. And now that storage I/O has become quicker (see below), we’re starting to see optimization above NAND, with SLC caching MLC and DRAM above both.

Regardless of the performance of the storage capacity layer, however, there will always be bottlenecks in the I/O channel (see “I/O As a Chain of Bottlenecks“). Faster storage (thanks largely to flash) has disrupted the industry, with the “big SAN” model giving way to top-of-rack flash, converged infrastructure, and even re-internalized storage. Next-generation solid state storage is so fast, in fact, that it requires some extreme re-engineering to take advantage of all these IOPS. Otherwise we’re left harnessing a racehorse to a wagon.

The most exciting area of enterprise storage today is the emergence of intelligence and integration (see “Get Smart“). No matter what a storage device can do, we can’t maximize return on investment without optimizing data access. This is why today’s trend toward data-aware storage, integrated caches, server/storage integration protocols, and software-defined storage is exciting. If the array’s capabilities (and limitations) are exposed to an intelligent data management layer, we can extract maximum performance from the whole system. Even those pesky hard disk drives.

If you’re in Minnesota in two weeks, come on down to the DeltaWare Data Solutions Emerging Technology Summit! I’ll also try to film my presentation and post it here. And if you’re organizing a conference yourself, might I suggest that you reach out to bring me in as a speaker?

© Stephen Foskett for Stephen Foskett, Pack Rat: The Four Horsemen of Storage System Performance Are Coming To Life!

For people of a certain age, nothing beats a green screen and a blinking block cursor. I can almost taste the Jolt Cola and Cool Ranch Doritos! Back at my alma mater, those terminals were so important that the sound they made became synonymous with hacking (in the true sense): Gweep!

Even with today’s “retina” displays, with all their astonishing color depth and TFT technology, gweeps like me spend much of our time gweeping in a terminal window. And there’s nothing that Terminal needs more than a sweet monospaced font!

Therefore, for your consideration, I present the ultimate Terminal font: Glass TTY VT220! That guy, bless him, transposed the original DEC fonts into a TrueType font file, complete with Cyrillic characters that were missing in the ROMs. Seriously – this is as accurate as you can get!

Back in the low-resolution dark ages (2010), I lamented that the font had to be used at an insane 20 points to be pixel-perfect. But today’s Retina MacBooks make a 20 point font downright desirable!

I did a little tweaking of the Terminal setup to get it how I liked it, but it’s pretty straightforward to have the ultimate Terminal font:

1. Download the TrueType font: Glass TTY VT220
2. Open it in Finder to launch FontBook and install it
3. Open Terminal preferences and create a new default profile with this font at 20 points and a proper green/black color scheme
4. Settle back with a Moxie or Zima and get on with the gweeping!

© Stephen Foskett for Stephen Foskett, Pack Rat: The Best Mac OS X Terminal Font: Glass TTY VT220

Before you read this, check out Today’s Storage: Same As It Ever Was for a bit of background…

For the past few decades, some of the best minds in computer science have been drawn to the question of storage. How can data be better protected, shared, and exploited? Given the price and performance profiles of various storage devices, how can we maximize benefit for a given application? What can we do to transcend inertia and make data available everywhere at once?

These fundamental questions have been tackled by a continuing stream of companies promoting waves of new technology. The limitations of block storage gave us CAS, object storage, and the cloud. RAID was invented because hard disk drives can not physically drive sufficient I/O operations, but wide striping, data dispersal, tiering, and caching are table stakes today. Data compression was pioneered as a transmission technology but has moved to array controllers and static data sets.

In each area, a predictable pattern emerged:

1. New algorithms and architectures are developed
2. Start-up companies are funded and launched, creating standalone products centered on them
3. These technologies become features as they are leveraged broadly across the industry, with the start-ups acquired or withering
4. Mainstream adoption is predictably slow and insider attention turns to the next wave

This last bit is pretty remarkable, and something I’ll be talking about in a follow-on post, but let’s focus on the early phases for now.

How many product, and even company, categories have become merely an obscure feature? I’ve watched this pattern repeat with SRM, object storage, data deduplication, thin provisioning, iSCSI, hybrid storage, and all-flash architectures (just off the top of my head). And now we’ve got RAM and flash caching, storage virtualization, and data-aware storage.

It is interesting too that each wave tends to build on those before it. Deduplication was once a marquee product but now is mainly a way to optimize the use of flash for hybrid or all-flash storage systems. SRM was the “next big thing” but is now an integrated feature of data-aware storage systems. Scale-out and virtual storage is easiest to do with an object store under the covers. And so on.

Each technology wave is accompanied by the same archetypal companies, too:

• Academics and technologists who can’t articulate a use case but usually have the best technology
• Re-packagers of outdated products trying to stay relevant by claiming they’re part of the current wave
• Marketing-first companies making noise with little or no IP of their own but capable of making lots of noise
• Super-funded leaders who hire and expand rapidly, are flush with cash, and become the standard-bearers for the wave
• Actual companies selling useful products (too few of these!) who have solid products and rely on real sales and use cases

It is rare that any of these companies lasts more than a few years. But their products tend to live on, occasionally in standalone companies (usually the last two) but more typically as a product line or feature inside one of the giant incumbents.

And so it goes. Waves of innovation and waves of companies, crash on the storage market, but the same incumbent leaders and product lines survive for decades. It’s actually pretty depressing. Are things changing? It’s hard to see sometimes, but real progress has been made. And perhaps all those waves have finally chipped away at the outdated storage paradigms I complained about last time to send it all toppling down.

© Stephen Foskett for Stephen Foskett, Pack Rat: Waves of Storage Innovation