Usage is Fluid

Too often, when I’ve pointed out that a marketer was using an industry term inappropriately, the response has been dismissive:

• “Who’s to say what that word really means?”
• “This is how the rest of the industry is using that term.”
• “Are you the word police?”
• “It doesn’t really matter what we call it.”

None of these responses is truly satisfying, but all get to a deeper truth: Words generally are defined by how they are used, not by the intentions of the originator. You may not like it if people say “premise” when they mean “premises” but that’s how English usage evolves. We can fight it, but eventually we might just have to accept that usage has changed.

The same is true of English usage overall: Spelling, pronunciation, and grammar is much more flexible than many would like to admit, and it is often wiser to accept “creative” usage than try to correct the speaker.

Words Have Meaning

This is all well and good for words in general, but the situation is different for marketers. Their job is to influence behavior, and specifically to drive purchasing. Non-standard usage or inappropriate identification of a product can have serious, often negative, consequences!

Let’s consider “cloud”, the lovely fluffy buzzword of the last decade. When “cloud mania” started, it seemed that every company wanted to jump on the bandwagon. Most simply applied “the c-word” to whatever they were selling and hoped it would convince buyers to keep buying. But that didn’t work. Most buyers weren’t actually interested in building a cloud, and they certainly weren’t fooled into trying to build one using the same old products they had been buying previously. “Cloudwashing” was a failure of epic proportions.

Now we are in the “software-defined” world, and the same thing is happening all over again. Marketers are eager to make their old products appear relevant and everything powered by software is being called “software-defined-whatever”. But buyers won’t be fooled into buying something they don’t want, especially in the IT space. They’ll see through the ruse and be left with a bad feeling about companies who do this.

Yes, some people will wonder if some old product is “cloud” or “software-defined”, but these aren’t credible customers! Unlike supermarket shoppers (who may prefer a “free-range” or “non-GMO” product without much thought), datacenter architects are going to investigate the product and determine if it lives up to the marketing claims being made. The only ones falling for this wouldn’t buy in the first place!

Even innocent malapropisms and flubbed usage can hurt marketers. Many audiences won’t audibly groan when they hear “on-premise”, but the speaker is still losing credibility. And credibility ought to be the prime goal of marketers!

Stephen’s Stance

Too many marketers and salespeople play fast and loose with words, but they’re only hurting themselves. Improper usage is embarrassing and causes a loss of credibility with the people they most want to reach. It would be wise to spend a lot more time being correct and a little less time jumping on bandwagons and buzzwords!

© Stephen Foskett for Stephen Foskett, Pack Rat: Marketers: Fudging the Meaning of Buzzwords Matters (To You!)

Memory Channel Storage and Memory Channel Memory

Diablo really wowed the storage world by bringing flash storage in a DRAM form factor to the memory channel. There, they delivered amazing performance: An order of magnitude lower latency than any other solution. And this was maximized by reducing the variability of latency seen on the PCI bus, let alone external storage solutions.

Memory Channel Storage, as Diablo calls this solution, is great for many applications but it has some limitations. For one, it requires special tweaks to the motherboard so the BIOS won’t try (and fail) to use this block flash capacity as regular memory. For another, it’s still block storage, complete with a LUN and (probably) a file system in the operating system.

Given the fact that Diablo had produced a DRAM form factor for NAND, the next logical step was making this flash appear as regular system memory, not storage. That’s Memory 1. In a way, it’s “Memory Channel Memory” though I’m glad no one is calling it that!

Diablo Memory 1

The physical product looks similar, and leverages the same technology, but the Memory 1 and MCS products are quite different in detail.

One major change is that Memory 1 no longer needs any special hardware or BIOS tweaks. The BIOS sees a few very large DIMMs and booths normally. Then a proprietary software driver starts, acting as an arbiter between the traditional RAM and Memory 1 flash. Diablo says most application execution happens in RAM, while in-memory caches are written to Memory 1. This makes it ideal for in-memory databases, caching solutions, and many web platforms but perhaps not for large, active applications. And the software will only work with Linux, Windows, and VMware initially.

Note that Memory 1 does not replace RAM in a system. You still need to buy some traditional RAM DIMMs. But you can buy a lot less (think 10:1 flash:RAM) and get a lot more.

The biggest change is simply what the product is. Memory 1 is memory; MCS is storage. Memory 1 looks like memory, acts like memory, and is accessed like memory. And it solves memory-related problems, like the fact that the biggest RAM DIMMs are much, much more expensive on a per-GB basis than smaller ones. Diablo isn’t talking pricing, but the graph above suggests that not only will Memory 1 be cheaper than the cheapest RAM, it’ll remain cheap even at large capacities.

And that’s really the killer app for this product: Too many purchase decisions are a delicate balance between RAM requirements and budgets. With Memory 1, buyers can simply buy as much RAM as they want (within reason) and not break the bank. No more buying too-big servers just for the DIMM slots or buying extra blades because you can’t get enough memory in one. And this should save tons of money when it comes to server acquisition as well as power.

Stephen’s Stance

Memory 1 is the next game-changer from Diablo. I’ve been very impressed by the company’s offerings in the past, and this is the logical next step for them. And it ought to be absolutely killer since it no longer requires special motherboard tweaks. I expect it’s going to be huge in the cloud datacenter.

Disclaimer: Diablo is a frequent Tech Field Day presenter and my company, Foskett Services, has done editorial work for them in the past, including a whitepaper written by yours truly. 

© Stephen Foskett for Stephen Foskett, Pack Rat: Diablo Memory 1 Takes Memory Channel Flash To The Next Level

A Bit About the Intel NUC Hardware

Intel’s NUC (“Next Unit of Computing”) hardware is a mini PC roughly the size of a current Mac Mini. It is designed to support a variety of end user computing applications and functions as a tiny desktop PC for most people. But as many have noticed, it can also support VMware vSphere and similar server workloads! That’s how I’m using it.

I love that the NUC is stable, low-power, compact, and well-supported. I connected three NUCs and a TP-Link 24 port managed Ethernet switch to my CyberPower UPS and the whole setup is drawing less than 60 watts! Plus, it’s quiet and cool and fits in a corner of my basement wiring closet. Score! Although the NUC is limited in RAM and peripheral expansion, these factors make them more attractive for home use than “real” PC’s, even compact ones.

The NUC has reasonably good laptop CPUs, including the latest i3, i5, and i7 models in dual-core variants. It includes two SODIMM slots (1.35 volt), currently supporting a maximum of 16 GB of RAM. Most NUC models are available with space for an internal laptop SATA hard disk drive, and all support some kind of internal SSD as well.

That SSD port is where things get interesting. The first four generations of NUC motherboards sported two Mini PCIe slots: One full length slot on top for mSATA SSDs and another half-length slot below, ostensibly for a Wi-Fi card. Instead of Wi-Fi, I added a wired Gigabit Ethernet NIC in that slot to give my NUCs two Ethernet ports rather than the standard single one.

mSATA, Mini PCI2, and M.2

Let me digress for a moment on Mini PCIe, mSATA, and a new standard called M.2. As I’ve covered before, mSATA is simply a SATA SSD packaged using the same form factor and pinout as a Mini PCIe card. Many modern laptops and small form factor PC’s (and even some full-size PC’s) come with one or more Mini PCIe slots on the board for adding Wi-Fi cards or mSATA SSDs and these can often be used to add additional PCIe Ethernet cards instead. Such is the case of the first four generations of Intel NUC.

Lately, however, the PC industry has adopted a new, even smaller form factor called M.2. On paper, M.2 is way better than Mini PCIe: M.2 has four PCIe 3.0 lanes plus 6 Gbps SATA and USB 3 all packed into that tiny connector! The old standard had a single PCIe 2.0 lane and USB 2.0, plus SMBUS and GSM SIM and such. Plus, the M.2 connector is substantially smaller, taking up less space in today’s super-thin laptops.

Unfortunately for hobbyists, M.2 is brand new and only SSDs and Wi-Fi NICs are available at this point. This means that buyers of fifth-generation Intel NUCs, which have two M.2 slots instead of two Mini PCIe slots, are limited in what they can add internally. Your SSD will be much faster, but you can’t add a wired Ethernet port at this point. I have seen M.2-to-Mini PCIe adapters on the Internets but I doubt these would fit in a NUC.

Adding a Mini PCIe Ethernet Adapter to a NUC

Now back to our discussion! It’s fairly straightforward to add a second Gigabit Ethernet port to your first- through fourth-generation Intel NUC. Just buy a half-length Mini PCIe Ethernet adapter like this Syba Realtek device on Amazon for just $16 and slap it in there. It works perfectly and there are drivers available for most operating systems, including VMware ESX!

The issue is that the Syba card is too tall to also install a mSATA SSD in the NUC and the bulky triple cable it uses doesn’t fit nicely along with a SATA HDD in the little NUC body.

The first problem is solved through the use of solder. I de-soldered the 10-pin Ethernet and 4-pin LED headers on the tiny NIC with my handy de-soldering iron and then pulled the pins out with pliers. I then soldered the Syba cable directly to the board, cutting off the connectors and tinning each little lead. I was careful to solder the wire low and slanted towards the back to allow the cable space in the cramped NUC and cut off the excess afterwards. And I used some plastic to keep everything from shorting out inside.

This gave just enough room to re-install my mSATA SSDs (I picked Samsung EVO because Microcenter had them on sale). But there is still the issue of routing the surprisingly bulky wires out of the case and locating the Ethernet connector. Nick solved this by soldering his own Ethernet pigtail directly to the Syba card, but I wanted to retain the RJ45 connector and activity lights.

So I went to Tinkercad and created a new base for the NUC, complete with a spot for the hard disk and controller and the Syba Ethernet connector. I then printed this out on my AIO Robotics Zeus 3D printer and, after some tinkering, got the whole thing to work! I’ve placed these on Thingiverse if you’d like to print one, too, but be warned that although it works, it’s still not perfect. And it’s a large, time-consuming print that might not fit on a Makerbot platter!

The hard disk drive and controller must be removed from the steel carrier that comes with the NUC (six screws and you’re done) and the Ethernet wires must be carefully routed around the disk. But everything does fit in place, and the NUC sits nicely closed afterward.

VMware ESX 6.0 and the Intel NUC

Happily, VMware added drivers for the Intel e1000 Gigabit Ethernet adapter found on all recent NUC motherboards to the standard set in ESX version 6. This means that it’s simple to install ESX on a NUC! No more creating a custom install! Just follow VMware’s directions to create a bootable USB drive and it’ll boot up perfectly on the NUC.

Once you’ve installed ESX on the USB drive, you might notice that the SATA and Realtek NICs are not detected. You need to install drivers for these. Happily, Andreas Peetz over at v-front.de maintains a depot of the correct drivers and you can install them from the command line in ESX! Just enable SSH and connect using your favorite terminal. Then install the drivers:

esxcli software acceptance set --level=CommunitySupported
esxcli software vib install -d http://vibsdepot.v-front.de -n sata-xahci
esxcli software vib install -d http://vibsdepot.v-front.de -n net55-r8168

Or something similar… You’ll get it. The SATA-XAHCI driver supports both the mSATA SSD and the SATA HDD in the NUC. The R8168 driver works with the R8111 chip in the Syba Ethernet NIC.

Stephen’s Stance

I hope this has been interesting and helpful to those wishing to use Intel’s NUC in a simple, low-power home “datacenter”. The Syba/Realtek NIC works great! Sadly, there’s no such solution for the fifth-generation Intel NUCs, so it’s best to stick to the older models for now. If there’s interest I might discuss my configuration next.

© Stephen Foskett for Stephen Foskett, Pack Rat: Adding a Second Ethernet Port to an Intel NUC via Mini PCIe

My Mini Datacenter

My mini datacenter is constructed from the following elements:

1. Three Intel NUC servers
   1. 2x D54250WYKH (fourth generation i5)
   2. 1x NUC5i3MYHE (fifth generation i3)
2. One TP-Link TL-SG2424 switch
3. One Raspberry Pi (model 2B) connected to a Drobo
4. An old Iomega ix4-200d
5. A pair of UPSes
   1. CyberPower 1000AVR LCD
   2. APC Back-UPS ES 550

I’m aiming for low power, low noise, low cost, and maximum utility in this mini datacenter. It’s not a lab really, since I’m running pfSense “in production” to manage my Internet access. But it’s definitely not an enterprise build!

I’m running vSphere 6.0 on all this, complete with VSAN (one SSD and one HDD per NUC) for all the “production” virtual machines. The Pi (backed by the Drobo) is serving an NFS datastore for logging, VIBs, and general messing about. The ix4 is hosting a datastore via iSCSI but I’m not using it right now. The two older NUCs have two Ethernet connections (thanks to the Syba Mini-PCIe NIC) but the new one has just one. I’m using 802.1Q VLANs to segment traffic a bit.

If you’re interested in any of the components mentioned here (why I chose them, what I’m doing with them, or how they work) just leave a comment or ask on Twitter. Maybe I’ll write some of that up too!

What is NUT?

Network UPS Tools (NUT) is one of the oldest and most useful open source projects most people have never heard of. Essentially, NUT is an extensible and highly configurable client/server application for monitoring and managing power sources. It includes a set of hardware-specific drivers (e.g. usbhid, apcupsd, snmp), a server daemon (upsd), and clients like upsmon and upsc.

NUT traces its history back to some serious software and hacks from the 1996/1997 era to monitor APC UPSes. Originally called Smart UPS Tools, the name was changed to Network UPS Tools due to the unfortunate similarity of that old name with a particular line of APC UPS products. NUT supports a wide variety of hardware, including most USB-connected UPSes, many of the old serial interfaces, and SNMP UPSes and PDUs. Sadly, many APC UPS models are unsupported because they use the proprietary “Microlink” protocol, though APC has recently begun offering a supported protocol.

A Raspberry Pi NUT Server

As a client/server application, the upsd daemon can run on one machine designated to monitor one or more UPSes and can support multiple clients querying the status of these devices. The tiny Raspberry Pi makes a wonderful NUT server, since it offers USB connectivity, sufficient CPU and memory, and a full Linux environment in a tiny and low-power device. The Raspbian operating system, based on Debian, includes pre-compiled nut-server and nut-client packages.

Since I already had a Raspberry Pi model 2B in my mini datacenter, it was a simple matter to connect my two UPSes to it via USB. I did need to order the proprietary RJ50-to-USB cable for the APC Back-UPS, but this was under $6 delivered from eBay. I plugged in the USB cables, configured NUT, and I was up and running with a fully functional home power monitoring server!

Here’s the simple config I’m using:

• Set “MODE=netserver” in /etc/nut/nut.conf
• Configure my UPSes in /etc/nut/ups.conf
• Set the server to “LISTEN” on its IP address in /etc/nut/upsd.conf
• Create a upsd user and password in /etc/nut/upsd.users
• Configure upsmon to monitor these UPSes by setting “MONITOR” lines in /etc/nut/upsmon.conf

Once all this was set I simply stopped and restarted nut-server and nut-client and my Raspberry Pi NUT server was up and running!

A NUT Client for VMware vSphere

VMware vSphere does not include NUT support, but a VMware ESX native port of upsmon exists and works great. It even supports the latest version of vSphere, 6.0. Thank you, René!

I installed upsmon on all three of my ESX hosts (the NUC servers mentioned above) by downloading René’s package to my NFS datastore and running the install script in an ssh shell on each host. Although the instructions say it won’t require a restart, I found that I did have to reboot all of the servers to load the daemon and set up the configuration variables properly.

Configuring upsmon for ESX involves customizing the following variables:

• UserVars.NutUpsName – The name of one or more UPSes in ups@server notation. If you enter more than one UPS name (separated by a space) the shutdown command will wait until all are exhausted.
• UserVars.NutUser – The upsd username set above.
• UserVars.NutPassword – The upsd password set above.
• UserVars.NutFinalDelay – The number of seconds to allow for a system shutdown after a low battery event is received.
• UserVars.NutSendMail – Enable (1) or disable (0) email from this upsmon instance.
• UserVars.NutMailTo – Email address to use for above.

Set these in the vSphere client by going to each host, clicking Settings, Advanced System Settings, and searching for “nut”.

I connected two of my NUCs to the larger CyberPower UPS and one to the smaller APC. I then configured each host to listed only for the UPS it is connected to. I also configured the NUT package in pfSense to monitor the UPS so it can shut down gracefully if there’s a power outage.

Stephen’s Stance

NUT is a wonderful and extensible power management framework, and the Raspberry Pi is an awesome platform on which to run the UPS monitoring drivers and upsd server daemon. Even if you’re not running vSphere, a Pi running NUT makes sense for the connected servers found everywhere today. My next task will be adding upsmon to the Iomega ix4 and other always-on systems!

© Stephen Foskett for Stephen Foskett, Pack Rat: Automated UPS Monitoring for vSphere with NUT and Raspberry Pi (Cheap!)

PCIe vs. Ethernet

Greg points out (rightly) that electrical signals copper traces on motherboards are currently limited to 15.75 Gbps in PCIe 4.0. With 16 lanes, this brings us to 252 Gbps of throughput on the PCIe 4.0 bus. Greg is also correct that current Ethernet switches operating at 25 Gb can handle this kind of throughput across 10 or so connections. QED, right?

Read Greg’s post: Musing: Could We Replace PCIe Bus With Ethernet Switch?

Sorry, Greg! Stuffing an Ethernet switch into a server is exactly the wrong direction for many reasons.

Most pressing is the issue of latency. PCIe latency is measured in hundreds of nanoseconds, while Ethernet interconnects are measured in tens of microseconds. This might not sound like much, but it’s literally two orders of magnitude difference and would be a huge step back in real-world use.

Just because you can push the same amount of data across a link (throughput) doesn’t mean you can do the same tasks. PCIe is like a fleet of shopping carts filling the aisles at your local Costco, while Ethernet is the street of SUVs taking those big boxes of cereal and lightbulbs back home. Although they are theoretically carrying the same payload, Explorers and Caravans just weren’t designed to navigate inside the store!

There are many other issues to consider as well. Ethernet NICs and switches are complex, being designed to handle the vagaries of topology changes, speed differences, and relatively frequent reconfiguration. An in-server Ethernet variant could be stripped down to the basics and integrated into the chipset just like PCIe but this would obviate the external connectivity benefits suggested by Greg. So every device would have to be a full-featured Ethernet endpoint, likely with TCP/IP besides!

Rack-Scale Computing, OPCIe, and SiPh

Greg mentions Intel’s work on rack-scale computing and silicon photonics. Good! But then he suggests running Ethernet over this lovely next-generation interface. Bad!

The intent is to run PCIe over all those integrated silicon/optical interconnects and extend it to rack-scale, rather than ingesting Ethernet. This has a whole raft of benefits, including better real-world performance (thanks to low latency and little protocol overhead) and easier integration, since PCIe is already in use at all points in a rack-scale infrastructure.

IT folks usually express some serious skepticism when I mention PCIe as an externally-exposed interconnect. But then I point out that this entire system is already in use! Apple’s Thunderbolt is simply PCIe over copper DisplayPort cables and long-range optical cables are on sale today. Intel’s silicon photonics (SiPh) optical PCIe (OPCIe) technology has been sampling for over a year now, and Fujitsu has demonstrated a server using these optical interconnects for peripheral interconnection.

Proponents of rack-scale computing seem poised to adopt OPCIe as an interconnect within the rack in the next year or so. This will encroach on the market for current server-to-server and server-to-storage interconnects like Ethernet, Fibre Channel, and InfiniBand. Enterprise products based on OPCIe are being developed as well, though few if any have yet been announced.

You might like to read my Rack Endgame series:

1. The Fat Middle: Today’s Enterprise Storage Array
2. Virtualized and Distributed Storage: This Time For Sure!
3. The Rack Endgame: A New Storage Architecture For the Data Center

Note that Fibre Channel, InfiniBand, RapidIO, and many other technologies besides have attempted to do just what Greg is suggesting: Unify internal and external connectivity with a “master” protocol. But none have succeeded. It seems more logical to standardize on a fast, scalable, low-latency bus like PCIe for short-range communication and a ubiquitous network like Ethernet for longer-range use.

Stephen’s Stance

Rather than pushing Ethernet into the server, the industry is pushing it out of the rack. Soon, racks will function like blade chassis, with high-speed interconnects for internal communication and Ethernet termination points for communication outside the rack. Probably the closest thing to reality in Greg’s vision is the concept of tunneling Ethernet over PCIe and integrating it into server chipsets. This would function something like FCoE, providing a path for a legacy interconnect (Ethernet) right into the heart of the new converged rack.

Note that the title of this piece if farcical, and based on Greg’s title. No, we cannot replace all of Ethernet with PCIe. It’s still the king of the campus and larger-radius networks. But it has no place in the world of PCIe!

© Stephen Foskett for Stephen Foskett, Pack Rat: Musing: Could We Replace Ethernet With PCIe?

A Little History

A long, long time ago, when Sun was the dominant server company and x86 was still a desktop CPU, computers needed to share files with each other over the network. Sun, being the dominant server company, created a networked file system called, surprisingly, Network File System or NFS for short. Like most things, it took a few revisions to take hold, but it eventually did with version 3. NFSv3 became so popular, in fact, that Big Daddy Sun “donated” it to the world as a standard. RFC-1813 never did become a real standard, but it was good enough to dominate network-attached storage for decades.

NFSv3 was created in the dark ages, when Microsoft hadn’t yet noticed the Internet and Windows didn’t even have a native IP stack. Seriously! Redmond was still trying to send files back and forth over NetBIOS before switching to something called CIFS, and if you don’t know what that is you should count yourself lucky and just move on along with this history lesson.

NFSv3 was a serious upgrade and a serious protocol, and UNIX nerds like me loved it. Well, as much as anyone loves a storage protocol. But it was never intended for the heavy use it has taken on in the 20 years (!) since it was introduced. For one thing, NFSv3 is stateless, making it difficult to maintain locks on shared data. It also has a rather naive and promiscuous approach to TCP port usage.

But the biggest issue with NFSv3 is that clients will only talk to one server at a time. This is seriously out of touch with today’s Internet-y scale-out world! This is especially true in high-I/O environments. You know, like VMware vSphere. Sadly, vSphere users have adopted NFSv3 like mad owing to the insanity of using block storage protocols that are even older and less flexible.

NFSv4 Rules?

The storage industry responded with a much-improved protocol: NFSv4. It has a concept of state, making locking easier to implement, and is more modern in the way it handles TCP ports. It also has a really nifty “pseudo filesystem” capability, allowing each client to see just what it needs to see. Plus, NFSv4 was ratified as an Internet standard, so everyone can coexist and get along and hold hands!

Or not. See, NFSv4 has been with us since 2003 and hasn’t really been adopted by anyone doing actual stuff. Sure, it exists as a protocol, but end-user enthusiasm has been pretty much nonexistent in the datacenter.

And there’s been a bit of criticism, too:

NFSv4 is not on our roadmap. It is a ridiculous bloated protocol which they keep adding crap to. In about a decade the people who actually start auditing it are going to see all the mistakes that it hides.

The design process followed by the NFSv4 team members matches the methodology taken by the IPV6 people. (As in, once a mistake is made, and 4 people are running the test code, it is a fact on the ground and cannot be changed again). The result is an unrefined piece of trash.

– Theo de Raadt, Leader of OpenBSD

NFS 4.1 and Parallel NFS!

NFSv4 didn’t address the inherent limits of a one-to-one protocol. No less a luminary than Garth Gibson wrote at length about the problems of NFSv4. So the industry responded by trying to develop 100 novel ways to parallelize I/O, mainly in the form of completely different shared file systems.

One of the better ideas was the parallel NFS (pNFS) concept from Panasas, Garth Gibson’s company. Like SDN over in the networking world, pNFS separates the “what” from the “how” of storage: A metadata server handles information about the files and directories and such separately from the actual data access. In practice, this allows clients to access data over multiple streams at once, yet still preserves backwards compatibility for non-parallel access.

pNFS became part of NFSv4.1, and the entire industry was transformed. Or not. Once again, end users responded with a yawn and, half a decade on, NFSv4.1 has almost no mainstream uptake. Very few storage devices support NFSv4.1 or the pNFS extensions anyway, so it’s not like they’re missing anything.

In fact, Microsoft’s competing SMB 3.0 protocol has much wider adoption even though it’s a relative whippersnapper, being introduced in 2012. No doubt much of the driving force behind SMB 3.0 is the fact that Microsoft controls the client (Hyper-V and Windows Server) as well as the “storage array” (Windows Server again), ensuring compatibility and supportability. No one in the NFS space can claim this kind of end-to-end support.

VMware vSphere 6 to the Rescue?

So now along comes VMware to the rescue, finally giving NFSv4.1 the client demand it has always needed. vSphere loves I/O and vSphere loves NFS, so this is a match made in heaven! Consummating this relationship, however, might take a bit longer…

First, VMware’s support for NFSv4.1 remains pretty iffy. It works with the basics (HA, DRS, vMotion) but not the more advanced features (Storage DRS, SRM, VVOLs). And of course it’s brand spanking new, so caveat user. Storage technologies usually have a bit of teething to do when they’re first launched, so I would expect a few bugs to crop up in short order.

But there are bigger issues. Although vSphere 6 includes NFSv4.1 support, it does not include pNFS! It does do multipathing or trunking (as Hans eloquently illustrates) but not parallel NFS. And you can’t mix NFSv3 and NFSv4.1 access so migration is problematic (as Chris discusses).

Then there’s the issue of array-side support. Even though NFSv4.1 has been around for a few years, storage array vendors have put it on the back burner so they can focus on things customers actually want. EMC claims NFSv4.1 support in the VNX, but who’s to say how “prime time” it is. And most other vendors are way behind in implementing it because people like me have been telling them for years not to bother.

Read more about vSphere 6.0 and NFSv4.1:

• An Overview of NFSv4 by SNIA
• VMware Embraces NFS 4.1, Supports Multipathing and Kerberos Authentication by Chris Wahl
• What’s New in vSphere 6.0: NFS Client by Julian Wood
• vSphere 6 NFS4.1 does not include parallel striping! by Hans De Leenheer
• vSphere 6.0 Storage Features Part 1: NFS v4.1 by Cormac Hogan

Stephen’s Stance

Storage is hard, and major changes need time before all the kinks are worked out. I applaud VMware for implementing NFSv4.1 since it checks a few of the empty boxes created by Microsoft’s excellent SMB 3.0. But it’s not time to celebrate yet.

Production use of NFSv4.1 in VMware vSphere environments is likely a year or two off, at least for people with half a brain. Unless your vendor sells a supported package and guarantees stability, it’s best to let it simmer a bit longer before taking the plunge. And enterprise environments are going to want to wait until the high-end vSphere features work on NFSv4.1, too.

Bonus snark! Maybe VMware shouldn’t have bothered with NFSv4.1 and should instead have implemented a modern, scalable, high-performance storage protocol like SMB 3! I’m sure Microsoft would welcome a new client…

Note: I am keenly aware that many people I respect spent years of their lives developing NFS and I didn’t do anything but snark at them. I really do feel bad about that.

Updated 2/4 to clarify the NFSv4.1/pNFS issue and removed the “one datastore” limit.

© Stephen Foskett for Stephen Foskett, Pack Rat: vSphere 6: NFS 4.1 Finally Has a Use?

Let’s get one thing out of the way from the start: The only thing protecting your data from corruption is some simple error checking on the disk drive itself and anything built into the software stack on your array or server. RAID doesn’t do any error checking at all, and neither does NTFS in Windows or HFS+ in Mac OS X. And none of those things can correct a read error if they encounter one. When people talk about disk or integrity checks they’re usually talking about the integrity of the file system or RAID set, not of the actual data itself.

Let this sink in: Your data is not protected. It can be corrupted. And you will never know until you need it.

Yes, I’m trying to scare you.

What Protects Your Data?

For most regular people, your only line of defense against random read and write errors is something called error correction coding (ECC), which is built into your hard disk drive’s controller. ECC is essential because magnetic media often has “bad” bits that aren’t readable, especially as information density increases. So hard disk controllers take care of recoverable read errors all the time.

As implemented in most modern hard disk drives, ECC works pretty well, but it’s not perfect. Most manufacturers claim that 1 bad bit will slip through every 10¹⁴ to 10¹⁶ bits, which is actually really good. But what about those unrecoverable read errors (URE’s)? They’re out of the disk drive’s hands. Hopefully something higher in the stack can recover the data. Maybe your filesystem, or maybe the storage array software.

The good news is that every enterprise storage array worthy of the name has data integrity checking built in, including all the big names and most of the smaller companies, too. After all, if an array can’t store data reliably it’s not really worth buying! So if you’re using a storage array, you’re probably good. Drobo apparently does integrity checking, too. So there’s that.

The bad news is that NTFS, ext3, and HFS+ don’t do any kind of data integrity checking. That means that the vast majority of user data is reliant on the ECC in the hard disk drive itself to ensure it meets the prime directive of storage.

The worse news is that unrecoverable read errors do happen, so all this data is at risk. Heavy data users (Greenplum, Amazon, CERN) report that errors really do happen about as often as hard disk drive manufacturers suggest they might. Furthermore, errors often come after the disk controller is done with the data: Faulty firmware, poor connections, bad cables, and even cosmic radiation can induce URE’s.

How Common is URE?

It’s hard to understand what one error in 10¹⁴, 10¹⁵, or 10¹⁶ really means in the real world. One easier way to think about it is that 10¹⁴ equals 12.5 TB, 10¹⁵ equals 125 TB, and 10¹⁶ is 1.25 PB. But this doesn’t really tell the correct story either. These are error rates, not error guarantees. You can read an exabyte of data and never encounter a URE, just like you can buy a lottery ticket and become a millionaire. The important thing is the probability.

As Matt Simmons points out, we can easily calculate the probability of a URE based on a given amount of data. The formula is Statistics 101 material, and he does a fine job of laying it out in his blog post, Recalculating Odds of RAID5 URE Failure. But even that was a little hard to grasp.

So here’s my take: Given a number of hard disk drives of a certain size in a set, how likely is a URE? I graphed it out for 1-10 drives of modern sizes, 1-10 TB. And the results are pretty scary.

Although a single 1 TB drive has less than an 8% chance of URE, those fancy new 10 TB drives start out over 55%, assuming a URE rate of 1 in 10¹⁴. Throw a few into a RAID set and you’ve got real trouble. If your risk threshold is a 50/50 chance, you can’t have more than three 3 TB drives (9 TB) in a set before you’re there. Even if you’re a crazy risk-taker, five 6 TB drives (30 TB) gets you over 90%. This is not good.

How about a URE rate of 1 in 10¹⁵? You’d reach 50% at around 90 TB, which is admittedly pretty high. But that’s still not a crazy huge amount of data, and it’ll be downright common in just a few years. And when you consider the reasonably likely issue of bad firmware and bad cables, it seems like URE isn’t an alien issue.

Play around with the numbers using my Google URE Spreadsheet.

What happens if you lose a bit? Maybe nothing. Video and audio files will probably keep playing. Photos might still look OK. But maybe not. And a faulty cable could wipe out the whole file, not just a bit of it (if you pardon the pun).

 Protect Your Data

What can you do about unrecoverable read errors? Simple answer: Use a better storage stack.

As mentioned above, most enterprise storage systems implement serious data integrity checking as part of their storage controller. Many use erasure coding, like the Reed-Solomon codes already used for ECC in the hard disk drive. Others prefer retaining a SHA1 hash for all data and recovering it from an alternate location if it gets corrupted. Either way the risk is reduced to such an extent that you don’t have to worry about it.

But what about servers and home users of non-enterprise storage? You’ve got trouble here. You can’t use NTFS, ext, or HFS+. Btrfs and ReFS have integrity features but they’re not effective out of the box: Btrfs only checks integrity with CRC32 and it’s not clear how it recovers data, while Microsoft engineered “integrity streams” into ReFS but only uses them for metadata by default. All they need to do is turn on integrity streams for user data, and this seems like a no-brainer for them in enterprise storage scenarios.

The only real option is ZFS. It has wonderful, robust integrity checking and data recovery. In fact, this was one of the design goals for ZFS! Honestly, if you care about your data and have more than a dozen terabytes of it, you must use something like ZFS or a real storage array.

Maybe not today, and maybe not tomorrow, but soon you’re going to need data integrity checking and error recovery. It’s time for Microsoft, Apple, and Btrfs to step up and provide it.

© Stephen Foskett for Stephen Foskett, Pack Rat: Why Big Disk Drives Require Data Integrity Checking

Hyperscale Versus Enterprise

Years ago, companies like Facebook, Yahoo, and Google discovered that conventional IT infrastructure just won’t cut it for their hyperscale data centers. What makes sense for even a large number of heterogeneous servers is entirely illogical for a massive, homogenous, software-defined cloud. So they set about creating a new kind of infrastructure that matched their needs and pared away the rest.

Hyperscale environments must shift as much intelligence as possible to free-to-use software, ruthlessly eliminating proprietary hardware. This is due to a simple economic fact: Scaling licensed or proprietary hardware costs serious money, while scaling license-free software on commodity hardware is much cheaper. The only way to be competitive in the cloud is to eliminate the IT infrastructure tax.

Another issue with conventional enterprise IT gear is that it was never designed for serious scale. “Scalable” solutions for the business typically use shared-memory clusters and can grow only by a factor of 10 or so. True hyperscale solutions reach thousands of nodes thanks to shared-nothing “sharded” architecture and software-defined integration.

The Open Compute Project Rack

Facebook and the rest of the Internet gang have spent the last decade driving this software-centric architecture, and this culminated last year with the introduction of the Open Compute Project (OCP).

First, they abandoned proprietary hardware (blade servers, storage arrays, and core switches) in favor of simplified alternatives. The mainstream vendors played along, introducing hyperscale servers that often looked like blades but were entirely different, typically with passive backplanes and no management hardware. Thus were born the Dell C5000, HP SL, and IBM iDataPlex. Although the margin on these servers was much lower, the volume of sales more than made up the difference.

But Facebook and friends kept pushing. They began exploring “white-box” servers, with SuperMicro stealing the spotlight if not the sales. The same was happening with networking, as OpenFlow and SDN opening the doors to new vendors, including some no-name alternatives. Storage, too, moved internal, leaving Fusion-io’s PCIe cards as the only big-budget name brand in hyperscale storage.

Open Compute was the next logical shift. Facebook and company invited hardware vendors like Intel and AMD to create a new simplified hyperscale server based around the “Group Hug” interconnect. These tiny servers fit in standard racks and used PCIe as their common I/O channel. This project has been moving forward over the last year, with many companies jumping onboard.

The endgame is a “Disaggregated Rack” where server components are dispersed within a rack, with Intel’s 100 GbE silicon photonics (SiPho) serving as the high-speed, low-latency interconnect. This rack would include a simple bulk storage server as “bottom of rack” capacity as well as a flash-based server for “top of rack” performance. This technology isn’t ready for production yet, but parts are becoming available and development is proceeding.

OCP Rack Versus Cisco UCS

All this may sound an awful lot like what I believe Cisco is building, but the similarities are somewhat superficial. Cisco UCS is a highly-integrated and tightly-managed enterprise product, while OCP Rack will be much simpler, relying on software for integration. Although there is some crossover between enterprise and hyperscale applications (name-drop Hadoop and Docker at your next business meeting!), this won’t go too far.

Read more about Cisco’s Trojan Horse

Consider Cisco’s new M-Series modular UCS servers. These are said to have been designed for hyperscale workloads but they don’t look much like Group Hug. M-Series uses a proprietary ASIC to aggregate servers on cartridges and share I/O resources. Like Group Hug, M-Series uses PCIe as the sole I/O protocol (a major departure for Ethernet-loving Cisco), but UCS is neither open nor really software-defined. M-Series relies on proprietary management for every function and is really designed to fit into an existing UCS environment, not an open cloud.

The same is true of Cisco’s Invicta flash appliance. It may look a bit like “Project Dragon”, Facebook’s Fusion-io-powered all-flash server, but Invicta will be as proprietary and enterprise-ready as anything from EMC and the rest of the enterprise storage old-guard.

In short, UCS is the kind of hyperscale that enterprises want and not at all what a software-driven cloud provider would be interested in. It’s very smart of Cisco to focus on enterprise needs rather than madly running off the cloud-shrouded cliff with an OCP clone.

Stephen’s Stance

Open Compute Project (OCP) is developing their own “Rack Endgame” solution, disaggregating conventional servers into rack-scale complexes interconnected by PCIe and driven by software. But these don’t really fit with the needs of the enterprise datacenter, which is much more interested in support and integration than acquisition costs and openness. They’re next-generation cousins rather than twins.

© Stephen Foskett for Stephen Foskett, Pack Rat: The Rack Endgame: Open Compute Project

The Rack Endgame

I recently wrote about “The Rack Endgame”, a serious challenge to the enterprise storage status quo in which the current datacenter architecture with centralized networked storage arrays is blown apart by virtualization and distributed storage. It’s worthwhile to read that whole series, but here it is in a nutshell, with special focus on Cisco.

Here’s the Rack Endgame series:

1. The Fat Middle: Today’s Enterprise Storage Array
2. Virtualized and Distributed Storage: This Time For Sure!
3. The Rack Endgame: A New Storage Architecture For the Data Center

Enterprise storage arrays appeared in the 1990’s as multitudes of smaller computers began to invade the enterprise datacenter. It made sense to centralize storage on a network in the heterogeneous computing environments of the time, and this architecture continues to this day. Both industry stalwarts and scrappy upstarts are focused on building “Jack of all trades” storage arrays that offer performance, capacity, and data management features.

But “do-it-all” arrays are seriously limited at the extremes. They will never offer the performance or scalability of specialized solutions. For this, we need a new dual-system architecture that brings fast flash close to the CPU and keeps scalable capacity at a distance.

This was impractical until now because such a solution would require specialized software running on every connected server. But the advent of virtualization makes it not just possible but downright likely! Already, VMware VSAN can intelligently locate data according to performance and capacity needs. And companies like PernixData and Infinio are building caching solutions that could turn into real storage virtualization layers in the future.

The Evolution of Cisco UCS

Cisco UCS was born into this environment. UCS was designed primarily to be a homogenous compute platform for the enterprise. All of the design decisions that went into “Project California” pointed to a future where server hardware was a universal blank slate for semi-portable (dare we say “containerized”?) operating system images. This is why UCS blades have no “personality” but what is defined at run-time by the UCS manager.

And UCS has been wildly successful. Although established companies like Dell and HP remain strong, UCS has become the standard-bearer for modern server hardware. It is the MacBook to the PC laptop world. And Cisco keeps innovating, with a UCS Mini and modular M-Series announced this month.

Until recently, Cisco has had no real storage solution. Apart from internal drive bays and PCIe flash cards, UCS required an enterprise storage array to function. But Cisco’s acquisition of Whiptail (now called Invicta) was a quiet step in a new direction.

Invicta gives Cisco a top-of-rack flash solution very much along the lines I described in my Rack Endgame article. It’s very fast but not all that scalable; a turbocharger, not an engine. This, and the fact that it’s not yet ready for production use, has led many to wonder what Cisco is up to. But I believe their plan was tipped at the recent UCS Grand Slam event.

Along with the UCS Mini and M-Series, Cisco introduced the fourth-generation UCS servers. And one in particular caught my eye: The UCS C3160 is a “Rack Storage Server” with two processors and up to 360 TB of hard disk storage in 4U. Cisco suggests that the C3160 could be used in media and analytics applications, but I see it differently.

The All-Cisco Rack

Imagine a rack of Cisco UCS servers – they could be B-Series blade servers, C-Series rack servers or M-Series modular servers. Now slap an Invicta flash unit on the top and a C3160 on the bottom, along with some Nexus switches for connectivity. Suddenly we have an all-Cisco rack!

This do-it-all rack could be used for VMware vSphere, with VSAN deciding whether to put data on the Invicta or ESX-running C3160. UCS-loving SimpliVity would be a perfect fit, too. It could run OpenStack or CloudStack, with the C3160 running Swift or Ceph. Or maybe the C3160 runs NexentaStor or Microsoft Windows Server!

Cisco would surely be happy with any such configuration, but I doubt EMC would be pleased. After all, UCS and Nexus make up “the other half” of the vBlocks offered by joint venture VCE. Once an all-Cisco rack is ready for production, why would Cisco spend much effort promoting cross-company solutions when they can keep all the money in-house?

Stephen’s Stance

The all-Cisco rack doesn’t mean EMC, NetApp, or even VCE is doomed. But a robust converged hardware offering from Cisco would challenge them just like UCS in the server space. And it’s clever, too, since Cisco still has plausible deniability. They can reassure EMC that the C3160 is just another UCS server and Foskett (and the customers) are inventing competition where none is intended. But the gloves will come off as soon as Cisco brings in-house a storage platform or converged compute software to run on this all-Cisco rack. Will it be Nexenta? SimpliVity? Or perhaps a homegrown offering leveraging OpenStack? Time will tell!

© Stephen Foskett for Stephen Foskett, Pack Rat: Cisco’s Trojan Horse

Here’s the Rack Endgame series:

1. The Fat Middle: Today’s Enterprise Storage Array
2. Virtualized and Distributed Storage: This Time For Sure!
3. The Rack Endgame: A New Storage Architecture For the Data Center

Two Kinds of Storage

Enterprise storage has two jobs to do: Long-term capacity and short-term retention. In other words, applications don’t treat primary storage resources the same, but we’ve historically used a single solution for both performance and capacity. As I discussed in “The Fat Middle: Today’s Enterprise Storage Array“, this was necessary because we lacked any mechanism to identify and move storage in the heterogeneous data centers of yesterday.

But things are changing. As I talked about in “Virtualized and Distributed Storage: This Time For Sure!“, virtualization and distribution is coming to the enterprise in the form of VMware VSAN, converged solutions from companies like Nutanix, and caching solutions like PernixData. And, as Dave McCrory points out with his “data physics“, storage has its own gravity that bends the shape of both physical infrastructure and application architectures.

The endgame of these shifts is the development of two flavors of enterprise storage, each with its own unique characteristics:

1. Capacity needs will be met by new storage systems optimized for scalability above all else. Take a look at X-IO, SwiftStack, and Exablox – notice any similarity in the job they serve? They’re all about capacity, ease of use, and scale.
2. Performance needs will be served by specialist flash on high-performance busses. Imagine a “top of rack” flash shelf like EMC DSSD or even in-server memory channel storage from SanDisk/Diablo. That’s a new world of performance!

Future enterprise data center architects will be able to realize dramatic reductions in cost for storage capacity as well as unheard-of performance by focusing on the edges rather than the “fat middle” of the market. And virtual server architecture will make this possible for the first time.

Top of Rack, Bottom of Rack

Walk into a datacenter in 5 years and things might look awfully different. Rather than a big, mysterious storage array at the physical and virtual center of the datacenter, you’ll see uniform racks of servers, each with its own storage. Sure, distributed storage could live within the server itself. But I think there’s another more-likely setup.

If servers are generic components to run software, why should storage be any different? Why not optimize servers for compute and memory and keep storage outside? Although it’s possible to pack disk drives inside converged servers, I’m not sure this is such a great idea. There’s no reason to keep low-performance capacity this close to the CPU, and disks waste space, power, and cooling that could be better used for CPUs and memory.

So why not stick to an external approach for storage by moving capacity to a “bottom-of-rack” pool? We could use inexpensive SAS or Ethernet to connect each server in a rack without any impact on performance. We could even start using simpler object storage rather than sticking with the outdated SCSI “fake disk” approach. But the protocol really doesn’t matter since we’re immediately abstracting everything in software anyway. It could even be FC, but that seems a bit of a waste of good tech with no SAN!

Then there’s performance storage. Advancements in PCI Express technology mean we can place high-throughput/low-latency flash memory just outside the server using an external PCIe bus. Keeping PCIe flash outside the server allows us to maintain something of a plug-and-play architecture for servers, since they could be swapped out without dramatically impacting the rest of the distributed storage network. But data would be “close” enough to offer orders-of-magnitude better performance.

I don’t hate the idea of keeping storage in-server, but this rack-oriented approach is a reasonable alternative. Sure, memory channel flash would be faster, but it could exist inside the top-of-rack array, right? And Nutanix is making hay with internal disk, but I doubt they’d be too upset to move disks just outside the box. It’s all good.

Stephen’s Stance

Top-of-rack flash and bottom-of-rack disk makes a ton of sense in a world of virtualized, distributed storage. It fits with enterprise paradigms yet delivers real architectural change that could “move the needle” in a way that no centralized shared storage system ever will. SAN and NAS aren’t going away immediately, but this new storage architecture will be an attractive next-generation direction!

Note: These are topics I discuss in my public speaking engagements, including my Truth in Storage seminar with Truth in IT. Check out the schedule for Truth in Storage and come listen to the whole story!

Disclaimer: I work with Diablo Technologies, X-IO, SanDisk, PernixData, Nutanix, Exablox, EMC, and most other companies in enterprise storage with Foskett Services and Tech Field Day. I don’t think I’m biased, but you can draw your own conclusions.

© Stephen Foskett for Stephen Foskett, Pack Rat: The Rack Endgame: A New Storage Architecture For the Data Center