Diablo Readies Memory Channel Flash For DDR4
The first generation of flash-based memory channel storage from Diablo Technologies and SanDisk is ramping up in high-end Xeon E5 and E7 machines made by IBM and is now getting put into machines by Supermicro. Diablo makes the chips that provide the interface between the memory bus and SATA-based flash memory, and SanDisk creates the UlltraDIMM modules that snap into memory slots and put flash that much closer to processing capacity. Now, Diablo is previewing its next generation of flash-memory interface chips, which will make use of the impending DDR4 interface and sport features that will allow flash memory channel sticks to be treated more like main memory and less like a block device, thus broadening the appeal of flash DIMMs.
Diablo has not really talked about this much before, but the first generation of its memory channel storage controller chip was called Carbon1, and SanDisk, by virtue of its acquisition last year of Smart Storage, had an exclusive license to that chip for a set period of time. Neither SanDisk nor Diablo has ever revealed what the term of that exclusivity was, but it was purposefully set to give the UlltraDIMM that SanDisk sells a chance to get through proofs of concept and prove the idea out. The two companies have been working closely with IBM to test UlltraDIMMs, but the idea has always been for Diablo to work with more memory and flash drive makers to create a broad array of memory channel storage products and have a larger number of server and storage array makers deploy the technology in their machines.
Kevin Wagner, vice president of marketing at Diablo, tells EnterpriseTech that eight large banks are testing out the UlltraDIMMs in proofs of concept right now, and a number of other large enterprises are also seeing how their applications might be accelerated by the flashy DIMMs. Wagner says that there is a misconception out there in the market that the UlltraDIMMs are only supported on IBM’s high-end System x X6 servers, which are based on Intel’s high-end “Ivy Bridge-EX” Xeon E7 processors. While it is true that IBM emphasized the System x3850 X6 (four socket) and x3950 X6 (eight socket) machines at its launch earlier this year, the UlltraDIMMs are also supported on IBM’s two-socket System x3650 M4 servers, which use Intel’s Xeon E5-2600 processors in either the “Sandy Bridge” or “Ivy Bridge” generations. Most of the IBM PoCs are running on these two-socket machines, says Wagner.
SanDisk and Diablo alike need for UlltraDIMMs to be used by more than IBM and in more than a few systems if it is to become a viable product. Boston, one of Supermicro’s largest value added resellers, had already certified UlltraDIMMs on its machines, and now Supermicro is lining up as customers.
Interestingly, a lot of the requests to run trials of UlltraDIMMs, which come in 200 GB and 400 GB capacities at the moment, are being tied to Supermicro machines as more and more enterprises are looking to Supermicro to supply their iron. (IBM is one of Supermicro’s largest customers by virtue of its acquisition of the SoftLayer cloud, and is on track to buy around 100,000 systems from Supermicro by the end of the year, doubling its server fleet.) Boston, one of Supermicro’s largest value added resellers that is always looking for an edge against the tier one server suppliers, has already certified UlltraDIMM memory on the Supermicro machines it resells. Wagner says that Supermicro originally wanted to certify on four of its X9 Series machines, but now it is already up to seven machines, including its Twin hyperscale servers, Hyper-Speed platforms aimed at high frequency traders, SuperStorage storage arrays, and GPU-enabled SuperServer hybrid systems. More importantly, now that Diablo, SanDisk, and Supermicro have completed an implementation of the UEFI interface between operating systems and BIOS firmware that knows how to speak UlltraDIMM, the memory channel storage can be more broadly supported across Supermicro platforms. By partnering with Diablo, Supermicro can also get in early on the future generations of the flash DIMM technology and not only tune up its platforms, but also provide input to steer how future Carbon interface chips and completed flash DIMMs are developed.
While Diablo provides drivers for memory channel storage for the Linux and Windows operating systems as well as for the ESXi hypervisor from VMware, IBM and Supermicro provide their own UEFI code and the drivers provided by Diablo have to be tweaked to account for the engineering these two companies do in their systems. Those UEFI tweaks are not in the open source Linux drivers provided by Diablo to the Linux community, and it seems likely that as other server makers get on board with UlltraDIMMs and other brands of memory channel storage as they come to market this will remain the case. The important thing is that IBM is working to certify more machines to use UlltraDIMMs, Supermicro is now on board, and several more OEMs will join the fold and be ready as the next-generation Carbon2 controller comes to market in 2015.
The big benefit with memory channel storage is that it has substantially lower write latency than even PCI-Express flash cards, which in turn are faster than solid state drives that hang off disk controllers inside of servers or storage arrays. Wagner says that the typical PCI-Express flash card can write with a latency that ranges from between 15 and 18 microseconds, depending on the file size and the load on the card, but that the UlltraDIMMs based on the Carbon1 interface chip from Diablo can do writes in around 3.3 microseconds. Because the DDR3 memory bus is faster and more parallel than the PCI-Express peripheral bus, as the load increases the UlltraDIMM can maintain its low write latency, while the PCI-Express flash cards see their latencies stretch out as the I/O load increases. Due to the nature of reads on flash memory, the UlltraDIMMs and PCI-Express cards have about the same latency – around 150 microseconds – and there is no big advantage there. That said, memory channel storage would be extremely useful as a fast write cache for servers and even as a second or third tier of cache in an all-flash or hybrid disk-flash storage array where write speed really mattered. So in supercomputing and financial services applications that have a lot of logging in their code, the state of the DRAM memory could ne snapshotted over to the flash DIMMs periodically, managed by the system itself and meaning that the code that is doing such logging and snapshotting now can be removed from applications. This will very likely speed up the applications, and in many cases, both resiliency and speed matter for such applications.
With the Carbon2 interface chip, Diablo is going to be pushing the write latencies down even lower than 3.3 microseconds on flash DIMMs – Wagner is not saying how low yet – and will be shifting to the emerging DDR4 interface, timed more or less to market with DDR4 memory slots. (Intel is widely expected to be the first vendor to support DDR4 memory in its forthcoming “Haswell” Xeon E5 v3 processors, chips that the company is already shipping to early adopter customers in the HPC and hyperscale arenas.) Generally speaking, DDR4 memory will run faster – the spec hits 3.2 Gb/sec, while DDR3 tops out at 2.1 Gb/sec – and use memory chips that range in density between 2 Gb and 16 Gb, twice that of DDR3. DDR4 memory will run at a lower voltage, use less power, and support higher sustained and burst data rates. And as fast as flash DIMMs are concerned, all of this merely means that vital data in memory will have to be replicated to flash all that much faster and in larger capacities.
The way the Diablo interface chip works, the memory interface is separated from the interface to the non-volatile memory. So Diablo can change either side independently. For the Carbon2, the memory interface is moving from DDR3 to DDR4, but in a future Carbon chip (or a derivative of the Carbon2 even), Diablo could shift to 3D NAND memory or even phase change memory or memristors as these storage options come to market.
There is more to the Carbon2 chip than changing to a DDR4 interface, though. One big change is a new feature of the memory channel storage processing engine called NanoCommit. This is part of Diablo’s log-stated plan of converging DRAM and flash memory on the DDR bus. With NanoCommit, the idea is to treat flash memory more like DRAM, reducing the gap between the two as far as applications and the memory subsystem are concerned.
The engine is tweaked to take very small pieces of data in main memory and transfer them at latencies on the order of nanoseconds to the non-volatile storage on the flash DIMM stick. This updated engine can handle hundreds of millions of transactions per second at those nanosecond latencies, and significantly, because it is moving small chunks of data in parallel between the two types of memory, it allows for the mirroring of DRAM to flash.
NanoCommit is only one of such enhancements that Diablo is working on other changes to the MCS processing engine to increase its functionality and boost its performance. Wagner did not tip his cards, but ideally, it would be interesting to see DRAM and flash look like a contiguous memory space with memory-style load/store.
Diablo and its memory channel storage partners are targeting the first quarter of 2015 for the launch of the Carbon2 chip, which is about when the Haswell Xeon E5 chips will start ramping in volume. These are the first processors to support DDR4 main memory that are coming to market, and Xeon E5 chips are by far the highest volume in terms of shipments worldwide, too.
