Testbed Setup and Testing Methodology

The rising popularity of USB 3.1 (both Gen 1 and Gen 2) Type-C direct-attached storage (DAS) devices and the upcoming Thunderbolt 3 DAS units made it clear that I had to work on some updates to our direct-attached storage testbed. Originally based on the Haswell platform, the DAS testbed used a Thunderbolt 2 PCIe add-on card and the USB 3.0 ports hanging off the PCH. For a brief while, I also added USB 3.1 Gen 2 Type-A and Type-C PCIe cards to evaluate a few DAS units.

The introduction of Skylake has been quite interesting from the viewpoint of fast local storage. While the 100-series chipset doesn't have native USB 3.1 Gen 2 support, it does have plenty of high-speed PCIe 3.0 lanes that enable high-speed bridges to other protocols. Motherboard vendors have decided to enable USB 3.1 on entry-level boards with an ASMedia bridge chip. However, premium boards can be equipped with Intel's own Alpine Ridge controller. As mentioned in the previous section, Thunderbolt 3 and Intel's Apline Ridge are interesting for a few reasons:

  • In addition to Thunderbolt 3, Alpine Ridge also integrates a USB 3.1 Gen 2 (10 Gbps) host controller
  • Thunderbolt 3 works over a Type-C interface, and supports a couple of additional protocols - USB 3.1 Gen 2 and DisplayPort 1.2

Considering these aspects, it made sense to migrate to Skylake for our DAS testbed. In particular, I looked out for a board with Alpine Ridge integrated. Ian published the review of the GIGABYTE Z170X-UD5 TH, and it turned out that the board perfectly fit the requirements. Note the DisplayPort output and PCIe 3.0 x 4 lanes from the Z170 PCH getting into the Alpine Ridge controller before producing two Type-C ports that can be used as '2x Intel Thunderbolt 3' or '2x DisplayPort' or '2x USB Type-C'.

Intel provided us with a sample of the Core i5-6600K to use in the board. G.Skill also came forward with four 8GB DDR4 DIMMs to give the testbed 32GB of DRAM (the same as our Haswell-based testbed)

The Corsair Carbide Series Air 540 chassis in our Haswell-based testbed has been great in terms of footprint, ventilation and easy access to components. Two hot-swap internal SATA slots turned out to be a boon for quick secure erases of SSDs as well as benchmarking of internal HDDs meant for NAS usage in the single-disk mode. However, this unintended usage model (I wasn't planning on doing this frequently when I first opted for the Corsair Air 540) was a bit of a hassle, since one of the chassis sides had to be dismounted to access the hot-swap slots. I wanted to address this issue in the new testbed.

In the lookout for a ATX chassis for the new testbed, I had three main requirements:

  • Hot-swap bays accessible without the need to open up the unit (similar to the drive slots in hot-swap NAS units)
  • Portability in terms of being easy to shift from one location in the lab to another (something I realized as important when trying to test daisy chaining with a Thunderbot 2 DAS unit last year)
  • Cubical footprint with horizontal motherboard orientation in order to better fit in a workbench and enable easy swapping out of PCIe cards in the future

The Cooler Master HAF XB EVO perfectly fit our requirements. The two X-Dock bays fulfilled our need for hot-swap bays for both 3.5" and 2.5" drives. Since the unit is marketed as a LAN box, it has two rigid carry handles on the side panels to enable portability. The unit can also easily serve as a testbench. Only the top cover (held in place by two screws at the back) needs to be removed in order to access the PCIe cards. The PSU slot also extends slightly out, enabling easier cable management inside the chassis. With plenty of additional drive slots in addition to the X-Dock, it was a no-brainer to go with the Cooler Master HAF XB EVO.

We have traditionally gone with the chassis vendor for the PSU also in our testbeds. Cooler Master suggested the fully modular V750 for use in our system.

Even though a 750W PSU is an overkill for a system with no discrete GPUs, the rating makes sure that we have the option in the future. The fully modular nature also helped greatly in cable management.

In addition to the above, we made use of a few components that were salvaged from earlier reviews / unused components from previous builds - a Corsair Hydro Series H105 liquid CPU cooler, a Samsung SM951 NVMe PCIe 3.0 x4 SSD for the boot drive, and an Intel 730 series 480 GB SSD and a Corsair Neutron XT 480 GB SSD for use as staging drives for temporary data. The gallery below provides some more pictures from our build process.

Evaluation of DAS units (both Thunderbolt 3-based and USB 3.x-based ones) on Windows is being done with the testbed outlined in the table below.

AnandTech DAS Testbed Configuration
Motherboard GIGABYTE Z170X-UD5 TH ATX
CPU Intel Core i5-6600K
Memory G.Skill Ripjaws 4 F4-2133C15-8GRR
32 GB ( 4x 8GB)
DDR4-2133 @ 15-15-15-35
OS Drive Samsung SM951 MZVPV256 NVMe 256 GB
SATA Devices Corsair Neutron XT SSD 480 GB
Intel SSD 730 Series 480 GB
Add-on Card None
Chassis Cooler Master HAF XB EVO
PSU Cooler Master V750 750 W
OS Windows 10 Pro x64
Thanks to Cooler Master, GIGABYTE, G.Skill and Intel for the build components

Our direct-attached storage testing involves artificial benchmarks (ATTO and CrystalDiskMark) as well as real-world data transfer scenarios (photographs, videos and documents). In addition, we run the PCMark 8 Storage Bench for select multimedia editing workloads. Finally, for simultaneous multi-target testing (as in, multiple drives in a JBOD, or, two or more daisy-chained systems), we utilize Iometer to get an idea of the total performance.

The Nuts and Bolts of Thunderbolt 3 Direct-Attached Storage Performance
Comments Locked

60 Comments

View All Comments

  • bill44 - Saturday, April 16, 2016 - link

    Agreed. If performance is what you seek, this maybe it;
    https://www.cinema5d.com/faster-than-fast-sonnet-f...
  • philipma1957 - Saturday, April 16, 2016 - link

    I am more of a Mac guy then a Windows or a linux guy.
    I own 3 macs
    I own 3 windows pcs
    I own a linux pc

    Thunderbolt is all over my home and I use it for a lot of IT work on Macs.
    A portable 2 drive piece of gear like this allows for rescue work.
    Simply have a ssd in one slot with a mac os and a blank ssd 2tb samsung. go to a clients place
    boot with it and then clone and rescue the dead mac's hhd on the big ssd in your other slot.

    I do this now with an older version of thunderbolt then this one. Works great.
    I have owned a dozen thunderbolt devices and they are day and nigh over most usb3 devices in terms of being a reliable stable external booter. None of this matters to most none mac users.

    One of the great features is I can plug in a thunderbolt mac os and boot most macs.

    Windows never really allowed this. So to complain about a system like thunderbolt costing too much is okay but the truth is when you realize what thunderbolt 1 and thunderbolt 2 can do to and with someone else's mac you would say that it is too cheap .
  • LuxZg - Sunday, April 17, 2016 - link

    Is it just me or did you really comparr a performance of USB 3.1 Gen.2 SSD connected directly to PC's GEN2 port, vs it connecting to GEN1 port of a DAS and claimed that performance drop is purely due to DAISY CHAINING?
  • ganeshts - Sunday, April 17, 2016 - link

    Nope.

    The USB 3.1 Gen 2 SSD is first connected to a PC's Gen 2 port.

    In the second case, it is connected to the Thunderbolt 3 port of the second DAS unit. Note that Thunderbolt 3 can support both Thunderbolt peripherals as well as USB 3.1 Gen 2 devices for daisy chaining. Only obvious restriction is that USB 3.1 Gen 2 devices can appear only at the end of the daisy chain.

    Also, please read specifications carefully for the USB 3.1 Gen 1 port of the DAS - The Gen 1 port is a device port - i.e, it can connect to a PC directly. It can't connect to another device like the Extreme 900 Portable SSD.
  • Zizy - Monday, April 18, 2016 - link

    1.) Could you please add USB 3.1 results for direct-attached? (just use USB cable instead of thunderbolt). Presumably the same, right? So, what is the point of Thunderbolt here? Daisy chaining?
    2.) Temperature is steadily increasing in your test. 10 minutes is quite short-ish test. What is the steady-state temperature and does it throttle at that temperature?
  • ganeshts - Monday, April 18, 2016 - link

    (1) The USB 3.1 port is USB 3.1 Gen 1 - Maximum possible bandwidth on that is theoretically 5 Gbps, but, in practice, for RAID-0 SSDs, one would get around 450 - 500 MBps. So, it will not be the same as that of Thunderbolt 3 - where we could get around 800 MBps

    (2) Temperature for SSD configuration was with the fans completely turned off. You can see that the fan 'on' case for HDDs shows that even after more than 250GB of continuous data transfer, the temperature of the HDDs is within 5C of the temperature prior to the start of the transfers. I don't expect throttling to be a concern. If it is (depending on the SSD or HDD in use), one should just turn on the fan.
  • Haravikk - Tuesday, April 19, 2016 - link

    Pretty underwhelming for the enormous price; USB3 is fast enough to handle three or even four drives if there's hardware RAID on the other end, so Thunderbolt only really offers an improvement if you're handling a JBOD enclosure, and even then two drives won't really justify it.

    Also, for the price, why aren't these hot-swap bays? Someone wanting to use paired drives in a mirror setup would benefit from being able to hot-swap in a replacement, and for the money you're spending I'd think that pretty reasonable.

    Lastly, couldn't this be bus-powered? Even 3.5" drives shouldn't require so much power the bus can't handle it, plus any good RAID controller should be able to spin them up separately (so you're not drawing peak power for both drives simultaneously).

    I dunno, I get that Thunderbolt is still a premium product, but it still feels as though the actual quality of design and features you're getting is severely lacking. Besides, being the first Thunderbolt 3 enclosure for two SATA III drives is meaningless as even Thunderbolt 1 is more than fast enough for that. Even the bigger "high end" enclosures are surprisingly poorly made and thought out for the amount of money you need to drop on them; a networked iSCSI device using ZFS with l2ARC will meet most needs, and the NAS/SAS options are just way better.
  • Questor - Tuesday, April 19, 2016 - link

    I believe computing and related has come to a point where it needs an official organization to manage and assign acronyms.
  • Chad - Tuesday, April 19, 2016 - link

    As a thunderbolt user, there are several other benefits (especially for Mac's) that is brings to the table. For example, I added an external HDD, then added 5 more, 1 by 1 as time went on and as needed, for a total of 6, but only using the 1 port. On Mac's, ports are a valuable commodity. On this same iMac, I only had a few USB ports and they were quickly used up for various things. So that left the thunderbolt port... but i was able to connect 5 devices on it. None of them came even remotely close to using it's bandwidth, but for me, it wasn't about that, or about speed... but expandability. That is something worth noting, for sure. Granted, in PC land, this isn't really a concern, but on Mac's, it's a huge thing. There's also an aesthetic factor to daisey chaining, over a spiderweb hub solution.
  • Haravikk - Wednesday, April 20, 2016 - link

    True, but my main point was that you're paying a pretty big premium for the Thunderbolt aspect of this device, for what is overall a pretty poor product; two 3.5" bays, not really setup well for 2.5" drives, not hot-swappable, not actually all that compact. Even little things haven't been thought through, like the place of the Thunderbolt ports under the fan, limiting you to a smaller fan size (bigger is always better when it comes to noise after all).

    Also, technically it's possible to create daisy-chained USB devices too (each one would integrate a three port hub, with one port for whatever it actually does), the only difference with Thunderbolt really is that it's a crucial part of the design so manufacturers are forced to enable chaining, whereas USB device manufacturers are happier cutting their costs and forcing us to rely on hubs.

    Also, if I've understood you right and you've got six HDDs, I'd seriously consider an 8-bay NAS instead. If you're fine with command line stuff then you can set that up with ZFS backed iSCSI, format the iSCSI volume as ZFS on your mac and assign it a chunk of your internal drive as a cache, and it's just better overall (if a little complex to setup in the first place). If someone manufactured a Thunderbolt cache device that I could plug straight between a Mac and a NAS then that would be the perfect option, but until then it's big local cache plus NAS with tons of drives for me =)

Log in

Don't have an account? Sign up now