Behind The Scenes: SSD Testing In 2019 With Quarch's HD Power Moduleby Billy Tallis on May 9, 2019 9:00 AM EST
- Posted in
- Testing Procedures
Every so often we are asked for more behind-the-scenes information on how we test products. So now that we've completed some updates to our SSD testing workflow, we're going to do just that, and give you a quick peek into our 2019 SSD testing setup.
As a bit of background to start things off, since December 2017, we have been using a Quarch XLC Programmable Power Module (PPM) to supply and measure power to SSDs during most of our benchmark runs. The XLC module is Quarch's second-generation power module, and while capable, it does have a few limitations as far as our SSD testing goes. Most of which, as it turns out, have been addressed in the XLC's successor, the third-generation HD Programmable Power Module.
So Quarch in turn has sent one of these new power modules our way, which we're using to upgrade our SSD testbed and refine our workflow. The newer power module allows us to take more accurate measurements of M.2 SSD power draw and gives us more flexibility in test configurations.
At a high level, the purpose of using a programmable power module is to allow us to record the power consumption of SSDs in great detail, with the power module feeding any SSDs we're testing. Complicating matters just a tad, these power modules are a bit larger than a standard 5.25" optical drive and require their own AC wall power input, so they don't fit easily inside any of the systems we use for SSD testing. Instead, they feed power to SSDs through cables that connect to any of several power injection fixtures—Quarch provides fixtures for almost all SSD form factors and connectors, including the very new EDSFF connector. These power injection fixtures pass through any data signals untouched, but connect the SSD's power supply lines to the Quarch power module instead of the host computer's power supply.
When it first launched, the HD Programmable Power Module added many convenience features over the XLC, but the core power supply and measurement capabilities we rely on were basically the same. Both modules provide two DC power rails: a 12V rail and one that can be configured for 3.3V or 5V depending on the type of drive being tested.
The power modules have several analog to digital converters (ADCs) to simultaneously measure current and voltage on both rails, and there's an extra low-range mode for the current measurements that allows for accurate measurement of sub-milliamp current drawn by SSDs in extremely low-power idle states. The ADCs are sampling at 250kHz, so the power modules can record very short-lived spikes in power draw. This for example means that when testing a mechanical hard drive, it is easy to observe individual seek operations separately from the read or write operations that occur after the heads are in the right place. The measurement resolution and sampling rate both fall somewhere between the typical values for bench multimeters and oscilloscopes, and for this specialized use case a Quarch programmable power module is able to replace a very large pile of traditional electronics lab equipment.
Last year, Quarch developed a hardware upgrade for the HD PPM to address a major limitation: When testing M.2 SSDs, the power module has to supply 3.3V over a cable that's about two feet long. High-power M.2 SSDs can draw in excess of 3A of current, which leads to significant voltage drop along the cable—and SSDs typically only tolerate about a 5% drop below nominal voltage before they stop working. The power modules allow their output rails to be set above nominal voltage, but some M.2 PCIe SSDs still pushed the limits of the XLC module's current delivery capabilities. Quarch has now upgraded the HD module to support 5A continuous and up to 9.2A for short term (10 seconds). More importantly, the HD PPM now automatically adjusts its output voltage to compensate for voltage drop along the cable, so that it regulates the programed voltage measured at the drive instead of at the module's output connector.
200mV drop on 3.3V supply with XLC PPM while writing 16GB test file
With the XLC power module, we were unable to feed high-performance M.2 PCIe SSDs 3.3V directly without running into problems, and instead we have been conducting performance testing with the drives mounted in a PCIe to M.2 adapter card that does its own 12V to 3.3V regulation. That adds some overhead to our power measurements, but this is easy to correct for after the fact. For idle power testing, voltage drop isn't a problem and we have always been using a direct 3.3V supply. The upgraded capabilities of the HD power module allow us to start using a direct 3.3V supply even during performance testing, and to use Quarch's M.2 power injection fixture to connect drives through the host system's M.2 slots instead of just through full-size PCIe card slots.
Powering M.2 SATA or PCIe SSDs with M.2 injection fixture
The net impact of all of this is that we're now able to measure M.2 SSD power draw in a laptop computer (albeit with some extra cables hanging out the bottom). So, for example, we can get proper power measurements from the Optane Memory H10 that isn't supported by any of our current desktop testbeds. We can also now measure power draw with the deepest PCIe link state power management modes that are not usable on most desktop systems. This gets most recent M.2 NVMe SSDs well below 10mW at idle, compared to the 50+mW that is the best most NVMe SSDs can achieve on our usual Skylake desktop test system. We will be re-testing idle power for all M.2 SSDs in our collection and will start reporting three sets of idle power measurements:
- Active idle with all NVMe and PCIe power management features disabled, as is typical for most desktop systems
- Desktop idle representing the best idle power achievable on most desktop systems with appropriate BIOS and operating system settings configured to save power
- Laptop idle representing the best-case that can be expected from modern laptops (and the occasional desktop with the right settings)
We still don't have the ability to test DevSleep idle power for SATA drives, which is the closest equivalent to the laptop idle that we are now measuring for M.2 PCIe SSDs. DevSleep is generally impossible on desktops because it requires signaling through the SATA power connector rather than the data connector, and the HP Spectre x360 13t notebook we are using with the Optane Memory H10 is an NVMe-only system. The notebook market in general has reached a tipping point and OEMs are rapidly switching from SATA to NVMe, so SATA DevSleep is becoming less relevant.
Going forward, all new M.2 SSDs will be tested with the HD Programmable Power Module for both idle and performance benchmarks. We will re-run performance testing for older M.2 drives as time permits, after updating all of the idle power measurements. On the software side, nothing is changing with our test procedures, and we will still be using the same Skylake desktop system for most of our tests except when testing drives that are not supported on that system (eg. the Optane Memory H10). We are currently not planning to make major changes to our SSD test procedures or host systems until it's time to move to PCIe 4.0.
Post Your CommentPlease log in or sign up to comment.
View All Comments
MrCommunistGen - Thursday, May 9, 2019 - linkImpressive. I wonder if the Phison Laptop Idle Wake-Up time is inherent to their design or if that's something they can tune with newer firmware.
Other than monitoring a drive's power consumption, is there a good way to tell if a drive is actually utilizing ASPM? I don't mean "check to make sure ASPM is enabled in BIOS/UEFI", but rather that the drive is actually entering the low power states.
I have a Toshiba XG4 that I've been messing around with. In some of my systems it idles at ~70C (hot to the touch) and in others it idles at ~45C. I'd assume that in the hot systems, there's no Power Management happening -- and I'd like to see if there's something I can look at in the OS that tells me "no Power Management is happening". Likewise, on the cooler idling systems I'd be curious to see what level of ASPM is being achieved.
Billy Tallis - Thursday, May 9, 2019 - linkIt might be possible to check some link status registers to figure out what state is actually being used on the drive's side of the PCIe link. But a lot of the time it's hard to query a drive's sleep state without waking it up.
I don't have an XG4, but as the TLC counterpart to the XG3/RD400 I should be getting around to re-testing that controller's idle behavior soon. My previous measurements have shown it to be tied for the highest active idle among M.2 drives, and second only to Phison E7 for desktop idle on the Skylake system with semi-broken ASPM. (https://www.anandtech.com/bench/SSD18/2453 and https://www.anandtech.com/bench/SSD18/2424)
FreckledTrout - Thursday, May 9, 2019 - linkThats a nice setup. In the PC photo whats the yellow discs at the bottom left hand of the photo?
James5mith - Thursday, May 9, 2019 - linkThey are push pins to hold drives in place in the drive cage.
Can't remember which case manufacturer uses them.
Billy Tallis - Thursday, May 9, 2019 - linkIt's an In-Win case. It seems to be the standard cheap desktop case that Intel's demo depot uses. I have two of them from the systems that they provided for Optane Memory reviews.
FreckledTrout - Thursday, May 9, 2019 - linkThanks. I don't play with cheap cases so never saw anything like that before. :)
evilspoons - Thursday, May 9, 2019 - linkMy local (otherwise good) PC store uses a variant of them and they're nasty. Everything is sharp, the panels don't go back on without a lot of "convincing" due to the weird way the tabs on the side panels lock in to those open rectangles in the channel it presses on to, and they barely retain their 90-degree corners. Not recommended. Whenever I can, I pay to upgrade to... well, basically anything else.
Billy Tallis - Thursday, May 9, 2019 - linkThe case definitely feels like a design from 15 years ago, save for the front panel USB 3.0 ports. Not something I'd ever buy, but it works fine for sitting on a shelf in the rack.
IntelUser2000 - Thursday, May 9, 2019 - linkI think this is great, but even this is going to fall short of battery life testing. I know, I know its unrealistic to expect battery life testing for every SSD.
The 970 Pro in TH testing is not the top SSD despite the fast transition and low idle power.
Even Notebookcheck testing shows a significant disparity between power measurement tests and actual battery life tests.
IntelUser2000 - Thursday, May 9, 2019 - linkI want to add power measurement has its uses, in Desktop and Server environments where its always plugged in.