The first major release of the Gen-Z systems interconnect specification is now available. The Gen-Z Consortium was publicly announced in late 2016 and has been developing the technology as an open standard, with several drafts released in 2017 for public comment.

Gen-Z is one of several standards that emerged from the long stagnation of the PCI Express standard after the PCIe 3.0 release. Technologies like Gen-Z, CAPI, CCIX and NVLink seek to offer higher throughput, lower latency and the option of cache coherency, in order to enable much higher performance connections between processors, co-processors/accelerators, and fast storage. Gen-Z in particular has very broad ambitions to blur the lines between a memory bus, processor interconnect, peripheral bus and even straying into networking territory.

The Core Specification released today primarily addresses connecting processors to memory, with the goal of allowing the memory controllers in processors to be media-agnostic: the details of whether the memory is some type of DRAM (eg. DDR4, GDDR6) or a persistent memory like 3D XPoint are handled by a media controller at the memory end of a Gen-Z link, while the processor itself issues simple and generic read and write commands over the link. In this use case, Gen-Z doesn't completely remove the need for traditional on-die memory controllers or the highest-performance solutions like HBM2, but Gen-Z can enable more scalability and flexibility by allowing new memory types to be supported without altering the processor, and by providing access to more banks of memory than can be directly attached to the processor's own memory controller.

At the lowest level, Gen-Z connections look a lot like most other modern high-speed data links: fast serial links, bonding together multiple lanes to increase throughput, and running a packet-oriented protocol. Gen-Z borrows from both PCI Express and IEEE 802.3 Ethernet physical layer (PHY) standards to offer per-lane speeds up to the 56Gb/s raw speed of 50GBASE-KR, and will track the speed increases from future versions of those underlying standards. The PCIe PHY is incorporated more or less as-is, while the Ethernet PHY standards have been modified to allow for lower power operation when used for shorter links within a single system, such as communication between dies on a multi-chip module. Gen-Z allows for asymmetric links with more links and bandwidth in one direction than the other. The Gen-Z protocol supports various connection topologies like basic point to point links, daisy-chaining, and switched fabrics, including multiple paths of connection between endpoints. Daisy-chain links are estimated to add about 5ns of latency per hop, and switch latencies are expected to be on the order of 10ns for a small 8-port switch up to 50-60ns for a 64-port switch, so using Gen-Z for memory access is reasonable, especially where the somewhat slower persistent memory technologies are concerned. The Gen-Z protocol expresses almost everything in memory terms, but with each endpoint performing its own memory mapping and translation rather than attempting to form a unified single address space across a Gen-Z fabric that could scale beyond a single rack in a data center.

Wide Industry Participation

The Gen-Z Consortium launched with the support of a dozen major technology companies, but its membership has now grown to the point that it is easier to list the big hardware companies who aren't currently involved: Intel and NVidia. Gen-Z has members from every segment necessary to build a viable product ecosystem: semiconductor design and IP (Mentor, Cadence, PLDA), connectors (Molex, Foxconn, Amphenol, TE), processors and accelerators (AMD, ARM, IBM, Cavium, Xilinx), switches and controllers (IDT, Microsemi, Broadcom, Mellanox), every DRAM and NAND flash memory manufacturer except Intel, software vendors (RedHat, VMWare), system vendors (Lenovo, HPE, Dell EMC). It is clear that most of the industry is paying attention to Gen-Z, even if most of them haven't yet committed to bringing Gen-Z products to market.

At the SuperComputing17 conference in November, Gen-Z had a multi-vendor demo of four servers sharing access to two pools of memory through a Gen-Z switch. This was implemented with heavy use of FPGAs, but with the Core Specification 1.0 release we will start seeing Gen-Z show up in ASICs. The focus for now is on datacenter use cases with products potentially hitting the market in 2019.

In the meantime, it will be interesting to see where industry support concentrates between Gen-Z and competing standards. Many companies are members or supporters of more than one of the new interconnect standards, and there's no clear winner at this time. Nobody is abandoning PCI Express, and it isn't clear which new interconnect will offer the most compelling advantages over the existing ubiquitous standards or over proprietary interconnects. Gen-Z seems to have one of the widest membership bases and the widest target market, but it could still easily be doomed to niche status if it only receives half-hearted support from most of its members.

Source: Gen-Z Consortium

Comments Locked

23 Comments

View All Comments

  • tuxRoller - Wednesday, February 14, 2018 - link

    You....didn't read the article.
  • CheapSushi - Tuesday, February 13, 2018 - link

    I'm guessing Intel doesn't like this because Gen-Z was aiming for having an "open and royalty-free" standard. Plus, since Gen-Zs standard is broad, so not just for PCIe, but could be used in similar place as Omni-Path, QPI, DMI, etc, they don't like it. I suppose it's in Intel's interest to continue market segmentation / product stratification through bus types, etc, when possible.
  • Dragonstongue - Wednesday, February 14, 2018 - link

    and there is Nvidia they seem to want complete NON open ways, they do not want to ceed control of anything, they want to be the only vendor or control the standard, Intel is very much the same way

    They want to be a cookie factory, push out the product nothing more, not make the product better (unless absolutely have to)

    for the first persons comment on without Intel dead in the water...they may still be number 1 producer for computer chips, they are far from the only one selling mega volumes worth of them annually.AMD is selling a crap ton of chips (Ryzen comes to mind) Threadripper, their mobile etc etc.

    Point is, many other makers are often times WAY more willing and wanting to work on open standards vs bottling everything up for their own benefit (Intel, Nvidia, Apple, are birds of a feather wanting proprietary BS for no real good reason in many cases)

    Work together to make it better, maybe things can be tweaked to lower cost, increase performance, reduce power etc, there is not a single perfect company (it is impossible) so consortiums such as this can be a very great thing if they have the support and guidance to "make it happen"
  • Yojimbo - Wednesday, February 14, 2018 - link

    Why should NVIDIA spend the money, manpower, or donate IP to join the consortium (whatever the requirements are)/when at this point they don't seem to get much out of it? They don't make host devices (CPUs), memory, or deal with networking, so they are entirely on the periphery of the standard. If it becomes a viable standard I'd imagine they'd support it just like they support PCI-express. Without Intel using it, however, it helps them very little.
  • Yojimbo - Wednesday, February 14, 2018 - link

    BTW, if this becomes a standard that Intel supports I can't see how NVIDIA wouldn't jump at the opportunity to use it. NVIDIA can't get a high speed connection to main memory on Intel's platform currently.
  • rahvin - Thursday, February 15, 2018 - link

    Nvidia most certainly does everything you listed that they don't. Either your ignorant of their actual products or you're trolling.
  • mode_13h - Saturday, February 17, 2018 - link

    Oh, puh-leez. They are openly hostile to standards. They are stuck at OpenCL 1.2, and don't support any version of it on Tegra. There's no technical justification for that.

    They also made NVLink, rather than embracing any open alternative or even just using the draft PCIe 4.

    They are also passively hostile towards the open source community, in that all the libraries needed to use their products effectively are proprietary and they give very little assistance (currently none) to the team writing open source linux drivers for their hardware.

    I'm not a Nvidia hater (I have some of their products and we buy more at my job), but it's only fair to call out their bad behavior.
  • ET - Wednesday, February 14, 2018 - link

    I'd love to see a more detailed analysis of the standard. I'm not sure what the standard allows exactly, but the possibilities seem tantalising. Having several types of RAM in the system (such as DDR4 and GDDR6), adding support for a new type of RAM via an expansion card, giving GPU and CPU pretty much direct access to each other's RAM, ...
  • mode_13h - Wednesday, February 14, 2018 - link

    You can already do that (CPU & GPU accessing each other's RAM). Try to step outside of that PC-centric mindset. This standard is really about cloud.

    IMO, what's most interesting about Gen-Z is that it's like a hybrid between a networking standard and an internal peripheral bus. This sentence says it all:

    "Gen-Z had a multi-vendor demo of four servers sharing access to two pools of
    memory through a Gen-Z switch."

    Enabling machines to share storage is a pretty big deal. Having banks of remote storage is a pretty big deal.
  • mode_13h - Wednesday, February 14, 2018 - link

    Oh, and because of that, I don't see it really competing with PCIe 4/5. In the near term, those will probably rule the inside of machines, where latency and bandwidth are of prime importance, while Gen-Z becomes more of a rack-scale interconnect standard.

    Then, Intel can go take their OmniPath and suck it.

Log in

Don't have an account? Sign up now