PCIe Gen3 I/O Expansion Drawer – Architecture and technical overview

2.7.2 PCIe Gen3 I/O Expansion Drawer

The PCIe Gen3 I/O Expansion Drawer (#EMX0) is a 4U high, PCIe Gen3-based and

rack-mountable I/O drawer. It offers two PCIe Fanout Modules (#EMXH), each providing six PCIe Gen3 full-high, full-length slots (two x16 and four x8).

The older Fanout Modules (#EMXF & #EMXG) that are used by Power E870, Power E870C, Power E880, and Power E880C systems cannot be ordered and is not supported with the Power system E1080.

For more information about the dimensions of the drawer, see 1.6, “I/O drawers” on page 26.

PCIe Gen 4 cable adapter (#EJ24) CXP 16X AOCs connects the system node to a PCIe Fanout Module in the I/O expansion drawer.

Concurrent repair and addition or removal of PCIe adapters is done by using HMC-guided menus or operating system support utilities.

A blind-swap cassette (BSC) is used to house the full-high adapters that are installed into these slots. The BSC is the same BSC that was used with the previous generation server’s #5802, #5803, #5877, and #5873 12X attached I/O drawers.

Power consumption is up to 50W per adapter slot.

Chapter 2. Architecture and technical overview 99

Figure 2-25 shows the rear view of the PCIe Gen3 I/O Expansion Drawer.

Figure 2-25 Rear view of the PCIe Gen3 I/O Expansion Drawer

2.7.3 PCIe Gen3 I/O Expansion Drawer optical cabling

I/O drawers are connected to the adapters in the system node by any of the following data transfer cables:

Ê 2.0 m CXP 16x Active Optical Cable (AOC) Pair (#ECCR)

Ê 10.0 m CXP 16x Active Optical Cable (AOC) Pair (#ECCY)

Ê 20.0 m CXP 16x Active Optical Cable (AOC) Pair (#ECCZ)

Cable lengths: Use the 2.0 m cables for intra-rack installations. Use the 10.0 m or 20.0 m cables for inter-rack installations.

Older cables #ECC6, #ECC8, and #ECC9 that are used in Power System E980 are not supported or oderable on Power System E1080. They are supported only as model conversions and announced as RPO.

100  IBM Power E1080: Technical Overview and Introduction

A minimum of one PCIe Gen 4 Optical Cable Adapter for PCIe Gen 3 Expansion Drawer (#EJ24) is required to connect to the PCIe Gen 3 6-slot Fanout Module in the I/O expansion drawer. The top port of the Fanout Module must be cabled to the top port of the #EJ24 port. Also, the bottom two ports must be cabled together.

To perform the cabling correctly, complete the following steps:

1. Connect an AOC to connector T0 on the PCIe Gen 4 cable adapter (#EJ24) in your server.

2. Connect the other end of the optical cable to connector T1 on one of the PCIe Gen 3 6-slot Fanout Modules in your I/O expansion drawer.

3. Connect another AOC cable to connector T1 on the PCIe Gen 4 cable adapter (#EJ24).

4. Connect the other end of the cable to connector T2 on the same PCIe Gen 3 6-slot Fanout Module in I/O expansion drawer.

5. Repeat the steps 1 to 4 for the other PCIe Gen 3 6-slot Fanout Module in the I/O expansion drawer, if required.

Drawer connections: Each Fanout Module in a PCIe Gen 3 Expansion Drawer (#EMX0) can be connected to only a single PCIe Gen 4 Cable Adapter (#EJ24). However, the two Fanout Modules in a single I/O expansion drawer can be connected to different system nodes in the same server.

Figure 2-26 shows the connector locations of the PCIe Gen3 I/O Expansion Drawer.

Figure 2-26 Connector locations for the PCIe Gen3 I/O Expansion Drawer

Chapter 2. Architecture and technical overview                       101

Table 2-25 lists the PCIe slots in the PCIe Gen3 I/O Expansion Drawer.

Table 2-25 PCIe slot locations and descriptions for the PCIe Gen3 I/O Expansion Drawer

Consider the following points:

Ê All slots support full-length, regular-height adapters or short (low-profile) adapters with a regular-height tailstock in single-wide, Gen3 BSCs.

Ê Slots C1 and C4 in each PCIe Gen 3 6-slot Fanout Module are x16 PCIe Gen 3 buses, and slots C2, C3, C5, and C6 are x8 PCIe buses.

Ê All slots support enhanced error handling (EEH).

Ê All PCIe slots are hot-swappable and support concurrent maintenance.

Typical Slot priorities are listed in Table 2-26.

Table 2-26 PCIe adapter slot priorities for the PCIe Gen3 I/O Expansion Drawer

For more information about slot priorities for individual adapters, see IBM Documentation.

If the EMX0 PCIe Gen 3 Expansion Drawer is configured with two PCIe Gen 3 6-slot Fanout Modules, distribute the PCIe adapters across both I/O modules whenever possible

Slots 1, 7, 4, and 10 support SR-IOV capabilities. However, if the total amount of physical memory is less than 128 GB, Slot 4 (P1-C4) and Slot 10 (P2-C4) are not SR-IOV capable.

102      IBM Power E1080: Technical Overview and Introduction

Figure 2-27 shows the connection schematics between system node and PCIe Gen3 I/O Expansion Drawer.

Figure 2-27 Connection schematics between system node and Expansion Drawer

General rules for the PCIe Gen3 I/O Expansion Drawer configuration The PCIe Gen 4 Cable Adapter (#EJ24) can be in any of the PCIe adapter slots in a

Power E1080 server node. However, we advise that you first populate the PCIe adapter slots as per the slot priorities that are discussed in this section. If an I/O expansion drawer is present, the PCIe Gen 4 cable adapter (#EJ24) must be given the highest priority.

Each processor module drives two PCIe Gen5 slots, and all slots are equal regarding their bandwidth characteristics. If you first use the slots following the slot priorities, you ensure that one PCIe Gen5 slot per processor module is populated before you use the second PCIe Gen5 slot of the processor modules. Table 2-27 lists the PCIe adapter slot priorities in the Power E1080 server. If the sequence is chosen as listed in the slot priority column, the adapters are assigned to the SCM in alignment with the internal enumeration order: SCM0, SCM1, SCM2, and SCM3

Table 2-27 PCIe adapter slot priorities

Chapter 2. Architecture and technical overview                       103

The following figures show several examples of supported configurations. (For simplification, we do not show every possible combination of the I/O expansion drawer to server attachments.)

Figure 2-28 shows an example of a Power 1080 that features a single system node and four PCIe Gen3 I/O Expansion Drawers. Each lane between the #EJ24 adapter and I/O drawer fanout module represent an CXP 16x AOC pair.

Figure 2-28 Single system node to four PCIe Gen3 I/O Expansion Drawer connections

104      IBM Power E1080: Technical Overview and Introduction

Figure 2-29 shows an example of a Power E1080 having two system nodes and four PCIe Gen3 I/O Expansion Drawers. Each lane between the #EJ24 adapter and I/O drawer fanout module represent an CXP 16x AOC pair.

Figure 2-29 Dual system node to four PCIe Gen3 I/O Expansion Drawer connections