A PCIe error log gives you a device address and an error name. Neither necessarily tells you which transaction failed, where it failed, or which component caused it. If the device captured a Transaction Layer Packet (TLP) header, that header is the next place to look. If it did not, you need other evidence.

That was the point of my ALPSS 2026 talk on using tlp-tool to analyze PCIe Advanced Error Reporting (AER) headers. The command-line program is called rtlp-tool; it turns the four hex words commonly printed by Linux into a packet type, Requester ID, address, and other fields. The slides are here.

I brought two storage incidents to the talk. The first had no TLP Header Log to decode; the useful evidence came from a PCIe analyzer. The second had a header, but its packet type looked wrong for the reported error. That one took a closer read of what CmpltAbrt actually said.

A read that could not get home

The first system had an NVMe drive and a Non-Transparent Bridge (NTB) switch on separate branches of an AMD EPYC I/O die. Peer-to-peer writes worked. Reads did not. The read request reached the switch, but the switch returned a Completion Abort before forwarding it to the far side.

Why the difference? A Memory Write is posted: the requester does not wait for a Completion TLP. A Memory Read is non-posted: the data has to come back to the requester. If the return path is wrong, writes can appear healthy while reads fail.

The analyzer trace showed that, on this path between the P and G links, the AMD fabric preserved the NVMe device’s original Requester ID in the read request. The switch had no return-route mapping for that ID. It could see the request, but could not route the completion back to its requester. Programming the NVMe IDs into the switch’s mapping table made the reads work in a proof of concept. A later BIOS change was reported to confirm the diagnosis by changing the platform’s Requester ID behavior.

The distinction between evidence sources matters. An AMD port reported the AER event, but its saved report contained no four-word TLP Header Log. I could not feed that report to rtlp-tool and claim the tool found the Requester ID. The analyzer trace showed the ID and abort point; the mapping experiment tested the explanation. AER told us where to start looking.

A write in a Completer Abort report

The second incident gave us an actual header. A Switchtec function reported CmpltAbrt (First) and saved these words:

TLP Header: 40001001 04090003 100047dc ae800000

I passed the line straight to rtlp-tool --aer:

printf 'TLP Header: 40001001 04090003 100047dc ae800000\n' | rtlp-tool --aer

The decode identifies a 3-DW Memory Write, length 1 DW, from Requester ID 0x0409 (04:01.1) to address 0x100047dc. The first three words give us that header. The fourth AER register word is undefined for this 3-DW format; the log does not establish it as write payload.

At first glance, a Memory Write in a Completer Abort report looks contradictory. Posted writes do not receive Completion TLPs, so where would a Completion Abort come from? The error name describes a Completer Abort condition recorded by the reporting function. It does not mean the captured packet is itself a Completion TLP with CA status. The header says the captured request was a write.

The header settles the packet type. It does not tell us why that write appears with this error, or explain the other NVMe link-down events around the incident. For that we still need topology, timestamps, device logs, and possibly a trace. Four DWORDs help; they do not close the incident.

Keep the reporter and the packet separate

For an AER or Downstream Port Containment (DPC) event, I now start with three questions:

  1. Which function reported the error?
  2. Which packet, if any, was captured?
  3. Where did the transaction actually fail?

Those can be three different places. rtlp-tool answers much of the second question quickly when a TLP header is available. The Linux PCIe AER guide now includes an example of decoding one with the tool. The rest is still engineering: line up the header with the topology and timeline, form an explanation, and try to break it with another trace or a configuration change.

After the talk, PCIe maintainers told me they had used the tool to debug issues the day before our conversation. That is the best kind of feedback for a decoder: it let them spend less time extracting bitfields and more time on the failure.