Peer to peer DMA issue

Hello,

I am trying to track the source of a bug that makes the use of multiple GTX1080’s impossible when I turn on the IOMMU in Linux 4.8 (using either standard iommu=on or iommu=pt for passthrough mode) on a X99 board.

The bug can be triggered by running any peer-to-peer memory transfer, for example running the CUDA 8.0 Samples code 1_Utilities/p2pBandwidthLatencyTest from the terminal triggers the problem: the video driver (and as a result the X server) crashes immediately, and after multiple Ctrl-C’s and waiting for tens of seconds the server eventually restarts and I am presented with a login prompt to X Windows…

The relevant kernel error messages are (thousands of these lines, just a snippet below:)

[   51.691440] DMAR: DRHD: handling fault status reg 2
[   51.691450] DMAR: [DMA Write] Request device [04:00.0] fault addr f8139000 [fault reason 05] PTE Write access is not set
[   51.691457] DMAR: [DMA Write] Request device [04:00.0] fault addr f8139000 [fault reason 05] PTE Write access is not set
[   51.691462] DMAR: [DMA Write] Request device [04:00.0] fault addr f8139000 [fault reason 05] PTE Write access is not set
[   51.691465] DMAR: [DMA Write] Request device [04:00.0] fault addr f8139000 [fault reason 05] PTE Write access is not set
[   51.691470] DMAR: DRHD: handling fault status reg 400
[   51.740674] DMAR: DRHD: handling fault status reg 402
[   51.740683] DMAR: [DMA Write] Request device [04:00.0] fault addr f8139000 [fault reason 05] PTE Write access is not set
[   51.740688] DMAR: [DMA Write] Request device [04:00.0] fault addr f8139000 [fault reason 05] PTE Write access is not set
[   51.740693] DMAR: [DMA Write] Request device [04:00.0] fault addr f8139000 [fault reason 05] PTE Write access is not set

Cleary the above suggest that the CUDA driver is attempting DMA at an address for which the corresponding iommu page table entry write flag is not set, presumably because the driver has not properly registered/requested access via the general dma_map() kernel interface (https://www.kernel.org/doc/Documentation/DMA-API-HOWTO.txt)

Scouting the net reveals a bug registered (188271 – IOMMU DMAR fault with NVIDIA CUDA peer to peer) for exactly the same reason on a totally different hardware (Supermicro Dual socket board) using Pascal Titan-X’s, so same architecture cards as mine. Interestingly enough, the kernel error messages in this report claim unauthorized access of exactly the same memory address! (f8139000, in bold below) :

[16193.666976] DMAR: [DMA Write] Request device [82:00.0] fault addr <b>f8139000</b> [fault reason 05] PTE Write access is not set (edited)

So this looks like a red flag that somehow the indirection afforded by the iommu is bypassed and the driver is using hardcoded DMA addresses. Please note that the author of the bug report claims that seting iommu=igfx_off somehow solves this, but really igfx_off per se should be irrelevant here, as there are no intel integrated graphics in these systems. What instead happens is that most likely iommu=igfx_off as opposed to iommu=on just turns off iommu altogether, allowing the dma to succeed. This is exactly what happens on my system too. So in other words the bug report merely states that turning off the iommu allows peer-to-peer tranfers to work. Still his detailed log files should be very useful for an independent manifestation of the same issue.

I am using an ASRock X99 board (x99e-itx/ac) with latest firmware, intel i6800k, dual Asus GTX-1080s Founder’s Edition, 32GB ram and Ubuntu 16.10 with all updates applied (kernel 4.8.0-37) with latest driver 378.13. All earlier drivers exhibit same symptoms. I have uploaded nvidia-bug-report.log.gz together with dmesg.txt, lspci.txt and_usr_lib_xorg_Xorg.0.crash which is the ubuntu bug report file that contains every imaginable system detail together with a full stack trace and coredump of the crashed X server at:

Any thoughts/suggestions would be very welcome at this stage, especially if someone has come across similar issues. I have tried posting this in the Linux driver forum but it shows up as hidden for whatever reason so I doubt anyone will read it there …

It seems to me you would want to file a bug report with NVIDIA to get this on to their radar. The bug reporting form is linked from the CUDA registered developer website. You would want to attach the smallest possible self-contained code that reproduces the issue.

Note: The bug reporting form has some “interesting” filters built in (presumably to protect against spamming), and seems to be a bit flaky in general. This can be a source of major frustration. My suggestion would be to first file a minimal report, then add supporting files and additional information in subsequent steps.

Thanks a lot, will try that and report back how it goes …

I have a similar issue with multiple Tesla K40m cards:

[319500.608366] DMAR: [DMA Write] Request device [0a:00.0] fault addr f7139000 [fault reason 05] PTE Write access is not set
[319663.465506] DMAR: DRHD: handling fault status reg 402
[319663.471400] DMAR: [DMA Write] Request device [0d:00.0] fault addr 3e400000000 [fault reason 05] PTE Write access is not set
[319663.484129] DMAR: [DMA Write] Request device [0a:00.0] fault addr 3ec00000000 [fault reason 05] PTE Write access is not set
[319794.230817] DMAR: DRHD: handling fault status reg 602
[319794.236696] DMAR: [DMA Write] Request device [0d:00.0] fault addr f5139000 [fault reason 05] PTE Write access is not set
[319798.309073] DMAR: DRHD: handling fault status reg 702
[319798.314964] DMAR: [DMA Write] Request device [0a:00.0] fault addr f7139000 [fault reason 05] PTE Write access is not set
[319835.781267] DMAR: DRHD: handling fault status reg 2
[319835.786957] DMAR: [DMA Write] Request device [0a:00.0] fault addr 3ec00000000 [fault reason 05] PTE Write access is not set
[319835.799871] DMAR: DRHD: handling fault status reg 102
[319835.805756] DMAR: [DMA Write] Request device [0d:00.0] fault addr 3e400000000 [fault reason 05] PTE Write access is not set
[320058.557742] DMAR: DRHD: handling fault status reg 202
[320058.563623] DMAR: [DMA Write] Request device [0d:00.0] fault addr f5139000 [fault reason 05] PTE Write access is not set
[320063.525084] DMAR: DRHD: handling fault status reg 302
[320063.530957] DMAR: [DMA Write] Request device [0a:00.0] fault addr f7139000 [fault reason 05] PTE Write access is not set
[320654.704396] DMAR: DRHD: handling fault status reg 402
[320654.710290] DMAR: [DMA Write] Request device [0d:00.0] fault addr 3e400000000 [fault reason 05] PTE Write access is not set
[320654.723020] DMAR: [DMA Write] Request device [0a:00.0] fault addr 3ec00000000 [fault reason 05] PTE Write access is not set
[320720.002516] DMAR: DRHD: handling fault status reg 602
[320720.008420] DMAR: [DMA Write] Request device [0d:00.0] fault addr f5139000 [fault reason 05] PTE Write access is not set
[320723.659083] DMAR: DRHD: handling fault status reg 702
[320723.664950] DMAR: [DMA Write] Request device [0a:00.0] fault addr f7139000 [fault reason 05] PTE Write access is not set