iommu unhandled context fault on PCI device DMA
I am having a problem with driver for a PCI device located behind three transparent P2P bridges. Unfortunately it's not possible to connect the device directly to the TX2 PCIe slot. The driver allocates a buffer for DMA descriptors which it then passes to the device. When the device attempts to fetch a descriptor, the following message gets logged on the console: arm-smmu 12000000.iommu: Unhandled context fault: iova=0x269ea1000, fsynr=0x10001, cb=21, sid=17(0x11 - AFI), pgd=0, pud=0, pmd=0, pte=0 the iova is correct bus address returned from dma_map_single. Here is the code fragment: buf->virt = kmalloc(buf->size, GFP_KERNEL); buf->phys = dma_map_single(&dev->dev, buf->virt, buf->size, DMA_BIDIRECTIONAL); I also tried the dma_alloc_coherent, it resulted in very same iommu error. Anyone knows how to fix/workaround this issue?
I am having a problem with driver for a PCI device located behind three transparent P2P bridges.
Unfortunately it's not possible to connect the device directly to the TX2 PCIe slot.

The driver allocates a buffer for DMA descriptors which it then passes to the device.
When the device attempts to fetch a descriptor, the following message gets logged on the console:

arm-smmu 12000000.iommu: Unhandled context fault: iova=0x269ea1000, fsynr=0x10001, cb=21, sid=17(0x11 - AFI), pgd=0, pud=0, pmd=0, pte=0

the iova is correct bus address returned from dma_map_single. Here is the code fragment:

buf->virt = kmalloc(buf->size, GFP_KERNEL);
buf->phys = dma_map_single(&dev->dev, buf->virt, buf->size, DMA_BIDIRECTIONAL);

I also tried the dma_alloc_coherent, it resulted in very same iommu error.

Anyone knows how to fix/workaround this issue?

#1
Posted 04/04/2017 07:29 PM   
IOVA 0x269ea1000 seems to be wrong. For PCIe, all IOVAs must fall with in 0x8000_0000 ~ 0xFFF0_0000 range. Are you saying the return value of dma_map_single() is 0x269ea1000??
IOVA 0x269ea1000 seems to be wrong.
For PCIe, all IOVAs must fall with in 0x8000_0000 ~ 0xFFF0_0000 range.
Are you saying the return value of dma_map_single() is 0x269ea1000??

#2
Posted 04/08/2017 11:04 AM   
I've been getting a similar message (on a TX2, r27.1): arm-smmu 12000000.iommu: Unhandled context fault: iova=0x68cf4000, fsynr=0x80011, cb=21, sid=17(0x11 - AFI), pgd=0, pud=0, pmd=0, pte=0 Which seems to occur when my PCI device sends an MSI interrupt (they don't get though to my ISR's). The iova address looks very similar to the MSI address given in lspci -vv, but it is only the bottom 32bits. ... Capabilities: [50] MSI: Enable+ Count=1/4 Maskable- 64bit+ Address: 0000000268cf4000 Data: 0000 ... Any ideas? Could it be a 64bit/32bit addressing problem?
I've been getting a similar message (on a TX2, r27.1):

arm-smmu 12000000.iommu: Unhandled context fault: iova=0x68cf4000, fsynr=0x80011, cb=21, sid=17(0x11 - AFI), pgd=0, pud=0, pmd=0, pte=0

Which seems to occur when my PCI device sends an MSI interrupt (they don't get though to my ISR's).

The iova address looks very similar to the MSI address given in lspci -vv, but it is only the bottom 32bits.

...
Capabilities: [50] MSI: Enable+ Count=1/4 Maskable- 64bit+
Address: 0000000268cf4000 Data: 0000
...

Any ideas? Could it be a 64bit/32bit addressing problem?

#3
Posted 04/12/2017 03:32 PM   
Following patch should solve the issue [code]diff --git a/drivers/pci/host/pci-tegra.c b/drivers/pci/host/pci-tegra.c index 85a1bbe..79d5ab6 100644 --- a/drivers/pci/host/pci-tegra.c +++ b/drivers/pci/host/pci-tegra.c @@ -3044,7 +3044,7 @@ static int tegra_pcie_enable_msi(struct tegra_pcie *pcie, bool no_init) } /* setup AFI/FPCI range */ - msi->pages = __get_free_pages(GFP_DMA32, 0); + msi->pages = __get_free_pages(GFP_DMA, 0); } base = virt_to_phys((void *)msi->pages); [/code] let us know the result.
Answer Accepted by Forum Admin
Following patch should solve the issue

diff --git a/drivers/pci/host/pci-tegra.c b/drivers/pci/host/pci-tegra.c
index 85a1bbe..79d5ab6 100644
--- a/drivers/pci/host/pci-tegra.c
+++ b/drivers/pci/host/pci-tegra.c
@@ -3044,7 +3044,7 @@ static int tegra_pcie_enable_msi(struct tegra_pcie *pcie, bool no_init)
}

/* setup AFI/FPCI range */
- msi->pages = __get_free_pages(GFP_DMA32, 0);
+ msi->pages = __get_free_pages(GFP_DMA, 0);
}
base = virt_to_phys((void *)msi->pages);


let us know the result.

#4
Posted 04/13/2017 03:49 PM   
Could it be that we have more of these 32/64 bit issues? On TX2 virt_to_page gives back 3ffffbcc203a040 which seems odd and leads to an unavoidable "Unable to handle kernel paging request at virtual address" when you try to use that. The addressing does not seem to be that off but virt_to_page seems to do something weird. TX2: (with patch at least driver loads ) [ 63.525223] : Allocate DMA buffer [ffffff8000e81000] bus: [ffffffc1ec9418c0] , size 0x00400000. [ 63.535384] : Allocate DMA buffer [ffffff8001281000] bus: [ffffffc1ec9418c8] , size 0x00400000. TX1: [ 10.835979] : Allocate DMA buffer [ffffffc057800000] bus: [ffffffc0f85624c0] , size 0x00400000. [ 10.835986] : Allocate DMA buffer [ffffffc057c00000] bus: [ffffffc0f85624c8] , size 0x00400000.
Could it be that we have more of these 32/64 bit issues?

On TX2
virt_to_page gives back 3ffffbcc203a040 which seems odd and leads to an unavoidable "Unable to handle kernel paging request at virtual address" when you try to use that.

The addressing does not seem to be that off but virt_to_page seems to do something weird.

TX2: (with patch at least driver loads )
[ 63.525223] : Allocate DMA buffer [ffffff8000e81000] bus: [ffffffc1ec9418c0] , size 0x00400000.
[ 63.535384] : Allocate DMA buffer [ffffff8001281000] bus: [ffffffc1ec9418c8] , size 0x00400000.


TX1:
[ 10.835979] : Allocate DMA buffer [ffffffc057800000] bus: [ffffffc0f85624c0] , size 0x00400000.
[ 10.835986] : Allocate DMA buffer [ffffffc057c00000] bus: [ffffffc0f85624c8] , size 0x00400000.

#5
Posted 04/21/2017 02:42 PM   
Thanks vidyas. That patch fixed the MSI interrupt issue.
Thanks vidyas. That patch fixed the MSI interrupt issue.

#6
Posted 04/21/2017 04:18 PM   
on memory that is created with dma_alloc_coherent my code works well on i386 / x86_64 and tx1 both 32 and 64 bit but not on TX2 TX2: [ 1052.721167] virt -> FFFFFF8000E81000 [ 1052.721170] phys -> FFFFFFC080E81000 [ 1052.721172] pfn -> FFFFFFC080E81 TX1: [ 1691.026382] virt -> FFFFFFC057800000 [ 1691.026388] phys -> D7800000 [ 1691.026392] pfn -> D7800 virt_to_phys result seems wrong
on memory that is created with dma_alloc_coherent my code works well on i386 / x86_64 and tx1 both 32 and 64 bit but not on TX2

TX2:

[ 1052.721167] virt -> FFFFFF8000E81000
[ 1052.721170] phys -> FFFFFFC080E81000
[ 1052.721172] pfn -> FFFFFFC080E81

TX1:

[ 1691.026382] virt -> FFFFFFC057800000
[ 1691.026388] phys -> D7800000
[ 1691.026392] pfn -> D7800

virt_to_phys result seems wrong

#7
Posted 04/22/2017 07:19 PM   
@MartijnBerger, I think there is some issue with dev pointer in this case. Bus addresses must fall with in 0x8000_0000 ~ 0xFFF0_0000 range.
@MartijnBerger,
I think there is some issue with dev pointer in this case. Bus addresses must fall with in 0x8000_0000 ~ 0xFFF0_0000 range.

#8
Posted 04/24/2017 05:54 AM   
Hi vidyas, I am attempting to fix this. Can you reproduce based on the information given or do you need something else ? calling virt_to_phys() on the result of dma_alloc_coherent is the smallest path to having this go wrong i can find. edit: You may mail me directly on this if there is anything I can help you with, This is a time critical issue for me
Hi vidyas,

I am attempting to fix this. Can you reproduce based on the information given or do you need something else ?


calling virt_to_phys() on the result of dma_alloc_coherent is the smallest path to having this go wrong i can find.

edit:
You may mail me directly on this if there is anything I can help you with, This is a time critical issue for me

#9
Posted 04/24/2017 06:20 AM   
BTW, whats the context in which virt_to_phys() is used? Given that dma_alloc_coherent() already gives CPU virtual address (to facilitate any access to this region from CPU), may I know whats the reason to use virt_to_phys()?
BTW, whats the context in which virt_to_phys() is used?
Given that dma_alloc_coherent() already gives CPU virtual address (to facilitate any access to this region from CPU), may I know whats the reason to use virt_to_phys()?

#10
Posted 04/24/2017 11:51 AM   
In the minimal case there is no reason. But in the control flow of my driver it is useful. But in general I am surprised that is does not work, so far all platforms i tried my code on the kernel has no problem and virt_to_phys() works as it should, this includes TX1. For some reason on TX2 is gives back what could be a pointer in the kernels address space. I can probably work around it by getting dma_addr_t and relying on physical space and DMA space being the same but I am of the opinion that I should not have to and and that virt_to_phys should work.
In the minimal case there is no reason. But in the control flow of my driver it is useful.

But in general I am surprised that is does not work, so far all platforms i tried my code on the kernel has no problem and virt_to_phys() works as it should, this includes TX1. For some reason on TX2 is gives back what could be a pointer in the kernels address space.

I can probably work around it by getting dma_addr_t and relying on physical space and DMA space being the same but I am of the opinion that I should not have to and and that virt_to_phys should work.

#11
Posted 04/24/2017 02:02 PM   
This is because of the fact that SMMU is being enabled for PCIe. Even in TX1, if 24.2 release is used (where SMMU is enabled for PCIe), virt_to_phys() macro would not give actual physical address. TX2 anyway has SMMU enabled for PCIe by default, so, virt_to_phys() doesn't work as expected. The reason being virt_to_phys() macro doesn't have idea of SMMU page tables where the actual mapping from bus_address to physical_address is present.
This is because of the fact that SMMU is being enabled for PCIe.
Even in TX1, if 24.2 release is used (where SMMU is enabled for PCIe), virt_to_phys() macro would not give actual physical address. TX2 anyway has SMMU enabled for PCIe by default, so, virt_to_phys() doesn't work as expected.
The reason being virt_to_phys() macro doesn't have idea of SMMU page tables where the actual mapping from bus_address to physical_address is present.

#12
Posted 04/28/2017 07:39 AM   
dma_mmap_attrs() is also not working for me on TX2. What API should I be using to get this memory into user space ? Also dma_alloc_attrs() is giving 0x8000000 on first mapping of 4M and next allocation creates an address exactly 5M from that alloaction when judgeing by dma_addr_t
dma_mmap_attrs() is also not working for me on TX2.

What API should I be using to get this memory into user space ?

Also dma_alloc_attrs() is giving 0x8000000 on first mapping of 4M and next allocation creates an address exactly 5M from that alloaction when judgeing by dma_addr_t

#13
Posted 05/11/2017 08:45 AM   
I am the same problem with V4L2 driver for a PCI device that supports three channel videos through TX2 PCIe slot. The first channel is OK. But the others have errors. I used the pci_alloc_coherent to alloc the memory for DMA, it return the virt and phy address correctly. [ 40.350211] Virt addr is 1281000, Phys addr is 80500000 But when receive data, there are some errors. [ 48.704536] arm-smmu 12000000.iommu: Unhandled context fault: iova=0x804e0600, fsynr=0x40013, cb=21, sid=17(0x11 - AFI), pgd=26ff63003, pud=26ff63003, pmd=24d572003, pte=0 Anyone knows how to fix/workaround this issue?
I am the same problem with V4L2 driver for a PCI device that supports three channel videos through TX2 PCIe slot.

The first channel is OK. But the others have errors.

I used the pci_alloc_coherent to alloc the memory for DMA, it return the virt and phy address correctly.

[ 40.350211] Virt addr is 1281000, Phys addr is 80500000

But when receive data, there are some errors.

[ 48.704536] arm-smmu 12000000.iommu: Unhandled context fault: iova=0x804e0600, fsynr=0x40013, cb=21, sid=17(0x11 - AFI), pgd=26ff63003, pud=26ff63003, pmd=24d572003, pte=0


Anyone knows how to fix/workaround this issue?

#14
Posted 07/27/2017 08:34 AM   
Hello Vidyas On TX2 board, r28.1. PCIE device using MSI interrupt. kernel report error: arm-smmu 12000000.iommu: Unhandled context fault: iova=0xebe40000, fsynr=0x60001, cb=22, sid=17(0x11 - AFI), pgd=0, pud=0, pmd=0, pte=0 arm-smmu 12000000.iommu: Unhandled context fault: iova=0x00000000, fsynr=0x60011, cb=22, sid=17(0x11 - AFI), pgd=0, pud=0, pmd=0, pte=0 after I apply patch: drivers/pci/host/pci-tegra.c - msi->pages = __get_free_pages(GFP_DMA32, 0); + msi->pages = __get_free_pages(GFP_DMA, 0); rebuild kernel and driver and reboot TX2 and running our PCIE device driver with MSI interrupt coming in. but the arm-smmu error is still there. Is there other patch we have to apply for our driver case ?
Hello Vidyas

On TX2 board, r28.1. PCIE device using MSI interrupt. kernel report error:

arm-smmu 12000000.iommu: Unhandled context fault: iova=0xebe40000, fsynr=0x60001, cb=22, sid=17(0x11 - AFI), pgd=0, pud=0, pmd=0, pte=0
arm-smmu 12000000.iommu: Unhandled context fault: iova=0x00000000, fsynr=0x60011, cb=22, sid=17(0x11 - AFI), pgd=0, pud=0, pmd=0, pte=0

after I apply patch:
drivers/pci/host/pci-tegra.c
- msi->pages = __get_free_pages(GFP_DMA32, 0);
+ msi->pages = __get_free_pages(GFP_DMA, 0);

rebuild kernel and driver and reboot TX2 and running our PCIE device driver with MSI interrupt coming in.
but the arm-smmu error is still there.

Is there other patch we have to apply for our driver case ?

#15
Posted 02/13/2018 04:57 PM   
Scroll To Top

Add Reply