Problem stopping an uncompleted read or write operation

[Arribas]

Hi Jeremy,
First of all, congratulations, your driver works very well!. However, I have an annoying problem if I have to stop an uncompleted DMA operation (either write or read).
I have a sample project in Vivado 2016.2 and a Zedboard hardware (AXI-DMA connected to a loopback AXI-Stream FIFO), with a AD Kernel 4.4 running on Linaro Debian distribution.

The loopback ezdma_speed_test is working:

sudo ./ezdma_speed_test
sent 100000 4096-byte packets in 17.003143051 sec: 22.974 MB/s

But, if I launch a read operation and the FIFO is empty, then, the device read function is waiting the DMA transaction to finish, as expected. In that situation, if I cancel the read operation, by hitting for instance, CTRL+C, then, ezdma driver tries to cancel the DMA operation of the xilinx-dma driver (by calling xilinx_dma_terminate_all() ), but it says that "cannot stop channel" and triggers a Kernel panic. Take a look to this Kernel crash (and system hangs!):

sudo ./ezdma_read
Receiving 100000 4096-byte packets
^C <<< CTRL+C is hit here to stop the process >>>
BUG: scheduling while atomic: ezdma_receive/2588/0x00000002
Modules linked in: ezdma(O)
CPU: 1 PID: 2588 Comm: ezdma_receive Tainted: G O 4.4.0-g6515388-dirty #7
Hardware name: Xilinx Zynq Platform
[] (unwind_backtrace) from [] (show_stack+0x10/0x14)
[] (show_stack) from [] (dump_stack+0x80/0xcc)
[] (dump_stack) from [] (__schedule_bug+0x44/0x60)
[] (__schedule_bug) from [] (__schedule+0x88/0x4c4)
[] (__schedule) from [] (schedule+0xb0/0xcc)
[] (schedule) from [] (schedule_hrtimeout_range_clock+0xc4/0x100)
[] (schedule_hrtimeout_range_clock) from [] (usleep_range+0x44/0x4c)
[] (usleep_range) from [] (xilinx_dma_terminate_all+0x98/0x154)
[] (xilinx_dma_terminate_all) from [] (ezdma_read+0x148/0x188 [ezdma])
[] (ezdma_read [ezdma]) from [] (__vfs_read+0x20/0xcc)
[] (__vfs_read) from [] (vfs_read+0x84/0xec)
[] (vfs_read) from [] (SyS_read+0x40/0x80)
[] (SyS_read) from [] (ret_fast_syscall+0x0/0x3c)
xilinx-dma 40400000.dma: Cannot stop channel df7614d0: 10008
Kernel panic - not syncing: Aiee, killing interrupt handler!

This is normal? is there a way to cancel a pending DMA operation gracefully?

Thank you for your time!
Javier.

[jeremytrimble]

@Arribas,

Thanks for the bug report. I haven't had much time to work on ezdma these days (and in fact I don't have any hardware to test on), but the immediate issue seems pretty clear -- Xilinx's driver is attempting to "schedule" (sleep) while we're in an atomic context (because I'm inside a spinlock when it is called).

I've pushed an (untested) candidate fix on the "issue8" branch (commit c706d1f) -- maybe you could let me know if that helps.

Best,
-Jeremy

[Arribas]

@jeremytrimble your fix worked!

Now the driver does not produce an exception and the program stops the read/write operation without any core dump. However, the xilinx_dma IP do not respond to the termination instruction.
I checked the AXI_lite DMA bus register activity with chipscope when the driver calls the xilinx_dma_terminate_all() function and it sends the correct register values for terminating the existing DMA transfer. Then, the function starts a polling period (here calls sleep()...), trying to read the DMA status register to verify the DMA channel stop. The DMA hardware never puts the status to stopped and then, the xilinx_dma_terminate_all() reports:

xilinx-dma 40400000.dma: Cannot stop channel df485050: 10008
xilinx-dma 40400000.dma: Cannot stop channel df485050: 10008

And, if I try to use again the DMA, it does not respond anymore. Neither the program or the driver hangs, but the DMA transfer never starts. Did you notice something similar? The IP I'm using is the one that comes with Vivado 2016.2. Is there any reference design that I can use to discard a hardware problem?

Thank you again!
Javi.

[jeremytrimble]

Javi,

I haven't used the AXI DMA as of Vivado 2016.2 yet. Not sure if I can help but maybe let me know what version of the xilinx_dma driver you're using -- I know they've changed it a good bit since I first wrote ezdma.

I'm not sure if the AXI DMA core is able to abort a DMA without receiving at least something on the S2MM AXI Stream channel (can't remember offhand at the moment). In the past when debugging AXI DMA stuff, I've often found it instructive to chipscope the Stream interface and see what's going on there.

[sonatique]

@Arribas
Hi,
Just wondering did you eventually solve your issues?
I am hitting the same here, in a similar configuration...
Any help appreciated!

@jeremytrimble : thanks for providing this piece of code in the first place. I am still fighting issues, but it looks promising!

[noelpedro]

@Arribas Are you still using EZDMA for gnss-sdr? Or are you using other driver?
Have u tried XIlinx udmabuf? or Xilinx dma-proxy?

Im currently getting the error that you had:
xilinx-dma 40400000.dma: Cannot stop channel df485050: 10008

Did you fix this?