Hi Yli,
No need to use two ISR's. Because the FS and the clock are locked (and probably routed to the same pins as in my situation) the data is received in both sports, even if just one is signaling the interrupt.
If you look into my code, you will see two interrupts because of the receive (2 sports, 16 channels) and transmit (2 ports 16 channels) implementation.
Since the sports do use DMA, you have plenty of time to handle the sport interrupts and there is no need for transferring buffers, just correcting the DMA setup.
The only difference from my code compared to what you need is that i send a deïnterlaced stream of 16 samples per channel (4 channels gives me 64 samples per FS). I have implemented the code like this to have efficiënt DMA transfer when using external DDR memory (for internal memory it would make any diffenernce since that memory is random access).