I don’t understand the background of your question.
I assume you already see the expected result in the IIO scope?
You set RX LO == TX LO, select a sample rate and filter settings.
Select a DDS tone(s).
And with antennas or loopback SMA cable you see the tones in the time and frequency domain.
Why are you asking me if this is the right method?
With that working you can move on two paths.
Add extra HDL for your baseband mod/demod or use a software only approach.
Start looking here:
https://github.com/analogdevicesinc/libiio/blob/master/examples/ad9361-iiostream.c
This is a simple example that receives and transmits IQ samples.
In this particular example the received samples are IQ swapped and transmitted back out.
Greetings,
Michael