An attempt to document things is started here:
http://wiki.analog.com/resources/tools-software/linux-drivers/iio-transceiver/ad9361-customization
Also you can use the GUI and look at the attribute using the TIP documented here:
http://wiki.analog.com/resources/tools-software/linux-software/fmcomms2_advanced_plugin
Like always the driver is the best manual:
https://github.com/analogdevicesinc/linux/blob/xcomm_zynq/drivers/iio/adc/ad9361.c
The base API is documented here:
http://wiki.analog.com/resources/tools-software/linux-drivers/iio-transceiver/ad9361
If the bool attribute adi,2rx-2tx-mode-enable is not set you are in 1RX1TX mode.
-Michael