The AD9540 software was create with visual basic, so I don't have any Matlab code to forward. That said, there's a few threads on DDS engineering zone that might help. See the link below for one. The SPI interface to the AD9540 is like most all other DDSs,
Here's some quick steps to program the AD9540 to see if you're communicating with the device.
Basically it turns off and on the SYNC_CLK.
1) Power up the part
2) Apply the REF CLK source
3) Send a master reset.(This places the internal registers in a known state)
4) Monitor the SYNC_CLK pin 18, it should be toggling.
5) Write to register 0x00h bit 4 (Disable SYNC_CLK) Consists of one byte for the instruction byte and 4 bytes of data so 40 SCLKs/SDIOs right?
6) Send an IO_UPDATE pulse. Note, the pulse must be wider that the SYNC_CLK period.
7) The SYNC_CLK should turn off. If so, you're communicating.
Hope this helps.