Hi Mr. Wang,
Thanks for posting your problem. I suspect that there is a switching noise getting into your signal path that generates these frequency peaks. Here are some recommendations and questions for further debugging:
1- Try with the 50ohm resistors on the input and with them shorted
2- Try with the Vgain at min, mid, and max settings. You can also try to connect the GAIN+- pins together at the common mode voltage to set it to mid gain precisely. The noise on the GAIN+- could cause these spectral broadening around dc.
3- Inspect the switching supply noise if you have any switching supplies
4- If you have clock multipliers then the switching noise may be caused by them
5- Inspect any switching source, HV pulsers, FPGAs, uProcessors, etc. It is recommended to sync all clocks on the system to avoid switching noise.
Regards
Ashraf