Hello,
Sideband suppression performance is a parameter of gain and phase imbalance between I and Q that are related to EVM performance in system level. By optimizing amplitude(gain) and phase more, then we can get better number on sideband suppression. However, around -50dB sideband suppression would be reasonable for that BB range and it would be enough for EVM at communication application. Please refer to below link.
http://www.analog.com/static/imported-files/circuit_notes/CN0134.pdf
For flatness over +/-24MHz signal band, there would be three factor mainly. Roll-off(sinx/x) from DAC, Baseband flatness of ADL5375-15, RF Flatness over 48MHz at 3GHz on ADL5375-15. It would be around -0.53dB from DAC roll-off at 24MHz. Approximately around 0.1dB at 24MHz from Baseband of ADL5375-15(Refer to Fig 64 in datasheet). Not much from RF Flatness (Refer to Figure 28 for RF flatness). So 3dB at +/-24MHz is big. It might be good to check DAC output and then move to Modulator to narrow down the issue.
Thanks.
Tony