Hi Kadir,
The AD6645 Data Outputs should be latched into your data-capture hardware by the rising edge of DRY during the TH_DR Hold-Time period shown in datasheet fig 2 below. If your data-capture hardware, cabling, FPGA receiver code, etc., has inadvertently introduced any delays or timing shifts between the DATAOUT and DRY signals you may actually be attempting to latch the DATAOUT before it has fully settled during the TS_DR Setup-Time period shown in fig 2. The fact that you're seeing occasional glitches at various analog input amplitudes and frequencies suggests you might be right on the edge with your DATAOUT Latch timing, so the glitches may be the result of occasionally latching transitioning, or otherwise unsettled ADC data outputs.
To fix this typically you would want to shift the rising edge of DRY so that it occurs more directly in the center of the optimum valid TH_DR latch window.The AD6645 Eval Brd includes a couple of gate delay options via Jumper E1 (BUFLAT can be driven from either DR_OUT or OPT_LAT) that can be selected to alter the timing relationships between DRY and DATAOUT. Our HSC-ADC-EVALB-DCZ capture hardware also includes additional gate provisions to further shift or invert the timing relationships between the data latching input DRY and the DATAOUT bits. Since you're using your own data capture hardware you might also be able to manipulate these DRY to DATAOUT timings in your hardware or FPGA code.
Best Regards,
Tony M