Hello Juergen,
I understand that you have external to the evaluation board , an active loop filter and VCO, for this configuration to work, you should isolate and power down the on board active loop filter and VCO, while at the same time connect your external loop filter and VCO.
USING AN EXTERNAL ACTIVE LOOP FILTER AND VCO WITH THE ADF4108 EVALUATION BOARD.
To power down the OP184, do not connect the +15V supply( powers down the OP184), and remove R23 ( removes the OPBIAS voltage).
To power down the VCO, remove R27 .
The CPOUT SMA connects to your loop filter ( see fig 7... populate R6 with a zero ohm link and remove R1 and C1 in fig.10 to isolate the on board loop filter from CPOUT).
Connect the RFOUT from your VCO back to the RFIN of the PLL ( ADF4108)...with reference to fig 10 of the evaluation board, you can connect your RFOUT to the VCO/2 SMA bearing in mind that the signal level will be attenuated by 9 dBs from VCO/2 SMA to RFIN, as U9 is a 6dB power splitter and U12 is a 3dB attenuator. The signal level range at RFIN is from +5dBs to -5dBs, so these components can be left provided your signal level out of our VCO is adequate. Isolate the on-board VCO by removing C29 and populating R34 with a 50ohm resistor.
I hope this helps.
Brigid.