I have seen cases in the past of the part oscillating when the input of the AD8304 does not see an impedance of some sort of photodiode (mainly seeing a relay or some type of switch).
I think there are two options going forward:
1) Experiment with the input compensation and perhaps solve the oscillation problems. This seems a little open ended as you are trying compensate for an unknown impedance that is seen looking back towards the switch.
2) I'm wondering if a buffer(s) would help your situation? How far along are you in the design process? Could use something like the ADL5315 between the switch and the log amp? What you would have to do, I think, is after the switch, terminate the PD current into a some R. So the voltage on that resistor would be I_pd*R. Connect the VSET pin of the ADL5315 to the resistor such that is senses the voltage I_pd*R, and then connect the same resistor value from the INPT pin to ground on the ADL5315. This would mirror the photodiode current to the IOUT pin. You could then connect the IOUT pin to the input of the log amp. I am hoping that the ADL5315 would act as a buffer and the oscillations would be nulled.
Does that all make sense?