The patch from Olivier is actually quite interesting. I wish I had seen it before.
Rather than checking for wakeup events like in [xfce/xfce4-panel!88 (merged)] , Olivier's patch implements wall-clock timers, as opposed to the monotonic-clock timers that GLIB implements.
That has the advantage that it also handles other causes of "sudden" time changes. But I am not sure if it works as-is when the clock is set backwards (or if the TFD_TIMER_CANCEL_ON_SET flag would have to be set)?