Core dump due to unhandled Xerror with dual screen

I've been experiencing xfwm4 restarts due to unhandled Xerrors. These happen mainly when I attach or detach my secondary screen. It's configured to extend my desktop.

Errors tend to happen when there is some window on the secondary screen and I turn it off (so disconnect it), but also happen when I turn it on and the system extends the desktop.

Here is the error message from xsession-errors:

(xfwm4:1351): Gdk-ERROR **: 11:20:33.986: The program 'xfwm4' received an X Window System error.
This probably reflects a bug in the program.
The error was 'BadColor (invalid Colormap parameter)'.
  (Details: serial 2181038 error_code 12 request_code 81 (core protocol) minor_code 0)
  (Note to programmers: normally, X errors are reported asynchronously;
   that is, you will receive the error a while after causing it.
   To debug your program, run it with the GDK_SYNCHRONIZE environment
   variable to change this behavior. You can then get a meaningful
   backtrace from your debugger if you break on the gdk_x_error() function.)

If I can help getting a better error for example by running xfwm4 synchronously please tell me and I'll try to collect the required information.