"Do nothing" in the "Display" dialog doesn't actually "do nothing"
Xfce's display management has always been a pile of junk on desktops with two or three or more monitors. There is a "Do nothing" option in the "Display" dialog which probably should make xfsettingsd actually do nothing when a display is turned on or off. Instead, it messes up the display configuration.
There could/should be an actual option to make xfsettingsd not touch or change the display configuration ever - which is what one might think "do nothing" would do.
For those who want a solution: just empty the if (event_num == RRScreenChangeNotify) function in displays-x11.c. you'd think compiling with --disable-xrandr would be a solution but it doesn't compile when that option is used thanks to a dumb change in common/Makefile.am so that's not an option.