Windows reshuffle when switching workspaces

All windows in the dock reshuffle to a default order when I switch workspaces in XFCE. I end up having to press ctrl and reshuffle them to how I want them to be again, which is annoying.

I want it to behave similarly to the default window buttons plugin where when the drag and drop feature is enabled it remembers window order.