Prevent windows move to external monitor on connect

I just got a bug (dunno if it presents only for me, or that's a well-known issue) that when I connect any external monitor - every window automatically moves to it. Even if the laptop monitor is set to Primary.