SSH_AUTH_SOCK is not set when using gpg-agent ssh support

xfce4-session version 4.16
Ubuntu 20.04: http://ppa.launchpad.net/xubuntu-dev/staging/ubuntu
I have also manually compiled for the same result.

I've been debugging a bit trying to get gpg-agent with ssh working. When starting up $SSH_AUTH_SOCK is unset when using gpg-agent's ssh support. SSH expects $SSH_AUTH_SOCK to be the gpg-agent's ssh socket but it is unset at line 292 of xfsm-startup.c. On my system .xsessionrc already is setting SSH_AUTH_SOCK to be the gpg-agent's socket but 292 ends unsetting this value.

I see a check for gpgagent_ssh_enabled in the ssh-agent handling section at 279 but cannot find any place where SSH_AUTH_SOCK is then reset to be the gpg-agent's ssh socket.

I have ssh-agent set to Enabled, and gpg-agent set to enabled and I have forced ssh-agent to be gpg-agent. /compat/LaunchGNOME is false.

xfconf-query -c xfce4-session -p /startup/ssh-agent/enabled -n -t bool -s true
xfconf-query -c xfce4-session -p /startup/gpg-agent/enabled -n -t bool -s true
xfconf-query -c xfce4-session -p /startup/ssh-agent/type -n -t string -s gpg-agent

.xsessionrc:
unset SSH_AGENT_PID
if [ "${gnupg_SSH_AUTH_SOCK_by:-0}" -ne $$ ]; then
  export SSH_AUTH_SOCK="$(gpgconf --list-dirs agent-ssh-socket)"
fi