Pro-Tip – SSH_AUTH_SOCK, tmux and you

Problem Definition

I use tmux (nearly) every day. One of the roles it fulfills is as a remote session persistence manager. I login to a remote host, connect to the tmux session I've been using for the last week and do things. That tmux session may persist for up to a month. Often my laptop stays at the office for days at a time. Assuming there have been no wifi router reboot, ISP hiccup, OS update-related laptop reboot, dog snagging the power adapter out the side of the laptop, fuse on the power strip the power adapter for the laptop is plugged into blowing because the hedgehog is cold and wants to snuggle events, tmux sessions are exactly as they were when last I interacted with them!

The one exception: ssh--specifically, ssh-agent forwarding. Each request for agent forwarding is assigned a local socket named /tmp/ssh-blanrndthng/agent.$( pidof local end of the agent link ). This path is assigned to the environment variable, SSH_AUTH_SOCKET.

When a new tmux session or window is created, the main tmux process reads the environment of the initiating process. Now, the prime directive of programs such as tmux is that it must persist the user's session. The best way for it to achieve this goal is to take on some daemon-like characteristics such as dissociating itself from its parent process and becoming a ward of the init process. This facilitates the feature of being able to disconnect from a tmux client-session leaving the editor sessions, log tail windows, etc. in whatever state they are currently in.

:# ps f -o session,pid,ppid,cmd -s 26804,26761
 SESS   PID  PPID CMD
26761 26761     1 tmux
26804 26804 26761  \_ -bash
26804 28003 26804      \_ ps f -o session,pid,ppid,cmd -s 2680

When tmux creates a new session (usually a shell), the shell environment as that tmux process knows it is passed to that session which is the state of the environment when the master tmux process was created.

If I have an active/connected terminal connection from the laptop to this host, there is no problem. ssh works as expected. It is when the user disconnects from all sessions that things get out of sync. So, let's say, one of the aforementioned events has occurred. Everything is back up. Systems are rebooted. Blah blah blah. I do the ssh dance and am reconnected to the remote host, reattach the tmux client-session I was working with previously. Everything works great until ssh-agent comes into play: Error connecting to agent: No such file or directory.

Fuck.

<Ctrl>-Bc (new tmux window with properly set SSH_AUTH_SOCK)

git push

<Ctrl>-Bp (return to previous window)

<Ctrl>-D (exit shell/close window with unset SSH_AUTH_SOCK)

NOTE

Absolutely nothing is malfunctioning. There is nothing that tmux or ssh-agent are doing (or not doing) that they shouldn't be (or should be) doing. When a new window/session is created, tmux learns the environment of the calling process and passes it on. It has no mechanism (nor should it) to influence the environment of existing processes.

Solution

The problem can be summed up as: there is a dynamic element involved when creating a static session. The solution is to make this element (our erstwhile problem variable, SSH_AUTH_SOCK) another static element for existing or new sessions.

check & repair

• is there a functional agent configuration (test against existing SSH_AUTH_SOCK present by default when ssh-agent forwarding is used)
• no
  • do nothing
• yes
  • reset SSH_AUTH_SOCK to point to the (static) filesystem location (~/.ssh/ssh_auth_sock)
  • is this a functional agent configuration?
    • yes: do nothing
    • no: ln -sf ${SSH_AUTH_SOCK} $HOME/.ssh/ssh_auth_sock

~/.bashrc_agent

# vim: ft=sh

function _check_ssh_agent() {
        return $( ssh-add -l >& /dev/null )
}

function set_ssh_agent() {

        local SAS=${SSH_AUTH_SOCK}

        _check_ssh_agent &&
                local SSH_AUTH_SOCK=${HOME}/.ssh/ssh_auth_sock
                _check_ssh_agent ||
                        ln -sf ${SAS} $HOME/.ssh/ssh_auth_sock

        # recall, "||" and "&&" operate on the 0/non-0 property
        # of the called function's return value. If the check succeeds
        # with the alternative socket path, the "ssh-add" call returns
        # 0, so there is nothing more to do. It is only if the alternative
        # path does not have a functional agent that a non-0 value will
        # be returned.  "&&" proceeds if 0 is returned. "||" proceeds
        # if non-0 is returned, thus, "||" is the correct glyph to
        # use since we have additional work to do.
}

set_ssh_agent

Run the check on login

~/.bashrc

[...]
[[ -f ~/.bashrc_agent ]] && . ~/.bashrc_agent

This directive lets ssh client utilities know where to look:

~/.ssh/config

Host *
  IdentityAgent ~/.ssh/ssh_auth_sock

Dark Corners

The one bit that popped into my head as I was writing this: This sort of thing should never be done with login accounts used by multiple individuals. The key material belonging to the last individual to login to the account and who is still logged in will be what every tmux session for that account has access to.