I was thinking about the same but I guess that's the downsize of having an extra security.

Something to make it less painful is to use the ControlMaster ssh feature which should allow to reconnect to your server during your session (as long you have the first ControlMaster still connected).. tutorial how to use it randomly taken from google here http://goo.gl/26Oxx