Sometimes badly configured router at my work makes my ssh connections get refused by timeout. At the same time internet works okay in general. It appears that some QoS settings interfere with SSH. Since I don’t have access to router settings this has to be fixed on my machine. To remediate this I just add following contents to my ~/.ssh/config file.

Host *
  IPQoS 0x00

This works for both OS X and Linux