SSH Summer School: jump hosts and file transferring

Written by ACA Group Team | Oct 9, 2018 12:00:00 AM

This blog post is the final part of a 3 part series about SSH (Secure Shell) connections with OpenSSH. In this last part of the series, I’ll talk about jump/bastion hosts, using ‘screen’ and transferring files. The aim of this blog post series is to teach you more about SSH. In these blog posts, I assume you are using a UNIX-like operating system with OpenSSH.

Let’s start off by talking about jump/bastion hosts and why they’re so handy. It’s becoming more and more common that the remote server you want to access isn’t directly reachable. Instead, you need to pass through a jump or bastion host. And that’s a good thing, because:

application servers, most of the times, should not be exposed directly to the internet.
if you are using IPv4, there is no need for a public IPv4 address for each of your servers.
only a single host’s firewall needs to be configured to allow your office IP, which saves a lot of work when this IP changes.

Passing through a jump host can be annoying because you need to SSH through multiple servers. Luckily, there are multiple ways to do this. I’ve listed 4 methods below!

How to pass through a jump host

1. 'Manually'

One method of passing through a jump host is to SSH into the jump host and then from the jump host SSH into your server.

user@local-computer $ ssh jumphostuser@my-jump-host
jumphostuser@my-jump-host $ ssh remoteserveruser@my-remote-server
[remoteserveruser@my-remote-server ~]$

You’ll soon come to the conclusion that connecting to your remote server from the jump host requires an SSH key on the jump host. A solution could be to create an SSH key on the jump host, but this isn’t an option for jump hosts that are replaced on a regular basis. This probably isn’t a suitable method for most of us, so let’s go on to the next method. Oh, and by the way: never copy your local SSH key to another host such as a jump host!

2. Agent forwarding

I have already explained the ssh-agent concept in the first blog post of this series. With agent forwarding, you can expose your local ssh-agent on a remote server, such as a jump host. You can either use the command line option -A or the config file option ForwardAgent.

user@local-computer $ ssh-add
Enter passphrase for /Users/janbeerden/.ssh/id_rsa:
Identity added: /Users/user/.ssh/id_rsa (/Users/user/.ssh/id_rsa)
 
user@local-computer $ ssh -A jumphostuser@my-jump-host
jumphostuser@my-jump-host $ ssh remoteserveruser@my-remote-server
[remoteserveruser@my-remote-server ~]$

By doing this, you no longer need an intermediary key. Your account on the jump host and the remote server should only contain the public part of your local SSH key. Easy enough, right?

However, because you are exposing your local ssh-agent to the jump host, the jump host can use your SSH key without needing the passphrase. If you are the only one using (and maintaining) this jump host, you’re probably fine. If that isn’t the case (probably most of the times), someone could use your key while it’s being forwarded to the jump host.

When starting an SSH connection with agent forwarding enabled, the SSH server will create a socket to expose your ssh-agent. Only your user on the jump host will have permissions on this socket, but these restrictions do not apply to the root user or any user that has enough permissions via sudo.

3. ProxyCommand

With the ProxyCommand option, you can specify a command to be used to connect to the remote server. This option can be used to ‘chain’ SSH connections. By using “ssh -W” as ProxyCommand, we request that standard input and output on the client is forwarded to Hostname over the secure channel to my-jump-host.

Host my-jump-host
  Hostname 192.168.0.10
  User jumphostuser
 
Host my-remote-server
  Hostname 192.168.5.10
  User remoteserveruser
  ProxyCommand ssh my-jump-host -W %h:%p

As you can see, this requires some configuration, but that only needs to be done once.

user@local-computer $ ssh my-remote-server
[remoteserveruser@my-remote-server ~]$

Once this configuration is in place, you can ssh ‘directly’ in to your remote server. Passing through the bastion host is taken care of by the ProxyCommand.

Since OpenSSH 7.3, there is an easier to configure alternative to ProxyCommand called ProxyJump.

Host my-jump-host
  Hostname 192.168.0.10
  User jumphostuser
 
Host my-remote-server
  Hostname 192.168.5.10
  User remoteserveruser
  ProxyJump my-jump-host

As you can see, this option’s configuration is a lot simpler.

user@local-computer $ ssh my-remote-server
[remoteserveruser@my-remote-server ~]$

Connecting to your remote server is no different from the ProxyCommand method. If you want to connect to a remote server without having to add it to your config file via a jump host (that you have added to your config file), ProxyJump is also very simple.

user@local-computer $ ssh remoteserveruser@192.168.0.20 -J my-jump-host

When you share an SSH config file with your team, you can also include the config shown above. They will certainly appreciate not having to create or add the configuration themselves!

I recommend this method instead of the others because:

method 1 is too much effort each time you want to SSH to your remote server,
method 2 is a security risk and still requires multiple steps each time you want to SSH to your remote server,
method 3’s syntax is more complex and thus less easy to remember than the ProxyJump method.

With methods 3 and 4, you will also have the benefit of being able to transfer files directly to your remote server using scp or sftp without having to copy them to the jump host first. But how exactly do you start transferring files then? We’ll dive into the details of file transfer, but first let me tell you about using ‘screen’.

Using 'screen'

Since each jump host introduces an extra point of failure, now is the perfect time to start using ‘screen’. Have you ever had your SSH session disconnect while running a lengthy and/or delicate process on a remote server? When your SSH session gets disconnected, all processes started in the session that are running on the foreground will be terminated. This will require you to start all over or, depending on the process, leave you with a mess to clean up manually. Luckily, you can prevent this from becoming a problem by using ‘screen’. From its man page:

Screen is a full-screen window manager that multiplexes a physical terminal between several processes (typically interactive shells).

Some benefits of using screen:

your shell remains active when your SSH session gets disconnected,
you can run lengthy processes on a remote server without the need of maintaining an active SSH connection,
you can use multiple windows from a single SSH session.

First of all, screen needs to be installed on your remote server. If screen isn’t installed yet, you can use your systems package manager to install it.

Starting a new screen

Starting a screen is very simple:

[remoteserveruser@my-remote-server ~]$ screen

Screen will create a new shell, similarly as SSH’ing into a server would.

Disconnecting from a screen

The difference from a ‘standard’ shell is that you can disconnect from it, without killing all the sub processes. First you need to enter the escape command, which is CTRL+a, followed by CTRL+d

[detached from 4389.pts-0.my-remote-server]
[remoteserveruser@my-remote-server ~]$

Screen will tell you it detached you from a screen session. In this case, the session is called 4389.pts-0.my-remote-server. If you want to exit the shell instead of disconnecting for the screen session, you can do so as if it is any other shell. Enter:

exit
CTRL + d
...

However, be careful! Screen does not protect you from accidentally exiting your shell.

Listing active screen sessions

With the -ls option, you can list all active screen sessions.

[remoteserveruser@my-remote-server ~]$ screen -ls
There are screens on:
    4405.pts-0.my-remote-server (Detached)
    4389.pts-0.my-remote-server (Detached)
2 Sockets in /var/run/screen/S-remoteserveruser.

Each line consists of the process id and the session name, separated by a period.

Starting a new screen with a custom session name

As you can see from the output above, screen sessions can be hard to identify. Therefore, it might be wise to name your screen session when starting it:

[remoteserveruser@my-remote-server ~]$ screen -S server-upgrade
[remoteserveruser@my-remote-server ~]$ screen -ls
There are screens on:
        4475.server-upgrade     (Attached)
1 Socket in /var/run/screen/S-remoteserveruser.

Attaching to a detached screen session

When you want to re-attach to a screen session, you can do so with the -r option. This option takes either the process id or the session name as parameter.

[remoteserveruser@my-remote-server ~]$ screen -r server-upgrade

Multiple windows in a single screen session

Each of the following subsections assume you are attached to a screen session.

Creating a new window inside your screen session

By issuing CTRL+a followed by CTRL+c, you create a new window.

List of open windows

By issuing CTRL+a followed by CTRL+w, you get a list of open windows.

[remoteserveruser@my-remote-server ~]$
 
 
0$ bash  1-$ bash  2*$ bash

As you can see from the output above, I have 3 open windows, identified by a number followed by its title.

Switching between windows

There are 4 (5) ways to switch between windows:

switch to a specific window
1. CTRL+a followed by the number of the window
2. CTRL+a followed by “, this method is probably the easiest since it presents you with an interactive window selector. By using the arrow keys, you can select the desired screen and press the return key to select it.
switch to the next window: CTRL+a followed by n
switch to the previous window: CTRL+a followed by p
switch to the last window: CTRL+a followed by o

Other screen key bindings

There are several keys bindings that can be used inside a screen session, after issuing the control command (CTRL+a). You can get an overview of the available key bindings by pressing the ? key. Hit CTRL+a first, then hit the ? key.

                     Screen key bindings, page 1 of 2.
 
                     Command key:  ^A   Literal ^A:  a
 
break       ^B b         license     ,            removebuf   =
clear       C            lockscreen  ^X x         reset       Z
colon       :            log         H            screen      ^C c
copy        ^[ [         login       L            select      '
detach      ^D d         meta        a            silence     _
digraph     ^V           monitor     M            split       S
displays    *            next        ^@ ^N sp n   suspend     ^Z z
dumptermcap .            number      N            time        ^T t
fit         F            only        Q            title       A
flow        ^F f         other       ^A           vbell       ^G
focus       ^I           pow_break   B            version     v
hardcopy    h            pow_detach  D            width       W
help        ?            prev        ^H ^P p ^?   windows     ^W w
history     { }          quit        \            wrap        ^R r
info        i            readbuf     <            writebuf    >
kill        K k          redisplay   ^L l         xoff        ^S s
lastmsg     ^M m         remove      X            xon         ^Q q
 
 
                [Press Space for next page; Return to end.]

One of my favorites is copy. As the name gives away, it can be used to copy text from one window to another. For a complete overview and explanation of the available command line options and key bindings, please have a look at the man page.

Transferring files

As promised above, we’ll touch upon transferring files. There are a number of methods to transfer files to, from and between remote hosts. In this section, I’ll go over file transfer via SCP, SFTP and SSH ‘streaming’. All three of these methods also work when using a jump host when you use the ProxyCommand or ProxyJump directive as described above.

SCP

scp or secure copy allows you to securely copy files from, to or between remote hosts.

From

user@local-computer $ scp my-remote-server:~/file.txt ~/Downloads/local/

To

user@local-computer $ scp ~/Downloads/local/file.txt my-remote-server:~/

Between

user@local-computer $ scp my-remote-server:~/file.txt my-other-remote-server:~/

For more information about using scp, have a look at it’s man page.

user@local-computer $ man scp

sftp or secure file transfer protocol is an interactive file transfer program similar to ftp which allows you to securely copy files to or from a remote server. By entering the “?” command you will get a list of possible commands.

user@local-computer $ sftp my-remote-server
sftp> ?
Available commands:
bye                                Quit sftp
cd path                            Change remote directory to 'path'
chgrp grp path                     Change group of file 'path' to 'grp'
chmod mode path                    Change permissions of file 'path' to 'mode'
chown own path                     Change owner of file 'path' to 'own'
df [-hi] [path]                    Display statistics for current directory or
                                   filesystem containing 'path'
exit                               Quit sftp
get [-afPpRr] remote [local]       Download file
reget [-fPpRr] remote [local]      Resume download file
reput [-fPpRr] [local] remote      Resume upload file
help                               Display this help text
lcd path                           Change local directory to 'path'
lls [ls-options [path]]            Display local directory listing
lmkdir path                        Create local directory
ln [-s] oldpath newpath            Link remote file (-s for symlink)
lpwd                               Print local working directory
ls [-1afhlnrSt] [path]             Display remote directory listing
lumask umask                       Set local umask to 'umask'
mkdir path                         Create remote directory
progress                           Toggle display of progress meter
put [-afPpRr] local [remote]       Upload file
pwd                                Display remote working directory
quit                               Quit sftp
rename oldpath newpath             Rename remote file
rm path                            Delete remote file
rmdir path                         Remove remote directory
symlink oldpath newpath            Symlink remote file
version                            Show SFTP version
!command                           Execute 'command' in local shell
!                                  Escape to local shell
?                                  Synonym for help

For more information about using sftp, have a look at it’s man page.

user@local-computer $ man sftp

‘Stream’ through SSH

Let’s say you need an archive of the log files on your remote server, but you don’t have enough space on that server to create the archive. Instead of creating the archive on the remote server and then copying the archive, you could also ‘stream’ it to your local machine. Just like scp, this method can be used to copy files from, to or between remote servers.

From

user@local-computer $ ssh remoteserveruser@my-remote-server "tar -zcf - /var/log/" > ~/log-archive.tgz

To

user@local-computer $ tar -zcf - /var/log/ | ssh remoteserveruser@my-remote-server "cat - > ~/log-archive.tgz"

Between

user@local-computer $ ssh remoteserveruser@my-remote-server "tar -zcf - /var/log/" | ssh remoteserveruser@my-remote-server "cat - > ~/log-archive.tgz"

In the examples above you will notice I use a dash “-” as the file name, this is the way of instructing tar, cat and several other programs to write to standard output (stdout) or read from standard input (stdin).
And there we have it! As always, if you have any questions or remarks, let us know.

View full post

SSH Summer School: jump hosts and file transferring

How to pass through a jump host

1. 'Manually'

2. Agent forwarding

3. ProxyCommand

4. ProxyJump (I recommend this one!)

Using 'screen'

Starting a new screen

Disconnecting from a screen

Listing active screen sessions

Starting a new screen with a custom session name

Attaching to a detached screen session

Multiple windows in a single screen session

Creating a new window inside your screen session

List of open windows

Switching between windows

Other screen key bindings

Transferring files

SCP

From

To

Between

‘Stream’ through SSH

From

To

Between