A Bastion or jump host has been THE solution for a long time when it comes to enabling access to servers/services that are not exposed to the Internet.
In this article, we’ll see how to totally get rid of those dinosaurs, increase security and even make it easier to do your work.
The old thinking is simple enough - use a dedicated Linux host. Allow SSH access from the Internet to that host, allow access to your private networks from that host. Perhaps jump some hoops when it comes to hardening this exposed host and set up filtering on the IP level to some IP’s you regularly use. Done and done.
Fair enough, then let the sysadmins that need access to private systems go via this Bastion/Jump Host.
Why is this bad?
This approach, as you probably are aware of, leads to some problems, for example:
You need to expose your Bastion host to the Internet on port 22. This is an attack vector.
You need to handle SSH keys. Either add a public key for every sysadmin, or you end up sharing a private key (even though you know it's not supposed to be handled like that).
SSH jumping, generally you don’t want to do work on the Bastion, then you have to SSH again to the target you want to reach. Workstation -> Bastion -> target, makes for some good complexity when wanting to fetch a file and extra cognitive load.
Public SSH keys - offboarding, do you really keep track of who has the private key for each public key present in your authorized_keys file in your Bastion host? In the case with a shared private key you face the risk of having the key compromised, and spread to people that should not have access to. And how would you know if that has happened?
Let’s fix it
If our systems are in AWS (EC2) we can solve this elegantly. By use of the quite well hidden, oddly named service, AWS Systems Manager - Session Manager.
With AWS Systems Manager - Session Manager (!) you can get rid of your Bastion. Yes, you can just remove it, easy as that.
Session Manager described by AWS:
Session Manager is a fully managed AWS Systems Manager capability that lets you manage your Amazon Elastic Compute Cloud (Amazon EC2) instances, on-premises instances, and virtual machines (VMs) through an interactive one-click browser-based shell or through the AWS Command Line Interface (AWS CLI).
It lets you connect to a shell in your EC2 through the AWS Web Console or your terminal via AWS CLI. Traffic goes through AWS and is encrypted via TLS 1.2.
What does it give you? It gives you a shell on your EC2 machine via the AWS console. IAM needs to allow this, see [5].
Head to your instance:
You can also get the same experience in your local terminal by using AWS CLI [6]- aws ssm start-session --target <your-aws-instance-id>
Wait, how is this authenticated?
You authenticate as you usually do when getting into the AWS Console, and for the command line you authenticate and set up the environment as you usually do.
And you can set it up so you can create a session directly to the host you want to connect to, no more jumping through one host to reach another one. This is true even if your hosts are in private subnets.
What do you get
Encryption as this communication is made through the AWS TLS endpoints.
Auditability can be set up to log the content of the sessions, either to CloudWatch or S3.
Control, you use IAM to control which users can use SSM, and on which hosts.
Offboarding ease, since you don’t have specific credentials for this, there is no such thing to offboard.
But wait, this is not SSH
What we have so far is something that looks like an SSH session but is not. You get a shell on your target systems.
You can run your commands. A lot of management tasks can be taken care of.
But there are some limitations, a Session Manager session does not allow you to tunnel traffic to a different host than the one you're connected to. Maybe, for example, you may want to be able to connect to your RDS database from your workstation.
Workstations -> Bastion -> RDS-DB
Didn’t I say you can scrap the Bastion?, yet there it is.
AWS Session Manager session doesn't let you do that.
Also, you can’t use SCP to copy files with Session Manager as we have it now.
But there is a way, you can run SSH over Session Manager.
This solution adds a fair bit of complexity, so please consider if you really need this. But you don’t need the Bastion, puh.
Still here? good, let's do it.
Set up SSH over Session Manager
Configure your local (on your workstation) SSH to use Session Manager
(but only when trying to connect to something beginning with i- like an AWS EC2 instance) by updating your SSH config.
~/.ssh/config. Make sure it contains:
# SSH over Session Manager host i-* mi-* ProxyCommand sh -c "aws ssm start-session --target %h --document-name AWS-StartSSHSession --parameters 'portNumber=%p'"Then you can ssh to your instance by doing `ssh <your-instance-ssh-user>@<your-aws-instance-id>`
SSH will be proxied over Session Manager.
But hold on, aren't we back to the ssh key handling mess again?
We are running or trying to run SSH over Session Manager, meaning that we tunnel SSH over the out-of-band AWS endpoints.
SSH is running without exposing port 22 to the Internet.
We have much better security that way, so sharing private ssh keys is not that bad then right?
No, no, no we don’t want to share private ssh keys. (see what I did there?).
Use temporary enabled public SSH keys
Instead push your public SSH key to the host you want to connect to, and let it sit there only for a short while.
Once again AWS magic can help, ec2-instance-connect.
aws ec2-instance-connect send-ssh-public-key --instance-id <your-aws-instance-id> --instance-os-user <your-instance-ssh-user> --ssh-public-key file://~/.ssh/id_rsa.pub --availability-zone <ec2-instance-az>Not the most beautiful combination of letters, but it really helps us keep things secure.
It uses AWS APIs to enable your public SSH key for this instance and user (How it's done: [2])
Now you can push whatever public key you want to use to the EC2 instance and start a SSH session.
Public key remains only for 60 seconds.
How we have SSH over Session Manager working, with temporary public SSH keys at the target.
SCP now works because it builds on SSH, and we have real SSH.
Tunneling works as it usually does with SSH, something like this will be fine:
ssh <your-instance-ssh-user>@<your-aws-instance-id> -L 3306:my-rds-connection-string:3306 -N
Ifs and buts
Depending on your setup you might bump into some things.
If you use pretty recent AWS AMI as base for your systems you’ll have a ssm agent installed that supports those operations.
If not you may have to install or update the ssm-agent. See [2]
If you plan to use ec2-instance-connect, with a recent AWS AMI you’re ok,
but in other cases you might have to change config for the SSH as described in [4]
Summary
AWS Systems Manager - Session Manager helps us to easily get a shell on an EC2 instance. Without exposing port 22 to the Internet.
Straight from the AWS Console or AWS CLI.
If real SSH is needed it can be tunneled over AWS Systems Manager - Session Manager enabling all the functions we are used to with SSH.
The result is, no Bastion, no exposure of port 22 to the Internet.
And best of all, No shared private SSH keys needed.
References: