AWS EC2 Instance Connect

Managing Secure Shell (SSH) access to systems can sometimes be a burden. The recommended secure standard is to utilize SSH keys to access systems. SSH keys provide a strong mechanism for controlling access to resources. In a typical scenario, an Administrator would create a public/private key pair and freely distribute the public key. But the private key should be guarded more discretely and selectively shared with trusted users that need access to resources.

The management of this process can get tricky. Administrators governing many systems and a large number of users can find themselves quickly overwhelmed by the management of SSH keys and this could lead to disastrous outcomes if the keys are compromised. Rotation of keys, accurate inventories and privileged access management are integral to an organization’s security layer that controls access to resources, assets and sensitive data.

A 2017 study on how well audits measure SSH security in environments, conducted by Venafi, a leading provider of machine identity protection, revealed a widespread lack of SSH audits. Over 400 IT security professionals participated in the study. The key results were:

1. Only a third (thirty-three percent) of respondents said auditors review SSH key rotation and retirement policies. Although SSH grants privileged access in the same ways that passwords do, they are rarely audited.
2. Less than half (forty-six percent) of respondents said auditors review the control of authorized key files. When SSH access is not limited to approved systems, attackers with SSH access can move easily across enterprise networks and remain undetected.
3. Just forty-three percent of respondents said auditors review their port forwarding policy. If port forwarding is not limited, malicious actors can use it to create encrypted connections that evade most security controls.
4. More than one-quarter (twenty-seven percent) of respondents said that none of these critical SSH best practices are audited. Without visibility into the efficacy of SSH security practices, organizations cannot accurately measure their security posture.

The fundamental takeaway here is that managing SSH keys is a lot harder than it appears, even among experienced security professionals and is clearly a blind spot for many organizations. Companies should strive to ensure a sustainable SSH policy that incorporates inventories, vulnerability identifications, remediation policies and audits. One method that can make this process less complicated and more streamline, if the organization is an Amazon Web Services patron, is AWS’s EC2 Instance Connect.

What is EC2 Connect?

In June of 2019, AWS released EC2 Instance connect. One of its biggest draw cards is that Administrators can control SSH access to instances using AWS Identity and Access Management (IAM) users and policies. Additionally, access to systems can be obtained by generating one-time, ephemeral keys each time an authorized user attempts to connect. The public key is sent to the instance’s meta data via an API call with the AWS CLI or the EC2 Instance Connect CLI and endures for 60 seconds before expiring. During the 60 second window, the authorized user can then SSH to the instance. All SSH connection logs will be recorded in Cloudtrail for connection audits.

This reduces the footprint of SSH keys and the operational burden that comes with SSH key management. It should be noted that EC2 Instance Connect is not a silver bullet but rather an additional method for organizations to continue developing their security awareness. It should also be noted that IAM identities with administrator access will be able to use this method without any additional policies associated. Administrator access to AWS accounts should always be confined to a select few, trusted privileged users. All other identities should be constrained by policies.

How Does it Work?

In this demo, I will be describing the process of setting up EC2 Connect on an instance in AWS. I’ve created an instance on AWS and assigned it a tag with the key of ec2connect and the corresponding value of True. This will be important when we look at the policy setup.

In the IAM dashboard, under the policies section, I’ve created a policy with the following permissions:

For the first permission, the Action parameter invokes the ec2-instance-connect:SendSSHPublicKey action. The Resource parameter allows the action against all instances in all regions (remember to use the account id number in place of "account-id") with the Condition parameter limiting it to instances with the tag key of ec2connect and value of True. The second permission allows for the ec2:DescribeInstances action against all resources. We will see this later when we need to list instances before attempting to connect.

One can now create a group and attach this policy to the group and then add users to this group. This will enable any user within this group to use the ec2connect method to connect to instances. In the demo’s case, I’ve created a group called ec2_connect, attached the policy, as above, called IAM_EC2InstanceConnect to the group and added my username to this group. My user will now be able to use the ec2connect method.

The instance I have provisioned is a version of Ubuntu. EC2 Connect is available for Amazon Linux and Ubuntu. The ec2-instance-connect package needs to be installed onto the instance. In the case of Ubuntu, this can be done with the apt-get install command:

sudo apt-get update
sudo apt-get install ec2-instance-connect

Once the package is installed, the following files will be in the /usr/share/ec2-instance-connect/ folder:

eic_curl_authorized_keys
eic_harvest_hostkeys
eic_parse_authorized_keys
eic_run_authorized_keys

These scripts are used by the ec2-instance-connect service to push the public key to the instance meta data. This service is invoked by the EC2 Instance Connect CLI or the AWS CLI command depending on the method used, which we will see later on.

At this point everything on the AWS side is configured. Let us now look at what is required on the client/user side attempting to make a secure connection to the instance.

Listing EC2 instances per region

In the below sections we will describe two connection methods. Both will need the AWS CLI installed and configured with a credentials file and a configuration file. Listing of EC2 instances will also require the AWS CLI. For more on AWS CLI configuration and credentials see configuring the AWS CLI.

Previously we discussed the policy allowing the ec2:DescribeInstances action. This enables users to get a listing of instances in a region of their choice. There are many ways to generate the output of a instance with the AWS CLI describe instance function, but in our case we need the instance ID per instance as that is what we will be connecting with. So for this case, the most practical method would be to list out the instances in a table format showing the name and the instance ID. The command to be used is:

aws ec2 describe-instances --filters Name=tag-key,Values=Name --query 'Reservations[*].Instances[*].{Instance:InstanceId,AZ:Placement.AvailabilityZone,ip_address: PrivateIpAddress, Name:Tags[?Key==`Name`]|[0].Value}' --output table --region af-south-1

This can be made even simpler by creating a bash function that takes in the region as an argument and adding it to your ~/.bash_profile file, like this:

listawsinstances() {
aws ec2 describe-instances --filters Name=tag-key,Values=Name --query 'Reservations[*].Instances[*].{Instance:InstanceId,AZ:Placement.AvailabilityZone,ip_address: PrivateIpAddress, Name:Tags[?Key==`Name`]|[0].Value}' --output table --region $1
}

This will list the instances, by invoking the listawsinstances function we created above, in the desired region in a table format like this:

The user now has a method to list the EC2 instances per region and obtain the instance ID which is required when making the connection.

Using the AWS CLI and the SSH command method

This method requires the AWS CLI installed and configured with a credentials file and a configuration file. You will also need a public and private key pair. This key pair can be generated by the user and does not need the usual stringent administrative requirements since it will be take on ephemeral qualities when used: the public key will only last 60 seconds in the instance meta data before expiring and usage is centralized by the AWS IAM service.

First use the AWS CLI to push the key to the instance meta data :

aws ec2-instance-connect send-ssh-public-key --instance-id i-0ddc130b08001c1cf --availability-zone us-east-1e --instance-os-user ubuntu --ssh-public-key file:///my_rsa_key.pub

On success, this will return the following :

{
"RequestId": "f87f3686-99c0-45f0-9908-7204d8e07ae1",

"Success": true
}

Using the EC2 Instance Connect CLI and the MSSH command method

The EC2 Instance Connect CLI provides a mechanism akin to the SSH command but also generates and publishes ephemeral public keys and establishes an SSH connection to the desired instance. There is no need to generate a key pair using this method as the CLI takes care of that for you. To install the command, use the pip tool:

pip install ec2instanceconnectcli

Once installed, the mssh command will now be available. Similar to the AWS CLI commands, configurations and credentials can be configured in the user’s home directory. For example, credentials can be configured in : ~/.aws/credentials and configurations can be configured in ~/.aws/config.

Once your configuration and credential files are in place (to specify a region that is not your default configuration, use the -r flag) you can run the following command with the intance ID, depending on the AMI operating system user. For the ubuntu user:

mssh ubuntu@i-0ddb130c08001c1cf

The default is the ec2-user. If your AMI uses this user, you can simply run:

mssh i-0ddb130c08001c1cf

Conclusion

The ability to securely connect to Linux instances with key pairs is vital for administrative purposes. However, the management of the key pairs poses security risks and as the Venafi study showed, this management is often overlooked. The EC2 Instance Connect method provides AWS customers with an alternative approach that lightens the burden of SSH key management and keeps access to EC2 instances bound to IAM.