Securely record SSH sessions on RHEL in a private VPC network

September 15, 2023

In this blog post, you’ll learn how to record SSH sessions on a Red Hat Enterprise Linux (RHEL) VSI in a private VPC network using in-built packages. The VPC private network is provisioned through Terraform and the RHEL packages are installed using Ansible automation. Additionally, you’ll learn how to set up a highly available bastion host.

What is session recording and why is it required?

A bastion host and a jump server are both security mechanisms used in network and server environments to control and enhance security when connecting to remote systems. They serve similar purposes but have some differences in their implementation and use cases. The bastion host is placed in front of the private network to take SSH requests from public traffic and pass the request to the downstream machine. Bastion host and jump servers are vulnerable to intrusion because they are exposed to public traffic.

Session recording helps an administrator of a system to audit user SSH sessions and make sure they comply with regulatory requirements. In the event of a security breach, the administrator will want to audit and analyze the user sessions. This is critical for a security-sensitive system.

What is a private VPC network?

A virtual private cloud is completely private if there is no public ingress or outgress network traffic. In simple technical terms, it’s private if there are no public gateways on the subnets (private subnets) and no floating IPs on the Virtual Server Instances (VSIs).

How do I connect to the private VPC network?

Client-to-site VPN for VPC is one of the two VPN options available on IBM Cloud, and it allows users to connect to IBM Cloud resources through secure, encrypted connections.

The client-to-site VPN is highly available, with two VPN servers that are created in two different availability zones in the same region. The bastions are highly available as well.

Prerequisites

Provision the private VPC network using Terraform

Once you have the IBM Cloud Secrets Manager secret with the certificate, launch your terminal and set the following Terraform variables:

export TF_VAR_ibmcloud_api_key=<IBM_CLOUD_API_KEY> export TF_VAR_secrets_manager_certificate_crn=<SECRET_CRN>

git clone https://github.com/VidyasagarMSC/private-vpc-network cd terraform

Run the Terraform commands to provision the VPC resources (e.g., subnets, bastion hosts (VSIs), VPN, etc.):

terraform init terraform plan terraform apply

Connect to client-to-site VPN

Once the VPC resources are successfully provisioned, you need to download the VPN client profile by navigating to VPN servers page on IBM Cloud.
Click the Client-to-site servers tab and then on the name of the VPN:

Download the profile from the Clients tab.
The VPN provisioned through Terraform uses certificates. Follow the instructions here to connect to the OpenVPN Client.
You should see the successful connection on your OpenVPN Client:

Verify the SSH connection

On a terminal, add the SSH private key to the SSH agent with the following command:

ssh-add <LOCATION_OF_PRIVATE_SSH_KEY>

Example: ssh-add ~/.ssh/<NAME_OF_THE_PRIVATE_KEY>
Run the following command to SSH into the RHEL VSI through a bastion host. You will be using the private IP address of the bastion in Zone 1:

ssh -J [email protected] [email protected]

Remember, you should be connected to the client-to-site VPN to access the RHEL VSI through the bastion host.
After SSH, You should see instructions to enable SSH session recording using the TLOG package on RHEL.

Deploy session recording using Ansible

To deploy the session recording solution, you need to have the following packages installed on the RHEL VSI:

tlog
SSSD
cockpit-session-recording

The packages will be installed through Ansible automation on all the VSIs—both bastion hosts and RHEL VSI.

Move to the Ansible folder:

cd ansible

Create hosts.ini from the template file:

cp hosts_template.ini hosts.ini

Run the Ansible playbook to install the packages from an IBM Cloud private mirror/repository:

ansible-playbook main_playbook.yml -i hosts.ini --flush-cache

You can see in Figure 1 that after you SSH into the RHEL machine, you will see a note saying: ATTENTION! Your session is being recorded!

Check the session recordings, logs and reports

If you closely observe the messages post-SSH, you will see a URL to the web console that can be accessed using the machine name or private IP over port 9090. To allow traffic on port 9090, in the Terraform code, change the value of allow_port_9090 variable to true and run terraform apply. The latest terraform apply will add ACL and security group rules to allow traffic on port 9090.

Now, open a browser and navigate to http://10.10.128.13:9090. To access using the VSI name, you need to set up a private DNS (out of scope for this article). You need a root password to access the web console:

Navigate to Session Recording on the left-hand side to see the list of session recordings. Along with session recordings, you can check the logs, diagnostic reports, etc.:

Conclusion

This article covered why session recording is required in bastion hosts for auditing and compliance and how session recording can be set up with the built-in RHEL packages using Ansible Automation.

While designing a secured virtual private cloud network, you learned the best practices in architecting a VPC private network. We also covered the need to build highly available VPN servers and bastion hosts. With the provisioning of cloud infrastructure using Terraform and Ansible for session recording, you got hands-on experience.

Learn more about IBM Cloud VPC

If you have any queries, feel free to reach out to me on Twitter or on LinkedIn.