Leveraging AWS Systems Manager Inventory and Fleet Manager to Mitigate XZ Utils SSH Backdoor Exploit
Every so often an exploit is discovered that sends shock waves through the technology industry. This was certainly the case for the exploit discovered by a Microsoft Postgres developer, Andres Freund.
Every so often an exploit is discovered that sends shock waves through the technology industry. This was certainly the case for the exploit discovered by a Microsoft Postgres developer, Andres Freund.
Andres was experiencing a significant delay in authentication responses for openSSH. Little did he know that this annoyance would lead to the discovery of a significant security vulnerability in the XZ Utils library, formerly known as LZMA Utils. This revelation reads like a scene straight out of a spy movie.
The vulnerability emerged from a meticulously orchestrated supply chain attack, reminiscent of covert operations portrayed on the silver screen. Malicious actors infiltrated the team responsible for maintaining the library, clandestinely operating within its ranks over an extended duration.
The YouTube channel, Seytonic provides a good explanation of the exploit methods and drama.
Proactive Response and Due Diligence in the Face of Security Threats
By the time I was contacted by our organisations security officer, I had heard of the exploit and had briefly done some research which revealed that our systems, all of which run a single, uniform flavour of Linux, were not susceptible to the exploit. I was also confident in the level of expertise exercised on our cloud network; there was no way SSH was publicly exposed! Nonetheless, in a case like this, and most cases involving security issues, due diligence should be exercised.
Although there was confidence that our systems were not at risk I wanted ( and needed ) to prove this. Were all of our systems running a version of XZ Utils that was impervious to the exploit, even though SSH was not exposed to the public? I could logon to one or two systems and check the versions running and then assume this was the case on all our fleets but this would not be a diligent way of confirming whether all systems were safe. I had to devise a way to automatically extract the version from all our fleets.
Combining some python automation with AWS Systems Manager Inventory and Fleet Manager did the trick!
AWS Systems Manager Fleet Manager and AWS Systems Manager Inventory
The combination of these services provided a robust way of gaining access to all the fleets in an account and regions and extracting the version of the XZ Utils package from each instance. This method will confirm that the version on the instances are not compromised.
AWS Systems Manager Fleet Manager
AWS Systems Manager Fleet Manager is a feature within AWS Systems Manager that enables you to view and manage your managed instances (servers or virtual machines) through a single interface. It provides a dashboard for monitoring the health and status of your instances, simplifying tasks such as patch management, software distribution, and troubleshooting. Fleet Manager streamlines operations and enhances visibility into your managed instances.
AWS Systems Manager Inventory
AWS Systems Manager Inventory allows you to collect metadata about your AWS resources and their configurations in a centralised inventory. This metadata includes details such as operating system, installed applications, network configurations, and more. It helps you track changes, perform audits, and ensure compliance across your infrastructure.
Setting Up Systems Manager Inventory
Systems Manager Inventory is a tool utilised for gathering information regarding installed application packages from running EC2 instances. It works seamlessly alongside Systems Manager's Fleet Manager. If it is not setup, one would first need to configure it to gather the required metadata.
Browse to the Systems Manager Inventory console in the required region. Then click on the "Setup Inventory" button.
Then provide a name. The only other parameter requiring adjustment is the Schedule. The default interval is set to every 30 minutes, which may be more frequent than needed. Once finished, click "Setup Inventory".
Once configured, a Systems Manager Association should be readily visible within the State Manager console.
To ensure the Inventory is gathering information accurately, navigate to the Fleet Manager console, choose one of the EC2 instances, and then click on the Inventory tab. Here, you should see that the installed applications and their corresponding versions are now being collected.
Streamlining Version Extraction through Automation
With a method in place for viewing the application version (in this case, the xz-utils application) across instances, automation becomes essential to scale the process efficiently. Python and the Boto3 library offer an effective solution.
The Python Script
This script essentially checks all your instances across different AWS regions to determine if they have a specified software package installed. By default, it checks for the "xz-utils" package, but you can specify any package you want to check. After the check, it generates a CSV file summarizing the results.
opensauce17The CSV Output
Conclusion
In response to the threat posed by the exploit, organizations must adopt a proactive stance, combining expertise, and technology to mitigate risks effectively. Leveraging AWS Systems Manager Inventory and Fleet Manager, along with automation, offers a robust solution for monitoring and managing instances across diverse environments.
The combination of AWS Systems Manager Inventory and Fleet Manager provides a centralized approach to gathering metadata about AWS resources and managing instances efficiently. By setting up Systems Manager Inventory and configuring it to collect relevant metadata, organisations can ensure comprehensive visibility into their infrastructure.