The ILOBleed Vulnerability and IPMI Network Security

On December 28th, 2021, I was snowed into my house and subsisting on cans of Campbell’s soup over the holiday interval. Scrolling Twitter, I came across an eyebrow-raising report from an Iranian cybersecurity firm called Amnpardaz. Amnparaz had announced that they had discovered the first known rootkit in a Hewlett-Packard Enterprise iLO baseboard management controller (BMC) and published an analysis of the infected firmware, as well as potential Indicators of Compromise (IoCs) that include both visible differences in the iLO management web interface login screen as well as MD5 checksums. Amnpardaz states in their report that they first observed this rootkit in the wild some time in 2020, and exclusively in Iran.

This report was picked up by TheHackerNews on December 30, but has thus far garnered relatively little attention in the wider industry. Knowing that HP iLO is widely used and how powerful a rooted baseboard management controller can be in the hands of an attacker, I wanted to highlight this vulnerability even though this activity has not been observed outside Iran.

This is because once an attack is discovered and publicized, the malicious code is often quickly shared with or replicated by third parties, and it may be used by a much wider range of threat actors in the future. It also brings attention to an important but rarely discussed area of network security: the Intelligent Platform Management Interface (IPMI), which is used primarily to remotely control server hardware.

Baseband management controllers and IPMI

Early in my career, I was the youngest employee of a startup and the only one without children. As a result, it was always my job to carry a pager – 24x7, 365 – and respond whenever one of our externally-facing servers crashed and needed to be rebooted. This always seemed to happen between 2:00 AM and 5:00 AM. And I often thought it would be really nice to be able to power servers on and off remotely, or at least force a reboot. Remote console access would be wonderful too. Oh – and just to top it all off I want to be able to mount removable media like DVD-ROMs so that I can reinstall the OS on the server if I have to!

Many other engineers felt the same pain as I did in the late 1990s, and that led to the development of the Intelligent Platform Management Interface, which is a common standard supported by over 0200 computer system vendors. IPMI allows system administrators to monitor and control the system hardware independently of the host system’s CPU, BIOS/UEFI firmware, and operating system.

This is done by putting what is essentially an entire second computer inside the host system, typically on a removable module with a serial or ethernet port for networked communications. (Some manufacturers use the same network interface for both IPMI and regular network traffic. This is not a configuration we recommend.)

This “second computer” is called a baseboard management controller (BMC), and it implements the IPMI standard as well as any additional features the manufacturer requires. It typically communicates “out of band” from the rest of the host system – it has its own physically independent network interface, storage, and processing capability.

In the case of the HP iLO product, the baseboard management controller is a PCI expansion card with a network or serial interface port that can be purchased with the device or sold separately. In general, any ProLiant 300-series system or higher comes with an iLO module.

IPMI Rootkit Vulnerability Risks

As you might imagine, any rootkit which operates at a lower level than the host system’s own OS is going to fly well below the radar of the normal antivirus/antimalware and endpoint detection tools, and gives the attacker an incredible degree of control over both the hardware and software of the host system. A baseboard management controller rootkit replaces the original firmware with a modified version that can be used by attackers to:

• Evade detection – iLO cannot be scrutinized by security tools
• Maintain persistence, even through server hard drive wipes and even through firmware upgrades of the baseboard management controller itself
• Modify or wipe the Integrated Management Log (IML) used to record BMC activity
• Execute console commands and modify OS files.

Leverage an API management interface, if one exists in that IPMI implementation. HP iLO offers an XML-based API for automated remote control that can be scripted via Remote Insight Board Command Language (RIBCL).

In the case of iLOBleed, this rootkit appears to have primarily been used to wipe data from servers, and conceals its presence by changing the product version number that appears in the product UI to reflect the latest upgrade when the user attempts a firmware upgrade, but not actually installing the new firmware. This ensures the rootkit continues to function and cannot be removed through normal means, while keeping the user from becoming suspicious.

Indicators Of Compromise

As its primary indicator of compromise, Amnparadaz noted that in version 2.55, HP updated the login splash page for the HP iLO management web interface with a new look and feel. The easiest way to check if your system is infected by this particular rootkit and verify that you are running the latest version (2.55).

If not, you’ll want to upgrade to that firmware version to take advantage of patches to known vulnerabilities in the firmware anyway – but it’s important to do this quickly, because when you upgrade the splash screen should change from the blue graphic on the left to the black and white graphic on the right:

If your iLO is running Firmware Version 2.55 and you’re still seeing the blue splash screen on the left, your iLO may be compromised by iLOBleed.

Note that the authors of this rootkit will probably update it to reflect the look and feel of iLO 2.55 in the very near future, now that its existence has been revealed to the world. They may already have. As a result, this IoC may not be very useful, and more direct methods are needed.

The normal way to verify whether a filesystem is infected with malware is to search for the malware’s unique MD5 hash. Unfortunately, HP does not enable end-users to read the iLO firmware directly. Amnparadaz’s report contains alleged MD5 hashes for both infected and original iLO firmware. Because HP does not allow the iLO firmware to be read normally, researchers used a series of known vulnerabilities in older iLO versions to dump the firmware and obtain its MD5 hash.

This makes it impractical for HP iLO users to scan for this vulnerability on their own. Amnparadaz offers a tool which they state is able to scan for this vulnerability, but they also state that it does so by exploiting several known vulnerabilities in the iLO firmware in order to perform read operations that are not normally permitted. This may violate your product warranty, and we do not recommend running this tool.

Instead, we suggest contacting HP Enterprise Support and asking them to address this issue.

IPMI Security Recommendations

Because of the level of control IPMI interfaces grant an attacker and the lack of visibility into the baseboard management controllers offered to network administrators, it is vital to implement stringent access controls, auditing and monitoring of activity and netflow data in the environment. These recommendations apply to any IPMI implementation, not just HP iLO.

1. Severely restrict access to this network – only system administrators who require access to the IPMI network to perform their work should be able to access it.
2. Do not connect your IPMI network directly to the Internet.
3. Follow strict network segmentation protocols.
4. Do not allow your IPMI network to route traffic to your “main” internal network, or other networks.
5. Dedicate the network to IT management systems exclusively.
6. Use a firewall and dedicated switch network to isolate the IPMI network from the rest of your environment.
7. Limit what traffic can pass in either direction through the firewall to specifically authorized host IPs and ports. Block egress – don’t let anything on the IPMI network reach out to the Internet.
8. Use Multi-Factor Authentication wherever possible.
9. Certificate-based MFA is supported by HP iLO.
10. Dell’s DRAC product supports a wider range of MFA options.
11. Secure the VPN connection that network admins who have access to the IPMI network use to perform their work remotely.
12. Keep VPN device firmware up-to-date (every 30 days)
13. Require app-based or token-based MFA on ALL remote sessions
14. SMS is not a secure transmission medium for MFA codes, as it is too easy to redirect SMS messages to multiple recipients without either sender or receiver being aware of it.
15. Stream logs from the baseboard management controllers (such as the IML log on HP iLO modules) to a central log collector for analysis.
16. Send alerts to IT staff if someone attempts to access a BMC or the IPMI network from a remote location.
17. Send alerts to IT staff when failed login attempts occur on any BMC.
18. BMCs typically ship with default passwords. These should be changed before the device is put into service, and changed periodically Dell DRAC modules have a single well-known default password that has never been changed. HP iLO modules each have a unique password that is generated at manufacturing time and printed on a tag that ships with the module.
19. Manage and audit what users have access to the IPMI network and your baseboard management controllers.
20. Audit user access regularly to ensure that there are no ‘stale’ unused accounts or individuals who have more system access than is required to perform their role.

References:

HackerNews article on iLOBleed https://thehackernews.com/2021/12/new-ilobleed-rootkit-targeting-hp.html

Amnpardaz Threat Report: Implant.ARM.iLOBleed.a https://threats.amnpardaz.com/en/2021/12/28/implant-arm-ilobleed-a/

HP iLO Product Page https://www.hpe.com/us/en/servers/integrated-lights-out-ilo.html

CISA Alert: Risks of Using IPMI https://www.cisa.gov/uscert/ncas/alerts/TA13-207A