Common attacks and defenses on vSphere

This post is a collection of public resources, techniques and knowledge around security on vShpere, ESXi and vCenter coupled with personal experience and insights acquired from security assessments.

The post is divided in the following sections:

Introduction

vSphere is a cornerstone of modern digital infrastructure. It is a popular platform used by many organizations with on-premises environments for hosting critical workloads. Virtualized infrastructure offers many benefits to organizations including and not limited to cost savings, improved scalability & flexibility and simplified management.

On the other hand, vSphere and its integration with other technologies introduces dangers to environments that rely on it. These dangers influenced by certain decisions on an architectural level as well as on the operational level of vSphere are amplified and can lead to severe impact on resilience and business continuity.

Terminology

This section includes product terminology that will help you quickly get up to speed with key components that consist this technology.

vSphere: VMWare’s (now Broadcom) virtualization platform / product family

Elastic Sky X integrated - ESXi: component of vSphere - bare-metal hypervizor that allows to run virtual machines (VMs) on physical server hardware

vSphere Client: GUI application that enables management of vSphere installation

VCenter Server Appliance - VCSA: component of vSphere - preconfigured virtual machine built on Photon OS used to control ESXi hosts and VMs

Vcenter Appliance Management Interface - VAMI: web portal for administering the VCenter Server Appliance (VCSA) - also known as vCenter Server Management Interface. By default, it is available on port 5480.

Common Security Gaps

This section describes common design and exposure flaws that increase the likelihood and the impact of compromise of vSphere infrastructure. In most cases, these gaps reflect a broader systemic issue: insufficient tiering of systems and inadequate enforcement of security controls across those systems.

MFA Not Enfroced on vSphere Client or VAMI

Second form of authentication is usually not configured on the sensitive management interfaces vSphere Client or the Vcenter Appliance Management Interface. This means that if a privileged account is compromised, the attacker will be able to access admin functionality using only credentials.

Local and Central Logging Not Enabled

There are different types of logs on vSphere infrastructure, detailed later in section Logging. However, if these logs are not enabled and if enabled they are only stored locally and not forwarded to a remote host and then ingested into a SIEM and monitored, this creates a blind spot that undermines troubleshooting as we as detection and response operations.

Intragrating with and Hosting Active Directory

In many production environments, vSphere is used to host Active Directory infrastructure. Active Directory in these environments is used as the identity platform. vSphere uses Active Directory to authorize access to itself. As a result, this creates a feedback loop in which a compromise of ESXi infrastructure cascades to the Active Directory, that in this case is a dependent service.

Similarly, when recovery systems such as backup servers or other sensitive systems used such as bastion hosts, jumphosts or management hosts are hosted on vSphere, a compromise of vSphere, cascades to these systems and makes recovery efforts challenging.

This scenario is covered in Google Cloud’s blog From Help Desk to Hypervisor: Defending Your VMware vSphere Estate from UNC3944 [2].

Active Directory user groups assigned with administrator privileges on ESXi or vCenter are often unmonitored. A threat actor with sufficient access could identify the Active Directory user groups that are used by vCenter or ESXi for authorization purposes and add attacker-controlled accounts in these groups for detection evasion, persistent access or other purposes.

Network Segregation

Quite often in production environments for ease of access purposes or other reasons, vSphere-related management interfaces are directly accessible from corporate network ranges. This indicates a “flat” network structure that allows any endpoint in the network to have line-of-sight access to systems that compose vSphere infrastructure. With line-of-sight attackers can then target the infrastructure with exploits such as CVE-2024-38812 and CVE-2024-38813.

Operational Knowledge & Skills Gap

A security gap is introduced when employees responsible for operating or securing vSphere are missing sufficient expertise. In such cases, security controls may be misconfigured, poorly monitored, or inconsistently enforced, and risky design decisions may go unrecognized. This skills and knowledge gap increases the likelihood of exposure, weakens incident detection and response, and can allow attackers to exploit issues that would otherwise be preventable through informed administration and governance.

Tactics on ESXi as Attack Vectors on Active Directory

This section documents various tactics that can impact vSphere infrastructure and potentially extend to Active Directory. The relationship between vSphere infrastructure and Active Directory has historically been exploited by threat actors who are looking to establish persistent access in a compromised environment, exfiltrate sensitive data or move laterally to other critical systems to maximize the impact of the compromise.

What is considered a tactic against a vSphere environment can transform into an attack vector for Active Directory. The following sub-sections describe such tactics.

The section discusses the following tactics:

vSphere as Launchpad for Further Attacks

Compromised vSphere infrastructure in the form of shell access to either an ESXi host or a vCenter Server, can be used by threat actors as launchpad for further attacks in an Active Directory environment. When it comes to monitoring and detections, vSphere is a blind spot. This is because antivirus or EDR software does not exist for the platform. As a result - similarly to edge devices - vSphere infrastructure can be abused to act as a staging platform for data exfiltration, as a gateway to other parts of the affected network or even as a host for persistent access to the affected network.

Additionally, attackers abuse native tools installed on ESXi hosts to achieve their objectives. The project located at [4] documents native binaries that can be abused for various purposes.

VM Disk Clone for Extracting AD Credentials

VM disk cloning is a tactic that accommodates lateral movement. By cloning a VM disk, the disk becomes available to attach to a different VM. This effectively allows access to the underlying filesystem. With access to the filesystem, files such as NTDS.dit - the Active Directory database - can then be extracted. This file can then be exfiltrated for offline password cracking. To clone a VM disk, the VM - on which the disk is originally attached to - needs to shut down.

This attack has been used by the ransomware group Akira [7].

Windows VM Memory Snapshot for Extracting Secrets from Memory

Snapshot of VM memory to extract credentials stored in memory - applicable to Active Directory (AD) servers including Domain Controllers (DCs) vmsd, vmsn

Logging

Logging is important and can help alert early to identify suspicious activity as well as troubleshout various issues. Broadcom has documented the location of log files for various Broadcom products in [27].

Log Sources

Logs are available on both ESXi and vCenter. There are three different types of logs on ESXi and vCenter. These are listed in the post From Help Desk to Hypervisor: Defending Your VMware vSphere Estate from UNC3944 by Google Cloud in [2]. For completeness, the logs are:

vCenter Server (VC Events) are documented in [28]. In general, these events operate at the management plane. A list of all vCenter Server Events is documented in [29].

ESXi Host Logs operate at the hypervisor level and are different per ESXi version. In general, they are located in the path /var/log on an ESXi host. Logs for ESXi version 7 are documented at [30] and for ESXi 8 at [31].

ESXi Host Audit Logs is not enabled by default. These logs are written to audit.*.log on ESXi’s filesystem. More information is available for vSphere 7 at [32], for vSphere 8 at [33] and vSphere 9 at [34].

Enabling Logs

Local Auditing can be enabled using the ESXCLI Python package, which offers the command-line interface utility esxcli. The utility can be used remotely or in the ESX shell. The following command shows how to enable Local Audit logs:

esxcli system auditrecords local enable

The underlying setting that controls the audit logs is Syslog.global.auditRecord.storageEnable = TRUE.

For more information, see the official documentation in [23] for ESXi 7, [24] for ESXi 8 and [25] for ESXi 9.

Detection Opportunities and General Hygiene Recommendations

Earlier, in section Common Security Gaps we discussed weaknesses that affect vSphere environments and create exploitation opportunities for threat actors. In this section we focus our attention on detection opportunities that allow defenders to protect vSphere infrastructure and general hygiene recommendations that will eventually limit the attack surface.

Unexpected VM Shutdown

Unusual activity or system behavior such as an unexpected shutdown of VMs that have a key role in an infrastructure, such as a Domain Controller, may indicate that an attacker is in the process of cloning a VM disk. It is therefore important to log and monitor these events to diagnose attacker activity.

Block ESXi & vCenter Internet Access

ESXi an vCentr shell access can be abused for purposes such as data exfilatration, as described earlier in section vSphere as Launchpad for Further Attacks.

changes to sensitive AD groups

This is a rather holistic recommendation that applies to any Active Directory environment, however it is explicitly called out in this post for vSphere, since many environments integrate vSphere with Active Directory. Changes of membership in sensitve groups that grant high privileges, such as ESXi administrators, should be monitored. Suspicious membership should be investigated and the root cause identified as soon as possible to limit the risk of unauthorized access.

vSphere execInstalledOnly

The vSphere security setting execInstalledOnly allows only signed VIB packages to run, preventing custom code from running. However, this controlled can be avoided by running Python scripts.

References

[1] https://cloud.google.com/blog/topics/threat-intelligence/vsphere-active-directory-integration-risks

[2] https://cloud.google.com/blog/topics/threat-intelligence/defending-vsphere-from-unc3944

[3] https://detect.fyi/vmware-esxi-logging-detection-opportunities-4fb56411ec21

[4] https://lolesxi-project.github.io/LOLESXi/#

[5] https://techdocs.broadcom.com/us/en/vmware-cis/vsphere/vsphere/7-0/vsphere-security-7-0/securing-esxi-hosts/managing-esxi-audit-records.html

[6] https://blog.ukotic.net/2020/08/06/enable-ssh-on-vcenter-server-7/

[7] https://www.itsecurityguru.org/2024/07/23/privilege-escalation-unravelling-a-novel-cyber-attack-technique/

[8] https://www.nakivo.com/blog/vmware-esxi-clone-vm/

[9] https://jamescoote.co.uk/Dumping-LSASS-with-SharpShere/

[10] https://www.trendmicro.com/en_gb/research/24/c/agenda-ransomware-propagates-to-vcenters-and-esxi-via-custom-pow.html

[11] https://www.trellix.com/blogs/research/ransomhouse-am-see/

[12] https://knowledge.broadcom.com/external/article/371714/faq-delete-all-snapshots-and-consolidate.html

[13] https://geek-university.com/delete-a-snapshot/

[14] https://www.truesec.com/hub/blog/secure-your-vmware-esxi-hosts-against-ransomware

[15] https://knowledge.broadcom.com/external/article/374277/how-to-disable-root-shell-access-on-esxi.html

[16] https://medium.com/@lubomir-tobek/vsphere-8-security-hardening-part-i-5b7c6cfd2224

[17] https://techdocs.broadcom.com/us/en/vmware-cis/vsphere/vsphere/9-0/vsphere-security.html

[18] https://techdocs.broadcom.com/us/en/vmware-cis/vsphere/vsphere/8-0/vsphere-security-8-0.html

[19] https://techdocs.broadcom.com/us/en/vmware-cis/vsphere/vsphere/7-0/vsphere-security-7-0.html

[20] https://knowledge.broadcom.com/external/article/342618/overview-of-virtual-machine-snapshots-in.html

[21] https://knowledge.broadcom.com/external/article/316623/configuring-the-esxi-host-with-active-di.html

[22] https://knowledge.broadcom.com/external/article/408693/configuring-local-audit-logging-esxi-hos.html

[23] https://techdocs.broadcom.com/us/en/vmware-cis/vsphere/vsphere-sdks-tools/7-0/esxcli-concepts-and-examples-7-0/managing-security/configuring-and-managing-the-audit-system-and-audit-data/enable-local-auditing-with-esxcli.html

[24] https://techdocs.broadcom.com/us/en/vmware-cis/vsphere/vsphere-sdks-tools/8-0/esxcli-concepts-and-examples-8-0/managing-security/configuring-and-managing-the-audit-system-and-audit-data/enable-local-auditing-with-esxcli.html

[25] https://techdocs.broadcom.com/us/en/vmware-cis/vsphere/vsphere-sdks-tools/9-0/esxcli-concepts-and-examples-8-0/managing-security/configuring-and-managing-the-audit-system-and-audit-data/enable-local-auditing-with-esxcli.html

[26] https://unit42.paloaltonetworks.com/ransomhouse-encryption-upgrade/

[27] https://knowledge.broadcom.com/external/article?articleNumber=322834

[28] https://knowledge.broadcom.com/external/article/312194

[29] https://www.virten.net/vmware/vcenter-events/

[30] https://techdocs.broadcom.com/us/en/vmware-cis/vsphere/vsphere/7-0/vsphere-security-7-0/securing-esxi-hosts/managing-esxi-log-files/esxi-log-file-locations.html

[31] https://techdocs.broadcom.com/us/en/vmware-cis/vsphere/vsphere/8-0/vsphere-monitoring-and-performance-8-0/system-log-files/collecting-log-files.html#GUID-832A2618-6B11-4A28-9672-93296DA931D0-en

[32] https://techdocs.broadcom.com/us/en/vmware-cis/vsphere/vsphere/7-0/vsphere-security-7-0/understanding-vsphere-hardening-and-compliance/audit-logging/single-sign-on-audit-events.html

[33] https://techdocs.broadcom.com/us/en/vmware-cis/vsphere/vsphere/8-0/vsphere-security-8-0/understanding-vsphere-hardening-and-compliance/audit-logging/single-sign-on-audit-events.html

[34] https://techdocs.broadcom.com/us/en/vmware-cis/vsphere/vsphere/9-0/vsphere-security/understanding-vsphere-hardening-and-compliance/audit-logging/single-sign-on-audit-events.htmls


tags:#vSphere - ESXi