Problem: VMware announced potential in-guest data inconsistency issue happen at virtual machine disks (VMDKs) that has been extended while running on VMware vSAN 6.6, 6.7 and later storage which ships with ESXi 6.5.0d and later including vSAN 6.6 (vSphere 6.5 EP2), vSAN 6.6.1 (vSphere 6.5U1), vSAN 6.6.1U2 (vSphere 6.5U2) and vSAN 6.7.
Content Summary
Workaround
Alternative
Workaround for vSAN on VMware Cloud Foundation 2.3.2 and 3.0
Patch Released
Reference
Workaround
VMware recommends applying the configuration changes to set the advanced setting in-place expansion VSAN.ClomEnableInplaceExpansion to 0 on all ESXi hosts that are part of the vSAN cluster including witness host in case of ROBO or stretch cluster.
Step 1: Log in to ESXi host through SSH as root.
Step 2: Run below command to set the value to 0:
esxcfg-advcfg -s 0 /VSAN/ClomEnableInplaceExpansion
Step 3: Run below command to confirm the value is set to 0:
esxcfg-advcfg -g /VSAN/ClomEnableInplaceExpansion
Step 4: Repeat this process on all ESXi hosts in the vSAN Cluster.
Step 5: No reboot or service restart is required for this change to take effect, and it will become effective within 60 seconds.
Alternative
Execute below PowerCLI script to get the cluster name, and then apply the setting across the entire cluster:
Foreach ($VMHost in (Get-Cluster -Name (Read-Host "Cluster Name") |Get-VMHost)) {Get-AdvancedSetting -Entity $VMHost -Name VSAN.ClomEnableInplaceExpansion | Set-AdvancedSetting -Value '0' -Confirm:$false}
Received notice from our TAM regarding a new critical vSAN Data Integrity issue. from r/vmware
Workaround for vSAN on VMware Cloud Foundation 2.3.2 and 3.0
Upgrade to VMware Cloud Foundation versions 2.3.2.3 and 3.0.1 which includes ESXi 6.5 EP9. For VMware Cloud Foundation environments that cannot be upgraded to 2.3.2.3 or 3.0.1, perform Workaround below:
Step 1: Download 58873_vcf-vsan-fix.zip file and copy it to /tmp folder on SDDC Manager Controller VM for 2.3.2 or the SDDC Manager VM for 3.0.
Step 2: Log in to SDDC Manager Controller VM as root user for 2.3.2, or to SDDC Manager VM as vcf user for 3.0. If the environment is VMware Cloud Foundation 3.0, switch to root user with su - command.
Step 3: Execute the following command:
unzip -d /tmp/58873_vcf-vsan-fix.zip
cd /tmp/vcf-vsan-fix
sh ./setup.sh
Step 4: Execute below commands to run automated remediation:
cd /opt/vmware/sddc-support/
python ondemandservice.pyc --setup-json cmd.json --domain-name MGMT
Step 5: Repeat Step 4 for any other workload domains in the VMware Cloud Foundation environment.
Patch Released
VMware ESXi 6.7, Patch Release ESXi670-201810001 (58848)
VMware ESXi 6.7, Patch Release ESXi-6.7.0-20181004001-standard (58849)
VMware ESXi 6.7, Patch Release ESXi-6.7.0-20181004001-no-tools (58850)
VMware ESXi 6.5, Patch Release ESXi650-201810001 (58852)
VMware ESXi 6.5, Patch Release ESXi-6.5.0-20181004001-standard (58853)
VMware ESXi 6.5, Patch Release ESXi-6.5.0-20181004001-no-tools (58854)
vSAN healthcheck – Advanced vSAN configuration supported alert may appear if advanced setting in-place expansion VSAN.ClomEnableInplaceExpansion is still disabled after patching to fixed release. Perform below steps to enable again in-place expansion.
Step 1: SSH into the host(s) reported.
Step 2: Execute below command:
esxcfg-advcfg -s 1 /VSAN/ClomEnableInplaceExpansion
Reference
VMware Knowledge Base: Virtual Machines running on VMware vSAN 6.6 and later report guest data consistency concerns following a disk extend operation (58715)
Real Time Services AB: Heads-up: Possible VMDK corruption on vSAN 6.6 and later!
LIFE IN A VIRTUAL SPACE: URGENT: VMDKs residing on vSAN 6.6 and later that have been extended may encounter data inconsistencies [RESOLVED]
VMware Knowledge Base: VMware Cloud Foundation patch to automate workaround for VSAN in-guest data inconsistency issue (58873)
VMware Knowledge Base: vSAN healthcheck – Advanced vSAN configuration supported (59191)
VMware Knowledge Base: Patch available for critical vSAN issue for vSAN 6.6 and 6.7 clusters regarding data consistency following a disk-extend operation (59194)
