Skip to main content

Posts

Showing posts with the label Troubleshooting

Troubleshooting vSAN Storage Policy Migration Failures in VMware Cloud Foundation 9.0

Recently I had a customer who migrated their VMware infrastructure from vSphere 7.0 to VMware Cloud Foundation 9.0. After the migration, they wanted to update their vSAN storage policies to take advantage of the new vSAN ESA (Express Storage Architecture) features. However, when they tried to change the storage policy for their production VMs, the operation kept failing with a cryptic error message. The error they were getting was: "Cannot complete operation due to insufficient resources to satisfy current storage policy." This was strange because they had plenty of disk space available. The vSAN cluster was only at 45% capacity, and all the hosts were healthy. Let me walk through how we troubleshooted and fixed this issue. Understanding the Problem First, let's understand what was happening in their environment: They had a 6-node vSAN cluster running VCF 9.0 vSAN was configured with the new ESA architecture They were trying to migrate VMs fro...

Fixing Tanzu Kubernetes Pod to External Services Connectivity Issues with NSX-T

Fixing Tanzu Kubernetes Pod to External Services Connectivity Issues with NSX-T Last month I got a call from a customer who was pulling their hair out over a networking issue. They had just deployed VMware Tanzu Kubernetes Grid on their vSphere with Tanzu environment, everything looked good in the dashboards, all pods were running, but their applications inside the pods could not reach external databases running on traditional VMs in the same datacenter. The frustrating part was that some pods could reach external services perfectly fine, while others would just timeout. There was no clear pattern. Let me tell you how we figured this out and fixed it. The Initial Problem Here is what the customer setup looked like: vSphere 8.0 with Tanzu enabled NSX-T 4.1.2 for networking Three Tanzu Kubernetes clusters running different microservices applications External PostgreSQL database running on traditional VMs (non-Kubernetes) External API services running on another se...

Building Guardrails and Conformity Bots in VMware Environments: A Practical Engineering Guide

In enterprise VMware environments, maintaining architectural standards at scale is a constant challenge. After years of working with large-scale virtualisation infrastructures, I've learned that the gap between what architects design and what exists in production grows exponentially with team size and deployment velocity. This post shares my hands-on experience building automated guardrails and conformity bots that enforce standards, detect drift, and maintain architectural hygiene across VMware estates. The Real Problem: Configuration Entropy Every VMware environment I've worked with faces the same pattern. It starts clean—well-tagged VMs, proper resource allocation, consistent network segmentation. Six months later, chaos. VMs get deployed without mandatory tags, making cost tracking nearly impossible Resource limits get bypassed during urgent deployments and never corrected Network placement becomes inconsistent as different teams interpret po...

VMware on AWS - How to restore NSX DFW firewall rules to previous state

Customers who uses NSX day-in, day-out would like to have a point-in time restore functionality of DFW firewall rules. Many customer have a large footprints in VMC and make changes to DFW quite often. This feature was missing for long time and we could see its included in recent versions . Let's see how DFW configuration roll back works  NSX DFW configuration has versioning, and it is stored in the NSX Manager.  Every time when someone update DFW configuration, NSX creates one more version but keep storing the previous ones. You can rollback for previous config but reapplying it once again.  You can find the options under Networking & Security tab , > Security > Distributed Firewall . In the right side we see an Actions drop down. Choose View to get to the below screen.  Let’s go through the use case:  1. Original state- default config with no custom rules:  a. There are no saved configurations during last 30 days: In my existing test setup, ...

[How to] Enable Multi-cast in VMware on AWS - NSX environment

I had a customer who were running few application VMs in their on-premises datacenter which uses multicasting as main mechanism in order to form cluster blocks. They are in process of migrating the workloads from On-premises to VMware on AWS SDDCs. The application Architect wanted to ensure that the VMC supports multicasting within AWS VMC so that they migrate the VMS ( Life & Shift) without major downtime/config changes to their applications clusters. Let's see how things works within VMC world In VMC setup the Multi casting feature is enabled by default. In SDDC networks, layer 2 multicast traffic is treated as broadcast traffic on the network segment where the traffic originates. It is not routed beyond that segment.   VMC Limitation:  Optimisation features such as IGMP snooping are not supported.  Layer 3 multicast (such as  Protocol Independent Multicast) is not supported in VMware Cloud on AWS.   In the above example case, the customer has L2 mul...

Update Manager 6.5 issue ""interface com.vmware.vim.binding.integrity.VcIntegrity is not visible from class loader""

This post is related to the issue what we faced today when we replaced the SSL certificates in our setup.  When I launched the web-client and access the update manager tab,  I get the message  "interface com.vmware.vim.binding.integrity.VcIntegrity is not visible from class loader" I started off by restarting the VMWare vSphere Update Manager Service for the affected vCSA: 1. Log into vCenter using the administrator@vsphere.local account. 2. Home - System Configuration - Services - Restart This did not resolve my issue...  And we tried restarting all the services by  SSH/Console into the affected server and run the following commands: service-control --stop --all service-control --start --all Still no luck.  Make sure the certs are applied and it gets reflected in the config file. ( verify if the thumbprint matches)  root@homelab71 [ /usr/lib/vmware-updatemgr/bin ]# pwd /usr/lib/vmw...