vthistle.com

You are here: Home / Archives for vExpert

Nutanix – Bully VM – Troubleshooting with Prism

By

I recently received a call from one of my favorite customers who was experiencing some slowness in their environment. They are running a Nutanix block that is about a year and a half old, utilizing hybrid storage. In larger environments, I would immediately open the customers vRealize Operations Manager and take a peek to see who is the causing the issue. In this case, the customer doesn’t use vRealize so that wasn’t available. I then immediately turned my attention to Nutanix Prism. For those not familiar with Prism, it is a single pane of glass to manage and monitor your Nutanix environment. You can retrieve a ton of information on each virtual machine running in the environment.

So what did I do to find the issue?

  1. When I logged into Prism, I immediately noticed that cluster IOPS more than tripled single the initial call into the help desk. (example of graph below)
  2. I then selected VM from the menu. This will bring you to the “VM Overview” page. This page gives you the following information by default. Top users VMs by IOPS, VM Alerts, Latency, Memory usage and CPU usage. 
  3. I could immediately see that the customers backup server was the suspect in this case as it was generating a ton of disk activity. This is the information I needed to log into the VM to find out what was going on.

Customer = happy!

To show you the kind of information and graphs you get when you click on the “table” section of VM, this is a snapshot of your virtual machines. 

If you want to drill down into a specific VM – click on that VM and you will see the following information. 

Hope this helps you when troubleshooting “bully” VMs in the future.

vSphere 6.5 – Enhanced Logging

By

This could be one of the best features that VMware added to the vSphere 6.5 release, Enhanced Logging. As a IT consultant, we often get calls such as, “X isn’t functioning, it was working 10 minutes ago”. You will immediately ask the four “W’s” – What, When, How – then someone will always ask “Who” once the issue is resolved. With enhanced logging, this is now all possible. Enhanced Logging might not be the sexiest thing, but its going to save you a lot of time troubleshooting.

In prior versions of vSphere, when you made a change to the VM, all you would see is “Reconfigured VM” and who made the change. How many times have you asked yourself, “What was the &*#* change”?

Lets look at how you can utilize Log Insight to view the “enhanced logs”.

After you make a change to a VM, you will the answers to our questions. – you can hover your mouse over “vc_username” to see made the change.

In this example, you can see the at 21:35, that the VM “ViewCS-01” had a change that disabled the NIC at Startup. You can also see what the value was prior to the change. As you can see below, the original state was “True” and was changed to false. changes

At this point, you can edit the configuration to the VM, and revert the change that was made – and the customer is happy.

Here is the awesome thing. If you didn’t know, if you are entitled to vCenter – you are also entitled to a 25 OSI license of Log Insight for your environment for FREE. Hopefully you will never need to use Log Insight, but it will ensure you have the logs necessary if you need to engage VMware GSS or see who made a change in one easy place.

vCenter – Upgrade from VCSA 6 to VCSA 6.5 – CLI

By

I recently have been on a mission to eradicate all vCenter Servers running on Windows and replacing them with the vCenter Server Appliance (VCSA). I’ve been very successful with this with the customers I have worked with, with very little push back. The only down fall with the 6.0 release was that Update Manager (VUM) still had to be on a Windows Sever. I got around this in most cases by creating a vCenter “Helper” Server. I would install VUM, PowerCLI and RVTools – All the essentials!

So, after doing a TON of these migrations by using the GUI, I thought I would give the CLI a shot.

Since vCenter 6.5 was introduced today (11/15/2016), I thought I would try to go through the upgrade process via the CLI. I will be going from VCSA 6.0U2 to VCSA 6.5. This upgrade will migrate your database, as well as migrate you from a Linux appliance onto VMware Photon OS.

Why do it from the CLI? Because, its the cool thing to do!

Note: Make sure all of your plugins are compatible with vCenter 6.5!

Note: Make sure SSH is enabled on your source vCenter prior to upgrade/migration.

First you will need to pick the template that you want to start from. VMware does a great job of telling you what you need to do to get this to work from these templates. To find it, mount the ISO and go to “X:\vcsa-cli-installer\templates\upgrade\vcsa6.0” – Pick  the template that works for your scenario and save it locally so that you can make changes that are necessary. I will be using the “embedded_vCSA_on_ESXi.json” template.

I’m not going to spell out what each option is, because VMware has done a great job of spelling that out in the Documentation Center for vSphere/vCenter 6.5.

Second, lets go through the upgrade. If you have VUM deployed on a windows server, you will want to run the upgrade script from the VUM server.

On the VUM server, run the Migration Assistant – Can be found at “X:\migration-assistant\VMware-Migration-Assistant.exe”. When the Migration Assistant starts, it will ask you for the Service Account password for the account you used to connect VUM to vCenter. *Note: Don’t close this window until the migration is completemigration-assistantNow we are going to execute the upgrade via the CLI – 
vcsa-upgrade-1-cli

  1. This will kick off 8 Steps:
    1. Verification of the template you specified above
    2. Verify Configuration
    3. Execute OVF Tool Command (Deploy the new appliance)step_3
    4. Install Services (on the new appliance)step4
    5. Export Data (from old vCenter)step5
    6. Configure vCenter Servicesstep6
    7. Import Data (to new appliance)step7
    8. Result and Log Filesstep8

SUCCESS!!

The HTML5 version still has limited functionality – but damn its fast! In my opinion even the FLASH site is quick. Yes, you get to choose!choose

vCenter – Solution Extension Removal

By

vCenter solutions extensions, everyone has them, everyone loves them. This post is going to focus on, what happens when you uninstall and extension, but the tags are still on the VM’s.

I recently ran into a situation with a customer that they were doing a VM migration with Zerto, they finished the migration and then uninstalled. Typically, no problem – however it appeared that after they uninstalled the product, vCenter was still showing that the “com.zerto.plugin” was still managing the VM.

zerto

This is actually very easy to fix. VMware KB2032366 references a similar issue with VMware Site Recovery Manager. Will will go through the steps listed in this KB, but removing the “com.zerto.plugin” extension.

  1. Download the “Managedby.ps1” script from the link above and extract the .ps1 file to a directory that you typically run scrips from.
  2. Open PowerCLI and connect to your vCenter “connect-viserver <vCenter hostname>
  3. I run the “.\managedby.ps1 -cmd scanVMs” to see what solutions are being used. This also ensures you are grabbing the exact name of the solution extension. This also lists all VM’s that are using this extension.
  4. Run – $hmsVms = .\ManagedBy.ps1 -Cmd getVms -extKey “com.zerto.plugin”
  5. To view the machines with the plugin, run $hmsVms
  6. Run – .\ManagedBy.ps1 -Cmd Clear -VMs $hmsVms
  7. Now, if you right click on the VM, you should not the the “confirm operation” dialog box you see above.

Complete! Wahoo!!

Nutanix: ESXi Storage I/O Control

By

I was running a health check on a customers Nutanix cluster this week and found a newer item that Nutanix Cluster Check (NCC) checks for. Its called “esx_sioc_status_check”. This item was added in NCC 2.2.5

Nutanix has stated for years that Storage I/O Control (SIOC) is not needed or recommended on datastore containers. The reason for this is due to how write I/O is serviced by every Nutanix CVM in the cluster – while 90-95% of read I/O is done on he local CVM. We are seeing more vendors call this “Data Locality” – side note: Nutanix has been talking about Data Locality for years and its advantages.

Nutanix now has now went a step further to ensure that SIOC stats are now disabled (Nutanix KB3358). SIOC is disabled by default, however, SIOC stats collection is enabled by default. SIOC Stats

To disable SIOC or SIOC stats mode, do the following: 

  1. Go to the vSphere Web Client and Log in
  2. Go to Storage View – Select the Datastore / Container – Manage – Settings and if SIOC is enabled, disable SIOC
  3. Select the “Disable Storage I/O statistics” check box to disable SIOC in stats mode.

Disabked SIOC Stats