When you resize the wrong hard disk on a vm

Category: vmware

12
Aug
2016

When you resize the wrong hard disk on a vm

I’m running WSUS on a 2012R2 server and the drive where the updates are tends to grow a lot. On this VM, I increased the provisioned size of the c:\ drive and not the drive where the updates reside. I didn’t notice I’d touched the wrong disk until I’d rescanned the disks on the VM and saw what I’d done.

As panic set in, I changed the provisioned size back to 60GB (I know, I know) and well, you know the rest. This article saved me today. I didn’t want to restore the VM unless I really had no choice, this did the trick.

Power off the VM and identify the host & datastore that the VM resided on, I enabled SSH and browsed to the VM location.

Cat out the descriptor file (vmname.vmdk) that outlines the data layout on the virtual disk. In the Extend Description section, you’ll see the extent(s) info. Next to RW (if it’s read/write) a number. That number represents the size of the vmdk in sectors, with the additional 200GB included.

To change the sector size to reflect the value I’m shrinking the drive back down to, 60GB, calculate it:

60GB = 60 * 1024 * 1024 * 1024 / 512 = 125829120 sectors

Edit the descriptor file and change the sectors to the new number, save and exit.

Come out of the directory where the VM resides and VMotion it to another datastore. Review the settings of the vm and take note of the new and improved disk size.

Power on the VM and review the size in Disk Management.

Now, grow the right disk and try not to ever do that again.

23
Mar
2016

Deprecated VMFS volumes found on the host Warning on ESXi after adding a datastore with Dell VSM

categories: Storage, vmware

I’m running ESXi version 6.1 U1 and Dell Virtual Storage Manager (VSM)version 4.5.2.974. I added a datastore to the cluster using VSM and on 2 of my hosts I got the following alert:

Enables SSH on the host and check out the logs.

In /var/log/hostd.log , I found the error:

warning hostd[xXxXxX] [Originator@6876 sub=Hostsvc.DatastoreSystem opID=123456-789-abc-def user=vpxuser] UpdateConfigIssues: Deprecated VMFS filesystems detected. These volumes should be upgraded to the latest version.

When I created the datastore, I did select VMFS 5, so I wasn’t sure why this error appeared.

According to vmware KB2109735, this is a known issue in version 6.0 and there is no resolution. Just restart the management services and the message goes away.

#  /etc/init.d/hostd restart

#  /etc/init.d/vpxa restart

Then the message goes away.

note

This only happens to me when I create a datastore with VSM. If I present a disk from my array, set up the iSCSI connections and rescan, I don’t get this message. Your mileage may vary.

15
Sep
2015

VDP backup fails due to error consolidating disks on VM

categories: vmware

We are running VDP 5.8 on vSphere 5.5. One of the appliances failed to backup a VM last night. Checking the VM in the client, there is a message to consolidate the disks. I attempt to consolidate by right-clicking on the VM > Snapshot > Consolidate and it failed with this error:

I checked the vmware.log file for the VM and saw this error:
2015-09-15T03:00:29.146Z| vcpu-0| I120: DISK: Failed to open disk for consolidate '/vmfs/volumes/5490a09a-eb6682ee-affb-bc305bef5520/mobxxxxxx/mobxxxxxx-000002.vmdk' : Failed to lock the file (16392) 8162

I check the settings on the VDP appliance and notice that the VMDK that belongs to the vm is attached.

Shut down the VDP appliance and remove the disk. Making sure to only remove from virtual machine and NOT remove from virtual machine and delete files from disk.

Run the consolidation and if it succeeds, re-run the VDP backup on out of date sources to remediate your failed backup.

14
May
2015

BSOD on a Windows VM bugcheck code 24

categories: vmware, Windows 2008 R2

A VM that is running Windows 2008 R2 has experienced a BSOD. The application owner notified me and sent me some useful information.

Problem signature:
Problem Event Name: BlueScreen
OS Version: 6.1.7601.2.1.0.400.8
Locale ID: 1033
Additional information about the problem:
BCCode: 24
BCP1: 00000000001904FB
BCP2: FFFFF88006C34E28
BCP3: FFFFF88006C34680
BCP4: FFFFF880014E81B2
OS Version: 6_1_7601
Service Pack: 1_0
Product: 400_3
Files that help describe the problem:
C:\Windows\Minidump50415-20202-01.dmp
C:\Users\servername\AppData\Local\Temp\WER-42663418-0.sysdata.xml
Read our privacy statement online:
http://go.microsoft.com/fwlink/?linkid=104288&clcid=0x0409
If the online privacy statement is not available, please read our privacy statement offline:
C:\Windows\system32\en-US\erofflps.txt

Good info here, but with all BSODs, you should look at the dump file for more definitive information.

I grabbed the dump file, and downloaded Windows SDK. During the install, I unchecked all the boxes except the Debugging Tools for Windows and installed.

Launch WinDGB (x64) as administrator. The first step is to configure the symbol file path. It can be any folder on your server, just create an empty one and go to File > Symbol File Path.

In the symbol search path, I entered the following and clicked OK.

SRV*c:\tmp\symbols*http://msdl.microsoft.com/download/symbols

Save the workspace. File > Save Workspace

Opened the dump file. File > Open Crash Dump.

Before you proceed, read the info under the bugcheck analysis. It tells you what probably caused the error.

NTFS.sys

Click the link in the Bugcheck Analysis section, !analyze -y to get more information.

Read through the info. Note the following:

Process Name
Bugcheck_str
Image Name
Area right above the debugging details

At the very top you’ll see a message and some arguments.

At the kd> prompt at the bottom of the debugger, enter

.bugcheck

In the analysis you see the Arguments. Do a .cxr on the 3rd argument as it suggests.

.cxr fffff88004523680

Next,

Run the kb command for a more informative stack trace.

There are now a few pieces of information to hone in on.

The bug check code
The stack track text

Look up the bugcheck code here on the msdn bug check code reference page.

https://msdn.microsoft.com/en-us/library/windows/hardware/hh994433%28v=vs.85%29.aspx

Google the stack trace text and this vmware KB article # 2115997 comes up.

http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=2115997

This appears to be a known issue with a few versions of vmware tools:

I looked in the vmware.log file for the entries mentioned and they matched.

The image name on the dump file referenced vds.exe and this VM is running tools version 9.4.11. This issue only appeared since the tools upgrade when we upgraded to 5.5 Patch 4

To confirm what version of tools you’re running, go to the VM and in the notifications bar, right click on the icon and select About VMware Tools

Time to downgrade tools.

Find the tools ISO that will have a version that is not listed in the KB, but one that is compatible with your version of vsphere.

http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=1036810

https://packages.vmware.com/tools/versions

Search for your current version and find the one closest to it that is not affected with this issue.

You’ll have to uninstall the current version of tools before installing older version. It will require a reboot.

The hardware version will not be an issue with this versions of tools, but confirm that for your specific environment.

04
Sep
2014

Coming down off my VMworld high

categories: vmware

This is my 3rd year attending VMworld and each time I walk away with a renewed focus and energy for my virtual infrastructure and what I can do to better manage and improve it. As I’m writing this post, I’m scrolling through my tweets and notes from VMworld and what stands out is that for my organization, VMware has the tools an SMB needs to deploy, monitor, upgrade, backup, administer, BD/DR, and grow its virtual infrastructure with ease.

My schedule was filled with sessions that covered just the things I needed to hear and to use day-one when I’m back in the office. Using the schedule builder as soon as it was available gave me time to make smarter choices and decide where my time should be spent in a more efficient manner. The VMworld app was a life saver! I knew right where I was suppose to be with notification turned on. The social media section keep me abreast of all the #vmworld tweets easily.

Here are my 3 favorite sessions. The sessions that made me want to roll up my sleeves the minute I had a WiFi connection to VPN back into the office and ‘test’ something out.

Ask the Experts: Design Advice for Small and Medium Business #INF1192
- It was invaluable to hear what happens in REAL SMB IT shops. The feedback and questions were very helpful.
Site Recover Manager and vSphere Replication : What’s New Technical Deep Dive # BCO2629
- There were several moments of applause. I’m surprised they didn’t get a standing ovation.
Managing vSphere Upgrades and Updates # INF1523
- All around great information and they pointed out specific caveats when upgrading that can be overlooked.

Here are my top 3 all-around favs from VMworld 2014:

The Solutions Exchange Welcome Reception
- I got to spend a lot of time looking at what vendors are bringing to the virtualization space. Software, products and services that are moving VMware forward as the leader.
- The Games at the CDW booth were more fun than I expected to have on opening day. The trivia questions were thought provoking and the ‘game show host’ and his assistant really kept us laughing.
- Free beer & wine.
Winning a scooter (razor type) from Ravello and terrorizing the Moscone Center.
Taking a selfie with my SRM mentor, Mike Laverick of VMware.

Here are a few things I would change for next year:

The volume of the music at the Monday General Session was deafening. I really didn’t understand why the AV people didn’t rectify that by the end of the keynote. I promised myself I wouldn’t listen to another GS in the Hall D. The hang space was just fine.
I was in just ONE session with a woman speaker and I think she was a vendor. ‘Nuff said.
I know they’re trying to accommodate those with dietary restrictions, but vegetarians do need protein too.

Now that I’m back in the office, I’m ready to, upgrade to vSphere 5.5, tweak SRM and test out the vSphere 6.0 beta. I can’t talk about it (NDA), but you can find out more about it here.

Looking forward to next year.

27
Jan
2014

Updating Java to 1.7 Breaks Dell EqualLogic VSM

categories: vmware

I loathe updating Java because you never know what it will break. This morning, it was the Dell EqualLogic VSM plug-in for vCenter.

Upon launching the plug-in, I got these errors:

I thought to disable the current version and enable the previous version, but there is a way to add this URL as an exception.

Launch the Java control panel (I’m running 2008 R2) and click on the security tab.

Click the edit Site List button:

Add the URL in the above mentioned error, by clicking Add, then typing in the URL.I also added the IP to vcenter as well. Click OK.

One more warning about the HTTP location you’ve entered. Just click continue. Apply the changes and close the java applet. Be sure to close the vCenter client as well.

Launch the client and log into VSM as usual.

Update- info on Windows 7

In Windows 7, there is no exception list under the security tab. To add your site to an exception list, you can create an exception.sites file and add the URL.

Browse to this location: C:\Users\username\AppData\LocalLow\Sun\Java\Deployment\security

If there is a file named exception.sites, add the URL to that file. One URL per line.

If not, you can create it and add your URL.

23
Jan
2014

The command has timed out as the remote server is taking too long

categories: vmware

I’m running vSphere 5.1 and I can’t launch the vSphere Client. I am able to launch the web client just fine. Upon launching the fat client, I get an error:

We updated our domain from 2003 to 2008 R2 and replaced all old domain controllers (DCs). I’d changed the primary and secondary server URL (there was no port number previously) during the cut over and all has been working well for the past 26 days and today, nothing.

I logged into the web client as admin@system-domain to confirm I was pointing to the new domain controllers and I was. So now I’m really stumped. KB article 203918 solved the riddle.

To view the identity sources: Administration > Sign-On and Discovery > Configuration. Here you can view the identity sources. Click the pencil to edit and see more detail.

Here are my current settings:

I added the port number to both the primary and secondary server URLs and tested the connection. Click OK to close the dialog box.

I was then able to launch the fat client without error.

Don’t forget to log out of the web client as admin@system-domain.

It’s safe to assume when I installed 5.1, I didn’t enter the port number as suggested in the Configuring vCenter Single Sign on doc and it worked just fine. Upon upgrading, the port became a required setting.

06
Nov
2013

Configure and Install the Multipath Extension Module for vSphere and EqualLogic

categories: Storage, vmware

The MEM (Multipath Extension Module) by EqualLogic (EQL) handles path selection and load balancing to the storage array. Upon install, it will add another path selection policy called ‘dell_psp_eql_routed’ in addition to the 3 default policies. Using this PSP is ideal when you’re datastores reside on EQL since the module is written by EQL, it has been designed to perform more efficiently in regards to path selection and load balancing to the array.

Here is how I installed and configured MEM on my esxi hosts. I’m running ESXi 5.1 Update 1 with vSphere CLI installed on vCenter.

Prerequisite: Be sure to configure an iSCSI vSwitch for multipathing before installing MEM. Please read TR1075 for more information on how to configure the vSwitch.

Download MEM from EQL support site. Login is required. Review the release notes, TR1074 as well as the installation & user guide before proceeding.

On the vCenter server, launch vCenter client. From the home screen, open Update Manager and click on the patch repository tab. Click import patches and browse to the MEM offline bundle zip and click next to upload.

Next, create a baseline. Enter a name and description for your new baseline, select ‘Host Extension’ then click next.

Add the extension to the baseline. Click next.

Review your setting and click finish.

Your newly created baseline is now listed under the Baselines and Groups tab.

Now it’s time to install. Begin by putting your host into maintenance mode.Click the update manager tab of the host and click attach.

Check the box next to the MEM install and click attach.

Highlight the attached baseline and click scan.

Confirm that you’re scanning for patches and extensions. Remove the check next to upgrades. Click scan.

The host will now be labeled as non-compliant. Click remediate in the lower right corner.

Click next twice in the remediate wizard if you’re accepting the defaults. On the schedule window, type in a new task name and description (optional) and select a remediation time. I did mine immediately, but this task can be scheduled for a later time. Click next.

Edit any host remediation options and click next. Edit any cluster remediation options and click next. Review your remediation settings and click finish.

Monitor the recent tasks pane to see the status of the installation. Upon completion, the host will be listed as compliant in the update manager tab.

Reboot.

From a system with vMware vSphere CLI installed on it, run the following command to verify the MEM installation. You will need the setup.pl script in order to run it. It’s included in the MEM download.

Enter the following command using your esxi hostname or IP:
# setup.pl --server=esxhostname_or_IP --query

Enter the credentials for the host. After a few moments, the command will display the version of the MEM installed, the default PSP that is now set as well as the vmkernal ports used by MEM.

To list the new Dell EQL PSP as well as the defaults, use the following command:

# esxcli --server esxhostname_or_IP storage nmp psp list

Enter the credentials for the host.

Multipathing is available immediately after installation. You can see the paths to the disk as well as the new PSP. On the host, go to the configuration tab+ hardware+storage+right-click the datastore + properties+ managed paths.

This is what it looked like before the install:

Here you can see that there were 2 active connections to each controller, however, only 1 was being used for I/O. Once MEM is installed, there are redundant active connections to each controller and the load is more evenly balanced.

As I mentioned before, the MEM is totally functioning, however, in order to use the new esxcli commands that are available to manage and report on, you’ll have to restart the hostd agent on the esxi host.

Enable SSH on your host and log into it.

Restart the hostd service:

# /etc/init.d/hostd restart

The new commands are now available. To list them:
# esxcli equallogic

Log out of the host and disable SSH.

Done!
For more information on Dell’s MEM, read this great blog post from Cormac Hogan of vMWare.

10
Jun
2013

ESXi 4.1 to 5.1 Upgrades with VUM

categories: vmware

This is a really good post about how to remove software modules that are no longer needed. I came upon this issue when upgrading from ESXi4.1 to ESXi 5.1. One of my hosts wouldn’t remove the drivers.

I uploaded the offline bundle zip to VUM and created a host extension baseline and ran it against the host. The host had a kernel panic and didn’t boot up.

I had to upgrade that particular host from the ISO and forgo update manager so all is well.

I was even given this post from VMware technical support when I called. I’d already happened upon it, but it was impressive that his blog post was on the tech’s radar.

05
Jun
2013

Having VT enabled on your CPU isn’t all you need to run a nested ESXi enviroment in VMware workstation 9

categories: Extended Page Table, Nested Environment, Virtualization, vmware, VMware Workstation

I built an ESXi 4.1 test environment in workstation 9 to test and document an upgrade from vSphere 4.1U2 to 5.1U1a

My test box is:

Dell PowerEdge 2900 with (2) dual core Intel Xeon 5160’s
24GB RAM
iSCSI Lun attached to hold my VMDKs
2008 R2 Enterprise SP1 with VMware workstation 9.0.2 (upgraded from 8.0.2)

I wanted to install some 64-bit VMs inside of my ESX servers and after some cursory reading and searching, I was under the impression that I could. The first thing was to make sure that CPU could handle a 64-bit OS (Intel® 64) VT (Intel® Virtualization Technology (VT-x) and XD (Execute Disable Bit). I went to the Intel site and looked up my processor. Looks like I was ready to go. (I missed the part about EPT=No at the bottom as Mike would say, “RTFM”)

After installing the DC,SQL, vCenter server and making sure communication between the 3 was working, I created my 1 ESXi VM and when I powered it on I got this warning:

There is a setting on the VM under processors that should be enabled. I checked the box, proceeded with my installation.

esxinstall-eror-fixed — Edit the settings on the ESXi VM and enable VT

I configured the host, added it to the cluster. So I thought I was in the clear. Somewhere along the line, before I tried to install a guest, I saw this message.

Virtualized Intel VT-x/EPT alert — I clicked yes to continue. Clicking no wouldn’t let me proceed.

I searched this message and the previous one and found several posts that suggested confirming the bios settings on the machine that’s running workstation, enabling VT in the settings (we did that) vmx or /etc/vmware/config hacks to fix it and nothing worked. I went so far as to update the bios on the server just in case there were some features that were not exposed on the version I was currently running.

Here are several errors I got when I tried to install 2003 R2 64bit on the ESXi host:

I posted on a vmware community for workstation and got a response that I really didn’t anticipate:

The implication of the message, “Virtualized Intel VT-x/EPT is not supported on this platform,” is that your host does not support EPT.

OK, so there is one more technology that is required to run a nested 64-bit environment. But what is EPT and when did it become so important?

The ability to run a nested environment is fairly new. The newer processors (AMD and Intel both) have begun to include Extended Page Tables (EPT) in to their procs (Nested page tables in AMD). EPT allows each guest VM to have it’s own page table to keep track of virtual & physical memory.

I have my nested environment running fine, there just aren’t any 64-bit VMs running on any of the ESXi 4.1 hosts.

For more info on EPT and Memory Management read here and here.

Category: vmware

*note*

Update- info on Windows 7

note