BSOD on a Windows VM bugcheck code 24

A VM that is running Windows 2008 R2 has experienced a BSOD. The application owner notified me and sent me some useful information.

Problem signature:
Problem Event Name: BlueScreen
OS Version: 6.1.7601.2.1.0.400.8
Locale ID: 1033
Additional information about the problem:
BCCode: 24
BCP1: 00000000001904FB
BCP2: FFFFF88006C34E28
BCP3: FFFFF88006C34680
BCP4: FFFFF880014E81B2
OS Version: 6_1_7601
Service Pack: 1_0
Product: 400_3
Files that help describe the problem:
C:\Windows\Minidump50415-20202-01.dmp
C:\Users\servername\AppData\Local\Temp\WER-42663418-0.sysdata.xml
Read our privacy statement online:
http://go.microsoft.com/fwlink/?linkid=104288&clcid=0x0409
If the online privacy statement is not available, please read our privacy statement offline:
C:\Windows\system32\en-US\erofflps.txt

 Good info here, but with all BSODs, you should look at the dump file for more definitive information.

I grabbed the dump file, and downloaded Windows SDK. During the install, I unchecked all the boxes except the Debugging Tools for Windows and installed.

Features to Install

Launch WinDGB (x64) as administrator. The first step is to configure the symbol file path. It can be any folder on your server, just create an empty one and go to File > Symbol File Path.

In the symbol search path, I entered the following and clicked OK.

SRV*c:\tmp\symbols*http://msdl.microsoft.com/download/symbols

Symbol Search Path

Save the workspace. File > Save Workspace

Opened the dump file. File > Open Crash Dump.

Open Dump File

Before you proceed, read the info under the bugcheck analysis. It tells you what probably caused the error.

NTFS.sys

Click the link in the Bugcheck Analysis section, !analyze -y to get more information.

Read through the info. Note the following:

  • Process Name
  • Bugcheck_str
  • Image Name
  • Area right above the debugging details

At the very top you’ll see a message and some arguments.

Analyze

At the kd> prompt at the bottom of the debugger, enter

.bugcheck

Bugcheck command

In the analysis you see the Arguments. Do a .cxr on the 3rd argument as it suggests.

arguments

.cxr fffff88004523680

CXR command

CXR output

Next,

Run the kb command for a more informative stack trace.

Run KB command

There are now a few pieces of information to hone in on.

  1. The bug check code
  2. The stack track text

Look up the bugcheck code here on the msdn bug check code reference page.

https://msdn.microsoft.com/en-us/library/windows/hardware/hh994433%28v=vs.85%29.aspx

Google the stack trace text and this vmware KB article # 2115997 comes up.

http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=2115997

This appears to be a known issue with a few versions of vmware tools:

I looked in the vmware.log file for the entries mentioned and they matched.

vmware-logs

The image name on the dump file referenced vds.exe and this VM is running tools version 9.4.11. This issue only appeared since the tools upgrade when we upgraded to 5.5 Patch 4

To confirm what version of tools you’re running, go to the VM and in the notifications bar, right click on the icon and select About  VMware Tools

VMware Tools version

Time to downgrade tools.

Find the tools ISO that will have a version that is not listed in the KB, but one that is compatible with your version of vsphere.

http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=1036810

https://packages.vmware.com/tools/versions

Search for your current version and find the one closest to it that is not affected with this issue.

Version of tools to downgrade to

You’ll have to uninstall the current version of tools before installing older version. It will require a reboot.

The hardware version will not be an issue with this versions of tools, but confirm that for your specific environment.

Updating the Drive Firmware on EqualLogic arrays

When you’re updating the firmware on EqualLogic arrays, this is also a good time to update the firmware on the hard disks as well. Check the recommended hard disk drive firmware on the eql support site (login required). Compare your hard drives revision number against the ones listed under the ‘affect hard drives’ section of the page. Open either group manager or San HQ and review the current firmware revision.

Confirm current drive firmware version

In group manager, Group + Members + Array + Disks tab.

version of your disk drives

In SanHQ, Default Server + Select Group + Hardware /Firmware + Disks

disk_fw_list_sanhq

If you find your disks drives require a firmware upgrade, plan to update.

FTP the firmware update to the array

Download the firmware update kit from the EQL support site.  FTP to the array.  Here is an example:

Open the connection to the array via IP or hostname and log in with an account that has admin privileges, like grpadmin.

Change to binary mode and ‘put’ the kit_vxxxx_DriveFW_xxxxxxxxxx.tgz file on the array. Once the transfer is complete, close and bye.

ftpupL

Run the update

Now, SSH into the array and begin the update.

Type ‘update‘ and confirm that you’d like to proceed with the update.

update

Depending on the number of drives in your array, this will only take a few minutes.

update-done

yes

Confirm new firmware version

Check group manager or SAN HQ and confirm your new hard drive firmware version.

newfw-sanhq newfw-group

 

 

 

 

Updating the firmware on Dell EqualLogic arrays with Dell Update Storage Manager

The Dell Storage Update Manager (DSUM) is a great new tool that makes updating your array firmware, drive firmware  and language packs easier. Launched summer 2014, this application that can be installed locally or ran remotely via a java based app, is your new recommended way to update your groups.  To use it, you must be running at least v5.0.0 PS series firmware or FS series firmware 3.0.0 or higher.  This wizard will walk you through assessing the current state of your groups and guide you through the updates step by step.

To download the application, login to your EQL support account. Also, download your firmware, disk drive firmware and your language packs (if applicable). To run the java web app, follow this link

http://psonlinehelp.dell.com/dell-storage-update-manager/

and open with Java web start launcher.

launch java app

Log into your group with administrative credentials.

Dell Storage Update Manager Login

Once logged in, you can review the status of your group firmware, disk drive firmware and language packs.

group inventory overview

Click on Update plan to being the update wizard.

selectupdates

Here is where you select the updates to be installed.

update plan summary

Review  the updates you’ve selected. The Update plan summary will give you an estimated time for the updates to completed.

Getting Started

On the getting started screen, you’ll get a quick reminder to perform the update when their is low activity and during a pre-planned maintenance windows.  Check the box to confirm you understand the ramifications.

Preparing your update files

Upload the files needed to update your array, drives, and language packs.

Prepare your group

The app will review if their are any issues detected on your group that may prevent updates from running. Some issues that may prevent an update are:

  • RAID is in a degraded state
  • There are active errors or warnings in the log file
  • There are disk errors or issues
  • Controller issues
  • Space issues
  • High I/O
  • Replication is occurring
  • Volumes are migrating

Install Updates

The next screen in the wizard is the 1st step in updating your array’s firmware. Review the information and click install update.

The installation begins and will display the progress  and the current status. you can also review this information in group admin under group operations.

  • The 1st step is the FTP transfer of the zip file to the array.
  • The 2nd step is the actual update.
  • The 3rd step is the update pending restart.

step1

step2

step3

Once the update is complete and pending a restart, click ‘restart’ to proceed.  You’ll be  warned about the restart and it’s ramifications. Repeat for each array.

restart

complete

The update is complete. The HIT Kit compatibility screen with remind you to confirm that all EqualLogic software in use in your environment  is at the versions listed.

hit

Follow the subsequent prompts for updating the drives and language pack (if aplicable)

Once all installs are done, you’ll get the ‘installation complete’ screen that summarizes what took place.

installcomplete

update successful

Coming down off my VMworld high

This is my 3rd year attending VMworld and each time I walk away with a renewed focus and energy for my virtual infrastructure and what I can do to better manage and improve it. As I’m writing this post, I’m scrolling through my tweets and notes from VMworld and what stands out is that for my organization, VMware has the tools an SMB needs to deploy, monitor, upgrade, backup, administer, BD/DR, and grow its virtual infrastructure with ease.

My schedule was filled with sessions that covered just the things I needed to hear and to use day-one when I’m back in the office. Using the schedule builder as soon as it was available gave me time to make smarter choices and decide where my time should be spent in a more efficient manner. The VMworld app was a life saver! I knew right where I was suppose to be with notification turned on. The social media section keep me abreast of all the #vmworld tweets easily.

Here are my 3 favorite sessions. The sessions that made me want to roll up my sleeves the minute I had a WiFi connection to VPN back into the office and ‘test’ something out.

Here are my top 3 all-around favs from VMworld 2014:

  • The Solutions Exchange Welcome Reception
    • I got to spend a lot of time looking at what vendors are bringing to the virtualization space. Software, products and services that are moving VMware forward as the leader.
    • The Games at the CDW booth were more fun than I expected to have on opening day. The trivia questions were thought provoking and the ‘game show host’ and his assistant really kept us laughing.
    • Free beer & wine.
  • Winning a scooter (razor type) from Ravello and terrorizing the Moscone Center.
  • Taking a selfie with my SRM mentor, Mike Laverick of VMware.
Me and Mike Laverick of VMware
Me and Mike Laverick of VMware

Here are a few things I would change for next year:

  • The volume of the music at the Monday General Session was deafening. I really didn’t understand why the AV people didn’t rectify that by the end of the keynote. I promised myself I wouldn’t listen to another GS in the Hall D. The hang space was just fine.
  • I was in just ONE session with a woman speaker and I think she was a vendor. ‘Nuff said.
  • I know they’re trying to accommodate those with dietary restrictions, but vegetarians do need protein too.

Now that I’m back in the office, I’m ready to, upgrade to vSphere 5.5, tweak SRM  and  test out the vSphere 6.0 beta. I can’t talk about it (NDA), but you can find out more about it here.

Looking forward to next year.

 

EqualLogic Cloning an Inbound Replica

I have a dev server that’s my sandbox for VMware VCA studying. This Dell PowerEdge 2900 is on it’s last leg. It has an iSCSI disk that houses a bunch of ISOs, installers, docs, etc that is being replicated to another EQL array. The old PE is located behind my desk and is loud as a 747 taking off when it powers up.I will NOT miss this thing at ALL!

I have a new PowerEdge R420 that I’ve moved my development environment to and it lives in an offsite data center. I want my iSCSI disk attached to my new server, but I don’t want to stop replication on my current disk until the server is wiped and hauled off. The beauty of EqualLogic is their arrays are wonderfully easy to administer and makes any takes relatively easy.

Since the disk is replicated and I don’t want it to stop replicating right away, but still would like to have the current data to use and access immediately on my new server. Yes, I could have mapped a drive, but if the 2900 dies, I’m still up and running without even a hitch.

That’s where cloning an inbound replica comes in.

From the outbound group manager, make sure you’ve replicated your volume.

confirm_volume_replicated

From the inbound group manager, Go to replication and expand inbound replicas. Select your replica so the information appears in the right pane. Click clone replica.

clone_replica

The wizard will guide you through the process of cloning your replica. On step 2, change the snapshot reserve if you need to. Step 4, review the summary and click finish.

change_snap_reserve

finish_clone_replica

Your new volume will appear in the volume list. Present it to the new server if you didn’t during the clone volume replica wizard and you’re in business.

 

new_vol

 

 

 

 

Activating Windows Server 2012 with MAK key

It appears the default activation in Windows Server 2012 is KMS.

win_isnt-activated

Since I’m a primarily a 2008 R2 shop and this is my first ‘real’ 2012 server, I was waiting for the prompt to activate it with my MAK key, but when I clicked Activate, it gave me an error about KMS.

The quick way to activate your server with MAK is to Open a command prompt and run as administrator.

Run As Administrator

At the prompt, enter: slmgr.vbs /ipk xxxxx-xxxxx-xxxxx-xxxxx

replace the Xs with your MAK Key and hit enter.

Enter Key

Your copy of Windows Server is now activated.

activated

Social media & sharing icons powered by UltimatelySocial