netnerds.net » Virtualization

22Jan/102

VMware ESXi 4.0: Create Virtual Machine Error Caused by NSF File on Western Digital MyBook WE

Such an awkward title, I know. It's just hard to encapsulate the following error message into a blog post title:

Create virtual machine 172.16.1.129 Error caused by file /vmfs/volumes/0-cb8d2a5-20-f15722/win-2k8.vmdk

Basically, I'm taking ghetto to the next level by using my Dell Optiplex 745 workstation as an ESXi 4.0 server, and using a Western Digital MyBook World Edition as an NFS server which will store the VM images. The MyBook is actually very cool; it's a quiet, visually appealing mirrored 1TB NAS. After manually enabling the NFS server, I was able to mount the shares in VMware but was unable to write to it. Attempting to create a VM would error out with the following in the messages log:

some bull

Hostd: [2010-01-22 19:45:36.384 5AA03B90 verbose 'ha-host'] ModeMgr::Begin: op = normal, current = normal, count = 0

Hostd: [2010-01-22 19:45:36.385 5AA03B90 info 'ha-eventmgr'] Event 18 : Creating win2k8 on host whateves.lan in ha-datacenter

Hostd: [2010-01-22 19:45:36.385 5AA03B90 verbose 'HostsvcPlugin'] CreateEntry '64'

Hostd: [2010-01-22 19:45:36.385 5AA03B90 verbose 'ResourcePool ha-root-pool'] Added child 64 to pool

Hostd: [2010-01-22 19:45:36.385 5AA03B90 verbose 'Vmsvc'] Create VM initiated [64]: /vmfs/volumes/0cb8d2a5-20f15722/win2k8/win2k8.vmx

Hostd: [2010-01-22 19:45:36.387 5AA03B90 verbose 'vm:/vmfs/volumes/0cb8d2a5-20f15722/win2k8/win2k8.vmx'] CreateVmDirectory: Creating vm dir (as vm principal user) '/vmfs/volumes/0cb8d2a5-20f15722/win2k8'

Hostd: [2010-01-22 19:45:36.388 5AA03B90 info 'App'] CreateDirectory: Calling _file->CreateDirectory with _file = [N7Vmacore6System8FileImplE:0x5af0ae58]

Hostd: [2010-01-22 19:45:36.388 5AA03B90 info 'App'] CreateDirectory: Calling _file->CreateDirectory for /vmfs/volumes/0cb8d2a5-20f15722/win2k8

Hostd: [2010-01-22 19:45:36.389 5AA03B90 verbose 'vm:/vmfs/volumes/0cb8d2a5-20f15722/win2k8/win2k8.vmx'] CreateVmDirectory: Failed to create vm dir (as vm principal user) '/vmfs/volumes/0cb8d2a5-20f15722/win2k8'.

Hostd: [2010-01-22 19:45:36.389 5AA03B90 verbose 'vm:/vmfs/volumes/0cb8d2a5-20f15722/win2k8/win2k8.vmx'] CreateVmDirectory: Creating vm dir (as superuser) '/vmfs/volumes/0cb8d2a5-20f15722/win2k8'

Hostd: [2010-01-22 19:45:36.390 5AA03B90 warning 'vm:/vmfs/volumes/0cb8d2a5-20f15722/win2k8/win2k8.vmx'] CreateVmDirectory: Failed to create vm dir '/vmfs/volumes/0cb8d2a5-20f15722/win2k8'

Hostd: [2010-01-22 19:45:36.391 5AA03B90 info 'vm:/vmfs/volumes/0cb8d2a5-20f15722/win2k8/win2k8.vmx'] Create failed with given spec: /vmfs/volumes/0cb8d2a5-20f15722/win2k8/win2k8.vmx

Hostd: (vim.vm.ConfigSpec) { dynamicType = <unset>,  changeVersion = <unset>,  name = "win2k8",  version = "vmx-07", uuid = <unset>, instanceUuid = <unset>, npivWorldWideNameType = <unset>, npivDesiredNodeW

wns = <unset>, npivDesiredPortWwns = <unset>, npivTemporaryDisabled = <unset>, npivOnNonRdmDisks = <unset>, npivWorldWideNameOp = <unset>, locationId = <unset>, guestId = "winLonghorn64Guest", alternateGuestName = "Microsoft Wi

ndows Server 2008 (64-bit)", annotation = <unset>, files = (vim.vm.FileInfo) { dynamicType = <unset>, vmPathName = "[VMs]", snapshotDirectory = "[VMs]", suspendDirectory = <unset>, logDirectory = <unset>, },

tools = (vim.vm.ToolsConfigInfo) { dynamicType = <unset>, toolsVersion = <unset>, afterPowerOn = true, afterResume = true, beforeGuestStandby = true, beforeGuestShutdown = true, beforeGuestReboot = true,

toolsUpgradePolicy = <unset>, pendingCustomization = <unset>, syncTimeWithHost = <unset>, }, flags = (vim.vm.FlagInfo) null, consolePreferences = (

Hostd: [2010-01-22 19:45:36.391 5AA03B90 info 'vm:/vmfs/volumes/0cb8d2a5-20f15722/win2k8/win2k8.vmx'] Exception thrown vim.fault.FileFault

Hostd: [2010-01-22 19:45:36.391 5AA03B90 info 'TaskManager'] Task Completed : haTask-ha-folder-vm-vim.Folder.createVm-172 Status error

Hostd: [2010-01-22 19:45:36.391 5AA03B90 verbose 'ha-host'] ModeMgr::End: op = normal, current = normal, count = 1

Hostd: [2010-01-22 19:45:36.391 5AA03B90 verbose 'vm:/vmfs/volumes/0cb8d2a5-20f15722/win2k8/win2k8.vmx'] RemoveFromAutoStart

Hostd: [2010-01-22 19:45:36.391 5AA03B90 verbose 'Hostsvc::AutoStartManager'] Request spec is (vim.host.AutoStartManager.Config) { dynamicType = <unset>, defaults = (vim.host.AutoStartManager.SystemDefaults) null, powerInfo

= (vim.host.AutoStartManager.AutoPowerInfo) [ (vim.host.AutoStartManager.AutoPowerInfo) { dynamicType = <unset>, key = 'vim.VirtualMachine:64', startOrder = -1, startDelay = -1, waitForHeartbeat = "n

o", startAction = "none", stopDelay = -1, stopAction = "none", } ], }

Hostd: [2010-01-22 19:45:36.391 5AA03B90 verbose 'Hostsvc::AutoStartManager'] Updated AutoStart sequence.

Hostd: [2010-01-22 19:45:36.392 5AA03B90 verbose 'Hostsvc::AutoStartManager'] Autostart info after reconfiguration: (vim.host.AutoStartManager.Config) { dynamicType = <unset>, defaults = (vim.host.AutoStartManager.SystemDefault

s) { dynamicType = <unset>, enabled = <unset>, startDelay = 120, stopDelay = 120, waitForHeartbeat = false, stopAction = "PowerOff", }, }

Hostd: [2010-01-22 19:45:36.392 5AA03B90 verbose 'Vmsvc'] Released Vm Id: 64.

Hostd: [2010-01-22 19:45:36.392 5AA03B90 verbose 'HostsvcPlugin'] RemoveEntry '64'

Hostd: [2010-01-22 19:45:36.392 5AA03B90 verbose 'HostsvcPlugin'] RemoveEntry succeeded

Hostd: [2010-01-22 19:45:36.392 5AA03B90 verbose 'ResourcePool ha-root-pool'] Removed child 64 from pool

Hostd: [2010-01-22 19:45:36.392 5A9C2B90 verbose 'App'] Looking up object with name = "64" failed.

Hostd: [2010-01-22 19:45:36.392 5AA03B90 info 'vm:/vmfs/volumes/0cb8d2a5-20f15722/win2k8/win2k8.vmx'] Create worker thread failed

Hostd: [2010-01-22 19:45:36.393 5AA03B90 verbose 'Statssvc'] EntityRemovedListener: Deleting stats for entity 64

Hostd: [2010-01-22 19:45:36.565 5AA44B90 verbose 'DvsTracker'] FetchDVPortgroups: added 0 items

Hostd: [2010-01-22 19:45:41.690 5AA03B90 verbose 'App'] Looking up object with name = "haTask-ha-host-vim.host.DatastoreSystem.removeDatastore-114" failed.

After searching the 'nets for about an hour, I found a post on MyBook World that addressed my issue. By default the MyBook mounts NFS shares as read-only. To change that, I modified the /etc/exports file, changing all instances of "ro" to "rw."

/nfs/Public *(rw,all_squash,sync,insecure,anonuid=65534,anongid=65534)

/nfs/Download *(rw,all_squash,sync,insecure,anonuid=65534,anongid=65534)

Then, I restarted the service by issuing /etc/init.d/S80nfsd restart. Once the restart was complete, my VM creation was successful

Posted by: Chrissy Filed under: Networking, Security, Virtualization 2 Comments

11Jan/091

VMWare ESXi 3.5: Creating an ISO Image from a CD/DVD-ROM

I always use ISOs in the datastore as CD/DVD devices in my virtual machines. Tonight, I had to make images of my Windows 2008 DVDs but because my tiny laptop doesn't have a DVD/CD-ROM build-in, I had to rely on my ESXi servers to create the ISOs. Creating an ISO in ESXi is as almost easy as creating one in Linux -- the only thing that presents a small challenge is VMware's use of an unconventional device path for the DVD/CD-ROM. Ultimately, I used VIC (VM->Properties->CD/DVD->Host Device) to find the name of my device, which turned out to be "/vmfs/devices/genide/vmhba0\:0\:0"

After opening up the SSH port on ESXI by using this tutorial, I created an ISOs folder in the datastore, placed the DVD in the drive and, finally, created the ISO by issuing the following command:

dd if=/vmfs/devices/genide/vmhba0\:0\:0 of=/vmfs/volumes/datastore3/isos/win2k8EEx64.iso

Posted by: Chrissy Filed under: Linux, Virtualization 1 Comment

10Jan/093

VMware Server 2 for Linux: HOW-TO Disable VMware Virtual Infrastructure Web Access

Many people are not aware that VMware Server's resource intensive Web Access/Web MUI is not required to administer VMware Server 2. As I detailed in a previous post, the same Windows-only thick client used to administer ESX Server, Virtual Infrastructure Client (VIC) 2.5, can also be used for administration.

While the "webAccess" MUI can be disabled, disabling the rather lightweight HTTP/HTTPS services entirely is not recommended, as they are required for VIC to work. I still can't believe how much RAM webAccess requires -- more than quadruple that of the virtualization engine itself!

According to pmap, webAccess (which can be found at /usr/lib/vmware/webAccess/java/jre1.5.0_15/bin/webAccess) uses ~485MB of RAM while the virtualization engine, hostd uses just 120MB. So if you'd like to disable that monstrosity, you will have to modify one file. If you don't want to do so without shutting down the virtualization engine itself, you will have to modify the same file twice. Here's how to do it without shutting down the engine.

Modify /etc/init.d/vmware, comment out line 1202:

#vmware_stop_hostd

Next, shut down webAccess.

service vmware-mgmt stop

Then, open up /etc/init.d/vmware again and uncomment line 1202 then comment out line 1901.

#vmware_start_webAccess

hostd is already running so you don't need to stop/restart anything at this point. Now you're done -- later hosebeast of a poorly designed MUI. Btw, If you want to turn off HTTP/HTTPS access completely (strangely, I didn't see any reduction in memory usage when I did this), comment out the _entire_ <proxysvc> section in the file /etc/vmware/hostd/proxy.xml and restart hostd.

service vmware-mgmt stop && service vmware-mgmt start

Edit: Joe @ 4sysops.net points out a downside:

The only down side to doing that is if you use the VI Client you can only create and manage virtual machines at version 4, not version 7 like the web interface uses. So, if you create a VM in WI, you can’t manage it with VI Client. You would want to pick one or the other to use primarily. I prefer the web interface, unless I were to add a VMware Server 2.0 machine to Virtual Center. I am hoping they update virtual center at some point to support version 7 virtual machines.

So you may want to consider that before disabling webAccess. Ultimately, I ended up wiping VMWare Server 2 and using ESXi 3.5 instead.

Posted by: Chrissy Filed under: Virtualization 3 Comments

8Jan/094

HP DL380 G4 and VMware: Will NOT Run 64-bit Virtualized Guests.

I mean, I'm not upset about it. I'm just sayin. I read all the Internet debates about how it might or it might not and I can confirm that I tried it and it didn't work. The server runs the host OS, SuSE Linux 64-bit, beautifully but it totally craps out when a 64-bit guest OS install is attempted. EM64T doesn't do VT so.. if that's what you're looking for, start with the HP DL380 G5.

This server still has its benefits however. Including 6 drive slots (2x72,2x147,2x36 SCSI YAY), capacity for up to 12 GB of RAM (I installed 10) and it's decently affordable. Another thing that I love about it is that I can change the POST text. How fun is that? So I made it say something that makes me laugh everytime. See if you can spot it:

Mmm 10240 MB Initialized... *drool*

Posted by: Chrissy Filed under: Virtualization 4 Comments

8Jan/0814

VMware Server 2 Beta: Use Virtual Infrastructure Client to Speed Up Administration

The web interface for VMware Server 2 Beta for Linux is garbage; it's both slow and unattractive. Even VMware Server 1 and ESX Server 2.5 from years ago had a faster/nicer web interface. VMware Server looks a bit like ESX and my experience with ESX taught me that it can be administered with both the web interface and Virtual Infrastructure Client (VIC). I wondered if perhaps Server 2 could be administered with VIC too and fortunately, my hunch was confirmed by the VMware forums.

VIC on VMware Server Beta 2 is much faster than the web interface and even provides a more information about the VMs resource histories. It can be assumed that pushing the slower web interface for the free product isn't so much tech driven as it is marketing/$$ driven but that's only a guess. VIC is a big part of the high end, high price ESX server but can be also be found hidden in the rpms and tarballs of VMware Server. I could not find it, however, in the Windows version of VMware Server 2, even after extracting contents of the executable using the /a option.

To find the VIC (Windows only client, Linux clients are out of luck) in an RPM, run the following commands:

mkdir vmware

mv VMware-server-e.x.p-63231.x86_64.rpm vmware/

cd vmware

rpm2cpio VMware-server-e.x.p-63231.x86_64.rpm  | cpio -i --make-directories

The file can then be found at ./usr/lib/vmware/hostd/docroot/client/VMware-viclient.exe. As for the tarball, expand it (tar -xvzf or WinRAR in Windows) and the file can be found at ./vmware-server-distrib/lib/hostd/docroot/client/VMware-viclient.exe.

The thick client is so much nicer; I know it's unlikely that I'll ever use the resource intensive MUI so I uninstalled it by running:

chkconfig httpd.vmware off

vmware-uninstall-mui.pl

Even though I ran the uninstaller, the MUI magically started up on the next reboot so I modified the permissions on /etc/init.d/vmware and then commented out the following line: $watchdog -s webAccess -u 30 -q 5 "$webAccess $webAccessOpts start" > /dev/null 2>&1 &. I then restarted the vmware service and it worked exactly as I hoped.

Aside from the bad web interface, I'm really impressed by this version of VMware server and I'm definitely recommending it at work once the final arrives. I honestly hope that Microsoft's new virtualization platform can impress me as much and even more once their product matures. As for xen, I successfully set it up in SuSE, it was eas as pie. However, my Opteron 270 doesn't appear to support hardware virtualization (even though AMD's docs say they do, perhaps I have to upgrade my BIOS) so I can't run Windows VMs. Totally unacceptable. xen is something I want to keep an eye on, though. Big companies like Citrix, Oracle and Sun are using it in their own virtualization platforms. Now to find a test server that supports hardware VT...

Update: You can also find the VMware-viclient.exe here on some .edu website. I haven't used it and can't vouch for its safety, but it's there (at least for now) in the event that you don't want to go through all of the above steps. The timestamp on it is December 2007 which is good for now, but I woudln't use it past June 2008.

Posted by: Chrissy Filed under: Linux, Virtualization 14 Comments

2Oct/072

Install VMware Server 1.0 on SuSE 10.2 x64

Ahh! One of my servers had a bad stick of RAM and caused all sorts of problems with VMWare ESX Server. At first, I thought ESX was too sensitive but later realized the stick was just super bad. Meanwhile, my evaluation version expired and so I decided to use VMware Server 1.0 (free) on top of SuSE 10.2 (also free).

Thankfully, this dude setup a really nice guide to get around some kernel issues in SuSE. It's pretty simple; before installing the VMware Server RPM, I ran the following:

# cd /usr/src/linux
# make mrproper; make cloneconfig; make modules_prepare

After installing the RPM, I ran vmware-config.pl and VMWare complained that a few files were missing. As it turns out, I needed the x86 version of a few packages. I loaded up
Yast -> Software -> Software Management -> Search -> [X] Provides -> [Missing Filename here]. I believe I ended up installing the following packages:

Several.. (many auto-selected themselves)

xorg-x11-libICE-32bit-7.2-13.x86_64.rpm

xorg-x11-libXau-32bit-7.2-8.x86_64.rpm

xorg-x11-libXdmcp-32bit-7.2-8.x86_64.rpm

xorg-x11-libSM-32bit-7.2-12.x86_64.rpm

xorg-x11-libX11-32bit-7.2-13.x86_64.rpm

xorg-x11-libXext-32bit-7.2-12.x86_64.rpm

xorg-x11-libXrender-32bit-7.2-12.x86_64.rpm

xorg-x11-libXt-32bit-7.2-13.x86_64.rpm

expat-32bit-2.0.0-32.x86_64.rpm

xorg-x11-libXfixes-32bit-7.2-13.x86_64.rpm

xorg-x11-libXmu-32bit-7.2-13.x86_64.rpm

xorg-x11-libXp-32bit-7.2-8.x86_64.rpm

xorg-x11-libXpm-32bit-7.2-12.x86_64.rpm

xorg-x11-libXv-32bit-7.2-8.x86_64.rpm

xorg-x11-libxkbfile-32bit-7.2-12.x86_64.rpm

zlib-32bit-1.2.3-33.x86_64.rpm

freetype2-32bit-2.2.1.20061027-11.x86_64.rpm

xorg-x11-libXprintUtil-32bit-7.2-8.x86_64.rpm

xorg-x11-libfontenc-32bit-7.2-12.x86_64.rpm

fontconfig-32bit-2.4.1-19.x86_64.rpm

xorg-x11-libs-32bit-7.2-19.x86_64.rpm

audit-libs-32bit-1.2.6-20.x86_64.rpm

cracklib-32bit-2.8.9-20.x86_64.rpm

libstdc++41-32bit-4.1.2_20061115-5.x86_64.rpm

libxcrypt-32bit-2.4-30.x86_64.rpm

db-32bit-4.4.20-16.x86_64.rpm

pam-32bit-0.99.6.3-24.x86_64.rpm

Next, used YaST to open up my firewall's port 902. Everything seemed to go well until I ran into PAM issues while attempting to remotely manage it using the VMWare Server Console (Windows). I received the error Permission denied: Login (username/password) incorrect. So I took a look at /var/log/messages and found this crappy news:

vmware-authd: PAM unable to dlopen(/usr/lib/vmware/lib/libpam.so.0/security/pam_unix2.so)

vmware-authd: PAM [error: /usr/lib/vmware/lib/libpam.so.0/security/pam_unix2.so: cannot open shared object file: No such file or directory]

After searching the web for a solution (thanks web!), I edited /etc/vmware/pam.d/vmware-authd and now it looks like the following:

#%PAM-1.0

#auth       sufficient       /usr/lib/vmware/lib/libpam.so.0/security/pam_unix2.so shadow nullok

#auth       required         /usr/lib/vmware/lib/libpam.so.0/security/pam_unix_auth.so shadow nullok

#account    sufficient       /usr/lib/vmware/lib/libpam.so.0/security/pam_unix2.so

#account    required         /usr/lib/vmware/lib/libpam.so.0/security/pam_unix_acct.so

auth sufficient /lib/security/pam_unix.so shadow nullok

auth required /lib/security/pam_unix_auth.so shadow nullok

account sufficient /lib/security/pam_unix.so

account required /lib/security/pam_unix_acct.so

Once that was done, I created a symbolic link to make restarting VMWare more comfy (ln -s /etc/init.d/vmware /usr/sbin/rcvmware), then I restarted the vmware service (rcvmware restart) and connected successfully from my remote machine. Now I'm happily installing Windows Server 2008 RC0. Hooray!

And my procrastination paid off -- while I was waiting for the motivation to troubleshoot the RAM issue, the price of my server's RAM dropped drastically -- from $160 to $99. Niiiice! I'm buying 5 for a total of 8 Gigs

Posted by: Chrissy Filed under: Linux, Virtualization 2 Comments

24Apr/073

Installing Longhorn x64 on VMWare ESX Server 3.0.x

I recently attended a Longhorn Roadshow in Santa Clara and learned quite a bit about Microsoft's emphasis on virtualization in Longhorn. A lot of companies are going towards virutalizing servers, even those still running NT or Exchange 5.5. The main reasons seem to be saving rackspace and saving electricity (fewer machines, less A/C) which both translate to saving money. Fortunately, my employer now has the infrastructure setup and virtualization on a mass scale seems like a possiblity. After a quick evaluation, I don't have much faith in Microsoft's current Virtual Server product but an evaluation of ESX Server 3.0 has proven impressive. VMWare has it together and it is likely the solution I'll be recommending in '08 when we're ready to move forward.

That said, it's been tough installing Longhorn x64 on VMWare ESX server. It should be expected, though; the support for Longhorn x64 isn't even experimental yet -- it's non-existent. I had to select Vista 64-bit Experimental as my base VM and hope for the best. What I've experienced is almost as painful as installing Windows 2003 R2 on a Macbook Most of the frustration revolves around the CD-ROM drivers. The initial install of Longhorn on ESX is so promising but then a message pops up that says: "A required CD/DVD drive device driver is missing." At first, I thought this was because I was using an external USB DVD-R drive but that turned out not to be the case. I figured that gem out only after going through all these steps:

I installed some dumb .flp that never loaded the CD-ROM drivers as promised
I asked a friend to bring me an internal CDROM only to find out it's EIDE and my server doesn't support it
I took my work workstation's IDE CD-ROM and hooked it half-up to my server (the IDE cable) and half-up to a workstation (the power cable because my server didn't have any free power cables left).
Enabled IDE in the BIOS and finally had ESX recognize the drive
Still had the same problem

So then I got creative and decided to create the Longhorn image on another workstation. Doh! The workstation's CPU was not 64-bit enabled. So then I tried it on my laptop.. doh! It's 64-bit enabled but doesn't have some special VT chip that's often times not found in laptops. This is why I hate hardware.

So I gave in and ..

Wiped ESX and reinstalled Longhorn 64-bit.
Installed the free VMware server, created a Longhorn Virtual Machine and installed Longhorn
Once the install was complete, I backed up the vmdk to another machine
I then wiped Longhorn on the server, resinstalled ESX and copied the vmdk to /vmfs/volumes/storage1/longhorn
Next, I ran vmkfstools -i longhorn-64ws.vmdk longhorn-64esx.vmdk
Once that was done, I created a new Virtual Machine within ESX and selected Custom then used the new image longhorn-64esx.vmdk

Ahhh, that worked! But now VMWare tools was giving me trouble. The CD-ROM still didn't work (the CD-ROM, listed as NECVMWar VMWare IDE CDR00 ATA Device, gives the status of "The Device Cannot Start" (Code 10)) so I had to figure out another way around the problem. I copied the windows.iso from /vmimages/tools-isoimages to my workstation using Veeam FastSCP, mounted the ISO, saved the files as a zip under my web root then used IE on the Longhorn server to fetch the zip. Installed and voila, it works!

Posted by: Chrissy Filed under: Virtualization, Windows 3 Comments