VMware vCD Guest Customization for Debian

One of the great features in vCloud Director is client’s capability to customize general specifications of a VM. Specifications like Hostname and more importantly IP address(es). Customer can even have some scripts for more advanced customizations like joining to a domain, … All these depend on ‘Guest OS Customization’ feature that should be enabled on a VM. Not all the operating systems support ‘Guest OS Customization’. For a list of supported OS’s in vCD 5.x see these links:

Supported guest operating systems in vCloud Director 5.5
– Supported guest operating systems in vCloud Director 5.1

As you can see, there is no support for Debian Linux! What a pity! If you deploy a Debian and want to change its IP through VM Properties in vCD portal, it will give you an error:

“Guest customization is not supported by the selected OS. Please disable guest customization to proceed.”

Debian_custom

Debian is a great OS and many clients may get disappointed! But fortunately, there is a simple work-around for it: change the Operating System type to: Other Linux and Guest Customization will be fine! Of course, try to choose the closest kernel version, for example choose ‘Other 2.6.x Linux (64-bit)’ for a Debian wheezy with kernel 3.2.0-amd64.

Debian_custom2

By this change, modifying Hostname or assigning IP addresses, Gateway, DNS to Debian NICs would be possible like any other supported OS.

VPN access for vCloud Director Customers

Providing VPN access to vCD customers is a great idea, because usually customers are behind a vCloud created firewall and most likely you created a routed organization VDC network to connect them to external network. So, how they have to get access to their VDC? One approach could be to define different sets of firewall and NAT rules for required access ports (SSH, RDP, …) but when the number of VM’s grow, this would be less flexible but of course still doable. Even a customer can get access to only one VM and go through this single VM to access the others; however sometimes it’s not a simple remote access and remote user wants to do a more advanced task.

By the way, I don’t want to go into the details of benefits of having a VPN for remote clients but it seems like a very helpful facility for cloud customers. We can leave it to the user to install its own VPN server to tunnel through to get access to organization VDC network but VMware provides this excellent capability to setup VPN gateways in vCloud Director or vSphere Cluster. For a Site-to-Site IPsec VPN, VMware vCD is pretty much straight forward. So, if you have a VPN gateway in place, easily you can establish a tunnel between your local network and your organization network in the cloud. I found this guide about setting up an IPSec tunnel in vCloud Director with useful examples, one with a Cisco WAN router. Here is another guide for a Cisco PIX and vCD; although the vCD version is old (1.5) but it’s too similar in terms of VPN tunnelling.

However, if you don’t have a VPN endpoint in-place and still want to establish a secure VPN-connection with your vCD organization network as a remote user, VMware provides this brilliant SSL VPN utility. It’s not as straight forward as IPsec VPN and it’s not present in vCD web portal but it worths deploying (especially for customers). VMware SSL VPN should be configured in vCloud Networking and Security solution (which is a new name for vShield Manager).

I’m not writing a How-To for this here and a complete step by step guide can be found here, very well explained by Ranga Maddipudi. I just wanted to give some idea and as you can see, deploying a SSL VPN gateway is fairly easy and an installable file (.exe file for Windows) will be provided. To get this file, on the client side use should use the browser to download the file. The URL for downloading the package would be: https://external-ip-address-of-gateway:443
After getting the file, user can easily install the VPN client and that’s it.

SSL-VPN1

Running the application and entering the right credentials, VPN connection will be established and given that the configurations are server side are well defined, remote user will get access to VDC organization network in the cloud. In fact, what excites me is that from engineering point of view, VMware did a great job to ease the whole procedure of setup a connection on both server and client side; in specific, generating a custom designed VPN client using SSL (as authentication and encryption protocol) VPN is a brilliant idea.

SSL-VPN2

Troubleshooting WordPress Screen of Death

Last week I was asked to look into an on-premise WordPress website with a very low performance. I’m not expert in WordPress but I could say that comparing to the structure of the website and contents, it was too slow. Examining the system logs, I understood that memory usage reaches to its limit very soon as a result of huge consumption of Apache processes; in fact eventually Apache was returning white screen (500 error) to the visitors.  So, definitely something was wrong and simple troubleshooting guides mentioning to disable plugin and themes and even overwriting wordpress files didn’t help me! Clueless!

But when I was backing up the wordpress database to move it to a fresh Linux machine, something came to my attention: the dumped file was too big for their contents. Also, as soon as I imported the db into new mySql, the website became slow and eventually went down! So, it turned out that the issue is in the WordPress database. Using ‘phpmyadmin’ I found the largest table which was ‘wp_options’! I ran a simple query on it to see what’s in it? browsing through the results, soon I saw some irrelevant stuff. OMG! WordPress database was hacked and some HTML pages were inserted into ‘wp_options’ table! No wonder that website was slow! the ‘option_name’ of this table was filled with stuff like ‘/?tid=michael-kors-sac-CclA21.html’ and the value was a complete HTML file! Around 35000 of these rows were inserted and made a huge database.

So, I started cleaning database and getting more information about hacking WordPress. By the way, I put the clean database in a fresh installation of WordPress in a new machine and asked them to follow the best practices for securing WordPress website. I didn’t find similar situation on Internet, so I though it worths sharing, although I’m not a WordPress expert!

Enlarge Linux Partition to sizes Larger Than 2TB

If you have partitioned a disk using ‘fdisk’, most probably the partition table is using ‘MBR‘. Nowadays, one important disadvantage of MBR is the lack of supporting larger than 2 TB partitions. So, if you want to extend a partition while disk is using MBR, first the disk partition table needs to be converted to ‘GPT‘. To find difference between MBR and GPT see here as well.

Generally, to resize a partition, it needs to be deleted and re-created using new size or end sector. A good general approach to enlarge a partition while not losing data is explained in this link. You can follow this how-to with a slight but important difference: instead of using ‘fdisk’, ‘gptdisk’ or ‘gdisk’ should be used. ‘gdisk’ supports GPT and if you use the existing specifications (first sector) for new partition, there would be no data loss. When doing conversion, a warning will be triggered:

THIS OPERATION IS POTENTIALLY DESTRUCTIVE! Exit by typing 'q' if you don't want to convert your MBR partitions to GPT format!

But it’s okay! Don’t worry and continue. Then, it will ask for partition type, because the current partition type is based on MBR partition table. You can choose ‘ef00’ which is ‘EFI System’. Go ahead and create the new partition with new size, save the partition table and you are done. Then ‘resize2fs’ can be used to enlarge volume.

p.s – instead of enlarging partition, another choice (rather than MBR to GPT) is using LVM to create large logical volumes containing multiple physical volumes.

Modify VMware vCenter 5.5 Alarm Notifications using PowerCLI

Configuring or modifying alarm notifications on vCenter can be time consuming because there are more than 80 pre-defined alarms (in vCenter 5.5) which need to be modified; and the point is that it’s not possible to apply a single change to all of them or a group of them at once. Via GUI (web client or native) alarms should be modified one by one. So, imagine that you want to add an email as recipient of some alarms, let’s say the ones which send email notification when status is changed from Yellow to Red and Yellow to Green. It’s difficult to go through all the alarms, find them and apply your changes. This kind of change is very common when a new administrator is added to the team or someone leaves.

It is one of the situations where using PowerCLI is really beneficial. You can customize and group the alarms as you want, change the variables whenever needed and run the script using PowerCLI. I found this template by Lauren Malhoit very useful but it was not compatible with vSphere 5.5. I modified the script for vSphere vCenter 5.5 and you can download it here. Using this script is easy by modifying email addresses and you can also change the groups (I mean for example adding to YRYG alarms) as you desired.

vCloud Director Resources Allocation

In vCloud Director (vCD) 5.x, you may have seen this error or heard a complaint from customer that when a new VM deployed, powering it on caused this error:

“The operation could not be performed, because there are insufficient memory resources” also “The available Memory resources in the parent resource pool are insufficient for the operation.”

This error might seem natural and you might suspect to the allocations of VDC but if the organization VDC is created using Allocation Pool model, the story is different. You sum up the memory assigned to all VMs and the total amount might be less than allocated amount in VDC. Most likely, it is because of the amount set in “Memory resources guaranteed” field of VDC. In fact, by default this is set to 20% and this value is the minimum number you can set. Since VMware reserves this amount of Memory, you should add it to the total amount (sum of all VMs) of RAM calculated before and then compare it to “Memory allocation”  that should be less or you will encounter the mentioned error message.

resource_alloc

So, to prevent this error, you must be more generous in assigning “Memory allocated” value of VDC because reducing “Memory resources guaranteed” less than 20% is not possible!

Quick Stats on Host not up-to-date

In VMware vSphere client (native and web client), sometimes you get this message in Summary tab of a host: “Configuration Issues: Quick stats on Host ‘xyz’ is not up-to-date”
Most of the times this message disappears after a while but sometimes it bothers for a long time. In those cases, a quick ‘Reconfigure for vSphere HA’ may clear the message.

VMware vSphere ESXi 5.5 and Heartbleed

Well, bad news for those who upgraded to version 5.5 is that: apparently VMware ESXi 5.5 is one of few VMware products that is being affected by heartbleed vulnerability! because of using OpenSSL 1.0.1. As VMware confirmed here, they are working on it to give out a patch; although because ESXi’s are usually deployed in a private, protected environment; the effects of this vulnerability is mitigated. Still, we should monitor VMware website for latest patches.

Upgrade Distributed vSwitch from 5.1 to 5.5

When you upgrade your VMware environment to version 5.5. remember to upgrade your distributed vSwitch as well; it will not be done automatically. In this way, you can utilize new features in dvSwitch 5.5, including:

The upgrade process is fairly easy and the good thing is that according to VMware documentation, it is non-disruptive which means there is no outage and no host and VM will get down or experience issues. Find your distributed vSwitch either in vSphere Client or Web client, right click and do upgrade.

VMware vSphere Big Data Extensions Installation – 3

After deploying Serengeti Management Server using OVF, it would be possible to install Big Data Extensions plugin in vCenter Server. Follow the instructions for doing this. After a Logout and Login (necessary), you will see Big Data Extensions plugin in vSphere Web Client Home, Inventories.

bigD_plugin3

Next step is connecting Big Data Extensions plugin to Serengeti Management Server. For a successful connection, 2 conditions should be met: First, Serengeti Management Server should be reached by vCenter in terms of networking. Ping is not blocked by default, so vCenter should be able to ping Serengeti Server. Secondly, SSO Lookup Service URL should be correct. If it’s not correct, you will get an error like: “Connection failed! Check the server has enabled SSO.” If you are not sure if SSO URL is entered and works properly, try to re-enter SSO Lookup service URL manually:

Login to Serengeti server and issue these commands:

– sudo /opt/serengeti/sbin/EnableSSOAuth https://VC-FQDN:7444/lookupservice/sdk
– sudo service tomcat restart

In my case, I first got the following error:

“com.vmware.vim.vmomi.core.exception.CertificateValidationException: Server certificate assertion not verified and thumbprint not matched
Return code is: SslHandShakeFailed
Please check if sso lookup service https url is correct, and sso service work normally.”

The issue was that I entered URL by IP and certificate was generated using FQDN. So, I recommend using FQDN of vCenter and not IP address.

Also, check the time of Serengeti server to by synced with vCenter.