NAT in Fenced vApps (vCloud Director)

An interesting feature in vCloud Director networking is the capability of creating a fenced vApp. Basically, it’s like having an extra  (in case you have one for Organization network which means routed) vShield router and firewall on the edge of vApp.

One of the coolest applications for fenced vApps is when you want to have identical machines (same IP and MAC) in your vDC; it means when you want to do a fast clone without customizing guest OS by changing IP’s and names, … In this case vApps are completely isolated while they can have connection to External networks or perhaps internet! See here for a how-to about creating fenced vApp.

After you created a fenced vApp, you will notice that the IP addresses in the vApp are in the same subnet with Organization Network (see the picture above), although a NAT gateway is operating between the vApp and Organization network. So when you want to do a DNAT (Destination NAT), there are 2 places you should configure. In the picture above, suppose you want to give access to a VM with IP in Fenced vApp from External Network. Assume that Edge 1 got IP (specified while fencing). First, you need to create appropriate rules in Edge Gateway of Organization Network, Edge 2 (if there is any) to NAT and open ports for the IP address of Edge 1 (


Next step, you need to do NAT and open ports from Edge 1 to specific VM but this configuration is not in Edge Gateways of vDC (unlike Edge 2) but can be found in Networking Tab of the vApp itself.
Click on the vApp, go to Networking tab,


right click on the selected network and choose ‘Configure Services’. there, you can define appropriate NAT and firewall rules.



MS SQL Server Clone in VMware

Having templates and cloning VM’s can be very handy for fast deployment. Suppose that you want to deploy an instance of a sophisticated Web application consisting of different functional servers like database, web, mail, messaging, etc. It is desirable to clone the whole application, saving lots of time to configure each server and establish connectivity between them. VMware enables us to do this by using vApp templates. vApp templates are also available in vCloud Director.

However, when it comes to Microsoft SQL Server, an issue is raised when you rename the server during cloning or customize operating system in vCloud! It’s because SQL server contains some internal databases (like master) and metadata that store system name and working with SQL server in this situation will cause problems. To prevent this issue, you can do the following:

1) Enable a sysadmin SQL account (like ‘sa’) before cloning.

2) After clone, login to new SQL Server using a non-Windows sysadmin (like ‘sa’)

3) issue the following commands in a Query window:
exec sp_dropserver ‘OldserverName’
exec sp_addserver ‘NewServerName’, ‘LOCAL’

4) Restart SQL Server services

Quick Stats on Host not up-to-date

In VMware vSphere client (native and web client), sometimes you get this message in Summary tab of a host: “Configuration Issues: Quick stats on Host ‘xyz’ is not up-to-date”
Most of the times this message disappears after a while but sometimes it bothers for a long time. In those cases, a quick ‘Reconfigure for vSphere HA’ may clear the message.

Upgrade Distributed vSwitch from 5.1 to 5.5

When you upgrade your VMware environment to version 5.5. remember to upgrade your distributed vSwitch as well; it will not be done automatically. In this way, you can utilize new features in dvSwitch 5.5, including:

The upgrade process is fairly easy and the good thing is that according to VMware documentation, it is non-disruptive which means there is no outage and no host and VM will get down or experience issues. Find your distributed vSwitch either in vSphere Client or Web client, right click and do upgrade.


To install VMware vSphere Big Data Extensions 1.1, if you satisfy the requirements mentioned in vmware document, go ahead with installation by deploying Big Data Extensions OVA as documented. But attention that:

  • Better to create a specific Resource Pool for your Big Data Cluster and specify the total amount of resources you want to assign and apply possible limits.
  • Create a port group dedicated to Big Data Extensions  as a communication link between management servers and working VMs.
  • When deploying Big Data Extensions Management server (OVA), ‘setup networks’ asks you to assign a destination port group. Note that: Management Network will use this network to communicate with vCenter server. So, if you use VLAN tags, the port group should be in the same VLAN (use same VLAN id) with vCenter network. If vCenter can not see Big Data Management server and vice versa, integration will not be made properly.


  • In ‘Customize template’ step, there are 2 important settings: SSO service and Management Server IP address. So, from right-pane open ‘VC SSO Lookup Service URL’ and ‘Management Server Networks Settings’. Enter appropriate values. For SSO Lookup Service URL, use vCenter server with the same format (if you didn’t change defaults), I mean port 7444/lookupservice/sdk. Use FQDN of vCenter and not IP address or certificate will not be accepted and you will see errors for connecting Big Data Extensions plugin to Serengeti server in the future.


VMware vSphere Big Data Extensions Installation – 1

Nowadays Big Data is everywhere. Many are talking about it and they are enthusiastic to deploy a Big Data instance in their environments. Installation and deployment can be difficult though. The fact is that there is no official mature Big Data standard and lots of open source standards are being developed, sometimes independently. Even if we accept Apache Hadoop as the dominant standard of Big Data, implementing Hadoop is a big challenge for IT departments. For example, according to this article: In addition to the technical challenges of deploying large-scale Hadoop systems and applications, another issue Manor cited is that IT operations often work in silos, with separate teams handling systems administration, database administration, storage, networking, security and application development. That approach can lead to problems in managing Hadoop clusters.

And it’s exactly where Virtualization, Cloud and SDN can help: integrating multiple administration tasks in a unified control center. And VMware did this beautifully by putting together all required Hadoop components in a package to create Clusters and control and scale the Hadoop Clusters by using VMware vSphere Big Data Extensions. Hadoop clusters which are created by vSphere Big Data Extensions are scalable, elastic and flexible. You can easily separate compute and data nodes or increase the number of working machines and so on. vSphere Big Data Extensions utilizes the open source project Serengeti that was initiated by VMware to implement Hadoop on a virtual platform. Serengeti or better to say VMware vSphere Big Data Extensions deploys HDFS, MapReduce, Pig, Hive and HBase on vSphere infrastructure.

You can find general installation instructions here, but there are some implementation tips which will help in vSphere Big Data Extensions installation. In my upcoming posts I will show the required steps and important considerations during installation.

VMware vCloud Windows Customization Requirements

Sometimes you may notice that your customization of a VM fails when you provision VM from template or import it from vSphere; for example Computer Name may not be changed or IP address can not be assigned.

There are a couple of documents about troubleshooting guest operating system customization in VMware kb: for vCloud Director and vCenter. But none of them worked in my specific case, so I’m sharing something important in Windows OS’s. This hint will be useful especially when you see this error in: C:\Windows\Temp\vmware-inc\guestcust.Log:
“Command Execution failed with exist code: 1, output: ‘The service can not be started.’ ”

Since Windows administrators tend to disable some unnecessary services to harden security, we should know which services are necessary for VMware, if there is any. In fact, a number of Windows services should be enabled and started so that VMware customization works properly.  The necessary services are:

  • DNS Client
  • DHCP Client
  • Workstation
  • TCP/IP NetBIOS Helper

So, if you face the same issue, besides viewing Logs on VM, check these services as well.

vCloud Automation Center Installation – 1

Installing VMware vCloud Automation Center is strongly recommended for the beautiful things that administrators or tenants can do. An example is deploying popular big data clusters using a very simple procedure (I will post a how-to soon for this). There is a perfect and comprehensive 7-part installation, configuration manual written by Kenny Coleman which can be found here.
So, if you didn’t deploy vCAC in your management environment yet, install and you will enjoy it!

p.s – I found Part 3 (Installing IAAS) of this guide the most difficult one. There are some hints that I would like to add:

  • To ease installation, don’t use external MS SQL server. Instead, install SQL Express 2012 on the same Windows machine (IAAS and Model Manager Server).
  • Make sure that DNS settings are correct and IAAS Server FQDN can be resolved.
  • If you don’t use Active Directory, YOU MUST specify domain name in Primary DNS suffix of System Properties to make computer full name like its FQDN. It’s very important that in Step 8, Current Server filled automatically with FQDN and not Stand-Alone Server name. Domain should be there. Or you will face with an error (in Logs) like:
    “Building Project “C:\Program Files (x86)\VMware\vCAC\Server\Model Manager Data\DeployRepository.xml” (VARegistrationFinalSteps target(s)) — FAILED.
    Build FAILED.
    Authentication issue
    this error occurs when either the username or password supplied to iis is invalid

vCloud Public Console Proxy IP Address

You may have noticed that vCloud Director uses 2 important IP addresses to provide public access to tenants/users. One is the well-known front-end VCD IP address which is access to web portal for managing the organization vDC (also known as HTTP access) and second one provides remote access to virtual console of VM which is in fact resided on ESXi server cluster (known as VRMC access), this latter one is sort of more back-end because it’s coming from ESXi server which never should be exposed to public! So, vCloud Director actually tunnels Remote Console communications between ESXi servers and users through a proxy agent on port 443. Apparently, the proxy service runs on vCloud Director machine. That’s why an extra IP is needed on vCloud Director. This IP address is also specified in initial setup but it can be changed later (of course everything can be changed!).

So, when you want to open up vCloud Director for public users, you should pay enough attention to VRMC IP address and port. If you  have to do NAT through your firewall you should specify a different IP for VRMC and introduce the public IP/URL to vCloud Director in administration web panel. See the picture below:

Also, port 443 should be opened for this public IP on the firewall.

If you need more information about publicizing the whole vCloud Director, I found this excellent blog post about this topic, although it’s very useful for a general architecture of vCD deployment:

Using NLB in VMware Environment

It’s very interesting that sometimes things work not in a way you expect. Well, it happens a lot in computer networking! By the way, using Microsoft Network Load Balancing in a VMware environment is one of them. In specific, when you intend to deploy Microsoft NLB in Unicast mode, you should be cautious. The reason for NLB not to work properly is well explained in the following Link:

Microsoft NLB not working properly in Unicast mode

In brief, NLB mechanism is based on hiding common, shared MAC address it assigns to all involved hosts from switch (by special kind of encapsulation I suppose) but ESX/ESXi hosts expose this common MAC address in certain conditions that will enable switch to learn the location and sends all the traffic to that specific port (ESX/ESXi host) which is against purpose of load balancer! There is a work around though which is suggested in the link above.