vCloud Director and vCloud Automation Center (vCAC)

There has been many discussions about the future of vCloud Director and if VMware intends to deprecate it. There is not much information from VMware about its vision but recently there is a great emphasis on vCloud Automation Center for cloud management that is now embedded in vRealize Suite. It seems a great product, especially for enterprises considering that it integrates management of different infrastructures (even physical and public cloud) into a single platform; but I still have some questions when I look at both vCloud Director and vCAC in regards to service providers who offer services to completely different, critical isolated organizations. I have read this blog post by VMware that explains VMware vision of Cloud management and transition plan, but still unsatisfied!

By the way, it seems that sooner or later vCAC will be dominant solution for Cloud management and VMware will add features to satisfy different requirements rapidly. So, it’s a good time to get familiar with this product. I saw this post that is a very good practical introduction and hand out that explains and clarifies basic concepts in vCAC. There is also another simple, more detailed manual here. Of course, more information can be found in VMware documentation.

p.s – From business perspective, it’s a smart move from VMware to focus more on enterprises rather than service providers because cloud service provider market is and will be dominated by other players.

NAT in Fenced vApps (vCloud Director)

An interesting feature in vCloud Director networking is the capability of creating a fenced vApp. Basically, it’s like having an extra  (in case you have one for Organization network which means routed) vShield router and firewall on the edge of vApp.

One of the coolest applications for fenced vApps is when you want to have identical machines (same IP and MAC) in your vDC; it means when you want to do a fast clone without customizing guest OS by changing IP’s and names, … In this case vApps are completely isolated while they can have connection to External networks or perhaps internet! See here for a how-to about creating fenced vApp.

After you created a fenced vApp, you will notice that the IP addresses in the vApp are in the same subnet with Organization Network (see the picture above), although a NAT gateway is operating between the vApp and Organization network. So when you want to do a DNAT (Destination NAT), there are 2 places you should configure. In the picture above, suppose you want to give access to a VM with IP 192.168.0.45 in Fenced vApp from External Network. Assume that Edge 1 got IP 192.168.0.3 (specified while fencing). First, you need to create appropriate rules in Edge Gateway of Organization Network, Edge 2 (if there is any) to NAT and open ports for the IP address of Edge 1 (192.168.0.3)

fenced1

Next step, you need to do NAT and open ports from Edge 1 to specific VM but this configuration is not in Edge Gateways of vDC (unlike Edge 2) but can be found in Networking Tab of the vApp itself.
Click on the vApp, go to Networking tab,

fenced2

right click on the selected network and choose ‘Configure Services’. there, you can define appropriate NAT and firewall rules.

fenced3

 

VPN access for vCloud Director Customers

Providing VPN access to vCD customers is a great idea, because usually customers are behind a vCloud created firewall and most likely you created a routed organization VDC network to connect them to external network. So, how they have to get access to their VDC? One approach could be to define different sets of firewall and NAT rules for required access ports (SSH, RDP, …) but when the number of VM’s grow, this would be less flexible but of course still doable. Even a customer can get access to only one VM and go through this single VM to access the others; however sometimes it’s not a simple remote access and remote user wants to do a more advanced task.

By the way, I don’t want to go into the details of benefits of having a VPN for remote clients but it seems like a very helpful facility for cloud customers. We can leave it to the user to install its own VPN server to tunnel through to get access to organization VDC network but VMware provides this excellent capability to setup VPN gateways in vCloud Director or vSphere Cluster. For a Site-to-Site IPsec VPN, VMware vCD is pretty much straight forward. So, if you have a VPN gateway in place, easily you can establish a tunnel between your local network and your organization network in the cloud. I found this guide about setting up an IPSec tunnel in vCloud Director with useful examples, one with a Cisco WAN router. Here is another guide for a Cisco PIX and vCD; although the vCD version is old (1.5) but it’s too similar in terms of VPN tunnelling.

However, if you don’t have a VPN endpoint in-place and still want to establish a secure VPN-connection with your vCD organization network as a remote user, VMware provides this brilliant SSL VPN utility. It’s not as straight forward as IPsec VPN and it’s not present in vCD web portal but it worths deploying (especially for customers). VMware SSL VPN should be configured in vCloud Networking and Security solution (which is a new name for vShield Manager).

I’m not writing a How-To for this here and a complete step by step guide can be found here, very well explained by Ranga Maddipudi. I just wanted to give some idea and as you can see, deploying a SSL VPN gateway is fairly easy and an installable file (.exe file for Windows) will be provided. To get this file, on the client side use should use the browser to download the file. The URL for downloading the package would be: https://external-ip-address-of-gateway:443
After getting the file, user can easily install the VPN client and that’s it.

SSL-VPN1

Running the application and entering the right credentials, VPN connection will be established and given that the configurations are server side are well defined, remote user will get access to VDC organization network in the cloud. In fact, what excites me is that from engineering point of view, VMware did a great job to ease the whole procedure of setup a connection on both server and client side; in specific, generating a custom designed VPN client using SSL (as authentication and encryption protocol) VPN is a brilliant idea.

SSL-VPN2

VMware vCloud Windows Customization Requirements

Sometimes you may notice that your customization of a VM fails when you provision VM from template or import it from vSphere; for example Computer Name may not be changed or IP address can not be assigned.

There are a couple of documents about troubleshooting guest operating system customization in VMware kb: for vCloud Director and vCenter. But none of them worked in my specific case, so I’m sharing something important in Windows OS’s. This hint will be useful especially when you see this error in: C:\Windows\Temp\vmware-inc\guestcust.Log:
“Command Execution failed with exist code: 1, output: ‘The service can not be started.’ ”

Since Windows administrators tend to disable some unnecessary services to harden security, we should know which services are necessary for VMware, if there is any. In fact, a number of Windows services should be enabled and started so that VMware customization works properly.  The necessary services are:

  • DNS Client
  • DHCP Client
  • Workstation
  • TCP/IP NetBIOS Helper

So, if you face the same issue, besides viewing Logs on VM, check these services as well.

vCloud Public Console Proxy IP Address

You may have noticed that vCloud Director uses 2 important IP addresses to provide public access to tenants/users. One is the well-known front-end VCD IP address which is access to web portal for managing the organization vDC (also known as HTTP access) and second one provides remote access to virtual console of VM which is in fact resided on ESXi server cluster (known as VRMC access), this latter one is sort of more back-end because it’s coming from ESXi server which never should be exposed to public! So, vCloud Director actually tunnels Remote Console communications between ESXi servers and users through a proxy agent on port 443. Apparently, the proxy service runs on vCloud Director machine. That’s why an extra IP is needed on vCloud Director. This IP address is also specified in initial setup but it can be changed later (of course everything can be changed!).

So, when you want to open up vCloud Director for public users, you should pay enough attention to VRMC IP address and port. If you  have to do NAT through your firewall you should specify a different IP for VRMC and introduce the public IP/URL to vCloud Director in administration web panel. See the picture below:

Also, port 443 should be opened for this public IP on the firewall.

If you need more information about publicizing the whole vCloud Director, I found this excellent blog post about this topic, although it’s very useful for a general architecture of vCD deployment:

Upgrade VMware vCloud Networking and Security to 5.5

When you want to upgrade VMware vCloud Director 5.1 to version 5.5, VMware vCloud Networking and Security should be upgraded to 5.5 as well because vCloud Director 5.5 is not backward compatible with vShield 5.1. Unfortunately and surprisingly, the documentation on VMware website to do so is not accurate! and it took some time for me to realize what’s the correct procedure. So, I thought it would be useful to share it here for those who can’t find the things mentioned in VMware website to upgrade VMware vCloud till they modify the documentation.

Actually, the part which is not accurate is where it explains the steps to upgrade vShield Edge appliances and it is a crucial part because failing to do this will result in failure in managing Edge gateways through vCloud Director interface. Since there would be one vShield Edge system appliance for each Edge Gateway that is created in Virtual Data Centers, you will have considerable number of vShield Edges in your environment and you should take care of them one by one. By the way, after upgrading vShield Manager to 5.5 is done (it’s easy, just uploading the upgrade bundle in vShield web console and reboot), the most important one is Upgrading vShield Edge.

Let’s look at the document on VMware website: Best practices for upgrading to VMware vCloud Networking and Security 5.5 , it says:

If you have vShield Edge 5.1.0 or later instances, upgrade each Edge:

  1. Log in to the vSphere Client.
  2. Click the data center for which vShield Edge instances are to be upgraded.
  3. Click the Network Visualization tab. All existing vShield Edge instances are shown in the listings page. An arrow icon is shown for each vShield Edge that must be updated.
  4. Click an Edge and click Upgrade from Actions to start the upgrade. When the Edge is upgraded, the arrow icon no longer appears.
  5. Repeat for each vShield that must be upgraded

After logging in to vSphere client, you will notice that there is no “Network Visualization” tab, but instead “Network Virtualization” tab. It must be a typo but even after clicking on “Network Virtualization” you will face some errors complaining about not having Acrobot Adobe Reader and so on! While the proper way to do this is to Log in to vShield Manager Web console (not vSphere Client) and look for Network Virtualization (instead of Visualization) under your DataCenter and the rest of steps are the same. You need to choose each Edge, select “Actions” on the top and choose “Upgrade” which will result in an automatic upgrade of that special Edge Gateway to new version.

vshield5.5

I hope they modify their document because it is considered as the most reliable one.

 

Is it safe to reboot MS SQL server in VMWare environment?

VMware vCenter and VMware vCloud need a database to store important information (most importantly, configuration). Due to critical nature of data, database server needs to be an enterprise class one. Supported databases are Microsoft SQL, Oracle (for a full list, see here). Of course, high availability should be considered for database server, but  you may wonder if it’s safe to restart database server for a short time? For example, say you still didn’t implement high availability and you need to do a Windows update. You want to reboot database server but you don’t intend to reboot the whole environment, I mean vCloud Director, vCenter itself, … So, the question would be: Is it possible that rebooting database server causes crash or any harm in other VMware components?
I decided to experience this in my Lab environment and the answer is: It’s generally safe to reboot! And it seems reasonable; as long as you are not changing configuration on your infrastructure.
Although, when I started some administration jobs in vCloud Director, like modifying a VM or adding a VM to a vApp, I got some weird error messages.  In fact, vCloud Director complained: “Error while connecting to sphere profile driven storage service”. I never saw this before and actually I’m not sure what profile driven storage service is! So, I looked into my vCenter server. In Administration, Management, there was an icon, named ‘VM Storage Profiles’. It looked relevant, so I clicked on it. The error message appeared here too! Looking into the issue more, It turned out that there is a Windows Service named ‘VMware vSphere Profile-Driven Storage Service’ that was stopped, while it was ‘Automatic’ service.  I started the service and everything got back to normal.
It means that we can’t say rebooting database server is completely safe and some unexpected issues may happen. If you have to reboot your database server, make sure to check the health of your other servers (vCloud Director and vCenter in specific) by looking into Logs, Services, …

p.s – My Lab environment included MS SQL 2008 R2, vCenter 5.1 (on Windows 2008 R2), vCloud Director 5.1.2 (on RedHat 6)