NAT in Fenced vApps (vCloud Director)

An interesting feature in vCloud Director networking is the capability of creating a fenced vApp. Basically, it’s like having an extra  (in case you have one for Organization network which means routed) vShield router and firewall on the edge of vApp.

One of the coolest applications for fenced vApps is when you want to have identical machines (same IP and MAC) in your vDC; it means when you want to do a fast clone without customizing guest OS by changing IP’s and names, … In this case vApps are completely isolated while they can have connection to External networks or perhaps internet! See here for a how-to about creating fenced vApp.

After you created a fenced vApp, you will notice that the IP addresses in the vApp are in the same subnet with Organization Network (see the picture above), although a NAT gateway is operating between the vApp and Organization network. So when you want to do a DNAT (Destination NAT), there are 2 places you should configure. In the picture above, suppose you want to give access to a VM with IP 192.168.0.45 in Fenced vApp from External Network. Assume that Edge 1 got IP 192.168.0.3 (specified while fencing). First, you need to create appropriate rules in Edge Gateway of Organization Network, Edge 2 (if there is any) to NAT and open ports for the IP address of Edge 1 (192.168.0.3)

fenced1

Next step, you need to do NAT and open ports from Edge 1 to specific VM but this configuration is not in Edge Gateways of vDC (unlike Edge 2) but can be found in Networking Tab of the vApp itself.
Click on the vApp, go to Networking tab,

fenced2

right click on the selected network and choose ‘Configure Services’. there, you can define appropriate NAT and firewall rules.

fenced3

 

Advertisement

vCloud Director Resources Allocation

In vCloud Director (vCD) 5.x, you may have seen this error or heard a complaint from customer that when a new VM deployed, powering it on caused this error:

“The operation could not be performed, because there are insufficient memory resources” also “The available Memory resources in the parent resource pool are insufficient for the operation.”

This error might seem natural and you might suspect to the allocations of VDC but if the organization VDC is created using Allocation Pool model, the story is different. You sum up the memory assigned to all VMs and the total amount might be less than allocated amount in VDC. Most likely, it is because of the amount set in “Memory resources guaranteed” field of VDC. In fact, by default this is set to 20% and this value is the minimum number you can set. Since VMware reserves this amount of Memory, you should add it to the total amount (sum of all VMs) of RAM calculated before and then compare it to “Memory allocation”  that should be less or you will encounter the mentioned error message.

resource_alloc

So, to prevent this error, you must be more generous in assigning “Memory allocated” value of VDC because reducing “Memory resources guaranteed” less than 20% is not possible!

vCloud Automation Center Installation – 1

Installing VMware vCloud Automation Center is strongly recommended for the beautiful things that administrators or tenants can do. An example is deploying popular big data clusters using a very simple procedure (I will post a how-to soon for this). There is a perfect and comprehensive 7-part installation, configuration manual written by Kenny Coleman which can be found here.
So, if you didn’t deploy vCAC in your management environment yet, install and you will enjoy it!

p.s – I found Part 3 (Installing IAAS) of this guide the most difficult one. There are some hints that I would like to add:

  • To ease installation, don’t use external MS SQL server. Instead, install SQL Express 2012 on the same Windows machine (IAAS and Model Manager Server).
  • Make sure that DNS settings are correct and IAAS Server FQDN can be resolved.
  • If you don’t use Active Directory, YOU MUST specify domain name in Primary DNS suffix of System Properties to make computer full name like its FQDN. It’s very important that in Step 8, Current Server filled automatically with FQDN and not Stand-Alone Server name. Domain should be there. Or you will face with an error (in Logs) like:
    “Building Project “C:\Program Files (x86)\VMware\vCAC\Server\Model Manager Data\DeployRepository.xml” (VARegistrationFinalSteps target(s)) — FAILED.
    Build FAILED.
    Authentication issue
    ……
    this error occurs when either the username or password supplied to iis is invalid
    …..

vCloud Public Console Proxy IP Address

You may have noticed that vCloud Director uses 2 important IP addresses to provide public access to tenants/users. One is the well-known front-end VCD IP address which is access to web portal for managing the organization vDC (also known as HTTP access) and second one provides remote access to virtual console of VM which is in fact resided on ESXi server cluster (known as VRMC access), this latter one is sort of more back-end because it’s coming from ESXi server which never should be exposed to public! So, vCloud Director actually tunnels Remote Console communications between ESXi servers and users through a proxy agent on port 443. Apparently, the proxy service runs on vCloud Director machine. That’s why an extra IP is needed on vCloud Director. This IP address is also specified in initial setup but it can be changed later (of course everything can be changed!).

So, when you want to open up vCloud Director for public users, you should pay enough attention to VRMC IP address and port. If you  have to do NAT through your firewall you should specify a different IP for VRMC and introduce the public IP/URL to vCloud Director in administration web panel. See the picture below:

Also, port 443 should be opened for this public IP on the firewall.

If you need more information about publicizing the whole vCloud Director, I found this excellent blog post about this topic, although it’s very useful for a general architecture of vCD deployment:

IP LAYER MONITORING IN VMWARE VSPHERE – 2

2 posts earlier, I talked about NetFlow in VMware 5.x and how to enable it in vSphere dvSwitch. I have also shown how you can send IP traffic flow information to a NetFlow collector. Nowadays, there are lots of commercial NetFlow collectors available; however, in this post I will introduce a simple, open-source NetFlow collector which you can use in your VMware environment to analyze IP traffic. This pretty piece of software is: ‘nfdump

As it’s shown, Nfdump has 2 major elements: ‘nfcapd‘ which is a daemon to gather and store relevant packets and ‘nfdump‘ which collects packets from all the daemons and interprets them. Apparently, nfcapd and nfdump could run on different machines and there could be multiple daemons but in case of VMware vSphere, it depends solely on the number of dvSwitches. If there is only one distributed switch, all the IP traffic flow information from all portgroups in that dvSwitch will be forwarded to one nfcapd. For test purposes, I also deployed both nfdump and nfcapd on a single linux machine but in cases that traffic is high, it maybe a good idea to deploy them on two different machines. Of course nfdump should have access to the storage in that case.

After installation, first you need to run daemon and specify a port and directory to store ip traffic information. Apparently, nfcapd will store information in binary. The command is simple, something like this:

  • nfcapd -w -D -l /var/netflow/dvswitch -p 23456

Then, daemon will run and listen to the specified port: 23456. If you have configured dvSwitch correctly (by specifying ip address of linux machine and 23456 as port) and activated monitoring on some portgroups in vCenter, this daemon will generate a couple of files in that directory.
Now, whenever you want to view the captured ip traffic flows, you should run nfdump. Since there are lots of files in that directory, you can interpret the whole directory using -R option with this command:

  • nfdump -R /var/netflow/dvswitch/

Filtering in nfdump is also possible, pretty much the same as tcpdump and you can view traffics of interest. You can find more information on nfdump website.

To view NetFlow captured traffic visually, you can mix nfsen with nfdump. It uses information that is dumped by daemon and utilizing rrdtool, it visualizes traffic flow. Installation is not difficult and you can see more information on their website. I’m really satisfied by this beautiful combination of nfdump and nfsen and if you intend to use NetFlow for monitoring, I recommend trying them. Good Luck!

DV portgroup is not found in the inventory after creation

Just for documentation that may help somebody else. The other day, I’ve got the following error when I intended to create an Organization VDC network in vCloud Director:

“Cannot deploy organization VDC network  (4a0c24d9-9f10-442b-8cb0-0fa9e8ccf0c8)
– com.vmware.ssdc.util.LMException: DV portgroup dvs.VCDVSNet1-a82df557-76db-4e37-9de3-53f4167db22c is not found in the inventory after creation
– DV portgroup dvs.VCDVSNet1-a82df557-76db-4e37-9de3-53f4167db22c is not found in the inventory after creation”

Normally vShield manager is the first thing I would suspect, but in this case it wasn’t the cause. I looked into many things and everything seemed normal. By the way, vCD Director is a software solution and unexpected things may happen. I thought restarting vCloud Director service is a good idea and yes, it worked! Therefore, the solution for me was running this command in vCloud Director server:

service vmware-vcd restart

Of course it may not fix the issue in your case. If it doesn’t help, look into VCD Cells page in cloud administrator console to see if there is an error message that maybe useful. Also, reconnecting vCenter is recommended.

p.s – VMware technical support told me to look into vCenter to monitor its CPU and memory usage. That’s a good point to be considered if everything else is okay and it’s not a configuration issue I believe. Also, database server may be slow not being able to return the result of a query in an acceptable time.

Extra Large Edge Gateway in vCloud Director 5.5

Sometimes you feel like implementing a powerful edge gateway in your VMware vCloud environment. Let’s say you have heavy load and you plan to use load balancer capability of edge gateway in VMware vCloud Director. Unfortunately hardware configuration of vShield edge gateways are not customizable through vCloud Director and changing hardware configuration through vCenter is not possible. Also, hardware templates for use as edge gateways are limited in terms of processing power and memory. There are 3 pre-defined hardware configurations in vCloud Director 5.5: Compact, Full and Full-4. Full-4 type is a new one in vCloud 5.5 and as I know Full gateways in vCloud 5 are upgraded to Full-4 automatically when you upgrade the infrastructure to version 5.5. In brief, hardware configurations for vShield edge gateways are as follows:

  • Compact: 1 * vCPU and 256 MB of Memory – 64000 concurrent sessions
  • Full: 2 * vCPU and 1024 MB of Memory – 1,000,000 concurrent sessions
  • Full-4 (new in vCloud 5.5): 4 * vCPU and 1024 MB of Memory

I didn’t find updated detailed information for vCloud 5.5 but you can see more details about edge gateway specifications and performance parameters in vCloud Director 5.1 at this useful link.
As you see, hardware power is limited especially in regards to memory. So, in case you need a memory intensive edge gateway (Load balancer is a good example) you need to upgrade the hardware. Although there is no direct method to this through vCloud Director admin panel, the fact is that vShield Manager has this capability to implement x-large gateways. x-large edge gateway in VMware Networking and Security 5.5 has 4 * vCPU and 8GB of Memory that is quite considerable.

As VMware recommended, if you need to upgrade hardware configuration of an edge gateway in vCloud Director, you can use vShield portal to do so. As it’s shown in the following picture, login to vShield Manager admin panel, choose your Datacenter, on ‘Network Virtualization’ tab select ‘Edges’, click on the edge gateway you intend to upgrade and finally from Actions menu choose: ‘Convert to X-Large’. That’s all.

Just keep in mind that in the picture above login to vShield Manager is done via vCenter. So, the ‘Network Virtualization’ tab shown in the figure is within vCenter; however it’s a bit difficult to get into vShield Manager through vCenter and I faced some weird errors about Acrobot Adobe! As a result, I recommend to use vShield Manager directly to avoid such issues.

Sticky sessions in vShield Edge Gateway Load Balancer

One of the features of edge gateways in VMware vCloud Director is the capability of implementing load balancer for HTTP, HTTPS and TCP-based applications in a virtual data center. For web applications (in specific HTTP), session management is an important matter. If web developers don’t implement session management in application level (using database, … to store sessions) and rely on Cookies, load balancer could be an issue. In these cases, network administrators are asked to configure load balancer with sticky session. Simply it means that if a client is forwarded to a web server for the first time (especially login page), it should stick to that specific server in later web requests. If it doesn’t happen, user may be forced to login again that would be frustrating!

By the way, when it comes to configuring vShield Edge Gateway to do load balancing, there is no obvious option to choose Sticky Session but it’s possible to do this by specifying proper value for Cookie name in the Virtual Server. As it’s shown in the picture, the procedure is as follows. I assume that you already know how to implement Load Balancer by creating Pool Servers and Virtual Server. See this link fore more information on how to create Load Balancer.

 

lb_vcns

  1. Right Click on the Edge Gateway and choose ‘Configure Services’
  2. Select ‘Load Balancer’ tab
  3. Go to ‘Virtual Servers’ section
  4. Edit selected Virtual Server
  5. Choose ‘Cookie’ as Persistence Method instead of default ‘None’
  6. Type proper value as Cookie Name; i.e, ‘ASP.NET_SessionId’ for .NET application, ‘PHPSESSID’ for PHP, … (ask your developer)

Migrate a VM by changing Datastore in vCloud Director

Migrating and relocating VMs is a great feature in Virtualized environments. You can do migration and perform your maintenance without disrupting any service. Migration in VMware is easy by utilizing vMotion in vCenter. You right click on VM, then choose Migrate and follow the instructions. But how we can relocate (change datastore or storage vMotion) a VM in vCloud Director?
Actually I was expecting the procedure in vCloud Director to be similar to vCenter, but when I right-clicked on the VM, I couldn’t find a ‘Migrate’ option or something like that. And apparently it’s not a good practice to migrate a VM which is controlled by vCloud Director through vCenter. But fortunately storage vMotion is possible in vCloud Director if you have separate Storage Profiles. To do this kind of relocation, right-click on the VM, then choose ‘Properties’; in ‘General’ tab you will find a pulldown menu for changing Storage Profile. Simply change Storage Profile to the desired one and bingo! Storage will be changed. You can even see the progress of relocation in vCenter.

Some tips:

  • As mentioned, changing datastore is possible if proper storage profiles are defined in the environment. That being said, it’s not possible to relocate to an individual, specific datastore. Actually, storage profile is another abstract layer over storage infrastructure that is being used by vCloud Director. To be honest, I didn’t have deeply realized what’s the main purpose of introducing storage profile and storage capability yet and why it doesn’t use datastore cluster instead. By the way, keep in mind that you need to create separate storage profiles if you have separate storages and you want more flexibility in vCloud. To get more information, look at this link: Using Storage Profiles with vCloud Director.
  • I couldn’t find much resources on the effects of migrating a VM from one host to another in vCloud Director environment; it is possible to perform this in vCenter and I did it in some cases with no issues. I suppose vCloud Director is working on higher level and will be notified of the changes.

NAT and PAT in vCloud Director 5.1

In VMware vCloud Director 5.1, NAT (Network Address Translation) and PAT (Port Address Translation) can be implemented using Edge Gateway of a vDC. Edge Gateway is created by Networking and Security component if you want a routed network in your Virtual Data Center.

Both NAT and PAT rules can be added/configured in Edge Network Services under NAT tab. There you can define Source NAT/PAT (SNAT) or Destination NAT/PAT (DNAT) rules. Apparently, SNAT provides connectivity to external network for your internal network users/machines and DNAT provides access to your internal network (the whole network or a specific machine or a specified port) from an external network.

NAT

The interesting point is that as it shown in the figure, in both cases, either SNAT or DNAT you have to choose your external network (‘Internet’ in this example) as the ‘Applied on’ network.

The other important thing is that you need to have a Firewall rule for NAT/PAT rules. For example if you are PAT’ing port 80 of an external IP to port 80 of an internal IP (DNAT), there must be a rule in Firewall that allows access to port 80 of external IP. In fact, in this case it is firewall that acts first; after firewall allows the connection, translation (DNAT) would be done.