eJabberd Puppet Module for Debian

PuppetForge is a great repository to find and utilize required modules; however, sometimes you need to make some changes to satisfy your own requirements. In my case, I needed an ‘eJabberd’ module for Debian based machines. I found this useful module by Lee Boynton that worked very well in CentOS but apparently not in Debian/Ubuntu. Those who are familiar with eJabberd installation, know that it’s a bit tricky when it needs to use mySql as its storage and requires specific drivers, schema. I modified Lee’s module slightly and the proper mysql manifest is as follows. I have tested this in Debian Squeeze as well as Wheezy:

# Installs the native erlang mysql driver
class ejabberd::mysql(
    $lib_dir = $ejabberd::params::lib_dir
) inherits ejabberd::params {
    if !defined(Package['git']) {
        package { 'git':
            ensure => installed,
        }
    }
    if !defined(Package['erlang-rebar']) {
	case $::osfamily {
	    'redhat': {
	        package { 'erlang-rebar':
        	    ensure => installed,
        	}
	    }
            'debian': {
                package { 'erlang':
                    ensure => installed,
                }
                file { "/home/debs":
                        ensure => directory
                }

                file { "/home/debs/rebar_2.0.0-5_amd64.deb":
                    owner   => root,
                    group   => root,
                    mode    => 644,
                    ensure  => present,
                    source  => "puppet:///modules/ejabberd/rebar_2.0.0-5_amd64.deb"  
                }

                package { 'erlang-rebar':
                        provider => dpkg,
                        ensure => installed,
                        source => "/home/debs/rebar_2.0.0-5_amd64.deb"
                }

            }
        }
    }


    vcsrepo { '/usr/local/src/mysql':
        ensure      => latest,
        provider    => git,
        source      => 'https://github.com/processone/mysql.git',
        require     => Package['git'],

        # use first version which is compatible with ejabberd 2.1.x
        revision    => '967f3a0bb7'
    }

    exec { 'compile-mysql':
        command     => '/usr/bin/rebar compile',
        creates     => '/usr/local/src/mysql/ebin/mysql.beam',
        cwd         => '/usr/local/src/mysql',
        environment => 'HOME=/root',
        require     => [
            Package['erlang-rebar'],
            Vcsrepo['/usr/local/src/mysql'],
        ]
    }

    file { "${lib_dir}/ebin/mysql.beam":
        ensure  => present,
        source  => '/usr/local/src/mysql/ebin/mysql.beam',
        require => Exec['compile-mysql'],
    }
    file { "${lib_dir}/ebin/mysql_auth.beam":
        ensure  => present,
        source  => '/usr/local/src/mysql/ebin/mysql_auth.beam',
        require => Exec['compile-mysql'],
    }
    file { "${lib_dir}/ebin/mysql_conn.beam":
        ensure  => present,
        source  => '/usr/local/src/mysql/ebin/mysql_conn.beam',
        require => Exec['compile-mysql'],
    }
    file { "${lib_dir}/ebin/mysql_recv.beam":
        ensure  => present,
        source  => '/usr/local/src/mysql/ebin/mysql_recv.beam',
        require => Exec['compile-mysql'],
    }
}
Advertisement

HAProxy Load Balancing IIS with Sticky Session and SSL

HAProxy is a very good candidate for load balancing in a web cluster with high availability, even for Windows IIS servers! In its newer versions (1.5.x), HAProxy supports native SSL which makes it suitable for even enterprise level web applications with high traffic. It also supports sticky session which is useful when no session management is implemented. I know that the best option is to use centralized session management out of the box, but considering the fact that this central session manager will be point of failure (at least in IIS) and needs care, sticky session can be a good choice for some small to medium environments with short aged session applications.

Here, I will show how to configure HAProxy 1.5.x to support backend IIS servers with SSL (https) and sticky sessions.

– If you have IIS certificate, export it and use ‘openssl’ in Linux to convert it to appropriate format and put it in a protected directory.

– For SSL termination (HAProxy sends certificate to the users and takes over https protocol between user and load balancer), configurations is as follows:

  • frontend https-in
    bind *:443 ssl crt /etc/ssl/private/company.com.pem
    reqadd X-Forwarded-Proto:\ https
    default_backend application-backend

– To deploy sticky session, specify ’round robin’ as balancing policy and configure backend cluster part as follows. the key line is ‘cookie SERVERID insert indirect’:

  • backend application-backend
    balance roundrobin
    option httpclose
    option forwardfor
    cookie SERVERID insert indirect nocache
    server WEB-001 192.168.x.1:80 cookie A check
    server WEB-002192.168.x.2:80 cookie B check
    server WEB-003 192.168.x.3:80 cookie C check

To have more information about different policies and different session behaviours, read here.

Mixing 802.1Q and 802.1ad in Linux

When it comes to networking, Linux kernel is really superior over Windows. Some will ask why? Apart from performance point of view, there are some great features in Linux that can not be deployed in Windows easily. To give an example, let’s think about 2 important features: support for VLAN and trunking (802.1q) and NIC teaming or Link aggregation (802.1ad).

As far as I know, Windows kernel doesn’t support 802.1q and it all depends on NIC driver and for 802.1ad Windows support starts from Windows 2012 which means it’s too young! and who knows how it works! but both are prolonged features in Linux kernel.

And these features are really useful; for example when one single computer needs to be part of different VLAN’s it needs to be connected to a trunk port on the switch; therefore should understand VLAN tags and decapsulate packets. This single computer can even act as a router between different VLAN segments. Connecting to different VLANs means more traffic, so it’s not a bad idea to double (as an example) its bandwidth by aggregating (bonding) two NIC’s to improve performance. I’m providing 2 links to show how to implement 802.1q and 802.1ad in a single Linux machine with 2 or more NIC’s:

And to have an idea about combining these 2 features, see:

Key authentication with SSH Secure Shell

Non-commercial version of SSH Secure Shell (can be obtained here) from SSH Communications Security is a decent ssh client that I have used for many years in my experiments and academic works. It lacks PKI and PKCS functionality, but still safe for experiments! However; when it comes to public key authentication, it needs some tweaks to work. Here are the steps required to enable key authentication over a Linux host; given that Linux host settings allow public/private key authentication:

  1. Connect to the host using SSH Secure Shell (by password)
  2. In Secure Shell client, go to: Edit -> Settings -> User Authentication -> Keys and click on ‘Generate New’
    ssh1
  3. When generation is done, it will ask you to upload the public key to the host. Let it upload to ‘.ssh ‘ as destination folder.
    ssh2
  4. It assumes that the host has the appropriate SSH server for this client (the company has SSH server too) but since standard Linux servers use OpenSSH as SSH server, uploading the public key to the host is not enough and needs some modifications that follows.
  5. In Linux host, you will see that a public key (KeyAuthTest.pub in this case) is uploaded in ‘.ssh’ directory. For this to work, there are 2 ways:
    • Edit ‘KeyAuthTest.pub’ manually! and give it the right format. Remove these lines (or something like this) in the beginning:
      —- BEGIN SSH2 PUBLIC KEY —-
      Comment: “[3072-bit rsa, yyyy@xxxx, Thu Oct 04 2012 21:33:49]”
      And this at the end:
      —- END SSH2 PUBLIC KEY —-
      Also, you need to remove all the carriage returns (CR) in this file. Then add ‘ssh-rsa’ in the beginning of the file. The file would be something like:
      ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC+…
      Finally, in shell append this file to the ‘authorized_keys’ file :
      cat ~/.ssh/KeyAuthTest.pub >> ~/.ssh/authorized_keys
    • Second approach: convert the key to proper OpenSSH format automatically and append it to the file:
      ssh-keygen -i -f ~/.ssh/KeyAuthTest.pub  >>  ~/.ssh/authorized_keys

Now, you will be able to connect to the host, using this public key.

System Administration, Cloud and Coding!

As virtualization and cloud are getting more common, deploying more and more servers is really tempting for companies because it’s easier and needs less cost.

However it brings the subject of managing servers into attention. System administrators now should find solutions to decrease the time of deployment and applying changes in configurations and fixes. Under the influence of Cloud and Virtualization, business and product owners expect quick reaction from IT department; although cloud and hypervisor infrastructures provide easy and quick ways to deploy servers but still there are many tasks that require novel ideas to be automated and here is where sort of coding and logic meets system administration.

There are some tools designed for this purpose. Web giants like Google and Amazon have long used software that automatically configures the vast collection of machines driving their online services. But as Luke Kanies (CEO and Founder of  Puppet Labs) says: “Google does [things] differently, and in many cases, they do it better. Amazon is the same way. But what’s really frustrating is that no one else can use their software, I wanted to build a tool that would help other companies solve the same problem.”

Among them, Puppet is getting more attention and recent investments from big IT groups like VMware and Cisco make it attractive for companies who use virtualization and cloud. By the way, if you are a system administrator, sooner or later you will see the need for using either Puppet or any other management and automation software. Get ready!

VMware vCD Guest Customization for Debian

One of the great features in vCloud Director is client’s capability to customize general specifications of a VM. Specifications like Hostname and more importantly IP address(es). Customer can even have some scripts for more advanced customizations like joining to a domain, … All these depend on ‘Guest OS Customization’ feature that should be enabled on a VM. Not all the operating systems support ‘Guest OS Customization’. For a list of supported OS’s in vCD 5.x see these links:

Supported guest operating systems in vCloud Director 5.5
– Supported guest operating systems in vCloud Director 5.1

As you can see, there is no support for Debian Linux! What a pity! If you deploy a Debian and want to change its IP through VM Properties in vCD portal, it will give you an error:

“Guest customization is not supported by the selected OS. Please disable guest customization to proceed.”

Debian_custom

Debian is a great OS and many clients may get disappointed! But fortunately, there is a simple work-around for it: change the Operating System type to: Other Linux and Guest Customization will be fine! Of course, try to choose the closest kernel version, for example choose ‘Other 2.6.x Linux (64-bit)’ for a Debian wheezy with kernel 3.2.0-amd64.

Debian_custom2

By this change, modifying Hostname or assigning IP addresses, Gateway, DNS to Debian NICs would be possible like any other supported OS.

Troubleshooting WordPress Screen of Death

Last week I was asked to look into an on-premise WordPress website with a very low performance. I’m not expert in WordPress but I could say that comparing to the structure of the website and contents, it was too slow. Examining the system logs, I understood that memory usage reaches to its limit very soon as a result of huge consumption of Apache processes; in fact eventually Apache was returning white screen (500 error) to the visitors.  So, definitely something was wrong and simple troubleshooting guides mentioning to disable plugin and themes and even overwriting wordpress files didn’t help me! Clueless!

But when I was backing up the wordpress database to move it to a fresh Linux machine, something came to my attention: the dumped file was too big for their contents. Also, as soon as I imported the db into new mySql, the website became slow and eventually went down! So, it turned out that the issue is in the WordPress database. Using ‘phpmyadmin’ I found the largest table which was ‘wp_options’! I ran a simple query on it to see what’s in it? browsing through the results, soon I saw some irrelevant stuff. OMG! WordPress database was hacked and some HTML pages were inserted into ‘wp_options’ table! No wonder that website was slow! the ‘option_name’ of this table was filled with stuff like ‘/?tid=michael-kors-sac-CclA21.html’ and the value was a complete HTML file! Around 35000 of these rows were inserted and made a huge database.

So, I started cleaning database and getting more information about hacking WordPress. By the way, I put the clean database in a fresh installation of WordPress in a new machine and asked them to follow the best practices for securing WordPress website. I didn’t find similar situation on Internet, so I though it worths sharing, although I’m not a WordPress expert!

Enlarge Linux Partition to sizes Larger Than 2TB

If you have partitioned a disk using ‘fdisk’, most probably the partition table is using ‘MBR‘. Nowadays, one important disadvantage of MBR is the lack of supporting larger than 2 TB partitions. So, if you want to extend a partition while disk is using MBR, first the disk partition table needs to be converted to ‘GPT‘. To find difference between MBR and GPT see here as well.

Generally, to resize a partition, it needs to be deleted and re-created using new size or end sector. A good general approach to enlarge a partition while not losing data is explained in this link. You can follow this how-to with a slight but important difference: instead of using ‘fdisk’, ‘gptdisk’ or ‘gdisk’ should be used. ‘gdisk’ supports GPT and if you use the existing specifications (first sector) for new partition, there would be no data loss. When doing conversion, a warning will be triggered:

THIS OPERATION IS POTENTIALLY DESTRUCTIVE! Exit by typing 'q' if you don't want to convert your MBR partitions to GPT format!

But it’s okay! Don’t worry and continue. Then, it will ask for partition type, because the current partition type is based on MBR partition table. You can choose ‘ef00’ which is ‘EFI System’. Go ahead and create the new partition with new size, save the partition table and you are done. Then ‘resize2fs’ can be used to enlarge volume.

p.s – instead of enlarging partition, another choice (rather than MBR to GPT) is using LVM to create large logical volumes containing multiple physical volumes.

IP LAYER MONITORING IN VMWARE VSPHERE – 2

2 posts earlier, I talked about NetFlow in VMware 5.x and how to enable it in vSphere dvSwitch. I have also shown how you can send IP traffic flow information to a NetFlow collector. Nowadays, there are lots of commercial NetFlow collectors available; however, in this post I will introduce a simple, open-source NetFlow collector which you can use in your VMware environment to analyze IP traffic. This pretty piece of software is: ‘nfdump

As it’s shown, Nfdump has 2 major elements: ‘nfcapd‘ which is a daemon to gather and store relevant packets and ‘nfdump‘ which collects packets from all the daemons and interprets them. Apparently, nfcapd and nfdump could run on different machines and there could be multiple daemons but in case of VMware vSphere, it depends solely on the number of dvSwitches. If there is only one distributed switch, all the IP traffic flow information from all portgroups in that dvSwitch will be forwarded to one nfcapd. For test purposes, I also deployed both nfdump and nfcapd on a single linux machine but in cases that traffic is high, it maybe a good idea to deploy them on two different machines. Of course nfdump should have access to the storage in that case.

After installation, first you need to run daemon and specify a port and directory to store ip traffic information. Apparently, nfcapd will store information in binary. The command is simple, something like this:

  • nfcapd -w -D -l /var/netflow/dvswitch -p 23456

Then, daemon will run and listen to the specified port: 23456. If you have configured dvSwitch correctly (by specifying ip address of linux machine and 23456 as port) and activated monitoring on some portgroups in vCenter, this daemon will generate a couple of files in that directory.
Now, whenever you want to view the captured ip traffic flows, you should run nfdump. Since there are lots of files in that directory, you can interpret the whole directory using -R option with this command:

  • nfdump -R /var/netflow/dvswitch/

Filtering in nfdump is also possible, pretty much the same as tcpdump and you can view traffics of interest. You can find more information on nfdump website.

To view NetFlow captured traffic visually, you can mix nfsen with nfdump. It uses information that is dumped by daemon and utilizing rrdtool, it visualizes traffic flow. Installation is not difficult and you can see more information on their website. I’m really satisfied by this beautiful combination of nfdump and nfsen and if you intend to use NetFlow for monitoring, I recommend trying them. Good Luck!

tcpkill to Kill a TCP connection!

I recently found a very useful command in Linux named ‘tcpkill’. Actually, the other day I was trying to find a way to kill a tcp connection between my server and a client. Not that it was an attack or needed firewall rule, but simply I wanted it to be killed in order to let the upper layer application to re-establish connection. There was no utility in the upper layer application to do this and it handed TCP connection management over to OS (TCP KEEPALIVE in linux kernel). So, I started looking for the solution to kill the connection.
So, the solution was easy, just issue ‘tcpkill’ command with appropriate parameters. Parameters are compliant with ‘tcpdump’ filter formats. So, if you are familiar with ‘tcpdump’ you will find it easy. For more explanation, examples see the amazing cyberciti website.