eJabberd Puppet Module for Debian

PuppetForge is a great repository to find and utilize required modules; however, sometimes you need to make some changes to satisfy your own requirements. In my case, I needed an ‘eJabberd’ module for Debian based machines. I found this useful module by Lee Boynton that worked very well in CentOS but apparently not in Debian/Ubuntu. Those who are familiar with eJabberd installation, know that it’s a bit tricky when it needs to use mySql as its storage and requires specific drivers, schema. I modified Lee’s module slightly and the proper mysql manifest is as follows. I have tested this in Debian Squeeze as well as Wheezy:

# Installs the native erlang mysql driver
class ejabberd::mysql(
    $lib_dir = $ejabberd::params::lib_dir
) inherits ejabberd::params {
    if !defined(Package['git']) {
        package { 'git':
            ensure => installed,
        }
    }
    if !defined(Package['erlang-rebar']) {
	case $::osfamily {
	    'redhat': {
	        package { 'erlang-rebar':
        	    ensure => installed,
        	}
	    }
            'debian': {
                package { 'erlang':
                    ensure => installed,
                }
                file { "/home/debs":
                        ensure => directory
                }

                file { "/home/debs/rebar_2.0.0-5_amd64.deb":
                    owner   => root,
                    group   => root,
                    mode    => 644,
                    ensure  => present,
                    source  => "puppet:///modules/ejabberd/rebar_2.0.0-5_amd64.deb"  
                }

                package { 'erlang-rebar':
                        provider => dpkg,
                        ensure => installed,
                        source => "/home/debs/rebar_2.0.0-5_amd64.deb"
                }

            }
        }
    }


    vcsrepo { '/usr/local/src/mysql':
        ensure      => latest,
        provider    => git,
        source      => 'https://github.com/processone/mysql.git',
        require     => Package['git'],

        # use first version which is compatible with ejabberd 2.1.x
        revision    => '967f3a0bb7'
    }

    exec { 'compile-mysql':
        command     => '/usr/bin/rebar compile',
        creates     => '/usr/local/src/mysql/ebin/mysql.beam',
        cwd         => '/usr/local/src/mysql',
        environment => 'HOME=/root',
        require     => [
            Package['erlang-rebar'],
            Vcsrepo['/usr/local/src/mysql'],
        ]
    }

    file { "${lib_dir}/ebin/mysql.beam":
        ensure  => present,
        source  => '/usr/local/src/mysql/ebin/mysql.beam',
        require => Exec['compile-mysql'],
    }
    file { "${lib_dir}/ebin/mysql_auth.beam":
        ensure  => present,
        source  => '/usr/local/src/mysql/ebin/mysql_auth.beam',
        require => Exec['compile-mysql'],
    }
    file { "${lib_dir}/ebin/mysql_conn.beam":
        ensure  => present,
        source  => '/usr/local/src/mysql/ebin/mysql_conn.beam',
        require => Exec['compile-mysql'],
    }
    file { "${lib_dir}/ebin/mysql_recv.beam":
        ensure  => present,
        source  => '/usr/local/src/mysql/ebin/mysql_recv.beam',
        require => Exec['compile-mysql'],
    }
}

HAProxy Load Balancing IIS with Sticky Session and SSL

HAProxy is a very good candidate for load balancing in a web cluster with high availability, even for Windows IIS servers! In its newer versions (1.5.x), HAProxy supports native SSL which makes it suitable for even enterprise level web applications with high traffic. It also supports sticky session which is useful when no session management is implemented. I know that the best option is to use centralized session management out of the box, but considering the fact that this central session manager will be point of failure (at least in IIS) and needs care, sticky session can be a good choice for some small to medium environments with short aged session applications.

Here, I will show how to configure HAProxy 1.5.x to support backend IIS servers with SSL (https) and sticky sessions.

– If you have IIS certificate, export it and use ‘openssl’ in Linux to convert it to appropriate format and put it in a protected directory.

– For SSL termination (HAProxy sends certificate to the users and takes over https protocol between user and load balancer), configurations is as follows:

  • frontend https-in
    bind *:443 ssl crt /etc/ssl/private/company.com.pem
    reqadd X-Forwarded-Proto:\ https
    default_backend application-backend

– To deploy sticky session, specify ’round robin’ as balancing policy and configure backend cluster part as follows. the key line is ‘cookie SERVERID insert indirect’:

  • backend application-backend
    balance roundrobin
    option httpclose
    option forwardfor
    cookie SERVERID insert indirect nocache
    server WEB-001 192.168.x.1:80 cookie A check
    server WEB-002192.168.x.2:80 cookie B check
    server WEB-003 192.168.x.3:80 cookie C check

To have more information about different policies and different session behaviours, read here.

Mixing 802.1Q and 802.1ad in Linux

When it comes to networking, Linux kernel is really superior over Windows. Some will ask why? Apart from performance point of view, there are some great features in Linux that can not be deployed in Windows easily. To give an example, let’s think about 2 important features: support for VLAN and trunking (802.1q) and NIC teaming or Link aggregation (802.1ad).

As far as I know, Windows kernel doesn’t support 802.1q and it all depends on NIC driver and for 802.1ad Windows support starts from Windows 2012 which means it’s too young! and who knows how it works! but both are prolonged features in Linux kernel.

And these features are really useful; for example when one single computer needs to be part of different VLAN’s it needs to be connected to a trunk port on the switch; therefore should understand VLAN tags and decapsulate packets. This single computer can even act as a router between different VLAN segments. Connecting to different VLANs means more traffic, so it’s not a bad idea to double (as an example) its bandwidth by aggregating (bonding) two NIC’s to improve performance. I’m providing 2 links to show how to implement 802.1q and 802.1ad in a single Linux machine with 2 or more NIC’s:

And to have an idea about combining these 2 features, see:

Key authentication with SSH Secure Shell

Non-commercial version of SSH Secure Shell (can be obtained here) from SSH Communications Security is a decent ssh client that I have used for many years in my experiments and academic works. It lacks PKI and PKCS functionality, but still safe for experiments! However; when it comes to public key authentication, it needs some tweaks to work. Here are the steps required to enable key authentication over a Linux host; given that Linux host settings allow public/private key authentication:

  1. Connect to the host using SSH Secure Shell (by password)
  2. In Secure Shell client, go to: Edit -> Settings -> User Authentication -> Keys and click on ‘Generate New’
    ssh1
  3. When generation is done, it will ask you to upload the public key to the host. Let it upload to ‘.ssh ‘ as destination folder.
    ssh2
  4. It assumes that the host has the appropriate SSH server for this client (the company has SSH server too) but since standard Linux servers use OpenSSH as SSH server, uploading the public key to the host is not enough and needs some modifications that follows.
  5. In Linux host, you will see that a public key (KeyAuthTest.pub in this case) is uploaded in ‘.ssh’ directory. For this to work, there are 2 ways:
    • Edit ‘KeyAuthTest.pub’ manually! and give it the right format. Remove these lines (or something like this) in the beginning:
      —- BEGIN SSH2 PUBLIC KEY —-
      Comment: “[3072-bit rsa, yyyy@xxxx, Thu Oct 04 2012 21:33:49]”
      And this at the end:
      —- END SSH2 PUBLIC KEY —-
      Also, you need to remove all the carriage returns (CR) in this file. Then add ‘ssh-rsa’ in the beginning of the file. The file would be something like:
      ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC+…
      Finally, in shell append this file to the ‘authorized_keys’ file :
      cat ~/.ssh/KeyAuthTest.pub >> ~/.ssh/authorized_keys
    • Second approach: convert the key to proper OpenSSH format automatically and append it to the file:
      ssh-keygen -i -f ~/.ssh/KeyAuthTest.pub  >>  ~/.ssh/authorized_keys

Now, you will be able to connect to the host, using this public key.

System Administration, Cloud and Coding!

As virtualization and cloud are getting more common, deploying more and more servers is really tempting for companies because it’s easier and needs less cost.

However it brings the subject of managing servers into attention. System administrators now should find solutions to decrease the time of deployment and applying changes in configurations and fixes. Under the influence of Cloud and Virtualization, business and product owners expect quick reaction from IT department; although cloud and hypervisor infrastructures provide easy and quick ways to deploy servers but still there are many tasks that require novel ideas to be automated and here is where sort of coding and logic meets system administration.

There are some tools designed for this purpose. Web giants like Google and Amazon have long used software that automatically configures the vast collection of machines driving their online services. But as Luke Kanies (CEO and Founder of  Puppet Labs) says: “Google does [things] differently, and in many cases, they do it better. Amazon is the same way. But what’s really frustrating is that no one else can use their software, I wanted to build a tool that would help other companies solve the same problem.”

Among them, Puppet is getting more attention and recent investments from big IT groups like VMware and Cisco make it attractive for companies who use virtualization and cloud. By the way, if you are a system administrator, sooner or later you will see the need for using either Puppet or any other management and automation software. Get ready!

VMware vCD Guest Customization for Debian

One of the great features in vCloud Director is client’s capability to customize general specifications of a VM. Specifications like Hostname and more importantly IP address(es). Customer can even have some scripts for more advanced customizations like joining to a domain, … All these depend on ‘Guest OS Customization’ feature that should be enabled on a VM. Not all the operating systems support ‘Guest OS Customization’. For a list of supported OS’s in vCD 5.x see these links:

Supported guest operating systems in vCloud Director 5.5
– Supported guest operating systems in vCloud Director 5.1

As you can see, there is no support for Debian Linux! What a pity! If you deploy a Debian and want to change its IP through VM Properties in vCD portal, it will give you an error:

“Guest customization is not supported by the selected OS. Please disable guest customization to proceed.”

Debian_custom

Debian is a great OS and many clients may get disappointed! But fortunately, there is a simple work-around for it: change the Operating System type to: Other Linux and Guest Customization will be fine! Of course, try to choose the closest kernel version, for example choose ‘Other 2.6.x Linux (64-bit)’ for a Debian wheezy with kernel 3.2.0-amd64.

Debian_custom2

By this change, modifying Hostname or assigning IP addresses, Gateway, DNS to Debian NICs would be possible like any other supported OS.

Troubleshooting WordPress Screen of Death

Last week I was asked to look into an on-premise WordPress website with a very low performance. I’m not expert in WordPress but I could say that comparing to the structure of the website and contents, it was too slow. Examining the system logs, I understood that memory usage reaches to its limit very soon as a result of huge consumption of Apache processes; in fact eventually Apache was returning white screen (500 error) to the visitors.  So, definitely something was wrong and simple troubleshooting guides mentioning to disable plugin and themes and even overwriting wordpress files didn’t help me! Clueless!

But when I was backing up the wordpress database to move it to a fresh Linux machine, something came to my attention: the dumped file was too big for their contents. Also, as soon as I imported the db into new mySql, the website became slow and eventually went down! So, it turned out that the issue is in the WordPress database. Using ‘phpmyadmin’ I found the largest table which was ‘wp_options’! I ran a simple query on it to see what’s in it? browsing through the results, soon I saw some irrelevant stuff. OMG! WordPress database was hacked and some HTML pages were inserted into ‘wp_options’ table! No wonder that website was slow! the ‘option_name’ of this table was filled with stuff like ‘/?tid=michael-kors-sac-CclA21.html’ and the value was a complete HTML file! Around 35000 of these rows were inserted and made a huge database.

So, I started cleaning database and getting more information about hacking WordPress. By the way, I put the clean database in a fresh installation of WordPress in a new machine and asked them to follow the best practices for securing WordPress website. I didn’t find similar situation on Internet, so I though it worths sharing, although I’m not a WordPress expert!