Installing qmailanalog and Generating daily qmail statistics with (qmail-stats.py) Daily qmail statististics notification via email

Finding a decent software to generate daily qmail statistics is a really tough job this days. Before time I always used eitherqmailalizer or isoqlog.
Presently qmailalizer is completely abondoned piece of software and I cannot force it to work on 64 bit architecture.Isoqlog is another story, it’s supposed to work with qmailrocks, however my qmail installation is based on Russ Nelson’s Linux Qmail and for some strange reason it’s generating empty statistics. It could be that isoqlog is not generating statistics because the log files’s feed to be processed is not enough. Anyways still I cannot figure out the reason why I cannot make work Isoqlog with the Qmail.

I needed a way to however at least have an overview statistics of what is happening inside qmail. Of course qmailmrtg which is explained how to be installed in my previous post is providing with some overall information, though the information acquired through it is too general.
I’ve spend some enormous time searching for something that could inform me on various qmail statistics based on the qmail logs, before I could find and tweak the qmail-stats.py report script to become usable with qmailanalog.

In the meantime It was necessery for me to investigate into qmailanalog and install it on the system.
Initially I instlaled the qmailanalog from source, latest current source release can be obtained via D.J. Bernstein’s qmaialanlog download page
You won’t be able to compile the qmailanalog piece of code in CentOS until substitute in the source file: error.h the line:

extern int errno;

with#include

After the above change your source should succesfully compile.
Right after I compile it I realized there is a CentOS source package installer called:
qmailanalog-installer

So on CentOS to install qmailanalog all I had to do was:

CentOS-server:~# apt-get install qmailanalog-installer
CentOS-server:~# /usr/bin/build-qmailanalog

Now as I already have qmailanalog properly installed on CentOS I decided to test it with a script called qmail-logs.sh
You can download the qmail-logs.sh script from here

Here I quote what exactly is written in the qmail-logs.sh header in order to provide you with a general idea what the script does.

## Purpose:
## Wrapper for qmailanalog scripts. Will analyze qmail multilog
## files for deferrals, failures, overall statistics, or convert
## them to sendmail-style logs.

After executing the script I realized the script is not working properly because of some errors issued by scripts included within the qmailanalog package.
The faced problems and their solution with the qmailanalog: zsenders, zsuccesses, zfailures, zrecipients, zfailures and zrhosts scripts I’ve reported as a CentOS bugs The qmailanalog bug report I’ve submitted can be seen here

The whole problem with qmailanalog scripts on CentOS is that the “gnu more” binary arguments passed during the script operations are not correct and needs to be fixed.
Similar issues and their solution is explained in Qmailrocks Forums threads

After fixing the issues with qmailanalog I tried once again the qmail-logs.sh script, this time some of the functions provided by the script prooved working however some of them weren’t okay still.
Therefore I took some more time searching on the internet and I’ve found on a mailing list the qmail-stats.py script which worked like a charm with a minor modifications.

In order to have the qmail-stats.py working you need the tai64nfrac binary.You can download the latest current version of tai64nfrac on tai64nfrac’s website from here
Installation of tai64nfrac is pretty straight forward and comes to the following:

CentOS-server:~# wget http://archives.eyrie.org/software/system/tai64nfrac-1.4.tar.gz
CentOS-server:~# tar -zxvf tai64nfrac-1.4.tar.gz
CentOS-server:~# cd tai64nfrac-1.4
CentOS-server:~# make
CentOS-esrver:~# make install

After the install you should have the tai64nfrac in /usr/local/bin/tai64nfrac
Now let’s go back to the qmail-stats.py script. I’ve mirrored the qmail-stats.py script the qmail-stats.py script can be downloaded here

The script reports statistics on Qmail Logs following criterias:

Overall Email Server Statistics
Failure Statistics (Reasons for Failure)
Deferrals Statistics (Reasons for Deferrals)
Top Ten Senders Statistics
Top Ten Recipients Statistics

To make the script working all you have to edit in the script is the LOGFILE_PATH the rest is preconfigured by me already in the version of qmail-stats.py which is provided for download above.
If you choose to compile the qmailanalog from source you might also need to change the CMDS options, which includes the directory locations and commands from qmailanalog.

Now if you want to have the reports generated from qmail-stats.py, you have to setup the qmail-stats.py to be running via the cron daemon.

To do so open your root crontab and put in it:

# report daily qmail statistics05
01 * * * /usr/local/bin/qmail-stats.py | mail -s “Qmail Daily Statistics for $(date)” admin@domain.com

Now you should have the qmail-stats.py mailed to you every day at 01:05 early in the morning!

Cheers

YGN Ethical Hacker Group has shared an amazing demo of scalp usage.

Scalp! is a log analyzer for the Apache web server that aims to look for security problems. The main idea is to look through huge log files and extract the possible attacks that have been sent through HTTP/GET (By default, Apache does not log the HTTP/POST variable).

CLICK HERE TO SEE A LIVE DEMO

NOTE:-
SHARING THIS PRESENTATION DOES NOT MEAN THAT I AM ASSOCIATED WITH THE GROUP. I FOUND IT INTERESTING AND THUS SHARED IT WITH ALL MY FRIENDS!!!

Scalp! is a log analyzer for the Apache web server that aims to look for security problems. The main idea is to look through huge log files and extract the possible attacks that have been sent through HTTP/GET (By default, Apache does not log the HTTP/POST variable).

How it works

Scalp is basically using the regular expression from the PHP-IDS project and matches the lines from the Apache access log file. These regexp has been chosen because of their quality and the top activity of the team maintaining that project.

You will then need this file https://svn.php-ids.org/svn/trunk/lib/IDS/default_filter.xml in order to run Scalp.

Scalp started as a simple python script which is still maintained, but I plan to focus my effort on the binary version (written in C++) for efficiency when it comes to scalp huge log files.

Usage

Scalp has a couple of options that may be useful in order to save time when scalping a huge log file or in order to perform a full examination; the default options are almost okay for log files of hundreds of MB.

Current options:

• exhaustive: Won't stop at the first pattern matched, but will test all the patterns
• tough: Will decode a part of potential attacks (this is done to use better the regexp from PHP-IDS in order to decrease the false-negative rate)
• period: Specify a time-frame to look at, all the rest will be ignored
• sample: Does a random sampling of the log lines in order to look at a certain percentage, this is useful when the user doesn't want to do a full scan of all the log, but just ping it to see if there is some problem…
• attack: Specify what classes of vulnerabilities the tool will look at (eg, look only for XSS, SQL Injection, etc.)

Example of utilization:

./scalp-0.4.py -l /var/log/httpd_log -f ./default_filter.xml -o ./scalp-output --html

Help

rgaucher@plop:~/work/scalp/branches$ ./scalp-0.4.py --help
Scalp the apache log! by Romain Gaucher - http://rgaucher.info
usage:  ./scalp.py [--log|-l log_file] [--filters|-f filter_file] [--period time-frame] [OPTIONS] [--attack a1,a2,..,an]
                   [--sample|-s 4.2]
   --log       |-l:  the apache log file './access_log' by default
   --filters   |-f:  the filter file     './default_filter.xml' by default
   --exhaustive|-e:  will report all type of attacks detected and not stop
                     at the first found
   --tough     |-u:  try to decode the potential attack vectors (may increase
                     the examination time)
   --period    |-p:  the period must be specified in the same format as in
                     the Apache logs using * as wild-card
                     ex: 04/Apr/2008:15:45;*/Mai/2008
                     if not specified at the end, the max or min are taken
   --html      |-h:  generate an HTML output
   --xml       |-x:  generate an XML output
   --text      |-t:  generate a simple text output (default)
   --except    |-c:  generate a file that contains the non examined logs due to the
                     main regular expression; ill-formed Apache log etc.
   --attack    |-a:  specify the list of attacks to look for
                     list: xss, sqli, csrf, dos, dt, spam, id, ref, lfi
                     the list of attacks should not contains spaces and comma separated
                     ex: xss,sqli,lfi,ref
   --output    |-o:  specifying the output directory; by default, scalp will try to write
                     in the same directory as the log file
   --sample    |-s:  use a random sample of the lines, the number (float in [0,100]) is
                     the percentage, ex: --sample 0.1 for 1/1000

I Hope it works for you as well !!

I recently came across an error for libc6.so which probably was overwritten and resulted in errors. Immediately after overwriting libc.so.6, the server kernel panicked.

Even when i tried to boot using rescue mode the system returned me with this below error:

-bin/sh: /usr/bin/chroot : /lib/ld-linux.so.2: Bad ELF interpreter : No Such File or Directory

and upon rebooting i got

/bin/sh: relocation error: /lib/tls/libc.so.6: symbol _dl_starting_up, version GLIBC_PRIVATE

To Resolve:

1. Reboot the Server with the Linux installation disc #1 and boot into rescue mode (at the prompt, enter linux rescue.

2. Note when using the rescue disc, that your server’s directories are in /mnt/sysimage. The contents (uncorrupted) of the rescue disc we need are in /lib/.

3. You will need to copy the libc.so.6 symbolic link in /lib/ and retain the source file’s ownership, links, attributes, etc… We used “cp -parf /lib/libc.so.6 /mnt/sysimage/lib/tls/” . Change directories into the target directory, and verify that the libc.so.6 > libc-2.3.4.so link is present.

4. Reboot, the server should boot into the correct runlevel now.

Hotmail……. it just annoys me when i hear that name…….I was a frequent hotmail user and suddenly one day i found my Inbox which holds not less than 5000 mails is now a count zero. I raised a concern to the hotmail support team and request them to atleast migrate my mails ot a newly created live.in id. But it just proved to be a failed attempt………….

This will sound harsh, but you need to learn from this lesson.

Stop using HotMail.

I’m serious. Stop using HotMail, or for that matter any web-based email service for anything that you would consider to be “important”.

If your email and your contacts are truly important, if losing them would be a serious problem for you, as it apparently has, then invest in a ‘real’ POP3 email account from an email service provider or from your ISP. Then use a ‘real’ email program like Outlook, or Eudora, or Thunderbird or any of a number of other good email programs that run on your machine to manage your email.

I am sharing the response i received from the support team which is more annoying,

Hi anandshah1983,

I tried to retrieve your e-mails but I wasn’t able to recover any. I’m sorry to tell you that we can no longer recover your mails.

Check out this Solution Article on the causes of lost e-mails and how you can prevent them:

Emails are missing from your Windows Live Hotmail account

https://windowslivehelp.com/solution.aspx?solutionid=66b7e7be-6723-4d38-8c62-a58f03e4130e

When i digged in i found that i was a coolhotmail user which allows you to create customised email accounts and my email id was anand.shah@coolaquarius.com ; the domain coolaqaurius was expired and none of the hotmail managers even cared about of saving a promoted product for the sake of company’s name. I am able to login but would never get my emails back ever.

STOP USING HOTMAIL !!!!!