CGI Super Statistics Log Analyzer Documentation

SUPERSTATS:  V1.0

System Requirements

  • Perl 5
  • Apache style combined (referer & access) logs recommended (not tested on transfer logs)
  • crontab
  • Unix recommended

Preliminaries

  • Determine the path to PERL 5 on your web server host.  Note that some web hosting companies run both PERL 4 and PERL 5.  Make ABSOLUTELY sure you are not setting this up under PERL 4.  Ask your administrator if you are not sure.
  • Download the tarfile for this program and save it to your desktop.
  • Unpack the tar archive on your desktop using a program that unpacks UNIX TAR ARCHIVES. If you don't have such a program then download WINZIP FREE from SHAREWARE.COM. 
  • After you have unpacked the TAR archive you will have a collection of folders and files on your desktop.  Now you have to do some basic editing of each of these files (or at least some of them).  Use a text editor such as wordpad, notepad, BBEdit, simpletext, or teachtext to edit the files.  These are NOT WORD PROCESSOR DOCUMENTS they are just simple TEXT files so don't save them as word processor documents or save them with extentions such as .txt or they will NOT WORK.   Note that there may be a some files inside of folders which are "blank".   This is normal.

Preparing the CGI scripts

Define Path To PERL 5

The first step is to open up each and every file that has a .cgi extention and edit line number one of each script.  Each of the cgi scripts is written in perl 5. For your scripts to run they must know where perl 5 is installed on your web server. The path to perl 5 is defined to a cgi script in the first line of the file. In each of the cgi scripts the first line of code looks something like this:

#!/usr/bin/perl

If the path to perl 5 on your web server is different from /usr/bin/perl you must edit the first line of each cgi script to reflect the correct path. If the path to perl 5 is the same no changes are necessary. If you do not know the path to perl 5 ask the webmaster or system administrator at your server site.  

Configure the .cgi files

stats.cgi

Only one thing to configure here.  Open up stats.cgi in your favorite text editor and find the USER CONFIGURATION AREA at the top of the script.  There are 6 variables.  Most are pretty self explanatory I hope.

  • $domain = "yourdomainname.com";  (your domain name goes here)
  • $logfile = "/full/path/to/your/logs"; (full path to your REFERER logs)
  • $outputdir = "/full/path/to/where/you/want/the/stats/";  (full path to the stats directory for the results)
  • $referaldepth = 10;  (how many top refering sites do you want to count?   top 10, 25, ?)
  • $pagereferaldepth = 10;  (how many pages do you want to count in the TOP and LEAST hit pages)
  • $hurlingthreshold = 100;  (this is the value you consider for password hurler alerts)

The only 3 that might be a bit confusing (the last 3) are $referaldepth, $pagereferaldepth, and $hurlingthreshold. 

$referaldepth is simply how deep you want to count and list the TOP refering sites.  If you set this value to 10 (default) then your stats will only show the TOP 10 REFERING SITES.  If you set $referaldepth to 25 then your stats will show the TOP 25 REFERING SITES.

$pagereferaldepth is similar but it affects the TOP AND LEAST visited webpages within your site.  Thus setting this variable to 16 (for example) would show you the TOP 16 visited pages and also the LEAST POPULAR 16 pages.

$hurlingthreshold is a value I would leave alone.  100 is a good number for this variable.  If your website is being attacked by brute force password hurlers then their IP address will show up in your logfile.  You can then contact their ISP and get them in trouble, or just hack their site or blow up their PC with a good ol' syn-flood.

Upload Your Edited CGI

  • Upload stats.cgi into your cgi-bin somewhere
  • Make a directory inside of your webspace called stats (http://www.yourdomain.com/stats/)

Chmod the files

  • Chmod stats.cgi to 755.  Chmod the stats DIRECTORY either to 666 or preferably 777.  You will also want to chmod your REFERER LOGFILE to 777 so it can be reset everynight.

Setup your crontab

Setup a crontab to run nightly (say at midnight).  The crontab will analyze your logs, print out an HTML dated webpage and store it inside the stats directory.  It will also clear out your logfiles for the next day.  The crontab entry should be something like this

0 0 * * * /full/path/to/cgi-bin/stats.cgi

For more information on crontab visit the tutorial

NOTE

This program is designed to work with REFERER LOGS.  I have not tested it on ACCESS (transfer) logs.  Your isp can easily setup REFERER logs if you ask.  If they can't then you need to get a better ISP.   REFERER logs contain important information MISSING in transfer logs such as who is sending you traffic, who is trying to hack your site, etc.