Setting Up Logwitch

Download the archive from sourceforge and decompress it somewhere such as /usr/local/logwitch. Everything should be root:root owned and logwitch.lua and also probably logwitch.sh must be executable. The logwitch.sh file is an example cron script to call logwitch.lua so it makes sense maybe to move it to /usr/local/bin or /etc/cron.daily

You will need to have a version of the lua language installed and depending on where that is on the file system, you may need to edit the first line of logwitch.lua to suit. You will certainly want to edit the logwitch.sh file to meet your requirements. What to do there is documented within the file, but as an examplle you could add a shell command to check RAID status. Apart from calling logwitch.lua with the -cron switch to monitor your log4j and unix/linux logs, logwitch.sh is used to add other information to a daily report of your server's performance. When logwitch is used in this way a logwitch was here tag is written to the end of the first log in each series, so that current information is never repeated unless you use the -grep switch. Any .gz logs are decompressed and recompressed after they have been checked.

The main part of the set up is done by adding lua modules to the same directory as logwitch.lua. There is no need for them to be specified in any config file as logwitch will look for them and work with what it finds. There are some examples in the modules directory; these files, when moved to the program's home directory containing logwitch.lua, are effectively the configuration. You do not need to be a lua expert; configuration is not at all difficult and it can be best understood by looking at a simple example and considering how it could be used as a template. The comments in the auth.lua module should be carefully read, but let's consider what's left in this configuration to check the auth logs if we remove these comments - not so much!

local auth = {}
auth.client = ''
auth.dir = '/var/log/'
auth.log = {'auth.log','auth.log.1','auth.log.2.gz'}
auth.alert = {'!closed','FAILED','Failed','su:','pam_unix(sshd:session):','su\\['}
auth.match = 1
function auth.present(line)
   return string.sub(line, 12)
end
return(auth)

There is little to add that is not already explained in the removed comments. The first line must match the filename minus the .lua exstention; all the other variable and function names in the module must follow suit. In this example the auth.client variable is not used, but the statement must be always be present. In fact this configuration is probably not one you would want to use as is, but the fact that I only allow ssh access from a minimal number of addresses makes this a simple first example.

The auth.dir variable simply tells where the programs logs are held on the file system; it must end with the path separator slash. The auth.log array is a list of the logs you wish to have checked, starting most recent first. These logs should already exist; logwitch does not attempt to catch errors here. If you see an error message attempt to index a nil value (local 'file'), that will be caused by a mistake here.

Logwitch strives to present you with no more than you need to see; it's up to you to choose what lines you consider important from the log files. You specify a word, or a list of words that "interesting" lines will contain. This is not pattern matching, rather simply matching a case sensitive string of characters. Note that if you specify ailed, both failed and Failed (and also nailed!) will be caught. There can be as many individual alerts as you consider necessary to catch (or ignore) the log lines you want to see. In the example, the first alert commences with !, an instruction to ignore rather than select any line containing this alert, whether or not positive alerts are also present. As with positive alerts, there can be as many negative alerts as you consider necessary. Often there will be none. Any negative alerts in the auth.alert array must precede the positve alerts in the comma separated, quoted list.

In most cases a single positive alert will indicate that the line containing it is of interest and in this example auth.match is set to 1. The setting is ignored if a line contains a negative alert. There may be occasions when you wish to select a line only if it contains more than a single positive alert and that would be specified here.

The next entry is a function, in this case only slightly more complicated than the essential return(line) statement. In this auth.present example the first line of the function strips out the date at the start of every auth log line. A date is essentila in a log file, but unnecessary in a daily report or a look at what's been going on since the last of those.

The final return(auth) line is also essential and there is no case for altering it.

All the supplied modules have informative comments, but please remember they are designed to work on my system (currently Debian 13) and to suit my inclinations. In every case they are likely to offer a good base, but what a particular application does on my system, in terms of it's logs, may not match yours. Indeed, an upgrade from one version to the next may require some recoding, particularly of the module.present function.

Advanced Use

The intent behind logwitch is that you should see at a glance what is going on by having all extraneous information removed. Advanced use of logwitch furthers this and implies little more than a minimal lua scripting ability and the knowledge of a few additional facilities that logwitch.lua makes available to the modules. The bind module illustrates the most important one - setting the module.client variable and aggregating mode and also accessing logwitch's flagged variable.

If you look at the bind section of both the example report and the later summary of subsequent activity, you see lines such as query-errors:141.98.83.48 *** 88 and query-errors: 134.122.121.84 *** 19. This represents a common scenario - a series of rapid attacks from a single source before any other activity intervenes, whether that be an incident from a different source or the automatic blocking of the offending ip address. With even 19 incidents to be listed one after the other as there would be without aggregation, the report becomes less immediately informative. See the comments in the bind.lua files that explain how aggregating is set up - the client ip is assigned to the bind.client variable within the bind.present function.

function bind.present(line)
   local ip
   local pos 
   local time = string.sub(line,12,20)
   line = string.sub(line, 25)
   if(flagged == 'rate%-limit:') then
   		pos = string.find(line, 'to')
   		ip = string.sub(line, pos + 3)
   		pos = string.find(ip, ' ')
   		ip = string.sub(ip, 1,pos)
   else
   		pos = string.find(line, '@')
   		line = string.sub(line, pos + 15)
   		pos = string.find(line, '#')
   		ip = string.sub(line, 0, pos -1)
   end
   bind.client = ip
   pos = string.find(flagged, '%-')
   if pos ~= nil then 
   		flagged = string.sub(flagged, 1, pos -2)..string.sub(flagged, pos)
   	end
   line = time .. " " .. flagged ..  " " .. ip
   return(line)
end

The function first catches the time of the event and stores it a the time variable. Next up is to separate out the client ip address from the log line; where that will be depends on the cause of the alert, hence the if / else fork in code execution. The flagged variable is acessible by all modules that you wish to make use it from logwitch.lua itself; it captures the alert word that triggered the selection of the line. Once we have the ip it is assigned to bind.client and that automatically triggers aggregating when an attack is repeated. The flagged variable is itself interesting as you may consider a line triggered by security to be more important than one triggered by query-error. In this example we simply tag the alert word (flagged) and ip address to the time and so produce an informative line to present in reports

You may wonder what happens if you fail to consider all possibilities in a module.present function - perhaps some corner case event that results in a log line your code cannot parse correctly. In such a case the line is simply presented as is from the log file, so you are not going to miss anything that you really need to see.

Instead of a list of alert words in the modules, there is the option of a single super negaive alert. This results in every line being accepted for display if it does not contain the alert word, which is indicated by setting module.alert = {'*!word'}. There must be no other alerts and setting the module.match to anything other than 1 makes no sense and will be ignored.

License

logwitch
Copyright (C) 2016 to 2026 David Matthews

    This program is free software; you can redistribute it and/or modify
    it under the terms of the GNU General Public License version 3 as published by
    the Free Software Foundation.

    This program is distributed in the hope that it will be useful,
    but WITHOUT ANY WARRANTY; without even the implied warranty of
    MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
    GNU General Public License for more details.

    You should have received a copy of the GNU General Public License
    along with this program; if not, write to the Free Software
    Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA  02111-1307  USA.

mail_AT_dmatthews_DOT_org
David Matthews 372 Danie Theron St, Pretoria North, 0182, South Africa.