If you want your system to be secured and hardened, you have to know what’s going on in that system, you can do that using logs. With logs, you can diagnose problems and determine the health of your system and applications.

The Logging Service

Most Linux distros come with Rsyslog (successor version of syslog) preinstalled as well as the logging component of systemd which is systemd–journald (journald).

Regardless of the software, they are the same; the difference is in some features.

Rsyslog Daemon

The rsyslog utility is used to create and store readable event notification messages so system administrators can manage their systems.

you can check if the service is running or not:

$ systemctl status rsyslog

Rsyslog can send its output to various destinations like:

Text files as /var/log/* files

SQL databases

Different hosts

Supported databases are MySQL, PostgreSQL, Oracle, SQLite, Microsoft SQL, Sybase, Firebird, and mSQL.

Each log entry contains the date, time, hostname, process name, PID, and the log message.

Configuring Rsyslog

The Rsyslog daemon configuration file is  /etc/rsyslog.conf.

Some Linux distros like Ubuntu or Linux Mint, you will find these lines in /etc/rsyslog.d/default.conf

Each line contains a selector and an action.

kern.*        /var/log/messages

The kernel is the facility and the priority is the asterisk (*), so all messages from kernel will be logged to /var/log/messages.

We need to know what kind of facilities we have on our system and what are the other priorities.

Rsyslog Port

Rsyslog uses TCP on port 514. And its TCP because of the @@ sign.

You can ensure your rsyslog port using the netstat command:

$ netstat -tnlp | grep rsyslog

Rsyslog Facilities

The source of the Rsyslog message is called the facility. There are some Linux functions, daemons and other applications have facilities attached to them.

The following list is the rsyslog facilities in Linux:

auth                                       Security related messages.

auth-priv                             Private authentication messages.

cron                                       Message generated by cron subsystem.

daemon                                Services messages.

kern                                       Kernel messages.

mail                                       Mail messages.

syslog                                    Syslog messages.

You have two additional special facilities the asterisk (*) which means all facilities and (none) which means no facility at all.

Look at the following examples to understand these special facilities.

*.emerg    /var/log/emerg

This line says send all messages of the emergency priority to /var/log/emerg file.

mail.none /var/log/maillog

This will tell rsyslog not to log any mail messages to the file /var/log/maillog.

Rsyslog Levels (Priorities)

Linux syslog server Priorities are the scale of importance. We have notice, debug, info, warning, err, crit, alert, and emerg.

You can use the asterisk for all priorities (*) and none for no one.

Also, you can use the equal sign (=) and exclamation mark(!).

For example, kern.=crit  means critical kernel messages is selected.

kern.!crit means select all kernel messages except the critical priority.

Rsyslog actions

Every event notification received by the Linux syslog server goes to the specified action, we saw the logs go to files in the previous examples, but there are more actions can be done.

Rsyslog can do the following actions:

Logging to a file.

Logging to a device file.

Sending to the user screen.

Send to named pipes.

Send to the remote host.

Look at the following examples to understand the difference between them:

auth.err                /var/log/messages

daemon.*           /dev/lpr2

auth-priv              root,likegeeks

kern.crit               |/var/log/mypipe

mail                       @@hackonology.local

auth.*                   >dbhost,dbname,dbuser,dbpassword;dbtemplate

In the 1st line sends all auth messages of err priority logged to /var/log/messages file.

The 2nd line sends all daemon messages of all priorities are sent to a local printer lpr2.

The 3rd line sends private authentication messages to both users root and hackonology if they are logged in.

The 4th line sends all kernel messages of crit priority to the named pipe /var/log/mypipe, you can create your named pipe using the mkfifo command.

The 5th line sends all mail messages to the host hackonology.local on TCP port 514. This requires Rsyslog daemon to start with -r option; otherwise, the port will not open.

You should use double @ sign @@hackonology.local, that means logs will be transmitted using TCP protocol.

The sixth line sends all auth messages to a database table with the specified parameters, and the dbtemplate field is optional.

Keep in mind that opening the port 514 on your system is dangerous, the attacker can flood your system with messages.

If you want remote logging it is recommended to use syslog-NG which we will discuss later on this post.

You may notice a line like the following in /etc/rsyslog.conf

mail.*    -/var/log/maillog

This tells Rsyslog to cancel file synchronization after writing logs, but you may lose data if the writing process doesn’t complete successfully.

Combining Rsyslog Selectors

The facility and priority together make up the selector.

You can combine selectors in your rsyslog.conf file like this:

mail,kern.emerg           /var/log/log

I think it’s easy to understand this line.

All mail and all kern messages with an emerg or higher priority will be sent to /var/log/log file.

Rsyslog Filters

You can use selectors for filtering, but what if you want to filter the messages?

You can filter your messages with property-based filters like this:

:property, compare-operation, "value"

The following examples show how to filter messages that contain error:

:msg, contains, "error"

The compare operations are:
contains, isequal, startswith, regex, ereregex.

:msg, regex, "login .* failed"

This filter says catch any message that contains the words “login” and “failed” and any text between them.

Systemd-journald Service

Systemd-Journald is a component of systemd that runs as a service to collect log data just like Rsyslog.

Systemd-journald stores logging data in indexed journals, which makes it faster than other tools.

Many Linux distros come with systemd-journald integrated alongside with Rsyslog.

You can run them concurrently on the same system and even feed the data from one to the other.

The main tool for interacting with journaled files is journalctl.

Journalctl is used for querying and viewing the contents of logging data collected by systemd-journald service.

$ journalctl

You can use your keyboard to navigate through messages and press q to quit.

Some of the output lines are colored and others are bold. The red color for the priority of crit [2], alert [1], and emerg [0].

The red color and bold for the priority of warn [4], crit [2], alert [1], and emerg [0].

The lines of priority debug (7) and higher info [6] are displayed normally.

The best thing about journalctl command is that you can filter the messages with some flexible options.

You can display a specific number of lines like this:

$ journalctl -n 3

You can show real-time messages generated like this:

$ journalctl -f

Also, you can show messages since a number of days:

report this ad$ journalctl –since=”3 days ago”

Or you can show messages between days like this:

$ journalctl --since="5 days ago" --until="2 days ago"

Also, you can specify the date you want:

$ journalctl --since="2017-03-14"

Maybe you want at a specific time:

$ journalctl --since="2017-03-14 15:00" --until="2017-03-14 15:30"

And you can specify the priority you want:

$ journalctl --since="2017-03-14 15:00" --until="2017-03-14 15:30" -p warning

You can display messages associated with a specific user like this:

$  journalctl _UID=1001

You can view messages generated by a program like /sbin/init:

$ journalctl /sbin/init

Also, you can check the messages on your disk:

$ journalctl /dev/sda1

Syslog-ng

Another successor to syslog which is syslog-ng. This tool is better and more secure.

Moreover, syslog-ng allows for more advanced message filtering, manipulation, and interaction.

Some people prefer syslog-ng because it has cleaner syntax than Rsyslog.

The configuration file is /etc/syslog-ng/syslog-ng.conf  file.

This file contains the following blocks:

options{}                             Global options.
source{}                               Define the source of the messages.
destination{}                      Define the destination of the messages (where it will be    saved).
filter{}                                   Defines a filter to filter the messages.
log{}                                      Write the logs from the source to the destination specified from the above statements.

The idea in syslog-ng is to write the source{},destination{} and filter{} then you use them in the log{} statement. We will see how to write each one of them and how to write the final log statement.

Syslog-ng Source Statements

You can write source statements like this:

source my_udp { udp(ip(1.2.3.4) ;};

The line the source of messages which is IP address 1.2.3.4.

You can also specify a file as a source of logging:

source my_file { file(“/proc/kmsg”); };

Also, you can specify named pipes as a source:

source my_pipe { pipe(“/var/mypipe”); };

Syslog-ng Destination Statements

You can write destination statements like that:

destination my_dest { file("/var/log/mylogs" owner(hackonology) group(hackonology) perm(0644)); };
You specify the filename and its ownership.

The destination can be named pipe like this:

destination my_dest { pipe("/var/mypipe"); };

You can use the ready to use file expansion macros like this:

destination my_dest { file("/var/log/hosts/$HOST/$FACILITY$YEAR$MONTH$DAY"); };

The destination can be a logged in user like this:

destination my_dest { usertty("root"); };

Syslog-ng Filters Statements

You can define a filter by specifying the facilies and the priorties that you want to select like this:

filter my_filter { facility(kern); priority(notice .. crit) };

The above filter statement selects all messages from the kernel with priorities from notice to crit.

You can filter by host:

filter my_filter { host(likegeeks); };

Or by specific program:

filter my_filter { program("telnet.*") };

Or even use regular expressions to filter messages:

filter my_filter { match("YOUR REGEX"); };

And even negate the regular expressions:

filter my_filter { not match("YOUR REGEX"); };

Syslog-ng Log Statements

Now we have the source, destination, and filter statements, we should write the log statements that will do the actual logging.

The log statements don’t need to be named, unlike other statements, since they will not be used elsewhere.

log { source(my_src); destination(my_dest); };

This line sends the messages from my_src to my_dest. You are not limited to one source, you can combine multiple sources.

log { source(my_src); source(my_kern); filter(my_filter); destination(my_dest); };

Logging syslog-ng Messages in SQL Database

Syslog-ng supports many databases as a backend, we will use MySQL in the following example.

This example sends the log messages to a MySQL database running on the localhost.

The syslog-ng automatically creates the necessary tables and columns.

Then you can use this destination on any log statement you want.

Log Files Locations

The default log files locations that you might need to take a look at:

/var/log/messages                           Contains general system messages.
/var/log/kern.log                              Kernel logs.
/var/log/cron                                     The cron daemon messages.
/var/log/btmp                                   Contains failed login attempts.
/var/log/mail.log                               Contains logs for the mail server.
/var/log/wtmp                                   Contains all login and logout attempts.
/var/log/dmesg                                 Contains kernel messages.
/var/log/secure                                 Security related messages will be logged here.
/var/log/mariadb                              MariaDB log directory.
/var/log/mysql                                   MySQL log directory.
/var/log/httpd/                                  Apache web-server log directory.

for CPanel apache logs are at this location

/usr/local/apache/logs/            Apache server logs.
/usr/local/apache/domlogs/   Domain specific logs.
/var/log/exim/                                   Exim mail server logs.
/var/log/yum.log                               Yum package manager logs.
/var/log/boot.log                              Contains information about when the system boots.

Using Logrotate

It’s very important to keep your log file at a manageable size so you can control it.

If you need to store log messages for the long term, it is preferred to use a database as we saw.

You can use the logrotate tool for log rotation, so you keep your log files in manageable sizes.

logrotate configuration is in  /etc/logrotate.conf file.

This configuration rotates logs weekly and after 4 times of rotation, the log file is deleted, then create new log files.

You can modify the log rotation process the way you want. Also, you can include your log files.

The following options can be used in your logrotate.conf file:

daily                                     rotate daily.
weekly                                 rotate weekly.
monthly                              rotate monthly.
compress                            compress old rotated logs.
include                                To include more conf file.

You can check the shell script that runs daily for log rotation in /etc/cron.daily/logrotate.

Let’s Join our Hacking Team

We Are Indian We Are Great

Hope this article helpful for you. Thank You

Indian Cyber Army | Make IT Secure

Enjoy…Stay Happy…Stay Secure…

Hope this article helpful for you. Thank You

---

If You Appreciate What We Do Here On Hackonology, You Should Consider:

Hackonology is the fastest growing and most trusted community site where you can find lots of courses, articles about Technology/Hacking/Cracking. Millions of people visit Hackonology! to search or browse the thousands of published articles available FREELY to all.

Let's be a part of Hacker's Community! Join our Hacking Team

We Are Indian We Are Great

---