![](https://i2.wp.com/hackonology.com/wp-content/uploads/2019/09/image0025.jpg?fit=624%2C308&ssl=1)
If you want your system to be secured and hardened, you have to know what’s going on in that system, you can do that using logs. With logs, you can diagnose problems and determine the health of your system and applications.
The Logging Service
Most Linux distros come with Rsyslog (successor version of syslog) preinstalled as well as the logging component of systemd which is systemd–journald (journald).
Regardless of the software, they are the same; the difference is in some features.
Rsyslog Daemon
The rsyslog utility is used to create and store readable event notification messages so system administrators can manage their systems.
you can check if the service is running or not:
$ systemctl status rsyslog
Rsyslog can send its output to various destinations like:
Text files as /var/log/* files
SQL databases
Different hosts
Supported databases are MySQL, PostgreSQL, Oracle, SQLite, Microsoft SQL, Sybase, Firebird, and mSQL.
Each log entry contains the date, time, hostname, process name, PID, and the log message.
Configuring Rsyslog
The Rsyslog daemon configuration file is /etc/rsyslog.conf.
Some Linux distros like Ubuntu or Linux Mint, you will find these lines in /etc/rsyslog.d/default.conf
Each line contains a selector and an action.
kern.* /var/log/messages
The kernel is the facility and the priority is the asterisk (*), so all messages from kernel will be logged to /var/log/messages.
We need to know what kind of facilities we have on our system and what are the other priorities.
Rsyslog Port
Rsyslog uses TCP on port 514. And its TCP because of the @@ sign.
You can ensure your rsyslog port using the netstat command:
$ netstat -tnlp | grep rsyslog
Rsyslog Facilities
The source of the Rsyslog message is called the facility. There are some Linux functions, daemons and other applications have facilities attached to them.
The following list is the rsyslog facilities in Linux:
auth Security related messages.
auth-priv Private authentication messages.
cron Message generated by cron subsystem.
daemon Services messages.
kern Kernel messages.
mail Mail messages.
syslog Syslog messages.
You have two additional special facilities the asterisk (*) which means all facilities and (none) which means no facility at all.
Look at the following examples to understand these special facilities.
*.emerg /var/log/emerg
This line says send all messages of the emergency priority to /var/log/emerg file.
mail.none /var/log/maillog
This will tell rsyslog not to log any mail messages to the file /var/log/maillog.
Rsyslog Levels (Priorities)
Linux syslog server Priorities are the scale of importance. We have notice, debug, info, warning, err, crit, alert, and emerg.
You can use the asterisk for all priorities (*) and none for no one.
Also, you can use the equal sign (=) and exclamation mark(!).
For example, kern.=crit means critical kernel messages is selected.
kern.!crit means select all kernel messages except the critical priority.
Rsyslog actions
Every event notification received by the Linux syslog server goes to the specified action, we saw the logs go to files in the previous examples, but there are more actions can be done.
Rsyslog can do the following actions:
Logging to a file.
Logging to a device file.
Sending to the user screen.
Send to named pipes.
Send to the remote host.
Look at the following examples to understand the difference between them:
auth.err /var/log/messages
daemon.* /dev/lpr2
auth-priv root,likegeeks
kern.crit |/var/log/mypipe
mail @@hackonology.local
auth.* >dbhost,dbname,dbuser,dbpassword;dbtemplate
In the 1st line sends all auth messages of err priority logged to /var/log/messages file.
The 2nd line sends all daemon messages of all priorities are sent to a local printer lpr2.
The 3rd line sends private authentication messages to both users root and hackonology if they are logged in.
The 4th line sends all kernel messages of crit priority to the named pipe /var/log/mypipe, you can create your named pipe using the mkfifo command.
The 5th line sends all mail messages to the host hackonology.local on TCP port 514. This requires Rsyslog daemon to start with -r option; otherwise, the port will not open.
You should use double @ sign @@hackonology.local, that means logs will be transmitted using TCP protocol.
The sixth line sends all auth messages to a database table with the specified parameters, and the dbtemplate field is optional.
Keep in mind that opening the port 514 on your system is dangerous, the attacker can flood your system with messages.
If you want remote logging it is recommended to use syslog-NG which we will discuss later on this post.
You may notice a line like the following in /etc/rsyslog.conf
mail.* -/var/log/maillog
This tells Rsyslog to cancel file synchronization after writing logs, but you may lose data if the writing process doesn’t complete successfully.
Combining Rsyslog Selectors
The facility and priority together make up the selector.
You can combine selectors in your rsyslog.conf file like this:
mail,kern.emerg /var/log/log
I think it’s easy to understand this line.
All mail and all kern messages with an emerg or higher priority will be sent to /var/log/log file.
Rsyslog Filters
You can use selectors for filtering, but what if you want to filter the messages?
You can filter your messages with property-based filters like this:
:property, compare-operation, "value"
The following examples show how to filter messages that contain error:
:msg, contains, "error"
The compare operations are:
contains, isequal, startswith, regex, ereregex.
:msg, regex, "login .* failed"
This filter says catch any message that contains the words “login” and “failed” and any text between them.
Systemd-journald Service
Systemd-Journald is a component of systemd that runs as a service to collect log data just like Rsyslog.
Systemd-journald stores logging data in indexed journals, which makes it faster than other tools.
Many Linux distros come with systemd-journald integrated alongside with Rsyslog.
You can run them concurrently on the same system and even feed the data from one to the other.
The main tool for interacting with journaled files is journalctl.
Journalctl is used for querying and viewing the contents of logging data collected by systemd-journald service.
$ journalctl
You can use your keyboard to navigate through messages and press q to quit.
Some of the output lines are colored and others are bold. The red color for the priority of crit [2], alert [1], and emerg [0].
The red color and bold for the priority of warn [4], crit [2], alert [1], and emerg [0].
The lines of priority debug (7) and higher info [6] are displayed normally.
The best thing about journalctl command is that you can filter the messages with some flexible options.
You can display a specific number of lines like this:
$ journalctl -n 3
You can show real-time messages generated like this:
$ journalctl -f
Also, you can show messages since a number of days:
report this ad$ journalctl –since=”3 days ago”
Or you can show messages between days like this:
$ journalctl --since="5 days ago" --until="2 days ago"
Also, you can specify the date you want:
$ journalctl --since="2017-03-14"
Maybe you want at a specific time:
$ journalctl --since="2017-03-14 15:00" --until="2017-03-14 15:30"
And you can specify the priority you want:
$ journalctl --since="2017-03-14 15:00" --until="2017-03-14 15:30" -p warning
You can display messages associated with a specific user like this:
$ journalctl _UID=1001
You can view messages generated by a program like /sbin/init:
$ journalctl /sbin/init
Also, you can check the messages on your disk:
$ journalctl /dev/sda1
Syslog-ng
Another successor to syslog which is syslog-ng. This tool is better and more secure.
Moreover, syslog-ng allows for more advanced message filtering, manipulation, and interaction.
Some people prefer syslog-ng because it has cleaner syntax than Rsyslog.
The configuration file is /etc/syslog-ng/syslog-ng.conf file.
This file contains the following blocks:
options{} Global options. source{} Define the source of the messages. destination{} Define the destination of the messages (where it will be saved). filter{} Defines a filter to filter the messages. log{} Write the logs from the source to the destination specified from the above statements.
The idea in syslog-ng is to write the source{},destination{} and filter{} then you use them in the log{} statement. We will see how to write each one of them and how to write the final log statement.
Syslog-ng Source Statements
You can write source statements like this:
source my_udp { udp(ip(1.2.3.4) ;};
The line the source of messages which is IP address 1.2.3.4.
You can also specify a file as a source of logging:
source my_file { file(“/proc/kmsg”); };
Also, you can specify named pipes as a source:
source my_pipe { pipe(“/var/mypipe”); };
Syslog-ng Destination Statements
You can write destination statements like that:
destination my_dest { file("/var/log/mylogs" owner(hackonology) group(hackonology) perm(0644)); }; You specify the filename and its ownership.
The destination can be named pipe like this:
destination my_dest { pipe("/var/mypipe"); };
You can use the ready to use file expansion macros like this:
destination my_dest { file("/var/log/hosts/$HOST/$FACILITY$YEAR$MONTH$DAY"); };
The destination can be a logged in user like this:
destination my_dest { usertty("root"); };
Syslog-ng Filters Statements
You can define a filter by specifying the facilies and the priorties that you want to select like this:
filter my_filter { facility(kern); priority(notice .. crit) };
The above filter statement selects all messages from the kernel with priorities from notice to crit.
You can filter by host:
filter my_filter { host(likegeeks); };
Or by specific program:
filter my_filter { program("telnet.*") };
Or even use regular expressions to filter messages:
filter my_filter { match("YOUR REGEX"); };
And even negate the regular expressions:
filter my_filter { not match("YOUR REGEX"); };
Syslog-ng Log Statements
Now we have the source, destination, and filter statements, we should write the log statements that will do the actual logging.
The log statements don’t need to be named, unlike other statements, since they will not be used elsewhere.
log { source(my_src); destination(my_dest); };
This line sends the messages from my_src to my_dest. You are not limited to one source, you can combine multiple sources.
log { source(my_src); source(my_kern); filter(my_filter); destination(my_dest); };
Logging syslog-ng Messages in SQL Database
Syslog-ng supports many databases as a backend, we will use MySQL in the following example.
destination my_dest { sql(type(mysql) host(“localhost”) username(“likegeeks”) password(“likegeeks”) database(“dblogs”) table(“logs”) columns(“datetime”, “host”, “program”, “pid”, “logmessage”) values(“$YEAR-$MONTH-$DAY $HOUR:$MIN:$SEC”, “${HOST}”, “${PROGRAM}”, “${PID}”, ” $MSG”) indexes(“datetime”, “host”, “program”, “pid”, “logmessage”)); }; |
This example sends the log messages to a MySQL database running on the localhost.
The syslog-ng automatically creates the necessary tables and columns.
Then you can use this destination on any log statement you want.
Log Files Locations
The default log files locations that you might need to take a look at:
/var/log/messages Contains general system messages. /var/log/kern.log Kernel logs. /var/log/cron The cron daemon messages. /var/log/btmp Contains failed login attempts. /var/log/mail.log Contains logs for the mail server. /var/log/wtmp Contains all login and logout attempts. /var/log/dmesg Contains kernel messages. /var/log/secure Security related messages will be logged here. /var/log/mariadb MariaDB log directory. /var/log/mysql MySQL log directory. /var/log/httpd/ Apache web-server log directory.
for CPanel apache logs are at this location
/usr/local/apache/logs/ Apache server logs. /usr/local/apache/domlogs/ Domain specific logs. /var/log/exim/ Exim mail server logs. /var/log/yum.log Yum package manager logs. /var/log/boot.log Contains information about when the system boots.
Using Logrotate
It’s very important to keep your log file at a manageable size so you can control it.
If you need to store log messages for the long term, it is preferred to use a database as we saw.
You can use the logrotate tool for log rotation, so you keep your log files in manageable sizes.
logrotate configuration is in /etc/logrotate.conf file.
weekly rotate 4 create |
This configuration rotates logs weekly and after 4 times of rotation, the log file is deleted, then create new log files.
You can modify the log rotation process the way you want. Also, you can include your log files.
The following options can be used in your logrotate.conf file:
daily rotate daily. weekly rotate weekly. monthly rotate monthly. compress compress old rotated logs. include To include more conf file.
You can check the shell script that runs daily for log rotation in /etc/cron.daily/logrotate.
Let’s Join our Hacking Team
We Are Indian We Are Great
Hope this article helpful for you. Thank You
Indian Cyber Army | Make IT Secure
Enjoy…Stay Happy…Stay Secure…
![](https://i0.wp.com/s3.amazonaws.com/docs.paykickstart.com/banner-images/1637981871_FO7130A2D7D85-2-320x100.jpg?resize=320%2C100&ssl=1)
Hope this article helpful for you. Thank You
If You Appreciate What We Do Here On Hackonology, You Should Consider:
Hackonology is the fastest growing and most trusted community site where you can find lots of courses, articles about Technology/Hacking/Cracking. Millions of people visit Hackonology! to search or browse the thousands of published articles available FREELY to all.
Let's be a part of Hacker's Community! Join our Hacking Team
We Are Indian We Are Great