In software, it’s essential to monitor logs of system activities. Today we’ll unravel what Linux logs are and how you can view them.
Logging is a must for today’s developers and why Retrace was designed with a built-in, centralized log management tool. Retrace collects all your log data for streamlined monitoring and insights and includes all the features you need in a log management tool – log tailing, full-text searching, structured logging, monitoring, alerting, and error-tracking functionalities for a seamless experience. And like all Stackify by Netreo solutions, Retrace works out-of-the-box with your stack, including Linux, Azure, MySQL, Windows, AWS, Oracle, MongoDB, and more.
Retrace is built to do the heavy lifting for you, but it’s still important to understand Linux logs, how to view them, and which logs are most important to your work. This mini-guide provides all the need-to-know essentials for you in an easily digestible format, so it won’t take up your entire lunch break to read – promise!
Linux logs provide a timeline of events for the Linux operating system, applications, and system and are a valuable troubleshooting tool when you encounter issues. When issues arise, analyzing log files is the first thing an administrator needs to do. To understand better, let’s see how Linux logs can be of help.
When something goes wrong on a Linux system, logs can help pinpoint the issue. By examining system logs, application logs, and service logs, it’s possible to identify errors, warnings, and other messages that indicate what went wrong.
System logs can help identify performance issues like memory leaks or disk I/O bottlenecks. Examining application logs can also help identify performance issues with specific applications.
Linux logs can be used to monitor system health and detect issues before they become critical. By monitoring system logs, administrators can identify trends and patterns that could indicate a problem is brewing.
Many organizations are required to maintain logs for compliance and auditing purposes. Linux logs can help organizations meet these requirements by providing a record of system activity.
Linux logs are an essential tool for monitoring and detecting security issues. System logs can be used to detect unauthorized access attempts, while application logs can help identify suspicious activity within specific applications. By monitoring logs, administrators can quickly identify and respond to security incidents.
For desktop app-specific issues, log files are written to different locations. Where a desktop application writes logs depends on the developer and whether or not the app allows for custom log configuration. Chrome, for example, writes crash reports to ‘~/.chrome/Crash Reports’.
Linux log files are stored in plain text and can be found in the /var/log directory and subdirectory. There are Linux logs for everything: system, kernel, package managers, boot processes, Xorg, Apache, MySQL, etc. In this article, we will focus specifically on Linux system logs.
First things first, you can change to this directory using the cd command. You also need to be the root user to view or access log files on Linux or Unix-like operating systems.
Syslog is a standard for message logging used by various Linux distributions. It collects logs from different system processes and applications, storing them in the /var/log directory. Key logs managed by syslog include /var/log/syslog, which captures general system activity, and /var/log/auth.log for authentication events. Syslog uses configuration files, typically located in /etc/rsyslog.conf or /etc/syslog.conf to determine where and how logs are stored, allowing log storage and filtering customization.
Systemd, a system and service manager for Linux, includes its logging system called journald. Unlike syslog, journald collects logs in a binary format, which can be viewed using the journalctl command. Logs are stored in /var/log/journal/ or in memory, depending on the configuration. Systemd logs provide detailed information about system services, including startup errors, status changes, and resource usage, making them invaluable for debugging and monitoring system health. Configuration settings for journald can be adjusted in /etc/systemd/journald.conf to manage log size, persistence, and other parameters.
Like any other OS, you can use certain commands to see Linux log files.
First, let’s switch to the directory where you can find Linux system logs with the command cd /var/log. Then, you can type ls to see the logs stored under this directory. One of the most important logs to view is the syslog, which logs everything but auth-related messages.
Issue the command vi syslog to view everything under the syslog. Zooming in on a specific issue will take a while since these files are long. You can use Shift+G to get to the end of the file, denoted by “END.”
You can also view logs via dmesg, which prints the kernel ring buffer and sends you to the end of the file. From there, you can use the command dmesg | less to scroll through the output. If you want to view log entries for the user facility, you need to issue the command dmesg –facility=user.
Lastly, you can use the tail command to view log files. It’s a handy tool that only shows the last part of the logs, where problems usually lie. For this, use the command tail /var/log/syslog or tail -f /var/log/syslog. tail will continue watching the log file and print out the next line written to the file. This allows you to follow what is written to syslog as it happens. Check out 20 ways to tail a log file post.
For a specific number of lines (for example, the last 5 lines), key in tail -f -n 5 /var/log/syslog, which prints the most recent 5 lines. Once a new line comes, the old one gets removed. To escape the tail command, press Ctrl+X.
To view logs collected by systemd’s journald, you can use the journalctl command. This tool allows you to browse through logs stored in binary format, providing a unified view of all system services and processes. Common commands include:
Systemd logs provide insights into service statuses, errors, warnings, and other system-related events. Users can find information on service start-ups, shutdowns, crashes, and resource usage. The logs are comprehensive, offering a detailed timeline of system operations, which is useful for diagnosing issues and monitoring system health.
Linux logs can be classified into different types based on the source or purpose of the log. Here are four common types of Linux logs:
These logs contain information about the system’s operation, such as boot messages, kernel messages, and hardware events. System logs are essential for troubleshooting system issues, and monitoring system performance.
These logs contain information about the behavior of an application, including errors, warnings, and other messages. Application logs are used to diagnose problems with applications and to analyze application performance.
These logs contain information about services running on the system, including network services and daemons. Service logs are used to monitor service activity, and optimize service performance.
These logs contain information about events on the system, such as user logins, system shutdowns, and security events. Event logs are used to audit system activity, track user activity, and investigate security incidents.
Monitoring every log is a monumental task and one reason we included centralized log management when we created Retrace. Log monitoring and management are essential for all developers, but the logs that you monitor will depend on your goals or other variables. There is some consensus about the most critical, must-monitor logs.
Stores Kernel logs and warning data. This log is valuable for troubleshooting custom kernels as well.
Messages relating to device drivers. The command dmesg can be used to view messages in this file.
Stores all Crond-related messages (cron jobs), such as when the cron daemon initiated a job, related failure messages, etc.
If you install packages using the yum command, this log stores all related information, which can be useful in determining whether a package and all components were correctly installed.
Contains information on all failed login attempts, which is useful for gaining insights on attempted security breaches, such as those attempting to hack login credentials as well as brute-force attacks.
General messages, as well as system-related information. Essentially, this log stores all activity data across the global system. Note that activity for Redhat-based systems, such as CentOS or Rhel, are stored in messages, while Ubuntu and other Debian-based systems are stored in Syslog.
Stores authentication logs, including both successful and failed logins and authentication methods. Again, the system type dictates where authentication logs are stored; Debian/Ubuntu information is stored in /var/log/auth.log, while Redhat/CentrOS is stored in /var/log/secure.
A repository of all information related to booting and any messages logged during startup.
Stores all logs related to mail servers, useful when you need information about postfix, smtpd, or any email-related services running on your server.
A directory containing error_log and access_log files of the Apache httpd daemon. The error_log contains all errors encountered by httpd. These errors include memory issues and other system-related errors. access_log contains a record of all requests received over HTTP.
MySQL log file that logs all debug, failure, and success messages. Contains information about the starting, stopping, and restarting of MySQL daemon mysqld. This is another instance where the system dictates the directory; RedHat, CentOS, Fedora, and other RedHat-based systems use /var/log/mysqld.log, while Debian/Ubuntu use the /var/log/mysql.log directory.
What does the output look like? Here’s an example of a Crontab edited by root log:
Sep 11 09:46:33 sys1 crontab[20601]: (root) BEGIN EDIT (root) Sep 11 09:46:39 sys1 crontab[20601]: (root) REPLACE (root) Sep 11 09:46:39 sys1 crontab[20601]: (root) END EDIT (root)
And here’s a case of Syslogd on Ubuntu (exiting and restarting):
Dec 19 07:35:21 localhost exiting on signal 15 Dec 19 16:49:31 localhost syslogd 1.4.1#17ubuntu3: restart.
And system shutdown from the Linux kernel:
Jun 1 22:20:05 secserv kernel: Kernel logging (proc) stopped. Jun 1 22:20:05 secserv kernel: Kernel log daemon terminating. Jun 1 22:20:06 secserv exiting on signal 15 Nov 27 08:05:57 galileo kernel: Kernel logging (proc) stopped. Nov 27 08:05:57 galileo kernel: Kernel log daemon terminating. Nov 27 08:05:57 galileo exiting on signal 15
A few other directories and their uses include:
Current login state, by user
Login/logout history
Tracks services running in the background that perform important tasks, but have no graphical output
Recordings of failed login attempts
Runs the pureftp process that listens for FTP connections. All connections, FTP logins, and authentication failures get logged here
Rarely used and often empty. When used, it contains messages from USENET
Information about the last logins for all users. This binary file can be read by the command lastlog.
Contains all FTP file transfer sessions, including information about the file name and user initiating FTP transfers
Understanding the usefulness and limitations of Linux logging is important for any professional working with them. To tie together all your system logging, use Retrace, which helps track system performance and centralizes all your logs. You can also try Netreo’s free code profiler, Prefix, to write better code faster, every time!
Stackify's APM tools are used by thousands of .NET, Java, PHP, Node.js, Python, & Ruby developers all over the world.
Explore Retrace's product features to learn more.
If you would like to be a guest contributor to the Stackify blog please reach out to [email protected]
