top is a very flexible tool and its manual page gives details on how to customize its display and adapt it to one's personal needs and habits.
The gnome-system-monitor and qps graphical tools are similar to top and they provide roughly the same features.
TIP Visual representations of activity
For more visual (and entertaining) representations of a computer's activity, one should investigate the lavaps, bubblemon and bubblefishymon packages. lavaps displays running processes as the wax bubbles in a lava-lamp. bubblemon is a desktop panel applet representing the amount of used memory and the processor usage as an aquarium with bubbles. bubblefishymon is rather similar, but it also adds fish representing network traffic (and even a duck).
14.3.2.2. History
Processor load, network traffic and free disk space are information that are constantly varying. Keeping a history of their evolution is often useful in determining exactly how the computer is used.
There are many dedicated tools for this task. Most can fetch data via SNMP (Simple Network Management Protocol) in order to centralize this information. An added benefit is that this allows fetching data from network elements that may not be general-purpose computers, such as dedicated network routers or switches.
This book deals with Munin in some detail (see Section 12.4.1, “Setting Up Munin”) as part of Chapter 12: “Advanced Administration”. Debian also provides a similar tool, cacti. Its deployment is slightly more complex, since it is based solely on SNMP. Despite having a web interface, grasping the concepts involved in configuration still requires some effort. Reading the HTML documentation (/usr/share/doc/cacti/html/index.html) should be considered a prerequisite.
ALTERNATIVE mrtg
mrtg (in the similarly-named package) is an older tool. Despite some rough edges, it can aggregate historical data and display them as graphs. It includes a number of scripts dedicated to collecting the most commonly monitored data such as processor load, network traffic, web page hits, and so on.
The mrtg-contrib and mrtgutils packages contain example scripts that can be used directly.
14.3.3. Detecting Changes
Once the system is installed and configured, and barring security upgrades, there's usually no reason for most of the files and directories to evolve, data excepted. It is therefore interesting to make sure that files actually do not change: any unexpected change would therefore be worth investigating. This section presents a few tools able to monitor files and to warn the administrator when an unexpected change occurs (or simply to list such changes).
14.3.3.1. Auditing Packages: debsums and its Limits
GOING FURTHER Protecting against upstream changes
debsums is useful in detecting changes to files coming from a Debian package, but it will be useless if the package itself is compromised, for instance if the Debian mirror is compromised. Protecting against this class of attacks involves using APT's digital signature verification system (see Section 6.5, “Checking Package Authenticity”), and taking care to only install packages from a certified origin.
debsums is an interesting tool since it allows finding what installed files have been modified (potentially by an attacker), but this should be taken with a grain of salt. First, because not all Debian packages provide the fingerprints required by this program (they can be found in /var/lib/dpkg/info/package.md5sums when they exist). As a reminder: a fingerprint is a value, often a number (even though in hexadecimal notation), that contains a kind of signature for the contents of a file. This signature is calculated with an algorithm (MD5 or SHA1 being well-known examples) that more or less guarantee that even the tiniest change in the file contents implies a change in the fingerprint; this is known as the “avalanche effect”. This allows a simple numerical fingerprint to serve as a litmus test to check whether the contents of a file have been altered. These algorithms are not reversible; in other words, for most of them, knowing a fingerprint doesn't allow finding the corresponding contents. Recent mathematical advances seem to weaken the absoluteness of these principles, but their use is not called into question so far, since creating different contents yielding the same fingerprint still seems to be quite a difficult task.
In addition, the md5sums files are stored on the hard disk; a thorough attacker will therefore update these files so they contain the new control sums for the subverted files.
The first drawback can be avoided by asking debsums to base its checks on a .deb package instead of relying on the md5sums file. But that requires dowloading the matching .deb files first:
# apt-get --reinstall -d install `debsums -l`
[ ... ]
# debsums -p /var/cache/apt/archives -g
It is also worth noting that, in its default configuration, debsums automatically generates the missing md5sums files whenever a package is installed using APT.
The other problem can be avoided in a similar fashion: the check must simply be based on a pristine .deb file. Since this implies having all the .deb files for all the installed packages, and being sure of their integrity, the simplest way is to grab them from a Debian mirror. This operation can be slow and tedious, and should therefore not be considered a proactive technique to be used on a regular basis.
# apt-get --reinstall -d install `grep-status -e 'Status: install ok installed' -n -s Package`
[ ... ]
# debsums -p /var/cache/apt/archives --generate=all
Note that this example uses the grep-status command from the grep-dctrl package, which is not installed by default.
14.3.3.2. Monitoring Files: AIDE
The AIDE tool (Advanced Intrusion Detection Environment) allows checking file integrity, and detecting any change against a previously recorded image of the valid system. This image is stored as a database (/var/lib/aide/aide.db) containing the relevant information on all files of the system (fingerprints, permissions, timestamps and so on). This database is first initialized with aideinit; it is then used daily (by the /etc/cron.daily/aide script) to check that nothing relevant changed. When changes are detected, AIDE records them in log files (/var/log/aide/*.log) and sends its findings to the administrator by email.
IN PRACTICE Protecting the database
Since AIDE uses a local database to compare the states of the files, the validity of its results is directly linked to the validity of the database. If an attacker gets root permissions on a compromised system, they will be able to replace the database and cover their tracks. A possible workaround would be to store the reference data on read-only storage media.
Many options in /etc/default/aide can be used to tweak the behaviour of the aide package. The AIDE configuration proper is stored in /etc/aide/aide.conf and /etc/aide/aide.conf.d/ (actually, these files are only used by update-aide.conf to generate /var/lib/aide/aide.conf.autogenerated). Configuration indicates which properties of which files need to be checked. For instance, the contents of log files changes routinely, and such changes can be ignored as long as the permissions of these files stay the same, but both contents and permissions of executable programs must be constant. Although not very complex, the configuration syntax is not fully intuitive, and reading the aide.conf(5) manual page is therefore recommended.