Patrick Schoenfeld: Let me introduce DPKG::Log and dpkg-report
We have customers, which require a report about what we've done during maintenance windows. Usually this includes a report about upgrades, newly installed packages etc. and obviously everything we've done apart from that.
Till now we've prepared them manually. For a greater bunch of systems this is a big PITA, because to be somehow useful you have to collect the data for all systems and after that, prepare a report where you have:
Its also error-prone because humans make mistakes.
Perl to the rescue!
At least the part about generating a report about installed/upgraded/removed packages could be automated, because dpkg writes a well-formed logfile in /var/log/dpkg.log. But I noticed that there appearently is no library specialised at parsing that file. Its not a big deal, because the format of that file is really simple, but a proper library would be nice anyway.
And so I wrote such a library.
It basically takes a logfile, reads it line by line and stores each line parameterized into a generic object.
Its features include:
Based on that, I wrote another library DPKG::Log::Analyse, which takes a log, parses it with DPKG::Log and then extracts the more relevant information such as installed packages, upgraded packages, removed packages etc.
This, in turn, features:
- Info about newly installed packages
- Info about removed packages
- Info about upgraded packages
- Info about packages which got installed and removed again
- Info about packages which stayed halfinstalled or halfconfigured at the end of the logfile (or defined reporting period)
These libraries are already uploaded to CPAN and packaged for Debian.
They passed the NEW queue very quickly and are therefore available for Sid:
http://packages.debian.org/sid/libdpkg-log-perl
As an example use (and for my own use case, as stated above), I wrote dpkg-report, which uses the module and a Template::Toolkit based template to generate a report about what happened in the given logfile.
It currently misses some documentation, but it works somehow like this:
Report for a single host over the full log:
dpkg-report
Report for a single host for the last two days:
dpkg-report --last-two-days
Report for multiple hosts (and logfiles):
The script expects that each log file has the name <systemname>.dpkg.log so that it can guess the hostname from the system and can grab all such log files from a directory if a directory is specified as log-file arguments:</systemname>
dpkg-report --log-file /path/to/logs
This will generate a report about all systems without any merging.
Report for multiple hosts with merging:
dpkg-report --log-file /path/to/logs --merge
This will do the following:
dpkg-Report for all:
------------------------
Newly Installed:
abc
Removed:
foo
Upgraded:
bar (0.99-12 - 1.00-1)
dpkg-Report for test*:
------------------------
Newly Installed:
blub
dpkg-Report for test1:
------------------------
Upgraded:
baz (1.0.0-1 -> 1.0.0-2)
dpkg-Report for test2:
------------------------
Upgraded:
zab (0.9.7 -> 0.9.8)
Currently this report generator is only included in the source package or in the git(hub) repository of the library. I wonder if it makes sense to let the source package build another binary package for it.
But its only a 238 lines perl script with a dependency on the perl library so I'm unsure if it warrants a new binary package. What do others think?
Till now we've prepared them manually. For a greater bunch of systems this is a big PITA, because to be somehow useful you have to collect the data for all systems and after that, prepare a report where you have:
- The upgrades done on all systems (e.g. a libc or kernel update)
seperated from - the updates specific to a certain system or system class
Its also error-prone because humans make mistakes.
Perl to the rescue!
At least the part about generating a report about installed/upgraded/removed packages could be automated, because dpkg writes a well-formed logfile in /var/log/dpkg.log. But I noticed that there appearently is no library specialised at parsing that file. Its not a big deal, because the format of that file is really simple, but a proper library would be nice anyway.
And so I wrote such a library.
It basically takes a logfile, reads it line by line and stores each line parameterized into a generic object.
Its features include:
- Parsing the logfile (obviously)
- Storing each line in a logfile as a DPKG::Log::Entry object that holds informations about the entry type (e.g. action or status update), the package associated (if any), timestamp, verbatim line, line number etc.
- Limiting the number of lines parsed to a time range specified by DateTime objects
Based on that, I wrote another library DPKG::Log::Analyse, which takes a log, parses it with DPKG::Log and then extracts the more relevant information such as installed packages, upgraded packages, removed packages etc.
This, in turn, features:
- Info about newly installed packages
- Info about removed packages
- Info about upgraded packages
- Info about packages which got installed and removed again
- Info about packages which stayed halfinstalled or halfconfigured at the end of the logfile (or defined reporting period)
These libraries are already uploaded to CPAN and packaged for Debian.
They passed the NEW queue very quickly and are therefore available for Sid:
http://packages.debian.org/sid/libdpkg-log-perl
As an example use (and for my own use case, as stated above), I wrote dpkg-report, which uses the module and a Template::Toolkit based template to generate a report about what happened in the given logfile.
It currently misses some documentation, but it works somehow like this:
Report for a single host over the full log:
dpkg-report
Report for a single host for the last two days:
dpkg-report --last-two-days
Report for multiple hosts (and logfiles):
The script expects that each log file has the name <systemname>.dpkg.log so that it can guess the hostname from the system and can grab all such log files from a directory if a directory is specified as log-file arguments:</systemname>
dpkg-report --log-file /path/to/logs
This will generate a report about all systems without any merging.
Report for multiple hosts with merging:
dpkg-report --log-file /path/to/logs --merge
This will do the following:
- Report packages which where installed/removed/etc. on all systems seperate from the upgrades only done on the specific systems
- If systems start with a common name and end on a number (e.g. mail1, mail2, mail3 etc.), report packages which where installed on all systems with that common name seperate from the upgrades only done on the specific systems
- For a specific system list only the changes on that particular system and nothing else.
dpkg-Report for all:
------------------------
Newly Installed:
abc
Removed:
foo
Upgraded:
bar (0.99-12 - 1.00-1)
dpkg-Report for test*:
------------------------
Newly Installed:
blub
dpkg-Report for test1:
------------------------
Upgraded:
baz (1.0.0-1 -> 1.0.0-2)
dpkg-Report for test2:
------------------------
Upgraded:
zab (0.9.7 -> 0.9.8)
Currently this report generator is only included in the source package or in the git(hub) repository of the library. I wonder if it makes sense to let the source package build another binary package for it.
But its only a 238 lines perl script with a dependency on the perl library so I'm unsure if it warrants a new binary package. What do others think?
I have been packaging for
The 0.9.2 release of
...For the cases where, yes, you actually do want it.
For the impatient
assuming you're in a clean checkout on the tip of your current branch:
You re eager to start contributing to Debian, your first action is to subscribe to some high-profile mailing lists (like
One of the reasons I like Scotland is the fun that Scottish people have with language. I'm going to use two examples to illustrate my point:


Yesterday I had my last day at $company and today is my first day in freedom. I'll write about my future plans in another blog post. Here I just want to express the joy I had yesterday afternoon when I did:
My server, which is among other things hosting this weblog, is a dedicated server originally hosted by the German webhoster webperoni. It was a bargain at that time, and still a good price, and I was satisfied with webperoni s performance and reliability. Unfortunately, they were taken over by another web hosting company, best known for their
At the
I've been very quiet on here for a few months, and the reason is (of course) because I've been doing other things. No surprise there, I guess!
What I've been doing, first of all, is writing
The interview at Munich seemed to go well. Well, at least, I had fun. We'll see how that turns out.
Another week is over and it's time to look again at state of
After almost six years from the original