Search Results: "ag"

16 August 2013

Sylvain Le Gall: Augeas tips and tricks for Puppet user: edit a complex node.

I have a recurring problem when trying to use augeas on a complex node: trying to edit a specific entry in a list which is uniquely defined by many attributes. You probably don't know that you have this problem, but it is easy to spot it into your augeas/puppet resource. Here are some symptoms of this problem:
 
 augeas  
   "setup-shorewall":
     changes =>
       [
         "set entry[last() + 1]/source 'all'",
         "set entry[last()]/dest 'all'",
         "set entry[last()]/policy 'REJECT'",
         "set entry[last()]/log_level 'info'",
       ],
     onlyif => "match entry[source = 'all'][dest = 'all'][policy = 'REJECT'] size == 0";
For a long time, I thought it was the only solution. But last week, I read again the documentation and found another solution. My main concerns are the onlyif and last() parts, it doesn't look clean to me. The problem is that I cannot define the entry all at once and if I use a value that will be set late, the node cannot be targeted in between. The clean way to do this was to define first the target attribute. Typically, in augeas changes:
 set spec[user = '$name']/user '$name'
This way if the node doesn't exist it is created and you can then use it directly:
 set spec[user = '$name']/host_group/host 'ALL'
 set spec[user = '$name']/host_group/command1 'ALL'
 set spec[user = '$name']/host_group/command1/tag 'PASSWD'
But sometimes it is not possible to set the attribute directly -- typically when you need to use multiple attribute. The solution in this case is to use defnode:
 defnode target entry[#comment = 'puppet: <%= name %>']/ "<%= name %>"
 set $target/action '<%= action %>'
 set $target/source '<%= source %>'
 set $target/#comment 'puppet: <%= name %>'
 clear $target
The big trick here is that defnode needs a value, but most of the time you cannot set a value for the node -- because it has none. To solve this, you set a value with defnode, process with your change and you clear the node at the end. This recent discovery has simplify a lot some augeas changes I use. Feel free to leave comment on your personal technique to deal with augeas and puppet.

Joey Hess: Debian at 20

RichiH used flint and steel to light the bonfire. We carefully fed it up from those sparks to a blaze. Put on the biggest logs we could find to make it last. Now I'm sitting on the hill above it watching folks gathered around. A poem is being read, in Hindi, then translated for us to English. Others have shared songs and poems in a dozen languages, both classics and their own. Out below the darkness of the lake. This is Debian at 20. At the start of this DebConf, I gave a talk on "Debian Cosmology. In that (and a later "dh_busfactor" talk) I shared my hopes and my fears. I was conflicted about giving the this talk, worked on it for weeks, felt it might not work, or be depressing. I've had nothing but good comments about it. 20 years is ages in internet time and technical projects ossify over time. The last session I was in this afternoon was a presentation of a new tool, which I hope & feel has the potential to fundamentally change an important and suboptimal part of Debian. Then I walked outside to a rainbow over crystal clear Swiss alps on the horizon. How encouraging, and what a nice story that will be around some future campfire.

Justus Winter: All the important bits are there - please test and review :)

Finally, more bootlog-pr0n:
start ext2fs: Hurd server bootstrap: ext2fs[device:hd0s1] exec init proc auth
INIT: version 2.88 booting
Using makefile-style concurrent boot in runlevel S.
Activating swap...done.
Checking root file system...fsck from util-linux 2.20.1
hd2 : tray open or drive not ready
hd2 : tray open or drive not ready
hd2 : tray open or drive not ready
hd2 : tray open or drive not ready
end_request: I/O error, dev 02:00, sector 0
/dev/hd0s1: clean, 44680/181056 files, 292234/723200 blocks
done.
Activating lvm and md swap...(default pager): Already paging to partition hd0s5!
done.
Checking file systems...fsck from util-linux 2.20.1
hd2 : tray open or drive not ready
hd2 : tray open or drive not ready
end_request: I/O error, dev 02:00, sector 0
done.
Cleaning up temporary files... /tmp.
Mounting local filesystems...done.
Activating swapfile swap...(default pager): Already paging to partition hd0s5!
done.
df: Warning: cannot read table of mounted file systems: No such file or directory
Cleaning up temporary files....
Configuring network interfaces...inetutils-ifconfig: invalid arguments
ifup: failed to open pid file /run/network/ifup-/dev/eth0.pid: No such file or directory
Internet Systems Consortium DHCP Client 4.2.2
Copyright 2004-2011 Internet Systems Consortium.
All rights reserved.
For info, please visit https://www.isc.org/software/dhcp/
can't create /var/lib/dhcp/dhclient./dev/eth0.leases: No such file or directory
Listening on Socket//dev/eth0
Sending on   Socket//dev/eth0
DHCPDISCOVER on /dev/eth0 to 255.255.255.255 port 67 interval 8
DHCPREQUEST on /dev/eth0 to 255.255.255.255 port 67
DHCPOFFER from 10.0.2.2
DHCPACK from 10.0.2.2
can't create /var/lib/dhcp/dhclient./dev/eth0.leases: No such file or directory
bound to 10.0.2.15 -- renewal in 34744 seconds.
done.
Cleaning up temporary files....
Setting up X socket directories... /tmp/.X11-unix /tmp/.ICE-unix.
INIT: Entering runlevel: 2
Using makefile-style concurrent boot in runlevel 2.
Starting enhanced syslogd: rsyslogd.
Starting deferred execution scheduler: atd.
Starting periodic command scheduler: cron.
Starting system message bus: dbusFailed to set socket option"/var/run/dbus/system_bus_socket": Protocol not available.
Starting OpenBSD Secure Shell server: sshd.
unexpected ACK from keyboard
GNU 0.3 (debian) (console)
login: root
root@debian:~# shutdown -h now
Broadcast message from root@debian (console) (Fri Aug 16 20:02:47 2013):
The system is going down for system halt NOW!
INIT: Switching to runlevel: 0root@debian:~#
INIT: Sending processes the TERM signal
INIT: Sending processes the KILL signal
Using makefile-style concurrent boot in runlevel 0.
Stopping deferred execution scheduler: atd.
task c10f72a8 deallocating an invalid port 2098928, most probably a bug.
Asking all remaining processes to terminate...done.
All processes ended within 1 seconds...done.
Stopping enhanced syslogd: rsyslogd.
Deconfiguring network interfaces...ifdown: failed to open pid file /run/network/ifdown-/dev/eth0.pid: No such file or directory
Internet Systems Consortium DHCP Client 4.2.2
Copyright 2004-2011 Internet Systems Consortium.
All rights reserved.
For info, please visit https://www.isc.org/software/dhcp/
can't create /var/lib/dhcp/dhclient./dev/eth0.leases: No such file or directory
Listening on Socket//dev/eth0
Sending on   Socket//dev/eth0
/bin/sh: 1: ifconfig: not found
done.
Deactivating swap...swapoff: /dev/hd0s5: 177152k swap space
done.
Unmounting weak filesystems...umount: /etc/mtab: Warning: duplicate entry for device /dev/hd0s1 (/dev/cons)
umount: could not find entry for: /dev/cons
done.
Unmounting local filesystems...done.
mount: cannot remount /: Device or resource busy
Will now halt.
init: notifying pfinet of shutdown...init: notifying tmpfs swap of shutdown...init: notifying tmpfs swap of shutdown...init: notifying tmpfs swap of shutdown...init: notifying ext2fs device:hd0s1 of shutdown...init: halting Mach (flags 0x8)...
In tight loop: hit ctl-alt-del to reboot
(You might note that df complains about not being able to read the mtab file. That is because it has been built with _PATH_MOUNTED being /var/run/mtab. This will correct itself when the coreutils package is being rebuilt against a patched libc.) I spent my last two weeks with polishing my patch series, that meant a lot of package rebuilds and that means a lot of waiting (even more so on the Hurd, short version: fakeroot-hurd could be fast but is not yet working properly, fakeroot-tcp is slow) and thus some frustration ;) Also I had to pay special attention so that the upgraded packages could be installed without accidentally breaking anything in the process. Making sysvinit pid 1 is surprisingly tricky in this regard since it breaks the ABI and requires a libc fix that also worked with the current Hurd servers. Here are the patches: So I had some spare time on my hand while waiting for numerous package rebuilds and I took this as an opportunity to read papers about Mach and to familiarize myself with mig, the Mach Interface Generator. While I have used it in the past, I had not yet looked at its implementation. And I had to patch the exec server, and there was both code implementing a questionable feature (on-demand unzipping of binaries) and code that was not even compiled (courtesy of the preprocessor) and had probably bit-rot by now. So I figured I could spend my time doing some cleanups: I have rebuild all the necessary packages and uploaded them into an apt repository:
deb http://teythoon.cryptobitch.de/gsoc/heap/debian unstable main
Please use unstable for now. Also make sure that you have a recovery plan for your Debian/Hurd installation if anything goes wrong. For your convenience there's a seed tarball containing packages with the appropriate sources.list.d snippets and the repository key: https://teythoon.cryptobitch.de/gsoc/heap/debian/seed.tar If you want to switch to the new runsystem.sysv, do:
# update-alternatives --config runsystem
Whenever you switch runsystems, please use reboot-hurd to reboot the system. This is the most robust way. Known issues: Next week I will address the network related issues. By now they are the source of most of the error messages in the bootlog.

Michael Stapelberg: My DebConf13 talks are online

I gave two talks at this year s DebConf, both about systemd. A huge thanks goes to the video team for their excellent work and putting up the videos that quickly! Find the recordings and slides here:
  1. Making your package work with systemd (508 MiB ogv) (Slides ( 230 KiB PDF))
  2. systemd myths debunked! (456 MiB ogv) (Slides ( 230 KiB PDF))

Richard Hartmann: Release Critical Bug report for Week 33

One more for DebConf. The UDD bugs interface currently knows about the following release critical bugs: How do we compare to the Squeeze release cycle?
Week Squeeze Wheezy Diff
43 284 (213+71) 468 (332+136) +184 (+119/+65)
44 261 (201+60) 408 (265+143) +147 (+64/+83)
45 261 (205+56) 425 (291+134) +164 (+86/+78)
46 271 (200+71) 401 (258+143) +130 (+58/+72)
47 283 (209+74) 366 (221+145) +83 (+12/+71)
48 256 (177+79) 378 (230+148) +122 (+53/+69)
49 256 (180+76) 360 (216+155) +104 (+36/+79)
50 204 (148+56) 339 (195+144) +135 (+47/+90)
51 178 (124+54) 323 (190+133) +145 (+66/+79)
52 115 (78+37) 289 (190+99) +174 (+112/+62)
1 93 (60+33) 287 (171+116) +194 (+111/+83)
2 82 (46+36) 271 (162+109) +189 (+116/+73)
3 25 (15+10) 249 (165+84) +224 (+150/+74)
4 14 (8+6) 244 (176+68) +230 (+168/+62)
5 2 (0+2) 224 (132+92) +222 (+132/+90)
6 release! 212 (129+83) +212 (+129/+83)
7 release+1 194 (128+66) +194 (+128/+66)
8 release+2 206 (144+62) +206 (+144/+62)
9 release+3 174 (105+69) +174 (+105/+69)
10 release+4 120 (72+48) +120 (+72/+48)
11 release+5 115 (74+41) +115 (+74/+41)
12 release+6 93 (47+46) +93 (+47/+46)
13 release+7 50 (24+26) +50 (+24/+26)
14 release+8 51 (32+19) +51 (+32/+19)
15 release+9 39 (32+7) +39 (+32/+7)
16 release+10 20 (12+8) +20 (+12/+8)
17 release+11 24 (19+5) +24 (+19/+5)
18 release+12 2 (2+0) +2 (+2/+0)
Graphical overview of bug stats thanks to azhag:

Lucas Nussbaum: Debian birthday

This upload was the first one of my very first package in Debian. It was sponsored by Dafydd Harries, who I ve finally met at DebConf13, and got out of NEW on 2005-08-16. Exactly 8 years ago today. I only realized that yesterday evening, and Debian s birthday feels even more special to me now. Dafydd, it looks like I owe you a lot! :)

Dirk Eddelbuettel: RcppArmadillo 0.3.910.0

A new minor release 3.910.0 of Armadillo came out a few days ago. A new RcppArmadillo release 0.3.910.0 was provided rightaway, and after a brief back-and-forth with CRAN (mostly having to do with the non-standard vignette corresponding to our CSDA paper and its use of LaTeX minted) is now on CRAN and also in Debian. Conrad made some standard cross-products faster, and added vectorise(), any() and all() functions. We also added an extension header for more automated conversion of sparse matrices (which is already used by a post on the Rcpp Gallery). The complete list of changes is below.
Changes in RcppArmadillo version 0.3.910.0 (2013-08-12)
  • Upgraded to Armadillo release Version 3.910.0 (Pyrenees)
    • faster multiplication of a matrix with a transpose of itself, ie. X*X.t() and X.t()*X
    • added vectorise() for reshaping matrices into vectors
    • added all() and any() for indicating presence of elements satisfying a relational condition
  • Added conversion support for sparse matrices (of type double) created by the Matrix package as class dgCMatrix
  • Moved vignette sources from inst/doc to vignettes; set BuildVignettes: FALSE as the minted mode for LaTeX upsets the CRAN builders.
Courtesy of CRANberries, there is also a diffstat report for the most recent release As always, more detailed information is on the RcppArmadillo page. Questions, comments etc should go to the rcpp-devel mailing list off the R-Forge page.

15 August 2013

Daniel Leidert: HP N54L Microserver - energy efficiency and power management

I recently worked on activating power management functions, reduce energy consumption and noise of my little HP N54L "toy". During this process I tried to avoid the usage of /etc/rc.local and set things by udev, hdparm and friends. Below are my results. Actual resultsWith the following steps my system (N54L + 3xWD20EFRX HDD +1xWD5003AZEX HDD + LCD-mod + case fan mod + Debian Wheezy) uses 27W in idle mode. The USB W-LAN card uses another 10W. In active mode, e.g. compiling source code, the system runs (and boots) with around 57W. The highest power consumption observed is during startup phase with 88W. First things firstFor the following steps it might be necessary to have some packages installed, that maybe do not occur in this post. If I missed something, I appreciate a hint. Further the following steps might produce even better results with a custom kernel. I'm using the stock linux-image-3.2.0-4-amd64 kernel image as the time of writing and I have these packages installed: amd64-microcode, firmware-linux, firmware-linux-free, firmware-linux-nonfree and firmware-atheros (the latter for my WLAN card). ASPM and ACPIFirst I enabled PCIE ASPM in my (non-modded) BIOS and forced it together with ACPI via grub by changing GRUB_CMDLINE_LINUX_DEFAULT in /etc/default/grub so it looks like this:

GRUB_CMDLINE_LINUX_DEFAULT="quiet splash acpi=force pcie_aspm=force nmi_watchdog=0"
ASPM has now been enabled as lspci prooves:

00:04.0 PCI bridge: Advanced Micro Devices, Inc. [AMD] RS780/RS880 PCI to PCI bridge (PCIE port 0) (prog-if 00 [Normal decode])
[..]
LnkCap: Port #1, Speed 5GT/s, Width x1, ASPM L0s L1, Latency L0 <64ns, L1 <1us
ClockPM- Surprise- LLActRep+ BwNot+
LnkCtl: ASPM L0s L1 Enabled; RCB 64 bytes Disabled- Retrain- CommClk+
ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
[..]
00:06.0 PCI bridge: Advanced Micro Devices, Inc. [AMD] RS780 PCI to PCI bridge (PCIE port 2) (prog-if 00 [Normal decode])
[..]
LnkCap: Port #3, Speed 5GT/s, Width x1, ASPM L0s L1, Latency L0 <64ns, L1 <1us
ClockPM- Surprise- LLActRep+ BwNot+
LnkCtl: ASPM L0s L1 Enabled; RCB 64 bytes Disabled- Retrain- CommClk+
ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
02:00.0 USB controller: Renesas Technology Corp. uPD720201 USB 3.0 Host Controller (rev 03) (prog-if 30 [XHCI])
[..]
LnkCap: Port #0, Speed 5GT/s, Width x1, ASPM L0s L1, Latency L0 <4us, L1 unlimited
ClockPM+ Surprise- LLActRep- BwNot-
LnkCtl: ASPM L0s L1 Enabled; RCB 64 bytes Disabled- Retrain- CommClk+
ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
[..]
03:00.0 Ethernet controller: Broadcom Corporation NetXtreme BCM5723 Gigabit Ethernet PCIe (rev 10)
[..]
LnkCap: Port #0, Speed 2.5GT/s, Width x1, ASPM L0s L1, Latency L0 <1us, L1 <64us
ClockPM+ Surprise- LLActRep- BwNot-
LnkCtl: ASPM L0s L1 Enabled; RCB 64 bytes Disabled- Retrain- CommClk+
ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
[..]
Even so /sys/module/pcie_aspm/parameters/policy will still show as below:

[default] performance powersave
I'll show how to set the powersave value in /sys/module/pcie_aspm/parameters/policy in the next section.JFTR: These are my ACPI related packages installed: acpi, acpid, acpi-support and acpi-support-base. Enable powersaving via UDEVThe following rules file /etc/udev/rules.d/90-local-n54l.rules has been inspired by a blog post. It enables powersaving modes for all PCI, SCSI and USB devices and ASPM. Further the internal RADEON cards power profile is set to the low value. There is no monitor connected usually. The file contains these rules:

SUBSYSTEM=="module", KERNEL=="pcie_aspm", ACTION=="add", TEST=="parameters/policy", ATTR parameters/policy ="powersave"

SUBSYSTEM=="i2c", ACTION=="add", TEST=="power/control", ATTR power/control ="auto"
SUBSYSTEM=="pci", ACTION=="add", TEST=="power/control", ATTR power/control ="auto"
SUBSYSTEM=="usb", ACTION=="add", TEST=="power/control", ATTR power/control ="auto"
SUBSYSTEM=="usb", ACTION=="add", TEST=="power/autosuspend", ATTR power/autosuspend ="2"
SUBSYSTEM=="scsi", ACTION=="add", TEST=="power/control", ATTR power/control ="auto"
SUBSYSTEM=="spi", ACTION=="add", TEST=="power/control", ATTR power/control ="auto"

SUBSYSTEM=="drm", KERNEL=="card*", ACTION=="add", DRIVERS=="radeon", TEST=="power/control", TEST=="device/power_method", ATTR device/power_method ="profile", ATTR device/power_profile ="low"

SUBSYSTEM=="scsi_host", KERNEL=="host*", ACTION=="add", TEST=="link_power_management_policy", ATTR link_power_management_policy ="min_power"
Set harddrives spindown timeoutI decided to sent my system drive to standby after 20 minutes and the RAID drives after 15 minutes. This is usually ok, because the RAID isn't always used. hdparm is the right tool to realize this. Many people use the /dev/disk/by-uuid/... syntax here, to avoid having to touch the configuration file if some system configuration changes. Because I'm running a RAID, I couldn't use this syntax, although it might be possible to use /dev/disk/by-id/... instead. Well for the moment I stay with the configuration below. The relevant part of /etc/hdparm.conf is:

[..]

# system harddrive
/dev/sda
spindown_time = 240


# below are the WD20EFRX drives
/dev/sdb
spindown_time = 180


/dev/sdc
spindown_time = 180


/dev/sdd
spindown_time = 180

Idle modeWhen there is nothing to do for the system, all I hear is the (still a bit noisy) fan of the power supply, which I might replace in the future too. Either by testing a different fan or by replacing the whole power supply unit by the fanless FORTRON FSP150-50TNF or (even better) a picoPSU.The system currently shows a power consumption of around 37W in idle mode whereas the USB W-LAN card itself needs around 10W. There is a possibility to enable power savings mode for this card too. I could add this entry to /etc/udev/rules.d/90-local-n54l.rules:

SUBSYSTEM=="net", ACTION=="add", KERNEL=="wlan*" RUN+="/usr/bin/iw dev %k set power_save on"
But it turned out that the connection became a bit unstable after it. So I don't use this rule. More on the roadThere are a lot more options one can easily find via $search_engine. The N54L system could be brought to sleep and woken up by LAN via Wake-on-LAN (WOL). This is a feature I don't use. I've also read rumors about enabling different sleep/suspend states of the system, which seems to require to install a modded BIOS. Well, I'll post news and changes if they happen to come;)

Bits from Debian: Happy 20th birthday Debian!

Today is Debian's 20 anniversary. This anniversary would have been impossible without a strong community of users and developers. That's why for its commemoration, we asked the Debian community what Debian meant to them. Below you have a selection of the answers we got.
Since I run Debian on my computers, I do not play anymore to 3D shooting games, not because of the lack of Free 3D drivers, but because developing Debian is more fun and addictive. Debian is a truly community based distro which is devoted to FOSS ideas and standards. It perfectly works on high variety of hardware. Users from all over the world have been using Debian and contributing to it during 20 years. And I am proud to be one of them. Happy Birthday, Debian! When I considered switching to Linux I asked friends which distribution to choose. I was told skip other distributions that were considered more newbie-friendly at the time and go straight to Debian instead. It might be more work initially, they said, but it would save me the hassle of switching to Debian later on, which I would inevitably do. Turns out they were right. I started my Linux experience more than ten years ago with Debian and have yet to see a better Linux distribution. [...] You are a worldwide community of volunteers working together for 20 years now. To me this is an encouraging example and, given the sorry state of the world we live in, more important than the technical quality of the operating system. I wish you another successful twenty years and to stay as independent as you are. I initially started in Debian as it was an interesting technical challenge. Over the years the community and having good standards around what Free Software is has become more important. As a user of Debian for 14 years, and a former Debian Developer for 10, I would like to wish a happy birthday to the best project on the internet, and the best Linux distribution ever. Thanks for all of your support over the years! "Rock Solid Stability and Absolute Freedom" Thats what Debian means to me. Debian gives me a feeling that I am using the best linux has to offer. You know your machine is in safe hands. As a long time Debian user and sometimes supporter [...] I wish Debian all the best for next 20 years and beyond! Debian is the universal operating system. And it's free (and not just as in beer ;-) Debian is an awesome combination of obsessive high quality software and software freedom. It's a pleasure to be able to use and contribute to this project. Thanks to all for their excellent work! Cheers to 2**20 years more! [...] I am so very thankful to all of the people who have contributed to and continue to contribute to such a wonderful ecosystem of tools. I love the commitment to the long term goals of security, freedom and transparency with respect to the computer systems that we all use and rely upon and the information that we store on them. Debian is the operating system that makes me free. Debian is a family gathered around great idea. Its pure love.

Richard Hartmann: DebConf13 group photo

Group photo and historic T-shirts for general consumption. Click through for full resolution. If you want to have all the JPGs and NEFs (raw files), feel free to grab them from this DebConf-local mirror. Those files still have the EXIF information intact to easen any post-processing done by people who can actually do this stuff. It would probably make sense to agree on a canonical version, crop, clean, and then put on numbers with names, etc. Group photo Historic T-shirts, in order

Russell Coker: Google web sites and Chromium CPU Use

Chromium is the free software build of the Google Chrome web browser. It s essentially the same as the Google code but will often be an older version, particularly when you get Chromium from Debian/Stable (or any other Linux distribution that doesn t track the latest versions all the time) and compare it to getting Chrome straight from Google. My wife is using Chromium on an AMD Opteron 1212 for all the usual web browsing tasks. Recently I ve noticed that it takes a lot of CPU time whenever she leaves a Google web site open, that can be Google+, Gmail, or Youtube. Web standards are complex and it s difficult to do everything the way that one might desire. Making a web browser that doesn t take 100% CPU time when the user is away from their desk may be a difficult technical challenge. Designing a web site that doesn t trigger such unwanted behavior in common web browsers might also be a challenge. But when one company produces both a web browser and some web sites that get a lot of traffic it s rather disappointing that they don t get this right. It could be that Google have fixed this in a more recent version of the Chrome source tree, and it could be that they fixed the browser code before rolling out a new version of Google+ etc which causes problems with the old version (which might explain why I ve never seen this problem). But even if that is the case it s still disappointing that they aren t supporting older versions. There is a real need for computers that don t need to be updated all the time, running a 3 month old Linux distribution such as Debian/Wheezy shouldn t be a problem. There s also a possibility that part of the cause of the problem is that an Opteron 1212 is a relatively slow CPU by today s standards and it s the slowest system I m currently supporting for serious desktop use. I don t even think it was one of the fastest CPUs available when it was released 4 years ago. But I think we should be able to expect systems to remain usable for more than 4 years. The Opteron 1212 system is a Dell PowerEdge tower server that is used as a workstation and a file server, so while I get desktop systems with faster CPUs for free I want to keep using the old PowerEdge server to avoid data corruption. As an aside I ve been storing important data on BTRFS for a year now and the only data loss I ve suffered has been due to a faulty DIMM. The filesystem checksums built in to modern filesystems such as BTRFS and ZFS mean that RAM corruption covers a greater portion of the risk to data integrity and the greater complexity of the data structures in such filesystems gives the possibility of corruption that can t be fixed without mkfs (as happened to me twice on the system with a bad DIMM). The consequences of such wasted CPU use are reduced CPU time for other programs which might be doing something useful, extra electricity use, and more noise from CPU cooling fans (which is particularly annoying for me in this case). Any suggesstions for reducing the CPU use of web browsers, particularly when idle?

Daniel Pocock: Free software for free markets

Co-inciding with the first DebConf in Switzerland, one of the world's leading financial centers, the first official packaging of open messaging and market data distribution framework OpenMAMA for a Linux distribution has just been uploaded. The packages, along with the Avis low-latency event-router middleware/transport were uploaded to the Debian unstable catalog this week and will soon be available conveniently to install with apt-get. OpenMAMA was developed by NYSE Technologies and includes code originally developed under their previous brand Wombat Financial Software. The project was recently placed under the strategic umbrella of the Linux Foundation. Packages for Fedora and EPEL The initial packaging effort has identified a number of build system bugs, patches have been contributed upstream. The resolution of these issues and the experience gained with the Debian package will make it relatively easy to package for RPM-based platforms in the near future. Not just for finance While it is clear that OpenMAMA's designers had financial services in mind (and the low-latency, high throughput transport of market price data in particular), the solution is generalised and likely to be useful in other domains where asynchronous messaging and inter-process communication is required. In fact, this distinction is made particularly clear by the division of OpenMAMA into two subprojects:
  • OpenMAMA itself is a general-purpose programming API for asynchronous messaging. It hides the details of various low-level transports, including the open source Avis event router and popular proprietary solutions like Tibco Rendezvous. Java programmers have had such a generalised API for many years (JMS), whereas OpenMAMA brings a similar paradigm to the C/C++ developer.
  • OpenMAMDA is a further layer on top of OpenMAMA designed to encapsulate market data. It competes directly with proprietary solutions such as those from Reuters and Bloomberg. OpenMAMDA comes with specific APIs for news, order books and options and could be extended for other markets.
Commercial background Reuters and Bloomberg are well known as two of the main vendors in the market data industry. Both of them (and other competitors) provide a range of data services and related software. Buying a data feed doesn't automatically give access to the software from the same franchise: the software and APIs are promoted as a valuable product in their own right. It is actually quite possible (and not unheard of) for an institution to run a Reuters Market Data System (RMDS) software stack fed by a data feed from Bloomberg or vice-versa. When an IT department or third-party vendor of trading systems wants to transport data between applications (or between multiple users of a single application) they will not typically do it themselves and one of these APIs is usually chosen. OpenMAMA appears to be a strong contender for such use cases. Bloomberg has recently made their API public, although OpenMAMA goes much further by making their source code public too. Development samples with the OpenMAMA API Programming the OpenMAMA API appears to be relatively consistent with the event-loop model that many developers are already comfortable with. For a very fast start, two compelling examples are the basic publisher and basic subscriber. Here is a sample of the code from the basic subscriber, demonstrating how to start the event loop and receive event notifications in a callback class:
static void createSubscriber (void)
 
    try
     
        gSubscription = new MamaBasicSubscription;
        gSubscription->createBasic (gTransport,
                                    gDefaultQueue,
                                    new SubscriptionCallback (),
                                    gTopic);
        gSubscription->setDebugLevel (gSubscLogLevel);
     
    catch (MamaStatus &status)
     
        cerr << "Error creating subscription: " 
             << status.toString () << endl;
        exit (1);
     
 
and here is some basic code that processes messages asynchronously:
class SubscriptionCallback : public MamaBasicSubscriptionCallback
 
public:
    virtual ~SubscriptionCallback ()  
    virtual void onCreate (MamaBasicSubscription*  subscription)
     
        if (gQuietLevel < 2)
         
            printf ("Created inbound subscription.\n");
         
     
    virtual void onError (MamaBasicSubscription*  subscription,
                          const MamaStatus&       status,
                          const char*             subject) 
     
        printf ("Error creating subscription: %s\n", 
                status.toString ());
        exit (1);
     
    virtual void onMsg (MamaBasicSubscription* subscription, 
                        MamaMsg&               msg)
     
        if (gQuietLevel < 2)
         
            printf ("Recieved msg.\n");
         
        DisplayIterator it;
        msg.iterateFields (it, NULL, NULL);
     
    //JG: need to add in onDestroy as this is abstract and must be implemented
    virtual void onDestroy (MamaBasicSubscription*  subscription, void * closure)
     
        if (gQuietLevel < 2)
         
            printf ("Destroyed subscription.\n");
         
     
 ;
Practical uses outside financial services Some people may feel that having a low-latency market-data system is overkill for anybody not running a trading desk. Not so. OpenMAMA may well prove useful and convenient as a one-stop-shop API for multi-currency accounting software (Postbooks, GnuCash, etc), for multi-currency web sites (e.g. UberCart) and many other applications.

Michal &#268;iha&#345;: DebConf day trip

Yesterday, we've spent nice day on a day trip of DebConf. I took the long hike, which ended up at Creux du Van, what is wonderful rocky place. The way up was through nice nature and we've met quite a lot of cows on the way: They we've finally arrived to Creux du Van: Of course we could not miss an opportunity to take a group photo: After wandering around the rocks, we took a lunch and headed to Neuch tel, where some music festival was going on: The whole day was nicely finished on a boat with a dinner. PS: More DebConf photos are available in my gallery.

Filed under: Debian English Photography phpMyAdmin 0 comments Flattr this!

Ingo Juergensmann: Exim4 and TLS with GMX/Web.de

Due to the unveiling of the NSA surveillance by Edward Snowden, some German mail providers decided last week to use TLS when sending mails out. For example GMX and Web.de. Usually there shouldn't be a problem with that, but it seems as if the Debian package of Exim4 (exim4-daemon-heavy) doesn't support any TLS ciphers that those providers will accept. The Debian package uses GnuTLS for TLS and there is Bug #446036 that asks for compilation against OpenSSL instead. Anyway, maybe it's something in my config as I don't use the Debian config but my own /etc/exim4/exim4.conf. Here are the TLS related parts:
tls_advertise_hosts = *
tls_certificate = /etc/exim4/ssl.crt/webmail-ssl.crt
tls_privatekey = /etc/exim4/ssl.key/webmail-server.key
That's my basic setup. After discovering that GMX and Web.de cannot send mails anymore, I added some more, following the Exim docs (it's commented out, because I don't use GnuTLS anymore):
#tls_dhparam = /etc/exim4/gnutls-params-2236
#tls_require_ciphers = $ if == $received_port 25 \
# NORMAL:%COMPAT \
# SECURE128
But still I got this kind of errors:
2013-08-14 22:49:27 TLS error on connection from mout.gmx.net [212.227.17.21] (gnutls_handshake): Could not negotiate a supported cipher suite.
As this didn't help either, I recompiled exim4-daemon-heavy against OpenSSL and et voila, it worked again. So, the question is if there's any way to get it working with GnuTLS ? Does the default Debian config work and if so, why? And if not, can a decision be made to use OpenSSL instead of GnuTLS? Reading the bug report it seems as if there are exemptions for linking against OpenSSL , so GPL wouldn't be violated. UPDATE 16.08.2013:
I reinstalled the GnuTLS version of exim4-daemon-heavy to test the recommendation in the comments with explicit tls_require_chiphers settings, but with no luck:
#tls_require_ciphers = SECURE256
#tls_require_ciphers = SECURE128
#tls_require_ciphers = NORMAL
These all resulted in the usual "(gnutls_handshake): Could not negotiate a supported cipher suite." error when trying one by one cipher setting. UPDATE 2 16.08.2013:
There was a different report about recent GnuTLS problem on the debian-user-german mailing list. It's not the same cause, but might be related.
Kategorie:

Russ Allbery: remctl 3.6

remctl is the network protocol that we use for nearly all of our middleware at Stanford. It's a flexible combination of GSS-API network authentication, ACLs, and mappings of commands to arbitrary local programs. The primary goal of this release is to fix a problem that I ran into with WebAuth: if a user of the remctl client library catches a signal during a connect with a timeout, the non-blocking connect support in the client library returned an error. Now, it returns to waiting for the connect if the select failed with EINTR. Also in this release, the Net::Remctl::Backend help formatting does something more reasonable with commands that have a very long syntax. Rather than push all of the command summaries into a tiny right-hand column, syntax lines that are too long will not affect the start column of the summary, just force the summary down one line. This release also fixes multiple compilation errors on Mac OS X and older Red Hat from a missing include, missing portability probes, and a workaround for Mac OS X's broken PCRE library installation. (The library exists, but without any header files.) You can get the latest release from the remctl distribution page.

Russ Allbery: rra-c-util 4.9

This is another rollup of various pending changes in my collection of supporting software. The libkafs portability layer now supports k_pioctl somewhat more completely by providing a fallback to lpioctl when using the AFS libraries and a system call failure when no library was found. The libkafs functions are also now properly prototyped if the library was found but no header files were available. The network_connection function, when given a timeout, restarts waiting for a non-blocking connect if interrupted by a signal. That means the timeout could in theory be much longer than intended if it's interrupted by a signal repeatedly, but hopefully that situation will be rare enough to not pose any problems. There are also a few fixes to the networking test suite. The standard Perl tests now check for POD in more places, don't check the Perl syntax of SYNOPSIS for scripts, work with packages without Build.PL or a t directory, and also check style for files under a usr directory. The PCRE Autoconf probe now copes with Mac OS X, which has the library but no headers. Finally, this release incorporates a combination test suite and utility script that checks that the versions of embedded Perl modules stay in sync with the overall package version. You can get the latest release from the rra-c-util distribution page.

14 August 2013

Joey Hess: words fail me







Russ Allbery: C TAP Harness 2.2

I've been incredibly, overwhelmingly busy lately, and I'm still fairly consumed by things other than writing journal posts (or reviews, sigh), but hopefully it's letting up a bit now. At the least, I'm back to writing code instead of doing other things, which is quite a relief. There's nothing particularly exciting in this release of my C test harness software, but since I'm doing some general releases anyway, it seemed like a good time to flush out pending changes. This release changes the exit status of the bail and sysbail functions to 255 to more closely match the BAIL_OUT function in Perl's Test::More, and documents why the runtests test harness does not match Test::More's handling of a bailing test case. You can get the latest version from the C TAP Harness distribution page.

13 August 2013

Kees Cook: TPM providing /dev/hwrng

A while ago, I added support for the TPM s pRNG to the rng-tools package in Ubuntu. Since then, Kent Yoder added TPM support directly into the kernel s /dev/hwrng device. This means there s no need to carry the patch in rng-tools any more, since I can use /dev/hwrng directly now:
# modprobe tpm-rng
# echo tpm-rng >> /etc/modules
# grep -v ^# /etc/default/rng-tools
RNGDOPTIONS="--fill-watermark=90%"
# service rng-tools restart
And as before, once it s been running a while (or you send SIGUSR1 to rngd), you can see reporting in syslog:
# pkill -USR1 rngd
# tail -n 15 /var/log/syslog
Aug 13 09:51:01 linux rngd[39114]: stats: bits received from HRNG source: 260064
Aug 13 09:51:01 linux rngd[39114]: stats: bits sent to kernel pool: 216384
Aug 13 09:51:01 linux rngd[39114]: stats: entropy added to kernel pool: 216384
Aug 13 09:51:01 linux rngd[39114]: stats: FIPS 140-2 successes: 13
Aug 13 09:51:01 linux rngd[39114]: stats: FIPS 140-2 failures: 0
Aug 13 09:51:01 linux rngd[39114]: stats: FIPS 140-2(2001-10-10) Monobit: 0
Aug 13 09:51:01 linux rngd[39114]: stats: FIPS 140-2(2001-10-10) Poker: 0
Aug 13 09:51:01 linux rngd[39114]: stats: FIPS 140-2(2001-10-10) Runs: 0
Aug 13 09:51:01 linux rngd[39114]: stats: FIPS 140-2(2001-10-10) Long run: 0
Aug 13 09:51:01 linux rngd[39114]: stats: FIPS 140-2(2001-10-10) Continuous run: 0
Aug 13 09:51:01 linux rngd[39114]: stats: HRNG source speed: (min=10.433; avg=10.442; max=10.454)Kibits/s
Aug 13 09:51:01 linux rngd[39114]: stats: FIPS tests speed: (min=73.360; avg=75.504; max=86.305)Mibits/s
Aug 13 09:51:01 linux rngd[39114]: stats: Lowest ready-buffers level: 2
Aug 13 09:51:01 linux rngd[39114]: stats: Entropy starvations: 0
Aug 13 09:51:01 linux rngd[39114]: stats: Time spent starving for entropy: (min=0; avg=0.000; max=0)us
I m pondering getting this running in Chrome OS too, but I want to make sure it doesn t suck too much battery.

2013, Kees Cook. This work is licensed under a Creative Commons Attribution-ShareAlike 3.0 License.
Creative Commons License

R&#233;mi Vanicat: Git annex and Calibre

The debconf BoF about gitify all things make we to want to report some of my git-annex use cases. The problem Calibre is a ebook manager that is available in debian. I use it to maintain my library, but also to dowload every day an epub version of a French newspaper and then put it on my kobo. Configuring git annex for this I wanted to use git-annex, so
    $ git init
    $ git annex init "some useful name"
But I don't want every thing in annex, because Calibre use some text file to save some metadata, so I used:
    $ git config annex.largefiles "include=* exclude=*.opf exclude=*.json"
then lets add everything
    $ git annex add *
    $ git add *
    $ git commit -m "first commit"
Calibre need read and write access on the its database, so let unlock it:
    $ git annex unlock metadata.db
On my other computer I only need to do
    $ git clone $user@$host:Calibre\ library
    $ cd Calibre\ library
    $ git annex init "another useful name"
    $ git annex get .
    $ git annex unlock metadata.db
The problem is that every time you will git annex sync, git annex will lock again the metadata.db, so lets unlock it automatically. I use git hooks, in .git/hooks/post-commit I have
    #!/bin/bash
    git annex edit metadata.db
don't forget to make this file executable
    $ chmod a+x .git/hooks/post-commit
Day to day operation
    $ git annex add .
Will put new file into the annex
    $ git add .
Will take care of the files that should no go into annex
    $ git annex sync
Will make the repositories exchange informations about all this, and make remote change local
    $ git annex get .
Will make remote book locally available Merge conflict You should not run calibre on the two computer simultaneously, or without syncing before it. If you do, you will have a conflict that git-annex will automatically solve by rename both of the file. You can then either: Checking the library You can use calibredb check_library to check you library is correct. If you use git for it, it will always tell you that it is not correct: there is this author ".git" it doesn't know about. Just don't care about it. Maybe this can be solved by using vcsh but apparently vcsh+git annex it not well tested yet. Automatic stuff I use mr to automatically run all this, but some config could be done (I believe) to have git annex copy --auto do what it should. There are also the git annex assistant for this kind of automatic synchronizations of contents, but I don't know if my automatic unlocking of one file will break this. It might be interesting to find someway to unlock and lock the library only when running calibre, a simple script to launch calibre will do that. Note that each time you will lock and unlock, you will have a new commit in git.

Next.

Previous.