Search Results: "zed"

5 February 2015

Dirk Eddelbuettel: Introducing drat: Lightweight R Repositories

A new package of mine just got to CRAN in its very first version 0.0.1: drat. Its name stands for drat R Archive Template, and an introduction is provided at the drat page, the GitHub repository, and below. drat builds on a core strength of R: the ability to query multiple repositories. Just how one could always query, say, CRAN, BioConductor and OmegaHat---one can now adds drats of one or more other developers with ease. drat also builds on a core strength of GitHub. Every user automagically has a corresponding github.io address, and by appending drat we are getting a standardized URL. drat combines both strengths. So after an initial install.packages("drat") to get drat, you can just do either one of 
library(drat)
addRepo("eddelbuettel")
or equally 
drat:::add("eddelbuettel")
to register my drat. Now install.packages() will work using this new drat, as will update.packages(). The fact that the update mechanism works is a key strength: not only can you get a package, but you can gets its updates once its author replaces them into his drat. How does one do that? Easy! For a package foo_0.1.0.tar.gz we do 
library(drat)
insertPackage("foo_0.1.0.tar.gz")
The default git repository locally is taken as the default ~/git/drat/ but can be overriden as both a local default (via options()) or directly on the command-line. Note that this also assumes that you a) have a gh-pages branch and b) have that branch as the currently active branch. Automating this / testing for this is left for a subsequent release. Also available is an alternative unexported short-hand function: 
drat:::insert("foo_0.1.0.tar.gz", "/opt/myWork/git")
show here with the alternate use case of a local fileshare you can copy into and query from---something we do at work where we share packages only locally. The easiest way to obtain the corresponding file system layout may be to just fork the drat repository. So that is it. Two exported functions, and two unexported (potentially name-clobbering) shorthands. Now drat away! Courtesy of CRANberries, there is also a copy of the DESCRIPTION file for this initial release. More detailed information is on the drat page.

This post by Dirk Eddelbuettel originated on his Thinking inside the box blog. Please report excessive re-aggregation in third-party for-profit settings.

3 February 2015

Thorsten Alteholz: USB 3.0 hub and Gigabit LAN adapter

Recently I bought an USB 3.0 Hub with three USB 3.0 ports and one Gigabit LAN port. It is manufactured by Delock and I purchased it from Reichelt (Delock 62440). Under Wheezy the USB part is recognized without problems but the kernel (3.2.0-4) does not have a driver for the ethernet part.
The USB Id is idVendor=0b95 and idProduct=1790, the manufacturer is ASIX Elec. Corp. and the product is: AX88179. So Google led me to a product page at Asix, where I could download the driver for kernel 2.6.x and 3.x.

mkdir -p /usr/local/src/asix/ax88179
cd /usr/local/src/asix/ax88179
wget www.asix.com.tw/FrootAttach/driver/AX88179_178A_LINUX_DRIVER_v1.13.0_SOURCE.tar.bz2
tar -jxf AX88179_178A_LINUX_DRIVER_v1.13.0_SOURCE.tar.bz2
cd AX88179_178A_LINUX_DRIVER_v1.13.0_SOURCE
apt-get install module-assistant
module-assistant prepare
make
make install
modprobe ax88179_178a.ko
 After editing /etc/network/interfaces and doing an ifup eth1, voila, I have a new network link. I hope the hardware is as good as the installation has been easy.

31 January 2015

John Goerzen: Home Automation, part 2: Z-Wave and ISY programming

In my part 1 post yesterday, I wrote about the start of the home automation project. I mentioned that I was using Insteon switches, and they mostly were working well (I forgot to mention an annoyance: you can dim, but not totally shut off, their LED status light.) Anyhow, the Insteon battery-operated sensors seem to not be as good as their Z-Wave competition. Setting up Z-Wave Z-Wave devices get joined to the Z-Wave network in much the same way Bluetooth devices get paired with each other. The first time you use a Z-Wave device, you join it to the controller. The controller assigns it an ID on your network, and both devices discover the best route to communicate with each other. I discovered that the Z-Wave module on the ISY-994i has particularly poor reception. Combined with the generally short range of battery-operated Z-wave devices, this meant that my sensors didn t work reliably. As with Insteon, AC-powered Z-wave devices tend to be repeaters, but I didn t have any AC-powered Z-wave devices. I went looking for Z-wave repeaters, and found some. But then I discovered that a Z-wave relay-based appliance switch was actually $5 cheaper, acted as a repeater, and could be used as a switch down the road if needed. A couple of those solved my communications issues. You join them to the network as usual, but then either re-join the battery-powered sensors (so they see the new route) or do a network heal (every device on the network re-learns about its neighbors and routes to the other devices) so they see the new route. Z-Wave Motion / Occupancy Sensors If you have been in new buildings, chances are you have seen light switches with built-in occupancy sensors. My doctor s office has these. They are typically the same size as a regular light switch, but with a passive infrared (PIR) motion sensor used to automatically turn the lights off after a timeout (or even on, when someone walks into the room.) These work fine for small rooms, but if you ve ever been in a bathroom with a PIR lightswitch that goes off while you re still in the room, you are aware of their faults for larger rooms! Most of the time, when you read about occupancy sensors , it s really a PIR sensor, and the term is used interchangeably with motion detector. Motion detectors in home automation systems are commonly used for a number of purposes: detecting occuption of rooms for automatic control of lighting or HVAC systems, triggering alerts, or even locking or unlocking doors. My friend told me he had poor luck with the Insteon PIR sensors, so I tried two different Z-Wave models: the $48 Aeon/Aeotec multi-sensor and the $30 Ecolink PIRZWAVE2-ECO. The Aeon device is clearly the more capable; it can be used indoors or out, and has a lot of options that can be configured by Z-Wave configuration parameters. It also has a ball/socket mount, so it can be easily aimed in different directions. It can draw power from 4 AAA batteries, or a mini USB cable (cable, but not power supply, included). The Aeon multi-sensor also has a temperature, humidity, and illumination (lux) sensor but, as you ll see, they have some drawbacks. The Ecolink device is more basic; it has a few settings that can be altered via jumpers, but none that can be altered via Z-Wave configuration commands. That makes it a bit of a hassle, since you have to open it up to change, and when you do, it triggers a tamper alarm that is both undocumented and never seems to go away. It is powered by a single CR123A 3V Lithium battery, which are about $2 each on Amazon. Both units have a default PIR timeout of 4 minutes. That means that after sensing motion, they will transmit the on signal (meaning motion detected) and then transmit no further signals for at least 4 minutes. This is because operating the radio consumes far more battery power than simply monitoring the PIR sensor, and it cuts down on repeated on/off transmissions. In this configuration, both units should have battery life of 6 months to a year, I figure, with an edge for the Ecolink, perhaps. Both can also be configured to transmit more frequently; the Ecolink has a jumper that can change its timeout to 10 seconds, whereas the Aeon can be configured over Z-Wave for any timeout between 10 seconds and 65535 seconds (values above 4 minutes are rounded to the nearest minute, for some reason.) Both also have adjustable sensitivity; on the Aeon this is via an adjustment knob, and on the Ecolink it s via jumpers. And both can report their battery level to enable software alerts when it s getting low. The Aeon ships with its temperature, humidity, battery, and lux sensors disabled. This appears to be the cause of much confusion online, as they send one transmission and then no more. Sadly, one has to resort to the hard-to-find but very helpful MultiSensor engineering spec document to figure out the way to enable those sensors and set their interval. (It must be said, however, that I doubt most consumers will understand how to set bitfields, and this is either not covered or covered incorrectly in their other manuals.) This can be a real power drain, so I just set it to report battery level every 6 hours (so I can alert myself if it gets low) and leave it at that. The Ecolink manual claims a detection radius of 39 feet and a total angle of 90deg (45deg left or right from center) and a 3-year battery life (I m skeptical). It also is rated for indoor use only. Aeotec claims a detection radius of 16 feet and a total angle of 120 deg. Be skeptical of all these figures. Once set up with good reception, both devices have been working fine. Programming the controller The ISY-994i-Zw Pro controller I mentioned in Part 1 has its own sort of programming language. It s a lot better than the sort of GUI clicky mess that is found in most of these things, but it still has a sort of annoying Java-based editor where you select keywords from a menu and such. Although you can backup and restore the device, and import and export the programs, the file formats are XML and not really suitable for hand-editing. Sigh. The language is limited, but gets the job done. Here is a simple program that turns off the fan in a bathroom 30 minutes after the light was turned off: 
If
        Status  '1st Floor / Bathroom + Laundry / Main Bath Fan' is On
    And Status  '1st Floor / Bathroom + Laundry / Main Bath Light' is Off
 
Then
        Wait  30 minutes 
        Set '1st Floor / Bathroom + Laundry / Main Bath Fan' Off
 
Else
   - No Actions - (To add one, press 'Action')
The if/then/else clauses should probably be more properly called when/do/finally. The if clauses are evaluated whenever a relevant event occurs. If the If evaluates to true, if executes the Then section. The program in Then is uninterruptible except for wait and repeat statements. So in this case, the program begins, and if the light stays off and the fan stays on, 30 minutes later it turns off the fan. But if the light comes back on, the program aborts (since there is nothing in else ). Similarly, if the fan goes off, the program aborts. A more complicated example: motion-activated lamp There is a table lamp in our living room controlled by Insteon. By adding a motion sensor in that room, it can be automatically turned on by someone walking through the room. Now, to be useful, I don t want it to turn on during the day. I also don t want it to turn on or adjust itself if other lights are on in the room, or if I turned it on myself; that could cause it to go on and off while I m watching TV, for instance. It s just to be helpful at night. Because the ISY-994i is pretty limited, having almost no control flow operations, this as with many tasks requires several programs . First, here s my main : 
If
        Status  '1st Floor / Living + Dining Room / LR Table Lamp' is Off
    And $LR_Lamp_Lockout is 0
    And Status  '1st Floor / Living + Dining Room / ZW 005 Binary Sensor' is On
    And From    Sunset 
        To      Sunrise (next day)
    And $LR_Lamp_Motion_Working is not 0
    And Status  '1st Floor / Living + Dining Room / LR Floor Lamp' is Off
    And Status  '1st Floor / Living + Dining Room / Dining Light' is Off
    And Status  '1st Floor / Living + Dining Room / LR Light' is Off
 
Then
        Set '1st Floor / Living + Dining Room / LR Table Lamp' 22%
        Set '1st Floor / Living + Dining Room / LR S Remote (Table Lamp)' 22%
 
Else
   - No Actions - (To add one, press 'Action')
So, let s look at the conditions. This program triggers if the lamp is off, the sensor is on, the time is between sunset and sunrise, and three other lights are off. (I will explain the lockout and motion_working variables later). If this is the case, it sets the lamp to 22% (and also informs the wall switch for it that the lamp is at 22%). I pick this precise value because it is unlikely I would manually set it via the wall switch, and therefore lamp is 22% bright doubles as a lamp was turned on by this program flag. So this is the turning the lamp on bit. Let s look at the program that turns it back off later: 
If
        Status  '1st Floor / Living + Dining Room / LR Table Lamp' is 22%
    And (
             Status  '1st Floor / Living + Dining Room / ZW 005 Binary Sensor' is Off
          Or $LR_Lamp_Motion_Working is 0
          Or Status  '1st Floor / Living + Dining Room / LR Light' is not Off
          Or Status  '1st Floor / Living + Dining Room / LR Floor Lamp' is not Off
          Or Status  '1st Floor / Living + Dining Room / Dining Light' is not Off
        )
 
Then
        Wait  15 seconds
        Set '1st Floor / Living + Dining Room / LR S Remote (Table Lamp)' Off
        Set '1st Floor / Living + Dining Room / LR Table Lamp' Off
 
Else
   - No Actions - (To add one, press 'Action')
This is using a sensor with a 4-minute timeout, so the lamp will always be on for at least 4 minutes and 15 seconds. This program runs if the lamp is still at the program-set level (so if, for instance, I turned the lamp full on, the program does nothing to override my setting.) Then, it looks for a condition to trigger turning the lamp off, which could be any of the sensor indicating no more motion, another program detecting it s lost communication with the sensor, or somebody turning on one of the bigger lights in the room. Then it simply sets the light (and the wall switch controlling it) to off. There are a couple more bits to this puzzle. What if the system turned the lamp on, but I really want it off? If I walked up to the wall switch and pushed off , the lamp would go off. And then, a couple seconds later, come back on, since the state of the system met the conditions for the lamp-on program. So we need a lockout that prevents this from happening. Here s my trigger lockout program: 
If
        Control '1st Floor / Living + Dining Room / LR Table Lamp' is switched Off
     Or Control '1st Floor / Living + Dining Room / LR Table Lamp' is switched Fast Off
     Or Control '1st Floor / Living + Dining Room / LR S Remote (Table Lamp)' is switched Fast Off
     Or Control '1st Floor / Living + Dining Room / LR S Remote (Table Lamp)' is switched Off
 
Then
        $LR_Lamp_Lockout += 1
        Run Program 'LR Lamp Clear Lockout' (If)
 
Else
   - No Actions - (To add one, press 'Action')
So if someone pushes the off button at the lamp switch box at the outlet it s plugged into (unlikely), or at the wall switch, it increments the lockout variable and runs another program. This other program is unique in that it is flagged disabled , meaning it is never run automatically by the system, only when called by another program. Here s the clear lockout program: 
If
        $LR_Lamp_Lockout > 0
 
Then
        Wait  5 minutes 
        $LR_Lamp_Lockout  = 0
 
Else
   - No Actions - (To add one, press 'Action')
Thus by pushing the off button on the switch, the motion-triggered program won t turn the lamp back on for at least 5 minutes. Before I had reliable Z-Wave communication to the device, I had some times where it would simply drop off the Z-Wave network until a reboot. This was particularly annoying if it occured after having detected motion, since the state of the sensor in the ISY controller is simply whatever state it last received. Therefore, I wrote this program to check if it believes the motion sensor is working: 
If
        Status  '1st Floor / Living + Dining Room / ZW 005 Binary Sensor' is On
 
Then
        $LR_Lamp_Motion_Working  = 1
        Wait  1 hour 
        $LR_Lamp_Motion_Working  = 0
 
Else
        $LR_Lamp_Motion_Working  = 1
We don t really care if the motion sensor is broken when the status is off; all that happens then is no lamp turns on. So this program activates when the status is set to on. It flips the working flag to true, then waits for an hour. If the sensor shows no motion within that hour, the program skips to the else (keeping the flag true). But if it is still on after an hour, it decides it must not be working and sets the working flag to false. You can see that flag used in the other programs logic. But because of the Else, which is run whenever the conditions that caused the Then clause to run become false, as soon as the system receives no motion , it will flag the sensor as working again. The final piece to this puzzle is a program flagged to run at boot time of the controller: 
If
   - No Conditions - (To add one, press 'Schedule' or 'Condition')
 
Then
        $LR_Lamp_Motion_Working  = 1
 
Else
   - No Actions - (To add one, press 'Action')
This simply initializes the motion working variable to a known state. Keypads The Insteon KeypadLinc is a nice device. It can control a single load directly, but all 8 buttons are fully responsive in the Insteon system. They each also have individually-addressible LED backlights. They are commonly used to do things like ALL OFF , TV (to set lights for watching TV), AWAY (to set the system for everyone being out of the house for awhile), etc. They are the size of a regular Decora switch, and I have installed one already, but haven t programmed much of it yet. REST API The ISY has an extensive REST API, which I ve used to integrate it a bit with my Debian systems. More on that another time. Mobile app Mobile apps are a common thing people look for in these systems. You can t use the Insteon app with the ISY, but they recommend Mobilinc Pro. It does the job. Mobilinc tries to sell a $10/mo connection service, which is totally unnecessary if you can figure out SSL, and has on-screen buttons to bypass, but judging by the Google Play reviews, a lot of people thought they had to pay for that and uninstalled it afterwards. Future directions Many people put in electronic door locks. I don t plan to do that. I do plan to have the house systems be aware of the general state of things (is the house empty? is everyone asleep?) and do appropriate things with lighting and HVAC. I don t really expect the savings in power for lighting control to pay for the system anytime soon. However, if it can achieve some savings in heating and cooling, it may well be able to pay for itself in a few years. So my big next step is thermostats that can integrate with all this. I have had a water mess in my basement before, and water leak sensors are a very common item people deploy in these setups. I certainly plan to add a few of them. Door-open sensors are also useful; they trigger more instantly and reliably than motion detectors and can be used in some nice ways (is it after dark and the door is opening when the house is vacant? If so, turn on the light nearby in case their hands are full.) Issues Some issues I ran into so far are already discussed above. One other major one involves SSL on the ISY-994i Pro. The method for adding SSL keys is cumbersome, but the processor on the device which appears to run some sort of Java is just not up to working with SSL. Apparently they only recently got it fast enough to work with 2048-bit keys. This is rather undocumented, though, so I obtained a cert for a 4096-bit key, my usual. Attempting to connect to the box with SSL appeared to hang not just that but confuse a lot of other things on it as well. Turned out it wasn t hung; it just too a minute and 45 seconds to complete the SSL handshake. Moral of the story: use 2048-bit keys, or stunnel4 or some such to re-wrap the SSL communications with a stronger key. The KeypadLinc backlights can be completely shut off, and both their on and off levels can be customized. I have it set to shut off the backlight during the day and turn it on at night. The wall switches, however, can t have their brightness status LED bar entirely shut off. They can be made dim, but don t ever go away. That s rather annoying. Also annoying is that Insteon doesn t make switches in the traditional toggle switch style in colors other than white. As our house had mostly black switches, I was forced into the Decora style. Overall thoughts This has been a great learning experience for me in a number of ways. I have only begun to tap what the system can do, and the real benefits will probably come once I get the heating and cooling into the mix. It s quite a nice way for a geek to go, and the improvements in lighting have also been popular with everyone else in the house.

John Goerzen: First Steps with Home Automation and LED Lighting

I ve been thinking about home automation automating lights, switches, thermostats, etc. for years. Literally decades, in fact. When I was a child, my parents had a RadioShack X10 control module and one or two target devices. I think I had fun giving people a light show turning on or off one switch and one outlet remotely. But I was stuck there are a daunting number of standards for home automation these days. Zigbee, UPB, Z-Wave, Insteon, and all sorts of Wifi-enabled things that aren t really compatible with each other (hellooooo, Nest) or have their own ecosystem that isn t all that open (helloooo, Apple). Frankly I don t think that Wifi is a great home automation protocol; its power drain completely prohibits it being used in a lot of ways. Earlier this month, my awesome employer had our annual meeting and as part of that our technical teams had some time for anyone to talk about anything geeky. I used my time to talk about flying quadcopters, but two of my colleagues talked about home automation. I had enough to have a place to start, and was hooked. People use these systems to do all sorts of things: intelligently turn off lights when rooms aren t occupied, provide electronic door locks (unlockable via keypad, remote, or software), remote control lighting and heating/cooling via smartphone apps, detect water leakage, control switches with awkward wiring environments, buttons to instantly set multiple switches to certain levels for TV watching, turning off lights left on, etc. I even heard examples of monitoring a swamp cooler to make sure it is being used correctly. The possibilities are endless, and my geeky side was intrigued. Insteon and Z-Wave Based on what I heard from my colleagues, I decided to adopt a hybrid network consisting of Insteon and Z-Wave. Both are reliable protocols (with ACKs and retransmit), so they work far better than X10 did. Both have all sorts of controls and sensors available (browse around on smarthome.com for some ideas). Insteon is a particularly interesting system an integrated dual-mesh network. It has both powerline and RF signaling, and most hardwared Insteon devices act as repeaters for both the wired and RF network simultaneously. Insteon packets contain a maximum hop count that is decremented after each relay, and the packets repeat in such as way that they collide and strengthen one another. There is no need to maintain routing tables or anything like that; it simply scales nicely. This system addresses all sorts of potential complexities. It addresses the split-phase problem of powerline-only systems, but using an RF bridge. It addresses long distances and outbuildings by using the powerline signaling. I found it to work quite well. The downside to Insteon is that all the equipment comes from one vendor: Insteon. If you don t like their thermostat or motion sensor, you don t have any choice. Insteon devices can be used entirely without a central controller. Light switches can talk to each other directly, and you can even set them up so that one switch controls dozens of others, if you have enough patience to go around your house pressing tiny set buttons. Enter Z-Wave. Z-Wave is RF-only, and while it is also a mesh network, it is source-routed, meaning that if you move devices around, you have to heal your network as all your nodes have to re-learn the path to each other. It also doesn t have the easy distance traversal of Insteon, of course. On the other hand, hundreds of vendors make Z-Wave products, and they mostly interoperate well. Z-Wave is said to scale practically to maybe two or three dozen devices, which would have been an issue for me, buut with Insteon doing the heavy lifting and Z-Wave filling in the gaps, it s worked out well. Controlling it all While both Insteon (and, to a certain extent, Z-Wave) devices can control each other, to really spread your wings, you need more centralized control. This lets you have programs that do things like if there s motion in the room on a weekday and it s dark outside, then turn on a light, and turn it back off 5 minutes later. Insteon has several options. One, you can buy their power line modem (PLM). This can be hooked up to a PC to run either Insteon s proprietary software, or something open-source like MisterHouse, written in Perl. Or you can hook it up to a controller I ll mention in a minute. Those looking for a fairly simpe controller might get the Insteon 2242-222 Hub, which has the obligatory smartphone app and basic schedules. For more sophisticated control, my friend recommended the ISY-994i controller. Not only does it have a lot more capable programming language (though still frustrating), it supports both Insteon and Z-Wave in an integrated box, and has a comprehensive REST API for integrating with other things. I went this route. First step: LED lighting I began my project by replacing my light bulbs with LEDs. I found that I could get Cree 4-Flow 60W equivs for $10 at Home Depot. They are dimmable, a key advantage over CFL, and also maintain their brightness throughout their life far better. As I wanted to install dimmer switches, I got a combination of Cree 60W bulbs, Cree TW bulbs (which have a better color spectrum coverage for more true colors), and Cree 100W equiv bulbs for places I needed more coverage. I even put in a few LED flood lights to replace my can lights. Overall I was happy with the LEDs. They are a significant improvement over the CFLs I had been using, and use even less power to boot. I have had issues with three Cree bulbs, though: one arrived broken, and two others have had issues such as being quite dim. They have a good warranty, but it seems their QA could be better. Also, they can have a tendency to flickr when dimmed, though this plagues virtually all LED bulbs. I had done quite a bit of research. CNET has some helpful brightness guides, and Insteon has a color temperature chart. CNET also had a nifty in-depth test of LED bulbs. Second step: switches Once the LED bulbs were in place, I was then able to start installing smart switches. I picked up Insteon s basic switch, the SwitchLinc 2477D at Menard s. This is a dimmable switch and requires a neutral wire in the box, but acts as a dual-band repeater for the system as well. The way Insteon switches work, they can be standalone, or controllers, responders, or both in a scene . A scene is where multiple devices act together. You can create virtual 3-way switches in a scene, or more complicated things where different lights are turned on at different levels. Anyhow, these switches worked out quite well. I have a few boxes where there is no neutral wire, so I had to use the Insteon SwitchLinc 2474D in them. That switch is RF-only and is supposed to have a minimum load of 20W, though they seemed to work OK albeit with limited range and the occasional glitch with my LEDs. There is also the relay-based SwitchLinc 2477S for use with non-dimmable lights, fans, etc. You can also get plug-in modules for controlling lamps and such. I found the Insteon devices mostly lived up to their billing. Occasionally I could provoke a glitch by changing from dimming to brightening in rapid succession on a remote switch controlling a load on a distant one. Twice I had to power cycle an Insteon switch that got confused (rather annoying when they re hardwared). Other than that, though, it s been solid. From what I gather, this stuff isn t ever quite as reliable as a 1950s mechanical switch, but at least in this price range, this is about as good as it gets these days. Well, this post got quite long, so I will have to follow up with part 2 in a little while. I intend to write about sensors and the Z-Wave network (which didn t work quite as easily as Insteon), as well as programming the ISY and my lessons learned along the way.

30 January 2015

Dirk Eddelbuettel: littler 0.2.2

max-heap image A new minor release of littler is available now. Several examples were added or extended: See the littler examples page for more details. Another useful change is that r now reads either one (or both) of /etc/littler.r and ~/.littler.r. These are interpreted as standard R files, allowing users to provide initialization, package loading and more. Carl Boettiger and I continue to make good use of these littler examples (to to install directly from CRAN or GitHub, or to run checks) in our Rocker builds of R for Docker. Full details for the littler release are provided as usual at the ChangeLog page. The code is available via the GitHub repo, from tarballs off my littler page and the local directory here. A fresh package has gone to the incoming queue at Debian; Michael Rutter will probably have new Ubuntu binaries at CRAN in a few days too. Comments and suggestions are welcome via the mailing list or issue tracker at the GitHub repo.

This post by Dirk Eddelbuettel originated on his Thinking inside the box blog. Please report excessive re-aggregation in third-party for-profit settings.

Laura Arjona: Going selfhosting: Installing Debian Wheezy in my home server

It was in my mind to open a new series of articles with topic selfhosting , because I really believe in free software based network services and since long time I want to plug a machine 24 7 at home to host my blog, microblog, MediaGoblin, XMPP server, mail, and, in conclusion, all the services that now I trust to very kind third parties that run them with free software, but I know I could run myself (and offer them to my family and friends). Last September I bought the domain larjona.net (curious, they say buy but it s a rent, for 1,2,3 years never yours. Pending another post about my adventures with the domain name, dynamic DNS, and SSL certs!) and I bought an HP Microserver G7 N54L, with 2 GB RAM. It had a 250GB SATA harddisk and I bought 2 more SATA harddisks, 1 TB each, to setup a RAID 1 (mirror). Total cost (with keyboard and mouse), 300 . A friend gave me a TFT monitor that was too old for him (1024 768) but it serves me well, (it s a server, no graphical interface, and I will connect remotely most of the times). Installing Debian stable (wheezy) I decided to install Debian stable. Jessie was not frozen yet, and since it was my first non-LAMP server install, I wanted to make sure that errors and problems would be my errors, not issues of the non-released-yet distro. I thought to install YunoHost or some other distro prepared for selfhosting, but I ve never tried them, and I have not much free time, so I decided to stick on Debian, my beloved distro, because it s the one that I know best and I m part of its awesome community. And maybe I could contribute back some bug reports or documentation. I wanted to try a crypto setup (just for fun, just for learn, for its benefits, and to be one more freecrypto-tester in the world) so after reading a bit: https://wiki.debian.org/DebianInstaller/SataRaid
https://wiki.archlinux.org/index.php/disk_encryption
http://madduck.net/docs/cryptdisk/
http://linuxgazette.net/140/kapil.html
http://smcv.pseudorandom.co.uk/2008/09/cryptroot/
http://www.linuxquestions.org/questions/linux-security-4/lvm-before-and-after-encryption-871379/ and some other pages, and try some different things, this is the setup that I managed to configure: Everything went well. Yay! Some doubts and one problem Everything went quite well except some doubts: After talking about this issues with friends (and in debian-women IRC channel), I decided to install the non-free driver, just in case, with the same reasoning as with the RAID: let the card do the job, so the CPU can care about other things. Again, I notice that learning a bit about benchmarking (and having some time to do some tests) would be nice And now, the problem: I left this problem apart and go on installing the software. I would think later what to do. Installing MediaGoblin The most urgent selfhosting service, for me, was GNU MediaGoblin, because I wanted to show my server to my family in Christmas, and upload the pictures of the babies and kids of the family. And it s a project where I contribute translations and I am a big fan, so I would be very proud of hosting my own instance. I followed the documentation to setup 2 instances of GNU MediaGoblin 0.7 (the stable release in the moment), with their corresponding PostgreSQL databases. Why two instances? Well, I want an instance to host and show my videos, images, and replicate videos that I like, and a private one for sharing photos and videos with my family. MediaGoblin has no privacy settings yet, so I installed separate instances, and the private one I put it in a different port, with a self-signed SSL cert, and enabled http-authorization in Nginx, so only authorized Linux users of my machine can accesss the website. Installing MediaGoblin was easier than what I thought. I only had some small doubts about the documentation, and they were solved in the IRC channel. You can access, for example, my user profile in my public instance, and see some different files that I already uploaded. I m very happy!! Face to face with the bug, again I had to solve the problem of the password not accepted in reboots. I began to think that it could be a bug in cryptsetup. Should I upgrade the package to the version in wheezy-backports? Jessie was almost frozen, maybe it was time to upgrade the whole system, to see if the problem was solved (and to see how my MediaGoblin was working or not in Jessie. It should work, it s almost packaged! But who knows). And if it didn t work, maybe it was time to file a bug So I upgraded my system to Debian Jessie. And after upgrade, the system didn t boot. But that s the story of another blog post (that I still need to finish to write don t worry, it has happy end, as you could see accessing my Mediagoblin site!). Comments? You can comment in this pump.io thread.
Filed under: My experiences and opinion Tagged: Debian, encryption, English, libre software, MediaGoblin, Moving into free software, N54L, selfhosting, sysadmin

27 January 2015

Laura Arjona: Upgrading my computers to Debian Jessie: Husband s laptop (Acer Aspire 5250)

This is an old laptop, with AMD E-300 processor, 6 GB RAM, Radeon HD 6310 VGA and Atheros AR9485 wireless network adapter. It was running Windows 7 (preinstalled). The hard disk failed, and I put the hard disk of another laptop (a broken Acer Aspire One D255) on it. Surprisingly, the Windows 7 on it booted (after some self-configuration that took quite long), but it was a Windows 7 Home 32bits, so it was only recognizing 4 GB RAM. That was the perfect excuse to convince my husband to install Debian in the laptop and begin his transition to a free OS. Yay! I installed Debian Jessie from scratch last summer. Everything went well (the installer went fine, 8 months before than its RC1 release, congrats Debian-boot team!). I needed the non-free radeon driver for the graphical display :/ Jessie is running GNOME3 desktop, and I ve been seeing all these months the transition to 3.14 version, and later, the integration of the Lines theme (by Juliette Belin), which I like very much. I have problems to watch high quality videos, in every player that I tried (VLC, Totem, mplayer) the audio and video are not synced, and video sometimes freezes. I m almost sure that the problem is what mplayer says: Your system is too SLOW to play this! . I tried to install the ATI non-free driver for better performance, but after successfully install it and reboot, GNOME was not starting (I got a black screen, no gdm greeting me). I could log in tty2, though. I don t know if I did something wrong, how to solve the problem, and I don t wanted to waste time, so I uninstalled it and returned to the non-free firmware that goes to the Linux kernel. For now, when I need to watch a video that gives those problems, I upload the file to my GNU MediaGoblin site, or use WinFF to reduce size/quality. Overall impression Fine! Both my husband and me are very happy. The installation went really well. I m not a GNOME expert user but I find it easy, intuitive, and he found it easy too. My husband uses the computer to surf the web, watch some videos and online series (we had to install non-free flash plugin from Adobe #grr), read mail from the browser, write something in LibreOffice and print it (hey! we just plug the printer/scanner and it works, no need to install drivers!), scan some image and send it by email I set Debian as default in GRUB, and the switch from Windows has been very natural for him (he was already using Firefox and LibreOffice in Windows. He still says I m a Windows user although he is just using Debian for months!). He bought an IPhone 4S (#grr!) and I tried to connect it as shown in the corresponding Debian wiki page, but it didn t work (I got segmentation fault when connecting the phone). However, it is recognized by Shotwell and we can copy all the photos and videos to the computer, which is what we wanted to do. So no problem on that side, either. In conclussion, one more computer at home running Debian ( future stable ), and we don t run Windows at home anymore :)
Filed under: My experiences and opinion Tagged: Debian, English, Moving into free software

Matthias Klumpp: AppStream 0.8 released!

Yesterday I released version 0.8 of AppStream, the cross-distribution standard for software metadata, that is currently used by GNOME-Software, Muon and Apper in to display rich metadata about applications and other software components. What s new? The new release contains some tweaks on AppStreams documentation, and extends the specification with a few more tags and refinements. For example we now recommend sizes for screenshots. The recommended sizes are the ones GNOME-Software already uses today, and it is a good idea to ship those to make software-centers look great, as others SCs are planning to use them as well. Normal sizes as well as sizes for HiDPI displays are defined. This change affects only the distribution-generated data, the upstream metadata is unaffected by this (the distro-specific metadata generator will resize the screenshots anyway). Another addition to the spec is the introduction of an optional <source_pkgname/> tag, which holds the source package name the packages defined in <pkgname/> tags are built from. This is mainly for internal use by the distributor, e.g. it can decide to use this information to link to internal resources (like bugtrackers, package-watch etc.). It may also be used by software-center applications as additional information to group software components. Furthermore, we introduced a <bundle/> tag for future use with 3rd-party application installation solutions. The tag notifies a software-installer about the presence of a 3rd-party application bundle, and provides the necessary information on how to install it. In order to do that, the software-center needs to support the respective installation solution. Currently, the Limba project and Xdg-App bundles are supported. For software managers, it is a good idea to implement support for 3rd-party app installers, as soon as the solutions are ready. Currently, the projects are worked on heavily. The new tag is currently already used by Limba, which is the reason why it depends on the latest AppStream release. How do I get it? All AppStream libraries, libappstream, libappstream-qt and libappstream-glib, are supporting the 0.8 specification in their latest version so in case you are using one of these, you don t need to do anything. For Debian, the DEP-11 spec is being updated at time, and the changes will land in the DEP-11 tools soon. Improve your metadata! This call goes especilly to many KDE projects! Getting good data is partly a task for the distributor, since packaging issues can result in incorrect or broken data, screenshots need to be properly resized etc. However, flawed upstream data can also prevent software from being shown, since software with broken data or missing data will not be incorporated in the distro XML AppStream data file. Richard Hughes of Fedora has created a nice overview of software failing to be included. You can see the failed-list here the data can be filtered by desktop environment etc. For KDE projects, a Comment= field is often missing in their .desktop files (or a <summary/> tag needs to be added to their AppStream upstream XML file). Keep in mind that you are not only helping Fedora by fixing these issues, but also all other distributions cosuming the metadata you ship upstream. For Debian, we will have a similar overview soon, since it is also a very helpful tool to find packaging issues. If you want to get more information on how to improve your upstream metadata, and how new metadata should look like, take a look at the quickstart guide in the AppStream documentation.

25 January 2015

Dirk Eddelbuettel: RcppArmadillo 0.4.600.4.0

Conrad put up a maintenance release 4.600.4 of Armadillo a few days ago. As in the past, we tested this with number of pre-releases and test builds against the now over one hundred CRAN dependents of our RcppArmadillo package. The tests passed fine as usual, and results are as always in the rcpp-logs repository. Changes are summarized below based on the NEWS.Rd file.
Changes in RcppArmadillo version 0.4.600.4.0 (2015-01-23)
  • Upgraded to Armadillo release Version 4.600.4 (still "Off The Reservation")
    • Speedups in the transpose operation
    • Small bug fixes
Courtesy of CRANberries, there is also a diffstat report for the most recent release. As always, more detailed information is on the RcppArmadillo page. Questions, comments etc should go to the rcpp-devel mailing list off the R-Forge page.

This post by Dirk Eddelbuettel originated on his Thinking inside the box blog. Please report excessive re-aggregation in third-party for-profit settings.

Joey Hess: making propellor safer with GADTs and type families

Since July, I have been aware of an ugly problem with propellor. Certain propellor configurations could have a bug. I've tried to solve the problem at least a half-dozen times without success; it's eaten several weekends. Today I finally managed to fix propellor so it's impossible to write code that has the bug, bending the Haskell type checker to my will with the power of GADTs and type-level functions. the bug Code with the bug looked innocuous enough. Something like this: 
foo :: Property
foo = property "foo" $
    unlessM (liftIO $ doesFileExist "/etc/foo") $ do
        bar <- liftIO $ readFile "/etc/foo.template"
        ensureProperty $ setupFoo bar
The problem comes about because some properties in propellor have Info associated with them. This is used by propellor to introspect over the properties of a host, and do things like set up DNS, or decrypt private data used by the property. At the same time, it's useful to let a Property internally decide to run some other Property. In the example above, that's the ensureProperty line, and the setupFoo Property is run only sometimes, and is passed data that is read from the filesystem. This makes it very hard, indeed probably impossible for Propellor to look inside the monad, realize that setupFoo is being used, and add its Info to the host. Probably, setupFoo doesn't have Info associated with it -- most properties do not. But, it's hard to tell, when writing such a Property if it's safe to use ensureProperty. And worse, setupFoo could later be changed to have Info. Now, in most languages, once this problem was noticed, the solution would probably be to make ensureProperty notice when it's called on a Property that has Info, and print a warning message. That's Good Enough in a sense. But it also really stinks as a solution. It means that building propellor isn't good enough to know you have a working system; you have to let it run on each host, and watch out for warnings. Ugh, no! the solution This screams for GADTs. (Well, it did once I learned how what GADTs are and what they can do.) With GADTs, Property NoInfo and Property HasInfo can be separate data types. Most functions will work on either type (Property i) but ensureProperty can be limited to only accept a Property NoInfo. 
data Property i where
    IProperty :: Desc -> ... -> Info -> Property HasInfo
    SProperty :: Desc -> ... -> Property NoInfo
data HasInfo
data NoInfo
ensureProperty :: Property NoInfo -> Propellor Result
Then the type checker can detect the bug, and refuse to compile it. Yay! Except ... Property combinators There are a lot of Property combinators in propellor. These combine two or more properties in various ways. The most basic one is requires, which only runs the first Property after the second one has successfully been met. So, what's it's type when used with GADT Property? 
requires :: Property i1 -> Property i2 -> Property ???
It seemed I needed some kind of type class, to vary the return type. 
class Combine x y r where
    requires :: x -> y -> r
Now I was able to write 4 instances of Combines, for each combination of 2 Properties with HasInfo or NoInfo. It type checked. But, type inference was busted. A simple expression like 
foo  requires  bar
blew up: 
   No instance for (Requires (Property HasInfo) (Property HasInfo) r0)
      arising from a use of  requires'
    The type variable  r0' is ambiguous
    Possible fix: add a type signature that fixes these type variable(s)
    Note: there is a potential instance available:
      instance Requires
                 (Property HasInfo) (Property HasInfo) (Property HasInfo)
        -- Defined at Propellor/Types.hs:167:10
To avoid that, it needed ":: Property HasInfo" appended -- I didn't want the user to need to write that. I got stuck here for an long time, well over a month. type level programming Finally today I realized that I could fix this with a little type-level programming. 
class Combine x y where
    requires :: x -> y -> CombinedType x y
Here CombinedType is a type-level function, that calculates the type that should be used for a combination of types x and y. This turns out to be really easy to do, once you get your head around type level functions. 
type family CInfo x y
type instance CInfo HasInfo HasInfo = HasInfo
type instance CInfo HasInfo NoInfo = HasInfo
type instance CInfo NoInfo HasInfo = HasInfo
type instance CInfo NoInfo NoInfo = NoInfo
type family CombinedType x y
type instance CombinedType (Property x) (Property y) = Property (CInfo x y)
And, with that change, type inference worked again! \o/ (Bonus: I added some more intances of CombinedType for combining things like RevertableProperties, so propellor's property combinators got more powerful too.) Then I just had to make a massive pass over all of Propellor, fixing the types of each Property to be Property NoInfo or Property HasInfo. I frequently picked the wrong one, but the type checker was able to detect and tell me when I did. A few of the type signatures got slightly complicated, to provide the type checker with sufficient proof to do its thing... 
before :: (IsProp x, Combines y x, IsProp (CombinedType y x)) => x -> y -> CombinedType y x
before x y = (y  requires  x)  describe  (propertyDesc x)
onChange
    :: (Combines (Property x) (Property y))
    => Property x
    => Property y
    => CombinedType (Property x) (Property y)
onChange = -- 6 lines of code omitted
fallback :: (Combines (Property p1) (Property p2)) => Property p1 -> Property p2 -> Property (CInfo p1 p2)
fallback = -- 4 lines of code omitted
.. This mostly happened in property combinators, which is an acceptable tradeoff, when you consider that the type checker is now being used to prove that propellor can't have this bug. Mostly, things went just fine. The only other annoying thing was that some things use a [Property], and since a haskell list can only contain a single type, while Property Info and Property NoInfo are two different types, that needed to be dealt with. Happily, I was able to extend propellor's existing (&) and (!) operators to work in this situation, so a list can be constructed of properties of several different types: 
propertyList "foos" $ props
    & foo
    & foobar
    ! oldfoo
conclusion The resulting 4000 lines of changes will be in the next release of propellor. Just as soon as I test that it always generates the same Info as before, and perhaps works when I run it. (eep) These uses of GADTs and type families are not new; this is merely the first time I used them. It's another Haskell leveling up for me. Anytime you can identify a class of bugs that can impact a complicated code base, and rework the code base to completely avoid that class of bugs, is a time to celebrate!

23 January 2015

Jaldhar Vyas: Mini-Debconf Mumbai 2015

Last weekend I went to Mumbai to attend the Mini-Debconf held at IIT-Bombay. These are my impressions of the trip. Arrival and Impressions of Mumbai Getting there was a quite an adventure in itself. Unlike during my ill-fated attempt to visit a Debian event in Kerala last year when a bureaucratic snafu left me unable to get a visa, the organizers started the process much earlier at their end this time and with proper permissions. Yet in India, the wheels only turn as fast as they want to turn so despite their efforts, it was only literally at the last minute that I actually managed to secure my visa. I should note however that Indian government has done a lot to improve the process compared to the hell I remember from, say, a decade ago. It's fairly straightforward for tourist visas now and I trust they will get around to doing the same for conference visas in the fullness of time. I didn't want to commit to buying a plane ticket until I had the visa so I became concerned that the only flights left would be either really expensive or on the type of airline that flies you over Syria or under the Indian Ocean. I lucked out and got a good price on a Swiss Air flight, not non-stop but you can't have everything. So Thursday afternoon I set off for JFK. With only one small suitcase getting there by subway was no problem and I arrived and checked in with plenty of time. Even TSA passed me through with only a minimal amount of indignity. The first leg of my journey took me to Zurich in about eight hours. We were only in Zurich for an hour and then (by now Friday) it was another 9 hours to Mumbai. Friday was Safala Ekadashi but owing to the necessity of staying hydrated on a long flight I drank a lot of water and ate some fruit which I don't normally do on a fasting day. It was tolerable but not too pleasant; I definitely want to try and make travel plans to avoid such situations in the future. Friday evening local time I got to Mumbai. Chhattrapati Shivaji airport has improved a lot since I saw t last and now has all the amenities an international traveller needs including unrestricted free wifi (Zurich airport are you taking notes?) But here my first ominous piece of bad luck began. No sign of my suitcase. Happily some asking around revealed that it had somehow gotten on some earlier Swiss Air flight instead of the one I was on and was actually waiting for me. I got outside and Debian Developer Praveen Arimbrathodiyil was waiting to pick me up. Normally I don't lke staying in Mumbai very much even though I have relatives there but that's because we usually went during July-August the monsoon season when Mumbai reverts back to the swampy archipelago it was originally built on. This time the weather was nice, cold by local standards, but lovely and spring-like to someone from snowy New Jersey. There have been a lot of improvements to the road infrastructure and people are actually obeying the traffic laws. (Within reason of course. Whether or not a family of six can arrange themselves on one Bajaj scooter is no business of the cops.) The Hotel Tuliip (yes, two i's. Manager didn't know why.) Residency where I was to stay while not quite a five star establishment was adequate for my needs with a bed, hot water shower, and air conditioning. And a TV which to the bellhops great confusion I did not want turned on. (He asked about five times.) There was no Internet access per se but the manager offered to hook up a wireless router to a cable. Which on closer inspection turned out to have been severed at the base. He assured me it would be fixed tomorrow so I didn't complain and decided to do something more productive thank checking my email like sleeping. The next day I woke up in total darkness. Apparently there had been some kind of power problem during the night which tripped a fuse or something. A call to the front desk got them to fix that and then the second piece of bad luck happened. I plugged my Thinkpad in and woke it up from hibernation and a minute later there was a loud pop from the power adapter. Note I have a travel international plug adapter with surge protector so nothing bad ought to have happened but the laptop would on turning on display the message "critical low battery error" and immediately power off. I was unable to google what that meant without Internet access but I decided not to panic and continue getting ready. I would have plenty of opportunity to troubleshoot at the conference venue. Or so I thought... I took an autorickshaw to IIT. There also there have been positive improvements. Being quite obviously a foreigner I was fully prepared to be taken along the "scenic route." But now there are fair zones and the rickshaws all have (tamperproof!) digital fare meters so I was deposited at the main gate without fuss. After reading a board with a scary list of dos and don'ts I presented myself at security only to be inexplicably waved through without a second glance. Later I found out they've abandoned all the security theatre but not got around to updating the signs yet. Mumbai is one of the biggest, densely populated cities in the world but the IIT campus is an oasis of tranquility on the shores of Lake Powai. It's a lot bigger than it looked on the map so I had to wander around a bit before I reached the conference venue but I did make for the official registration time. Registration I was happy to meet several old friends (Such as Kartik Mistry and Kumar Appiah who along with Praveen and myself were the other DDs there,) people who I've corresponded with but never met, and many new people. I'm told 200+ people registered altogether. Most seemed to be students from IIT and elsewhere in Mumbai but there were also some Debian enthusiasts from further afield and most hearteningly some "civilians" who wanted to know what this was all about. With the help of a borrowed Thinkpad adapter I got my laptop running again. (Thankfully, despite the error message, the battery itself was unharmed.) However, my streak of bad luck was not yet over. It was that very weekend that IIT had a freak campus-wide network outage something that had never happened before. And as the presentation for the talk I was to give had apparently been open when I hibernated my laptop the night before, the sudden forced shutdown had trashed the file. (ls showed it as 0 length. An fsck didn't help.) I possibly had a backup on my server but with no Internet access I had no way to retrieve it. I still remained cool. The talk was scheduled for the second day so I could recover it at the hotel. Keynotes Professor Kannan Maudgalya of the FOSSEE (Free and Open Source Software for Education) Project which is part of the central government Ministry for Human Resource Development spoke about various activities of his project. Of particular interest to us are: FOSSEE is well funded, backed by the government and has enthusiastic staff so we should be seeing a lot more from them in the future. Veteran Free Software activist Venky Hariharan spoke about his experiences in lobbying the government on tech issues. He noted that there has been a sea change in attitudes towards Linux and Open source in the bureacracy of late. Several states have been aggressively mandating the use of it as have several national ministries and agencies. We the community can provide a valuable service by helping them in the transition. They also need to be educated on how to work with the community (contributing changes back, not working behind closed doors etc.) Debian History and Debian Cycle Shirish Agarwal spoke about the Debian philosophy and foundational documents such as the social contract and DFSG and how the release cycle works. Nothing new to an experienced user but informative to the newcomers in the audience and sparked some questions and discussion. Keysigning One of my main missions in attending was to help get as many isolated people as possible into the web of trust. Unfortunately the keysigning was not adequately publicized and few people were ready. I would have led them through the process of creating a new key there and then but with the lack of connectivity that idea had to be abandoned. I did manage to sign about 8-10 keys during other times. Future Directions for Debian-IN BOF I led this one. Lots of spirited discussion and I found feedback from new users in particular to be very helpful. Some take aways are: Lil' Debi Kumar Sukhani was a Debian GSoC student and his project which he demonstrated was to be able to install Debian on an Android phone. Why would you want to do this? Apart from the evergreen "Because I can", you can run server software such as sshd on your phone or even use it as an ARM development board. Unfortunately my phone uses Blackberry 10 OS which can run android apps (emulated under QNX) but wouldn't be able to use this. When I get a real Android phone I will try it out. Debian on ARM Siji Sunny gave this talk which was geared more towards hardware types which I am not but one thing I learned was thee difference between all the different ARM subarchitectures. I knew Siji first from a previous incarnation when he worked at CDAC with the late and much lamented Prof. R.K. Joshi. We had a long conversation about those days. Prof. Joshi/CDAC had developed an Indic rendering system called Indix which alas became the Betamax to Pango's VHS but he was also very involved in other Indic computing issues such as working with the Unicode Consortium and the preseration of Sanskrit manuscripts which is also an interest of mine. One good thing that cameout of Indix was some rather nice fonts. I had thought they were still buried in the dungeons of CDAC but apparently they were freed at one point. That's one more thing for me to look into. Evening/Next morning< My cousin met me and we had a leisurely dinner together. It was quite late by the time I got back to the hotel. FOSSEE had kindly lent me one of their tablets (which incidently are powerful enough to run LibreOffice comfortably.) so I thought I might be able to quickly redo my presentation before bedtime. Well, wouldn't you know it the wifi was not fixed. As I should have guessed but all the progress I'd had made me giddily optimistic. There was an option of trying to find an Internet cafe in a commercial area 15-20 minutes walk away. If this had been Gujarat I would have tried it but although I can more or less understand Hindi I can barely put together two sentences and Marathi I don't know at all. So I gave up that idea. I redid the slides from memory as best I could and went to sleep. In the morning I checked out and ferried myself and my suitcase via rickshaw back to the IIT campus. This time I got the driver to take me all the way in to the conference venue. Prof. Maudgalya kindly offered to let me keep the tablet to develop stuff on. I respectfully had to decline because although I love to collect bits of tech the fact it is it would have just gathered dust and ought to go to someone who can make a real contribution with it. I transferred my files to a USB key and borrowed a loaner laptop for my talk. Debian Packaging Workshop While waiting to do my talk I sat in on a workshop Praveen ran taking participants through the whole process of creating a Debian package (a ruby gem was the example.) He's done this before so it was a good presentation and well attended but the lack of connectivity did put a damper on things. Ask Me Anything It turned out the schedule had to be shuffled a bit so my talk was moved later from the announced time. A few people had already showed up so I took some random questions about Debian from them instead. GNOME Shell Accessibility With Orca Krishnakant Mane is remarkable. Although he is blind, he is a developer and a major contributor to Open Source projects. He talked about the Accessibility features of GNOME and compared them (favorably I might add) with proprietary screen readers. Not a subject that's directly useful to me but I found it interesting nonetheless. Rust: The memory safe language Manish Goregaokar talked about one of the new fad programming languages that have gotten a lot of buzz lately. This one is backed by Mozilla and it's interesting enough but I'll stick with C++ and Perl until one of the new ones "wins." Building a Mail Server With Debian Finally I got to give my talk and, yup, the video out on my borrowed laptop was incompatible with the projector. A slight delay to transfer everything to another laptop and I was able to begin. I talked about setting up BIND, postfix, and of course dovecot along with spamassassin, clamav etc. It turned out I had more than enough material and I went atleast 30 minutes over time and even then I had to rush at the end. People said they liked it so I'm happy. The End I gave the concluding remarks. Various people were thanked (including myself) mementos were given and pictures were taken. Despite a few mishaps I enjoyed myself and I am glad I attended. The level of enthusiasm was very high and lessons were learned so the next Debian-IN event should be even better. My departing flight wasn't due to leave until 1:20AM so I killed a few hours with my family before the flight. Once again I was stopping in Zurich, this time for most of a day. The last of my blunders was not to take my coat out of my suitcase and the temperature outside was 29F so I had to spend that whole time enjoing the (not so) many charms of Zurich airport. Atleast the second flight took me to Newark instead of JFK so I was able to get home a little earlier on Monday evening, exhausted but happy I made the trip.

Michael Prokop: check-mk: monitor switches for GBit links

For one of our customers we are using the Open Monitoring Distribution which includes Check_MK as monitoring system. We re monitoring the switches (Cisco) via SNMP. The switches as well as all the servers support GBit connections, though there are some systems in the wild which are still operating at 100MBit (or even worse on 10MBit). Recently there have been some performance issues related to network access. To make sure it s not the fault of a server or a service we decided to monitor the switch ports for their network speed. By default we assume all ports to be running at GBit speed. This can be configured either manually via: 
cat etc/check_mk/conf.d/wato/rules.mk
[...]
checkgroup_parameters.setdefault('if', [])
checkgroup_parameters['if'] = [
  (  'speed': 1000000000 , [], ['switch1', 'switch2', 'switch3', 'switch4'], ALL_SERVICES,  'comment': u'GBit links should be used as default on all switches'  ),
] + checkgroup_parameters['if']
or by visting Check_MK s admin web-interface at WATO Configuration -> Host & Service Parameters -> Parameters for Inventorized Checks -> Networking -> Network interfaces and switch ports and creating a rule for the Explicit hosts switch1, switch2, etc and setting Operating speed to 1 GBit/s there. So far so straight forward and this works fine. Thanks to this setup we could identify several systems which used 100Mbit and 10MBit links. Definitely something to investigate on the according systems with their auto-negotiation configuration. But to avoid flooding the monitoring system and its notifications we want to explicitly ignore those systems in the monitoring setup until those issues have been resolved. First step: identify the checks and their format by either invoking cmk -D switch2 or looking at var/check_mk/autochecks/switch2.mk: 
OMD[synpros]:~$ cat var/check_mk/autochecks/switch2.mk
[
  ("switch2", "cisco_cpu", None, cisco_cpu_default_levels),
  ("switch2", "cisco_fan", 'Switch#1, Fan#1', None),
  ("switch2", "cisco_mem", 'Driver text', cisco_mem_default_levels),
  ("switch2", "cisco_mem", 'I/O', cisco_mem_default_levels),
  ("switch2", "cisco_mem", 'Processor', cisco_mem_default_levels),
  ("switch2", "cisco_temp_perf", 'SW#1, Sensor#1, GREEN', None),
  ("switch2", "if64", '10101',  'state': ['1'], 'speed': 1000000000 ),
  ("switch2", "if64", '10102',  'state': ['1'], 'speed': 1000000000 ),
  ("switch2", "if64", '10103',  'state': ['1'], 'speed': 1000000000 ),
  [...]
  ("switch2", "snmp_info", None, None),
  ("switch2", "snmp_uptime", None,  ),
]
OMD[synpros]:~$
Second step: translate this into the according format for usage in etc/check_mk/main.mk: 
checks = [
  ( 'switch2', 'if64', '10105',  'state': ['1'], 'errors': (0.01, 0.1), 'speed': None ), # MAC: 00:42:de:ad:be:af,  10MBit
  ( 'switch2', 'if64', '10107',  'state': ['1'], 'errors': (0.01, 0.1), 'speed': None ), # MAC: 00:23:de:ad:be:af, 100MBit
  ( 'switch2', 'if64', '10139',  'state': ['1'], 'errors': (0.01, 0.1), 'speed': None ), # MAC: 00:42:de:ad:be:af, 100MBit
  [...]
]
Using this configuration we ignore the operation speed on ports 10105, 10107 and 10139 of switch2 using the the if64 check. We kept the state setting untouched where sensible ( 1 means that the expected operational status of the interface is to be up ). The errors settings specifies the error rates in percent for warnings (0.01%) and critical (0.1%). For further details refer to the online documentation or invoke cmk -M if64 . Final step: after modifying the checks configuration make sure to run cmk -IIu switch2 ; cmk -R to renew the inventory for switch2 and apply the changes. Do not forget to verify the running configuration by invoking cmk -D switch2 : Screenshot of 'cmk -D switch2' execution

21 January 2015

Jo Shields: mono-project.com Linux packages, January 2015 edition

The latest version of Mono has released (actually, it happened a week ago, but it took me a while to get all sorts of exciting new features bug-checked and shipshape). Stable packages This release covers Mono 3.12, and MonoDevelop 5.7. These are built for all the same targets as last time, with a few caveats (MonoDevelop does not include F# or ASP.NET MVC 4 support). ARM packages will be added in a few weeks time, when I get the new ARM build farm working at Xamarin s Boston office. Ahead-of-time support This probably seems silly since upstream Mono has included it for years, but Mono on Debian has never shipped with AOT d mscorlib.dll or mcs.exe, for awkward package-management reasons. Mono 3.12 fixes this, and will AOT these assemblies optimized for your computer on installation. If you can suggest any other assemblies to add to the list, we now support a simple manifest structure so any assembly can be arbitrarily AOT d on installation. Goodbye Mozroots! I am very pleased to announce that as of this release, Mono users on Linux no longer need to run mozroots to get SSL working. A new command, cert-sync , has been added to this release, which synchronizes the Mono SSL certificate store against your OS certificate store and this tool has been integrated into the packaging system for all mono-project.com packages, so it is automatically used. Just make sure the ca-certificates-mono package is installed on Debian/Ubuntu (it s always bundled on RPM-based) to take advantage! It should be installed on fresh installs by default. If you want to invoke the tool manually (e.g. you installed via make install, not packages) use 
cert-sync /path/to/ca-bundle.crt
On Debian systems, that s 
cert-sync /etc/ssl/certs/ca-certificates.crt
and on Red Hat derivatives it s 
cert-sync /etc/pki/tls/certs/ca-bundle.crt
Your distribution might use a different path, if it s not derived from one of those. Windows installer back from the dead Thanks to help from Alex Koeplinger, I ve brought the Windows installer back from the dead. The last release on the website was for 3.2.3 (it s actually not this version at all it s complicated ), so now the Windows installer has parity with the Linux and OSX versions. The Windows installer (should!) bundles everything the Mac version does F#, PCL facades, IronWhatever, etc, along with Boehm and SGen builds of the Mono runtime done with Visual Studio 2013. An EXPERIMENTAL OH MY GOD DON T USE THIS IN PRODUCTION 64-bit installer is in the works, when I have the time to try and make a 64-build of Gtk#.

19 January 2015

Daniel Pocock: Storage controllers for small Linux NFS networks

While contemplating the disk capacity upgrade for my Microserver at home, I've also been thinking about adding a proper storage controller. Currently I just use the built-in controller in the Microserver. It is an AMD SB820M SATA controller. It is a bottleneck for the SSD IOPS. On the disks, I prefer to use software RAID (such as md or BtrFs) and not become dependent on the metadata format of any specific RAID controller. The RAID controllers don't offer the checksumming feature that is available in BtrFs and ZFS. The use case is NFS for a small number of workstations. NFS synchronous writes block the client while the server ensures data really goes onto the disk. This creates a performance bottleneck. It is actually slower than if clients are writing directly to their local disks through the local OS caches. SSDs on an NFS server offer some benefit because they can complete write operations more quickly and the NFS server can then tell the client the operation is complete. The more performant solution (albeit with a slight risk of data corruption) is to use a storage controller with a non-volatile (battery-backed or flash-protected) write cache. Many RAID controllers have non-volatile write caches. Some online discussions of BtrFs and ZFS have suggested staying away from full RAID controllers though, amongst other things, to avoid the complexities of RAID controllers adding their metadata to the drives. This brings me to the first challenge though: are there suitable storage controllers that have a non-volatile write cache but without having other RAID features? Or a second possibility: out of the various RAID controllers that are available, do any provide first-class JBOD support? Observations I looked at specs and documentation for various RAID controllers and identified some of the following challenges: Next steps Are there other options to look at, for example, alternatives to NFS? If I just add in a non-RAID HBA to enable faster IO to the SSDs will this be enough to make a noticeable difference on the small number of NFS clients I'm using? Or is it inevitable that I will have to go with one of the solutions that involves putting a vendor's volume metadata onto JBOD volumes? If I do go that way, which of the vendors' metadata formats are most likely to be recognized by free software utilities in the future if I ever connect the disk to a generic non-RAID HBA? Thanks to all those people who provided comments about choosing drives for this type of NAS usage. Related reading

17 January 2015

Tim Retout: CPAN PR Challenge - January - IO-Digest

I signed up to the CPAN Pull Request Challenge - apparently I'm entrant 170 of a few hundred. My assigned dist for January was IO-Digest - this seems a fairly stable module. To get the ball rolling, I fixed the README, but this was somehow unsatisfying. :) To follow-up, I added Travis-CI support, with a view to validating the other open pull request - but that one looks likely to be a platform-specific problem. Then I extended the Travis file to generate coverage reports, and separately realised the docs weren't quite fully complete, so fixed this and added a test. Two of these have already been merged by the author, who was very responsive. Part of me worries that Github is a centralized, proprietary platform that we now trust most of our software source code to. But activities such as this are surely a good thing - how much harder would it be to co-ordinate 300 volunteers to submit patches in a distributed fashion? I suppose you could do something similar with the list of Debian source packages and metadata about the upstream VCS, say...

14 January 2015

Ritesh Raj Sarraf: Apport in Debian

Looking at the PTS entries, I realized that it has been more than 2 yrs, since I pushed the first Apport packages into Debian. We have talked about it in the past, and do not see a direct need for apport yet. That is one reason why it still resides (and will continue to) in Experimental. Even though not used as a bug reporting tool, Apport can still be a great tool for (end) users to detect crashes. It can also be used to find further details about program crashes and pointers to look further. This post is a call for help if there is anybody, who'd be interested to work on maintaining Apport in Debian. Most work include maintaining new upstream releases, and porting the Debian CrashDB to newer versions, as and when necessary. As said above, it is not going to be a bug reporting tool, but rather a bug monitoring tool.

Categories:

Keywords:

13 January 2015

Erich Schubert: Big data predictions for 2015

My big data predictions for 2015:
  1. Big data will continue to fail to deliver for most companies.
    This has several reasons, including in particular: 1: lack of data to analyze that actually benefits from big data tools and approaches (and which is not better analyzed with traditional tools). 2: lack of talent, and failure to attract analytics talent. 3: stuck in old IT, and too inflexible to allow using modern tools (if you want to use big data, you will need a flexible "in-house development" type of IT that can install tools, try them, abandon them, without going up and down the management chains) 4: too much marketing. As long as big data is being run by the marketing department, not by developers, it will fail.
  2. Project consolidation: we have seen hundreds of big data software projects the last years. Plenty of them on Apache, too. But the current state is a mess, there is massive redundancy, and lots and lots of projects are more-or-less abandoned. Cloudera ML, for example, is dead: superseded by Oryx and Oryx 2. More projects will be abandoned, because we have way too many (including much too many NoSQL databases, that fail to outperform SQL solutions like PostgreSQL). As is, we have dozens of competing NoSQL databases, dozens of competing ML tools, dozens of everything.
  3. Hype: the hype will continue, but eventually (when there is too much negative press on the term "big data" due to failed projects and inflated expectations) move on to other terms. The same is also happening to "data science", so I guess the next will be "big analytics", "big intelligence" or something like that.
  4. Less openness: we have seen lots of open-source projects. However, many decided to go with Apache-style licensing - always ready to close down their sharing, and no longer share their development. In 2015, we'll see this happen more often, as companies try to make money off their reputation. At some point, copyleft licenses like GPL may return to popularity due to this.

8 January 2015

Wouter Verhelst: ExtreMon example

About a month ago, I blogged about extremon. As a reminder, ExtreMon is a monitoring tool that allows you to view things as they are happening, rather than with the ~5 minute delay that munin gives you, and also avoiding the quad-state limitation of Nagios' "good", "bad", "ugly", and "unknown" states. No, they're not really called that. Yes, I know you knew that. Anyway. In my blog post, I explained how you can set up ExtreMon, and I also set up a fairly limited demo version on my own server. But I have since realized that while it is functional, it doesn't actually show why ExtreMon is so great. In an effort to remedy that, I present you an example of what ExtreMon can do. Let's start with a screenshot of the ExtreMon console at the customer for which I spent time trying to figure out how to get it up and running: Click for full sized version. You'll note that even in that full-sized version, many things are unreadable. This is because the ExtreMon console allows one to move around (right mouse button drag for zoom; left mouse button drag for moving around; control+RMB for rotate; center mouse button to reset to default); so what matters is that everything fits on the screen, not whether it is all readable (if you need to read, you zoom). The image shows 18 rectangles. Each rectangle represents a single machine in this particular customer's HPC cluster. The top three rectangles are the cluster's file servers; the rest are its high performance nodes. You'll note that the left fileserver has 8 processor cores (top row), 8 network cards (bottom row, left part), and it also shows information on its memory usage (bottom row, small rectangle in the middle) as well as its NFS client and server procedure calls (bottom row, slightly larger rectangles to the right). This file server is the one on which I installed ZFS a while back; hence the large amount of disks visible in the middle row. The leftmost disk is the root filesystem (which is an ext4 off a hardware RAID1); the two rightmost "disks" are the PCIe-attached SSDs which are used for the ZFS L2ARC and write log. The other disks in this file server nicely show how ZFS does write load balancing over all its disks. The second file server has a hardware RAID1 on which it stores all its data; as such, there is only one disk graph there. It is also somewhat more limited in network, as it has only two NICs. It does, however, also have 8 cores. The last file server has no more than four processor cores; in addition, it also does not have a hardware RAID controller, so it must use software RAID over its four hard disks. This server is used for archival purposes, mostly, since it is insufficient for most anything else. As said, the other nodes are the "compute nodes", where the hard work is done. Most of these compute nodes have 16 cores each; two have 12 instead. When this particular screenshot was taken, four of the nodes (the ones showing red in their processor graphs) were hard at work; the others seem to have been mostly idling. In addition to the familiar memory, NFS (client only), network, and processor graphs, these nodes also show a "swap space" graph (just below the memory one), which seems fine for most nodes, except for the bottom left one (which shows a few bars that are coloured yellow rather than green). The green/yellow/red stuff is supposed to represent the "ok", "warning", "bad" states that would be familiar from Nagios. In this particular case, however, where "processor is busy all the time" is actually a wanted state, a low amount of idleness on the part of the processor isn't actually a problem, on the contrary. I did consider, therefore, to modify the ExtreMon configuration so that the processor graphs would not show red when the system was under high load; however, I found that differences in colour like this actually makes it more easy to see, at a glance, which machines are busy -- and that's one of the main reasons why we wanted to set this up. If you look carefully, you can find a particular processor core in the graph which shows 100% usage for "idle", "system", and "softirq", at the same time. Obviously that can't be the case, so there's a bug somewhere. Frank seems to believe it is a bug in CollectD; I haven't verified that. At any rate, though, this isn't usually a problem, due to the high update frequency of ExtreMon. The amount of data that's flowing through ExtreMon is amazing: Which renders a grand total of 2887 data points that are shown in this particular screenshot; and then I'm not even counting all the intermediate values, some of which also pass through ExtreMon. Nor am I counting the extra bits which have since been added (this screenshot is a few days old, now, and I'm still finetuning things). Yet even so, ExtreMon manages to update those values once every few seconds, in the worst case. As a result, the display isn't static for a moment, constantly moving and updating data so that what you see is never out of date for more than a second or two. Awesome.

6 January 2015

Russ Allbery: Review: Code Complete, Second Edition

Review: Code Complete, Second Edition, by Steve McConnell
Publisher: Microsoft
Copyright: June 2004
ISBN: 0-7356-1967-0
Format: Kindle
Pages: 960
As mentioned in the title, this is a review of the second edition of Code Complete, published in 2004. There doesn't appear to be a later edition at the time of this writing. I should say, as a prefix to this review, that I'm the sort of person who really likes style guides. When learning a language, a style guide is usually the second or third document I read. I enjoy debates over the clearest way to express a concept in code, trying to keep all the code in a large project consistent, and discussing the subtle trade-offs that sit on the boundary between mechanical style issues and the expressiveness of programming. I try to spend some time reading good code and getting better at expressing myself in code. Presumably, therefore, I'm the target audience for this book. It sounded good from the descriptions, so I picked it up during one of the Microsoft Press sales. The stated goal of Code Complete is to collect in one place as much as possible of the oral tradition and lore of the programming field, to try to document and communicate the techniques and approaches that make someone a good programmer. The table of contents sounds like a style guide, with entire sections on variables and statements in addition to topics like how to improve existing code and how to design a new program. If you're starting to think that a 960 page style guide sounds like a bad idea, you're wiser than I. (In my defense, I grabbed this as an ebook and didn't realize how large it was before I bought it.) I have not actually finished this book. I hate to do this: I don't like reviewing books I haven't finished (this will be the first), and I hate starting books and not finishing them. (This is probably not particularly wise, since some books aren't worth finishing, but I've gotten into a rhythm of reading and reviewing that works for me, so I try not to mess with it.) But I've been trying to finish this book off and on for about a year, I don't think it's worth the time investment, and I think I've gotten far enough in it to provide some warnings by others who are deceived by the very high ratings that it gets on Amazon and other places. The primarily problem with Code Complete is its sheer, mind-numbing comprehensiveness. It tries to provide a set of guidelines and a checklist to think about at each level of writing code. This is one of those ideas that might sound good on paper, but which completely doesn't work for me. There is no way I'm going to keep this many rules in my head, in the form of rules, while programming. Much of good style has to be done by feel, and the book I'm looking for is one that improves my feel and my sense of taste for code. What Code Complete seems to provide instead is a compilation of every thought that McConnell has ever had about programming. There's a lot of basic material, a few thoughtful statements, a ton of style advice, an endless compilation of trade-offs and concepts that one should keep in mind, and just a massive, overwhelming pile of stuff. Each chapter (and there are a lot of chapters) ends in a checklist of things that you should think about when doing a particular programming task. To give you a feel for the overwhelming level of trivia here, this is the checklist at the end of the chapter where I stopped reading, on quality assurance in software. This is one picked at random; a lot of them are longer than this.
  • Have you identified specific quality characteristics that are important to your project?
  • Have you made others aware of the project's quality objectives?
  • Have you differentiated between external and internal quality characteristics?
  • Have you thought about the ways in which some characteristics might compete with or complement others?
  • Does your project call for the use of several different error-detection techniques suited to finding several different kinds of errors?
  • Does your project include a plan to take steps to assure software quality during each stage of software development?
  • Is the quality measured in some way so that you can tell whether it's improving or degrading?
  • Does management understand that quality assurance incurs additional costs up front in order to save costs later?
I'm not saying those are bad things to think about with quality assurance, but you may notice a few issues immediately. They're very general and vague, they're not phrased in a particularly compelling or memorable way, and there are a lot of them. This falls between two stools: it's too much for the programmer who is thinking about quality as part of an overall project but not focusing on it (particularly when you consider that the book is full of checklists like this for everything from variable naming to how to structuring if statements to program debugging), but it's not nearly specific or actionable enough for someone who is focusing on quality assurance. It's not that the information isn't organized: there's a lot of structure here. And there are bits and pieces here that are occasionally interesting. McConnell is very data-driven and tries to back up recommendations with research on error rates and similar concrete measurements. It's just insufficiently filtered and without elegant or memorable summary. There is far too much here, an overwhelming quantity, and hopelessly mixed between useful tidbits and obvious observations that anyone who has been programming for a while would pick up, all presented in the same earnest but dry tone. It didn't help that there's a lot here I didn't agree with. Some of that is to be expected: I've never agreed completely with any style guide. But McConnell kept advocating variable and function naming conventions that I find rather ugly and tedious, and the general style of code he advocates feels very "bureaucratic" to me. It's not exactly wrong, but one of the things that I look for in style discussions is to be inspired by the elegant and simple way someone finds to phrase something in code. A lot of the code in this book just felt mind-numbing. It's functional, but uninteresting; perfectly adequate for a large project, but not the sort of discussion that inspires me to improve the quality of my craft. So, I didn't finish this. I gave up about halfway through. It's frustrating, since I was occasionally finding an interesting nugget of information. But they were too few and far between, and the rest of the book was mostly... boring. It's possible that I just know too much about programming to be the person for whom that McConnell was writing this book. It's certainly true that the book has not aged particularly well; it's focused on fairly old-school languages (C, C++, Java, and Visual Basic) and says almost nothing about modern language techniques, although it does have a bit about extreme programming. But whatever the reason is, it didn't work for me at all. I would rate it as one of the worst books about programming I've tried to read. And that's notably different enough from its reviews that it seems worth throwing this out there as a warning. I'm quite disappointed, since I'd heard nothing but praise for this book before picking it up. But it's not for me, and I'm now dubious of its value for any programmer outside of a fairly narrow, large-team, waterfall development process involving large numbers of people writing very large quantities of code in languages that aren't very expressive. And, well, in that situation I think one would get more benefit from changing that environment than reading this book. Rating: unfinished

4 January 2015

Russ Allbery: Review: Programming Ruby

Review: Programming Ruby, by Dave Thomas, et al.
Publisher: Pragmatic Bookshelf
Copyright: 2005
Printing: May 2006
ISBN: 0-9745140-5-5
Format: Trade paperback
Pages: 785
There are a few different editions of this book. The version I read is the second edition, written by Dave Thomas with Chad Fowler and Andy Hunt and published in 2005, covering Ruby 1.8.2. There's now a fourth edition, covering Ruby 1.9 and 2.0, which is probably what you'd want if you were buying this book today. This book, in whatever edition, is called the Pickaxe in the Ruby community after its cover. I've used a lot of different programming languages, so I can usually pick one up on the fly reasonably well, but I still like to read a good introductory book before using one seriously. It's a bit too easy to get lost or to fall into habits that don't match the best practices of the language community without a solid introduction. I've been using a bit of Ruby off and on since I started using Puppet, but I'm looking at doing more serious development using Chef, so I decided it was time to get that introduction. (It helped that I had this book sitting around, although that's also why I read an older edition.) Programming Ruby starts with the obligatory introduction to installing and running Ruby, and then provides a high-level introduction to the language and its basic types just enough to make Ruby comprehensible before starting into the object system. Everything is an object in Ruby, so the book introduces the object system as early as possible, and then shows the rest of the language from constants up in the light of that object system. The rest of part one follows the normal language introduction path, building up from constants and methods to exceptions, modules, and basic IO. It closes with chapters about threads and processes, unit testing, and the debugger. Part two is a grab-bag of one-chapter topics describing how to use Ruby in a particular setting, or showing one angle of the language. The best of those chapters for me was the one on RDoc, partly because I'm quite impressed by Ruby's documentation system. A few of these chapters are oddly in-depth for an introductory book I doubt I'm ever going to use all the details about special irb configuration, and if I do, I'd just look them up but I greatly appreciated the solid chapter on how to write Ruby extensions in C. There is also the obligatory chapter on writing GUI applications with Tk, which always seems to show up in these sorts of introductions and which always baffles me. Does anyone actually do this any more instead of writing a web application? Part three dives back into the language and provides a more complete and formal description. The authors aren't afraid to get into some of the internals, which I appreciated. There is a good chapter here on the details of the type system and how objects and classes interact, and a much-needed extended discussion of duck typing. This type of weak typing and runtime binding is fundamental to how Ruby approaches objects, for better or worse. (I have mixed opinions; it makes some things easier, but I increasingly appreciate strong typing and more formal interface definitions.) Some discussion of marshalling and introspection closes out the discussion portion of the book. That's about 420 pages of the material. The rest of the book is a detailed reference on all of the core classes, and a quicker overview of the standard library. Normally, this sort of thing is thrown into language introductions to pad out the page count, but usually the language's official documentation is better at this sort of reference. But I found Programming Ruby to be an exception. The reference is succinct, sticking to a paragraph or two for each method, and did a great job of providing enough cross-reference and discussion to put each class into a broader perspective. It's the most useful example of this type of reference section I've seen. I still probably won't use it after this initial reading, but I think I got a better feel for the language from reading through it. It's hard to review a book like this without reviewing the language it documents, at least a little bit. I'll indulge: it entertains me how much Ruby is obviously based on Perl, including borrowing some of Perl's more dubious ideas. The global punctuation variables will look familiar to any Perl programmer, and the oddly-named global variables for the interpreter flags are in the same spirit. The language unfortunately has similar problems as Perl with safely running commands without using the shell; it's possible, but not the default and not what the built-ins do. There are places where I wish Ruby were a little less like Perl. The plus side for an experienced Perl programmer is that Ruby feels quite familiar and has made some clear improvements. The ? and ! convention for methods that return booleans or modify objects in-place is brilliant in its simplicity, and something I'd love to see in more languages. And the way Ruby implements ubiquitous code blocks for both iterators and for any temporary objects is lovely once one gets used to it. It's similar to Python's context managers, except more general and built deeper into the language. Returning to the review of the book, rather than the topic, Programming Ruby has a good, clear explanation of blocks, iterators, and yield. If you're interested in getting a grounding in Ruby, this book still feels like a solid introduction. The edition I read is getting a bit long in the tooth now that we're on Ruby 2.1, but the pace of language change has slowed, and most of the book is still applicable. (If you're buying it new, you should, of course, get the later edition.) The table of contents makes it seem like the book is covering the same ground multiple times, but that organizational strategy worked better than I expected. Ruby is not the most organized language in the world, so I still felt a bit overwhelmed with random method names in places, but I never felt lost in the mechanics of the language. In short, recommended if you want a good introduction to the language, although probably in a later edition. Rating: 8 out of 10

Next.

Previous.