Search Results: "spa"

30 January 2013

James Morrison: Issues with Appengine

I like App Engine, my wedding website runs on App Engine as well as a couple recent commercial projects. However, there are some annoyances that make a lot of the useful things I learned at Google not applicable.

Chunked responses
App Engine doesn't have a way of doing chunked responses.
1.5. Processing after sending the response My annoyance here is that I can't send an early response and wait for some resource. Say I want to do 5 url fetches that I cache for a short period of time. If I want to reply in 250ms, I want to use any of the results I've gotten and reply to the user. Then once the rest of the fetches come in, I want to put them in the cache, so they will come back quickly the next time.
Counters
App Engine has crappy support for counters. You can shard your counters into the datastore: https://developers.google.com/appengine/articles/sharding_counters#counter_python or you can use taskqueues: http://blog.appenginefan.com/2009/10/non-sharded-counters-part-2-using-task.html . Now, how do you graph these counters over time, well, that is left as an exercise for the reader. Google has special datastores for stores and retrieving counters.
Backends
Backends in appengine are versioned separately from frontends. This sounds like a good idea, but to run the devserver with backends, the backends need to be in the same source tree. Thus backends share the same url space. Backends also share the same app.yaml, so again the url space is shared. Thus, if you make a change that you want to coordinate between the frontend and the backend, you need to change the application version in app.yaml, then rename all the backends in backends.yaml. If backends are supposed to be in the same source tree as frontends, then they should be prefixed with the application version. If backends should be versioned separately from frontends, then they shouldn't share url handlers from app.yaml.
3.5 Uploading backends is slow
Each backend is uploaded through appcfg.py separately. So if you are crazy and have one backend per backend level, then you need to upload 4 copies of the source code each time the backends get updated.
Logging
I want structured logs. Instead what you have to do is use the python logging module to write out human readable logs, then parse them out into the structure you want.
http://code.google.com/p/google-app-engine-samples/source/browse/trunk/logparser/logparser.py

As a preemptive counter, 1.5 could be dealt with using backends and a background thread, except backends don't scale to the number of queries, so my use case above isn't fixed.

Michal Čihař: phpMyAdmin translations status

Next round of phpMyAdmin 4.0 translation status report is coming. So let's look at which translations are at 100% right now:

There are also several languages which need just few strings to be complete:

As you can see, there is still lot of languages missing, this might be your opportunity to contribute to phpMyAdmin. Also you are welcome to translate phpMyAdmin 4.0 using translation server. If your language is already fully translated and you want to help as well, you can translate our documentation as well.

Filed under: English Phpmyadmin 0 comments Flattr this!

Uwe Hermann: libsigrokdecode 0.1.1 released, more protocol decoders supported

Just a quick announce: We released libsigrokdecode 0.1.1 today, a new version of one of the shared libraries part of the open-source sigrok project (for signal acquisition/analysis of various test&measurement gear, like logic analyzers, scopes, multimeters, etc). I will update the Debian package soonish. As you probably know, in addition to the infrastructure for protocol decoding, this library also ships with a bunch of protocol decoders written in Python. Currently we support 29 different ones (in various states of "completeness", improvements are ongoing). This release adds support for the following new protocol decoders:

avr_isp: AVR In-System Programming
can: Controller Area Network
jtag: Joint Test Action Group (IEEE 1149.1)
jtag_stm32: Joint Test Action Group / ST STM32
lm75: National LM75
lpc: Low-Pin-Count
maxim_ds28ea00: Maxim DS28EA00 1-Wire digital thermometer
onewire_link: 1-Wire serial communication bus (link layer)
onewire_network: 1-Wire serial communication bus (network layer)
sdcard_spi: Secure Digital card (SPI mode)
tlc5620: Texas Instruments TLC5620 DAC
uart_dump: UART dump

Please check the announce on the sigrok blog and/or the NEWS file for the full list of changes and improvements. Happy hacking and decoding!

29 January 2013

Daniel Kahn Gillmor: visualizing MIME structure

Better debugging tools can help us understand what's going on with MIME messages. A python scrap i wrote a couple years ago named printmimestructure has been very useful to me, so i thought i'd share it. It reads a message from stdin, and prints a visualisation of its structure, like this:

0 dkg@alice:~$ printmimestructure < 'Maildir/cur/1269025522.M338697P12023.monkey,S=6459,W=6963:2,Sa' 
 multipart/signed 6546 bytes
  text/plain inline 895 bytes
  application/pgp-signature inline [signature.asc] 836 bytes
0 dkg@alice:~$

You can fetch it with git if you like:

git clone git://lair.fifthhorseman.net/~dkg/printmimestructure

It feels silly to treat this ~30 line script as its own project, but i don't know of another simple tool that does this. If you know of one, or of something similar, i'd love to hear about it in the comments (or by sending me e-mail if you prefer). If it's useful for others, I'd be happy to contribute printmimestructure to a project of like-minded tools. Does such a project exist? Or if people think it would be handy to have in debian, i can also package it up, though that feels like it might be overkill. and oh yeah, as always: bug reports, suggestions, complaints, and patches are welcome :)Tags: debugging, mime, python

Tollef Fog Heen: Abusing sbuild for fun and profit

Over the last couple of weeks, I have been working on getting binary packages for [Varnish] modules built. In the current version, you need to have a built, unpacked source tree to build a module against. This is being fixed in the next version, but until then, I needed to provide this in the build environment somehow. RPMs were surprisingly easy, since our RPM build setup is much simpler and doesn't use mock/mach or other chroot-based tools. Just make a source RPM available and unpack + compile that. Debian packages on the other hand, they were not easy to get going. My first problem was to just get the Varnish source package into the chroot. I ended up making a directory in /var/lib/sbuild/build which is exposed as /build once sbuild runs. The other hard part was getting Varnish itself built. sbuild exposes two hooks that could work: a pre-build hook and a chroot-setup hook. Neither worked: Pre-build is called before the chroot is set up, so we can't build Varnish. Chroot-setup is run before the build-dependencies are installed and it runs as the user invoking sbuild, so it can't install packages. Sparc32 and similar architectures use the linux32 tool to set the personality before building packages. I ended up abusing this, so I set HOME to a temporary directory where I create a .sbuildrc which sets $build_env_cmnd to a script which in turns unpacks the Varnish source, builds it and then chains to dpkg-buildpackage. Of course, the build-dependencies for modules don't include all the build-dependencies for Varnish itself, so I have to extract those from the Varnish source package too. No source available at this point, mostly because it's beyond ugly. I'll see if I can get it cleaned up.

27 January 2013

Jan Wagner: Fixing muninlite for up interfaces without link

Usually I'm monitoring stuff with Icinga (Nagios in the past). But for my small network, I primary needed monitoring of bandwidth. In our commercial environment we are using a closed source software for accounting traffic. There is also a license for testing purpose with a reduced number of sensors available. But I'm neither running windows in this network nor feeling happy with this. Cacti is a bit bloated for this small network and zabbix is (caused by what?) removed in wheezy, beside that I'm not getting the concept behind it. So I thought I could give munin a try and on the first view it doesn't look so bad. Monitoring my half dozens openwrt devices works like a charm by installing muninlite just the package. One central part of the network is a QNAP TS-459 Pro+, hosting a BackupPC and TimeMachine service, proving SMB/AFS data store and running SqueezeBox Server for another half dozen streaming devices. Unfortunately there is no optware package to provide a munin node. So I just copied the shell script of muninlite and the xinet config over from an openwrt device. At first it looked not bad, but than munin wasn't able to collect the data. After a while I realized, that munin was failing when collecting the network informations. A look into the muninlite script revealed that it was failing when trying to discover the interface speed of eth1 via ethtool. In my setup the QNAP is just connected with with one network interface, the second one is unconnected. Unfortunately all network interfaces on QNAP devices are up and therefore listed in /proc/net/dev where muninlite is discovering the network interfaces:

[~] # grep '^ *\(ppp\ eth\ wlan\ ath\ ra\ ipsec\ tap\ br-\)\([^:]\)\ 1,\ :' /proc/net/dev   cut -f1 -d:   sed 's/ //g
> s/\-/_/g'
eth1
eth0

Let's look into it:

[~] # ethtool eth0  grep Speed:
    Speed: 1000Mb/s
[~] # ethtool eth0  grep "Link detected:"
    Link detected: yes
[~] # ethtool eth1  grep Speed:
    Speed: Unknown! (65535)
[~] # ethtool eth1  grep "Link detected:"
    Link detected: no

Maybe you see .. the interface eth1 is up but has no link, so there is no speed negotiated and muninlite is failing. Thus I hacked the scripted and now it's working like a charme.

<notextile><figure class="code"><figcaption> (muninlite_fix-unused-up_interface.diff) download</figcaption>

--- /opt/sbin/munin-node.orig	2013-01-27 15:13:51.869007214 +0100
+++ /opt/sbin/munin-node	2013-01-27 16:11:20.536006950 +0100
@@ -133,7 +133,7 @@
   if [ -n "$(which ethtool)" ]; then
 	if [ -x "$(which ethtool)" ]; then
   		if ethtool $1   grep -q Speed; then
-    			MAX=$(($(ethtool $1   grep Speed   sed -e 's/[[:space:]]\ 1,\ / /g' -e 's/^ //' -e 's/M.*//'   cut -d\  -f2) * 1000000))
+    			MAX=$(($(ethtool $1   grep Speed   sed -e 's/[[:space:]]\ 1,\ / /g' -e 's/^ //' -e 's/M.*//'   sed -e 's/Unknown\!/0/'   cut -d\  -f2) * 1000000))
     			echo "up.max $MAX"
     			echo "down.max $MAX"
 		fi
@@ -535,19 +535,31 @@
     for INTER in $(grep '^ *\(ppp\ eth\ wlan\ ath\ ra\ ipsec\ tap\ br-\)\([^:]\)\ 1,\ :' /proc/net/dev   cut -f1 -d:   sed 's/ //g
 s/\-/_/g');
     do
-      INTERRES=$(echo $INTER   sed 's/\./VLAN/')
-      RES="$RES if_$INTERRES"
-      eval "fetch_if_$ INTERRES ()   fetch_if $INTER $@;  ;"
-      eval "config_if_$ INTERRES ()   config_if $INTER $@;  ;"
+      if [ -n "$(which ethtool)" ]; then
+        if [ -x "$(which ethtool)" ]; then
+          if [ -n "$(ethtool $INTER   grep 'Link detected: yes')" ]; then
+            INTERRES=$(echo $INTER   sed 's/\./VLAN/')
+            RES="$RES if_$INTERRES"
+            eval "fetch_if_$ INTERRES ()   fetch_if $INTER $@;  ;"
+            eval "config_if_$ INTERRES ()   config_if $INTER $@;  ;"
+          fi
+        fi
+      fi
     done
   elif [ "$PLUG" = "if_err_" ]; then
     for INTER in $(grep '^ *\(ppp\ eth\ wlan\ ath\ ra\ ipsec\ tap\ br-\)\([^:]\)\ 1,\ :' /proc/net/dev   cut -f1 -d:   sed 's/ //g
 s/\-/_/g');
     do
-      INTERRES=$(echo $INTER   sed 's/\./VLAN/')
-      RES="$RES if_err_$INTERRES"
-      eval "fetch_if_err_$ INTERRES ()   fetch_if_err $INTER $@;  ;"
-      eval "config_if_err_$ INTERRES ()   config_if_err $INTER $@;  ;"
+      if [ -n "$(which ethtool)" ]; then
+        if [ -x "$(which ethtool)" ]; then
+          if [ -n "$(ethtool $INTER   grep 'Link detected: yes')" ]; then
+            INTERRES=$(echo $INTER   sed 's/\./VLAN/')
+            RES="$RES if_err_$INTERRES"
+            eval "fetch_if_err_$ INTERRES ()   fetch_if_err $INTER $@;  ;"
+            eval "config_if_err_$ INTERRES ()   config_if_err $INTER $@;  ;"
+          fi
+        fi
+      fi
     done
   elif [ "$PLUG" = "netstat" ]; then
     if netstat -s >/dev/null 2>&1; then

</figure></notextile>

Gregor Herrmann: RC bugs 2013/04

yeah, the RC bug count is going down, mostly due to the bug squashing party taking place in cambridge, uk, this weekend (with participation of some release team members). here are my contributions during the last week:

~~#677054~~ nut-client: "nut-client: prompting due to modified conffiles which were not modified by the user"
add some information to the bug log, later NMUd by Ivo De Decker/Andreas Beckmann
~~#695710~~ src:adplug: "adplug: includes non-free documentation (GFDL with unmodifiable sections)"
repack again, upload to DELAYED/5
#696144 spamassassin: "20_head_tests.cf: regex for illegal IP address contains valid network 5/8"
add a comment to the bug report, then marked as fixed in wheezy+sid by other contributor
~~#698764~~ libio-compress-perl: "libio-compress-perl: breaks "debmirror" and possibly other packages."
some investigation (pkg-perl), later closed as user error
#698979 src:libmail-cclient-perl: "libmail-cclient-perl: FTBFS with DEB_BUILD_OPTIONS=nocheck"
fix debian/rules (pkg-perl), severity later downgraded by release team

26 January 2013

Eddy Petrișor: (Serial) console flooded with kernel messages?

(If you want to ignore the explanations and see how to stop the Linux kernel from flooding the console with low importance messages, go straight to the bottom of the article, is the small bit at end with larger font.)

After connecting to the serial console on my Linksys WRT160NL router I was faced with the problem that the console was flooded with all sorts of messages such as:

DROP IN=eth1 OUT= SRC=X.Y.Z.W DST=255.255.255.255 PROTO=UDP SPT=58488 DPT=2008 LEN=26
DROP IN=eth1 OUT= SRC=X.Y.Z.U DST=178.156.183.255 PROTO=UDP SPT=137 DPT=137 LEN=58
DROP IN=eth1 OUT= SRC=X.Y.Z.W DST=255.255.255.255 PROTO=UDP SPT=58488 DPT=2008 LEN=26
ACCEPT IN=br0 OUT=eth1 SRC=a.b.c.d DST=69.171.246.16 PROTO=TCP SPT=3651 DPT=443
DROP IN=eth1 OUT= SRC=X.Y.Z.U DST=178.156.183.255 PROTO=UDP SPT=137 DPT=137 LEN=58
DROP IN=eth1 OUT= SRC=X.Y.Z.W DST=255.255.255.255 PROTO=UDP SPT=58488 DPT=2008 LEN=26
DROP IN=eth1 OUT= SRC=X.Y.Z.W DST=255.255.255.255 PROTO=UDP SPT=58488 DPT=2008 LEN=26
DROP IN=eth1 OUT= SRC=X.Y.Z.W DST=255.255.255.255 PROTO=UDP SPT=58488 DPT=2008 LEN=26
DROP IN=eth1 OUT= SRC=X.Y.Z.W DST=255.255.255.255 PROTO=UDP SPT=58488 DPT=2008 LEN=26
DROP IN=eth1 OUT= SRC=X.Y.Z.W DST=255.255.255.255 PROTO=UDP SPT=58488 DPT=2008 LEN=26
DROP IN=eth1 OUT= SRC=X.Y.Z.W DST=255.255.255.255 PROTO=UDP SPT=58488 DPT=2008 LEN=26
DROP IN=eth1 OUT= SRC=X.Y.Z.W DST=255.255.255.255 PROTO=UDP SPT=58488 DPT=2008 LEN=26
DROP IN=eth1 OUT= SRC=178.156.183.146 DST=255.255.255.255 PROTO=UDP SPT=17500 DPT=17500 LEN=120
DROP IN=eth1 OUT= SRC=178.156.183.146 DST=178.156.183.255 PROTO=UDP SPT=17500 DPT=17500 LEN=120
DROP IN=eth1 OUT= SRC=178.156.183.146 DST=255.255.255.255 PROTO=UDP SPT=17500 DPT=17500 LEN=120
DROP IN=eth1 OUT= SRC=X.Y.Z.W DST=255.255.255.255 PROTO=UDP SPT=58488 DPT=2008 LEN=26
DROP IN=eth1 OUT= SRC=X.Y.Z.W DST=255.255.255.255 PROTO=UDP SPT=58488 DPT=2008 LEN=26
DROP IN=eth1 OUT= SRC=178.156.177.142 DST=255.255.255.255 PROTO=UDP SPT=17500 DPT=17500 LEN=153
DROP IN=eth1 OUT= SRC=178.156.177.142 DST=255.255.255.255 PROTO=UDP SPT=17500 DPT=17500 LEN=153

The serial console was working, but it was impossible to do anything practical in these conditions. I tried looking on the net for 'linux stop console flooding' and similar terms, but didn't get too far, except the fact the problem was the loglevel.

Here is the explanation of what this means (quote from Documentation/kernel-parameters.txt):

        loglevel=       All Kernel Messages with a loglevel smaller than the
                        console loglevel will be printed to the console. It can
                        also be changed with klogd or other programs. The
                        loglevels are defined as follows:
                        0 (KERN_EMERG)          system is unusable
                        1 (KERN_ALERT)          action must be taken immediately
                        2 (KERN_CRIT)           critical conditions
                        3 (KERN_ERR)            error conditions
                        4 (KERN_WARNING)        warning conditions
                        5 (KERN_NOTICE)         normal but significant condition
                        6 (KERN_INFO)           informational
                        7 (KERN_DEBUG)          debug-level messages

This was enough to go through my local git tree of the kernel in the Documentation directory and grep for loglevel. This brought me to this interesting bit from Documentation/sysctl/kernel.txt

==============================================================
printk:
The four values in printk denote: console_loglevel,
default_message_loglevel, minimum_console_loglevel and
default_console_loglevel respectively.
These values influence printk() behavior when printing or
logging error messages. See 'man 2 syslog' for more info on
the different loglevels.
- console_loglevel: messages with a higher priority than
  this will be printed to the console
- default_message_loglevel: messages without an explicit priority
  will be printed with this priority
- minimum_console_loglevel: minimum (highest) value to which
  console_loglevel can be set
- default_console_loglevel: default value for console_loglevel
==============================================================

So I ran 'cat /proc/sys/kernel/printk' and got (I managed to read it through the flood of messages from the firewall):

7       4       1       7

According to the explanations above, that meant that console_loglevel was too, so to fix it I ran:

echo '2 4 1 7' > /proc/sys/kernel/printk

And, behold, the serial console was usable.

25 January 2013

Tanguy Ortolo: Do not reCAPTCHA!

You probably know reCAPTCHA already: for the webmasters and the end user, it is an antispam system which asks you to read distorted words in order to prove that you are a human and not a spamming bot. This service has another end: instead of generating its distorted texts, it takes them from printed books to help digitalizing them.The problem In theory, this is an excellent idea, which should help preserving and distributing parts of the human culture. And, in the beginning, it was used in such a way, with reCAPTCHA participating in the Gutenberg project which digitalizes books from the public domain to offer them freely (as in free speech) to everyone. In 2009 however, reCAPTCHA was bought by Google. Now they do not participate in the Gutenberg project any more, instead they are digitalizing stuff from the New York Times and from Google Books. They do not provide much detail about that, but it does not seem that they distribute them in a free way afterwards . To put it boldly: they use people's brain power to digitalize books for their own exploitation monopoly, which is wrong. These books should be made available under a free license. Suggestion Do not help reCAPTCHA. If you are a webmaster, try to use another more ethical system. If you are an end user, and you are facing a reCAPTCHA test, avoid putting the exact answer but try inserting small mistakes (not big ones, which get detected and make you fail the test). And ask Google to free the books they digitalized using your brain. Notes

I may be wrong but I learnt to be suspicious in the context of digital culture. Please Google, if you do publish your books digitalized with reCAPTCHA under a free license, tell me, or rather, tell it to everyone and display it proudly in the reCAPTCHA website.

24 January 2013

Gunnar Wolf: The joys of Internet The Bolivarian dream

Oh, the joys of the Internet. A Mexican and an Argentinian listening to a Spanish cantaor singing Mexican music for an Argentinian audience, remembering a Costa Rican woman. What? Regina and me are finally back home in Mexico, after a month (me) and six weeks (her) of vacations in Argentina. And this week, in the city of Cosqu n (C rdoba, Argentina), they celebrate most important Argentinian folkloric festival. The Cosqu n Festival can be followed live on the TV P blica website. Right now, while I finish writing a short article and Regina fights her way to learn some of the GNOME 3 tricks, we are following Cosqu n. Among many great Argentinian folklorists, they invited a Spanish cantaor, David Palomar, who is remembering Chabela Vargas, a great singer, born in Costa Rica, but who became famous in Mexico, singing very heartfelt Mexican music, and deceased earlier this year. Trivia: Q: What do Mexico, Argentina and Spain have in common (besides a language that can be almost-understood)? A: They all have a city called C rdoba.

Dirk Eddelbuettel: Rcpp reaches 100 dependents on CRAN

With the arrival earlier today of the stochvol package onto the CRAN network for R, our Rcpp project reached a new milestone: 100 packages have either a Depends:, Imports: or LinkingTo: statement on it. The full list will always be at the bottom of the CRAN page for Rcpp; I also manually edit a list on my Rcpp page. But for the record as of today, here is the current list as produced by a little helper script I keep:

 acer                apcluster           auteur             
 bcp                 bfa                 bfp                
 bifactorial         blockcluster        ccaPP              
 cda                 classify            clusteval          
 ConConPiWiFun       EpiContactTrace     fastGHQuad         
 fdaMixed            forecast            fugeR              
 GeneticTools        gMWT                gof                
 gRbase              gRim                growcurves         
 GUTS                jaatha              KernSmoothIRT      
 LaF                 maxent              mets               
 minqa               mirt                mRMRe              
 multmod             mvabund             MVB                
 NetworkAnalysis     ngspatial           oem                
 openair             orQA                parser             
 pbdBASE             pbdDMAT             phom               
 phylobase           planar              psgp               
 quadrupen           Rchemcpp            Rclusterpp         
 RcppArmadillo       RcppBDT             rcppbugs           
 RcppClassic         RcppClassicExamples RcppCNPy           
 RcppDE              RcppEigen           RcppExamples       
 RcppGSL             RcppOctave          RcppRoll           
 RcppSMC             RcppXts             rforensicbatwing   
 rgam                RInside             Rmalschains        
 Rmixmod             robustgam           robustHD           
 rococo              RProtoBuf           RQuantLib          
 RSNNS               RSofia              rugarch            
 RVowpalWabbit       SBSA                sdcMicro           
 sdcTable            simFrame            spacodiR           
 sparseHessianFD     sparseLTSEigen      SpatialTools       
 stochvol            surveillance        survSNP            
 termstrc            tmg                 transmission       
 trustOptim          unmarked            VIM                
 waffect             WideLM              wordcloud          
 zic

And not to be forgotten is BioConductor which has another 10:

 ddgraph            GeneNetworkBuilder GOSemSim          
 GRENITS            mosaics            mzR               
 pcaMethods         Rdisop             Risa              
 rTANDEM

As developers of Rcpp, we are both proud and also a little humbled. The packages using Rcpp span everything from bringing new libraries to R, to implementing faster ways of doing things we have before to doing completely new things. It is an exciting time to be using R, and to be connecting R to C++, especially with so many exciting things happening in C++ right now. Follow the Rcpp links for more, and come join us on the Rcpp-devel mailing list to discuss and learn.

23 January 2013

Michal Čihař: Weblate 1.4

Weblate 1.4 has been released today. It comes with lot of improvements, especially in configurability, admin interface and usability. Full list of changes for 1.4:

Fixed deleting of checks/comments on unit deletion.
Added option to disable automatic propagation of translations.
Added option to subscribe for merge failures.
Correctly import on projects which needs custom ttkit loader.
Added sitemaps to allow easier access by crawlers.
Provide direct links to string in notification emails or feeds.
Various improvements to admin interface.
Provide hints for production setup in admin interface.
Added per language widgets and engage page.
Improved translation locking handling.
Show code snippets for widgets in more variants.
Indicate failing checks or fuzzy strings in progressbars.
More options for formatting commit message.
Fixed error handling with machine translation services.
Improved automatic translation locking behaviour.
Support for showing changes from previous source string.
Added support for substring search.
Various quality checks improvements.
Support for per project ACL.
Basic unit tests coverage.

You can find more information about Weblate on it's website, the code is hosted on Github. If you are curious how it looks, you can try it out on demo server. You can login there with demo account using demo password or register your own user. Ready to run appliances will be soon available in SUSE Studio Gallery. Weblate is also being used https://l10n.cihar.com/ as official translating service for phpMyAdmin, Gammu, Weblate itself and others. If you are free software project which would like to use Weblate, I'm happy to help you with set up or even host Weblate for you (this will be decided case by case as my hosting space is limited).

Filed under: English Phpmyadmin Suse Weblate 0 comments Flattr this!

Russ Allbery: Log parsing and infinite streams

I have a problem I have to solve for work that involves correlating Apache access and error logs. Part of WebAuth logs successful authentications to the Apache error log, and I want to correlate User-Agent strings with users so that we can figure out what devices our users are using by percentage of users rather than percentage of hits. The problem, as those who have tried to do this prior to Apache 2.4 know, is that Apache doesn't provide any easy way to correlate access and error log entries (made even more complex because two separate components are involved). I could have just hacked something together, but I've written way too many ad hoc log parsers, and, see, I was reading this book.... The book in question is Mark Jason Dominus's Higher Order Perl. I'm not quite done with it, and will post a full review when I am. I have some problems with it, mostly around the author's choice of example problems. But there is one chapter on infinite streams, and the moment I read that chapter on the train, it struck me as the perfect solution to log parsing problems. I'm not much of a functional programming person (which is where Dominus is drawing most of the material for this book), so I don't know if this terminology is standard or somewhat unique to the book. An infinite stream in this context is basically a variation on an interator that lets you look at the next item without consuming it. The power comes from putting this modified iterator in front of a generator and use it to consume one element at a time, and then compose this with transformation and filtering functions. That gives you all the power of the iterator to store state and lets you process one logical element from the stream at a time, without inverting your flow of control. Dominus provides code in the book for a very nice functional implementation of this that's about as close as you're probably going to get to writing Haskell in Perl. Unfortunately, his publisher decided to write their own free software license, so the license is kind of weird and doesn't explicitly permit redistribution of modified code. It's probably fine, but I didn't feel like dealing with it, and I'm more comfortable with writing object-oriented code at the moment (at least in Perl), so I decided to write an object-oriented version of the same code specific for log parsing. That's what I've been doing since shortly after lunch, and I can't remember the last time I've had this much fun writing code. I have a reasonable first cut at a (fully tested and fully test-driven) log parsing framework built on top of a reasonably generic implementation of the core ideas behind infinite streams. I also used this as an opportunity to experiment with Module::Build, and have discovered that the things I most disliked about it have apparently all been fixed. And I'm also using Perl 5.10 features. (I was tempted to start using some things from 5.12, but I do actually need to run this on Debian stable.) It's rather satisfying to write a thoroughly modern Perl module. There are some definite drawbacks to writing this in an object-oriented fashion. There's rather more machinery and glue that has to be set up, it's probably a bit slower, and it tends to accumulate layers of calls. One of the advantages of the method with standalone functions and a very simple, transparent data structure is that it's easier to eliminate unnecessary call nesting. But I suspect the object-oriented version will do what I want without any difficulties, and if I feel very inspired, I can always fiddle with it later. Maybe I'll eventually use this as a project to experiment with Moose as well. I'm surprised that no one else has done this, but I poked around on CPAN a fair bit and couldn't find anything. This will all show up on CPAN (as Log::Stream) as soon as I've finished enough of it to implement my sample application. And then I'll hopefully find some time to rewrite our metrics system using it, which should simplify it considerably....

Joey Hess: in Sydney

After arriving yesterday, and doing the obligatory jetlagged wander around Circular Quay to the Opera House, I crashed for 15 hours sleep and seem reasonably well over the jetlag now. I'm staying near Central Station, which is great because it's very convenient to transport and there's lots of cheap ethnic food.

view down my street

Today I took the ferry over to Manly Beach. I had to get in some in swimming of course, although today was cloudy and only in the 80's (F). That was ok, but the really nice bit was walking over to Shelly Beach and up along the sandstone cliffs near North Head. Highlight was finding a small gotto high on the cliff, with some beautiful sandstone filigree overhead, that seemed carved by water (but probably really by wind), and contained a shady bench overlooking the Pacific. Great place to enjoy a lunch of crusty bread and salami if you're ever in the area. Topped off a great day with a random trip to Neutral Bay (the D&Der in me liked the name, and why not, ferries are free) and a wander through the Royal Botanic Gardens. I'm hoping to spend either Thursday or Friday at a co-working or hackerspace in Sydney, but I don't know where yet. Any locals have ideas?

22 January 2013

Matthew Garrett: ITWire still has problems with basic accuracy

ITWire ran a story on the 12th of January entitled "Fedora still has problems with secure boot". It ends up discussing two issues with the author's experience with the Fedora installer - that he can't reclaim any free space on existing drives, and that the installer didn't automatically add entries for Windows to the grub menu. These are certainly legitimate issues, and I don't want to suggest that it's reasonable for people to have to manually alter their configuration in order to support dual boot. But they're not issues with Fedora's support for secure boot, despite the enthusiasm with which certain people have jumped on the story. We've received one credible report of a secure boot related problem with Fedora on a couple of Toshiba laptops, which appears (and I want to stress that we're still working on diagnosing it) to be a firmware bug rather than any kind of problem with Fedora.

Mistakes happen in journalism, and sometimes there are differences of opinion. But this story is simply wrong. When asked about it in the comments, the author failed to support his position. When contacted, the editor in chief was willing to add a note saying that I disputed the arguments but was unwilling to remove the incorrect claims. As a result, the internet remains full of links and reposts of an article that unashamedly tells users that the current Linux distribution with the best UEFI hardware support has issues with something it has no issues with.

For reasons that are unclear to me, ITWire seems to have some sort of well regarded status in the Australian technical industry. It seems entirely undeserved.

comments

Michal Čihař: phpMyAdmin translations status

With release of first alpha for phpMyAdmin 4.0, it's time to resurrect my regular translation status posts. So let's look at which translations are at 100% right now:

There are also several languages which need just few strings to be complete:

Filed under: English Phpmyadmin 0 comments Flattr this!

Julien Danjou: Extending Swift with middleware: example with ClamAV

In this article, I'm going to explain you how you can extend Swift, the OpenStack Object Storage project, so it performs extra action on files at upload or at download time. We're going to build an anti-virus filter inside Swift. The goal is to refuse uploaded data if they contain a virus. To help us with virus analyses, we'll use ClamAV. WSGI, paste and middleware

In a pipeline, a black cat can become a white cat with the help of some middleware.

To do our content analysis, the best place to hook in the Swift architecture is at the beginning of every request, on swift-proxy, before the file is actually stored on the cluster. Swift proxy uses, like many other OpenStack projects, paste to build his HTTP architecture. Paste uses WSGI and provides an architecture based on a pipeline. The pipeline is composed of a succession of middleware, ending with one application. Each middleware has the chance to look at the request or at the response, can modify it, and then pass it to the following middleware. The latest component of the pipeline is the real application, and in this case, the Swift proxy server. If you've already deployed Swift, you encountered a default pipeline in the swift-proxy.conf configuration file:

[pipeline:main]
pipeline = catch_errors healthcheck cache ratelimit tempauth proxy-logging proxy-server

This is a really basic pipeline with a few middleware. The first one catches error, the second one is in charge to return 200 OK response if you send a GET /healthcheck request on your proxy server. The third one is in charge of caching, the fourth one is used for rate limiting, the fifth for authentication, the sixth one for logging, and the final one is the actual proxy server, in charge of proxying the request to the account, container, or object servers (the others components of Swift). Of course, we could remove or add any of the middleware here at our convenience. Be aware that the order matters: for example, if you put healthcheck after tempauth, you won't be able to access the /healthcheck URL without being authenticated! ClamAV

If you don't know ClamAV, it's an antivirus engine designed for detecting trojans, viruses, malware and other malicious threats. Wwe're going to use it to scan every incoming file. To build the middleware, we'll use the Python binding pyclamd. The API is quite simple, see:

>>> import pyclamd
>>> pyclamd.init_unix_socket('/var/run/clamav/clamd.ctl')
>>> print pyclamd.scan_stream(pyclamd.EICAR)
 'stream': 'Eicar-Test-Signature(44d88612fea8a8f36de82e1278abb02f:68)' 
>>> print pyclamd.scan_stream("safe!")
None

Anatomy of a WSGI middleware Your WSGI middleware should consist of a callable object. Usually this is done with a class implementing the __call__ method. Here's a basic boilerplate:

class SwiftClamavMiddleware(object):
    """Middleware doing virus scan for Swift."""
 
    def __init__(self, app, conf):
        # app is the final application
        self.app = app
 
    def __call__(self, env, start_response):
        return self.app(env, start_response)
 

def filter_factory(global_conf, **local_conf):
    conf = global_conf.copy()
    conf.update(local_conf)
 
    def clamav_filter(app):
        return SwiftClamavMiddleware(app, conf)
    return clamav_filter

I'm not going to expand more on why this is built this way, but if you want to have more info on this kind of filter middleware, you can read their documentation on Paste. This middleware will just do nothing as it is. It's going to simply pass all requests it receives to the final application, and returns the result. Testing our basic middleware

Now is a really good time to add unit tests. I hope you didn't think we were going to write code without some tests, right? It's really easy to test a middleware, as we're going to use WebOb for that.

import unittest
from webob import Request
 

class FakeApp(object):
    def __call__(self, env, start_response):
        return Response(body="FAKE APP")(env, start_response)
 

class TestSwiftClamavMiddleware(unittest.TestCase):
 
    def setUp(self):
        self.app = SwiftClamavMiddleware(FakeApp(),  )
 
    def test_simple_request(self):
        resp = Request.blank('/',
                             environ= 
                                 'REQUEST_METHOD': 'GET',
                              ).get_response(self.app)
        self.assertEqual(resp.body, "FAKE APP")

We create a FakeApp class, that represents a fake WSGI application. You could also use a real application, or write a fake application looking like the one you want to test. It'll require more time, but your tests will be closer to the reality. Here we write the simplest test we can for our middleware. We're just sending a GET / request to it, so it passes the request to the final application and returns the result. It is transparent, it does nothing. Now, with that solid base we'll able to add more features and test these features incrementally. Plugging ClamAV in With our base ready, we can start thinking about how to plug ClamAV in. What we want to check here, is the content of the file when it's uploaded. If we refer to the OpenStack object storage API , a file upload is done via a PUT request, so we're going to limit the check to that kind of requests. Obviously, more checks could be added, but we'll keep things simple here for the sake of comprehensibility. With WSGI, the content of the request is available in env['wsgi.input'] as an object implementing a file interface. We'll scan that stream with ClamAV to check for viruses.

import pyclamd
from webob import Response
 

class SwiftClamavMiddleware(object):
    """Middleware doing virus scan for Swift."""
 
    def __init__(self, app, conf):
        pyclamd.init_unix_socket('/var/run/clamav/clamd.ctl')
        # app is the final application
        self.app = app
 
    def __call__(self, env, start_response):
        if env['REQUEST_METHOD'] == "PUT":
            # We have to read the whole content in memory because pyclamd
            # forces us to, but this is a bad idea if the file is huge.
            scan = pyclamd.scan_stream(env['wsgi.input'].read())
            if scan:
                return Response(status=403,
                                body="Virus %s detected" % scan['stream'],
                                content_type="text/plain")(env, start_response)
        return self.app(env, start_response)
 

def filter_factory(global_conf, **local_conf):
    conf = global_conf.copy()
    conf.update(local_conf)
 
    def clamav_filter(app):
        return SwiftClamavMiddleware(app, conf)
    return clamav_filter

That's it. We only check for PUT requests and if there's a virus in the file, we return a 403 Forbidden error with the name of the detected virus, bypassing entirely the rest of the middleware chain and the application handling. Then, we can simply test it.

import unittest
from cStringIO import StringIO
from webob import Request, Response
 

class FakeApp(object):
    def __call__(self, env, start_response):
        return Response(body="FAKE APP")(env, start_response)
 

class TestSwiftClamavMiddleware(unittest.TestCase):
    def setUp(self):
        self.app = SwiftClamavMiddleware(FakeApp(),  )
 
    def test_put_empty(self):
        resp = Request.blank('/v1/account/container/object',
                             environ= 
                                 'REQUEST_METHOD': 'PUT',
                              ).get_response(self.app)
        self.assertEqual(resp.body, "FAKE APP")
 
    def test_put_no_virus(self):
        resp = Request.blank('/v1/account/container/object',
                             environ= 
                                 'REQUEST_METHOD': 'PUT',
                                 'wsgi.input': StringIO('foobar')
                              ).get_response(self.app)
        self.assertEqual(resp.body, "FAKE APP")
 
    def test_put_virus(self):
        resp = Request.blank('/v1/account/container/object',
                             environ= 
                                 'REQUEST_METHOD': 'PUT',
                                 'wsgi.input': StringIO(pyclamd.EICAR)
                              ).get_response(self.app)
        self.assertEqual(resp.status_code, 403)

The first test test_put_empty simulates an empty PUT request. The second one, test_put_no_virus simulates a regular PUT request but with a simple file containing no virus. Finally, the third and last test simulates the upload of a virus using the EICAR test file. This is a special test file that is recognized as a virus, even if it's not real one. Very handy for testing virus detection software! Configuring Swift proxy Our middleware is ready! We can configure Swift's proxy server to use it. We need to add the following lines to our swift-proxy.conf to teach it how to load the filter:

[filter:clamav]
paste.filter_factory = swiftclamav:clamav_filter

We'll assume that our Python modules is named swiftclamava here. Now that we've defined our filter and how to load it, we can use it in our pipeline:

[pipeline:main]
pipeline = catch_errors healthcheck cache ratelimit tempauth clamav proxy-logging proxy-server

Just before reaching the proxy-server, and after the user being authenticated, the content will be scanned for viruses. It's important here to put this after authentication for example, because otherwise we may scan content that will get rejected by the authtemp module, thus scanning for nothing! Beyond scanning And voil , we now have a simple middleware testing uploaded content and refusing infected files. We could enhance it with various other things, like configuration handling, but I'll let that as an exercise for the interested readers. We didn't exploited it here, but note that you can also manipulate request headers and modify them if needed. For example, we could have added a header X-Object-Meta-Scanned-By: ClamAV to indicates that the file has been scanned by ClamAV. You should now be able to build your own middleware doing whatever you want with uploaded data. Happy hacking!

Matthew Palmer: When is a guess not a guess?

when it s a prediction . In the 4th January edition of the Guardian Weekly , the front page story, entitled Meet the world s new boomers ¹ contained this little gem:

Back in 2006, [PricewaterhouseCoopers] made some forecasts about what the global economy might look like in 2050, and it has now updated the predictions in the light of the financial crisis and its aftermath.

Delightful. They made some forecasts about what the global economy might look like. Given that they clearly didn t include any impact of the GFC in their forecasts, it clearly wasn t a particularly accurate forecast. Y know what an inaccurate prediction is called? Guesswork. Let s call a spade a spade here. I see this all the time, and it s starting to shit me. People making predictions and forecasts and projections hither and yon, and they re almost always complete bollocks, and they never get called on it. I read the Greater Fool blog now and then, and that blog is chock full of examples of people making predictions which have very little chance of being in any way accurate. While Dr Ben Goldacre and others are making inroads into requiring full disclosure in clinical trials, I m not aware of anyone taking a similar stand against charlatans making dodgy-as-hell predictions over and over again, with the sole purpose of getting attention, without any responsibility for the accuracy of those predictions. Is anyone aware of anyone doing work in this area, or do I need to register badpredictions.net and start calling out dodginess?

Sorry, I don t have a link, the whole site s behind a paywall. I read it on dead tree.

Eddy Petrișor: (Rsnapshot) backup and security - I see problems

In my previous post I was asking for suggestions for backup solutions that would be open/free software, do backups over the network to a local HDD, be cross platform to allow Windows and Linux clients and not be too CPU/memory hungry (on the server).

Several people suggested rsnapshot, BackupPC, areca-backup, and rsync. Thank you for all your suggestions, you have been a tremendous help. I have decided to give rsnapshot a try since it was suggested to me that it would actually do what is supposed to do for Windows clients, too (which was initially my perceived show stopper for rsnapshot).

Still, when getting to the implementation, I was a little disappointed by the very permissive access that needs to be provided on the client machines, since the backup is initiated from the backup server. Even the so called more secure suggested solutions seem way too permissive for my taste, since losing the control over the backup system means basically giving total access to the data from all client machines, which is quite a big problem in my opinion.

The data-transfer mechanism employed by rsnapshot is simply

S ==(connects and reads all data)==> C
S stores data in the final storage area

Am I the only one seeing a problem with this idea? If the server can connect to all your client machines and read all areas as it pleases, even if you restrict it to some directories, the data is already compromised when the backup server is compromised (think .ssh private keys, files with wireless network passwords and so on; I won't say card information - you don't keep credit/debit card information on your computer, or at least not in plain text, do you?).

What I would consider a better alternative would be a server-initiated dialogue which goes a little like this (S is server, C is client, '=' represent connections via ssh):

S ---(requests backup initiation procedure)---> C
S waits for a defined period of time that C connects back to send (already encrypted) data; if it doesn't arrive, it aborts
S <===(sends encrypted data to be backed up)=== C
S <-(signals the completion of data transfer)-- C
S stores the data in the final storage area

This way, the server can allow access from the clients only in designated areas (even a chroot is possible) from designated clients, access can even be provided only after a port knocking procedure and only during the backup time frame (since the server initiates the negotiation, it can expect only then the knocks, but only then), so the server is quite well secured. The connection to the server can even be done through an unprivileged account, it can even be one account per client machine which can be limited to a scponly shell, if you care for that level of security.

On the other hand, the client information is secure since it can be encrypted directly on the client machine and sent only after encryption, the client machine can decide and control what it sends, while the backup server can only store what the client provides. Also, if the server is compromised, the clients' data and system aren't compromised at all, since the data is on the backup machine, but is encrypted with a key only known on the client (and a backup copy of it can be stored somewhere safe).

I am aware this approach can be problematic for permission (user/group preservation), but it doesn't happen if there is a local <-> remote user mapping or simply the numeric IDs are kept.

I am also aware this means smarter clients and might mean the Windows machine might not be able to implement this completely, but a little more security than "here is all my data" can still be achieved, can't it?

What do other people think? Am I insane or paranoid?

I think I can implement this type of protocol in some scripts (at least one for server and one for clients) and use the backup_script feature of rsnapshot to keep this clean and nice within rsnapshot.

What might prove problematic with this approach is that rsync spedup is lost (might be?) because the copy is done to a temporary directory which, I assume, is empty, so tough luck. Another problem seems to be that every time the backup is done, the client has to encrypt each of the files to backup, which seems to be a real performance penalty, especially if the data to be backed up is quite large.

Is there an encryption layer that does this automatically at file level in the same/similar manner that LUKS does for entire block devices? Having the right file names, but with scrambled/encrypted contents seems to be the ideal solution from this PoV.

Thanks for reading and possible suggestions you might point me to.

P.S.: I just thought of this, if there was an encryption layer implemented with fuse which is mounted in some directory on the client machine, the default rsnapshot mechanism could actually work, and this would mitigate the data accessibility issue and the performance issue since that file system could be contained within a chroot and the encryption/scrambling would be done transparently on the client, so no data is plainly accessible. Does anybody know such a FUSE implementation that does on-the-fly file encryption?

P.P.S.: EncFS does exactly what I want with its --reverse option which is exactly designed for this purpose:

Normally EncFS provides a plaintext view of data on demand. Normally it stores enciphered data and displays plaintext data. With --reverse it takes as source plaintext data and produces enciphered data on-demand. This can be useful for creating remote encrypted backups, where you do not wish to keep the local files unencrypted.

Great!

19 January 2013

Dmitrijs Ledkovs: #takethestage

#stagetaken

I have now permanently settled in the UK.

Next.

Previous.