Search Results: "tar"

23 February 2016

Enrico Zini: pdftk-inkscape

In praise of pdftk and inkscape To appease the utterly evil, capricious and monstrous gods of Italian bureaucracy, I needed to craft several PDF files to sacrifice to the altar of the Italian Certified Email. Three of those PDFs (let's call them A.pdf, B.pdf and C.pdf) are copies of multi-page PDFs where I need to sign each page and then send them back. One of them (let's say C.pdf) needs to be signed on each page except the last, which needs instead to be signed twice in two special blank spaces. I have a transparent png with my signature (let's call it signature.png). Let's start with C.pdf: 
# Split C.pdf into single-page PDFs
pdftk C.pdf burst
# Edit a random page in inkscape, add a signature to the corner of the
    # page, delete the rest of the contents and save as signature.pdf
    inkscape pg_0001.pdf
# Edit the last page in inkscape, add the signatures where needed, save
    # as pg_0005-signed.pdf
    inkscape pg_0005.pdf
# Stamp the signatures all over all pages but the last one
for f in pg_000[1-4].pdf
do
    pdftk $f stamp signature.pdf output  basename $f .pdf -signed.pdf
done
# Put it all together
pdftk pg_000*-signed.pdf cat output C-signed.pdf
And now that I have signature.pdf, I can easily process A and B: 
pdftk A.pdf stamp signature.pdf output A-signed.pdf
pdftk B.pdf stamp signature.pdf output B-signed.pdf
pdftk is much smarted than that, and it would have been able to extract a single document with all pages of C.pdf except the last one, but with Italian bureaucracy it is painful enough to get the job done that afterwards one is more likely to need eating chocolate to avoid depression, rather than to try and redo it all again more elegantly. So, there are times when bureaucracy calls for printing a pile of sheets, mindlessly sign them all, scan them again, shred them, plant a new tree, and then finally book a MRI scan of your wrist to assess the state of your carpal tunnel. In those times, pdftk and inkscape can save the day.

Enrico Zini: In praise of pdftk and inkscape

To appease the utterly evil, capricious and monstrous gods of Italian bureaucracy, I needed to craft several PDF files to sacrifice to the altar of the Italian Certified Email. Three of those PDFs (let's call them A.pdf, B.pdf and C.pdf) are copies of multi-page PDFs where I need to sign each page and then send them back. One of them (let's say C.pdf) needs to be signed on each page except the last, which needs instead to be signed twice in two special blank spaces. I have a transparent png with my signature (let's call it signature.png). Let's start with C.pdf: 
# Split C.pdf into single-page PDFs
pdftk C.pdf burst
# Edit a random page in inkscape, add a signature to the corner of the
    # page, delete the rest of the contents and save as signature.pdf
    inkscape pg_0001.pdf
# Edit the last page in inkscape, add the signatures where needed, save
    # as pg_0005-signed.pdf
    inkscape pg_0005.pdf
# Stamp the signatures all over all pages but the last one
for f in pg_000[1-4].pdf
do
    pdftk $f stamp signature.pdf output  basename $f .pdf -signed.pdf
done
# Put it all together
pdftk pg_000*-signed.pdf cat output C-signed.pdf
And now that I have signature.pdf, I can easily process A and B: 
pdftk A.pdf stamp signature.pdf output A-signed.pdf
pdftk B.pdf stamp signature.pdf output B-signed.pdf
pdftk is much smarted than that, and it would have been able to extract a single document with all pages of C.pdf except the last one, but with Italian bureaucracy it is painful enough to get the job done that afterwards one is more likely to need eating chocolate to avoid depression, rather than to try and redo it all again more elegantly. So, there are times when bureaucracy calls for printing a pile of sheets, mindlessly sign them all, scan them again, shred them, plant a new tree, and then finally book a MRI scan of your wrist to assess the state of your carpal tunnel. In those times, pdftk and inkscape can save the day.

Steinar H. Gunderson: Multithreaded OpenGL driver quality

Multithreaded OpenGL is tricky, both for application programmers and drivers. Based on some recent experience with developing an application that would like to run on all three major desktop GPU vendors, allow me to present my survey with sample size 1. I'll let you draw your own conclusions: Curiously, somewhat similar to an Intel/Mesa bug I reported back in July, and which has no response yet. Update: The NVIDIA driver is now up to exposing three bugs in my code. One of them even came with a textual error message when running with a debug context (in apitrace).

Alessio Treglia: Analog thinking: the boldness to be creative

The binary code, although necessary for major social and technological developments, is annihilating the Homo Technologicus, stifling his innate freedom for analog connections.

Rodin1The first widespread use of the binary code was the Morse code alphabet. For decades, this communication system allowed to transmit information over long distances, between ships in the ocean and the mainland, between one continent and another, and today its use is still active in emergency situations. In its disarming simplicity (connect or interrupt two electrical wires), this system proved to be the shortest way to transcribe an alphabet.

Then, the Boolean algebra (binary) represented the language of communication between humans and computers, enabling the translation of instructions and commands towards the processing unit (microprocessor / CPU). This determined the Big Bang of the computers and the technological era

<Read More...>

Uwe Kleine-K nig: Christoph Hellwig vs VMware

On Thursday the lawsuit about GPL compliance of VMware's ESXi products starts. I support Christoph's position and share his view that VMware's product is in conflict with the GPL. Read more on the software freedom conservancy's FAQ about the case. I keep my fingers crossed!

Michal &#268;iha&#345;: Weekly phpMyAdmin contributions 2016-W07

As the flow of incoming bugs for upcoming 4.6.0 has slowed down a bit it was more time for code cleanups and related tasks. But it's also time where potential Google Summer of Code students come to our organization and want to get involved. On the cleanup side the biggest was change to remove embedded PHP libraries which are available on Packagist from our Git and use Composer to manage the dependencies. This change will happen in 4.7.0, so it's still some time ahead, but it's already in our master branch. There still some third party libraries which we use and can not be installed using Composer, so we keep these for now. Besides the usual bug fixing stuff, I've noticed that we lack issues which can be easily understood and fixed by potential Google Summer of Code. We require them to get involved before the program starts, so that we can see they are capable of useful contributions and also to see how they behave if asked for patch improvements. To fix this deficit we're prepared few small cleanup or refactoring tasks, where the students can show their skills. All handled issues:

Filed under: English phpMyAdmin 2 comments

21 February 2016

Mario Lang: Generating C++ from a DTD with Jinja2 and lxml

I recently stumbled across an XML format specified in a DTD that I wanted to work with from within C++. The XML format is document centric, which is a bit of a pain with existing data binding compilers according to my limited experience. So to learn something new, and to keep control over generated code, I started to investigate what it would take to write my own little custom data binding compiler.
Writing a program that writes a program It turns out that there are two very helpful libraries in Python which can really make your life a lot easier:
To keep my life simple, I am focusing on generating accessors for XML attributes only for now. I leave it up to the library client to figure out how to deal with child elements.
A highly simplified DOM Inspired by the hybrid example from libstudxml, we define a simple base class that can store raw XML elements. 
class element  
public:
  using attributes_type = std::map<xml::qname, std::string>;
  using elements_type = std::vector<std::shared_ptr<element>>;
  element(const xml::qname& name) : tag_name_(name)  
  virtual ~element() = default;
  xml::qname const& tag_name() const   return tag_name_;  
  attributes_type const& attributes() const   return attributes_;  
  attributes_type&       attributes()         return attributes_;  
  std::string const& text() const   return text_;  
  void text(std::string const& text)   text_ = text;  
  elements_type const& elements() const  return elements_; 
  elements_type&       elements()         return elements_;  
  element(xml::parser&, bool start_end = true);
  void serialize (xml::serializer&, bool start_end = true) const;
  template<typename T> static std::shared_ptr<element> create(xml::parser& p)  
    return std::make_shared<T>(p, false);
   
private:
  xml::qname tag_name_;
  attributes_type attributes_;
  std::string text_;           // Simple content only.
  elements_type elements_;     // Complex content only.
 ;
For each element name in the DTD, we're going to define a class that inherits from the element class, implementing special methods to make attribute access easier. The element(xml::parser&) constructor is going to create the corresponding class whenever it sees a certain element name. This calls for some sort of factory: 
class factory  
public:
  static std::shared_ptr<element> make(xml::parser& p);
protected:
  struct element_info  
    xml::content content_type;
    std::shared_ptr<element> (*construct)(xml::parser&);
   ;
  using map_type = std::map<xml::qname, element_info>;
  static map_type *get_map()  
    if (!map) map = new map_type;
    return map;
   
private:
  static map_type *map;
 ;
template<typename T>
struct register_element : factory  
  register_element(xml::qname const& name, xml::content const& content)  
    get_map()->insert( name, element_info content, &element::create<T> );
   
 ;
shared_ptr<element> factory::make(xml::parser& p)  
  auto name = p.qname();
  auto iter = get_map()->find(name);
  if (iter == get_map()->end())  
    // No subclass found, so store plain data so we do not loose on roundtrip.
    return std::make_shared<element>(p, false);
   
  auto const& element = iter->second;
  p.content(element.content_type);
  return element.create(p);
The header template Now that we have our required infrastructure, we can finally start writing Jinja2 templates to generate classes for all elements in our DTD: 
 %- for elem in dtd.iterelements() % 
   %- if elem.name in forwards_for % 
     %- for forward in forwards_for[elem.name] % 
class  forward ;
     %- endfor % 
   %- endif % 
class  elem.name  : public dom::element  
  static register_element< elem.name > factory_registration;
public:
   elem.name (xml::parser& p, bool start_end = true) : dom::element(p, start_end)  
   
   %- for attr in elem.iterattributes() % 
     %- if attr is required_string_attribute % 
  std::string  attr.name () const;
  void  attr.name (std::string const&);
     %- elif attr is implied_string_attribute % 
  optional<std::string>  attr.name () const;
  void  attr.name (optional<std::string>);
     # more branches to go here # 
     %- endif % 
   %- endfor % 
 ;
 %- endfor %
required_string_attribute and implied_string_attribute are so-called Jinja2 tests. They are a nice way to isolate predicates such that the Jinja2 templates can stay relatively free of complicated expressions: 
templates.tests['required_string_attribute'] = lambda a: \
  a.type in ['id', 'cdata', 'idref'] and a.default == 'required'
templates.tests['implied_string_attribute'] = lambda a: \
  a.type in ['id', 'cdata', 'idref'] and a.default == 'implied'
That is nice, but we have only seen C++ header declarations so far. Lets have a look into the implementation of some of our attribute accessors.
Enum conversion One interesting aspect of DTD based code generation is the fact that attributes can have enumerations specified. Assume that we have some extra data-structure in Python which helps us to define a nice name for each individual enumeration attribute. Then, a part of the Jinja2 template to generate the implementation for an enumeration attribute looks like: 
     %- elif attr is known_enumeration_attribute % 
       %- set enum = enumerations[tuple(attr.values())]['name'] % 
       %- if attr.default == 'required' % 
 enum   elem.name :: attr.name () const  
  auto iter = attributes().find(qname " attr.name " );
  if (iter != attributes().end())  
         %- for value in attr.values() % 
     % if not loop.first % else  % else %       % endif -% 
    if (iter->second == " value ") return  enum :: value   mangle ;
         %- endfor % 
    throw illegal_enumeration ;
   
  throw missing_attribute ;
 
void  elem.name :: attr.name ( enum  value)  
  static qname const attr " attr.name " ;
  switch (value)  
         %- for value in attr.values() % 
  case  enum :: value   mangle :
    attributes()[attr] = " value ";
    break;
         %- endfor % 
  default:
    throw illegal_enumeration ;
   
 
       %- elif attr.default == 'implied' % 
 # similar implementation using boost::optional # 
       %- endif % 
     %- endif %
Putting it all together The header for the library is generated like this: 
from jinja2 import DictLoader, Environment
from lxml.etree import DTD
LIBRARY_HEADER = """
 # Our template code # 
"""
bmml = DTD('bmml.dtd')
templates = Environment(loader=DictLoader(globals()))
templates.filters['mangle'] = lambda ident: \
   '8th_or_128th': 'eighth_or_128th',
   '256th': 'twohundredfiftysixth',
   'continue': 'continue_'
   .get(ident, ident)
def template(name):
  return templates.get_template(name)
def hpp():
  print(template('LIBRARY_HEADER').render(
     'dtd': bmml,
     'enumerations': enumerations,
     'forwards_for':  'ornament': ['ornament_type'],
                      'score': ['score_data', 'score_header'] 
     ))
With all of this in place, we can have a look at a small use case for our library.
Printing document content I haven't really explained anything about the document format we're working with until now. Braille Music Markup Language is an XML based plain text markup language. Its purpose is to be able to enhance plain braille music scores with usually hard-to-calcuate meta information. Almost all element text content is supposed to be printed as-is to reconstruct the original plain text. So we could at least define one very basic operation in our library: printing the plain text content of an element. I found an XML stylesheet that is supposed to convert BMML documents to HTML. This stylesheet apparently has a bug, insofar as it forgets to treat the rest_data element in the same way as it already treats the note_data element. note to self, I wish I would've done a code review before the EU-project that developed BMML was finished. It looks like resurrecting maintainance is one of the things I might be able to look into in a meeting in Pisa in the first three days of March this year. If we keep this in mind, we can easily reimplement what the stylesheet does in idiomatic C++: 
template<typename T>
typename std::enable_if<std::is_base_of<element, T>::value, std::ostream&>::type
operator<<(std::ostream &out, std::shared_ptr<T> elem)  
  if (!std::dynamic_pointer_cast<note_data>(elem) &&
      !std::dynamic_pointer_cast<rest_data>(elem) &&
      !std::dynamic_pointer_cast<score_header>(elem))
   
    auto const& text = elem->text();
    if (text.empty()) for (auto child : *elem) out << child; else out << text;
   
  return out;
The use of std::enable_if is necessary here so that operator<< is defined on the element class and all of its subclasses. Without the std::enable_if magic, client code would be forced to manually make sure it is passing std::shared_ptr<element> each time it wants to use the operatr<< on any of our specially defined subclasses. Now we can easily print BMML documents and get their actual plain text representation. 
#include <fstream>
#include <iostream>
#include <xml/parser>
#include <xml/serializer>
#include "bmml.hxx"
using namespace std;
using namespace xml;
int main (int argc, char *argv[])  
  if (argc < 2)  
    cerr << "usage: " << argv[0] << " [<filename.bmml>...]" << endl;
    return EXIT_FAILURE;
   
  try  
    for (int i = 1; i < argc; ++i)  
      ifstream ifs argv[i] ;
      if (ifs.good())  
        parser p ifs, argv[i] ;
        p.next_expect(parser::start_element, "score", content::complex);
        cout << make_shared<bmml::score>(p, false) << endl;
        p.next_expect(parser::end_element, "score");
        else  
        cerr << "Unable to open '" << argv[i] << "'." << endl;
        return EXIT_FAILURE;
       
     
    catch (xml::exception const& e)  
    cerr << e.what() << endl;
    return EXIT_FAILURE;
That's it for now. The full source for the actual library which inspired this posting can be found on github in my bmmlcxx project. If you have an comments or questions, send me mail. If you like bmmlcxx, don't forget to star it :-).

Mario Lang: Generating C++ from a DTD with Jinja2 and lxml

I recently stumbled across an XML format specified in a DTD that I wanted to work with from within C++. The XML format is document centric, which is a bit of a pain with existing data binding compilers according to my limited experience. So to learn something new, and to keep control over generated code, I started to investigate what it would take to write my own little custom data binding compiler.
Writing a program that writes a program It turns out that there are two very helpful libraries in Python which can really make your life a lot easier:
To keep my life simple, I am focusing on generating accessors for XML attributes only for now. I leave it up to the library client to figure out how to deal with child elements.
A highly simplified DOM Inspired by the hybrid example from libstudxml, we define a simple base class that can store raw XML elements. 
class element  
public:
  using attributes_type = std::map<xml::qname, std::string>;
  using elements_type = std::vector<std::shared_ptr<element>>;
  element(const xml::qname& name) : tag_name_(name)  
  virtual ~element() = default;
  xml::qname const& tag_name() const   return tag_name_;  
  attributes_type const& attributes() const   return attributes_;  
  attributes_type&       attributes()         return attributes_;  
  std::string const& text() const   return text_;  
  void text(std::string const& text)   text_ = text;  
  elements_type const& elements() const  return elements_; 
  elements_type&       elements()         return elements_;  
  element(xml::parser&, bool start_end = true);
  void serialize (xml::serializer&, bool start_end = true) const;
  template<typename T> static std::shared_ptr<element> create(xml::parser& p)  
    return std::make_shared<T>(p, false);
   
private:
  xml::qname tag_name_;
  attributes_type attributes_;
  std::string text_;           // Simple content only.
  elements_type elements_;     // Complex content only.
 ;
For each element name in the DTD, we're going to define a class that inherits from the element class, implementing special methods to make attribute access easier. The element(xml::parser&) constructor is going to create the corresponding class whenever it sees a certain element name. This calls for some sort of factory: 
class factory  
public:
  static std::shared_ptr<element> make(xml::parser& p);
protected:
  struct element_info  
    xml::content content_type;
    std::shared_ptr<element> (*construct)(xml::parser&);
   ;
  using map_type = std::map<xml::qname, element_info>;
  static map_type *get_map()  
    if (!map) map = new map_type;
    return map;
   
private:
  static map_type *map;
 ;
template<typename T>
struct register_element : factory  
  register_element(xml::qname const& name, xml::content const& content)  
    get_map()->insert( name, element_info content, &element::create<T> );
   
 ;
shared_ptr<element> factory::make(xml::parser& p)  
  auto name = p.qname();
  auto iter = get_map()->find(name);
  if (iter == get_map()->end())  
    // No subclass found, so store plain data so we do not loose on roundtrip.
    return std::make_shared<element>(p, false);
   
  auto const& element = iter->second;
  p.content(element.content_type);
  return element.create(p);
The header template Now that we have our required infrastructure, we can finally start writing Jinja2 templates to generate classes for all elements in our DTD: 
 %- for elem in dtd.iterelements() % 
   %- if elem.name in forwards_for % 
     %- for forward in forwards_for[elem.name] % 
class  forward ;
     %- endfor % 
   %- endif % 
class  elem.name  : public dom::element  
  static register_element< elem.name > factory_registration;
public:
   elem.name (xml::parser& p, bool start_end = true) : dom::element(p, start_end)  
   
   %- for attr in elem.iterattributes() % 
     %- if attr is required_string_attribute % 
  std::string  attr.name () const;
  void  attr.name (std::string const&);
     %- elif attr is implied_string_attribute % 
  optional<std::string>  attr.name () const;
  void  attr.name (optional<std::string>);
     # more branches to go here # 
     %- endif % 
   %- endfor % 
 ;
 %- endfor %
required_string_attribute and implied_string_attribute are so-called Jinja2 tests. They are a nice way to isolate predicates such that the Jinja2 templates can stay relatively free of complicated expressions: 
templates.tests['required_string_attribute'] = lambda a: \
  a.type in ['id', 'cdata', 'idref'] and a.default == 'required'
templates.tests['implied_string_attribute'] = lambda a: \
  a.type in ['id', 'cdata', 'idref'] and a.default == 'implied'
That is nice, but we have only seen C++ header declarations so far. Lets have a look into the implementation of some of our attribute accessors.
Enum conversion One interesting aspect of DTD based code generation is the fact that attributes can have enumerations specified. Assume that we have some extra data-structure in Python which helps us to define a nice name for each individual enumeration attribute. Then, a part of the Jinja2 template to generate the implementation for an enumeration attribute looks like: 
     %- elif attr is known_enumeration_attribute % 
       %- set enum = enumerations[tuple(attr.values())]['name'] % 
       %- if attr.default == 'required' % 
 enum   elem.name :: attr.name () const  
  auto iter = attributes().find(qname " attr.name " );
  if (iter != attributes().end())  
         %- for value in attr.values() % 
     % if not loop.first % else  % else %       % endif -% 
    if (iter->second == " value ") return  enum :: value   mangle ;
         %- endfor % 
    throw illegal_enumeration ;
   
  throw missing_attribute ;
 
void  elem.name :: attr.name ( enum  value)  
  static qname const attr " attr.name " ;
  switch (value)  
         %- for value in attr.values() % 
  case  enum :: value   mangle :
    attributes()[attr] = " value ";
    break;
         %- endfor % 
  default:
    throw illegal_enumeration ;
   
 
       %- elif attr.default == 'implied' % 
 # similar implementation using boost::optional # 
       %- endif % 
     %- endif %
Putting it all together The header for the library is generated like this: 
from jinja2 import DictLoader, Environment
from lxml.etree import DTD
LIBRARY_HEADER = """
 # Our template code # 
"""
bmml = DTD('bmml.dtd')
templates = Environment(loader=DictLoader(globals()))
templates.filters['mangle'] = lambda ident: \
   '8th_or_128th': 'eighth_or_128th',
   '256th': 'twohundredfiftysixth',
   'continue': 'continue_'
   .get(ident, ident)
def template(name):
  return templates.get_template(name)
def hpp():
  print(template('LIBRARY_HEADER').render(
     'dtd': bmml,
     'enumerations': enumerations,
     'forwards_for':  'ornament': ['ornament_type'],
                      'score': ['score_data', 'score_header'] 
     ))
With all of this in place, we can have a look at a small use case for our library.
Printing document content I haven't really explained anything about the document format we're working with until now. Braille Music Markup Language is an XML based plain text markup language. Its purpose is to be able to enhance plain braille music scores with usually hard-to-calcuate meta information. Almost all element text content is supposed to be printed as-is to reconstruct the original plain text. So we could at least define one very basic operation in our library: printing the plain text content of an element. I found an XML stylesheet that is supposed to convert BMML documents to HTML. This stylesheet apparently has a bug, insofar as it forgets to treat the rest_data element in the same way as it already treats the note_data element. note to self, I wish I would've done a code review before the EU-project that developed BMML was finished. It looks like resurrecting maintainance is one of the things I might be able to look into in a meeting in Pisa in the first three days of March this year. If we keep this in mind, we can easily reimplement what the stylesheet does in idiomatic C++: 
template<typename T>
typename std::enable_if<std::is_base_of<element, T>::value, std::ostream&>::type
operator<<(std::ostream &out, std::shared_ptr<T> elem)  
  if (!std::dynamic_pointer_cast<note_data>(elem) &&
      !std::dynamic_pointer_cast<rest_data>(elem) &&
      !std::dynamic_pointer_cast<score_header>(elem))
   
    auto const& text = elem->text();
    if (text.empty()) for (auto child : *elem) out << child; else out << text;
   
  return out;
The use of std::enable_if is necessary here so that operator<< is defined on the element class and all of its subclasses. Without the std::enable_if magic, client code would be forced to manually make sure it is passing std::shared_ptr<element> each time it wants to use the operatr<< on any of our specially defined subclasses. Now we can easily print BMML documents and get their actual plain text representation. 
#include <fstream>
#include <iostream>
#include <xml/parser>
#include <xml/serializer>
#include "bmml.hxx"
using namespace std;
using namespace xml;
int main (int argc, char *argv[])  
  if (argc < 2)  
    cerr << "usage: " << argv[0] << " [<filename.bmml>...]" << endl;
    return EXIT_FAILURE;
   
  try  
    for (int i = 1; i < argc; ++i)  
      ifstream ifs argv[i] ;
      if (ifs.good())  
        parser p ifs, argv[i] ;
        p.next_expect(parser::start_element, "score", content::complex);
        cout << make_shared<bmml::score>(p, false) << endl;
        p.next_expect(parser::end_element, "score");
        else  
        cerr << "Unable to open '" << argv[i] << "'." << endl;
        return EXIT_FAILURE;
       
     
    catch (xml::exception const& e)  
    cerr << e.what() << endl;
    return EXIT_FAILURE;
That's it for now. The full source for the actual library which inspired this posting can be found on github in my bmmlcxx project. If you have an comments or questions, send me mail. If you like bmmlcxx, don't forget to star it :-).

Vincent Sanders: Stack 'em, pack 'em and rack 'em.

As you may be aware I have a bit of a problem with Single Board Computers in that I have a lot of them. Keeping them organised has turned into a bit of a problem.

cluttered shelf of SBC
I designed clip cases for many of these systems giving me a higher storage density on my rack shelves and built a power supply to reduce the cabling complexity. These helped but I still ended up with a cluttered shelf full of SBC.

I decided I would make a rack enclosure to hold the SBC, I was limited to material I could easily CNC machine which limited me to acrylic plastics or wood.

laser cutting the design, viewed through heavily tinted filterInitially I started with the idea of housing the individual boards in a toast rack arrangement. This would mean that the enclosure would have to be at least 2U high to fit the boards all the existing cases would have to be discarded. This approach was dropped when the drawbacks of having no flexibility and only being able to fit the units that were selected at design time became apparent (connector cutouts and mounting hole placement.

Instead I changed course to try and incorporate the existing cases which already solved the differing connector and mounting placement problem and gave me a uniform size to consider. Once I had this approach the design came pretty quickly. I used a tube girder construction 1U in height to get as much strength as possible from the 3mm acrylic plastic I would use.

laser cut pieces arranged for assembly still with protective film on
The design was simply laser cut from sheet stock and fastened together with M3 nut and bolts. Once I corrected the initial design errors (I managed to get almost every important dimension wrong on the first attempt) the result was a success.

working prototype resting on initial version
The prototype is a variety of colours because makespace ran out of suitably sized clear acrylic stock but the colouring has no effect on the result other than aesthetical. The structure gives a great deal of rigidity and there is no sagging or warping, indeed testing on the prototype got to almost 50Kg loading without a failure (one end clamped and the other end loaded at 350mm distance)

I added some simple rotating latches at the front which keep the modules held in place and allow units to be removed quickly if necessary.

rack slots installed and in use
Overall this project was successful and I can now pack five SBC per U neatly. It does limit me to using systems cased in my "slimline" designs (68x30x97mm) which currently means the Raspberry Pi B+ style and the Orange Pi PC.

Once small drawback is access to I/O and power connectors. These need to be right angled and must be unplugged before unit removal which can be a little fiddly. Perhaps a toast rack design of cases would have given easier connector access but I am happy with this trade off of space for density.

As usual the design files are freely available, perhaps they could be useful as a basis for other laser cut rack enclosure designs.

20 February 2016

Mark Brown: Expedient ABIs

The biggest change we ve seen in the Linux kernel for ARM over the past few years has been the transition to providing descriptions of the hardware in systems via device tree. This splits out the description of the devices in the system that can t be automatically enumerated from the kernel into a separate binary instead of being part of the kernel binary. Currently for most systems that are actively used upstream the device tree source code is kept in the kernel but the goal is to allow people to use device trees that are distributed separately to the kernel, especially device trees that are shipped as part of the board firmware. This is something that other platforms have done for a long time, PowerPC Macs and Sun SPARC systems use device tree as the mechanism for describing the hardware to the operating system. One consequence of this desire to allow the kernel and device tree to be shipped separately is that the device tree becomes an ABI. This is a really big change for people working in the embedded and consumer electronics areas where ARM has been most widely deployed, it means that any descriptions of the hardware need to be something that can stand the test of time. Anything we release is something we have to expect to carry code for indefinitely. When everything was done as part of the kernel binary we could easily do something that doesn t quite represent the hardware with the intention of replacing it later, now it is much harder to do that. An example of this is the SAW in Qualcomm SoCs. This is a block in the SoC which provides control of some regulators used for the CPU cores in the PMIC in very low power states and also allows the CPU to control the regulator with fast memory mapped registers rather than the slower buses used to control the PMIC. Unfortunately it doesn t fully replace direct access to the PMIC, it supports a subset of the control we need for the PMIC but not all of it. We could represent the SAW as an independent regulator but from a system integration point of view it is functioning as an extra control interface for the external PMIC and if we want to use the extra functionality that is only available via direct access to the PMIC we need to consider that and represent the SAW as an extension of it. If we don t need that extra PMIC functionality at the current time this means we need to do some extra work to make sure we describe the PMIC before we can use the SAW even if we have no intention to use anything other than the SAW. Now, few if any people are actually using the device tree as an ABI at present so those working on enabling platforms often forget about the requirement and find it an obstacle to getting things done they have pressure to get things done, they don t have quite the same pressure to make sure that attention is paid to device tree compatibility so it can easily get forgotten. Over time this may change, especially if people start to take advantage of the device tree as an ABI that become more and more important, but for now if we want to enable that in the future it s something we have to actively think about and work on, accepting that this means we won t always be able to do the most expedient thing.

19 February 2016

Joey Hess: Ubuntu trademark nonsense

Canonical appear to require that you remove all trademarks entirely even if using them wouldn't be a violation of trademark law.
-- Matthew Garrett Each time Matthew brings this up, and as evidence continues to mount that Canonical either actually intends their IP policy to be read that way, or is intentionally keeping the situation unclear to FUD derivatives, I start wondering about references to Ubuntu in my software. Should such references be removed, or obscured, like "U*NIX" in software of old, to prevent exposing users to this trademark nonsense? 
joey@darkstar:~/src/git-annex>git grep -i ubuntu  wc -l
457
joey@darkstar:~/src/ikiwiki>git grep -i ubuntu  wc -l
80
joey@darkstar:~/src/etckeeper>git grep -i ubuntu  wc -l
14
Most of the code in git-annex, ikiwiki, and etckeeper is licensed under the GPL or AGPL, and so Canonical's IP policy probably does not require that anyone basing a distribution on Ubuntu strip all references to "Ubuntu" from them. But then, there's Propellor: 
joey@darkstar:~/src/propellor>git grep -i ubuntu  wc -l
10
Propellor is BSD licensed. It's in Ubuntu universe. It not only references Ubuntu in documentation, but contains code that uses that trademark: 
data Distribution
        = Debian DebianSuite
          Ubuntu Release
So, if an Ubuntu-derived distribution has to remove "Ubuntu" from Propellor, they'd end up with a Propellor that either differs from upstream, or that can't be used to manage Ubuntu systems. Neither choice is good for users. Probably most small derived distributions would not have expertise to patch data types in a Haskell program and would have to skip including Propellor. That's not good for Propellor getting wide distribution either. I think I've convinced myself it would be for the best to remove all references to "Ubuntu" from Propellor.
done Similarly, Debconf is BSD licensed. I originally wrote it, but it's now maintained by Colin Watson, who works for Canonical. If I were still maintaining Debconf, I'd be looking at removing all instances of "Ubuntu" from it and preventing that and other Canonical trademarks from slipping back in later. Alternatively, I'd be happy to re-license all Debconf code that I wrote under the AGPL-3+. Update: Another package that comes to mind is Debootstrap, which is also BSD licensed. Of course it contains "Ubuntu" in lots of places, since it is how Ubuntu systems are built. I'm no longer an active developer of Debootstrap, but I hope its current developers carefully consider how this trademark nonsense affects it. PS: Shall we use "*buntu" as the, erm, canonical trademark-free spelling of "Ubuntu"? Seems most reasonable, unless Canonical has trademarked that too.

18 February 2016

Enrico Zini: gui-debugging-output

Debugging output of GUI apps I have been hit by #643726 again. I was happily working on a terminal while an apt upgrade was running on another, and suddenly kbuildsycoca4 started vomiting on my command line because I had run okular once on that terminal last tuesday. Among the possible workarounds, one can run kdebugdialog from package kde-runtime and check the "Disable all debug output" button. This is the output of kdebugdialog when run on a terminal after enabling "Disable all debug output": 
$ kdebugdialog
QDBusConnection: session D-Bus connection created before QCoreApplication. Application may misbehave.
QDBusConnection: session D-Bus connection created before QCoreApplication. Application may misbehave.
I see it as debugging output turned into an art performance, and I have an appreciation for this kind of things. kdebugdialog, my hat's off to you.

Enrico Zini: gui-debugging-output

Debugging output of GUI apps I have been hit by #643726 again. I was happily working on a terminal while an apt upgrade was running on another, and suddenly kbuildsycoca4 started vomiting on my command line because I had run okular once on that terminal last tuesday. Among the possible workarounds, one can run kdebugdialog from package kde-runtime and check the "Disable all debug output" button. This is the output of kdebugdialog when run on a terminal after enabling "Disable all debug output": 
$ kdebugdialog
QDBusConnection: session D-Bus connection created before QCoreApplication. Application may misbehave.
QDBusConnection: session D-Bus connection created before QCoreApplication. Application may misbehave.
I see it as debugging output turned into an art performance, and I have an appreciation for this kind of things. kdebugdialog, my hat's off to you.

Enrico Zini: Debugging output of GUI apps

I have been hit by #643726 again. I was happily working on a terminal while an apt upgrade was running on another, and suddenly kbuildsycoca4 started vomiting on my command line because I had run okular once on that terminal last tuesday. Among the possible workarounds, one can run kdebugdialog from package kde-runtime and check the "Disable all debug output" button. This is the output of kdebugdialog when run on a terminal after enabling "Disable all debug output": 
$ kdebugdialog
QDBusConnection: session D-Bus connection created before QCoreApplication. Application may misbehave.
QDBusConnection: session D-Bus connection created before QCoreApplication. Application may misbehave.
I see it as debugging output turned into an art performance, and I have an appreciation for this kind of things. kdebugdialog, my hat's off to you.

Stein Magnus Jodal: A guide to poor API management

This is the story of libspotify, as experienced by a Spotify customer and libspotify developer for six years.

Step 1: Support your API for years Since April 2009, libspotify has been a mostly nice although proprietary C library for integrating with the Spotify music streaming service, providing both music metadata and full playback capabilities. Language wrappers have been written for a plenitude of programming languages (including my own pyspotify). libspotify is integrated into numerous open source projects (including my own Mopidy), networked AV receivers, and rumour has it: even cars. In addition to being closed source, there was another catch: in order to use it, you need the Spotify premium subscription. Speaking from my own experience supporting a project integrating with Spotify for the last six years: lots of end users upgraded to premium in order to use the projects built on top of libspotify.

Step 2: Stop accepting bug reports and stop releasing updates In May 2012, Spotify released libspotify 12.1.51 for all supported architectures, which now even included Android. After months of endless requests from the quickly growing hoards of Raspberry Pi users, Spotify released libspotify 12.1.103 for hardfloat armv6 on January 22 2013. The release included minor API additions, like the addition of explicit lyrics and means for accepting Spotify s Terms of Service. The additions in 12.1.103 never reached the other supported architectures and 12.1.51 remains the last release for those. There was no clear communication about this being the end of libspotify, releases have just ceased. libspotify has not been mentioned in a single developer blog post or developer email newsletter since. That s now more than three years ago.

Step 3: Create new APIs Over the next couple of years Spotify released a new Web API and iOS and Android SDKs. Through their developer blog, they communicated quite a bit both about these and about the deprecation of the apps inside the Spotify desktop client. Here are some pieces from their developer blog to illustrate: The Web API grew steadily, as documented by the Web APIs changelog. Over time, its feature list improved, supporting anonymous endpoints for generic music metadata as well as OAuth protected endpoints for managing your own music collection and playlists. The only major part missing that prevented developers replacing libspotify was music playback, as the Web API only provides 30 second previews. From what I recall of 2014 IRC discussions with Spotify employees, they had built a new much smaller library for streaming music. The iOS and Android SDKs were simply platform specific wrappers around this new playback library and the new Web API. My impression is that, after a number of beta releases, the iOS and Android SDKs seamlessly replaced libspotify on those platforms. The plan was apparently to release the new playback library for desktop too, so that it, in combination with the Web API, could replace libspotify there as well. That was all back in 2014.

Step 4: Deprecate old APIs At some point, Spotify deprecated their old web API, the Metadata API. There was no blog post about it and it was not mentioned in any Spotify Developer News email I ve received in recent years. I guess they only published this fact as a note on the Metadata API web page; it is therefore hard to track down exactly when it was deprecated and how the deprecation was communicated. Anyway, it was deprecated, and the natural upgrade path was to the new Web API. Spotify even has a migration guide mapping the concepts of the Metadata API to the new Web API. All good. In May 2015, after 28 months without a single release or any news, libspotify is officially deprecated. The communication channel of choice? A note on the libspotify web page. No post on the developer blog. No Spotify Developer News email. To be able to use libspotify, all developers have registered and requested a personal application key. To my knowledge, none of these registered library users were notified directly.

Step 5: Shut down an old API On January 20 2016 Spotify end-of-lifed their Metadata API. I don t know if this date was communicated on the Metadata API web page well in advance of this shutdown. There s no blog post on the topic. There s no news email. There s no tweet from @SpotifyPlatform or @SpotifyStatus. I can t find anything except users commenting on various Spotify documentation pages that the text is outdated because the API is no longer merely deprecated, but entirely shut down. Hopefully they did publish the shutdown date ahead of time. It had a full replacement API and a migration guide. Fair and square. Nothing much to complain about here. It s time to get porting your code to the Web API! But frankly, I don t really care about this Metadata API because I don t use the Metadata API, do I?

Step 6: Break your API without warning February 3 2016, Mopidy users start reporting General transient error when searching Spotify. This is a libspotify error message known as SP_ERROR_OTHER_TRANSIENT with error code 16. This error usually means that there is some unspecified issue with the Spotify service. We ve actually seen it before and like its name suggests, it is indeed transient and soon goes away. According to a series of tweets between my fellow Mopidy developer Nick Steel and @SpotifyCares it appears that the libspotify search functionality was backed by the Metadata API s search functionality. Who knew? I m not aware of any relation between these APIs ever being mentioned publicly. No libspotify user could possibly have known that the Metadata API shutdown would affect them. Let s summarize: libspotify hasn t had any releases for three years and it was officially deprecated in May last year. But, there have been no warnings about any shutdown. The libspotify project is likely no longer staffed at all. Did someone just forget that libspotify depended on the Metadata API they were shutting down? Many newer networked AV receivers use the Spotify Connect libraries that are exclusive to Spotify s commercial partners. But some receivers probably still use libspotify. If so, did Spotify just break hardware sold with Spotify support as a major feature? Something that people are paying a monthly fee to use? It seems they did! In the Spotify Community forums there are numerous reports of broken devices and software that all have two things in common: the date the problem started occurring, and that search is the only feature that is broken. Surely, Spotify will quickly explain themselves and fix this?

Step 7: Stay quiet Since the API shutdown, Spotify has been very quiet. No posts on the developer blog. No email newsletter. No tweets from @SpotifyPlatform since December 1 last year. Some users at The Spotify Community forum are quoting responses from Spotify. Most responses seem to link to the Metadata API migration docs and state that application developers must migrate. In other cases the responses point the finger at the commercial partner which only adds to the confusion. Some of these devices may use the Metadata API directly, and in those cases the application maintainers are probably to blame. However, I m quite certain that the libspotify search issue is responsible for a fair share of these complaints. libspotify, in contrast to the Metadata API, has never been officially shut down. The responsibility for any breakage caused by libspotify is Spotify s alone.

What I want from Spotify
  1. Apologize to your paying customers and supporters.
  2. Get substantially better at communicating deprecations and shutdowns. We want all news, not just good news.
  3. Give us a new supported C library for audio playback. Now! Not later this year. I ve heard that promise for three consecutive years now. We can live with pre-release software quality initially. Getting the library out sooner rather than later shows us that there is a road ahead for all the projects that stream music from Spotify. Let us get started making new language wrappers.
  4. Release Spotify Connect specifications and libraries. We want to make music players that can be controlled by the Spotify mobile app. We want to make music controllers that can play music on all the Spotify Connect software and hardware players we ve invested in. You ve created this technology, now let us use it.
Thanks to Nick Steel for reviewing this blog post.

17 February 2016

Riku Voipio: Ancient Linux swag

Since I've now been using Linux for 20 years, I've dug up some artifacts from the early journey.
  1. First the book, from late 1995. This from before Tux, so the penguin in the cover is just a co-incidence. The book came with a slackware 3.0 CD, which was my entrance to Linux. Today, almost all of the book is outdated - slackware and lilo install? printing with lpr? mtools and dosemu? ftp, telnet with SLIP dialup? Manually configuring XFree86 and fvwm? How I miss those times!* The only parts of the book are still valid are: shell and vi guides. I didn't read latter, and instead imported my favorite editor from dos FTE.
  2. Fast forward some years, into my first programming job. Ready to advertise the Linux revolution, I bought the mug on right. Nobody else would have a Tux mug, so nobody would accidentally take my mug from the office dishwasher. That only worked for my first work place (a huge and nationally hated IT consultant house). The next workplace, a mobile gaming startup (in 2001, I was there before it was trendy!) - and there was already plenty of Linux mugs when I joined...
  3. While today it may be hard to imagine, those days using Microsoft office tools was mandatory. That leads to the third memorabilia in the picture. Wordperfect for Linux existed for a brief while, and in the box (can you imagine, software came in physical boxes?) came a Tux plush.
* Wait no, I don't miss those times at all

Antoine Beaupr : My free software activities, february 2016

Debian Long Term Support (LTS) This is my third month working on Debian LTS, started by Raphael Hertzog at Freexian. This month was my first month working on the frontdesk duty and did a good bunch of triage. I also performed one upload and reviewed a few security issues.

Frontdesk duties I spent some time trying to get familiar with the frontdesk duty. I still need to document a bit of what I learned, which did involve asking around for parts of the process. The following issues were triaged:
  • roundcube in squeeze was happily not vulnerable to CVE-2015-8794 and CVE-2015-8793, as the code affected was not present. roundcube is also not shipped with jessie but the backport is vulnerable
  • the php-openid vulnerability was actually just a code sample, a bug report comment clarified all of CVE-2016-2049
  • ffmpeg issues were closed, as it is not supported in squeeze
  • libxml2 was marked as needing work (CVE-2016-2073)
  • asterisk was triaged for all distros before i found out it is also unsupported in squeeze (CVEs coming up, AST-2016-001, AST-2016-001, AST-2016-001)
  • libebml and libmatroska were marked as unsupported, although an upload of debian-security-support will be necessary to complete that work (bug #814557 filed)

Uploads and reviews I only ended up doing one upload, of the chrony package (CVE-2016-1567), thanks to the maintainer which provided the patch. I tried my best trying to sort through the issues with tiff (CVE-2015-8668 and CVE-2015-7554), which didn't have obvious fixes available. OpenSUSE seem to have patches, but it is really hard to find them through their issue trackers, which were timing out on me. Hopefully someone else can pick that one up. I also tried and failed to reproduce the cpio issue (CVE-2016-2037), which, at the time, didn't have a patch for a fix. This ended up being solved and Santiago took up the upload. I finally spent some time trying to untangle the mess that is libraw, or more precisely, all the packages that embed dcraw code instead of linking against libraw. Now I really feel even more strongly for the Debian policy section 4.13 which states that Debian packages should not ship with copies of other packages code. It made it really hard to figure out which packages were vulnerable, especially because it was hard to figure out which versions of libraw/dcraw were actually vulnerable to the bug, but also just plain figure out which packages were copying code from libraw. I wish I had found out about secure-testing/data/embedded-code-copies earlier... Still, it was interesting to get familiar with codesearch.debian.net to try to find copies of the vulnerable code, which was not working so well. Kudos to darktable 2.0 for getting rid of their embedded copy of libraw, by the way - it made it completely not vulnerable to the issue, the versions in stretch and sid not having the code at all and older versions having non-vulnerable copies of the code.

Issues with VMs again I still had problems running a squeeze VM - not wanting to use virtualbox because of the overhead, I got lost for a bit trying to use libvirt and KVM. A bunch of issues crept up: using virt-manager would just fail on startup with an error saying interface mtu value is improper, which is a very unhelpful error message (what is a proper value??) - and, for the record, the MTU on eth0 and wlan0 is the fairly standard 1500, while lo is at 65536 bytes, nothing unusual there as far as I know. Then the next problem was actually running a VM - I still somewhat expected to be able to boot off a chroot, something I should definitely forget about it seems like (boot loader missing? not sure). I ended up calling virt-install with the live ISO image I was previously using: 
virt-install --virt-type kvm --name squeeze-amd64 --memory 512 --cdrom ~/iso/Debian/cdimage.debian.org_mirror_cdimage_archive_6.0.10_live_amd64_iso_hybrid_debian_live_6.0.10_amd64_gnome_desktop.iso --disk size=4 --os-variant debiansqueeze
At least now I have an installed squeeze VM, something I didn't get to do in Virtualbox (mostly because I didn't want to wait through the install, because it was so slow). Finally, I still have trouble getting a commandline console on the VM: somehow, running virtsh console squeeze-amd64 doesn't give me a login terminal, and worse, it actually freezes the terminal that I can actually get on virt-viewer squeeze-amd64, which definitely sounds like a bug. I documented a bit more of that setup in the Debian wiki KVM page so hopefully this will be useful for others.

Other free software work I continued my work on improving timetracking with ledger in my ledger-timetracking git repository, which now got a place on the new plaintextaccounting.org website, which acts as a portal for ledger-like software projects and documentation.

Darktable 2.0 I had the pleasure of trying the new Darktable 2.0 release, which only recently entered Debian. I built a backport for jessie, which works beautifully: much faster thumbnail rendering, no dropping of history when switching views... The new features are great, but I also appreciate how they are being very conservative in their approach. Darktable is great software: I may have trouble approaching the results other are having with lightroom and snapseed, but those are proprietary software that I can't use anyways. I also suspect that I just don't have enough of a clue of what I'm doing to get the results I need in Darktable. Maybe with hand-holding, one day, I will surpass the results I get with the JPEGs from my Canon camera. Until then, I turned off RAW exports in my camera to try and control the explosion of disk use I saw since I got that camera: 
41M     2004
363M    2005
937M    2006
2,2G    2007
894M    2008
800M    2009
1,8G    2010
1,4G    2011
9,8G    2012
31G     2013
26G     2014
9,8G    2015
The drop in 2015 is mostly due to me taking less pictures in the last year, for some reason...

Markdown mode hacks I ended up writing some elisp for the markdown mode. It seems I am always writing links like [text](link) which seems more natural at first, but then the formatting looks messier, as paragraph wrapping is all off because of the long URLs. So I always ended up converting those links, which was a painful series of keystrokes. So I made a macro, and while I'm a it, why not rewrite it as a lisp function. Twice. Then I was told by the markdown-mode.el developers that they had already fixed that (in the 2.1 version, not in Debian jessie) and that the C-c C-a r key binding actually recognized existing links and conveniently converted them. I documented my adventures in bug #94, but it seems I wrote this code for nothing else than re-learning Emacs lisp, which was actually quite fun.

More emacs hacking Another thing I always wasted time doing by and is "rename file and buffer". Often, you visit a file but it's named wrong. My most common case is a .txt file that i rename to .mdwn. I would then have to do: 
M-x rename-file <ret> newfile
M-x rename-buffer <ret> newfile
C-x C-s <ret> newfile
Really annoying. Turns out that set-visited-file-name actually does most of the job, but doesn't actually rename the file, which is really silly. So I wrote this small function instead: 
(defun rename-file-and-buffer (newfname)
  "combine rename-file and rename-buffer
set-visited-file-name does most of the job, but unfortunately
doesn't actually rename the file. rename-file does that, but
doesn't rename the buffer. rename-buffer only renames the buffer,
which is pretty pointless.
only operates on current buffer because set-visited-file-name
also does so and we don't bother doing excursions around.
"
  (interactive "GRename file and bufer: ")
  (let ((oldfname (buffer-file-name)))
    (set-visited-file-name newfname nil t)
    (rename-file oldfname newfname)
    )
  )
Not bound to any key, really trivial, but doing this without that function is really non-trivial, especially since set-visited-file-name needs special arguments to not mark the file as modified.

IRC packages updates I updated the Sopel IRC bot package to the latest release, 6.3.0. They have finally switched to Requests, but apart from that, no change was necessary. I am glad to finally see SNI support working everywhere in the bot! I also update the Charydbis IRC server package to the latest 3.5.0 stable release. This release is great news, as I was able to remove 5 of the 7 patches I was dragging along the Debian package. The previous Charybdis stable release was over 3 years old, as 3.4.2 was released in (December) 2012! I spend a good chunk of time making the package reproducible. I filed a bug upstream and eventually made a patch to make it possible to hardcode a build timestamp, which seems to have been the only detectable change in the reproducible build infrastructure. Charybdis had been FTBS for a while in sid now, and the upload should fix that as well. Unfortunately, Charybdis still doesn't build with hardening flags - but hopefully a future update of the package should fix that. It is probably because CFLAGS are not passed around properly. There's really interesting stuff going on in the IRC world. Even though IRC is one of the oldest protocols still in operation (1988, even before the Web, but after SMTP and the even more venerable FTP), it is still being actively developed, with a working group drafting multiple extensions to the IRCv3 protocol defined in RFC 1459. For example, IRCv3.3 includes a Strict Transport Security extension, which tries to ensure users use encrypted channels as much as possible, through warnings and STARTTLS support. Charybdis goes even further by proposing a reversal of the +S ("secure channel" flag) where all channels are secure by default, and you need to deliberately mark a channel as insecure with the +U flag if you actually want to allow users on an clear-text connection to join the channel. A transition mechanism is also proposed.

Miscellaneous bug reports En vrac... I fell face-first in this amazing game that is endless-sky. I made a small pull request on the documentation, a bug report and a feature request. I forwarded a bug report, originally filed against monkeysign, to the pyqrencode maintainers. I filed a usability bug against tails-installer, which just entered Debian, mostly usability issues. I discovered the fim image viewer, which re-entered Debian recently. It seemed perfect to adjust my photos-import workflow, so I added it to my script, to be able to review photos prior to importing them into Darktable and git-annex.

16 February 2016

Michal &#268;iha&#345;: Weekly phpMyAdmin contributions 2016-W06

As we're getting closer to release of new major release (4.6.0), the focus moves to the bug fixing. Most of the fixed issues came from our error reporting server, which collects error reports from the installations. Vast majority if bugs were affecting older releases as well, so these were fixed for upcoming 4.5.5 as well. Besides bugfixing there was also some cleanups in the master branch - shared code for processing sprites and covering it by tests and removed caching of MySQL server information as that didn't bring any speed improvements and lead to cache consistency issues. All handled issues:

Filed under: English phpMyAdmin 0 comments

15 February 2016

Sven Hoexter: distribution specific details

To state the obvious: my personal preference is to run Debian GNU/Linux. My current workplace is a CentOS shop and usually I'm the first to claim that it doesn't matter at all, and distribution specific implementation details are irrelevant for what we do (running a JVM). Let's take a short detour to the RedHat network-scripts. Two weeks ago we found some systems we originally installed in a different network segment, with different DNS servers and different search domains, came back up after a reboot with a rewritten 'resolv.conf'. Later on cfengine replaced the generated 'resolv.conf' with the intended one, so it wasn't that obvious to spot in the first place. A colleague found the origin of the rewritten 'resolv.conf' in a device specific configuration file that defined the 'DNS 1,2 ' variables with the installation time DNS server IPs from the other segment. I expected to experience the same behaviour (resolv.conf rewritten during startup and replaced by cfengine later on) in other locations, but assumed we just didn't notice it because the main difference would be a slightly different list of search domains. And I was wrong. I checked the timestamps of several 'resolv.conf' files and their cfengine backup file. None were recently created or related somehow to a reboot. grep-ing through parts of the network-scripts I found the following conditional in '/etc/sysconfig/network-scripts/ifup' 
if [ "$PEERDNS" != "no" ]  [ -n "$RESOLV_MODS" -a "$RESOLV_MODS" != "no" ]; then
   [ -n "$MS_DNS1" ] && DNS1=$MS_DNS1
   [ -n "$MS_DNS2" ] && DNS2=$MS_DNS2
      if [ -n "$DNS1" ] && ! grep -q "^nameserver $DNS1" /etc/resolv.conf &&
         tr=$(mktemp /tmp/XXXXXX) ; then
 ...
So if you adjust only the second nameserver IP you stored in "DNS2" in your configuration you end up without an update to your 'resolv.conf'. Now knowing that, I'd say this is relevant distribution specific knowledge, and I'm wondering how many of such subtle behaviours we've hidden in Debian specific solutions? Maybe knowledge about distribution specific implementation details even matters after all. Regardig the 'resolv.conf' issue itself the fault is on us. We as the responsible team did not read the documentation properly and thus deployed a configuration that later on let to some unexpected consequences. I try to remember those issues as an example for the next NetworkManager/systemd-networkd vs old-school-network-scripts argument.

Julien Danjou: Timeseries storage and data compression

The first major version of the scalable timeserie database I work on, Gnocchi was a released a few months ago. In this first iteration, it took a rather naive approach to data storage. We had little ideas about if and how our distributed back-ends were going to be heavily used, so we stuck to the code of the first proof-of-concept written a couple of years ago. Recently we got more feedbacks from our users, ran a few benchmarks. That gave us enough feedback to start investigating in improving our storage strategy. Data split Up to Gnocchi 1.3, all data for a single metric are stored in a single gigantic file per aggregation method (min, max, average ). This means that the file can grow to several megabytes in size, which make it slow to manipulate. For the next version of Gnocchi, our first work has been to rework that storage and split the data into smaller parts. Gnocchi Carbonara archives split The diagram above shows how data are organized inside Gnocchi. Until version 1.3, there would have been only one file for each aggregation methods. In the upcoming 2.0 version, Gnocchi will split all these data into smaller parts, where each data split is stored in a file/object. This allows to manipulate smaller pieces of data and to increase the parallelism of the CRUD operations on the back-end leading to large speed improvement. In order to split timeseries into several chunks, Gnocchi defines a maximum number of N points to keep per chunk, to limit their maximum size. It then defines a hash function that produces a non-unique key for any timestamp. It makes it easy to find in which chunk any timestamp should be stored or retrieved. Data compression Up to Gnocchi 1.3, the data stored for each metric is simply serialized using msgpack, a fast and small serialization format. Though, this format does not provide any compression. That means that storing data points needs 8 bytes for a timestamp (64 bits timestamp with nanosecond precision) and 8 bytes for a value (64 bits double-precision floating-point), plus some overhead (extra information and msgpack itself). After looking around on how to compress all these measures, I stumbled upon a paper from some Facebook engineers called about Gorilla, their in-memory timeserie database, entitled "Gorilla: A Fast, Scalable, In-Memory Time Series Database". For reference, part of this encoding is also used by InfluxDB in its new storage engine. The first technique I implemented is easy enough, and it's inspired from delta-of-delta encoding. Instead of storing each timestamp for each data point, and since all the data points are aggregated on a regular interval, we transpose points to be the time difference divided by the interval. For example, the suite of timestamps timestamps = [41230, 41235, 41240, 41250, 41255] is encoded into timestamps = [41230, 1, 1, 2, 1], interval = 5. This allows regular compression algorithms to reduce the size of the integer list using run-length encoding. To actually compress the values, I tried two different algorithms: I then benchmarked these solutions: Gnocchi Carbonara compression speed The XOR algorithm implemented in Python is pretty slow, compared to LZ4. Truth is that python-lz4 is fully implemented in C, which makes it fast. I've profiled my XOR implementation in Python, to discover that one operation took 20 % of the time: count_lead_and_trail_zeroes, which is in charge of counting the number of leading and trailing zeroes in a binary number. Gnocchi Carbonara compression XOR profiling I tried 2 Python implementations of the same algorithm (and submitted them to my friend and Python developer Victor Stinner by the way). The first version using string search with .index() is 10 faster than the second one that only do integer computation. Ah, Python As Victor explained, each Python operation is slow and there's a lot in the second version, whereas .index() is implemented in C and really well optimized and only needs 2 Python operations. Finally, I ended up optimizing that code by leveraging cffi to use directly ffsll() and flsll(). That decreased the run-time of count_lead_and_trail_zeroes by 45 %, making the entire XOR compression code speed increased by a small 7 %. This is not enough to catch up with LZ4 speed. At this stage, the only solution to achieve a high-speed would probably to go with a full C implementation. Gnocchi Carbonara compression size Considering the compression ratio of the different algorithms, they are pretty much identical. The worst case scenario (random values) for LZ4 compress down to 9 bytes per data point, whereas XOR can go down to 7.38 bytes per data point. In general XOR encoding beats LZ4 by 15 %, except for cases where all values are 0 or 1. However, LZ4 is faster than XOR by a factor of 4 -70 depending on cases. That means that we'll use LZ4 for data compression in Gnocchi 2.0. It's possible that we could achieve as fast compression/decompression algorithm, but I don't think it's worth the effort right now it'd represent a lot of code to write and to maintain.

Next.

Previous.