Search Results: "js"

13 January 2013

Neil Williams: Status of klibc on AArch64

With significant assistance from Steve McIntyre and some judicious delving into the ARM Information Centre, I've now got the assembly portions of klibc sorted (but untested) for AArch64 (arm64).

Andy & I started by copying the old ARM support as a new directory and one of the final steps was to remove a whole bunch of legacy code from the days before Thumb and all the #ifdef lines which went with it. Some files disappeared entirely. setjmp.S was the largest amount of work as the load&store multiple support of ARMv7 has gone in ARMv8, so the stmia mnemonic had to be expanded to multiple stp calls but that makes it more explicit, so it's not a bad thing.

Steve & I borrowed from the glibc AArch64 code from upstream by Marcus Shawcroft, simplifying the glibc macros for klibc and also got clues about what extra registers needed handling compared to ARMv7.

I've now got to look at some traditional cross-compilation issues because the Linaro AArch64 toolchain doesn't install to typical Debian cross-building paths and the build now moves past the AArch64-specific assembly and fails later when the C code gets the wrong include path and ends up including /usr/include/i386-linux-gnu/asm/byteorder.h with predictable results.

If someone has an AArch64 toolchain already set up, feel free to clone my modified klibc tree and let me know if there are subsequent build errors. Of course, if you fancy testing a build in the Foundation Model for ARMv8, that would be good too! (Report issues via github.)

Whilst I'm sorting out the toolchain, I've also been updating perl-cross-debian (which has also seen some more upstream testing and improvement).

12 January 2013

John Sullivan: Aaron Swartz

Aaron was an inspiration to me personally, politically, and professionally ever since we met (ice cream and word games with a small group in a bank vault at Herrell's in Harvard Square) several years ago. I don't understand how things got to this point, but I know I'm angry along with Lessig. I'm so sorry for all of his family and friends; all the rest of us can do is try to make even a tiny sliver of the difference he did.

11 January 2013

Julien Danjou: Overriding cl-json object encoding

CL-JSON provides an encoder for Lisp data structures and objects to JSON format. Unfortunately, in some case, its default encoding mechanism for CLOS objects isn't exactly doing the right thing. I'll show you how Common Lisp makes it easy to change that. Identifying the problem CL-JSON & CLOS CL-JSON mechanism encoding CLOS object is really neat. Let's see how it works for a simple case:

(defclass kitten ()
  ((tail :initarg :tail)))
 
(json:encode-json-to-string (make-instance 'kitten :tail 'black))

will produce:

 "tail":"black"

Still using CL-JSON, we can also decode the JSON object to a CLOS object:

(slot-value
 (json:with-decoder-simple-clos-semantics
   (json:decode-json-from-string " \"tail\":\"black\" "))
 :tail)

That code will return "black". Note that it's also possible to specify which class should be used when decoding objects, but that's beyond the purpose of this article. Postmodern Now, let's introduce Postmodern, a wonderful Common Lisp system providing access to the wonderful PostgreSQL database. It also provides a simple system to map rows in a database to CLOS classes, called DAO for Database access objects. With this, we can easily store our kitten into a table.

(defclass kitten ()
  ((tail :initarg :tail))
  (:metaclass postmodern:dao-class))

If we try to encode this to JSON, it will produce the exact same result seen previously. The problem is what happens when one of our column has a NULL value. Postmodern encodes this using the :null symbol. So this code:

(defclass kitten ()
  ((tail :initarg :tail :col-type (or s-sql:db-null text)))
  (:metaclass postmodern:dao-class))
 
(postmodern:deftable kitten
  (postmodern:!dao-def))
 
(postmodern:connect-toplevel  )
 
(postmodern:create-table 'kitten)
 
(json:encode-json-to-string
  (postmodern:make-dao 'kitten))

will return:

" "tail":"null" "

Fail! The fact that the column is NULL is represented by the :null symbol. And CL-JSON encodes all symbols as string. This is not at all what we want here! Overriding encode-json CL-JSON provides and uses the encode-json method to encode all kind of object. It is defined as a generic function, and a lot of different methods are implemented to handle the different standard Common Lisp types. The one used for standard-object is defined liked that:

(defmethod encode-json ((o standard-object)
                        &optional (stream *json-output*))
  "Write the JSON representation (Object) of the CLOS object O to
STREAM (or to *JSON-OUTPUT*)."
  (with-object (stream)
    (map-slots (stream-object-member-encoder stream) o)))

All we need to do here, is to create a new method for our kitten objects, that handles correctly the :null case.

(defclass kitten ()
  ((tail :initarg :tail :col-type (or s-sql:db-null text)))
  (:metaclass postmodern:dao-class))
 
(export 'kitten)
 
;; Switch package just to define the new method
(in-package :json)
(defmethod encode-json ((o cl-user:kitten)
                        &optional (stream json:*json-output*))
  "Write the JSON representation (Object) of the postmodern DAO CLOS object
O to STREAM (or to *JSON-OUTPUT*)."
  (with-object (stream)
    (map-slots (lambda (key value)
                 (as-object-member (key stream)
                   (encode-json (if (eq value :null) nil value) stream)))
               o)))
 
;; Go back into our package
(in-package :cl-user)
 
(postmodern:deftable kitten
  (postmodern:!dao-def))
 
(postmodern:connect-toplevel  )
 
(postmodern:create-table 'kitten)
 
(json:encode-json-to-string
  (postmodern:make-dao 'kitten))

With that new method, as soon as we encounter a :null symbol as a value for an object's slot, we replace it by nil. Now if we try to encode another kitten, we'll get:

 "tail":null

which is far better for our JavaScript data consumers! In the end, I think that this kind of trick is feasible that easily because of the way CLOS provides its generic method implementation. The fact that methods don't belong to any class makes the extension of every program, library and class so much easier. Doing this in another language like Java would likely by impossible, and in Python it would unlikely be as clean as it is done in Common Lisp. The ability to teach any library about how it should handle your class just by defining a new method is really handy!

3 January 2013

Richard Hartmann: GnuPG key transition statement

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1,SHA512
I am transitioning GPG keys from an old 1024-bit DSA key to a new
4096-bit RSA key.  The old key will continue to be valid for some
time, but I prefer all new correspondance to be encrypted to the new
key, and will be making all signatures going forward with the new key.
This transition document is signed with both keys to validate the
transition.
If you have signed my old key, I would appreciate signatures on my new
key as well, provided that your signing policy permits that without
re-authenticating me.
The old key, which I am transitioning away from, is:
  pub   1024D/DFCA34A3 2007-10-07 [expires: 2013-10-24]
    Key fingerprint = FE23 BE62 DF18 72FB C58D  D637 FFAE 0427 DFCA 34A3
The new key, to which I am transitioning, is:
  pub   4096R/95206DD3 2013-01-02 [expires: 2016-01-02]
    Key fingerprint = DF0B FDFF 4A4D DA01 7944  1B8F 6906 4B01 9520 6DD3
To fetch the full new key from a public key server using GnuPG, run:
  gpg --keyserver keys.gnupg.net --recv-key 95206DD3
If you have already validated my old key, you can then validate that
the new key is signed by my old key:
  gpg --check-sigs 95206DD3
If you then want to sign my new key, a simple and safe way to do that
is by using caff (shipped in Debian as part of the "signing-party"
package) as follows:
  caff 95206DD3
Please contact me via e-mail at <richih.mailinglist@gmail.com>
if you have any questions about this document or this transition.
    Richard Hartmann
    richih.mailinglist@gmail.com
    2013-01-03
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.12 (GNU/Linux)
iEYEARECAAYFAlDk27gACgkQ/64EJ9/KNKMGDgCeLyQy6yhEu3OB8UarJ6LYOfVY
ihgAnRSbYMrVxK+28souMHiLx4y0rSQ+iQIcBAEBCgAGBQJQ5Nu4AAoJEGkGSwGV
IG3TcbYP/je8mu+MCIiNNY8w/gN+/9S8gEGWd2p9LeN71xpCuSpBVSevAxxStV9R
2XRTfI8tf/nk27y+4073IscUTPaBkb2IhvIfCnFmOtZjP9slS2JMHOylqH3KLulQ
Cm/IbpKCAwTmQtRMzFxKoRjNPJDCVzZYpoD9r7TmIrs+iKtDQ/w/zQmtwEFWncYK
2JJqVEd+qN9REpWlKDDifcUDPGiugsICelAaUF8VMvp2ouVxQTGbz53IdAgvrX9G
vM+BAfel3W73nfF3L+08vgad0BGGMnEWcfcZ6r+DKu2WXgC8AXT0Y32sD8TiGJSd
JrWMqwec1iOd6lfOpFDdhSCFyMl5r1NiOLw5iNsETHYjysW2N1wZCicmF/8ukm/y
EpIXf3z8vtLrofGLN1jTC34Jugts8C5NNRil195Hkim36t2F97OJswNLhxt7nQ41
vhyXaBK3kwOZ1nVp6xM/yMnQzi5eVO1W/BE8Wr27O6pe3KqEx0Lv8zBzQjj4ly64
shyNRY3gzxs/KZ3p/3nrSn1AYOXIuCM3nILh89qQ/ynotrqYmpFkUorUSSOvraoQ
LN5IPH0wh8F2eDUCui2fGDEmnQ5S5mlHnIZmL3gkPVGLhNaoFQh0R8UwUO6HIxMQ
cTxv5TGL1YWtIvl0bvlEXK6MA4g0ZBXk15JZkWUD84zycr5cb79d
=DFzt
-----END PGP SIGNATURE-----

26 December 2012

Gunnar Wolf: Brazilian electronic booths hacked in a real election Surprised, anyone?

Doing my regular news scan, I stumbled across this: Hacker reveals in Rio how he rigged an election (in Portuguese; you can try the Google-translated version). Why am I reposting this? Because, even after the reported studies by Diego Aranha and the information disclosure exploited by Sergio Freitas, Brazil is still portrayed as the biggest example on how electronic voting can be 100% secure and tamper-proof. Well, in this case, Rangel (his full name ahs not yet been disclosed), a 19 year old hacker, not only demonstrated how elections could be rigged, but admitted on doing so together with a small group, and even pointed at who was benefitted from this. Rangel's attack was done during the transmission phase After ~50% of the electoral results had been sent over the Oi network. And yes, the provider will most likely close the hole that was pointed at, but this basically shows (again!) that no system can be 100% tamperproof, and that the more electronic devices are trusted for fundamental democratic processes, the more we as a society will be open to such attacks. The security-minded among us will not doubt even for a second that, as this attack was crafted, new attacks will continue to be developed. And while up to some years ago the attack surface was quite smaller (i.e. booths didn't have a communications phase, just stored the votes, and communication was done by personal means), earlier booths have been breached as well. And so will future booths be breached. So, the news of this attack are indeed very relevant for the field. The presentation I am quoting was held around two weeks ago And December will surely dillute attention from this topic. Anyway, I will look for further details on the mechanism that was used, as well as to the process that follows in the TSE (Supreme Electoral Tribunal). I hope we have news to talk about soon!

15 December 2012

Jose Luis Rivas: Why Jekyll

Why? I switched to Jekyll but it wasn't a fast decision. I really tried to give WordPress a chance, again, but I'm not getting along pretty well with PHP since like 2 years ago. In fact, is a headache to maintain on my server. In the course of the years I have had several downtimes on my PHP-based sites because of changes like PHP-FPM. Some sites just dropped working for me while others kept doing OK. This is in the same server, with the same database and the same codebase (WordPress). After a lot of time programming on Node.js and been learning more Ruby (CoffeeScript made me love the syntax) I just got tired of using more resources of what I really needed, and with a regular CMS that means a database. Why would I need a database if I just update like once a day? For a comment-system? There's Disqus already, and people can just comment you on Twitter or make a Branch for it, or just write you an email! Plain simple. Jekyll So there was Jekyll, a pretty nice tool creating static-sites very fast and easy to learn. It didn't needed a database and because all is static is pretty fast. Even the hosting requirements got lowered to the point I could (and I did) just upload this to GitHub and served it from there as a GitHub Page! One more thing One thing made me decided to upload to GitHub as well was Prose.io, a simple text-editor for GitHub with support for MarkDown made by the guys @ developmentseed.com. So while it was made for being updated on my terminal I could just do it as well from the web, from anywhere with just my GitHub account and having previews on-the-fly. The downside Since I don't control this server I couldn't setup redirections for my old feeds. Of course, I could have just copied the same pattern in a folder and put there an XML feeding the category X, but I think is easier to update the planets being feeded to the new URIs. On the bright-side I even could put pretty-permalinks to work! So the structure for my old links kept the same.

11 December 2012

Russell Coker: Links December 2012

Steven Johnson gave an interesting TED talk about where good ideas come from [1]. He starts by attributing coffee replacing alcohol as a standard drink for some good ideas and then moves on to how ideas develop. Erez Lieberman Aiden and Jean-Baptiste Michel gave an interesting and amusing TED talk about the ngram analysis of books that Google scanned [2]. Here is the link for the Google Books ngram search [3]. Clay Shirky gave an insightful TED talk about how the Internet is changing the world [4]. He cites Open Source programmers as the modern day equivalent to the Invisible College based on our supposed great ability to get along with other people on the Internet. If we really are so much better than the rest of the Internet then things must be bad out there. He ends with ways of using Git to draft legislation. Hans Rosling gave an interesting TED talk about religion and the number of babies that women have [5]. His conclusion is that it s more about income and social stability and that the world s population can stabilise at 10 billion if we provide family planning to everyone. Alexis C. Madrigal wrote an interesting interview with Genevieve Bell about her work at Intel and the way people use technology [6]. Indigogo is raising funds for the Cuddle Mattress , it s a mattress with foam slats and a special fitted sheet to allow your arm to slide between the slats [7]. So you could have your arm underneath your SO for an extended period of time without risking nerve damage. They also show that when sleeping on your side your shoulder can go between the slats to avoid back problems. Nate Silver (who is famous for predicting US elections gave an interesting TED talk about racism and politics [8]. One of his main points is to show the correlation between racism and lack of contact of members of other races. Sociological Images has an interesting article by Lisa Wade about whether marriage is a universal human value [9]. In regard to historical marriage she says women were human property, equivalent to children, slaves, servants, and employees . The general trend in the comments seems to be that there are so many types of marriage that it s difficult to make any specific claims to traditional marriage unless you count a tradition of a short period in a single geographic region. Plurality is an excellent sci-fi short movie on youtube [10]. TED has an interesting interview with Hakeem Oluseyi about his research about astrophysics and how he achieved a successful career after being a gangster as a teenager [11]. He has some good ideas about helping other children from disadvantaged environments become successful. Paul Dwerryhouse wrote an interesting blog post about his work in designing and implementing a filesystem based on a Cassandra data store with FUSE [12]. Paul also wrote a post about using Apache Zookeeper to lock metadata to permit atomic operations [13]. The documentary Monumental Myths provides an interesting and insightful analysis of the processes of creating, maintaining, and explaining monuments [14]. It focusses on some significant monuments in the US and explains both sides to each story. Howard Zinn makes the insightful point that when people present a certain point of view of history it s not controversial, as soon as you present the other side they call it controversial . That happens even in debates about current issues. Howard also says to criticise whatever the government does is not anti-America, it s anti-government, it s pro-America, it s pro the people, it s pro the country . The song that plays during the closing credits is interesting too. The music video Same Love is one of the best presentations of the argument for marriage equality [15]. Chris Samuel wrote an interesting post about systems locked down for Windows 8 and options for purchasing PCs for running Linux [16]. His solution is to buy from ZaReason. I saw his laptop in action at the last LUV meeting and it looks really nice. Unfortunately a byproduct of the extremely thin form factor is the fact that it lacks a VGA port, this meant that Chris had to use my Thinkpad T61 (which is rather clunky by comparison) for his presentation.

8 December 2012

Eddy Petrișor: Consent is needed, but not the point

Russ makes a good point about a nuance of practical jokes that makes them special, the characteristic of involving a third party which is the target of the joke, and how that relates to consent. He is right, consent is an important fact, I acknowledge that. Russ also says at the end of his post that being an asshole doesn't make you responsible for somebody else's decisions, I ackowledge that, too. I even somewhat agree to the position that the employer might fire you on assholery reasons, but in this particular case, I think that's a hypocrite position. If assholery was an issue, it should have been before, too, and I don't think the employer thought of it before this sad even, as long as it meant more money for them. I think this only shows how cynical they are, the cost analysis for them showed them they now might lose more money by not firing, than by keeping them. It's just cynical.

In spite of that, my initial post, just as I wrote, was not about the DJs, the people involved, the assholery, the taste or lack thereof of the joke. That's why I was intentionally leaving out names or specific context. It was about the bigger picture, irrational reactions and how in the face of a tragedy we surrender our thinking capabilities and react in an irrational manner to the least significant things.

I have had quite recent unfortunate events happening to people close to me, and, invariably, I observe this loss of rational faculties in the face of sad events. Even when this kind of disproportionate response is shown for what it is and how it is harmful in itself, people still refuse to give up on being irrational or they do after a long struggle with the facts.

I'll finish my article with a similar question: can we, as a species, go beyond our condition and make a conscious effort to keep calm and think things through before making decisions in the face of tragedy, or are we incapable to do that?

And no, I am not advocating to drop feelings and affection, I find it sad that I have to say that again, but I am almost sure that in spite of it, I will be wrongly understood by somebody.

7 December 2012

Thorsten Glaser: Collision resolution in open addressing hashtables

Before we begin, everyone should read up on hashtables and what open addressing / closed hashing is. The context is lines 111 190 of Python s Objects/dictobject.c as of today (so we get the line numbers straight). (I ve reworded this wlog entry a bit; I originally wrote it too late at night for it to read coherent.) Basically, I ve got an application where I d like to use a hashtable for a number of things not as generic as Python, and with focus on small footprint. I d like to offer associative arrays in a scripting language, where the keys are always arbitrary byte strings excluding NUL. Also, I d like to use the hashtable as backend for indexed arrays, where the keys are uint32_t and the usual use case is sequential. Finally, I m using it for several internal tables, such as a list of keywords, one of builtins, one of special variables, etc. which is a reason for me to not use a self-balancing binary tree as data structure (reading further below might suggest that, but getting a sorted list of hashtable keys is not the focus, though not unimportant).
My questions on this are:

Why is the shift on perturb done after its first use? In my experiments (using 32-bit width everywhere), for the pathological case of an 8-element (i = 3) table with three entries 0, 0x40000000 and 0x800000000, the second round yields 1 for all three, so it cannot have to do with the upper bits. My lookup looks like:

	mask = 2  - 1;
	j = perturb = hash(key);
	goto find_first_slot;
	 find_next_slot:
	j = (j << 2) + j + perturb + 1;
	perturb >>= PERTURB_SHIFT;
	/* FALLTHROUGH */
	 find_first_slot:
	entry = table[j & mask];
	if (!match(entry)) goto find_next_empty_slot;

This means that my first check is always the bare hash (so only do it if needed is no reason) and, since I m using gotos, I could just move the perturb >>= PERTURB_SHIFT; line before the line recalculating the next j to use. This seems to make more sense, even in the face of Python. (I actually looked at the Python file s comments again today because I thought to use a different resolution, but they have a good rationale for using the multiplication by 5.) Why can t we just use i as the PERTURB_SHIFT? Sure, this changes a shift-right by a constant, which can possibly be encoded as immediate value in assembly (unless you re on a pre-80186, which can only do SHR AX,1 and SHR AX,CL but not SHR AX,4, but that s outside of mksh s scope) into a right-shift by a variable, but i is already known, and I think the behaviour is better (it wouldn t eat any bits; assume the same 8-entry hashtable and pathologic keys 0, 8 and 16). Again: who do I think I am to go against the wisdom of the Python people, who seem to have shed more thought on this than everyone else I saw, asked, read about (including Spammipedia). That s why I m asking here. On that reference: I don t support spammers or people nagging for donations or premium accounts, like Xing and Groundspeak/Geocaching.COM, at all. In fact, I urge others to do the same, so it really hurts them; it may be their business model, but not if they spam me. Besides, OpenCaching.DE exists. Another thing is: to avoid CVE-2011-4815, I m randomising the hash used, with one seed value per hashtable, changed before a resize operation. I originally thought to seed it with nonzero, but then I have to rehash on hashtable resize, so I ll be XORing the final hash value instead (thanks ciruZ for the idea). I m thinking of omitting that for indexed arrays, as an attacker almost certainly cannot determine the keys there. (To directly use the indexed array keys, which are already uint32_t, as hashes makes using i from even more important.) The hash I m using is a modified Jenkins one-at-a-time called NZAAT: it s my new generic standard n n-cryptographic hash, and the changes are thus: while adding a byte, another increment of the hash is done (so NUL counts), and the finaliser got prefixed with the shift-left-add+shift-right-xor sequence of the adder (but not adding any value or the +1), to get best avalanche for all bytes. I actually compiled several versions of Hash.cs on a Windows VM at work to analyse the original one-at-a-time and all of my modifications; these turned out to be the simplest ones (I originally had added 0x100 instead of 1, but the effect was the same, and +1 is usually cheaper on most CPUs). Also, to avoid people being able to get to the seed, a user will always get only a sorted list of hashtable keys (numeric for indexed arrays, ASCIIbetically otherwise; see also my thoughts on JSON from the previous wlog entry). What algorithm do I use? For strings, comparisons are much more expensive, so I d like to keep them low. Memory use is also a factor; allocating one large(r) block is better than many small ones due to the pool allocator overhead and due to portability to ancient Unic s (which is another reason for me to use a hashtable which is a small struct plus an array of pointers, and then pass the list of keys as array of string pointers, instead of a tree). For both reasons, I m thinking a relatively simple MergeSort: I need to allocate the result array anyway, so I can just get two and free the one that isn t the end result, and it s AFAICT the cheapest on comparisons other than Tree Sort (which nobody really seems to use, and which would effect to using a balanced binary tree again). Since keys are unique, stability and duplicate handling is never an issue. I d like to use only one algorithm and one data structure, not a combination, as compactness is a design goal. Please drop your thoughts on Freenode, e.g. by /msg MemoServ send mirabilos your text here or per eMail to the domains debian, freewrt or mirbsd, which are organisations, with the localpart tg. Or just contact me as usual, if you re already acquainted. Or lookup 0xE99007E0. Thanks in advance! (Especially, Python Developers thoughts are welcome.)

6 December 2012

Steve Kemp: December 2012 Software Updates

Some brief software updates:

Custodian: This is the monitoring tool I wrote for Bytemark. It still rocks, and has run over 10 million tests without failure.I'd love more outside feedback, even if just to say "documentation needs work".
Slaughter: This is my sysadmin tool for multiple hosts - consider it cfengine-lite, or cfengine-trivial more likely. The 2.x release is finally out, and features pluggable transports. That means your central server can be running HTTP, RSYNC, FTP, or anything you like. 90% of the changes came from or were inspired by Csillag Tamas, to whom I owe a debt of thanks.
Templer: A static-site generator, written in Perl. I use this to generate blogspam.net, and other sites from simple layouts. Tutorial available online.
redis-document-store: A trivial hack which allows using Redis as a schema-less document storage system. Assuming you never delete documents it is simple, transparent, and already in live use for Debian Administration

Random Comment on Templer:

Although I've made extensive notes on common static site generators, and they will be discussed at length in the near future, I do want to highlight one problem common to 90% of them: Symbolic links. For example webgen fails my simple test:
~/hg/websites$ webgen create test.example.com
~/hg/websites$ cd test.example.com/src/
~/hg/websites/test.example.com/src$ mkdir jquery-1.2.3
~/hg/websites/test.example.com/src$ touch jquery-1.2.3/jquery.js
~/hg/websites/test.example.com/src$ ln -s jquery-1.2.3 jquery
~/hg/websites/test.example.com$ webgen
Starting webgen...
...
Finished
~/hg/websites/test.example.com$ ls out/    grep jq
jquery-1.2.3
Here we see creating a symlink to a directory has not produced a matching symlink in the output. Something I use frequently. for example. Some tools mangled symlinked directories, or files, some ignore them completely. Neither is acceptible.

Petter Reinholdtsen: Scripting the Cerebrum/bofhd user administration system using XML-RPC

Where I work at the University of Oslo, we use the Cerebrum user administration system to maintain users, groups, DNS, DHCP, etc. I've known since the system was written that the server is providing an XML-RPC API, but I have never spent time to try to figure out how to use it, as we always use the bofh command line client at work. Until today. I want to script the updating of DNS and DHCP to make it easier to set up virtual machines. Here are a few notes on how to use it with Python. I started by looking at the source of the Java bofh client, to figure out how it connected to the API server. I also googled for python examples on how to use XML-RPC, and found a simple example in the XML-RPC howto. This simple example code show how to connect, get the list of commands (as a JSON dump), and how to get the information about the user currently logged in:

#!/usr/bin/env python
import getpass
import xmlrpclib
server_url = 'https://cerebrum-uio.uio.no:8000';
username = getpass.getuser()
password = getpass.getpass()
server = xmlrpclib.Server(server_url);
#print server.get_commands(sessionid)
sessionid = server.login(username, password)
print server.run_command(sessionid, "user_info", username)
result = server.logout(sessionid)
print result

Armed with this knowledge I can now move forward and script the DNS and DHCP updates I wanted to do.

1 December 2012

Thorsten Glaser: Proposed extensions to the JSON specification

The following proposal extends the JSON specification, with the idea of using JSON as an information interchange format, rather than just a way of writing certain ECMAscript values. They do not add anything but only restrict valid JSON content and encoders with some rationale. First of, I d like to remind everyone, including JSON s author, that JSON is case-sensitive, except in the four hexdigits after a backslash-u sequence in a String. Second, I d like to remind everyone that JSON is not binary-safe. No way around that, it implements Unicode (actually, 16-bit UCS-2, and it doesn t guarantee that UTF-16 surrogates are correctly paired) text. I also consider only UTF- 8, 16,32 B,L E valid encodings for JSON. (No PDP endian, either. Sorry, guys.) For my first proposal, I d like to point out CVE-2011-4815 which was about overflowing hashtables. The obvious fix is to randomise the hash per hashtable; to ensure this doesn t leak, we sort ASCIIbetically the keys of an Object in the encoder. (Using Unicode is good here we can just sort the keys as UTF-8 strings by their uint8_t value or as Unicode (UCS-2 or even UCS-4 or UTF-16) strings by the codepoints.) JSON was never preserving the order of elements in an Object anyway so we make it standardised (we still accept any order, and, when parsing, in collision cases, the later value wins). This also helps diffs. For my second proposal, I d like to forbid \u0000, \uFFFE, \uFFFF in strings. The first because many implementations use C strings, and for an information interchange format this is better; it also has security implications to allow NUL in a string. The other two, but not unpaired UTF-16 surrogates (as ECMAscript uses UCS-2 and got UTF-16 only later) because they re not valid Unicode; JSON was not binary-safe already so why bother. Among other benefits, this also helps implementations. For my third proposal, I d like to agree that implementations should impose a nesting depth limit that may be user-defined, and in the face of which, cyclic checking may be ignored by an encoder. I emit nesting depth overflows as literalnull; might also throw an error. Since I was asked, the common standard value is to restrict nesting depth is 32, unless the user specified one. (I also saw 8, but 32 WFM pretty well.) Most seem to use it even if it may seem low at first. Only specialised applications probably need more, and they can always pass a value. For my forth proposal, backslash-escape U+007F U+009F always. It may upset humans, editors, databases, etc. (This paragraph is newly added, after some IRC discussion.) All these do not permit anything that wasn t accepted to be accepted afterwards. I ve got a fifth proposal which changes acceptance rules but only for a subset of parsers: formally JSON is defined in ECMA-262 as industry standard that, in contrast to RFC 4627, always allowed any Value as top-level element of a JSON text. I d like to make it so, and ignore the RFC s requirement for it to be an Object or Array. Even so, the first two characters (after the BOM, if any) of a JSON text always are in the non-NUL 7-bit ASCII range, allowing for encoding detection. (This is done by the NUL octet pattern in the first four octets.) JSON has only taken off because it s a tightly defined simple format that can be used everywhere and isn t too awful for humans (escaping not needed for U+0020 U+D7FF and U+E000 U+FFFD after all, although I d also take the C1 control characters out, see my forth proposal above). I ve started to use a trailing comma in indexed and associative arrays in code I write at work, when the array values are one a line, to help version control systems to do their diffs, but refrain from asking for a JSON extension to permit that in order to not endanger compatibility any (no comment needed, it s just not worth it), but I d like my above proposals to be followed by implementators (and I m one of them). Some more discussion with Jonathan pointed out that JSON5 allows for trailing commata in Object and Array; IMHO the only feature of it that is not bad or outright harmful. I ll probably keep from accepting them because, on their own, they re not that useful, and I usually would run JSON texts, even configs, through a parser/encoder roundtrip to pretty-print them which would lose them anyway. As for binary-safeness: probably best to just use base64 and let the outer layers worry about compression. The data is usually unrelated to the JSON-encoded structure, and even if it s related to other data the base64 representation is usually similar (unless misaligned). Update 02.12.2012 Wrong I was about the first two characters: " " is a valid JSON text. Still possible to peek at four octets and determine the encoding by ordering the tests; updated my notes.

26 November 2012

John Sullivan: Santiago de Compostela

I've put up my nearly unedited, unsorted, and uncommented photos from my recent trip to Santiago de Compostela, Spain, for the Libre Software World Conference. It was a beautiful place (and a great conference) -- I hope to write more about it soon.

23 November 2012

Benjamin Mako Hill: Cultivated Disinterest in Professional Sports

Like many of my friends, I have treated professional sports with cultivated indifference. But a year and a half ago, I decided to become a football fan. Several years ago, I was at a talk by Michael Albert at MIT where he chastised American intellectuals for what he claimed was cultivated disdain of professional sports. Albert suggested that sports reflect the go-to topic for small talk and building rapport across class and context. But he suggested that almost everybody who used the term "working class struggle" was incapable of making small talk with members of the working class because unlike most working class people (and most people in general) educated people systematically cultivate ignorance in sports. Professional sports are deeply popular. In the US, Sunday Night Football is now the most popular television show among women in its time slot and the third most popular television in America among 18-49 year old women. That it is also the most popular television show in general is old news. There are very few things that anywhere near half of Americans have in common. Interest in football is one of them. An enormous proportion of the US population watches the Superbowl each year. I recognized myself in Albert's critique. So I decided to follow a local team. I picked football because it is the most popular sport in America and because their strong revenue sharing system means that either team has a chance to win any given match. My local team is the New England Patriots and I've watched many of the team's games or highlights over the last season and a half. I've also followed a couple football blogs. A year and half in, I can call myself a football fan. And I've learned a few things in the process:

With a little effort, getting into sports is easy. Although learning the rules of a sport can be complicated, sports are popular because people, in general, find them fun to watch. If you watch a few games with someone who can explain the rules, and if you begin to cheer for a team, you will find yourself getting emotionally invested and excited.
Sports really do, as Albert implied, allow one to build rapport and small talk across society. I used to dread the local cab driver who would try to make small talk by mentioning Tom Brady or the Red Sox. No more! Some of these conversations turn into broader conversations about life and politics.
Interest in sports can expand or shrink to fill the time you're willing to give it. It can mean just glancing through the sports sections of the paper and watching some highlights here or there. Or it can turn into a lifestyle.
It's not all great. Football, like most professional sports, is deeply permeated with advertisements, commercialism, and money. Like other sports, it is also violent. I don't think I could ever get behind a fight sport where the goal is to hurt someone else. The machoness and absence of women in the highest levels of most professional sports bothers me deeply.

I've also tried to think a lot about why I, like most of my friends, avoided sports in the past. Disinterest in sports among academics and the highly educated is, in my experience, far from passive. I've heard people almost compete to explain the depth of their ignorance in sports one doesn't even know the rules, one doesn't own a television, one doesn't know the first thing about the game. I did the same thing myself. Bethany Bryson, a sociologist at JMU has shown that increased education is associated with increased inclusiveness in musical taste (i.e., highly educated people like more types of music) but that these people are most likely to reject music that is highly favored by the least educated people. Her paper's title sums up the attitude: "Anything But Heavy Metal". For highly educated folks, it's a sign of cultivation to be eclectic in one's tastes. But to signal to others that you belong in the intellectual elite, it can pay in cultural capital to dislike things, like sports, that are enormously popular among the least educated parts of society. This ignorance among highly educated people limits our ability to communicate, bond, and build relationships across different segments of society. It limits our ability to engage in conversations and build a common culture that crosses our highly stratified and segmented societies. Sports are not politically or culturally unproblematic. But they provide an easy and enjoyable way to build common ground with our neighbors and fellow citizens that transcend social boundaries.

14 November 2012

Michal Čihař: Key signing party in Prague

Tomorrow (Thursday 15th November), there will be a PGP key signing and CAcert assurance party in Prague. It will be held in SUSE offices and it will start at 17:00. If you are interesting in having your PGP key signed, bring papers with your key fingerprint (you can use for example these slips) and some government issued ID. For CAcert assurance it is recommended to bring printed out assurance papers (you can get them from CAcert website) and of course IDs as well. You can find more information on Facebook or Google+ event pages.

Filed under: English Suse 2 comments Flattr this!

11 November 2012

Gregor Herrmann: RC bugs 2012/45

again, not one of my best RCBW weeks. still, here's the short report:

#683100 pnp4nagios-web: "pnp4nagios-web: libjs-jquery-ui (>= 1.8.14) may be not enough"
add a comment to the bug report
#683138 pnp4nagios-web: "pnp4nagios-web: crashes immediately due to debian/patches/adjust-template-path"
add a comment to the bug report
~~#689687~~ src:guile-1.6: "guile-1.6: FTBFS on i386, kfreebsd-i386: testsuite failures"
sponsor NMU prepared by Daniel Hartwig, upload to DELAYED/2
#690128 inn2: "inn2: conffile disappearing during squeeze->wheezy upgrade: /etc/news/motd.news"
some triaging
#691215 pinpoint: "X request failed: BadDrawable"
try to reproduce, unsuccessful
~~#691535~~ mumble: "Plugins: Failed to load libmumble.so: Cannot load library /usr/lib/mumble/libmumble.so: (/usr/lib/mumble/libmumble.so: undefined symbol: glPopClientAttrib)"
send a patch proposal to the BTS, then upload to unstable with maintainer's approval

Nathan Handler: Debian Developer

Today, I officially got approved by the Debian Account Managers as a Debian Developer (still waiting on keyring-maint and DSA). Over the years, I have seen many people complain about the New Member Process. The most common complaint was with regards to the (usually) long amount of time the process can take to complete. I am writing this blog post to provide one more perspective on this process. Hopefully, it will prove useful to people considering starting the New Member Process. The most difficult part about the New Member Process for me had to do with getting my GPG key signed by Debian Developers. I have not been able to attend any large conferences, which are great places to get your key signed. I also have not been able to meet up with the few Debian Developers living in/around Chicago. As a result, I was forced to patiently wait to start the NM process. This waiting period lasted a couple of years. It wasn't until this October, at the [Association for Computing Machinery at Urbana-Champaign's Reflections Projections Conference], that this changed. Stefano Zacchiroli was present to give a talk about Debian. Asheesh Laroia was also present to lead an OpenHatch Workshop about contributing to open source projects. Both of these Developers were more than willing to sign my key when I asked. If you look at my key, you will see that these signatures were made on October 7 and October 9, 2012. With the signatures out of the way, the next step in the process was to actually apply. Since I did not already have an account in the system, I had to send an email to the Front Desk and have them enter my information into the system. Details on this step, along with a sample email are available here. Once I was in the system, the next step was to get some Debian Developers to serve as my advocates. Advocates should be Debian Developers you have worked closely with, and usually include your sponsor(s). If these people believe you are ready to become a Debian Developer, they write a message describing the work you have been doing with them and why they feel you are ready. Paul Tagliamonte had helped review and sponsor a number of my uploads. I had been working with him for a number of years, and he really helped encourage and help me to reach this milestone. He served as my first advocate. Gregor Herrmann is responsible for getting me started in contributing to Debian. When I first tried to get involved, I had a hard time finding sponsors for my uploads and bugs to work on. Eventually, I discovered the Debian Perl Group. This team collectively maintains most of the Perl modules that are included in the Debian repositories. Gregor and the other Debian Developers on the team were really good about reviewing and sponsoring uploads in a very timely manner. This allowed me to learn quickly and make a number of contributions to Debian. He served as my second advocate. With my advocations taken care of, the next step in the process was for the Front Desk to assign me an Application Manager and for the Application Manager to accept the appointment. Thijs Kinkhorst was appointed as my Application Manager. He also agreed to take on this task. For those of you who might not know, the Application Manager is in charge of asking the applicant questions, collecting information, and ultimately making a recommendation to the Debian Account Managers about whether or not they should accept the applicant as a Developer. They can go about this in a variety of ways, but most choose to utilize a set of template questions that are adjusted slightly on a per-applicant basis. Remember that period of waiting to get my GPG key signed? I had used that time to go through and prepare answers to most of the template questions. This served two purposes. First, it allowed me to prove to myself that I had the knowledge to become a Debian Developer. Second, it helped to greatly speed up the New Member process once I actually applied. There were some questions that were added/removed/modified, but by answering the template questions befrehoand, I had become quite familiar with documents such as the Debian Policy and the Debian Developer's Rerference. These documents are the basis for almost all questions that are asked. After several rounds of questions, testing my knowledge of Debian's philosophy and procedures as well as my technical knowledge and skills, some of my uploads were reviewed. This is a pretty standard step. Be prepared to explain any changes you made (or chose not to make) in your uploads. If you have any outstanding bugs or issues with your packages, you might also be asked to resolve them. Eventually, once your Application Manager has collected enough information to ensure you are capable of becoming a Debian Developer, they will send their recommendation and a brief biography about you to the debian-newmaint mailing list and forward all information and emails from you to the Debian Account Managers (after the Front Desk checks and confirms that all of the important information is present). The Debian Account Managers have the actual authority to approve new Debian Developers. They will review all information sent to them and reach their own decision. If they approve your application, they will submit requests for your GPG key to be added to the Debian Keyring and for an account to be created for you in the Debian systems. At this point, the New Member process is complete. For me, it took exactly 1 month from the time I officially applied to become a Debian Developer until the time of my application being approved by the Debian Account Managers. Hopefully, it will not be long until my GPG key is added to the keyring and my account is created. I feel the entire process went by very quickly and was pain-free. Hopefully, this blog post will help to encourage more people to apply to become Debian Developers.

Nathan Handler: Introducing nmbot

While going through the NM Process, I spent a lot of time on https://nm.debian.org. At the bottom of the page, there is a TODO list of some new features they would like to implement. One item on the list jumped out at me as something I was capable of completing: IRC bot to update stats in the #debian-newmaint channel topic. I immediately reached out to Enrico Zini who was very supportive of the idea. He also explained how he wanted to expand on that idea and have the bot send updates to the channel whenever the progress of an applicant changes. Thanks to Paul Tagliamonte, I was able to get my hands on a copy of the nm.debian.org database (with private information replaced with dummy data). This allowed me to create some code to parse the database and create counts for the number of people waiting in the various queues. I also created a list of events that occurred in the last n days. Enrico then took this code and modified it to integrate it into the website. You can specify the number of days to show events for, and even have the information produced in JSON format. This information is generated upon requesting the page, so it is always up-to-date. It took a couple of rounds of revisions to ensure that the website was exposing all of the necessary information in the JSON export. At this stage, I converted the code to be an IRC bot. Based on prior experience, I decided to implement this as an irssi5 script. The bot is currently running as nmbot in #debian-newmaint on OFTC. Every minute, it fetches the JSON statistics to see if any new events have occurred. If they have, it updates the topic and sends announcements as necessary. While the bot is running, there are still a few more things on its TODO list. First, we need to move it to a stable and permanent home. Running it off of my personal laptop is fine for testing, but it is not a long term solution. Luckily, Joerg Jaspert (Ganneff) has graciously offerred to host the bot. He also made the suggestion of converting the bot to a Supybot plugin so that it could be integrated into the existing ftpmaster bot (dak). The bot's code is currently pretty simple, so I do not expect too much difficulty in converting it to Python/supybot. One last item on the list is something that Enrico is working on implementing. He is going to have the website generate static versions of the JSON output whenever an applicant's progress changes. The bot could then fetch this file, which would reduce the number of times the site needs to generate the JSON. The code for the bot is available in a public git repository, and feedback/suggestions are always appreciated.

8 November 2012

Jo Shields: Evil, or why Douglas Crockford is harmful to Free Software

Yesterday I received a new serious bug report against Mono in Debian. For those not in the know, serious severity is release critical, and can trigger removal of a package from the distribution in order to make a shipping release (e.g. if Debian is in deep freeze, as is the case right now). Bug 692614 relates to a single source file, ./mcs/class/System.Web.Extensions/System.Web.Script.Serialization/JsonDeserializer.cs, which is part of the System.Web.Extensions assembly. This file was written in 2008 by an upstream Mono developer, and is based on some code from json.org specifically JSON_parser.c and JSON_checker.c json.org code carries a non-standard license. Specifically, it s MIT/X11 with an added clause: The Software shall be used for Good, not Evil. I ve had some discussions with Mono upstream, and they believe that they have resolved the matter to their satisfaction their solution will make its way into Debian soon. But in the meantime, let s talk about the clause. The Free Software Foundation s Freedom 0 reads:

A program is free software if the program s users have the four essential freedoms:

The freedom to run the program, for any purpose (freedom 0).

The Debian Free Software Guidelines and Open Source Initiative s Open Source Definition clause 6 reads:

No Discrimination Against Fields of Endeavor The license must not restrict anyone from making use of the program in a specific field of endeavor. For example, it may not restrict the program from being used in a business, or from being used for genetic research.

It s fairly clear that the JSON.org license clause goes against both of these, making any piece of software using that license neither Open Source nor Free Software. It is not distributable by any organization which mandates Freedom for its users not in Debian, not in Fedora, not on Google Code. Anybody who cares about their users will reject a clause like this, because it has awful chilling effects. Think it s funny? It really isn t. Who can use the code without being at risk of a lawyer knocking on their doors? Can the Catholic Church? The ACLU? Republicans? Democrats? Military contractors? Genetic engineers? Big pharma? Petrochemicals firms? Communists? Fascists? Mono developers? WikiLeaks? Without a clear definition of good and evil , people need to seriously consider whether they are safe to use the code because if the developer and users interpretations of the terms differ, there could be hell to pay. Would you want to ship a hardware device let s say you re making a smart TV and have some developer of a library send his lawyers around to tell you sorry, TV rots kids brains, it s clearly evil to get your entire distribution channel shut down by injunction?

It s not an oversight, kids. Upstream are well aware of the pain this childishness causes anyone who takes software licenses seriously. They giggle about it at conferences, like tweenagers who think they ve one-upped the adults in the room. This non-Free license is intended to mock people who take licensing seriously. In fact, one could easily categorise efforts to pollute the Free Software ecosystem with fake, non-Free software as evil, which would mean all JSON.org code fails to comply with its own license, due to shipping with its license (Inception, anyone?). Before the jokers in the room claim that this kind of problem is deserved by Mono, let s take a look at all the software in Debian which Douglas Crockford endangers with his childishness. PHP? OwnCloud? jQuery? You think Debian serves its users well by pulling jQuery from the next release in order to serve the ego of a man behaving like an eleven year old? It is also interesting to note that the author of this do no evil clause works at PayPal. Read that twice, go and repair your irony meters, then come back. In conclusion: thank you so VERY much Doug Crockford for making the world of Free Software measurably worse.

2 November 2012

Neil Williams: Introducing pyBit - Buildd Integration Toolkit

pyBit - cross-platform package building using AMQP

Message queues provide a simple way to create a distributed, cross-platform buildd toolkit to build packages using a collection of buildds, direct from various VCS clients. pyBit is intended to support rapidly evolving software collections and can support multiple VCS frontends and multiple build backends. Cross building is expected to be supported for some backends. The initial backend uses dpkg for Debian with subversion providing the source and sbuild doing the actual build.

pyBit includes support for cancelling selected builds and using multiple buildd clients per architecture, per platform and per suite.

Hooks are available or in development for subversion and git, other VCS hooks can be added. A RESTful web API provides live build reports and can generate build jobs for specific packages using particular VCS branches on selected architectures to support re-building packages at any point in the development process. Build history is stored using postgresql.

Other buildd systems can rebuild long lists of packages or build lots of binary packages from relatively slow moving source packages. PyBit exists to handle much more rapid software development across a wide range of platforms, VCS inputs and architectures. Buildd clients which are under-used can be tasked with building multiple suites or adding cross-build support. Buildd clients which are over-utilised are easily identified and adding new machines to an existing architecture / platform / suite pool should be easy. Hook activation automatically cancels any ongoing build for the same architecture, platform and suite to avoid wasting time on an interim version.

The emphasis in pyBit is to have fast builds with redundant clients, reliable reporting using a flexible and intuitive frontend. To this end, there is no need for a source package to be uploaded. Depending on the VCS hook in use, builds can happen every time a particular file is changed (e.g. debian/changelog for a native Debian package) or at every push (for a distributed VCS) or whatever is appropriate for a particular software collection. Builds are checked for available build-dependencies and packages re-queued if the build-dependencies are not yet met.

In the longer term, it may well be possible to use more than one server / database combination to support more builds and more platforms.

So far, we've got to a working model and tagged 0.1.0 as the first downloadable release. There is a lot more to do, especially adding more documentation, more VCS hooks, support for more VCS methods on the buildd clients and more buildd client scripts for platforms other than Debian. (The git hook and git source client are expected to let pyBit be self-buildable but a certain amount of configuration will be required for the server and each client which makes it not-quite self-hosting.)

pyBit concentrates on preparing collections of binary packages which can be used to build others, rather than trying to rebuild everything every time - this allows more rapid upstream software development and encourages modular, re-usable software. pyBit will also support rebuilding specific versions, architectures, suites or platforms via the RESTful web API. Access to this frontend can be controlled through any of the standard methods.

Components are loosely coupled via JSON encoded messages sent using rabbitMQ and curl. A new client can be added at any time and it will simply pick up buildd jobs from the relevant queue.

Development has now moved to Github and if anyone wants to look at new clients and new hooks, just contact the team by email or via IRC (#pybit on irc.oftc.net). Current development is based on Debian Squeeze 6.0.6 with backports and also on Debian Wheezy - more testing is welcome. Patches are also very welcome, pyBit is licensed under GPL2+.

There is no intrinsic reason why pyBit could not support any buildd platform capable of taking source from a known location and building a set of binaries. Packages are added to the queues whenever the hook is activated, so adding new packages to the collection is simply a case of triggering the existing hook.

Another interesting challenge for pyBit would be to trigger a hook on a new source code repository and just let it work through every package until everything is built. That probably won't work with the current 0.1.0 code but if that is what you'd like to work on, join the team.

Next.

Previous.