Search Results: "spa"

12 March 2015

Marko Lalic: solicit - an HTTP/2 Library for Rust

For quite some time now, I've been following Rust, a new(ish) programming language by Mozilla. It's been around for a while, but recently it's been attracting a lot of buzz. Around the new year, I finally decided to take a deeper look into the language. I find that the best way to learn a new programming language is to simply dive in and try to build something nontrivial. At that point I'd also been interested in learning more about the specifics behind the new HTTP/2 spec, so I decided to combine the two... The result is a (work in progress) client library for HTTP/2 written in Rust, which can be found at this GitHub repository: https://github.com/mlalic/solicit What follows is a description of what I aim to achieve with the library, as well as an overview of its current design. If that doesn't interest you, you can just check out the GitHub repo for some examples or the source code itself. Goals Given that there are already some fairly popular HTTP libraries for Rust (hyper leading the way), I decided that I wouldn't aim to replace them. Rather, the idea is to focus on the lower levels of the HTTP/2 protocol what is essentially another transport protocol and expose an API that allows those libraries to use an HTTP/2 connection to send requests and receive responses. HTTP/2 doesn't change the semantics of the protocol, so once any HTTP/1.1 library gets the raw headers and body, it should be able to represent the response regardless of the fact that it came on an HTTP/2 connection. The same holds for sending requests: only the representation of the request changes, not its content. There are some minor differences, such as HTTP/2 doing away with the reason phrases being part of the response, but those were only meant to be human readable and "MUST NOT" have had any influence on the semantics of the response. Therefore, the primary goal for the library is to provide an API for using an HTTP/2 connection on the lowest level, but making it easy for clients to customize or augment any particular step of handling a response, without having to know the details of functionality below the layer they want to customize. As an example, if one wanted to build a client that sends out progress notifications for downloads, it wouldn't be good if they needed to muck about with raw HTTP/2 frames ideally, they'd only need to implement a new way to handle incoming data chunks for streams. The same applies for implementing a client that streams the download into a file (instead of saving it in memory) or one that sends the chunks to a different component for processing as soon as they're received (e.g. parsing HTML). Since the Rust standard library only exposes blocking IO primitives, I decided to base this library on blocking IO, as well. However, the goal was to make it flexible enough so that clients that would like to allow multiple requests to be sent concurrently on the same HTTP/2 connection can be built, but without the library itself spawning any threads. Finally, on a less conceptual level (and more of an implementation level), the code should have extensive test coverage. Design The library is split into several layers, each of which provides services to the one above it. A brief description of each layer follows. Transport Layer This layer provides the API for raw byte stream communication with the peer. Essentially, it's a wrapper around the particular transport protocol that is used for the HTTP/2 connection. As the HTTP/2 spec allows the transport protocol to be either a cleartext TCP connection or a TLS-protected one, those are the ones that need to be implemented at this level. The API of the layer is represented by a single trait TransportStream that simply requires io::Read and io::Write to be implemented and (for now) provides a single convenience method based on the underlying io::Read implementation. For a clear-text TCP connection, the only thing that was required to implement it was one line of code: 
impl TransportStream for TcpStream
HTTP/2 Connection Layer The connection layer provides an API for reading and writing to a raw HTTP/2 connection. This means reading and writing HTTP/2 frames, not arbitrary bytes. Each HTTP/2 frame has to follow the same basic structure(a 9-byte header followed by a sequence of byte representing the payload of the frame). This is represented by the RawFramestruct. Each HTTP/2 frame type can have its own specific format for the content of the payload and assign meaning to the flags found in the header. Therefore, each frame needs to have its own implementation that can parse a RawFrame or return an error if the raw frame does not represent a valid frame of that type. Some common methods that each frame type implementation needs to support are given by the Frametrait that must be implemented by the implementations. Finally, the HttpConnectionstruct handles reading the byte stream from the underlying TransportStream instance, correctly delineating frames and parsing them into the appropriate frame type implementation (by delegating to the correct Frame implementation based on the frame type found in the header). Obviously, varying the implementation of the transport layer is a matter of choosing the appropriate TransportStream implementation. This includes using mock implementations for testing the HttpConnection itself. HTTP/2 Client Connection The ClientConnectionbuilds upon the HttpConnection by using it to expose functionality that is specific to the client side of an HTTP/2 connection. This includes sending requests and reading responses. The struct takes care of creating frames that need to be sent for a given request (including HPACK header encoding) and shipping them off to the server (using the underlying HttpConnection instance). As for handling a response, the ClientConnection expects to get an instance of a struct implementing the Sessiontrait when constructed. This trait essentially defines a number of callbacks: methods that will be called on this instance when the corresponding event occurs in the ClientConnection. Therefore, the ClientConnection understands the semantics of the particular frames that it reads (e.g. performs the decoding of the HEADERS frames' payloads, passing the raw data received in a DATA frame to the session, etc.). However, it does not know anything about streams; it only passes the stream ID up to the session layer, without thinking about whether the stream is open, valid, etc. HTTP/2 Session As mentioned in the previous section, each ClientConnection requires an instance of a struct implementing the Sessiontrait to be passed to it at construction time. The Session trait defines a number of callbacks that are invoked in the appropriate moments during the processing of frames read on a client connection. For example, new_headers (takes a list of heaers and a stream on which they were received) or new_data_chunk (a buffer and a stream on which the data arrived). The session is in charge of handling these events as they arise. It is up to it to decide what (if anything) should be done with the data that it receives. There is a default implementation of the Session trait (unimaginatively named DefaultSession), which provides an implementation that tracks which streams are currently open and delegates the processing of new events on particular streams to instances of a Streamtrait. This allows for client implementations to simply vary the implementation of a Stream that they provide to the DefaultSession in order to change how a particular part of a response is handled, but let the DefaultSession take care of tracking which streams are open and valid. Different implementations of a Session still can exist, if the particular client needs more fine-grained customization over how these events are handled, but for most purposes the default session, along with Stream customization should probably be enough. HTTP/2 Stream Again, as mentioned previously, the DefaultSessionimplementation of the Session trait relies on a Streamtrait providing the handling for events arising on individual streams. This level should be the one that gets customized most often in alternate implementations. A default stream implementation DefaultStreamhandles the messages by building up a response in-memory. Different implementations could instead send progress notifications on an associated channel, for instance. HTTP/2 Client The final layeris that of a client that exposes methods that would usually be expected for a client HTTP library: sending requests, obtaining responses. This layer is not the primary focus, though, and it only exposes two simple implementations that showcase how the lower level APIs can be used to build a client. The second purpose it fulfills is as a "sanity check" that the APIs actually do allow the goals to be met. Therefore, one of the clients allows for requests to be sent from concurrently running threads and the responses to those requests to be obtained asynchronously (in the form of a Future<Response>). However, the responses are basic and very raw: they are stored fully in memory (not available until they've been read until the end) and both the body, as well as the headers, exposed as raw byte sequences. As an example how this clientcan be used (as well as the simpler single-threaded version), check out the Examplessection in the GitHub repository's README file. Future Work Refactoring APIs As the project was started as a Rust learning exercise, the initial APIs are slightly awkward and fairly inefficient to work with. For example, the solicit::hpackmodule requires headers to be represented as a tuple of Vec<u8>, which means that whenever we need to encode a header list, even though their values might be (statically known) byte sequence literals, such as (b":method", b"GET"), we need to make a bunch of to_vec calls on the &[u8]s before passing them to the encoder, leading not only to a really ugly and tedious way of obtaining the encoded representation of the headers, but also to a lot of unnecessary allocations. Another API that will see some refactoring and improvement is the TransportStream, as well as the HttpConnection in order to make it possible to have separate threads perform reads and writes on the connection. Currently, the socket is owned by the HttpConnection instance and all writes and reads require a &mut reference, making this impossible. This change would tackle this by embracing Rust's new try_clone method on the TcpStream into the TransportStream and refactoring the HttpConnection to allow splitting it into a write and read handle. HTTP/2 Features As already mentioned, only a small subset of the full HTTP/2 spec is currently implemented, so, obviously, implementing the rest of it falls under the most important future work. Some near- to mid-term priorities that will be tackled are: Obviously, those aren't all the features of HTTP/2, but it's a rough list of things I'm planning on working on myself as the next steps. Integration With an (existing?) HTTP Library I would like it if in the end the library or at least some of its pieces ended up being used in the scope of one of the fully featured HTTP libraries. If there's any interest in this, I wouldn't mind putting in some extra legwork to help make it happen. Conclusion While I'm certain that there can be better, more efficient, more elegant implementations, I've had a lot of fun working on this so far (learning new things is always fun!) and if nothing else I'll continue in order to gain the deepest possible understanding of HTTP/2. For anyone that might have read all the way through, thanks for reading and I'm open for any sort of feedback!

Erich Schubert: The sad state of sysadmin in the age of containers

System administration is in a sad state. It in a mess.
I'm not complaining about old-school sysadmins. They know how to keep systems running, manage update and upgrade paths.
This rant is about containers, prebuilt VMs, and the incredible mess they cause because their concept lacks notions of "trust" and "upgrades".
Consider for example Hadoop. Nobody seems to know how to build Hadoop from scratch. It's an incredible mess of dependencies, version requirements and build tools.
None of these "fancy" tools still builds by a traditional make command. Every tool has to come up with their own, incomptaible, and non-portable "method of the day" of building.
And since nobody is still able to compile things from scratch, everybody just downloads precompiled binaries from random websites. Often without any authentication or signature.
NSA and virus heaven. You don't need to exploit any security hole anymore. Just make an "app" or "VM" or "Docker" image, and have people load your malicious binary to their network.
The Hadoop Wiki Page of Debian is a typical example. Essentially, people have given up in 2010 to be able build Hadoop from source for Debian and offer nice packages.
To build Apache Bigtop, you apparently first have to install puppet3. Let it download magic data from the internet. Then it tries to run sudo puppet to enable the NSA backdoors (for example, it will download and install an outdated precompiled JDK, because it considers you too stupid to install Java.) And then hope the gradle build doesn't throw a 200 line useless backtrace.
I am not joking. It will try to execute commands such as e.g.
/bin/bash -c "wget http://www.scala-lang.org/files/archive/scala-2.10.3.deb ; dpkg -x ./scala-2.10.3.deb /"
Note that it doesn't even install the package properly, but extracts it to your root directory. The download does not check any signature, not even SSL certificates. (Source: Bigtop puppet manifests)
Even if your build would work, it will involve Maven downloading unsigned binary code from the internet, and use that for building.
Instead of writing clean, modular architecture, everything these days morphs into a huge mess of interlocked dependencies. Last I checked, the Hadoop classpath was already over 100 jars. I bet it is now 150, without even using any of the HBaseGiraphFlumeCrunchPigHiveMahoutSolrSparkElasticsearch (or any other of the Apache chaos) mess yet.
Stack is the new term for "I have no idea what I'm actually using".
Maven, ivy and sbt are the go-to tools for having your system download unsigned binary data from the internet and run it on your computer.
And with containers, this mess gets even worse.
Ever tried to security update a container?
Essentially, the Docker approach boils down to downloading an unsigned binary, running it, and hoping it doesn't contain any backdoor into your companies network.
Feels like downloading Windows shareware in the 90s to me.
When will the first docker image appear which contains the Ask toolbar? The first internet worm spreading via flawed docker images?
Back then, years ago, Linux distributions were trying to provide you with a safe operating system. With signed packages, built from a web of trust. Some even work on reproducible builds.
But then, everything got Windows-ized. "Apps" were the rage, which you download and run, without being concerned about security, or the ability to upgrade the application to the next version. Because "you only live once".
Update: it was pointed out that this started way before Docker: Docker is the new 'curl sudo bash' . That's right, but it's now pretty much mainstream to download and run untrusted software in your "datacenter". That is bad, really bad. Before, admins would try hard to prevent security holes, now they call themselves "devops" and happily introduce them to the network themselves!

11 March 2015

Joey Hess: 7drl 2015 day 5 type directed spell system development

I want my 7drl game Scroll to have lots of interesting spells. So, as I'm designing its spell system, I've been looking at the types, and considering the whole universe of possible spells that fit within the constraints of the types. My first throught was that a spell would be a function from World -> World. That allows any kind of spell that manipulates the game map. Like, for instance a "whiteout" that projects a stream of whitespace from the player's mouth. Since Scroll has a state monad, I quickly generalized that; making spell actions a state monad M (), which lets spells reuse other monadic actions, and affect the whole game state, including the player. Now I could write a spell like "teleport", or "grow". But it quickly became apparent this was too limiting: While spells could change the World map, the player, and even change the list of supported spells, they had no way to prompting for input. I tried a few types of the Event -> M () variety, but they were all too limiting. Finally, I settled on this type for spell actions: M NextStep -> M NextStep. And then I spent 3 hours exploring the universe of spells that type allows! To understand them, it helps to see what a NextStep is: 
type Step = Event -> M NextStep
data NextStep = NextStep View (Maybe Step)
Since NextStep is a continuation, spells take the original continuation, and can not only modify the game state, but can return an altered continuation. Such as one that prompts for input before performing the spell, and then calls the original continuation to get on with the game. That let me write "new", a most interesting spell, that lets the player add a new way to cast an existing spell. Spells are cast using ingredients, and so this prompts for a new ingredient to cast a spell. (I hope that "new farming" will be one mode of play to possibly win Scroll.) And, it lets me write spells that fail in game-ending ways. (Ie, "genocide @"). A spell can cause the game to end by returning a continuation that has Nothing as its next step. Even better, I could write spells that return a continuation that contains a forked background task, using the 66 line contiuation based threading system I built in day 3. This allows writing lots of fun spells that have an effect that lasts for a while. Things like letting the player quickly digest letters they eat, or slow down the speed of events. And then I thought of "dream". This spell stores the input continuation and game state, and returns a modified continuation that lets the game continue until it ends, and then restores from the point it saved. So, the player dreams they're playing, and wakes back up where they cast the spell. A wild spell, which can have different variants, like precognitive dreams where the same random numbers are used as will be used upon awaking, or dreams where knowledge carries back over to the real world in different ways. (Supports Inception too..) Look how easy it was to implement dreaming, in this game that didn't have any notion of "save" or "restore"! 
runDream :: M NextStep -> M NextStep -> (S -> S) -> M NextStep
runDream sleepcont wakecont wakeupstate = go =<< sleepcont
   where
         go (NextStep v ms) = return $ NextStep v $ Just $
        maybe wake (go <=<) ms
         wake _evt = do
                 modify wakeupstate
                 wakecont
I imagine that, if I were not using Haskell, I'd have just made the spell be an action, that can do IO in arbitrary ways. Such a spell system can of course do everything I described above and more. But, I think that using a general IO action is so broad that it hides the interesting possibilities like "dream". By starting with a limited type for spells, and exploring toward more featureful types, I was able to think about the range of possibilities of spells that each type allowed, be inspired with interesting ideas, and implement them quickly. Just what I need when writing a roguelike in just 7 days!
Fun comment thread on reddit

9 March 2015

Joey Hess: 7drl 2015 day 3 movement at last

Got the player moving in the map! And, got the map to be deadly in its own special way. 
        HeadCrush -> do
                showMessage "You die."
                endThread
Even winning the game is implemented. The game has a beginning, a middle, and an end. I left the player movement mostly unconstrained, today, while I was working on things to do with the end of the game, since that makes it easier to play through and test them. Tomorrow, I will turn on fully constrained movement (an easy change), implement inventory (which is very connected to movement constraints in Scroll), and hope to start on the spell system too.
At this point, Scroll is 622 lines of code, including content. Of which, I notice, fully 119 are types and type classes. Only 4 days left! Eep! I'm very glad that scroll's central antagonist is already written. I don't plan to add other creatures, which will save some time.
Last night as I was drifting off to sleep, it came to me a way to implement my own threading system for my roguelike. Since time in a roguelike happens in discrete ticks, as the player takes each action, normal OS threads are not suitable. And in my case, I'm doing everything in pure code anyway and certianly cannot fork off a thread for some background job. But, since I'm using continuation passing style, I can just write my own fork, that takes two continuations and combines them, causing both to be run on each tick, and recursing to handle combining the resulting continuations. It was really quite simple to implement. Typechecked on the first try even! 
fork :: M NextStep -> M NextStep -> M NextStep
fork job rest = do
        jn <- job
        rn <- rest
        runthread jn rn
  where
        runthread (NextStep _ (Just contjob)) (NextStep v (Just contr)) =
                return $ NextStep v $ Just $ \i -> do
                        jn <- contjob i
                        rn <- contr i
                        runthread jn rn
        runthread (NextStep _ Nothing) (NextStep v (Just contr)) =
                return $ NextStep v (Just contr)
        runthread _ (NextStep v Nothing) =
                return $ NextStep v Nothing
endThread :: M NextStep
endThread = nextStep Nothing
background :: M NextStep -> M NextStep
background job = fork job continue
demo :: M NextStep
demo = do
    showMessage "foo"
    background $ next $ const $
        clearMessage >> endThread
That has some warts, but it's good enough for my purposes, and pretty awesome for a threading system in 66 LOC.

Petter Reinholdtsen: Frikanalen, Norwegian TV channel for technical topics

The Norwegian Unix User Group, where I am a member, and where people interested in free software, open standards and UNIX like operating systems like Linux and the BSDs come together, record our monthly technical presentations on video. The purpose is to document the talks and spread them to a wider audience. For this, the the Norwegian nationwide open channel Frikanalen is a useful venue. Since a few days ago, when I figured out the REST API to program the channel time schedule, the channel has been filled with NUUG talks, related recordings and some Creative Commons licensed TED talks (from archive.org). I fill all "leftover bits" on the channel with content from NUUG, which at the moment is almost 17 of 24 hours every day. The list of NUUG videos uploaded so far include things like a one hour talk by John Perry Barlow when he visited Oslo, a presentation of Haiku, the BeOS re-implementation, the history of FiksGataMi, the Norwegian version of FixMyStreet, the good old Warriors of the net video and many others. We have a large backlog of NUUG talks not yet uploaded to Frikanalen, and plan to upload every useful bit to the channel to spread the word there. I also hope to find useful recordings from the Chaos Computer Club and Debian conferences and spread them on the channel as well. But this require locating the videos and their meta information (title, description, license, etc), and preparing the recordings for broadcast, and I have not yet had the spare time to focus on this. Perhaps you want to help. Please join us on IRC, #nuug on irc.freenode.net if you want to help make this happen. But as I said, already the channel is already almost exclusively filled with technical topics, and if you want to learn something new today, check out the Ogg Theora web stream or use one of the other ways to get access to the channel. Unfortunately the Ogg Theora recoding for distribution still do not properly sync the video and sound. It is generated by recoding a internal MPEG transport stream with MPEG4 coded video (ie H.264) to Ogg Theora / Vorbis, and we have not been able to find a way that produces acceptable quality. Help needed, please get in touch if you know how to fix it using free software.

8 March 2015

Joey Hess: 7drl 2015 day 2 level generation and game concept

Much as I want to get my @ displayed and moving around screen, last night and today have focused on level generation instead. Scroll has kinda strange level generation method, compared to how I suppose most roguelikes do it. There are only 3 calls to rand in all of Scroll. Just a 3 random parameters, but that's enough to ensure a different level each time. 
-- Random level generation function.
level :: Bool -> StdGen -> [String]
level randomize r = concat
        [ final (length tutorial + extra) $ concat $ rand mariner1body
        , concat $ rand mariner1end
        , concatMap rand kubla
        ]
  where
    -- here be spoilers
You could say there are two influences in Scroll's level generation method: Nick Montfort and Samuel Taylor Coleridge.
I have thought some about Scroll before starting the 7drl week, but my idea for the game was missing some key concepts. There was nothing to keep the player engaged in moving forward, an unclear victory condition, no clue how to generate appropriate random levels, a large potential for getting stuck, and no way to lose the game. This is all a problem for a roguelike. But, I had an idea finally last night, about a single unified thing that all of that stuff falls out from. And it's right there in the name of the game! Now that I understand Scroll better, I wrote the tutorial level. It's a very meta tutorial level that sets the scene well and serves more purposes than are at first apparent. I count a total of 6 things that this "tutorial level" will let the user do. And interestingly, while the tutorial level is static, it interacts with the rest of the game in a way that will make it be experienced differently every time through.
The strangest line of code I wrote today is: 
import GPL
Somehow, I have never before, in all my time programming, written a line like that one.
Finally, after 7 hours of nonstop coding, I got ncurses to display the generated game world, scrolling around in the display viewport. No @ yet; that will need to wait for tonight or tomorrow!

Carl Chenet: Verify your backups or lose them all

Follow me on Identi.ca or Twitter or Diaspora*diaspora-banner As a good sysadmin, you thought you had backups for your server but you didn t verify them before the major migration you attempted? When you need them, they re empty or full of useless files. And now it s too late
nobackup

Wen you discover that you don t have usable backups lesjoiesdusysadmin.fr

You won t guess how often this situation occurs. Backup Checker is a command line software developed in Python 3.4 on GitHub (stars appreciated :) ) allowing users to verify the integrity of archives (tar, gz, bz2, lzma, zip, tree of files) and the state of the files inside an archive in order to find corruptions or intentional of accidental changes of states or removal of files inside an archive. github-logo Install Backup Checker from PyPI The easiest way to install Backup Checker is from PyPi using the following command: 
$ pip3.4 install backupchecker
Debian Wheezy and Squeeze packages for Backup Checker Backup Checker Debian packages are now available for your stable servers Wheezy and Squeeze in the MyTux Debian repositories. Debian Squeeze Just copy/paste the following command on your server to add the MyTux Debian Squeeze repository and install Backup Checker: 
apt-key adv --keyserver pgp.mit.edu --recv-keys 2B24481A \
&& echo "deb http://debian.mytux.fr squeeze main" > /etc/apt/sources.list.d/mytux.list \
&& apt-get update \
&& apt-get install backupchecker
debian-squeeze-toy Debian Wheezy Just copy/paste the following command on your server to add the MyTux Debian Wheezy repository and install Backup Checker: 
apt-key adv --keyserver pgp.mit.edu --recv-keys 2B24481A \
 && echo "deb http://debian.mytux.fr wheezy main" > /etc/apt/sources.list.d/mytux.list \
 && apt-get update \
 && apt-get install backupchecker
wheezy Using Backup Checker 2 steps are needed to secure your backups. First you need to generate the configuration files of your backups, using the following command: $ backupchecker -G /backups/backup-08032015.tar.gz This generates 2 files /backups/backup-08032015.conf and /backups/backup-08032015.list you need to store in order to verify this archive later (the -O option lets you define a custom location to store the files). Second step is to verify your backup. It is as simple as: $ backupchecker -c /etc/backupchecker/backup-08032015.conf -l /var/log/backupchecker.log Then check if warnings have been sent to /var/log/backupchecker.log. Really simple isn t it? Scripting this command, your backups are now verified and secured. If any modification occurs, it will be detected and pinpointed. What do you thing about it? As usual, any feedback is welcome, through bug reports, emails of the author or comments on this blog. Several companies now use Backup Checker to secure their backups. Let us know if we can help you.

7 March 2015

Joey Hess: 7drl 2015 day 1 groundwork

Scroll is a roguelike, with a twist, which I won't reveal until I've finished building it. I'll just say: A playable roguelike pun, set in a filesystem near you. I'm creating Scroll as part of the 7DRL Challange. If all goes well, I'll have a usable roguelike game finished in 7 days. This is my first time developing a roguelike, and my first time writing a game in Haskell, and my first time writing a game to a time limit. Wow!
First, some groundwork. I'm writing Scroll in Haskell, so let's get the core data types and monads and IO squared away. Then I can spend days 2-7 writing entirely pure functional code, in the Haskell happy place. To represent the current level, I'm using a Vector of Vectors of Chars. Actually, MVectors, which can be mutated safely by pure code running inside the ST monad, so it's fast and easy to read or write any particular location on the level. 
-- Writes a Char to a position in the world.
writeWorld :: Pos -> Char -> M ()
writeWorld (x, y) c = modWorld $ \yv -> do
    xv <- V.read yv y
    V.write xv x c
showPlayer :: M ()
showPlayer = writeWorld (5,8) '@'
(I wish these Vectors had their size as part of their types. There are vector libraries on hackage that do, but not the standard vector library, which has mutable vectors. As it is, if I try to access outside the bounds of the world, it'll crash at runtime.) Since the game will need some other state, I'm using the state monad. The overall monad stack is type M = StateT S (ST RealWorld). (It could be forall s. StateT S (ST s), but I had some trouble getting that to type check, so I fixed s to RealWorld, which is ok since it'll be run using stToIO. Next, a concept of time, and the main event loop. I decided to use a continutation passing style, so the main loop takes the current continuation, and runs it to get a snapshot of the state to display, and a new continutation. The advantage of using continuations this way is that all the game logic can be handled in the pure code. I should probably be using the Cont monad in my monad stack, but I've not learned it and lack time. For now I'm handling the continuations by hand, which seems ok. 
updateWorld :: Step
updateWorld (Just 'Q') = do
        addMessage "Are you sure you want to quit? [yn]"
        next $ \i -> case i of
                Just 'y' -> quit
                _ -> continue
updateWorld input = do
        addMessage ("pressed " ++ show input)
        continue
Finally, I wrote some ncurses display code, which is almost working.
Start time: After midnight last night. Will end by midnight next Friday. Lines of code written today: 368 Craziest type signature today: writeS :: forall a. ((Vec2 a -> ST RealWorld ()) -> M ()) -> Pos -> a -> M ()
By the way, there's a whole LambdaHack library for Haskell, targeted at just this kind of roguelike construction. It looks excellent. I'm not using it for two reasons:
  1. Scroll is going to be unusual in a lot of ways, and LambdaHack probably makes some assumptions that don't fit.
  2. mainSer :: (MonadAtomic m, MonadServerReadRequest m) => [String] -> COps -> (m () -> IO ()) -> (COps -> DebugModeCli -> ((FactionId -> ChanServer ResponseUI RequestUI -> IO ()) -> (FactionId -> ChanServer ResponseAI RequestAI -> IO ()) -> IO ()) -> IO ()) -> IO ()
    That's a lot of stuff to figure out! I only have a week, so it's probably easier to build my own framework, and this gives me an opportunity to learn more generally useful stuff, like how to use mutable Vectors.

6 March 2015

Rapha&#235;l Hertzog: My Free Software Activities in February 2015

My monthly report covers a large part of what I have been doing in the free software world. I write it for my donators (thanks to them!) but also for the wider Debian community because it can give ideas to newcomers and it s one of the best ways to find volunteers to work with me on projects that matter to me. Debian LTS This month I have been paid to work 14.5 hours on Debian LTS. I worked mostly on CVE triage (41 commits in the security tracker) and organizational issues. One maintainer complained that he had not been kept in the loop for an LTS update of his package. After some discussion, I decided to change the way I did CVE triage so that any time that I add a package to our list of packages needing an update, I also send a mail to the maintainer, thus offering him the opportunity to step in. To make this sustainable, I wrote a small helper script that will generate a mail out of a template. And to kickstart the process I mailed all maintainers of packages that were already listed in our queue of packages to update. To improve the email generated, I requested a JSON export of the security tracker data (see discussions in #761859). In the mean time, Holger worked on this already and after a few iterations we did converge on an output format that will be really useful both for my needs in terms of CVE triage but also for the Package Tracker to be able to display the list of security vulnerabilities affecting each release (see #761730). Last but not least, I don t want to be the only one doing CVE triage for our LTS release so I documented the process in our wiki page. As a side note, I sponsored an e2fsprogs update prepared by Nguyen Cong and I sent the DLA for the embargoed samba update that had been prepared by Ivo de Decker (thanks to both of them!). Tryton Like last month, I invested again a copious amount of time on Tryton, fixing some bugs that were affecting me and improving the French chart of accounts to properly manage purchases and sales within the European Union. Here are some links for more details: Debian I did some work on Distro Tracker, I fixed #777453 (password reset not working because the generated email was using an invalid From email) and #779247 (obsolete build reproducibility action items were not dropped). I also started to work on restructuring the mail handling in distro-tracker (cf #754913) but it s not public yet. While I have no plans to stop contributing to Debian (it s part of my day job!), I reduced my non-work related involvement by officially recognizing that I was no longer properly assuming some of my responsibilities and that I was following too many mailing lists and RSS feeds. The most notable changes are that I removed myself from the maintenance of dpkg, developers-reference, quilt, sql-ledger, and a few perl/python modules. Misc Voting software. Part of the reason why I m reducing my involvement in Debian is that I got more involved in Nouvelle Donne (a French political party) and in particular in the handling of its digital infrastructure (currently running on Ubuntu, doh!). As part of this, I was looking for free software to handle secure votes and elections (and if possible adhering to the principles of liquid democracy). There s no perfect solution and no clear winner. That said I started following the evolution of AgoraVoting because it seems to have a good momentum and has some interesting features (it already supports votes with ranked choices, supports good crypto, has been used for elections involving large numbers of voters in the context of Podemos in Spain). But it still has some ways to go to establish itself as a truly international and community-backed project. GDM bug. Due to my work on Kali, I filed a bug against GDM (this one has been quickly fixed upstream, it s still open in Debian) and another one against accountsservice to request the possibility to define the default graphical session. Dirvish formula for Salt. I contributed another formula to manage backups with dirvish. Thanks See you next month for a new summary of my activities.

One comment Liked this article? Click here. My blog is Flattr-enabled.

Dirk Eddelbuettel: Rcpp 0.11.5

The new release 0.11.5 of Rcpp just reached the CRAN network for GNU R, and a Debian package has also been be uploaded. Rcpp has become the most popular way of enhancing GNU R with C++ code. As of today, 345 packages on CRAN depend on Rcpp for making analyses go faster and further; BioConductor adds another 41 packages, and casual searches on GitHub suggests dozens mores. This release continues the 0.11.* release cycle, adding another large number of small bug fixes, polishes and enhancements. Since the previous release in January, we incorporated a number of pull requests and changes from several contributors. This time, JJ deserves a special mention as he is responsible for a metric ton of the changes listed below, making Rcpp Attributes even more awesome. As always, you can follow the development via the GitHub repo and particularly the Issue tickets and Pull Requests. And any discussions, questions, ... regarding Rcpp are always welcome at the rcpp-devel mailing list. See below for a detailed list of changes extracted from the NEWS file.
Changes in Rcpp version 0.11.5 (2015-03-04)
  • Changes in Rcpp API:
    • An error handler for tinyformat was defined to prevent the assert() macro from spilling.
    • The Rcpp::warning function was added as a wrapper for Rf_warning.
    • The XPtr class was extended with new checked_get and release functions as well as improved behavior (throw an exception rather than crash) when a NULL external pointer is dereferenced.
    • R code is evaluated within an R_toplevelExec block to prevent user interrupts from bypassing C++ destructors on the stack.
    • The Rcpp::Environment constructor can now use a supplied parent environment.
    • The Rcpp::Function constructor can now use a supplied environment or namespace.
    • The attributes_hidden macro from R is used to shield internal functions; the R_ext/Visibility.h header is now included as well.
    • A Rcpp::print function was added as a wrapper around Rf_PrintValue.
  • Changes in Rcpp Attributes:
    • The pkg_types.h file is now included in RcppExports.cpp if it is present in either the inst/include or src.
    • sourceCpp was modified to allow includes of local files (e.g. #include "foo.hpp"). Implementation files (.cc; .cpp) corresponding to local includes are also automatically built if they exist.
    • The generated attributes code was simplified with respect to RNGScope and now uses RObject and its destructor rather than SEXP protect/unprotect.
    • Support addition of the rng parameter in Rcpp::export to suppress the otherwise automatic inclusion of RNGScope in generated code.
    • Attributes code was made more robust and can e.g. no longer recurse.
    • Version 3.2 of the Rtools is now correctly detected as well.
    • Allow 'R' to come immediately after '***' for defining embedded R code chunks in sourceCpp.
    • The attributes vignette has been updated with documentation on new features added over the past several releases.
  • Changes in Rcpp tests:
    • On Travis CI, all build dependencies are installed as binary .deb packages resulting in faster tests.
    • Thanks to CRANberries, you can also look at a diff to the previous release As always, even fuller details are on the Rcpp Changelog page and the Rcpp page which also leads to the downloads page, the browseable doxygen docs and zip files of doxygen output for the standard formats. A local directory has source and documentation too. Questions, comments etc should go to the rcpp-devel mailing list off the R-Forge page.

    This post by Dirk Eddelbuettel originated on his Thinking inside the box blog. Please report excessive re-aggregation in third-party for-profit settings.

    4 March 2015

    Zlatan Todori : Interviews with FLOSS developers: Paul Wise

    After starting with Joey Hess, we continue with Paul Wise. What makes his star to shine are many things such as being a DSA (Debian System Administrator), a helpful hand on mailings list, encouraging people to join Debian teams but most of all - he has encyclopedia knowledge on Debian as a whole which he gladly shares with anyone who asks (very fast response on IRC channels). It is almost impossible for any single person to count all Debian teams, work and places - to know most of those things, you can image the vast knowledge which Paul has. The legend says that his brain has better and faster search engine algorithm on Debian related queries than all other engines combined. So lets see what he has to share with world. me: Who are you? pabs: Paul Wise (pabs) and I have to say that I'm no-where near as knowledgeable as your intro suggests. me: How did you start programming? pabs: Messing around with fractals and graphics things in MS BASIC. me: How would you now advise others to start programming? pabs: Pick an issue in a tool you use, investigate how the tool works and how you can change it, fix that and contribute the change back to the project that created that tool. In the process you will learn skills, interact with the community and contribute to the project. me: Setup of your development machine? pabs: Lenovo Thinkpad with external monitor, Debian testing and some tweaks me What is your preferable language (for hacking)? Why? How do you compare it to other languages? pabs: I currently prefer Python for its readability. It still has some rough edges though the documentation covers them fairly well. I generally pick up new languages when working on projects written in them. Haskell is next on the horizon due to Nikki and the Robots. me: Describe your current most memorable situation as software developer/hacker? pabs: I had a great time creating fractals in BASIC, learning about the Mandelbrot set, L-systems and more. My days and nights of hacking on frhed (a GPLed hex editor for Windows) to help me cheat at Civilisation were pretty memorable. frhed led to my work on reverse engineering the CHM file format (a documentation format for Windows programs). A stand-out moment during my time with Debian was hacking on the derivates census patch generation code during the Debian UK BBQ weekend, surrounded by geeks playing Portal, cooking things, hacking on Debian and generally having a good time (thanks Steve!). me: Some memorable moments from Debian conferences? pabs: There are so many; meeting Debian folks, playing Mao once and then never again, late night games of werewolf, both delectably delicious and hideously disgusting cheeses, fried insects, day trips to beautiful landscapes, inspiring keynotes, exciting BoFs, secret IRC channels for planning surprise birthday parties, blue hair, wet air, blocks of fried cheese, a vast quantity of icecream, pants, geeks in the surf, volcanoes, hiking, a wonderful view, a uni-cycling stormtrooper & more. me: How do you see future of Debian development? pabs: I hope we will continue to exist and uphold our principles for the foreseeable future. I don't have any crystal balls though. me: You recently became member of Debian DSA - what is that like, what roles do you have and what tasks are in front of DSA? pabs: We wrote a bit of text about that for DPN recently. me: You have large knowledge on Debian and you share it with anyone who wants to know more. What motivates you to do so? pabs: I want the operating system I personally rely on to exist into the future, helping folks work on and join Debian can help with that. me: Why should developers and users join Debian community? What makes Debian a great and happy place? pabs: Every Debian contributor has different reasons for joining the community. Personally the Social Contract, the DFSG and the spirit and culture behind them are the main reason to be involved. I also like our many efforts towards technical excellence and correctness. Of course I've made a number of good friends over the years, especially as a result of attending DebConf every year since 2007. me: You are member of Debian publicity team which writes Debian news - do you need more people to join that team and how can they start? pabs: Since there is an infinite amount of work to do, pretty much every part of Debian always needs help, that includes the publicity team. We published a post about ways to help here. me: If someone wants to contribute to Debian in terms of packaging, can they do it anonymously (for example over Tor network, does Debian have .onion address)? pabs: Due to Debian's penchant for transparency it is harder but there are definitely package maintainers who have built up a reputation for good work under a pseudonym over the years and become Debian contributors as a result. I'm not aware of completely anonymous package maintainers but there are definitely people who file bugs using one-off pseudonyms, which is almost the same thing as anonymously. There are definitely Debian contributors and members who use Tor while contributing to Debian. In fact, as Debian is very highly dependent on OpenPGP and the best practices for OpenPGP include refreshing your keyring slowly over Tor, so probably quite a number of Debian contributors use Tor. As far as I know Debian itself does not run any Tor relays or onion services. me: What are places that non-packaging developers and people could join and help spread Debian even more? pabs: There are many ways to help Debian, including non-technical ones. Unfortunately our web page about helping Debian isn't quite up-to-date with all of them but a few more are to volunteer at DebConf, helo with artwork requests, speak about Debian at events or even come up with ideas for projects. Whatever skills you have, Debian can probably make use of them. If you aren't sure where to start, jump on the debian-mentors mailing list or IRC channel and we can probably guide you to the right place within Debian. Don't worry about not being skilled enough, everyone starts somewhere. me: How do you see Debian will manage webapps? pabs: Personally I prefer locally installed software, standard data formats and standard data transfer protocols to the wild webapps world but I understand they are becoming very popular to produce and use due to the ubiquity of the web browser platform. Antonio Terceiro is mentoring a project for this year's newcomer mentorship programs (outreachy/gsoc) that aims to improve support for installing web apps on Debian installations. I hope it succeeds as it could help make Debian more popular on servers and home servers in particular. me: How would you advise Debian (and other FLOSS users) to setup their machine in terms of security and anonymity? pabs: All technology has upsides and downsides. I would advise anyone to analyse their situation and protect themselves accordingly. For example if you have a bad memory, full disk encryption, which is based on pass-phrases might lead to data loss and physical security might be a better choice for protecting your data. The right choices around technology are very much a personal thing. me: Is it better to setup xmonad (because it is Haskell based WM) with small dependency chain or GNOME (because it is getting sandboxed apps) in term of security and privacy implications? pabs: Again, the right choices around technology are very much a personal thing. Due to the design of X11, both of these are approximately equivalent from a window-manager security properties point of view, that is to say, pretty bad. Wayland is one of the possible X11 successors and offers much better security properties. GNOME folks are working on switching to Wayland. Ultimately though it comes down to how each person uses their window manager and which software they run under it. me: Should Debian join Tor project as distro that installs Tor relays by default - should it offer that as option in installer in Debian 9? pabs: Running a Tor relay requires a reasonably fast and reliable Internet connection and should be a conscious decision on behalf of the sysadmin for a computer so Debian probably shouldn't install them by default. If tasksel gets support for installing tasks from Debian Pure Blends, then we could add a Tor relay task to the Debian Sanctuary Pure Blend. me: Have you ever considered joining initiatives such as FreedomBox? pabs: I was quite moved by Eben Moglen's talk at DebConf10 in New York and the resulting BoF. It seemed like a very ambitious project but I didn't really have the knowledge, skills or time to contribute yet. me: Are you a gamer? Valve Steam games are offered for free to Debian Developers - do you use steam and play Valve games? Your thoughts on Steam and non-free Linux gaming? pabs: I play computer games occasionally, all from Debian main or ones that I'm packaging. 0ad is my current go-to for a bit of gaming. I don't have any experience with Steam or non-free games on Linux. me: Is there something you would change in FLOSS ecosystem? pabs: Various folks have highlighted new and ongoing challenges for the FLOSS ecosystem in various places in recent years. Something that I would like to highlight that does not get talked about enough is the choices we make around our digital artefacts. This is the discussion around "preferred form for modification" or "source". The "source" for a particular digital artefact is a deliberate choice on behalf of the authors. Often generated files are distributed alongside the "source" without any instructions for reproducing the generated files from the "source". It sometimes happens that FLOSS contributors forget to distriute what they have chosen as "source", instead just distributing the generated files. This is a fairly well known issue but still happens. What isn't thought about quite as much is that the choice of "source" has consequences for future development possibilities of that "source". Some forms of "source" are more expressive than others, can be modified in a wider variety of ways and are better choices in general. Sometimes the consequences of choosing less expressive forms are mild and other times they are quite important. I hope more people will start to think about these choices. Some examples where, in my opinion, various people could have made better choices are listed in the mail I sent to the games team list last year. Another thing I would like to highlight is the work that organisations like Software Freedom Conservancy and Software in the Public Interest do to protect, defend, promote and support FLOSS projects. It is very important work that needs our interest and support. me: Can FLOSS world create great alternatives to Viber, Dropbox, WhatsUp, Facebook, Skype and other non-free services? pabs: I think that the FLOSS world has already created alternatives to all of those. The success of non-free services doesn't take these alternatives away but it does mean some of them are less useful because some of them are the kind of tools that become more useful with a larger amount of people using them. I don't know what it would take for the FLOSS alternatives to achieve similar success as network effects are hard to overcome. Hopefully mako is right and the network effects are overrated. me: Your thoughts and compare Cloud, IaaS, PaaS, SaaSS? To what should the FLOSS world pay more attention and energy? pabs: Initially I dismissed these as buzzwords and a threat to Free Software. These days I view them as potential opportunities for Free Software. Cloud-related technologies such as OpenStack and virtual machines can make private compute farm hardware more flexible and useful to their owners. IaaS providers can be used to run Debian more simply and cheaply and therefore bring Debian to more people than possible with hardware. PaaS providers can be used to run Free Software services. SaaSS can be based entirely on Free Software and respect users. Of course, just like running Free Software on hardware (proprietary or libre), cloud technology, IaaS, PaaS and SaaSS all come with downsides. The FLOSS world should aim to inform users of our software of these downsides. For example, the Debian installer could note that it is running on Intel CPUs with a proprietary BIOS and various proprietary software running, that it is running on a mobile phone with a locked bootloader, that it is running in a Xen VM on machines owned by Amazon. Free Software services could note they are running on Google App Engine etc. Free Software web browsers, chat clients etc could note when they are connecting to proprietary network services. All these notes could inform users about the downsides present in the particular situation encountered. There is also much work to be done making it easier to run Free Software on top of or use Free Software to connect to all manner of platforms from lowRISC to UEFI to VMware to Google App Engine to GitHub to Facebook. The more places Free Software can reach, the more people will be exposed to the philosophy behind it and the more potential there is for folks to join the community. While co-option of the FLOSS world is a dangerous certainty, co-option of proprietary platforms might be able to expand the reach of the philosophy behind Free Software. me: Your thoughts on Purism (the open hardware laptop initiative that got recently funded on CrowdSupply)? pabs: I don't know enough about that to comment but personally I am more interested in a laptop based on a libre CPU architecture. The RISC-V ISA and the lowRISC project seems to be one of the more promising possibilities at this point in time. me: Did you watch Citizenfour - comments on it? pabs: I've seen the trailer and look forward to watching it at some point, I read there might be a screening at DebConf15.

    2 March 2015

    Gergely Nagy: Distributed Version Control from another angle

    I've been using distributed version control systems for a good while, started with TLA back in 2001, when it was still a bunch of shell scripts and was simply called "Arch". I jumped the Git bandwagon quite early too, sometime around May 2005, I believe, though it was only something to test and play with on the side: I didn't migrate my projects over yet. There are many reasons why I preferred these over the systems I used in the past (RCS and CVS; never considered Subversion an option), most of those reasons have been discussed at length over the years, and there's no reason to do so again.The reason for this post, is because I read a blog post about why Git and the other DVCSes fall short. There are a number of things in that article that I find... well, lets just say misinformed. I'll try to explain why.But here's the rub
    I have wanted to have the full history while offline a grand total of maybe about six times. And this is merely going down over time as bandwidth gets ever more readily available. If you work as a field tech, or on a space station, or in a submarine or something, then okay, sure, this is a huge feature for you. But for the rest of us, I am going to assert that this is not the use case you need to optimize for when choosing a VCS.
    I'm neither a field tech, nor do I work on a space station or a submarine. Yet, having the full history available locally has been crucial for my work for at least the past six years. Initially, I worked offline primarily, because my internet connection was incredibly limited (slow, expensive and unreliable), yet, I had a habit of committing small changes often. I needed a full clone to work effectively, and conveniently. Throw-away branches and all that - I needed them local, I didn't have the resources to push them online every time I wanted a merge.But now that I have fast and reliable internet, why do I need the full history on my own computer? I do, because I do much more research than I do development. The vast majority of my work between 2010 and 2012 involved reading histories, cherry picking and so on. About six hours a day, I was reading the past. Having the full history locally is a terrific aid when your job involves digging through it.Between 2013 and 2014, the amount of history reading dropped significantly, but I still worked with long-lived branches a lot, branches where history matters, where having easy and fast access to them is indispensable. I still had to find changes based on content (not commit message, branch, date or anything - content) on a daily basis. Hitting the network for each and every search would have wasted not only tremendous bandwidth and processing power on the server, but also a significant amount of time.If you do more maintenance and research, or follow a good number of branches (or forks), with the intent to merge and cherry-pick actively from them, having all the histories available locally is a huge improvement. I'm happy to pay the price of "wasting" disk space to have this available. I couldn't work otherwise. Without full, local history, the things I did for syslog-ng would not have been possible. No maintenance, none of the features I implemented, none of the fixes I found. Why? Because I rely on the project's history to aid me in learning the whys, not just the hows.Thank you, but - perhaps unlike others - I like to learn from the past, and prefer doing it efficiently.Say goodbye to blobs (or your sanity (and sometimes both))
    You know what kind of external binary asset management system works really well? Subversion. Turns out it works tolerably for source control, too. Maybe you should take a look at it.
    Sure, some systems are better at managing binary assets. Git - and usually DVCSes - are terrible at it. There's nothing stopping anyone from using two systems: one for source, another for data. Mind you, I am extremely happy with git-annex. I disagree with Subversion being tolerable for source control. I'd turn this assertion around instead: You know what kind of system works really well for source code? Git. Turns out it works tolerably as an external binary assest management system, too. Maybe you should take a look at it.But then, I rarely need to work with blobs, so I'm likely not the best person to disagree here.Say goodbye to sane version synchronizationOkay, so this part of the article is ill-titled, I believe. Large repositories and version synchronisation are two different, not necessarily related topic. Granted DVCSes may not be the best fit for large repositories, but neither are centralised version control systems, based on my past experience. I had the misfortune of working with a roughly 3-4Gb (checked out) project, with extensive history, managed with a centralised version control system, without local history. It was terrible (and blame was so slow and expensive that we were told never to use it, because it crashed the server). While the system allowed us to have a reasonable overview over the whole project, but it utterly failed at allowing us to work at the component level. How do I see which version of component X we had in project release A? Or the version of component Y we had two weeks ago? Or what if we have component M, made up from components J and K, and belonging to component Z? Good luck managing that in a single repository.With TLA, we had configs, which were a bit cumbersome to use, but captured most of the essential parts of managing component bundling and hierarchical versioning far better than any other solution I've seen so far. I've yet to find an acceptable solution, but I'm convinced it will not have anything to do with centralised systems. You can't capture the delicate relations there.Say what again, I dare you, I double dare you
    All of the above might be almost tolerable if DVCSes were easier to use than traditional SCMs, but they aren t.
    I'm sorry, but have you tried managing a decently sized project, with a large history, built from multiple components with a traditional SCM? Because that is a colossal pain in the backside.Does Git's UI suck? Yes. But so did CVS's and Subversion's. If you rely on the command-line tools, you're going to feel the pain. That is why you use the IDE integration you surely have, which makes the most common tasks a breeze. That may sound like a git-apology, but remember, we did the same with every other version control system, because smart people prefer integration, and want the computer to work for them.
    And yet, when someone dares to say that Git is harder than other SCMs, they inevitably get yelled at, in what I can only assume is a combination of Stockholm syndrome and groupthink run amok by overdosing on five-hour energy buckets.
    If you feel the need to understand the innards of Git - then yes, it is harder than traditional SCMs. It is distributed, and different than what you're used to. If you do not have prior knowledge though, the difference in learning the internal working of Git and - say - Subversion all that different. And if you do not care how it works down below (and I dare say, a lot of developers should not, and do not care), then there is nothing to learn, apart from your IDE integration, which ain't different from a traditional SCM's.Mind you, I found it both interesting, and beneficial to learn Git's inner workings. Not only because it allowed me to make better use of my VCS, but because it taught me a bunch of things I could make use of elsewhere. I learned nothing from traditional SCMs.Do you hear the people sing, singing the song of angry men
    There is one last major argument that I hear all the time about why DVCSes are superior, which is that they somehow enable more democratic code development.
    The reason Git and other tools make code contribution easier is not GitHub pull requests (although, for drive-by contributions those are pretty useful too). The reason they make contribution easier is because you have your own copy of the full history. You can create your own branches easily, you can work with it as if you were the owner. For one-off contributions, that is no real advantage, I'll give you that. It takes more effort to fork & do a PR on GitHub, than to send a patch, I'll also agree with that.But once you start working with a project for more than a few patches, when you want to work with its history too, once you have long-lived branches, the local copy quickly becomes a great asset. Once you want to submit more than a few changes, you won't have to repeat a all steps. And not only that: since you have the full history, merging upstream changes (or just cherry picking from them), rebasing your work, and many other tasks become simpler.That is what it buys you, that is why it makes contribution easier. GitHub pull requests are an icing over the cake at best (and there are a good number of cases where the icing is completely unnecessary, because it doesn't go well with the cake).In closingIn closing, I'd like to highlight that different projects have different requirements. For example, I understand why Facebook, Google and a number of others use a single, huge repository. I happen to hate that setup, and would find working in such an environment horrible. But luckily, I don't work for those companies, and they don't listen to me, either, so everyone's happy. That does not mean I would go out and tell them they're wrong to use such a centralised, single-repository setup. For their use, in their situation, for their requirements, that may be the better option. For mine, not so much.In the end, what we need to realise is that there is no single tool that will fit all needs. There are tasks where Git is a terrible choice, but that does not make it terrible for all cases. Similarly, there are cases where Subversion is superior (I assume - I didn't have the misfortune of meeting with such a case).Instead of bashing the DVCS world, and assuming the Big Company use cases are all that matter, and that everyone uses the tools the same way, accept that some use it to achieve things you didn't think of, jobs you didn't consider.

    28 February 2015

    Zlatan Todori : Interviews with FLOSS developers: Joey Hess

    Edit: Now translated to Chinese. Thanks zhang wei! There is really hardly a better way to open a series of interviewing with developers behind Free Libre Open Source Software project, then with incredible mind such as Joey Hess. To write his contributions to Free software ecosystem, especially in Debian, would be a book by itself. His impact exceeds even his projects - people literally follow his blog posts to see what he is doing and how is he living. A hacker from cabin. If you really need to have a picture of true hacker, then Joey is the one. As this isn't a book I will just mention few projects that he has been behind - git-annex, ikiwiki, etckeeper, debian installer, parts of dpkg, debhelper, devscripts, taskel. So without further waiting here it is. Picture of Joey Hess me: Who are you? joeyh: I'm Joey -- https://joeyh.name/ me: How did you start programming? joeyh: Atari 130XE which came with BASIC and a boring word processor and not much else. No other friends had one, so the only way to get software was to type in demo programs from manual and then begin to change and write my own. So, the easy way to learn. Also some Logo in school. me: How would you now advise others to start programming? joeyh: Difficult question, it seems much harder to get an intimate understanding of things than when I started, and much harder to be motivated to program when there's so much stuff easily available. Maybe simple bare-metal systems like Arduino coupled with real-world interaction are the answer. I've recently been mentoring my nephew who is learning python and Python the Hard Way has gotten him far impressively fast. me: Setup of your development machine? joeyh: Lenovo laptop de-spywared with Debian unstable, xmonad, xfce, vim. me: Your thoughts on Purism (the open hardware laptop initiative that got recently funded on CrowdSupply)? joeyh: I don't know much about that one, but it seems that consumer level hardware has gotten so low quality, and so closed and untrustworthy that it makes sense to either build alternatives that are open, or pick out, as a community, the stuff we can adapt to our needs and concentrate on it. Several projects are trying, I hope they succeed. me: How do you see future of Debian development? joeyh: Well, I've mostly stopped worrying about it. If you look back at my presentations at the past 2 or 3 DebConfs, you'll find my best thoughts on the matter. me: You retired as Debian developer - do you intend sometime soon to come back and/or do you plan to join some other communities? joeyh: It would be glorious to come back, wouldn't it? But I don't think I will. Can't step in the same river twice, and all. Instead, Debian will probably have to put up with me as an annoying upstream author who doesn't ship tarballs, but does ship debian/ directories, and as a bug reporter who enjoys reporting amusing bugs like -0 NaN. I seem to have more time to spend in other online communities since I left Debian, but in a more diffuse way. Maybe that's just what it's like, to be involved in Free Software but not in the embrace of a big project like Debian. me: Some memorable moments from Debian conferences? joeyh: There are so many! Picnicing on berries and tamales at the Portland farmer's market right outside the venue; rainbows and bonfire in Switzerland after crazy busy days; impromptu pipe organ repair in a weird night venue in Edinburgh; walking through Porto Alegre at night with Ian Murdock and how humble he was about what he'd started; hacking all night in Spain; failing to sleep through midnight sun and incessent partying Finland; hanging out in the hotel lobby in Atlanta where we designed Build-Depends. me: Are you a gamer? Valve Steam games are offered for free to Debian Developers - do you use steam and play Valve games? joeyh: I've played through Half Life and Portal, but nethack has claimed more of my time. I mostly enjoy short, indie games, or games that tell us something new about the medium of games, A recent favorite was A Dark Room. But really, I have more pure fun playing real world Tabletop games with friends, like Carcassanne Discovery and Hive. In March, I am going to try to write a roguelike game in one week, in Haskell, for the Seven Day Roguelike Challenge and I'll be blogging about my progress daily. me: You are nowdays a Haskell hacker (git-annex) - what would you like to say about this language and how does it compare to Python, C, JavaScript, Ruby and Perl? joeyh: Not just git-annex; all my current projects are written in Haskell. I think it's amazing how much we expect programmers to keep in their heads while writing code. Is that buffer going to overflow? Is changing the value of that global variable going to break some other part of the code? Is that input sanitized yet? Did that interface change? Haskell solves some of these outright, but more, it makes you start noticing this kind of pervasive issue, and it provides ways to completely eliminate a class of problems from your code. For example http://joeyh.name/blog/entry/making_propellor_safer_with_GADTs_and_type_families/. The class of bugs I avoided there had never affected my code even once, but it was still worth preventing that whole class of bugs, so I don't have to worry about them ever again. me: Would you suggest Haskell as first language to learn especially for those that have an itch for mathematics? joeyh: I think that can work well. Or it can go other the way -- I had an affinity to mathematics when I was young, but it got knocked out of me in the way that happens to many people, and languages like perl and C don't do much to make you want to learn more about higher-order math. I've been picking up a bit more here and there via Haskell. me: How do you compare your productivity in Haskell compared to your Perl days? joeyh: It's very different; I'm a very different programmer now. I probably would bang out quick hacks more quickly when I was writing Perl. But, they tended to stay quick hacks. Now, I might take a little longer to get there, but the code seems a lot more solid, while also being more malleable to turn into larger or different programs. I'm also a lot more drawn toward writing software libraries. me: Can you describe your philosophy of life (you live in cabin, in forest, using a lot of solar power - many people are intrigued (including myself) what drives you towards that kind of life and how does it impact your overall quality of life and happiness. Looking the todays modern predator capitalistic society, in which you could easily earn more then $10.000 a month, you seem to be an anarchist and very humble human)? joeyh: I want to build worthwhile things that might last. Which is super hard in the world of software, both because it's hard to think far ahead at all, and because most jobs don't emphasize that kind of real value. I've been lucky and bootstrapped up to a point where I've been able to work full time on free software for years, and I'm willing to forgo a lot to continue that. Living in the woods without modern conveniences is great, because it's quiet and you can think as much as you like; the internet is just as close as it is anywhere else (maybe a bit slower); and when you've spent too much time quietly thinking you'll need to go chop wood, or haul water, or jump in the river to cool off, depending on the season. (Humble? Like most programmers, I am internally a flaming tower of ego...)
    Vote on Hacker News

    Dirk Eddelbuettel: RcppEigen 0.3.2.4.0

    A new release of RcppEigen is now on CRAN and in Debian. It synchronizes the Eigen code with the 3.2.4 upstream release, and updates the RcppEigen.package.skeleton() package creation helper to use the kitten() function from pkgKitten for enhanced package creation. The NEWS file entry follows.
    Changes in RcppEigen version 0.3.2.4.0 (2015-02-23)
    • Updated to version 3.2.4 of Eigen
    • Update RcppEigen.package.skeleton() to use pkgKitten if available
    Courtesy of CRANberries, there is also a diffstat report for the most recent release.

    This post by Dirk Eddelbuettel originated on his Thinking inside the box blog. Please report excessive re-aggregation in third-party for-profit settings.

    27 February 2015

    Enrico Zini: python-api-stability

    Another day in the life of a poor developer 
    try:
    # After Python 3.3
    from collections.abc import Iterable
except ImportError:
    # This has changed in Python 3.3 (why, oh why?), reinforcing the idea that
    # the best Python version ever is still 2.7, simply because upstream has
    # promised that they won't touch it (and break it) for at least 5 more
    # years.
    from collections import Iterable
import shlex
if hasattr(shlex, "quote"):
    # New in version 3.3.
    shell_quote = shlex.quote
else:
    # Available since python 1.6 but deprecated since version 2.7: Prior to Python
    # 2.7, this function was not publicly documented. It is finally exposed
    # publicly in Python 3.3 as the quote function in the shlex module.
    #
    # Except everyone was using it, because it was the only way provided by the
    # python standard library to make a string safe for shell use
    #
    # See http://stackoverflow.com/questions/35817/how-to-escape-os-system-calls-in-python
    import pipes
    shell_quote = pipes.quote
import shutil
if hasattr(shutil, "which"):
    # New in version 3.3.
    shell_which = shutil.which
else:
    # Available since python 1.6:
    # http://stackoverflow.com/questions/377017/test-if-executable-exists-in-python
    from distutils.spawn import find_executable
    shell_which = find_executable

    Dirk Eddelbuettel: RcppArmadillo 0.4.650.1.1 (and also 0.4.650.2.0)

    A new Armadillo release 4.650.1 was released by Conrad a few days ago. Armadillo is a powerful and expressive C++ template library for linear algebra aiming towards a good balance between speed and ease of use with a syntax deliberately close to a Matlab. It turned out that this release had one shortcoming with respect to the C++11 RNG initializations in the R use case (where we need to protect the users from the C++98 RNG deemed unsuitable by the CRAN gatekeepers). And this lead to upstream release 4.650.1 which we wrapped into RcppArmadillo 0.4.650.1.1. As before this, was tested against all 107 reverse dependencies of RcppArmadillo on the CRAN repo. This version is now on CRAN, and was just uploaded to Debian. Its changes are summarized below based on the NEWS.Rd file.
    Changes in RcppArmadillo version 0.4.650.1.1 (2015-02-25)
    • Upgraded to Armadillo release Version 4.650.1 ("Intravenous Caffeine Injector")
      • added randg() for generating random values from gamma distributions (C++11 only)
      • added .head_rows() and .tail_rows() to submatrix views
      • added .head_cols() and .tail_cols() to submatrix views
      • expanded eigs_sym() to optionally calculate eigenvalues with smallest/largest algebraic values fixes for handling of sparse matrices
    • Applied small correction to main header file to set up C++11 RNG whether or not the alternate RNG (based on R, our default) is used
    Now, it turns out that another small fix was needed for the corner case of a submatrix within a submatrix, ie V.subvec(1,10).tail(5). I decided not to re-release this to CRAN given the CRAN Repository Policy preference for releases no more than every 1 2 months . But fear not, for we now have drat. I created a drat package repository in the RcppCore account (to not put a larger package into my main drat repository often used via a fork to initialize a drat). So now with these two simple commands 
    ## if needed, first install 'drat' via:   install.packages("drat")
drat:::add("RcppCore")
update.packages()
    you will get the newest RcppArmadillo via this drat package repository. And course install.packages("RcppArmadillo") would also work, but takes longer to type :) Lastly, courtesy of CRANberries, there is also a diffstat report for the most recent CRAN release. As always, more detailed information is on the RcppArmadillo page. Questions, comments etc should go to the rcpp-devel mailing list off the R-Forge page.

    This post by Dirk Eddelbuettel originated on his Thinking inside the box blog. Please report excessive re-aggregation in third-party for-profit settings.

    23 February 2015

    Enrico Zini: akonadi-client-example

    Akonadi client example After many failed attemps I have managed to build a C++ akonadi client. It has felt like one of the most frustrating programming experiences of my whole life, so I'm sharing the results hoping to spare others from all the suffering. First thing first, akonadi client libraries are not in libakonadi-dev but in kdepimlibs5-dev, even if kdepimlibs5-dev does not show in apt-cache search akonadi. Then, kdepimlibs is built with Qt4. If your application uses Qt5 (mine was) you need to port it back to Qt4 if you want to talk to Akonadi. Then, kdepimlibs does not seem to support qmake and does not ship pkg-config .pc files, and if you want to use kdepimlibs your build system needs to be cmake. I ported by code from qmake to cmake, and now qtcreator wants me to run cmake by hand every time I change the CMakeLists.txt file, and it stopped allowing to add, rename or delete sources. Finally, most of the code / build system snippets found on the internet seem flawed in a way or another, because the build toolchain of Qt/KDE applications has undergone several redesignins during time, and the network is littered with examples from different eras. The way to obtain template code to start a Qt/KDE project is to use kapptemplate. I have found no getting started tutorial on the internet that said "do not just copy the snippets from here, run kapptemplate instead so you get them up to date". kapptemplate supports building an "Akonadi Resource" and an "Akonadi Serializer", but it does not support generating template code for an akonadi client. That left me with the feeling that I was dealing with some software that wants to be developed but does not want to be used. Anyway, now an example of how to interrogate Akonadi exists as is on the internet. I hope that all the tears of blood that I cried this morning have not been cried in vain.

    Enrico Zini: akonadi-build-hth

    The wonders of missing documentation Update: I have managed to build an example Akonadi client application. I'm new here, I want to make a simple C++ GUI app that pops up a QCalendarWidget which my local Akonadi has appointments. I open qtcreator, create a new app, hack away for a while, then of course I get undefined references for all Akonadi symbols, since I didn't tell the build system that I'm building with akonadi. Ok. How do I tell the build system that I'm building with akonadi? After 20 minutes of frantic looking around the internet, I still have no idea. There is a package called libakonadi-dev which does not seem to have anything to do with this. That page mentions everything about making applications with Akonadi except how to build them. There is a package called kdepimlibs5-dev which looks promising: it has no .a files but it does have haders and cmake files. However, qtcreator is only integrated with qmake, and I would really like the handholding of an IDE at this stage. I put something together naively doing just what looked right, and I managed to get an application that segfaults before main() is even called: 
    /*
 * Copyright   2015 Enrico Zini <enrico@enricozini.org>
 *
 * This work is free. You can redistribute it and/or modify it under the
 * terms of the Do What The Fuck You Want To Public License, Version 2,
 * as published by Sam Hocevar. See the COPYING file for more details.
 */
#include <QDebug>
int main(int argc, char *argv[])
 
    qDebug() << "BEGIN";
    return 0;
 

    QT       += core gui widgets
CONFIG += c++11
TARGET = wtf
TEMPLATE = app
LIBS += -lkdecore -lakonadi-kde
SOURCES += wtf.cpp
    I didn't achieve what I wanted, but I feel like I achieved something magical and beautiful after all. I shall now perform some haruspicy on those oscure cmake files to see if I can figure something out. But seriously, people?

    Dirk Eddelbuettel: drat Tutorial: Publishing a package

    Introduction The drat package was released earlier this month, and described in a first blog post. I received some helpful feedback about what works and what doesn't. For example, Jenny Bryan pointed out that I was not making a clear enough distinction between the role of using drat to publish code, and using drat to receive/install code. Very fair point, and somewhat tricky as R aims to blur the line between being a user and developer of statistical analyses, and hence packages. Many of us are both. Both the main point is well taken, and this note aims to clarify this issue a little by focusing on the former. Another point make by Jenny concerns the double use of repository. And indeed, I conflated repository (in the sense of a GitHub code repository) with repository for a package store used by a package manager. The former, a GitHub repository, is something we use to implement a personal drat with: A GitHub repository happens to be uniquely identifiable just by its account name, and given an (optional) gh-pages branch also offers a stable and performant webserver we use to deliver packages for R. A (personal) code repository on the other hand is something we implement somewhere---possibly via drat which supports local directories, possibly on a network share, as well as anywhere web-accessible, e.g. via a GitHub repository. It is a little confusing, but I will aim to make the distinction clearer.

    Just once: Setting up a drat repository So let us for the remainder of this post assume the role of a code publisher. Assume you have a package you would like to make available, which may not be on CRAN and for which you would like to make installation by others easier via drat. The example below will use an interim version of drat which I pushed out yesterday (after fixing a bug noticed when pushing the very new RcppAPT package). For the following, all we assume (apart from having a package to publish) is that you have a drat directory setup within your git / GitHub repository. This is not an onerous restriction. First off, you don't have to use git or GitHub to publish via drat: local file stores and other web servers work just as well (and are documented). GitHub simply makes it easiest. Second, bootstrapping one is trivial: just fork my drat GitHub repository and then create a local clone of the fork. There is one additional requirement: you need a gh-pages branch. Using the fork-and-clone approach ensures this. Otherwise, if you know your way around git you already know how to create a gh-pages branch. Enough of the prerequisities. And on towards real fun. Let's ensure we are in the gh-pages branch: 
    edd@max:~/git/drat(master)$ git checkout gh-pages
Switched to branch 'gh-pages'
Your branch is up-to-date with 'origin/gh-pages'.
edd@max:~/git/drat(gh-pages)$

    Publish: Run one drat command to insert a package Now, let us assume you have a package to publish. In my case this was version 0.0.1.2 of drat itself as it contains a fix for the very command I am showing here. So if you want to run this, ensure you have this version of drat as the CRAN version is currently behind at release 0.0.1 (though I plan to correct that in the next few days). To publish an R package into a code repository created via drat running on a drat GitHub repository, just run insertPackage(packagefile) which we show here with the optional commit=TRUE. The path to the package can be absolute are relative; the easists is often to go up one directory from the sources to where R CMD build ... has created the package file. 
    edd@max:~/git$ Rscript -e 'library(drat); insertPackage("drat_0.0.1.2.tar.gz", commit=TRUE)'
[gh-pages 0d2093a] adding drat_0.0.1.2.tar.gz to drat
 3 files changed, 2 insertions(+), 2 deletions(-)
 create mode 100644 src/contrib/drat_0.0.1.2.tar.gz
Counting objects: 7, done.
Delta compression using up to 8 threads.
Compressing objects: 100% (7/7), done.
Writing objects: 100% (7/7), 7.37 KiB   0 bytes/s, done.
Total 7 (delta 1), reused 0 (delta 0)
To git@github.com:eddelbuettel/drat.git
   206d2fa..0d2093a  gh-pages -> gh-pages
edd@max:~/git$
    You can equally well run this as insertPackage("drat_0.0.1.2.tar.gz"), then inspect the repo and only then run the git commands add, commit and push. Also note that future versions of drat will most likely support git operations directly by relying on the very promising git2r package. But this just affect package internals, the user-facing call of e.g. insertPackage("drat_0.0.1.2.tar.gz", commit=TRUE) will remain unchanged. And in a nutshell that really is all there is to it. With the newly drat-ed package pushed to your GitHub repository with a single function call), it is available via the automatically-provided gh-pages webserver access to anyone in the world. All they need to do is to point R's package management code (which is built into R itself and used for e.g._ CRAN and BioConductor R package repositories) to the new repo---and that is also just a single drat command. We showed this in the first blog post and may expand on it again in a follow-up. So in summary, that really is all there is to it. After a one-time setup / ensuring you are on the gh-pages branch, all it takes is a single function call from the drat package to publish your package to your drat GitHub repository.

    This post by Dirk Eddelbuettel originated on his Thinking inside the box blog. Please report excessive re-aggregation in third-party for-profit settings.

    21 February 2015

    Francesca Ciceri: Dudes in dresses, girls in trousers

    "As long as people still think of people like me as "a dude in a dress" there is a lot work to do to fight transphobia and gain tolerance and acceptance."
    This line in Rhonda's most recent blogpost broke my heart a little, and sparked an interesting conversation with her about the (perceived?) value of clothes, respect and identity. So, guess what? Here's a pic of a "girl in trousers". Just because. MadameZou in her best James Dean impersonation (Sorry for the quality: couldn't find my camera and had to use a phone. Also, I don't own a binder, so I used a very light binding)

    Next.

    Previous.