Search Results: "paultag"

12 December 2016

Paul Tagliamonte: DNSync MAC Addresses

I ve been hacking on a project on and off for my LAN called DNSync. This will take a DNSMasq leases file and sync it to Amazon Route 53. I ve added a new feature, which will create A reccords for each MAC address on the LAN. Since DNSync won t touch CNAME records, I use CNAME records (manually) to point to the auto-synced A records for services on my LAN (such as my Projector, etc). Since It s easy for two machines to have the same name, I ve decided to add A records for each MAC as well as their client name. They take the fomm of something like ab-cd-ef-ab-cd-ef.by-mac.paultag.house., which is harder to accedentally collide.

18 September 2016

Paul Tagliamonte: DNSync

While setting up my new network at my house, I figured I d do things right and set up an IPSec VPN (and a few other fancy bits). One thing that became annoying when I wasn t on my LAN was I d have to fiddle with the DNS Resolver to resolve names of machines on the LAN. Since I hate fiddling with options when I need things to just work, the easiest way out was to make the DNS names actually resolve on the public internet. A day or two later, some Golang glue, and AWS Route 53, and I wrote code that would sit on my dnsmasq.leases, watch inotify for IN_MODIFY signals, and sync the records to AWS Route 53. I pushed it up to my GitHub as DNSync. PRs welcome!

05 September 2016

Paul Tagliamonte: go-haversine

In the spirit of blogging about some of the code i ve written in the past year or two, I wrote a small utility library called go-haversine, which uses the Haversine Forumla to compute the distance between two points. This is super helpful when working with GPS data - but remember, this assumes everything s squarely on the face of the planet.

22 August 2016

Paul Tagliamonte: go-wmata - golang bindings to the DC metro system

A few weeks ago, I hacked up go-wmata, some golang bindings to the WMATA API. This is super handy if you are in the DC area, and want to interface to the WMATA data. As a proof of concept, I wrote a yo bot called @WMATA, where it returns the closest station if you Yo it your location. For hilarity, feel free to Yo it from outside DC. For added fun, and puns, I wrote a dbus proxy for the API as weel, at wmata-dbus, so you can query the next train over dbus. One thought was to make a GNOME Shell extension to tell me when the next train is. I d love help with this (or pointers on how to learn how to do this right).

15 August 2016

Paul Tagliamonte: Minica - lightweight TLS for everyone!

A while back, I found myself in need of some TLS certificates set up and issued for a testing environment. I remembered there was some code for issuing TLS certs in Docker, so I yanked some of that code and made a sensable CLI API over it. Thus was born minica! Something as simple as minica tag@domain.tls domain.tld will issue two TLS certs (one with a Client EKU, and one server) issued from a single CA. Next time you re in need of a few TLS keys (without having to worry about stuff like revocation or anything), this might be the quickest way out!

08 August 2016

Paul Tagliamonte: Using PKCS#11 on GNU/Linux

PKCS#11 is a standard API to interface with HSMs, Smart Cards, or other types of random hardware backed crypto. On my travel laptop, I use a few Yubikeys in PKCS#11 mode using OpenSC to handle system login. libpam-pkcs11 is a pretty easy to use module that will let you log into your system locally using a PKCS#11 token locally. One of the least documented things, though, was how to use an OpenSC PKCS#11 token in Chrome. First, close all web browsers you have open.
sudo apt-get install libnss3-tools
certutil -U -d sql:$HOME/.pki/nssdb
modutil -add "OpenSC" -libfile /usr/lib/x86_64-linux-gnu/opensc-pkcs11.so -dbdir sql:$HOME/.pki/nssdb
modutil -list "OpenSC" -dbdir sql:$HOME/.pki/nssdb 
modutil -enable "OpenSC" -dbdir sql:$HOME/.pki/nssdb
Now, we'll have the PKCS#11 module ready for nss to use, so let's double check that the tokens are registered:
certutil -U -d sql:$HOME/.pki/nssdb
certutil -L -h "OpenSC" -d sql:$HOME/.pki/nssdb
If this winds up causing issues, you can remove it using the following command:
modutil -delete "OpenSC" -dbdir sql:$HOME/.pki/nssdb

31 July 2016

Paul Tagliamonte: Hacking a Projector in Hy

About a year ago, I bought a Projector after I finally admitted that I could actually use a TV in my apartment. I settled on buying a ViewSonic PJD5132. It was a really great value, and has been nothing short of a delight to own. I was always a bit curious about the DB9 connector on the back of the unit, so I dug into the user manual, and found some hex code strings in there. So, last year, between my last gig at the Sunlight Foundtion and USDS, I spent some time wandering around the US, hitting up DebConf, and exploring Washington DC. Between trips, I set out to figure out exactly what was going on with my Projector, and see if I could make it do anything fun. So, I started off with basics, and tried to work out how these command codes were structured. I had a few working codes, but to write clean code, I'd be better off understanding why the codes looked like they do. Let's look at the "Power On" code. 0x06 0x14 0x00 0x04 0x00 0x34 0x11 0x00 0x00 0x5D Some were 10 bytes, other were 11, and most started with similar looking things. The first byte was usually a 0x06 or 0x07, followed by two bytes 0x14 0x00, and either a 0x04 or 0x05. Since the first few bytes were similarly structured, I assumed the first octet (either 0x06 or 0x07) was actually a length, since the first 4 octets seemed always present. So, my best guess is that we have a Length byte at index 0, followed by two bytes for the Protocol, a flag for if you're Reading or Writing (best guess on that one), and opaque data following that. Sometimes it's a const of sorts, and sometimes an octet (either little or big endian, confusingly).
Length
           Read / Write
                 
     Protocol                Data
       ----           ------------------------ 
0x06 0x14 0x00 0x04 0x00 0x34 0x11 0x00 0x00 0x5D
Right. OK. So, let's get to work. In the spirit of code is data, data is code, I hacked up some of the projector codes into a s-expression we can use later. The structure of this is boring, but it'll let us both store the command code to issue, as well as define the handler to read the data back.
(setv *commands*
  ;  function                       type family         control
  '((power-on                         nil nil            (0x06  0x14 0x00  0x04  0x00 0x34 0x11 0x00 0x00 0x5D))
    (power-off                        nil nil            (0x06  0x14 0x00  0x04  0x00 0x34 0x11 0x01 0x00 0x5E))
    (power-status                   const power          (0x07  0x14 0x00  0x05  0x00 0x34 0x00 0x00 0x11 0x00 0x5E))
    (reset                            nil nil            (0x06  0x14 0x00  0x04  0x00 0x34 0x11 0x02 0x00 0x5F))
    ...
As well as defining some of the const responses that come back from the Projector itself. These are pretty boring, but it's helpful to put a name to the int that falls out.
(setv *consts*
  '((power        ((on           (0x00 0x00 0x01))
                   (off          (0x00 0x00 0x00))))
    (freeze       ((on           (0x00 0x00 0x01))
                   (off          (0x00 0x00 0x00))))
    ...
After defining a few simple functions to write the byte arrays to the serial port as well as reading and understanding responses from the projector, I could start elaborating on some higher order functions that can talk projector. So the first thing I wrote was to make a function that converts the command entry into a native Hy function.
(defn make-api-function [function type family data]
   (defn ~function [serial]
      (import [PJD5132.dsl [interpret-response]]
              [PJD5132.serial [read-response/raw]])
      (serial.write (bytearray [~@data]))
      (interpret-response ~(str type) ~(str family) (read-response/raw serial))))
Fun. Fun! Now, we can invoke it to create a Hy & Python importable API wrapper in a few lines!
(import [PJD5132.commands [*commands*]]
        [PJD5132.dsl [make-api-function]])
(list (map (fn [(, function type family command)]
               (make-api-function function type family command)) *commands*)))
Cool. So, now we can import things like power-on from *commands* which takes a single argument (serial) for the serial port, and it'll send a command, and return the response. The best part about all this is you only have to define the data once in a list, and the rest comes for free. Finally, I do want to be able to turn my projector on and off over the network so I went ahead and make a Flask "API" on top of this. First, let's define a macro to define Flask routes:
(defmacro defroute [name root &rest methods]
  (import os.path)
  (defn generate-method [path method status]
     (with-decorator (app.route ~path) (fn []
       (import [PJD5132.api [~method ~(if status status method)]])
       (try (do (setv ret (~method serial-line))
               ~(if status  (setv ret (~status serial-line)))
                (json.dumps ret))
       (except [e ValueError]
          (setv response (make-response (.format "Fatal Error: ValueError:  " (str e))))
          (setv response.status-code 500)
          response)))))
  (setv path (.format "/projector/ " name))
  (setv actions (dict methods))
   (do ~(generate-method path root nil)
       ~@(list-comp (generate-method (os.path.join path method-path) method root)
                    [(, method-path method) methods])))
Now, we can define how we want our API to look, so let's define the power route, which will expand out into the Flask route code above.
(defroute power
  power-status
  ("on"  power-on)
  ("off" power-off))
And now, let's play with it!
$ curl http://192.168.1.50/projector/power
"off"
$ curl http://192.168.1.50/projector/power/on
"on"
$ curl http://192.168.1.50/projector/power
"on"
Or, the volume!
$ curl 192.168.1.50/projector/volume
10
$ curl 192.168.1.50/projector/volume/decrease
9
$ curl 192.168.1.50/projector/volume/decrease
8
$ curl 192.168.1.50/projector/volume/decrease
7
$ curl 192.168.1.50/projector/volume/increase
8
$ curl 192.168.1.50/projector/volume/increase
9
$ curl 192.168.1.50/projector/volume/increase
10
Check out the full source over at github.com/paultag/PJD5132!

22 July 2016

Paul Tagliamonte: HOPE 11

I ll be at HOPE 11 this year - if anyone else will be around, feel free to send me an email! I won t have a phone on me (so texting only works if you use Signal!) Looking forward for a chance to see everyone soon!

16 July 2016

Paul Tagliamonte: The Open Source License API

Around a year ago, I started hacking together a machine readable version of the OSI approved licenses list, and casually picking parts up until it was ready to launch. A few weeks ago, we officially announced the osi license api, which is now live at api.opensource.org. I also took a whack at writing a few API bindings, in Python, Ruby, and using the models from the API implementation itself in Go. In the following few weeks, Clint wrote one in Haskell, Eriol wrote one in Rust, and Oliver wrote one in R. The data is sourced from a repo on GitHub, the licenses repo under OpenSourceOrg. Pull Requests against that repo are wildly encouraged! Additional data ideas, cleanup or more hand collected data would be wonderful! In the meantime, use-cases for using this API range from language package managers pulling OSI approval of a licence programatically to using a license identifier as defined in one dataset (SPDX, for exampele), and using that to find the identifer as it exists in another system (DEP5, Wikipedia, TL;DR Legal). Patches are hugly welcome, as are bug reports or ideas! I'd also love more API wrappers for other languages!

10 July 2016

Paul Tagliamonte: SNIff

A while back, I found myself in need of two webservers that would terminate TLS (with different rules). I wanted to run some custom code I d written (which uses TLS peer authentication), and also nginx on port 443. The best way I figured out how to do this was to write a tool to sit on port 443, and parse TLS Client Hello packets, and dispatch to the correct backend depending on the SNI name. SNI, or Server Name Indication allows the client to announce (yes over cleartext!) what server it s looking for, similar to the HTTP Host header. Sometimes, like in the case above, the Host header won t work, since you ve already done a TLS handshake by the time you figure out who they re looking for. I also spun the Client Hello parser out into its own importable package, just in case someone else finds themselves in this same boat. The code s up on github.com/paultag/sniff!

02 July 2016

Paul Tagliamonte: Hello, InfluxDB

Last week, I posted about python-sense, and API wrapper for the internal Sense API. I wrote this so that I could pull data about myself into my own databases, allowing me to use that information for myself. One way I'm doing this is by pulling my room data into an InfluxDB database, letting me run time series queries against my environmental data.
#!/usr/bin/env python
from influxdb import InfluxDBClient
import json
import datetime as dt
from sense.service import Sense
api = Sense()
data = api.room_sensors(quantity=20)
def items(data):
    for flavor, series in data.items():
        for datum in reversed(series):
            value = datum['value']
            if value == -1:
                continue
            timezone = dt.timezone(dt.timedelta(
                seconds=datum['offset_millis'] / 1000,
            ))
            when = dt.datetime.fromtimestamp(
                datum['datetime'] / 1000,
            ).replace(tzinfo=timezone)
            yield flavor, when, value
client = InfluxDBClient(
    'url.to.host.here',
    443,
    'username',
    'password',
    'sense',
    ssl=True,
)
def series(data):
    for flavor, when, value in items(data):
        yield  
            "measurement": " ".format(flavor),
            "tags":  
                "user": "paultag"
             ,
            "time": when.isoformat(),
            "fields":  
                "value": value,
             
         
client.write_points(list(series(data)))
I'm able to run this on a cron, automatically loading data from the Sense API into my Influx database. I can then use that with something like Grafana, to check out what my room looks like over time.

27 June 2016

Paul Tagliamonte: Hello, Sense!

A while back, I saw a Kickstarter for one of the most well designed and pretty sleep trackers on the market. I fell in love with it, and it has stuck with me since. A few months ago, I finally got my hands on one and started to track my data. Naturally, I now want to store this new data with the rest of the data I have on myself in my own databases. I went in search of an API, but I found that the Sense API hasn't been published yet, and is being worked on by the team. Here's hoping it'll land soon! After some subdomain guessing, I hit on api.hello.is. So, naturally, I went to take a quick look at their Android app and network traffic, lo and behold, there was a pretty nicely designed API. This API is clearly an internal API, and as such, it's something that should not be considered stable. However, I'm OK with a fragile API, so I've published a quick and dirty API wrapper for the Sense API to my GitHub.. I've published it because I've found it useful, but I can't promise the world, (since I'm not a member of the Sense team at Hello!), so here are a few ground rules of this wrapper: This module is currently Python 3 only. If someone really needs Python 2 support, I'm open to minimally invasive patches to the codebase using six to support Python 2.7. Working with the API: First, let's go ahead and log in using python -m sense.
$ python -m sense
Sense OAuth Client ID: xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx
Sense OAuth Client Secret: xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx
Sense email: paultag@gmail.com
Sense password: 
Attempting to log into Sense's API
Success!
Attempting to query the Sense API
The humidity is **just right**.
The air quality is **just right**.
The light level is **just right**.
It's **pretty hot** in here.
The noise level is **just right**.
Success!
Now, let's see if we can pull up information on my Sense:
>>> from sense import Sense
>>> sense = Sense()
>>> sense.devices()
 'senses': [ 'id': 'xxxxxxxxxxxxxxxx', 'firmware_version': '11a1', 'last_updated': 1466991060000, 'state': 'NORMAL', 'wifi_info':  'rssi': 0, 'ssid': 'Pretty Fly for a WiFi (2.4 GhZ)', 'condition': 'GOOD', 'last_updated': 1462927722000 , 'color': 'BLACK' ], 'pills': [ 'id': 'xxxxxxxxxxxxxxxx', 'firmware_version': '2', 'last_updated': 1466990339000, 'battery_level': 87, 'color': 'BLUE', 'state': 'NORMAL' ] 
Neat! Pretty cool. Look, you can even see my WiFi AP! Let's try some more and pull some trends out.
>>> values = [x.get("value") for x in sense.room_sensors()["humidity"]][:10]
>>> min(values)
45.73904
>>> max(values)
45.985928
>>> 
I plan to keep maintaining it as long as it's needed, so I welcome co-maintainers, and I'd love to see what people build with it! So far, I'm using it to dump my room data into InfluxDB, pulling information on my room into Grafana. Hopefully more to come! Happy hacking!

19 June 2016

Paul Tagliamonte: Go Debian!

As some of the world knows full well by now, I've been noodling with Go for a few years, working through its pros, its cons, and thinking a lot about how humans use code to express thoughts and ideas. Go's got a lot of neat use cases, suited to particular problems, and used in the right place, you can see some clear massive wins. I've started writing Debian tooling in Go, because it's a pretty natural fit. Go's fairly tight, and overhead shouldn't be taken up by your operating system. After a while, I wound up hitting the usual blockers, and started to build up abstractions. They became pretty darn useful, so, this blog post is announcing (a still incomplete, year old and perhaps API changing) Debian package for Go. The Go importable name is pault.ag/go/debian. This contains a lot of utilities for dealing with Debian packages, and will become an edited down "toolbelt" for working with or on Debian packages. Module Overview Currently, the package contains 4 major sub packages. They're a changelog parser, a control file parser, deb file format parser, dependency parser and a version parser. Together, these are a set of powerful building blocks which can be used together to create higher order systems with reliable understandings of the world. changelog The first (and perhaps most incomplete and least tested) is a changelog file parser.. This provides the programmer with the ability to pull out the suite being targeted in the changelog, when each upload was, and the version for each. For example, let's look at how we can pull when all the uploads of Docker to sid took place:
func main()  
    resp, err := http.Get("http://metadata.ftp-master.debian.org/changelogs/main/d/docker.io/unstable_changelog")
    if err != nil  
        panic(err)
     
    allEntries, err := changelog.Parse(resp.Body)
    if err != nil  
        panic(err)
     
    for _, entry := range allEntries  
        fmt.Printf("Version %s was uploaded on %s\n", entry.Version, entry.When)
     
 
The output of which looks like:
Version 1.8.3~ds1-2 was uploaded on 2015-11-04 00:09:02 -0800 -0800
Version 1.8.3~ds1-1 was uploaded on 2015-10-29 19:40:51 -0700 -0700
Version 1.8.2~ds1-2 was uploaded on 2015-10-29 07:23:10 -0700 -0700
Version 1.8.2~ds1-1 was uploaded on 2015-10-28 14:21:00 -0700 -0700
Version 1.7.1~dfsg1-1 was uploaded on 2015-08-26 10:13:48 -0700 -0700
Version 1.6.2~dfsg1-2 was uploaded on 2015-07-01 07:45:19 -0600 -0600
Version 1.6.2~dfsg1-1 was uploaded on 2015-05-21 00:47:43 -0600 -0600
Version 1.6.1+dfsg1-2 was uploaded on 2015-05-10 13:02:54 -0400 EDT
Version 1.6.1+dfsg1-1 was uploaded on 2015-05-08 17:57:10 -0600 -0600
Version 1.6.0+dfsg1-1 was uploaded on 2015-05-05 15:10:49 -0600 -0600
Version 1.6.0+dfsg1-1~exp1 was uploaded on 2015-04-16 18:00:21 -0600 -0600
Version 1.6.0~rc7~dfsg1-1~exp1 was uploaded on 2015-04-15 19:35:46 -0600 -0600
Version 1.6.0~rc4~dfsg1-1 was uploaded on 2015-04-06 17:11:33 -0600 -0600
Version 1.5.0~dfsg1-1 was uploaded on 2015-03-10 22:58:49 -0600 -0600
Version 1.3.3~dfsg1-2 was uploaded on 2015-01-03 00:11:47 -0700 -0700
Version 1.3.3~dfsg1-1 was uploaded on 2014-12-18 21:54:12 -0700 -0700
Version 1.3.2~dfsg1-1 was uploaded on 2014-11-24 19:14:28 -0500 EST
Version 1.3.1~dfsg1-2 was uploaded on 2014-11-07 13:11:34 -0700 -0700
Version 1.3.1~dfsg1-1 was uploaded on 2014-11-03 08:26:29 -0700 -0700
Version 1.3.0~dfsg1-1 was uploaded on 2014-10-17 00:56:07 -0600 -0600
Version 1.2.0~dfsg1-2 was uploaded on 2014-10-09 00:08:11 +0000 +0000
Version 1.2.0~dfsg1-1 was uploaded on 2014-09-13 11:43:17 -0600 -0600
Version 1.0.0~dfsg1-1 was uploaded on 2014-06-13 21:04:53 -0400 EDT
Version 0.11.1~dfsg1-1 was uploaded on 2014-05-09 17:30:45 -0400 EDT
Version 0.9.1~dfsg1-2 was uploaded on 2014-04-08 23:19:08 -0400 EDT
Version 0.9.1~dfsg1-1 was uploaded on 2014-04-03 21:38:30 -0400 EDT
Version 0.9.0+dfsg1-1 was uploaded on 2014-03-11 22:24:31 -0400 EDT
Version 0.8.1+dfsg1-1 was uploaded on 2014-02-25 20:56:31 -0500 EST
Version 0.8.0+dfsg1-2 was uploaded on 2014-02-15 17:51:58 -0500 EST
Version 0.8.0+dfsg1-1 was uploaded on 2014-02-10 20:41:10 -0500 EST
Version 0.7.6+dfsg1-1 was uploaded on 2014-01-22 22:50:47 -0500 EST
Version 0.7.1+dfsg1-1 was uploaded on 2014-01-15 20:22:34 -0500 EST
Version 0.6.7+dfsg1-3 was uploaded on 2014-01-09 20:10:20 -0500 EST
Version 0.6.7+dfsg1-2 was uploaded on 2014-01-08 19:14:02 -0500 EST
Version 0.6.7+dfsg1-1 was uploaded on 2014-01-07 21:06:10 -0500 EST
control Next is one of the most complex, and one of the oldest parts of go-debian, which is the control file parser (otherwise sometimes known as deb822). This module was inspired by the way that the json module works in Go, allowing for files to be defined in code with a struct. This tends to be a bit more declarative, but also winds up putting logic into struct tags, which can be a nasty anti-pattern if used too much. The first primitive in this module is the concept of a Paragraph, a struct containing two values, the order of keys seen, and a map of string to string. All higher order functions dealing with control files will go through this type, which is a helpful interchange format to be aware of. All parsing of meaning from the Control file happens when the Paragraph is unpacked into a struct using reflection. The idea behind this strategy that you define your struct, and let the Control parser handle unpacking the data from the IO into your container, letting you maintain type safety, since you never have to read and cast, the conversion will handle this, and return an Unmarshaling error in the event of failure. Additionally, Structs that define an anonymous member of control.Paragraph will have the raw Paragraph struct of the underlying file, allowing the programmer to handle dynamic tags (such as X-Foo), or at least, letting them survive the round-trip through go. The default decoder contains an argument, the ability to verify the input control file using an OpenPGP keyring, which is exposed to the programmer through the (*Decoder).Signer() function. If the passed argument is nil, it will not check the input file signature (at all!), and if it has been passed, any signed data must be found or an error will fall out of the NewDecoder call. On the way out, the opposite happens, where the struct is introspected, turned into a control.Paragraph, and then written out to the io.Writer. Here's a quick (and VERY dirty) example showing the basics of reading and writing Debian Control files with go-debian.
package main
import (
    "fmt"
    "io"
    "net/http"
    "strings"
    "pault.ag/go/debian/control"
)
type AllowedPackage struct  
    Package     string
    Fingerprint string
 
func (a *AllowedPackage) UnmarshalControl(in string) error  
    in = strings.TrimSpace(in)
    chunks := strings.SplitN(in, " ", 2)
    if len(chunks) != 2  
        return fmt.Errorf("Syntax sucks: '%s'", in)
     
    a.Package = chunks[0]
    a.Fingerprint = chunks[1][1 : len(chunks[1])-1]
    return nil
 
type DMUA struct  
    Fingerprint     string
    Uid             string
    AllowedPackages []AllowedPackage  control:"Allow" delim:"," 
 
func main()  
    resp, err := http.Get("http://metadata.ftp-master.debian.org/dm.txt")
    if err != nil  
        panic(err)
     
    decoder, err := control.NewDecoder(resp.Body, nil)
    if err != nil  
        panic(err)
     
    for  
        dmua := DMUA 
        if err := decoder.Decode(&dmua); err != nil  
            if err == io.EOF  
                break
             
            panic(err)
         
        fmt.Printf("The DM %s is allowed to upload:\n", dmua.Uid)
        for _, allowedPackage := range dmua.AllowedPackages  
            fmt.Printf("   %s [granted by %s]\n", allowedPackage.Package, allowedPackage.Fingerprint)
         
     
 
Output (truncated!) looks a bit like:
...
The DM Allison Randal <allison@lohutok.net> is allowed to upload:
   parrot [granted by A4F455C3414B10563FCC9244AFA51BD6CDE573CB]
...
The DM Benjamin Barenblat <bbaren@mit.edu> is allowed to upload:
   boogie [granted by 3224C4469D7DF8F3D6F41A02BBC756DDBE595F6B]
   dafny [granted by 3224C4469D7DF8F3D6F41A02BBC756DDBE595F6B]
   transmission-remote-gtk [granted by 3224C4469D7DF8F3D6F41A02BBC756DDBE595F6B]
   urweb [granted by 3224C4469D7DF8F3D6F41A02BBC756DDBE595F6B]
...
The DM     <aelmahmoudy@sabily.org> is allowed to upload:
   covered [granted by 41352A3B4726ACC590940097F0A98A4C4CD6E3D2]
   dico [granted by 6ADD5093AC6D1072C9129000B1CCD97290267086]
   drawtiming [granted by 41352A3B4726ACC590940097F0A98A4C4CD6E3D2]
   fonts-hosny-amiri [granted by BD838A2BAAF9E3408BD9646833BE1A0A8C2ED8FF]
   ...
...
deb Next up, we've got the deb module. This contains code to handle reading Debian 2.0 .deb files. It contains a wrapper that will parse the control member, and provide the data member through the archive/tar interface. Here's an example of how to read a .deb file, access some metadata, and iterate over the tar archive, and print the filenames of each of the entries.
func main()  
    path := "/tmp/fluxbox_1.3.5-2+b1_amd64.deb"
    fd, err := os.Open(path)
    if err != nil  
        panic(err)
     
    defer fd.Close()
    debFile, err := deb.Load(fd, path)
    if err != nil  
        panic(err)
     
    version := debFile.Control.Version
    fmt.Printf(
        "Epoch: %d, Version: %s, Revision: %s\n",
        version.Epoch, version.Version, version.Revision,
    )
    for  
        hdr, err := debFile.Data.Next()
        if err == io.EOF  
            break
         
        if err != nil  
            panic(err)
         
        fmt.Printf("  -> %s\n", hdr.Name)
     
 
Boringly, the output looks like:
Epoch: 0, Version: 1.3.5, Revision: 2+b1
  -> ./
  -> ./etc/
  -> ./etc/menu-methods/
  -> ./etc/menu-methods/fluxbox
  -> ./etc/X11/
  -> ./etc/X11/fluxbox/
  -> ./etc/X11/fluxbox/window.menu
  -> ./etc/X11/fluxbox/fluxbox.menu-user
  -> ./etc/X11/fluxbox/keys
  -> ./etc/X11/fluxbox/init
  -> ./etc/X11/fluxbox/system.fluxbox-menu
  -> ./etc/X11/fluxbox/overlay
  -> ./etc/X11/fluxbox/apps
  -> ./usr/
  -> ./usr/share/
  -> ./usr/share/man/
  -> ./usr/share/man/man5/
  -> ./usr/share/man/man5/fluxbox-style.5.gz
  -> ./usr/share/man/man5/fluxbox-menu.5.gz
  -> ./usr/share/man/man5/fluxbox-apps.5.gz
  -> ./usr/share/man/man5/fluxbox-keys.5.gz
  -> ./usr/share/man/man1/
  -> ./usr/share/man/man1/startfluxbox.1.gz
...
dependency The dependency package provides an interface to parse and compute dependencies. This package is a bit odd in that, well, there's no other library that does this. The issue is that there are actually two different parsers that compute our Dependency lines, one in Perl (as part of dpkg-dev) and another in C (in dpkg). To date, this has resulted in me filing three different bugs. I also found a broken package in the archive, which actually resulted in another bug being (totally accidentally) already fixed. I hope to continue to run the archive through my parser in hopes of finding more bugs! This package is a bit complex, but it basically just returns what amounts to be an AST for our Dependency lines. I'm positive there are bugs, so file them!
func main()  
    dep, err := dependency.Parse("foo   bar, baz, foobar [amd64]   bazfoo [!sparc], fnord:armhf [gnu-linux-sparc]")
    if err != nil  
        panic(err)
     
    anySparc, err := dependency.ParseArch("sparc")
    if err != nil  
        panic(err)
     
    for _, possi := range dep.GetPossibilities(*anySparc)  
        fmt.Printf("%s (%s)\n", possi.Name, possi.Arch)
     
 
Gives the output:
foo (<nil>)
baz (<nil>)
fnord (armhf)
version Right off the bat, I'd like to thank Michael Stapelberg for letting me graft this out of dcs and into the go-debian package. This was nearly entirely his work (with a one or two line function I added later), and was amazingly helpful to have. Thank you! This module implements Debian version comparisons and parsing, allowing for sorting in lists, checking to see if it's native or not, and letting the programmer to implement smart(er!) logic based on upstream (or Debian) version numbers. This module is extremely easy to use and very straightforward, and not worth writing an example for. Final thoughts This is more of a "Yeah, OK, this has been useful enough to me at this point that I'm going to support this" rather than a "It's stable!" or even "It's alive!" post. Hopefully folks can report bugs and help iterate on this module until we have some really clean building blocks to build solid higher level systems on top of. Being able to have multiple libraries interoperate by relying on go-debian will be a massive ease. I'm in need of more documentation, and to finalize some parts of the older sub package APIs, but I'm hoping to be at a "1.0" real soon now.

11 June 2016

Paul Tagliamonte: It's all relative

As nearly anyone who's worked with me will attest to, I've long since touted nedbat's talk Pragmatic Unicode, or, How do I stop the pain? as one of the most foundational talks, and required watching for all programmers. The reason is because netbat hits on something bigger - something more fundamental than how to handle Unicode -- it's how to handle data which is relative. For those who want the TL;DR, the argument is as follows: Facts of Life:
  1. Computers work with Bytes. Bytes go in, Bytes go out.
  2. The world needs more than 256 symbols.
  3. You need both Bytes and Unicode
  4. You cannot infer the encoding of bytes.
  5. Declared encodings can be Wrong
Now, to fix it, the following protips:
  1. Unicode sandwich
  2. Know what you have
  3. TEST
Relative Data I've started to think more about why we do the things we do when we write code, and one thing that continues to be a source of morbid schadenfreude is watching code break by failing to handle Unicode right. It's hard! However, watching what breaks lets you gain a bit of insight into how the author thinks, and what assumptions they make. When you send someone Unicode, there are a lot of assumptions that have to be made. Your computer has to trust what you (yes, you!) entered into your web browser, your web browser has to pass that on over the network (most of the time without encoding information), to a server which reads that bytestream, and makes a wild guess at what it should be. That server might save it to a database, and interpolate it into an HTML template in a different encoding (called Mojibake), resulting in a bad time for everyone involved. Everything's awful, and the fact our computers can continue to display text to us is a goddamn miracle. Never forget that. When it comes down to it, when I see a byte sitting on a page, I don't know (and can't know!) if it's Windows-1252, UTF-8, Latin-1, or EBCDIC. What's a poem to me is terminal garbage to you. Over the years, hacks have evolved. We have magic numbers, and plain ole' hacks to just guess based on the content. Of course, like all good computer programs, this has lead to its fair share of hilarious bugs, and there's nothing stopping files from (validly!) being multiple things at the same time. Like many things, it's all in the eye of the beholder. Timezones Just like Unicode, this is a word that can put your friendly neighborhood programmer into a series of profanity laden tirades. Go find one in the wild, and ask them about what they think about timezone handling bugs they've seen. I'll wait. Go ahead. Rants are funny things. They're fun to watch. Hilarious to give. Sometimes just getting it all out can help. They can tell you a lot about the true nature of problems. It's funny to consider the isomorphic nature of Unicode rants and Timezone rants. I don't think this is an accident. U n i c o d e timezone Sandwich Ned's Unicode Sandwich applies -- As early as we can, in the lowest level we can (reading from the database, filesystem, wherever!), all datetimes must be timezone qualified with their correct timezone. Always. If you mean UTC, say it's in UTC. Treat any unqualified datetimes as "bytes". They're not to be trusted. Never, never, never trust 'em. Don't process any datetimes until you're sure they're in the right timezone. This lets the delicious inside of your datetime sandwich handle timezones with grace, and finally, as late as you can, turn it back into bytes (if at all!). Treat locations as tzdb entries, and qualify datetime objects into their absolute timezone (EST, EDT, PST, PDT) It's not until you want to show the datetime to the user again should you consider how to re-encode your datetime to bytes. You should think about what flavor of bytes, what encoding -- what timezone -- should I be encoding into? TEST Just like Unicode, testing that your code works with datetimes is important. Every time I think about how to go about doing this, I think about that one time that mjg59 couldn't book a flight starting Tuesday from AKL, landing in HNL on Monday night, because United couldn't book the last leg to SFO. Do you ever assume dates only go forward as time goes on? Remember timezones. Construct test data, make sure someone in New Zealand's +13:45 can correctly talk with their friends in Baker Island's -12:00, and that the events sort right. Just because it's Noon on New Years Eve in England doesn't mean it's not 1 AM the next year in New Zealand. Places a few miles apart may go on Daylight savings different days. Indian Standard Time is not even aligned on the hour to GMT (+05:30)! Test early, and test often. Memorize a few timezones, and challenge your assumptions when writing code that has to do with time. Don't use wall clocks to mean monotonic time. Remember there's a whole world out there, and we only deal with part of it. It's also worth remembering, as Andrew Pendleton pointed out to me, that it's possible that a datetime isn't even unique for a place, since you can never know if 2016-11-06 01:00:00 in America/New_York (in the tzdb) is the first one, or second one. Storing EST or EDT along with your datetime may help, though! Pitfalls Improper handling of timezones can lead to some interesting things, and failing to be explicit (or at least, very rigid) in what you expect will lead to an unholy class of bugs we've all come to hate. At best, you have confused users doing math, at worst, someone misses a critical event, or our security code fails. I recently found what I regard to be a pretty bad bug in apt (which David has prepared a fix for and is pending upload, yay! Thank you!), which boiled down to documentation and code expecting datetimes in a timezone, but accepting any timezone, and silently treating it as UTC. The solution is to hard-fail, which is an interesting choice to me (as a vocal fan of timezone aware code), but at the least it won't fail by misunderstanding what the server is trying to communicate, and I do understand and empathize with the situation the apt maintainers are in. Final Thoughts Overall, my main point is although most modern developers know how to deal with Unicode pain, I think there is a more general lesson to learn -- namely, you should always know what data you have, and always remember what it is. Understand assumptions as early as you can, and always store them with the data.

31 May 2016

Paul Tagliamonte: Iron Blogger DC

Back in 2014, Mako ran a Boston Iron Blogger chapter, where you had to blog once a week, or you owed $5 into the pot. A while later, I ran it (along with Molly and Johns), and things were great. When I moved to DC, I had already talked with Tom Lee and Eric Mill about running a DC Iron Blogger chapter, but it hasn t happened in the year and a half I ve been in DC. This week, I make good on that, with a fantastic group set up at dc.iron-blogger.com; with more to come (I m sure!). Looking forward to many parties and though provoking blog posts in my future. I m also quite pleased I ll be resuming my blogging. Hi, again, planet Debian!

07 June 2015

Thomas Goirand

There s a lot of things I d like to blog about. The last version of OpenStack, the OpenStack Liberty design summit, Kilo in the official jessie-backports repositories, etc. Maybe the most interesting part of this blog post is the last bit at the end, about a major change in the packaging workflow for OpenStack in Debian. Please read on OpenStack release names reminder
Just a reminder to make it easier for the average Debian reader who may know Debian well, but not OpenStack. OpenStack 2014.1, is Icehouse, and is the version in Jessie. 2014.2 is Juno and was released right before the freeze of Jessie. 2015.1.0 is what has been released just right after jessie, on the 30th of April. Liberty, which probably will be called 12 (as this will be the 12th release of OpenStack), and not 2015.2 (this has been discussed in Vancouver), will be released in about 5 months form now. The last summit, in Vancouver, BC, Canada, was the Liberty summit, as the OpenStack conventions are always named after the next release (since we are discussing what we will be doing during the next development cycle). OpenStack 2015.1.0, aka Kilo, release in Debian
5 days after the release of Jessie, OpenStack 2015.1.0, aka Kilo, was released. Since I couldn t upload to unstable during the freeze, I was holding a lot of packages, and when I did upload them, there was about 20 packages of mine in the FTP master s NEW queue. Though, since the DSA want to use OpenStack for the Debian infrastructure, the 20 packages were fast track into Sid, thanks to the work of Paultag (thanks man!). OpenStack Kilo in the official Jessie backports
Previously, I was only uploading OpenStack packages to Debian unstable, and maintaining a non-official Debian repositories for backports to Debian stable. However, for multiple reasons, this wasn t satisfying. Then, after packages migrated to Stretch, I started to upload to Debian backports. And right before the summit, almost everything went in. Only python-pysaml2 was missing (as I discovered too late that version 2.0.0 breaks Keystone which needs version 2.4.0). In fact, the last bits of the Kilo release reached jessie-backports in the middle of the OpenStack Liberty summit. Removal of the Debian install-guide from the official site
As there was not enough efforts working on the documentation, unfortunately, the link to the Debian install-guide has been removed from docs.openstack.org. IMO, this is mostly due to a bad communication between myself and the doc team, and also because one person who promised to work on the Debian side of the install-guide failed to warn everyone that he finally couldn t (as his managers assigned him to something else). I hope this will soon be reverted. During the Vancouver summit, I had the opportunity to discuss with the doc team about re-inclusion of the Debian install-guide. Unfortunately, as they are moving away from the XML source format to a more standard RST-based system, the current documentation is frozen, so it seems more realistic to hold on until all of the install-guide is switched to RST. OpenStack Debian image listed on apps.openstack.org
There s a new area on the openstack.org where images and apps for OpenStack are listed. Under the glance image tab, you will see that both the Jessie and the weekly testing image are listed. There s also a nice, easily identifiable Debian logo to link to these images. Also, as there are trademark problems with the Ubuntu images which makes them harder to redistribute, the Murano project (which is shipping a system to automatically install apps that to installed within a few clicks on an OpenStack cloud) decided to switch to Debian for their base image. Debian listed in the OpenStack market place
On the openstack.org site, there s a section called Marketplace. In there, vendors supporting OpenStack are listed. To get there, a vendor needs to 1/ have a defined set of OpenStack project supported by the distribution (Debian already has a way more than the required set), 2/ sign some kind of agreement with the OpenStack foundation, and 3/ pay some sponsoring money. During the summit, I discussed this with Jonathan Bryce, from the OpenStack foundation, and he agreed that Debian would not have to pay for this (since we aren t a big company with big money). I have put Jonathan and Neil (our Debian Project Leader) in touch so that signing the document may happen, though since we were all busy with the summit, I do not expect Jonathan to send the documents right away. Hopefully, this will be fixed before the end of this month of May 2015. Debian (and Ubuntu) packages collaboratively maintained upstream
Since about forever (forever is 5 years in the OpenStack world ), I pushed for more collaboration on OpenStack packaging between Debian package maintainers and Canonical. However, for some reasons which I do not wish to expand on in this blog post, it has been socially hard to do so. Also, Canonical always used BZR, which wasn t to the tastes of everyone. But during the Liberty summit, some very good things happened. First of all, Launchpad is now able to support Git (it s been a few weeks it does in fact). Even though it will take a bit of time before the Canonical server team switches to it, we can consider that this problem is already out of the way. Then it looks like Canonical are now more open than before for collaboration with Debian on the OpenStack packaging. Note that we actually did some work together already, but now we both would like a full alignment of *all* of our packages. I have discussed this with James Page, who is the head of Canonical s server team. We will first start to do so on the dependencies: this includes all of the python-*client libraries, but also all of python-oslo.* (the Oslo libs are use by all of the projects and are kind of unifying the project), plus all the third party dependencies the project relies on. James already pushed new versions of some Oslo libraries to Experimental (in order to not overwrite Kilo), which are adding transition packages needed for Ubuntu. We wont need those in Debian, but we want to welcome them to keep the same source packages. We will then later try to merge the core projects if we can. Unfortunately, since the packaging of the core projects (ie: Nova, Neutron, Cinder, Glance, etc.) was forked, merging probably will be a bit painful. We will have to make some decisions on how this happen. I am however confident that it will be done during the Liberty release cycle. Move of the packaging to upstream Gerrit
A few weeks after the summit, I wrote a proposal to upstream OpenStack dev list, with as subject: Adding packaging as an OpenStack project . What it means is that I have proposed to have Debian/Ubuntu packaging to happen in upstream infrastructure, using Gerrit, and building packages using upstream cloud. We will add all the tests we can, like building with unit tests, lintian, piuparts, adequate, but also maybe a full installation of the packages with functional tests. My proposal is here: http://lists.openstack.org/pipermail/openstack-dev/2015-May/064848.html As everything, this translates into a Gerrit review process: https://review.openstack.org/#/c/185187/ As you can read in the above thread, Fedora/RDO people, which have used a Gerrit work-flow for a long time already, also would like to join. But it looks like we ll be doing 2 teams: one for RPMs and one for debs. The proposal is currently under review by the OpenStack technical committee, which will accept (or not) if the packaging project can be fully considered as an OpenStack project. I expect a final answer next Tuesday. Note that if they deny, we can still use the stackforge namespace instead, their decision is just about the TC blessing the project as being OpenStack or not. What s very nice about this, is that not only we will have a better collaboration between Debian & Ubuntu, better automated testing and Q/A, this also opens contributions to potentially anyone. Especially, we welcome operation people, those who are doing actual big deployments. Sure, it was possible before, but I often had the feedback that many were scared to break anything when trying to contribute. Thanks to the CI/CD form upstream infra, and the Gerrit peer review process, it wont be a problem anymore. So we do expect operation people to contribute more. I will also push more upstream packaging within Mirantis, so that MOS (Mirantis OpenStack) aligns fully with Debian & Ubuntu as well. Another good thing, is that it will be easier for the puppet team to support Debian (they historically were more Ubuntu oriented), and it s going to be super easy for them to request for packaging fixes. I hope we will be able to work hand-to-hand with them, adding puppet deployment checks in the packaging repo, and packaged deployments within the puppet Gerrit review process.

01 June 2015

Paul Tagliamonte: Soylent Sherry Negroni

paultagskitchen:
DELICIOUS COCKTAIL PHOTOIngredients
  • 1 tsp soylent
  • 1 tsp simple syrup
  • 1 oz Palo Cortado sherry
  • oz Rosso Vermouth
  • oz Campari
Assembly Combine Soylent and Simple Syrup. Create what I m going to start to call Soylent Syrup . Enjoy that one, folks. Add ice to a rocks glass, pour Soylent Syrup over ice. Add Sherry, Vermouth and Campari. Stir. Garnish with an orange twist. Big thanks to Matthew Garrett for sparking this one.

08 May 2015

Daniel Kahn Gillmor: Cheers to audacity!

When paultag recently announced a project to try to move debian infrastructure to python3, my first thought was how large that undertaking would likely be. It seems like a classic engineering task, full of work and nit-picky details to get right, useful/necessary in the long-term, painful in the short-term, and if you manage to pull it off successfully, the best you can usually hope for is that no one will notice that it was done at all. I always find that kind of task a little off-putting and difficult to tackle, but I was happy to see someone driving the project, since it does need to get done. Debian is potentially also in a position to help the upstream python community, because we have a pretty good view of what things are being used, at least within our own ecosystem. I'm happy to say that i also missed one of the other great benefits of paultag's audacious proposal, which is how it has engaged people who already knew about debian but who aren't yet involved. Evidence of this engagement is already visible on the py3porters-devel mailing list. But if that wasn't enough, I ran into a friend recently who told me, "Hey, I found a way to contribute to debian finally!" and pointed me to the py3-porters project. People want to contribute to the project, and are looking for ways in. So cheers to the people who propose audacious projects and make them inviting to everyone, newcomers included. And cheers to the people who step up to potentially daunting work, stake out a task, roll up their sleeves, and pitch in. Even if the py3porters project doesn't move all of debian's python infrastructure to pyt3 as fast as paultag wants it to, i think it's already a win for the project as a whole. I am looking forward to seeing what comes out of it (and it's reminding me i need to port some of my own python work, too!) The next time you stumble over something big that needs doing in debian, even something that might seem impossible, please make it inviting, and dive in. The rest of the project will grow and improve from the attempt.Tags: py3-porters

02 April 2015

Paul Tagliamonte: Oatmeal Raisin Cookies

Oatmeal Raisin Cookies: paultagskitchen:
DELICIOUS COOKIE PHOTOIngredients
  • cups soylent
  • 1 cups rolled oats
  • cup sugar (white & dark brown)
  • cup flour
  • cup raisins
  • tsp baking soda & powder
  • tsp salt
  • 1 stick butter (roomtemp - NOT melted. Don t even try that. Stop. You. I see you.)
  • 1 egg
  • 1 tsp vanilla
Assembly Combine butter,

29 March 2015

Zlatan Todori : Its all about fun

The percentage that women in Debian occupy as DDs is ~2%. Yes, just ~2% ladies that are DDs! So that means ~98% of DDs are gentelmen. some picture with rage meme I know there are more of ladies in Debian, so I firstly urge you, for love of Debian, to apply if you are contributing to this project, love its community and want to see Debian taking over the universe (okay, it seems that we conquered outer space so we need a help on Earth). So why is the number this low? Well maybe it's too precious to us currently inside that we want to prevent it being spoiled from outside. Also there seems to be not that much of younger DDs. Why is that important - well, young people like to do it and not to think about it. Many time they just break it, but many time they also do a breakthrough. Why is difference important and why should we embrace it? It's very important because it breaks a monopoly on view and behavior. It brings views not just from a larger number of people, but also from people from different backgrounds, and in constructive conversation it can put even more pluses on current workflow or it can counter it with good arguments. In a project of its size and worldwide geolocation of its developers, this is true for Debian more then any other projects I know. We need more women so we can balance our inner workings and have a better understanding of humanity and how is it moving, what and why does it need and where is it steering. That way we can produce a community which will improve quality of OS that we produce - because of sheer number of different people working on the same thing bringing to it its own personal touch. So, ladies and youth all over the world, unite and join in Debian because without diversity Debian can't grow beyond its current size. Also, no, Debian is not about code only, it needs painters, musicians, people that want to talk about Debian, people that share love and happiness, people that want to build better communities, UI/UX designers, makers, people who know how to repair a bike, athletes, homebrew beer producers, lawyers (just while world gets rid of laws, then we don't need you), actors, writters... Why, well because world and communities are made up from all that diversity and that's what makes it a better and not a monotone place. But I just use Debian. Well, do you feel love towards Debian and its work? Would you like to feel more as integral part of community? If the answer is big fat YES, then you should be a DD too. Every person that feels it's part of Debians philosophy about freedom and behaving in good manner should join Debian. Every person that feels touched and enhanced by Debian's work should become part of community and share its experience how Debian touched their soul, impacted their life. If you love Debian, you should be free to contribute to it in whatever manner and you should be free to express your love towards it. If you think lintian is sexy, or shebang is a good friends of yours, or you enjoy talking to MadameZou about Debian and zombies (yeah, we do have all kinds of here), or you like Krita, or you hate the look of default XFCE theme, or you can prove that you a more crazy developer then paultag - just hop into community and try to integrate in it. You will meet great folks, have a lot of conversation about wine and cheese, play some dangerous card games and even learn about things like bokononism (yeah I am looking at you dkg!). Now for the current Debian community - what the hell is packaging and non-packaging Debian Developer? Are one better then others? Do others stink? They don't know to hug? WHAT? Yes I know that inexperienced person shouldn't have a permission to access Debian packaging infrastructure, but I have the feeling that even that person knows that. Every person should have a place in Debian and acknowledge other fields. So yes, software developers need access to Debian packaging infrastructure, painters don't. I think we can agree on this. So lets abolish the stupid term and remove the difference in our community. Lets embrace the difference, because if someone writes a good poem about Debian heroism I could like it more then flashplugin-nonfree! Yep, I made that comparison on purpose so you can give a thought about it. Debian has excellent community regarding operating system that it's producing. And it's not going away, not at least anytime soon. But it will not go forward if we don't give additional push as human beings, as people who care about their fellow Debianites. And we do care, I know that, we just need to push it more public. We don't hide bugs, we for sure shouldn't hide features. It will probably bring bad seeds too, but we have mechanisms and will to counter that. If we, on average 10 bad seeds, get some crazy good hacker or crazy lovely positive person like this lady, we will be on right path. Debian is a better place, it should lead in effort to bring more people into FLOSS world and it should allow people to bring more of diversity into Debian. draw a picture where it says next year 3 dpl candidates should be only women and at least one of them not involved in packaging

Next.