This is another post about EVM/IMA which has it s main purpose providing useful web search results for problems. However if reading it on a planet feed inspires someone to play with EVM/IMA then that s good too, it s interesting technology. When using EVM/IMA in the Linux kernel if dmesg has errors like op=appraise_data cause=missing-HMAC the missing-HMAC means that the error code in the kernel source is INTEGRITY_NOLABEL which has a comment No security.evm xattr . You can check for the xattr on a file with the following command (this example has the security.evm xattr):

# getfattr -d -m - /etc/fstab 
getfattr: Removing leading '/' from absolute path names
# file: etc/fstab
security.evm=0sAwICqGOsfwCAvgE9y9OP74QxJ/I+3eOSF2n2dM51St98z/7LYHFd9rfGTvssvhTSYL9G8cTdRAH8ozggJu7VCzggW1REoTjnLcPeuMJsrMbW3DwVrB6ldDmJzyenLMjnIHmRDDeK309aRbLVn2ueJZ07aMDcSr+sxhOOAQ/GIW4SW8L1AKpKn4g=
security.ima=0sAT+Eivfxl+7FYI+Hr9K4sE6IieZ+
security.selinux="system_u:object_r:etc_t:s0"

If dmesg has errors like op=appraise_data cause=invalid-HMAC the invalid-HMAC means that the error code in the kernel source is INTEGRITY_FAIL which has a comment Invalid HMAC/signature . These errors are from the evm_verifyxattr() function in Linux kernel 5.11.14. The error evm: HMAC key is not set means that the evm key is not initialised, this means the key needs to be loaded into the kernel and EVM is initialised by the command echo 1 > /sys/kernel/security/evm (or possibly some equivalent from a utility like evmctl). When the key is loaded the kernel gives the message evm: key initialized and after that /sys/kernel/security/evm is read-only. If there is something wrong with the key the kernel gives the message evm: key initialization failed , it seems that the way to determine if your key is good is to try writing 1 to /sys/kernel/security/evm and see what happens. After that the command cat /sys/kernel/security/evm should return 3 . The Gentoo wiki has good documentation on how to create and load the keys which has to be done before initialising EVM [1]. I ll write more about that in another post.

[1] https://wiki.gentoo.org/wiki/Extended_Verification_Module

Because posting private keys on the Internet is a bad idea, some people like to redact their private keys, so that it looks kinda-sorta like a private key, but it isn t actually giving away anything secret. Unfortunately, due to the way that private keys are represented, it is easy to redact a key in such a way that it doesn t actually redact anything at all. RSA private keys are particularly bad at this, but the problem can (potentially) apply to other keys as well. I ll show you a bit of Inside Baseball with key formats, and then demonstrate the practical implications. Finally, we ll go through a practical worked example from an actual not-really-redacted key I recently stumbled across in my travels.

The Private Lives of Private Keys Here is what a typical private key looks like, when you come across it:

Obviously, there s some hidden meaning in there computers don t encrypt things by shouting BEGIN RSA PRIVATE KEY! , after all. What is between the BEGIN/END lines above is, in fact, a base64-encoded DER format ASN.1 structure representing a PKCS#1 private key. In simple terms, it s a list of numbers very important numbers. The list of numbers is, in order:

A version number (0);
The public modulus , commonly referred to as n ;
The public exponent , or e (which is almost always 65,537, for various unimportant reasons);
The private exponent , or d ;
The two private primes , or p and q ;
Two exponents, which are known as dmp1 and dmq1 ; and
A coefficient, known as iqmp .

Why Is This a Problem? The thing is, only three of those numbers are actually required in a private key. The rest, whilst useful to allow the RSA encryption and decryption to be more efficient, aren t necessary. The three absolutely required values are e, p, and q. Of the other numbers, most of them are at least about the same size as each of p and q. So of the total data in an RSA key, less than a quarter of the data is required. Let me show you with the above toy key, by breaking it down piece by piece¹:

MGI DER for this is a sequence
CAQ version (0)
CxjdTmecltJEz2PLMpS4BX n
AgMBAA e
ECEDKtuwD17gpagnASq1zQTY d
ECCQDVTYVsjjF7IQ p
IJANUYZsIjRsR3 q
AgkAkahDUXL0RS dmp1
ECCB78r2SnsJC9 dmq1
AghaOK3FsKoELg== iqmp

Remember that in order to reconstruct all of these values, all I need are e, p, and q and e is pretty much always 65,537. So I could redact almost all of this key, and still give all the important, private bits of this key. Let me show you:

-----BEGIN RSA PRIVATE KEY-----
..............................................................EC
CQDVTYVsjjF7IQIJANUYZsIjRsR3....................................
........
-----END RSA PRIVATE KEY-----

Now, I doubt that anyone is going to redact a key precisely like this but then again, this isn t a typical RSA key. They usually look a lot more like this:

People typically redact keys by deleting whole lines, and usually replacing them with [...] and the like. But only about 345 of those 1588 characters (excluding the header and footer) are required to construct the entire key. You can redact about 4/5ths of that giant blob of stuff, and your private parts (or at least, those of your key) are still left uncomfortably exposed.

But Wait! There s More! Remember how I said that everything in the key other than `e`, `p`, and `q` could be derived from those three numbers? Let s talk about one of those numbers: `n`. This is known as the public modulus (because, along with `e`, it is also present in the public key). It is very easy to calculate: `n = p * q`. It is also very early in the key (the second number, in fact). Since `n = p * q`, it follows that `q = n / p`. Thus, as long as the key is intact up to `p`, you can derive `q` by simple division.

Real World Redaction At this point, I d like to introduce an acquaintance of mine: Mr. Johan Finn. He is the proud owner of the GitHub repo johanfinn/scripts. For a while, his repo contained a script that contained a poorly-redacted private key. He since deleted it, by making a new commit, but of course because git never really deletes anything, it s still available. Of course, Mr. Finn may delete the repo, or force-push a new history without that commit, so here is the redacted private key, with a bit of the surrounding shell script, for our illustrative pleasure:

#Add private key to .ssh folder
cd /home/johan/.ssh/
echo  "-----BEGIN RSA PRIVATE KEY-----
MMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMM
NNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNN
KKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKK
 
MIIJKgIBAAKCAgEAxEVih1JGb8gu/Fm4AZh+ZwJw/pjzzliWrg4mICFt1g7SmIE2
TCQMKABdwd11wOFKCPc/UzRH/fHuQcvWrpbOSdqev/zKff9iedKw/YygkMeIRaXB
fYELqvUAOJ8PPfDm70st9GJRhjGgo5+L3cJB2gfgeiDNHzaFvapRSU0oMGQX+kI9
ezsjDAn+0Pp+r3h/u1QpLSH4moRFGF4omNydI+3iTGB98/EzuNhRBHRNq4oBV5SG
Pq/A1bem2ninnoEaQ+OPESxYzDz3Jy9jV0W/6LvtJ844m+XX69H5fqq5dy55z6DW
sGKn78ULPVZPsYH5Y7C+CM6GAn4nYCpau0t52sqsY5epXdeYx4Dc+Wm0CjXrUDEe
Egl4loPKDxJkQqQ/MQiz6Le/UK9vEmnWn1TRXK3ekzNV4NgDfJANBQobOpwt8WVB
rbsC0ON7n680RQnl7PltK9P1AQW5vHsahkoixk/BhcwhkrkZGyDIl9g8Q/Euyoq3
eivKPLz7/rhDE7C1BzFy7v8AjC3w7i9QeHcWOZFAXo5hiDasIAkljDOsdfD4tP5/
wSO6E6pjL3kJ+RH2FCHd7ciQb+IcuXbku64ln8gab4p8jLa/mcMI+V3eWYnZ82Yu
axsa85hAe4wb60cp/rCJo7ihhDTTvGooqtTisOv2nSvCYpcW9qbL6cGjAXECAwEA
AQKCAgEAjz6wnWDP5Y9ts2FrqUZ5ooamnzpUXlpLhrbu3m5ncl4ZF5LfH+QDN0Kl
KvONmHsUhJynC/vROybSJBU4Fu4bms1DJY3C39h/L7g00qhLG7901pgWMpn3QQtU
4P49qpBii20MGhuTsmQQALtV4kB/vTgYfinoawpo67cdYmk8lqzGzzB/HKxZdNTq
s+zOfxRr7PWMo9LyVRuKLjGyYXZJ/coFaobWBi8Y96Rw5NZZRYQQXLIalC/Dhndm
AHckpstEtx2i8f6yxEUOgPvV/gD7Akn92RpqOGW0g/kYpXjGqZQy9PVHGy61sInY
HSkcOspIkJiS6WyJY9JcvJPM6ns4b84GE9qoUlWVF3RWJk1dqYCw5hz4U8LFyxsF
R6WhYiImvjxBLpab55rSqbGkzjI2z+ucDZyl1gqIv9U6qceVsgRyuqdfVN4deU22
LzO5IEDhnGdFqg9KQY7u8zm686Ejs64T1sh0y4GOmGsSg+P6nsqkdlXH8C+Cf03F
lqPFg8WQC7ojl/S8dPmkT5tcJh3BPwIWuvbtVjFOGQc8x0lb+NwK8h2Nsn6LNazS
0H90adh/IyYX4sBMokrpxAi+gMAWiyJHIHLeH2itNKtAQd3qQowbrWNswJSgJzsT
JuJ7uqRKAFkE6nCeAkuj/6KHHMPsfCAffVdyGaWqhoxmPOrnVgECggEBAOrCCwiC
XxwUgjOfOKx68siFJLfHf4vPo42LZOkAQq5aUmcWHbJVXmoxLYSczyAROopY0wd6
Dx8rqnpO7OtZsdJMeBSHbMVKoBZ77hiCQlrljcj12moFaEAButLCdZFsZW4zF/sx
kWIAaPH9vc4MvHHyvyNoB3yQRdevu57X7xGf9UxWuPil/jvdbt9toaraUT6rUBWU
GYPNKaLFsQzKsFWAzp5RGpASkhuiBJ0Qx3cfLyirjrKqTipe3o3gh/5RSHQ6VAhz
gdUG7WszNWk8FDCL6RTWzPOrbUyJo/wz1kblsL3vhV7ldEKFHeEjsDGroW2VUFlS
asAHNvM4/uYcOSECggEBANYH0427qZtLVuL97htXW9kCAT75xbMwgRskAH4nJDlZ
IggDErmzBhtrHgR+9X09iL47jr7dUcrVNPHzK/WXALFSKzXhkG/yAgmt3r14WgJ6
5y7010LlPFrzaNEyO/S4ISuBLt4cinjJsrFpoo0WI8jXeM5ddG6ncxdurKXMymY7
:::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::.::
:::::::::::::::::::::::::::.::::::::::::::::::::::::::::::::::::
LLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLlL
 
 
 
YYYYYYYYYYYYYYYYYYYYYyYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYY
gff0GJCOMZ65pMSy3A3cSAtjlKnb4fWzuHD5CFbusN4WhCT/tNxGNSpzvxd8GIDs
nY7exs9L230oCCpedVgcbayHCbkChEfoPzL1e1jXjgCwCTgt8GjeEFqc1gXNEaUn
O8AJ4VlR8fRszHm6yR0ZUBdY7UJddxQiYOzt0S1RLlECggEAbdcs4mZdqf3OjejJ
06oTPs9NRtAJVZlppSi7pmmAyaNpOuKWMoLPElDAQ3Q7VX26LlExLCZoPOVpdqDH
KbdmBEfTR4e11Pn9vYdu9/i6o10U4hpmf4TYKlqk10g1Sj21l8JATj/7Diey8scO
sAI1iftSg3aBSj8W7rxCxSezrENzuqw5D95a/he1cMUTB6XuravqZK5O4eR0vrxR
AvMzXk5OXrUEALUvt84u6m6XZZ0pq5XZxq74s8p/x1JvTwcpJ3jDKNEixlHfdHEZ
ZIu/xpcwD5gRfVGQamdcWvzGHZYLBFO1y5kAtL8kI9tW7WaouWVLmv99AyxdAaCB
Y5mBAQKCAQEAzU7AnorPzYndlOzkxRFtp6MGsvRBsvvqPLCyUFEXrHNV872O7tdO
GmsMZl+q+TJXw7O54FjJJvqSSS1sk68AGRirHop7VQce8U36BmI2ZX6j2SVAgIkI
9m3btCCt5rfiCatn2+Qg6HECmrCsHw6H0RbwaXS4RZUXD/k4X+sslBitOb7K+Y+N
Bacq6QxxjlIqQdKKPs4P2PNHEAey+kEJJGEQ7bTkNxCZ21kgi1Sc5L8U/IGy0BMC
PvJxssLdaWILyp3Ws8Q4RAoC5c0ZP0W2j+5NSbi3jsDFi0Y6/2GRdY1HAZX4twem
Q0NCedq1JNatP1gsb6bcnVHFDEGsj/35oQKCAQEAgmWMuSrojR/fjJzvke6Wvbox
FRnPk+6YRzuYhAP/YPxSRYyB5at++5Q1qr7QWn7NFozFIVFFT8CBU36ktWQ39MGm
cJ5SGyN9nAbbuWA6e+/u059R7QL+6f64xHRAGyLT3gOb1G0N6h7VqFT25q5Tq0rc
Lf/CvLKoudjv+sQ5GKBPT18+zxmwJ8YUWAsXUyrqoFWY/Tvo5yLxaC0W2gh3+Ppi
EDqe4RRJ3VKuKfZxHn5VLxgtBFN96Gy0+Htm5tiMKOZMYAkHiL+vrVZAX0hIEuRZ
JJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJ
MMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMM
-----END RSA PRIVATE KEY-----" >> id_rsa

Now, if you try to reconstruct this key by removing the obvious garbage lines (the ones that are all repeated characters, some of which aren t even valid base64 characters), it still isn t a key at least, openssl pkey doesn t want anything to do with it. The key is very much still in there, though, as we shall soon see. Using a gem I wrote and a quick bit of Ruby, we can extract a complete private key. The irb session looks something like this:

>> require "derparse"
>> b64 = <<EOF
MIIJKgIBAAKCAgEAxEVih1JGb8gu/Fm4AZh+ZwJw/pjzzliWrg4mICFt1g7SmIE2
TCQMKABdwd11wOFKCPc/UzRH/fHuQcvWrpbOSdqev/zKff9iedKw/YygkMeIRaXB
fYELqvUAOJ8PPfDm70st9GJRhjGgo5+L3cJB2gfgeiDNHzaFvapRSU0oMGQX+kI9
ezsjDAn+0Pp+r3h/u1QpLSH4moRFGF4omNydI+3iTGB98/EzuNhRBHRNq4oBV5SG
Pq/A1bem2ninnoEaQ+OPESxYzDz3Jy9jV0W/6LvtJ844m+XX69H5fqq5dy55z6DW
sGKn78ULPVZPsYH5Y7C+CM6GAn4nYCpau0t52sqsY5epXdeYx4Dc+Wm0CjXrUDEe
Egl4loPKDxJkQqQ/MQiz6Le/UK9vEmnWn1TRXK3ekzNV4NgDfJANBQobOpwt8WVB
rbsC0ON7n680RQnl7PltK9P1AQW5vHsahkoixk/BhcwhkrkZGyDIl9g8Q/Euyoq3
eivKPLz7/rhDE7C1BzFy7v8AjC3w7i9QeHcWOZFAXo5hiDasIAkljDOsdfD4tP5/
wSO6E6pjL3kJ+RH2FCHd7ciQb+IcuXbku64ln8gab4p8jLa/mcMI+V3eWYnZ82Yu
axsa85hAe4wb60cp/rCJo7ihhDTTvGooqtTisOv2nSvCYpcW9qbL6cGjAXECAwEA
AQKCAgEAjz6wnWDP5Y9ts2FrqUZ5ooamnzpUXlpLhrbu3m5ncl4ZF5LfH+QDN0Kl
KvONmHsUhJynC/vROybSJBU4Fu4bms1DJY3C39h/L7g00qhLG7901pgWMpn3QQtU
4P49qpBii20MGhuTsmQQALtV4kB/vTgYfinoawpo67cdYmk8lqzGzzB/HKxZdNTq
s+zOfxRr7PWMo9LyVRuKLjGyYXZJ/coFaobWBi8Y96Rw5NZZRYQQXLIalC/Dhndm
AHckpstEtx2i8f6yxEUOgPvV/gD7Akn92RpqOGW0g/kYpXjGqZQy9PVHGy61sInY
HSkcOspIkJiS6WyJY9JcvJPM6ns4b84GE9qoUlWVF3RWJk1dqYCw5hz4U8LFyxsF
R6WhYiImvjxBLpab55rSqbGkzjI2z+ucDZyl1gqIv9U6qceVsgRyuqdfVN4deU22
LzO5IEDhnGdFqg9KQY7u8zm686Ejs64T1sh0y4GOmGsSg+P6nsqkdlXH8C+Cf03F
lqPFg8WQC7ojl/S8dPmkT5tcJh3BPwIWuvbtVjFOGQc8x0lb+NwK8h2Nsn6LNazS
0H90adh/IyYX4sBMokrpxAi+gMAWiyJHIHLeH2itNKtAQd3qQowbrWNswJSgJzsT
JuJ7uqRKAFkE6nCeAkuj/6KHHMPsfCAffVdyGaWqhoxmPOrnVgECggEBAOrCCwiC
XxwUgjOfOKx68siFJLfHf4vPo42LZOkAQq5aUmcWHbJVXmoxLYSczyAROopY0wd6
Dx8rqnpO7OtZsdJMeBSHbMVKoBZ77hiCQlrljcj12moFaEAButLCdZFsZW4zF/sx
kWIAaPH9vc4MvHHyvyNoB3yQRdevu57X7xGf9UxWuPil/jvdbt9toaraUT6rUBWU
GYPNKaLFsQzKsFWAzp5RGpASkhuiBJ0Qx3cfLyirjrKqTipe3o3gh/5RSHQ6VAhz
gdUG7WszNWk8FDCL6RTWzPOrbUyJo/wz1kblsL3vhV7ldEKFHeEjsDGroW2VUFlS
asAHNvM4/uYcOSECggEBANYH0427qZtLVuL97htXW9kCAT75xbMwgRskAH4nJDlZ
IggDErmzBhtrHgR+9X09iL47jr7dUcrVNPHzK/WXALFSKzXhkG/yAgmt3r14WgJ6
5y7010LlPFrzaNEyO/S4ISuBLt4cinjJsrFpoo0WI8jXeM5ddG6ncxdurKXMymY7
EOF
>> b64 += <<EOF
gff0GJCOMZ65pMSy3A3cSAtjlKnb4fWzuHD5CFbusN4WhCT/tNxGNSpzvxd8GIDs
nY7exs9L230oCCpedVgcbayHCbkChEfoPzL1e1jXjgCwCTgt8GjeEFqc1gXNEaUn
O8AJ4VlR8fRszHm6yR0ZUBdY7UJddxQiYOzt0S1RLlECggEAbdcs4mZdqf3OjejJ
06oTPs9NRtAJVZlppSi7pmmAyaNpOuKWMoLPElDAQ3Q7VX26LlExLCZoPOVpdqDH
KbdmBEfTR4e11Pn9vYdu9/i6o10U4hpmf4TYKlqk10g1Sj21l8JATj/7Diey8scO
sAI1iftSg3aBSj8W7rxCxSezrENzuqw5D95a/he1cMUTB6XuravqZK5O4eR0vrxR
AvMzXk5OXrUEALUvt84u6m6XZZ0pq5XZxq74s8p/x1JvTwcpJ3jDKNEixlHfdHEZ
ZIu/xpcwD5gRfVGQamdcWvzGHZYLBFO1y5kAtL8kI9tW7WaouWVLmv99AyxdAaCB
Y5mBAQKCAQEAzU7AnorPzYndlOzkxRFtp6MGsvRBsvvqPLCyUFEXrHNV872O7tdO
GmsMZl+q+TJXw7O54FjJJvqSSS1sk68AGRirHop7VQce8U36BmI2ZX6j2SVAgIkI
9m3btCCt5rfiCatn2+Qg6HECmrCsHw6H0RbwaXS4RZUXD/k4X+sslBitOb7K+Y+N
Bacq6QxxjlIqQdKKPs4P2PNHEAey+kEJJGEQ7bTkNxCZ21kgi1Sc5L8U/IGy0BMC
PvJxssLdaWILyp3Ws8Q4RAoC5c0ZP0W2j+5NSbi3jsDFi0Y6/2GRdY1HAZX4twem
Q0NCedq1JNatP1gsb6bcnVHFDEGsj/35oQKCAQEAgmWMuSrojR/fjJzvke6Wvbox
FRnPk+6YRzuYhAP/YPxSRYyB5at++5Q1qr7QWn7NFozFIVFFT8CBU36ktWQ39MGm
cJ5SGyN9nAbbuWA6e+/u059R7QL+6f64xHRAGyLT3gOb1G0N6h7VqFT25q5Tq0rc
Lf/CvLKoudjv+sQ5GKBPT18+zxmwJ8YUWAsXUyrqoFWY/Tvo5yLxaC0W2gh3+Ppi
EDqe4RRJ3VKuKfZxHn5VLxgtBFN96Gy0+Htm5tiMKOZMYAkHiL+vrVZAX0hIEuRZ
EOF
>> der = b64.unpack("m").first
>> c = DerParse.new(der).first_node.first_child
>> version = c.value
=> 0
>> c = c.next_node
>> n = c.value
=> 80071596234464993385068908004931... # (etc)
>> c = c.next_node
>> e = c.value
=> 65537
>> c = c.next_node
>> d = c.value
=> 58438813486895877116761996105770... # (etc)
>> c = c.next_node
>> p = c.value
=> 29635449580247160226960937109864... # (etc)
>> c = c.next_node
>> q = c.value
=> 27018856595256414771163410576410... # (etc)

What I ve done, in case you don t speak Ruby, is take the two chunks of plausible-looking base64 data, chuck them together into a variable named b64, unbase64 it into a variable named der, pass that into a new DerParse instance, and then walk the DER value tree until I got all the values I need. Interestingly, the q value actually traverses the split in the two chunks, which means that there s always the possibility that there are lines missing from the key. However, since p and q are supposed to be prime, we can sanity check them to see if corruption is likely to have occurred:

>> require "openssl"
>> OpenSSL::BN.new(p).prime?
=> true
>> OpenSSL::BN.new(q).prime?
=> true

Excellent! The chances of a corrupted file producing valid-but-incorrect prime numbers isn t huge, so we can be fairly confident that we ve got the real p and q. Now, with the help of another one of my creations we can use e, p, and q to create a fully-operational battle key:

>> require "openssl/pkey/rsa"
>> k = OpenSSL::PKey::RSA.from_factors(p, q, e)
=> #<OpenSSL::PKey::RSA:0x0000559d5903cd38>
>> k.valid?
=> true
>> k.verify(OpenSSL::Digest::SHA256.new, k.sign(OpenSSL::Digest::SHA256.new, "bob"), "bob")
=> true

and there you have it. One fairly redacted-looking private key brought back to life by maths and far too much free time. Sorry Mr. Finn, I hope you re not still using that key on anything Internet-facing.

What About Other Key Types? EC keys are very different beasts, but they have much the same problems as RSA keys. A typical EC key contains both private and public data, and the public portion is twice the size so only about 1/3 of the data in the key is private material. It is quite plausible that you can redact an EC key and leave all the actually private bits exposed.

What Do We Do About It? In short: don t ever try and redact real private keys. For documentation purposes, just put KEY GOES HERE in the appropriate spot, or something like that. Store your secrets somewhere that isn t a public (or even private!) git repo. Generating a dummy private key and sticking it in there isn t a great idea, for different reasons: people have this odd habit of reusing demo keys in real life. There s no need to encourage that sort of thing.

Technically the pieces aren t 100% aligned with the underlying DER, because of how base64 works. I felt it was easier to understand if I stuck to chopping up the base64, rather than decoding into DER and then chopping up the DER.

What happened in the Reproducible Builds effort between Sunday August 28 and Saturday September 3 2016: Media coverage Antonio Terceiro blogged about testing build reprodubility with debrepro . GSoC and Outreachy updates The next round is being planned now: see their page with a timeline and participating organizations listing. Maybe you want to participate this time? Then please reach out to us as soon as possible! Packages reviewed and fixed, and bugs filed The following packages have addressed reproducibility issues in other packages:

ruby-ronn/0.7.3-5 by Antonio Terceiro, original patch by Chris Lamb.

The following updated packages have become reproducible in our current test setup after being fixed:

check-all-the-things/2016.09.03 by Paul Wise, original patch by Chris Lamb.
e2fsprogs/1.43.2-2 by Theodore Y. Ts'o.
gnupg1/1.4.21-1 by Daniel Kahn Gillmor, #834755 by Chris Lamb.
gnupg2/2.1.15-2 by Daniel Kahn Gillmor.
inform6-compiler/6.33-2 by Ben Finney.
libhat-trie/0.0~git25f9e946-2 by Sascha Steinbiss.
mrd6/0.9.6-13 by Thomas Preud'homme, original patch by Reiner Herrmann.
safecat/1.13-3 by Teemu Hukkanen, original patch by Reiner Herrmann.
shibboleth-sp2/2.6.0+dfsg1-1 by Ferenc W gner.
sweethome3d-furniture-editor/1.15-2 by Markus Koschany, original patch by Reiner Herrmann.
traceroute/1:2.1.0-2 by Laszlo Boszormenyi, original patch by Reiner Herrmann.
wise/2.4.1-19 by Sascha Steinbiss.
xmltooling/1.6.0-2 by Ferenc W gner.

The following updated packages appear to be reproducible now, for reasons we were not able to figure out yet. (Relevant changelogs did not mention reproducible builds.)

comitup/0.5-1 by David Steele.
dita-ot/1.5.3-2 by Mathieu Malaterre.
dput/0.10.2 by Ben Finney.
gnome-settings-daemon/3.21.90-2 by Michael Biebl.
libkf5kipi/4:16.08.0-1 by Pino Toscano.
libmpc/2:0.1~r475-2 by James Cowgill.
taurus/4.0.1+dfsg-1 by Picca Fr d ric-Emmanuel.
tcpwatch-httpproxy/1.3.1-2 by Toni Mueller.

The following 4 packages were not changed, but have become reproducible due to changes in their build-dependencies:

Some uploads have addressed some reproducibility issues, but not all of them:

amanda/1:3.3.9-1 by Jose M Calhariz.
dispcalgui/3.1.6.0-2 by Christian Marillat, original patch by Chris Lamb.
fastqtl/2.184+dfsg-4 by Dylan A ssi.
grass/7.0.5~rc1-1~exp1 by Bas Couwenberg.
kdevplatform/5.0-1 by Pino Toscano, original patch by Scarlett Clark.
leveldb/1.19-1 by Alessio Treglia.
libdevel-cover-perl/1.23-2 by gregor herrmann, original patch by Chris Lamb.
linux/4.7.2-1 by Ben Hutchings, #830268 by Reiner Herrmann.
lmms/1.1.3-4 by Javier Serrano.
mayavi2/4.4.3-2.2 by Anton Gladky.
opensaml2/2.6.0-2 by Ferenc W gner.
perl/5.22.2-4 by Dominic Hargreaves, original patch and original patch by Chris Lamb.
radare2/0.10.5+dfsg-1 by Sebastian Reichel, original patch by Chris Lamb.
splint/3.1.2.dfsg1-3 by Rapha l Hertzog, original patch by Chris Lamb.

Patches submitted that have not made their way to the archive yet:

#835673 filed against dacs by Chris Lamb.
#835805 filed against dh-python by Chris Lamb.
#835985 filed against nmh by Chris Lamb.
#836609 filed against nostalgy by Chris Lamb.
#835807 filed against pygtksourceview by Chris Lamb.
#836004 filed against python-gflags by Chris Lamb.
#835816 filed against rt-extension-customfieldsonupdate by Chris Lamb.
#835834 filed against ruby-compass by Chris Lamb.
#836605 filed against torque by Chris Lamb.
#833176 filed against trafficserver by Reiner Herrmann.

Reviews of unreproducible packages 706 package reviews have been added, 22 have been updated and 16 have been removed in this week, adding to our knowledge about identified issues. 5 issue types have been added:

1 issue type has been updated:

timestamp_in_enc_files_added_by_texlive_fontinst

Weekly QA work FTBFS bugs have been reported by:

Chris Lamb (8)
Lucas Nussbaum (3)

diffoscope development diffoscope development on the next version (60) continued in git, taking in contributions from:

Mattia Rizzolo:
- Better and more thorough testing
- Improvements to packaging
- Improvements to the ppu comparator

strip-nondeterminism development Mattia Rizzolo uploaded strip-nondeterminism 0.023-2~bpo8+1 to jessie-backports. A new version of strip-nondeterminism 0.024-1 was uploaded to unstable by Chris Lamb. It included contributions from:

Chris Lamb:
- Improve code quality of zip, jar, ar, png processors
AYANOKOUZI, Ryuunosuke:
- Preserve file attribute information of target file (#836075)

Holger added jobs on jenkins.debian.net to run testsuites on every commit. There is one job for the master branch and one for the other branches. disorderfs development Holger added jobs on jenkins.debian.net to run testsuites on every commit. There is one job for the master branch and one for the other branches. tests.reproducible-builds.org Debian: We now vary the GECOS records of the two build users. Thanks to Paul Wise for providing the patch. Misc. This week's edition was written by Ximin Luo, Holger Levsen & Chris Lamb and reviewed by a bunch of Reproducible Builds folks on IRC.

What happened about the reproducible builds effort this week: Media coverage Daniel Stender published an English translation of the article which originally appeared in Linux Magazin in Admin Magazine. Toolchain fixes Fixes landed in the Debian archive:

Lunar uploaded docbook-to-man/1:2.0.0-34 which removes a timestamp in generated manpages. Original patch by Chris Lamb.
Stefano Rivera uploaded dh-python/1.20150628-1 which now sorts namespace files. Original patch by Chris Lamb.
Christian Hofstaedtler uploaded ruby2.2/2.2.2-2 which now uses UTC for the dates in gemspec files. Original patch by Chris Lamb.

Lunar submitted to Debian the patch already sent upstream adding a --clamp-mtime option to tar. Patches have been submitted to add support for SOURCE_DATE_EPOCH to txt2man (Reiner Herrmann), epydoc (Reiner Herrmann), GCC (Dhole), and Doxygen (akira). Dhole uploaded a new experimental debhelper to the reproducible repository which exports SOURCE_DATE_EPOCH. As part of the experiment, the patch also sets TZ to UTC which should help with most timezone issues. It might still be problematic for some packages which would change their settings based on this. Mattia Rizzolo sent upstream a patch originally written by Lunar to make the generate-id() function be deterministic in libxslt. While that patch was quickly rejected by upstream, Andrew Ayer came up with a much better one which sadly could have some performance impact. Daniel Veillard replied with another patch that should be deterministic in most cases without needing extra data structures. It's impact is currently being investigated by retesting packages on reproducible.debian.net. akira added a new option to sbuild for configuring the path in which packages are built. This will be needed for the srebuild script. Niko Tyni asked Perl upstream about it using the __DATE__ and __TIME__ C processor macros. Packages fixed The following 143 packages became reproducible due to changes in their build dependencies: alot, argvalidate, astroquery, blender, bpython, brian, calibre, cfourcc, chaussette, checkbox-ng, cloc, configshell, daisy-player, dipy, dnsruby, dput-ng, dsc-statistics, eliom, emacspeak, freeipmi, geant321, gpick, grapefruit, heat-cfntools, imagetooth, jansson, jmapviewer, lava-tool, libhtml-lint-perl, libtime-y2038-perl, lift, lua-ldoc, luarocks, mailman-api, matroxset, maven-hpi-plugin, mknbi, mpi4py, mpmath, msnlib, munkres, musicbrainzngs, nova, pecomato, pgrouting, pngcheck, powerline, profitbricks-client, pyepr, pylibssh2, pylogsparser, pystemmer, pytest, python-amqp, python-apt, python-carrot, python-crypto, python-darts.lib.utils.lru, python-demgengeo, python-graph, python-mock, python-musicbrainz2, python-pathtools, python-pskc, python-psutil, python-pypump, python-repoze.sphinx.autointerface, python-repoze.tm2, python-repoze.what-plugins, python-repoze.what, python-repoze.who-plugins, python-xstatic-term.js, reclass, resource-agents, rgain, rttool, ruby-aggregate, ruby-archive-tar-minitar, ruby-bcat, ruby-blankslate, ruby-coffee-script, ruby-colored, ruby-dbd-mysql, ruby-dbd-odbc, ruby-dbd-pg, ruby-dbd-sqlite3, ruby-dbi, ruby-dirty-memoize, ruby-encryptor, ruby-erubis, ruby-fast-xs, ruby-fusefs, ruby-gd, ruby-git, ruby-globalhotkeys, ruby-god, ruby-hike, ruby-hmac, ruby-integration, ruby-ipaddress, ruby-jnunemaker-matchy, ruby-memoize, ruby-merb-core, ruby-merb-haml, ruby-merb-helpers, ruby-metaid, ruby-mina, ruby-net-irc, ruby-net-netrc, ruby-odbc, ruby-packet, ruby-parseconfig, ruby-platform, ruby-plist, ruby-popen4, ruby-rchardet, ruby-romkan, ruby-rubyforge, ruby-rubytorrent, ruby-samuel, ruby-shoulda-matchers, ruby-sourcify, ruby-test-spec, ruby-validatable, ruby-wirble, ruby-xml-simple, ruby-zoom, ryu, simplejson, spamassassin-heatu, speaklater, stompserver, syncevolution, syncmaildir, thin, ticgit, tox, transmissionrpc, vdr-plugin-xine, waitress, whereami, xlsx2csv, zathura. The following packages became reproducible after getting fixed:

ansible/1.9.2+dfsg-1 by Harlan Lieberman-Berg.
antpm/1.16-9 uploaded by Kristof Ralovich, original patch by Dhole.
ccseapps/2.5-6 by Alastair McKinstry.
clblas/2.4-2 uploaded by Ghislain Antony Vaillant, original patch by akira.
dictionaries-common/1.26.1 by Agustin Martin Domingo.
djvulibre/3.5.27.1-2 uploaded by Barak A. Pearlmutter, original patch by Dhole.
exim4/4.86~RC4-1 by Andreas Metzler.
geis/2.2.17-1 uploaded by Stephen M. Webb, original patch by akira.
guymager/0.7.4-2 uploaded by Michael Prokop, original patch by Reiner Herrman.
irrlicht/1.8.1+dfsg1-2 uploaded by Vincent Cheng, original patch by akira.
ismrmrd/1.2.3-2 by Ghislain Antony Vaillant.
ispell-et/1:20030606-24 by Agustin Martin Domingo.
jacktrip/1.1~repack-1 uploaded by IOhannes m zm lnig, original patch by akira.
ksshaskpass/4:5.3.1-1 by Maximiliano Curia, report by Daniel Kahn Gillmor.
libencode-jis2k-perl/0.03-1 by gregor herrmann.
libosmium/2.1.0-4 by Bas Couwenberg.
libzorpll/3.9.4.1-2 by SZALAY Attila.
linop/0.8.2-3 uploaded by Ghislain Antony Vaillant, original patch by Juan Picca.
lmdb/0.9.15-1 uploaded by Ond ej Sur , original patch by akira.
manuel/1.8.0-3 by Daniel Stender.
nss-wrapper/1.0.3-3 by Jakub Wilk.
orthanc/0.9.0+dfsg-1 prepared by Sebastien Jodogne, issues fixed in upstream version.
osmcoastline/2.0.1-2 by Bas Couwenberg.
osmium-tool/1.0.1-2 uploaded by Bas Couwenberg, original patch by Chris Lamb.
pelican/3.6.0-4 uploaded by Vincent Cheng, original patch by Chris Lamb.
py-lmdb/0.86-1 uploaded by Robert Edmonds, original patch by Chris Lamb.
python-pysqlite2/2.6.3-4 uploaded by Joel Rosdahl, original patch by Juan Picca.
stx-btree/0.9-2 uploaded by Yury Stankevich, original patch by akira.
wxwidgets3.0/3.0.2+dfsg-1 by Olly Betts.
y-u-no-validate/2013052401-4 by Jakub Wilk.
yudit/2.9.6-4 uploaded by Hideki Yamane, original patch by Chris Lamb.

Some uploads fixed some reproducibility issues but not all of them:

cdo/1.6.6+dfsg.1-1 uploaded by Alastair McKinstry.
dsdo/1.6.36-6 by Agustin Martin Domingo.
liboro-java/2.0.8a-11 by Emmanuel Bourg.
simgrid/3.11.1-10 uploaded by Martin Quinson, original patch by akira.

Patches submitted which have not made their way to the archive yet:

#790357 on clanlib by Chris Lamb: sort keys from Perl hash when generating documentation.
#790374 on oxygen-icons by Dhole: removes the timestamps from the the generated png icons and make symlink deduplication deterministic
#790380 on gearmand by Chris Lamb: don't package unreproducible example files
#790387 on kbtin by Chris Lamb: use UTC timezone when fixing manual date
#790389 on xtrkcad by Chris Lamb use time of latest debian/changelog entry as build date
#790410 on pyke by Chris Lamb: use C locale for time strings
#790503 on libnet-interface-perl by Chris Lamb: sort Perl hash keys when generating C code
#790696 on python-pygraphviz by Chris Lamb: set documentation date for Sphinx.
#790697 on python-repoze.who by Chris Lamb: set documentation date for Sphinx.
#790741 on libapache-poi-java by Chris Lamb: set build stamp to time of latest debian/changelog entry.
#790763 on openchange by Chris Lamb: use UTC when writing timestamp in manpage.
#790783 on jargon-text by Daniel Kahn Gillmor: set C.UTF-8 as locate to always output unicode when converting HTML to text.
#791423 on linuxtv-dvb-apps by Reiner Herrmann: set LC_ALL to C before sorting #defines..

reproducible.debian.net A new package set for the X Strike Force has been added. (h01ger) Bugs tagged with locale are now visible in the statistics. (h01ger) Some work has been done add tests for NetBSD. (h01ger) Many changes by Mattia Rizzolo have been merged on the whole infrastructure:

IRC notifications when known reproducible packages stops buildig successfully.
Packages marked ftbfs_due_to_obsolete_dependencies now appears as FTBFS on the package tracker.
When listing packages affected by an issue, packages without bugs are grouped together, making it easier to spot the ones who requires work.
Both build logs are now saved separately. A diff between the two files is available.
The text output of debbindiff is now available as well for easier search and reports.
Build logs and debbindiff output are now stored compressed with gzip.
The builder used for a given test is now recorded in the database.
The manual scheduler available from Alioth gained new options:
- -i/--issues: schedule all packages affected by the given issue.
- -r/--status: schedule all packages with the given status.
- -b/--before: schedule all packages built before the given date
- -t/--after: schedule all packages built after the given date.
- --noisy: notify the IRC channel also when the build starts, with a URL to watch it in real time.

debbindiff development Version 26 has been released on June 28th fixing the comparison of files of unknown format. (Lunar) A missing dependency identified in python-rpm affecting debbindiff installation without recommended packages was promptly fixed by Michal iha . Lunar also started a massive code rearchitecture to enhance code reuse and enable new features. Nothing visible yet, though. Documentation update josch and Mattia Rizzolo documented how to reschedule packages from Alioth. Package reviews 142 obsolete reviews have been removed, 344 added and 107 updated this week. Chris West (Faux) filled 13 new bugs for packages failing to build from sources. The following new issues have been added: snapshot_placeholder_replaced_with_timestamp_in_pom_properties, different_encoding, timestamps_in_documentation_generated_by_org_mode and timestamps_in_pdf_generated_by_matplotlib.

It is still 2012 in this part of the world and the world is still intact. Since nothing major happened, I thought of spending the new gifted time to add a long pending item to apt-offline. As shown in the screen shots, apt-offline's GUI now has support to detect and display the downloaded offline bug reports.

This is part of the just released, version 1.3.

Categories:

Keywords:

I bought an EEE 901. More on that later.

For the past few weeks I've been trying to work out which "mobile broadband" provider to go with for my daily commute. It's a train journey from the north coast of Northern Ireland to Belfast and goes through some sparsely populated areas, so I wasn't expecting to get anything that would provide 3G all the way. However I do want a provider that can do their best to keep a session up for the journey, handing over between 2G + 3G as necessary. Coverage maps/checkers aren't too bad if you just want to check a static point, but really even then it's better to actually try the provider in the location yourself. So, armed with a bluetooth GPS (Bl eNEXT BN-909GR), an Option GT 3G Quad PCMCIA card and my laptop I set about measuring signal strength along my train journey for each of the 5 providers available. I also made sure to actually try a session with them for at least one journey, because signal strength is not everything.

I wrote some Perl using Net::GPSD and Device::Gsm to query my current location, signal strength, 2G/3G status and network every 10 seconds. I did this for 3 train journeys (except for O2) and then also produced an "optimistic" file of all the data points combined for that provider.

A second piece of Perl took this data and drew the maps you see below. The graph on the right for each provider is all data points. Red means 2G, blue means 3G, green means "Limited Service" (ie it could see a signal, but not one it was allowed to connect to). White means that we had a GPS location and no signal at all. Click on the maps for bigger versions. You can see the route on OpenStreetMap - my graphs are a little squished horizontally, but the journey is from Castlerock (top left on the coast near Coleraine) to Belfast (bottom right).

At first glance 3 look to be the best bet. Lots of blue, a reasonably priced data plan. Except they can't hold up a session to save themselves. Not only does a 2G/3G handover almost always seem to result in a drop, but I had periods of failing to get a 2G session up at all and had some random drops while in 3G coverage as well. So strike them off.

O2 and T-Mobile don't manage any 3G outside Belfast (that's the bottom right corner btw). I found that a few periods of 3G along the journey helped a lot; it meant I could do a quick mail sync rather than the long drawn out affair it was on a 2G connection. They can both handle handover between 2G/3G and keep a session up for most of the journey, but the lack of speed counts them out.

Vodafone managed the worst reception, which is disappointing. I found them quite good when I lived in England (my datacard is from a Vodafone Pay as You Use data contract). I did find sessions being dropped during the no signal regions, so they just don't seem to be a good bet.

Which leaves Orange. I've had issues with Orange and data in the past (pre 2004 IIRC) but in testing they were able to keep a session up between Belfast and Castlerock, handling 2G/3G handover fine. Coverage was reasonably good and they manage 3G at several points along the journey (the blue points roughly correspond to Coleraine, Ballymena, Antrim and Belfast). I understand Orange have some EDGE capability, but my card doesn't do EDGE so I've no idea if that'll be an extra gain or not. I had a few issues with getting dud DNS servers but this happened with the other providers too so I was beginning to suspect the datacard of playing up. Manually setting known good servers in resolv.conf worked fine.

So, before I go and sign up to an 18 month Orange contract, what have I forgotten?

I updated ikiwiki, the software used to generate this blog, to the version in backports (2.50 from 1.50). I tried very hard to not spam any aggregators, as the new version uses a different link format. Hopefully I succeeded, apologies if I didn't.

I have just uploaded an important update to apt-p2p that is highly recommended for all users. At some point over the last 48 hours bittorrent nodes have started to infiltrate the apt-p2p DHT (you may have noticed many ValueError tracebacks in your log file). The (mainline) bittorrent DHT is very similar in protocol to apt-p2p, and so the nodes are able to partially communicate and pollute each others routing table. I didn't think this would ever happen, as there wouldn't seem to be a reason for them to ever come into contact, but somehow it did. I've made some minor changes that exploit the differences between the two protocols to exclude bittorrent nodes from the apt-p2p routing table, and to drop any requests from bittorrent nodes (which should prevent apt-p2p nodes from polluting the bittorrent DHT). However, it is very important that all users upgrade to this new version to prevent any further mixing of the DHTs.

This new version of DebTorrent finalizes the work I previously uploaded to experimental. This new version is the first in unstable to use unique piece numbers to create long-lived torrents for testing and unstable. It also comes with a new version of the apt-transport-debtorrent package, which can now tell APT not to cache any package files downloaded with DebTorrent, which should save some disk space for some people. The locations of the long-lived torrent files and piece files that DebTorrent uses has also changed, as they are now hosted on my merkel page and should be more up to date. However, I was not able to get the ssh trigger for the sync from ftp-master setup, nor am I able to access the projectb database through pygresql. If anyone knows how I can get those working, please email me. Here is the changelog for debtorrent:

Display the torrent identifier on the local status page rather than the info hash (Closes: #465339)
Add support for the new No Content message to apt-transport-debtorrent so that apt treats debtorrent sources as local
Make the pieces and unique piece number locations config options, and use the new ones at merkel.debian.org/~camrdale/.
Decompress the files needed to create torrents so they are created and started faster (this and the previous Closes: #463676)

And for apt-transport-debtorrent:

Add a config option to make debtorrent sources local (Closes: #477383)
Upgrade the recommends on debtorrent to 0.1.7.
Add a config file for APT to load, and a manpage describing the options.

I have updated apt-p2p to version 0.1.3, which is available now in unstable and my repository (see my previous post introducing apt-p2p). This update brings mostly speed improvements to finding peers to download from, and performing the download. Nodes in the DHT are now checked for responsiveness more thoroughly before adding them to the routing table, and rechecked more frequently afterwards. This should prevent bad nodes from propagating through the system, and departing nodes remaining in the system long after they have left. Downloading from peers is also quicker, as bad peers are dropped with a quicker timeout value and after a small number of errors. There was also a bug revealed by about 10% of the mirrors that caused downloads to fail, and was fixed in version 0.1.2. This update is STRONGLY recommended, as keeping unresponsive nodes out of the DHT will make everyone's experience better. If you tried apt-p2p and had some problems, I hope you'll consider trying again now. Unfortunately, these are enhancements I could not predict needing before releasing to the public, as the problems they solve are all caused by having a larger number of users, some of which are unresponsive. There are still a large number of peers (maybe 50% of them) that are unreachable, and so can not share any files with other peers. I don't have a good way to check yet, but you can go to sites such as this one to probe your client for you. Just enter the remote IP address and port (available on the status page) of your client in the form http://ipaddress:port/ (e.g. http://1.2.3.4:9977/). If the check returns any HTTP headers (even a 404 Not Found response) then you're fine, but if it doesn't return anything then your peer is firewalled or NATted and should probably be fixed. Here are the changelog entries for 0.1.2 and 0.1.3:

Speed up downloading from peers
- Set a new peer's ranking values so they don't get an unfair advantage.
- Reduce the HTTP connection timeout to 10s.
- Drop peers after a limited number of errors.
Speed up the DHT requests when nodes fail
- Schedule a re-ping message after adding a new node.
- When a node fails, schedule a future ping to check again.
- Send periodic finds to nodes that are stale.
- Increase the stored value redundancy to 6.
- Increase the concurrency of DHT requests to 8.
- Add early termination and ignoring slow responses to recursive DHT actions when timeouts occur.
Remove the debconf note about port forwarding (Closes:#479492)
- Add a NEWS entry for port forwarding
Fixed a bug in the HTTP downloader that caused errors with some mirrors that always close the connections (Closes: #479455)

Going just from the title of Cameron's post about his apt-p2p tool: How about automatically running a zeroconf-enabled webserver serving /var/cache/apt/archives (and any mounted local repositories?) during install and while aptitude is running (and, if the user allows, as a daemon by default on a running system) and of course a corresponding sources.list line and apt retrieval module. This would hugely improve installation time for people with just a few machines who are too lazy to set up their own apt cache. Obviously, the details would be tricky, but since there is a security chain from Release.gpg to the .deb, downloading packages from untrusted peers shouldn't be a problem, even if the package name / version pair is not really unique (or even if someone actually tries a spoofing attack.) (There's a thought: running a tftp / dhcp server with a fai network boot in addition would be a step towards world domination, too, closely followed by automated hacking tools to install Debian on all computing devices on the network. Support call: “My printer only prints one page saying debian:~# after power on and then stops.”) Update 2008-05-07: Both Nijel and Florian Ludwig himself pointed out his apt-zeroconf project, which is unfortunately not very actively developed at the moment. Yet another thing I should/could do if I had the time.

After many long months of planning and work, I have completed another peer-to-peer downloader for Debian. If you've been keeping track, that makes 2 now. This one is called apt-p2p, and as of yesterday it is available in unstable. The functionality is very similar to the first one I wrote, DebTorrent, so if you've used that one you should feel very comfortable. After installation you just add a localhost:9977 to the start of your sources.list entries (see the man page). The only difference is the port number (DebTorrent uses 9988 by default) and that you can use it on all your sources.list entries, whether they be deb, deb-src, official Debian archive, or any other archive. Then an apt-get update gets it started, and you can begin installing packages. Point your web browser to http://localhost:9977 to see what's going on. IMPORTANT: as with any P2P program, it works much better if you open a port through your NAT or firewall. Without this crucial step, you won't be able to share with any peers, and your lookup of peers to download from may take longer. Make sure to forward both TCP and UDP ports 9977, or whichever port you set in the config file. For more details, see the port forwarding section of the FAQ. The similarities with DebTorrent are all external, so let's look at how it differs from DebTorrent internally:

it's very general, it doesn't matter what you're trying to download, a source package or a Packages file, from constantly-updated unstable or a year-old stable, for i386 or hppa architectures (DebTorrent only works for .deb files, is only supported for the official archive, and breaks downloaders into groups by architectures)
it doesn't require anything other than what's available to apt (DebTorrent uses piece hashes of large files, and ordering information, both of which are downloaded separately)
it can be very fast when downloading from mirrors (mirror downloads with DebTorrent are not so fast)
the code is simple, and makes use of available code, such as Twisted, Khashmir, and python-apt, which should make future enhancements and maintenance easier (DebTorrent is large and monolithic)
requires less memory and CPU power (50% to 75% less memory than DebTorrent)

Here's some technical details for those interested:

makes use of hashes to uniquely identify files
uses a Distributed Hash Table (DHT) to find peers
also stores piece hashes in the DHT for efficient downloading
uses HTTP/1.1 requests to download from peers
no peers available causes it to fallback to a download from the mirror

Though I know it can be fast, I'm not yet sure if the peer lookup in the DHT will be quick enough to keep up with the downloading. All my tests so far show that it is, but until there are a number of peers out there trying it, I can't be sure. Also, I have some improvements in mind to enhance the speed, in particular the wait for a timeout to occur, so this may improve in the future. If you see a delay in downloading where apt seems to be stalled saying 'Waiting for headers', for now be patient and see what happens. It may be that apt-p2p is downloading in the background (it does this sometimes), or that it's waiting for a lookup in the DHT to complete. If it hangs for more than a minute, or there's errors in the log file, please file a bug so that I can look into it. Finally, for you DebTorrent fans don't worry, I haven't given up on it. Stay tuned for more info on it coming soon.

A quick update on the situation of the video streams from Linux Audio Conf; yesterday we had over 200 viewers watching the live streams, today even more. If there was a talk on yesterday that you wanted to see, the videos are available online here.

As usual, the live streams are available here.

The streams from the Linux Audio Conference 2008, being held in K ln this year are now online. If you are interested in music, audio and video related topics in Linux, some of the talks may be very interesting to you, and definitely worth watching.
The streams can be found at http://lac.linuxaudio.org/?page_id=29.

As some of you may know, we have been streaming video streams from the Debian room at FOSDEM. This has been working with rather variable results. The machine handling the encoding is a 2 GHz P4-m laptop. It seems to not be able to keep a video stream bigger than the size of a postage stamp going without crashing. Now although this is better than nothing (you can at least still hear the audio live, and see a humanoid figure walking about), being able to see the slides clearly would also be nice.
Tomorrow, this may be improved on. The video stream (when it's online) is at http://fosdem.debconf.org:8000/stream.ogg

I have uploaded a new version of DebTorrent to experimental and to my personal repository. This new version implements the unique piece numbers, which has been planned for quite some time now. Well, it's finally here. To summarize, unique piece numbers keep torrent's alive longer, by assigning files unique piece numbers that never get reused in that torrent. New files get new piece numbers added to the end, but peers in the old torrent can still share most of the old files with peers in the new torrent (in fact, it's the same torrent, but old and new peers have different ideas about what it contains). The creation of torrents from Packages files has also changed, as now 2 torrents are created, one for whatever arch the Packages file was for, and one for the Architecture:all files. Since the Arch:all files are the same for all architectures, this also allows for more sharing of common files between peers on different architectures. This change required a change in the cache directories, which is described in more detail in the NEWS file, but is handled almost automatically. The only thing to do is the make sure to do an apt-get update after upgrading so that the torrents can be restarted. There was also some changes in the statistics reporting. In the client, the uploaded and downloaded statistics will now persist over restarts so that you can see how much you've done over a longer period. The tracker status page also got an update, and now shows the total uploaded and downloaded bytes for each torrent, as well as some more descriptive names for some of the torrents. Due to the large changes in this release, I opted for uploading to experimental (and my repository) for now, so that it can be tested a bit. Please do test it, and let me know of any bugs, problems or concerns that you may have. Here is the changelog:

Add support for unique piece numbers
- increases duration of oft-updated torrents so that more peers can participate
- currently supported only by debian testing and unstable
- see http://wiki.debian.org/DebTorrent/UniquePieces for more info
Switch to using 2 torrents per Packages file: one for architecture- specific files, and one for architecture-independent files
- also added a new script splitcachefor_all to ease the upgrade
Use python-debian for all reading of RFC 822 type files
- also requires python-apt
Add torrent names to the tracker display
Make the download/upload statistics persist over restarts
Report more and better statistics on the tracker's info page

It's been a while, but a new update to DebTorrent is available in unstable. There is also a new version of the helper apt-transport-debtorrent (0.2.0) to go with it, which was mostly the cause of the long delay (due to a pending apt transition to testing). It is now highly recommended to install both, read below for new reasons why. One of the previous problems with debtorrent was the incorrect status updates displayed by apt, due to it not being aware of pieces of large files that have already downloaded. Using these new versions together will help to improve the update status messages shown during a download. Now, apt-get update will display a status at the bottom that will look something like:

DebTorrent: 837MiB left at 319 KiB/s (46m03s)

You may not be able to see it all if there are multiple updates to different mirrors under way. In aptitude, the status will appear on a line by itself at the bottom of the list of downloaded files. The status line will update about once per second, and should give a good indication of how much time is really left. The other (incorrect) status information presented by apt/aptitude will still be there but you can ignore it. I've only tested apt and aptitude, so I'm not sure how (or if) synaptic, adept or gnome-apt will display these messages, though they are general apt status messages, usually used for displaying the status while logging into an FTP server. This should also fix another problem that debtorrent can have with downloading large files (usually on slower connections). Since apt can't see the download happening for a long time, the connection would sometimes time out. With these status updates the connection is constantly active, so the timeouts shouldn't be a problem when downloading large packages (although they may still occur when doing an apt-get update, as there are no status updates during that process). I did implement a better solution, as suggested by Micha Politowski on the wiki. This solution uses the creation of sparse files to let apt know how much of a file has actually been downloaded. However, apt doesn't understand how a single process can download multiple files in prarallel, so it only follows the last file that was started, and complains when a file finishes that was not the last one that started. I've disabled that for now while I see if it's possible to modify apt to work better in this situation. Here are the changes for debtorrent:

Update to support apt debtorrent transport version 0.2
- send piece downloaded status messages (currently disbaled due to apt not liking it)
- send general status update messages
Fix some minor packaging issues
Upgrade to standards version 3.7.3 (no changes)
Remove the unneeded binary-arch rule
Changes the XS-Vcs-* headers to Vcs-*
Moved Homepage from description to Source package fields

And for apt-transport-debtorrent:

Upgrade transport version to 0.2
- Create sparse files based on status messages (102) from DebTorrent
- Send general status updates to apt (103) from DebTorrent
Remove support for Range headers as they may confuse the sparse file allocation
Fix typo in long description
Upgrade to standards version 3.7.3 (no changes)
Changes the XS-Vcs-* headers to Vcs-*
Moved Homepage from description to Source package fields

After having tried contacting tech support at Lenovo a few times, they finally realised that there was a problem with my 3g/WWAN card, and promptly sent me a replacement one and an unlock code for said card without any further hassle.
The new card is now in my machine and working well. Woot.

Since the conclusion of the Google Summer of Code, the DebTorrent project has been without a sponsor. I was going to email debian-mentors to see if anyone was interested, but I decided to first post to my blog to see if there are any Debian Developers reading this who have the time and are especially interested in sponsoring it. Here are some of the details of the package:

Package name: debtorrent
Current version: 0.1.4.1
Author: Cameron Dale camrdale@gmail.com
URL: http://debtorrent.alioth.debian.org/
License: MIT
Section: net
Priority: optional
Language: python
VCS: subversion http://svn.debian.org/wsvn/debtorrent/debtorrent/trunk/

It is currently maintained using an svn-buildpackage subversion repository, but I might be interested in switching to git if a sponsor is interested.

I recently upgraded my Internet service with Telus, which included a new wireless router to replace my old high-speed modem. The new router is a 2Wire 2700HG-E which is configured through a web interface. The web interface worked fine at first on some other machines, but whenever I accessed it from my main desktop machine the pages would either fail to load, or have the images in the wrong locations. After much debugging (though it seems obvious now) I finally discovered that this was due to my enabling HTTP pipelining in FireFox on that machine (as suggested by Mozilla). Being the diligent citizen that I am, I of course filed a support request so that this could be fixed in the next version of the 2Wire software. Here is the response I received:

The HomePortal GUI was designed for internet browsers on default settings. We recommend keeping the pipelineing [sic] setting on "false".

Not exactly a good solution, and a little surprising, considering it's relatively straight-forward to implement. I know because I recently upgraded a simple HTTP handler from 1.0 to 1.1 specifically to support pipelining, as part of my work on DebTorrent. Now I've just read "Structured streams: a new transport abstraction", in which the authors state:

implementing pipelining correctly in web servers has proven challenging enough that seven years after the standardization of HTTP/1.1, popular browsers still leave pipelining disabled for compatibility

So it seems pipelining is disabled for a reason, though I had it enabled for years in FireFox and saw no problems (that I know of) until now. Maybe the problems are a thing of the past. Does anyone else have any experiences (good or bad) with pipelining to share?

Search Results: "mrd"

10 May 2021

17 May 2020

7 September 2016

7 July 2015

31 December 2012

28 September 2008

7 July 2008

22 June 2008

20 June 2008

10 May 2008

5 May 2008

4 May 2008

29 February 2008

28 February 2008

23 February 2008

28 January 2008

9 January 2008

19 October 2007

14 October 2007

21 September 2007