Erich Schubert: ELKI applying for GSoC 2012
A General Framework for Increasing the Robustness of PCA-Based Correlation Clustering Algorithms(part of my diploma thesis).
Hans-Peter Kriegel, Peer Kr ger, Erich Schubert, Arthur Zimek
Scientific and Statistical Database Management (SSDBM 2008)
One of the earliest features I wrote for the Debian bug tracking
system (Debbugs) after joining the team was support for forcibly
merging bugs. Originally, merging two bugs required that the bugs be
in exactly the same state before merging them; forcemerge removed
this requirement.
Unfortunately, the way I originally implemented this was shortsighted,
and merely forced the merge partners to have the same values as the
merge master. This meant that owners, blocking bugs, and many other
things were silently changed, which meant that people weren't notified
of changes, and bugs could end up in an inconsistent state.
A while ago, I decided to fix this by calculating the changes required
to actually merge the bugs, making those changes, and then merging the
bugs normally; thus, doing everything that a maintainer would normally
have done for them. This necessitated abstracting out the entire
control apparatus into the Debbugs::Control module.
Now that it's complete, you can do the following:
> forcemerge 1 2
Bug #1 [foo] new title
Bug #2 Done: foo@bugs.something [foo] foo
Unset bug forwarded-to-address
Severity set to 'wishlist' from 'grave'
3 was blocked by: 2
3 was not blocking any bugs.
Removed blocking bug(s) of 3: 2
2 was blocked by: 4
2 was not blocking any bugs.
Removed blocking bug(s) of 2: 4
Bug reopened
Removed annotation that bug was owned by bar@baz.com.
Removed indication that 2 affects bleargh
Removed tag(s) unreproducible and moreinfo.
Merged 1 2
> thanks
Stopping processing here.
and bug 2 now is merged with 1 and matches the state of 1.
[The above is the control output from the appropriate bit of the 06_mail_handling.t test.]
This change also means that I'll be able to finally write support for
control@ operations at submit@ time. Also, all of the bug
modifications that happen at submit@ or nnn@ time (setting title,
found, etc.) will be implemented as calls to Debbugs::Control so we
can eventually keep a postgresql database updated in addition to the
flatfile database.
We're heading towards final phpMyAdmin 3.5, so let's again look at translations status. Here comes fifth round of translation status update.
Since last update we one less translation (it was not really complete, the po file missed few strings) at 100%:
Filed under: English Phpmyadmin 0 comments Flattr this!
During the last week, I created ghc-heap-view, a library to investigate the actual memory representation of Haskell values. It is inspired by vacuum and the GHCi debugger, but goes beyond them by allowing the user to look inside thunks and functions and see what other values they refer to. Let me demonstrate it by running the included demo:
ghc-heap-view-demo
Here are a four different lists, where the first three are already evaluated.> l = [1,2,3]and is now found at 0x00000000006d1750/2 (where the /2 is the pointer tag information) and fully evaluated:
ConsClosure info = StgInfoTable ptrs = 2, nptrs = 0, tipe = CONSTR_STATIC, srtlen = 1 ,
ptrArgs = [0x00000000006d16e0/1,0x00000000006d1730/2],
dataArgs = [], descr = "ghc-prim:GHC.Types.:"
The second one, l2, is locally defined
> let l2 = 4:land now found at 0x00007fdce19fe4b0/2. See how the cons-cell references l!
ConsClosure info = StgInfoTable ptrs = 2, nptrs = 0, tipe = CONSTR_2_0, srtlen = 1 ,
ptrArgs = [0x00000000006dca50/1,0x00000000006d1750/2],
dataArgs = [],
descr = "ghc-prim:GHC.Types.:"
And the binding
> args <- map length fmap getArgsevaluates to the one , global empty list at 0x00000000006db640/1:
ConsClosure info = StgInfoTable ptrs = 0, nptrs = 0, tipe = CONSTR_NOCAF_STATIC, srtlen = 0 ,
ptrArgs = [],
dataArgs = [],
descr = "ghc-prim:GHC.Types.[]"
And now we have, at 0x00007fdce19fe4c8, the concatenation of them, but unevaluated:
> let x = l ++ l2 ++ argsThe thunk keeps a reference to l2 and args, but not l, as that is at a static address, unless you are running this in GHCi:
ThunkClosure info = StgInfoTable ptrs = 2, nptrs = 0, tipe = THUNK_2_0, srtlen = 1 ,
ptrArgs = [0x00007fdce19fe4b0/2,0x00000000006db640/1],
dataArgs = []
Now to some more closure types. m and m' locally bound of type the unboxed type Int#, with values 42 resp. 23.
> let f = \x n -> take (I# m + I# x) n ++ args t = f m' l2So here is (0x00007fdce1937d50/2), referencing its free variables args and 42:
FunClosure info = StgInfoTable ptrs = 1, nptrs = 1, tipe = FUN_1_1, srtlen = 65553 ,
ptrArgs = [0x00000000006db640/1],
dataArgs = [42]
And t is a thunk that applies f (also referenced here) to an unboxed value (23) and l2:
ThunkClosure info = StgInfoTable ptrs = 2, nptrs = 1, tipe = THUNK, srtlen = 0 ,
ptrArgs = [0x00007fdce19fe4b0/2,0x00007fdce1937d50/2],
dataArgs = [23]
Lastly, here is the standard example for self reference:
> let x = id (:) () xThis is what x (0x00007fdce1947940) looks like, at least without -O:
ThunkClosure info = StgInfoTable ptrs = 0, nptrs = 0, tipe = THUNK, srtlen = 1 ,
ptrArgs = [],
dataArgs = []
So it is unevaluated. Let us evaluate it using seq. Now we have, still at 0x00007fdce1947940:
IndClosure info = StgInfoTable ptrs = 1, nptrs = 0, tipe = BLACKHOLE, srtlen = 0 ,
indirectee = 0x00007fdce194cc98/2
The thunk was replaced by an indirection. If we look at the target, 0x00007fdce194cc98/2, we see that it is a newly created cons-cell referencing the original location of x:
ConsClosure info = StgInfoTable ptrs = 2, nptrs = 0, tipe = CONSTR_2_0, srtlen = 1 ,
ptrArgs = [0x00000000006db620/1,0x00007fdce1947940],
dataArgs = [],
descr = "ghc-prim:GHC.Types.:"
After running the garbage collector (performGC), we find that the address of x is now 0x00007fdce19f30d0/2 and that the self-reference is without indirections:
ConsClosure info = StgInfoTable ptrs = 2, nptrs = 0, tipe = CONSTR_2_0, srtlen = 1 ,
ptrArgs = [0x00000000006db620/1,0x00007fdce19f30d0/2],
dataArgs = [],
descr = "ghc-prim:GHC.Types.:"
Future plans
The output of ghc-heap-view is not really pretty yet; even the indentation in this blog post was added manually by me, so this really needs a pretty printer providing a nicer, possibly more compact representation, including something like what vacuum provides. Maybe vacuum can be ported to use this library, and also include the thunk s and function s references in the output. Maybe also the GHCi debugger can be extended to show more information about unevaluated expressions using this. Internally, the library is not very polished yet either. It only handles those closures types that I have seen so far, and is likely to break horribly if run in a threaded or debugging enabled runtime.
How it works
Obviously, this is not standard Haskell 98 code, but rather deep trickery involving the GHC API and some C code. Initially I tried to use the API that vacuum and the GHCi debugger rely on, which is an operation
unpackClosure# :: a -> (# Addr#, Array# b, ByteArray# #)which takes any Haskell value and returns the address to its info table, the pointers and the non-pointer-data in the closure. Unfortunately, it was not complete in that it was meant only for data closures and will for other closure types, e.g. thunks, return no data and no pointers (as can be seen in the code). So I implemented my own version of this operation:
foreign import prim "slurpClosurezh" slurpClosure# :: Any -> (# Addr#, ByteArray#, Array# b #)where the returned ByteArray# contains the complete closure, including extra information fields such as the arity of a function. The Array# is again the list of pointers in the closure. At first glance, this is a duplication, as the pointers are of course also contained in the ByteArray#. But as soon as the GHC runtime reigns again, a garbage collector run can happen, the referenced values will move somewhere else, and the words that once were pointers in the ByteArray# become useless. But the corresponding entries in the Array# are updated by the garbage collector, as it knows that these are pointers, and not just words. This way, we get both a faithful copy of the closure on the heap and useful references to the contained data. Here is a demonstration of this effect:
$ ghci -XMagicHash -package ghc-heap-view GHCi, version 7.4.1: http://www.haskell.org/ghc/ :? for help Loading package ghc-prim ... linking ... done. [..] Loading package ghc-7.4.1 ... linking ... done. Loading package ghc-heap-view-0.1 ... linking ... done. Prelude> let a = [1,2,3,4]; b = 5:a Prelude> :m + GHC.HeapView Prelude GHC.HeapView> rawHeapData <- getClosureRaw b Prelude GHC.HeapView> rawHeapData (0x000000004080d658,[1082185320,140040739366568,140040739365928],[0x00007f5dc68626a8,0x00007f5dc6862428]) Prelude GHC.HeapView> System.Mem.performGC Prelude GHC.HeapView> rawHeapData (0x000000004080d658,[1082185320,140040739366568,140040739365928],[0x00007f5dc41b3ad8,0x00007f5dc41b3b28])The function rawHeapData is a thin wrapper around slurpClosure# which turns the primitive array in normal lists. Note that the second component of the triple is unchanged, but the third is updated by the garbage collector. Of course this means that the Show instance for the data type that ghc-heap-view uses to reference values is not referential transparent either. The foreign function import above is of type prim , i.e. does not call a C function but rather a Cmm function. Cmm is a reduced C that GHC uses internally to compile the Haskell code to, and most primitive operations are implemented in this language although I do quickly call regular C from my Cmm code to do the more complicated stuff, mainly figuring out what words of the closure are pointers. The knowledgeable reader might notice that I am passing a boxed value of type Any to the foreign function. This is currently not possible with foreign prim functions, and to actually use that code, you need the patch in GHC ticket #5931. But you can use ghc-heap-view without that as well (and the Cabal package will by default use that path), using the following hack to obtain the pointer to a Haskell value on the Heap as an unboxed type that can pass to the primitive operation:
foreign import prim "slurpClosurezh" slurpClosure'# :: Word# -> (# Addr#, ByteArray#, Array# b #) data Ptr' a = Ptr' a aToWord# :: Any -> Word# aToWord# a = case Ptr' a of mb@(Ptr' _) -> case unsafeCoerce# mb :: Word of W# addr -> addr slurpClosure# :: Any -> (# Addr#, ByteArray#, Array# b #) slurpClosure# a = slurpClosure'# (aToWord# a)This works because a Word and a Ptr' have the same closure layout, only differing in the fact that one stores an a, and the other stores a Word#. Once we obtained the raw representation of the closure, we do the parsing in Haskell. Using the info table and the raw closure, we have enough information to tell which words have to be replaced by the appropriate pointer (which might already have been updated by the garbage collector) in the pointers list. This work was supported by a scholarship from the Deutsche Telekom Stiftung.
I have an old motherboard that requires creating a DOS boot floppy in order to upgrade its BIOS. Fortunately, it's not too hard to do this with FreeDOS and a USB stick.apt-get install makebootfat syslinux
cd /tmpand then create a directory for the files that will end up in the root directory of the "floppy":
wget http://www.ibiblio.org/pub/micro/pc-stuff/freedos/files/distributions/1.0/pkgs/commandx.zip
wget http://www.ibiblio.org/pub/micro/pc-stuff/freedos/files/distributions/1.0/pkgs/kernels.zip
wget http://www.ibiblio.org/pub/micro/pc-stuff/freedos/files/distributions/1.0/pkgs/substx.zip
wget http://www.ibiblio.org/pub/micro/pc-stuff/freedos/files/distributions/1.0/pkgs/unstablx.zip
for ZIP in *.zip; do unzip $ZIP; done
cp ./source/ukernel/boot/fat16.bin .
cp ./source/ukernel/boot/fat12.bin .
cp ./source/ukernel/boot/fat32lba.bin .
cp /usr/lib/syslinux/mbr.bin .
mkdir /tmp/fs-rootand copy the BIOS image and update program into that same directory (/tmp/fs-root/).
cp ./bin/command.com /tmp/fs-root/
cp ./bin/kernel.sys /tmp/fs-root/
/usr/bin/makebootfat -o /dev/sdb -E 255 -1 fat12.bin -2 fat16.bin -3 fat32lba.bin -m mbr.bin /tmp/fs-root
Justin TV is a streaming service that uses the broken flash concept in order to allow you to view channels.
Unfortunately it's the only source of Arabic movies, series and shows.
I wrote a GStreamer source that retrieves the channel information and uses libRTMP to connect to the Justin streaming server. The rest is handled by GStreamer ;-)
If you want to watch http://www.justin.tv/ammartv for example, you simply do
gst-launch-0.10 playbin uri=jtv://ammartvThe 2nd plugin is a Totem plugin that indirectly utilizes the GStreamer source.
Given a language code (ar or en or ...), it will download the list of channels in that language and shows them in Totem's side panel. One can then easily double click on a channel and simply watch it. I originally wanted to do the Totem plugin in python but the inability to do asynchronous callback based HTTP requests prevented me :-(
Be warned that the code is of course a bit buggy and needs a lot of cleanups, it's been years since I coded anything in Gtk and that Justin might find out and either block the source somehow or even change the protocol but let's wait until that happens ;-) read more
Recently I accidentally flooded Planet Debian with my blog feed. This was an accident caused by some of my older blog entries not having valid "Date:" headers. (I use chronicle which parses simple text files to build a blog, and if there is no Date: header present in entries it uses the CTIME of the file(s).)
So why did my CTIMEs get lost? Short version I had a drive failure and a PSU failure which lead to me rebuilding a few things and cloning a fresh copy of my blog to ~/hg/blog/.
My host is now once again OK, but during the pain the on-board sound started to die. Horribly crackly and sounding bad. I figure the PSU might have caused some collateral damage, but so far thats the only sign I see.
I disabled the on-board sound and ordered a cheap USB sound device which now provides me with perfect sound under the Squeeze release of Debian GNU/Linux.
In the past I've ranted about GNU/Linux sound. So I think it is only fair to say this time things worked perfectly - I plugged in the device, it was visible in the output of dmesg, and /proc/asound/cards and suddenly everything just worked. Playing music (mpd + sonata) worked immediately, and when I decided to try playing a movie with xine just for fun sound was mixed appropriately - such that I could hear both "song" + "movie" at the same time. Woo.
(I'm not sure if I should try using pulse-audio, or similar black magic. Right now I've just got ALSA running.)
Anyway as part of the re-deployment of my desktop I generated and pass-phrased a new SSH key, and then deployed that with my slaughter tool. My various websites all run under their own UID on my remote host, and a reverse-proxy redirects connections. So far example I have a Unix "s-stolen" user for the site stolen-souls.com, a s-tasteful user for the site tasteful.xxx, etc. (Right now I cannot remember why I gave each "webserver user" an "s-" prefix, but it made sense at the time!)
Anyway once I'd fixed up SSH keys I went on a spree of tidying up and built a bunch of meta-packages to make it a little more straightforward to re-deploy hosts in the future. I'm quite pleased with the way those turned out to be useful.
Finally I decided to do something radical. I installed the bluetile window manager, which allows you to toggle between "tiling" and "normal" modes. This is my first foray into tiling window managers, but it seems to be going well. I've got the hang of resizing via the keyboard and tweaked a couple of virtual desktops so I can work well both at home and on my work machine. (I suspect I will eventually migrate to awesome, or similar, this is very much a deliberate "ease myself into it" step.)
ObQuote: "Being Swedish, the walk from the bathroom to her room didn't need to be a modest one. " - Cashback.
Summary
The JSON License may
seem interesting, but it a bad license, both non-free and ambiguous: do
not use it. If you are the author of a piece of software that
uses JSMin
or its PHP port,
consider dropping that non-free part or at least rendering it
optional.The JSON license
Do you know the JSON License?
It is a software license inspired by the MIT/Expat license with one
additional morality clause:
The Software shall be used for Good, not Evil.Non-free For novices that clause can be perceived as a good idea serving a noble goal. But the problem here is that is would make your software non-free. Indeed, there are two common definitions of free software: the FSF definition, and the Debian Free Software Guidelines, also known as the Open Source Definition. Basically, both definitions indicate that to be free, a piece of software must be usable with no restrictions, and forbidding evil uses is such a restriction. Specifically, it violates:
The Software should rather be used for Good, not Evil.The JSMin case This infamous JSON License is known to be used by JSMin and its PHP port. If you are a Web application author, be careful when embedded external content, since it can render you whole piece of software non-free! If you are using JSMin, I would suggest that you drop it, or at least make it optional by using a conditional (if JSMin is here, use it, otherwise do nothing) so that distributors such as Debian can expurgate that non-free part.
I have just bought our plane tickets to Managua, so I can finally say this:
Yes, many of you will ask what happened, I was bragging everywhere I wanted to go by land, driving from Mexico City to Managua. I'd love to, and I'm sure it's completely doable... But we have family issues to attend on July 21, in Argentina. So we will have a beautiful flight schedule (and carbon footprint) for this July:
Next.