Search Results: "ema"

8 June 2010

Russell Coker: Is the PC Dying?

I just read an interesting article about the dispute between Microsoft and Apple about types of PC [1]. Steve Jobs predicted a switch from desktop PCs to portable devices, while Steve Ballmer of Microsoft claimed that the iPad is just a new PC. Defining a PC I think that the defining characteristic of the IBM Compatible PC was it s open architecture. Right from the start the PC could have it s hardware expanded by adding new circuit boards into slots on the motherboard (similar to other PC systems of that era such as the Apple 2 and the S-100 bus). The deal with IBM included Intel sharing all it s CPU designs with other manufacturers such as NEC and AMD from the 8086 until the mid-90 s. AMD specialised in chips that were close copies of Intel chips at low prices and higher clock rates while NEC added new instructions. Compaq started the PC clone market as well as the laptop market, and system software for the IBM compatible PCs was primarily available from IBM and Microsoft in the early days, along with less popular variants such as CP/M86, Novell Netware and others. In the late 80 s there was OS/2 as an alternate OS and Windows as one of several optional GUI environments to run on top of MS-DOS or PC-DOS. In the mid 90 s PCs were used for running protected mode OSs such as Linux and Windows/NT. Now if we look at a system such as a Netbook then it clearly misses some of the defining characteristics of the desktop PC. I can t upgrade a Netbook in any meaningful way changing a storage device or adding more RAM does not compare to adding an ISA/MCA/EISA/VL-Bus/PCI/PCIe expansion card. With my EeePC 701 I don t even have an option of replacing the storage as it is soldered to the motherboard! A laptop allows me to add a PCMCIA or PC-Card device to expand it, but with a maximum of two cards and a high price this isn t a great option. What is Best for Home Users? For a while now my parents have been using 3G net access for their home Internet use [2]. So it seems that a laptop provides greater benefits for their use now than it previously did when they used Cable and ADSL net access. My parents have been considering getting a new monitor (1920*1080 resolution monitors are getting insanely cheap nowadays) and driving such a monitor effectively might require a more capable PC. I recently bought myself a nice refurbished Thinkpad for $796 [3], it seems likely that I could find a refurbished Thinkpad at auction which is a little older and slower for a lower price, even buying an old T41p would be a reasonable option. This would give my parents not only the option of using the Internet when on holidays, but also in a different part of their house when they are at home. The Apple iPad would probably be quite a reasonable Internet platform for my parents if it wasn t for the fact that it uses DRM. While it s not a great platform for writing, my parents probably don t do enough that it would be a huge problem for them. So I might look for a less restrictive tablet platform for my parents. At the moment the best resolution for a tablet seems to be 1024*768, but I expect that some tablets (maybe with a hybrid tablet/laptop design like the Always Innovating Smartbook [4]) with a higher resolution will be released soon. I hope that the iPad and other closed devices don t get any serious market share, but it seems likely that OSs such as Android which are only slightly more open will have a significant market share. Ultra-Mobile Design vs PCs Design One significant problem with ultra-mobile devices is that they make significant engineering trade-offs to get the small size. For a desktop system there are lots of ways of doing things inefficiently, running the AMD64 or i386 architecture which is wasteful of energy and having lots of unused space inside the box in case you decide to upgrade it. But for a laptop there are few opportunities for being inefficient, and for a tablet or smart phone everything has to be optimised. When the optimisation of a device starts by choosing a CPU that s unlike most other systems (note that there is a significant range of ARM CPUs that are not fully compatible with each other) it makes it very difficult to produce free software to run it. I can salvage a desktop PC from a rubbish bin and run Linux on it (and I ve done that many times), but I wouldn t even bother trying to run Linux on an old mobile phone. It seems that in the near future my parents (and many other people with similar needs) will be best suited by having a limited device such as a tablet that stores all data on the Internet and not having anything that greatly resembles a PC. In many ways it would be easier for me to support my parents by storing their data in the cloud and then automatically backing it up to removable SATA disks than with my current situation of supporting a fully capable PC and backing it up to a USB device whenever I visit them. I m also considering what to do for some relatives who are about to go on a holiday in Europe, they want to be able to send email etc. It might not be possible just yet, but it seems like an ideal way of doing this would be to provide them with something like an iPad that they can use with a local 3G SIM for the country that they stay in and they could then upload all their best photos to some server that I can backup and send email to everyone they know. An iPad isn t good for this now as you don t want to go on holidays in another country while carrying something that is really desirable to thieves. Ultra Mobile Devices are Killing PCs It seems to me that Google Android and the Apple iPad/iPhone OS are taking over significant parts of the PC market. The people who are doing traditional PC things are increasingly using Laptops and Netbooks, and the number of people who get the freedom that a PC user did in the 80 s and 90 s is decreasing rapidly. I predict that by 2012 the majority of Linux systems will be running Google Android on hardware that doesn t easily allow upgrading to more open software. At the moment probably the majority of Linux systems are wireless routers and other embedded devices that people don t generally think about. But when iPad type devices running a locked-down Linux installation start replacing Ubuntu and Fedora desktop systems people will take notice. I don t think that the death of the PC platform as we know it will kill Linux, but it certainly won t do us any good. If there were smarter people at Microsoft then they would be trying to work with the Linux community on developing innovative new ways of using desktop PCs. Of all the attempts that Microsoft has made to leave the PC platform the only success has been the X-Box which is apparently doing well. Tablet devices such as the iPad could work really well in a corporate environment (where MS makes most of it s money). On many occasions I ve been in a meeting and we had to adjourn due to someone needing to go to their desk to look something up. If everyone had an iPad type device at their desk that used a wired network when it was available and encrypted wireless otherwise then for a meeting everyone could take their tablet without it s keyboard and be able to consult all the usual sources of data without any interruption. Could a high-resolution version of the iPad kill MS-Windows in the corporate environment?

7 June 2010

Daniel Stone: and one more thing ...

So, I spent an hour or two this afternoon following the iPhone 4 liveblog. It all looked fairly compelling (the screen!), right up until Steve's 'and one more thing': video calling.



HELLO I'M IN 2007, CAN YOU HEAR ME
(Photo of the Nokia N800, which shipped in January 2007, from rnair.)

The moral of the story? If you want to be four years ahead of the WWDC closing bombshell, email sales@collabora.co.uk. :)


PS: The 2010 'fuck it, we're going to fivefour blades' version; we also had a six-way video call going earlier today.

Martín Ferrari: Python WTF

Lately, I have recurring WTF moments when working with Python. I'm still more or less a novice in Python, as I started to use it in my day job last year, so sometimes it could be blamed to ignorance. Today I was writing some code that takes any file descriptor as argument, and thought it could be nice to accept also objects that represent one. But it seems there is no common ancestor to file objects and socket objects, for example. In Perl, I'd just check for IO::Handle. Then, I wrote code to check if a passed argument was a socket, or a file descriptor to one, with some obvious check: isinstance(s, socket.socket). WRONG, when you create sockets with socketpair, they are of an undocumented type which is not a descendant of socket, nor of SocketType! (There is a patch for this since 2007) In general, I have found a lack of any reasonable object hierarchy, and many inconsistencies in the low-level interfaces, which is a pity, because Python is in general quite tidy. Comparing with Perl is unavoidable, and I would have expected that Python had learnt more from it. Another thing that amazes me is the lack of anything like CPAN, which is one of the reasons Perl was so successful. The cheese factory doesn't come close, as you need to publish your files somewhere else, there is no automated testing, no automatically generated documentation, or anything that requires access to the source code. In Perl-land, many people loathe the venerable MakeMaker, and some of the alternatives are ugly as hell. The Python equivalent (or so) to Makemaker, distutils, looks to me like a bad joke: advanced customisation is black magic (if possible at all), and doesn't provide absolutely basic things like running your unit tests! After fighting with it for some days, I decided to go back writing my own Makefile and calling distutils just to build and install. Tags: Planet Debian, Planet Lugfi, Programaci n

Julien Danjou: Announcing erc-track-score

A couple of months ago, I've started using ERC to hang out on IRC. I've read all the pages on EmacsWiki about it, just to see how far I could customize it. I must admit that I was not disappointed, even if I expected to be. It's quite a nice software, and once well configured it's more convenient that my old irssi setup. While browsing EmacsWiki, I read an interesting idea about channel scoring/temperature on the erc-track page. The idea is to see if it's worth jumping to an IRC channel to see what people are talking about. Challenge accepted! I sat up and started to dig though ERC source code to find the information I needed about variables and functions. I finally did write something nice, which I called erc-track-score. And yet another piece of software I wrote for my lovely Emacs! How does it work? Ha-ha, I was sure you would ask. You're so predictable, dude! Read the following, and you'll know everything you ever wanted to know about it since the moment you read the title of that blog entry. Which probably turned you on. Nasty you. First of all, the score of a channel starts at zero. Zero means "seriously, don't bother, nothing is happening here". Upon each new message arrival, the score is incremented by 1. If a new message contains a keyword, your nickname or is sent by a pal, the score is increased by configurable values, by default between 2 and 20 points, depending on the match type. On the other hand, when a message is send by some fool, the score is decreased by 1 by default. Obviously, if the score is going negative, you really should not jump to the channel. Finally, the score is permanently and slowly brought back to 0. By default, the score is decreased by 1 point every 10 seconds. Overall, reading the score should gives you a good idea of the channel temperature. I'm still not sure what is the best formula to compute the score, but so far the default values seem quite good. We'll see.

6 June 2010

Christian Perrier: D-I string freeze for Squeeze about to be started

Today, I launched a proposal for a string freeze in Debian Installer. A string freeze is a formal agreement among developers to avoid changing localizable material without concertation. This is a commitment towards translators to guarantee them that, while they're doing an effort to complete their work, this effort won't be broken by a developer adding of modifying translatable material. This can be seen as a first step towards the preparation of a release of D-I. Indeed, the only release we did for Squeeze (D-I alpha1 release) had no formal translation update round. We currently have 45 languages that are considered to be "in good shape" (even if not all complete) and very likely to remain activated in Squeeze. However, 20 (TWENTY) languages are in danger of being deactivated because of low activity. To avoid this, I will once again take my superhero suit, add some leather to it, take a whip and start nagging translators, doing noise and, in short, annoying the entire world (including blogging too often), in order to shake people enough so that we reduce this number to zero. Be prepared. Interesting links:

5 June 2010

Ingo Juergensmann: Drupal - direkt import into database

I was trying the last week to import data from one database into a Drupal database. Basically this worked fairly well. The needed fields from the old database are easy to get by simple SELECT statements. Because I don't know anything about PHP, I doing the export/import with a Python script directly into the database. After all this is about articles, comments and tags, that need to be imported into a Drupal database.

The articles will be stored into tables node_revisions, node and node_counter. The tags into tables term_data and term_node. Comments will go into table comments. Here are the INSERTs:

Articles:

"insert into node_revisions (uid, body, teaser, log, timestamp, format, nid, title) values ('3', '%s', '%s', '', '%s', '1', '%s', '%s')" % (body, body, timestamp, i, title)

"insert into node (vid, type, uid, status, created, changed, comment, promote, moderate, sticky, language, tnid, translate, title) values ('%s', 'blog', '3', '1', '%s', '%s', '2', '1', '0', '0', '', '0', '0', '%s')" % (vid, timestamp, timestamp, title)

"insert into node_counter (nid,totalcount,daycount,timestamp) values ('%s', '100', '0','%s')" % (nid, timestamp)

Tags:

"insert into term_data (vid, name) values ('3', '%s')" % tag

"insert into term_node (nid, vid, tid) values ('%s', '%s', '%s')" % (nid, vid, tagtid)

Comments:

"insert into comments (pid, nid, uid, subject, comment, hostname, timestamp, status, format, thread, name, mail, homepage) values ('%s', '%s', '0', '%s', '%s', '%s', '%s', '0', '1', '01', '%s', '%s', '%s')" % (parentid, nid, ctitle, cbody, cip, ctimestamp, cauthor, cemail, curl)

sql="insert into term_node (nid, vid, tid) values ('%s', '%s', '%s')" % (nid, vid, tid)

The import is successful as there are comments and tags attached to each article or node. But there are still some problems. When accessing Recent Posts there are no entries listed. Yet, Drupal still thinks there are so many articles that it lists more than 30 pages:

Recent Post Errors

Something similar happens to the tags. The tags are actually assigned to and are displayed as well with the nodes. But when accessign taxonomys vocabular and trying to list the terms, the resulting page is empty:

Tag Errors

If my understanding of the Drupal6 database (on PostgreSQL this is) is correct, everything should be fine and properly stored into the database. But apparently there's something missing. But what? As already mentioned I have no clue about PHP, so using node_api or any other API calls by Drupal/PHP is no option for me. Except when the calls can be placed via Pythons system() call.

If anyone has some tips or hints for me, I would appreciate when you write a comment.

When there's interest I can provide the entire Python import script. I'm going to make it public anyway, when the last problems are solved.

Ingo Juergensmann: Drupal - direct import into database

I was trying the last week to import data from one database into a Drupal database. Basically this worked fairly well. The needed fields from the old database are easy to get by simple SELECT statements. Because I don't know anything about PHP, I doing the export/import with a Python script directly into the database. After all this is about articles, comments and tags, that need to be imported into a Drupal database.

The articles will be stored into tables node_revisions, node and node_counter. The tags into tables term_data and term_node. Comments will go into table comments. Here are the INSERTs:

Articles:

"insert into node_revisions (uid, body, teaser, log, timestamp, format, nid, title) values ('3', '%s', '%s', '', '%s', '1', '%s', '%s')" % (body, body, timestamp, i, title)

"insert into node (vid, type, uid, status, created, changed, comment, promote, moderate, sticky, language, tnid, translate, title) values ('%s', 'blog', '3', '1', '%s', '%s', '2', '1', '0', '0', '', '0', '0', '%s')" % (vid, timestamp, timestamp, title)

"insert into node_counter (nid,totalcount,daycount,timestamp) values ('%s', '100', '0','%s')" % (nid, timestamp)

Tags:

"insert into term_data (vid, name) values ('3', '%s')" % tag

"insert into term_node (nid, vid, tid) values ('%s', '%s', '%s')" % (nid, vid, tagtid)

Comments:

"insert into comments (pid, nid, uid, subject, comment, hostname, timestamp, status, format, thread, name, mail, homepage) values ('%s', '%s', '0', '%s', '%s', '%s', '%s', '0', '1', '01', '%s', '%s', '%s')" % (parentid, nid, ctitle, cbody, cip, ctimestamp, cauthor, cemail, curl)

sql="insert into term_node (nid, vid, tid) values ('%s', '%s', '%s')" % (nid, vid, tid)

The import is successful as there are comments and tags attached to each article or node. But there are still some problems. When accessing Recent Posts there are no entries listed. Yet, Drupal still thinks there are so many articles that it lists more than 30 pages:

Recent Post Errors

Something similar happens to the tags. The tags are actually assigned to and are displayed as well with the nodes. But when accessign taxonomys vocabular and trying to list the terms, the resulting page is empty:

Tag Errors

If my understanding of the Drupal6 database (on PostgreSQL this is) is correct, everything should be fine and properly stored into the database. But apparently there's something missing. But what? As already mentioned I have no clue about PHP, so using node_api or any other API calls by Drupal/PHP is no option for me. Except when the calls can be placed via Pythons system() call.

If anyone has some tips or hints for me, I would appreciate when you write a comment.

When there's interest I can provide the entire Python import script. I'm going to make it public anyway, when the last problems are solved.

UPDATE:

Killes on IRC hinted me that node_comment_statistics and term_hierarchy needs to be filled as well. Thanks for the hint!

Russell Coker: Bugs in Google Chrome

I m currently running google-chrome-beta version 5.0.375.55-r47796 on Debian/Unstable. It s the fastest web browser I ve used in recent times it s the first time that I ve run a browser that feels faster than my recollection of running IBM WebExplorer for OS/2 on a 486-66 system! It has a good feature set, and it s the only browser I ve used that in a typical configuration will make proper use of the screen space by not having a permanent status bar at the bottom of the sceen and by having tabs in the title-bar. But it s not perfect, here is a list of some bugs: Chrome Titlebar when maximisedChrome Titlebar when not maximisedRight of Chrome Titlebar when not maximised Above are three partial screen captures of Chrome, the first is when maximised and the second is when the window isn t maximised. Notice the extra vertical space above the tab in the title bar in the second picture. The third picture shows the right side of the titlebar and you can see a space below the three buttons where you can drag the window around no matter how many tabs you open that space below the three buttons is reserved. If the Chrome developers had removed the extra vertical space in the titlebar and reserved slightly more horizontal space then you would be able to drag the window around. While an anonymous commentator made a good point that the extra vertical space can be used to drag the window around when the maximum number of tabs are open, it seems that there are other ways of achieving that goal without wasting ~18 vertical pixels. Doing so would be a lot less ugly than what they did with finding text in the page. When I visit a web site that uses cookies from an Incognito Window (which means that cookies etc aren t stored) there is no option to say allow all cookies . This is really annoying when you get to a web site such as the IBM one which stores 5 cookies when you first load the page and then at least one new cookie write for every page you visit. Given that cookie data will be discarded as soon as the window is closed it seems like a good idea to have an option to allow all cookies for Incognito Windows even if all cookies aren t allowed for regular windows. Blocking all cookies would be OK too, anything but having to click on Block or Allow multiple times for each page load. The J and K keys don t work in a view of Venus version 0~bzr95-2+lenny1 (the latest version in Debian/Lenny). I once had a situation where I entered a . at the end of a domain name (which is quite legal there is always an implied dot) and Chrome then wouldn t take note of my request to accept all cookies from the domain. I haven t been able to reproduce that bug, but I have noticed that it stores the settings for whether cookies should be stored separately for domains that end in . , so www.cnn.com. is different from www.cnn.com . Iceweasel seems to just quietly strip the trailing dot. Of course this is better than Konqueror which won t even load a URL with a dot at the end. Chrome can be relied on to restore all windows rapidly after a crash, unlike Iceweasel which restores them at it s normal load speed (slow) and Konqueror which doesn t tend to restore windows. This is good as it does seem to crash regularly. In a response to my post about Chrome and SE Linux [1] Ben Hutchings pointed out that the --no-sandbox option to chrome disables the creation of a PID namespace and therefore makes debugging a lot easier, if I get a lot of spare time I ll try and track down some of the Chrome SEGVs. The JavaScript compiler is either buggy or it s not buggy in situations where people expect IE bugs. When using the Dell Australia web site I can t always order all options. When trying to order a Dell R300 1RU server with hot-plug disks in a hardware RAID array it seems impossible to get all the necessary jumpers which is a precondition to completing the order fortunately I only wanted to blog about how cheap Dell servers are so I don t actually need to complete an order. Dell s web site is also difficult in Iceweasel on occasion, so it s obviously more demanding than most sites. It might be a good test site for people who work on browsers as it s both demanding and important. When I select a URL to be opened in a new window (or when JavaScript does this) then the new tab is opened with about:blank listed as the URL. If the URL is for a PDF file (or something else that is to be downloaded) then the URL entry field is never updated to give the real URL. I believe that this is wrong, either the new tab shouldn t be opened or it should have the correct URL on display there is no benefit with a tab open to show nothing but about:blank in the URL entry field. Also if a URL takes some time to load then it may keep about:blank in the URL entry field for some time. This means that if you use the middle mouse button to rapidly open a few new tabs you won t be able to see what is to be loaded in each one. Sometimes I have several tabs loading and I m happy to close some unimportant ones if they are slow but some are worth waiting for. Overall that s not too bad. I can use Dell s site in Iceweasel, so the only critical bug is the cookies issue in Incognito Windows which makes the Incognito feature almost unusable for some sites.

4 June 2010

Marco Túlio Gontijo e Silva: Summer of Code weekly report #4

My project. I m publishing my report earlier this week, because there was a lot to talk about. This week I started to make changes in the code going in the direction of what I want to do. I haven t started a final implementation, but I m studying about how what I want to do will affect the rest of the Garbage Collection (GC). I noticed the code in rts/sm/Sweep.c was simple and similar to what I m planning to do, so I started changing how it works. Sweep in the Glasgow Haskell Compiler (GHC) is done by a bitmap, which contains a bit for each word in a memory block, and is set to 1 when there s an object starting in the mapped area and 0 otherwise. When there s a block with no objects starting at it, that is, all bits of the bitmap are set to 0, the block is freed.
        if (resid == 0)
         
            freed++;
            gen->n_old_blocks--;
            if (prev == NULL)  
                gen->old_blocks = next;
              else  
                prev->link = next;
             
            freeGroup(bd);
         
The bits are analyzed in a group of BITS_IN(W_), where BITS_IN(W_) is the number of bits in a word.
        for (i = 0; i < BLOCK_SIZE_W / BITS_IN(W_); i++)
         
            if (bd->u.bitmap[i] != 0) resid++;
         
If more than of the groups are completely set to 0, the block is considered fragmented.
            if (resid < (BLOCK_SIZE_W * 3) / (BITS_IN(W_) * 4))  
                fragd++;
                bd->flags  = BF_FRAGMENTED;
             
Immix, the GC algorithm I plan to implement in GHC, divides the blocks of memory in lines. My initial plan was to identify free lines. I decided to consider a the size of a line fixed in BITS_IN(W_) words, because it will map to a word in the bitmap, and the code was already analyzing in groups of BITS_IN(W_) words. This was very easy with the current code.
            if (bd->u.bitmap[i] != 0) resid++;
            else printf("DEBUG: line_found(%p)\n", bd->start + BITS_IN(W_) * i);
This worked, and showed some free lines. I m sure there are other ways of logging in GHC, but printf was the simplest way I could thought of. I measured the occurrence of free lines using the bernouilli program from the NoFib benchmark suite, calling it with 500 +RTS -w, to make it uses sweep. In 782 calls to GarbageCollect(), sweep() was called 171 times, for 41704 blocks to be swept and found 230461 free lines. This gives us about 5.5 free lines per block, from the 8 lines in each block (on 64 bits systems). The problem is that the bitmap is marked only in the start of the objects allocated, so even in a line that all bits are marked with 0 we can t assume that it s completely free, because there may be an object that starts in the previous line that is using the space of the line. Checking only for the previous line doesn t work either, because a big object can span several lines. What we can do here is a variation of conservative marking, as proposed in the Immix paper, checking only the previous line and working only with objects smaller than a line. To make sure I was working only with objects smaller than a line, I had to mark the blocks that contains medium objects and avoid them when seeking free lines. The block flags are defined in includes/rts/storage/Block.h, so I included another flag in this file, BF_MEDIUM.
/* Block contains objects evacuated during this GC */
#define BF_EVACUATED 1
/* Block is a large object */
#define BF_LARGE     2
/* Block is pinned */
#define BF_PINNED    4
/* Block is to be marked, not copied */
#define BF_MARKED    8
/* Block is free, and on the free list  (TODO: is this used?) */
#define BF_FREE      16
/* Block is executable */
#define BF_EXEC	     32
/* Block contains only a small amount of live data */
#define BF_FRAGMENTED 64
/* we know about this block (for finding leaks) */
#define BF_KNOWN     128
/* Block contains objects larger than a line */
#define BF_MEDIUM    256
The GHC GC is generational, there is, objects are allocated in a generation and, after a time, the ones that are still being used are moved the next generation. This idea assumes the death probability of younger objects is higher, so few objects are moved to the next generation. Sweep and Immix work only in the last generation so, to mark blocks with medium objects we have to check the size of the objects that are moved to the next generation. This is done in the copy_tag function of rts/sm/Evac.c. I inserted a code that checks for the object size and marks the block when it s bigger than BITS_IN(W_).
STATIC_INLINE GNUC_ATTR_HOT void
copy_tag(StgClosure **p, const StgInfoTable *info,
         StgClosure *src, nat size, generation *gen, StgWord tag)
 
    StgPtr to, from;
    nat i;
    to = alloc_for_copy(size,gen);
    if(size > 8)  
        Bdescr(to)->flags  = BF_MEDIUM;
     
So I updated the code in rts/sm/Sweep.c to only inspect for free lines in blocks without BF_MEDIUM mark.
            if (bd->u.bitmap[i] != 0) resid++;
            else if(!(bd->flags & BF_MEDIUM))  
                printf("DEBUG: line_found(%p)\n", bd->start + BITS_IN(W_) * i);
             
This also worked. Now, in the 32012 blocks there were 189015 free lines, found in the same number of GCs, making about 5.9 free lines per block. We considered only blocks with small objects, but we didn t ignore the first line of each group of free lines. This can be achieved by checking if the previous line was also free.
            if (bd->u.bitmap[i] != 0) resid++;
            else if(!(bd->flags & BF_MEDIUM) && i > 0 && bd->u.bitmap[i] == 0)  
                printf("DEBUG: line_found(%p)\n", bd->start + BITS_IN(W_) * i);
             
Now, from the 32239 blocks, 165547 free lines were found, giving 5.1 free lines per block. But there are more things to improve. If the whole block is free, we want to free it, instead of marking it s lines as free. So it s better to mark the lines after we know that the blocks are not completely free. So I left the code that checks the bitmap as it was, and included a line check only for blocks that are not completely free. At this point, I also associated the fragmentation test with blocks with medium objects, because in blocks of small objects we plan to allocate in free lines, so fragmentation is not a (big) issue.
            if (resid < (BLOCK_SIZE_W * 3) / (BITS_IN(W_) * 4) &&
                (bd->flags & BF_MEDIUM))  
                fragd++;
                printf("DEBUG: BF_FRAGMENTED\n");
                bd->flags  = BF_FRAGMENTED;
             
            else if(!(bd->flags & BF_MEDIUM))  
                for(i = 1; i < BLOCK_SIZE_W / BITS_IN(W_); i++)
                 
                    if(bd->u.bitmap[i] == 0 && bd->u.bitmap[i - 1] == 0)  
                        printf("DEBUG: line_found(%p)\n", bd->start + BITS_IN(W_) * i);
                     
                 
             
The total ammount of blocks increased dramatically: the blocks that become fragmented and were not called again in sweep made a huge difference. From the 345143 blocks, 1633268 free lines were found, or about 4.7 free lines per block. 9434 blocks were free, so, from the remaining blocks, we have about 4.9 free lines per block. Something we ll need then is a way to access these lines latter. The simplest way I thought to achieve it is constructing a list of lines, in each the first word of each free line is a pointer to the next free line, and the first word of the last free line is 0. It s useful to keep reporting the lines to stdout, so that we can then follow the list and check if we went to the same lines.
                    if(bd->u.bitmap[i] == 0 && bd->u.bitmap[i - 1] == 0)  
                        StgPtr start = bd->start + BITS_IN(W_) * i;
                        printf("DEBUG: line_found(%p)\n", start);
                        if(line_first == NULL)  
                            line_first = start;
                         
                        if(line_last != NULL)  
                            *line_last = (StgWord) start;
                         
                        line_last = start;
                        *line_last = 0;
                     
                 
             
         
     
    for(line_last = line_first; line_last; line_last = (StgPtr) *line_last)  
        fprintf(stderr, "DEBUG: line_found(%p)\n", line_last);
     
I printed the inclusion of the lines on the list to stdout, and the walking on the list in stderr, so that it d be easy to diff. There was no difference between the lists. There re another improvements that can be made, like using a list of groups of free lines, but I think it ll be better to think about this after studying how the allocation in the free lines will be done. That s where I m going to now.
There are some minor things I learned, and thought they worth blogging. The current GHC uses three strategies for collecting the last generation: copying, mark-compact and mark-sweep. Copying is the default until the memory reaches 30% of the maximum heap size; after that, mark-compact is used. Sweep can be chosen by a Real Time System (RTS) flag, -w. To use mark-compact always, the flag is -c. I ve been submitting small patches to the cvs-ghc mailling list, mostly about outdated comments. Most of them were accepted, except for one which contained a lot of commentary, and that indeed was not completely correct. I corrected it and resend to the list, but the message is waiting for approval because the message header matched a filter rule. I believe this is because I replied the message generated from darcs. There s a very useful ghc option, specially for testing the compiler, because in this case you need to rebuild the source, even when there re no changes in it. It s -fforce-recomp, and it makes only sense when used with --make.

Colin Watson: Hacking on grub2

Various people observed in a long thread on debian-devel that the grub2 package was in a bit of a mess in terms of its release-critical bug count, and Jordi and Stefano both got in touch with me directly to gently point out that I probably ought to be doing something about it as one of the co-maintainers. Actually, I don't think grub2 was in quite as bad a state as its 18 RC bugs suggested. Of course every boot loader failure is critical to the person affected by it, not to mention that GRUB 2 offers more complex functionality than any other boot loader (e.g. LVM and RAID), and so it tends to accumulate RC bugs at rather a high rate. That said, we'd been neglecting its bug list for some time; Robert and Felix have both been taking some time off, Jordi mostly only cared about PowerPC and can't do that any more due to hardware failure, and I hadn't been able to pick up the slack. Most of my projects at work for the next while involve GRUB in one way or another, so I decided it was a perfectly reasonable use of work time to do something about this; I was going to need fully up-to-date snapshots anyway, and practically all the Debian grub2 bugs affect Ubuntu too. Thus, with the exception of some other little things like releasing the first Maverick alpha, I've spent pretty much the last week and a half solidly trying to get the grub2 package back into shape, with four uploads so far. The RC issues that remain are: If we can fix that lot, or even just the ones that are reasonably well-understood, I think we'll be in reasonable shape. I'd also like to make grub-mkconfig a bit more robust in the event that the root filesystem isn't one that GRUB understands (#561855, #562672), and I'd quite like to write some more documentation. On the upside, progress has been good. We have multiple terminal support thanks to a new upstream snapshot (#506707), update-grub runs much faster (#508834, #574088), we have DM-RAID support with a following wind (#579919), the new scheme with symlinks under /dev/mapper/ works (#550704), we have basic support for btrfs / as long as you have something GRUB understands properly on /boot (#540786), we have full info documentation covering all the user-adjustable settings in /etc/default/grub, and a host of other smaller fixes. I'm hoping we can keep this up. If you'd like to help, contact me, especially if there's something particular that isn't being handled that you think you could work on. GRUB 2 is actually quite a pleasant codebase to work on once you get used to its layout; it's certainly much easier to fix bugs in than GRUB Legacy ever was, as far as I'm concerned. Thanks to tools like grub-probe and grub-fstest, it's very often possible to fix problems without needing to reboot for anything other than a final sanity check (although KVM certainly helps), and you can often debug very substantial bits of the boot loader - the bits that actually go wrong - using standard tools such as strace and gdb. Upstream is helpful and I've been able to get many of the problems above fixed directly there. If you have a sound knowledge of C and a decent level of understanding of the environment a boot loader needs to operate in - or for that matter specialist knowledge of interesting device types - then you should be able to find something to do.

Adrian von Bidder: Filesystems Quo Vadis: Clients

Probably it's just a question of me not paying enough attention to the news... So: pointers welcome. There are quie a few shiny new filesystems for local storage, like btrfs or HAMMER (and nilfs, Tux3, ext4, Reiser4...) It seems the distributed storage side is covered as well, with ceph being merged recently. There are other systems (like, for example, Lustre), but they haven't appeared much in the news channels I tend to read. What I'd be curious is if any of these support hierarchical storage architectures like pushing out rarely used data to tape libraries. (But this is just idle speculation, I don't need this anywhere.) But what I would really need is a replacement for NFS (v3): a classical client-server filesystem. I'm not sure NFSv4 is the right solution (where I'd use it, I currently can't because we rely too much on POSIX ACLs there, making the transition to NFSv4 quite a chore.) I think POHMELFS might be a solution in the long term, or CRFS, but I'm not sure how much progress there is on these; apt-cache search is silent, not a good sign. There's Samba 4 I guess I'll have to look at it, since it's supposed to be much cleaner and nicer to use than its predecessors and might be a good solution even if no Microsoft systems are involved. Is this what small environments (a fileserver and 100 clients or so) will want to use? Comments are very welcome.

John Goerzen: Hardware/Machine Reporting Tools

We have a tool at work, Kaseya, for our Windows machines. It handles updates, but also can report back information gathered from them: OS version, patches installed, time since last checkin, last user logged in, user most frequently logged in, plus details about hardware: RAM, disk size, serial number, etc. I m looking for a tool that can do this on Linux as well. Of course, we could roll our own; all the above is readily cleaned from standard commands plus examining /proc and /sys. But before we roll our own, we want to be sure there s nothing else out there worth using. Down the road, we might also like it to grow in another direction: centralized configuration, storing links between machines and people in a database, and generating DNS, DHCP, etc. configurations out of it. Whether that would be the same tool as this is another question. There are, of course, expensive and overkilling well-known commercial packages to do this. We re after something simple that gets the job done. Any ideas?

MJ Ray: Talks: RMS Kosovo, MJR Plymouth

RMS will be visiting Kosovo today to speak at the national library. There is no entrance fee, however the number of seats is limited to 250, so please register by writing to register@flossk.org. Less illustriously, MJR will be visiting Plymouth to run a social media surgery as part of Doing business the co-operative way on Friday 25 June (tickets 25 or 75 email now). I ve a few good ideas for what to cover if people are interested but don t have specific questions (and I ll post a few of them here later), but what tip would you give to a new social media user?

2 June 2010

Christoph Egger: [FAIL] Security

I'm all for security and really like encryption (my Notebook's harddrive is encrypted, I've recently got a GPG Smartcard, ...) but sometimes you see big failes where security is atemted but doesn't actually secure anything but only hinders the legitimate user. Today one of these candidates ate way to much of my time again. I'm currently getting more and more used to GNU Emacs and currently experimenting with emacs-jabber. Therefore copying my jabber accounts over from psi. As with these passwords you never type in I couldn't remember some of my jabber passwords -- no problem psi has to store them so it should be easy to get them, right? Well actually not. The configuration file (XML) had a password entry but all that was in it was just obviously hex-encoded numbers. These numbers turned out to be be 16bit packages of characters that are XOR-ed against the JID So now you have to read them in in junks of 16bit, XOR them against the JID and get the password. Time to recapitulate what this security helped. I've written a hacky 10 lines C Program that can reliably retrieve passwords from any config file I might come across. Seems you can do the same in 2 lines of perl. Ergo no security at all was added. Next question: What did it cost? Needed an hour or so of researching the encryption and trial&error out the right program fragment. For nothing gained at all. Fail.

1 June 2010

Russell Coker: Can SE Linux Implement Traditional Unix Users and Groups?

I was asked by email whether SE Linux could implement traditional Unix users and groups. The Strictly Literal Answer to that Question The core of the SE Linux access control is the domain-type model where every process has a domain and every object that a process can access (including other processes) has a type. Domains are not strongly differentiated from types. It would be possible to create a SE Linux policy that created a domain for every combination of UID and GID that is valid for a user shell given that such combinations are chosen by the sysadmin who could limit them to some degree. There are about 2^32 possible UIDs and about 2^32 possible GIDs, as every domain is listed in kernel memory we obviously can t have 2^64 domains, but we could have enough to map a typical system that s in use. Of course the possible combinations of supplemental groups probably makes this impossible for even relatively small systems, but we can use a simpler model that doesn t emulate supplemental groups. For files there are more possible combinations because anyone who is a member of a group can create a SETGID directory and let other users create files in it. But in a typical system the number of groups is not much greater than the number of users the maximum number of groups is typically the number of users plus about 60. So if we had 100 users then the number of combinations of UID and GID would be something like 100*(100+60)=16,000 it should be possible to have that many domains in a SE Linux policy (but not desirable). Then all that would be needed is rules specifying that each domain (which is based on a combination of UID and GID) can have certain types of access to certain other types based on them having either the same UID or the same GID. Such a policy would be large, it would waste a lot of kernel memory, it would need to be regenerated whenever a user is added, and it s generally something you don t want to use. No-one has considered implementing such a policy, I merely describe it to illustrate why certain configuration options are not desirable. The rest of this post is about realistic things that you can do with SE Linux policy and how it will be implemented in Debian/Squeeze. My previous post titled Is SE Linux Unixish? addresses this issue at a more conceptual level and also considers MAC vs DAC [1]. The History of mapping Unix Accounts to SE Linux Access Control In the early releases of SE Linux (long before it was included in Fedora) every user who could login to a system needed to have their user-name compiled into the policy. The policy specified which roles the user could access, the roles specified which domains could be accessed, and therefore what the user could do. The identity of files on disk was used for two purposes, one was logging (you could see who created a file) and the other was a permission check for the SE Linux patched version of Vixie cron which would not execute any command on behalf of a user unless the identity on the crontab file in the cron spool matched the identity used to run it this is an analogy of the checks that Vixie cron makes on the Unix UID of the crontab file (some other cron daemons do fewer checks). Having to recompile policy source every time you added a user was annoying. So a later development was to allow arbitrary mappings between Unix account names and SE Linux Identities (which included a default identity) and another later development was to have a utility program semanage to map particular Unix account and group names to SE Linux identities. This was all done years ago. Fedora Core 5 which was released in 2006 had the modular policy which included these features (apart from mapping Unix groups to SE Linux identities which was more recent). Fedora Core 5 also introduced MCS which was comprised of a set of categories that a security context may have. The sysadmin would configure the set of categories that each account would have. A recent development has been a concept named UBAC (User Based Access Control) which basically means that a process running directly on behalf of a regular user (IE with a SE Linux identity that s not system_u) can only access files that have an identity of system_u or which have the same identity as the process. This means that you can only access your own files or system files not files of other users which may have inappropriate Unix permissions. So for example if a user with a SE Linux identity of john gives their home directory the Unix permission mode of 0777 then a user with a SE Linux identity of jane can t access their files. Of course this means that if you have a group of people working together on a project then they probably need to all have the same Identity and in practice you would probably end up with everyone having the same identity. I ve given up on the idea of using UBAC in Debian. The Current Plan for Users and SE Linux Access Control in Debian My plan is to have things work in Squeeze in much the same way as in Lenny. You have a SE Linux identity assigned to a login session and everything related to it (including cron jobs) based on the Unix account name or possibly the Unix group name (if there are login entries for both the user-name and the group-name then the user-name entry has precedence). The mapping between Unix accounts and SE Linux identities is configured by the sysadmin and SE Linux identities don t matter much for the Targeted configuration (which is what most people use). The identity determines which roles may be used and also has a limit on the MCS categories. The MCS categories are also specified in the login configuration which has to be a sub-set of the categories used by the identity record. So for example the following is the output of a couple of commands run on a Debian/Unstable system. They show that the test Unix account is assigned a SE Linux identity of staff_u and an MCS range of s0-s0:c1 (this means it creates files by default at level s0 and can also write to other files at that level, but can also have read/write access to files at the level s0:c1 ). The staff_u identity (as shown in the output of semanage user -l can be used with all categories in the set s0:c0.c1023 where the dot means the set of categories from c0 to c1023 inclusive) but in the case of the test user only one category will be used. The test group however (as expressed with %test ) is given the identity user_u and is not permitted to use any categories.
# semanage login -l
Login Name    SELinux User    MLS/MCS Range           
%test         user_u          s0                     
__default__   unconfined_u    s0-s0:c0.c1023         
root          unconfined_u    s0-s0:c0.c1023         
system_u      system_u        s0-s0:c0.c1023         
test          staff_u         s0-s0:c1               
# semanage user -l
             Labeling   MLS/       MLS/                         
SELinux User Prefix     MCS Level  MCS Range         SELinux Roles
root         sysadm     s0         s0-s0:c0.c1023    staff_r sysadm_r system_r
staff_u      staff      s0         s0-s0:c0.c1023    staff_r sysadm_r
sysadm_u     sysadm     s0         s0-s0:c0.c1023    sysadm_r
system_u     user       s0         s0-s0:c0.c1023    system_r
unconfined_u unconfined s0         s0-s0:c0.c1023    system_r unconfined_r
user_u       user       s0         s0                user_r
I hope to get the policy written to support multiple user roles in time for the release of Squeeze. If I don t make it then I will put such a policy on my own web site and try to get it included in an update. The policy currently basically works for a Targeted configuration where the users are not confined (apart from MCS). How MMCS Basically Works The vast majority of SE Linux users run with the MCS policy rather than the MLS policy. For Debian I have written a modified version of MCS that I call MMCS. MMCS is mandatory (you can t relabel files to escape it) and it prevents write-down. If a process has the range s0-s0:c1,c3 then it has full access to files labelled as s0, s0:c1, s0:c3, and s0:c1,c3 and any files it creates will be labeled as s0. If a process has the range s0:c1-s0:c1,c3 then it has read-only access to files labelled as s0 and s0:c3 and read-write access to files labelled as s0:c1 and s0:c1,c3. This means that any secret data it accesses that was labelled with category c1 can t be leaked down to a file that is not labelled with that category. Now MCS currently has no network access controls, so there s nothing stopping a user from using scp or other network utilities to transfer files. But that s the way with most usable systems. I don t think that this is necessarily a problem, the almost total lack of network access controls in a traditional Unix model doesn t seem to concern most people. Now to REALLY Answer that Question SE Linux is a Mandatory Access Control (MAC) system, this makes it inherently different to a Discretionary Access Control (DAC) system such as traditional Unix access controls. Unix permissions are based on each file having a UID, a GID, and a set of permissions and each process having a UID, a GID, and a set of supplementary GIDs. If a user runs a setuid or setgid program then the process will have extra privileges. It also has a lot of stuff that most people aren t aware of such as real vs effective UIDs, the tag bit, setgid directories, and lots more including some quite arbitrary things like making ports <1024 special. SE Linux is based on every object (process, file, socket, etc) having a single security label which includes an identity, a role, a type, and a sensitivity label (MCS categories or an MLS range). There is no support for an object to have more than one label. The SE Linux equivalent to setuid/setgid files is a label for a file which triggers a domain transition when it s executed. This differs from setuid files in that the domain transition is complete (the old privileges can t be restored) and the transition is generally not to a strict super-set of the access (usually a different sub-set of possible access and sometimes to lesser access). These differences are quite fundamental. So really SE Linux can t implement traditional Unix access control. What SE Linux is designed to do is to provide a second layer of defense and to also provide access controls that have different aims than that of Unix permissions such as being mandatory and implementing features such as MLS.

MJ Ray: Hello Oslo, This is Emacs Calling

The Eurovision Song Contest has been and gone with a better-than-usual winner and much of the snarkiness was on microblog sites like identi.ca this year. Search for #ESC, #eurovision or incomprehensibly-to-me #eurovison (no second i is it spelt like that in some language, or can thousands not spell?). Anyway, enough of the title, on with the message I ve been irritated by my linphone SIP internet phone (Skype without lock-in, with standards) taking up my screen space for a while. I ve been meaning to upgrade to one of the recent versions that includes the linphonecsh shell and write some client that only displays windows when it s needed, but I ve been writing programs for work so much that I ve not even found time to write blog posts recently, much less fun code, so that s not happened. So I m quite happy to find linphone.el for my GNU Emacs text editor and find it mostly works. I changed one line at the end to (define-key global-map [menu-bar tools linphone] '("Linphone" . linphone)) to put it on the tools menu, but other than that, it just worked. Well, the Mute doesn t, but that s probably my phone headset being a different ALSA device. I don t care: I use the hardware mute button. In previous times, I would have added the link to the venerable EmacsWiki too, but it has the evil bad wrong Google reCaptcha on the edit page to stop disabled users, so screw it. Google s reCaptcha seems to be spreading again, obstructing more people when accessing more websites. Is there a reason for that? The re in reCaptcha stands for replace with real anti-spam, please! Anyway, now I m controlling phone calls from Emacs. Whatever next? Voice recognition, M-x doctor and a speech synth?

31 May 2010

Christian Perrier: Samba 3.5.3 in experimental - 3.4.8 in testing

During the week-end, samba 3.5.3 hit the archive. It is still in experimental (and will remain there as Squeeze will be released with 3.4.* as finally decided during SambaXP). For lenny users, I also build backported packages. They are not on backports.org as the policy for these "as official as possible" backports is to have the same version than testing. So, they are provided "as is", for i386 and amd64, on the pkg-samba team's unofficial repository. The Release file on that repository is signed by my GPG key. Please note that these backports also need talloc from backports.org so you need to add entries for backports.org in your sources.list files to be able to use them. Since I last blogged about samba packages, the 3.4.8 version was also released. It entered testing without any problem and I also uploaded a backported version to backports.org. Finally, on May 5th, new versions of the 3.2.5 packages of lenny were updated to 2:3.2.5-4lenny11, currently in stable-proposed-updates. So, they will be part of the next point release of lenny. They fix two important bugs (#575951 for interdomain trust with w2k8 servers, and #538819 for memory leaks in some situations).

Stefano Zacchiroli: document your team - redux

wiki.d.o/Teams to the rescue About 3 years ago (too bad I've actually missed the 3-year anniversary by a few days!), Raphael set up http://wiki.debian.org/Teams. In my recent encounters and contacts with people interested in contributing to Debian, I've found that page to be of invaluable help. In particular, people find it very useful in understanding the macro-structures of Debian and in understanding where they can start to contribute. Approaching a team is most likely "less scary" than contacting a larger forum, and that page offers a good service of team indexing. Of course, the usefulness of wiki.d.o/Teams is directly proportional to how much complete it is and to how much individual team pages are current. So, in case you didn't know about the page or that you just remembered that years ago you set up a page there and then forgot about it, this is probably a good moment to add or update your team information there. To eat my own dog food, I've recently set up Teams/DPL, which is probably not terribly useful, but it contains interesting stuff such as where the daily activity bits are stored. (Yes: I'm just a one-person-team, but given the index is useful to document Debian "parts" in general, I believe we should allow for a slightly semantic abuse of the name.) Incidentally, that shows another way in which wiki team pages can come to the rescue: improve the documentation of our processes. For instance, we are grown accustomed to the fact that d-d-a is an authoritative information source. Well, of course it is, but when you need to find out a 4-year old d-d-a post of some core team, since it is the only documentation of some still current (is it?) procedure, let's be fair and acknowledge that a mailing list archive it's not that handy. So, summarizing relevant d-d-a posts in team pages (with references to the originals) is another way, accessible to everybody, to help in keeping docs current. I've been doing that recently for a handful of teams, and people seem to appreciate (of course it's your responsibility to check that they do appreciate).

Gunnar Wolf: Update: A correspondingly pleasant dinner

Update: A correspondingly pleasant dinner
And just to round off my last post (and of course, following a completely non-technical thematic), what is the logical consequence of feeling introspective and blogging just before preparing dinner? Of course Dinner gets prepared with sharing it in mind. The dish ends up even looking as if meant to be served! So this was it: One of my simplest and still favorite dishes: Tostadas. But, yes, these are heterodox tostadas, as they share the basic tostadiness (a hard, roasted and cracky toasted corn tortilla with a soft leguminous layer to give it some consistency, and with... stuff on top. Yes, stuff is sometimes too generic, but that's the beauty of it). What did I come up with? In the strict order with which they were approached at feeding time, and described bottom to top:
  1. Closest to traditional, top left: Mashed beans, bits of panela cheese with little bits of chipotle and soybean-based vegetarian chorizo
  2. Something I have only seen in Guatemala, and which I intend to take a better look at next time I'm there (top right): Mashed frijoles, , cochinita pibil (canned, shame on me, and frankly lacking in taste), grated beet (betabel/remolacha/betarraga/whatnot), grated cabbage
  3. The heterodoxiest of them all, and the idea that led me to the others (center): Mashed lima/fava beans (according to Google habas), cochinita pibil, nopales, and a hint of habanero sauce.
Yum! [update] On a completely unrelated notice, but not worth opening a third post in a row... Some minutes after I published the earlier post, I got a visit to http://gwolf.org/blog/pleasant-perception-changes?utm_source=twitterfeed&utm_medium=twitter. Now, it beats me: I don't use twitter. I don't even care about twitter. And were my message so deep somebody just twitted (twat it?) right away, I still find the time lapse too short. Who's auto-twitting me? Maybe a planet or other such aggregator?

30 May 2010

Kees Cook: Linux Security Summit 2010

The Call For Participation is open for the 2010 Linux Security Summit, being held just before this year s LinuxCon. If you re interested in helping make Linux more secure, you ve got ideas to present, want to have your opinion heard, or generally just want to hang out, please join us and/or suggest a topic for discussion (CFP ends June 4th, so please hurry). I m hoping to get a chance to discuss what I m calling the popular kernel hardening patches which appear in a lot of distros yet remain missing from the upstream Linux kernel.

Next.

Previous.