Search Results: "solomon"

27 July 2020

Russ Allbery: Summer haul

I'm buying rather too many books at the moment and not reading enough of them (in part because I got back into Minecraft and in part because I got a bit stuck on a few difficult books). I think I've managed to get myself unstuck again, though, and have started catching up on reviews. 2020. It's kind of a lot. And I'm not even that heavily affected. Katherine Addison The Angel of the Crows (sff)
Marie Brennan A Natural History of Dragons (sff)
Kacen Callender Queen of the Conquered (sff)
Jo Clayton Diadem from the Stars (sff)
Jo Clayton Lamarchos (sff)
Jo Clayton Irsud (sff)
Clifford D. Conner The Tragedy of American Science (nonfiction)
Kate Elliott Unconquerable Sun (sff)
Rory Fanning & Craig Hodges Long Shot (nonfiction)
Michael Harrington Socialism: Past & Future (nonfiction)
Nalo Hopkinson Brown Girl in the Ring (sff)
Kameron Hurley The Stars Are Legion (sff)
N.K. Jemisin Emergency Skin (sff)
T. Kingfisher A Wizard's Guide to Defensive Baking (sff)
T. Kingfisher Nine Goblins (sff)
Michael Lewis The Fifth Risk (nonfiction)
Paul McAuley War of the Maps (sff)
Gretchen McCulloch Because Internet (nonfiction)
Hayao Miyazaki Nausica of the Valley of the Wind (graphic novel)
Annalee Newitz The Future of Another Timeline (sff)
Nick Pettigrew Anti-Social (nonfiction)
Rivers Solomon, et al. The Deep (sff)
Jo Walton Or What You Will (sff)
Erik Olin Wright Stardust to Stardust (nonfiction) Of these, I've already read and reviewed The Fifth Risk (an excellent book).

2 January 2017

Shirish Agarwal: India Tourism, E-Visa and Hong Kong

A Safe and Happy New Year to all. While Debconf India is still a pipe-dream as of now, did see that India has been gradually doing it easier for tourists and casual business visitors to come visit India. This I take as very positive development for India itself. The 1st condition is itself good for anybody visiting India
Eligibility International Travellers whose sole objective of visiting India is recreation , sight-seeing , casual visit to meet friends or relatives, short duration medical treatment or casual business visit.
https://indianvisaonline.gov.in/visa/tvoa.html That this facility is being given to 130 odd countries is better still
Albania, Andorra, Anguilla, Antigua & Barbuda, Argentina, Armenia, Aruba, Australia, Austria, Bahamas, Barbados, Belgium, Belize, Bolivia, Bosnia & Herzegovina, Botswana, Brazil, Brunei, Bulgaria, Cambodia, Canada, Cape Verde, Cayman Island, Chile, China, China- SAR Hong-Kong, China- SAR Macau, Colombia, Comoros, Cook Islands, Costa Rica, Cote d lvoire, Croatia, Cuba, Czech Republic, Denmark, Djibouti, Dominica, Dominican Republic, East Timor, Ecuador, El Salvador, Eritrea, Estonia, Fiji, Finland, France, Gabon, Gambia, Georgia, Germany, Ghana, Greece, Grenada, Guatemala, Guinea, Guyana, Haiti, Honduras, Hungary, Iceland, Indonesia, Ireland, Israel, Jamaica, Japan, Jordan, Kenya, Kiribati, Laos, Latvia, Lesotho, Liberia, Liechtenstein, Lithuania, Luxembourg, Madagascar, Malawi, Malaysia, Malta, Marshall Islands, Mauritius, Mexico, Micronesia, Moldova, Monaco, Mongolia, Montenegro, Montserrat, Mozambique, Myanmar, Namibia, Nauru, Netherlands, New Zealand, Nicaragua, Niue Island, Norway, Oman, Palau, Palestine, Panama, Papua New Guinea, Paraguay, Peru, Philippines, Poland, Portugal, Republic of Korea, Republic of Macedonia, Romania, Russia, Saint Christopher and Nevis, Saint Lucia, Saint Vincent & the Grenadines, Samoa, San Marino, Senegal, Serbia, Seychelles, Singapore, Slovakia, Slovenia, Solomon Islands, South Africa, Spain, Sri Lanka, Suriname, Swaziland, Sweden, Switzerland, Taiwan, Tajikistan, Tanzania, Thailand, Tonga, Trinidad & Tobago, Turks & Caicos Island, Tuvalu, UAE, Ukraine, United Kingdom, Uruguay, USA, Vanuatu, Vatican City-Holy See, Venezuela, Vietnam, Zambia and Zimbabwe.
This should make it somewhat easier for any Indian organizer as well as any participants from any of the member countries shared. There is possibility that this list would even get longer, provided we are able to scale our airports and all and any necessary infrastructure that would be needed for International Visitors to have a good experience. What has been particularly interesting is to know which ports of call are being used by International Visitors as well as overall growth rate
The Percentage share of Foreign Tourist Arrivals (FTAs) in India during November, 2016 among the top 15 source countries was highest from USA (15.53%) followed by UK (11.21%), Bangladesh (10.72%), Canada (4.66%), Russian Fed (4.53%), Australia (4.04%), Malaysia (3.65%), Germany (3.53%), China (3.14%), France (2.88%), Sri Lanka (2.49%), Japan (2.49%), Singapore (2.16%), Nepal (1.46%) and Thailand (1.37%).
And port of call
The Percentage share of Foreign Tourist Arrivals (FTAs) in India during November 2016 among the top 15 ports was highest at Delhi Airport (32.71%) followed by Mumbai Airport (18.51%), Chennai Airport (6.83%), Bengaluru Airport (5.89%), Haridaspur Land check post (5.87%), Goa Airport (5.63%), Kolkata Airport (3.90%), Cochin Airport (3.29%), Hyderabad Airport (3.14%), Ahmadabad Airport (2.76%), Trivandrum Airport (1.54%), Trichy Airport (1.53%), Gede Rail (1.16%), Amritsar Airport (1.15%), and Ghojadanga land check post (0.82%) .
The Ghojadanga land check post seems to be between West Bengal, India and Bangladesh. Gede Railway Station is also in West Bengal as well. So all and any overlanders could take any of those ways.Even Hardispur Land Check post comes in the Bengal-Bangladesh border only. In the airports, Delhi Airport seems to be attracting lot more business than the Mumbai Airport. Part of the reason I *think* is the direct link of Delhi Airport to NDLS via the Delhi Airport Express Line . The same when it will happen in Mumbai should be a game-changer for city too. Now if you are wondering why I have been suddenly talking about visas and airports in India, it came because Hong Kong is going to Withdraw Visa Free Entry Facility For Indians. Although, as rightly pointed out in the article doesn t make sense from economic POV and seems to be somewhat politically motivated. Not that I or anybody else can do anything about that. Seeing that, I thought it was a good opportunity to see how good/Bad our Government is and it seems to be on the right path. Although the hawks (Intelligence and Counter-Terrorist Agencies) will probably become a bit more paranoid , their work becomes tougher.
Filed under: Miscellenous Tagged: #Airport Metro Line 3, #CSIA, #Incredible India, #India, #International Tourism

19 December 2014

Dirk Eddelbuettel: Rocker is now the official R image for Docker

big deal Something happened a little while ago which we did not have time to commensurate properly. Our Rocker image for R is now the official R image for Docker itself. So getting R (via Docker) is now as simple as saying docker pull r-base. This particular container is essentially just the standard r-base Debian package for R (which is one of a few I maintain there) plus a mininal set of extras. This r-base forms the basis of our other containers as e.g. the rather popular r-studio container wrapping the excellent RStudio Server. A lot of work went into this. Carl and I also got a tremendous amount of help from the good folks at Docker. Details are as always at the Rocker repo at GitHub. Docker itself continues to make great strides, and it has been great fun help to help along. With this post I achieved another goal: blog about Docker with an image not containing shipping containers. Just kidding.

This post by Dirk Eddelbuettel originated on his Thinking inside the box blog. Please report excessive re-aggregation in third-party for-profit settings.

7 December 2014

Miriam Ruiz: Falling Trees, by Robert Fulghum

In the Solomon Islands in the south Pacific some villagers practice a unique form of logging. If a tree is too large to be felled with an ax, the natives cut it down by yelling at it. (Can t lay my hands on the article, but I swear I read it.) Woodsmen with special powers creep up on a tree just at dawn and suddenly scream at it at the top of their lungs. They continue this for thirty days. The tree dies and falls over. The theory is that the hollering kills the spirit of the tree. According to the villagers, it always works. Ah, those poor nave innocents. Such quaintly charming habits of the jungle. Screaming at trees, indeed. How primitive. Too bad thay don t have the advantages of modern technology and the scientific mind. Me? I yell at my wife. And yell at the telephone and the lawn mower. And yell at the TV and the newspaper and my children. I ve been known to shake my fist and yell at the sky at times. Man next door yells at his car a lot. And this summer I heard him yell at a stepladder for most of an afternoon. We modern, urban, educated folks yell at traffic and umpires and bills and banks and machines especially machines. Machines and relatives get most of the yelling. Don t know what good it does. Machines and things just sit there. Even kicking doesn t always help. As for people, well, the Solomon Islanders may have a point. Yelling at living things does tend to kill the spirit in them. Sticks and stones may break our bones, but words will break our hearts . by Robert Fulghum (All I Really Need To Know I Learned In Kindergarten)

23 October 2014

Dirk Eddelbuettel: Introducing Rocker: Docker for R

You only know two things about Docker. First, it uses Linux
containers. Second, the Internet won't shut up about it. -- attributed to Solomon Hykes, Docker CEO

So what is Docker? Docker is a relatively new open source application and service, which is seeing interest across a number of areas. It uses recent Linux kernel features (containers, namespaces) to shield processes. While its use (superficially) resembles that of virtual machines, it is much more lightweight as it operates at the level of a single process (rather than an emulation of an entire OS layer). This also allows it to start almost instantly, require very little resources and hence permits an order of magnitude more deployments per host than a virtual machine. Docker offers a standard interface to creation, distribution and deployment. The shipping container analogy is apt: just how shipping containers (via their standard size and "interface") allow global trade to prosper, Docker is aiming for nothing less for deployment. A Dockerfile provides a concise, extensible, and executable description of the computational environment. Docker software then builds a Docker image from the Dockerfile. Docker images are analogous to virtual machine images, but smaller and built in discrete, extensible and reuseable layers. Images can be distributed and run on any machine that has Docker software installed---including Windows, OS X and of course Linux. Running instances are called Docker containers. A single machine can run hundreds of such containers, including multiple containers running the same image. There are many good tutorials and introductory materials on Docker on the web. The official online tutorial is a good place to start; this post can not go into more detail in order to remain short and introductory.

So what is Rocker? rocker logo At its core, Rocker is a project for running R using Docker containers. We provide a collection of Dockerfiles and pre-built Docker images that can be used and extended for many purposes. Rocker is the the name of our GitHub repository contained with the Rocker-Org GitHub organization. Rocker is also the name the account under which the automated builds at Docker provide containers ready for download.

Current Rocker Status

Core Rocker Containers The Rocker project develops the following containers in the core Rocker repository
  • r-base provides a base R container to build from
  • r-devel provides the basic R container, as well as a complete R-devel build based on current SVN sources of R
  • rstudio provides the base R container as well an RStudio Server instance
We have settled on these three core images after earlier work in repositories such as docker-debian-r and docker-ubuntu-r.

Rocker Use Case Containers Within the Rocker-org organization on GitHub, we are also working on
  • Hadleyverse which extends the rstudio container with a number of Hadley packages
  • rOpenSci which extends hadleyverse with a number of rOpenSci packages
  • r-devel-san provides an R-devel build for "Sanitizer" run-time diagnostics via a properly instrumented version of R-devel via a recent compiler build
  • rocker-versioned aims to provided containers with 'versioned' previous R releases and matching packages
Other repositories will probably be added as new needs and opportunities are identified.

Deprecation The Rocker effort supersedes and replaces earlier work by Dirk (in the docker-debian-r and docker-ubuntu-r GitHub repositories) and Carl. Please use the Rocker GitHub repo and Rocker Containers from Docker.com going forward.

Next Steps We intend to follow-up with more posts detailing usage of both the source Dockerfiles and binary containers on different platforms. Rocker containers are fully functional. We invite you to take them for a spin. Bug reports, comments, and suggestions are welcome; we suggest you use the GitHub issue tracker.

Acknowledgments We are very appreciative of all comments received by early adopters and testers. We also would like to thank RStudio for allowing us the redistribution of their RStudio Server binary. Published concurrently at rOpenSci blog and Dirk's blog.

Authors Dirk Eddelbuettel and Carl Boettiger

This post by Dirk Eddelbuettel originated on his Thinking inside the box blog. Please report excessive re-aggregation in third-party for-profit settings.

31 March 2014

Russell Coker: Links March 2014

Typing Animal wrote an interesting article about the dangers of stainless steel in a medical environment [1]. Apparently silver and copper are best due to the oligodynamic effect. Instead of stainless steel drinking bottles they should sell silver plated drinking bottles for kids, I m sure that lots of parents would pay extra for that. Mark Kendall gave an interesting TED talk about a replacement for the hypodermic syringe in vaccinations [2]. His invention can reduce the cost of immunisation while increasing the effectiveness and avoiding problems with people who have a needle phobia. The TED blog has an interesting interview with Will Potter about the use of the war on terror to silence journalists and the invention of the term eco terrorism for non-violent people who are politically active [3]. The TED blog has an interesting article by Kate Torgovnick May about designing products for sustainability [4]. It links to an insightful TED talk by Leyla Acaroglu about some of the complex issues related to sustainability [5]. Manoush Zomorodi wrote an informative article about How one college went from 10% female computer-science majors to 40% [6]. Slate has an interesting article by Jamelle Bouie showing the way that support for capital punishment in the US is linked to racism [7]. The Southern California Public Radio blog has an interesting article by Josie Huang about Suey Park and her success in using twitter to oppose racism [8]. Andrew Solomon wrote an insightful interview with the father of Adam Lanza for the New Yorker [9]. Waleed Aly wrote an insightful article about George Brandis attempt to change the Racial Discrimination Act specifically to allow Andrew Bolt to be racist [10]. He describes it as the whitest piece of proposed legislation I ve encountered which is significant in a country with as much racism as Australia. Really we need stronger laws against racism, there should be no right to be bigoted. A German Court has ruled that non commercial licenses don t permit non-commercial organisations to re-publish material [11]. This seems bogus to me, I d be happy to have my non-commercial licensed work published by a non-commercial publishing organisation just as long as they don t run adverts on the page. Professors Woolley and Malone wrote an interesting article about their research into group performance, apparently having more women in a group improves the collective intelligence of a group, but having smarter men in the group doesn t [12]. Susie Hill wrote an article about the SPARX computer game that is designed to treat adolescent depression [13]. They are working on a rainbow edition for GLBT kids and a version for Maoris. Unfortunately their web site is down right now and the version at archive.org says that it s currently only available to participants in a clinical trial. Tim Chevalier wrote an insightful article explaining why people who campaign against equality shouldn t be given senior positions in corporations [14]. Zeynep Tufekci wrote an insightful article about how French High Theory and Dr. Seuss can help explain gender problems in geek communities [15]. Hannah Levintova wrote an informative article for Mother Jones about how the US based hate group the World Congress of Families incites homophobic violence in Russia [16]. Josh Sanburn wrote an article for Time about people in the Deep South who claim to be Christian giving away guns to encourage people to attend church [17]. This is the same part of the world where people who claimed to be Christian used their religion as an excuse for supporting slavery. I m quitting bourbon, too much evil comes from that part of the world and I m not buying anything that comes from there.

7 March 2014

Russell Coker: Depression

Andrew Solomon gave an interesting TED talk about depression [1]. I ve had problems with depression at various times through my life, about 18 months ago I recognised it as depression after reading a list of depression symptoms on the Beyond Blue site. I think that they have changed their site recently they now have an interactive checklist for depression on their web site [2] (or maybe I just missed the interactive part last time). There is a strong correlation between Autism and depression, this is noted both in research and described on the web, Elspeth s article on Bluehackers is a good description of this [3]. Her experiences differ from mine in some ways, but it s within the common range of experiences you see described on Autism forums etc. Depression is getting more widely known, organisations such as Beyond Blue and Bluehackers are doing good work in spreading information to people who might be depressed and people who know them. The general advice is to see a psychologist, which is good advice for average people. Alexithymia and Choice of Psychologists One problem with such advice is that it doesn t apply so well to people with alexithymia (read the Wikipedia page) [4], that means most people on the Autism Spectrum. The Wikipedia page says psychosomatic illness or substance abuse is frequently exacerbated should these individuals enter psychotherapy . Based on people I know and accounts I ve read on the Internet I expect that anyone on the Autism Spectrum who sees a psychologist that doesn t specialise in Autism (which means most psychologists) will get a result that s about the opposite of what one might desire. In theory a psychologist could recognise someone as being possibly on the Autism Spectrum and refer them to an expert for assessment, but I haven t heard of that happening to an adult. I think that most people who have some degree of alexithymia will avoid psychologists, without ever seeing one you can just know that it s going to be unpleasant. So while you wouldn t want someone who has alexithymia to visit a random psychologist in practice that shouldn t happen too often as such people will be more likely to reject any advice about seeing a psychologist. My page of Autism self-diagnosis tests has a link to an Alexithymia test [5]. If you get a high score on that test (or if taking the test seems too unpleasant) then it s best to see a psychologist who specialises in Autism. Such psychologists are usually happy to work for people who don t quite meet the Autism diagnostic criteria, but they may strongly recommend an Autism assessment so that they can determine the best strategies for treatment. In terms of addressing such problems it seems that the best thing we can do is try and reduce the stigma associated with Autism. The vast majority of people on the Autism Spectrum have little in common with Rain Man. Many of the so-called Autism advocacy organisations make things worse by implying that everyone who is Autistic is unable to live an independent life which helps them in fundraising but doesn t help us.

25 January 2014

Russell Coker: Links January 2014

Fast Coexist has an interesting article about the art that Simon Beck creates by walking in snow [1]. If you are an artist you can create art in any way, even by walking in patterns in the snow. Russ Altman gave an interesting TED talk about using DNA testing before prescribing drugs [2]. I was surprised by the amount of variation in effects of codeine based on genetics, presumably many other drugs have a similar range. Helen Epstein wrote an interesting article about Dr. Sara Josephine Baker who revolutionised child care and saved the lives of a huge number of children [3]. Her tenacity is inspiring. Also it s interesting to note that the US Republican party was awful even before the Southern Strategy . The part about some doctors opposing child care because it s the will of God for children to die and keep them in employment is chilling. Jonathan Weiler wrote an insightful article about the problems with American journalism in defending the government [4]. He criticises the media for paying more attention to policing decorum than to content. Tobias Buckell wrote an interesting post about the so-called socialised health-care in the US [5]. He suggests that Ronald Reagan socialised health-care by preventing hospitals from dumping dying people on the street. I guess if doing nothing for people until they have a medical emergency counts as socialised health-care then the US has it. Kelvin Thomson MP made some insightful comments about climate change, the recent heat-wave in Australia, and renewable energy [6]. Iwan Baan gave an interesting TED talk about ways that people have built cheap homes in unexpected places [7], lots of good pictures. Racialicious has an interesting article by Arturo R. Garc a about research into the effects of concussion and the way the NFL in the US tried to prevent Dr. Bennet Omalu publicising the results of his research [8]. Stani (Jan Schmidt) wrote an interesting post about how they won a competition to design a commemerative Dutch 5 Euro coin [9]. The coin design is really good (a candidate for the geekiest coin ever), I want one! Seriously if anyone knows how to get one at a reasonable price (IE close to face value for circulated or not unreasonably expensive for uncirculated) then please let me know. When writing about Edward Snowden, Nathan says Imagine how great a country would be if if it were governed entirely by people who Dick Cheney would call Traitor [10]. That s so right, that might make the US a country I d be prepared to live in. Andrew Solomon gave an interesting TED talk Love No Matter What about raising different children [11]. Aditi Shankardass gave an interesting TED talk about using an ECG to analyse people diagnosed wit severe Autism and other developmental disorders [12]. Apparently some severe cases of Autism have a root cause that can be treated with anti-seizure medication. George Monbiot wrote an insightful article about the way that Bono and Bob Geldoff promote G8 government intervention in Africa and steal air-time that might be given to allow Africans to represent themselves in public debates [13]. Daniel Pocock wrote an informative article about racism in Australian politics and how it is bad for job-seekers and the economy (in addition to being horribly wrong) [14]. Aeon Magazine has an interesting article by Anne Buchanan about the difference between scientists and farmers [15]. She has some interesting points about the way that the lack of general knowledge impacts research, but misses the point that in most fields of study there is a huge problem of people not knowing about recent developments in their own field. I don t think it s a pipe dream to be well educated in humanities and science, but I guess that depends on the definition of well educated . Brian Cox gave an interesting TED talk titled Why We Need the Explorers about the benefits of scientific research [16]. Yupu Zhang, Abhishek Rajimwale, Andrea C. Arpaci-Dusseau, and Remzi H. Arpaci-Dusseau from the University of Wisconsin-Madison wrote an interesting paper about ZFS corruption in the face of disk and memory errors [17]. One thing to note is that turning off atime can reduce the probability of a memory error leading to corrupt data being written to disk, run zfs set atime=off tank to fix this. The comedian Solomon Georgio celebrated Martin Luther King day by tweeting I love you to racists [18]. It s an interesting approach and appears to have worked well.

15 March 2011

Colin Watson: Wubi bug 693671

I spent most of last week working on Ubuntu bug 693671 ("wubi install will not boot - phase 2 stops with: Try (hd0,0): NTFS5"), which was quite a challenge to debug since it involved digging into parts of the Wubi boot process I'd never really touched before. Since I don't think much of this is very well-documented, I'd like to spend a bit of time explaining what was involved, in the hope that it will help other developers in the future. Wubi is a system for installing Ubuntu into a file in a Windows filesystem, so that it doesn't require separate partitions and can be uninstalled like any other Windows application. The purpose of this is to make it easy for Windows users to try out Ubuntu without the need to worry about repartitioning, before they commit to a full installation. Wubi started out as an external project, and initially patched the installer on the fly to do all the rather unconventional things it needed to do; we integrated it into Ubuntu 8.04 LTS, which involved turning these patches into proper installer facilities that could be accessed using preseeding, so that Wubi only needs to handle the Windows user interface and other Windows-specific tasks. Anyone familiar with a GNU/Linux system's boot process will immediately see that this isn't as simple as it sounds. Of course, ntfs-3g is a pretty solid piece of software so we can handle the Windows filesystem without too much trouble, and loopback mounts are well-understood so we can just have the initramfs loop-mount the root filesystem. Where are you going to get the kernel and initramfs from, though? Well, we used to copy them out to the NTFS filesystem so that GRUB could read them, but this was overly complicated and error-prone. When we switched to GRUB 2, we could instead use its built-in loopback facilities, and we were able to simplify this. So all was more or less well, except for the elephant in the room. How are you going to load GRUB? In a Wubi installation, NTLDR (or BOOTMGR in Windows Vista and newer) still owns the boot process. Ubuntu is added as a boot menu option using BCDEdit. You might then think that you can just have the Windows boot loader chain-load GRUB. Unfortunately, NTLDR only loads 16 sectors - 8192 bytes - from disk. GRUB won't fit in that: the smallest core.img you can generate at the moment is over 18 kilobytes. Thus, you need something that is small enough to be loaded by NTLDR, but that is intelligent enough to understand NTFS to the point where it can find a particular file in the root directory of a filesystem, load boot loader code from it, and jump to that. The answer for this was GRUB4DOS. Most of GRUB4DOS is based on GRUB Legacy, which is not of much interest to us any more, but it includes an assembly-language program called GRLDR that supports doing this very thing for FAT, NTFS, and ext2. In Wubi, we build GRLDR as wubildr.mbr, and build a specially-configured GRUB core image as wubildr. Now, the messages shown in the bug report suggested a failure either within GRLDR or very early in GRUB. The first thing I did was to remember that GRLDR has been integrated into the grub-extras ntldr-img module suitable for use with GRUB 2, so I tried building wubildr.mbr from that; no change, but this gave me a modern baseline to work on. OK; now to try QEMU (you can use tricks like qemu -hda /dev/sda if you're very careful not to do anything that might involve writing to the host filesystem from within the guest, such as recursively booting your host OS ... [update: Tollef Fog Heen and Zygmunt Krynicki both point out that you can use the -snapshot option to make this safer]). No go; it hung somewhere in the middle of NTLDR. Still, I could at least insert debug statements, copy the built wubildr.mbr over to my test machine, and reboot for each test, although it would be slow and tedious. Couldn't I? Well, yes, I mostly could, but that 8192-byte limit came back to bite me, along with an internal 2048-byte limit that GRLDR allocates for its NTFS bootstrap code. There were only a few spare bytes. Something like this would more or less fit, to print a single mark character at various points so that I could see how far it was getting:
	pushal
	xorw	%bx, %bx	/* video page 0 */
	movw	$0x0e4d, %ax	/* print 'M' */
	int	$0x10
	popal
In a few places, if I removed some code I didn't need on my test machine (say, CHS compatibility), I could even fit in cheap and nasty code to print a single register in hex (as long as you didn't mind 'A' to 'F' actually being ':' to '?' in ASCII; and note that this is real-mode code, so the loop counter is %cx not %ecx):
	/* print %edx in dumbed-down hex */
	pushal
	xorw	%bx, %bx
	movb	$0xe, %ah
	movw	$8, %cx
1:
	roll	$4, %edx
	movb	%dl, %al
	andb	$0xf, %al
	int	$0x10
	loop	1b
	popal
After a considerable amount of work tracking down problems by bisection like this, I also observed that GRLDR's NTFS code bears quite a bit of resemblance in its logical flow to GRUB 2's NTFS module, and indeed the same person wrote much of both. Since I knew that the latter worked, I could use it to relieve my brain of trying to understand assembly code logic directly, and could compare the two to look for discrepancies. I did find a few of these, and corrected a simple one. Testing at this point suggested that the boot process was getting as far as GRUB but still wasn't printing anything. I removed some Ubuntu patches which quieten down GRUB's startup: still nothing - so I switched my attentions to grub-core/kern/i386/pc/startup.S, which contains the first code executed from GRUB's core image. Code before the first call to real_to_prot (which switches the processor into protected mode) succeeded, while code after that point failed. Even more mysteriously, code added to real_to_prot before the actual switch to protected mode failed too. Now I was clearly getting somewhere interesting, but what was going on? What I really wanted was to be able to single-step, or at least see what was at the memory location it was supposed to be jumping to. Around this point I was venting on IRC, and somebody asked if it was reproducible in QEMU. Although I'd tried that already, I went back and tried again. Ubuntu's qemu is actually built from qemu-kvm, and if I used qemu -no-kvm then it worked much better. Excellent! Now I could use GDB:
(gdb) target remote   qemu -gdb stdio -no-kvm -hda /dev/sda
This let me run until the point when NTLDR was about to hand over control, then interrupt and set a breakpoint at 0x8200 (the entry point of startup.S). This revealed that the address that should have been real_to_prot was in fact garbage. I set a breakpoint at 0x7c00 (GRLDR's entry point) and stepped all the way through to ensure it was doing the right thing. In the process it was helpful to know that GDB and QEMU don't handle real mode very well between them. Useful tricks here were: Single-stepping showed that GRLDR was loading the entirety of wubildr correctly and jumping to it. The first instruction it jumped to wasn't in startup.S, though, and then I remembered that we prefix the core image with grub-core/boot/i386/pc/lnxboot.S. Stepping through this required a clear head since it copies itself around and changes segment registers a few times. The interesting part was at real_code_2, where it copies a sector of the kernel to the target load address, and then checks a known offset to find out whether the "kernel" is in fact GRUB rather than a Linux kernel. I checked that offset by hand, and there was the smoking gun. GRUB recently acquired Reed-Solomon error correction on its core image, to allow it to recover from other software writing over sectors in the boot track. This moved the magic number lnxboot.S was checking somewhat further into the core image, after the first sector. lnxboot.S couldn't find it because it hadn't copied it yet! A bit of adjustment and all was well again. The lesson for me from all of this has been to try hard to get an interactive debugger working. Really hard. It's worth quite a bit of up-front effort if it saves you from killing neurons stepping through pages of code by hand. I think the real-mode debugging tricks I picked up should be useful for working on GRUB in the future.

14 March 2011

Colin Watson: Wubi bug 693671

I spent most of last week working on Ubuntu bug 693671 ("wubi install will not boot - phase 2 stops with: Try (hd0,0): NTFS5"), which was quite a challenge to debug since it involved digging into parts of the Wubi boot process I'd never really touched before. Since I don't think much of this is very well-documented, I'd like to spend a bit of time explaining what was involved, in the hope that it will help other developers in the future. Wubi is a system for installing Ubuntu into a file in a Windows filesystem, so that it doesn't require separate partitions and can be uninstalled like any other Windows application. The purpose of this is to make it easy for Windows users to try out Ubuntu without the need to worry about repartitioning, before they commit to a full installation. Wubi started out as an external project, and initially patched the installer on the fly to do all the rather unconventional things it needed to do; we integrated it into Ubuntu 8.04 LTS, which involved turning these patches into proper installer facilities that could be accessed using preseeding, so that Wubi only needs to handle the Windows user interface and other Windows-specific tasks. Anyone familiar with a GNU/Linux system's boot process will immediately see that this isn't as simple as it sounds. Of course, ntfs-3g is a pretty solid piece of software so we can handle the Windows filesystem without too much trouble, and loopback mounts are well-understood so we can just have the initramfs loop-mount the root filesystem. Where are you going to get the kernel and initramfs from, though? Well, we used to copy them out to the NTFS filesystem so that GRUB could read them, but this was overly complicated and error-prone. When we switched to GRUB 2, we could instead use its built-in loopback facilities, and we were able to simplify this. So all was more or less well, except for the elephant in the room. How are you going to load GRUB? In a Wubi installation, NTLDR (or BOOTMGR in Windows Vista and newer) still owns the boot process. Ubuntu is added as a boot menu option using BCDEdit. You might then think that you can just have the Windows boot loader chain-load GRUB. Unfortunately, NTLDR only loads 16 sectors - 8192 bytes - from disk. GRUB won't fit in that: the smallest core.img you can generate at the moment is over 18 kilobytes. Thus, you need something that is small enough to be loaded by NTLDR, but that is intelligent enough to understand NTFS to the point where it can find a particular file in the root directory of a filesystem, load boot loader code from it, and jump to that. The answer for this was GRUB4DOS. Most of GRUB4DOS is based on GRUB Legacy, which is not of much interest to us any more, but it includes an assembly-language program called GRLDR that supports doing this very thing for FAT, NTFS, and ext2. In Wubi, we build GRLDR as wubildr.mbr, and build a specially-configured GRUB core image as wubildr. Now, the messages shown in the bug report suggested a failure either within GRLDR or very early in GRUB. The first thing I did was to remember that GRLDR has been integrated into the grub-extras ntldr-img module suitable for use with GRUB 2, so I tried building wubildr.mbr from that; no change, but this gave me a modern baseline to work on. OK; now to try QEMU (you can use tricks like qemu -hda /dev/sda if you're very careful not to do anything that might involve writing to the host filesystem from within the guest, such as recursively booting your host OS ...). No go; it hung somewhere in the middle of NTLDR. Still, I could at least insert debug statements, copy the built wubildr.mbr over to my test machine, and reboot for each test, although it would be slow and tedious. Couldn't I? Well, yes, I mostly could, but that 8192-byte limit came back to bite me, along with an internal 2048-byte limit that GRLDR allocates for its NTFS bootstrap code. There were only a few spare bytes. Something like this would more or less fit, to print a single mark character at various points so that I could see how far it was getting:
	pushal
	xorw	%bx, %bx	/* video page 0 */
	movw	$0x0e4d, %ax	/* print 'M' */
	int	$0x10
	popal
In a few places, if I removed some code I didn't need on my test machine (say, CHS compatibility), I could even fit in cheap and nasty code to print a single register in hex (as long as you didn't mind 'A' to 'F' actually being ':' to '?' in ASCII; and note that this is real-mode code, so the loop counter is %cx not %ecx):
	/* print %edx in dumbed-down hex */
	pushal
	xorw	%bx, %bx
	movb	$0xe, %ah
	movw	$8, %cx
1:
	roll	$4, %edx
	movb	%dl, %al
	andb	$0xf, %al
	int	$0x10
	loop	1b
	popal
After a considerable amount of work tracking down problems by bisection like this, I also observed that GRLDR's NTFS code bears quite a bit of resemblance in its logical flow to GRUB 2's NTFS module, and indeed the same person wrote much of both. Since I knew that the latter worked, I could use it to relieve my brain of trying to understand assembly code logic directly, and could compare the two to look for discrepancies. I did find a few of these, and corrected a simple one. Testing at this point suggested that the boot process was getting as far as GRUB but still wasn't printing anything. I removed some Ubuntu patches which quieten down GRUB's startup: still nothing - so I switched my attentions to grub-core/kern/i386/pc/startup.S, which contains the first code executed from GRUB's core image. Code before the first call to real_to_prot (which switches the processor into protected mode) succeeded, while code after that point failed. Even more mysteriously, code added to real_to_prot before the actual switch to protected mode failed too. Now I was clearly getting somewhere interesting, but what was going on? What I really wanted was to be able to single-step, or at least see what was at the memory location it was supposed to be jumping to. Around this point I was venting on IRC, and somebody asked if it was reproducible in QEMU. Although I'd tried that already, I went back and tried again. Ubuntu's qemu is actually built from qemu-kvm, and if I used qemu -no-kvm then it worked much better. Excellent! Now I could use GDB:
(gdb) target remote   qemu -gdb stdio -no-kvm -hda /dev/sda
This let me run until the point when NTLDR was about to hand over control, then interrupt and set a breakpoint at 0x8200 (the entry point of startup.S). This revealed that the address that should have been real_to_prot was in fact garbage. I set a breakpoint at 0x7c00 (GRLDR's entry point) and stepped all the way through to ensure it was doing the right thing. In the process it was helpful to know that GDB and QEMU don't handle real mode very well between them. Useful tricks here were: Single-stepping showed that GRLDR was loading the entirety of wubildr correctly and jumping to it. The first instruction it jumped to wasn't in startup.S, though, and then I remembered that we prefix the core image with grub-core/boot/i386/pc/lnxboot.S. Stepping through this required a clear head since it copies itself around and changes segment registers a few times. The interesting part was at real_code_2, where it copies a sector of the kernel to the target load address, and then checks a known offset to find out whether the "kernel" is in fact GRUB rather than a Linux kernel. I checked that offset by hand, and there was the smoking gun. GRUB recently acquired Reed-Solomon error correction on its core image, to allow it to recover from other software writing over sectors in the boot track. This moved the magic number lnxboot.S was checking somewhat further into the core image, after the first sector. lnxboot.S couldn't find it because it hadn't copied it yet! A bit of adjustment and all was well again. The lesson for me from all of this has been to try hard to get an interactive debugger working. Really hard. It's worth quite a bit of up-front effort if it saves you from killing neurons stepping through pages of code by hand. I think the real-mode debugging tricks I picked up should be useful for working on GRUB in the future.

29 December 2009

Neil Williams: Intermittent tasks

Sometimes, the main task at any one time is simply going to take time and cannot be shortened. With my poor network connection, this tends to occur any time I need to do network tasks like multistrap and debootstrap tests, archive updates and the rest. The problem is that my network connection is so bad, most apt type tasks completely saturate all available bandwidth, especially when downloading a single large package like those built from gcc or OOo.

Thankfully, gpdftext has provided an intermittent task that is useful, intermittent, non-network dependent and relatively straightforward (so as not to be too distracting from the main task) without being sufficiently routine to make me think about automating it.
I've downloaded some 70 public domain novels as A4 PDF and I'm gradually working through them in gpdftext to tidy up the chapter headings and reformat as an A5 PDF, ready to be transferred back to my ebook reader.

This version of gpdftext isn't released yet (it's current SVN, 0.1.0) as it's in string freeze. However, it provides a fairly robust test of the next release. :-)

Current list includes: Little Women, Fairy Tales of Hans Christian Andersen, Lady Susan, Northanger Abbey, Pride and Prejudice, Sense and Sensibility, Peter Pan (Peter and Wendy), Lorna Doone: A Romance Of Exmoor, The Gap in the Curtain, The Thirty-Nine Steps, Through the Looking Glass (And What Alice Found There), Fanny Hill: Memoirs of a Woman of Pleasure, Moll Flanders, Robinson Crusoe, The Further Adventures of Robinson Crusoe, A Tale of Two Cities, David Copperfield, Great Expectations, Little Dorrit, Oliver Twist, The Life And Adventures Of Nicholas Nickleby, The Adventures of Sherlock Holmes, The Casebook of Sherlock Holmes, The Hound of the Baskervilles, The Memoirs of Sherlock Holmes, The Return of Sherlock Holmes, Cranford, The Wind in the Willows, King Solomon's Mines, Tess of the d'Urbervilles, The Hunchback of Notre Dame, Declaration of Independence, Just so Stories, Kim, The Jungle Book, The Man Who Would be King, The Arabian Nights, Lady Chatterley's Lover, Sons and Lovers, Women in Love, The Story of Doctor Dolittle, The Call of the Wild, The Game, White Fang, Moby Dick, Anne of Green Gables, The Well at the World's End, Pygmalion, An Inquiry into the Nature and Causes of the Wealth of Nations, East is West, Strange Case of Dr Jekyll and Mr Hyde, The Jewel of Seven Stars, Anna Karenina, The Adventures of Huckleberry Finn, The Adventures of Tom Sawyer, The Kama Sutra, 20,000 Leagues Under the Sea, The Aeneid of Virgil.

That should be enough for now.... (21 converted so far).

15 March 2009

Chris Lamb: Joachim Raff and the orchestration of Bach's Chaconne

Earlier today I stumbled across a recording of a work I had not heard in a few years by the obscure Swiss-German composer Joachim Raff. The composition is an orchestral transcription of the Chaconne from Bach's Partita in D minor, without doubt the finest work ever written for solo violin. This piece has been transcribed countless times; the source material is clearly a masterpiece, but what makes the movement really attractive is its implicit nature - dispite the violin's limited facilities for counterpoint and polyphony in general, Bach succeeds in alluding to a multitude of voices within the same line. Larry Solomon provides a brief overview of what to look out for. Raff's transcription thus endeavours to render the implicit explicit; for example, where there is the implication of a held note, it can played for its implied duration. Indeed, Raff believed that Chaconne was in itself a "reduction" from an orchestrated original. Whilst I don't share that literal belief, it is certainly good working model for a transcription. And what a transcription! I've often thought of the romantic transcriptions such as these to be a guilty pleasures; they clearly violate any sensibilities I have about performing early music. However, what is most surprising (and guilt-relieving) about Raff's effort is the sheer amount of novel material involved. Here is an example starting at bar 216 - I have overlayed Bach's original line with a melody of Raff's own invention:
http://chris-lamb.co.uk/wp-content/2009/raff_chaconne.png
There are countless other examples which, following Bach's example, vary greatly in their subtlety. No prizes for guessing why I chose this excerpt though. However, dispite these additions I feel some crucial aspects are actually lost in translation. Firstly, we can muse over whether Bach would have approved of such a rendering - not wishing to dwell too deeply in well-tread arguments, many would point to Bach's re-use of his own material (as well as his transcriptions of other composers) to illustrate that he was not against the practice, but I find that argument difficult to apply to the Chaconne - the dualism between the implicit and the explicit would have greatly appealed to Bach, so to rob the work of it would seem to be doing him a serious discourtesy. Furthermore, the implications in Bach's work are certainly not followed to the same conclusions by all - a truly solo musician has the advantage of being able to make their own decisions about the music, perhaps even on the spur of the moment. In contrast, an ensemble is effectively forced to obey Hr. Raff's inferences. This lack of spontaneity is particularly damning; I am sure one would tire of differing interpretations of Raff's transcription quicker than one would of Bach's original. Lastly (and certainly most subjectively) there is something disarmingly solitary about the original work which is only amplified by being played solo - dispite my inability to provide a satisfactory rendition I would find performing this piece with (or even to) others rather discomforting. In conclusion, I would recommend listening to this work (and the rest of the CD). I have found it extremely illuminating, if only about Bach's original than of Raff's.

18 May 2006

Ted Walther: Why didn't King Solomon kill the two prostitutes?

On the long bus ride back from the Mexican pyramids, I got to think about a long standing problem. If Israel's laws forbade prostitution, why didn't Israel's greatest judge enforce those laws? We've all heard the story of King Solomon and the two harlot's, where he finds the real mother by threatening to chop the baby in two. Since the laws of Israel carried a death penalty for adultery, why didn't Solomon enforce it? People today equate prostitution with adultery, but the Bible does not. Not always. As the wisest man who ever lived, a walking embodiment of true justice, Solomons judgement deserves a deeper look. Here is the text of the whole incident:
1 Kings 3:16-28 Then came there two women, that were harlots, unto the king, and stood before him. And the one woman said, O my lord, I and this woman dwell in one house; and I was delivered of a child with her in the house. And it came to pass the third day after that I was delivered, that this woman was delivered also: and we were together; there was no stranger with us in the house, save we two in the house. And this woman's child died in the night; because she overlaid it. And she arose at midnight, and took my son from beside me, while thine handmaid slept, and laid it in her bosom, and laid her dead child in my bosom. And when I rose in the morning to give my child suck, behold, it was dead: but when I had considered it in the morning, behold, it was not my son, which I did bear. And the other woman said, Nay; but the living is my son, and the dead is thy son. And this said, No; but the dead is thy son, and the living is my son. Thus they spake before the king. Then said the king, The one saith, This is my son that liveth, and thy son is the dead: and the other saith, Nay; but thy son is the dead, and my son is the living. And the king said, Bring me a sword. And they brought a sword before the king. And the king said, Divide the living child in two, and give half to the one, and half to the other. Then spake the woman whose the living child was unto the king, for her bowels yearned upon her son, and she said, O my lord, give her the living child, and in no wise slay it. But the other said, Let it be neither mine nor thine, but divide it. Then the king answered and said, Give her the living child, and in no wise slay it: she is the mother thereof. And all Israel heard of the judgment which the king had judged; and they feared the king: for they saw that the wisdom of God was in him, to do judgment.
Whoredom, harlotry, prostitution; they defile the land. When Judah thought his daughter-in-law Tamar had become a harlot, he ordered her to be burnt. The problem was not with the sex for pay, but with violation of Judah's property rights. Tamar had been bought and paid for, and belonged to his family for the purpose of increasing their manpower. Biblically, sex for pay is fine. A ketubah (marriage contract) guarantees a woman a certain financial allowance in return for performing her sexual duties. The problem is that prostitution lowers the birthrate of the population, thus reducing its manpower, thus reducing its brainpower, thus reducing its ability to protect and defend itself. The connection between prostitution and lower birth rates isn't obvious to the layman, so just take it for granted right now.
Leviticus 18:20,28 Moreover thou shalt not lie carnally with thy neighbour's wife, to defile thyself with her. That the land spue not you out also, when ye defile it, as it spued out the nations that were before you. Leviticus 19:29 Do not prostitute thy daughter, to cause her to be a whore; lest the land fall to whoredom, and the land become full of wickedness.
Prostitution can fall into two categories; the woman is still the property of her father, or she belongs to a husband. Widows, divorced, and single women fall into the category of belonging to their fathers; they have a right to return to the family home and be supported there. How would a father prostitute his daughter? If he treats her harshly, refuses to marry her off to a suitable husband, does not feed or clothe her, train her or discipline her, she may well turn to whoredom to support herself. Look closely at the story of the two harlots. They say they have no man. They fall into the category of single women. Sex with a single woman is not adultery. Adultery carries the death penalty, but as single women, their prostitution was to be regulated by different laws. If a girl took up the prostitution trade, and then got married under false pretenses, pretending to be a virgin, the penalty was death. However no death penalty is indicated for the circumstance where the woman and her family were up front about her past. A man taking a wife needs to be able to accurately assess the risks of disease and future infidelity.
Deuteronomy 22:13-21 If any man take a wife, and go in unto her, and hate her, And give occasions of speech against her, and bring up an evil name upon her, and say, I took this woman, and when I came to her, I found her not a maid: Then shall the father of the damsel, and her mother, take and bring forth the tokens of the damsel's virginity unto the elders of the city in the gate: And the damsel's father shall say unto the elders, I gave my daughter unto this man to wife, and he hateth her; And, lo, he hath given occasions of speech against her, saying, I found not thy daughter a maid; and yet these are the tokens of my daughter's virginity. And they shall spread the cloth before the elders of the city. And the elders of that city shall take that man and chastise him; And they shall amerce him in an hundred shekels of silver, and give them unto the father of the damsel, because he hath brought up an evil name upon a virgin of Israel: and she shall be his wife; he may not put her away all his days. But if this thing be true, and the tokens of virginity be not found for the damsel: Then they shall bring out the damsel to the door of her father's house, and the men of her city shall stone her with stones that she die: because she hath wrought folly in Israel, to play the whore in her father's house: so shalt thou put evil away from among you.
Suppose a single woman was to work as a whore. What is the penalty? For the girl, no penalty at all. If she is a virgin turning her first trick, the man has to pay a hefty fine of fifty shekels to her father, who has the choice of giving her in marriage or not.
Exodus 22:16,17 And if a man entice a maid that is not betrothed, and lie with her, he shall surely endow her to be his wife. If her father utterly refuse to give her unto him, he shall pay money according to the dowry of virgins. Deuteronomy 22:28,29 If a man find a damsel that is a virgin, which is not betrothed, and lay hold on her, and lie with her, and they be found; Then the man that lay with her shall give unto the damsel's father fifty shekels of silver, and she shall be his wife; because he hath humbled her, he may not put her away all his days.
When she is no longer a virgin, her father either allows her to be a prostitute, violating the law at Leviticus 19:29, or he doesn't allow her. If her father allows her, then he becomes liable for her behavior and she is not punished, as described in Hosea 4:13,14 quoted below. If her father doesn't allow her, she is again subject to the death penalty, as set forth in the next set of verses:
Deuteronomy 21:18-21 If a man have a stubborn and rebellious son, which will not obey the voice of his father, or the voice of his mother, and that, when they have chastened him, will not hearken unto them: Then shall his father and his mother lay hold on him, and bring him out unto the elders of his city, and unto the gate of his place; And they shall say unto the elders of his city, This our son is stubborn and rebellious, he will not obey our voice; he is a glutton, and a drunkard. And all the men of his city shall stone him with stones, that he die: so shalt thou put evil away from among you; and all Israel shall hear, and fear.
In the case of Solomon's two harlots, there is no evidence at all that their father disciplined or otherwise acted to prevent them from their trade. Perhaps their father was dead, but in that case his responsibility to his daughters would have fallen on his next of kin. It really is a matter of family responsibility, not of persecuting individuals. There is one remaining corner case; what of the married woman who becomes a prostitute? What if the two harlots had been married after all? It is a principle of the Torah that it must be applied equally. You cannot hit one party with the full penalty then let another get off free for the same crime. It is better to err on the side of mercy, so if you do not convict one, you should not convict another. This is violated daily by un-Godly governments who selectively enforce their laws to "make examples" of people. Such selective enforcement is neither fair or just. In Solomon's case, assuming the women were married, the other parties to the crime of adultery were missing. Without them, there was no way to proceed. This is why Jesus was able to say "Let he who is without sin cast the first stone." The two witnesses rule also comes into it; before a person can be put to death two witnesses to the crime are required. Solomon's two harlots had no witnesses against them. When people engage in false worship, and use the services of whores, God will not punish the women who turn to adultery and infidelity. He will punish those who are responsible; the men, who failed to take their patriarchal prerogative in setting a good example of just, fair, respectful, and temperate behavior.
Hosea 4:13,14 They sacrifice upon the tops of the mountains, and burn incense upon the hills, under oaks and poplars and elms, because the shadow thereof is good: therefore your daughters shall commit whoredom, and your spouses shall commit adultery. I will not punish your daughters when they commit whoredom, nor your spouses when they commit adultery: for themselves are separated with whores, and they sacrifice with harlots: therefore the people that doth not understand shall fall.
If you have any Torah questions, feel free to write in to ted AT reactor-core DOT org.