Search Results: "petere"

13 May 2021

Shirish Agarwal: Population, Immigration, Vaccines and Mass-Surveilance.

The Population Issue and its many facets Another couple of weeks passed. A Lot of things happening, lots of anger and depression in folks due to handling in pandemic, but instead of blaming they are willing to blame everybody else including the population. Many of them want forced sterilization like what Sanjay Gandhi did during the Emergency (1975). I had to share So Long, My son . A very moving tale of two families of what happened to them during the one-child policy in China. I was so moved by it and couldn t believe that the Chinese censors allowed it to be produced, shot, edited, and then shared worldwide. It also won a couple of awards at the 69th Berlin Film Festival, silver bear for the best actor and the actress in that category. But more than the award, the theme, and the concept as well as the length of the movie which was astonishing. Over a 3 hr. something it paints a moving picture of love, loss, shame, relief, anger, and asking for forgiveness. All of which can be identified by any rational person with feelings worldwide.

Girl child What was also interesting though was what it couldn t or wasn t able to talk about and that is the Chinese leftover men. In fact, a similar situation exists here in India, only it has been suppressed. This has been more pronounced more in Asia than in other places. One big thing in this is human trafficking and mostly women trafficking. For the Chinese male, that was happening on a large scale from all neighboring countries including India. This has been shared in media and everybody knows about it and yet people are silent. But this is not limited to just the Chinese, even Indians have been doing it. Even yesteryear actress Rupa Ganguly was caught red-handed but then later let off after formal questioning as she is from the ruling party. So much for justice. What is and has been surprising at least for me is Rwanda which is in the top 10 of some of the best places in equal gender. It, along with other African countries have also been in news for putting quite a significant amount of percentage of GDP into public healthcare (between 20-10%), but that is a story for a bit later. People forget or want to forget that it was in Satara, a city in my own state where 220 girls changed their name from nakusha or unwanted to something else and that had become a piece of global news. One would think that after so many years, things would have changed, the only change that has happened is that now we have two ministries, The Ministry of Women and Child Development (MoWCD) and The Ministry of Health and Welfare (MoHFW). Sadly, in both cases, the ministries have been found wanting, Whether it was the high-profile Hathras case or even the routine cries of help which given by women on the twitter helpline. Sadly, neither of these ministries talks about POSH guidelines which came up after the 2012 gangrape case. For both these ministries, it should have been a pinned tweet. There is also the 1994 PCPNDT Act which although made in 1994, actually functioned in 2006, although what happens underground even today nobody knows  . On the global stage, about a decade ago, Stephen J. Dubner and Steven Levitt argued in their book Freakonomics how legalized abortion both made the coming population explosion as well as expected crime rates to be reduced. There was a huge pushback on the same from the conservatives and has become a matter of debate, perhaps something that the Conservatives wanted. Interestingly, it hasn t made them go back but go forward as can be seen from the Freakonomics site.

Climate Change Another topic that came up for discussion was repeatedly climate change, but when I share Shell s own 1998 Confidential report titled Greenhouse effect all become strangely silent. The silence here is of two parts, there probably is a large swathe of Indians who haven t read the report and there may be a minority who have read it and know what already has been shared with U.S. Congress. The Conservative s argument has been for it is jobs and a weak we need to research more . There was a partial debunk of it on the TBD podcast by Matt Farell and his brother Sean Farell as to how quickly the energy companies are taking to the coming change.

Health Budget Before going to Covid stories. I first wanted to talk about Health Budgets. From the last 7 years the Center s allocation for health has been between 0.34 to 0.8% per year. That amount barely covers the salaries to the staff, let alone any money for equipment or anything else. And here by allocation I mean, what is actually spent, not the one that is shared by GOI as part of budget proposal. In fact, an article on Wire gives a good breakdown of the numbers. Even those who are on the path of free markets describe India s health business model as a flawed one. See the Bloomberg Quint story on that. Now let me come to Rwanda. Why did I chose Rwanda, I could have chosen South Africa where I went for Debconf 2016, I chose because Rwanda s story is that much more inspiring. In many ways much more inspiring than that South Africa in many ways. Here is a country which for decades had one war or the other, culminating into the Rwanda Civil War which ended in 1994. And coincidentally, they gained independence on a similar timeline as South Africa ending Apartheid in 1994. What does the country do, when it gains its independence, it first puts most of its resources in the healthcare sector. The first few years at 20% of GDP, later than at 10% of GDP till everybody has universal medical coverage. Coming back to the Bloomberg article I shared, the story does not go into the depth of beyond-expiry date medicines, spurious medicines and whatnot. Sadly, most media in India does not cover the deaths happening in rural areas and this I am talking about normal times. Today what is happening in rural areas is just pure madness. For last couple of days have been talking with people who are and have been covering rural areas. In many of those communities, there is vaccine hesitancy and why, because there have been whatsapp forwards sharing that if you go to a hospital you will die and your kidney or some other part of the body will be taken by the doctor. This does two things, it scares people into not going and getting vaccinated, at the same time they are prejudiced against science. This is politics of the lowest kind. And they do it so that they will be forced to go to temples or babas and what not and ask for solutions. And whether they work or not is immaterial, they get fixed and property and money is seized. Sadly, there are not many Indian movies of North which have tried to show it except for oh my god but even here it doesn t go the distance. A much more honest approach was done in Trance . I have never understood how the South Indian movies are able to do a more honest job of story-telling than what is done in Bollywood even though they do in 1/10th the budget that is needed in Bollywood. Although, have to say with OTT, some baggage has been shed but with the whole film certification rearing its ugly head through MEITY orders, it seems two steps backward instead of forward. The idea being simply to infantilize the citizens even more. That is a whole different ball-game which probably will require its own space.

Vaccine issues One good news though is that Vaccination has started. But it has been a long story full of greed by none other than GOI (Government of India) or the ruling party BJP. Where should I start with. I probably should start with this excellent article done by Priyanka Pulla. It is interesting and fascinating to know how vaccines are made, at least one way which she shared. She also shared about the Cutter Incident which happened in the late 50 s. The response was on expected lines, character assassination of her and the newspaper they published but could not critique any of the points made by her. Not a single point that she didn t think about x or y. Interestingly enough, in January 2021 Bharati Biotech was supposed to be share phase 3 trial data but hasn t been put up in public domain till May 2021. In fact, there have been a few threads raised by both well-meaning Indians as well as others globally especially on twitter to which GOI/ICMR (Indian Council of Medical Research) is silent. Another interesting point to note is that Russia did say in its press release that it is possible that their vaccine may not be standard (read inactivation on their vaccines and another way is possible but would take time, again Brazil has objected, but India hasn t till date.) What also has been interesting is the homegrown B.1.617 lineage or known as double mutant . This was first discovered from my own state, Maharashtra and then transported around the world. There is also B.1.618 which was found in West Bengal and is same or supposed to be similar to the one found in South Africa. This one is known as Triple mutant . About B.1.618 we don t know much other than knowing that it is much more easily transferable, much more infectious. Most countries have banned flights from India and I cannot fault them anyway. Hell, when even our diplomats do not care for procedures to be followed during the pandemic then how a common man is supposed to do. Of course, now for next month, Mr. Modi was supposed to go and now will not attend the G7 meeting. Whether, it is because he would have to face the press (the only Prime Minister and the only Indian Prime Minister who never has faced free press.) or because the Indian delegation has been disinvited, we would never know.

A good article which shares lots of lows with how things have been done in India has been an article by Arundhati Roy. And while the article in itself is excellent and shares a bit of the bitter truth but is still incomplete as so much has been happening. The problem is that the issue manifests in so many ways, it is difficult to hold on. As Arundhati shared, should we just look at figures and numbers and hold on, or should we look at individual ones, for e.g. the one shared in Outlook India. Or the one shared by Dr. Dipshika Ghosh who works in Covid ICU in some hospital
Dr. Dipika Ghosh sharing an incident in Covid Ward

Interestingly as well, while in the vaccine issue, Brazil Anvisa doesn t know what they are doing or the regulator just isn t knowledgeable etc. (statements by various people in GOI, when it comes to testing kits, the same is an approver.)

ICMR/DGCI approving internationally validated kits, Press release.

Twitter In the midst of all this, one thing that many people have forgotten and seem to have forgotten that Twitter and other tools are used by only the elite. The reason why the whole thing has become serious now than in the first phase is because the elite of India have also fallen sick and dying which was not the case so much in the first phase. The population on Twitter is estimated to be around 30-34 million and people who are everyday around 20 odd million or so, which is what 2% of the Indian population which is estimated to be around 1.34 billion. The other 98% don t even know that there is something like twitter on which you can ask help. Twitter itself is exclusionary in many ways, with both the emoticons, the language and all sorts of things. There is a small subset who does use Twitter in regional languages, but they are too small to write anything about. The main language is English which does become a hindrance to lot of people.

Censorship Censorship of Indians critical of Govt. mishandling has been non-stop. Even U.S. which usually doesn t interfere into India s internal politics was forced to make an exception. But of course, this has been on deaf ears. There is and was a good thread on Twitter by Gaurav Sabnis, a friend, fellow Puneite now settled in U.S. as a professor.
Gaurav on Trump-Biden on vaccination of their own citizens
Now just to surmise what has been happened in India and what has been happening in most of the countries around the world. Most of the countries have done centralization purchasing of the vaccine and then is distributed by the States, this is what we understand as co-operative federalism. While last year, GOI took a lot of money under the shady PM Cares fund for vaccine purchase, donations from well-meaning Indians as well as Industries and trade bodies. Then later, GOI said it would leave the states hanging and it is they who would have to buy vaccines from the manufacturers. This is again cheap politics. The idea behind it is simple, GOI knows that almost all the states are strapped for cash. This is not new news, this I have shared a couple of months back. The problem has been that for the last 6-8 months no GST meeting has taken place as shared by Punjab s Finance Minister Amarinder Singh. What will happen is that all the states will fight in-between themselves for the vaccine and most of them are now non-BJP Governments. The idea is let the states fight and somehow be on top. So, the pandemic, instead of being a public health issue has become something of on which politics has to played. The news on whatsapp by RW media is it s ok even if a million or two also die, as it is India is heavily populated. Although that argument vanishes for those who lose their dear and near ones. But that just isn t the issue, the issue goes much more deeper than that Oxygen:12%
Remedisivir:12%
Sanitiser:12%
Ventilator:12%
PPE:18%
Ambulances 28% Now all the products above are essential medical equipment and should be declared as essential medical equipment and should have price controls on which GST is levied. In times of pandemic, should the center be profiting on those. States want to let go and even want the center to let go so that some relief is there to the public, while at the same time make them as essential medical equipment with price controls. But GOI doesn t want to. Leaders of opposition parties wrote open letters but no effect. What is sad to me is how Ambulances are being taxed at 28%. Are they luxury items or sin goods ? This also reminds of the recent discovery shared by Mr. Pappu Yadav in Bihar. You can see the color of ambulances as shared by Mr. Yadav, and the same news being shared by India TV news showing other ambulances. Also, the weak argument being made of not having enough drivers. Ideally, you should have 2-3 people, both 9-1-1 and Chicago Fire show 2 people in ambulance but a few times they have also shown to be flipped over. European seems to have three people in ambulance, also they are also much more disciplined as drivers, at least an opinion shared by an American expat.
Pappu Yadav, President Jan Adhikar Party, Bihar May 11, 2021
What is also interesting to note is GOI plays this game of Health is State subject and health is Central subject depending on its convenience. Last year, when it invoked the Epidemic and DMA Act it was a Central subject, now when bodies are flowing down the Ganges and pyres being lit everywhere, it becomes a State subject. But when and where money is involved, it again becomes a Central subject. The States are also understanding it, but they are fighting on too many fronts.
Snippets from Karnataka High Court hearing today, 13th March 2021
One of the good things is most of the High Courts have woken up. Many of the people on the RW think that the Courts are doing Judicial activism . And while there may be an iota of truth in it, the bitter truth is that many judges or relatives or their helpers have diagnosed and some have even died due to Covid. In face of the inevitable, what can they do. They are hauling up local Governments to make sure they are accountable while at the same time making sure that they get access to medical facilities. And I as a citizen don t see any wrong in that even if they are doing it for selfish reasons. Because, even if justice is being done for selfish reasons, if it does improve medical delivery systems for the masses, it is cool. If it means that the poor and everybody else are able to get vaccinations, oxygen and whatever they need, it is cool. Of course, we are still seeing reports of patients spending in the region of INR 50k and more for each day spent in hospital. But as there are no price controls, judges cannot do anything unless they want to make an enemy of the medical lobby in the country. A good story on medicines and what happens in rural areas, see no further than Laakhon mein ek.
Allahabad High Court hauling Uttar Pradesh Govt. for lack of Oxygen is equal to genocide, May 11, 2021
The censorship is not just related to takedown requests on twitter but nowadays also any articles which are critical of the GOI s handling. I have been seeing many articles which have shared facts and have been critical of GOI being taken down. Previously, we used to see 404 errors happen 7-10 years down the line and that was reasonable. Now we see that happen, days weeks or months. India seems to be turning more into China and North Korea and become more anti-science day-by-day

Fake websites Before going into fake websites, let me start with a fake newspaper which was started by none other than the Gujarat CM Mr. Modi in 2005 .
Gujarat Satya Samachar 2005 launched by Mr. Modi.
And if this wasn t enough than on Feb 8, 2005, he had invoked Official Secrets Act
Mr. Modi invoking Official Secrets Act, Feb 8 2005 Gujarat Samachar
The headlines were In Modi s regime press freedom is in peril-Down with Modi s dictatorship. So this was a tried and tested technique. The above information was shared by Mr. Urvish Kothari, who incidentally also has his own youtube channel. Now cut to 2021, and we have a slew of fake websites being done by the same party. In fact, it seems they started this right from 2011. A good article on BBC itself tells the story. Hell, Disinfo.eu which basically combats disinformation in EU has a whole pdf chronicling how BJP has been doing it. Some of the sites it shared are

Times of New York
Manchester Times
Times of Los Angeles
Manhattan Post
Washington Herald
and many more. The idea being take any site name which sounds similar to a brand name recognized by Indians and make fool of them. Of course, those of who use whois and other such tools can easily know what is happening. Two more were added to the list yesterday, Daily Guardian and Australia Today. There are of course, many features which tell them apart from genuine websites. Most of these are on shared hosting rather than dedicated hosting, most of these are bought either from Godaddy and Bluehost. While Bluehost used to be a class act once upon a time, both the above will do anything as long as they get money. Don t care whether it s a fake website or true. Capitalism at its finest or worst depending upon how you look at it. But most of these details are lost on people who do not know web servers, at all and instead think see it is from an exotic site, a foreign site and it chooses to have same ideas as me. Those who are corrupt or see politics as a tool to win at any cost will not see it as evil. And as a gentleman Raghav shared with me, it is so easy to fool us. An example he shared which I had forgotten. Peter England which used to be an Irish brand was bought by Aditya Birla group way back in 2000. But even today, when you go for Peter England, the way the packaging is done, the way the prices are, more often than not, people believe they are buying the Irish brand. While sharing this, there is so much of Naom Chomsky which comes to my mind again and again

Caste Issues I had written about caste issues a few times on this blog. This again came to the fore as news came that a Hindu sect used forced labor from Dalit community to make a temple. This was also shared by the hill. In both, Mr. Joshi doesn t tell that if they were volunteers then why their passports have been taken forcibly, also I looked at both minimum wage prevailing in New Jersey as a state as well as wage given to those who are in the construction Industry. Even in minimum wage, they were giving $1 when the prevailing minimum wage for unskilled work is $12.00 and as Mr. Joshi shared that they are specialized artisans, then they should be paid between $23 $30 per hour. If this isn t exploitation, then I don t know what is. And this is not the first instance, the first instance was perhaps the case against Cisco which was done by John Doe. While I had been busy with other things, it seems Cisco had put up both a demurrer petition and a petition to strike which the Court stayed. This seemed to all over again a type of apartheid practice, only this time applied to caste. The good thing is that the court stayed the petition. Dr. Ambedkar s statement if Hindus migrate to other regions on earth, Indian caste would become a world problem given at Columbia University in 1916, seems to be proven right in today s time and sadly has aged well. But this is not just something which is there only in U.S. this is there in India even today, just couple of days back, a popular actress Munmun Dutta used a casteist slur and then later apologized giving the excuse that she didn t know Hindi. And this is patently false as she has been in the Bollywood industry for almost now 16-17 years. This again, was not an isolated incident. Seema Singh, a lecturer in IIT-Kharagpur abused students from SC, ST backgrounds and was later suspended. There is an SC/ST Atrocities Act but that has been diluted by this Govt. A bit on the background of Dr. Ambedkar can be found at a blog on Columbia website. As I have shared and asked before, how do we think, for what reason the Age of Englightenment or the Age of Reason happened. If I were a fat monk or a priest who was privileges, would I have let Age of Enlightenment happen. It broke religion or rather Church which was most powerful to not so powerful and that power was more distributed among all sort of thinkers, philosophers, tinkers, inventors and so on and so forth.

Situation going forward I believe things are going to be far more complex and deadly before they get better. I had to share another term called Comorbidities which fortunately or unfortunately has also become part of twitter lexicon. While I have shared what it means, it simply means when you have an existing ailment or condition and then Coronavirus attacks you. The Virus will weaken you. The Vaccine in the best case just stops the damage, but the damage already done can t be reversed. There are people who advise and people who are taking steroids but that again has its own side-effects. And this is now, when we are in summer. I am afraid for those who have recovered, what will happen to them during the Monsoons. We know that the Virus attacks most the lungs and their quality of life will be affected. Even the immune system may have issues. We also know about the inflammation. And the grant that has been given to University of Dundee also has signs of worry, both for people like me (obese) as well as those who have heart issues already. In other news, my city which has been under partial lockdown since a month, has been extended for another couple of weeks. There are rumors that the same may continue till the year-end even if it means economics goes out of the window.There is possibility that in the next few months something like 2 million odd Indians could die
The above is a conversation between Karan Thapar and an Oxford Mathematician Dr. Murad Banaji who has shared that the under-counting of cases in India is huge. Even BBC shared an article on the scope of under-counting. Of course, those on the RW call of the evidence including the deaths and obituaries in newspapers as a narrative . And when asked that when deaths used to be in the 20 s or 30 s which has jumped to 200-300 deaths and this is just the middle class and above. The poor don t have the money to get wood and that is the reason you are seeing the bodies in Ganges whether in Buxar Bihar or Gajipur, Uttar Pradesh. The sights and visuals makes for sorry reading
Pandit Ranjan Mishra son on his father s death due to unavailability of oxygen, Varanasi, Uttar Pradesh, 11th May 2021.
For those who don t know Pandit Ranjan Mishra was a renowned classical singer. More importantly, he was the first person to suggest Mr. Modi s name as a Prime Ministerial Candidate. If they couldn t fulfil his oxygen needs, then what can be expected for the normal public.

Conclusion Sadly, this time I have no humorous piece to share, I can however share a documentary which was shared on Feluda . I have shared about Feluda or Prodosh Chandra Mitter a few times on this blog. He has been the answer of James Bond from India. I have shared previously about The Golden Fortress . An amazing piece of art by Satyajit Ray. I watched that documentary two-three times. I thought, mistakenly that I am the only fool or fan of Feluda in Pune to find out that there are people who are even more than me. There were so many facets both about Feluda and master craftsman Satyajit Ray that I was unaware about. I was just simply amazed. I even shared few of the tidbits with mum as well, although now she has been truly hooked to Korean dramas. The only solace from all the surrounding madness. So, if you have nothing to do, you can look up his books, read them and then see the movies. And my first recommendation would be the Golden Fortress. The only thing I would say, do not have high hopes. The movie is beautiful. It starts slow and then picks up speed, just like a train. So, till later. Update The Mass surveillance part I could not do justice do hence removed it at the last moment. It actually needs its whole space, article. There is so much that the Govt. is doing under the guise of the pandemic that it is difficult to share it all in one article. As it is, the article is big

11 April 2016

Peter Eisentraut: Some git log tweaks

Here are some tweaks to git log that I have found useful. It might depend on the workflow of individual projects how applicable this is. Git stores separate author and committer information for each commit. How these are generated and updated is sometimes mysterious but generally makes sense. For example, if you cherry-pick a commit to a different branch, the author information stays the same but the committer information is updated. git log defaults to showing the author information. But I generally care less about that than the committer information, because I m usually interested in when the commit arrived in my or the public repository, not when it was initially thought about. So let s try to change the default git log format to show the committer information instead. Again, depending on the project and the workflow, there can be other preferences. To create a different default format for git log, first create a new format by setting the Git configuration item pretty.somename. I chose pretty.cmedium because it s almost the same as the default medium but with the author information replaced by the committer information.
[pretty]
cmedium="format:%C(auto,yellow)commit %H%C(auto,reset)%nCommit:     %cn <%ce>%nCommitDate: %cd%n%n%w(0,4,4)%s%n%+b"
Unfortunately, the default git log formats are not defined in terms of these placeholders but are hardcoded in the source, so this is my best reconstruction using the available means. You can use this as git log --pretty=cmedium, and you can set this as the default using
[format]
pretty=cmedium
If you find this useful and you re the sort of person who is more interested in their own timeline than the author s history, you might also like two more tweaks. First, add %cr for relative date, so it looks like
[pretty]
cmedium="format:%C(auto,yellow)commit %H%C(auto,reset)%nCommit:     %cn <%ce>%nCommitDate: %cd (%cr)%n%n%w(0,4,4)%s%n%+b"
This adds a relative designation like 2 days ago to the commit date. Second, set
[log]
date=local
to have all timestamps converted to your local time. When you put all this together, you turn this
commit e2c117a28f767c9756d2d620929b37651dbe43d1
Author: Paul Eggert <eggert@cs.ucla.edu>
Date:   Tue Apr 5 08:16:01 2016 -0700
into this
commit e2c117a28f767c9756d2d620929b37651dbe43d1
Commit:     Paul Eggert <eggert@cs.ucla.edu>
CommitDate: Tue Apr 5 11:16:01 2016 (3 days ago)
PS: If this is lame, there is always this: http://fredkschott.com/post/2014/02/git-log-is-so-2005/

23 May 2015

Francois Marier: Usual Debian Server Setup

I manage a few servers for myself, friends and family as well as for the Libravatar project. Here is how I customize recent releases of Debian on those servers.

Hardware tests
apt-get install memtest86+ smartmontools e2fsprogs
Prior to spending any time configuring a new physical server, I like to ensure that the hardware is fine. To check memory, I boot into memtest86+ from the grub menu and let it run overnight. Then I check the hard drives using:
smartctl -t long /dev/sdX
badblocks -swo badblocks.out /dev/sdX

Configuration
apt-get install etckeepr git sudo vim
To keep track of the configuration changes I make in /etc/, I use etckeeper to keep that directory in a git repository and make the following changes to the default /etc/etckeeper/etckeeper.conf:
  • turn off daily auto-commits
  • turn off auto-commits before package installs
To get more control over the various packages I install, I change the default debconf level to medium:
dpkg-reconfigure debconf
Since I use vim for all of my configuration file editing, I make it the default editor:
update-alternatives --config editor

ssh
apt-get install openssh-server mosh fail2ban
Since most of my servers are set to UTC time, I like to use my local timezone when sshing into them. Looking at file timestamps is much less confusing that way. I also ensure that the locale I use is available on the server by adding it the list of generated locales:
dpkg-reconfigure locales
Other than that, I harden the ssh configuration and end up with the following settings in /etc/ssh/sshd_config (jessie):
HostKey /etc/ssh/ssh_host_ed25519_key
HostKey /etc/ssh/ssh_host_rsa_key
HostKey /etc/ssh/ssh_host_ecdsa_key
KexAlgorithms curve25519-sha256@libssh.org,ecdh-sha2-nistp521,ecdh-sha2-nistp384,ecdh-sha2-nistp256,diffie-hellman-group-exchange-sha256
Ciphers chacha20-poly1305@openssh.com,aes256-ctr,aes192-ctr,aes128-ctr
MACs hmac-sha2-512-etm@openssh.com,hmac-sha2-256-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-512,hmac-sha2-256,umac-128@openssh.com
UsePrivilegeSeparation sandbox
AuthenticationMethods publickey
PasswordAuthentication no
PermitRootLogin no
AcceptEnv LANG LC_* TZ
LogLevel VERBOSE
AllowGroups sshuser
or the following for wheezy servers:
HostKey /etc/ssh/ssh_host_rsa_key
HostKey /etc/ssh/ssh_host_ecdsa_key
KexAlgorithms ecdh-sha2-nistp521,ecdh-sha2-nistp384,ecdh-sha2-nistp256,diffie-hellman-group-exchange-sha256
Ciphers aes256-ctr,aes192-ctr,aes128-ctr
MACs hmac-sha2-512,hmac-sha2-256
On those servers where I need duplicity/paramiko to work, I also add the following:
KexAlgorithms ...,diffie-hellman-group-exchange-sha1
MACs ...,hmac-sha1
Then I remove the "Accepted" filter in /etc/logcheck/ignore.d.server/ssh (first line) to get a notification whenever anybody successfully logs into my server. I also create a new group and add the users that need ssh access to it:
addgroup sshuser
adduser francois sshuser
and add a timeout for root sessions by putting this in /root/.bash_profile:
TMOUT=600

Security checks
apt-get install logcheck logcheck-database fcheck tiger debsums corekeeper
apt-get remove john john-data rpcbind tripwire
Logcheck is the main tool I use to keep an eye on log files, which is why I add a few additional log files to the default list in /etc/logcheck/logcheck.logfiles:
/var/log/apache2/error.log
/var/log/mail.err
/var/log/mail.warn
/var/log/mail.info
/var/log/fail2ban.log
while ensuring that the apache logfiles are readable by logcheck:
chmod a+rx /var/log/apache2
chmod a+r /var/log/apache2/*
and fixing the log rotation configuration by adding the following to /etc/logrotate.d/apache2:
create 644 root adm
I also modify the main logcheck configuration file (/etc/logcheck/logcheck.conf):
INTRO=0
FQDN=0
Other than that, I enable daily checks in /etc/default/debsums and customize a few tiger settings in /etc/tiger/tigerrc:
Tiger_Check_RUNPROC=Y
Tiger_Check_DELETED=Y
Tiger_Check_APACHE=Y
Tiger_FSScan_WDIR=Y
Tiger_SSH_Protocol='2'
Tiger_Passwd_Hashes='sha512'
Tiger_Running_Procs='rsyslogd cron atd /usr/sbin/apache2 postgres'
Tiger_Listening_ValidProcs='sshd mosh-server ntpd'

General hardening
apt-get install harden-clients harden-environment harden-servers apparmor apparmor-profiles apparmor-profiles-extra
While the harden packages are configuration-free, AppArmor must be manually enabled:
perl -pi -e 's,GRUB_CMDLINE_LINUX="(.*)"$,GRUB_CMDLINE_LINUX="$1 apparmor=1 security=apparmor",' /etc/default/grub
update-grub

Entropy and timekeeping
apt-get install haveged rng-tools ntp
To keep the system clock accurate and increase the amount of entropy available to the server, I install the above packages and add the tpm_rng module to /etc/modules.

Preventing mistakes
apt-get install molly-guard safe-rm sl
The above packages are all about catching mistakes (such as accidental deletions). However, in order to extend the molly-guard protection to mosh sessions, one needs to manually apply a patch.

Package updates
apt-get install apticron unattended-upgrades deborphan debfoster apt-listchanges update-notifier-common aptitude popularity-contest
These tools help me keep packages up to date and remove unnecessary or obsolete packages from servers. On Rackspace servers, a small configuration change is needed to automatically update the monitoring tools. In addition to this, I use the update-notifier-common package along with the following cronjob in /etc/cron.daily/reboot-required:
#!/bin/sh
cat /var/run/reboot-required 2> /dev/null   true
to send me a notification whenever a kernel update requires a reboot to take effect.

Handy utilities
apt-get install renameutils atool iotop sysstat lsof mtr-tiny
Most of these tools are configure-free, except for sysstat, which requires enabling data collection in /etc/default/sysstat to be useful.

Apache configuration
apt-get install apache2-mpm-event
While configuring apache is often specific to each server and the services that will be running on it, there are a few common changes I make. I enable these in /etc/apache2/conf.d/security:
<Directory />
    AllowOverride None
    Order Deny,Allow
    Deny from all
</Directory>
ServerTokens Prod
ServerSignature Off
and remove cgi-bin directives from /etc/apache2/sites-enabled/000-default. I also create a new /etc/apache2/conf.d/servername which contains:
ServerName machine_hostname

Mail
apt-get install postfix
Configuring mail properly is tricky but the following has worked for me. In /etc/hostname, put the bare hostname (no domain), but in /etc/mailname put the fully qualified hostname. Change the following in /etc/postfix/main.cf:
inet_interfaces = loopback-only
myhostname = (fully qualified hostname)
smtp_tls_security_level = may
smtp_tls_protocols = !SSLv2, !SSLv3
Set the following aliases in /etc/aliases:
  • set francois as the destination of root emails
  • set an external email address for francois
  • set root as the destination for www-data emails
before running newaliases to update the aliases database. Create a new cronjob (/etc/cron.hourly/checkmail):
#!/bin/sh
ls /var/mail
to ensure that email doesn't accumulate unmonitored on this box. Finally, set reverse DNS for the server's IPv4 and IPv6 addresses and then test the whole setup using mail root.

Network tuning To reduce the server's contribution to bufferbloat I change the default kernel queueing discipline (jessie or later) by putting the following in /etc/sysctl.conf:
net.core.default_qdisc=fq_codel

17 February 2015

Peter Eisentraut: Listing screen sessions on login

There is a lot of helpful information about screen out there, but I haven t found anything about this. I don t want to forget any screen sessions, so I d like to be notified when I log into a box and there are screens running for me. Obviously, there is screen -ls, but it needs to be wrapped in a bit logic so that it doesn t annoy when there is no screen running or even installed. After perusing the screen man page a little, I came up with this for .bash_profile or .zprofile:
if which screen >/dev/null; then
    screen -q -ls
    if [ $? -ge 10 ]; then
        screen -ls
    fi
fi
The trick is that -q in conjuction with -ls gives you exit codes about the current status of screen. Here is an example of how this looks in practice:
~$ ssh host
Last login: Fri Feb 13 11:30:10 2015 from 192.0.2.15
There is a screen on:
        31572.pts-0.foobar      (2015-02-15 13.03.21)   (Detached)
1 Socket in /var/run/screen/S-peter.
peter@host:~$ 

9 January 2015

Peter Eisentraut: Directing output to multiple files with zsh

Normally, this doesn t work as one might naively expect:
program > firstfile > secondfile
The second redirection will override the first one. You d have to use an external tool to make this work, maybe something like:
program   tee firstfile secondfile
But with zsh, this type of thing actually works. It will duplicate the output and write it to multiple files. This feature also works with a combination of redirections and pipes. For example
ls > foo   grep bar
will write the complete directory listing into file foo and print out files matching bar to the terminal. That s great, but this feature pops up in unexpected places. I have a shell function that checks whether a given command produces any output on stderr:
! myprog "$arg" 2>&1 >/dev/null   grep .
The effect of this is: (Note the ordering of 2>&1 >/dev/null to redirect stderr to stdout and silence the original stdout, as opposed to the more common incantation of >/dev/null 2>&1, which silences both stderr and stdout.) The reason for this is that myprog has a bug that causes it to print errors but not produce a proper exit status in some cases. Now how will my little shell function snippet behave under zsh? Well, it s quite confusing at first, but the following happens. If there is stderr output, then only stderr is printed. If there is no stderr output, then stdout is passed through instead. But that s not what I wanted. This can be reproduced simply:
ls --bogus 2>&1 >/dev/null   grep .
prints an error message, as expected, but
ls 2>&1 >/dev/null   grep .
prints a directory listing. That s because zsh redirects stdout to both /dev/null and the pipe, which makes the redirection to /dev/null pointless. Note that in bash, the second command prints nothing. This behavior can be changed by turning off the MULTIOS option (see zshmisc man page), and my first instinct was to do that, but options are not lexically scoped (I think), so this would break again if the option was somehow changed somewhere else. Also, I think I kind of like that option for interactive use. My workaround is to use a subshell:
! ( myprog "$arg" 2>&1 >/dev/null )   grep .
The long-term fix will probably be to write an external shell script in bash or plain POSIX shell.

2 December 2014

Peter Eisentraut: ccache and clang, part 3

In part 1 and part 2 I investigated how to use ccache with clang. That was more than three years ago. I got an email the other day that ccache bug 8118, which I filed while writing part 1, was closed, as ccache 3.2 was released. The release notes of clang 3.2 contain several items related to clang. So it was time to give this another look. Basically, the conclusions from part 2 still stand: You cannot use ccache with clang without using CCACHE_CPP2. And it is now becoming clear to me that this is an issue that is not going to go away, and it s not really even Clang s fault. Warnings! The problem is that clang s -Wall can cause warnings when compiling the preprocessed version of otherwise harmless C code. This can be illustrated by this piece of C code:
int
foo()
 
        int *p, *q;
        p = q = 0;
        p = p;
        if (p == p)
                return 1;
        if ((p == q))
                return 2;
        return 0;
 
When compiled by gcc-4.9 -Wall, this gives no warnings. When compiled by clang-3.5 -Wall, this results in test.c:7:4: warning: explicitly assigning value of variable of type 'int *' to itself [-Wself-assign] test.c:8:8: warning: self-comparison always evaluates to true [-Wtautological-compare] test.c:10:9: warning: equality comparison with extraneous parentheses [-Wparentheses-equality] test.c:10:9: note: remove extraneous parentheses around the comparison to silence this warning test1.c:10:9: note: use '=' to turn this equality comparison into an assignment You wouldn t normally write code like this, but the C preprocessor could create code with self-assignments, self-comparisons, extra parentheses, and so on. This example represents the issues I saw when trying to compile PostgreSQL 9.4 with ccache and clang; there might be others. You can address this issue in two ways:
  1. Use CCACHE_CPP2, as discussed in part 2. With ccache 3.2, you can now also put this into a configuration file: run_second_cpp = true in ~/.ccache/ccache.conf
  2. Turn off the warnings mentioned above: -Wno-parentheses-equality, -Wno-tautological-compare, -Wno-self-assign (and any others you might find). One might think that these are actually useful warnings that one might want to keep, but GCC doesn t warn about them, and if you develop primarily with GCC, your code might contain these issues anyway. In particular, I have found that -Wno-tautological-compare is necessary for legitimate code.
I think CCACHE_CPP2 is the way to go, for two reasons. Firstly, having to add more and more options to turn off warnings is obviously somewhat stupid. Secondly and more importantly, there is nothing stopping GCC from adding warnings similar to Clang s that would trigger on preprocessed versions of otherwise harmless C code. Unless they come up with a clever way to annotate the preprocessed code to the effect of this code might look wrong to you, but it looked OK before preprocessing, so don t warn about it , in a way that creates no extra warnings and doesn t lose any warnings, I don t think this issue can be solved. Speed! Now the question is, how much would globally setting CCACHE_CPP2 slow things down? To test this, I have built PostgreSQL 9.4rc1 with clang-3.5 and gcc-4.8 (not gcc-4.9 because it creates some unrelated warnings that I don t want to deal with here). I have set export CCACHE_RECACHE=true so that the cache is not read but new cache entries are computed. That way, the overhead of ccache on top of the compiler is measured. Results: (There different percentages between gcc and clang arise because gcc is faster than clang (yes, really, more on that in a future post), but the overhead of ccache doesn t change.) 10% or so is not to be dismissed, but let s remember that this applies only if there is a cache miss. If everything is cached, both methods do the same thing. Also, if you use parallel make, the overhead is divided by the number of parallel jobs. With that in mind, I have decided to put the issue to rest for myself and have made myself a ~/.ccache/ccache.conf containing
run_second_cpp = true
Now Clang or any other compiler should run without problems through ccache. Color! There is one more piece of news in the new ccache release: Another thing I talked about in part 1 was that ccache will disable the colored output of clang, and I suggested workarounds. This was actually fixed in ccache 3.2, so the workarounds are no longer necessary, and the above configuration change is really the only thing to make Clang work smoothly with ccache.

5 November 2014

Peter Eisentraut: Checking whitespace with Git

Whitespace matters. Git has support for checking whitespace in patches. git apply and git am have the option --whitespace, which can be used to warn or error about whitespace errors in the patches about to be applied. git diff has the option --check to check a change for whitespace errors. But all this assumes that your existing code is cool, and only new changes are candidates for problems. Curiously, it is a bit hard to use those same tools for going back and checking whether an existing tree satisfies the whitespace rules applied to new patches. The core of the whitespace checking is in git diff-tree. With the --check option, you can check the whitespace in the diff between two objects. But how do you check the whitespace of a tree rather than a diff? Basically, you want
git diff-tree --check EMPTY HEAD
except there is no EMPTY. But you can compute the hash of an empty Git tree:
git hash-object -t tree /dev/null
So the full command is
git diff-tree --check $(git hash-object -t tree /dev/null) HEAD
If have this as an alias in my ~/.gitconfig:
[alias]
    check-whitespace = !git diff-tree --check $(git hash-object -t tree /dev/null) HEAD
Then running
git check-whitespace
can be as easy as running make or git commit.

17 July 2013

Peter Eisentraut: Tricky shell local variables

I have a word of warning against improper use of local in shell functions. If you are using shell functions, you might want to declare some variables local to the shell function. That is good. The basic syntax for that is
local a b c
In some shells, you can also combine the local declaration and assignment, like this:
local foo=$1
local bar=$2
(The Debian policy even explicitly allows it.) This is somewhat dangerous. Bare shell assignment like
foo=$bar
does not perform word splitting, so the above is safe even if there are spaces in $bar. But the local command does perform word splitting (because it can take multiple arguments, as in the first example), so the seemingly similar
local foo=$bar
is not safe. This can be really confusing when you add local to existing code and it starts breaking. You can avoid this, of course, by always quoting everything to like
local foo="$bar"
but overquoting isn't always desirable, because it can make code less readable when commands are nested, like
local foo="$(otherfunc "other arg")"
(Nesting is legal and works fine in this case, however.) I suggest using local only for declaring variables, and using separate assignment statements. That way, all assignments are parsed in the same way.

3 April 2013

Peter Eisentraut: Installing multiple PostgreSQL versions on Homebrew

I was going to post this yesterday, but some might have thought that it was a joke. April 1st is always an annoying day to communicate real information. If you have been fond of the way Debian and Ubuntu manage multiple PostgreSQL versions in parallel, you can now have the same on OS X with Homebrew:
brew tap petere/postgresql
brew install postgresql-9.2
# etc.
brew install --HEAD postgresql-common
postgresql-common is the same code as in Debian, only mangled a little. Now you have all the client programs symlinked through pg_wrapper, and you can use the server management tools such as:
pg_createcluster 9.2 main
pg_ctlcluster 9.2 main start
pg_lsclusters
Let me know if you find this useful. Links:

14 August 2012

Peter Eisentraut: Funny version numbers

Often, I install a new Debian package using apt-get install, and as the progress output flies by, I wonder, Whoa!, should I really be using a piece of software with that version number? It says a lot, after all. If I see
tool 2.4.1-2
then I (subconsciously) think, yeah, the upstream maintainers are obviously sane, the tool has been around for a while, they have made several major and minor releases, and what I'm using has seen about one round of bug fixing, and a minimal amount of tweaking by the Debian maintainer. On the other hand, when I see
tool 7.0.50~6.5~rc2+0.20120405-1
I don't know what went on there. The original release version 7.0.50 was somehow wrong and had to be renamed 6.5? And then the 2nd release candidate of that? And then even that wasn't good enough, and some dated snapshot had to be taken? Now, of course, there are often reasons for things like this, but it doesn't look good, and I felt it was getting out of hand a little bit. I tried to look into this some more and find a regular expression for a reasonably sane version number. It's difficult. This is how far I've gotten: https://gist.github.com/3345974. But this still lists more than 1500 packages with funny version numbers. Which could be cause for slight concern. Take a look at what this prints. You can't make some of that stuff up.

18 July 2012

Peter Eisentraut: Tracing shell scripts with time stamps

A random tip for shell script hackers. You know that with set -x you can turn on tracing, so that every command is printed before being executed. In bash, you can also customize the output prefix by setting the PS4 variable. The default is PS4='+ '. Here is an example. I wanted to "profile" a deployment script, to see why it took so long. Ordinarily, I might have sprinkled it with date calls. Instead, I merely added
set -x
PS4='+\t '
near the top. \t stands for time stamp. (The script was already using bash explicitly, as opposed to /bin/sh.) That way, every line is prefixed by a time stamp, and the logs could easily be analyzed to find a possible performance bottleneck.

16 May 2012

Christian Perrier: tar taf

Hey, thanks, Peter for the hint! Never heard of that option until you blogged about it. So, now I'm also ready to "tar taf", "tar xaf", "tar caf"..:-). Harder for me than you because I was used to "tar tfz" or "tar tfj"..:-) And I suspect that "tar taf" is prone to typos...we'll see.

14 May 2012

Peter Eisentraut: Time to retrain the fingers

For years, no decades, I've typed tar tzf something, tar xzf something. Except when someone annoying sent an uncompressed tar file and I had to then go and take out the z in the middle. Then came bzip2, and we learned tar tjf, tar xjf. OK, I could live with that. One emerging problem was that the tab completion now worked the wrong way around conceptually, because you had to pick and type the right letter first in order to see the appropriate set of files to unpack offered for completion. Then came lzma, which was (quick, guess?), tar tJf, tar xJf. And then there was lzop, which was too boring to get its own letter, so you had to type out tar -x --lzop -f. But lzma was short-lived, because then came xz, which was also J, because lzma was now too boring as well to get its own letter. Oh, and there is also the old compress, which is Z, and lzip, which I'd never heard of. But stop that. Now there is
 -a, --auto-compress
use archive suffix to determine the compression program
This handles all the above compression programs, and no compression. So from now on, I always use tar taf and tar xaf. Awesome. The finger movements will be almost the same on QWERTY and AZERTY, and easier than before on QWERTZ. Actually, this option is already four years old in GNU tar. Funny I'd never heard of it until recently.

29 April 2012

Peter Eisentraut: Setting the time zone on remote SSH hosts

The tasks: I have one or more desktop/laptop machines with varying local time zones (because the persons using them are actually in different time zones, or because the one person using them travels). I also have a number of servers configured in some random time zones. (It could be the time zone where they are physically located, or the time zone of the person who installed it, or UTC for neutrality.) Now what I would like to have happen is that if I log in using SSH from a desktop to a server, I see time on that server in my local time zone. For things like ls -l, for example. Obviously, this illusion will never be perfect. Nothing (except something very complicated) will adjust the time stamps in the syslog output, for example. But the ls -l case in particular seems to come up a lot, to check how long ago was this file modified. This should be completely doable in principle, because you can set the TZ environment variable to any time zone you like, and it will be used for things like ls -l. But how do you get the TZ setting from here to there? First, you have to make the remote SSH server accept the TZenvironment variable. At least on Debian, this is not done by default. So make a setting like this in /etc/ssh/sshd_config:
# Allow client to pass locale environment variables
AcceptEnv LANG LC_* TZ
You also need to make the equivalent setting on the client side, either in /etc/ssh/ssh_config or in ~/.ssh/config:
SendEnv LANG LC_* TZ
Which leaves the question, how do you get your local time zone into the TZ variable to pass to the remote server? The actual time zone configuration is the file /etc/localtime, which belongs to glibc. In current Debian, this is (normally) a copy of some file under /usr/share/zoneinfo/. In the distant past, it was a symlink, which would have made things easier, but now it's a copy, so you don't know where it came from. But the name of the time zone is also written to /etc/timezone, so you can use that. The format of the TZ environment variable can be found in the glibc documentation. If you skip past most of the text, you will see the relevant parts:
The third format looks like this: :CHARACTERS Each operating system interprets this format differently; in the GNU C library, CHARACTERS is the name of a file which describes the time zone.
So what you could do is set
TZ=":$(cat /etc/timezone)"
Better yet, for hopping through multiple SSH hosts in particular, make sure to preserve an already set TZ:
TZ=$ TZ:-":$(cat /etc/timezone)" 
And finally, how does one hook this into ssh? The best I could think of is a shell alias:
alias ssh='TZ=$ TZ:-":$(cat /etc/timezone)"  ssh'
Now this set up has a number of flaws, including: But it practice it has turned out to be quite useful. Comments? Improvements? Better ideas? Related thoughts:

13 April 2012

Peter Eisentraut: Enabling core files for PostgreSQL on Debian

The other day, I was a bit puzzled over a seemingly simple task: Enable core files to be generated from a PostgreSQL instance running on Debian. That instance has unfortunately been segfaulting on occasion, but never left a core file.

Now in principle it is clear that
ulimit -c unlimited
is the incantation to get this done. But where do you put this? You could hack it into the init script, but that seemed a bit ugly, and I wanted a sustainable solution.

A useful thing in the meantime is to check the current settings. That information is available in /proc/$PID/limits with the PID of the postmaster process (or any child process, really), and it looked like this to begin with:
Limit                     Soft Limit           Hard Limit           Units     
Max cpu time unlimited unlimited ms
Max file size unlimited unlimited bytes
Max data size unlimited unlimited bytes
Max stack size 8388608 unlimited bytes
Max core file size 0 unlimited bytes
...
Use sudo grep core /proc/$(sudo cat /var/run/postgresql/8.4-main.pid)/limits if you want it automated.

So it's good to know that we only need to set the soft limit.

One way to configure this properly would appear to be in /etc/security/limits.conf. There you can add a line like
*               soft    core            unlimited
to enable core dumps globally. I'm not actually sure whether that would work if the service is started during the boot without PAM. In any case, I didn't want to enable core files globally; who knows what that would lead to.

One could replace the * by a user name, such as postgres, and then enable pam_limits.so in /etc/pam.d/su. But the postgresql init script in Debian is nested about four levels deep, so it wasn't clear whether it called su at all.

Now as it turns out, the init script ends up changing the user using this Perl code:
$) = $groups;
$( = $gid;
$> = $< = $uid;
(see change_ugid in /usr/share/postgresql-common/PgCommon.pm), so the whole PAM line of thought wasn't going to work anyway. (Other packages such as pgbouncer and slony1 do got through su, so that would be a solution for those.)

The way to solve this is the pg_ctl -c option, which sets the soft limit for core files to unlimited. And the way to pass this option through the init script maze is the file /etc/postgresql/8.4/main/pg_ctl.conf, which should contain a line like this:
pg_ctl_options = '-c'
Then restart postgresql, and check /proc/$PID/limits again:
Limit                     Soft Limit           Hard Limit           Units     
Max cpu time unlimited unlimited ms
Max file size unlimited unlimited bytes
Max data size unlimited unlimited bytes
Max stack size 8388608 unlimited bytes
Max core file size unlimited unlimited bytes
OK.

Another thing that's recommendable in this context is to change the core file names to have a unique element, so that if multiple backends crash before you can take a look, they don't overwrite each other's core files. The core(7) man page explains the configuration options; I went with this sysctl setting:
kernel.core_pattern = core.%e.%p
which includes process name and PID. The PID file still ends up in the data directory of the PostgreSQL instance, which could also be changed, but I didn't find it necessary.

Stick the above line in /etc/sysctl.d/local.conf and reload with
service procps force-reload
I actually use a setting like that on all machines now; it's just nicer.

OK, and now I'll wait for the next core file. Or not.

23 November 2011

Peter Eisentraut: git whoami

My favorite feature in bzr (Bazaar) is the bzr whoami command, which prints what your current identity (name and email) is, as far as the repository is concerned. You can tell I haven't used bzr much if that's as far as I have gotten. But seriously, with so many Git repositories around, several project identities, directory-specific shell configuration, and so on, it's easy to get confused, and it's annoying to have to check and repair commits for correct user name and email all the time. So here is git whoami. This has already saved me so many headaches.

12 November 2011

Peter Eisentraut: Switching desktop environments

So, like many people, I have received GNOME 3 in the latest upgrade on Debian testing, and switched away to Xfce very soon after. What's amazing (to me) about this whole experience, however, is how easy it is to switch desktop environments in Debian. Two years ago, I fled KDE in very similar circumstances. I have since used Openbox, LXDE, GNOME, Xfce, as well as a few more obscure window managers in short-lived experiments. All of this was easy to do and worked flawlessly. Most of my settings, my preferred programs, and startup applications were preserved, and if I didn't like it, I could switch back without a problem. (Ironically, the upgrades from KDE 3 to KDE 4 and from GNOME 2 to GNOME 3 lost far more settings than changes to other environments.) And of course you can mix and match applications from different environments, and these days this doesn't result in a visual crime. I don't know whom to thank for this, probably a combination of upstream developers, freedesktop.org guys, and Debian maintainers. Keep it up.

16 September 2011

Peter Eisentraut: ccache and clang, part 2

There's more funny business when using ccache in combination with clang. Last time I suggested that you use the invocation
./configure CC='ccache clang -Qunused-arguments -fcolor-diagnostics'
to get rid of the "argument unused during compilation" warnings.But you still might get loads of warnings that you wouldn't normally get without ccache, such as this example (from the PostgreSQL source code):

extension.c:382:35: warning: equality comparison with extraneous parentheses [-Wparentheses]
if (( (((control->directory)[0]) == '/') ))
~~~~~~~~~~~~~~~~~~~~~~~~~~^~~~~~
extension.c:382:35: note: use '=' to turn this equality comparison into an assignment
(This is the opposite of the warning that tells you to put two pairs of parentheses around an assignment used as a truth value.)Or:

path.c:727:7: warning: explicitly assigning a variable of type 'char *' to itself [-Wself-assign]
path = (path);
~~~~ ^ ~~~~
The problem is, these come from macro expansions, so wouldn't normally see them, because (I guess) the compiler driver is smart enough not to warn about such things when they come from macro expansions.The way ccache works is approximately
  1. preprocess the input file
  2. look for it in the cache
  3. if not found, compile the preprocessed file
What would be better in this situation is
  1. preprocess the input file
  2. look for it in the cache
  3. if not found, compile the original file
And indeed you can turn on that second behavior by setting the obscure environment variable CCACHE_CPP2 (as in, run cpp twice):
export CCACHE_CPP2=yes
Then all these extra warnings disappear.(The ccache man page is worth a read. There are a few interesting settings to play with.)I'm currently playing around with a shell script ccache-clang that looks like this:
CCACHE_CPP2=yes exec ccache clang -Qunused-arguments  test -t 2 && echo -fcolor-diagnostics  "$@"

23 June 2011

Peter Eisentraut: Bootstrapping Debian packages

I read something about dh_make again a while ago, which made me wonder, besides dh_make, how do people generally bootstrap new Debian packages. I don't think I've ever actually used dh_make for anything but experimenting. For the longest time, copying over the packaging files from some similar package worked pretty well.

Recently, however, I have applied a completely new strategy: I start with nothing, call debuild, fix the error, retry, until it works. This might sound pretty bizarre, but it works pretty well and enhances your understanding of how the packaging tools work.

Here is about how it works:

Start with nothing, call debuild, it complains:
cannot find readable debian/changelog anywhere!
Call dch --create, which says:
Cannot find debian directory!
OK, mkdir debian, call dch --create again, fill in the template. Call debuild again, now it says:
but there's no debian/rules there!
Create a debian/rules, that's easy to do by heart nowadays, at least as a start:
#!/usr/bin/make -f

%:
dh $@
Call debuild again, now dpkg-buildpackage actually starts, but stops with:
dpkg-source: error: cannot read mypackage/debian/control: No such file or directory
At this point I'm too lazy to figure out what is necessary to put into debian/control, so let's just keep it simple and touch debian/control. At this point dpkg-source gets into a bit of trouble:
Use of uninitialized value $sourcepackage in sprintf at /usr/bin/dpkg-source line 290.
So let's put that in and also follow the subsequent advice to add debian/source/format and the Maintainer and Standards-Version fields. So far we have:
Source: mypackage
Maintainer: Peter Eisentraut <petere@debian.org>
Standards-Version: 3.9.2
The next hint of a message is:
dh: No packages to build.
This means we need a binary package stanza, so I'll just add
Package: mypackage
Architecture: any
Now debhelper complains:
dh: Compatibility levels before 5 are deprecated.
Not sure why it didn't complain about that earlier. Let's stick 8 in there.

At this point I had to do actual work and mess around with debian/rules a bit to get the package to actually build, but a few minutes later I had a functioning provisional package.

The next step is to clean up the warnings from the various dpkg-* workers:
dpkg-gencontrol: warning: missing information for output field Description
dpkg-gencontrol: warning: package mypackage: unused substitution variable $ shlibs:Depends
dpkg-deb: warning: parsing file 'debian/mypackage/DEBIAN/control' near line 6 package 'mypackage': missing description
dpkg-genchanges: warning: missing Section for binary package mypackage; using '-'
dpkg-genchanges: warning: missing Priority for binary package mypackage; using '-'
dpkg-genchanges: warning: missing Section for source files
dpkg-genchanges: warning: missing Priority for source files
So we add Description, Depends, Section, and Priority.

And finally we have a list of complaints from Lintian to address:
W: mypackage source: debhelper-but-no-misc-depends mypackage
E: mypackage source: package-uses-debhelper-but-lacks-build-depends
W: mypackage source: package-needs-versioned-debhelper-build-depends 8
W: mypackage source: debhelper-overrides-need-versioned-build-depends (>= 7.0.50~)
E: mypackage source: temporary-debhelper-file debhelper.log
E: mypackage: no-copyright-file
The only things I added manually after that were Vcs-*, Homepage, and Enhances.

Now the only things left to do are running the thing through cowbuilder a few times and putting in all the necessary build dependencies, and writing a nice changelog entry.

Note, this method does not replace putting in some thought. But it's an interesting way to get a relatively clean package.

5 May 2011

Peter Eisentraut: ccache and clang

Just a note for the Internet: When you use ccache and clang together, you will probably get a lot of warnings like these:
clang: warning: argument unused during compilation: '-c'
clang: warning: argument unused during compilation: '-I .'
These are harmless, but if you want to get rid of them, use the clang option -Qunused-arguments, which will hide them. (The first one is already fixed in ccache.)

The reason for this is that ccache splits the compilation into separate calls to the preprocessor and the compiler proper, and it tries to sort out which of the options that you called it with go with which call. But since gcc doesn't complain about passing -c to the preprocessor or -I to the compiler, ccache doesn't bother about sorting this out (bug). That's why you don't lose any information relative to using gcc if you use the -Qunused-arguments option.

Also, if you like clang's colored diagnostics messages, you'll have to turn them on explicitly with -fcolor-diagnostics, because when running through ccache, clang doesn't think it's printing to a terminal and turns off the color by default.

So a complete invocation might look like this:
./configure CC='ccache clang -Qunused-arguments -fcolor-diagnostics'

Next.