Formic 0.9beta8 is out

This release introduces a new feature and fixes a couple of annoying bugs. Note that the semantics for ** were incorrect in previous versions. Get it at http://www.aviser.asia/formic.

New feature:

Added the ability to pass in the function that walks the directory path, which allows for alternate implementations or supplying a mock function that provides values completely unrelated to the OS. This is available only from the API and not from the command line::

      files = ["CVS/error.py", 
               "silly/silly1.txt", 
               "1/2/3.py", 
               "silly/silly3.txt", 
               "1/2/4.py", 
               "silly/silly3.txt"]
      for dir, file in FileSet(include="*.py", walk=walk_from_list(files)):
          print dir, file

Bug fixes:

Fixed #10: Paths like //network/dir caused an infinite loop
Fixed #11: Incorrect handling of globs ending /** and /. Ant Glob semantics for::

      **/test/**

are that they should match "all files that have a test element in their path, including test as a filename."

Locked out of my HTC ONE after full device encryption!

I bought a new HTC ONE yesterday, and, boy, is it lovely! The speakers and the screen are simply gorgeous and the camera massively out-performs my old Galaxy SII's. I sincerely hope that HTC can capitalise on the ONE and get share back from Samsung - it's far more interesting when there is tough competition.

Device encryption

I have to admit my particular OCB: I'm paranoid on data and security. I back up my data regularly and ensure it's pushed to another, different locations. I have Avast! installed to catch malware and track my phone if it's lost or stolen. I auto-upload the photos and videos I take. I have a PIN screen lock. Today I went the whole hog with full device encryption. Under the covers, Android is using the tried and tested Linux dmcrypt module, so this should be a slam-dunk.

The process was simple if rather lengthy. One hour later and after multiple device reboots, the ONE prompted me for the decryption PIN. That entered, I held my breath as the reboot continued. At last the lock screen appeared. Home and dry!

Impenetrable lock screen

But no. The lock screen keyboard are somehow switched during the device encryption. I was greeted with:

Locked out! Sorry for the poor quality, I was panicking...

Panic - there's no numbers! I checked out page 2 of the keyboard - still a noticeable lack of arabic numerals. What is "HI"? No, that makes things worse. What about the keyboard button down there on the left? Ah, it makes the keyboard disappear. Let's not do that again. What am I going to do? The obvious answer is a factory reset followed by several hours of reconfiguration. Bugger - tedious, repetitive but doable.

Ruling out a factory reset

Then the ONE started cheekily beeping: "Hello," it was saying, "I've got an SMS for you when you log in". "Bing bing! Here's another!" But how was I to read it?

While I was still digesting this, the damn thing woke up again: "Bing bing! You've got a third SMS! Your lucky day".

I rebooted twice - a soft and then hard reboot. On restart I was still presented with a keyboard without the keys I needed.

Now, a factory reset was out of the question. I would lose those SMSes and never be able to find out who was messaging me.

HTC Customer Service

I called HTC Customer Service. Well,  I use the word "Service" in the broadest and possibly incorrect sense. Reboot was suggested, but discarded because I'd already trudged this path. Only after this did the agent appear to want to understand the situation at all. Then I had 5 minutes of classical music (to better soothe away the pain) while the agent sought help from his seniors. Or to Google, take a personal break or whatever it is they do in these circumstances.

Ah, human again! "Sir, change the keyboard to a different locale". I reminded the agent that that was why I was calling them. Last option: Factory reset.

I suggested that this was quite a serious issue with their new flagship device and that it might be an idea to capture more details from me to help their customers. However, the agent had an hourly call target to achieve and the greater good was somebody else's problem. All problems can be cracked with either a factory reset or a ONE-for-ONE replacement, finesse be damned!

Voice of the Customer

One does not simply factory reset. Not when I could not reply to my mystery SMS sender, anyway. So what to do?

The Android keyboard does not let you change locale from the lock screen. Basic security, and basically a good design.

I had ensured that the developer options were off. There was no hope in connecting with adb and hacking it.

I was Googling, refining my searches for encryption, keyboards, locales and similar for 15 minutes. Other than attempting to sell me about 100 apps for encryption, replacement keyboards, lock screens and the like, there was no useful pointers at all.

I went from page to page in the keyboard, shifted, downshifted, press-and-holded every key I could - still no luck.

I called my phone from my landline to see if I could backdoor it while on the call. No luck.

Then I noticed it: The icon for a microphone buried on the second screen as a press-and-hold option.

After selecting this, a big warning from Google appeared. I had to accept an option acknowledging my (presumed) non-English accent could confuse Google's voice recognition engine. Then the Google Voice input box appeared. I spoke my PIN. If there was a tremble in my voice, Google's algorithm bested it without breaking a sweat. I was in!

Some concluding thoughts

I am documenting this in case you, dear reader, have the same issue and Google lead you here. I guess that it's some weird combination of the following:

• HTC ONE (or maybe just Android Jelly Bean)
• Using a lockscreen PIN
• Using the absolutely excellent Swift keyboard
• Then encrypting the device

If you are going to encrypt the device, I suggest switching off the lockscreen security for the duration.

If you are stuck, don't expect much from HTC Customer Service. But find the Google Voice icon. Thanks, Google, for saving me.

And the "HI"? My Indian friends are laughing at me. This means Hindi.

UPDATE 13th June:

This happens every time I reboot my ONE. Luckily the ONE is so stable I haven't realized until now.

Also, every reboot switches my Swift keyboard back to the HTC ONE's :(

I have another solution: I went into settings, selected 'Apps' and swiped into the 'ALL' tab. There I went down and disabled all the foreign keyboards so helpfully installed on the device. That includes Google Korean, Indic, Thai and Vietnamese. On reboot (I have to say I was extremely nervous. Back up your phone before doing this) I was presented with the default number pad for the screen lock. Victory!

Update 2 on DDoS

The Register reports that a man has been arrested in connection with the DDoS attack on SpamHaus - the one that involved a throughput of 300Gbps in one major Internet exchange.

Without commenting on the arrest itself, there are now more details on the DDoS. It's estimated that the attack used about 30,000 DNS resolvers to generate the traffic, so we can update the statistics:

While there may be 21m open resolvers, the attack used perhaps 30,000. Each open resolver was receiving an average of about 100kb/s traffic inbound and emitting an average of 10Mb/s. This is a lot more noticeable than the 15kb/s, but still not necessarily a big deal in a major data centre.

It did, however, trigger an alarm for Trevor Pott, the author of the Sysadmin blog on The Register. In his article, he reports:

The alarm went off late Tuesday night reporting DNS traffic of 10Mbit. 

My mistake stems from the simple assumption that BIND disables recursion by default. The change was made with BIND 9.4 way back in 2007. For reasons incomprehensible to me CentOS 5.9 ... is running BIND 9.3.6 which means that by default recursion requests are honoured. 

The fix required is simple ... I needed to ... instruct BIND to only honour recursion requests from servers inside my datacenter. 

But what of the future? Unfortunately, this attack is likely to be repeated. The Open DNS Resolver Project tracks open resolvers and has been reporting a steady growth in number. At the time of the attack, it estimated about 21m open resolvers. This week's survey has grown by another 4m. Cloudflare's future looks bright.

Update on DDoS

This is a follow-up to my previous post, and is itself followed up by "Update 2 on DDoS".

There's a great article on The Register about a recent, giant attack on Spamhaus, based on the Cloudflare blog. At its peak, for one hour, one of their upstream networks was receiving 300Gb/s of traffic - the highest ever reported for a DDoS.

Details below the fold...

Some thoughts on Distributed Denial of Service (DDoS) attacks

I recently answered a question on StackOverflow on how web applications could be protected from a DDoS. This was a popular question, and (for me) a very popular answer, and a few other members added their thoughts.  I have expanded on my post, done more research, and rewritten large portions.

UPDATE: Spamhaus was subject to an enormous DDoS in March 2013 and they turned to Cloudflare, a professional CDN I mentioned in this posting. I discuss the attack in a follow-up posting here and a second here.

What is a Distributed Denial of Service?

DDoS is a family of attacks which overwhelm key systems in the datacenter, disabling your web site and the web applications you are running on it. DDoS attacks are very common and carried out for a number of reasons:

• Political or ideological individuals, or groups as Anonymous
• Criminal groups using them either to:
• Extort money from business that don't know how or can't afford to protect themselves or
• Are paid to damage business competitors reputation or ability to trade on the web

The first DDoS attacks were very simple, but they have evolved over the years taking advantage of any weakness in the network protocols, firewalls, operating systems and web servers. However, at the most basic level, a DDoS can be indistinguishable from just a lot of users accessing your services - almost like a digital Occupy.

DDoS can attack any one (or indeed several) of the many different components of the application stack including:

• The hosting center's network connection to the internet
• The hosting center's internal network and routers
• Your firewall and load balancers
• Your web servers, application servers and database.

DDoS is just one way hackers can attack your systems and services. When securing your systems, you should consider defense against DDoS as just one aspect of your overall security posture.

There's a lot of information below the break.