dnsh.io

Why Mass Surveillance Is Counter Productive

Sat, 19 Nov 2016 00:00:00 +0000

Last Thursday(17th Nov 2016), The UK Parliament approved a bill called Investigatory Powers Act that legalise a plethora of digital spying activities by government agencies. This bill is also known as “The Snoopers Charter” and grants following powers:

Equipment interference by government agencies becomes legal

Upon obtaining a warrant, government agencies may hack into computers, networks, mobile devices, servers or any other connected or offline device. This would allow government agencies to do things like:

download data from a mobile phone at that is stolen on left unattended
install intrusive software that track every key pressed on a laptop.

While this is nothing new, it is the first time the practice has been made legal in a western democracy. Given that a warrant is required before any attacks, most people would be fine with this law. But…

Bulk hacking and collecting bulk data sets also becomes legal

This is where things start to get too intrusive. “Bulk hacking” grants government agencies to perform proximity attacks on networks and devices in bulk, and almost indiscriminately. For example, it would become legal for government agents to hack your company’s network if they think one of your colleagues is emailing ISIS. Not only that, it’s also legal for them to keep whatever data they might find about YOU even though you had nothing to do with it.

IP Bill and web logs

Internet service providers are now obliged to store internet history data for all their customers for a minimum of 12 months. Internet history data logs will be made up of websites that you may have visited and devices that you have connected with. Government agencies are to be given full access to all of this data without any need for a warrant.

OK so what? What’s wrong with all this?

Let’s not worry about equipment interference since this is quite similar to other breaking and entering laws that many think is fair. However, very few would find indiscriminate attacks on private data to be fair.

But I’m not fussed about the fairness of this bill, actually. What I’m concerned about is how ineffective are these policies at doing what the government claim they do: counter-terrorism and provide security. Before going further, let me introduce some security engineering jargon:

Policy (or the objective goal)
Mechanism (ciphers, access controls, hardware tamper-resistance and other machinery that you assemble in order to implement the policy)
Assurance (the amount of reliance you can place on each particular mechanism)
Incentive (the motive for people to either guard or try and defeat the policy)

Snoopers Charter is a collection of ineffective policies backed by weak mechanisms brought forward to give the illusion of assurance.

These policies are ineffective mainly due to weak mechanisms. IP bill can be defeated quite easily by proxying through a VPN. Someone who has a vested interest in maintaining anonymity and communicating secretly can still do that quite easily. You don’t need to be a genius to learn how to use Tor browser or PGP encryption.

So what’s actually happening is that criminals and terrorists can go about their business without having to change anything. ISPs will have to spend millions of pounds to collect and store weblogs. Surely, this is a cost they will just pass on to their customers. Digital intrusion is a paid service, you see.

But what is dangerous about this is that there will no doubt be a demand in the digital blackmarket for all this web-log data. Yahoo were the most recent company to come out clean about how successful they were at securing their customer data. When it comes to privacy, it’s very difficult to put the genie back in the bottle. So the fewer the data that is stored about you, the better your privacy.

Since the 9/11 attacks the Transport Security Administration in USA has spent $14.7 billion on aggressive passenger screening, which is fairly ineffective since well below half of the weapons taken through screening (accidentally or for testing purposes) were picked up, while $100m spent on reinforcing cockpit doors would remove most of the risk.

String concatenation in Rust

Thu, 06 Oct 2016 00:00:00 +0000

So I’ve been messing with Rust language for a couple of weeks now and I am thoroughly enjoying it. Rust is a systems programming language developed by Mozilla foundation. It’s becoming popular with plenty of rave reviews about seg-fault safety and the modern syntax. I highly recommend taking a look if you haven’t already.

Rust is obsessed about memory from ground up. This is understandable given that avoiding seg-faults is one of its unique selling points. When you write Rust, you are contantly thinking about where things are in memory. Sometimes that makes it very difficult to do some of the most trivial things in programming. Take string concatenation for example.

This is how you do it in Javascript:

var x = "Hello ";
var y = "World";
var xy = x + y;

And in PHP:

<?php
$x = "Hello ";
$y = "World";
$xy = $x . $y;

So you would think that the following would work in Rust:

let x = "Hello ";
let y = "World";
let xy = x + y;

That doesn’t compile. This is because Rust has 2 types of strings. &str and String. Both x and y are of type &str. These are called string literals, string slices or static strings. They have a fixed size and cannot be mutated.

A string literal is a string slice that is statically allocated, meaning that it’s saved inside the compiled program, and exists for the entire duration it runs. The reference binding is a reference to this statically allocated string. Any function expecting a string slice will also accept a string literal.

So what about String? Strings are allocated on the heap and can be mutated. It’s essentially a vector of unicode chars. Strings are generally created from a &str by using to_string() function. So lets try again:

let mut x = "Hello ".to_string();
let mut y = "World".to_string();
let xy = x.push_str(y);

That doesn’t work either. It’ll give you expected &str, found struct std::string::String. Ok. So how about this then:

let mut x = "Hello ".to_string();
let mut y = "World".to_string();
let xy = x + y;

Again, the compiler will throw the same error as before. So in order to get around it, you have to dereference one of the Strings to obtain the &str version of it. This is done easily by prefixing an ampersand to the second argument, which essentially dereferences the String and passes a pointer of the string slice to the concat function. So the working version of the previous example would look like this:

let mut x = "Hello ".to_string();
let mut y = "World".to_string();
let xy = x + &y;

If you are making a call to a function that’s returning a String and you want to concatenate it to an existing string then you just add the ampersand to the beginning of the function call.

fn get_world() -> String {
    "World".to_string()
}

fn main() {
    let mut x = "Hello ".to_string();
    let mut xy = x + &get_world();
}

But WTF? why do you have to do that?

The simple answer is because how concatenation is done. When you are using the + operator for Strings in Rust, you are essentially calling the std::ops::Add Trait on the String on the left hand side. And this Trait is designed so that the left hand side argument should be String and the right hand side argument should be an &str.

So now we understand that this is because of a particular design decision of the standard library. But why? Why not just allow for two arguments like add(a: String, b: String) -> String?

This is basically down to how and where things are stored in the memory and the operational cost of mutating them.

impl<'a> Add<String> for &'a str requires prepending, which is not as efficient as appending
impl Add<String> for String needlessly consume two arguments when one is sufficient
impl<'a, 'b> Add<&'a str> for &’b str hides an unconditional memory allocation

Label centric music discovery

Sun, 02 Oct 2016 00:00:00 +0000

Music is big business. In fact it’s a business that entertained $15bn (USD) from global record sales alone. 45% of this $15bn was generated through digital sales channels. So it would be easy to make the assumption that digital sales are on the up because digital consumption of music is so much easier and cheaper for the consumer.

But as it is always the case with statistics, numbers doesn’t always paint the full picture. Sure it’s easier now than 15 - 20 years ago to go and buy a number off the top 10; but if you venture 1 degree from the mainstream taste, you’ll find yourself spending hours clicking through music (as opposed to digging in the past - I’ll come to that too) trying to find the perfect tuna.

Let me describe the struggle in detail: You hear this incredible piece of music somewhere and you get instantly hooked. You’ve never heard this stuff before (and you blame yourself for it) and you are sure hearing this sound has changed your life. Now you HAVE to spend your next two weeks listening to some more of the same. In the most ideal situation, you heard this music while you were sitting in your friend’s sofa and you interrogate him until he gives up all his knowledge of that piece. And in less than ideal situations, you hear it while you’re out and about - in a pub or a festival. If you are really lucky, one of your friends (or Shazam) would have successfully id’ed the track. Great - but you want more of the same. Which finally brings us to the topic of this article.

More of the same please!

You managed to find out the name of the artist and the name of the song. This is the beginning of your hunt. You do your basic searches on Google, or apps like Shazam will even attempt to recommend similar. This might yield some decent results depending on the popularity of the type of music that you are hunting. Underground music tends to get the least amount of exposure (for obvious reasons). Things like Spotify can be very useful in your road to new music discovery because it allows you to listen to a incredibly large collection of recorded music. But it falls short on the discovery bit.

When you know the artist and the song, you can try and listen to more from the same artist or find other artists that they collaborated with or even try “similar artists” recommended by Spotify in your quest to find more of that flavour. Sometimes this doesn’t work because the the song that you heard from that artist may have been a special release and the rest of their releases are a completely different genre altogether (yes, this happens - quite a lot in niche genres).

Algorithmic music discovery doesn’t quite work for creative content because art is often varied and doesn’t always follow a consistent and a predictable pattern.

The thing is, the music industry had been around for a long time and this problem had already been solved. Algorithmic music discovery doesn’t quite work for creative content because art is often varied and doesn’t always follow a consistent and a predictable pattern. So human curation is such a big part of music discovery process. But it’s a lot of work to manually curate and categorise all this amount of digital content that you would think this is not an economically viable option. But in fact, this is exactly how underground music is pushed.

Piggy back on the work of the labels

Record labels often times act like human curators. They work with artists during the recording and release process and the label themselves decide to release music through their brand that they can identify with. So check out what label released that song that you really liked. If it’s not one the major labels, then there’s a good chance that you will like most of the content released by that label. Hunting this way will introduce you to new artists that you would otherwise wouldn’t know to look up.

While you are doing this, you can use Spotify’s Advanced Search functionality to filter by labels. Just typing in the label name in search wouldn’t get you any results on Spotify. In fact, labels aren’t even top level entities like Artist, Albums and Songs. I think this is a mistake from Spotify. So you have to use Advanced Search.

So to filter by label, type in label:"Name of the Label" in to Spotify’s search bar. The example below shows how to find all releases made by the record label 1080p:

label:"1080p"

Go out and try it and zero in on the music you reeeeaaaalllly want.

PHP 7.0 short-change

Fri, 23 Sep 2016 00:00:00 +0000

So, It’s been almost a year now since PHP 7 had been released. I have to say that regardless of the tone of this article’s title, I am over the moon about this major version. I think it’s really come a long way from the old “Personal Home Page” laughing-stock. And I’m not even talking about the PHP 7 vs Python 3 benchmark game.

No, I mean it had actually become a decent language and I am not that sad about having to work with it day in day out. I sometimes even admit at parties to being a PHP developer.

PHP 5.6 was already pretty good. We had short syntax arrays, consistent and standard compliant JSON serialization/de-serialisation, Traits, Reflection and so on. So the big thing apart from all the we’re faster than python performance improvements has to be the inclusion of optional typing for scalars (again, PHP 5.6 fell a little short on this one).

I love it! Now I don’t have to spend time type checking at the beginning of every method like we used to:

<?php
function multiply($x, $y)
{
  if (!is_numeric($x) || !is_numeric($y)) {
    throw new \InvalidArgumentException("Both X and Y has to be a number");
  }
  return $x * $y;
}

Look at that shit! Anyone who says dynamic typing makes it easier to write code probably doesn’t type check their input arguments. In the above example, 3 lines of code went in to type checking and the actual business logic was a one-liner. So I’m glad that PHP 7 allows us to type hint scalars. Another bonus is that we can now declare the return type as well. This now cuts down the amount of code on the calling method. Check what the same example looks like on PHP 7:

<?php
function multiply(int $x, int $y) : int
{
  return $x * $y;
}

Beautiful! And this is great when taking two integers and returning one. But people pay me money for my code so life is never that simple. I have to do things from time to time that isn’t discussed in the release notes. Like return null sometimes (or all the time for that matter). Check this:

<?php
function getOneLemon() : Lemon
{
  if (count($this->lemons)) {
      return $this->lemons[0];
  }
  return null;
}

Looks pretty reasonable but the code above will throw a fatal because you promised to return a Lemon but sometimes you can’t. One way around it was to get rid of the return type declaration and maybe describe the return types in the docblock so your IDE knows what you are up to. I mean… we just lost all the power that this feature was supposed to give us.

Another way was to throw an exception when you can’t return the type you promised. This is a little bit better but if you really think about it, you are relying on a construct designed to signal failures and errors as a part of the logic flow. This makes me uncomfortable. If I go to a shop and ask for some sugar I don’t want the shop keeper to start shutting down the shop when he notice that he doesn’t have any sugar to sell. No is a perfectly acceptable answer.

But there is light at the end of the tunnel. An RFC was accepted and this issue is ironed out in the next minor version upgrade (7.1) - along with a heap of other improvements. The official released is set to be in Nov this year. So in 7.1 the last example would look like this:

<?php
function getOneLemon() : ?Lemon
{
  if (count($this->lemons)) {
      return $this->lemons[0];
  }
  return null;
}

Boom. The question mark in front of the Lemon indicates that it’s optional. That means you can return null as well as Lemon (but no other type). Pretty solid. Another thing - you can also return a new void type. Looking forward to that!

PhpSpec is a bloody nightmare

Thu, 22 Sep 2016 00:00:00 +0000

Where I work, we use PhpSpec for unit-testing our code. One might argue that these should not be called unit tests since you write specifications rather than tests. Call it what you will - we can have that argument another day. Today I will rant about the good and some of the bad things about it.

In a nutshell, PhpSpec helps bring the TDD buzzword to the PHP world. By allowing you to mock shit straight off the bat, PhpSpec makes it difficult to come up with excuses like “I wrote the feature but I haven’t written any tests yet”. Again, I’m not pissed off about that.

In fact, I actually like it because I’m lazy and PhpSpec generates shit for me. I just make grand statements like “Oh, I got a Some\Bullshit. And it takes a CrunkJuiceFactory as a constructor argument. And I can getJuice() from CrunkJuiceFactory anytime I want and it’ll give me n number of juice hits”. But I haven’t written any of that behaviour yet. Better get that scaffolded… So I do:

bin/phpspec desc Some/Bullshit

Then PhpSpec just makes a specification for you with nothing much in it but you can’t argue with that since you haven’t done any work yet yourself. But you don’t care. This is TDD so you run the test anyway:

bin/phpspec run spec/Some/BullshitSpec.php

Oh look! PhpSpec wants to help you create the actual test subject!

Some/Bullshit
  10  - it is initializable
      class Some\Bullshit does not exist.

1 specs
1 example (1 broken)
36ms

  Do you want me to create Some\Bullshit for you? [Y/n]

You take Y, and spec gives you your new file and you are ready to write some code. At this point, I know the IDE warriors are up on my case pointing out that one could achieve the same with 3 clicks and you don’t have to keep chatting to a terminal. You are correct. But that’s not where Spec saves you. PhpSpec gives you the ability to focus on TDD, writing your subject and mocking the behaviour of its dependencies right from the start.

For example, you can just scaffold out CrunkJuiceFactory object just like your Bullshit object. And then you can mock return values of methods in CrunkJuiceFactory so that you can just focus on writing the actual behaviour of Bullshit:

<?php
namespace spec\Some;

use PhpSpec\ObjectBehavior;
use Prophecy\Argument;

class BullshitSpec extends ObjectBehavior
{
    public function let(CrunkJuiceFactory $cjFactory) {
        $cjFactory->getJuice()->willReturn(5);
        $this->beConstructedWith($cjFactory);
    }

}

You just noticed that you started describing behaviour of CrunkJuiceFactory and making claims about what it will do before you even thought about implementing it. This is great for productivity because you don’t have to write both classes in parallel.

But how does that even work?? This witchcraft makes you productive but unless you understand the underlying mechanics of the framework, you will run in to problems. Essentially, PhpSpec makes heavy use of another buzzword. Reflection.

Reflection allows PhpSpec to get a description of the object and the dependencies that you are trying to test. And using this information, PhpSpec creates stunt doubles of your objects which it pushes through your test subject. And this stunt double essentially intercepts calls made from your test subject to it and then returns values that you have mocked. This is what you did when you described $cjFactory->getJuice()->willReturn(5);.

All this stuff sounds great! Why the damn rant? Well, the problem is that this makes it very difficult to test your tests. Spec gets weird when you are testing some code that you have wrapped around a try .. catch block.

If you cause a PhpSpec error, inside a try catch block in your subject (like making a call to a method that you didn’t describe in the spec) then you are going to spend hours trying to figure out why PhpSpec keep running forever and eventually die without giving you any bacon. Or anything for that matter.

Let me explain better:

<?php

public function getBacon()
{
    try {
        $bacon = $this->baconFactory->getSome();
    } catch (\Exception $e) {
        $this->logger->critical("OUT OF BACON!!!! :O");
    }
}

In the above scenario, if you had forgotten to mock getSome() method in your BaconFactory object inside your spec then PhpSpec will throw an Exception right where that ‘Undefined method’ was called. This means that the exception gets caught by your subject and not PhpSpec itself. And then PhpSpec hangs. And you have no idea WTF just happened. Nothing in the terminal. Just hangs. If you ran it with php -d memory_limit=-1 then it hangs even longer. Possibly until the end of the world.

So how do you deal with this?

If you come across this sorry situation in your life, try and narrow down exactly which spec is hanging. And then you will probably notice that your subject has a try catch block. Then stick a var_dump in the catch block with the exception message and die. It’ll show you the message on your terminal so you can move forward.

<?php

public function getBacon()
{
    try {
        $bacon = $this->baconFactory->getSome();
    } catch (\Exception $e) {
        var_dump($e->getMessage());die;
        $this->logger->critical("OUT OF BACON!!!! :O");
    }
}

Oh, and don’t forget to remove the die statement once you are done debugging.