Wednesday, July 29, 2015

Building a Click-Tracking System, Part the First

In my position as Special Projects Lead for, I have several interesting projects on my plate. They mostly fall under the analytics umbrella: onboarding of users, dynamic pricing, that sort of thing. The first one we decided to build is a custom click-tracking system.

In this series, I want to document the process of building said click-tracking system: what software was used, what I learned by using said software, problems I came up against, etc. Hopefully others can learn from my experience (and mistakes). So let's go...

Why a custom click-tracking system? 

Doesn't Google Analytics do that for you? There's plenty of other free web analytics available. Why go through the hassle? Yes, Google Analytics is very good at what it does, but it's not granular enough for our tastes. We want to do things such as determine how the system is being used ("Why do users take ten clicks to get through this process instead of four?"), how people move through the system ("Why did they go from the payment page, then to the about page, then never returned?") or through a page. Also we want to aggregate the raw data in ways that we're not aware of yet. Finally, playing with lots of data is fun! :-)

Lil Brother

When I develop solutions, I'm of the "Don't reinvent the wheel" school of philosophy, so naturally I went looking for software that already did what I wanted.  Most click-tracking software out there does what Google Analytics does, which is a fine thing but as discussed in the previous paragraph, they're not for us. Fortunately, I came across Lil Brother from the fine people at ShutterstockLil Brother "tracks clicks and other events in the browser and reports back in real time". Perfect!

While the documentation is sufficient I think there are some things that should be spelled out better and one or two things that are just plain typos, hence this document. So let's get technical!

First off, grab the code from their repo. There are two aspects to the code, the client-side and the server-side. Let's start with the client-side code.

Client-Side Code

The first thing you need to be aware of is which lilbro.js library to load. If you read their documentation, it says to include the following line in your client-side code:

<script src="http://server:8000/lilbro.js"></script>

This is just plain wrong. The lib/lilbro.js file you find in the repo is for the server-side code, not the client-side. The client-side actually uses the three files located in the client/src directory: LilBro.BrowserDetect.js, LilBro.Schema.js, and LilBro.js. Include those in your client-side code and you're almost ready to rock. 

In the usual Javascript fashion, you need to create a LilBro object. preferably on the body of the DOM. That's all you really need to do; Lil Brother will track every click on the page.  It will also track focus and blur events if you add the watch_focus: true attribute. Be careful here! The example wrongly says track_focus; the attribute is actually called watch_focus.

At this point, I'm going to tell you to RTFM as the examples given are pretty good. I will, of course, make a few comments. :-)
  • LilBro essentially passes just an array of numbers as key-value pairs back to the server. It uses the LilBro.Schema.js to determine what the keys mean. So be sure both the front-end and back-end are using the same schema file(s). If they don't, you will see odd things, e.g. the fields will be in your object in Firebug but won't see the fields in the back-end and you'll be all, like, 0_o
  • You can add extra attributes to the LilBro object you create by assigning the key-value pairs as a JSON object to event_base but remember to update the schema (see above bullet)
  • You're going to need some type of taxonomy (structure) to your html ids and classes. If an element doesn't have an id or class, LilBro will climb the DOM looking for one. This is a Good Thing. However, we came across the problem where two calendar widgets on our page both have a class="day" on the date elements for styling, but we don't know which widget it is from.

Server-side Code

On the backend, LilBro comes with a node.js based server called, confusingly enough, lilbro.js. It works pretty well and easily.  You will probably have to install some dependencies, mainly nomnom.js. To do that run 

$ cd lil-brother
$ npm install

When I did that, I got an error that npm wanted to install nomnom version > 2.0.0 but the highest available was 1.8.2. If you get the same error, just edit the packages.json and set the version number to > 1.8.1. Worked for me!

The server can write to files or to a ZeroMQ queue.  We're writing to files for the moment. When I get back into the office later this week, I'm going to write a Python script to tail -f the file to dump the data into a MySQL database.

That's All For Now

This should get you up and running with Lil Brother so you can start playing around with it. I'll keep you posted about my progress. Let me know about yours.

No comments:

Post a Comment