Technology, Open Source, and Me

DADS: Massaging Data

2011-12-28T18:16:00.000-08:00

I've decided that "Diary of an Anomaly Detection System" is too wordy to keep writing in the title of the posts in this series, so I'm shortening it to "DADS" hence the title of this post "DADS: Massaging Data".

Anywho, as I said in the previous post, I'm going to talk a bit about what I needed to do to get my data ready for the anomaly detection algorithm. This post has nothing to do with machine learning, per se, but is an important part of designing an ML algorithm.

I'm going to use seven metrics ("features" in ML parlance) to start with: short-, medium-, and long-term load averages; memory use; number of processes; and the number of zombies. You can argue whether or not these are useful metrics but I'm not interested in that argument at this point. I'm currently building the framework for the ML algo; I'll be adding, subtracting, and inventing metrics once I have something to manipulate them with.

I'm using Python since that is one of the scripting languages of choice at my day job; Perl, unfortunately, is frowned upon and the consensus is Ruby can't do scientific programming just yet. Don't even get me started with Java.

Let's read some data

The data originally resides in RRDtool and needs to be put into a standard matrix form. Shouldn't be that difficult, right? RRDtool has a Python interface, so it's just a matter of reading the data in, right? I wish! The RRDTool Python API is essentially a wrapper around the command-line tool but the output is "Python-esque". For example, the CLI ouput for the load average looks like this:

 [faber@fabers-desktop data] rrdtool fetch load/load.rrd  AVERAGE -s 1321160400 

                      shortterm             midterm            longterm

1321164000: 5.3888888889e-03 1.2805555556e-02 5.0000000000e-02
1321167600: 3.0555555556e-03 1.1388888889e-02 5.0000000000e-02
1321171200: 3.7500000000e-03 1.1861111111e-02 5.0000000000e-02
...

where the first column is the number of seconds from the epoch and the three remaining colums are short-, medium- and long-term load averages; a very handy format. Unfortunately the Python output looks like this:

>>> mydata = rrdtool.fetch('load/load.rrd', 'AVERAGE', '--start=1321160400')
>>> mydata

((1321160400, 1325098800, 3600), ('shortterm', 'midterm', 'longterm'), [(0.005388888888888891, 0.012805555555555468, 0.05000000000000019), (0.0030555555555555557, 0.011388888888888818, 0.05000000000000019), (0.0037500000000000016, 0.011861111111111041, 0.0500000000000002), ...]

which is not a very handy format. For reasons which I'll get into later, I want the format to be this:


shortterm = ((1321164000, 5.3888888889e-03),

   (1321167600, 3.0555555556e-03),
   (1321171200, 3.7500000000e-03),
   ...

            )

mediumterm = ((1321164000, 1.2805555556e-02), (1321167600, 1.1388888889e-02), (1321171200, 1.1861111111e-02_,...)

longterm = ((1321164000, 5.0000000000e-02), (1321167600, 5.0000000000e-02), (1321171200, 5.0000000000e-02),...)

So the next step is to format the data.

List Comprehensions to the Rescue

I've always thought Python was just an okay language but its list comprehensions are kinda cute. It wasn't until this project that I found out just how useful they are. Here's the blow-by-blow action:


# mydata[0] = timestamp begin, end, and interval
# mydata[1] = labels
# mydata[2] = list of 3-tuples
mydata = rrdtool.fetch('load/load.rrd', 'AVERAGE', '--start=1321160400')

# create a list of timestamps at the appropriate intervals
tses = [ i for i in range(mydata[0][0], mydata[0][1], mydata[0][2]) ]

# create three lists from the 3-tuple list
st, mt, lt = zip(*mydata[2])

mydict = {}
mydict['shortterm'] = zip(tses, st)
mydict['midterm']   = zip(tses, mt)
mydict['longterm']  = zip(tses, lt)

Seven lines of code. I don't know about you, but I'm impressed when a language allows me to do that with native functions.

So what's with the key/value format?

There's a subtle problem with the raw data that's not obvious until you start reading in other RRDtool files and try munging them together: you don't always have data for all the same timestamps. memory.rrd might have data for timestamps t1 and t2 while load.rrd might have data for t2 and t3. How do you manage your lists so that you don't duplicate timestamps (two t2s in the above case) AND fill in values for data you don't have and don't know you don't have? Easy:

SQL.

I'm going to store my data into an SQLite3 database then generate a matrix from the database table. If I do my SQL correctly (and I will :-), SQLite3 will fill in missing data, order by timestamp and I don't have to keep track of values or timestamps across rrd files! This is why I break every metric.rrd file into a (timestamp, value) data structure and put it into a dictionary called mydict['metric']; so I can easily insert and update the metric column in the database!

How that is actually done, I'll talk about in the next post since it's late.

Diary of an Anomaly Detection System

2011-12-26T09:14:00.000-08:00

I recently finished Stanford's Machine Learning course offered as part of SEE. It was one of the best courses I've ever taken. Not only did I learn a lot about ML algorithms, but I learned a lot about the applications thereof as well as some math applications.

One of the algoritms that grabbed my attention was "anomaly detection"; this is the algorithm credit card companies use to flag possibly fraudulent activity. It can also be used for monitoring computers and, I believe, web pages but more about that later.

Which brings us to this series of blog posts. Since I'm on winter break, I decided to spend part of my time designing, coding, and blogging about building an anomaly detection system.

The Motivation

My current and on-going project at work is called Sentry, a system that processes URLs. The system involves 23 (and counting) virtual machines and four physical machines so obviously, system administration is a not-insubstantial part of the project.

Currently, the overall health of the system can be measured by throughput; I use collectd and RRDtool to monitor the system. If the throughput is too low, I know something is wrong with one or more of the machines but that doesn't tell me which machine(s) is having a problem, so I look at the CPU load graph for each machine to see if anything looks odd, if not, I look at the network transfers graph for each machine to see if anything looks off, if not, then I... Since there are 27 machines, each with over a dozen metrics, this is a chore, as you can imagine.

Anomaly detection will reduce the the dozen-plus metrics for each computer down to a range of numbers that we consider "normal behavior" and, by extension, the health of all 27-plus machines down to a range of numbers. So how do we define the range of numbers?

Assume a spherical cow

All of my metrics measure different things: load average, memory use, etc. If I plot the range of numbers over time I'll get different looking graphs, but for now, assume all the graphs are Gaussian. What I'm going to do is take each metric's numbers and figure out which Gaussian curve (in other words, the values for μ and σ) best fits the data. I'll then take the latest reading, for say, memory use, and see where on the curve it sits (since I know the equation for the Gaussian distribution function) call that value p(memory), and for any value of p(memory) greater than say, ε, I'll tell the computer to flag that reading as an anomaly.

I'll then take all the latest metric readings (the x-s), calculate the p(x)s and multiply them together; that gives me a single number, super-ε, that tells me if my computer is acting "weird". If I multiply all the p(x)s for all 27+ computers, I get another super-ε number telling me if my cluster of computers is acting "weird". Neat, huh?

First Things First

But first, I need to gather the data and put it into matrix form that the algorithm can handle. The raw data resides in RRDtool tables. I'll be using Python as my main language. The vectorization libraries will be numpy and scipy. Even though my main machine at home is a Macbook Pro, installing numpy and scipy on a MBP is a PITA, so I'll be doing all of our work on a (Ubuntu) Linux box.

Massaging the data will be the topic of my next blog post.

The Budget Game

2011-02-06T09:45:00.001-08:00

I've been doing Tim Ferris' "Slow Carb Diet" from his book "The Four Hour Body". To help out, I've been tracking calories using the MyFitnessPal Calorie Counter app on my Android phone. So far, so good; I'm down five pounds in two weeks and this is the first time I've been below 190 in a decade.

So I got to thinking, why not do with dollars what I'm doing with calories, IOW, set up a budget? Now, I've done that in the past and it's been BORING! so I thought about how to make it more fun and came up with the idea of making it a game.

Apparently, the idea of making budgeting a game isn't a new idea but it's not that widespread either. So I'm thinking how to turn budgeting into a game. Here's what I've gotten so far:

1. A "game" is a month-long and you play 30 day-long "sessions".

2. You allocate a specific amount of health points for the game (your monthly budget) and session (your daily budget). The daily allotment shows up on the screen as your avatar (think the Marine from DOOM; I am ;-).

3. As you spend money, your avatar's health deteriorates. If you go over budget, your avatar dies and the game is over until the next day.

4. If you do get through the day without killing your avatar, you win points (?) and the remaining health points goes into a vault. At the end of the game, you see how well you've scored by seeing how many health points you have in the vault.

The idea here is to track day-to-day spending, not the big monthly outlays like rent and heating.

Since this is going to run on my Android phone and I really dislike Java, I'm going to do this as an HTML5 app. If I understand correctly, I can write the app using vi and Chrome, then wrap it in an Android "package" and I'm done. I assume something similar can be done for the iPhone, but, come on, who uses an iPhone these days? Really?

The game idea still needs work, but that's what development is for. :-) If you have any suggestions, comment below or tweet me using the hashtag #TheBudgetGame.

Help me to choose some new frames

2011-01-16T16:29:00.000-08:00

From @erked via Twitter: @faberfedor You have now taken more time choosing glasses, using a more formal process, than McCain did selecting a running mate.

2009-06-22T19:37:00.000-07:00

Okay, I have no idea what this mean, but a crocheting friend of mine thinks this is the bee's knees for crocheters everywhere, so I'm passing it along:

With most afghans that run back-and-forth (in contrast to the concentric type), the base row can be inflexible compared to the rest of the afghan, which results in one end (usually the on shorter sides) being a consistent length and the opposite side getting longer as the afghan is stretched or pulled as it ages. I had about 10 rows done when I realized the base row was too tight, and I experimented and figured out a way to make the base row and the first row at the same time, and make it as flexible/stretchy as the rest of the crocheting. One small step for me, one giant leap for crocheters everywhere. I should publish this technique somewhere.

[Here's the the technique]
chain 4, yarn over, insert hook in first chain stitch, yarn over, pull through a loop. Yarn over and pull through the last loop, then yarn over and pull through the last loop (two chain stitches on the end of the hook), then yarn over and finish like a normal double crochet: pull through two loops, yarn over and pull through two loops. For the next stitch, yarn over, insert hook in the second chain stitch at the bottom of the previous eccentric double crochet, pull through a loop, do the two chain stitches at the end of the hook, then complete the double crochet.

Got that? Good! Now someone explain it to me! On second thought, don't bother. :-)

Thoughts about the Panopticon

2009-05-25T11:13:00.000-07:00

I finally got around to reading Joshua-Michéle Ross's three articles over on O'Reilly's Radar. Nice overview of the topics but I was hoping for somehing more in depth. The last one, The Digital Panopticon, gave me an idea.

While I do love the idea of location based services (I'm even writing one of my own), I'm beginning to wonder if there is a way to anonymize such a service so the end-user can have the benefits of LBS without giving up information to the Watchers. Ideas are welcome.

As an aside, Joshua-Michéle states:

In the age of social networks we find ourselves coming under a vast grid of surveillance - of permanent visibility. The routine self-reporting of what we are doing, reading, thinking via status updates makes our every action and location visible to the crowd. This visibility has a normative effect on behavior (in other words we conform our behavior and/or our speech about that behavior when we know we are being observed).

He doesn't take into account that we (some of us, at least) are not reporting all of our activities and locations. True, we may be few and far between, but we do exist.

2009-05-20T20:18:00.000-07:00

I loved this TED Talks about 10 things you didn't know about orgasm, mostly because I think Mary Roach is hot! :-)

Last year's MP3 Experiment

2009-05-16T05:17:00.000-07:00

With the latest NYC MP3 Experiment coming up next week, I thought this might be a good time to reminisce about my first experience with the MP3 Experiment from last year. So here's a write-up of last years activities. I'm really looking forward to this years!

Here are some pictures of last year's event.

------------------------------------------------------------------------------
Okay, if you followed my Twitter feed last Saturday (20080927), you have a pretty good idea how my Day In The city went and my thoughts on Governor's Island, dinner and girl-watching in Little Italy, and the Museum of Sex. What you don't know is what actually happened during the MP3 Experiment:NYC because I was busy doing the experiment. ImprovEverywhere will have film and commentary up on their site in a few weeks. Here's my experience with it all.

Background: We were told to show up at Governor's Island, just south of Manhattan, wearing a red, green, yellow, or blue t-shirt, bring an umbrella and a balloon. At precisely 3:15 PM, we were to push the play button on our MP3 players and do what the Voice of Steve told us.

I showed up about an hour early. The weather was warm, if misty/raining. After walking around awhile, I wasn't feeling too happy. Anyway, at 3 PM I head for the large field in the middle of the island, put in my earphones, checked the time on my cell phone, sent one last tweet, and waited. Fortunately, the weather started to cooperate; the sun hadn't come out but it did stop raining/misting. There were maybe 50 to 75 people scattered about the field which is a couple of acres large.

At precisely 3:15 PM, I pushed "play" and music played. And played. And played. I heard someone mention maybe they were just screwing with us and there was nothing but music on the MP3. After a minute, we heard The Voice Of Steve.

The Voice of Steve (which was obviously computer modulated but not computer generated) welcomed us to Governor's Island. After a few minutes of talk, Steve told us to stand and stretch. It was interesting watching the others; some were ten or even twenty seconds later than others. Along with not synchronizing properly, it turns out that MP3 players don't all play at the exact same rate.

After we stood and stretched, Steve had us point to NYC, then to our homes, then to Nicaragua. At this point, I saw some people point at the sky! Steve commented on our lack of geographical knowledge.

Steve then told us to look around for a person wearing a different color shirt (I was wearing green) and give them a big hug. The first person I saw was a large woman wearing blue, so I walked up to her and gave her a hug. And she hugged back. Hard!

Steve said to hug an inanimate object. Not being near a tree or anything, I hugged my umbrella. Then He said to hug an animal. Any animal would do: squirrel, goose, an ant. Unfortunately, I couldn't even find an ant to hug. :-(

Steve then declared we would have thumb wars! This cute little chick (she barely came up to my shoulders) in blue denim jacket and jeans was walking by so I grabbed her and we got into position. "One, two, three, four, I declare a thumb war!" boomed in our headsets and we started thumb wrestling! I easily one the first round. My opponent decided I wasn't going to win a second time; she put down her umbrella, got into a fighting stance and "One, two, three, four I declare a thumb war" was heard and she fought hard! And she won! So with a tied score, we fought one last battle which I handily one. She bowed to my impressive thumb-fighting skills and we went our different ways.

Steve then told us to walk to the field in the middle of the island where there would be an "epic battle" later. (Every time Steve mentioned the words "epic battle", a deep baritone would say "EPIC! BATTLE!" behind him.) This is when I noticed that many people were not on the field to begin with! He told us to take out our umbrellas, hold them high over our heads and walk around the field. We did this for several minutes; it actually got kinda boring after a while but I got to see some interesting ppl.

One interesting person I saw was a young woman with The Most Elegant Equation in Mathematics tattooed across her shoulder blades in letters a hand high! When she walked past me a few minutes later, I tapped her on her shoulder and said "Cool tat!". She looked at me, confused, and said "Burn" and walked away. That's when I noticed that it wasn't a tattoo; the equation was burned into her skin! Kids these days!

We played a couple of "motion games" as I call them: "Equilateral Triangle" and "Attacker Defender". In the former, you choose two other people on the field and you move in such a way as to form and equilateral triangle. It sounds easy until you realize they're doing the same thing with two *other* people. "Attacker Defender" is similar except you keep the person you've chosen as the Defender between you and the Attacker. Again, they're doing the same thing with two other people. Then it got interesting.

Steve told us to find three other people with the same color shirt and form a square with them, so there I was standing shoulder to shoulder forming a tight square with an Irish woman, a swarthy fellow and an Asian fellow. It's just weird standing that close to strangers, what with them in your personal space, you know?

At this point, Steve tells us we need to learn how to do a "fife and drum" shtick for the epic battle (EPIC! BATTLE!). He instructed the reds and the blues to tap out a rhythm on their thighs. He instructed the greens and the yellows to play the fife part and we whistled the tune he gave us. Imagine the scene: over 200 people standing on field, forming tight little squares, half of them drumming on their thighs and the other half whistling.

After we practiced that a few times, Steve congratulated us on a job well done. Then he said to find three other squares of the same color and form a larger geometric object, you know, like the shapes in Tetris. I'm sure you can guess what comes next. After we form a larger object, we were told to move around the field and find a shape we could fit in with. Yes, we were playing human Tetris. So now those squares of people were now scrunched together even tighter! There was, literally, no room to move.

Steve then told us to take out our umbrellas and life them over our heads. We did, and we blotted out the sky! One minute we're in an open field and the next minute we're under a canopy of plastic. That felt really weird! And then we started humming, cuz Steve said to.

After a few minutes of this, Steve congratulated us again, told us to put our umbrellas away and to find people wearing different colored shirts. That was easy, I just turned around and I was standing next to another little cutie in blue (I refer to her as Smurfette), a tall guy in red, etc. I notice Smurfette wasn't wearing headphones and I asked her if she was hearing The Voice Of Steve? She shook her head "no" so I took out one of my earplugs and put it on her. I was rather surprised she didn't pull away or anything when I did it, but where was she going to pull away to? We were all packed tightly together.

Steve mentioned that we were now going to play "Human Twister"; he would mention a color and either "head", "elbow", "shoulder" or "left/right foot" and you had to put your hand on the head/elbow/shoulder or put your foot next to the foot of someone with that color shirt. So when Steve said "Red head" we all put our hands on the head of the tall guy wearing red. With "blue shoulder" I put my hand on Smurfette's shoulder, etc. That was fun, but at one point I thought some people were going to fall over and take us (and everyone else?) with them. Fortunately, that was averted by Steve telling us to prepare for the "epic battle" (EPIC! BATTLE!).

To do so, the blues and greens went to one end of the field and the reds and the yellows to the other end. We got out our "weapons", the balloons, and blew them up. Smurfette was still with me, so I was echoing Steve's instructions since she wasn't wearing my other headphone anymore. The two groups started yelling at each other: "Red rules!" "Get them all!", ""Death to the Yellows!".

Steve told us to get our "weapons" ready and for each group to walk towards the other and stop when we were twenty yards apart while we did the fife and drum shtick. So there we were: two groups of people walking across the field of battle towards one another, whooping and hollering, brandishing our "weapons", some whistling and some tapping their thighs. That was so much fun and exciting. I actually thought back to eariier battles in history and wondered if this wasn't just a little like them.

Anyway, we stopped twenty yards from each other and commenced with more whooping and hollering. Steve said "Fight" and each group rushed the other one balloons flailing! Oh, it was an epic battle (EPIC! BATTLE!) to be sure! A Yellow had broken through the lines and I was attacking him when four Reds come around our left flank and attacked me. As they beat me with their balloons, I fell to my knees, crying for help, swinging my weapon uselessly around me, and then I fell over "dead". Cries of "Medic!" where heard all across the battlefield.

I just laid on my back and watched the whole thing. :-)

Eventually, Steve called a halt to the battle. By this time over half
of the people were laying on the ground "dead". Steve then had us close our eyes and meditate for a few moments. Then He said goodbye and we all waved as He left.

And with that, the MP3 Experiment:NYC was over.

As I and 200 of my closest friends walked over to the docks, I saw my thumb-war opponent. She saw me and stuck her thumb up in the air. As I passed, we did a quick thumb war which, alas, was a draw.

I was wondering how they were going to handle having 200-some people trying to get back to Manhattan all at the same time, especially when the ferries only run once every half hour. It turns out that they got some really big ferries and they were loading two up at the same time.

So we formed two queues each about six abreast and we were allowed onto the ferries in groups of ten or so. While we were waiting, someone led us in various activities like singing "If you're happy and you know it" and "Row, row, row your boat". Eventually, I did get on the first boat out, had a nice time talking to some other people, and eventually got into a cab and headed for dinner in Little Italy.

Washington D.C. Trip, 2009-04-03

2009-04-05T08:51:00.000-07:00

I had made tentative plans to go to the National Gallery of Art (NGA) in Washington, D.C. this summer with my friend Tish. Now that I have a "real job" it looks like that wouldn't happen, so I decided to make the trip before starting my new job.

Since D.C. is a pretty good haul from the north-central Jersey boonies, I headed down on Thursday afternoon. I arrived just in time to go to "game night" with Tish and her brother Mark. We ended up playing a couple of hands of The Great Dalmuti. Afterwards, we got some pizza then headed back home where we sat around drinking wine and shooting the breeze until 3:30 AM.

Up way too early the next morning, Tish and I headed down to the Metro in the rain. That part of Maryland/Virginia reminded me a lot of a New Jersey landscape with more modern infrastructure: better and wider roads, no construction, etc. The other drivers on the road were just as bad as Jersey drivers though.

So we got to D.C. around ten-ish and headed over to the East Wing of the NGA. I hadn't been in D.C. since I was a teenager but I hadn't been in this area. I was impressed with the architecture in this part of town, for example the Department of Agriculture's North Building; the architecture exudes big-bloated bureaucracy and power.

Even though the East Wing of the NGA is dedicated to modern and abstract art, the best exhibit was the illuminated manuscripts. The artwork on some of those were incredible and I couldn't believe some of them were over 500 years old and still so brilliantly, well, illuminated!

Next to the illuminated manuscripts, the best part about the East Wing was the Multiverse light tunnel you traverse to get to the West Wing. Other than that, I was meh about the East Wing; I expected more...important works, or at least more works that I would recognize.

The West Wing holds more traditional, or should I say classical, pieces. I liked it better then the East Wing but still had the same thought; I expected to see more "famous" paintings. There were a few, of course; Da Vinci's Ginerva de' Benci, Napoleon in his study and a couple of Monets come to mind, but still, I expected more.

Don't bother eating at the cafe in the West Wing. The cafe was called "Cafe' Amsterdam" and I assume it was themed to go along with the Dutch Cityscapes exhibit. They had a nice blonde Belgian beer and the stew was good but the rest of the food was meh. Of course, because it was in a museum, the price was a bit much ($20 for the buffet).

Afterwards we walked around some more. I think we covered about 80% of the West Wing's exhibits. By that time, I was just tired of it all. Oh, the noisy high-schoolers who were on some sort of artsy scavenger hunt didn't help my mood either.

By this time the weather had turned beautiful, so we walked across the National Mall and made our way over to the United States Holocaust Memorial Museum. As we approached, it looked like there was a throng of people waiting to get in; think Ruby Tuesday's on a Friday night. Turns out most of the people were coming out of the museum although there were long lines to get in. The lines were due to everyone having to go through metal detectors to get in. I thought it was mostly security theater; after all, the other museums don't make you walk through metal detectors. Tish pointed out other museums don't have wingnuts running around saying that Picasso never existed.

I have to say I was pleased to see that the Holocaust Memorial Museum is not the Jewish Holocaust Memorial Museum; unless they were talking about specific incidents related to the Jews, the museum made it a point to mention that the Roma were also targeted for extermination.

I must say the museum is the quietest museum I've ever been in.
Sure, the lobby is loud and noisy, but once people got off the elevator on the top floor (you start on the fourth floor and work you way down) their mood suddenly changes and, if they say anything at all, it's in a low whisper.

There were some exhibits in the basement that were overlooked by most people unfortunately: a small display talking about the ethnic cleansing in Kosovo, the 1994 Rwandan genocide was mentioned in passing, a display about "The Protocols of The Elders of Zion" was interesting, and an elaborate exhibit on the Nazi's use propaganda was very good.

I highly recommend the Holocaust Memorial Museum if you're in the Washington, D.C. area.

Afterwards, we headed over to the Potomac basin to see the cherry blossoms in bloom. I don't understand the attraction myself. Then we headed to Reiter's bokstore which is an awesome bookstore; nothing but technical and professional books. If I lived in the area, I'd be there a lot just to peruse the math and computer books.

We finally got back home around 10 PM that night. It was a long day, but fun.

My NYC Android Presnetation

2009-03-26T15:04:00.000-07:00

A couple of you have been asking me how my presentation at the March 24, 2009 meeting of the NYC Android Developer's Meetup went. Overall, not bad, but it did have its rocky moments.

The point of the presentation was to demo the mini eco-system I've been writing that uses Google's App Engine (GAE) and Google's Android phone. There's nothing really special about it (other than I wrote it ;-) but the group needed someone to present something and you know how much I like to stand in front of a technical audience.

For those of you in the audience that are new, you can read what my little eco-system does here.

The meetings are held at Fast Company's NYC headquarters, 29th floor of WTC 7. The meeting room has an awesome view of New York and the Hudson but this was the first time I had seen it in the light (due to Daylight Savings Time change). With the sun setting behind skyscrapers, it was even awesomer [sic].

I got there early and started setting up. Of course, the hardware didn't want to cooperate: the GPS on the phone wasn't kicking in; the Macbook display was doing weird things until one of the other members pointed out I had to put it into "mirror" mode; and Droidex, the application that displays the phone screen on the computer in real-time stopped working! Without Droidex, my presentation was worthless. Eventually, though, I got everything working (except the GPS).

There was a good turnout for the meeting, probably 25 to 30 people, over half of them new. We introduced ourselves and mentioned why we were interested in Android. When the last person finished speaking, Rana the Assistant Organizer looked at me like "Well? Go for it" so I did.

The presentation was called "Scratching an Itch". The phrase "scratching an itch" is used in programming to describe the act of a programmer writing a program ("scratching") for his need (an "itch") instead of someone else's need, like his client or boss.

So the first thing I had to do was explain my "itch". I did it by tellning the story in a Lessig-style presentation. The story revolved around me missing Banksy's NYC exhibit in October 2008 but first, I had to explain who Banksy was, so I showed several of his art works. Things were going well until I showed this one; about half the audience (mostly the older people) didn't appreciate it for some reason. :-)

Anyway, I got through the "describing the itch" part and started demoing my programs. That could have gone a lot better! I think I'll practice that some more before trying it out on some more audiences.

Afterwards, I got to talk to several cool people about how to do things in Android and even got into a discussion about AWS.
At that point, there was nothing left to do but head home.

I want to put up a screencast of my presentation and demo, so if anyone knows of some decent [fF]ree software to do that on a Macbook or Linux, let me know.

Android and ProximityAlerts: Have I found "the trick"?

2009-03-18T12:03:00.000-07:00

As you know, I've been fighting with Proximity Alerts (PAs) in Android. The idea behind them is I can tell the LocationManager to notify me when I come within X distance of a geopoint (geographical point, a.k.a. latitude/longitude). There's is practically no discussion of this on the Google Groups, for the 1.0 version of the SDK anyway, and no one has been answering my postings.

My conclusion? PAs don't work, which is a shame cause it seems to be a killer feature of the Android framework.

Well, I Was running errands today and I decided to try something. I deleted every location in my Android save one. I then field-tested my app. The PA worked! I then drove to a local WiFi hotpsot (the local liquor store :-), changed the geopoint and tested it again. It worked again!

So my current theory is this: PAs work, but only one at a time. You can't load up several PAs becasue, presumably only the last one is read/active.

This means I have to write my own routine to determine what geopoints are nearby which kind of defeats the point of using PAs in the first place if you ask me.

Maybe I'll actually have a working demo for my presentation on Tuesday at the NYC Android Developer Meetup!

Musings for a rainy day

2009-03-09T14:50:00.000-07:00

All throughout my consultancy, my business model was "build it and they will come" and sure enough, for ten years, people kept finding me and hiring me for projects. Well, that's not working any more; the market has moved and I need to find another market and/or business model.

The question is "how?" How do you find something that is both profitable and enjoyable? I've been told by marketers "Pick an industry. Anything. Selling dog food. Dry cleaning. Just pick something and focus on it." That is unsatisfying to me.

I also need to move "up the stack". What I've been doing is selling services (programing, admin, training); basically being a journeyman or, in your terminology, a non-full-time contractor. But what's above me in the stack? What skills or knowledge do I have that others would pay for that don't ential billing by the hour for slinging code or doing maintenance?

And then there's the technical side of it. I'm familiar with what's going on technically (Twitter and Facebook APIs, Google Android, the iPhone, Cloud Computing) but how do you form a business around them? A specific example: I have the docs and understand the Twitter API but what can I do with it? What can I build with it that others will find useful and, even better, pay for?

Suggestions?

Last night's presentation (2009-02-26)

2009-02-27T19:38:00.000-08:00

Last night, I gave a presentation on my Google App Engine application at the NYC Cloud Computing Group's (semi-regular) monthly meeting held at Gemini Systems down in the Financial district (thanks for hosting us, Mick!). I must say, it went over much better than I expected.

I've been concentrating on my Android Developer's presentation slated for March 10th and didn't give enough attention to the NYCCC presentation, but then I never think I give enough attention to the job at hand. Anyway, I put together a NeoOffice (that's OpenOffice for the Mac, doncha' know) Impress presentation which was filled with bullet points. The difference this time was I showed each bullet point individually. Simple technique but it kept everyone with me instead of having some people listening to me and others reading ahead.

The original idea was to have a hands-on workshop, so I had planned to give the presentation, lead everyone in programming a simple app, then show a simple app that I had written to "scratch an itch"

However, given that I was sharing the evening with Dustin Whitney who was discussing Amazon Web Services and we each had only an hour, that idea went out the window real fast. Instead, I gave my presentation and then walked through my App Engine datastore for my Tag Your World program.

As I said, it went over very well. The audience was full of questions and kept me on my toes. I really like presenting to a technical audience.

At the end of my hour, there were still questions and comments, but I had to turn things over to Dustin. After a five minute break, Dustin gave a good presentation on the CloudTools that he has built and showed us some of the ways he uses Amazon Web Services in his day job.

Overall, it was a great meeting. Good presentations (if I do say so myself), good audience, and good food. I can't wait to see what John has slated for the next meeting.

World Domination is one step closer!

2009-02-24T11:51:00.000-08:00

I spent half of the day getting my Android app, "Tag Your World", to download locations from my Google App Engine application! And it works!

Okay, it's not perfect, mind you. I'm not authenticating or anything so I'm just pulling down the "public" info. However, Lenza just posted some code on how to do that so I should have no problem! Thank you, Lenza!

Now I can finish my presentation for this Thursday (although I'd really like to hack some more ;-)!

2009-02-23T19:08:00.001-08:00

Not much to report, I'm afraid.

I had two job interviews last week. I'll report on those once I hear back from them.

I'm working on a presentation on Google's App Engine for the NYC Cloud Computing Group this Thursday (20090226). Problem is I can't find the right hook to make it entertaining.

Yes, you can make a technical talk entertaining. I'm also working on a presentation for the NYC Android Developers Group about my App Engine and Android ecosystem. The hook there is (I hope) very entertaining: a Lessig-style story followed by a functional demonstration followed by code discussion and Q&A.

I need to get at least two more functions working to pull off the demo. If I'm lucky, I might be able to demo my talk at the next ABE-PM meeting or even the LUG/IP meeting (although I think the next one is spoken for already).

I'll let you know.

That was a short day of programming

2009-02-17T10:33:00.001-08:00

Today I decided to work on the Google AppEngine (GAE) side of my application's ecology. If you have a Google account, you can see it here. I know, not much to look at.

What I wanted to do today was to get the AppEngine and the Android talking to one another via RESTful calls. I wrote a routine to take the data from GAE and convert it into a type of KML format. When I uploaded it, I found I couldn't save or update data. :-( There is something wrong with the urlfetch() call; strange becasue I didn't touch that code. Now I'm waiting to hear back from the GAE mailing list.

Maybe I'll go work on a website for this app for marketing purposes.

2009-02-16T19:18:00.000-08:00

I just spent the last twelve hours hacking on my Android app, Tag Your World. I made some progress but the code is still pretty bad (whaddya' expect? It's still my first Java program!).

Most of the morning was spent figuring out how to parse a JSON-formatted KML response from Google's Maps API. After looking at various XML parsers, most notably SAX, I decided I might as well just do it manually.

By "manually" I mean I took the JSON string, converted it into a JSONObject and looped through the keys. When I found the "Placemarks" key, which holds several addresses of different resolutions (address level, street level, etc.), I converted that to a JSONArray and passed it off to a function which populated an instance variable ("my first use of polymorphism!" he says proudly).

Overall, I went from a JSON string -> JSONObect -> JSONArray -> JSONObject just to fetch the first address line in the KML. There's got to be a better way! I find it hard to believe that there's no easy way to take a KML-encoded string and magically produce a Java object. Unless I'm on the bleeding edge and I don't know it.

Spent the evening doing some cleanup work (the TimePicker wasn't handling hours properly) and adding a few cute features (Satellite View v Map View).

The next major thing I need to do is to integrate the Android app with my AppEngine app to show them both off at the next NYC Cloud Computing Meetup on the 26th and the next NYC Android Developer Meetup, whenever that will be.

On top of that, I've got two day-long job interviews this week! For someone who hasn't had any work for awhile, I'm awfully busy!

Thoughts on Ocaml

2009-02-08T12:16:00.000-08:00

I'm interviewing with a financial company in Manhattan. The position requires wearing many hats, but it's mostly administration. However, I will be doing a little bit of coding, mostly in the basics (shell, Perl, Python) but also in Ocaml. So I spent the weekend writing two programs in Ocaml based on the problems listed over on Project Euler, numbers 8 and 18 to be specific. This was my first foray into functional programing. I must say I like the paradigm although Ocaml itself leaves something to be desired.

Functional programming treats functions differently than in imperative programing, i.e. "normal" programming. In FP, functions are treated more like mathematical functions, so it is well-suited for some of the problems over on Project Euler. Not everything can be done in a non-imperative way (circular data structures, for example) but it is a good tool to have in your developer's tool box. I was pleased to see that Perl is considered a good functional language, so this weekend's exercise will definitely have an effect on my usual coding style.

Ocaml itself, well, I dunno. It's a relatively young language, started in 1998, so it doesn't have a lot of libraries or even tools; I couldn't find a decent IDE that was being actively developed. Now, normally, un*x/Linux is my IDE (Vim, make, etc.) but I do like to use a graphical IDE every now and then (my current fave is Eclipse) especially when learning a new language/framework.

Since I code quite a bit in Perl, I'm used to using Vim and the internal Perl debugger, a paradigm that Ocaml fits into very well, but the Ocaml debugger needs some work. In the last program I wrote, it kept telling me I had a syntax error on blank lines and even on a line after the end-of-file.

I think I'll go learn Scheme to learn more about functional programing and, of course, Perl.

Geocoding on Android

2009-02-01T05:20:00.001-08:00

If you want to do geocoding, that is, if you want to get the latitude and longitude of an address in Android, do not use the Geocoder object. Yes, yes, I know that's what it's for and the docs give you very simple examples but they fail to mention one thing: it doesn't work!

I lost three frackin' days trying to figure out what I was doing wrong until I came across this thread. Turns out there's a bug in Android and forward geocoding doesn't work. Sure, it's been fixed in "cupcake" (the next version Android that is, as of this moment, pre-alpha) but you figured someone would have seen my posts of the past three days and said something about it.

I guess no one is geocoding on the Android, eh?

udev fun!

2009-01-27T16:20:00.001-08:00

That was a rough day! I spent most of it trying to get my Android Dev phone to be recognized by my CentOS box. It turns out the Google supplied udev rules don't work under Fedora or CentOS.

The output of udevinfo is different on the different OSes, hence the udev rule is different. This one works for Fedora 10:

SUBSYSTEM=="usb", ATTR{idVendor}=="0bb4", ATTR{idProduct}=="0c01", SYMLINK+="android_adb"

and this one works on CentOS 5:

SYSFS{idVendor}=="0bb4", SYSFS{idProduct}=="0c02", SYMLINK+="android_adb"

Note that the spaces after the commas are important and they're both supposed to be on one line.

If you put these rules into 50-android.rules like it's suggested, you will get a device node called /dev/android_adb. With this setup you'll have to run adb as root. To fix that problem, do these two things:

put MODE="0666" (with a comma and a space!) in either of the above line, and
change the name of the file to 51-android.rules.

You can restart udev with start_udev or udevcontrol reload_rules.

I'm really hating Apple...

2009-01-21T18:35:00.001-08:00

...or at least the Macbook. Not only does it not have enough keys, the overall interface isn't as consistent as everyone maes (and the damn 'k' ey has been flaky since day one!) it out to be.

Here's what pissing me off tonight. I'm using Eclipse to write an Android application. I want to learn a few shortcuts like how to jump to a specific line (Ctrl-L), got to the last edited position (Ctrl-E), or jump to another class member (Ctrl-Q). Except on an Apples, the function of the Control ey (ARGH!) is usurped by the Command (Open- Apple, Flower thingie) key. So it's Command-L, Command-E and Command-Q, right? Amirite? WRONG! It's Ctrl-Q because Command-Q closes the program! So maybe Ctrl-E also goes to the last position? No, because that key combo does nothing!

Go head, fanboi! Argue all you want that Command-Q to close is consistent across applications. Can you Command-Q the Finder? Huh? Can you?

And I'm not going to blame this on the Eclipse people. I'm blaming Apple. Here's another one: the function keys are set, by default to be Special Features Keys (dim/brighten monitor, volume up/down, etc.). Fine. So if I want the function eys (oh, screw it!) to act like function keys, I need to push a modifier key. Fine again, except it's never the same frackin' modifier key! If I want to quit out of Midnight Commander, it's Command-F10, but to make a directory, it's Control-F7 and to move a file it's Fn-F6! WTF?

As soon as I can, I'm getting a Lenovo Thinkpad running Linux. The only thing this MB is good for is as an iPod updater. I was going to say "as a multimedia box" but I can't even watch The Daily Show on this thing.

Bah!

More Android Fun! (Yes, I really mean it this time!)

2009-01-17T17:36:00.000-08:00

I just spent the past 12 hours working on my Android app. I think I made a fair amount of progress. For 12 hours, I better have!

Reverse geocoding (where you look up the address given the latlong) is working so I can now say "Save where I am right now!" and also say "Use this address for this new location". If you're paying attention, you can see how they both share a lot of code.

I got Preferences working too! That was pretty easy, I must say.

I still can't get my Linux boxen to recognize the Android, so I'm stuck doing hardware debugging on my @#$%^ Macbook. Although it was kinda neat to fire up the program in Eclipse and watch it run on the handheld.

I think I have only two major pieces left now; doing the proximity checking and drawing the proximity area. The first is mostly done by the Android framework, unfortunately I'm having difficulting finding code I can ste^H^H^H look at for inspiration. I still don't know how to do the proximity area drawing.

Tip of the day: how to properly transfer your app from one machine to another. Tar up the workspaces on one machine, transfer them to the new one, delete the old dir, untar and then....right-click, select Import -> From Existing Files. Things work much better that way.

Tomorrow it's the LocationManager coding! Wish me luck!

More Android fun (NOT!)

2009-01-16T11:10:00.000-08:00

Man, today is not my day for doing Android work!

I've been trying for the whole morning and the past two hours to either a) get my Dev phone recognized by my Linux boxes (no luck so far) and b) get my app running on the Dev phone via my Macbook.

A major problem is moving an Eclipse/Android app from one machine to another. I can understand needing to do a Project -> Build All on the new machine but there were still weird problems. I ended up just rerunning the app several times until all the gremlins were shaken out.

For future reference, if you do move Android projects from one machine to another, remember to copy your ~/.android/debug.keystore from one machine to another. I don't know exactly what it does, but your maps won't work otherwise.

Now I have to prep for a phone conference about putting on a cloud computing conference on Wall Street and then take my second Google interview. Wish me luck!

NYC Cloud Computing Meetup

2009-01-15T11:34:00.001-08:00

Last night's NYC Cloud Computing Meetup was another successful event. I think there were twenty attendees, not including the two speakers, Geir Magnusson from 10Gen and Helena May of AppNexus.

Geir gave a good overview of the state of data persistence in the cloud, from Amazon's S3 to Google BigTable to his company's offering. It was quite informative. The tech guys got pretty into it.

Helena's pesentation on what her company offers really jazzed up the business types in the crowd. Succintly, AppNexus offers a production-grade alternative to Amazon's EC2 with 100% uptime, SLAs and the like.

If you're interested in cloud computing are are near NYC, you really should check us out. The crowd is pretty interactive and there are always good discussions going on.

Next month's meeting should be very interesting; the organizer, John D'Esposito, volunteered me to give a presentation on Google's AppEngine. I just have to figure out what I'm going to say! Stay tuned!