Detection of realtime noteworthy Twitter information in your areas of interest.

My team has been working on a project aimed at extracting realtime, anomalous, relevant information from the Twitter public feed. Part of our engagement is to investigate commercial applications of the technology we develop and to that end we are going through some early alpha tests of a UI for the technology. The technology works, and in many cases seems to beat both local and national media to items of emerging interest; it also sometimes generates false positives. My focus is creating a compelling user product and experience with this technology and I could use some feedback to help direct my efforts. I am asking for a few friends to take an early look and provide any feedback they care to give, especially on the potential value of the product to your daily routine. My team will be iterating on the feedback as quickly as we are able, so initially I’ll just be taking groups of ten or so and seeing how it goes until I can get out a next revision. If you have any interest in participating in one of the early groups please let me know and I will send the full page description of what we have built and put you on the list for one of the test groups. The standard alpha tester caveats apply.

Also any direct connection to a reporter I could talk to at the Bloomberg Speed Desk would be appreciated.

Thanks! – Contact via LinkedIn please.

 

Found Software

Most of the software I write these days makes use of found software. Like the Dada art movement, I turn software I find on the internet into works that please me, and my clients. [ http://en.wikipedia.org/wiki/Found_object ]. This style is different I think than what is taught in CS and CE, where knowledge of algorithms, and BigO, are primary. The spectrum from original algorithm development to consumer product with large embedded software component is broad. In the strictest sense I focus on the need of my client, or my perception of their need and what might best address the gap. Rarely does the need start with a wholly novel algorithm. Thinking about it, the last time I developed such an algorithm may have been in 1992 for a CD-i compression scheme called DYUV where I enabled blits into the compressed data structure.
Recently my team has been working on a route generation package for UAVs. The constraint is it must be open software or developed in-house. People wanted to start in-house using standard A * search techniques. But we looked to see what was available – easy to do with Google. Looking for path finding I came across [ https://code.google.com/p/straightedge/  ]. A terrific place to start, it is in Java, it is BSD licensed, it has some nice visualization capabilities – always helpful for debugging. Now what is my software engineering point here? It is that we can probably save a great deal of effort by looking for found software that meets our needs but the task is not effort free, and in fact we may find that we go well down the path of using this software only to find some showstopper. My point is, I did not have to author an A * search algorithm, at most I need to know about it so I could work my Google search. What I do need is the long term experience of patching together various Java projects into a working cohesive structure that solves my clients’ problems. On a larger scale consider the software we will probably never have to write; an SQL data base (Postgres, MySQL, etc) a no-SQL DB (Couch, Mongo, Neo4j, etc), a fast messaging system (ZeroMQ, RabbitMQ, some other MQ d’ jour), your OS ( Linux, OpenBSD, even Darwin, and if you don’t like how it works the code is there to change :-) ). So being facile with many of these pieces helps, if you get stuck, being familiar with method of debugging is most important, and in making a large project being familiar with SCM and project architecture is most beneficial.

Wipe Nokia 3650

I wanted to recycle my Nokia 3650, but not to give up the memory card with any info. Turns out the Macbook Air has a slot. Let’s see what happens.

Using the System Information App - I get
 Built in SD Card Reader:
Vendor ID:    0x05ac
 Product ID:    0x8404
 Revision:    2.00
 Serial Number:    310
SDSC Card:
Capacity:    16.1 MB (16,089,088 bytes)
 Removable Media:    Yes
 BSD Name:    disk1
 Partition Map Type:    MBR (Master Boot Record)
 S.M.A.R.T. status:    Not Supported
 Volumes:
 NO NAME:
 Available:    10.1 MB (10,133,504 bytes)
 Capacity:    16 MB (16,039,936 bytes)
 Writable:    Yes
 File System:    MS-DOS FAT12
 BSD Name:    disk1s1
 Mount Point:    /Volumes/NO NAME
 Content:    DOS_FAT_12

Cool.
Lets get a dd of what was there.

dd if=/dev/disk1 of=nokiasdcard.dd

That worked to get a mountable version, and then I wiped the card using Disk Utility.
In the end my son wanted the retro phone, no doubt to show his friends the ancient technology.

Mathematica

I’ve been an occasional user of Mathematica since version 2.0. While it is a rare and lucky project that has enough math making it necessary to break out Mathematica, when it does happen it is nice to have the Swiss Army Knife of symbolic math and graphics manipulation there in my box of tools.

 

Turtles all the way down

I am working a DARPA project called MRC, the BAE portion of which is called SOUND.

One part of this uses the enhanced Xen Flux Time Travel work, which looks to me like a great advance in debugging intermittent bugs.

To make it easy to test and integrate I am trying to run a guest VM inside of the modified Xen inside of VMWare Fusion on my Mac. This is to save me the trouble of acquiring and stacking more hardware into my tiny office.

I miss the days when I was confused because I had ssh’d into some server and then into another and could not recall which system I was editing files on.

simple java server monitor

Circa 2004

Update: Check out Nagios which is an open source service monitor. This looks like a good solution. Wish I had found it before I did my little project!

ServerMonitor is a small java program that monitors a server and sends email to the administrator when the server is not available.I have been having some reliability problems with a server that I depend on and wanted an easy way to be notified if there server had stopped running. This program is very simple to use.

Use Case Diagram

Deployment Diagram

The program consists of a single java file ServerMonitor.java [text] and a script smctl.sh to start and stop the monitor.

Description
I need feedback regarding a service running on Server. I run a ServerMonitor daemon on Monitor that periodically checks to see if it can connect to Server on a specific port. If it can, I assume all is well. If not, ServerMonitor checks to see if Reference Host is visible. This is to distinguish between an actual failure in Server and a local problem with the connectivity of Monitor to either Server or Reference Host. If there is any failure to connect with Server then email is sent through Notification Server to the administrator.

Installation
Create a directory serverMon on Monitor and put ServerMonitor.java and smctl.sh in that directory. Then cd to that directory and follow these directions.

Set smctl.sh to be executable with $chmod u+x
Compile Servermonitor.java with $javac ServerMonitor.java
You should now have a directory that looks something like this.

[drew@drewsG5 serveMon]$ chmod u+x smctl.sh [drew@drewsG5 serveMon]$ javac ServerMonitor.java [drew@drewsG5 serveMon]$ ls -la total 16 drwxr-xr-x 5 drew drew 170 May 2 22:22 . drwxr-xr-x 11 drew drew 374 May 2 22:20 .. -rw-r–r– 1 drew drew 3617 May 2 22:22 ServerMonitor.class -rw-r–r– 1 drew drew 5794 May 2 14:22 ServerMonitor.java -rwxr–r– 1 drew drew 493 May 2 22:18 smctl.sh [drew@drewsG5 serveMon]$

Configuration
Now you need to edit smctl.sh to set the relevant parameters.

# Edit these lines for your particular setup HOST=www.host.com PORT=80 INTERVAL=10 REFHOST=refhost REFPORT=81 ADMIN=you@localhost SMTPHOST=mail.localhost

Operation
To start ServerMonitor, $./smctl.sh start
To stop the monitor, $./smctl.sh

To see that everything is running correctly $tail -f log.out
You should see something like this:

[drew@drewsG5 serverMon]$ ./smctl.sh start [drew@drewsG5 serverMon]$ tail -f log.out Starting ServerMonitor. host:mail.host.edu port:25 refHost:www.host.edu refPort:80 interval:10 adminEmail:you@localhost smtpHost:mail.smtp.localhost [Mon May 03 17:18:31 PDT 2004] mail.host.edu seems to be running.

Note that only one email will be sent on server failure. When the message is received, you should examine log.out for details about the failure. Then I suggest that you stop and start ServerMonitor again to check for the next problem.

# TODO: check is pid file already exists – and ServerMonitor is already running
# so that we don’t start twice.
# Check if running before we try to stop the process. i.e. make sure calling
# stop twice is not a problem. Would be nice to verify that the pid we are killing
# is actually the ServerMonitor process!

Model Driven Architecture

Circa 2005/’06

Model Driven Architecture (MDA) is an OMG specified approach to software development intended to ,”…separate[s] the specification of functionality from the specification of the implementation of that functionality on a specific technology platform.”

I have had an opportunity over the last two years to become quite familiar with one large scale, leading edge development using MDA.

The short answer is, Brooks is still right. There is no silver bullet to software development.

What I seem to notice is that there is no lack of smart people working on the project and the main distinction between folks is the years of experience.

Redux

If you come across iUML or the corpse of Kennedy Carter, run away, the zombies will eat your brain.

Intel C++ and OpenMP Experiment Notes

From  Sept 2006:

The Problem

I have some legacy software in need of performance improvement. The software is single thread, single process. How effective are OpenMP and Intel C++ on the pathway to improvement?

I have the Intel C++ compiler (icc) available, which supports OpenMP along with some other auto parallelize features, and a four processor Xeon box to experiment on.

Now, I’m experianced enough not to expect a silver bullet fired out of my C++ compiler will kill the performance monster. On the other hand, there is some interest in finding out how hard it will be to refactor this software into a form that benefits from an SMP platform. If nothing else, shaking out the makefile to support an alternative compiler to the GCC 3.4.4 will make for a stronger build process. And, in looking over some discussion threads regarding the Intel compiler and this particular feature I see that icc will report on the loops in the code that are deemed safe for parallelization and will tell me something about why the other loops are not safe for the speed treatment.

Progress

So far I am still getting GCC to compile the code base cleanly.

Addendum

The work was fun and seemed to be helpful in digging into the code, but eventually the client decided other issues, like clearly stating the problem were more relevant to success than doubling the speed of the code. And I could not agree more.