In July of 2013, my colleague Mike Dewar started building out the streaming data infrastructure for the lab. Faced with a desire to better prototype and see the data that he was working with he proposed the idea of Streamtools, a series of tools designed to easily work with streaming data. Confronted with my own desire to both see and prototype with the data, I joined in the effort.
The first version consisted of a loosely-coupled set of command-line binaries based around NSQ that allowed us to quickly filter and characterize streams of data. This worked well for our streaming data infrastructure, but with our prototypes the topologies became increasingly complicated. To ameliorate this problem our next version included a graphical interface that allowed users to lay out topologies on a live data stream from an in-browser interface. Despite this version working quite well in a variety of circumstances, our desire for even more flexibility has culminated in a total redesign of how Streamtools fundamentally operates.