In July of 2013, my colleague Mike Dewar started building out the streaming data infrastructure for the lab. Faced with a desire to better prototype and see the data that he was working with he proposed the idea of Streamtools, a series of tools designed to easily work with streaming data. Confronted with my own desire to both see and prototype with the data, I joined in the effort.
The first version consisted of a loosely-coupled set of command-line binaries based around NSQ that allowed us to quickly filter and characterize streams of data. This worked well for our streaming data infrastructure, but with our prototypes the topologies became increasingly complicated. To ameliorate this problem our next version included a graphical interface that allowed users to lay out topologies on a live data stream from an in-browser interface. Despite this version working quite well in a variety of circumstances, our desire for even more flexibility has culminated in a total redesign of how Streamtools fundamentally operates.
st-core is the next version of Streamtools that is currently under development. It eschews many design choices from previous versions, such as a complicated DSL, asynchronous/lossy event handling, and complicated high-level “blocks” (what we call individual functions in Streamtools) in favor of a more lower-level, flexible language that affords powerful abstractions that can be reasoned about. st-core is written in Go and Javascript, with React utilized for the interface.