VectorVis: Visualizing Distributed Logs At Scale

Previous approaches to visualize distributed system logs focused mostly on detailed views at the scale of single message traces or short groups of log lines, however, a full distributed log can be as large as a terabyte. While detailed views are quite useful, the problem of deciding which traces, nodes, or time-spans to inspect remains unsolved. We created VectorVis, a tool that enables interactive exploration with coordinated detail and summary views. The premise of the tool is that it is meant as a pre-processing step to filter large log files and output a reduced log to ShiViz, a sophisticated distributed log visualization tool that focuses on inspecting the “happens-before” relationship. We contribute (1) a task analysis that motivates the major design choices of VectorVis; (2) a detailed description of our design process that articulates design trade-offs and requirements; (3) a prototype that demonstrates key interaction motifs necessary for mediating summary and detail views.

I created this application together with Paul Bucci for the Distributed Systems grad course (538B) at UBC.