Microscopy images are a significant source of insight and raw information for neuroscience. Modern techniques in electron microscopy (EM) allow scientists the ability to image at such high resolution that every single synaptic connection can be distinguished . Furthermore, acquisition automation has enabled us to acquire large volumes of microscopy data spanning several resolutions with minimal human involvement in the acquisition.

The popularization of these automated imaging systems has made acquiring large amounts of data the standard operation for many laboratories , and although most of them are able to physically handle the amount of data, there is an increasing need for streamlining the pipeline. This necessity arises because of the growing acquisition speed of microscopes, leading to an exponential growth in data throughput. While different techniques are emerging to solve each step of the upstream process, they still have their own independent development communities and there are very few laboratories with the capability of carrying out the entire process by themselves. Individually these algorithms contribute to the study of neuroscience image data; it is, however, non-trivial to chain these modules together and deploy them in one coherent environment for end-to-end connectomics projects.

The continuous use of electron microscopes can produce single datasets that reach multiple petabytes of data, which cannot be processed on local workstations or small clusters and therefore require High Performance Computing (HPC) facilities. We propose to deploy and chain these different processing libraries into a single microscope to HPC workflow and provide a way for the user to interact with the data and its processing in real time. Our package is implemented in Python and is called HAPPYNeurons (HPC Automated Pipeline for Processing Yotta Neurons).