17 Simon O'Toole, Day 3 Keynote, DataEngBytes 2021

Описание: From telescope to data centre: adventures in astronomical data pipelines

Simon O'Toole
I am interested in data authorisation and access control, astronomical data models, systems integration and data interoperability. I have a diverse background; with a PhD in astrophysics, I worked as a research fellow, provided technical support, and managed data systems at the Australian Astronomical Observatory. I'm now based in the Australian Astronomical Optics group at Macquarie University, working as a senior specialist research systems engineer and leading projects to build a next generation astronomical data archive.

I will describe the data flow we have developed to move astronomical data from Siding Spring Observatory in central west NSW to data centres in Sydney and Canberra, and the subsequent processing stages to produce science-ready data. This is a story that involves the coupling of Apache Nifi, MongoDB and Docker with astronomical data reduction software to allow researchers to explore some of the most violent phenomena in the Universe: colliding black holes and exploding stars. I’ll also look at some of the related future data engineering challenges involved in the Square Kilometre Array and the European Southern Observatory.