The Human Factor: Results of a Data Analysis Round Robin

Описание: This is a remote presentation I gave at the 2022 Small-angle Scattering conference in Campinas, Brazil. The video has been obtained from the conference organisers with their explicit permission for use on YouTube. I've tried to spruce up the audio from the remote recording the best I could.

The conference abstract for this talk was:

"How much do we, the small-angle scatterers, influence the results of an investigation? What uncertainty do we add by our human diversity in thoughts and approaches, and is this significant compared to the uncertainty from the instrumental measurement factors?

After our previous Round Robin on data collection, we know that many laboratories can collect reasonably consistent small-angle scattering data on easy samples[1]. To investigate the next, human component, we compiled four existing datasets from globular (roughly spherical) scatterers, each exhibiting a common complication, and asked the participants to apply their usual methods and toolset to the quantification of the results (https://lookingatnothing.com/index.ph....

Accompanying the datasets was a modicum of accompanying information to help with the interpretation of the data, similar to what we normally receive from our collaborators. More than 30 participants reported back with volume fractions, mean sizes and size distribution widths of the particle populations in the samples, as well as information on their self-assessed level of experience and years in the field.

While the Round Robin is still underway (until the 25th of April, 2022), the initial results already show significant spread in the results. Some of these are due to the variety in interpretation of the meaning of the requested parameters, as well as simple human errors, both of which are easy to correct for. Nevertheless, even after correcting for these differences in understanding, a significant spread remains. This highlights an urgent challenge to our community: how can we better help ourselves and our colleagues obtain more reliable results, how could we take the human factor out of the equation, so to speak?

In this talk, we will introduce the four datasets, their origins and challenges. Hot off the press, we will summarize the anonymized, quantified results of the Data Analysis Round Robin. (Incidentally, we will also see if a correlation exists between experience and proximity of the result to the median). Lastly, potential avenues for improving our field will be offered based on the findings, ranging from low-effort yet somehow controversial improvements, to high-effort foundational considerations."

[1] B. R. Pauw, C. Kästner, A. F. Thünemann, Nanoparticle size distribution qu­antification: results of a small-angle X-ray scattering inter-laboratory comparison, Journal of applied crystallography, 50: 1280–1288, DOI: 10.1107/S160057671701010X

B. R. Pauw, Data Analysis Round Robin, Looking At Nothing, a SAXS Weblog, 3274, 2022 https://lookingatnothing.com/index.ph...