KERLUMPH

A lossy JPEG2000 Data Compression software

Description

Recent work evaluating the JPEG2000 (ISO/IEC 15444) standards as a future data format standard in astronomy has shown promising results on observational data. KERLUMPH, the GERLUMPH extension of the Kakadu Software Development Kit (KDU), has been developed to evaluate the standards suitability for data generated in numerical simulation — using GERLUMPH data as a case study. KERLUMPH enables the conversion of magnification maps (.bin) — pixelated versions of the caustic pattern in the background source plane created by the foreground microlenses — into a compressed JPEG2000 file (.jp2), and the inversion of the process in order to enable work with raw-like data.

When approaching lossy data compression, one should keep in mind the intended purposes of the data to be compressed, and evaluate the effect of the loss on future analysis. We showed that lossy JPEG2000 can deliver higher compression ratios (mean ratio of 7.7:1) for magnification maps than its lossless counterparts (e.g. bzip2, lzma) while keeping a low level of introduced errors. We also showed that for magnification maps convolved with a quasar profile, the compression ratios are much higher than in our lossless experiment (up to 325,812:1) while also being in a near lossless regime (RMSE≤0.01). Our case study results suggest that JPEG2000 could be suitable for other numerical datasets. In particular, well suited candidate are likely to be data structure such as gridded data or volumetric data. In our case study, results suggest that such compression will still enable us to do our science.

More details can be found in the following paper: