Modern sky surveys are producing astronomical catalogs with billions of stars and galaxies. What is often important for science is cross-correlating these catalogs and finding the matching objects in several catalogs so that new insights can be gained from all observations at once. This operation, commonly known as ‘cross-matching’, can be extremely computationaly expensive because of the large number of comparisons that need to be performed.

DIRAC team has designed and implemented a system called AXS, or Astronomical Extensions for Spark, that comprises a new cross-matching approach that significantly outperforms other such systems and is capable of cross-matching multi-billion catalogs in tens of seconds on commodity hardware. AXS also contains other functionalities useful to astronomers and is based on Apache Spark, an industry-standard, open-sourced, distributed data processing system.”