J-PAS data management pipeline and archiving
J-PAS survey is going to observe 8000 deg<SUP>2</SUP> in 54 optical
narrow band filters plus 3 broad band ones. The survey will produce 1.2
PB of raw data in six years of observations. The treatment of about 1.5
TB per night, coming from the 14 detectors of the JPCam camera in the
JST/T250, plus one detector in the JAST/T80 camera, shall be performed
during the day after the operations. This contribution presents the
software and hardware architecture designed to process and validate the
data. The processing of the images is divided in two main stages. The
first one, which deals with instrumental correction and data validation,
is run daily. The second stage is run when a tile is completed and do
the combination of the individual corrected frames and weight maps. To
perform the astrometric calibration, image coadding and source
extraction the data management pipeline uses software from the community
which is integrated through Python. The software uses a database to
control the process by storing the operations performed, parameters used
and quality checks. This allows fast reprocessing to retrieve
intermediate stages of the treatment from the raw data for any data
release. This approach saves disk space by avoiding the storage of the
processed individual frames. The data archiving and processing will be
done in a data center 30 km away from the observatory. It will be
equipped with ~2.5 PB of storage capacity to store the raw data and the
final mosaics of the 57 filters, and processing power to deal with the
incoming data.