Downloading and Installation

Instructions for downloading and installing the Pre-Processing Pipeline are summarised below.

Obtaining the Pre-Processing Pipeline

The pipeline is written in the Common Workflow Language (CWL) and consists of CWL workflows, which can be obtained with:

$ git clone https://git.astron.nl/RD/preprocessing-cwl

Manual Installation

To run the pipeline, you need a CWL runner, such as cwltool or toil. It’s recommended to install these in a virtual environment:

$ python3 -m venv venv
$ source venv/bin/active
$ pip install cwltool toil[cwl]

Note that it’s also necessary to install NodeJS, which can either be installed via your favourite package manager or using the nodejs-wheel Python package.

Additionally, the Pre-Processing Pipeline depends on the following software for processing:

• LofarStMan

• DP3

• AOFlagger

• SAGECal

• Casacore

• EveryBeam

Please follow their respective installation instructions.

Docker Installation

To run the pipeline within a Docker container, please build the image located in the docker/pipeline directory of the repository:

$ docker build ../.. -f Dockerfile -t preprocess:latest

Please modify and give the image an appropriate tag with the -t option. This requires one to install Docker on their system, either Docker Engine (only available for Linux) or Docker Desktop (available for Windows, MacOS, and Linux). Instead of Docker, one could also use Apptainer (previously called Singularity) on Linux.