Downloading and Installation
============================

Instructions for downloading and installing the Pre-Processing Pipeline are summarised below.

Obtaining the Pre-Processing Pipeline
-------------------------------------

The pipeline is written in the `Common Workflow Language (CWL) <https://www.commonwl.org/>`_ and consists of CWL workflows, which can be obtained with:

.. code:: console

    $ git clone https://git.astron.nl/RD/preprocessing-cwl


Manual Installation
-------------------

To run the pipeline, you need a CWL runner, such as ``cwltool`` or ``toil``. It's recommended to install these in a virtual environment:

.. code:: console

    $ python3 -m venv venv
    $ source venv/bin/active
    $ pip install cwltool toil[cwl]

Note that it's also necessary to install NodeJS, which can either be installed via your favourite package manager or using the ``nodejs-wheel`` Python package.

Additionally, the Pre-Processing Pipeline depends on the following software for processing:

* `LofarStMan <https://github.com/lofar-astron/LofarStMan>`_
* `DP3 <https://git.astron.nl/RD/DP3>`_
* `AOFlagger <https://gitlab.com/aroffringa/aoflagger>`_
* `SAGECal <https://github.com/nlesc-dirac/sagecal>`_
* `Casacore <https://github.com/casacore/casacore>`_
* `EveryBeam <https://git.astron.nl/RD/EveryBeam>`_

Please follow their respective installation instructions.

Docker Installation
-------------------

To run the pipeline within a Docker container, please build the image located in the ``docker/pipeline`` directory of the `repository <https://git.astron.nl/RD/preprocessing-cwl>`_:

.. code:: console

    $ docker build ../.. -f Dockerfile -t preprocess:latest

Please modify and give the image an appropriate tag with the ``-t`` option. This requires one to install Docker on their system, either `Docker Engine <https://docs.docker.com/engine/>`_ (only available for Linux) or `Docker Desktop <https://docs.docker.com/desktop/>`_ (available for Windows, MacOS, and Linux). Instead of Docker, one could also use `Apptainer <https://apptainer.org/>`_ (previously called Singularity) on Linux.