Ingestor Installation

The OpenEM Ingestor reads datasets from a facility data directory, extracts metadata, creates the corresponding SciCat records, and asks the PSI Globus Proxy to transfer the data. The supported deployment is maintained in the OpenEM deployment repository. Use the files in that repository as the source of truth instead of maintaining a separate copy of the Ingestor Compose configuration.

This guide describes the standard ExtGlobus setup. It assumes that the data directory, Globus Connect Server endpoint, storage gateway, and mapped collection have already been configured as described in the Globus installation guide.

Before You Begin

The target host requires:

• Docker Engine with the Compose plugin; see the official Docker Engine installation guide and Compose plugin guide;
• a local data directory that is also exposed by the facility's Globus mapped collection;
• DNS for the public Ingestor hostname;
• a valid TLS certificate for that hostname; and
• outbound HTTPS access to GitHub, GitHub Container Registry, the selected SciCat environment, the identity provider, and the PSI Globus Proxy.

The examples use the following placeholders. Replace them with values for the facility; do not copy the example values unchanged into production.

Required value	Example placeholder	Source
Facility code	example	Agree with SciCat Support
Ingestor hostname	ingestor.example.org	Facility DNS administrator
Data directory	/srv/openem/data	Facility storage administrator
Display name for the directory	OpenEM Data	Facility choice
OIDC client ID	openem-ingestor-example	SciCat Support
Globus source facility	EXAMPLE	SciCat Support
Deployment	qa initially	Facility and SciCat Support

Register the Globus endpoint and collection with SciCat Support before configuring the Ingestor. Support supplies or confirms the OIDC client ID and the source facility identifier used by the PSI Globus Proxy. The Globus endpoint ID and collection ID are registered on the proxy side; they are not Ingestor .env variables in the current deployment.

Important

The public OIDC callback URL must match the selected deployment exactly: https://ingestor.example.org/qa/callback for QA, https://ingestor.example.org/dev/callback for development, and https://ingestor.example.org/callback for production. Ask SciCat Support to confirm the callback before testing login.

1. Check Out the Deployment Repository

Choose a directory managed by the service operator. /opt/openem is used here as an example:

export OPENEM_INSTALL_DIR=/opt/openem
sudo install -d -o "$(id -u)" -g "$(id -g)" "$OPENEM_INSTALL_DIR"
git clone https://github.com/SwissOpenEM/openem-deployment.git \
  "$OPENEM_INSTALL_DIR/openem-deployment"
cd "$OPENEM_INSTALL_DIR/openem-deployment"

Run all subsequent ./compose.sh commands from this repository directory.

2. Configure the Facility

Create the facility configuration from the repository template:

cp .env.example .env
chmod 600 .env

Edit .env and replace the facility values. A minimal ExtGlobus configuration looks like this:

FACILITY=example
INGESTOR_DOMAIN=ingestor.example.org

HOST_COLLECTION_PATH=/srv/openem/data
HOST_COLLECTION_NAME="OpenEM Data"

GLOBUS_SOURCE_FACILITY=EXAMPLE
OIDC_CLIENT_ID=openem-ingestor-example

LIFESCIENCE_EXTRACTOR_ADDITIONAL_PARAMS="--cs 2.7"

# Recommended for reproducible production deployments. Select a published tag.
INGESTOR_VERSION=v1.1.0

Available Ingestor tags and their changes are listed on the Ingestor releases page. If INGESTOR_VERSION is omitted, the deployment uses latest.

The variables have the following meanings:

Variable	Meaning
FACILITY	Facility code used in naming defaults
INGESTOR_DOMAIN	Public hostname without https:// or a path
HOST_COLLECTION_PATH	Absolute path to the readable data directory on the host
HOST_COLLECTION_NAME	Name shown to users in the Ingestor file browser
GLOBUS_SOURCE_FACILITY	Source identifier configured in the PSI Globus Proxy
OIDC_CLIENT_ID	Facility client registered in the identity provider
INGESTOR_VERSION	Container image tag; pin a release for predictable upgrades
LIFESCIENCE_EXTRACTOR_ADDITIONAL_PARAMS	Optional arguments for the life-science metadata extractor

By default, GLOBUS_COLLECTION_ROOT_PATH is the same as HOST_COLLECTION_PATH. Set it explicitly only if the root path known to Globus differs from the host path:

GLOBUS_COLLECTION_ROOT_PATH=/path/known/to/globus

The Compose file mounts HOST_COLLECTION_PATH read-only at the same absolute path inside the container. The account used by the container therefore needs read permission on files and search permission on every parent directory. To run the container with a dedicated host account rather than the default user, add its numeric IDs:

UID=1001
GID=1001

Check the path and permissions before starting the service:

test -d /srv/openem/data
namei -l /srv/openem/data

Note

Do not edit services/ingestor/compose.yaml for a normal installation and do not create services/ingestor/config/.env. The current deployment generates the Ingestor configuration from Compose and merges the supported environment files through compose.sh.

3. Select the SciCat Deployment

The repository contains PSI-owned settings for three environments:

Deployment argument	SciCat environment	Local port	Public path
dev	Development	8080	/dev
qa	Quality assurance	8081	/qa
production	Production	8082	/

Start with QA unless SciCat Support directs otherwise. compose.sh merges configuration in this order, with later files taking precedence:

1. services/ingestor/config/<deployment>/env.<deployment>;
2. .env for shared facility settings; and
3. .env.<deployment> for local, deployment-specific overrides.

Do not change the versioned files below services/ingestor/config/. For example, to ensure that QA is reachable only through an Apache reverse proxy on the same host, create .env.qa containing:

INGESTOR_PORT=127.0.0.1:8081

Create equivalent .env.dev or .env.production overrides when those deployments are enabled. The value works with the Compose short port syntax and prevents direct network access to the unencrypted backend port.

Render and inspect the final configuration before creating a container:

./compose.sh qa config

Confirm at least the image tag, public hostname, /qa path prefix, source and destination facility identifiers, SciCat URLs, OIDC issuer, port binding, and read-only data-directory mount. The implementation of this merge is available in compose.sh, and the generated application settings are defined in the Ingestor Compose file.

4. Start and Test the Ingestor

Pull the selected image and start QA:

./compose.sh qa pull
./compose.sh qa up -d

Check the container state, logs, and local version endpoint:

./compose.sh qa ps
./compose.sh qa logs --tail=100
curl --fail --show-error http://127.0.0.1:8081/version

The container should be Up, the version endpoint should return JSON, and the logs should contain no startup errors. Follow the logs while diagnosing a problem:

./compose.sh qa logs --follow

Press Ctrl+C to stop following the output; this does not stop the container.

5. Publish the Service with HTTPS

A reverse proxy terminates TLS and forwards requests to the local Ingestor port. Use one of the following approaches.

Option A: Existing Apache from Globus Connect Server

Globus Connect Server installs Apache on port 443. When the Ingestor and GCS share a host, add a separate virtual host for the Ingestor hostname instead of starting a second proxy on the same port. The hostname must resolve to this host and its certificate must already be available.

Enable the required Apache modules:

sudo a2enmod proxy proxy_http ssl

Create /etc/apache2/sites-available/openem-ingestor.conf. This QA-only example uses the same hostname and path configured above; replace the hostname and certificate paths:

<VirtualHost *:443>
    ServerName ingestor.example.org

    SSLEngine On
    SSLCertificateFile /etc/letsencrypt/live/ingestor.example.org/fullchain.pem
    SSLCertificateKeyFile /etc/letsencrypt/live/ingestor.example.org/privkey.pem

    ProxyRequests Off
    ProxyPreserveHost On

    RedirectMatch 308 ^/qa$ /qa/
    ProxyPass        /qa/ http://127.0.0.1:8081/ retry=0
    ProxyPassReverse /qa/ http://127.0.0.1:8081/

    ErrorLog ${APACHE_LOG_DIR}/openem-ingestor-error.log
    CustomLog ${APACHE_LOG_DIR}/openem-ingestor-access.log combined
</VirtualHost>

The trailing slashes are intentional: Apache removes the public /qa/ prefix when forwarding the request, while the Ingestor still uses that prefix when it constructs public callback URLs. Do not add permissive CORS headers; the Ingestor handles its own browser-access policy.

Enable and validate the virtual host, then reload Apache without interrupting active services:

sudo a2ensite openem-ingestor.conf
sudo apache2ctl configtest
sudo systemctl reload apache2

This arrangement follows the Globus recipe for concurrent hosting of GCS and another application on port 443. Apache's official documentation explains ProxyPass and ProxyPassReverse and name-based virtual hosts. Certificate issuance and renewal are site-specific; if Certbot is used, follow its Apache instructions and ensure it does not replace the GCS virtual-host configuration.

For multiple deployments on one hostname, place the more specific routes before the production / route:

RedirectMatch 308 ^/dev$ /dev/
RedirectMatch 308 ^/qa$ /qa/
ProxyPass        /dev/ http://127.0.0.1:8080/ retry=0
ProxyPassReverse /dev/ http://127.0.0.1:8080/
ProxyPass        /qa/  http://127.0.0.1:8081/ retry=0
ProxyPassReverse /qa/  http://127.0.0.1:8081/
ProxyPass        /     http://127.0.0.1:8082/ retry=0
ProxyPassReverse /     http://127.0.0.1:8082/

Option B: Bundled Traefik Proxy

Use the bundled proxy when no existing service owns ports 80 and 443. Set the certificate paths in .env:

TLS_CERT_FILE=/etc/letsencrypt/live/ingestor.example.org/fullchain.pem
TLS_KEY_FILE=/etc/letsencrypt/live/ingestor.example.org/privkey.pem

Then start the proxy together with QA:

./compose.sh proxy qa up -d

The proxy implementation and mounts are documented in the deployment repository's proxy Compose file. Do not use this option on a GCS host where Apache already listens on ports 80 or 443.

6. End-to-End Verification

Test the external endpoint:

curl --fail --show-error https://ingestor.example.org/qa/version

Then verify the complete workflow:

1. Open the matching SciCat environment and sign in.
2. Open the Ingestor and complete the OIDC login.
3. Confirm that OpenEM Data is shown and that only the intended data tree can be browsed.
4. Ingest a small, non-sensitive test dataset.
5. Confirm metadata extraction, SciCat record creation, and the Globus transfer to PSI.

If the local version endpoint works but the public endpoint does not, inspect DNS, the TLS certificate, Apache or Traefik logs, and the proxy path. If login fails, compare the generated callback URL with the OIDC client's registered redirect URL. If files are missing, check host permissions and verify that HOST_COLLECTION_PATH and GLOBUS_COLLECTION_ROOT_PATH describe the same data tree from their respective perspectives.

Updating and Restarting

Read upstream changes before updating, especially changes to .env.example and the deployment-specific environment files. Then update the repository and recreate only the intended deployment:

cd /opt/openem/openem-deployment
git pull --ff-only
./compose.sh qa config
./compose.sh qa pull
./compose.sh qa up -d --force-recreate
./compose.sh qa logs --tail=100
curl --fail --show-error http://127.0.0.1:8081/version

docker compose down is not required for a routine image update and causes an avoidable outage. To roll back, restore the previous INGESTOR_VERSION in .env, run pull, and recreate the deployment again.

To stop QA deliberately:

./compose.sh qa down

For application-level configuration details, API documentation, and source code, see the OpenEM Ingestor repository.