Skip to content
Snippets Groups Projects

WS-3239 EVLA SDM ingest_obs

Merged Daniel Nemergut requested to merge ws3239-evla_sdm_ingest_obs into development

These are the changes to ingest_envoy necessary for EVLA SDM ingestion. It gets from receiving the workflow request via curl to calling ingest. No changes to the workflow service and wrester ended up being necessary.

Main changes:

  • Converted the bash observation product collector to Python.
  • Modified the product collector to only use hard links if the source and staging directories are on the same device, otherwise it'll perform a copy (to prevent cross-device errors).
  • Cleaned up path vs string usage in the collectors.

To test:

  1. Put an SDM directory somewhere with some XML files in it.
  2. Submit an ingest_obs workflow request pointing to the SDM directory. Here's what I did (essentially what StackStorm will do):
import requests
url = "http://workflow:3456/workflows/ingest_obs/requests/create-and-submit"
params = {
    "telescope": "EVLA",
    "data_src": "/lustre/mchammer/evla/sdm/123-ABC_sb30512915_1.57112.35963144676",
    "project_code": "123-ABC",
}
response = requests.post(url, json=params)
  1. View results. The source directory will be renamed to *.running and then *.failed or *.ingested, the staging directory should be named *.running and the ingest_obs spool will contain the logs.
Edited by Daniel Nemergut

Merge request reports

Merge request pipeline #21702 passed

Merge request pipeline passed for 5e4a823b

Merged by Daniel NemergutDaniel Nemergut 6 days ago (Apr 11, 2025 8:46pm UTC)

Loading

Pipeline #21762 canceled

Pipeline canceled for 570b2da8 on development

Activity

Filter activity
  • Approvals
  • Assignees & reviewers
  • Comments (from bots)
  • Comments (from users)
  • Commits & branches
  • Edits
  • Labels
  • Lock status
  • Mentions
  • Merge request status
  • Tracking
  • Loading
  • Loading
  • Loading
  • Loading
  • Loading
  • Loading
  • Loading
  • Loading
  • Loading
  • Loading
Please register or sign in to reply
Loading