WS-3239 EVLA SDM ingest_obs
These are the changes to ingest_envoy
necessary for EVLA SDM ingestion. It gets from receiving the workflow request via curl to calling ingest. No changes to the workflow service and wrester ended up being necessary.
Main changes:
- Converted the bash observation product collector to Python.
- Modified the product collector to only use hard links if the source and staging directories are on the same device, otherwise it'll perform a copy (to prevent cross-device errors).
- Cleaned up path vs string usage in the collectors.
To test:
- Put an SDM directory somewhere with some XML files in it.
- Submit an
ingest_obs
workflow request pointing to the SDM directory. Here's what I did (essentially what StackStorm will do):
import requests
url = "http://workflow:3456/workflows/ingest_obs/requests/create-and-submit"
params = {
"telescope": "EVLA",
"data_src": "/lustre/mchammer/evla/sdm/123-ABC_sb30512915_1.57112.35963144676",
"project_code": "123-ABC",
}
response = requests.post(url, json=params)
- View results. The source directory will be renamed to
*.running
and then*.failed
or*.ingested
, the staging directory should be named*.running
and theingest_obs
spool will contain the logs.
Edited by Daniel Nemergut
Merge request reports
Activity
Please register or sign in to reply