No story: ingest_envoy fixes for EVLA SDM ingestion
MR Work
- Updating unit tests to use and expect the Workspaces staging directory for EVLA EB ingestion.
- Changing
trigger_ingest
'ssubprocess.run
call to not write out the stdout and stderr tosys.stdout
andsys.stderr
, because that causes an error (run this locally to check it). The error when I raningst_envoy
on dev wasValueError: stdout and stderr arguments may not be used with capture_output.
Please see the code at the bottom of this MR description for an example of this. - Logging out the calls to the
ingest
pex and curator. - Fixing a passed parameter to the curator launcher so it gets a
Path
object instead of a string. - Updating other unit tests as appropriate.
- The logging changes from !1915 (merged).
How to Test
- Run
faultchecker
on dev. - Choose an EB missing from
/home/mchammer
and search for it on the dev Archive UI (to make sure it's not there). - Put a
metadata.json
in that EB's/home/mchammer
directory, filling in the information in the workflow template foringest_obs
(you can just copy the one from/home/mchammer/evla/mcaf/workspace/25A-099.sb47832121.eb48262287.60762.35817085648
and change the project code and data source). IMPORTANT: Please note that in the end, this file will be in the workflow directory generated by Workspaces on lustre as with VLBA ingestion. - Run
ingest_envoy
on dev:/lustre/aoc/cluster/pipeline/dsoc-dev/workspaces/sbin/ingest_envoy --observation [path to EB source dir on /home/mchammer]/metadata.json [path to EB source dir on /home/mchammer]
. - You may need to re-index Solr.
- See the EB on the dev Archive UI!
Outstanding Issues
As far as I know, the only outstanding issue is the logging changes already in discussion for the Addressed in !1915 (merged).ingest
pex run using subprocess.run
.
Code Example
import subprocess
# Correct usage with capture_output
result = subprocess.run(['ls', '-l'], capture_output=True, text=True)
print(result.stdout)
print(result.stderr)
# Correct usage without capture_output, capturing stdout and stderr separately
result = subprocess.run(['ls', '-l'], stdout=subprocess.PIPE, stderr=subprocess.PIPE, text=True)
print(result.stdout)
print(result.stderr)
# Incorrect usage, will raise ValueError
try:
result = subprocess.run(['ls', '-l'], capture_output=True, stdout=subprocess.PIPE)
except ValueError as e:
print(f"Caught expected error: {e}")
Edited by Daniel Lopez-Sanders
Merge request reports
Activity
Please register or sign in to reply