Skip to content
Snippets Groups Projects

No story: ingest_envoy fixes for EVLA SDM ingestion

Merged Daniel Lopez-Sanders requested to merge no-story-ingest-envoy-evla-sdm-fixes into development

MR Work

  1. Updating unit tests to use and expect the Workspaces staging directory for EVLA EB ingestion.
  2. Changing trigger_ingest's subprocess.run call to not write out the stdout and stderr to sys.stdout and sys.stderr, because that causes an error (run this locally to check it). The error when I ran ingst_envoy on dev was ValueError: stdout and stderr arguments may not be used with capture_output. Please see the code at the bottom of this MR description for an example of this.
  3. Logging out the calls to the ingest pex and curator.
  4. Fixing a passed parameter to the curator launcher so it gets a Path object instead of a string.
  5. Updating other unit tests as appropriate.
  6. The logging changes from !1915 (merged).

How to Test

  1. Run faultchecker on dev.
  2. Choose an EB missing from /home/mchammer and search for it on the dev Archive UI (to make sure it's not there).
  3. Put a metadata.json in that EB's /home/mchammer directory, filling in the information in the workflow template for ingest_obs (you can just copy the one from /home/mchammer/evla/mcaf/workspace/25A-099.sb47832121.eb48262287.60762.35817085648 and change the project code and data source). IMPORTANT: Please note that in the end, this file will be in the workflow directory generated by Workspaces on lustre as with VLBA ingestion.
  4. Run ingest_envoy on dev: /lustre/aoc/cluster/pipeline/dsoc-dev/workspaces/sbin/ingest_envoy --observation [path to EB source dir on /home/mchammer]/metadata.json [path to EB source dir on /home/mchammer].
  5. You may need to re-index Solr.
  6. See the EB on the dev Archive UI!

Outstanding Issues

As far as I know, the only outstanding issue is the logging changes already in discussion for the ingest pex run using subprocess.run. Addressed in !1915 (merged).

Code Example

import subprocess

# Correct usage with capture_output
result = subprocess.run(['ls', '-l'], capture_output=True, text=True)
print(result.stdout)
print(result.stderr)

# Correct usage without capture_output, capturing stdout and stderr separately
result = subprocess.run(['ls', '-l'], stdout=subprocess.PIPE, stderr=subprocess.PIPE, text=True)
print(result.stdout)
print(result.stderr)

# Incorrect usage, will raise ValueError
try:
    result = subprocess.run(['ls', '-l'], capture_output=True, stdout=subprocess.PIPE)
except ValueError as e:
    print(f"Caught expected error: {e}")
Edited by Daniel Lopez-Sanders

Merge request reports

Loading
Loading

Activity

Filter activity
  • Approvals
  • Assignees & reviewers
  • Comments (from bots)
  • Comments (from users)
  • Commits & branches
  • Edits
  • Labels
  • Lock status
  • Mentions
  • Merge request status
  • Tracking
  • Loading
  • Loading
  • Loading
  • Loading
  • Loading
  • Loading
  • Loading
  • Loading
  • Loading
  • Loading
Please register or sign in to reply
Loading