Provenance in Practice - ASOV meeting

14 déc. 2021, 09:30 → 15 déc. 2021, 12:00 Europe/Paris

Online

Mathieu Servillat (LUTH, Observatoire de Paris - CNRS)

The objective of this meeting is to bring together the ASOV community and beyond to exchange on the needs expressed and solutions proposed to manage provenance information in the astronomy domain.

Provenance in practice has been discussed within the ESCAPE European project (with a dedicated workshop [1]), and at the last ADASS and IVOA conferences (see e.g. [2]). Building on prototype implementations, a full provenance management system has been designed [3]. For simpler use, the concept of last-step provenance was also recently proposed to make provenance easier to adopt.

We plan to have presentations on the morning of Tuesday 14, so that participants express their needs and solutions (under consideration or in place) with regard to provenance. Tuesday afternoon, we will have a discussion on identified common solutions, and demonstrations of some existing tools (voprov, OPUS, logprov, ...). On Wednesday 15 morning, we plan to specify the necessary VO standards or notes, and potential further developments required.

Here are some topics that may be addressed during the meeting :

• Existing and considered provenance recommendations at IVOA
  • Data models
  • Simplified views of the data model (last-step provenance)
  • Access protocols (ProvSAP, ProvTAP)
  • Serializations
• Use cases for the capture and the use of provenance information
  • Inputs from ESCAPE partners
  • Finding the right granularity for the capture
  • Connections with workflows
• Demo of implemented tools (voprov, ProvSAP, ProvTAP, logprov...)
• Towards a full provenance management system

[1] https://indico.in2p3.fr/event/21913/page/2641-summary
[2] https://arxiv.org/abs/2101.08691
[3] https://arxiv.org/abs/2109.07751
 

---

Collaborative notes:
https://etherpad.in2p3.fr/p/prov-in-practice

Ice-breaker document:
https://docs.google.com/document/d/1CbxUtHun4_iSJJppmjTiPZ0pHd7LmZMD8gThqRg8v5I/edit?usp=sharing

---

 

mardi 14 décembre

1 Welcome

Status and Use cases

Presentation of the Provenance vision at IVOA, and round table of participants to describe their needs and their data products for which provenance is relevant

Président de session: François Bonnarel (CDS ObAS)

2 Introduction and status of Provenance at IVOA

Orateur: Dr Mathieu Servillat (LUTH, Observatoire de Paris - CNRS)

2021-12-14 provenance in practice @ProvASOV.pdf

3 Catalog of Coronal Hole detection in TAP service

Coronal holes (CH) are seen as dark features on EUV images of the Sun. They are the source of the fast solar wind, and as such are closely monitored. The CH feature recognition module named SPoCA-CH was developped at the Royal Observatory of Belgium. Since 2010, one version of this software is running at LMSAL and provides near-real time detection of coronal holes. We would like to provide the outputs from an updated version of SPOCA-CH as a catalog for a VESPA TAP service. In this talk, we will discuss our progress, as well as questions regarding the encoding of provenance information within this TAP service.

Orateur: Véronique Delouille (Observatoire royal de Belgique)

CH_Catalog_DPF_VDelouille.pdf

4 Planetary data - VESPA

Orateur: Stephane Erard (LESIA/PADC - Obs Paris)

Provenance-Erard-2021.key.pdf

5 The Outer Solar System Origins Survey

Orateur: Jean-Marc Petit (UTINAM)

Provenance_workshop_20211214_OSSOS_JMP.pdf

6 AMHRA and MP3C

AMHRA: Analysis and Modeling at High Angular Resolution
MP3C: Minor Planet Physical Properties Catalogue

Orateur: Nicolas Bruot (OCA)

20211214_provenance_en_pratique_oca_intro.pdf

Demo and discussions

Président de session: Mireille Louys

7 Provenance SKA requirements

Orateur: Jesus Salgado (SKAO)

Provenance for SKA.pdf

8 Pipeline execution and tracing with CTADIRAC

Orateurs: Michèle Sanguillon (LUPM - IN2P3 - CNRS), Patrick Maeght

211215_Provenance_CTADIRAC.pdf

9 In-pipeline provenance capture for LST1 (CTA prototype telescope)

Orateur: Jose Enrique Ruiz (IAA)

LSTProvASOV.pdf

10 How to use the voprov Python package

Orateur: Dr Mathieu Servillat (LUTH, Observatoire de Paris - CNRS)

voprov.ipynb

11 Provenance capture with OPUS

ProvSAP access to provenance graphs

Orateur: Dr Mathieu Servillat (LUTH, Observatoire de Paris - CNRS)

15:20 Coffee break

12 Provenance integration to Vizier

Orateur: Gilles Landais (CDS)

vofrance-Provenance_Vizier.pdf

13 Query provenance using ProvTAP

Orateur: François Bonnarel (CDS ObAS)

ProvTAPstatusASOVDec2021_v2.pdf

14 Definition of a last-step flat provenance

Orateur: Dr Mathieu Servillat (LUTH, Observatoire de Paris - CNRS)

15 Preparation of next day topics - discussion

mercredi 15 décembre

Conclusion and next steps

discussion/conclusions that may be focused on specific questions, and should lead to a road map of future implementations and recommendations.

Président de session: Dr Mathieu Servillat (LUTH, Observatoire de Paris - CNRS)

Discussion

• Last-step flat provenance in more details
  • requirements
  • mandatory fields?
  • FITS keywords?
  • map with other standards (OGIP?)
• Discuss logprov Python package
  • demo
  • doc topics
  • associated coding guidelines
• voprov improvement 
  • should simplify the declaration of descriptions and configuration
  • example of internal OPUS code to autogenerate the prov graph
  • doc and debugging
• Provenance and workflows
  • connector to workflow in last-step provenance
  • mapping CWL to Provenance descriptions
  • workflow attributes: ref to file, and format 
• Storage
  • what is a "provenance record"
  • specifications for a DB ingestion system: from a file, ingest in a relational database structured with the IVOA ProvDM
  • test of NoSQL solution / graph databases
• Query provenance
  • ProvTAP views: should answer use cases
  • query an element of a graph?

 

Road-map

• logprov
  • create tutorial to build an interface that executes any code (outside Python) with prov tracking
  • documentation
  • dev:
    • use voprov instead of prov
    • to be tested with last version of gammapy
• voprov
  • publish notebook on voprov github
    • use generic filenames
    • add location
    • add EntityDescription.content_type
  • documentation
  • link in doc to the OPUS function that auto generates a graph
  • bugs:
    • update dependency to prov==2.0
  • dev:
    • shortcuts to declare parameter and descriptions
    • shortcut to prov:label
    • YAML output
      • to be defined (IVOA Note?)
    • JSON schema?
    • serializations proposed in voprov
• Last-step flat provenance
  • Prepare an IVOA Note
  • based on ids
  • important to include used and generated ids
  • How to resolve those ids?
    • ProvURL: link to a ProvSAP that can resolve the ids
      • PROVURL keyword in FITS
      • ServiceDescriptor in VOTable?
• ProvTAP
  • Draft before Christmas
  • Discussion in January on IVOA draft
  • Discussion with Gilles on ProvTAP usage
• Provenance and workflows
  • test CWL on a use case, check mapping with prov
    • HESS DR1
    • CTA data challenge
  • internship?
• Domain specific discussions
  • to take place in January-February within ESCAPE
  • SKA (with SKAO, Jesus and IAA, Lourdes, Julian, etc.) 
  • Solar data (with ROB)
  • CTA