• • IPAW'06 presentation and paper gLiteJob Provenance.

• slides from the workshop

• cesnet-slides.pdf:

Full implementation

The query client is the same as for #1.

• query3.log: Query #3 results

• query5.log: Query #5 results

• query4-pondeli.log: Query #4 - Monday

• query4-ctvrtek.log: Query #4 - Thursday

• JP Primary Storage (JPPS), where the data on jobs are stored permanently, can be queried for any attribute of a particular job. However, a concrete jobid must be known.

• JP Index Server (JPIS) is a configurable cache of subset of jobs and attributes. It can search for jobs matching query criteria, specified as comparison of an attribute with a constant value. Concrete JobIds^? needn't be known.

Similarly to Query #9, we might introduce dummy "producer jobs" (i.e. having the particular data file assined as their output), that would carry the annotation. However, we consider this approach too artifitial.

Pseudocode:

• JPIS query: find jobs (workflows) having http://twiki.ipaw.info/Challenge/CESNET/Annotations:studyModality = input value
• for each found workflow W
  • JPIS query: find jobs where http://egee.cesnet.cz/en/Schema/LB/Attributes:parent = W and IPAW_STAGE = 5
  • JPPS query: retrieve IPAW_OUTPUT and other annotations of the jobs

• A simple workaround is storing all annotations in a similar way as IPAW_HEADER tag, i.e. an attribute holding both annotation name and value. However, this approach would not allow more complicated queries on the annotation values.
• A better workaround is attaching a known "Annotation" attribute to each job. This attribute would hold names of all existing annotations for this job. The user would first query for values of this attribute, and then choose which real annotations (JP attributes) to retrieve.
• Extending JPPS query interface to support queries like "all attributes of this job falling into namespace X".

The code can be also easily modified to record the graph traversal (details on workflow nodes) leading to a particular file, and display it with the file in a similar way as in previous queries.

Query #4

Inputs

Outputs

Implementation

Full implementation

Sample output

Comments

The filter "ran on Monday" is quite challenging. Currently, we implement it at client side which is not a scalable solution. However, the JP concept foresees a solution of the issue via an already defined interface to type plugin. A plugin, for a concrete type, defines the following methods:

• transformation of a value to a "queryable database form", which is stored at JPIS when a value of this type arrives there (in addition to the literal value)
• user query comparison operators that transform a compared value into a SQL expression

Then, upon arrival to JPIS, weekday number would be extracted from the timestamp and stored in an extra database column. The plugin would also define an operator isWeekDay(x) that would be tranformed at query time to an expression refering to the new column. Therefore the condition would be evaluated at the SQL level, i.e. in the most effective way.

Query #5

Inputs

Outputs

Implementation

Sample output

Comments

Query #6

Inputs

Outputs

Implementation

Sample output

Comments

Query #7

Inputs

Outputs

Implementation

Sample output

Comments

Query #8

Inputs

Outputs

Implementation

Sample output

Comments

Query #9

Inputs

Outputs

Implementation

Sample output

Comments

The output bellow is cut and reformated, here is the original output.

jobid https://skurut1.cesnet.cz:9000/hvkpZCsRsiqrxs5K_bo7Ew: attr IPAW_STAGE: 5 attr IPAW_PROGRAM: convert attr IPAW_INPUT: gsiftp://umbar.ics.muni.cz:1414/home/mulac/pch06/blabla-x.pgm attr IPAW_OUTPUT: gsiftp://umbar.ics.muni.cz:1414/home/mulac/pch06/blabla-x.gif attr CE: skurut17.cesnet.cz:2119/jobmanager-lcgpbs-voce

jobid https://skurut1.cesnet.cz:9000/02ZaAADKyebzggYPp4M9tA: attr IPAW_STAGE: 4 attr IPAW_PROGRAM: slicer attr IPAW_INPUT: gsiftp://umbar.ics.muni.cz:1414/home/mulac/pch06/blabla.hdr gsiftp://umbar.ics.muni.cz:1414/home/mulac/pch06/blabla.img attr IPAW_OUTPUT: gsiftp://umbar.ics.muni.cz:1414/home/mulac/pch06/blabla-x.pgm attr CE: skurut17.cesnet.cz:2119/jobmanager-lcgpbs-voce

jobid https://skurut1.cesnet.cz:9000/wGMnTvCILtiSTi7ZOQwfTQ: attr IPAW_STAGE: 3 attr IPAW_PROGRAM: softmean attr IPAW_INPUT: gsiftp://umbar.ics.muni.cz:1414/home/mulac/pch06/anatomy1-resliced.img ... attr IPAW_OUTPUT: gsiftp://umbar.ics.muni.cz:1414/home/mulac/pch06/blabla.img gsiftp://umbar.ics.muni.cz:1414/home/mulac/pch06/blabla.hdr attr CE: skurut17.cesnet.cz:2119/jobmanager-lcgpbs-voce

jobid https://skurut1.cesnet.cz:9000/9d0XMwfPuefR9woAFkDplQ: attr IPAW_STAGE: 2 attr IPAW_PROGRAM: reslice attr IPAW_INPUT: gsiftp://umbar.ics.muni.cz:1414/home/mulac/pch06/anatomy3.warp ... attr IPAW_OUTPUT: gsiftp://umbar.ics.muni.cz:1414/home/mulac/pch06/anatomy3-resliced.img ... attr CE: skurut17.cesnet.cz:2119/jobmanager-lcgpbs-voce

jobid https://skurut1.cesnet.cz:9000/RglBtUz0IzwSeM32KLnHPg: attr IPAW_STAGE: 2 attr IPAW_PROGRAM: reslice attr IPAW_INPUT: gsiftp://umbar.ics.muni.cz:1414/home/mulac/pch06/anatomy4.warp ... ...

jobid https://skurut1.cesnet.cz:9000/wdWQHL0-RXkd3VeNcSrTaw: attr IPAW_STAGE: 2 attr IPAW_PROGRAM: reslice attr IPAW_PARAM: attr IPAW_INPUT: gsiftp://umbar.ics.muni.cz:1414/home/mulac/pch06/anatomy1.warp ... ...

jobid https://skurut1.cesnet.cz:9000/xwIsN2JgGfsRuvYwh0QXsw: attr IPAW_STAGE: 2 attr IPAW_PROGRAM: reslice attr IPAW_INPUT: gsiftp://umbar.ics.muni.cz:1414/home/mulac/pch06/anatomy2.warp ... ...

jobid https://skurut1.cesnet.cz:9000/yM3sz8v6WCIPgi5-0m8L4w: attr IPAW_STAGE: 1 attr IPAW_PROGRAM: align_warp attr IPAW_PARAM: -m 12, -q attr IPAW_INPUT: gsiftp://umbar.ics.muni.cz:1414/home/mulac/pch06/anatomy4.img gsiftp://umbar.ics.muni.cz:1414/home/mulac/pch06/reference.img attr IPAW_OUTPUT: gsiftp://umbar.ics.muni.cz:1414/home/mulac/pch06/anatomy4.warp attr CE: skurut17.cesnet.cz:2119/jobmanager-lcgpbs-voce

jobid https://skurut1.cesnet.cz:9000/s47ihjBHQXqPkkNwA2iazg: attr IPAW_STAGE: 1 attr IPAW_PROGRAM: align_warp attr IPAW_PARAM: -m 12, -q attr IPAW_INPUT: gsiftp://umbar.ics.muni.cz:1414/home/mulac/pch06/anatomy2.img ... ...

...

Query #2

Inputs

Outputs

Implementation

Full implementation

Sample output

Query #3

Inputs

Outputs

Implementation

Full implementation

Sample output

Comments

• query1.log: Query #1 results

• query2.log: Query #2 results

• query3.log: Query #3 results

• Project Overview: Job Provenance (JP for short) is a part of the gLite Grid middleware implementation

The LB trace is uploaded as a raw LB dumpfile, three sample snapshots are available here (files dump[123]). JP provides the user with an interface to retrieve such raw files, and their format is public in principle (NetLogger^? ULM according to draft-abela-05, LB specific fields are documented in LB User's Guide). However, access to the raw files is not supposed to be a typical JP usage.

On the contrary, the end user of JP sees all that available data transformed into the form of logical JP attributes, "namespace:name = value" pairs. Attribute values are digested from the raw traces JP plug-in modules, hiding internal structure, syntax, format version, and other implementation details.

At this level the provenance trace of executed workflow is represented by a set of JP attributes and its values assigned to both the workflow and all its subjobs (nodes).

There are the following classes of attributes:

• JP system ones (namespace http://egee.cesnet.cz/en/Schema/JP/System):
  • jobId
  • owner: identity of the job submitter
  • regtime: when the job was registered with the middleware
• digested from LB trace, conforming to schema =http://egee.cesnet.cz/en/Schema/LB/JobRecord=
• digested from JDL, describing workflow structure (namespace http://egee.cesnet.cz/en/Schema/JP/Workflow):
  • ancestor: JobId^?(s) of immediately preceeding job in the workflow
  • successor: JobId^?(s) of immediately following jobs in the workflow
• unqualified user tags, logged via LB (see above); they are reported in the namespace http://egee.cesnet.cz/en/WSDL/jp-lbtag

For the specific implementation of the challenge workflow we use LB user tags to store additional information about the workflow nodes. JP turns these values into attributes of the 4th kind on the list above. The following table summarizes their meaning:

Find the process that led to Atlas X Graphic / everything that caused Atlas X Graphic to be as it is. This should tell us the new brain images from which the averaged atlas was generated, the warping performed etc.

Inputs

Outputs

• JobId^?, user, and execution time of the workflow that produced the graphics
• List of nodes (subjobs) of the workflow that contributed to the queried file:
  • input and output files
  • stage of the workflow, program name and parameter values

Implementation

The query is implemented as a graph search where the vertices are nodes of the DAG and oriented edges are given by the ANCESTOR attribute. The search is seeded with a JPIS query, retrieving JobId^? of the last node of the workflow which produced the queried file directly, i.e. typically the convert utility.

Pseudocode:

• JPIS query: JobId^? of DAG node having IPAW_OUTPUT = 'Atlas X Graphic'
• initialise job_list with the retrieved JobId^?
• (graph search) while there are unprocessed elements in job_list
  • pick a list element job
  • JPPS query: all values of ANCESTOR attribute of job
  • insert each retrieved value into job_list unless it is already there
• for each element of job_list
  • JPPS query: attributes IPAW_INPUT, IPAW_OUTPUT, IPAW_PROGRAM, IPAW_PARAM, IPAW_STAGE
• sort job_list according to IPAW_STAGE
• pretty-print job_list, including all the retrieved attributes

Full implementation

Sample output

TODO

Comments

Challenge workflow

• procedures become nodes of the DAG, i.e. they are turned into normal gLite jobs during the DAG processing, and executed on the Grid computing resources. Besides down- and uploading the data files (see bellow) each such job involves running the appropriate AIR, FSL, or ImageMagic^? utility.
• dependencies among procedures are reflected in dependencies of the DAG. Therefore e.g. all four align_warp invocations can run in parallel but softmean must be preceeded by successfull completition of all four reslice instances.
• data items, both input and output, are external files wrt. the workflow implementation because a unified shared filesystem cannot be exepected on the Grid computing resources. Therefore each job is responsible for downloading all its inputs and uploading all its outputs.

In our experimental runs we put the files on a dedicated GridFTP^? server and access (both down- and upload) with the gsiftp:// protocol (solving also access control -- a running gLite job possesses delegated user credentials). Consequently, the data items are identified with their full URL in our implementation.

We might have used the gLite data services, identifying files with GUID's or logical file names. However, this approach would make the implementation more obscure while not exhibiting any important provenance features.

We provide a template for the workflow JDL. It contains placeholders for the data files, details on instantiating and submitting it with gLite command-line tools can be found at this page.

• Project Overview: Job Provenance (JP for short) is a part of the gLite Grid middleware implementation
• Provenance-specific Overview: JP is a job centric system. The Grid job is the primary entity of interest, all data are organised on a per-job basis. JP collects data about job life cycle including job inputs and outputs, infrastructure state and user annotations.

gLite job processing in a nutshell

Upon creation the job is assigned a unique immutable Job Identifier (jobid). The jobid is used to refer to the job all the time during the job life and afterwards.

The user describes the job (i.e. executable, parameters, input files etc.) using the Job Description Languate (JDL), using the extensible Classified Advertisement (classad) syntax. The description may grow fairly complex, including requirements on the execution environment, proximity of input and output storage etc.

Processing of the job can be summarised as follows:

• the job is submitted via a User Interface (either command line or graphical)
• Workload Manager (WM) queues the job and starts finding a suitable Computing Element to execute it
• the job is passed to the chosen Computing Element and runs there
• after completition, the user can retrieve the job output
• all the time, the job is tracked by Logging and Bookkeeping (LB) service, providing the user with the view on the job state and further details
• after retrieving the job output all the middleware data (namely the job trace in LB) on the job are passed to Job Provenance and purged in their original locations
• annotations can be added to the job in the form of tags (name = value pairs) during its life time via LB (even from inside of the running application) or any time afterwards via JP

Besides simple jobs gLite supports also complex ones, job workflows in the form of Directed Acyclic Graphs (DAG). A DAG is completely described, using a nested JDL syntax, as a set of its nodes (simple jobs) and execution dependencies among them. DAG processing is implemented by interfacing the WM planning machinery with the Condor DAGMan.

TODO: references JDL, WMS, LB

The workflow as a DAG in a Grid

To show the nature of Job Provenance (JP) we must use the context of Grid. The reader doesn't need a deep understanding of Grid, but we must introduce many terms and tools that forms a Grid environment.

The Grid environment