First Provenance Challenge

First Provenance Challenge

The inputs to a workflow are a set of new brain images (Anatomy Image 1 to 4) and a single reference brain image (Reference Image). All input images are 3D scans of a brain of varying resolutions, so that different features are evident. For each image, there is the actual image and the metadata information for that image (Anatomy Header 1 to 4). The image data was published with article Frontal-Hippocampal Double Dissociation Between Normal Aging and Alzheimer's Disease by Head, D, Synder, AZ, Girton, LE, Morris, JC, Buckner, RL in the fMRI Data Center Accession Number: 2-2004-1168X.

The stages of the workflow are as follows.

1. For each dimension (x, y and z), the averaged image is sliced to give a 2D atlas along a plane in that dimension, taken through the centre of the 3D image. The output is an atlas data set, using slicer. This tool can be downloaded as part of the FSL suite, available at http://www.fmrib.ox.ac.uk/fsl/.
2. For each atlas data set, it is converted into a graphical atlas image using (the ImageMagick utility) convert.

The full steps, procedures data and parameters are enumerated in the table below. The procedure names are linked to the manual pages for those utilities, and the input and output names to the actual data exchanged between procedures.

First Provenance Challenge

Timetable

• 2006-June: Challenge finalised, participants start!

• 2006-October-15: Comparisons performed, minutes of discussion, proposed next steps uploaded

First Provenance Challenge

• For any data given above, each team should provide a link to an explanation of the representation used so that other participants can interpret it.

As it may be useful to some, we provide sample implementations of the workflow here. This should not preclude the use of any other technology. The implementations assume that the executables referenced above are all installed; they are provided by the two packages AIR (automated image registration) suite and ImageMagick.

• Shell script workflow

Timetable

• 2006-June: Challenge finalised, participants start!

First Provenance Challenge

The inputs to a workflow are a set of new brain images (Anatomy Image 1 to 4) and a single reference brain image (Reference Image). All input images are 3D scans of a brain of varying resolutions, so that different features are evident. For each image, there is the actual image and the metadata information for that image (Anatomy Header 1 to 4). The image data was published with article Frontal-Hippocampal Double Dissociation Between Normal Aging and Alzheimer's Disease by Head, D, Synder, AZ, Girton, LE, Morris, JC, Buckner, RL in the fMRI Data Center Accession Number: 2-2004-1168X.

1. For each new brain image, align_warp compares the reference image to determine how the new image should be warped, i.e. the position and shape of the image adjusted, to match the reference brain. The output of each procedure in the stage is a _warp parameter set_ defining the spatially transformation to be performed (Warp Params 1 to 4).
2. For each warp parameter set, the actual transformation of the image is done by reslice, which creates a new version of the original new brain image with the configuration defined in the warp parameter set. The output is a resliced image.
3. All the resliced images are averaged into one single image using softmean.

1. For each atlas data set, it is converted into a graphical atlas image using (the ImageMagick utility) convert.

The full steps, procedures data and parameters are enumerated in the table below. The procedure names are linked to the manual pages for those utilities, and the input and output names to the actual data exchanged between procedures.

First Provenance Challenge

1. Find all invocations of procedure align_warp using a twelfth order nonlinear 1365 parameter model (see model menu describing possible values of parameter "-m 12" of align_warp) that ran on a Monday.
2. Find all Atlas Graphic images outputted from workflows where at least one of the input Anatomy Headers had an entry global maximum=4095. The contents of a header file can be extracted as text using the scanheader AIR utility.
3. Find all output averaged images of softmean (average) procedures, where the warped images taken as input were align_warped using a twelfth order nonlinear 1365 parameter model, i.e. "where softmean was preceded in the workflow, directly or indirectly, by an align_warp procedure with argument -m 12."

1. A user has annotated some anatomy images with a key-value pair center=UChicago. Find the outputs of align_warp where the inputs are annotated with center=UChicago.
2. A user has annotated some atlas graphics with key-value pair where the key is studyModality. Find all the graphical atlas sets that have metadata annotation studyModality with values speech, visual or audio, and return all other annotations to these files.

Timetable

• 2006-September: Deadline for challenge results to be uploaded
• 2006-September: Face-to-face meeting at which results are discussed
• 2006-October-15: Comparisons performed, minutes of discussion, proposed next steps uploaded

First Provenance Challenge

Timetable

• • Fix provenance challenge proposal TWiki page, with aims, objectives, workflow, sample queries, timetable
  • Upload input, output, intermediate data used in workflow
  • Send provenance challenge call to other possible participants, including:

-- SimonMiles - 7 June 2006

First Provenance Challenge

• Extensions to the example workflow to best illustrate the unique aspects of their system
• Any categorisation of queries that the project considers to have practical value

Example Workflow

We propose an example workflow for creating population-based "brain atlases" from the fMRI Data Center's archive of high resolution anatomical data. The workflow is shown below (click for a pdf version of the image).

First Provenance Challenge

1. Find the process that led to Atlas X Graphic / everything that caused Atlas X Graphic to be as it is. This should tell us the new brain images from which the averaged atlas was generated, the warping performed etc.
2. Find the process that led to Atlas X Graphic, excluding everything prior to the averaging of images with softmean.
3. Find the Stage 3, 4 and 5 details of the process that led to Atlas X Graphic.

1. Find all output averaged images of softmean (average) procedures, where the warped images taken as input were align_warped using a twelfth order nonlinear 1365 parameter model, i.e. "where softmean was preceded in the workflow, directly or indirectly, by an align_warp procedure with argument -m 12."
2. A user has run the workflow twice, in the second instance replacing each procedures (convert) in the final stage with two procedures: pgmtoppm, then pnmtojpeg. Find the differences between the two workflow runs.
3. A user has annotated some anatomy images with a key-value pair center=UChicago. Find the outputs of align_warp where the inputs are annotated with center=UChicago.

• 2006-September: Face-to-face meeting at which results are discussed
• 2006-October-15: Comparisons performed, minutes of discussion, proposed next steps uploaded

Aims

• The capabilities of each system in answering provenance-related queries
• What each system considers to be within scope of the topic of provenance (regardless of whether the system can yet achieve all problems in that scope)

• Representations of provenance for the example workflow
• Representations of the result of the core (and other) queries

Optionally, the participants may like to contribute the following.

• Additional queries (beyond the core queries) that illustrate the scope of their system

• Extensions to the example workflow to best illustrate the unique aspects of their system
• Any categorisation of queries that the project considers to have practical value

Example Workflow

The stages of the workflow are as follows.

1. For each new brain image, align_warp compares the reference image to determine how the new image should be warped, i.e. the position and shape of the image adjusted, to match the reference brain. The output of each procedure in the stage is a _warp parameter set_ defining the spatially transformation to be performed (Warp Params 1 to 4).

Core Provenance Queries

1. Find the process that led to Atlas X Graphic / everything that caused Atlas X Graphic to be as it is. This should tell us the new brain images from which the averaged atlas was generated, the warping performed etc.
2. Find the process that led to Atlas X Graphic, excluding everything prior to the averaging of images with softmean.
3. Find the Stage 3, 4 and 5 details of the process that led to Atlas X Graphic.

1. Find all output averaged images of softmean (average) procedures, where the warped images taken as input were align_warped using a twelfth order nonlinear 1365 parameter model, i.e. "where softmean was preceded in the workflow, directly or indirectly, by an align_warp procedure with argument -m 12."
2. A user has run the workflow twice, in the second instance replacing each procedures (convert) in the final stage with two procedures: pgmtoppm, then pnmtojpeg. Find the differences between the two workflow runs.
3. A user has annotated some anatomy images with a key-value pair center=UChicago. Find the outputs of align_warp where the inputs are annotated with center=UChicago.

Sample Worflow Implementations

• Shell script workflow

Timetable

-- SimonMiles - 6 June 2006

Provenance Challenge

Our focus in this challenge is on provenance and not on running the experiment. Hence, to facilitate take-up, while based on a real experiment, the procedures can be implemented as "dummies", i.e. we provide the input, output and intermediate data and participants can use fake procedures that take the right input and produce the right output. Alternatively, if participants wish to enact the real workflow, this is also possible. In addition to this, we define a set of core queries that all partipicipants should show how they address, so we can compare systems.

• Representations of provenance for the example workflow
• Representations of the result of the core (and other) queries
• Contributions to a matrix of queries vs systems, indicating for each that: (1) the query can be answered by the system, (2) the system cannot answer the query now but considers it relevant, (3) the query is not relevant to the project.

The inputs to a workflow are a set of new brain images (Anatomy Image 1 to 4) and a single reference brain image (Reference Image). All input images are 3D scans of a brain of varying resolutions, so that different features are evident. For each image, there is the actual image and the metadata information for that image (Anatomy Header 1 to 4). The image data was published with article Accession Number: 2-2004-1168X Frontal-Hippocampal Double Dissociation Between Normal Aging and Alzheimer's Disease by Head, D, Synder, AZ, Girton, LE, Morris, JC, Buckner, RL in the fMRI Data Center.

Provenance Challenge

The inputs to a workflow are a set of new brain images (Anatomy Image 1 to 4) and a single reference brain image (Reference Image). All input images are 3D scans of a brain of varying resolutions, so that different features are evident. For each image, there is the actual image and the metadata information for that image (Anatomy Header 1 to 4). The image data was published with article Accession Number: 2-2004-1168X Frontal-Hippocampal Double Dissociation Between Normal Aging and Alzheimer's Disease by Head, D, Synder, AZ, Girton, LE, Morris, JC, Buckner, RL in the fMRI Data Center.

The stages of the workflow are as follows.

1. For each warp parameter set, the actual transformation of the image is done by reslice, which creates a new version of the original new brain image with the configuration defined in the warp parameter set. The output is a resliced image.
2. All the resliced images are averaged into one single image using softmean.
3. For each dimension (x, y and z), the averaged image is sliced to give a 2D atlas along a plane in that dimension, taken through the centre of the 3D image. The output is an atlas data set, using slicer.

1. Find the process that led to Atlas X Graphic / everything that caused Atlas X Graphic to be as it is. This should tell us the new brain images from which the averaged atlas was generated, the warping performed etc.
2. Find the process that led to Atlas X Graphic, excluding everything prior to the averaging of images with softmean.
3. Find the Stage 3, 4 and 5 details of the process that led to Atlas X Graphic.

Participant Instructions

• 2006-September: Face-to-face meeting at which results are discussed
• 2006-October-15: Comparisons performed, minutes of discussion, proposed next steps uploaded

Provenance Challenge

It is comprised of procedures, shown as orange ovals, and data items flowing between them, shown as rectangles. It can be seen as five stages, where each stage is depicted as a horizontal row of the same procedure in the figure. The procedures employ the AIR (automated image registration) suite to create an averaged brain from a collection of high resolution anatomical data. In addition to the data items shown in the figure, there are other inputs to procedures (constant string options), defined below.

The stages of the workflow are as follows.

1. For each new brain image, align_warp compares the reference image to determine how the new image should be warped, i.e. the position and shape of the image adjusted, to match the reference brain. The output of each procedure in the stage is a [[http://bishopw.loni.ucla.edu/AIR5/file_types.html#warp][_warp parameter set_] defining the spatially transformation to be performed (Warp Params 1 to 4).

Provenance Challenge

Our focus in this challenge is on provenance and not on running the experiment. Hence, to facilitate take-up, while based on a real experiment, the procedures can be implemented as "dummies", i.e. we provide the input, output and intermediate data and participants can use fake procedures that take the right input and produce the right output. Alternatively, if participants wish to enact the real workflow, this is also possible. In addition to this, we define a set of core queries that all partipicipants should show how they address, so we can compare systems.

• Representations of provenance for the example workflow
• Representations of the result of the core (and other) queries
• Contributions to a matrix of queries vs systems, indicating for each that: (1) the query can be answered by the system, (2) the system cannot answer the query now but considers it relevant, (3) the query is not relevant to the project.

Core Provenance Queries

1. Find all the graphical atlas sets that have metadata annotation studyModality with values speech, visual or audio, and return all the annotation tags of this set of data
2. Find all the metadata tags study on actual results of softmean that were linearly aligned with model= affine, and whose outputs have annotation state = IL

Sample Worflow Implementations

As it may be useful to some, we provide sample implementations of the workflow here. This should not preclude the use of any other technology. The implementations assume that the executables referenced above are all installed.

Timetable

• • Fix provenance challenge proposal TWiki page, with aims, objectives, workflow, sample queries, timetable
  • Upload input, output, intermediate data used in workflow
  • Send provenance challenge call to other possible participants, including:

• 2006-September: Face-to-face meeting at which results are discussed
• 2006-October-15: Comparisons performed, minutes of discussion, proposed next steps uploaded

Provenance Challenge

It is comprised of procedures, shown as orange ovals, and data items flowing between them, shown as rectangles. It can be seen as five stages, where each stage is depicted as a horizontal row of the same procedure in the figure. The procedures employ the AIR (automated image registration) suite to create an averaged brain from a collection of high resolution anatomical data. In addition to the data items shown in the figure, there are other inputs to procedures (constant string options), defined below.

1. For each new brain image, align_warp compares the reference image to determine how the new image should be warped, i.e. the position and shape of the image adjusted, to match the reference brain. The output of each procedure in the stage is a [[http://bishopw.loni.ucla.edu/AIR5/file_types.html#warp][_warp parameter set_] defining the spatially transformation to be performed (Warp Params 1 to 4).
2. For each warp parameter set, the actual transformation of the image is done by reslice, which creates a new version of the original new brain image with the configuration defined in the warp parameter set. The output is a resliced image.
3. All the resliced images are averaged into one single image using softmean.

Core Provenance Queries

1. Find all the graphical atlas sets that have metadata annotation studyModality with values speech, visual or audio, and return all the annotation tags of this set of data
2. Find all the metadata tags study on actual results of softmean that were linearly aligned with model= affine, and whose outputs have annotation state = IL

Timetable

• 2006-May-30: First full draft of challenge, asking potential participants to help finalise

• 2006-September: Face-to-face meeting at which results are discussed
• 2006-October-15: Comparisons performed, minutes of discussion, proposed next steps uploaded

Provenance Challenge

Aims

The provenance challenge aims to establish an understanding of the capabilities of available provenance-related systems and, in particular, the following details.

• The capabilities of each system in answering provenance-related queries
• What each system considers to be within scope of the topic of provenance (regardless of whether the system can yet achieve all problems in that scope)

Each project participating in the challenge will have its own page on this TWiki where they can inform the rest of their efforts in meeting the challenge. During the provenance challenge, we expect the participants to upload the following to their page, to then allow comparison.

• Contributions to a matrix of queries vs systems, indicating for each that: (1) the query can be answered by the system, (2) the system cannot answer the query now but considers it relevant, (3) the query is not relevant to the project.

Optionally, the participants may like to contribute the following.

• Additional queries (beyond the core queries) that illustrate the scope of their system

• Any categorisation of queries that the project considers to have practical value

It is comprised of procedures, shown as orange ovals, and data items flowing between them, shown as rectangles. It can be seen as five stages, where each stage is depicted as a horizontal row of the same procedure in the figure. The procedures employ the AIR (automated image registration) suite to create an averaged brain from a collection of high resolution anatomical data. In addition to the data items shown in the figure, there are other inputs to procedures (constant string options), defined below.

1. For each dimension (x, y and z), the averaged image is sliced to give a 2D atlas along a plane in that dimension, taken through the centre of the 3D image. The output is an atlas data set, using slicer.
2. For each atlas data set, it is converted into a graphical atlas image using (the ImageMagick utility) convert.

The full steps, procedures data and parameters are enumerated in the table below. The procedure names are linked to the manual pages for those utilities, and the input and output names to the actual data exchanged between procedures.

Core Provenance Queries

1. Find the process that led to Atlas X Graphic / everything that caused Atlas X Graphic to be as it is. This should tell us the new brain images from which the averaged atlas was generated, the warping performed etc.
2. Find the process that led to Atlas X Graphic, excluding everything prior to the averaging of images with softmean.

1. Find all calls to procedure alignwarp, and their runtimes, with argument model=rigid that ran in less than 30 minutes on non-ia64 processors. The argument is given as a string option to the alignwarp procedure.
2. Find the average runtime of all alignwarp calls with argument model=rigid that ran in less than 30 minutes.
3. Find all procedure calls within runs of the workflow named ‘/prod/2005/0305/prep’, a version of the brain atlas workflow given above, whose inputs were linearly aligned with model=affine

• 2006-September: Face-to-face meeting at which results are discussed
• 2006-October-15: Comparisons performed, minutes of discussion, proposed next steps uploaded

Provenance Challenge

1. For each warp parameter set, the actual transformation of the image is done by reslice, which creates a new version of the original new brain image with the configuration defined in the warp parameter set. The output is a warped image.
2. All the warped images are averaged into one single image using softmean.
3. For each dimension (x, y and z), the averaged image is sliced to give a 2D atlas along a plane in that dimension, taken through the centre of the 3D image. The output is an atlas data set, using slicer.

The full steps, procedures data and parameters are enumerated in the table below. The procedure names are linked to the manual pages for those utilities, and the input and output names to the actual data exchanged between procedures.

Core Provenance Queries