Skip to end of metadata
Go to start of metadata
You are viewing an old version of this page. View the current version. Compare with Current ·  View Page History

This page serves as a wishlist for FireCloud features needed by the Broad GDAC to match the productivity of FireHose.

Requested Features

WDL 

  • "Map" data type

    A common use case in FireHose is to select input files from samples in a sample set, and pass these files to an analysis via a two-column tsv file that maps sample ids to the data files. An analagous method exists in FireCloud, allowing you to accept as input an array of sample ids and an array of the data files. The problem arises when the data is sparse – the two arrays are no longer parallel, and the mapping is broken. From a task-authors perspective, this could be solved if there was a Map data type in WDL. In Firecloud, you could pass the input as "this.samples.name->this.samples.data", and not require any sort of Null sentinel value in the bucket. 


  • Compostability/ Imports

    Different workflows will often share common tasks, such as a preprocessor or a report generator. Since WDLs are not composable, each workflow must independently maintain a copy of the task. A temporary solution used by gdac-firecloud is to use a script to sync task definitions within the repository, but this has limitations. First, it only works with tasks defined in the repository. It also requires manual intervention by the workflow developer, via 'make sync'. Import statements are currently part of the WDL spec, but are not implemented in FireCloud or any of the development tools (i.e. Cromwell, wdltool, etc.). 

     

Developer Tools

  • Debugging
    • Running an on-prem instance
    • SSH access to observe running jobs, look at files in a workspace
    • Fail-Fast --> Don't let me submit a workflow that has invalid inputs

UI

 

API

  • Accurate Documentation
    The current api page is intended to list all available orchestration (read user-available) api calls. But this page is either incomplete, or does not accurately represent the available API calls. On a couple occasions, Alex has directed me to an API endpoint that is not listed on this page, but is listed on the page for the service (Rawls, Agora, etc.) at https://swagger.dsde-dev.broadinstitute.org/ . Some of these APIs are publicly accessible as pass-throughs, but the distinction has not been made clear.
Labels: