Challenge #5 - 6 PI Leads: James Bassingthwaighte (jbb2@u.washington.edu), Rajanikanth Vadigepalli (Rajanikanth.Vadigepalli@jefferson.edu), Jonathan Lederer (jlederer@umaryland.edu), and Eric Sobie (eric.sobie@mssm.edu)
IMAG POCs: Michele Grimm (mgrimm@nsf.gov), Christina Payne (cpayne@nsf.gov), Jerry Myers (jerry.g.myers@nasa.gov), Kenneth Wilkins (kenneth.wilkins@nih.gov), Haluk Resat (haluk.resat@nih.gov)
9:30-9:45am: Update on the Accomplishments for Challenges #5 and #6 thusfar
Challenge 5 Presentation:
Challenge 5 additional slides:
Challenge 6 Presentation:
9:45-10:00am: Discussion of the gaps to be addressed in the future.
Questions to pose to the MSM Consortium related to these challenges:
- (Challenge 5 leads) Do you see the need for a single point resource, potentially setup and maintained by NCBI/NLM, to share and download models (like the Gene Expression Omnibus resource is for the gene expression data https://www.ncbi.nlm.nih.gov/geo/)?
- (Challenge 6 leads) Are there other success stories that weren’t highlighted in pre-meeting discussions?
- (Challenge 6 leads) How do experiments and model parameterization need to be documented for protocols to become standardized more readily?
- (Challenge 6 leads) What additional challenges are faced besides those already mentioned?
- From Jacob Barhak: Jim's definitions of repeatability, replicability, and reproducibility are very good. However, in many cases during development we need even stricter definitions where a model needs debugging and repeatability has to be exact , not statistical. How would you define such debug level reproducibility?
- Fom Gary An: What would the panel consider to be "similar" results for Q2 (Replicated) and Q3 (Reproduced)?
- (Bill Lytton) How can we define appropriate levels of Verification and Validation, Replicability and Reproducibility outside of the domain of purpose-built individual models. For example, 1. the modeling platform itself is an instantiation of the modeling domain and has issues of Verification and Validation; 2. model modules can be shared even across organ systems -- for example in brain modeling we have sometimes reused voltage-sensitive ion channels from cardiac.
- (PERSON) QUESTION
- (PERSON) QUESTION
- (PERSON) QUESTION
Comments
I was surprised that the panel didn't mention the COMBINE organization which is heavily involved in the standardization, repositories and modeling in a variety of biological domains.
Herbert Sauro
The modeling standards for genome-scale metabolic models are a nice example of standards facillitating model reuse.
Mike Henson
I like to second James Glazier's comment on separating the execution from the underlying biological description.
Herbert Sauro
Crucial issue for reproducibilility is that there needs to be a formal structured "biological" or "abstract" or "conceptual" model developed before the computation starts and presented in a reaable and unambiguous way. Such a model is a method independent description of the structure and concepts of the biology being modeled. It isn't, in general, possible to back this information out of a specific numerical implementation. Still don't have standards for such descriptions. Perhaps as a result, they aren't required or included in most modeling papers or projects.... Without these higher-level abstract models as underpinnings for numerical methods, we can't ever hope to do model reproduction or reuse. By the way, this idea isn't new, it goes all the way back to Diderot in the 18th Century! It's interesting that it is so hard to achieve and that the community is still so resistant to doing it.