Working Group Leads

Jerry G. Myers (Jerry.G.Myers@nasa.gov)

Andrew Drach (andrew.drach@utexas.edu)

The Committee on Credible Practice of Modeling & Simulation in Healthcare  was established in order to provide a collaborative platform to outline good practice of simulation-based medicine. The Committee was an initiative started under the IMAG and MSM stemming from discussions at previous years’ MSM Consortium meetings, and monthly IMAG/MSM meetings. Throughout these discussions it became clear that research activities in computational medicine are growing at a significant rate and notable discoveries are being made. However, the mechanisms or processes necessary to appropriately translate these research activities and discoveries in computational methods to clinical practice are lacking. Moreover, there is substantial research diversity in the field such that subject matter experts within and across mathematical and biological disciplines tend to have their own interpretation of credible practice in M&S. Additionally, tools and good practice guidelines established by individual disciplines or research areas do not readily transfer across other disciplines or are not adopted by different fields. To help fill these critical gaps, the "Committee on Credible Practice of Modeling & Simulation in Healthcare" was established.

This page provides a summary of the Committee for IMAG/MSM audience. For more details of the Committee and its activities, please refer to Committee wiki site at http://wiki.simtk.org/cpms.

Goals and Objectives:

The primary focus of the Committee is to enhance the credibility of computational modeling & simulation for clinical research and decision making. The specific charges are:

• to adopt a consistent modeling & simulation terminology
• to develop guidelines & procedures for credible practice
• to demonstrate workflows for credibility assessment
• to promote credible practice of modeling and simulation

Ten Simple Rules for Credible Models - see Announcement

Please refer to the following publication for an in-depth review of the Ten (not so) Simple Rules for Credible Practice

*https://www.imagwiki.nibib.nih.gov/resources/publications/credible-prac…*

For a summary of the ten simple rules and proposed scoring rubric please refer to the

https://www.imagwiki.nibib.nih.gov/content/10-simple-rules-conformance-rubric

****NOTE****  The IMAG wiki now has a Resource Credibility Assessment Form to post and continuously update your Model Credibility Assessment activities. (February 2020) -- see Instructions below

Instructions with Conformance Rubric

2020 Example of Completed TSR Form - https://www.imagwiki.nibib.nih.gov/resource_credibility_assessment/cpms…

To upload your Model Credibility Assessment:

1. Make sure you are logged into the IMAG wiki
2. Add Content:  Choose Resource Credibility Assessment form - complete form (wiki Support links at the bottom navigation bar)
3. Update your Models, Tools & Databases entry with your Resource Credibility Assessment (IF YOU DON'T HAVE AN ENTRY - follow instructions below.)
4. The title you entered for your Credibility Assessment will auto populate in this box:  Link to current Resource Credibility Assessment, Choose and Save Models, Tools & Database Form
5. Update your Credibility Assessment regularly (use Edit tab).

To upload/update your Model in the IMAG wiki:

• Make sure you are logged into the IMAG wiki
• Add Content:  Choose Models, Tools & Databases Form - complete form (wiki Support links at the bottom navigation bar)
• Update your Model regularly (use Edit tab) - with publications, simulations, images.

 

---

2018 Mid-Term Review - October 2018

 

MSM 2018 Model Credibility Session March 22, 2018

---

MSM 2017 (10^th Anniversary) Breakout Session Summary

March 22, 2017 - 100 PM - 3:00 PM

Attendees

42 showed interest during registration, 13 attended.

• Brian Colder
• Jerry Day
• Ahmet Erdemir (Co-Chair)
• Jason Haugh (MSM Steering Committee)
• Marc Horner (Vice Co-Chair)
• Joy Ku (Executive Committee Member)
• Jason Halloran
• Donna Lochner (Advisory Council Member)
• Tina Morrison (Executive Committee Member)
• Lealem Mulugeta (Co-Chair)
• Pras Pathmanathan (Advisory Council Member)
• John Rice
• Herbert Sauro

Highlights

Goals

1. Developing and adapting guidelines and procedures for credible practice of M&S in healthcare,
2. cultivating consistent terminology,
3. demonstrating workflows for credible practice, and
4. promoting credible practice

Structure

Executive Committee (10 members) - hands on activities
Advisory Council (12 members) - for feedback

Breakout feedback

• Progress:
  • 2 webinars
  • 6 conference posters/presentations
  • 2 manuscripts in progress
  • Ten "Not So" Simple Rules Poster at #1
  • Theme 6: Session on Model Credibility Plan(s)
• Leadership transitioning mechanism is in place (Vice Co-Chair positions); volunteers are always welcome.
• Members are sought after. DISCLAIMER: Upto 8 hours of effort level per month is expected.
• The need for establishing clarity in terminology is pressing.
  • Glossary needs to be revived (see the Committee wiki).
  • High priority terms will be defined in manuscripts
• Interest in documenting M&S workflows - need contributions from other working groups

Notes

1. Introductions. The session started with the attendees introducing themselves.
2. MSM Steering Committee perspective. Jason Haugh provided the role of the MSM Steering Committee to establish accountability with the Working Groups and provided his view of Committee activities. Jason noted that the Steering Committee would like to assess Working Group operations, if there is a need to change the Working Group or consolidate it with other Working Groups, how the leadership and membership works. A response to a survey in this regard was sent to him by the Co-Chairs before the meeting. Ahmet also provided the survey response as a report in the IMAG wiki. Jason noted that the MSM Steering Committee will meet and provide their opinions based on this information and Jason's observations in the breakout session.  His personal impression is that the Committee has well-defined goals, strong leadership, and a lot of activity.
3. Summary of the Committee. Lealem provided a description of the Committee, its history, and its motivations. The Committee started as an IMAG initiative and initially was not designed as a Working Group. The leadership is not necessarily U01 awardees. The membership is by invitation or nomination, specifically focusing on the motivation to do active work for the Committee. The Committee is formed by an Executive Committee and an Advisory Council. The Executive Committee is more hands-on, commonly requiring up to 8 hours of effort level per month, Advisory Council is more designed to acquire intellectual feedback on Committee operations and requires less effort level.
4. Overall progress on Committee activities. Ahmet went over the progress report, which started discussion on anticipated Committee operations and activities (notes below).
5. Progress on manuscripts. Pras asked about the progress on the Committee papers. Ahmet noted that two manuscripts are under development, albeit the progress had been slow. The first one is Ten Simple Rules for Credible Practice - Committee Perspective. The drafts of Introduction and Discussion needs to be written by Lealem and Ahmet respectively. Other relevant sections were completed and the draft is available in the source code repositort. Tina volunteered to work on the manuscript throughout MSM meeting. The group also agreed on incorporating the example Lealem used for the MSM 2017 poster to demonstrate strategies to respond to Ten Simple Rules. The group also agreed to provide definitions of high-priority terms (credible, model) as part of that manuscript as a segway to glossary activity (also see notes below). The second manuscript will be based on survey data to establish Ten Simple Rules Community Perspective. The data has been analyzed. The analysis needs to be wrapped up to start drafting the manuscript. Jason also mentioned that the MSM Steering Committee is interested in two articles =by contributions from all working groups - one for modelers, the other for non-modelers. These articles may be a good venue to cite the manuscripts on Ten Simple Rules.
6. Leadership. Ahmet and Lealem are interested in transitioning the leadership. When the Committee was founded, term limits on co-chairs were placed. The reason for this was twofold; to bring new energy to the Committee and to prevent Co-Chairs bias and comfort influence the Committee's direction. The Committee has established a Vice Co-Chairs position to assist with the transitioning process, i.e. to familiarize with the mechanics of the Committee and its overall goals. Currently, Marc Horner is a Vice Co-Chair and it is likely that one of the new members will be asked to fill in the other position. There is no set timeline; the leadership transition will depend on who steps up and when. Lealem proposed to look into it in June, 2017.
7. Membership. There was inquiry made on the requirements and expecations of the Committee members or those who want to become members. It was explained that the membership is comprised of individuals both from the MSM community and those who are not in that community. Yet, they all are expected to have in modeling and simulation and its credibility. The membership is also quite multi-faceted with respect of dicipline and industry/academia (also see notes on Summary of the Committee).
8. MSM 2017 Session on Credibility Plan(s). The Committee is moderating Theme 6 session on Model Credibility Plan(s). It is hoped that the awardees will have a chance to present their approach to credibility of their processes and get feedback from the Committee and session attendees. There are 11 awardees and all are asked to provide their Credibility Plan and presentation on the wiki. The Committee is planning to document the discussions and draft a summary of the experience.
9. M&S terminology. The participants had long discussion on modeling and simulation terminology in regard to achieving clarity in the broad community. This is a charge of the Committee and the Committee has started a glossary activity. Unfortunately, the activity did not have much traction possibly due to enormity of the problem and the required workload. Tina suggested to connect the Committee with the FDA's modeling and simulation workgroup to assist with this effort. The group also agreed to approach this problem by first defining key terms in upcoming manuscripts (see discussion above) as a starting point. Lealem also noted the possibility to introduce this glossary to the medical dictionary of national library. Ahmet also noted the manuscript effort led by Tony Hunt to understand differences between mechanistic models,. models of mechanisms, and model mechanisms.
10. Documenting Workflows. A while ago, as part of the Committee activities, Tony Hunt has started an initiative to document modeling and simulation workflows in different disciplines to identifiy overlaps and differences. At that time, Tony, Ahmet and Lealem provided documentation on the modeling and simulation workflow of their respective disciplines. Tony also reached out to other Working Group leads to request workflow documentation for specific disciplines of the Working Groups. He has not received any feedback. The Committee agreed to ask the MSM Steering Committee to follow up with Working Groups to revive this activity.

---

MSM 2017 (10^th Anniversary) Report

Recap of Goals

The goals of the Committee on Credible Practice of Modeling and Simulation in Healthcare remain the same:

(1) Developing and adapting guidelines and procedures for credible practice of M&S in healthcare,
(2) cultivating consistent terminology,
(3) demonstrating workflows for credible practice, and
(4) promoting credible practice.

Obstacles

The Committee is an organic effort relying on contributions from diverse members from the modeling and simulation community. Fundamental obstacles are related to (1) balancing members' sustained commitment to the Committee with their primary scientific, engineering, and clinical responsibilities, and (2) recruiting new members to infuse energy to Committee activities and renew the vision of the Committee. As co-chairs, we had difficulty to finalize the reporting of guidelines in a publication form due to competing commitments.

Accomplishments

The Committee drafted the Ten Simple Rules - Committee Perspective for credibility and also completed the public survey to compile Ten Simple Rules - Community Perspective. See https://simtk.org/plugins/moinmoin/cpms/Ten%20Simple%20Rules%20of%20Credible%20Practice  for more details.

The Committee also provided the following webinars, see https://simtk.org/plugins/moinmoin/cpms/Events  for more details.

(1) BioGears Framework for Multiscale Physiology: Lessons Learned
Wednesday, September 7, 2016 at 9:00 am PDT
Jeff Webb, Applied Research Associates, Inc.

(2) In silico Design of Novel Surgical Methods for Children with Single Ventricle Hearts: From Computation to Clinic
Thursday, January 22, 2015 at 10:00 am PST
Dr. Alison Marsden, U.C. San Diego

The following is a list of conference abstracts by the Committee (past year), see https://simtk.org/plugins/publications/index.php/?group_id=848  for more details.

(1) J.E. Bischoff, L. Mulugeta, A. Erdemir and A. Hunt, “Towards the Establishment of Guidelines for the Credible Practice of Modeling and Simulation in Healthcare”, Latsis Symposium ETH Zurich on Personalized Medicine 2016, 27-29 June 2016, Zurich, Switzerland. (2016)

(2) M. Horner, L. Mulugeta, A. Erdemir, G. An, D.M. Eckmann, J.E. Bischoff, C.A. Hunt, J. Ku, D. Lochner, W.W. Lytton, V. Marmarelis, J.G. Myers, G. Peng, M. Steele and M. Walton, “A Comparison of Community-Based Guidelines and Standards for the Credible Use of Computational Methods in Healthcare”, 2016 BMES/FDA Frontiers in Medical Devices, Washington, DC. (2016)

(3) M. Horner, L. Mulugeta, A. Erdemir, G. An, D.M. Eckmann, J.E. Bischoff, C.A. Hunt, J. Ku, D. Lochner, W.W. Lytton, V. Marmarelis, J.G. Myers, G. Peng, M. Steele and M. Walton, “Credibility of Computational Methods in Healthcare: A Comparison of Community-based Standards and Guidelines”, 2016 ASME V&V Symposium, Las Vegas Nevada. (2016)

(4) L. Mulugeta, L. Tian, M.J. Steele, M. Horner, A. Erdemir, J.E. Bischoff, D.M. Eckmann, C.A. Hunt, J.P. Ku, D.R. Lochner, W.W. Lytton, V. Marmarelis, T.M. Morrison, J.G. Myers, G.C.Y. Peng, and M. Walton, “Developing Credible Practice Guidelines for Modeling and Simulation in Healthcare: A Multifaceted Approach”, 38th Annual International Conference of the IEEE Engineering in Medicine and Biology Society, 16-20 August 2016, Orlando, FL. (2016)

(5) Morrison TM, Horner M, Mulugeta L, Tian L, Steele MJ, Hunt CA, Ku JP, Pathmanathan P, Lochner DR, Lytton WW, Myers JG, Peng GCY, Erdemir A. Credible Practice of Modeling and Simulation in Healthcare.  Virtual Physiological Human Conference. Amsterdam, The Netherlands. (2016)

(6) Ahmet Erdemir, Gary An, Jeff E. Bischoff, David M. Eckmann, Marc Horner, C. Anthony Hunt, Joy P. Ku, Donna, R. Lochner, William W. Lytton, Vasilis Z. Marmarelis, Alison L. Marsden, Tina M. Morrison, Lealem Mulugeta, Jerry G. Myers, Grace C.Y. Peng, Gaurav N. Pradhan, Martin J. Steele, Rajanikanth Vadigepalli, Marlei E. Walton, Ten “Not So” Simple Rules for Credible Practice of Modeling & Simulation vs Considerations of Medical Device Development Tools Program, Accepted for 2016 BMES/FDA Frontiers in Medical Devices, Washington, DC. (2017)

Work in progress to present Committee's activities in the upcoming Computational Neuroscience Meeting.

In MSM 2017, the Committee supports:

(1) A poster titled "Guidelines for Credible Practice of Modeling and Simulation in Healthcare"
(2) THEME 6:  Model Credibility Plans - Consortium Review

Activity Level

As mentioned above, the Committee continues to be active in promotion of its work in conferences and through webinars.

The Committee members meet regularly through conference calls, at least once a month (although targeted for every two weeks). Meeting minutes are available at https://simtk.org/docman/?group_id=848

All Committee activities can be accessed through the Committee wiki at
https://simtk.org/plugins/moinmoin/cpms/FrontPage

New members joined:
(1) Andrew Drach, University of Texas at Austin (Executive Committee)
(2) Rajanikanth Vadigepalli, Thomas Jefferson University (Executive Committee)
(3) Alison Marsden, Stanford University (Advisory Council)
(4) Gaurav Pradhan, Mayo Clinic (Advisory Council)

Potential Improvements

The Committee will improve by refreshing membership portfolio and through handover of leadership to infuse new ideas and energy to the Committee. In regard to former, the Committee is actively recruiting new members from the IMAG/MSM community and elsewhere. The MSM 2017 Consortium meeting and particularly, the session on Model Credibility Plans, will be a great opportunity to bring in new investigators to the Committee. In regard to latter, the Committee devised a Vice Chairs structure to pass on leadership.

The Committee is not planning to merge with other WGs nor separate as additional WGs. The Committee will continue to synergistically work with other WGs as credibility is at the foundation of activities across IMAG/MSM.

Potential Publications

The Committee is still interested in publication of Ten Simple Rules, Committee and Community Perspectives. Target journal is PLOS Computational Biology.

Some Committee members has started an article on semantics of model types, which has been revised for resubmission to PLOS Computational Biology - Mechanistic Models, Model Mechanisms, and Computational Models of Explanation for Biological Phenomena by Hunt et al.

Ahmet Erdemir, Committee Co-Chair, along with Herbert Sauro, published a special section  IEEE Transactions on Biomedical Engineering on model sharing and reproducibility, which is synergistic with Committee activities - http://ieeexplore.ieee.org/xpl/tocresult.jsp?isnumber=7573011 .

As noted above, the Committee is interested in publication of Ten Simple Rules, Committee and Community Perspectives. We welcome other WGs to contribute these initiatives. Both these manuscripts will provide guidance both for modelers and non-modelers. The Committee is also interested in providing credibility perspective in any position papers that may be developed through a collaboration between WGs and the steering committee of the meeting.

Leadership Changes

The Committee Co-Chairs are interested in pursuing leadership exchange, particularly through the Vice Chairs mechanism instituted in the Executive Committee structure.

---

MSM 2017 (10^th Anniversary) Discussion

MSM 2017 participants, please provide your comments, questions here.

 

---

MSM 2015 Breakout Summary: https://simtk.org/websvn/wsvn/cpms/doc/presentations/2015_IMAG_MSM_Briefing.pdf

---

ROSA Worldwide Webinar Series: Impact of Modeling & Simulation in Drug Development

The Committee on Credible Practice of Modeling & Simulation in Healthcare (CPMS) was honored to give a webinar in May at the ROSA Worldwide Webinar Series: Impact of Modeling & Simulation in Drug Development  on May 17, 2018. THe topic of the webinar discussed the application of our ten simple rules for promoting credibility of modeling and simulation in healthcare.

 

Abstract: Computational modeling and simulation (M&S) has the potential to play a critical role in precision medicine and personalized healthcare. However, there are few formalized processes and procedures to support the credible use of M&S in healthcare and biomedical research, limiting their adoption. To address this gap, the Committee on Credible Practice of Modeling & Simulation in Healthcare (CPMS) was established under the Interagency Modeling and Analysis Group (IMAG) and the Multiscale Modeling (MSM) Consortium, which includes representatives from government agencies such as the National Institutes of Health, NASA, and the U.S. Food and Drug Administration. One of CPMS’ primary goals is to develop “The Ten Simple Rules of Credible Practice of M&S in Healthcare.” These rules were derived via synthesis of the Committee’s expertise in this domain and by surveying the global stakeholder community to ensure a balanced representation of interests and perspectives concerning credible practice of M&S in healthcare. This presentation will introduce these rules, present two case studies (modeling heart valves and bone remodeling) to demonstrate their value, and discuss strategies for incorporating these rules into workflows.

 

Speakers: 

Lealem Mulugeta is currently Chief Scientist and Executive Director of InSilico Labs, LLC and Director of Medalist Fitness, LLC and Co-founder of CPMS. His current business ventures are focused on translation of biomedical, computational and engineering research to enable individualized healthcare and peak performance. Prior to his venture with InSilico Labs and Medalist Fitness, Lealem worked at NASA as the Project/Lead Scientist of NASA's Digital Astronaut Project (DAP). The DAP was dedicated to implementing well-validated computational models to help predict and assess spaceflight health and performance risks, and to enhance the development of health risk countermeasure. During his tenure with the DAP, Lealem played a strong role in the development and implementation of standardized methods for verification, validation and credibility assessment of NASA’s biomedical computational models.

Andrew Drach is an Executive Committee Member of CPMS. He is currently a Senior Research Fellow in the Institute for Computational Engineering and Sciences at the University of Texas at Austin and was previously the Assistant Director of the Center for Cardiovascular Simulation at the Institute. Andrew’s research focuses on computational materials science and biomedical engineering, examining the biomechanics of growth and remodeling of biological tissues, multi-scale modeling of materials, and chemo-mechanical behavior materials. He is currently a co-chair of the BMES Special Interest Group on Medical Devices.

Webinar Recording:  https://youtu.be/U4G4dYtJ-SI

Presentations: 

Lealem Mulugeta -

 

Andrew Drach -