July 2015, VOLUME136 /ISSUE 1

Comparative Effectiveness Research Through a Collaborative Electronic Reporting Consortium

  1. Alexander G. Fiks, MD, MSCEa,b,c,d,e,
  2. Robert W. Grundmeier, MDb,
  3. Jennifer Steffes, MSWe,
  4. William G. Adamsf,
  5. David C. Kaelber, MD, PhD, MPHg,
  6. Wilson D. Pace, MDh, and
  7. Richard C. Wasserman, MD, MPHe,i
  8. for the Comparative Effectiveness Research Through Collaborative Electronic Reporting (CER2) Consortium
  1. aThe Pediatric Research Consortium,
  2. bDepartment of Biomedical and Health Informatics,
  3. cCenter for Pediatric Clinical Effectiveness, and
  4. dPolicyLab at The Children’s Hospital of Philadelphia, Philadelphia, Pennsylvania;
  5. ePediatric Research in Office Settings, American Academy of Pediatrics, Elk Grove Village, Illinois;
  6. fBoston University School of Medicine, Boston, Massachusetts;
  7. gDepartments of Internal Medicine, Pediatrics, and Epidemiology and Biostatistics and the Center for Clinical Informatics Research and Education, The MetroHealth System, Case Western Reserve University, Cleveland, Ohio;
  8. hAmerican Academy of Family Physicians National Research Network, Leawood, Kansas; and
  9. iDepartment of Pediatrics, University of Vermont College of Medicine, Burlington, Vermont


The United States lacks a system to use routinely collected electronic health record (EHR) clinical data to conduct comparative effectiveness research (CER) on pediatric drug therapeutics and other child health topics. This Special Article describes the creation and details of a network of EHR networks devised to use clinical data in EHRs for conducting CER, led by the American Academy of Pediatrics Pediatric Research in Office Settings (PROS). To achieve this goal, PROS has linked data from its own EHR-based “ePROS” network with data from independent practices and health systems across the United States. Beginning with 4 of proof-of-concept retrospective CER studies on psychotropic and asthma medication use and side effects with a planned full-scale prospective CER study on treatment of pediatric hypertension, the Comparative Effectiveness Research Through Collaborative Electronic Reporting (CER2) collaborators are developing a platform to advance the methodology of pediatric pharmacoepidemiology. CER2 will provide a resource for future CER studies in pediatric drug therapeutics and other child health topics. This article outlines the vision for and present composition of this network, governance, and challenges and opportunities for using the network to advance child health and health care. The goal of this network is to engage child health researchers from around the United States in participating in collaborative research using the CER2 database.

  • Abbreviations:
    American Academy of Pediatrics
    clinical decision support
    comparative effectiveness research
    Comparative Effectiveness Research Through Collaborative Electronic Reporting
    The Children’s Hospital of Philadelphia
    electronic health record
    Electronic National Quality Improvement and Research Network
    institutional review board
    Observational Medical Outcomes Partnership
    Pediatric Research Consortium
    Pediatric Research in Office Settings
  • A 2010 Institute of Medicine report notes that clinical data, “because of their potential to enable the development of new knowledge and to guide the development of best practices from the growing sum of individual clinical experiences, . . . represent the resource most central to healthcare progress.”1 The Comparative Effectiveness Research Through Collaborative Electronic Reporting (CER2) Consortium ( was created to develop a pediatric “learning healthcare system”2 infrastructure to harvest routinely collected clinical data from electronic health records (EHRs) and other sources to generate new knowledge and improve care. Such approaches are increasingly valuable as more children receive care using EHRs.3 Although many of the individual sites represented on this consortium’s research team had begun separate and limited efforts in this direction, a system using clinical data from a broad array of practices providing pediatric primary care to conduct comparative effectiveness research (CER), here defined as “the direct comparison of existing health care interventions to determine which work best for which patients and which pose the greatest benefits and harms,”4 had not yet been created.

    Although such a system may benefit pediatric care in many ways, the need for such an infrastructure is especially acute with regard to drug therapeutics in pediatric populations. It has been estimated repeatedly over the past 40 years that 75% to 80% of pharmaceuticals possess insufficient labeling information for dosing, safety, or efficacy in children.510 Even when research is available to guide use, the impact of these medications on children in real-world practice settings is often poorly understood. To address these knowledge gaps, large EHR databases from diverse practice settings are a particularly helpful resource because they can link prescribing data with clinical outcomes, can identify cohorts of children for more detailed study, and can drive decision support to improve care based on medical evidence.11 Because most children are healthy, drug use and adverse events in most settings are uncommon. Therefore, large populations are needed to study rare pediatric conditions, and many questions cannot be readily answered without data pooled across health systems.12 In addition, we focus on data from the primary care medical home, because EHRs from this setting provide a longitudinal record of care throughout childhood, and although only a subset of children receive specialty or hospital care, nearly all receive primary care.

    This article describes the creation of an EHR research network sufficiently large to conduct CER, the CER2 Consortium (Fig 1). We detail the network structure, initial research focus, governance, and future vision. The consortium was created through a partnership between the American Academy of Pediatrics (AAP), Health Resources and Service Administration Maternal and Child Health Bureau, Eunice Kennedy Shriver National Institute of Child Health & Human Development and independent practices and health systems. The goal is to enable widespread collaboration to generate new knowledge to improve child health and health care. The collaboration of the 2 federal agencies that focus on research in child populations is especially significant, because although each has invested resources in developing and maintaining research networks, neither had based such a network on clinical data from EHRs.

    FIGURE 1

    Map of CER2 participating sites. Two hundred and twenty-two practice sites in 27 states representing 1.2 million covered lives.

    A Scholarly Focus on Pharmacoepidemiology Coupled With the Ability to Support a Broad Research Agenda

    Given the evidence that more information is needed about medication safety and effectiveness in children, the short-term focus of the CER2 Consortium is pharmacoepidemiologic research directed at those issues. As reported recently, the Best Pharmaceuticals for Children Act and the Pediatric Research Equity Act have collectively resulted in >500 labeling changes for pediatric medications, providing evidence to support well-informed prescribing.13 However, additional information is needed to guide medication decisions because, as the AAP Committee on Drugs writes, an overwhelming number of drugs still have no information in the labeling for use in pediatrics.13 Even when pediatric labeling does exist, medications are often used for off-label indications and among off-label age groups. Although off-label prescribing does not imply that use is harmful or unwarranted,13 expanding the evidence base is likely to increase the confidence of both clinicians and families that medications are likely to improve health without detrimental side effects.

    With this background, the CER2 Consortium was funded with a focus on developing a national resource to evaluate medication safety and effectiveness in high-priority areas. Specifically, 4 retrospective, observational proof of concept studies are being conducted, and 1 full-scale prospective trial is planned to demonstrate the utility of CER2 as a tool to improve medication safety, effectiveness, and prescribing. These studies will assess the prevalence of use of specific atypical antipsychotics in children and adolescents (ages 3–18 years) and concomitant medications; evaluate the effects of exposure to atypical antipsychotics and metabolic effects of usage of these medications by basic demographic factors including self-reported race and ethnicity, child age, and gender; assess the prevalence of on- and off-label use of frequently prescribed asthma medication in children aged 0 to 5 years through EHR data; and examine rates of psychotropic polypharmacy in children and adolescents (ages 3–18 years) and acquire information on drug safety, efficacy, and side effects. This combination of prospective studies guided by EHR data, combined with the secondary data analyses, will develop methods applicable to future work in diverse areas.

    In addition to these proof of concept studies, the initial grant that funds the CER2 Consortium also supports a larger-scale evaluation of pediatric hypertension. The estimated prevalence of hypertension is increasing, and it now affects 2% to 5% of children,1418 which makes it one of the top 10 chronic diseases in childhood. Pediatric hypertension predisposes children to hypertension in adulthood. In fact, although additional evidence is needed to substantiate the direct link between routine blood pressure measurement and the identification of children at increased risk of adult cardiovascular disease,19 multiple early markers of cardiovascular disease such as left ventricular hypertrophy, arterial intima-media thickness, altered arterial compliance, atherosclerosis, and diastolic dysfunction have been recognized in childhood.2027 Additionally, secondary hypertension is more common in children than adults, so prompt recognition may be particularly important to potentially diagnose and treat underlying diseases.28 Despite clear criteria for the diagnosis and well-defined recommendations for the evaluation and management of hypertensive children (the National Heart, Lung, and Blood Institute recommends screening starting at 3 years of age),29 this condition often remains undiagnosed. Therefore, the opportunity exists to greatly increase the diagnosis and improve the evaluation and management of hypertension in the study’s target population (children and adolescents 3–18 years old). The consortium seeks to better understand the epidemiology and effectiveness of treatment of this condition and, by using advanced clinical decision support (CDS) tools in EHRs, to automatically alert physicians at the point of care to improve recognition, evaluation, and management of pediatric hypertension.

    The goal of CER2 is to engage scholars from around the country in a diverse range of research projects focused on pediatric medication use, safety, and effectiveness, as well as other areas including preventive care, treatment of acute conditions, and chronic disease management. For example, PROS and the Children’s Hospital of Philadelphia (CHOP) Pediatric Research Consortium (PeRC) have used data from CER2 sites to study the impact of patient portals on decision making for pediatric asthma.30 In the future, we intend to engage epidemiologists and health service researchers in generalist and specialist fields in a range of research topics by using this database.

    A Network Structure to Support Generalizable Research

    The CER2 Consortium was designed with the explicit goal of including health systems and independent practices that, taken together, would reflect the broader range of settings where children in the United States receive care. The use of this approach greatly increases the likelihood that findings from the network will be generalizable. Toward this end, PROS’s own network of largely independent pediatric practices31 partnered with CHOP and its PeRC network,31 the American Academy of Family Physicians Electronic National Quality Improvement and Research Network (eNQUIRENet), the MetroHealth System/Case Western Reserve School of Medicine, and Boston Medical Center/Boston HealthNet i2b2 database.32 In addition, the Allied Physicians group, a unified practice group of 25 suburban practices in the New York area sharing a common EHR, and Eskenazi Health, a multisite health system in the Indianapolis area, have also joined the consortium and begun to contribute data. Details on the characteristics of these networks are shown in Table 1. Together, the CER2 sites have the capacity to perform primary care drug therapeutics CER on a racially, ethnically, and economically diverse group of >1.2 million children seen in urban, suburban, and rural private and public sector practices and clinics by pediatricians, family physicians, nurse practitioners, and physician assistants.

    TABLE 1

    CER2 Practice Characteristics

    Network Governance

    A governance document for CER2 was created by the consortium to clarify the goals, responsibilities, and protections afforded to those contributing data and performing research. Specific details of the governance document address data storage and stewardship (including administrative, technical, and physical safeguards) as well as required training for researchers using the data. Overall, protections address risks to data privacy and, to avoid public comparisons between health systems, emphasize that data presented publicly will not single out health systems or practices by name. As an independent organization representing pediatricians nationally, the AAP owns the data contributed to the consortium and, through the existing AAP Department of Research and PROS network, acts as an honest broker to oversee research through CER2. Personal identifying information has been limited to birth dates, dates of service, and 5-digit zip codes of residence (a limited data set). Within the governance document, research allowable by the consortium was broadly defined to include studies that compare various interventions, including therapeutics, diagnostics, and practice-level delivery system characteristics; identify best practices; and inform innovations in care delivery and payment mechanisms. The governance structure allows 2 levels of participation for institutions: Partners of the consortium, who contribute data, and Affiliates, who may or may not provide data. In addition, permission may be granted for individual researchers to use the data even if their institution has not joined the consortium. This flexible structure was created to encourage participation by researchers from varied organizations while also recognizing the particular contributions of organizations that share data. To ensure that the interests of institutions sharing data are protected, an executive committee made up of the Partners decides which studies should proceed using the consortium data. A framework is provided for settling any disputes that arise. The guiding principle of this group is to support the conduct of studies by Partners and Affiliates designed to advance health and health care for children. Therefore, the governance document explicitly emphasizes the importance of publishing studies conducted with consortium data.

    Data Organization and Access

    Researchers seeking to assemble data from disparate health systems can use either a federated or a centralized database model. A federated database structure facilitates sharing and interchanges of data between autonomous databases, such as EHRs located within different practice or clinic sites or organizations, uniting independent database systems to share and exchange information.33 Authorized users access these data through Web portals that query data that is aggregated virtually for analyses. A primary advantage of this data model is that data never leave the host institution, avoiding challenges such as slow transfer speeds and potential privacy threats that may arise when data are moved between institutions. Prominent networks that have used a federated model include the Health Maintenance Organization Research Network,34 the Scalable Architecture for Therapeutic Inquiries Network,35 and the Shared Health Research Information Network, a Web-based approach led by the Harvard Clinical and Translational Science Center and involving Harvard-affiliated academic teaching hospitals that allows researchers to query aggregate data about demographics, diagnoses, medications, and selected laboratory values across hospitals.

    The CER2 Consortium has chosen to centralize data within a contracted data coordinating center at CHOP, which is also a data partner (Fig 2). The consortium provides methods for collaborators to directly and securely access the data where they reside at CHOP. This model is particularly well suited to the consortium, and potentially very broadly generalizable, because data are aggregated from sites that may lack sufficient staff (in number, training, or both) to locally extract and transform data into a prespecified format, validate the data, and adequately maintain the database to ensure that researchers can access it. With this centralized model, sites assemble data in whatever structure is most convenient and then review the data for compliance with the Health Insurance Portability and Accountability Act and applicable local institutional review board (IRB) protocols. Then, after an additional round of review to confirm consistency with IRB protocols and avoid breaches of privacy, clinical informaticians, statisticians, and data analysts working with the data at CHOP systematically review the data for quality. This process involves a thorough review of data tables to flag missing data, outliers, and implausible values. As needed, CHOP works with sites contributing data to validate data elements by inspection and chart review. To date, this iterative review has proven particularly helpful to ensure that needed data elements are extracted, that elements outside the scope of approved projects and regulatory approvals are excluded, and that the strengths and limitations of data from different sites are well characterized. This process helps to identify networks with data that are relevant to specific study questions and may also lead to additional work to address data gaps. For example, 1 contributing network structured data at the patient level, without providing links to specific encounters. Review ultimately triggered revisions to the data structure, facilitating the extraction of visit-level diagnostic and billing codes needed to achieve research goals.

    FIGURE 2

    Data flow for the CER2 Consortium. Data from each primary care practice are initially extracted at the local level from the on-site EHR to the local clinical data repository, which is located behind each practice or organization’s firewall. Study data are then deidentified and securely transferred to the Data Coordinating Center at CHOP, where they are stored on a secure offsite server and aggregated with data from other participating practices and networks.

    The lack of agreed-upon EHR data standards complicates efforts to aggregate and validate the data. For the CER2 Consortium, adoption of a common final data model for all EHR data was necessary. Those contributing data to the CER2 Consortium use a mix of data formats including the Clarity model used for Epic (Verona, WI), the i2b2 model36 used by Boston Medical Center/Boston HealthNet, the Observational Medical Outcomes Partnership (OMOP)37 used by eNQUIRENet and ePROS, and the Regenstrief Medical Record System used in the Eskenazi system. Because of its history as a tool for pharmacoepidemiologic research, the presence of a data conversion tool already developed by data analysts in CHOP’s Department of Biomedical and Health Informatics, the inclusion of both patient- and encounter-level data, and the ability to handle claim data, the team ultimately chose by consensus to use the OMOP data model and standardize all data using readily available standard-based vocabularies from OMOP. Given that not all participating sites and investigators have expertise in OMOP, the Department of Biomedical and Health Informatics has assumed the role of converting data from OMOP as needed to support data analyses for particular projects.

    In the process of data extraction, additional steps were taken to facilitate future growth of the consortium. Specifically, when we extracted data from sites with a common data model (eg, Epic and i2b2), we developed reusable scripts. These scripts were packaged to facilitate easy installation at participating sites. The CHOP data coordinating center maintains these scripts and provides assistance as needed to the sites. This approach enables us both to pull new data from current collaborators and to more readily extract data from collaborators who join the consortium and use the same EHRs as existing members.

    Innovations in CER2 to Address the Limitations of EHR Data for CER

    Standardized EHR data, pooled across health systems in the CER2 database, provide readily accessible, detailed information on children’s demographic characteristics, medical problems, growth, medications prescribed, allergies, and test results, including laboratory studies. However, specialized approaches are needed to address certain limitations of EHR data. Examples of these approaches follow.

    Central to assessing child health is the monitoring of growth, yet growth data in EHRs are vulnerable to quality problems. Common errors in growth data include weights of clothed children, incorrect placement of decimal points, missing digits due to keystroke errors, and unintentional miscoding of metric and English system values. Although these errors are common, it is not feasible to detect them with manual chart review because of the volume of data. As a result, our group is working with an expert in growth statistics to develop and validate algorithms to facilitate the automated cleaning of growth data. Preliminary data indicate that these strategies, which use an exponentially weighted moving average of all other SD scores for a particular growth parameter to determine whether a growth value should be retained, will be highly effective in ensuring the validity of data points used in CER studies with growth as a main outcome, exposure, or covariate.38 For future and ongoing studies, we expect to implement validated algorithms before data analyses focused on growth to ensure that erroneous values are excluded and true outlier values retained. This work provides a model for developing rigorous, automated methods to clean large EHR data sets. Such approaches will be increasingly needed to support valid research involving these data sets and fulfill the promise of “clinical data” to meaningfully advance health and health care.

    Evaluating health disparities is another high priority area for the consortium. However, despite the prioritization of capturing information on race and ethnicity in EHRs as part of the Federal Meaningful Use Program,39 data on race and ethnicity are often missing. To address this knowledge gap, consortium researchers are evaluating algorithms, validated in adults, that use data on surname and address in addition to standard covariates to impute data on ethnicity and race. Our results suggest that these methods are effective for children as well as adults, even given the challenges of a growing number of children living in multiracial, multiethnic families.40 With studies such as these, the consortium hopes to advance both the fields of growth measurement and disparities research and the broader goal of addressing missing or erroneous data in EHRs.

    Although EHR data offer substantial advantages over administrative data in providing detailed clinical information that extends beyond billing codes, insights into clinical decision making often entail a review of free text, especially clinician notes. Therefore, consortium members are working to obtain funding that will both test and advance the ability of commercially available text parsers to make sense of text elements such as medical histories (including the documentation of side effects) and assessments that are found in notes. Although this work is at an early stage, the application of mathematical and engineering approaches will increasingly make it possible to extract information in notes and reports across millions of encounters and diverse EHR environments. In addition, with the analysis of free text, a major concern is privacy, calling for the development of workflows that allow individual sites to code notes by using standardized algorithms that remove identifiers before the aggregation of these data.41

    In addition to developing methods to improve the utility of secondary EHR data for CER, the research team plans to develop and test strategies by using secondary data to identify participants from whom supplementary prospective data collection is warranted. These strategies, already tested at individual sites within the consortium, require additional development to function effectively across the network. Initial evaluations of these approaches will be part of the first set of planned consortium studies and will focus on identifying adverse effects of medication on children 6 to 18 years old receiving psychotropic polypharmacy. For these studies, the EHR database will be used to identify children meeting inclusion criteria and their families who, with approval from applicable IRBs, will be contacted for prospective data collection.

    To proactively improve care, consortium studies also plan to implement CDS. Such studies will address a key barrier in the CDS field: the difficulty in replicating the benefits of CDS approaches for diverse practices that use multiple EHRs.42 Our hypertension CDS will focus on helping clinicians improve the identification, evaluation, and management of the many children who have unrecognized hypertension by prompting clinicians to recognize and address abnormal values and use evidence-based guidelines.43 For example, the planned decision supports will encourage clinicians to recheck abnormal blood pressure values using appropriate-sized blood pressure cuffs to help distinguish “white coat” hypertension from truly elevated blood pressures. For this study, EHR modifications will be implemented at multiple sites, and the impact of the modifications will be monitored. This work will build on the success of CDS approaches in one of the consortium sites, the MetroHealth System.44 Ultimately, our efforts will demonstrate our ability to generalize CDS strategies developed at a single site across the network.

    Even with these innovations, certain limitations of EHR data derived from pediatric practices will persist. Although data will reflect the details of clinical care provided in highly varied practice settings and the network includes sites caring for many underserved communities, children in CER2 sites are not a random sample meant to reflect regional or overall US populations. As in any resource that uses data from medical records, variation in physician documentation may introduce bias in the data that are captured. Prospective data collection, as well as manual chart review, may be needed to prevent or address these biases in certain circumstances. After a detailed review of the data, consortium researchers will adopt these strategies as needed. Furthermore, it is not possible to link children across health systems in CER2. However, given the geographic diversity of study sites, we expect few children will be cared for across multiple study sites, mitigating this potential problem.

    Conclusions and Future Directions

    The CER2 Consortium leverages the clinical data in EHRs, primarily in primary care settings, for >1.2 million children (∼1.8% of the US population <18 years of age) with the short-term goal of promoting medication safety and effectiveness for children and the longer-term objective of deriving insights from real-world data to broadly improve child health and health care. To achieve these goals, the consortium has established a flexible governance structure that prioritizes both patient privacy and data availability to researchers. Data are available within a secure research environment and stored in the standard OMOP data model, a format increasingly familiar to health service researchers around the United States. Advancing the network’s mission, multiple studies are under way that will detail research methods to improve the analysis of secondary data, supplement routinely collected EHR data with prospective data collection, and use CDS to support health care decision-making. To meet long-term goals in specific areas, data may need to be linked from subspecialty clinics, emergency departments, and hospitals, following children across the continuum of care to better extract new knowledge. As the consortium evolves, the longitudinal record in the primary care EHR will provide a foundation for understanding the diagnosis, treatment, and evaluation of children over time for a broad population of children with varying degrees of medical complexity.


    We thank Stephanie Mayne, Banita McCarn, and Weiwei Liu for help with background research and manuscript preparation. In addition, we recognize Benyamin Margolis, PhD, Stella Yu, ScD, MPH, Romuladus Azuine, DrPH, Anne Zajicek, MD, PharmD, Perdita Taylor-Zapata, MD, and Gary Mirkin, MD, for their contributions to the network. Finally, we recognize the rest of the CER2 team: Russell Localio, PhD, Michelle Ross, PhD, Elias Brandt, Janeen Leon, Elena DeBartolo, Carolina Freuder, Rob Valuck, PhD, Laurel Leslie, MD, Laura Shone, DrPH, Carrie Daymont, MD, and Angela Czaja, MD.


      • Accepted April 22, 2015.
    • Address correspondence to Alexander G. Fiks, MD, MSCE, The Children’s Hospital of Philadelphia, 3535 Market Street, Room 1546, Philadelphia, PA 19104. E-mail: fiks{at}
    • Dr Fiks contributed to the conception and design of the study and the acquisition and analysis of data, and drafted the manuscript; Drs Grundmeier, Adams, Kaelber, Pace, and Wasserman and Ms Steffes contributed to the conception and design of the study and the acquisition and analysis of data and critically reviewed the manuscript; and all authors approved the final manuscript as submitted.

    • FINANCIAL DISCLOSURE: The authors have indicated they have no financial relationships relevant to this article to disclose.

    • FUNDING: Supported by the Health Resources and Services Administration (HRSA) of the US Department of Health and Human Services (HHS) under grant R40MC24943 and title “Primary Care Drug Therapeutics CER in a Pediatric EHR Network,” number UB5MC20286 and title “Pediatric Primary Care EHR Network for CER,” and number UA6MC15585 and title “National Research Network to Improve Child Health Care.” This information or content and conclusions are those of the authors and should not be construed as the official position or policy of, nor should any endorsements be inferred by, HRSA, HHS, or the US government. Funding was provided, in part, by a grant from the Eunice Kennedy Shriver National Institute of Child Health and Human Development under the Best Pharmaceuticals for Children Act.

    • POTENTIAL CONFLICT OF INTEREST: Alexander G. Fiks is the associate director and Richard C. Wasserman is the director of Pediatric Research in Office Settings. Jennifer Steffes is an employee of the American Academy of Pediatrics who works for Pediatric Research in Office Settings. All other authors collaborate with Pediatric Research in Office Settings.