The birth certificate and death certificate are important sources of population-based data for assessing the extent of risk and the quality of perinatal outcome. The birth certificate contains the hospital of birth and many items, such as birth weight and race, that can serve as important risk adjusters for neonatal mortality. To assess mortality a second vital record, the death certificate, must be linked to the birth certificate. If the analysis is to be stratified by level of neonatal care or other hospital characteristics, a third file providing these details must also be utilized. The exact vital record formats, recording protocols, and quality control efforts are determined by and differ across each state. Even with these differences, the quality and completeness of vital records and their linkage are reasonable for population-based analyses. Although the most important vital outcome from a neonatologist's perspective is neonatal mortality, vital records can also be used to assess fetal, perinatal, postneonatal, and infant mortality.
The analytic paradigm that is used in quality analysis performed on data derived from the vital record states that observed outcome is a function of risk, chance, and care. Risk is a characteristic or condition such as low birth weight or low 1-minute Apgar score that elevates the probability of an adverse outcome but is beyond the control of the agent responsible for the outcome. Using risk matrices or regression analysis one determines the expected mortality for a specific institution's case-mix. This expectation is usually based on the statewide analysis of infants with a similar risk profile. A standardized mortality ratio is calculated by dividing observed by expected mortality. A hospital with a high observed mortality (12 deaths per 1000) and an even higher expected mortality based on the risk characteristics of its neonates (24 per 1000) would have a standardized mortality ratio of 0.5. Once the effects of chance have been accounted for by statistical testing this finding could indicate that mortality in this hospital is 50% lower then expected.
Although initially intended for legal and broad-based public health purposes, vital records represent an important source of data to inform perinatal quality improvement activities. The optimal usefulness of information derived from vital records requires that clinicians take an active role in assuring that data entry is complete and accurately reflects risk status, clinical factors, and outcomes. However, even a superb database will be of limited usefulness unless it is linked to an initiative that actively involves clinicians committed to quality improvement.
The birth certificate and death certificate are an important source of population-based data for assessing the extent of risk and the quality of perinatal outcome. In addition to legal and administrative uses, the 1998 revision added many items to obtain information on demographic, behavioral, and medical factors that influence the outcome of pregnancy.1,,2 A primary intention of this expansion was to gain a broader understanding of the source of socioeconomic and racial/ethnic disparities and to shed light on the relative lack of progress in reducing low birth weight and prematurely. Fortunately these factors could also be used for risk adjustment opening up the possibility of assessing the quality of perinatal care provided in a state's delivery hospitals. Guiding these analyses is the basic paradigm; observed outcome is dependent on risk, chance, and care. If our database has items that can adequately adjust for those differences in risk that are not under the practitioner's control and if our statistical techniques can reduce misinterpretation due to random fluctuations in outcome, the differences in outcome observed across institutions can be attributed to the quality of care provided. The purpose of this article is to examine the extent to which these requirements can be satisfactorily met and to illustrate several examples of quality analysis based on the linked birth and death certificate. Although the birth certificate has a number of items that could be used to assess the quality of perinatal care, the focus of this article will be on the use of vital records to estimate facility specific, risk-adjusted mortality rates. The birth certificate contains many items, such as birth weight and race, that are important risk adjusters. To assess mortality a second vital record, the death certificate, must be linked to the birth certificate. The resulting linked birth certificate-death certificate cohort file3for all births during a calendar year is used to perform quality analysis. If the evaluation includes an analysis for specific levels of care (for example, the risk-adjusted mortality of a regional neonatal intensive care unit [NICU] compared with all regional NICUs), a third file that gives the level of care for each birth facility must also be merged with the linked birth-death cohort file. To also evaluate risk-adjusted fetal mortality, a file containing information on fetal deaths4 can also be included in the database. These linked records can be grouped in a variety of ways depending on what provider information is available on the birth certificate. Although the most common analytic grouping is by hospital of birth, records can also be grouped by a specific “birth attendant” to provide outcomes for specific attendants or groups of attendants. For example, one could review the outcomes of a preferred provider organization (PPO) by grouping the linked birth certificates for all the physician birth attendants in the PPO. In registration areas where information on payer source is included, the birth certificate records could also be grouped to evaluate the risk-adjusted outcomes in the various payer sources or health maintenance organizations.5 In states where specific hospitals are part of a provider network one can easily combine the data from all these hospitals to provide an outcomes profile for the provider network.
THE VITAL RECORD
The main data source that is used to perform these analyses is constructed by linking two vital records, the certificate of live birth and the infant death (0–365 days) certificate for all infants born during a cohort year. This data source is considered secondary in that its collection and format are determined by the requirements of vital registration. The birth and death registration system in this country is decentralized. The federal government does not issue regulations governing state vital statistics systems nor does it heavily invest in the technology to improve their operation.6 The legislative mandate to record births and deaths as well as the specific forms to be used and the procedures to be followed for registration and quality control occurs at the state level. With no centralized control, achieving uniformity of information is difficult. The most important strategy for promoting uniformity in data collection are the US Standard Certificates and Forms. The first US Standard Certificate of Live Birth was developed in 1900. The 1989 revised certificate expanded information on birth attendant and location of delivery, Hispanic status, and added a number of new items on medical and behavioral risk factors of pregnancy and birth, obstetric procedures, the condition of the infant, and congenital abnormalities. In addition, the format of the certificate was changed. A number of open-ended questions on medical risk factors, obstetric procedures, and condition of the newborn that had poor response rates in the past were replaced by check boxes.1,,7 Vital records are collected from 57 registration areas; the 50 states, the District of Columbia, New York City, and the five US territories. An analysis of the items included on the birth certificates across the 55 registration areas that report to the National Center for Health Statistics by computer tape was performed in preparation for the year 2000 update of the Standard Certificate.8 As seen in Table 1, the majority of items are collected in a format that produces acceptable data. In addition to these items, 41 registration areas include the social security number of parents, 28 include parental occupation, 21 include prenatal blood test screening for human immunodeficiency virus, syphilis, and/or hepatitis, and 12 include primary financial coverage or source of payment. A number of other items such as neonatal intensive care, prepregnancy infectious illness, and weight are also collected in various registration areas.
As in the case of the birth certificate, regulations, forms, and procedures for death registration are mandated at the state level. In an attempt to promote uniformity, the National Center for Health Statistics also develops and routinely updates a standard certificate of death. Information on the death certificate includes a limited set of sociodemographic descriptors that include age, sex, race, and Hispanic status. Several sociodemographic descriptors such as educational level or occupation of the deceased do not apply to the neonate. In addition the birth certificate includes place of death and causes of death. However “place of death” varies across registration areas from specific location (for example, Bakerville General Hospital) to type of location (home, hospital, work, etc). Information on the death certificate is provided by two groups of persons. The certifying physician, medical examiner, or coroner certifies the causes of death and the funeral director provides demographic information on age, race, sex, etc and files the certificate with the state office of vital records. In an effort to improve both timeliness and accuracy, most registration areas are rapidly moving toward an electronic death certificate system.
For the purpose of quality improvement the three most important features of the death certificate are 1) the fact that the infant has died, which allows one to calculate mortality rates, 2) the age of death, which allows one to identify an infant death and classify it as occurring during the neonatal or postneonatal period, and 3) the causes of death. For this information to be maximally useful, one must link the death certificate with its corresponding birth certificate. This linkage allows one to investigate the death, its timing, and its causes with respect to the various factors recorded on the birth certificate such as hospital of birth, payer source, infant birth weight, sex, and gestational age as well as the mother's medical and sociodemographic characteristics.
The construction of a linked birth death cohort file for all births occurring during a calendar year is performed by the individual registration areas using a variety of methods. A national linkage is also conducted by the National Center for Health Statistics and the initial results are used to quality control the state files. The usual strategy is to match the birth and death records based on the similarity of factors recorded on both certificates. The National Center for Health Statistics uses four items common to both certificates to link the files and assess the quality of the match. These are: date of birth, sex, state of birth, and race. For 1987 the National linkages had a final match rate of 98.3% for infants whose births and deaths took place in the same state and 87.3% for infants whose births and deaths took place in different states.26
Although the list of items included on vital records is extensive, before we can use this data for outcomes analysis we must be assured of its completeness and accuracy. Unfortunately the completeness and accuracy of birth certificate data and the completeness of birth and death certificate linkage have been shown to vary across time, location, and data items.9 Variation across time is important because it may effect the validity of hospital time trends for mortality. Variation across time stems from changes in the actual items included and/or changes in their definition, changes in the way items are entered on the birth certificate, and changes in the registration process. Since its inception there have been eleven revisions of the recommended standard birth certificate. The most recent 1989 revision introduced a number of sociodemographic and medical data items as well as the use of a check box format. The most recent change in vital statistics recording procedures has been the changeover from a paper record system to the electronic birth certificate (EBC).6 Let us consider the implications of these changes.
Before the 1989 revision there were a number of studies on data completeness and quality. The studies indicated that some variables such as birth weight had a low percentage of missing values and were generally of high quality.10,,1 Other variables such as gestational age,11,,12 parental occupation,13birth defects,14 and prenatal care15 were felt to be problematic for public health epidemiologic and research purposes. An important finding from these early studies was that missing and erroneous data for birth certificates and the underregistration of fetal and neonatal deaths were not randomly distributed. These problems were concentrated in those subgroups at greatest risk for an adverse pregnancy outcome such as teens, unmarried, and non-white women.11,,16,17 These studies also demonstrated that there was a disproportionately high mortality in infants whose birth certificates had missing and erroneous data. A recent evaluation of the 1993 California linked birth cohort confirmed these findings. For example, in the 5341 records that did not include the mother's level of education, neonatal mortality was increased from 3.4 to 33.3 deaths per 1000 As compared with birth certificates that included maternal education, postneonatal mortality increased from 2.55 to 7.3 deaths per 1000. Because records with missing data have a greatly increased incidence of death, they should not be dropped from a risk adjustment analysis. The best way to handle missing data are to treat it as a specific category of the variable under consideration. In a risk adjustment strategy based on constructing a multiple risk matrix18 one should attempt to include specific cells for missing data within the risk matrix. If one uses logistic regression with categorical risk adjustment variables, it is straightforward to add a missing value category. Dealing with missing variables in linear regression (for length of stay as the outcome for example) is more complex in that one should use a statistical technique that estimates the missing value based on the values of the other variables on the specific birth certificate.19 The advantage of this approach is that the imputed missing value will be similar to what one would expect in an infant with a similar risk profile. An unacceptable approach to missing data are to assign an average cohort value. This is because infants with missing values on their birth certificate are rarely average in terms of their risk profile or probability of mortality. A possible exception to this approach is missing information on birth weight. Because birth weight is such an important predictor of mortality, and mortality changes so dramatically across birth weight, we believe that a record with unknown birth weight should not be included in the analysis. Table 2 shows an analysis of missing birth weight and gestational age for the 1994 California linked birth death file.20 The 1994 cohort consisted of 567 469 live births with gestational age at least 20 weeks or unknown and a birth weight of at least 500 g or unknown. Only 215 births had neither birth weight nor gestational age recorded. Their neonatal mortality was 590.7 compared with 3.0 per 1000 for records that contained birth weight and gestational age. The neonatal mortality rate (NMR) was increased to 5.7 in the 27 830 infants whose records (4.9% of the cohort) had a birth weight in excess of 500 g but missing gestational age. Although very few California records have missing birth weight, we drop them from the analysis. The title of Table 2 introduces another important aspect of the vital record, the definition of a live birth. The US standard is to report as a live birth any product of conception that shows any evidence of life regardless of birth weight or gestational age. Although this definition ensures compatibility of coding across registration areas, for the purpose of quality improvement only those live births that could be expected to benefit from perinatal care should be included in the analytic cohort. In California we restrict analysis to infants who weight at least 500 g. Although US reporting requirements for fetal death is based solely on a gestational age of at least 20 weeks,3 we also use the 500-g criteria for fetal deaths.
After the introduction of the 1989 revised birth certificate there were several studies designed to evaluate data quality. Again some variables were found to be fairly complete and accurate, while others were not as useful. Comparing a sample of 1989 North Carolina birth certificates to hospital medical records, Buescher and colleagues21 found 92% to 100% agreement for the commonly used risk adjusters birth weight, Apgar, and method of delivery. Agreement for tobacco use (84.4%), prenatal care (82.1%), weight gain during pregnancy (82.8%), obstetric procedures (68.8%), and events during labor and delivery (62.5%) were fair to good. Agreement was poor for medical history (58.5%) and alcohol during pregnancy (56.2%). In a validation study of the 1989 Tennessee birth certificate using the hospital records of 1016 cases who were either very low birth weight (VLBW) or neonatal deaths and 634 randomly selected controls, Piper et al22 found good agreement for birth weight and descriptive demographic data. Even with the introduction of the check box format “caution is needed in using birth certificate data for assessment of maternal medical risk factors, complications of labor and delivery, abnormal conditions of the newborn and congenital anomalies.” Two very important observations emerged from this study. The first is that the broader the category the higher the likelihood of completeness and accuracy. For example, in defining the categories to be used for risk adjustment, it is important to consider that although finer categories many have a better ability to distinguish gradations of risk, this ability could be degraded by a higher level of missing and erroneous values. For example, it is reasonable to include a measure of inadequate prenatal care as a risk adjuster in that mortality has been observed to be higher in the infants of mothers with inadequate prenatal care. However, the decision to use the Kotelchuck index,23 month of prenatal care initiation or trimester of prenatal care initiation to indicate the adequacy of prenatal care can only be determined by testing the predictive usefulness of each of these measure for one's specific data set. In the absence of the ability to carry out the necessary exploratory analyses, I believe that one should review the literature and opt for the broadest category that has been shown to demonstrate a significant increase in relative risk for the outcome under evaluation. A second observation that has emerged from the studies of the revised 1998 certificate confirms earlier findings from birth certificate,10,,11 fetal death certificate,4,,24and infant death certificate.17,,25,26 Missing data are more common in higher risk patients. If these records are dropped from the analysis or the missing data are replaced with the cohort's average, the true extent of risk will be underestimated in a facility with missing or out of range risk adjustment data. Although there has been a great deal of demand from facilities that the best possible risk adjustment procedures be used so as to assure a“ level playing field,” the importance of the facility's role in providing complete and accurate data has not been emphasized. If a facility does not provide optimal documentation of it's patients' risk status in the form of complete and accurate data, even the best adjustment strategy will underestimate the facility's case-mix. On the other hand a facility whose vital record system fails to report adverse outcomes such as neonatal death could exaggerate the quality of the care it provides. Although there is underreporting of deaths to vital registrars,3,25–27 this may be more of a potential problem to outcomes analysis in clinical data bases that attempt to exclude infants that the institution has defined as moribund. Although by definition a moribund infant should not be treated, it is possible that some infants who do not respond to clinical interventions could be retrospectively labeled as moribund. A troubling potential source for underreporting death exists when infants are transferred from their hospital of birth to a NICU in another state. Deaths occurring in the host state must be reallocated back to the state of birth introducing a delay in the construction of the linked birth death file and increasing the possibility of an unsuccessful linkage.25 As is the case with all aspects of vital records, each registration area has its own method for linking birth and death certificates. For the 1995 cohort year, the percent of infant deaths that were successfully linked to a birth certificate averaged 97.5% and ranged from a very low 84.3% to 100% across registration areas.3
Earlier in this article the important potential of the electronic vital record to improve both the accuracy and timeliness of the data was introduced.6 Paradoxically, in recent years the accuracy and completeness of the birth certificate and of birth certificate-death certificate linkage has been challenged by the rapid transition from a paper to an EBC.28 During the last 10 years completion of an EBC at the hospital of birth and its electronic transmission to the state office of vital statistics has rapidly replaced the traditional paper record and its system for quality control. In 1998 it was estimated that 90% of all registration areas were using the EBC. The promise of the EBC is the potential for more accurate and timely data. Data accuracy can be improved by using suspect and missing data flags during data entry at the facility of birth. The ready accessibility of data, birth attendant, and mother to clarify uncertainties raised by data entry error detection protocols and the ability for the mother to review the completed record for accuracy before discharge are important potential advantages. On-site quality review at the time of data entry and electronic transmission also have the promise of greatly facilitating birth death linkage and improving the overall timeliness of vital statistics data. A further advantage of the EBC is its suitability for incorporation into a Community Health Information Management System.29Unfortunately, the incorporation of the EBC has been accompanied by a variety of quality control problems that tend to be unique to each specific registration area. Although these problems could occur with a paper system, the EBC poses additional problems stemming from potential software, hardware, and operator errors. A recent review of the 1995 US Natality File revealed a number of items that were problematic. These include: race/ethnicity, prenatal visits, weight gain, facility codes, birthplace of mother, education, residence, last menstrual period, medical risk factors, birth order, and obstetric procedures. The problems identified included: increases in unknowns, missing data, and unexpected changes in valid responses for some items. In response to these problems the National Center for Health Statistics convened a working group to investigate the issues related to the quality and completeness of birth certificate data and make recommendations to improve the quality of medical and health information on the birth certificate.28 Because the type and extent of problems will vary across registration areas, the potential usefulness of vital records for assessing the quality of perinatal care will also vary across registration areas. Fortunately, a reasonable general assessment of data quality and the identification of highly suspect data items can usually be obtained by contacting the person responsible for building the linked birth death file in your registration area. An important consideration in this assessment is the percent of death certificates that could not be matched with birth certificates. Although this varies across registration areas, the national goal is for <5%.3 A second consideration is the extent of missing data and quality of the reporting of birth weight the most important of the risk adjusters.
Even with all the concerns raised above, neonatal and fetal mortality are useful outcome variables for quality improvement and a great deal of risk adjustment is possible using data from vital records. Table 3 shows the observed mortality and risk standardized mortality for four California hospitals with different levels of neonatal care. Based on observed mortality one might opt for delivery at the primary care facility as it has the lowest neonatal mortality. When one adjusts for the complexities of case-mix, a very different picture emerges. In this example case-mix–adjusted mortality has been estimated by taking into account each hospital's distribution of race/ethnicity, birth weight, sex, and plurality.
The most common outcomes evaluated from vital statistics files are fetal, neonatal, perinatal, postneonatal, and infant mortality. As we consider each of these outcome measures, it is important to consider what specific process or set of processes are reflected by each measure. In California during the 1980s for example a county hospital was singled out for its high perinatal mortality rate. The public interpretation was that this indicated inferior perinatal care and one should avoid delivery at this facility if at all possible. However the risk-adjusted neonatal mortality for births in this hospital was well within normal expectation. It was excessive fetal mortality that had produced the elevated perinatal mortality. On further examination the majority of fetal deaths reported by the facility were to women who presented to this hospital for the very first time with a dead fetus. Although this raises important public health questions regarding the availability and quality of prenatal care received by these women, the high perinatal mortality was not an accurate reflection of the quality of perinatal care provided by the institution. If outcomes are to be used to inform intervention activities within a hospital or birthing facility, it is important to identify the specific process or set of processes that are reflected by each outcome.
Fetal mortality usually expressed as the number of fetal deaths meeting a birth weight and or gestational age criteria (for example, 500 g or 20 weeks gestation) per 1000 live and dead births is an important outcome.4 Although the overall quality of reporting is difficult to assess, the smaller the fetus the more likely the underreporting.24 The interpretation of high fetal mortality derived from the vital record even when risk adjusted will always require further evaluation based on more detailed clinical data sources. Of particular importance is whether the excess fetal mortality is intrapartum and potentially attributable to the quality of care provided during labor or antepartum and attributable to the availability and quality of prenatal care.
Neonatal mortality is usually expressed as the number of deaths in liveborn infants with a birth weight ≥500 g, during the first completed 27 days of life per 1000 live births weighing >500 g.26 A higher then expected risk-adjusted neonatal mortality reflects the quality of care provided the infant; but, provided by whom? Because the linked birth death certificate may not specify the institutional location of death, a death in the linked birth death cohort file is attributed to the hospital of birth. Is it possible for a superb community hospital with a reputation for always transferring high-risk mothers before birth and effectively stabilizing and quickly transferring small sick neonates immediately after birth to have a higher then expected risk-adjusted neonatal mortality? Although quite unexpected by the referring hospital, the answer is yes, if infants are transferred to a facility whose neonatal mortality is higher then expected because of the quality of the care and services it provides (J. B. Gould, personal confidential communication, June 1998). If the second facility has a delivery service, the excess mortality experienced by the transferred infants would most probably also be seen in the risk-adjusted mortality analysis of its inborn births. However, if the second facility is a stand free children's hospital without birthing services, the risk-adjusted mortality of its outborn infants can not be evaluated by a monitoring system based on vital statistics data.
Perinatal mortality combines fetal and neonatal deaths and is usually expressed as the number of live births and fetal deaths per 1000 births. By definition high perinatal mortality could derive from only excess fetal, from only excess neonatal, or from both excess neonatal and fetal mortality. Because the causes of fetal and neonatal deaths are so disparate, I feel that this measure should not be used in quality of care analysis. If the analysis already includes risk-adjusted fetal and risk-adjusted neonatal mortality, combining fetal and neonatal deaths into a composite measure adds little but confusion. It should be noted that as a vital statistics measure to compare mortality across states or countries, the potential advantage of perinatal mortality is that it gets around the problems inherent in classifying a tiny birth as a fetal death or as a live birth well along in the process of dying.9
Postneonatal mortality has two very different but equally valid definitions. The usual vital statistics definition is the number of deaths from the 27th completed day of life until the 364th completed day of life per 1000 liveborn infants.7 When defined this way the infant mortality rate, the number of deaths in the first year of life per 1000 live births, will be the sum of the NMR and the postneonatal mortality rate (PNMR). For stratified and/or risk-adjusted PNMR analysis epidemiologists commonly use a second definition, the number of deaths from day 28 to day 364 per 1000 survivors to the 28th day of life (J. L. Kiely, National Center for Health Statistics, personal communication, Winter 1997). This measure is perfectly valid and has some advantageous mathematical properties but can not be combined with the NMR to estimate the infant mortality rate. A potential disadvantage of presenting an institutional risk-adjusted PNMR is the attribution of a high rate to the quality of care provided by the institution. Unlike neonatal mortality, an outcome measure that is responsive to the quality of hospital-based medical services, postneonatal mortality is a reflection of community issues such as sanitation, access to primary care, immunization, and even sleep position.26 Although there has been some concern that the increased survival brought about by modern medical advances of the 1980s and 1990s may have delayed some neonatal deaths into the postneonatal period, studies have shown that most small sick infants still die quite early in the neonatal period.30,,31 The advantage of identifying an institution whose population experiences a high postneonatal mortality is that it provides a possible location for intervention. For example, one could set up a hospital-based postpartum program that stresses the importance of sleep position, immunization, etc and facilitates obtaining primary care. In some clinical NICU databases it is useful to measure death that occurs after 28 days. In this situation I prefer the measure “death during the first 28 days or during the index hospitalization if it extends beyond the first 28 days of life per 1000 NICU admissions”.
Infant mortality is defined as the number of deaths in the first 365 days of life per 1000 liveborn infants.25 As pointed out above approximately two-thirds of these deaths take place in the neonatal period and stem from perinatal causes while the one third that takes place after the first month of life reflect predominantly nonperinatal etiologies.26 Because it reflects two very different sets of etiologies, I feel that it should not be used to assess a hospital's quality of perinatal care. It is important to keep in mind that any “confidential” outcome is potentially subject to three interpretations, that of the medical community, the public, and the press. Choosing outcomes that may be fairly attributed to the caregivers, and clearly defining the processes that an outcome variable reflects both in the accompanying text and on the tables themselves are important strategies to align these three interpretations. Regardless of presentation strategy, any outcome variable that is included on a facility's risk-adjusted analysis runs the risk of being interpreted as reflecting the quality of care provided by the institution.
In addition to mortality, the vital record contains other information that can be used to assess the quality of perinatal care. In states that report Apgar it is possible to perform an analysis of low Apgar rates. Observed and risk-adjusted cesarean section rates and vaginal birth after cesarean rates can also be estimated from vital records for an entire state, a perinatal region, a facility, a practitioner or group of practitioners. The percent of VLBW infants who are delivered at a hospital without neonatal intensive care beds can also be determined by combining the vital record file, which always has the name of the birth facility with a file that lists the facility's level of care. This measure is a health goal for the year 2000 and provides an estimate of the effectiveness of regionalization.32
The level of analysis depends on what variables are available for grouping and how the birth certificates are grouped. All birth certificates contain the name of the birth attendant and the facility or place of birth. In a facility level analysis, vital records are grouped by facility. For a perinatal regional analysis the records from each facility in a particular region are grouped. If provider is included on the birth certificate it is possible to do an analysis at the provider level.
It is also possible to use linkage strategies to enhance the usefulness of the vital record as a data source for quality improvement. The first example is New York State's Quality Assurance Reporting Requirements Program to assess the quality of care provided by managed care plans.5 For 1994, rates of risk-adjusted low birth weight and first trimester entry into prenatal care as well as the observed rates for VLBW, prenatal care utilization, cesarean section, and vaginal birth after cesarean were compared for Medicaid and commercial managed care providers. To build the data set for this analysis managed care plans submitted identifying information on mothers who had a delivery during the reporting year, which was, matched through a linkage algorithm to the vital statistics birth files. The birth certificate files not only provided the outcomes listed above but provided clinical, demographic, and socioeconomic information for risk adjustment. A second example of linkage, supported by the California's Office of State Health Planning and Development, linked the state-linked infant birth/death file and a modification of the Universal Billing Form 1992 file.33 This database allows one to select outcomes from the International Classification of Diseases (9th revision) and procedure codes available on the discharge-billing file and adjust these outcomes based on the birth weight, clinical, demographic, and socioeconomic information from the birth certificate. Although the New York example uses personal data, such as the mother's name, to link its file, the California linkage is done solely based on ZIP code of residence, clinical and sociodemographic factors.33 In a project sponsored by the Pacific Business Group on Health, it was possible to successfully link the vital statistics linked birth/death file, the infant's hospital discharge billing file, the mother's hospital discharge billing file, and the hospital discharge file for any readmissions the infant may have experienced during the first year of life. Preliminary analysis suggests that this file will be capable of a facility level analysis of neonatal readmission rates during the first year of life as a function of the institution's discharge practices risk adjusted based on birth weight and sociodemographic information from the birth certificate, and clinical information from the International Classification of Diseases (9th revision) discharge codes of the mother and the infant (B. Herrchen, Health Information Solutions, personal communication, March 1998). Of note is that these linkages developed by Dr Beate Herrchen were performed very successfully without using personal data such as names, hospital numbers, or social security numbers.
The analytic paradigm that is used in quality analysis performed on data derived from the vital record states that observed outcome is a function of risk, chance, and care. The purpose of this section is to discuss what is meant by risk, and describe several strategies that have been used to adjust for the risk characteristics based on vital records. I define a case-mix risk characteristic as a condition that elevates the probability of an adverse outcome and is beyond the control of the agent accountable for the rate of adverse outcome. Consider the following example: Hospital A has a NMR of 9 per 1000 live births in a state where the average NMR is 3 per 1000. The question “Does Hospital A have a high neonatal mortality” seems reasonable at first glance; however, if it were asked in the context,“if so we should avoid sending patients to deliver there,” the question as stated above would not provide enough information to make this decision. According to our quality analysis paradigm one must ask “Does Hospital A have a NMR that is higher then expected for the risk profile of the population it cares for?” If the expected mortality observed across the state for patients similar in risk to those delivered in Hospital A is 12 per 1000, Hospital A's observed mortality of 9 per 1000 is much better then expected. Just how much better then expected can be expressed by a standardized mortality ratio ([SMR] = observed rate/expected rate). The SMR for Hospital A is 0.75 (SMR = 9 ÷ 12). If we put statistical concerns aside until the next section, an SMR of 0.75 indicates that the observed mortality in Hospital A is 25% lower than expected based on its case-mix.
An alternative to the SMR would be to calculate a risk-adjusted mortality for Hospital A using a case-mix severity index defined as the ratio of the average mortality observed in the entire state cohort (3 per 1000) divided by the institution's expected mortality (12 per 1000). The neonatal mortality of 3 per 1000 experienced by the entire state is one forth of the 12 per 1000 neonatal mortality expected in Hospital A based on the risk profile of its patients. One can use the case-mix severity index (state observed NMR ÷ Hospital A expected NMR = 3 ÷ 12 = 0.25) to adjust for the increased risk experienced in Hospital A to estimate a risk-adjusted NMR for Hospital A (case-mix severity index × Hospital A observed NMR = 0.25 × 9 = 2.25). The risk-adjusted NMR for hospital A is 2.25. Statistical considerations aside, Hospital A's risk-adjusted NMR of 2.5 per 1000 is much lower then the average state's NMR of 3 per 1000. Returning to the original question rephrased as “Given Hospital A's high observed NMR of 9 per 1000 in a state where the average NMR is 3 per 1000, should we refer our patents for delivery?”, we can answer as follows. The high NMR observed in Hospital A is a reflection of the very high-risk population that it cares for. Both the SMR of 0.75 and the risk-adjusted NMR of 2.5 indicate that Hospital A has a lower NMR then expected and would be a desirable facility for our patients.
Tables based on case-mix–adjusted mortality that present “crude” and “risk-adjusted” mortality and a test of significance are concise and easy to read. However, a presentation that gives the observed mortality, the expected mortality, the SMR, and its statistical significance provides a much clearer picture of the extent of risk and quality of outcome.
The fundamental notion of risk adjustment is that it is possible to estimate an expected level of outcome that will account for factors that are not under the control of the agent being assessed. To be included in a risk adjustment procedure there should be evidence that the factor alters the probability of adverse outcome and is not under the control of the entity being evaluated. For example infants with low 1-minute Apgar scores have a higher probability of neonatal mortality.34 If a hospital's perinatal services are being evaluated, it would not be appropriate to risk adjust for the 1-minute Apgar in that after adjusting for maternal medical conditions the 1-minute Apgar will be dependent in large part on the quality of obstetric management. However, if NMR is being used to assess the quality of neonatal care, the 1-minute Apgar is a reasonable risk adjuster in that it is not primarily under the control of the neonatologist. The first step in evaluating the potential validity of a risk adjustment procedure is to consider if any of the adjustment factors are under the control of the entity being evaluated. For example, cesarean delivery is associated with a decreased risk of fetal mortality, but should not be used as a fetal mortality risk adjuster as it represents an obstetric intervention directed against fetal mortality. The second step is to be certain that the model considers the inclusion of the major available outcomes predictors that meet this criterion. All inclusiveness may not be as critical when risk adjustment is based on variables available in the vital record. This is because birth weight accounts for the majority of the variation in models based on birth certificate variables. In an evaluation of logistic modeling for risk-adjusted mortality based on the 1990 California cohort of 617 689 births, “pseudo R squares (R2)” (the estimate of the percent variability in the dataset explained by a logistic model35) of 0.392, 0.380, and 0.085 were obtained for neonatal, fetal, and postneonatal mortality. The contribution of birth weight classified in 250-g categories to these “R squares” was 0.379, 0.312, and 0.065, respectively. In the 1990 California cohort birth weight alone accounted for 97% of the explanatory power of the neonatal, 82% for the fetal, and 77% of the postneonatal model.35Table 4 shows the marginal “R squares” contributed by forwardly selected variables that significantly (P < .05) improved the neonatal model.
We are most familiar with the use of regression models for hypothesis testing. In this mode only variables that make a significant contribution to the model are retained in the final model. However, as seen in Table 4, with the exception of birth weight most of the adjustment variables contribute very little to the model. In preliminary analyses we have found that minor variations in the cohort from year to year change which risk predictor variables will meet the “significance” inclusion criteria. A way around this dilemma is to consider the regression equation as a convenient tool for fitting data based on a large number of risk factors to the pattern of mortality seen in the cohort. Because this “curve fitting” approach does not assume that the relationship between risks and mortality is generated by a probability model, the notion of P values for the coefficients associated with the risk variables is not appropriate. All risk factors on the birth certificate that have been demonstrated to effect mortality (or the outcome under consideration) can be included in the regression formula as they have an a priori potential for improving the curve's fit.
Regardless of which view one takes with respect to risk predictor inclusion (as discussed in the section on the vital record), there are several advantages in using categorical risk factors in the regression equation. The first is that using a few broad categories will decrease the likelihood of missing or inaccurate data.22 The second is that missing or improbable data can be assigned to a missing category. Because infants whose records have missing data have been shown to have increased mortality, assigning missing data to a variable specific missing data category will assure that these records are not excluded by the analytic software. Once the regression equation has been formulated based on the entire cohort, it can be used to estimate a birth's expected mortality based on its risk profile. The expected mortality is estimated for all births in a facility. These values are then averaged to compute the facility's overall expected mortality. The final step is to calculate the hospital's indirectly standardized SMR by dividing the observed mortality by the expected mortality.
The risk matrix approach is also used as a risk adjustment strategy. In the risk matrix approach the population is divided by factors that are known to influence mortality. In the California matrix developed by Williams and Chen18 each birth is characterized by sex, plurality, birth weight (250-g intervals for singletons, 500-g intervals for multiple births), and for singletons by race/ethnicity (black, white, Hispanic, or other). All births are then assigned to 1 of 150 distinct cells. A mortality rate is calculated for each cell and represents the statewide mortality for that particular risk category. The statewide mortality observed in each cell defines the expected mortality for infants with the cell's risk profile. For example in 1993 the rate of neonatal mortality experienced by the cohort's 260 singleton, female, Hispanic, 750-g to 999-g infants was 135 per 1000. To determine the expected neonatal mortality in a facility one assigns each of its births to the appropriate risk matrix cell and records the associated statewide neonatal mortality for the cell. The facility's overall expected neonatal mortality is obtained by averaging the expected mortality for each of the facility's births. The SMR is then used to compare to the facility's observed mortality to its expected mortality.
In both the regression and the risk matrix approach the average experience across the entire state (or analytic cohort) is used to set the “gold standard.” A more lofty expectation could be developed by excluding poor performers. Based on an initial analysis one could determine the third of hospitals with the highest SMRs. Note that a high SMR indicates poor performance in that the observed rate exceeds the expected rate. The births at these hospitals could then be removed from the analytic cohort and the risk matrix or regression coefficients recomputed. This would generate an expectation based on the two-thirds of facilities with the best performance. The revised expectations would then be applied to the births at all the facilities.
Although our discussion of risk adjustment has been focused on mortality the same principles and procedures can be applied to any outcome, for example low 5-minute Apgar, cesarean rates, or the percent of births delivered postterm.
To facilitate presentation, the discussion has been limited to an evaluation of all birthing facilities within a state. The inclusion of facilities with vastly different missions, staffing patterns, and patient risk profiles to build a universal adjustment model has been criticized as mixing apples and oranges. Volkswagens and Mack trucks is a much better analogy. Although both are motor vehicles and can transport from point A to B, it is difficult to conceptualize their inclusion in a universal automotive adjustment strategy. A reasonable solution is to evaluate within category. In California all hospitals are evaluated using statewide expectations. In addition regional NICUs, community NICUs, and intermediate NICUs are also evaluated based on the risk profile and pattern of mortality observed in each of these hospital groupings.
Returning to the paradigm “outcome is a function of risk, chance, and care,” let us now consider the element of chance. Our strategy to adjust for risk was to develop an SMR. This indirectly standardized ratio of observed to expected mortality is subject to chance variation, and the magnitude of this chance variation is roughly proportional to the inverse of the number of expected deaths.18 Because death during the first year of life is a relatively rare event, small fluctuations in outcome can lead to large differences in risk interpretation, especially for facilities with a small delivery volume. It is important to account for the chance component by using tests of significance or confidence intervals. Following the small sample size recommendations of the National Center for Health Statistics,7 we have chosen to base the statistical tests for the analysis of risk-adjusted mortality in California's birthing faculties on the Poisson distribution.37 The SMR allows one to evaluate if a hospital's performance is better or worse then expected. This is important for quality improvement in that hospitals whose observed mortality is lower then expected have the potential of serving as benchmarks. Because we test to see if a hospital has a higher or has a lower mortality then expected. it is important to use a statistical test that is two-sided. A hospital whose observed and expected mortality is equal will have an SMR of 1. A hospital whose observed mortality is 10% higher then that expected based on the entire cohort will have an SMR of 1.10. Statistics are used to assess the probability that the SMR of 1.10 has not occurred by chance. When observing the results of several institutions it is often tempting to rank-order them based on their SMRs, especially if one hospital has a high and statistically significant SMR (for example, 1.25, P < .05) and the second hospital has a seemingly lower SMR (such as .85,P < .10). This is a misuse of the SMR test statistic. As described above, the statistic only tests likelihood that an SMR differs from 1 by chance. Hospitals should not be ranked based on their SMRs or based on their risk-adjusted mortality rates unless specific statistical procedures have been set up to test for significant differences between institutions.
Regardless of the sophistication of one's statistical analysis, hospitals with a small number of births will always be difficult to analyze because of the large degree of chance variation in their outcomes. From the quality improvement perspective three possible solution are possible. If the number of births is too small for statistical testing (less the 207), one can conduct a multidisciplinary detailed analysis of each death in collaboration with the staff of a regional perinatal center. In larger services combining several years worth of data can often provide a more stable estimate. The third (most optimal) approach is to compliment mortality by using a morbidity database to monitor adverse outcomes that are more common than death. Because morbidity is more common than mortality, for any given number of births a more confident estimate can be obtained for morbidity measure.
In this final section I will present a community and a statewide example of using vital records data for quality improvement.
Rochester, New York
The first example is of a community network of 6 hospitals in Rochester, New York. Dr Henry Thiede writes, “About 7 years ago our community of 6 hospitals, all which have an obstetrical service, embarked on a three-part venture. The goal was/is to provide timely, high-quality aggregate data for each hospital and the community. Because each hospital must complete a birth certificate for every liveborn, the bottom half of which contains considerable clinical information, we wrote a computer program to generate the birth certificate electronically, thus giving each hospital its own dataset. Since then New York State has adopted the Genesis EBC software to which we then switched. Concurrently, we organized data extractors from each hospital into a cohesive group that meets monthly to address a wide array of issues relating to definitions, interpretations, conflicting information, omissions, etc. A week or so before each meeting, the extractors are provided with a deidentified copy of an actual obstetrical record to code. The results of their codings are collated and discussed at the meeting.
Last, we reached an agreement with each hospital to upload its birth certificate data every week to a nonpartisan agency (Rochester Healthcare Information Group) that would maintain the community database and generate perinatal reports of all sorts on aggregated deidentified data. Release of the data are governed by carefully crafted guidelines. Rochester Healthcare Information Group also observes the data quality; works closely with the regional perinatal center; and runs the ongoing record extractor educational program. Within the data access guidelines, deidentified and provisional aggregate data are given on request to 1) providers (hospitals, physicians, nurse midwives) to enable them to compare their individual performance to the whole community; 2) county health department, Healthy Start, Health Systems Agency, and perinatal network for surveillance, grant applications, needs assessment, etc; and 3) perinatologists in the regional perinatal center who do outreach education for physicians and hospitals in their region. New York State has recently implemented a statewide initiative similar to what is described above, contracting with the regional perinatal centers to carry out the program. Thus, we now have a 9-county data set and a greatly expanded medical extractor training program (it's not part of the New York State initiative). A really nice benefit of the new New York State software is a report-generating module with which each hospital can easily generate “canned” monthly Continuous Quality Improvement reports for internal use. While I was still chair of our department, we generated semiannual reports giving each member of the staff his/her results as well as aggregate deidentified results of the entire staff performance for comparison. I counseled those providers whose performance fell outside two standard deviations of the mean.
New York is now providing us, through the birthing hospitals, quarterly lists of infant deaths, and we are beginning the process of linking those to the birth certificates. In the future, I hope to be able to import birth certificate data directly into the National Fetal Infant Mortality Review Program (NFIMR) software to save some of the time and effort involved with the NFIMR data gathering process“ (H. Thiede, personal communication, June 7, 1998).
A major criticism of the use of vital records for quality analysis is that they are entered by nonclinicians who are not sensitive to or knowledgeable about clinical issues. The purpose and education of these registrars is solely to create birth certificates, not a database for evaluating the quality of clinical care. Although vital record databases are secondary data sources in that they were not created for the purpose of quality analysis, they have great potential for this purpose. This potential will only be realized when clinicians take an active role in assuring that data entry is complete and accurately reflects risk status, clinical factors, and outcomes. Rochester's ongoing data extractor training program is a superb model for us all. Only when clinicians begin to take a major role in the data collection and entry process can they have confidence in the vital record's ability to inform quality improvement.
A second important aspect of the Rochester collaborative is the many uses to which the data are put. A database is only worthwhile if data are used to inform action. The larger the number of local stakeholders receiving data that they consider to be useful, the more assured the program's longevity.
My final comment is to point out that the timeliness of the Rochester system is in marked contrast to the usual lag in the release of vital statistics data. The recent widespread adoption of the EBC, and local report generating software makes possible analyses that are close to real time. New York's quarterly release of infant deaths and Rochester's plan to link these to the birth certificate will greatly enhance both the timeliness and usefulness of their evaluations. In summary, the Rochester example provides an exceptional local model for harnessing the power of vital statistics data to improve the quality of perinatal care.
The second case example is the California Perinatal Profiles.38 California was one of the first states to develop a program of statewide risk-adjusted mortality during the perinatal period. The initial analytic strategy developed by Ronald Williams in the early 1980s used a risk matrix approach to estimate mortality and used the SMR as the evaluation index.18 The program was discontinued in the late 1980s in part because of controversy surrounding the release of the evaluations and criticism concerning the timeliness of the data. By the mid-1990s the California State Bureau of Maternal and Child Health who had sponsored the original effort received a number of requests for the program to be reinstated. Many of the requests were from hospitals, which reported that the data provided in the past had been important for quality assessment and improvement. There was also the impression that this information could be very helpful to an institution in California's highly competitive managed care market. A second need for examining the quality of perinatal outcomes stemmed from the rapid increase in the number of community neonatal intensive care units and a breakdown in the traditional pathways of regionalization that had taken place during the 1990s. These forces led to the decision to develop a statewide program of perinatal quality improvement. The program has three major activities: 1) performing a risk-adjusted fetal, neonatal, and postneonatal mortality analysis for California's birthing facilities and 15 perinatal regions, 2) offering assistance in reviewing and interpreting the data for the purpose of quality improvement, and 3) improving the quality, timeliness, and scope of the analysis. In contrast to the consumer approach (for example, the Missouri Department of Heath's 1994 Show Me Buyer's Guide: Obstetrical Services that published each hospital's charges, length of stay, level of care and services, cesarean section, transfer agreement, risk-adjusted neonatal mortality, patient satisfaction, etc39) the California program is confidential and directed toward assisting facilities and their staff in assessing and improving the quality of their perinatal care. The analyses based on an all California-linked birth/death cohort file are performed at the Perinatal Projects Laboratory, School of Public Health University of California, Berkeley. Risk adjustment is performed using the risk matrix approach with significance testing based on the Poisson distribution as described previously. For each perinatal region and each birthing facility a data sheet is developed which presents a sociodemographic profile, and a mortality analysis based on all infants born in California weighing at least 500 g for each of 5 years and the 5 years combined. The sociodemographic variables are representative of factors that have been demonstrated to be associated with perinatal outcome and may be incorporated into a logistic model in the future. An upper quartile value is indicated by an ↑ (up arrow) and a lower quartile value by a ↓ (down arrow). The mortality analysis presents the observed mortality, expected mortality, SMR, and its statistical significance for fetal, neonatal, and postneonatal mortality. An additional level specific mortality analysis is presented for intermediate, community, and regional NICUs. The data sheet also gives the percentage of records with one or more missing data items.Fig 1 is an example of a data sheet. A great deal of effort went into its design and extensive notes are included to facilitate interpretation.
An essential part of the project is the technical assistance provided to the facilities by the staff of the Regional Perinatal Programs of California. The regional staffs provide the facilities with a list of their deaths for the purpose of quality control. They then review the regional and facility specific data on-site to facilitate interpretation for quality improvement activities. The regional staffs are provided extensive ongoing training in the details of the data analysis as well as well as the rationale behind its construction. Each year an extensive “teaching” slide set is created that explains the analysis and presents regional as well as the confidential facility specific data. This is usually presented at a facility's perinatal morbidity and mortality conference.
A third component consists of an effort to explore ways in which to improve the quality and timeliness of the analysis. Timeliness of the data are a problem because the linked birth death cohort file is constructed by linking deaths during the first year of life to the corresponding birth certificates for all infants born during a calendar year. For example, an infant born on December 31, 1993 must be observed for the possibility of an infant death until December 31, 1994. Thus, the actual linkage for the 1994 cohort is not begun until January 1995. Data clean-up, the linkage, and risk-adjusted analysis take substantial time resulting in a lag of several years between the year of the birth cohort and release of the data. Our 1994 cohort's analysis is scheduled for release in the fall of 1998. We are now exploring ways in which the electronic birth and death certificate process can be used to facilitate more rapid linkage and data processing. Because mortality is becoming increasingly less frequent, random variation in mortality rates has become more problematic. The construction of an analytic system that combines information derived from vital records with data obtained from sources that capture morbidity is under development.
With >500 000 births at more than 300 facilities, the California project is extremely complex. An important aspect has been to develop effective data presentation. This includes report forms that are easy to read and interpret, teaching materials, and a knowledgeable staff to work on site with clinicians and administrators. Maryland has also emphasized the importance of technical assistance in its analysis of VLBW mortality. Their on-site technical assistance team includes a nurse, obstetrician, neonatologist, maternal and child health professional, and data analyst. Although this may seem like a large investment, over the last several years they have been extremely pleased with the results that have been achieved (I. Horon, MD, Chief, Division of Health Statistics, personal communication, June 5, 1998).
I would like to conclude this article by emphasizing that despite the many limitations we have considered, I feel that vital records can play an important role in perinatal quality improvement. However, for the potential to be maximized, clinicians must become actively involved in assuring the uniformity of definition, accuracy, and completeness of the vital record. The key to the effectiveness of vital statistics data for quality improvement is for a consortium of clinicians, maternal and child health professionals, vital records personnel, and data analysts to work together to provide timely, accurate data that has been fairly risk-adjusted, and the technical assistance to interpret and act on the data.
I thank Ms Connie Gee for her assistance in preparing this article.
- NICU =
- neonatal intensive care unit •
- PPO =
- preferred provider organization •
- EBC =
- electronic birth certificate •
- NMR =
- neonatal mortality rate •
- VLBW =
- very low birth weight •
- PNMR =
- postneonatal mortality rate •
- SMR =
- standardized mortality ratio •
- NFIMR =
- National Fetal Infant Mortality Review Program
- ↵Centers for Disease Control and Prevention, National Center for Health Statistics, US Department of Health and Human Services. Public Use Data File Documentation. Linked Birth/Infant Death Data Set—1995 Period Data. Hyattsville, MD: US Department of Health and Human Services; 1997
- ↵Gaudino JA, Hoyert DL, MacDorman MF, et al. Fetal death. In: Wilcox LS, Marks JS, eds. Data to Action: CDC's Public Health Surveillance for Women, Infants, and Children. CDC Monograph. Centers for Disease Control and Prevention, Public Health Service, US Department of Health and Human Services. Hyattsville, MD: US Department of Health and Human Services; 1993:163–178
- ↵New York State Department of Health, Office of Managed Care, Bureau of Quality Management and Outcomes Research. 1994 Quality Assurance Reporting Requirements: A Report of Birth Outcomes Data. Albany, NY: New York State Department of Health; 1997
- ↵National Center for Health Statistics. Vital Statistics of the United States, 1992, I. Natality, Section 4, Technical Appendix. Public Health Service. Hyattsville, MD: US Department of Health and Human Services; 1995:1–23
- ↵National Center for Health Statistics. Items included on the 1989, US Standard Certificate of Live Birth in comparison with items included on Certificates of Live Birth from other registration areas, Public Health Service. Hyattsville, MD: US Department of Health and Human Services; 1996:1–35
- ↵Kleinman JC. Methodological issues in the analysis of vital statistics. In: Kiely M, ed. Reproductive and Perinatal Epidemiology. Roca Raton, Ann Arbor, Boston: CRC Press; 1991:447–467
- Shaw GM,
- Melcoe LH,
- Croen LA,
- Smith DF
- Clark K,
- Fu CM,
- Burnett C
- ↵Williams RL, Chen PM. 1978 California Birth Cohort Perinatal Statistics Organized by Hospital and Geographic Units. Prepared for the Maternal and Child Health Branch, Department of Health Services, State of California. Contract Number 80–63687. 1981
- ↵Rubin DB. Multiple Imputation for Nonresponse in Surveys. New York, NY: Wiley; 1987
- ↵Tashiro MA. Description of the California Birth Cohort Perinatal File. Sacramento, CA: Health Data and Statistics Branch, Department of Health Services; 1992–1994
- Piper JM,
- Mitchell EF Jr.,
- Snowden M,
- Hall C,
- Adams M,
- Taylor P
- ↵MacDorman MF, Rowley DL, Iyasu S, Kiely JL, Gardner PG, Davis MS. Infant mortality. In: Wilcox LS, Marks JS. eds. Data to Action: CDC's Public Health Surveillance for Women, Infants, and Children. CDC Monograph. Centers for Disease Control and Prevention, Public Health Service, US Department of Health and Human Services; Hyattsville, MD: US Department of Health and Human Services; 1993:231–249
- ↵Rowley DL, Iyasu S, MacDorman MF, Atrash HK. Neonatal and postneonatal mortality. In: Wilcox LS, Marks JS, eds. Data to Action: CDC's Public Health Surveillance for Women, Infants, and Children. CDC Monograph. Centers for Disease Control and Prevention, Public Health Service, US Department of Health and Human Services; Hyattsville, MD: US Department of Health and Human Services; 1993:251–262
- ↵US Department of Health and Human Services, Public Health Service, Centers for Disease Control and Prevention, National Center for Health Statistics, Division of Vital Statistics. Report of the Working Group to Improve the Quality of Birth Data, Executive Summary. Hyattsville, MD: US Department of Health and Human Services; 1998
- ↵Moynihan JJ, Norman K. CHIN provides vital healthcare linkages. Healthcare Finan Manage. 1994;59–64
- Meadow W,
- Reimshisel T,
- Lantos J
- ↵Public Health Service. Healthy Children 2000: National Health Promotion and Disease Prevention Objectives Selected for Mothers, Infants, Children, Adolescents, and Youth. Adapted from: Public Health Service. Healthy People 2000: National Health Promotion and Disease Prevention Objectives (full report, with commentary). Washington, DC: US Department of Health and Human Services, Public Health Service; 1991. DHHS Publication No. (PHS)91–50212
- Nagelkerke NJD
- ↵Blackwell TL, Gould JB, Heilig CM. Technical Report on Modeling Perinatal Mortality for California Hospitals, 1990. Prepared for Maternal and Child Health Branch, Department of Health Services, State of California under Contract Number 96–25766. Berkeley, CA: University of California, Berkeley; 1997
- ↵Selvin S. Practical Biostatistical Methods. Belmont, CA: Duxbury Press, Wadsworth Publishing Co; 1995:493
- ↵Gould JB, Heilig CM, Blackwell TL, Chavez G, et al. Perinatal Profiles of California Regions and Hospitals 1989–1993. Prepared for Maternal and Child Health Branch, Department of Health Services, State of California under Contract Number 96–25766. Berkeley, CA: University of California, Berkeley; 1997
- ↵Kivlaham C, Land G, Hoskins B, Fraas J. Show Me Buyer's Guide: Obstetrical Technical Report, II. Missouri Department of Health, Division of Health Resources, State Center for Health Statistics. Publication No. 18.6. Jefferson City, MO: Missouri Department of Health; 1994
- Copyright © 1999 American Academy of Pediatrics