Reliability of Telemedicine in the Assessment of Seriously Ill Children
Abstract
BACKGROUND AND OBJECTIVE: Data are limited that establish the clinical reliability of telemedicine in evaluating children who are seriously ill. Evaluation of a seriously ill child poses a challenge in that telemedicine is primarily visual, without the ability to perform a “hands-on” physical examination. Previous studies evaluating observation in assessing febrile children and children in respiratory distress have validated observation as both predictive and reliable in detecting underlying illness. The purpose of this study was to determine the interobserver reliability of telemedicine observations, compared with bedside observations, in assessing febrile children and children in respiratory distress.
METHODS: Children 2 to 36 months old presenting with a fever were evaluated by using the Yale Observation Scale; patients aged 2 months to 18 years presenting with respiratory symptoms were evaluated by using the Respiratory Observation Checklist, a list of observational signs of respiratory distress derived from validated studies and developed specifically for the present study by the authors. Telemedicine communication used commercially available tablet devices that provided 2-way, live-streamed images with audio.
RESULTS: A total of 132 febrile subjects were evaluated by using the Yale Observation Scale. Strong agreement (Pearson’s correlation coefficient, 0.81) was found between bedside and telemedicine observers. A total of 145 subjects were evaluated by using the Respiratory Observation Checklist. Excellent agreement between bedside and telemedicine observers was found for the impression of respiratory distress (κ = .85) and good agreement (κ > .6) for the majority of the remaining components of the checklist.
CONCLUSIONS: Telemedicine, using commercially available telecommunications equipment, is reliable in the assessment of febrile children and children with respiratory distress.
- CI —
- confidence interval
- PED —
- pediatric emergency department
- YOS —
- Yale Observation Scale
What’s Known on This Subject:
Previous studies have evaluated the feasibility and effectiveness of telemedicine in assessing and treating ill children. However, no studies have formally evaluated how reliable telemedicine is and whether data presented with telemedicine are similar to observations made at the bedside.
What This Study Adds:
This study found that the application of telemedicine, using commercially available telecommunications equipment, is reliable between bedside and telemedicine observers in the assessment of febrile children and children with respiratory distress.
Telemedicine is the use of electronic information and communications technologies to provide and support health care when distance separates the participants.1 Telemedicine exists in many forms, including telephone consultations and more technologically advanced systems such as teleradiology and telestroke. It is evolving into an important tool for transforming patient management by providing specialized care to areas with limited access. Currently, implementation of telemedicine applications exists in most medical and surgical specialties, with few examples in pediatrics.2–8
In emergency and acute care settings, data that establish the clinical reliability of telemedicine in evaluating children who are seriously ill are limited. Previous studies have evaluated the feasibility of telemedicine in assessing and treating ill children.9–11 Few studies have formally evaluated how reliable telemedicine is; that is, whether data presented with telemedicine are similar to observations made at the bedside.12
In addition, the evaluation of a seriously ill child poses a challenge for telemedicine in that telemedicine is primarily visual in its application and has limited ability to perform nonvisual components of the physical examination (eg, palpation, percussion). However, previous studies evaluating the role of observation in assessing febrile children and children in respiratory distress have validated observation as both predictive and reliable in identifying underlying serious illness.13–16
The purpose of the present study was to determine the interobserver reliability of telemedicine observations, compared with bedside observations, in assessing febrile children and children in respiratory distress.
Methods
This prospective observational study was performed in a pediatric emergency department (PED) located in an urban tertiary care children’s hospital with an annual census of 35 000 visits. Our institutional review board approved this study.
A convenience sample of 2- to 36-month-old children presenting with a fever were evaluated by using the Yale Observation Scale (YOS) (Fig 1). The YOS is an observational assessment tool, which, when performed in a systematic manner, predicts serious illness in a febrile child. It consists of 6 observational items: quality of cry, reaction to parent stimulation, state variation or ability to be aroused, color, hydration status, and response to social overtures. Each item is scored from 1 to 5. A cumulative score ≥16 predicts underlying serious illnesses (eg, pneumonia, meningitis, bacteremia).13 The YOS has been validated in follow-up studies.14,16
Components of the YOS.
A convenience sample of children aged 2 months to 18 years presenting with respiratory symptoms (eg, cough, dyspnea, wheezing) were evaluated by using the Respiratory Observation Checklist (Fig 2). This tool, developed specifically for the present study by the authors, is a checklist of common observational signs of respiratory distress. It is readily applicable to telemedicine in that it contains only visual signs. The components of the checklist were derived from previously published and validated respiratory scores, including the Pediatric Asthma Severity Score and the Pediatric Respiratory Assessment Measure.17–19
Components of the Respiratory Observation Checklist.
Each component of the Respiratory Observation Checklist was assessed in a dichotomous fashion (present or absent). Specific physical signs included: age-appropriate tachypnea; perioral cyanosis; nasal flaring; tripoding; thoracoabdominal asynchrony; and supraclavicular, substernal, and intercostal retractions. In addition, the subject’s mental status (normal or depressed) was assessed. The final component was a global assessment as to whether the subject was in respiratory distress.
Subjects eligible for enrollment were identified during the initial PED triage assessment. Verbal consent/assent was obtained for enrollment in the study. Subjects deemed clinically too unstable were excluded from the study.
Before traditional bedside history taking and physical examination, 2 observers (the bedside observer and the telemedicine observer) evaluated each subject. The bedside observer evaluated the subject by using either the YOS or the Respiratory Observation Checklist tool, or both if eligible. Video images of the subject were simultaneously transmitted to the telemedicine observer, who evaluated the same subject by using the same tool(s) as the bedside observer. The telemedicine observer was typically stationed in the PED or was remotely viewing images from a different site (eg, home).
Video image acquisition used the high-definition video camera on Apple’s iPad tablet device (720 HD, fourth generation; Apple, Cupertino, CA). Images were transmitted by using Apple’s FaceTime application. The telemedicine observer viewed images of the subject on a separate iPad high-resolution display (2048 × 1536; fourth generation). Using the 2-way communication capabilities of the FaceTime application, the telemedicine observer was able to command the direction and positioning of the video camera at all times via a separate assistant controlling the angles and distance of the camera.
The telemedicine observer did not directly observe the bedside observer’s assessment of the subject during video imaging acquisition. The assistant was instructed not to capture any images of the bedside observer’s evaluation of the subject, necessitating “taking turns” by the observers. The observers were blinded to each other’s results, and there was no direct communication between the bedside and telemedicine observers during the evaluation of the subject.
Images of the subject were securely transmitted by using our institution’s secure, password-protected Wi-Fi connection (Wi-Fi Protected Access II). Similarly, the telemedicine observer viewed images by using a secure, password-protected Wi-Fi connection that is local to where the telemedicine observer was stationed (eg, PED, home). Images were live-streamed with no video recordings. No patient identifiers were recorded.
Observers for this study, for both bedside and telemedicine observations, were pediatric emergency medicine attending physicians, fellows, and mid-level providers. Each observer underwent detailed instruction on the application of the YOS and Respiratory Observation Checklist tools before study initiation.
To further evaluate interobserver reliability, simultaneous bedside–bedside observations on the same patient by 2 observers scoring the same instrument (YOS or Respiratory Observation Checklist) on the same patient or by 2 observers simultaneously scoring telemedicine images using the same instrument on the same patient were assessed.
Data Analysis
The primary objective of the present study was to determine the reliability of telemedicine observations, compared with bedside observations, in evaluating febrile children by using the YOS and evaluating clinical signs of respiratory distress by using the Respiratory Observation Checklist. The secondary objective was to determine the interobserver reliability between 2 observers simultaneously assessing the child at the bedside or both by telemedicine.
Reliability and agreement between telemedicine and bedside observers for febrile children according to the YOS were analyzed by using Pearson’s correlation coefficient statistics. Reliability and agreement between telemedicine and bedside observers for clinical signs of respiratory distress according to the Respiratory Observation Checklist were performed by using weighted κ statistics. Excellent agreement was defined as a weighted κ ≥ .8; good agreement as κ values between .6 and .8; and fair agreement as κ values between .4 and .6. Analyses were performed by using SPSS software, PASW version 18 (IBM SPSS Statistics, IBM Corporation, Armonk, NY).
Results
A total of 132 febrile subjects were evaluated by using the YOS (Table 1). The median age of subjects was 15 months, 55% were male, and the mean temperature of the febrile infants at home or at triage was 38.7°C. Fifteen percent of subjects underwent laboratory testing (eg, complete blood cell count, urinalysis, blood or urine culture) and/or radiographic evaluations (eg, chest radiograph) as part of their evaluation of fever in the PED. Scores determined by the bedside observers were taken as the gold standards in the comparisons. The median YOS score determined by these observers was 9.
Study Assessment Tools
Comparing bedside versus telemedicine observer reliability for the YOS, there was excellent agreement between observers, with an overall Pearson’s correlation coefficient of 0.81 (Table 2). Although the majority of subjects had YOS scores <16, there was strong agreement between observers across all YOS scores.
YOS: Telemedicine Versus Bedside Observer Reliability
Reliability between 2 simultaneous bedside–bedside observations showed excellent agreement (Pearson’s correlation coefficient, 0.89 [95% confidence interval (CI), 0.78–0.94]). Similarly, excellent agreement was seen with 2 observers simultaneously evaluating with telemedicine (Pearson’s correlation coefficient, 0.81 [95% CI, 0.61–0.91]).
A total of 145 subjects were evaluated by using the Respiratory Observation Checklist (Table 1). The median age of subjects was 4.7 years, and 52% were male. Nearly one-half of the subjects were tachypneic for age on presentation to the PED; 15% were found to be hypoxic with pulse oxygen saturation <92% on room air.
Each of the components of the Respiratory Observation Checklist was analyzed in a dichotomous fashion (present or absent). The bedside observer’s scoring of each component determined prevalence for each component.
Agreement between bedside and telemedicine observers is shown in Fig 3. Occurrence of each component (ie, scored as present) is indicated in parentheses. Excellent agreement was found for the overall impression of respiratory distress, with a κ of .85. This action was followed by good agreement for thoracoabdominal breathing, age-appropriate tachypnea, supraclavicular retractions, nasal flaring, and substernal retractions (κ values between .6 and .8). Fair agreement was seen for intercostal retractions. The components tripoding, mental status change, and cyanosis were not observed in our study group.
Respiratory Observation Checklist. Telemedicine versus bedside observer reliability. n = 145. Data in parentheses indicate prevalence of each component in the study group, as determined by the bedside observer.
To account for variability of observers in applying the Respiratory Observation Checklist, interobserver variability was determined. As shown in Figs 4 and 5, there was excellent to good agreement in applying the checklist, either simultaneously at bedside or with telemedicine.
Respiratory Observation Checklist. Interobserver variability (bedside–bedside), n = 30.
Respiratory Observation Checklist. Interobserver variability (telemedicine–telemedicine), n = 30.
Discussion
Telemedicine has gained widespread application and usage, but the reliability of telemedicine observations, compared with bedside observations, has not fully been proven. The importance of determining whether observations made via telemedicine are similar enough to observations made in person underlie the important application of telemedicine: to make clinical assessments and treatment plans for patients being evaluated with telemedicine when bedside evaluation is not possible.
In our study, we sought to determine the reliability of telemedicine in assessing serious illness in children by evaluating 2 distinct groups of subjects: the febrile child and children in respiratory distress. The study data indicate that there was excellent interobserver reliability between bedside and telemedicine scoring of the YOS. There was excellent to good agreement between observers for overall assessment of respiratory distress. Moreover, there was excellent interobserver reliability when 2 observers were simultaneously scoring telemedicine video images or performing bedside observations by using the YOS and excellent to good interobserver reliability scoring the Respiratory Observation Checklist.
These 2 specific groups were chosen for study because they are often challenging to providers who are not primarily pediatric trained or who are inexperienced in caring for the acutely ill child. These pediatric patients potentially will benefit the most from the application of telemedicine with consultation with pediatric experts.
Currently, there are many different telemedicine communications systems and devices available. For our study, we elected to use iPad devices and FaceTime technology. The iPad and FaceTime application are readily available. The camera and visual display of the iPad aresimilar in specifications to other commercially products currently used for telemedicine. In addition, the iPad camera is portable and more versatile compared with stationary, pan–zoom cameras seen in some telemedicine systems. Lastly, the iPad is reasonably priced compared with traditional telemedicine workstations. It was important for us to study a technology that is economically feasible to acquire and has greater potential for widespread implementation.
In our study, febrile children were evaluated by using the YOS. Overall, we found excellent agreement between bedside and telemedicine observers, suggesting that the YOS can be applied reliably by using telemedicine. The YOS predicted serious underlying illness with scores >16. Although the majority of our study subjects had YOS scores <16, we found strong agreement across all YOS scores. The limited number of subjects with scores ≥16 was identified as a limitation in our study. In addition, we found excellent agreement between 2 observers, both at the bedside and with telemedicine. This finding suggests that there was little variability among the observers in applying the YOS tool when assessing the same subjects.
To assess respiratory distress by using telemedicine, we developed a de novo Respiratory Observational Checklist. This checklist was derived from previously validated respiratory assessment tools (the Pediatric Asthma Severity Score and the Pediatric Respiratory Assessment Measure). Because visual signs are most commonly assessed with telemedicine, the visual signs of respiratory distress from these tools were used to develop our Respiratory Observation Checklist.
We found the highest degree of agreement between bedside and telemedicine observers with the clinical impression of respiratory distress. The prevalence of a score of “present” for respiratory distress was high, noted in 56% in our study subjects. Such strong agreement by observers that a subject was in respiratory distress suggests that clinical impressions or “gestalt” is not lost with telemedicine.
The remaining physical signs of respiratory distress, including thoracoabdominal breathing, age-appropriate tachypnea, and supraclavicular retractions, all demonstrated good agreement (κ between .6 and .8) between bedside and telemedicine observers. However, only fair agreement was found for intercostal retractions (κ < .6). A potential explanation is that the camera speed and resolution may not easily discriminate the visual subtlety of intercostal retractions. Unfortunately, we could not evaluate tripoding, mental status change, or cyanosis because none of our study subjects presented with these signs.
In evaluating interobserver reliability in applying the Respiratory Observation Checklist, we found excellent to good agreement for the majority of components, for simultaneous bedside–bedside observations, and for telemedicine–telemedicine observations. Again, these findings suggest little variability among the observers in applying the Respiratory Observation Checklist when presented with the same observational data.
There were several limitations to our study. Patients who were critically ill could not be evaluated because they were too unstable to be consented and were excluded from the study. This limitation potentially explains why few subjects had YOS scores ≥16 and why some of the signs of severe respiratory distress of the Respiratory Observation Checklist were not seen. In addition, our study was performed in a tertiary care children’s hospital with observers who have specific pediatric emergency medicine expertise. The results of this study may therefore not be generalizable to a broader group of providers who do not have this same knowledge and training. Further studies will be needed to determine any differences in the performance and reliability of telemedicine comparing observers who have an expertise in the care of the seriously ill child versus those who do not. Ultimately, it is this potential application of telemedicine (to provide pediatric expertise remotely) that may offer the greatest impact for the care of the seriously ill child. Lastly, scores for simultaneous bedside–bedside observations were slightly higher than bedside–telemedicine or telemedicine–telemedicine observations. A possible explanation for this fact may be the inherent limitation of the technology used for our study, including the camera and display resolutions, camera speed, and strength of the telecommunications network (Wi-Fi bandwidth). In addition, because the camera must be manually directed, there may be some variability in the real-time images with telemedicine compared with the bedside. Lastly, our study, which used FaceTime technology and iPad devices, may not be generalizable to other telemedicine technologies.
Our future studies will aim to study the reliability and application of telemedicine in the evaluation and potential treatment of critically ill children. As technology continuously evolves, we will continue to study other telecommunications technologies (eg, electronic stethoscopes) that can be feasibly and reliably used for telemedicine.
Conclusions
The findings of our study found that telemedicine, using commercially available telecommunications equipment, was reliable in the assessment of these febrile children and children with respiratory distress.
Acknowledgments
The authors offer special thanks to the attending physicians, fellows, mid-level providers, and staff of the PED at Yale–New Haven Children’s Hospital.
Footnotes
- Accepted December 10, 2015.
- Address correspondence to Lawrence Siew, MD, 100 York Street, Suite 1F, Yale University School of Medicine, New Haven, CT 06511. E-mail: lawrence.siew{at}yale.edu
FINANCIAL DISCLOSURE: The authors have indicated they have no financial relationships relevant to this article to disclose.
FUNDING: All phases of this study were supported by the Ken Graff Young Investigator Award, sponsored by the American Academy of Pediatrics, Section of Emergency Medicine.
POTENTIAL CONFLICT OF INTEREST: The authors have indicated they have no potential conflicts of interest to disclose.
References
- Copyright © 2016 by the American Academy of Pediatrics
