Objective. AI/RHEUM is a multimedia expert system developed originally to assist in the diagnosis of rheumatic diseases in adults. In the present study we evaluated the usefulness of a modified version of this diagnostic decision support system in diagnosing childhood rheumatic diseases.
Methodology. AI/RHEUM was modified by the addition of 5 new diseases to the knowledge base of the system. Criteria tables for each of the diseases included in the knowledge base were modified to suit the needs of children. The modified system was tested on 94 consecutive children seen in a pediatric rheumatology clinic.
Results. AI/RHEUM made the correct diagnosis in 92% of the cases when the diagnosis was available in the knowledge base of the system. It was also shown to be effective in the education of pediatric trainees through its multimedia features.
Conclusions. AI/RHEUM is an expert system that may be helpful to the nonspecialist as a diagnostic decision support system and as an educational tool.
Computers have come to play a major part in patient care activities, including literature searches, patient monitoring, and drug prescriptions. Imaging techniques such as magnetic resonance imaging and nuclear medicine would be impossible without computer technology. All of these areas are grouped under computer-based clinical decision support systems by Habbema,1 who recognized four levels of medical decision-making: 1) collecting background information on patients, both general and specific; 2) assessing diagnostic and therapeutic methods that include algorithms; 3) developing a global clinical policy, such as protocols and guidelines, which are becoming increasingly important in this era of cost-containment; and 4) developing an individualized clinical strategy that includes medical expert systems.
The term expert system refers to a computer program that simulates the diagnostic reasoning of an experienced clinician in specific areas of medicine such as rheumatology. Expert systems belong to the general area of artificial intelligence in which symbolic reasoning methods are used. An expert system has two components—a knowledge base and an inference mechanism. The knowledge base can be in any field of knowledge. In medicine, the knowledge base can include information about specific diseases and an inference mechanism designed to work with the knowledge base and with patient data to reach a conclusion such as a diagnosis or treatment plan.
There are complex software packages that can build general expert systems, called expert system shells. CTX, written at the National Library of Medicine (Bethesda, MD), is one such shell.2AI/RHEUM, a knowledge-based medical consultant system built using the CTX shell, is a multimedia expert system originally developed by Kingsland et al3 in the 1980s to assist general physicians with the diagnosis of rheumatic diseases in adults. We now report on a modification of this system to address childhood rheumatic diseases.
MATERIALS AND METHODS
AI/RHEUM runs on IBM-compatible personal computers and UNIX systems. The knowledge base includes diagnostic criteria tables for 54 common rheumatic diseases (Table 1). The criteria tables are designed using the most important clinical and laboratory features of each of the diseases. These tables were originally developed at the University of Missouri, Columbia, MO, and reviewed by a panel of rheumatologists. More recently, H. James Williams, MD, at the University of Utah has modified some of these tables.
One of the authors (B.H.A.) modified the criteria tables to suit the needs of children, prepared a set of screening criteria to enter patient data into the system, and also added 5 more diseases to the knowledge base.
AI/RHEUM's reasoning uses the criteria table paradigm with 1 criteria table for each of the diseases covered. Criteria tables (Fig 1) consist of the following elements: major criteria, minor criteria, required criteria, and exclusions. The decision elements may be symptoms, signs, or laboratory data from the patient data file. The decision element may also be an intermediate hypothesis derived from a combination of findings. An example of an intermediate hypothesis is when the data entry includes pain in the joints and limitation of range of movement and the program generates arthritis as the decision element.
The expert system matches the individual patient findings with the criteria tables and reaches diagnostic conclusions at three different levels of certainty: definite, probable, and possible. An example of a patient with definite diagnosis of Wegener's granulomatosis is given in Fig 2. When a patient has vasculitis of small vessels, midline granuloma, and antineutrophil cytoplasmic antibody the diagnosis is definite. If the same patient has small vessel vasculitis and antineutrophil cytoplasmic antibody, the diagnosis of Wegener's granulomatosis will be probable. Whenever the diagnosis is suggested at the possible level, the system recommends that you keep the diagnosis in the differential list and look for further clues to exclude or include it.
Patient findings are entered on data forms online for each patient. The data form consists of multiple screens to enter clinical findings and laboratory data. The system can capture >500 patient findings but does not require entry for each finding.
During data entry, the physician can use the multimedia capability of the system to obtain expert consultation on each of the patient findings. For example, if the physician wants to know what synovial biopsy is, he or she can choose one of the three options associated with that finding, choosing from: tell-me-more, show-me-more, or search-for-more (Fig 3).
The tell-me-more section consists of texts that define the clinical terms under four categories: what (is the observation), why (is it important), how (is it obtained), and a reference list. Figure 4 is an example of a tell-me-more screen for enthesitis.
The show-me-more option connects the user directly to a video disc containing >6000 slides and several video sequences. These include clinical photographs, microscopic slides, and roentgenograms. Figure 5 is an image from the collection on Wegener's granulomatosis.
By clicking on the search-for-more option, the physician is connected to Medline through the Grateful-Med or Internet-Grateful-Med program. All of these options are accessible during data entry without having to move in and out of the data entry mode.
At the completion of the data entry, the physician requests a summary of the patient data (Fig 2), followed by a request for the results (Fig 6). The results show the diagnosis at different levels of certainty, and the output is available in a fraction of a second. The output also lists findings that support the diagnosis, findings present in the patient that do not fit the diagnosis, and findings which, if present, will strengthen the diagnosis (Fig 7).
If the physician wants to know why the diagnosis was or was not triggered, he or she can request information on that particular diagnosis in relation to the findings present in that specific patient. The system will give the findings that triggered that diagnosis. Figure 8 shows the output from the program on the patient with Wegener's granulomatosis when the program was asked for the reasons why Henoch-Schonlein purpura was also triggered. Similar explanations can be obtained for the inclusion of juvenile rheumatoid arthritis in the list of differential diagnoses.
We conducted a prospective study to test the ability of this system to offer consultation on patients with pediatric rheumatic diseases. Case histories of consecutive patients seen by one of the authors (B.H.A.) at either the Children's Hospital of Philadelphia or at the Alfred I. duPont Hospital for Children were entered into the AI/RHEUM case entry screens after the initial visit. Most of the patients had a few laboratory data elements available from the referring physicians. Some had none. Six of the patients were from Brazil and were entered by a pediatric rheumatologist from that country. All patients except the 6 from Brazil had a follow-up period of 2 months to 1 year to confirm or reject the diagnosis.
The results were analyzed to determine whether the initial diagnosis entertained by the physician was suggested by the AI/RHEUM expert system at least at the possible level (Table 2). There were a total of 97 diagnoses in 94 patients (3 patients had a dual diagnosis). Overall, 78 (80%) of the 97 diagnoses were correctly made by AI/RHEUM. Eighteen of the 97 diagnoses did not have criteria tables as part of the knowledge base included in the AI/RHEUM program. Of the 79 conditions for which criteria tables were available in the program, the system made the correct diagnosis in 73 (92%).
In 5 patients, the program arrived at an incorrect conclusion although the correct diagnosis was available in the knowledge base. One was a patient with Behcet's disease who did not have oral ulcers. According to strict criteria, the expert system was correct in rejecting the diagnosis of Behcet's disease by suggesting nonspecific vasculitis as a possibility. In a second patient, with monarticular arthritis and positive antinuclear antibody, the expert system suggested the diagnosis of pauciarticular type I or rheumatoid factor negative polyarticular arthritis. After 1 year of follow-up, the arthritis has cleared and there is no final diagnosis in this patient. In 2 patients with Lyme disease and in 1 with acute rheumatic fever, the expert system did not arrive at the correct conclusion because of the requirements of the criteria tables. In 1 patient, the program incorrectly refused to make the diagnosis of Lyme disease although the diagnosis was available in the system. This was also because of the stringent requirements of the criteria table.
Of the 18 patients with diagnoses not included in the knowledge base, the expert system correctly recognized that criteria tables were not available for 5 diseases and refused to come to a conclusion. In the other 13 patients, the system offered some other diagnosis. Seven of them were appropriate. For example, panniculitis and retroperitoneal fibrosis are not included in the system; however, the system suggested nonspecific vasculitis as a possibility. In 6 other patients, the alternatives suggested by the system were not appropriate. Three of these children had leukemia, 1 had reflex sympathetic dystrophy syndrome, 1 had lymphangioma of the synovium, and 1 had lymphedema.
Computer-assisted decision making is currently confined to specific areas of patient management such as routine follow-up and drug monitoring. It has not played a major part in clinical diagnosis despite the availability of some excellent programs. The CTX expert system can be adapted to many areas of medicine for developing a consultation and an educational program for nonspecialists in that field. It is ideal for performing triage functions for primary care physicians and emergency room physicians. The multimedia capacity of the program allows for interactive learning and immediate feedback.
AI/RHEUM developed originally to assist in the diagnosis of rheumatic diseases in adults has been tested extensively. In the initial study, the system was tested in 384 carefully selected adult cases with an accuracy of 94%.4 Another study, from Keio University in Japan, involved 59 patients with various connective tissue diseases. The diagnosis made by AI/RHEUM was in full or partial agreement with those of the Japanese rheumatologists in 54 cases (92%). The criteria table for mixed connective tissue disease was found to have a sensitivity of 90% and specificity of 96%.5
Moens and Van der Korst6 tested the performance of AI/RHEUM in 570 different cases from a Dutch outpatient rheumatology clinic and compared it with predictions of diagnostic outcome made by rheumatologists. In this study, physicians made 839 diagnoses. Of these, 694 were definite and 145 were possible. Of these, the original model ranked 735 (88%) and the modified model 727 (87%) among the 5 most common diagnoses. Sensitivity was 62% but the specificity was 97% to 98%.
The modified AI/RHEUM system when tested in children was shown to make the diagnosis correctly in 92% of patients when the diagnosis was available in the knowledge base. Obviously the system cannot make those diagnoses for which there are no criteria tables (Table 3) in the system and should refuse to make a diagnosis for these conditions. This was what we found for 5 conditions. In 13 patients, the program offered a different diagnosis because some of the signs and symptoms are common to many rheumatic diseases.
Two diseases account for 6 of the 18 incorrect conclusions—Lyme disease accounting for 2 of 5 among the group of diseases for which criteria tables are available and leukemia and malignancy accounting for 5 of the 13 among the diseases without criteria tables. We propose to review the criteria table for Lyme disease and add a criteria table for leukemia. We also plan to perform sensitivity and specificity analysis on all the criteria tables.
More than the consultative role, this expert system provides a strong educational component. All through the case-entry mode, are prompts requesting written and visual information pertaining to the clinical features of the disease in question. This aspect was not tested in formal way. However, trainees in pediatrics who used the system found it to be educational. Some of them said that AI/RHEUM helped them consider a diagnosis they would not have considered otherwise.
The system is simple to use and capable of performing some functions of an expert in pediatric rheumatology, but it needs to be tested at other pediatric centers. The educational component needs formal testing. The sensitivity of the system in diagnosing conditions listed in the knowledge base needs to be tested.
Finally, the legal ramifications of an expert system for use by generalists is an uncharted territory. Although an expert system such AI/RHEUM can be made available through the Internet, the legal and financial issues have to be resolved before such expert systems are available more widely.
We thank Joseph Santoroski for providing programming and technical assistance.
- Received January 8, 1998.
- Accepted May 19, 1998.
Reprint requests to (B.H.A.) duPont Hospital for Children, 1600 Rockland Rd, Wilmington, DE 19899.
- Habbema JDF
- ↵Cheh ML, Kingsland LC III. Criteria table representation in the medical domain. In: Kahn M, Smith J Jr, Buchanan B, Musen M, Szolovitz P, eds. Working Notes of the AAAI Spring Symposium Series 199. Artificial Intelligence in Medicine. Menlo Park, CA: American Association for Artificial Intelligence; 1992:11–24
- ↵Kingsland LC, Lindberg DB, Sharp GC. Anatomy of a knowledge-based consultant system: AI/RHEUM. MD Comput. 1986:3:18–26
- ↵Kingsland LC III. The evaluation of medical expert systems: experience with the AI/RHEUM knowledge-based consultant system in rheumatology. In: Proceedings of the Ninth Annual Symposium on Computer Applications in Medical Care. Washington, DC: IEEE Computer Society Press; 1985:292–295
- Moens HJ
- Copyright © 1998 American Academy of Pediatrics