Drug safety 23: 533-542, dec 2000

ORIGINAL RESEARCH ARTICLE
Adis International Limited. All rights reserved.
A Retrospective Evaluation of a Data
Mining Approach to Aid Finding New
Adverse Drug Reaction Signals in the
WHO International Database
Marie Lindquist, Malin Ståhl, Andrew Bate, I. Ralph Edwards and Ronald H.B. Meyboom The Uppsala Monitoring Centre, Uppsala, Sweden Abstract
Background: The detection of new drug safety signals is of growing importance
with ever more new drugs becoming available and exposure to medicines increas-
ing. The task of evaluating information relating to safety lies with national agen-
cies and, for international data, with the World Health Organization Programme
for International Drug Monitoring.
Rationale: An established approach for identifying new drug safety signals from
the international database of more than 2 million case reports depends upon clin-
ical experts from around the world. With a very large amount of information to
evaluate, such an approach is open to human error. To aid the clinical review, we
have developed a new signalling process using Bayesian logic, applied to data
mining, within a confidence propagation neural network (Bayesian Confidence Pro-
pagation Neural Network; BCPNN). Ultimately, this will also allow the evalua-
tion of complex variables.
Methods: The first part of this study tested the predictive value of the BCPNN
in new signal detection as compared with reference literature sources (Martindale’s
Extra Pharmacopoeia in 1993 and July 2000, and the Physicians Desk Reference
in July 2000). In the second part of the study, results with the BCPNN method were
compared with those of the former signalling procedure.
Results: In the study period (the first quarter of 1993) 107 drug–adverse reaction
combinations were highlighted as new positive associations by the BCPNN, and
referred to new drugs. 15 drug–adverse reaction combinations on new drugs became
negative BCPNN associations in the study period. The BCPNN method detected
signals with a positive predictive value of 44% and the negative predictive value
was 85%. 17 as yet unconfirmed positive associations could not be dismissed with
certainty as false positive signals.
Of the 10 drug–adverse reaction signals produced by the former signal detectionsystem from data sent out for review during the study period, 6 were also identifiedby the BCPNN. These 6 associations have all had a more than 10-fold increase ofreports and 4 of them have been included in the reference sources. The remaining4 signals that were not identified by the BCPNN had a small, or no, increase in thenumber of reports, and are not listed in the reference sources.
Conclusion: Our evaluation showed that the BCPNN approach had a high and
promising predictive value in identifying early signals of new adverse drug reactions.
Background
tional pharmacovigilance programme for consider-ation of public health implications.
The Uppsala Monitoring Centre, as the WHO Col- The aim of the 2 parts of this study was to evaluate laborating Centre for international drug monitor- ing, is responsible for the technical and scientific • pick up signals early, i.e. before publication in a maintenance and development of the WHO Internat- ional Drug Monitoring Programme. The programme • pick up valid signals, i.e. drug–adverse reaction now has 56 member countries, annually contribut- associations that are described in a standard ref- ing around 150 000 suspected adverse drug reaction (ADR) reports to the WHO database in Uppsala.
• pick up signals identified in the previous signal One of the main aims of the international pharma- covigilance programme is to identify early signals of safety problems related to medicines. To aid this,a new ADR signalling system has been implemented by the Uppsala Monitoring Centre. It complementsthe previous signal generation procedure which in- The BCPNN methodology[2,3] uses a neural net- volved the examination of large, unwieldy amounts work architecture to identify unexpectedly strong of sorted and tabulated material by an expert panel.
dependencies between variables (e.g. drugs and ad- We have previously published an overview of the new verse reactions) within the WHO database, and how signalling approach, including results from an dependencies change on addition of new data. The evaluation including a comparison against another dependencies are selected using a measure of dis- signalling system.[1] The new system is based on a proportionality called the Information Component(IC): data mining technique, using a Bayesian ConfidencePropagation Neural Network (BCPNN) to scan in- coming ADR reports and compare them statistical- ly with what is already stored in the database.[2,3] The new quarterly output contains statistical in- formation from the BCPNN scan. It also contains fre- x = probability of a specific drug being listed on quency counts for each drug and ADR listed, indi- vidually and occurring together. The figures from the y = probability of a specific ADR being listed previous quarter are also included and the data is Pxy = probability that a specific drug–adverse re- action combination is listed on a case report.
Drug–adverse reaction combinations that are sta- tistically significantly different from the back- • the number of case reports with drug X (Cx) ground of reports (‘associations’) are sent to a panel of • the number of case reports with ADR Y (Cy) reviewers for evaluation and expert opinion. Within • the number of reports with the specific combina- the WHO Programme a ‘signal’ concerns ‘informa- tion regarding a possible relationship between a drug • the total number of reports (C).
and an adverse event or interaction’.[4] As before, Positive IC values indicate that the particular com- signals of possible safety problems are circulated to bination of variables is reported to the database more all national centres participating in the interna- often than statistically expected from reports al-  Adis International Limited. All rights reserved.
Data Mining to Find New Adverse Drug Reaction Signals ready in the database. The higher the value of the IC, 2000 online version of the US Physicians’ Desk the more the combination stands out from the back- Reference.[6] The latter reference source contains labelling information approved by the US Food and From the distribution of the IC, expectation and Drug Administration (FDA) and was used as a sec- variance values are calculated using Bayesian sta- ond reference, because of its comprehensive listing tistics. The standard deviation for each IC provides of ADRs, recognised as well as suspected.
a measure of the robustness of the value. The higher All reports in the WHO database are coded using the Cx, Cy and Cxy levels are, the narrower the con- the WHO Adverse Reaction Terminology. This is a fidence interval becomes. If a positive IC value in- hierarchical classification, with the following levels: creases over time and the confidence interval nar- • system organ class: a group of adverse reaction rows, this shows a likelihood of a positive quantitative terms pertaining to the same body organ system association between the studied variables.
• high level term: a grouping term for qualitatively In this study we used drug as variable ‘x’ and ad- verse reaction as variable ‘y’. The term ‘association’ • preferred term: main terms for coding of adverse denotes a drug-adverse reaction combination where the lower 95% confidence limit of the IC value is • included term: lower level terms, e.g. synonyms with, or more specific terms than, the preferredterms.
In the analysis we used the WHO preferred terms of the selected associations and compared those a-gainst the listed terms or descriptions used in Mar- A retrospective standard quarterly BCPNN data- tindale and the Physicians’ Desk Reference.
base screening was made for the first quarter of 1993. We selected for analysis drug–adverse reac- • N = the drug was found in the source but no match- tion combinations which in this quarter became pos- ing ADR or corresponding high level terms were itive ‘associations’ (the lower 95% confidence limit described for the drug or for the drug group; of the IC value changed from a negative to a posi- tive value), and which included new drugs (first NA = not applicable, i.e. the drug was not found reported to the WHO database in 1990 or later). We in the source, or was noted as being withdrawn also selected combinations referring to new drugs, for which the upper 95% confidence limit of the IC Y+ = a high level term pertaining to the ‘preferred changed from a positive to a negative value in the term’ of the ADR, but not the specific ADR, was study period. In this paper these are referred to as described for the drug; or, the same ADR, or a high level term, was listed for the group to which We then analysed if these positive and negative the drug was referred to but not listed for the associations were widely known at the time. This was done by checking if they were listed in the 30th ed- • Y = the drug was found and the same ADR, or ition of Martindale[5] published in 1993. Martindale was chosen as it is a standard compendium of druginformation, available worldwide and containing monographs based on published information.
We subsequently analysed if the selected asso- ciations had been strengthened or confirmed over We made retrospective BCPNN scans to identify the 7 year period from 1993 to 2000. The associations if, and when, drug–adverse reaction safety signals were therefore checked against Martindale, the circulated to national pharmacovigilance centres July 2000 online edition,[6] and also against the July fulfilled the association threshold criteria.
 Adis International Limited. All rights reserved.
Before the introduction of the BCPNN, quarterly printouts at 2 threshold levels were sent to a reviewpanel for the selection of possible signals:• ‘level 2’: at least 2 case reports of the combination, with the drug being reported as suspected of hav-ing caused the reaction In the first quarter of 1993, 682 drug–adverse re- • ‘level 5’: at least 5 case reports of the combina- action combinations fulfilled the quantitative thres- tion, with the drug being reported as suspected of hold criteria for new BCPNN positive associations.
107 of these concerned new drugs. Another 32 com- For this test, we selected all drug–adverse reaction binations became negative associations in the study combinations, which had been listed in the ‘level 2’ period. 15 of those referred to new drugs.
listing of the first quarter of 1993 and had gone on The literature reference sources (30th edition of Martindale,[5] Martindale online July 2000 and Physician’s Desk Reference online July 2000[6]) We checked if these were included in the 30th ed- were checked for references to the 107 positive and ition of Martindale,[5] Martindale June 1999 online the 15 negative associations selected for analysis.
version[7] and the June 1999 online version of the The results are shown in table I and II.
Physicians’ Desk Reference,[7] and we analysed the As seen in table I, 71 of the positive associations increase of reports from 1993 to 1999 for the sig- were not listed at all in the 30th edition of Martin- dale.[5] 29 were listed by a higher level term, or indi-rectly, by referral to another drug in the same group,while only 7 positive associations were specifically Table I. Result of an analysis of the 30th edition of Martindale
(1993)[5] for positive and negative associations selected from a
BCPNN retrospective screening of drug–adverse reaction combi- For the negative associations, more than half nations entered into the WHO database in the first quarter of 1993 were listed in the 30th edition of Martindale,[5] ei- Table II shows that 78 positive associations have Positive
Associations not listed in Martindale (N or NA) been strengthened or confirmed in current litera- Association listed on high level term, or by referral ture, whereas 29 were not listed. Of the negative as- Association listed on preferred term level (Y) The aim of the Uppsala Monitoring Centre sig- nalling process is to identify previously undetected Negative
Associations not listed in Martindale (N or NA) adverse reactions to medicines. Thus, we consider Association listed on high level term, or by referral drug–adverse reactions already known as nonsig- nals (in this study there were 36 positive and 8 neg- Association listed on preferred term level (Y) ative such associations). In addition there were 10 positive and 2 negative associations relating to a ADR = adverse drug reaction; BCPNN = Bayesian Confidence
drug withdrawn immediately before the study period, Propagation Neural Network; N = the drug was found in the source
but no matching ADR or corresponding high level terms were des-
and 2 positive associations relating to a drug with- cribed for the drug or for the drug group; NA = not applicable (the
drawn during the study period. These 14 associations drug was not found in the source, or was noted as being withdrawn were excluded. Hence we obtained the results from the market); Y = the drug was found and the same ADR, or a
synonym, was listed for the drug; Y+ = high level term pertaining to
the ‘preferred term’ of the ADR, but not the specific ADR, was From this, the positive predictive value is 44% described for the drug; or, the same ADR, or a high level term, waslisted for the group to which the drug was referred to but not listed (42 of 95) and the negative predictive value is 85%  Adis International Limited. All rights reserved.
Data Mining to Find New Adverse Drug Reaction Signals Table II. Result of an analysis of the July 2000 online versions of
at the end of the first quarter of 1999. On the other Martindale and Physicians Desk Reference[6] for positive and hand, the 6 signals that were BCPNN associations negative associations selected from a BCPNN retrospective have all had more than a 10-fold increase in number screening of drug-ADR combinations entered into the WHOdatabase first quarter 1993 Four of the 6 signals that passed the associations threshold did so before being circulated within the Positive
WHO Programme. Two did not, and, although sum- Associations not listed in Martindale or PDR (N or atriptan and confusion became an association in the Associations listed in Martindale or PDR (Y or Y+) fourth quarter of 1993, the quantitative strength of the relationship has since decreased.
Negative
Associations not listed in Martindale or PDR (N or Discussion
Associations listed in Martindale or PDR (Y or Y+) At the start of the WHO International Drug Mon- ADR = adverse drug reaction; BCPNN = Bayesian Confidence
itoring Programme in the late 1960s quantitative Propagation Neural Network; N = the drug was found in the source
and statistical methods were proposed for adverse but no matching ADR or corresponding high level terms were des- reaction signalling purposes.[8] Because of con- cribed for the drug or for the drug group; NA = not applicable (the
drug was not found in the source, or was noted as being withdrawn
straints in computational power these were not re- from the market); Y = the drug was found and the same ADR, or a
alised at the time. Lately, however, there has been synonym, was listed for the drug; Y+ = high level term pertaining
a renewed interest in statistical methods applied to to the ‘preferred term’ of the ADR, but not the specific ADR, wasdescribed for the drug; or, the same ADR, or a high level term, was signal generation in pharmacovigilance. We are listed for the group to which the drug was referred to but not listed aware of work being done in several countries based on proportional reporting ratios and odds ratios,and, in the US, a Bayesian data mining tool for signalgeneration has been developed for the FDA.[9] Table IV lists as yet unconfirmed positive asso- The assessment of an ADR signalling system is ciations, excluding the 12 on withdrawn drugs. For difficult because there is no ‘gold standard’ for com- each there is a short commentary based on a pre- parison. Also there are different definitions of the Table III. Predictive value of the Bayesian Confidence Propagation
There were a total of 10 drug–adverse reaction combinations from the first quarter of 1993 ‘level 2’ Associations referring to withdrawn drugs are excluded.
listing which were subsequently signalled in the pre- Listed (Y/Y+) in the July 2000 online versions of Martindale and vious procedure. The result of a BCPNN scan of the Physicians’ Desk Reference[6] and not listed (N) in the30th edition of Martindale.[5] these, and checks against the June 1999 online ver- Not listed (N) in the July 2000 online versions of Martindale or sion of Martindale[7]and the June 1999 online version the Physicians’ Desk Reference;[6] or listed (Y/Y+) in the of Physicians’Desk Reference[7] are shown in table V. The increase in the number of reports from the N = the drug was found in the source but no matching ADR or
corresponding high level terms were described for the drug or for
first quarter of 1993, to the first quarter of 1999, is the drug group; Y = the drug was found and the same ADR, or a
also shown in table V. On analysis, 6 of the 10 sig- synonym, was listed for the drug; Y+ = high level term pertaining
nals have fulfilled the BCPNN association criteria.
to the ‘preferred term’ of the ADR, but not the specific ADR, wasdescribed for the drug; or, the same ADR, or a high level term, was The remaining 4 drug–adverse reaction combina- listed for the group to which the drug was referred to but not listed tions still had no more than 4 case reports for each  Adis International Limited. All rights reserved.
Table IV. Positive associations identified by the Bayesian Confidence Propagation Neural Network from the first quarter of 1993 which are not listed in the July 2000 online versions
of Martindale (MD) or the Physicians’ Desk Reference (PDR)[6]a These reports suggest that occasionally the therapeutic effect may decrease after some time of use. A minority of reports refers to suspected interactions with otherdrugs Only 3 patients, who all simultaneously had several other serious suspected adverse reactions, e.g. pancreatitis, duodenal ulcer, GI haemorrhage, ileus Hiccup occurred in association with, and probably secondary to, other suspected adverse reactions (e.g. vomiting, ulcer and haematemesis, abdominal pain): probably nonspecific stimulation of the phrenic nerve and not a pharmacological In all but 1 report peritonitis occurred in patients with intestinal perforation, i.e. as acomplication of another suspected adverse reaction, GI ulcer Concern over the high incidence of adverse reactions, including acute renal failure,has led to regulatory actions and, in some countries, withdrawalb Concern over the high incidence of adverse reactions, including acute renal failure,has led to regulatory actions and, in some countries, withdrawalb These reports refer to signs of – mainly cardiac or nervous system – toxicity. Allpatients simultaneously used theophylline and the term suggests that during use oflomefloxacin increased blood concentrations of theophylline were found, i.e. asuspected interaction. According to the July 2000 online version of Martindale,[6]lomefloxacin is considered not to interact significantly with theophylline or caffeine In these 4 reports no other drugs were recorded, and no other explanation was given Age and concomitant illness in hypertension patients are associated with a high riskof atherosclerosis and angina pectoris. Several other groups of antihypertensivedrugs are known occasionally to cause (increased) angina pectoris. These 22reports suggest that moxonidine occasionally precipitates or aggravated anginapectoris, that the effect promptly disappears after stopping and that it may also fadewhen the drug is continued Lacrimation disturbance and xerophthalmia have also been reported to the WHOdatabase with the related drugs buserelin, goserelin, leuprorelin and octreotide Disturbances of taste and smell have also been reported to the WHO database withthe related drugs buserelin, goserelin, leurporelin and octreotide Breast enlargement is reported in 11 pre- or postmenopausal women in 4 countries; 5 patients simultaneously used hormone preparations. In addition, there are 2 reports of men with gynaecomastia during the use of nicotine patches. Perhaps thisis a secondary effect to decreased enzyme induction after stopping heavy smoking Data Mining to Find New Adverse Drug Reaction Signals term ‘signal’. According to the definition used inthe WHO Programme a signal is essentially a hy-pothesis together with data and arguments, and it isnot only uncertain but also preliminary in nature:the situation may change substantially over time.[4,10] For the purpose of the paper we felt we would achieve a reasonable estimate of the predictive powerof the BCPNN tool by checking historical associ-ations identified by the BCPNN against standardreference sources. Martindale has worldwide cov-erage, recognition and wide availability and was usedas a standard for well known, recognised ADRs.
The Physicians’ Desk Reference, though not inter-national, gives very recent information on drugs. Ithas a comprehensive ADR listing, generally moreinclusive than that of Martindale. However, thePhysicians’ Desk Reference also includes suspec-ted adverse reactions, whether substantiated or not.
We considered an ADR listed in the Physicians’Desk Reference an indication of a possible drug–adverse reaction relationship. Table IV lists thepositive associations still not mentioned in the ref-erence sources. These cannot simply be dismissedas ‘false positives’, since at least some of them maybe true signals of ADRs that are not yet established.
The reader can draw some conclusions about themin addition to the comments in the table. Several ofthe associations in table IV raise the point that theremay well be alternative explanations, relating, forexample, to the way in which the drug is used, orconfounding underlying disease. However, the re-viewer should not dismiss the drug as causal tooreadily. Similarly also ‘true negatives’ might be asyet unrecognised signals.
The length of time chosen for the retrospective check against the literature was not arbitrary, but basedon the assumption that 7 years would be sufficientfor ADRs to be included in the reference sources,allowing for the maximum reporting for new drugsto have taken place (the Weber effect). We knowhowever that 1 new association appeared in Martin-dale between 1999 and 2000, and 7 years still maynot be long enough. Publishing delay must be con-sidered in the use of these reference sources, but  Adis International Limited. All rights reserved.
Table IV. Contd.
Reports in 2 countries. A follow up may be interesting Small number of poorly documented cases from a single country Small number of poorly documented cases from a single country Cause of death not specified. It is known, however, that nicotine may cause serious cardiac arrhythmia. One probable duplicate report Nausea usually occurred as part of a more general reaction after intravenous injection, together with urticaria, myalgia, agitation, headache In the study, Martindale and Physicians’ Desk Reference were the standards, but for all unconfirmed associations a comprehensive literature search would be indicated.
July 2000 online version of Martindale.[6] GI = gastrointestinal; N = the drug was found in the source but no matching ADR or corresponding high level terms were described for the drug or for the drug group; NA = not
applicable (the drug was not found in the source, or was noted as being withdrawn from the market); WHO-ART = World Health Organization Adverse Reaction Terminology.
Table V. Signals circulated within the WHO Programme in 1993 to 1996 originating from the first quarter of 1993, together with the result of
the Bayesian Confidence Propagation Neural Network scan of these and the subsequent check in the 30th edition of Martindale (MD 1993),[5]
and the June 1999 online versions of Martindale (MD 1999) and the Physician’s Desk Reference (PDR 1999).[7] The increase in the number
of reports from the first quarter of 1993 to the first quarter of 1999, is also shown
ADR = adverse drug reaction; N = the drug was found in the source but no matching ADR or corresponding high level terms were described
for the drug or for the drug group; NA = not applicable (the drug was not found in the source, or was noted as being withdrawn from the
market); Y = the drug was found and the same ADR, or a synonym, was listed for the drug; Y+ = high level term pertaining to the ‘preferred
term’ of the ADR, but not the specific ADR, was described for the drug; or, the same ADR, or a high level term, was listed for the group to
which the drug was referred to but not listed for the drug itself; WHO-ART = World Health Organization Adverse Reaction Terminology.
this is minimised now by their availability online us- associations may become associations with time. To avoid this we have considered the inverse of a pos- The use of our selected literature sources as a ‘gold itive association a definite negative association in standard’ is open to debate. The literature is not in- this paper. Another asymmetry is that the negative tended as an early signalling system, and uses many associations are a selection of all nonassociations.
sources for its information other than the WHO data- This assumes that definite negative associations re- base: the biases affecting inclusion and exclusion present all nonassociations, though it is clear that of ADR information therefore may be very differ- some nonassociations will became positive associ- ent. Factors, such as those affecting the differential ations in time. This again shows the difficulty of reporting to WHO and the inclusion of new infor- mation in the reference sources will have an effect An assumption was made that a substantial in- which is independent of the performance of the crease in the number of reports of an association over BCPNN. The BCPNN is run every quarter, and we the period indicated ongoing clinical interest in an selected just one quarter: since the BCPNN is used association. More reports may be seen as a support in continuous analysis, the specificity and sensitiv- for the validity of the associations, though there is ity are subject to necessary time-dependent changes often a tendency for ADRs that are becoming well in classification of ‘positives’ and ‘negatives’. It is known to be reported more frequently. Therefore, difficult to consider something as a ‘nonassociation’ the associations in table IV for which the number because of this time dependency, and it is clear that of reports have increased are of particular interest.
there is an asymmetry in the effect of time on our Another obvious limitation is that our method for results: associations cannot become nonassociations signal generation is dependent on the terminology with time, whereas nonassociations can whereas non- used for recording of adverse reactions. Very little  Adis International Limited. All rights reserved.
Data Mining to Find New Adverse Drug Reaction Signals work has been done on any of the medical termino- however, that all signals identified in the previous logies in use or proposed to determine their relative system that went on to become frequently reported value in searching for new drug signals.[11] in the WHO database were also identified in the ret- We found that 44% of the BCPNN signals are strengthened or confirmed in the current referencesources while not mentioned at all in the 30th edi- Conclusions
tion of Martindale (1993).[5] The 84% negative pre- This retrospective evaluation of the new statis- dictive value indicates that combinations not high- tical signalling tool used at the Uppsala Monitor- lighted for review, if not already known, are unlikely ing Centre has shown that the BCPNN has a high to become signals. This indicates that the BCPNN predictive value, and that it can identify early sig- is a valuable tool in the filtering of combinations for nals of adverse drug reactions. It has further strength- clinical review, and that it has the ability to find early ened our view that the BCPNN will provide a use- signals. The normal methods for assessing the power ful tool in international pharmacovigilance.
of a method are difficult to apply to the BCPNN, be- To our knowledge, this is the first time an ADR- signalling method has been subjected to a rigorous The BCPNN associations, which are not yet re- performance analysis. The lack of a ‘gold standard’.
ported in the current literature, are included in table and the dynamic nature of signal finding with time IV. If these associations were to emerge in the liter- make conventional validation methods difficult to ature in the coming years, it would increase the pos- itive predictive value of the BCPNN.
The BCPNN has the power to analyse signals Acknowledgements
further.[3] We are developing its use for looking at The authors are indebted to national centres contributing complex variables to see whether parameters such as data to the WHO International Drug Monitoring Programme.
gender, age, and other drug use increase the strength The opinions and conclusions, however, are not necessarily of association, and whether ‘syndromes’ of reported those of the various centres nor of the WHO.
terms are present. However, as with any subdivision Disclaimer
of data, a very large amount is necessary initially,to attain statistical significance in subsets. This is a The WHO database contains summary reports of individ- ual suspected adverse reactions to medicines, received from major advantage of using the large pooled WHO data- national centres in countries participating in the WHO Inter- base, and we are trying to maximise this potential.
national Drug Monitoring Programme. No causality assess- The BCPNN is not a panacea for drug safety mon- ment is made at the Uppsala Monitoring Centre, but if such an itoring. The drug–ADR combinations which reach assessment has been made by the national centre submitting statistical significance, do so only in comparison with the report, this is stored in the database. Since these reportsconstitute suspicions of adverse drug reactions, further investi- the background experience of 2 million case reports.
gation and research is needed for a full interpretation of the This is particularly important for commonly re- ported ADRs, which, however serious, would notreach significance until the quantitative experience References
for a drug and such an ADR is excessive. Sumatri- 1. Lindquist M, Edwards IR, Bate A, et al. From association to alert - a revised approach to International Signal analysis.
ptan and confusion is an example of this issue, pass- Pharmacoepidemiol Drug Saf 1999; 8: S15-25 ing the BCPNN association threshold after being cir- 2. Bate A, Lindquist M, Edwards IR, et al. A Bayesian neural network method for adverse drug reaction signal generation.
Eur J Clin Pharmacol 1998; 54: 315-21 We have stressed[1] that although the BCPNN 3. Orre R, Lansner A, Bate A, et al. Bayesian neural networks approach has its limitations and is not a substitute for with confidence estimations applied to data mining. Compu-tational Statistics and Data Analysis 2000; 34 (4): 473-93 expert review, it does have a place particularly where 4. Edwards IR, Biriell C. Harmonisation in pharmacovigilance.
large volumes of data are involved. It is reassuring,  Adis International Limited. All rights reserved.
5. Reynolds JEF. 30th ed. Martindale: the extra pharmacopoeia.
10. Meyboom RH, Egberts AC, Edwards IR, et al. Principles of signal detection in pharmacovigilance. Drug Saf 1997; 16 (6): 6. Micromedex vol 105. Micromedex® healthcare series vol.105 [online]. Available from URL: http://mdx.com. [Accessed 2000 11. Yokotsuka M, Aoyama M, Kutoba K. The use of a medical dictionary for regulatory activities terminology (MedRA) in 7. Micromedex vol 100. Micromedex® healthcare series vol.100 prescription-event monitoring in Japan (J-PEM). Int J Med [online]. Available from URL: http://mdx.com. [Accessed 1999 8. Finney DJ. Systematic signalling of adverse reactions to drugs.
9. DuMouchel W. Bayesian data mining in large frequency tables, Correspondence and offprints: Marie Lindquist, Uppsala with an application to the FDA spontaneous reporting system.
Monitoring Centre, Stora Torget 3, 753 20 Uppsala, Sweden.
 Adis International Limited. All rights reserved.

Source: http://www.pvtoolkit.com/toolkit/readers/retrospective_evaluation_data_mining_ADR_signals.pdf