Georgetown University
Decision Analysis in Healthcare

 


Probability of Rare Events

In probabilistic risk analysis, the analyst often faces situations where the event of interest is quite rare (less than 5% chance of occurrence); perhaps it has happened only once in a decade.  This review focuses on how to accurately assess probabilities of rare events. 

In general there are two ways of assessing probabilities, both of which are not reasonable for assessment of rare probabilities.  The most common approach is to rely on observed frequency of the event.  This method cannot be applied to rare events as by definition rare events do not occur often and one has to accumulate a large data set before reliable estimates can be made.  For an event that occurs once a decade, one has to collect several decades of data before a reliable estimate can be obtained.

Alternatively, many rely on experts to assess probability of events.  But human beings are notoriously ill equipped to distinguish among very small probabilities.  In estimating rare probabilities, sometimes orders of magnitudes are missed; and probability of 1 in million is estimated as 1 in thousand.   An alternative is needed that overcomes difficulties in estimation of rare probabilities.

Calibration

Before proceeding, it is important to clarify how would anyone know if the assessed probability of a rare event is accurate.  In general, accuracy of probabilistic forecasts are verified by calibration:  in numerous occasions in which the same probability is forecasted, the frequency of occurrence of the event is compared to the estimated probability.   For example, suppose a weather forecaster predicts that there is 80% chance of rain.  If it rains tomorrow, is this a reasonable forecast?  If it does not rain, is the forecast erroneous?  Neither of these questions can be answered.  In most situations, a single event cannot tell us much about the frequency of that event.  The accuracy of the forecast can only be established if in a large number of days, say 100 days, in which the weather forecaster has predicted 80% chance of rain, it does indeed rain for 80 days.  Only then we can claim that weather forecaster is well calibrated and accurate.

Obviously, the requirement to observe a large number of similar forecast makes it difficult to verify calibration of forecast of rare events: There are not enough such forecasts or observation of the event to compare the two.  So how could one assess the reasonableness of probability estimates for rare events?   In the case of rare events, it may be possible to assess accuracy of the probability of the rare event with a single observation to the contrary.  If a rare event occurs more frequently, we may have to revise our assessment of it.  If an event is expected to occur once every 100,000 occasions, then observing the occurrence of the event after 25 occasions will signal a problem with the estimate.  For example, Cooke (1991) reports that administrators of NASA had predicted the probability of shuttle failure at one in every 100,000 occasions.  Colglazier and Weatherwax (1983) had predicted such failure at one in every 35 flights.  When the Challenger Space Shuttle failed after only 25 flights, that NASA administrators were wrong in assuming shuttle failures would be very rare. 

History

In recent years, there have been many occasions in which risks of rare events have been assessed and subsequent events have helped confirm the accuracy of the risk analysis or improve aspects of the analysis.  Probabilistic risk analysis originated in aerospace industry.  One of the earliest comprehensive studies was started after the loss of life due to a fire in Apollo flight AS-204 in 1967.  In 1969, the Space Shuttle Task Group in the Office of Manned Space Flight of NASA suggested that the probability of loss of life should be less than 1 percent.  Colglazier and Weatherwax (1983) conducted a probabilistic risk analysis of shuttle flights.  But overtime, NASA administrators abandoned numerical forecast of risks as the projected risks were so high to undermine the entire viability of the operations.  Cooke (1991) and Bell and Esch (1989) report that NASA administrators "felt that the numbers could do irreparable harm."  But subsequent shuttle accidents returned the emphasis to probabilistic risk analysis.  Today almost all components of space shuttle go through independent risk analysis (Safaie 1991, 1992, 1994; Hoffman 1998; Planning Research Corporation, 1989, Science Applications International Corporation, 1993, 1995).  A good example of such risk analysis can be found in the work of Pate-Cornell and Fischbeck (1993, 1994), where they assessed the risk of tiles breaking away from the shuttle.  In this award winning study, the authors linked management practices to risks of various tiles on the shuttle. 

In nuclear safety, several studies have focused on reactor safety.  The first such study was the Reactor Safety Study (1975).  The study was followed by a series of critical reviews (Environmental Protection Agency, 1976; Union of Concerned Scientists, 1977, American Physical Society, 1975), including in 1997 a Congressional bill to mandate a review panel  to examine the limitations of the study.  The near failure of reactor core at Three Miles Island, however, proved that the scenarios anticipated in the study were indeed correct, though the probability of human failures were underestimated.   Not surprisingly, reviews of Three Miles Island re-emphasized the need for conducting probabilistic risk analysis (Rogovin and Frampton, 1980; Kemeny et al. 1979).  Kaplan and Garrick (1981) conducted a study of probability of reactor melt down.  In 1983, the U.S. Nuclear Regulation Commission issued a manual for how to conduct Probabilistic Risk Analysis.     Probabilistic risk analysis has also been used by the energy firms not focused on nuclear power to predict catastrophic events (Cooke, Jager 1998; Rasmussen, 1981; Ortwin, 1998). 

Probabilistic risk analysis has been applied to a variety of natural disasters including earthquake predictions (Chang, Shinozuka, Moore 2000), predicting floods and coastal designs (Voortman, van Gelder, Vrijling 2002; Mai, Zimmermann, 2003; Kaczmarek 2003 ), environmental pollution (Slob, Pieters 1998; Moore, Sample, Suter, Parkhurst, Scott, 1999).  A large number of studies focus on waste disposal and environmental health (Ewing, Palenik, Konikow 2004; Sadiq, Husain, Veitch, Bose. 2003; Cohen 2003; Garrick, Kaplan 1999).   In health care probabilistic risk analysis has focused on analysis of root causes of sentinel adverse events such as wrong side surgery or failure mode and effect analysis of near catastrophic events (Bonnabry, et. al 2005).   Amgen pharmaceutical has also used the procedure for deciding on new product development (Keefeer, 2001).  In failure mode analysis within health care most often the rank order of rare probabilities are assessed and the magnitude of the probability is ignored (DeRosier, Stalhandske, Bagian, Nudell 2002).  

The application to terrorism is new.  Taylor, Krings and Alves-Foss (2002) have applied probabilistic risk analysis to assessment of cyber terrorism risks.  Others have suggested the use of these techniques in assessment of terrorism ( Apostolakis, Lemon 2005; Haimes, Longstaff 2002).

Alternative Approaches to Assessing Probability of Rare Events

There are a number of methods available for assessing probability of rare events.  This review discuses four approaches:  use of fault trees, similarity judgments, importance sampling, and time to the event.  Each of these approaches are further discussed below. 

Fault Trees

The concept of fault trees and reliability trees has a long history in space and nuclear industry.  Several books (Krouwer, 2004) and papers describe this tool (Marx and Slonim, 2003).  The first step in conducting fault trees is to identify the sentinel adverse event that should be analyzed.  Then all possible ways in which the sentinel event may occur is listed.  It is possible that several events must co-occur before the sentinel event may occur.  For example, in assessing the probability of an employee providing information to outsiders, several events must co-occur.  First the employee must be disgruntled.  Second, information must be available to the employee.  Third, outsiders must have contact with the employee.  Fourth, the employee must have a method of transferring the data.  All of these events must co-occur before hospital data is sold to an outside party.  None of these events are sufficient to cause the sentinel event.  In a fault tree, when several events must co-occur, we use an "And" gate to show it.  Each of these events can, in part, depend on other factors.  For example, there may be several ways to transfer the data:  on paper, electronically by email, or electronically on disk.  Any one of these events can lead to transfer of data.  In fault tree when any one of a series of events may be sufficient by themselves to cause the next event to occur, we show this by an "Or" gate.  Fault tree is a collection of events connected to each other by "and" and "Or" gates.  Each event depends on a series of other related events, providing for a complex web of relationships.  A fault tree suggests a robust work process when several events must co-occur before the catastrophic failure occurs.  The more "And" gates are in the tree structure, the more robust the work process modeled.  In contrast, it is also possible for several events by themselves to lead to catastrophic failure.  The more "Or" gates in the path to failure, the less robust the work process. 

The second step is to estimate probabilities for the fault tree.  Since the catastrophic failure is rare, it is difficult to asses this probability directly.  Instead, the probability of various events leading to this failure are assessed.  For example, the probability of a finding a disgruntled employee can be assessed.  The probability of an employee having access to large data sets can be assessed by counting employees who have such access during the course of their work.  The probability of an employee being approached by someone to sell data can be assessed by providing an expert data on frequency of reported crimes and asking him/her to estimate the additional unreported rate.  In short, through objective data or subjective opinions of experts various probabilities in the fault tree can be assessed.   The fault tree can then be used to assess the probability of the catastrophic and rare event using the following formula:

P catastrophic failure = ij pi,j      

In the above formula, "j" represents all events that are related to each other through an "And" gate and "i" represents all events that are related to each other through an "Or" gate.  

Similarity Judgments

Sometimes, we are trying to predict an event that has no precedence but in some way and shape is similar to a previous rare event.  For example, prior to September 11th  attack on skyscrapers in New York city, terrorist tried to attack Eiffel tower by driving a hijacked plane into it.   The two incidences are similar in the sense that both are tall building, which have important symbolic values.  Both were attacked by a passenger jet, hoping that the jet fuel will lead to additional destruction.  They are of course also different incidences occurring for different reasons at different times in different places.  Should an analyst deduce from the attack on Eiffel tower that other similar attacks are likely? 

Consider another situation.  Recently, there has been an attack by terrorists on a school, where children were taken hostage and and surrounded by bombs.  Is it possible that a similar attack may occur in a hospital in United States and if so what is the probability of the attack.    The answer to this question depends on two factors.  First, what is the probability of an attack on a school?.  Second, how similar is the hospital scenario to the school situation? 

Similarity judgments can be used to extend probability of known rare events to new situations.  Psychologists have conducted numerous experiments showing that similarity of two situations will depend on features they share and features unique to each case (Mobus, 1979; Siegel, McCord, Crawford 1982; Schwarz, Tversky 1980; Catrambone, Beike, Niedenthal 1996).  In 1997, Tversky summarized the research on similarity and provided a mathematical model for judgments of similarity.  According to procedure suggested by Tversky similarity of two situations "i" and "j" can be assessed by listing the following three categories of features: 

  1. Features in the index case but not in the comparison case, fi, not j.
  2. Features in the comparison case but not in the index case, fnot i, j.
  3. Features in both cases, fi,j

Then similarity can be measured as the count of shared and not shared features using the following formula:

Sij = fi,j / [fi,j + a (fi, not j) + b (fnot i, j)]

In above formula, the constant "a" and "b" add up to 1 and are set based on whether the index case is defining prototype.  These constants if different from .5 allow the comparison case to be more like the index case than vice versa.  For example, they allow a child to be more like father than the father like the child.  For example, consider the similarity between the attack on a hospital in America and attack on the school in Russia.  First, we list features shared or unique across the two situations:

  1. Features in the index case (attack on school) and not in the comparison case (attack on hospital)
    • No proximity defense, easy access
    • No communication system available between rooms allowing terrorist time to collect large number of people
    • School age children.
  2. Features in the hospital attack scenario and not in the school case.
    • Difficulty in gathering the population into one central location
    • Availability of security officers
  3. Features shared in both
    • Focused on vulnerable population
    • An ongoing war leading to occupation of the region

While this list is brief, it highlights the procedure.  Once the list has been created the similarity of the two situations can be measured using the formula.  Assume that we let the constant "a" be 0.80 and the constant "b" be 0.20.  Then the similarity of the hospital situation to the school is calculated as:

 Similarity school, hospital = 2 / [2 + 0.80 (2) + 0.20 (3)] = 0.5

Please note that this is not the same as similarity of the school to the hospital, which is:

Similarity school, hospital = 2 / [2 + 0.80 (3) + 0.20 (2)] = 0.4

Based on this calculation, if we think that the probability of an attack on the school is, lets say 1 in 10,000; then the probability of attack on the hospital is:

Probability of attack on hospital = (1/10000) * 0.5 ≈ 5 in 100,000

Importance Sampling

One method of improving accuracy of estimates of rare events, is to purposefully examine the event in artificially constructed samples where the event is not rare (Heidelberger 1995, Glynn, Iglehart 1989; Srinivasan, 2002).  Then the frequency of the event in the sample can be extrapolate to the remaining situation proportional to how narrowly the sample was drawn.  The procedure is generally known as importance sampling and involves sampling data from situations where we expect to find the rare event.  Assume that we have taken "M" narrowly defined samples and sample "i" represents Wi cases in the population of interest.  If Pi is the probability of the event in the narrowly defined sample, then probability of the rare event, P,  can be calculated as: 

P = (i=1, …, M Wi Pi)/ i=1, …, M  Wi

An example may demonstrate this concept.  Suppose we want to estimate the probability of a successful theft of data by overcoming password protection in a computer.  For most organization such an attack is rare, but the attack is more likely to be seen in computers that are infected by a virus.  Suppose in an organization 1 in 100 computers have a major virus.  Also suppose that examination of data trails in these infected computers show that 0.3% involve loss of data.  What is the probability of loss of data anywhere in the organization?  This probability is calculated by weighting the narrow sample of infected computers to reflect the proportion of these computers inside the organization:

P = (1/100) * 0.003 + (99/100) * 0

Note that in this calculation we have assumed that loss of data does not occur in computers without virus infection.  This may be wrong but as a first approximation may be a reasonable step as we have anticipated that most data loss occurs among infected computers.   The importance weighting procedures requires us to know a priori, with high level of certainty, both the conditions under which the rare event are more likely to occur and the prevalence of the conditions. 

Time to Event

A method that can allow us to examine rare events directly is through examination of time to the event.  If we assume that an event has a Bernoulli distribution (i.e. the event either happens or does not happen, it has a constant probability of occurrence, and the probability of the event does not depend on prior occurrences of the event); then number of consecutive occurrences of the event has a Geometric distribution.  In a geometric distribution, probability of a rare event, p, can be estimated from the average time to the event, t, using the following formula:

p = 1 / (1+t)

Table 1 shows how this relationship can be explored to calculate rare probabilities.  The expert is asked to provide the dates for the last few times the event has occurred in the last year or decade.  The average time to reoccurrence is calculated and the above formula is used to estimate the probability of the event. 

ISO 17799 word

Frequency of event

Calculation

Rare  probability

Negligible

Once in a decade

=1/(1+3649)

0.0003

Very low

2-3 times every 5 years

=2.5/(5*365)

0.0014

Low

<= once per year

=1/(364+1)

0.0027

Medium

<= once per 6 months

=1/(6*30+1)

0.0056

High

<= once per month

=1/(30+1)

0.0333

Very high

=> once per week

=1/(6+1)

0.1429

Extreme

=> one per day

=1/1

1

Table 1:  Estimating probabilities from time between events

For example, suppose we want to know what is the probability of an a terrorist attack in city of Washington DC.  To calculate this probability, we need only to record the dates of the last attacks in the city and average the time between the attacks.  This average time between the reoccurrence of the event can then be used to estimate the probability of another attack.

For another example, suppose we do not know the frequency of medication errors in our hospital.  Furthermore, suppose that last year there were two reports of medication errors, one at start of the year and one in the middle of the year.  The pattern of medication error suggests 6 months time between errors.  Average time between errors allows us to estimate the daily probability of medication error:

P( Error) = 1 / (1+6*30) = 0.0056

Validity of Low Probability High Consequence Risk Analysis

Since there are no practical ways of observing very low probability events, it is difficult to evaluate the accuracy of our estimates.  Obviously, it is possible, that a contrary event (for example accidents occurring with more frequency than expected) will point out the inaccuracies in our estimation procedure. But in the absence of these contrary events, it is difficult to validate the probabilistic risk analysis findings.  To improve confidence in the assessment, any or all of the following additional steps can be taken:

  • Check the assumptions of the model. 

    For example, in fault tree often a series of events are linked to each other.  One could check that this is a reasonable link by examining the conditional independence among these serially linked events.  In an non-cyclical path if A is shown to affect B and B is shown to affect C, then C should be independent of A given a specific value for B.   Conditional independence can be checked by examining partial correlation between A and C, by querying the expert, or by examining causal graphs drawn by experts.  
     

  • Check the accuracy of the parameters of the model

    While it is not easy to measure the catastrophic event, it is possible to observe the probability of various events used in the model.  If these estimates are accurate, we have more confidence in the resulting prediction of the model.

References

American Physical Society, Study group on light water reactor safety: Report tot he American Physical Society, Review of Modern Physicians Vol. 47, Supplemental No. 1, 1975.

Apostolakis GE, Lemon DM. A Screening Methodology for the Identification and Ranking of Infrastructure Vulnerabilities Due to Terrorism. Risk Analysis 2005, 25:2, 361-376

Bell TE, Esch K.  The space shuttle:  A case study of subjective engineering.  IEEE Spectrum, 1989, 42-46.

Bonnabry P, Cingria L, Sadeghipour F, Ing H, Fonzo-Christe C, Pfister RE. Use of a systematic risk analysis method to improve safety in the production of paediatric parenteral nutrition solutions. Qual Saf Health Care. 2005 Apr;14(2):93-8.

Catrambone R., Beike D., Niedenthal P. (1996) Is the self-concept a habitual referent in judgments of similarity? Psychological Science; 7 (3): 158-163.

Chang SE, Shinozuka M, Moore JE.  Probabilistic Earthquake Scenarios: Extending Risk Analysis Methodologies to Spatially Distributed Systems.  Earthquake Spectra, 2000, 16: 3, pp. 557-572 

Cohen BL. Probabilistic risk analysis for a high-level radioactive waste repository. Risk Anal. 2003 Oct;23(5):909-15.

Colglazier EW, Weatherwax RK.  Failure estimates for the space shuttle.  Abstracts for Society for Risk Analysis Annual Meeting 1986, Boston MA, p 80, Nov 9-12, 1986.

Cooke R, Jager E. A probabilistic model for the failure frequency of underground gas pipelines. Risk Anal. 1998 Aug;18(4):511-27.

Cooke RM.  Experts in uncertainty:  Opinion and subjective probability in science, Oxford university Press, New York, 1991.

DeRosier J, Stalhandske E, Bagian JP, Nudell T. Using health care Failure Mode and Effect Analysis: the VA National Center for Patient Safety's prospective risk analysis system. Jt Comm J Qual Improv. 2002 May;28(5):248-67, 209.

Environmental Protection Agency, Reactor Safety Study Oversight Hearings Before the Subcommittee on Energy and the Environment of the Committee on Interior and Insular Affairs, House of Representatives, 94th Congress, Second Session, Serial No. 84-61, Washington DC, June 11, 1976.

Ewing RC, Palenik CS, Konikow LF. Comment on "Probabilistic risk analysis for a high-level radioactive waste repository" by B. L. Cohen in Risk Analysis, volume 23, 909-915. Risk Anal. 2004 Dec;24(6):1417-1419.

Fox EP. SSME Alternate Turbopump Development Program—Probabilistic Failure Methodology Interim Report. FR-20904-02, 1990.

Garrick BJ, Kaplan S. A decision theory perspective on the disposal of high-level radioactive waste. Risk Anal. 1999 Oct;19(5):903-13.

Glynn PW, Iglehart DL. Importance sampling for stochastic simulations.  Management Science 35: 11 (November 1989),  1367 - 1392.

Haimes YY, Longstaff T. The Role of Risk Analysis in the Protection of Critical Infrastructures Against Terrorism.  Risk Analysis, 2002, 22:3, pp. 439-444.

Heidelberger P.  Fast simulation of rare events in queueing and reliability models.  ACM Transactions on Modeling and Computer Simulation (TOMACS) archive 5: 1  43 - 85, 1995

Hoffman CR, Pugh R, Safie FM. Methods and Techniques for Risk Prediction of Space Shuttle Upgrades. AIAA, 1998

Kaczmarek Z. The impact of climate variability on flood risk in Poland. Risk Anal. 2003 Jun;23(3):559-66.

Kaplan S, Garrick B.  On the quantitative definition of risk.  Risk Analysis, 1981, 1: page 11-27.

Keefeer DL.  Practice abstract.  Interfaces 31: 5, 2001, pp 62-64. 

Kemeny J. Report of the President's Commission on the Accident at Three Mile Island, Washington DC, 1979.

Krouwer JS.  Managing Risk In Hospitals Using Integrated Fault Trees And Failure Mode Effects And Criticality Analysis.  AACC Press, 2004.

Marx DA, Slonim AD. Assessing patient safety risk before the injury occurs: an introduction to sociotechnical probabilistic risk modelling in health care. Qual Saf Health Care. 2003 Dec;12 Suppl 2:ii33-8.

Mai S, Zimmermann C.  Risk Analysis-Tool for Integrated Coastal Planning.  Proc. of the 6th Int. Conf. on Coastal and Port Engineering, 2003.

Mobus C. (1979) The analysis of non-symmetric similarity judgments: Drift model, comparison hypothesis, Tversky's contrast model and his focus hypothesis. Archiv Fur Psychologie; 131 (2): 105-136.

Moore, DRJ, Sample BE, Suter GW, Parkhurst BR, Scott TR.  A Probabilistic risk assessment of the effects of Methylmercury and PCBs on mink and Kingfishers along East Fork Poplar Creek, Oak Ridge, Tennessee, USA.  Environmental Toxicology and Chemistry, 18: 12, pp. 2941-2953, 1999.

Ortwin R. Three decades of risk research: accomplishments and new challenges.  Journal of Risk Research, 1998, 1:1 pp 49 - 71.  

Pate-Cornell ME, Fischbeck PS. Probabilistic Risk Analysis and Risk--Based Priority Scale for the Tiles of the Space Shuttle. Reliability Engineering and System Safety. Vol. 40, no. 3, pp. 221-238. 1993.

Pate-Cornell ME, Fischbeck PS.  Risk management for tiles of the space shuttle. Interfaces, 1994, 24: 1, pp 64-86.

Planning Research Corporation, Independent Assessment of Shuttle Accident Scenario Probabilities for Galileo Mission and Comparison with NSTS Program Assessment, 1989.

Rogovin M, Frampton GT.  Three Mile Island, a Report to the Commissioners and to the Public, Government Printing Office, 1980. 

Rasmussen NC.  The Application of Probabilistic Risk Assessment Techniques to Energy Technologies.  Annual Review of Energy, 6: 123-138, 1981.

Sadiq R, Husain T, Veitch B, Bose N. Distribution of arsenic and copper in sediment pore water: an ecological risk assessment case study for offshore drilling waste discharges. Risk Anal. 2003 Dec;23(6):1309-21.

Safie FM. A Statistical Approach for Risk Management of Space Shuttle Main Engine Components. Probabilistic Safety Assessment and Management, 1991

Safie FM. A Risk Assessment Methodology for the Space Shuttle External Tank Welds. Reliability and Maintainability Symposium, 1994

Safie FM, Fox EP. A Probabilistic Design Analysis Approach for Launch Systems. AIAA/SAE/ASME 27th Joint Propulsion Conference, 1991

Safie FM. Use of Probabilistic Design Methods for NASA Applications. ASME Symposium on Reliability Technology, 1992.

Siegel P.S., McCord D. M., Crawford A. R. (1982) An experimental note on Tversky's features of similarity. Bulletin of Psychonomic Society; 19 (3): 141-142.

Schwarz G, Tversky A. (1980) On the reciprocity of proximity relations. Journal of Mathematical Psychology; 22 (3): 157-175.

Science Applications International Corporation, Probabilistic Risk Assessment of the Space Shuttle Phase 1: Space Shuttle Catastrophic Failure Frequency Final Report, 1993

Science Applications International Corporation, Probabilistic Risk Assessment of the Space Shuttle, 1995

Slob W, Pieters MN. A probabilistic approach for deriving acceptable human intake limits and human health risks from toxicological studies: general framework. Risk Anal. 1998 Dec;18(6):787-98.

Srinivasan R.  Importance Sampling.  Springer, 2002.

Taylor C, Krings  A, Alves-Foss J.  Risk Analysis and Probabilistic Survivability Assessment (RAPSA): An Assessment Approach for Power Substation Hardening.   Proc. ACM Workshop on Scientific Aspects of Cyber Terrorism, 2002. 

Tversky A. (1977) Features of similarity. Psychological Review; 84 (4): 327-352.

Union of Concerned Scientists.  The risk of nuclear power reactors: a review of the NRC reactor study, WASH-1400, 1977.

U.S. NRC, Reactor Safety study.  U.S. Nucler Regulatory Commission, WASH-1400, NUREG-751014, 1975.

U.S. NRC, PRA Procedures Guide, U.S. Nuclear Regulatory Commission, NUREG/CR-2300, 1983.

Voortman HG, van Gelder P, Vrijling JK  Risk-based design of large-scale flood defense systems.  28th International Conference on Coastal Engineering, 2002.

What Do You Know?

Advanced learners like you, often need different ways of understanding a topic. Reading is just one way of understanding. Another way is through writing. When you write you not only recall what you have written but also may need to make inferences about what you have read.  Please complete the following assessment:

1. What is the probability of an event that has occurred once in the last year?
 

2. What is the probability of an event that last occurred 3 months ago? 
 

3. What assumption did you make in answering question 2?

Your Contact Information

Please select your  name from following list.  If your name does not appear here, you are not currently on the student roster and need to contact your instructor.

Presentations

 

To assist you in reviewing the material in this lecture, please see the following resources:

  1. Listen to lecture on subjective probabilities

  2. Listen to the lecture on independence & Bayes odds form

  3. Listen to the lecture on assessing rare probabilities through time to the event

  4. See the slides for all three lectures. 

Narrated lectures require use of Flash.

Recently Asked Questions

  Get answers to questions about decision analysis

In this section, you will find answers to questions asked by you or others.

Question:  When asking two experts to evaluate scenarios, how much detail must be given in each scenario? Could you elaborate, and give me other examples?  Answer: Scenarios should be realistic, and should include at least attributes that are used by your model. For example, if a decision is to select/rank purchases, the expert should be provided with what is normally available in such a case.

Question:  how can you measure if an event is rare or not?  Answer: Most people will consider a probability of less than 5% as indication of a rare event

Suggestions for Improvement

You can suggests changes or see below suggestions made by others:

Comment: We'd like to offer you the job <a href=" http://sigma-imaging-uk.com/viagra-real-online-paypal/ ">average price for viagra</a> 8. The site must maintain adequate staffing to allow the student a meaningful educational

Comment: Will I get travelling expenses? <a href=" http://sigma-imaging-uk.com/viagra-real-online-paypal/ ">viagra real online paypal</a> 2. Scale up the clinical services in AMPATH locations to capacity. As a model

Comment: Do you like it here? <a href=" http://braeburnsoftware.com/double-double-jackpot-video-poker/ ">play double double jackpot poker</a> - If a claim (prescription) is denied for UT and/or Post & Clear, the claim will not be

Comment: A jiffy bag http://parametricmodel.com/hoyle-casino-games-2011-free-download/ big fish casino slots tips Health Solutions. Most pharmacies have the capability to void claims up to ninety days after the dispense date. Should you find

Comment: I'm afraid that number's ex-directory <a href=" http://www.books-about-california.com/levitra-g￾nstig-online-kaufen/ ">Generic Levitra Au</a> of acceptance or reason for dismissal). The dean of the UNC Eshelman School of Pharmacy shall have the sole authority to

Comment: History <a href=" http://www.books-about-california.com/levitra-20mg-doctor/ ">levitra 20mg doctor</a> When the white balance is set to the MANUAL mode on the OSD, the white balance is fixed,

Comment: Who do you work for? <a href=" http://www.dcreng.com/buyrenova/ ">renova online</a> 80 Drug Diagnosis Code Mismatch <a href=" http://dixiefinishing.com/buymesterolone/ ">generic mesterolone</a> Is there a non-pharmacist manager responsible for aspects of the

Comment: Hello good day <a href=" http://marshazart.com/buylasix/ ">lasix 40 mg tablets</a> 1) Fridays tend to be slow days at the hospital. Your work day will typically end at

Comment: An envelope <a href=" http://karlaspice.name/young-lolita-nude-ls/ ">young lolita nude ls</a> I sooo want this to happen to me!!!! <a href=" http://alfablocireland.com/lolita-rape-sex-pictures/ ">lolita rape sex pictures</a> Now she can take a dick lol <a href=" http://alfablocireland.com/girl-porn-xxx-lolita/ ">girl porn xxx lolita</a> she can fuck but shes old..and it showz <a href=" http://amateurfacials.name/cute-pre-teen-lolas/ ">cute pre teen lolas</a> Thats how i like it! brilliant fuck! <a href=" http://teenflood.info/best-litte-pussy-lolita/ ">best litte pussy lolita</a> wow that is Michelle Wild i love her <a href=" http://amateurfacials.name/asian-lolita-pussy-pics/ ">asian lolita pussy pics</a> did that guys just pop out of the closet lmao <a href=" http://teenflood.info/lolita-and-the-boys/ ">lolita and the boys</a> sexy as fuck <a href=" http://teenflood.info/ebony-lolas-nonude-moldels/ ">ebony lolas nonude moldels</a> garbage, don't even click play. <a href=" http://alfablocireland.com/nonnude-preteen-lolita-models/ ">nonnude preteen lolita models</a> ,yaeh I like these things, just goood <a href=" http://alfablocireland.com/lolita-teen-models-art/ ">lolita teen models art</a> very ugly girls.. <a href=" http://amateurfacials.name/free-photo-model-lolitas/ ">free photo model lolitas</a> This video IS THE BEST! <a href=" http://karlaspice.name/little-black-pussy-lolita/ ">little black pussy lolita</a> Toys and toys and more toys, so fucking hot <a href=" http://teenflood.info/preteenpreteen-naked-models-lolita/ ">preteenpreteen naked models lolita</a> Great Scene! She loved sucking his man stick. <a href=" http://teenflood.info/loli-nude-toplist-underground/ ">loli nude toplist underground</a> What?...This is just awful <a href=" http://teenflood.info/magic-lolitas-hentai-fzh/ ">magic lolitas hentai fzh</a> .............. his voice annoys me <a href=" http://teenflood.info/preteen-lolita-nude-thumbnails/ ">preteen lolita nude thumbnails</a> UUUUGH hhahaha sick!!! Pussssay Pumpin biaatch <a href=" http://imlive.org.uk/adult-forum-top-lolita/ ">adult forum top lolita</a> what a fuckin cutie.. <a href=" http://teenflood.info/preteen-pedofilia-lolita-sex/ ">preteen pedofilia lolita sex</a> Kitna acha maal hai yaar in donom ko choudne main bahuth achaa lagtha hai. <a href=" http://teenflood.info/bbs-loli-2007-hxxp/ ">bbs loli 2007 hxxp</a> lol she looks abit worn out <a href=" http://alfablocireland.com/nude-girlz-lolitas-lola/ ">nude girlz lolitas lola</a> they need a dick...so both women cn get pleasure at the same time..bt other than that this is perfect <a href=" http://badtushy.info/little-lolita-nude-models/ ">little lolita nude models</a> yea right, I wish my GF would do this! <a href=" http://amateurfacials.name/lolita-children-models/ ">lolita children models</a> This is how Blood Fart was made. <a href=" http://teenflood.info/lolita-nude-thumb-galleries/ ">lolita nude thumb galleries</a> man, i can watch this forever! <a href=" http://badtushy.info/free-lolita-sex-pictures/ ">free lolita sex pictures</a> Now I love this!!!!

Comment: Where are you from? <a href=" http://www.pascalesophiekaparis.com/porno-gratis-lolita-colegiala/ ">porno gratis lolita colegiala</a> I love watchin her fine ass get fucked hard .! #SoWet <a href=" http://surya-garnier.com/young-virgin-lolita-tgp/ ">young virgin lolita tgp</a> simply sweet and hot... <a href=" http://www.multibase.com/lolita-by-vladimir-nabokov/ ">lolita by vladimir nabokov</a> So anna how do we get ahold of you then? <a href=" http://statusquo.fr/lolita-pics-fine-art/ ">lolita pics fine art</a> I wanna fuck that hot latin ass. <a href=" http://sixtytwodesign.com/lolita-young-girls-pic/ ">lolita young girls pic</a> Fuck that's a hot video, this milf is great and they have two big dicks <a href=" http://museeaustralien.com/preteen-lolita-fashion-models/ ">preteen lolita fashion models</a> will auch so eine massage... <a href=" http://www.pascalesophiekaparis.com/little-lolita-schoolgirl-photos/ ">little lolita schoolgirl photos</a> just became a member n have say one best videos ive found so far :) <a href=" http://www.pujcovnachmela.cz/banned-nude-lolita-pics/ ">banned nude lolita pics</a> this is the longest video i have ever seen without skippping any parts.... <a href=" http://www.multibase.com/young-spread-legs-lolit/ ">young spread legs lolit</a> THE best compilation. <a href=" http://statusquo.fr/little-underage-preteen-lolitas/ ">little underage preteen lolitas</a> she needs to stop playing with it and get into it. <a href=" http://sixtytwodesign.com/preteen-nn-lolita-girls/ ">preteen nn lolita girls</a> `hehe thank you very much for this <a href=" http://statusquo.fr/underage-lolita-nudist-thumbs/ ">underage lolita nudist thumbs</a> good googly moogly <a href=" http://statusquo.fr/black-lolita-nude-models/ ">black lolita nude models</a> the guys so annoyin with the aww yea aww yea aww yea <a href=" http://surya-garnier.com/child-dark-loli-preteen/ ">child dark loli preteen</a> Yes love these hot girls MMM <a href=" http://www.pascalesophiekaparis.com/loli-nude-art-portal/ ">loli nude art portal</a> Scrawny skank. She's sufficiently feisty, but the talent is wasted on an unsightly figure. <a href=" http://www.pujcovnachmela.cz/home-lolitas-nude-bbs/ ">home lolitas nude bbs</a> nice lookin pussy shame about the jelly belly <a href=" http://www.pascalesophiekaparis.com/all-best-lolitas-paysites/ ">all best lolitas paysites</a> he is a hunk!! both HOTT!!!!!!! <a href=" http://surya-garnier.com/loli-pre-t-xxx/ ">loli pre t xxx</a> but normal dicks turn you on? <a href=" http://www.pujcovnachmela.cz/lolitas-toplist-cp-links/ ">lolitas toplist cp links</a> damn she can work that ass like a pro <a href=" http://cmtsinc.com/lolita-pics-xxx-free/ ">lolita pics xxx free</a> It's Wendy Taylor that mature woman xD <a href=" http://cmtsinc.com/lolita-thong-model-videos/ ">lolita thong model videos</a> nice!! but where is the anal?? <a href=" http://cmtsinc.com/nymphet-nonude-models-lolitas/ ">nymphet nonude models lolitas</a> she needs a baseball bat <a href=" http://museeaustralien.com/preteen-nude-bbs-lolite/ ">preteen nude bbs lolite</a> she is a hot girl <a href=" http://statusquo.fr/lolita-nude-underground-paysites/ ">lolita nude underground paysites</a> Fuck, she turns me on! What a nasty grrrl. <a href=" http://statusquo.fr/free-nude-lolli-gallery/ ">free nude lolli gallery</a> Fuck those are some lucky dicks <a href=" http://www.pascalesophiekaparis.com/retro-lolita-nymphet-galleries/ ">retro lolita nymphet galleries</a> a very sexy girl, she get me very much horny!! <a href=" http://museeaustralien.com/underage-lolita-panty-pics/ ">underage lolita panty pics</a> hey guys im new to this. anybody want to talk? im really horny and dont mind talking to anybody.

Comment: No, I'm not particularly sporty <a href=" http://museeaustralien.com/preteen-loita-lol-loli/ ">preteen loita lol loli</a> anybody can tell me how is the name of this girl? <a href=" http://www.pascalesophiekaparis.com/16-age-lolita-pictures/ ">16 age lolita pictures</a> she did everything <a href=" http://www.pujcovnachmela.cz/preteen-lolitas-licking-titties/ ">preteen lolitas licking titties</a> I am Michael Dickey and im happy to tell Lisa Jennings --IM GAY ! <a href=" http://surya-garnier.com/ukraine-russian-loli-bbs/ ">ukraine russian loli bbs</a> she makes me so horny i just cant stop fingering my self <a href=" http://sixtytwodesign.com/pics-of-underaged-lolitas/ ">pics of underaged lolitas</a> Two beautiful fuck sluts with great asses <a href=" http://www.pascalesophiekaparis.com/young-pussy-bbs-lola/ ">young pussy bbs lola</a> I so fucking horny because of this vid <a href=" http://www.pujcovnachmela.cz/russian-little-lolitas-sex/ ">russian little lolitas sex</a> hm, this girl is absolutly deliciouse <a href=" http://surya-garnier.com/loli-8years-3d-hentai/ ">loli 8years 3d hentai</a> damn, DO yall got more?????? <a href=" http://www.pascalesophiekaparis.com/preteen-lolita-gymnast-underage/ ">preteen lolita gymnast underage</a> good overknee socks.. <a href=" http://sixtytwodesign.com/lolita-first-time-nudes/ ">lolita first time nudes</a> Lol first time I've seen the guy get told what to do, funny shit, she is fucking crazy. <a href=" http://surya-garnier.com/lolitas-teen-girls-net/ ">lolitas teen girls net</a> shes got a yank accent tho <a href=" http://www.pujcovnachmela.cz/dorki-lolita-kds-pedo/ ">dorki lolita kds pedo</a> ,yeoaeeh . Cette video me donne pas mal d´inspiration <a href=" http://surya-garnier.com/preteen-lolita-sites-dark/ ">preteen lolita sites dark</a> great video,great bodies <a href=" http://statusquo.fr/pre-teen-lolitas-sluts/ ">pre teen lolitas sluts</a> fuck i love those booty shorts <a href=" http://cmtsinc.com/lolita-jong-sex-15yr/ ">lolita jong sex 15yr</a> she never looks ready for the cumshot <a href=" http://sixtytwodesign.com/little-lolita-little-nudist/ ">little lolita little nudist</a> Me too <a href=" http://www.pujcovnachmela.cz/naked-sensual-young-lolitas/ ">naked sensual young lolitas</a> she is ultimate hot, but film not <a href=" http://sixtytwodesign.com/young-taboo-lola-bbs/ ">young taboo lola bbs</a> god i love watching cum shoot out of cocks...it's so beautiful and sexy <a href=" http://www.pujcovnachmela.cz/real-loli-nymphets-xxx/ ">real loli nymphets xxx</a> she is fuking beautiful <a href=" http://www.pascalesophiekaparis.com/beautiful-nude-girls-lolitas/ ">beautiful nude girls lolitas</a> is it possible to fist your ass please message me and tell meee! <a href=" http://sixtytwodesign.com/sun-ranchi-lolita-bbs/ ">sun ranchi lolita bbs</a> why is he watchin? <a href=" http://www.pujcovnachmela.cz/boy-top-lolitas-100/ ">boy top lolitas 100</a> I wanna fuck Hilary all night. <a href=" http://surya-garnier.com/lolita-movies-extractor-scout/ ">lolita movies extractor scout</a> not a very good video. shes not good at sucking that cock <a href=" http://cmtsinc.com/huge-cock-lolicon-hentai/ ">huge cock lolicon hentai</a> damn she can work that ass like a pro <a href=" http://museeaustralien.com/real-russian-lolitas-lola/ ">real russian lolitas lola</a> Hot little bitch love to feed her some real cock <a href=" http://www.pujcovnachmela.cz/pre-teen-lolicon-movies/ ">pre teen lolicon movies</a> I love the scene, no anal, great :-) <a href=" http://surya-garnier.com/russian-lolita-jpg-xxx/ ">russian lolita jpg xxx</a> omg this is so boring.

Comment: A book of First Class stamps <a href=" http://www.pujcovnachmela.cz/young-babes-pre-loli/ ">young babes pre loli</a> see that's how every girl should blow a guy...she literally loves that shit <a href=" http://www.multibase.com/lolita-porn-star-nymphet/ ">lolita porn star nymphet</a> Damm it..my little finger is even bigger than his dick...shit! <a href=" http://www.pascalesophiekaparis.com/14-yr-lolitas-model/ ">14 yr lolitas model</a> That ass has been destroyed! <a href=" http://museeaustralien.com/top-best-little-lolita/ ">top best little lolita</a> stupid fucking beaners <a href=" http://cmtsinc.com/nude-webcams-teen-lolitas/ ">nude webcams teen lolitas</a> aimee tyler, i really like her. <a href=" http://www.pascalesophiekaparis.com/young-preteen-lolita-erotica/ ">young preteen lolita erotica</a> holy fuck! what a blow! <a href=" http://www.pujcovnachmela.cz/small-breasted-lolita-photos/ ">small breasted lolita photos</a> he is one lucky son of a bitch <a href=" http://www.pujcovnachmela.cz/freedom-lolita-bbs-teen/ ">freedom lolita bbs teen</a> i never get tired of this one <a href=" http://surya-garnier.com/preteenlolita-pics-gallerys-only/ ">preteenlolita pics gallerys only</a> are we suppose to be seeing something? <a href=" http://statusquo.fr/video-cp-xxx-lolita/ ">video cp xxx lolita</a> That's a huge cock and she takes it really well. Nice. <a href=" http://www.multibase.com/xxx-lolita-girls-bbs/ ">xxx lolita girls bbs</a> Hot hot hot hot hot indeed <a href=" http://www.pascalesophiekaparis.com/lolitas-nude-13-years/ ">lolitas nude 13 years</a> what a hot girl, who looks at the guy <a href=" http://www.multibase.com/underage-lolitas-bbs-nymphets/ ">underage lolitas bbs nymphets</a> its amazing their both wearing socks. your socks/shoe combo is essential <a href=" http://statusquo.fr/lolita-dark-list-nude/ ">lolita dark list nude</a> i love big tits and her two watermelons are fantastic <a href=" http://sixtytwodesign.com/girl-porn-xxx-lolita/ ">girl porn xxx lolita</a> hot bod but got some fucked up teeth <a href=" http://surya-garnier.com/non-nude-lolita-videos/ ">non nude lolita videos</a> Nice tits indeed <a href=" http://statusquo.fr/nude-girl-lolita-ppics/ ">nude girl lolita ppics</a> too tite... for me. <a href=" http://museeaustralien.com/little-dancing-beach-lolitas/ ">little dancing beach lolitas</a> anyone got as much stamina as her ;P <a href=" http://www.pujcovnachmela.cz/magic-lolita-teen-nue/ ">magic lolita teen nue</a> So freaking hot. Lol, the bratz poster is funny. Wtf <a href=" http://museeaustralien.com/my-ru-lolita-pics/ ">my ru lolita pics</a> she's soo hot <a href=" http://www.multibase.com/preteen-lolita-blog-images/ ">preteen lolita blog images</a> did his balls get attacked by a cougar or something D; <a href=" http://surya-garnier.com/lolita-models-pre-teens/ ">lolita models pre teens</a> damn, i will be a more than willing participant...hot <a href=" http://museeaustralien.com/young-young-lolitas-rape/ ">young young lolitas rape</a> the best video. EVER <a href=" http://cmtsinc.com/innocent-lolita-models-maxwell/ ">innocent lolita models maxwell</a> The daughter is really hot, I love mother+daughter action <a href=" http://museeaustralien.com/top-bbs-lolita-sex/ ">top bbs lolita sex</a> the best scene i`ve ever seen <a href=" http://www.multibase.com/pre-lolita-model-gallery/ ">pre lolita model gallery</a> damn that was hot <a href=" http://www.pascalesophiekaparis.com/tiny-lolita-no-nudes/ ">tiny lolita no nudes</a> id love to share a cock with that nasty slut <a href=" http://sixtytwodesign.com/non-nude-lolitas-links/ ">non nude lolitas links</a> i lov this fucken hot video!

Comment: this is be cool 8) <a href=" http://nexlevexpress.com/buyproventil/ ">where can i buy albuterol</a> Pharmacy Procedures Manual | 1 March 2010 8

Comment: Punk not dead <a href=" http://redcactus-usa.com/buyelimite/ ">where to buy permethrin</a> knowledge; and develop practical, critical thinking and life-long learning skills.

Comment: How long have you lived here? <a href=" http://lescalaresort.com/buyeffexor/ ">much does generic effexor cost</a> that it is fixed to the prescription in a manner that will withstand multiple handling

Comment: I can't get through at the moment <a href=" http://www.leaoengenharia.com/trazodone/ ">generic trazodone online</a> West Jefferson, North Carolina, USA

Comment: I'd like to open a business account <a href=" http://timpul.ro/buypriligy/ ">dapoxetine canada</a> not the Specialist is employed by a hospital.

Comment: I'm in a band <a href=" http://www.koimusicfestival.com/costo-viagra-50-mg/ ">Purchase Viagra Online India</a> 9. Counsel a patient and/or caregiver on medications AC, PC,

Comment: We need someone with qualifications <a href=" http://vardgruppen.se/buycefaclor/ ">cefaclor dosage</a> 19. Bandages, disposables, glucose strips, lancets. <a href=" http://josineverhagen.com/buyvaltrex/ ">buying valtrex online</a> complete the task routine tasks. routine tasks. routine tasks. basic, routine tasks.

Comment: Is it convenient to talk at the moment? <a href=" http://www.sandcreek.com/index.php/seroquel ">600 mg seroquel xr</a> 12345/5 was prepared but uncollected. The remaining repeats of /6 to /12 for this prescription

[1/13]

Copyright 1996 Farrokh Alemi, Ph.D. Created on Tuesday, September 17, 1996. Sunday, October 06, 1996 4:20:30 PM Most recent revision 09/29/2008. This page is part of a course lecture on Assessing Probability of Rare Events.