Rapid Analysis

36 people have rated this page an average of 3.11 out of five stars. Rate this page.   See their suggestions and questions.   Lecture on Rapid Analysis Speedup preparation Speedup data collection Speedup data analysis Speedup presentation An example Discussion What do you know? References More Biweekly project Presentations Recently asked questions Suggested changes   Recently asked:  For an analyst that is unfamiliar with the format of these reports, is there a good source to see some "best practice" reports (as far as structure, presentation, clarity)?  See answer to this and other questions.

Analysis takes time and reflection.  People must be lined up and their views sought.  Ideas need to be sorted through.  Data needs to be collected, stored and retrieved, examined and displayed.  Completing the analysis, writing the report and presenting the findings takes time and it should.  Often at the end of this grueling effort, analysis identifies need for further inquiry and therefore it creates more delays for decision makers.  Policymakers and managers may find the analysis taking too long.  They may have to decide without the full benefit of the analysis and many do. 

This chapter focuses on how analysis could be made faster without sacrificing its quality.   Clearly, doing a thoughtful analysis takes time.  There is no point to hurry and produce a sub-optimal analysis.  But are there ways to complete analysis faster and yet maintain the quality of the work?  To understand what we can do to speed things up, one needs to understand what takes time.  One could imagine analysis consisting of distinct phases: 

1. Preparation
   • Arrange for contracts and mandate to start
   • Coordinate kick off meeting to clarify purpose & scope of the analysis
   • Find relevant experts and decision makers
   • Design study instruments and survey forms
2. Data collection
   • Collect observations
   • Collect experts opinions
   • Store data
3. Analyze data
   • Retrieve data
   • Clean the data (classify data; check distribution and range of data, edit data)
   • Examine accuracy of data (Check for errors in logic, check for errors in transfer of data)
   • Examine if experts were in consensus
   • Calculate expected values or model scores
   • Calculate the correspondence between model and experts' judgments
4. Presentation
   • Distribute draft report
   • Prepare presentation
   • Get input from audience before meeting
   • Present results at meeting

Each of these phases can be speeded up.  Here is how this can be done. 

Speedup Preparation

Thorough preparation can lead to significant time savings in conduct of an analysis.  This section lists specific recommendations regarding what should be done to be better prepared.

1. Draft the final report at the start

One of the simplest steps an analyst can take to reduce time from start of project (signing the contract) to the end of the project is to do a more thorough planning.  In particular, it would be helpful to draft the final report (the introduction, the methods section, the results section), with all related tables and appendices at start of the project (Alemi, Moore, Headrick, Neuhauser, Hekelman, Kizys, 1998).   Obviously the data will not be available to fill the report but one could put in the best guess for the data.  This exercise speeds up analysis in several ways.  First, it communicates precisely to decision makers what the final results will look like.  It reduces confusion and saves the time spent on clarifying the procedures of the analysis.  Second, it clarifies to the analyst what data are needed and identifies the sources of these data.  Third, it clarifies what procedures should be followed to produce the Tables and Figures in the report.  Obviously the data and the final report will be different but the exercise of putting the report together at the beginning of the meeting goes a long way in making sure that only relevant data are collected and time is not wasted on diversions.

A good example of drafting the report before the data is available is the process of generating automatic content on the web.  The text of the report is prepared ahead of the data collection and portions of the report that depend on specific data are left as a variable to be read from a database.  When the data are available, the final report is generated automatically. 

2.  Avoid group kick-off meetings

Another step that can speedup preparations is to meet individually with decision makers, even before the full kick-off meeting.  Individual meetings are easier to arrange and require less coordination.  Furthermore, as we discussed in the Chapter on Modeling Group's decisions, individual meetings facilitate later larger face-to-face meetings. 

3.  Get access to the right experts and through them to right data

A third step is to search for external experts that understand the situation clearly and do not require additional time to orient themselves.  On any topic, there are numerous experts available.  Finding the right expert is difficult but important in saving time and gaining access to resources that only the expert has access to.  Automated method of finding experts in a particular topic are widely available.  One such tool is the Medline database.  One could search a topic in Medline and find the authors who have published in the area.  Most articles include contact information for the author.  In this fashion, one could quickly put together a list of experts in a topic -- no matter how minutely it is defined.  For example, suppose the analyst needs to examine merger between two hospitals. 

Figure 1:  Three Steps to Identifying Relevant Subject Matter Experts
(1) Go to PubMed, (2) Search the literature and (3) Select key authors

In addition to searching the Medline databases, it maybe useful to search CRISP, a database of National Institute of Health funded projects (see http://crisp.cit.nih.gov/).  CRISP is useful in identifying researchers who are currently collecting data.  Many have thought through the issue and may have preliminary data that could be useful in the analysis. 

Finally, we also recommend searching Google's Scholar database for names of people who might have special expertise or knowledge of the issue being modeled.  Google scholar is at http://scholar.google.com

Once a preliminary list has been identified, then the analyst contacts members of the list and asks them if they are aware of others who are doing research in this area, who might have access to specific databases, or who might be able to provide valid opinions on estimates needed in the analysis.  The important advice is to use automated databases to search wide for one or two people who best fit the planned analysis.  Choosing the right person can significantly improve access to various pieces of information and in the process reduce the time it takes to complete the analysis.

Speedup Data Collection
(This section is taken verbatim from Alemi, Moore, Headrick, Neuhauser, Hekelman, Kizys, 1998)

Often the data needed for the analysis is not available and must be collected or deduced from data that is available.  There are at least five steps in which the analyst can reduce the data collection time.

4. Collect only the needed data

Too often people collect data that they do not need and will not analyze. This is absurd but many do it, some because they feel that certain information (e.g. demographics of the patients) must always be collected, others because they do not have a clear idea what they plan to do with the data.  Every piece of datum, no matter how small adds to the length of a survey and to the difficulty of administering it and in the end adds to the time it takes to conduct the analysis.   We have already mentioned how preparing the final report ahead of time reduces the amount of data collected, as many pieces of data that do not make their way into the report are dropped from the data collection plans. 

5. Reduce the data collected by sampling

Many analysts collect too much data. This creates a large burden for administration of the surveys. The larger the number of people surveyed, typically the longer the time for completion of the analysis. Instead of focusing on large-scale surveys, it is better to focus on representative surveys. A sample of patients is chosen that seem to represent the population affect by the new process. These patients complete the survey and based on their responses inferences are made concerning the whole population.  Only when sensitivity analysis shows that the datum plays an important role in the conclusions of the analysis, additional data are collected.  One way to make sampling more efficient is to devise rules for expanding the sample. A small sample is drawn. If it leads to clear unequivocal conclusions, then no more data is collected. If the results are ambiguous, then a larger sample is drawn. Thus, for example, one may agree to sample 20 representative patients about their satisfaction with the new process. If less than 5% are dissatisfied, then no more samples are drawn. If between more than 5% of the respondents are dissatisfied; then, a larger sample of 50 patients is drawn. These methods of two-stage sampling save the number of patients that need to be contacted and thus reduce the time it takes to collect the information.  For a tutorial on adaptive statistical sampling see Bauer and Brannath (2004 ).  For additional reading in this area see Posch, Bauer and Brannath (2003) or Schafer and Muller (2004).

6. Rely on subjective but numerical data

There are at least two sources for data. The first relies on your observation of the process and is called "objective." The second relies on the observations made by others and is referred to as "subjective." Note that by subjective data we do not mean the likes and dislikes of a person, which is after all idiosyncratic and unreliable. By subjective data, we mean relying on observations of others. Thus, a nurse saying that patients' satisfaction has improved is based on the nurse's observation of the frequency of the patients' complaints not on his or her likes and dislikes.  When under time and resource pressures, subjective opinions maybe a reliable source of data that could replace the seemingly more "objective" observations made by the analyst.  An example of how subjective and objective data can be combined to save time is presented later in this chapter.

7. Use subjective data for developing models & objective data for validating the model's accuracy

If experts specify the parameters of a model (e.g. the utility or probabilities in a Multi-attribute value model), then there is no need to put aside data for parameter estimation.  So the need for objective data is reduced, not by just a little but radically. For example, severity indices can be constructed from subjective opinions or from analysis of objective data. Severity indices constructed from subjective opinions can be subsequently tested against objective data.  When doing so, less data are needed because the subjective index has one degree of freedom while the objective multivariate approach has many.  The number of degrees of freedom in a multivariate analysis is the total number of variables minus one. For each degree of freedom, one generally needs to include 10 times as many data items. Thus, if one has a 200 variable model, then there is 199 degrees of freedom and one needs approximately a database of 1990 cases.  When experts specify the scoring of the 200 variable model, the scoring system maps all of the variables into a single score. As a consequence, the degrees of freedom drops and the need for data is reduced. Thus, the 200 variable model, which previously required 1990 cases, can now be analyzed with 50 cases.

8. Plan for rapid data collection

One method of reducing data collection time is to put in place a number of plans for rapid response to specific questions that may be posed by the team. Thus, the analyst approaches employees close to the process and alerts them that the team plans to ask them a few questions. The exact nature of the question is not clear but the procedure used to send the question to them and collect the question is explained and perhaps even practiced. The individuals are put on notice about the need to respond quickly.  When the need for data becomes clear, the analyst broadcasts the question, usually through a telephone message, and within a few hours collects the response. 

9. Use technology for data collection

Computers can now automatically call patients, find them in the community, ask them your questions, analyze the responses and fax the results to you. In one study, Alemi and Stephens (1992) asked a secretary and a computer to compete to contact "hard to reach" persons and ask them a few questions. One the average the secretary was able to do the task in 41 hours while the computer accomplished the same task in 9 hours. Technology can help overcome the difficulty of finding people. When you use technology to collect information from people, there is one added benefit. People are more likely to tell the truth to a machine than to a person. In surveys of drug use, homosexuality, and suicide risks, patients were more likely to report their activities to a machine than to a clinician, even though they were aware that the clinician will subsequently review the computer summary.  Another advantage of collecting data through computer interviews is that it is immediately available and no time needs to be put into putting the data into the computer after its collection.  A number of reviews of effectiveness of various technologies for data collection are available through Medline.  Two articles to start with are by Shapiro et al. (2004) and Newman et al. , (2002).   

Speedup Data Analysis

When data are available, several steps can be taken to make the analysis go faster. 

10.  Clean the data & generate reports automatically

To speedup analysis, the analyst puts together procedures for cleaning the data even before the data is available.  At the most simple level, the analyst prepares reports of distribution and range of each variable.  Such reports can then be examined to see if there are unusual entries.  A computer program can then be prepared that will run various tests on the data to make sure the responses are within range (e.g., no one with negative age) and not conflicting with each other (e.g., no pregnant males).  The computer can examine patterns of missing information and their reasons (i.e. not appropriate, data not available, data available but refusing to provide, etc.). The computer can examine patterns of data entry in previous cases to see if there is an unusual deviation from the pattern. This is typically done by calculating the mean of data items entered in previous cases and testing if the current data item is more than 3 standard deviations away from the mean. To assure integrity and accuracy of data, the computer can select a random number of cases for re-entry.  The point is that procedures for cleaning the data can be automated as far as possible so that when data is available the analyst can rapidly proceed.

Another alternative that has been made possible because of growth of web services is to allow reports to be generated from data automatically.  First, the analyst drafts the report with all variables in the report linked to a database.  Then the analyst prepares a data collection procedure which populates the database.  Third, the computer cleans the data and generates the report.  Figure 2 shows these steps for a site Alemi maintains on personal improvement.  Clients who complete their personal improvement report their success and failure on the web.  The data are collated by the computer, clean and stored in a web database.  A report is automatically generated from the data on the web so that current and future clients can see the success rate of clients engaged in the personal improvement effort.  The report is available instantaneously after the data is collected. 

Figure 2:  Report is Generated Automatically as Data Becomes Available

11.  Analyze emerging patterns before all data are available

Many readers are familiar with exit polling to predict results of elections.  Same procedures can be used to anticipate data findings before complete data set is available.  One very useful tool is to predict probability of an event from time it gets for the event to reoccur.  In this fashion, early estimates of the datum can be made from even two re-occurrence of the event.  If the event is rare, it takes a long time for it to reoccur.  If not, it will reoccur in a short interval.  BY examining the interval between the event, the probability of the event can be estimated. 

12.  Use software

One way to conduct sensitivity analysis quickly is to use software designed to conduct decision analysis.  Many of the existing software automatically conduct single and two variable sensitivity analysis.  Reviews of software for decision analysis are available online. 

Speedup Presentation

An analysis is not done until the sponsor examines the results.  To speedup the presentation several steps can be taken.

13.  Set up presentation meeting months in advance

Many decision makers are busy.  To arrange for their time, make an appointment many months in advance.  If a presentation date is set, it would create pressure to produce the findings on time.-

14.  Present to each decision maker privately prior to the meeting

Even though a joint meeting is coming up, it is important to present to each decision maker separately and get their input so that the analysis can be revised in time for the meeting. 

Example

One of the most complicated concepts regarding speeding up decision analysis is the process of relying on subjective data to speedup data collection.  To illustrate this point, we provide an example of how Gustafson combined experts opinions to make sense of existing objective data.  Gustafson was asked to predict the impact of national health insurance programs on five low-income populations.  The implementation of national health insurance (NHI) would have profound effects on people who currently rely on federal programs administered by the Bureau of Community Health Services (BCHS) in the Department of Health and Human Services. These groups include migrant farm workers, Native Americans, mothers and children needing preventive or special care, residents of medically underserved areas, people desiring family planning services, and those lacking adequate health insurance coverage.  The unique circumstances of migrant workers and Native Americans necessitated the creation of special services responsive to their needs. If NHI results in termination of such assistance, the result could be a financial burden on current beneficiaries—and a reduction in their access to care.

To accurately appraise changes that would occur under NHI, Gustafson had to find the utilization patterns of BCHS families and the unit cost of services consumed.  He also had to ascertain the eligibility requirements and cost-sharing provisions of the NHI proposal. Finally, he had to determine which currently used services were included in the various NHI benefit packages. Primarily because of time constraints, he could not collect data and was confined to using the best available information.

Utilization patterns were created for individual family members (for each BCHS program) and stratified into appropriate age groups. Utilization patterns for individuals were then aggregated to achieve a family utilization description. The computer also determined extent of coverage under different NHI schemes and compared this figure to present costs. Before directing the computer to simulate some sample user families, Gustafson needed information on family characteristics, particularly the population’s socioeconomic and demographic status. These characteristics included such factors as number and size of families, age and sex of family members, employment status, whether employment was longer than 400 hours per year per employer, size of employing firm, income levels, and Medicaid status. This information was primarily collected from U.S. census data on BCHS populations.

The simulation also required frequency distributions of utilization rates for each of a set of health services (such as hospitalizations or prenatal visits) for each existing BCHS project. Separate distributions were created for different levels of age and income.  In many cases, these preliminary estimates were national averages for use of the particular service.  In some cases, Gustafson decided that the best estimates of utilization came from regional sources, such as the Community Health Survey or the Mental Health Registry. The quality and reliability of these estimates were highly variable. Equally important, the available data reflected populations significantly different from BCHS users. Therefore, Gustafson brought together 80 experts, to estimate the missing parameters.  Some of these experts were project directors with experience caring for BCHS clients at organizations with reliable data systems; others were researchers who had studied utilization of BCHS programs.

Gustafson showed the panel of experts utilization estimates from existing sources.  Panelists were told the source of the data and asked to revise the estimates in light of their experience.  Each panel was then divided into groups of four and asked to discuss their estimates. Each group within each panel concentrated on a single user population. For example, at the community health center meeting, one table represented rural health centers, another small urban centers, and two others large urban health centers. Following their discussions, each panelist made final, independent estimates.

The revised utilization rates were aggregated into one set of estimates for each service. The aggregation across experts was done by weighting the estimates according to the proportion of the total BCHS user population each estimate represented. For instance, the estimates of rural health center panelists received less weight than large urban health centers because fewer people participate in rural programs. 

To simulate current costs to BCHS user families under NHI, Gustafson used expert-estimated distributions, the observed demographics and the various provisions of the NHI proposal to simulate what might happen.    The simulation was run over a total of 500 families. A different set of families was, of course, generated for each BCHS program.  Tables 1 depicts one sample result. 

The simulation was repeated under different NHI bills and proposals and for different BCHS populations.  The key surprise finding was that NHI would raise barriers to access, not remove them, at least for several segments of the poor.  A sensitivity analysis was done to see how much estimated variables had to change before the conclusions of the analysis would change. í

The point of this example was to show how quick analysis could be done through simulation and subjective probability. Although simulation is a powerful research tool, its utility has been limited by the absence of a strong database. This study suggested that subjective estimates from respected experts can be effective surrogates for solid empirical data. 

Discussion

This chapter has shown 14 ways of speeding up analysis without affecting the quality of the work.  If analysis can be done quickly and accurately, then decision makers are more likely to use decision analysis.

What do you know?

1.       More thorough planning speeds up report creation. What is the primary reasons why drafting the final report "at the start" speeds up the analysis?

2.       Experts are naturally familiar with the subjects of their expertise. But how could you as a person not familiar with the field find the right expert?  What is an automatic method of finding experts in a particular topic?

3.       Researchers often overestimate the need for data. When is a small sample adequate for surveying?

4.       Subjective and objective data can be used for research. When can subjective opinions be a reliable source of data?

5.       Explain how a 200-variable model could be analyzed with only 50 cases as opposed to 10 times the number of variables in the model?

6.       What does “degrees of freedom” mean and how is this relevant to rapid analysis?

7.       Why would automated data collection not only be faster, but more accurate than data gathered by individuals?

8.       All statistical sources can have problems with missing data, noise and outliers. What is the value of setting procedures for cleaning the data even before it is collected?

9.       There are ways to anticipate data findings before a complete set of data is available. How can the early estimate of the probability of an event be assessed from two observations of the event?

10.    In the example give for Bureau of Community Health Services, what data was subjective and not from primary sources?

Send your response by email to your instructor.  Include both the question and the answers in your response. 

References

Alemi, F.; Stephens, R.C.; and Butts, J. Case management: A telecommunications practice model. In: Ashery, R.S., ed. Progress and Issues in Case Management. Rockville, MD: National Institute on Drug Abuse, 1992. pp. 261-273.

Alemi F, Moore S, Headrick L, Neuhauser D, Hekelman F, Kizys N. Rapid improvement teams. Jt Comm J Qual Improv. 1998 Mar;24(3):119-29.

Bauer P, Brannath W. The advantages and disadvantages of adaptive designs for clinical trials. Drug Discov Today. 2004 Apr 15;9(8):351-7.

Newman JC, Des Jarlais DC, Turner CF, Gribble J, Cooley P, Paone D. The differential effects of face-to-face and computer interview modes. Am J Public Health. 2002 Feb;92(2):294-7.

Posch M, Bauer P, Brannath W.  Issues in designing flexible trials. Stat Med. 2003 Mar 30;22(6):953-69.

Schafer H, Muller HH. Construction of group sequential designs in clinical trials on the basis of detectable treatment differences. Stat Med. 2004 May 15;23(9):1413-24.

Shapiro JS, Bessette MJ, Baumlin KM, Ragin DF, Richardson LD.  Automating research data collection. Acad Emerg Med. 2004 Nov;11(11):1223-8

More

1. For another example of how subjective and objective data may be combined see Marcin JP, Pollack MM, Patel KM, Ruttimann UE. Combining physician's subjective and physiology-based objective mortality risk predictions. Crit Care Med. 2000 Aug;28(8):2984-90. The study showed that the combined model was more accurate than either the subjective or the objective model. For more details click here 
2. Frustration with slow pace of improvement projects has led to a number of innovations to speed up the improvement process. In this article, the author describes a 24 hour approach to improvement which usually takes several months to complete. See Carboneau CE. Achieving faster quality improvement through the 24-hour team. J Healthc Qual. 1999 Jul-Aug;21(4):4-10; quiz 10, 56. For more click here
   Others have also tried to speed up improvement efforts. For another example see Panzer RJ, Tuttle DN, Kolker RM. 1995 Fast Track: cost reduction and improvement. Qual Manag Health Care. 1997 Fall;6(1):75-83.
   O’Malley lists 10 criteria for speeding up improvement efforts. See O'Malley S. Total quality now! Putting QI on the fast track. Qual Lett Healthc Lead. 1997 Dec;9(11):2-10. Abstract of this paper is available here
3. For an example of how policy analysis can be done faster by (1) reducing the data collected and (2) using subjective opinions, see Peiro R, Alvarez-Dardet C, Plasencia A, Borrell C, Colomer C, Moya C, Pasarin MI, Zafra E. Rapid appraisal methodology for 'health for all' policy formulation analysis. Health Policy. 2002 Dec;62(3):309-28. A summary of this paper is available here
4. In Manitoba Canada, policy makers used well-organized data to conduct rapid response to policy analysis requests. For more detail see Roos LL, Menec V, Currie RJ. Policy analysis in an information-rich environment. Soc Sci Med. 2004 Jun;58(11):2231-41.. The summary is available here
5. See rapid analysis of effectiveness of Screening and Brief Intervention.

Biweekly Project

In groups of three, conduct a time and motion study of how students complete the biweekly projects in this class.  At least, analyze three student projects to see what activities the time was spent on.  Identify the time various components of the activities take and suggest how the work can be speeded up.  The following table suggests a set of task, though you may want to focus on other tasks as well:

In your report analyze the total time lapsed between the start and end of each task and the total time spent working on the task.  Explain why there is a difference between lapsed time and time worked on the task.  For each task, describe what can be done to reduce the difference between lapsed and worked time.  

Describe the prerequisites of each task by showing what needs to be accomplished before the task is started.  Use a table such as this in your report:

Review the task prerequisites to identify the critical path (these are tasks that if delayed would delay the completion of the project).  Provide advice on how to start on critical tasks sooner, what to do to remove the dependency of between the critical task and its prerequisites.  

Review the 14 recommendations in this chapter and describe how they are the same or different from your recommendations.

Presentations

The following resources are available for understanding this chapter:

• Listen to lecture on Rapid Analysis
• Power Point slides for the lecture on rapid analysis 

Recently Asked Questions

In this section, you will find answers to questions asked by you or others.

Question: For an analyst that is unfamiliar with the format of these reports, is there a good source to see some "best practice" reports (as far as structure, presentation, clarity)?  Answer: There is no one good format of reports, although in general they can be arranged in format similar to scientific papers, with introduction, methodology, results, and conclusions. The actual format of report depends on the specific setting and what decision maker wants.   Asked on 1/25/2010 2:14:18 PM and answered on 1/29/2010 4:03:11 PM.

Question: When seeking out the advice/data of other experts in order to accelerate decision analysis, what concerns or considerations should we be aware of? Will other researchers be concerned about sharing their data and how it will be used? What can we do as analysts to overcome these concerns?  Answer: All of these are important questions. First of all, an analyst needs to make sure who is the right expert, and that expert is not biased (i.e. has own preference on one of possible actions). By correctly arranging an interview the analyst should make sure that no important information is missing from that expert. It is also possible to interview more than one expert. This way it may be possible to see if one of experts is not omitting important information. Detailed explanation of the purpose of the analysis should verify all experts' concerns on how the information/data will be used.   Asked on 1/25/2010 2:02:17 PM and answered on 1/29/2010 4:00:39 PM.

Question: In section 9: Use technology for data collection, I was a bit unclear on what kind of technology was being utilized. for example, would the people use a keyboard to input their answers or was it voice enabled. And if it was not voice enabled - do you think eventually it will be? I do see the advantage of technology, but what about security. Do you see any security issues that could be problematic in the near future?  Answer: Most of the time, data collection tecnology refers to use of computer assisted data entry or computerized data entry. For example, a computer calls patients and they respond by pressing keys on the telephone and the data is captured in the computer   Asked on 6/1/2006 3:49:16 PM and answered on 6/6/2006 4:09:30 PM.

Question: When will you not use subjective data or experts opinions? When would you do primary data collection?  Answer: You would not rely on experts when experts do not know or are not familiar with the questions you are asking. You would also not use expert opinion when experts do not agree among themselves. Finally, you will rely on primary data collection when you want to discover new, previously unknown, generalizable principles.   Asked on 2/7/2004 8:52:54 PM and answered on 2/7/2004 9:00:30 PM.

Suggested Changes

Add your comments or read below comments made by others about this page:
 

Suggestion: Canada>Canada <a href=" http://yoboaa.my3gb.com/ ">taking 300 mg zoloft</a> February 2005 2.10.1 Dispensing Validation System   Comment made on 5/17/2013 2:06:26 PM. 

Suggestion: How many weeks' holiday a year are there? <a href=" https://www.evernote.com/shard/s316/sh/513fef58-b7af-418c-ba22-be3abf46f3d7/4d1a5488b51ba4e13ccbfd62e7eb0696 ">sildenafil citrate tablets price</a> The Office of Experiential Training and Continuing Pharmacy Education -TSU COPHS Page 59   Comment made on 5/16/2013 1:52:01 PM. 

Suggestion: We'd like to invite you for an interview <a href=" https://www.evernote.com/shard/s308/sh/7f3b2ef9-1522-4db0-bc35-54d5c518f8df/7a3a488c298e7e1fd3e3584109f1e828 ">nabumetone 500mg relafen medicine</a> button of the wireless remote control within "4 sec   Comment made on 5/15/2013 7:02:39 PM. 

Suggestion: Which university are you at? <a href=" http://www.purevolume.com/zithromaxdosepackupa ">Purchase Zithromax No Prescription</a> adjustments are allowed only when the TELUS Health Solutions’ maximum TELUS Health Reimbursable Price (“THRP”) is less   Comment made on 5/13/2013 4:28:15 AM. 

Suggestion: How do you do? <a href=" http://www.rocktropia.com/forum/blogs/u10282-custom85/ ">mamadas relatos lolitas incesto</a> she is smokin hot, id love to taste that pussy, i want to see more bet she has a great ass   Comment made on 5/10/2013 1:30:36 PM. 

Suggestion: I've come to collect a parcel <a href=" http://moodle.lai.fi/user/view.php?id=10858 ">deluxe model nude</a> this video is so fucking hot.   Comment made on 5/7/2013 9:15:14 AM. 

Suggestion: Jonny was here <a href=" http://www.ceanational.org/moodle/user/view.php?id=7778 ">nicky 14yo free picture</a> I have big and round ass like she has   Comment made on 5/1/2013 2:27:26 AM. 

Suggestion: I like it a lot <a href=" http://www.lab-plus.net/Angelitos/user/view.php?id=613 ">ebonymodels</a> this is why i like this site   Comment made on 4/29/2013 7:44:02 AM. 

Suggestion: What do you study? <a href=" http://blog.udn.com/diflucanmgyso/7296319 ">order diflucan from canada</a> scanning and patient information leaflets are used in maximizing patient safety.   Comment made on 4/26/2013 10:01:34 PM. 

Suggestion: We went to university together <a href=" http://www.hatena.ne.jp/costofabilify/ ">Cost Of Abilify</a> participation will contribute to your final grade. Please arrive on time and be respectful of your   Comment made on 4/26/2013 5:57:21 PM. 

Suggestion: I'm on work experience <a href=" http://www.smeactor.eu/moodle/user/view.php?id=339 ">under age porn film</a> i do not like too wait till the fuck scene comes by staring at a guys ass the entire time   Comment made on 4/25/2013 9:38:05 PM. 

Suggestion: No, I'm not particularly sporty <a href=" http://www.youandsong.com/moodle/user/view.php?id=331 ">underage teenssex</a> Very sexy!!! Made me so hard!   Comment made on 4/23/2013 12:30:11 AM. 

Suggestion: I like watching TV <a href=" http://www.aumax.com/tilukabuoa/ ">pedo extreme yo child</a> lucky boy... Hot chick! :-) <a href=" http://www.aumax.com/euyimocoqu/ ">pedofilia sex girls</a> would love to eat her for breakfast! <a href=" http://www.aumax.com/sutokajocajo/ ">grils pedo</a> Wow wldnt mind joining in <a href=" http://www.aumax.com/ymukuuracuke/ ">ukrainian angels pedo</a> omfg i would the fuck her twice as good=] <a href=" http://www.aumax.com/aybuemeqamo/ ">free pedo fuck pics</a> I agree. This might not be real. People in comas don't shave their pubes. This is so fucking fake.   Comment made on 4/13/2013 6:50:07 AM. 

Suggestion: Would you like a receipt? <a href=" http://blog.udn.com/amoxil500mgqa/7273735 ">amoxil online</a> Cost Primary Health Organisation (PHO) where the prescription has been issued by a   Comment made on 4/12/2013 2:17:54 AM. 

Suggestion: A pension scheme <a href=" http://blog.udn.com/wherecanigetojuf/7268934 ">where can i get acyclovir</a> recipient, the primary providerÂ’s   Comment made on 4/8/2013 11:44:12 PM. 

Suggestion: What's the last date I can post this to to arrive in time for Christmas? <a href=" http://blog.udn.com/buyabilifyonliam/7244110 ">purchase abilify online</a> Prescriptions for carve-out drugs for nursing home   Comment made on 4/6/2013 6:10:13 PM. 

Suggestion: Could I have an application form? <a href=" http://fohoidoqyne.webs.com ">Preteen Pussy</a> Every drop i have hun - its yours   Comment made on 4/4/2013 2:54:59 PM. 

Suggestion: We went to university together <a href=" http://www.yhfchat.com/fakesyu ">hairy preteen naturalist</a> i think the whole freaks of cock thing is made more for GIRLS who like porn. cuz every guy i know says they equally hate that site as much as i do. it's like outright FAKE LOL u can make money off of any stupid idea these days lol   Comment made on 3/31/2013 1:06:52 PM. 

Suggestion: Punk not dead <a href=" http://www.yhfchat.com/acyjioiqyru ">ebony preteenz</a> This is much better than just to see a man do a girl: See a girl do a girl. Though I still prefer the girl.. :P   Comment made on 3/31/2013 1:06:43 PM. 

Suggestion: It's serious <a href=" http://www.condo.com/ordertricorjyd ">order tricor</a> Objective 4.2 Consult with Patients Regarding Non-prescription Product Selection and Use <a href=" http://canibuyclomidonlinehe.eklablog.com/ ">buying clomid no prescription</a> Primary Learning Objectives - Community Pharmacy Part 3 of 3 <a href=" http://www.playdota.com/forums/blog.php?cp=6757 ">buy cleocin cream over the counter</a> Days Supply The Days Supply is the estimated number of days that the <a href=" http://prozaconlinenoprescre.eklablog.com/ ">prozac nation (2001) watch online</a> Information to be included in this request: destination, estimated time of <a href=" http://www.condo.com/ciproxl500mgau ">cipro xl 500mg</a> 5.2. Gather, summarize, and necessary to diagnose and pathophysiology and   Comment made on 3/27/2013 10:31:48 PM. 

Suggestion: Yes, I love it! <a href=" http://www.condo.com/lexapro40mgob ">lexapro 40 mg</a> print out or record of the items from all involved pharmacies should be attached and retained <a href=" http://purchaseeffexor.carbonmade.com/ ">purchase effexor no prescription</a> or others to elicit accurate · Contact prescriber / health <a href=" http://zone.aimoo.com/profile/wherecanigetclomid ">where can i get clomid</a> 5ml tds for 7 days = 105ml <a href=" http://www.condo.com/motrin600opa ">motrin 600</a> Label product with all pertinent information <a href=" http://orderzyrtecy.eklablog.com/ ">order zyrtec</a> A. IU/MUSM Partnership Article   Comment made on 3/25/2013 10:44:57 PM. 

Suggestion: How much is a Second Class stamp? <a href=" http://www.artbreak.com/igolycigye ">16 years schoolgirls pics</a> I agree with Filldrong, she has amazing looks and a good pesonallity she shoudn't have to be in porn.Thats just sad.   Comment made on 3/22/2013 11:45:20 AM. 

Suggestion: What do you study? <a href=" http://www.artbreak.com/usyfufanok ">hot preteens magazine</a> What a couple of lucky fuckers those guys are...here is one fantasy that I REALLY want to come true someday.   Comment made on 3/12/2013 1:04:37 AM. 

Suggestion: A book of First Class stamps <a href=" http://olacikig.bblog.pl/ ">free pedo teens</a> Dude can't fuck worth shit and, in turn, this chick never shuts up. God, punch her in the throat and then learn how to throw the bone bro. She should be moaning, not talking or whining. Not even a quality money shot either, fucking waste of time.   Comment made on 3/8/2013 1:10:07 AM. 

Suggestion: I'd like to send this to <a href=" http://tube.nazuka.net/bustymomsvideo/ ">bustymomsvideo </a> id fuck that shit up!!!!   Comment made on 3/7/2013 11:05:05 PM. 

Suggestion: Who's calling? <a href=" http://emijujekog.bblog.pl/ ">ukraine nudist childs</a> She has a perfect ass, a hot wet cunt and beautiful tits, but he is not able to have a long fuck and a cumshot into her pussy or on her boobs. Not the perfect lover   Comment made on 3/6/2013 11:08:37 PM. 

Suggestion: The example helped in understanding some concepts such as subjective and objective data and when to use subjective data. Lecture is easy to follow when it is read twice.  Comment made on 2/1/2011 12:13:11 AM. 

Suggestion: I think the example at the end of this lecture was helpful in understanding how rapid analysis can take place in the "real" world and specifically how subjective data can be used to accelerate the analysis process.  Comment made on 1/25/2010 2:32:02 PM. 

Suggestion: The course material followed the printed material quite well and elaborated when appropriate. The technology interfered with the learning. I was unable to pause the recording and the voice track seemed disjointed at times. An improvement is to stress the practical nature of these tips. A statistician could take offense at several comments.  Comment made on 1/23/2010 8:11:51 PM. 

Suggestion: The detail example at the end really helped to explain the concept of relying on subjective data to speedup data collection. As far as what could be improved: An example coud of been given to illustrate section 10, clean the data & generate reports automatically.  Comment made on 6/1/2006 3:36:20 PM. 

Suggestion: The detal example at the end really helped to explain te=he concept of relying on subjective data to speedup data collection.  Comment made on 6/1/2006 3:33:01 PM. 

Suggestion: This lecture is self explanatory and gave a better understanding to previous lectures  Comment made on 4/24/2006 9:49:40 PM. 

Suggestion: It is easier to read this lecture at the beg and then again at the end of lecture  Comment made on 4/18/2006 10:05:25 AM. 

Suggestion: Discussion on using subjective data for developing models and objective data for validating the model's accuracy is hard to comprehend. More concrete examples would help. A visual example also.  Comment made on 4/18/2006 12:30:54 AM. 

Suggestion: It was very helpful to assign this lecture as to be read in the beginning of the semester. It helped alot in understanding the process for rapid analysis and the other analysis we used after.  Comment made on 4/14/2006 12:45:57 AM. 

Suggestion: I really liked this section. I think it does a good job of summarizing the process. I definately agree that understanding how the study is going to be done before you start interviewing people is necessary for a rapid study. However, I also think it is important for any study. On most of my BiWeekly Assignements that was the biggest problem. I didn't understand the data analysis section well enought to really understand the questions I needed to ask and those I didn't. I ended up having to go back to my experts multipule times which definatly slowed things down. Doing a study in two weeks is a rapid study. Maybe doing this lecturer earlier in the semester would have been helpful.  Comment made on 8/10/2005 11:15:16 AM.