CUSUM for monitoring competency: Computer software is useful for bootstrapping and real time CUSUM plotting.
Br J Ophthalmol 2011;95:295e296. doi:10.1136/bjo.2010.188805. http://bjo.bmj.com/content/94/4/445/reply#bjophthalmol_el_9709
CUSUM for monitoring competency:
Computer software is useful for bootstrapping and real time CUSUM plotting.
We congratulate Salowi and colleagues on their study of the use of CUSUM to monitor competency in cataract extraction (1). However there are two modifications that could make their system more responsive and reliable.
Firstly the authors use a CUSUM chart that fluctuates on only one side of the zero line. Traditionally CUSUM charts fluctuate on both sides of the zero line (2,3). The consultant in their Figure 1 performed 48 procedures and 43 were successful while 5 were failures. The weight for a failure would be -1.791666 and for a success it would be + 0.208333. Figure 1 depicts the traditional CUSUM graph with this data. Against this, the data from his trainee is also drawn. The trainee CUSUM score keeps going further and further away from the zero line, suggesting that he has not reached the bottom of his learning curve. Once his learning is over, his mean CUSUM line will run parallel to the zero line.
The second modification relates to the decision interval. The authors discuss in detail the arbitrary decision intervals (control lines) that they have employed, based on a trade-off between the need to detect poor performance quickly and that to avoid a large number of false alarms.
Decision lines need not be arbitrary. With the help of computers, bootstrapping techniques can be employed, so that these lines are placed where they are statistically meaningful. As an illustration, in Figure 1 with the consultant performance, there were 5 failures and 43 successes. The 5 failures need not be evenly interspersed among the successes. The sequence of failures is purely a matter of chance and 2 or even 3 failures may be clustered together. Bootstrapping allows random reordering of the failures and successes in a way that the overall numbers of success and failures are the same for each iteration. The computer can calculate the maximum and minimum score for each iteration. If a 1000 iterations are performed it is possible to calculate the mean of the highest scores (maximum score in the iteration) and the mean of the lowest scores (minimum score in the iteration) and also the standard deviation around the means. The upper decision line is the limit drawn with the mean upper score plus 2SD. The lower decision line is the mean lower score minus 2SD. If surgery is performed by a person of comparable competence as the consultant, his CUSUM score will lie within the two decision lines, 95% of the time.
We have recently used CUSUM for a clinical trial and for this we developed software that allows for easy bootstrapping, drawing of control lines and plotting of CUSUM score. This software is available free on the internet. (http://jacob.puliyel.com/foresee/).
Figure 2 can also be redrawn using the acceptable rate for posterior capsule rupture (PCR) of 5%. Here the acceptable standard is 1 failure for 19 successes. For Figure 3 using the acceptable rate for impaired vision as 10% (using data reported by the authors from the Malaysian National Cataract Surgery Registry) there can be 1 failure for 9 successes.
The software is interactive and allows CUSUM plotting in real time (meaning that it allows one to see how the CUSUM graph evolves, with each new success or failure). We hope that the free software available on the net will encourage more widespread use of CUSUM in various clinical situations.
Figure 1 Standard reference cumulative sum (CUSUM) showing performance by a trainee and a consultant can be viewed at http://jacob.puliyel.com/#paper_199
Reference
1. Salowi MA, Choong YF, Goh PP, Ismail M, Lim TO. CUSUM: a dynamic tool for monitoring competency in cataract surgery performance. Br J Ophthalmol. 2010;94:445-9.
2. Van Rij AM, McDonald JR, Pettigrew RA, Putterill MJ, Reddy CK, Wright JJ. Cusum as an aid to early assessment of the surgical trainee. Br J Surg. 1995;82:1500-3.
3. Sibanda T, Sibanda N. The CUSUM chart method as a tool for continuous monitoring of clinical outcomes using routinely collected data. BMC Med Res Methodol. 2007;7:46.
The CUSUM software available free on the internet, was developed by the authors in the context of a clinical trial.
Ashish Puliyel, Tech Guru gonzoBuzz, Singapore
ashishpuliyel@gmail.com
Jacob Puliyel, Consultant Pediatrician, St Stephens Hospital, Delhi, India
puliyel@gmail.com