A database released in July arguably changed the dynamics of the surgical profession. When the performance of more than 16,000 surgeons became easily searchable online, surgery was suddenly a public-facing profession.
The Surgeon Scorecard belongs to nonprofit investigative newsroom ProPublica, which has worked on a number of medical transparency projects, including a recent push to add CMS data to Yelp's review pages. The scorecard is unique among online ratings in that it shines a spotlight on the individual surgeon's outcomes, rather than those of the hospital. For some surgeons, that light is unforgiving.
About a month and a half after the Surgeon Scorecard's launch, it has been viewed more than 1.7 million times and gained skeptics and supporters alike. Critics call it flawed; some even consider it public shaming. Advocates see it as a stride for patient safety and transparency.
Its makers acknowledge the scorecard has some limitations, like any study, but say this doesn't overshadow its importance.
"You are going to take some heat if you do it," said Marshall Allen, one of the ProPublica reporters behind the Surgeon Scorecard. "But it prompts the medical community to provide more information to patients, and I think there is a hunger for it. Traffic on the database online has been a big affirmation."
However, considering the page views and the outpouring of reaction, it's hard to pin down the repercussions of this newly available information. We set out to see how life has or hasn't changed for surgeons.
Responsible, not at fault
A group of ProPublica's investigative reporters created Surgeon Scorecard after accessing five years worth of Medicare data on eight elective surgical procedures, including knee replacement, hip replacement and laparoscopic gallbladder removal. They only analyzed elective procedures since those are scheduled in advance and performed on patients generally in good health. Cases where patients were admitted through the emergency room or from nursing homes were omitted.
This left the reporters with approximately 2.3 million elective procedures for Medicare fee-for-service patients. They then consulted a panel of two dozen physicians to determine which complications were appropriate to gauge outcomes. They counted cases where patients died in the hospital and those where patients were readmitted within 30 days with complications that were most likely surgically-related.
Within this sample, overall complication rates looked low on paper — only about 2 to 4 percent, according to ProPublica. Yet when 2 to 4 percent is illustrated in lives, it equates to 3,405 deaths of Medicare patients during a hospital stay and a staggering 63,173 readmissions.
The results also showed great variation in performance between surgeons, even at some of the nation's top medical centers. For example, 756 surgeons who performed at least 50 operations did not have one recorded complication. On the other hand, about 25 percent of complications could be attributed to just 11 percent of physicians, according to ProPublica.
Imagine you are a passenger on a flight with a pilot who is one of few responsible for one in four plane crashes — that's something you'd want to know before boarding the aircraft. While the whole flight crew, the plane and even the weather may have contributed to some or all of those crashes, the pilot is still ultimately responsible for everything from takeoff to landing.
It's the same drill in the operating room. While a surgeon may not be at fault for a surgical complication, they are ultimately responsible for them, and that's what the Surgeon Scorecard tries to get at. It's not meant to point fingers or overlook the influence of team-based care.
"The surgeon isn't necessarily to blame, but the surgeon is responsible for the overall episode of care," said Mr. Allen. "The surgeon is best able to figure out what might have gone wrong and best able to prevent it in the future. That's what we are trying to motivate and spur along — the commitment to improvement."
The American College of Surgeons and patient safety experts confirmed that surgeons are considered responsible for all aspects of a patient's care, just like a pilot is ultimately responsible for landing safely at the destination.
While this idea of holding surgeons responsible for outcomes is an accepted principle, publishing those outcomes online is a whole new animal.
What report card?
We spoke with a handful of surgeons and other medical professionals to get their take on the new tool. Each of the surgeons we spoke with had low adjusted complication rates in at least one procedure. They were also affiliated with organizations that ranked well overall in their region. The scorecard ranks a healthcare organization by its surgeon with the lowest adjusted rate of complications and by the combined performance of surgeons and hospitals under its Medicare ID for those procedures.
Everyone we talked to was familiar with the Surgeon Scorecard, but not necessarily concerned by it.
John Cooper, MD, associate medical director of quality and surgical services at the Palo Alto (Calif.) Medical Foundation, was so impressed by the tool that he wrote in to ProPublica and said he was sharing the information with his colleagues. "I was stunned by it in that it's a bold step," Dr. Cooper said in an interview with Becker's Hospital Review. "I was impressed just on the basis of sharing it publically. They did a great job with the data they have. There are some limitations, but they made a nice effort to risk adjust and use analytics."
Another physician, James Kudrna, MD, orthopedic surgeon with NorthShore Orthopaedic Institute, part of NorthShore University HealthSystem in Evanston, Ill., had a closer brush with the tool. Dr. Kudrna clocked the lowest adjusted complication rate for hip replacements in the state of Illinois, at 1.7 percent.
He said he had only heard positive things about the Surgeon Scorecard from his colleagues, but added, "If I were to extrapolate, there were likely a few physicians who were maybe taken aback by this. The reason for this is depending on where [their] institution is located; the mix of patients is variable. If they are quite sick and haven't had a lot of care available to them or have a lot of comorbidities...The data has to be looked at by what kind of patient pool you treat."
However, as Mr. Allen noted ProPublica's reporters adjusted data for risk and took each surgeon's case mix into account. Trauma and high-risk cases were thrown out and statistical methods were used to adjust for age, health, hospital performance and luck. Extremely high raw rates were adjusted downward, assuming some amount of bad luck was involved, and extremely low rates were also adjusted, assuming some amount of good luck was involved. For this reason, the lowest reported rate is 1.1 percent and the highest reported rate is 12.2 percent.
Washington's top hip replacement surgeon, James Crutcher, MD, an orthopedic surgeon with Orthopedic Physician Associates, which is affiliated with Seattle-based Swedish First Hill Campus, has a 2.0 percent adjusted complication rate. "I became aware of the database a month ago when [a fellow physician] asked me if I'd seen the data and they forwarded it to me. Really, I haven't heard much publicity about it since," Dr. Crutcher said. He added, "It's really early; we haven't felt any impact from this data release at this time."
The surgeons confirm that the scorecard has garnered attention within the medical community, as Mr. Allen guessed. ProPublica's reporters in New York City continue to receive emails about the database, but Mr. Allen says they have not received as many as one might think. "We just published 17,000 complication rates and we've gotten maybe fifty-ish emails — that's from patients, hospitals, that's everybody. In a way that's not a lot of response."
Based on the emails he has received — many from physicians and researchers with questions, concerns and corrections — he guessed most of the scorecard's web traffic so far stems from the medical community.
Despite this attention, the Surgeon Scorecard hasn't changed how hospitals are addressing their lowest performers, based on interviews with physicians. A month and a half after its publication, none of the surgeons' affiliated hospitals had taken action directly in response to the tool. Drs. Cooper, Kudrna and Crutcher each confirmed that the data has not yet influenced changes within their current clinical quality improvement programs, which are already focused on continuous improvement.
Dr. Cooper of the Palo Alto Medical Foundation, part of Sacramento, Calif.-based Sutter Health, said, "None of our doctors have a score they need to feel bad about."
Dr. Crutcher of Orthopedic Physician Associates in Seattle made a similar confirmation. "It's not something that regularly occurs, where a doctor has a high complication rate in one area. We all operate with the same care processes and measures so we maintain a standard level of performance," he said.
Dr. Crutcher is included under Swedish Medical Center's Medicare ID, which does not have any hip replacement surgeons in the "red" zone with high rates of adjusted complications. Almost all of its hip replacement surgeons span the tool's medium range of complications, which means the surgeons' patients fared no better or worse than their peers' patients. A medium rate can also mean there is not enough data to put the physician definitively in the low or high range.
However, not all Swedish Medical Center surgeons are operating outside of the "red" zone. A couple of knee replacement surgeons in particular fall into the area for high adjusted rates of complications.
This is not unique to Swedish Medical Center — a quick search for Evanston Hospital, the Medicare ID for the four-hospital NorthShore University HealthSystem where Dr. Kudrna works, also produces a range of results. Its surgeons span the scale of low to high adjusted rates of complications, with several surgeons in the red zone for various procedures.
If anything, the Surgeon Scorecard illuminates that wide variation still exists among surgeons, even at the same facilities, and even when clinical quality programs are in place.
Doctor's orders
While the data may not be changing clinical quality agendas yet, it is fairly revolutionary for patients. It helps inform their decision-making and makes it easier to differentiate clinical quality from amenities and customer service. Though ProPublica recently partnered with Yelp to beef up its review pages with CMS data, many reviews authored by patients on Yelp's site, or similar sites like ZocDoc, focus on bedside manner, wait times and other qualitative factors.
ProPublica published a response from a reader who was married to a nursing supervisor at a local hospital. He said his wife was not aware of the Surgeon Scorecard.
"When my mother required gallbladder surgery, my wife specifically ensured that a certain surgeon wasn't on call for the procedure," he wrote, according to ProPublica. "While I was at the hospital visiting my wife, I mentioned casually to two of her coworkers (separately) that my mother was upstairs awaiting surgery. Both nurses asked cautiously who was on call and when they found out it was Dr. [redacted] ... they breathed a sigh of relief."
The Scorecard helps patients without a connection to a nursing supervisor or individual with inside knowledge get that same information. Studies show a thirst for this information, too. In a 2015 survey of more than 3,000 consumers, the National Research Corp. found 70 percent of consumers are interested in seeing performance data before choosing a healthcare provider and 45 percent have viewed physician ratings or reviews online. Almost a third said they looked at online ratings or reviews as a first step in finding a physician.
That said, none of the physicians we talked to had heard a patient mention the Surgeon Scorecard. This may be because it is still too new, or patients say they found a physician "online" with no further specification, or use it alongside other methods, such as word-of-mouth or physician referral.
"In my experience, the most common referral source is other patients. You have a patient that has a good outcome and they become a megaphone," Dr. Crutcher said. "Way down the line are online rating systems. I've heard from others before: People value referrals from friends, family and primary care physicians much more than an Internet source."
With 40 years of experience and an estimated 19,000 hip replacements under his belt, Dr. Kudrna confirmed word-of-mouth was also the top way his patients found him. For younger physicians, he guessed physician referrals were king. "Many of those types of referrals are based on success of patients. As [your career] builds it's a patient referral, but earlier on, it's physician referral."
While personal and professional referrals may be the norm for the Drs. Crutcher and Kudrna, who are already ranked among the nation's top surgeons for Medicare hip replacements, established online tools are increasingly factoring into patient decisions.
According to the National Research Corp. survey, 60 percent of consumers feel online ratings and reviews are important when choosing a new physician, especially when it's on a "site which I recognize and trust." Based on this data, the Scorecard has the potential, with time, to become very influential.
Making the grade
It may still be too early to see if the Surgeon Scorecard will make a difference in healthcare. The utility of the tool for consumers hinges on how patients define what makes a good physician. For busy patients, a good physician may be one with short wait times and flexible schedules. For nervous patients, a good physician may be one who is friendly and understanding. Yet for those who prioritize outcomes, the Surgeon Scorecard is an innovative tool to support informed decision-making.
Nonetheless, many patients may never seek out this information — they may continue to rely on family, friend and physician referrals to choose surgeons. Even in that case, ProPublica's Mr. Allen remains optimistic.
"Let's just say no patient ever looks at it. Well you still have the medical community looking at it, and surgeons alone are competitive. They want to be the best, and even if they see they have wide variation, that will motivate them to see how to improve," Mr. Allen said.
To that point, the Surgeon Scorecard demonstrates just how far healthcare transparency has come. Data was still institutional a couple of years ago. It pertained to hospitals as organizations, not so much the individual physicians practicing inside of them. Two years ago, it was monumental for CMS to add to its Hospital Compare website ratings of hospitals based on readmissions and complications following hip and knee replacements. Now, drilling down complications by surgeon provides an even clearer picture. This increased sense of transparency created by the Surgeon Scorecard, whether patients actually use it or not, engenders accountability and can be an incredible motivator for improvement on the individual level.
To share comments, concerns or advice with ProPublica on the Surgeon Scorecard, contact scorecard@propublica.org.
More articles on integration and physician issues:
Don't call this physician a saint
Depressed, stressed out and breaking up: What this top researcher learned from medical
interns
'But I will, if I have to': How one surgeon has repeatedly saved his loved ones