Ranking The Rankings: Bias In Hospital Scores

Major companies like U.S. News & World Report and Leapfrog publish annual hospital ratings both to reward top hospitals for their level of care and to help patients stay informed about the strengths and weakness of their local hospitals. But new reports call the reliability of these ratings into question.

Doctors at the Johns Hopkins Hospital in Baltimore, Maryland researched data used to create these rankings and found inconsistencies. About 68% of all Maryland hospitals, including Johns Hopkins, received the worst possible ratings in patient safety. Nationally, only 21% of hospitals received this lowest rating.

Doctors at Johns Hopkins researched the methods used to create these ratings and found that at least one factor that counted towards 10% of the patient safety rating was being unfairly considered in Maryland.

U.S. News & World Report uses Medicare billing claims to differentiate between medical conditions that are “present on admission” (or POA) and medical conditions that are the result of poor hospital care.

Since 2007 most hospitals in the United States have been required to inform Medicare about POA conditions by using codes in the billing claims. Maryland however, has the nation’s only all-payer hospital regulation system. This system requires all payers—including the government, private insurance companies, businesses, and patients—to pay the same rates for health care.

This plan eliminates both the price variation by region and the tendency for hospitals to overcharge patients due to low Medicare reimbursements. All-payer plans encourage patients to judge hospitals for their level of care and not on their costs.

Ironically, it is this part of the plan that may have left Maryland with low patient care rankings.

Until 2014 (the last year the ratings focused on), Maryland did not need to report POA data to Medicare. This made Maryland hospitals appear to cause many incidences of patient harm that were actually present in the patient at arrival. Compounding this glitch in the ratings was the growing concern about preventable patient harm that led U.S. News to increase the weight of the patient safety score from 5% to 10% of the overall hospital score last year.

Avery Comarow, the Health Rankings Editor at U.S. News & World Report, responded to the criticism by stating that though not required, some of the POA data was available for Maryland hospitals at the time of the ratings, and that for most of the other patient safety ratings Maryland hospitals score similarly to many hospitals in the country. Comarow did note that the software used in the study did have limitations and biases, but he couldn’t say whether that would work for or against Maryland ratings, or it if it would even have any effect on the ratings.

Bias in hospital ratings is not unique to the U.S. News & World Report list. A different set of ratings completed by LeapFrog found New York’s Harlem Hospital to be among the top 98 hospitals in the entire country, though just two years earlier Consumer Reports ranked it to be one of the worst in the country. Seven of the hospitals on the list were slated to receive payment reductions from Medicare due to “high rates of potentially avoidable infections and complications such as blood clots, bed sores and falls.”

The clearest takeaway from the discrepancies in these rankings is that there are no clear hospital rankings. In the absence of nation-wide controls for hospital funding and patient care reporting, rankings should be read with some degree of skepticism.