Monthly Archives: November 2011

The UK Quality and Outcomes Framework for NHS GPs – Did it work, and should we copy it?

Quality and Outcomes Framework – summary report of a meeting held by the primary care research groups of the Royal Statistical society (RSS) meeting Monday 21st November, Errol St. London.

Speakers

  • Nick Steel
  • David Reeves
  • Tim Doran

What I report here is my interpretation of what the speakers said combined with the audience discussion afterwards. The speakers are not responsible for anything they disagree with!

Background

The Quality and Outcomes Framework (QOF) is a system of payments to British GPs introduced in April 2004. In brief, GPs have a set of targets, mostly oriented towards certain chronic diseases, and they receive substantial payments for hitting these targets for a high proportion of relevant patients. There is evidence supporting the benefit of hitting each of these targets. The relevant patients are declared by each practice on a series of practice registers, for example one for people with diabetes, one for people with known heart disease, and so on.

For example, a GP practice would register patients at risk of coronary heart disease. If a recent blood pressure (within the last 15 months) is under 150/90 a patient has met that target. A GP where less than 40% of the relevant patients meet the target gets nothing. A GP where 70% of the relevant patients have met the target gets 30 points, and there is a sliding scale between these points.

The QOF contains four main components, known as domains. The four domains are: Clinical Domain, Organisational Domain, Patient Experience Domain and Additional Services Domain. Each domain consists of a set of achievement measures, known as indicators, against which practices score points according to their level of achievement. The 2010/11 QOF measured achievement against 134 indicators; practices scored points on the basis of achievement against each indicator, up to a maximum of 1,000 points.

  • clinical care: the domain consists of 86 indicators across 20 clinical areas (e.g. coronary heart disease, heart failure, hypertension) worth up to a maximum of 697 points.
  • organisational: the domain consists of 36 indicators (worth up to 167.5 points) across five organisational areas – records and information; information for patients; education and training; practice management and medicines management.
  • patient experience: the domain consists of three indicators (worth up to 91.5 points) that relate to length of consultations and to patient experience of access to GPs.
  • additional services: the domain consists of nine indicators across four service areas – cervical screening, child health surveillance, maternity service and contraceptive services.

The QOF gives an indication of the overall achievement of a surgery through a points system. Practices aim to deliver high quality care across a range of areas for which they score points. Put simply, the higher the score, the higher the financial reward for the practice. The final payment is adjusted to take account of surgery workload and the prevalence of chronic conditions in the practice’s local area (Source http://www.qof.ic.nhs.uk/). A practice getting maximum points across all targets gets about £ST130,000 extra a year. QOF payments come to GP practices through the Primary Care Trust with which they are affiliated. Overall the QOF costs about £ST1 billion a year.

The targets are measured as reported by the GPs from their IT systems. The PCTs do not have direct access to these, but an abstract is supplied to the PCTs by each practice.

Evaluations

The QOF was introduced in a big bang across the whole of the UK in April 2004. This was at the insistence of the BMA, who opposed a phased roll out. No baseline data were collected beforehand. In particular, no data on the proportion of GP practices meeting the various targets before the QOF started were collected. This was because of political pressure to do something fast about General Practice. The result was that no assessment was done of the QOF. A number of studies have now been done with ambivalent results. These were done using synthetic indicators made from existing GP data gathered for research purposes, for example, the GP Research Database. These show significant improvement in the indicators over time, but little evidence for most indicators, that the rate of improvement sped up when the QOF was introduced. There are indicators which rose faster in the first year of the QOF, but returned to the earlier trend of improvement from the second year on. An extensive systematic review of the QOF will be published in the next few weeks, and these results are from a preview given by one of the authors.
Pay-for-performance, and the cost-effectiveness of the QOF

The OQF is a pay-for-performance system. The research evidence on these systems is that they benefit productivity if the work being done is mechanical and repetitive. If the work involves even rudimentary cognition pay-for-performance does not work. There are case studies from Maine in the USA, where strict targets for diabetes care were set, and primary care doctors got a small sum of money ($10) for hitting these. This was modestly effective. Overall QOF has reduced inequalities in the indicators measured between practices in poor, middle income, and rich areas, but there is little evidence of markedly improved health for a very large investment. Nurses generally like it because it has improved their working patterns, and expanded their practice. Doctors tolerate it, claim not to like it, but feel that it has improved practice.

Issues

There is extensive gaming of the QOF. Most PCTs do not police it with any rigour. This is partly because to do so requires significant skills, and partly because they do not want to rock the boat. The major games relate to minor shading of results, for example the hypertension target is under 150/90, which is not exactly demanding. There seem to be quite a few patients coming ins at 148/88 and similar values.

The targets are not very tough (See Figure 1)! Most practices get maximum marks. There are a few exceptions, for example university student practices, where there are so few people with chronic diseases that it is nearly impossible to get a high score. This is dealt with by local informal special funding arrangements. The targets were originally set with no baseline data, so the maximum and minimum cut-offs were largely made up.

There is a practice called exception reporting. This allows a GP to declare that specific patients, for stated reasons, will not be counted. A good example is the use of beta-blockers after a heart attack. For some patients these are not advised because the patient has a condition which would be worsened by the drug, so these patients are ‘exception reported’ from the heart attack register. However, there is evidence that many practices use exception reporting to game the system, and that practices with high rates of exception reporting are likely to be of poor quality overall.

The targets have been revised somewhat since they were set up, but the targets for maximum payment have hardly been shifted at all, even though most practices are hitting these. A proposal that these targets should be reset to the 75th centile (the cut-off between the top quarter of practices, and the bottom three quarters), was rejected.

There is modest evidence that treatments and patients with diseases which were not incentivised by the QOF scheme have suffered.

Mental health is not covered by QOF at all, partly because it was too hard to come up with indicators.

Advice

The question of whether the speakers would recommend other countries to introduce the QOF was extensively discussed at the meeting. The final answer was ‘Yes-ish’. The recommendations below are what I would recommend before thinking about a QOF for Ireland.

  • Establish baseline values for any proposed indicators for a year or two before introducing anything.
  • Extract data directly from the GP EHR systems, with no practice-level intervention.
  • Monitor patterns of care closely.
  • Allow exception reporting, but monitor it closely.
  • Prohibit dumping difficult patients (which is common in the USA).
  • Set tough targets, based on evidence.
  • Possibly give full payment only for 100% compliance with the target, or for a 75th centile performance, revised each year.
  • Revise these targets regularly
  • Look at the Kaiser system, where practices are evaluated on targets, but do not know till the end of the relevant year what those targets were. This makes gaming pointless, and encourages a wide view of practice improvement

Paying for third level – a fairer alternative to fees?

There are two widely held beliefs about paying for third level. The first is that universities cannot continue to be largely funded from general taxation, as at present. The second is that Irish universities are underfunded, and this is why they have slipped down the league tables. As a result there is a modest panic about paying for universities.

Prof. Michael Murphy, the president of UCC, in an interview with Sean Kelly (Irish Times, Nov 12th), blew on the flames, saying that “Student fees of at least €4,500-€5,000 per year are necessary to maintain the quality of higher education in Ireland” and “as graduates gained a significant income premium from their degree, it was right that those who can afford to pay fees be asked to make a more significant contribution.”

Prof. Murphy is of course right about the income premium. OECD figures for Ireland up to 2004, suggest that graduates, as compared with someone who only completes secondary school, can earn 30% to 70% more. More recent Irish data are not available, but the more recent figures for other OECD states are similar. These figures are for extra earnings across a working lifetime (20 to 64).

How do we pay for third-level education now? It’s hard to get a good picture of the finances of the whole sector, and there are big differences between institutions, especially between the ITs and the universities. The HEA pay the bulk of the costs, apparently 65% to 90%+, depending on the institution. They paid just under €1.35 billion in ordinary grants to all third level institutions in 2009. Of this €769 million went to the university sector. Student ‘registration fees’, now running at €2,000 a year, are said to pay about half the costs of ‘direct student services’, a rather vague term.

Income from research grants, campus companies, patent licensing, conferences, accommodation, catering, rental income, and other university enterprises, makes up the rest. It is not clear whether these other activities even cover their own costs, as there are weak costing and management information systems in the sector. In any event, the bulk of funding comes direct, as a per-capita grant, from HEA, i.e. from general taxation, and so, non-graduates subsidize graduates. This seems unfair, as Prof. Murphy rightly suggests.

The return of fees is touted as a solution. Our neighbours in England have just brought in fees of £9,000 a year, and got a 12 % drop in applications to third level. They do have an efficient scheme of student loans, where repayments are deducted directly from salaries, once the student earns above a modest level, and interest rates are subsidized. The US has an odd system, with very high fees, student loans, and default rates running at 10% overall, and up to 80% in the worst of the private colleges (think mortgage misselling tactics applied to degrees). Most other EU countries have either no fees, or modest fees. We have nothing.

What would bringing in fees do? This would help to fix the funding gap in the sector, although to do this they would need to be closer to €10,000 or €12,000, than the figures of €3,000 to €4,000 being discussed. This would throw much of the burden on current students, and their families. Students entering in the next few years seem likely to be hit for fees. Those whose families can afford it, will graduate with few debts. Those whose families cannot, will either not go, as the recent English experience suggests, or will take on significant debts to do so.

Assuming we develop a system more like England than the US, these students will then repay their loans over 20 to 30 years, by direct deduction from their salaries. Older graduates, like me, would pay nothing for our education. We have already benefited from third level education, which received significant public subsidies, and, in many cases, our parents paid the fees.

Wouldn’t it be better to think of a graduate tax? The effect of this would be similar to a loan repayment deducted from salary, but a lot more people would contribute, so it would spread the costs more fairly. Every graduate earning above a set level would pay, and it would be deducted directly, like PRSI, and the universal social charge. No costly new systems would be required, and the administrative costs of tracking 40,000 graduates a year, across the world, for twenty or thirty years, would be avoided. There would be a once-off cost for registering graduates, but this would not be too hard. The tax would cover all graduates, whether of Irish, or foreign, third level institutions. This would ensure that all graduates would contribute to the costs of third level education.

This might be seen to be unfair to older graduates, as students paid tuition fees up to 1996. However, most graduates under the age of sixty have benefited from significant public subsidies to third level, which have been a feature of the Irish system since the late 1960’s, so it seems fair that we should contribute now.

What rate of tax would be needed to raise €2 billion a year? A rough estimate can be made. In 2010 the top 1% of income earners paid 25% of income tax, the top 8% paid 60% of of the tax, and the top 21% 83% of the tax. In 2010 it was estimated, by the Dept. of Finance, that a 1% rise in the health levy would raise €600m. If we assume the same ratio applies, then 21% of the population paying a 3% levy would raise €1.4 billion.

This suggests that a graduate tax paid at zero for low earning graduates (e.g. under €30,000), 2% for lower earners (e.g under €50,000), and 3% for higher earners would pay for Irish third level into the foreseeable future. These are not trivial sums, but they are not especially onerous either. In the UK new graduates now pay up to 9% of their income for up to 25 years to repay their loans.

There is a major effort to bounce us into bringing back student fees, but it’s not too late to change our minds. There are better, cheaper, and fairer options, and a graduate tax is one.

Scandal Exposed in Major Study of Autism and Mercury – Not!

Irish Autism Action pointed me to a piece picked up by Yahoo news from October 25th, written by staff from the Coalition for Mercury-Free Drugs (CoMeD) . It is a critique of sorts, of a paper published in Pediatrics, in 2003, on “Thimerosal and the Occurrence of Autism: Negative Ecological Evidence from Danish Population-Based Data” which may be read (free, thanks to the AAP) here.

The paper first.

It is written by seven Danish scientists. Poul Thorsen, of whom more later, is the fourth author. To understand the importance of this paper, you need to know that Denmark has one of the best health information systems in the world. The vast majority of contacts between Danish residents and their health services are recorded, with a unique identifier, which allows one to identify the first contact, and to avoid counting the same people twice. This system is reliable, not infallible, but far better than equivalent systems in the UK. There are no such systems in Ireland.

Thiomersal or Thimerosal

Thimerosal or Thiomersal is a chemical, containing mercury. In the body it is converted to ethylmercury. A lot is known about the toxicity of a related compound, methylmercury, which caused the terrible tragedy of Minanmata disease. Much less is known about the toxicity of ethylmercury, but it is felt to be wise to assume that it too is toxic. Thiomersal has been used as a preservative in multi-dose vaccine vials for many years, but, because of concerns about possible toxicity, is now largely phased out in the EU, and the USA. There has been an extensive controversy about the links between thiomersal exposure in infancy and childhood, and the development of childhood autism.

What the Danish paper adds

The Danes analyzed first diagnoses of autism, (the ‘incidence’ of autism in the jargon of my profession), in children, aged between 2 and 10, and diagnosed between 1971 and 2000. Thiomersal was used as a vaccine preservative up to March 1992. They found just under 1000 cases of autism diagnosed in Denmark over this period. The rate of occurrence of autism is shown in Figure 1 from their paper.
Figure 1 for Madsen's paper form Pediatrics

Briefly this shows autism rates low, and steady from 1971 to about 1987 or 1988. After this autism rates rise steadily, first in the youngest children (ages 2 to 4), then in the next group (ages 5 to 6), and finally in the oldest children, (ages 7 to 9). These are children newly diagnosed with autism. Rates are beginning to dip in the two older groups in 2000, as compared with 1999. The vertical line on the graph, in 1992, shows when thimerosal containing vaccines stopped being produced in Denmark.

What does this mean?

This graph shows a significant increase in the incidence of autism. This is similar to that recorded in many other countries at the same time. No-one really knows the cause of this increase. It is widely believed by experts that a big part of it is due to a mix of better access to child psychiatric services, better and more rigorous diagnosis, and to the broadening of autism diagnoses into the Autism Spectrum disorders (see the recent review by Fombonne). This does not mean that some part of the increase may not be real, but it is widely believed that most of it reflects other things besides a real increase in autism.

There are also specific factors which very likely contributed to the rise in Denmark – and are mentioned in their paper. First they moved classification systems, for autism, and all other diagnoses, in 1994, from ICD-8 to ICD-9. Secondly, outpatient diagnoses were accepted from 1995, but the authors examined inpatient diagnoses separately, and found the same pattern in these data. They did not show the inpatient only data, but this is a common practice, enforced by editors, when there is pressure on space in published papers.

It’s easier to say what this does not mean. If you believe that thiomersal in vaccines caused any significant proportion of cases of autism, you should no longer believe this. I can think of no mechanism by which removing a major cause of autism, could lead to a rise in the incidence of the syndrome. The dip in incidence in 2000 is interesting, though modest, but cannot be related to the removal of mercury from vaccines eight years earlier.

And the scandal?

The COMED story makes two specific allegations

  • The authors suppressed data from 2001, which showed a further fall in autism prevalence. CDC officials knew about this.
  • One of the authors, Poul Thorsen, has been sacked by his employers, Aarhus university, and is under investigation for large scale financial fraud against the US government, related to grants for autism research.

So, let’s take these in order.

Suppressing data?

First, the fall in autism incidence in 2001. As I mentioned the paper shows data up to 2000, with some indication that incidence is falling in 2000 compared with 1999. The email mentioning the 2001 data is largely redacted, so it’s hard to make much sense out of it. But if we suppose that the rate had indeed fallen further, what implication does this have for the published paper? The paper shows clearly a large increase in autism incidence starting a few years before thimerosal was removed in 1992, and continuing at the same pace for at least six further years (up to 1999). Removing thimerosal in 1992 cannot affect autism rates in children aged 5 to 6 in 2000, it is simply impossible. Neither can it affect rates in 2001.

Many scientists in my discipline struggle with the ‘just one more year’ problem. Data are seldom complete on December 31st. It takes time to clean data, to check it, and to make it available. It then takes more time to get it, analyze it, and write a paper about it. By then the next year’s data may have come out. Most of us decide, usually after a quick look at the new data, if we want to hold up the paper for another six months while we repeat the process. At some point, you just have to publish. This is life, not a scandal!

Poul Thorsen

First, a declaration. I know Dr. Thorsen. I have met him two or three times, when I was working on a study to design methods to estimate the prevalence of autism across Europe. If, as alleged, he has defrauded scientific funds paid by the US, to Aarhus university for autism research, I do not defend him. However, even if every charge in the grand jury indictment were proved, I do not see that this ought to affect how we read the paper.

Thorsen was one of seven authors, and was neither the first, nor the last, the two most significant slots in our business. The paper was not funded by his research funds, and had no organic connection with them. No allegation of scientific fraud has been made against him. An ancient, and disreputable debating strategy is the ‘ad hominem’ attack – I can’t attack what you say, so instead I will attack you. Scientists didn’t attack Andrew Wakefield personally, they attacked his paper, because the conclusions he presented, did not follow from the evidence he presented. Similarly, an attack on Poul Thorsen, justified, or otherwise, does not speak to the correctness, or otherwise, of this paper.

One swallow does not a summer make

One of the mottoes of my profession is ‘Never believe one study’. Any single study can come up with a completely wrong answer, no matter how well it is done, no matter how skilled, conscientious, and honest, those who work on it. In the case of thiomersal and autism, there is a small flock of good studies, all saying pretty much the same thing – there is no evidence for a causal link between autism and thiomersal exposure. The paper by Madsen et al. is only one of the flock, but it is flying in the same direction as the others.

Is thiomersal safe?

Many countries still need to use thiomersal, because it is cheap, effective, and allows vaccines to be used safely in hot climates. We don’t use it, because we can afford not to. José G. Dórea, from the Universidade de Brasilia in Brazil, has a very thoughtful review article assessing the evidence, and concludes that, if we have to use it, and Brazil does, we need to be very careful with thiomersal in babies. He finds no good evidence of harm, but this not the same as evidence of safety. If thiomersal has to be used, there are steps that can be taken to protect and develop babies’ brains in other ways, and these ought to be adopted widely in any event. I would fully agree with him.

Scandal?

I do not speak for the Coalition for Mercury-Free Drugs, and I make no assertion about their objectives in writing this story. However, they are wrong. There is no scandal here, and the problems they identify with the paper are acknowledged and addressed by the authors. There is a
long report, sent by CoMED, to Béatrice Sloth of the Secretariat for the Danish Committees on Scientific Dishonesty, which forms, I think, the basis for the press release. I have read it carefully, and I see no evidence of scientific wrong-doing in the report. There are critcisms of the paper, but the major scientific issues raised, the change in diagnosis codes, and the inclusion of outpatient care, are addressed in the paper by the authors. This is not stated in the report.

All in all, this is a non-story. It is typical of too many others, written by people who seem to only half understand the science, and seemingly intended to strike fear into the hearts of parents. Some children died, and many more were unnecessarily brain damaged, by the original measles vaccine scare. Is that not enough? Isn’t it time to stop?