Archive for the ‘UK parochial’ Category

Simple maths of a fairer USS deal


In yesterday’s post I showed a graph, followed by some comments to suggest that future USS proposals with a flatter (or even increasing) “percent lost” curve would be fairer (and, as I argued earlier in my Robin Hood post, more affordable at the same time).

It’s now clear to me that my suggestion seemed a bit cryptic to many (maybe most!) who read it yesterday.  So here I will try to show more specifically how to achieve a flat curve.  (This is not because I think flat is optimal.  It’s mainly because it’s easy to explain.  As already mentioned, it might not be a bad idea if the curve was actually to increase a bit as salary levels increase; that would allow those with higher salaries to feel happy that they are doing their bit towards the sustainable future of USS.)

Flattening the curve

The graph below is the same as yesterday’s but with a flat (blue, dashed) line drawn at the level of 4% lost across all salary levels.


I drew the line at 4% here just as an example, to illustrate the calculation.  The actual level needed — i.e, the “affordable” level for universities —  would need to be determined by negotiation; but the maths is essentially the same, whatever the level (within reason).

Let’s suppose we want to adjust the USS contribution and benefits parameters to achieve just such a flat “percent lost” curve, at the 4% level.  How is that done?

I will assume here the same adjustable parameters that UUK and UCU appear to have in mind, namely:

  • employee contribution rate E (as percentage of salary — currently 8; was 8.7 in the 12 March proposal; was 8 in the January proposal)
  • threshold salary T, over which defined benefit (DB) pension entitlement ceases (which is currently £55.55k; was £42k in the 12 March proposal; and was £0 in the January proposal)
  • accrual rate A, in the DB pension.  Expressed here in percentage points (currently 100/75; was 100/85 in the 12 March proposal; and not relevant to the January proposal).
  • employer contribution rate (%) to the defined contribution (DC) part of USS pension.  Let’s allow different rates C_1 and C_2 for, respectively, salaries between T and £55.55k, and salaries over £55.55k. (Currently C_1 is irrelevant, and C_2 is 13 (max); these were both set at 12 in the 12th March proposal; and were both 13.25 in the January proposal.)

I will assume also, as all the recent proposals do, that the 1% USS match possibility is lost to all members.

Then, to get to 4% lost across the board, we need simply to solve the following linear equations.  (To see where these came from, please see this earlier post.)

For salary up to T:

 (E - 8) + 19(100/75 - A) + 1] = 4.

For salary between T and £55.55k:

  -8 + 19(100/75) - C_1 + 1 = 4.

For salary over £55.55k:

 13 - C_2 = 4.

Solving those last two equations is simple, and results in

 C_1 = 14.33, \qquad C_2 = 9.

The first equation above clearly allows more freedom: it’s just one equation, with two unknowns, so there are many solutions available.  Three example solutions, still based the illustrative 4% loss level across all salary levels, are:

 E=8, \qquad A = 1.175 = 100/85.1

 E = 8.7, \qquad A = 1.21 = 100/82.6

 E = 11, \qquad A = 100/75.

At the end here I’ll give code in R to do the above calculation quite generally, i.e., for any desired percentage loss level.  First let me just make a few remarks relating to all this.


Choice of threshold

Note that the value of T does not enter into the above calculation.  Clearly there will be (negotiable) interplay between T and the required percentage loss, though, for a given level of affordability.

Choice of C_2

Much depends on the value of C_2.

The calculation above gives the value of C_2 needed for a flat “percent lost” curve, at any given level for the percent lost (which was 4% in the example above).

To achieve an increasing “percent lost” curve, we could simply reduce the value of C_2 further than the answer given by the above calculation.  Alternatively, as suggested in my earlier Robin Hood post, USS could apply a lower value of C_2 only for salaries above some higher threshold — i.e., in much the same spirit as progressive taxation of income.

Just as with income tax, it would be important not to set C_2 too small, otherwise the highest-paid members would quite likely want to leave USS.  There is clearly a delicate balance to be struck, at the top end of the salary spectrum.

But it is clear that if the higher-paid were to sacrifice at least as much as everyone else, in proportion to their salary, then that would allow the overall level of “percent lost” to be appreciably reduced, which would benefit the vast majority of USS members.

Determination of the overall “percent lost”

Everything written here constitutes a methodology to help with finding a good solution.  As mentioned at the top here, the actual solution — and in particular, the actual level of USS member pain (if any) deemed to be necessary to keep USS afloat — will be a matter for negotiation.  The maths here can help inform that negotiation, though.

Code for solving the above equations

## Function to compute the USS parameters needed for a
## flat "percent lost" curve
## Function arguments are:
## loss: in percentage points, the constant loss desired
## E: employee contribution, in percentage points
## A: the DB accrual rate
## Exactly one of E and A must be specified (ie, not NULL).
## Example calls:
## flatcurve(4.0, A = 100/75)
## flatcurve(2.0, E = 10.5)
## flatcurve(1.0, A = 100/75)  # status quo, just 1% "match" lost

flatcurve <- function(loss, E = NULL, A = NULL){

    if (is.null(E) && is.null(A)) {
        stop("E and A can't both be NULL")}
    if (!is.null(E) && !is.null(A)) {
        stop("one of {E, A} must be NULL")}

    c1 <- 19 * (100/75) - (7 + loss)
    c2 <- 13 - loss

    if (is.null(E)) {
        E <- 7 + loss - (19 * (100/75 - A))

    if (is.null(A)) {
        A <- (E - 7 - loss + (19 * 100/75)) / 19

return(list(loss_percent = loss,
            employee_contribution_percent = E,
            accrual_reciprocal = 100/A,
            DC_employer_rate_below_55.55k = c1,
            DC_employer_rate_above_55.55k = c2))

The above function will run in base R.

Here are three examples of its use (copied from an interactive session in R):

###  Specify 4% loss level, 
###  still using the current USS DB accrual rate

> flatcurve(4.0, A = 100/75)
[1] 4

[1] 11

[1] 75

[1] 14.33333

[1] 9

###  This time for a smaller (2%) loss, 
###  with specified employee contribution

> flatcurve(2.0, E = 10.5)
[1] 2

[1] 10.5

[1] 70.80745

[1] 16.33333

[1] 11

### Finally, my personal favourite:
### --- status quo with just the "match" lost

> flatcurve(1, A = 100/75)
[1] 1

[1] 8

[1] 75

[1] 17.33333

[1] 12

© David Firth, March 2018

To cite this entry:
Firth, D (2018). Simple maths of a fairer USS deal. Weblog entry at URL

USS proposals: Tail wagging the dog?


Update on 16 March: There’s now a follow-up post to this one, which gives more detail on how (mathematically) to achieve a fairer sharing-out of whatever level of USS member pain might ultimately be deemed necessary.  See Simple maths of a fairer USS deal (but ideally only after reading the necessary background, below!).

In response to my previous post, “Latest USS proposal: Who would lose most?“, someone asked me about doing the same calculation for the USS JNC-supported proposals from January.  For a summary of those January proposals and my comments about their fairness, please see my earlier post “USS pension scheme and fairness“.

Anyway, the calculation is quite simple, and it led to the following graph.  The black curve is as in my previous post, and the red one is from the same calculation done for the January USS proposal.

lost-comparisonThe red curve shows just over 5% effective loss of salary for those below the current £55.55k USS threshold, and then a fairly sharp decline to less than 2% lost at the salaries of the very highest-paid professors, managers and administrators.  Under the January proposals, higher-paid staff would contribute proportionately less to the “rescue package” for USS — less, even, than under the March proposals.  (And if the salary axis were to be extended indefinitely, the red curve would actually cross the zero-line: that’s because in the January proposals the defined-contribution rate from employers would actually have increased from (max) 13% to 13.25%.)

In terms of unequal sharing of the “pain”, then, the January proposal was even worse than the March one.

At the bottom here I’ll give the R code and a few words of explanation for the calculation of the red curve above.

But the main topic of this post arises from a remarkable feature of the above graph! At the current USS threshold salary of £55.55k, the amount lost is the same — it’s 5.08% under both proposals.  Which led me to wonder: is that a coincidence, or was it actually a (pretty weird!) constraint used in the recent UUK-UCU negotiations?  And then to wonder: might the best solution (i.e., for the same cost) be to do something that gives a better graph than either of the two proposals seen so far?

Tail wagging the dog?

The fact that the loss under the March proposal tops out at 5.08%, exactly (to 2 decimals, anyway) the same as in the January proposal, seems unlikely to be a coincidence?

If it’s not a coincidence, then a plausible route to the March proposal, at the UUK-UCU negotiating table, could have been along the lines of:

How can we re-work the January proposal to

  • retain defined benefit, up to some (presumably reduced) threshold and with some (presumably reduced) accrual rate,

while at the same time

  • nobody loses more than the maximum 5.08% that’s in the January proposal
  • the employer contribution rate to the DC pots of high earners is not reduced below the current standard (i.e., without the “match”) level of 12%


Those constraints, coupled with total cost to employers, would lead naturally to a family of solutions indexed by just two adjustable constants, namely

  • the threshold salary up to which DB pension applies (previously £55.55k)
  • the DB accrual rate (previously 1/75)

— and it seems plausible that the suggested (12 March 2018) new threshold of £42k and accrual rate of 1/85 were simply selected as the preferred candidate (among many such potential solutions) to offer to UUK and UCU members.

But the curve ought to be flat, or even increasing!

The two constraints listed as second and third bullets in the above essentially fix the position of the part of the black curve that applies to salaries over £55.55k.  That’s what I mean by “tail wagging the dog”.  Those constraints inevitably result in a solution that implies substantial losses for those with low or moderate incomes.

Once this is recognised, it becomes natural to ask: what should the shape of that “percentage loss” curve be?

The answer is surely a matter of opinion.

Those wishing to preserve substantial pension contributions at high salary levels, at the expense of those at lower salary levels, would want a curve that decreases to the right — as seen in the above curves for the January and March proposals.

For myself, I would argue the opposite: The “percent lost” curve should either be roughly constant, or might reasonably even increase as salary increases.  (The obvious parallel being progressive rates of income tax: those who can afford to pay more, pay more.)

I had made a specific suggestion along these lines, in this earlier post:

The details of any solution that satisfies the “percent loss roughly constant, or even increasing” requirement clearly would need to depend on data that’s not so widely available (mainly, the distribution of all salaries for USS members).

But first the principle of fairness needs to be recognised.  And once that is accepted, the constraints underlying future UUK-UCU negotiations would need to change radically — i.e., definitely away from those last two bullets in the above display.

Calculation of the red curve

In the previous post I gave R code for the black curve.  Here is the corresponding calculation behind the red curve:

sacrifice.Jan <- function(salary) { # salary in thousands
    old_threshold <- 55.55
    s <- salary

## sacrifice arising from income up to old_threshold
    s2 <- min(s, old_threshold)
    r2 <- s2 * (19/75 + 1/100 - (13.25 + 8)/100)

## sacrifice (max) arising from income over the old threshold
## -- note that this is negative
    r3 <- (s > old_threshold) * (s - old_threshold) * 
                (13 - 13.25)/100

    return(r2 + r3)

## A vector of salary values up to £150k
salaries <- (1:1500) / 10

## Compute percent of salary that would be lost, 
## at each salary level
sacrifices <- 100 * sapply(salaries, sacrifice.Jan) / salaries

In essence:

  • salary under £55.55k would lose the defined benefit (that’s the 19/75 part) and the 1% “match”, and in its place would get 21.25% as defined contribution.  The sum of these parts is the computed loss r2.
  • salary over £55.55k would gain the difference between potential 13% employer contribution and the proposed new rate of 13.25% (that’s the negative value r3 in the code).

© David Firth, March 2018

To cite this entry:
Firth, D (2018). USS proposals: Tail wagging the dog?. Weblog entry at URL


Latest USS proposal: Who would lose most?


Update on 16 March: After reading this post, you might perhaps be interested in these follow-ups:

Update, 14 March: Some details in the original post yesterday were not quite right, and so the graph/numbers that appear in the now-corrected version below are different in detail from yesterday’s.  But the overall picture is unchanged.  (If you really want to know about those changes in the detail, please see my note in Appendix 2 at the bottom of the post about that.)

Yesterday (March 12th) the UUK/UCU negotiations at ACAS concluded with an agreement document.

In this post I’ll look at the numbers in those proposed interim changes to the Universities Superannuation Scheme, to work out how much money would effectively be lost by USS members at each salary level.

This is inevitably a fairly rough calculation, but its results don’t really demand more precision.  The picture is very clear: the cost of “saving” USS would be felt most by USS members with low or moderate incomes.

The effective marginal rates at which money is lost by members are (as calculated below):

  • 4.7% on salary up to £42k
  • 6.3% on salary between £42k and the current USS threshold salary of £55.55k
  • 1.0% (at most) on salary over £55.55k

This translates into the following relationship between salary and the percentage of total salary lost:


The two “kinks” in that graph reflect the discontinuities in marginal rates, at £42k and at £55.55k.

The vertical lines drawn in green are current full-time pay grades at a typical university (with no London allowance or other extras): grade 6 is the pay of many Research Associates and Teaching Fellows, for example; grade 7 is the pay of most Lecturers; grade 8 is the pay of Senior Lecturers and Readers; and grade 9 is the pay of Professors and other senior staff.  (I have mentioned only academic and research staff here, but the same grades apply also to administrative and technical staff in UK universities.)

The long decay to the right continues indefinitely, ultimately approaching an asymptote at 1% lost, i.e., for those with absolutely stratospheric salaries (if such people are actually members of USS, still, that is — though I would guess that many are not).

In the rest of this post I’ll give the details of the calculation that leads to the above numbers and graph.  (For people who prefer a list of numbers to a graphical display, I have also added the numbers as an Appendix at the bottom of this post.)

Just here, though, let me again comment on how unfair this “remedy” would be.  The unfairness should be obvious from the above graph: those who are paid most, and would stand to benefit most from being in USS, would contribute least, in percentage terms, in this proposed move towards the future sustainability of USS.  For a more general view on this unfairness, see also my previous two posts in this “USS” category:

The calculation

It suffices to consider salaries in three distinct bands.  In each salary band, we can calculate how much is lost, per unit of salary.

The following code in R reproduces the graph drawn above.  A brief explanation is then given, beneath the displayed code.

## This code runs in base R.

## Function to compute the amount that would be lost annually (£k)
## at any given salary level
sacrifice <- function(salary) { # salary in thousands
    old_threshold <- 55.55
    new_threshold <- 42
    s <- salary

## sacrifice arising from income up to the new threshold
    r1 <- min(s, new_threshold) * ((8.7 - 8)/100 +
                                    19 * (1/75 - 1/85) +

## sacrifice arising from income between the thresholds
    s2 <- (s > new_threshold) * (min(s, old_threshold) - 
    r2 <- s2 * ((8.7 - 8)/100 + (19/75 - (12 + 8.7)/100) + 1/100)

## sacrifice (max) arising from income over the old threshold
    r3 <- (s > old_threshold) * (s - old_threshold) * (1/100)

    return(r1 + r2 + r3)

## A vector of salary values up to £150k
salaries <- (1:1500) / 10

## Compute percent of salary that would be lost, 
## at each salary level
sacrifices <- 100 * sapply(salaries, sacrifice) / salaries

## Plot the result
svg(file = "lost.svg", width = 8, height = 4)
plot(salaries, sacrifices, type = "l",
 xlab = "salary (thousands)", ylab = "percent lost",
 main = "Percent of salary lost under UUK-UCU agreement 2018-03-12")
abline(v = c(29, 39, 48, 61), col = "green")
text(x = c(34, 44, 54, 75), y = 2.8,
 labels = c("6", "7", "8", "9"), col = "green")

Band 1: Salary up to £42k

Most contributions from this part of salary go to the “defined benefit” part of USS. The new proposal would see 8.7% of member’s salary up to £42k going in to this, as opposed to 8.0% at present. The return (i.e., the value of the defined-benefit pension) can readily be calculated using the standard HMRC formula, the one that is used for Annual Allowance purposes. Under current USS, the value of this part is 19 times (s/75), where s is either £42k or the member’s salary if the salary is less than £42k. Under yesterday’s proposals, the value of this part would fall to 19 times (s/85). Under yesterday’s proposals, USS members would also lose the possibility to add 1% “matching” employer contribution to an additional, defined-contribution pension pot. The amount lost to each member, relating to salary in this first band, is then the sum of the additional contribution made and the amount of pension value lost: that is r1 in the above code.

Band 2:  Salary between £42k and £55.55k

Now, for salaries greater than £42k, let s2 be the smaller of (salary minus £42k) and (£55.55k minus £42k). Then current USS has members contributing 8% of s2 in the defined-benefit part, for a return of 19 times s2/75. Yesterday’s proposal would change the contribution to 8.7% of s2, for a return of s2 times (12% + 8.7%). And again, the possibility of 1% matching employer contribution to the defined-contribution pot would be lost. The amount lost to each member, relating to salary in this second band, is again just the sum of the additional contribution made and the amount of pension value lost: that is r2 in the above code.

Band 3: Salary over £55.55k

Relating to salary above the current £55.55k threshold, the loss would be limited to loss of the 1% matching employer contribution.  This is computed as r3 in the above code. (In practice this will be an upper bound on what is lost.  Those USS members with the very highest salaries are likely also to face issues relating to the HMRC Annual Allowance and Lifetime Allowance limits, in which case the loss of the matching employer contribution could be worth substantially less than 1% to them.)


I have reproduced the full calculation here, with code, because I found the result of the calculation so shocking!  If anyone reading this thinks I have made a mistake in the calculation, please do let me know. If it is correct — and right now I have no reason to suspect otherwise — then I confess I’m alarmed that this is actually being proposed as a potential solution, even as an interim solution for the next 3 years, to the perceived problems with USS.  It shakes my faith in those who have been involved in negotiating it.  With seemingly intelligent people on both sides of the table, how could they possibly come up with something as bad as this?

© David Firth, March 2018

To cite this entry: Firth, D (2018). Latest USS proposal: Who would lose most?  Weblog entry at URL

Appendix 1: A tabular view of what’s in the graph

## Make a table for anyone who wants more detail than the graph
salary <- c(10:55, 55.55, 56:100, 150)
percent_lost <- round(100 * sapply(salary, sacrifice) / salary, 2)
salary <- 1000 * salary
my_table <- data.frame(salary, percent_lost)

That’s the code for making a little table, showing the same numbers as those in the above graph.

Here is the resulting table:

salary    %
 10000 4.68 -- I started the table at £10k for no good reason
 11000 4.68
 41000 4.68
 42000 4.68 -- the proposed new threshold
 43000 4.72
 44000 4.76
 45000 4.79
 46000 4.82
 47000 4.86
 48000 4.89
 49000 4.92
 50000 4.94
 51000 4.97
 52000 5.00
 53000 5.02
 54000 5.05
 55000 5.07
 55550 5.08 -- current USS threshold, highest % of salary lost
 56000 5.05
 57000 4.98
 58000 4.91
 59000 4.84
 60000 4.78
 61000 4.72
 62000 4.66
 63000 4.60
 64000 4.54
 65000 4.49
 66000 4.44
 67000 4.39
 68000 4.34
 69000 4.29
 70000 4.24
 71000 4.19
 72000 4.15
 73000 4.11
 74000 4.07
 75000 4.02
 76000 3.98
 77000 3.95
 78000 3.91
 79000 3.87
 80000 3.84
 81000 3.80
 82000 3.77
 83000 3.73
 84000 3.70
 85000 3.67
 86000 3.64
 87000 3.61
 88000 3.58
 89000 3.55
 90000 3.52
 91000 3.49
 92000 3.47
 93000 3.44
 94000 3.41
 95000 3.39
 96000 3.36
 97000 3.34
 98000 3.31
 99000 3.29
100000 3.27
150000 2.51 -- possibly there are even some salaries this high?!

Appendix 2: Details of the update made on 14 March

Many thanks to all who gave feedback on the original posting, yesterday (13 March).

In response to that feedback, I made two substantive changes to the calculation.  This Appendix gives details of those changes, for those who are interested (and for the record).

Neither change affects the story qualitatively: only the detailed numbers have changed a bit.

Change 1: Use of HMRC multiplier 19 rather than 23

The HMRC calculations for Annual Allowance and Lifetime Allowance purposes are different in detail: the former uses a multiplier of 19 times pension to value USS defined benefits, while the latter uses 23 (i.e., in place of 19).  In yesterday’s post I had used 23.  The updated figures calculated above use multiplier 19 instead.

Mainly I decided to use the smaller figure as it’s a bit more conservative, in relation to the value lost through the proposed reduction of defined benefits.  (I certainly don’t want to be accused of bias in the other direction, through having picked the larger multiplier.)

The effect on the calculated numbers is mainly to reduce the height of the “spike” that appears in the graph, around the £55k salary level.  The spike is still there; it’s just a bit smaller.

My friend Jon commented that the actual value of a defined-benefit pension is harder to quantify than the HMRC formula would suggest — and that it’s likely to be dependent on age and perhaps other factors.  This is undoubtedly true, and certainly I would not suggest that anyone should use the above numbers for their own financial planning!  Rather, the aim here was (only) to show through a simple, transparent calculation how the losses arising from current proposals would differ — in rough, average terms — between pay levels.

Since writing my post yesterday I found that I am not alone in having done a calculation like this: see also (and maybe there are others too?).

Change 2: Inclusion of the USS “Match” at all salary levels

Several people pointed out to me that the USS “Match” possibility is available at all salary levels.  So it’s a benefit that would be lost at all salary levels, under the 12 March agreement.  In yesterday’s post I had taken it into account only at salaries over £55.55k: that (relatively minor) error is now corrected, in the revised figures shown above.


Mathematical operations with the Normal distribution [a re-blog from corp.ling.stats]


Following a recent Twitter exchange with Ian Dryden, I was thinking I’d write something about the risk of default for the USS pension scheme. But then I came across this other new post at corp.ling.stats — which has already done what I was intending!


This post is a little off-topic, as the exercise I am about to illustrate is not one that most corpus linguists will have to engage in.

However, I think it is a good example of why a mathematical approach to statistics (instead of the usual rote-learning of tests) is extremely valuable.

Case study: The declared ‘deficit’ in the USS pension scheme

At the time of writing nearly two hundred thousand university staff in the UK are active members of a pension scheme called USS. This scheme draws in income from these members and pays out to pensioners. Every three years the pension is valued, which is not a simple process. The valuation consists of two aspects, both uncertain:

  • to value the liabilities of the pension fund, which means the obligations to current pensioners and future pensioners (current active members), and
  • to value the future asset value of the pension fund…

View original post 2,753 more words

Future USS: Robin Hood can help?


Update on 16 March: After reading this post, you might perhaps be interested in some of these follow-ups:

In this post I’ll follow up on the previous one, USS pension scheme and fairness, in the light of this week’s UCU proposals (proposals which provide a basis for renewed talks with Universities UK about the future of USS).

My purpose here is to suggest that a fairer solution could also be more affordable — in that it would help fund the defined-benefit section of USS, but would also reduce (and might perhaps even eliminate?) the upward pressure on employer and employee contribution rates.

1. The current UCU proposals

The following summary excerpt is taken from


The University and College Union (UCU) deserves all of the many congratulations it is currently getting, for having produced such a proposal this week — in particular, a proposal that has at last persuaded the employers (Universities UK) to take part in further talks about the future of USS.

That said: I should admit to feeling disappointed by the 3rd and 4th bullet points above!  USS members would all be getting a slightly worse pension, in return for an appreciable increase in the contribution rates (both employee and employer contribution rates).

More crucially — if I have understood it correctly —  the implied loss of pension (as a percentage of contributions paid) would be greatest for those USS members whose salary is £55k or lower.  Higher-paid members of USS would be required to make a smaller sacrifice — substantially smaller, in the case of those with the very highest salaries — relative to the overall size of their pensions.

Recognising this unfair distribution of the pain, and thinking about how best to fix it, leads naturally to an additional device that could make the proposed changes fairer and more easily affordable.

2.  Robin Hood to the rescue

The current USS setup has a slightly progressive aspect, which is that the 18% employer contribution for salary over the £55k threshold pays only 12% into the member’s defined-contribution (DC) pension pot.  The remaining 6% therefore helps to support the running costs of USS, and (mainly, it seems safe to presume) the defined-benefit part of USS.

The suggestion I want to make here is that future USS should become more progressive — that is, USS should move further in that same, progressive direction.  (As mentioned in the previous post, the current USS proposal would increase that 12% figure to 13.25%, thereby making the scheme appreciably less progressive.  And as I have argued in that previous post, that seems completely indefensible.)

The specific suggestion:

  1. For salary between the current £55k threshold and some specified higher threshold, pay x% from the employer contribution into the employee’s DC pot.  The value of x could be left at 12, for example.
  2. For salary above the higher of the two thresholds, pay y% from the employer contribution into the employee’s DC pot, where y is smaller than x.

Even if x remains at 12, such a device would allow (through suitable choice of the higher threshold and the value of y) substantial savings to be made from the employer contributions on higher salaries — savings which could then be used directly to support the threatened, defined-benefit part of USS.

The precise arithmetic on this becomes possible only with detailed knowledge (not available to me) of the distribution of USS-member salaries over £55k.  Some general considerations on the choice of the higher threshold, and of y, are:

  • The higher threshold clearly should not be so high as to make the resultant savings too small to be of much consequence.
  • The value of y probably needs to be at least as high as the employee contribution rate (currently 8%), otherwise it could become unattractive for those with the highest salaries to remain in USS.

Illustrative example:

Just to give a sense of what might be possible through this device.  Let’s suppose:

  • salary between £55k and £75k gets 12% into the DC pot, from the employer contribution — as now.
  • salary over £75 gets 8% (i.e., the current employee contribution rate) instead of 12%.

Then the amount saved, which could then be used to support the defined-benefit part of USS, would be 4% of all member salaries over £75k.

3. Conclusion

It is unclear to me, in absence of enough data to determine it, whether the ‘Robin Hood’ device just described would be enough to completely eliminate the need for a change in the accrual rate and/or increased contributions (as proposed by UCU in their points 3 and 4 above).

What is clear to me, though, is that such a device would help eliminate the unfairness described above.  And it would, at least, reduce the need for any changes in accrual or contribution rates, even if such need is not completely eliminated.

© David Firth, March 2018

To cite this entry:
Firth, D (2018). Future USS: Robin Hood can help?  Weblog entry at URL

Postscript: about Robin Hood

The legend of Robin Hood, ‘feared by the bad, loved by the good’, will already be known to most people who have grown up in England.  There are countless stories of Robin Hood ‘persuading’ the rich to part with their money, for the benefit of poorer folk.

For anyone interested, there’s a lot to read about it here:

One aspect that I particularly like is that my home city of Wakefield is one of the (many!) places that lay claim to Robin Hood as one if its own townspeople.  But then there’s all that romantic Sherwood Forest nonsense…

USS pension scheme and fairness


Update on 16 March: After reading this post, you might perhaps be interested in some of these follow-ups:

The Universities Superannuation Scheme (USS) is among the largest pension schemes in the UK.  It provides pensions for academic and other staff in most of the UK’s universities — and right now it is the subject of strike action in around 60 universities.  The strike relates to substantial proposed changes to the pension scheme.

Most of the current arguments are about the long-term affordability of USS in its present form.  I do not consider myself sufficiently expert to add much that would be useful on that (crucial) aspect of USS — so that’s not the subject of this post.  Instead, here I will take a look at the current USS scheme in regard to aspects of fairness; and I’ll examine the new proposals and their implied direction of travel, in relation to fairness.

Why am I writing this?  Well, I needed to find out enough about USS and the new proposals to be able to explain — to friends, family, colleagues in other countries, etc. — what all the current argument is about.  One thing I found — quite separately from the arguments about affordability of a defined-benefit pension scheme — was that the new proposals seem to have the wrong direction of travel in relation to (at least my notions of) fairness.

At the end I’ll make a specific suggestion towards improving the currently proposed changes to USS.

1.  A bit of history

Back in the mists of time, when I first joined USS, the university that appointed me was (rightly) very keen to stress how great the pension scheme was.  I would need to pay 6.35% of salary, and the university would contribute an additional 18.55% of my salary, to provide a guaranteed pension based on whatever salary level I had achieved at retirement.  This was regarded as an important part of the deal: our salaries were considerably lower than we could be paid elsewhere, but the promise of a decent pension would compensate to some extent.

I remember well, a few years later in 1997, the widespread feeling of outrage and dismay when the universities collectively decided to reduce their contributions from 18.55% to 14%.  In 1997 the USS pension fund was substantially in surplus, and the universities succumbed to the natural temptation of a ‘contributions holiday’.  (It turned out to be a very long holiday: the universities’ contribution rate remained at 14% until 2011.)

Right now, in 2018, employer contribution is 18%, and USS members themselves pay 8% of salary into the scheme (with an option to increase that to 9% in return for a matching 1% increase of the employer contribution to the USS member’s “defined contribution” account with USS).

In relation to this history, let me mention here two aspects relating to fairness:

  1. The ‘final salary‘ basis for determining the amount of pension payable was (in my opinion!) blatantly unfair.  As someone who stands to benefit from it, I think I am in a good position to say this.  Members’ life-long contributions to USS were being used disproportionately to pay the pensions of those who were paid the highest salaries, and especially those whose salaries became high relatively late in their careers.
  2. The long contributions holiday taken by universities between 1997 and 2011 is clearly connected with whatever problems USS has today.  The universities ought to be prepared to substantially increase their contributions to the USS pot when the historic surplus has been run down partly as a result of the earlier (substantial!) ‘holiday’.  It would be completely unfair to expect future USS members to pay the cost of the ‘holiday’.

2.  Current USS

This excerpt from gives a very rough summary of the current USS setup:


In relation to fairness of the current setup, I want to note three things:

  1. The unfair final salary aspect has now gone (following the changes to USS that were made in 2016).  Instead, we now have ‘Career Revalued Benefits’ (CRB), based on contributions from annual salary up to the ‘threshold’ amount of around £55k.  This results in lower pensions for many (perhaps most) USS members, especially those whose salaries increase substantially in mid-late career.  But CRB does seem a fairer basis than ‘final salary’, for determining the relationship between life-long contributions and the level of pension ultimately provided.  (The 1/75 multiplier is arguably not high enough; but that relates more to affordability than to fairness.)
  2. The current USS setup has an appealing, progressive aspect: the employer contribution of 18% supports only a 12% contribution to the ‘USS Investment Builder’ pension pot that relates to to annual salary in excess of the £55k threshold.  The implication of this is that some (maybe most?) of the remaining 6% of employer contribution, for salaries over £55k, gets used in support of the defined-benefit CRB scheme that applies to the first £55k of every member’s salary.  As a principle, this seems right and fair: a priority for USS should be to ensure that even the least well-paid members get a decent pension.  Appropriateness of the precise details (12% versus perhaps some lower figure or a decreasing schedule for higher salaries; and the £55k level of the salary ‘threshold’) is of course open to debate, still.
  3. The previous two points were about ways in which USS currently is fair.  For balance, let me mention one aspect of current USS that does seem unfair, still.  The USS scheme has always, as far as I know, included specific pension provision for the surviving spouse/partner and/or other dependants of a member who dies.  While this clearly is a benefit to those members who have got such dependants, it implies some level of subsidy from the contributions — whether made directly, or by the employer — of any member without such dependants.  (Again I write as someone who does benefit from this; but I don’t regard it as fair!)

3.  The USS proposals for change

The following excerpt is from


My thoughts on the specific numbers there, in terms of fairness, are as follows.  (The first item in the following list is by far the most important of the three; the other two are quite minor by comparison.)

  1. For salary amounts over the current threshold of £55k, the proposal would actually increase the employers’ pension-pot contributions (from the current 12%, to 13.25%).  This runs counter to the ‘progressive’ aspect, mentioned above, of the current USS setup.  Moreover, it implies that less of the employer contribution on salary amounts over £55k — that is to say, less than now — would actually be used to service the defined-benefit commitments of USS.  This seems absurd.  It represents a direction of travel that’s completely unfair: it would systematically shift the future cost of those defined-benefit commitments, away from the higher-paid members, onto those whose salary is mostly or entirely below the current £55k threshold.
  2. The suggested flexibility to adjust member contributions downwards to 4%, while still getting 13.25% employer contributions, looks good on the face of it.  But who would it benefit most?  Most members with low to middling incomes will probably want or need to contribute 8%, still, in order to (aim to) ensure that they have a decent pension at retirement.  Probably the main beneficiaries of this flexibility would be those members with the highest salaries, who could use it to reduce or eliminate potential breaches of the HMRC allowances (either the Annual Allowance for pension contributions, or the Lifetime Allowance for total pension-pot size).
  3. That last bullet-point in the above excerpt looks relatively minor.  It suggests shifting some of the cost of USS death benefits away from employers, to members — if I have understood it correctly.  The details of this are not yet clear, at least to me; but it could imply that the cost of pensions for widows/partners and other dependants — which I have argued above is already unfair — would be carried more directly than before by members, i.e., through their own direct contributions rather than their employers’.

4.  Conclusions

My comments in this post really are quite separate from the on-going arguments about affordability of the defined-benefit part of USS.  But…

If those arguments about affordability do result in substantial changes being made, then I really hope that considerations of fairness will play a major role.  As things stand at present, the proposed USS changes would appear to be quite neutral or even beneficial to those members with the very highest salaries, but clearly detrimental to the majority.

A more positive take on the points made above would be that there is plenty of room for manoeuvre, towards the negotiation of a future USS that could work better for all.  The universities plainly are not currently at the absolute limit of what is affordable.  The proposal to contribute more to the pension pots of the highest paid, rather than less, makes this abundantly clear (as indeed does the USS history, as mentioned above).  A future USS that’s designed to be more progressive than the present version — i.e., with the highest paid receiving gradually lower marginal rates of employer contribution to their pension-pots — could perhaps make the defined-benefit component of USS work out to be (even more) clearly affordable?

Update (4 March 2018): The follow-up post Future USS: Robin Hood can help? formulates a concrete suggestion along the lines just indicated above — a simple device that would make USS both fairer and more affordable.


© David Firth, February 2018

To cite this entry:
Firth, D (2018). USS pension scheme and fairness. Weblog entry at URL

Exit poll for June 2017 election (UK)



It has been a while since I posted anything here, but I can’t resist this one.

Let me just give three numbers.  The first two are:

  • 314, the number of seats predicted for the largest party (Conservatives) in the UK House of Commons, at 10pm in Thursday (i.e., before even a single vote had been counted) from the exit poll commissioned jointly by broadcasters BBC, ITV and Sky.
  • 318, the actual number of seats that were won by the Conservatives, now that all the votes have been counted.

That highly accurate prediction changed the whole story on election night: most of the pre-election voting intention polls had predicted a substantial Conservative majority.  (And certainly that’s what Theresa May had expected to achieve when she made the mistake of calling a snap election, 3 years early.)  But the exit poll prediction made it pretty clear that the Conservatives would either not achieve a majority (for which 326 seats would be needed), or at best would be returned with a very small majority such as the one they held before the election.  Media commentary turned quickly to how a government might be formed in the seemingly likely event of a hung Parliament, and what the future might be for Mrs May.  The financial markets moved quite substantially, too, in the moments after 10pm.

For more details on the exit poll, its history, and the methods used to achieve that kind of predictive accuracy, see Exit Polling Explained.

The third number I want to mention here is

  • 2.1.0

That’s the version of R that I had at the time of the 2005 General Election, when I completed the development of a fairly extensive set of R functions to use in connection with the exit poll (which at that time was done for BBC and ITV jointly).  Amazingly (to me!) the code that I wrote back in 2001–2005 still works fine.  My friend and former colleague Jouni Kuha, who stepped in as election-day statistician for the BBC when I gave it up after 2005, told me today that (with some tweaks, I presume!) it all works brilliantly still, as the basis for an extremely high-pressure data analysis on election day/night.  Very pleasing indeed; and strong testimony to the heroic efforts of the R Core Development Team, to keep everything stable with a view to the long term.

As suggested by that kind tweet reproduced above from the RSS President, David Spiegelhalter: Thursday’s performance was quite a triumph for the practical art and science of Statistics.  [And I think I am allowed to say this, since on this occasion I was not even there!  The credit for Thursday’s work goes to Jouni Kuha, along with John Curtice, Steve Fisher and the rest of the academic team of analysts who worked in the secret exit-poll “bunker” on 8 June.]


Facts checked: The UK general election, 7-way TV debate between Westminster party leaders


I was part way through doing some of this myself, when I found that has done a great job already with checking some of the main numbers quoted in last night’s televised debate:

RAE 2008: How much weight did research outputs actually get?


In the 2008 UK Research Assessment Exercise each subject-area assessment panel specified and published in advance the weight to be given to each of the three parts of the assessment, namely “research outputs”, “research environment” and “esteem”. The quality “sub-profiles” for those three parts were then combined into an overall quality profile for each department assesed, by using the published weights. The overall quality profiles have since been used in HEFCE funding allocations, and in various league tables published by newspapers and others.

For example, RAE Panel F (Pure and Applied Maths, Statistics, Operational Research, Computer Science and Informatics) specified the following weights:

  • Research outputs: 70%
  • Research environment: 20%
  • Esteem: 10%

The weight specified for research outputs varied between 50% (for RAE Panel G, engineering disciplines) to 80% (RAE Panel N, humanities disciplines).

When the RAE sub-profiles were published in spring 2009, it became clear that the assessments for the three parts were often quite different from one another. For example, some of the assessment panels awarded many more 4* (“world leading”) grades for research environment and esteem than for research outputs. These seemingly systematic differences naturally prompt the question: to what extent are the agreed and published weights for the three parts reflected in funding allocations, league tables, etc.?

Let’s leave the consideration of league tables for another time. Here we’ll calculate the actual relative weights of the three parts in terms of their effect on funding outcomes, and compare those with the weights that were published and used by RAE assessment panels.

The formula used by HEFCE in 2009 awarded quality-related research funding to departments in proportion to

\displaystyle 7 p_{4d} + 3 p_{3d} + p_{2d}

where the p’s come from the department’s overall RAE profile (being the percentages at quality levels 4*, 3* and 2*). Now, from the published sub-profile for research outputs, it can also be calculated how much of the any department’s allocated funding came from the research outputs component, in the obvious way. The actual weight accorded to research outputs in the 2009 funding outcomes by a given RAE Sub-panel is then

\displaystyle{\sum_d(\textrm{funding from RAE research outputs profile for department } d) \over\sum_d(\textrm{funding from overall RAE profile for department } d)}

where the summations are over all of the departments d assessed by the Sub-panel. (In the calculation here I have used the un-rounded overall profiles, not the crudely rounded ones used by HEFCE in their 2009 funding allocation. I’ll write more about that in a later post. Rounded or un-rounded doesn’t really affect the main point here, though.)

For 2010 it seems that the HEFCE funding rates will be in the ratio 9:3:1 rather than 7:3:1, i.e., proportionately more funds will be given to departments with a high percentage of work assessed at the 4* quality level. The table below lists the discrepancies between the actual and intended weight given to Outputs, by RAE Sub-panel, using the 2009 and 2010 HEFCE funding rates. For example, the RAE Sub-panel J41 (Sociology) decided that 75% of the weight should go to Outputs, but the reality in 2009 was that only 56.6% of the HEFCE “QR” funding to Sociology departments came via their Outputs sub-profiles; the corresponding figure that appears in the table below is 56.6 – 75 = -18.4. An alternative view of the same numbers is that the Sociology Sub-panel intended to give combined weight 25% to “research environment” and “esteem”, but those two parts of the assessment actually accounted for a very much larger 43.4% of the 2009 funding allocation to Sociology departments (and with the new funding rates for 2010 that will increase to 45.4%).

RAE Panel RAE Sub-panel name 2009 2010
A Cardiovascular Medicine -2.9 -3.4
A Cancer Studies -3.8 -4.3
A Infection and Immunology -7.7 -9.0
A Other Hospital Based Clinical Subjects -13.4 -16.1
A Other Laboratory Based Clinical Subjects -4.5 -4.9
B Epidemiology and Public Health -10.7 -12.5
B Health Services Research -9.2 -10.6
B Primary Care and Other Community Based Clinical Subjects -5.0 -5.4
B Psychiatry, Neuroscience and Clinical Psychology -6.4 -7.2
C Dentistry 0.2 -0.5
C Nursing and Midwifery -2.3 -3.6
C Allied Health Professions and Studies -1.9 -2.5
C Pharmacy -4.1 -5.3
D Biological Sciences -5.5 -6.0
D Pre-clinical and Human Biological Sciences -4.9 -5.6
D Agriculture, Veterinary and Food Science -7.0 -8.2
E Earth Systems and Environmental Sciences -4.9 -5.6
E Chemistry -3.5 -3.9
E Physics -3.2 -4.1
F Pure Mathematics -3.8 -4.2
F Applied Mathematics -6.0 -7.2
F Statistics and Operational Research -3.2 -3.5
F Computer Science and Informatics 1.8 1.6
G Electrical and Electronic Engineering 2.0 1.8
G General Engineering and Mineral & Mining Engineering 3.2 2.7
G Chemical Engineering 7.1 7.8
G Civil Engineering 1.2 0.6
G Mechanical, Aeronautical and Manufacturing Engineering 3.7 3.7
G Metallurgy and Materials 2.6 2.2
H Architecture and the Built Environment -2.8 -3.7
H Town and Country Planning -3.2 -3.3
H Geography and Environmental Studies -3.8 -4.5
H Archaeology -10.9 -12.6
I Economics and Econometrics -1.2 -1.1
I Accounting and Finance -4.0 -4.7
I Business and Management Studies -5.3 -6.2
I Library and Information Management -8.9 -10.0
J Law -13.4 -14.7
J Politics and International Studies -9.6 -10.0
J Social Work and Social Policy & Administration -8.4 -9.7
J Sociology -18.4 -20.4
J Anthropology -18.6 -21.2
J Development Studies -11.5 -12.9
K Psychology -3.6 -4.6
K Education -5.9 -7.5
K Sports-Related Studies -4.5 -5.5
L American Studies and Anglophone Area Studies -13.3 -14.7
L Middle Eastern and African Studies -14.3 -16.0
L Asian Studies -14.1 -15.5
L European Studies -11.1 -13.1
M Russian, Slavonic and East European Languages -11.1 -12.8
M French -6.9 -7.6
M German, Dutch and Scandinavian Languages -4.5 -5.3
M Italian -8.6 -9.7
M Iberian and Latin American Languages -12.2 -13.8
M Celtic Studies -9.8 -11.3
M English Language and Literature -10.9 -12.7
M Linguistics -7.9 -9.1
N Classics, Ancient History, Byzantine and Modern Greek Studies -8.8 -10.1
N Philosophy -11.9 -14.1
N Theology, Divinity and Religious Studies -9.6 -11.4
N History -8.4 -9.5
O Art and Design -13.9 -14.8
O History of Art, Architecture and Design -1.0 -1.0
O Drama, Dance and Performing Arts -2.8 -2.9
O Communication, Cultural and Media Studies 0.7 0.8
O Music -4.5 -4.6

RAE 2008: relation between 2009 and 2010 funding rates

Most of the discrepancies are negative: the actual weight given to research outputs, in terms of funding, is less than was apparently intended by most of the assessment panels. Some of the discrepancies are very large indeed — more than 20 percentage points in the cases of Sociology and Anthropology, under the HEFCE funding rates that will be applied in 2010.

Click on the image for a graphical view of the relationship between the discrepancies for 2009 (funding rates 7:3:1:0:0 for the five RAE quality levels) and 2010 (funding rates 9:3:1:0:0).


In RAE 2008 the agreed and published weights were the result of much discussion and public consultation, most of which centred on the perceived relative importance of the three components (research outputs, research environment, esteem) in different research disciplines. The discrepancies that are evident here arise from the weighted averaging of three separate profiles without (it seems) careful consideration of the differences of distribution between them. In the case of funding, it’s (mainly) differences in the usage of the 4* quality level that matter: if 4* is a relatively rare assessment for research outputs but is much more common for research environment, for example, the upshot is that the quality of research outputs actually determines less of the funding than the published weights would imply.

It is to be hoped that measures will be put in place to rectify this in the forthcoming replacement for the RAE, the Research Excellence Framework. In particular, the much-debated weight of 25% for the proposed new “impact” part of the REF assessment might actually turn out to be appreciably more if we’re not careful (the example of Sociology, see above, should be enough to emphasise this point).



The calculation done here was suggested to me by my friend Bernard Silverman, and indeed he did the same calculation independently himself (for the 2009 funding formula) and got the same results. The opinion expressed above is mine, not necessarily shared by Bernard.

© David Firth, February 2010

To cite this entry:

Firth, D (2010). RAE 2008: How much weight did research outputs actually get? Weblog entry at

RAE 2008: Assessed quality of research in different disciplines


RAE 2008 aggregate quality assessments, by discipline (1295 x 788 pixels)

This graph was drawn with the help of my daughter Kathryn on her “take your daughter to work” day in Year 10 at school. Her skill with spreadsheet programs was invaluable!

The graph shows how different disciplines — that is, different RAE “sub-panels” or “units of assessment” — emerged in terms of their average research quality as assessed in RAE 2008. The main data used to make the graph are the overall (rounded) quality profiles and submission-size data that were published in December 2008. Those published quality profiles were the basis (in March 2009) of ‘QR’ funding allocations made for 2009-10 by HEFCE to universities and other institutions.

Each bar in the graph represents one academic discipline (as defined by the remit of an RAE sub-panel). The blue and pink colouring shows how the sub-panels were organised into 15 RAE “main panels”. A key task of the main panels was to try to ensure comparability between the assessmenta made for different disciplines. Disciplines within the first seven main panels are the so-called “STEM” (Science, Technology, Engineering and Mathematics) subjects.

The height of each bar is calculated as the average, over all “full-time equivalent” (FTE) researchers whose work was submitted to the RAE, of a “quality score” calculated directly from the published RAE profile of each researcher’s department. The quality score for members of department d is calculated as a weighted sum

\displaystyle w_4 p_{4d} + w_3 p_{3d} + w_2 p_{2d} + w_1 p_{1d} + w_0 p_{0d}\ ,

where the p’s represent the department’s RAE profile and the w’s are suitably defined weights (with w4w3 ≥ … ≥ w0). The particular weights used in constructing such a quality score are rather arbitrary; here I have used 7:3:1:0:0, the same weights that were used in HEFCE’s 2009 funding allocation, but it would not make very much difference, for the purpose of drawing this graph to compare whole disciplines, to use something else such as 4:3:2:1:0.
Example: for a department whose RAE profile is

          4*   3*   2*   1*   0*
          0.10 0.25 0.30 0.30 0.05

the quality score assigned to each submitted researcher is

\displaystyle (7 \times 0.10) + (3\times 0.25) + (1 \times 0.30) = 1.75\ .

The average quality score over the whole RAE was about 2.6 (the green line in the graph).

The graph shows a fair amount of variation between disciplines, both within and between main panels of the RAE. The differences may of course reflect, at least to some extent, genuine differences in the perceived quality of research in different disciplines; the top-level RAE assessment criteria were the same for all disciplines, so in principle this kind of comparison between disciplines might be made (although in practice the verbally described criteria would surely be interpreted differently by assessors in different disciplines). However, it does appear that some main panels were appreciably tougher in their assessments than others. Even within main panels it looks as though the assessments of different sub-panels might not really be comparable. (On this last point, main panel F even went so far as to make an explicit comment in the minutes of its final meeting in October 2008 (available in this zip file from the RAE website): noting the discrepancy between assessments for Computer Science and Informatics and for the other three disciplines under its remit, Panel F minuted that ”…this discrepancy should not be taken to be an indication of the relative strengths of the subfields in the institutions where comparisons are possible.” I have not checked the minutes of other panels for similar statements.)

Note that, although the HEFCE funding weights were used in constructing the scores that are summarized here, the relative funding rates for different disciplines cannot be read straightforwardly from the above graph. This is because HEFCE took special measures to protect the research funding to disciplines in the “STEM” group. Within the non-STEM group of disciplines, relative heights in the above graph equate to relative funding rates; the same applies also within each main panel among the STEM disciplines. (On this last point, and in relation to the discrepancy minuted by Panel F as mentioned above: Panel F also formally minuted its hope that the discrepancy would not adversely affect the QR funding allocated to Pure Maths, Applied Maths, and Statistics & Operational Research. But, perhaps unsurprisingly, that expression of hope from Panel F was ignored in the actual funding formula!)


HEFCE relies heavily on the notion that assessment panels are able to regulate each other’s behaviour, so as to arrive at assessments which allow disciplines to be compared (for funding purposes at least, and perhaps for other purposes as well). This strikes me as wishful thinking, at best! By allowing the relative funding of different disciplines funding to follow quality scores so directly, HEFCE has created a simple game in which the clear incentive for the assessors in any given discipline is to make their own scores as high as they can get away with. The most successful assessment panel, certainly in the eyes of their own colleagues, is not the one that makes the best job of assessing quality faithfully, but the one with the highest scores at the end! This seems an absurd way to set up such an expensive, and potentially valuable, research assessment exercise. Unfortunately in the current plans for the REF (Research Excellence Framework, the RAE’s successor) there is little or no evidence that HEFCE has a solution to this problem. The REF pilot study seems to have concluded, as perhaps expected, that routinely generated “bibliometric” measures cannot be used at all reliably for such inter-discipline comparisons.

Since I don’t have an alternative solution to offer, I strongly favour the de-coupling of allocation of funds between disciplines from research quality assessment. If the Government or HEFCE wishes or needs to increase its funding for research in some disciplines at the expense of others, it ought to be for a good and clear reason; research assessment panels will inevitably vary in their interpretation of the assessment criteria and in their scrupulousness, and such variation should not be any part of the reason.


© David Firth, November 2009

To cite this entry:

Firth, D (2009). RAE 2008: Assessed quality of research in different disciplines. Weblog entry
at URL