Far be it from me to position myself as the queen of healthy eating. However, even I know that heavily processed food is less good for us than food in its natural, unprocessed state. Whether raw food is really better that cooked food, I very much doubt. However, let’s run with the raw=good, processed=bad metaphor when applied to the wonderful world of school data.

I think the grip that excessive marking had on our schools is finally beginning to wane. At a recent event I was at for early career stage teachers, I asked how many people worked at schools where heavy duty marking was still the norm. About half the room put their hand up. Last year it would have been at least two thirds, if not three quarters. At our open days about marking, our approach seems less revolutionary now to most who attend and more about tweaking their own approach. Just anecdotes, but maybe signs that more sensible time are on the way. Data however, that’s a different matter. How long before the Teacher Workload Advisory Group on Making Data Work. has any impact?

I’ve written before about how we have attempted to strip back our approach to data.  We’ve stripped it back even more since then, and I’m still not quite convinced we’ve stripped it back enough yet. So I eagerly sought out any workshops on data at ResearchEd Kent yesterday, attending excellent sessions by both Becky Allen and Ben White.. I also reread Tom Sherrinton’s blog about the futile quest to ‘measure’ progress. This is my attempt at a synthesis/summary: a smoothie, if you like, of wholesome data goodness.

Processed food is easy to prepare; pop it in the microwave and ping. And the product is pretty uniform. But that ease and uniformity comes at a price to its nutritional worth. Binders, stabilisers, emulsifiers and preservatives are added to the basic ingredients; salt and sugar are added to make it all palatable. The more the basic raw ingredients are mucked above with, the more their essential goodness is compromised.

Raw, unprocessed food, on the other hand, is much more likely to be high in fibre, low in sugar and salt, and free from various kinds gunk. But it doesn’t come conveniently ready to cook. The individual ingredients are stubbornly individual. Transforming them into a meal will take work and each ingredient needs different work: some need peeling, some chopping, some blanching.  The resultant meals are likely to vary somewhat, even when using the same ingredients.

For years we have consumed highly processed data. Data has been aggregated, averaged and spreadsheetified into a highly reportable, easily consumable product. Contrast this with the messy business of trying to report progress when all you have is as assortment of raw data which you can only describe.  Governors and that host of others who would seek to make us accountable are used to being able to swallow down data purée without having to chew; no wonder they turn their noses up at the prospect of roughage-heavy, honest-to-goodness data wholefood.

Flightpaths, graphs and charts often the illusion of certainty, of rigour. They are (relatively) easy to understand. They blend complex individuality into homogenised simplicity; progress in art can be compared with progress in RE or maths or history. Yet this consistency is a consistency in meaninglessness. This is fake rigour, delusional certainty, but oh so deliciously tempting! Who wouldn’t want the sweet taste of certainty, of ‘sciency’ looking graphs, rather than unpalatable messy reality? No wonder we keep on swallowing those blue pills.

The links will take you to better analysis than I can do here, but processed data is bad for us because:

• Progress isn’t linear. It happens in fits and starts. We might wish it to be otherwise but wish fulfilment is not a good basis for policy.
• Any attainment measure is an approximation with a margin of error. Yet we imbue data with spurious accuracy.
• This margin is magnified when ‘measuring’ progress. This is because progress relies on two measures of attainment, both with margins of error. The concomitant statistical ‘noise’ is therefore deafening
• Even standardised assessments taken across many schools are unreliable because the stakes attached to these tests will differ from setting to setting and class to class. Children do better when they know the test is really high stakes. So schools which (sensibly) downplay the significance of tests are likely to get lower scores.
• Schools often set ‘off the peg’ tests that don’t match with what they’ve taught. Unsurprisingly, children do less well on stuff they haven’t been taught. This is therefore an exercise in utterly futility.
• These pointless tests are very expensive, wasting money that could be better spent elsewhere.
• Teacher assessment that results in grades is notoriously unreliable.
• Accountability pressures such as linking performance related pay to data amplifies this unreliability.
• Trying to compare progress in different subjects is a fool’s errand. Progress in writing is different from progress in science. Reducing it to a number because you want to compare things easily is statistically illiterate. [1]
• The same is true of trying to compare progress of different age groups.
• Processing and aggregating all this data is very time consuming. It adds to teacher workload. Its very meaningless makes that work all the more irksome.
• Unless your school is enormous, analysis of results by groups is statistical nonsense, even if the data made sense. Which it doesn’t.
• Bu the time children have done these assessments, it’s too late to help them. They are a post mortem rather than an x-ray.
• And none of this this anything to actually help any child learn anything any better – at best it helps managers allocate scarce extra resources at the chosen few. These ‘few’ will probably not include the most disadvantaged as they won’t look like a safe bet for improving progress scores. This is immoral.

So what’s this raw data you mention?

Instead of highly processed rubbish, what will really help us is limited amounts of raw data, by which I mean the results of highly focused tests/quizzes/assessments that tell you something really specific about what each child can or can’t do. For example, a times table quiz, a check of which sounds a child knows, a multiple choice quiz from a recent history topic, an analysis of a ‘cold write’ for a small number of highly specific grammatical features, a fitness ‘bleep’ test, a test of reading fluency. These kind of assessments are ‘granular; they check how much a child has learnt about a small, specific component of whatever they are learning: the fine ‘grain’ of learning. Such assessments might be reported as a simple test score or percentage. What must be resisted at all costs is aggregating all these results in some sort of statistical blender to make one score. That makes as much sense as putting your starter, main course and pudding into a blender in the hope of making a great tasting meal.

With raw data, you can see where the gaps are and go on to do something about them. There is little point in collecting data in the first place, unless you have a plan to use it in some way to do something as a result beyond reporting it.  In fact, as Becky Allen cautions, think about what you want to do – support those whose reading is less fluent, for example – and then collect data that enables you to pinpoint who those children are. Becky goes on to suggest that there is other raw data we could collect that would really help us deploy those precious extra resources to those who most need them.  Instead of CATS tests, year 7’s should start the year with a series of assessments that will pinpoint those with obvious barriers to progress in secondary school.   For example, a simple handwriting task so that schools can then intervene with those whose handwriting is poor or laborious, tests of times tables and number bonds, reading fluency tests. Then in addition to these, support to help those whose attendance or behaviour is poor or who do not do their homework[2]. For all of these, raw ‘live’ data is readily available and useful.  Throw in a test of phonics and substitute analysis of time spent on independent reading for homework and this list makes perfect sense of primary schools too.

I’d add in a bit of benchmarking, checking how we are doing compared with other schools in case we are deluding ourselves that things are fine when they’re really not. We are using comparative judgment for writing and star reader and star maths standardised assessments. Accelerated reader enables us to track how much independent reading children are doing and support those not doing enough.

When it comes to reporting to governors, local authorities, MATs or Ofsted, you will not be able to pull out a conditionally formatted, high processed chart with neatly aggregated data. All you will be able to do is talk about how you know which children are not doing as well as you want them to and what you are doing about it. Which is exactly what Sean Harford says we should be doing. Not that we do things ‘for Ofsted’ but it sure does help when they talk sense.

Unfortunately, we are still obliged to report certain things that make no sense whatsoever, so here we do what we have to, possibly signalling our disapproval along the way. For example, since we still have to report on the impact of pupil progress on our websites, headteachers might like to consider doing this (look at the end).

 

 

 

 

 

 

 

 

[1] Although Deep Ghatura would argue otherwise. Maybe I should say that how we do things now is illiterate but there may be another way.

[2] Ben White’’s blog outlines various things they ask pupils about homework which yields really interesting information.