Gridwatch sources its data off ELEXON which is collecting data from generation connected to the National Grid at 275/400kV levels. However, to participate in the balancing mechanism a generator has to connect to the ELEXON metering system system so over the last few years progressively more generation, and BESS, located within the DNO system has become visible in real time. There are other sources of generator info from OFGEM renewablesandchp.ofgem.gov.uk which provides historical data on all sites accredited under the RO scheme (be warned there are plenty) but it does provide high level summaries. Problem is some of the larger sites are also reporting through ELEXON so you would need to remove them.
Anyhow only recently found this substack but great work.
One question occurs - if the Gridwatch data gives actual generation after grid balancing action, what happened to the ‘surplus’ generation in the corresponding DUKES figures? If supply must always equal demand, how was the excess supply implied by DUKES used?
I've satisfied myself that Gridwatch data is recorded after grid balancing i.e. net of grid balancing.
But I also had to reconcile that the DUKES total energies from wind and solar are greater than I get from numerical integration of Gridwatch data for complete years 2020 and 2021.
Assuming that the DUKES numbers are official and therefore correct, I needed to increase the Gridwatch powers / energies by the factors I noted in my 23 December post.
There was no excess per DUKES, there cannot be an excess.
I believe gridwatch looks at the amount of energy distributed in the grid, whilst dukes also includes embedded power applications, I.e. large energy consumers near wind and solar farms that buy the electricity directly via a dedicated connection, rather than from the grid.
As I understand, Gridwatch numbers report actual generation volume which, at any given time, must equal demand.
You have noted that the ‘renewables’ contribution appears to ‘peak’ at demand minus ~2GW, and conclude this is explained by National Grid constraining renewable (principally wind) so that grid frequency can be maintained.
If wind output has been constrained, how can DUKES report a higher output than Gridwatch? One explanation, I suppose, is that the constrained wind generation is being consumed off-grid, so not visible to Gridwatch.
Is that your conclusion? Otherwise how can Gridwatch and DUKES both be correct?
Hello again, Tim, and thank you for the clarification.
Fundamentally the Gridwatch numbers do not claim to be exact reality.
Under some of the 'dials' on the dashboard are explanations of the expected acccuracies/errors.
Because of that I never attempted energy balances across sets of data - there are bits of energy missing from the Gridwatch record.
I can't remember exactly why, but I decided to download whole-year solar PV data from Sheffield Uni to compare with Gridwatch (which says its solar PV is *based* on Sheffield Uni) but there were differences. I then compared whole-year 2021 numerical integration (SheffUni) with DUKES 2021 and they were different: SheffUni 11,557,429 MWh vs. DUKES 12,137,960 MWh, i.e. 95.22%
The Gridwatch cumulative total for 2021 was less again by the factor included in the post.
So I decided that I need to treat DUKES (the official record) as true.
I assume the DUKES people have better ways of estimating all the off-grid solar PV and off-grid wind. Plus all the unmetered losses occurring throughout the system which are a hidden 'demand'.
So DUKES is correct (I assume, as it's official Gov statistical info) while Gridwatch admits it's not.
But to counter your proffered explanation: *constrained* power is *potential* grid power that was never generated.
National Grid ESO tells a generator to turn down or turn off because there is insufficient demand, the windmill blades are feathered (or whatever), and the output from that facility drops, possibly to zero. And the facility-owner gets a nice 'constraint payment' cheque in the post afterwards.
Hmm. Thanks for that, Chris. I agree/accept your explanations. However, the differences seem too large to be glossed over. Is there not some way of attempting a reconciliation of total demand to total supply over a given period.
Hello Tim, most kind of you to accept my explanation. It's the only one I have.
I am not 'glossing over' the differences. I have found the differences, they are significant to my analysis of "will renewables keep GB's lights on or not?", the necessary 'fudge factors' for wind and solar PV are quite consistent across the two full years the data is available, so I have applied those fudge factors to my recent analysis and will continue to apply them going forward.
I have no interest in trying to reconcile data from a 3rd party (Gridwatch). I admire the construction of their data-gathering app, I applaud that it puts this data into the public domain in a very accessible format, but I accept it was put together by private individuals and has its limits.
Chris, I apologise, I didn’t intend to be patronising. I’m simply noting what appear to me to be anomalies and trying to resolve them.
You may have done this but I can’t know that without asking.
Personally, I don’t see a 5% anomaly between DUKES and Gridwatch as material or surprising. It can be expected, I think, because we know that domestic solar will be, to some extent, consumed off grid whilst actual generation will be included in FITs claims (which may be the source of DUKES data?)
It’s a bit harder to explain the difference in wind, which is much larger in absolute and relative terms.
Refreshing all 3 then looking at their data, the numbers differ!
Hovering your mouse over Gridwatch/Templar's dials provides background info. E.g the wind dial reveals:
“This is the total contributed by metered wind farms. Wind power contributes about another 30% from embedded (or unmetered) wind turbines that shows only as a drop in demand."
Likewise, hover over the solar dial and we learn:
"Solar PV: As no solar PV to date is metered centrally, we cannot show accurate real time figures on solar PV power.
Estimated power (data provided by Sheffield University) is shown here. There is good evidence from the lack of decrease in recorded demand at midday to suppose this is somewhat overestimated, however."
Earlier this week, I discovered Elexon Insights has recently produced its own more user-friendly set of info - in Beta version. There are various tabs & drop-down menus, and filters & calendars to play with.
On its 'Generation by fuel type' page, if you deselect 'All fuel type' i.e. leaving All Interconnects, interconnectors' contributions are all positive!? It doesn't appear to recognise fuels are used to generate exports.
Its 'Wind Generation' page is interesting, showing info on Outturn, Latest Forecast and Initial Forecast. Discrepancies between 'latest forecast' and Outturn are 'interesting'.
I've still got to assimilate all you've written, but the above recognises some the challenges we seem to face.
I still like Gridwatch because it gathers all the data in one place and that data is easy to download and then manipulate in Excel.
I have not attempted to generate energy balances for each data-set because yes, in the early days I read those guidance notes behind the 'dials' on the dashboard. I thought then that there are large hidden / unrecorded power flows so attempting energy balances would be a pointless exercise. (Each year's Excel file with the calcs and charts is already 60-62 MB.)
However, for this post I realised that the corrections needed for wind and solar PV were significant to my analysis.
I decided to treat the DUKES figures as definitive - the correction factors derived from that look reasonable in light of the dashboard guidance notes.
Gridwatch sources its data off ELEXON which is collecting data from generation connected to the National Grid at 275/400kV levels. However, to participate in the balancing mechanism a generator has to connect to the ELEXON metering system system so over the last few years progressively more generation, and BESS, located within the DNO system has become visible in real time. There are other sources of generator info from OFGEM renewablesandchp.ofgem.gov.uk which provides historical data on all sites accredited under the RO scheme (be warned there are plenty) but it does provide high level summaries. Problem is some of the larger sites are also reporting through ELEXON so you would need to remove them.
Anyhow only recently found this substack but great work.
Hi, thank you for the additional info and the kind words.
I am currently persisting with Gridwatch data as it's so convenient *and* it's 'good enough for government work' as they say.
May I ask, how *did* you find my substack?
Hi Chris.
This is interesting reading as always.
One question occurs - if the Gridwatch data gives actual generation after grid balancing action, what happened to the ‘surplus’ generation in the corresponding DUKES figures? If supply must always equal demand, how was the excess supply implied by DUKES used?
Hello Tim,
Not sure I completely understand your Q.
I've satisfied myself that Gridwatch data is recorded after grid balancing i.e. net of grid balancing.
But I also had to reconcile that the DUKES total energies from wind and solar are greater than I get from numerical integration of Gridwatch data for complete years 2020 and 2021.
Assuming that the DUKES numbers are official and therefore correct, I needed to increase the Gridwatch powers / energies by the factors I noted in my 23 December post.
There was no excess per DUKES, there cannot be an excess.
I hope that helps / answers...?
I believe gridwatch looks at the amount of energy distributed in the grid, whilst dukes also includes embedded power applications, I.e. large energy consumers near wind and solar farms that buy the electricity directly via a dedicated connection, rather than from the grid.
Sorry I wasn’t clear in phrasing the question.
As I understand, Gridwatch numbers report actual generation volume which, at any given time, must equal demand.
You have noted that the ‘renewables’ contribution appears to ‘peak’ at demand minus ~2GW, and conclude this is explained by National Grid constraining renewable (principally wind) so that grid frequency can be maintained.
If wind output has been constrained, how can DUKES report a higher output than Gridwatch? One explanation, I suppose, is that the constrained wind generation is being consumed off-grid, so not visible to Gridwatch.
Is that your conclusion? Otherwise how can Gridwatch and DUKES both be correct?
Hello again, Tim, and thank you for the clarification.
Fundamentally the Gridwatch numbers do not claim to be exact reality.
Under some of the 'dials' on the dashboard are explanations of the expected acccuracies/errors.
Because of that I never attempted energy balances across sets of data - there are bits of energy missing from the Gridwatch record.
I can't remember exactly why, but I decided to download whole-year solar PV data from Sheffield Uni to compare with Gridwatch (which says its solar PV is *based* on Sheffield Uni) but there were differences. I then compared whole-year 2021 numerical integration (SheffUni) with DUKES 2021 and they were different: SheffUni 11,557,429 MWh vs. DUKES 12,137,960 MWh, i.e. 95.22%
The Gridwatch cumulative total for 2021 was less again by the factor included in the post.
So I decided that I need to treat DUKES (the official record) as true.
I assume the DUKES people have better ways of estimating all the off-grid solar PV and off-grid wind. Plus all the unmetered losses occurring throughout the system which are a hidden 'demand'.
So DUKES is correct (I assume, as it's official Gov statistical info) while Gridwatch admits it's not.
But to counter your proffered explanation: *constrained* power is *potential* grid power that was never generated.
National Grid ESO tells a generator to turn down or turn off because there is insufficient demand, the windmill blades are feathered (or whatever), and the output from that facility drops, possibly to zero. And the facility-owner gets a nice 'constraint payment' cheque in the post afterwards.
That's my understannding of it, anyway.
Hmm. Thanks for that, Chris. I agree/accept your explanations. However, the differences seem too large to be glossed over. Is there not some way of attempting a reconciliation of total demand to total supply over a given period.
Demand = supply (generation) + ICT import - ICT export
Is that unachievable?
Hello Tim, most kind of you to accept my explanation. It's the only one I have.
I am not 'glossing over' the differences. I have found the differences, they are significant to my analysis of "will renewables keep GB's lights on or not?", the necessary 'fudge factors' for wind and solar PV are quite consistent across the two full years the data is available, so I have applied those fudge factors to my recent analysis and will continue to apply them going forward.
I have no interest in trying to reconcile data from a 3rd party (Gridwatch). I admire the construction of their data-gathering app, I applaud that it puts this data into the public domain in a very accessible format, but I accept it was put together by private individuals and has its limits.
Chris, I apologise, I didn’t intend to be patronising. I’m simply noting what appear to me to be anomalies and trying to resolve them.
You may have done this but I can’t know that without asking.
Personally, I don’t see a 5% anomaly between DUKES and Gridwatch as material or surprising. It can be expected, I think, because we know that domestic solar will be, to some extent, consumed off grid whilst actual generation will be included in FITs claims (which may be the source of DUKES data?)
It’s a bit harder to explain the difference in wind, which is much larger in absolute and relative terms.
Hi Chris
Thanks for your efforts!
Trying to understand our generation and consumptions is a nightmare.
Apols if I missed anything you've covered. My comments are based my understandings *which may not necessarily be correct*. ;-)
Leo Smith, who runs the Gridwatch website https://www.gridwatch.templar.co.uk
Mark Stolworthy writes and maintains https://gridwatch.co.uk
DUKES is the Digest of UK Energy Statistics - i.e. inc Northern Ireland.
As I understand it, both draw their data from Elexon, but that covers GB only.
https://www.bmreports.com/bmrs/?q=generation/fueltype/current
Refreshing all 3 then looking at their data, the numbers differ!
Hovering your mouse over Gridwatch/Templar's dials provides background info. E.g the wind dial reveals:
“This is the total contributed by metered wind farms. Wind power contributes about another 30% from embedded (or unmetered) wind turbines that shows only as a drop in demand."
Likewise, hover over the solar dial and we learn:
"Solar PV: As no solar PV to date is metered centrally, we cannot show accurate real time figures on solar PV power.
Estimated power (data provided by Sheffield University) is shown here. There is good evidence from the lack of decrease in recorded demand at midday to suppose this is somewhat overestimated, however."
On the Gridwatch.co.uk page, scroll down for explanations.
Earlier this week, I discovered Elexon Insights has recently produced its own more user-friendly set of info - in Beta version. There are various tabs & drop-down menus, and filters & calendars to play with.
https://bmrs.elexon.co.uk/generation-by-fuel-type
However, there's no solar info.
On its 'Generation by fuel type' page, if you deselect 'All fuel type' i.e. leaving All Interconnects, interconnectors' contributions are all positive!? It doesn't appear to recognise fuels are used to generate exports.
Its 'Wind Generation' page is interesting, showing info on Outturn, Latest Forecast and Initial Forecast. Discrepancies between 'latest forecast' and Outturn are 'interesting'.
I've still got to assimilate all you've written, but the above recognises some the challenges we seem to face.
Apols if it seems like a rant ;-)
Hello Ron, thank you and Merry Xmas!
I still like Gridwatch because it gathers all the data in one place and that data is easy to download and then manipulate in Excel.
I have not attempted to generate energy balances for each data-set because yes, in the early days I read those guidance notes behind the 'dials' on the dashboard. I thought then that there are large hidden / unrecorded power flows so attempting energy balances would be a pointless exercise. (Each year's Excel file with the calcs and charts is already 60-62 MB.)
However, for this post I realised that the corrections needed for wind and solar PV were significant to my analysis.
I decided to treat the DUKES figures as definitive - the correction factors derived from that look reasonable in light of the dashboard guidance notes.
As for the rest, happy reading!