Model Comparison – Future🔗
This section describes how we test En-ROADS projections against future scenarios from other scientific models. These comparisons give us an opportunity to look for ways to improve En-ROADS and build our confidence that En-ROADS is appropriate to its purpose of improving decision-maker understanding of the dynamics of the climate-energy-land-economic system.
What Do We Mean by Comparisons? — One Example🔗
To illustrate how En-ROADS compares with other models, consider the example of coal primary energy demand. The graph below compares coal demand in the En-ROADS Baseline Scenario with six scenarios produced by leading modeling organizations, including the Network for Greening the Financial System (NGFS), the International Energy Agency (IEA), and the Production Gap Report. The remaining graphs in the grid compare En-ROADS versions of NGFS scenarios to results from the models used by NGFS. Scroll down to see comparisons to additional key variables.
Table of Contents🔗
- Preliminary Findings
- Exceptions and Caveats
- Testing En-ROADS Against Other More Disaggregated Integrated Assessment Models
- Understanding En-ROADS Scenario Comparisons
- Economic Input Assumptions Comparisons
- Energy Comparisons
- Total Primary Energy
- Coal Primary Energy Demand
- Coal with CCS Primary Energy Demand
- Oil Primary Energy Demand
- Natural Gas Primary Energy Demand
- Natural Gas with CCS Primary Energy Demand
- Bioenergy Primary Energy Demand
- Bioenergy with CCS (BECCS) Primary Energy Demand
- Nuclear Primary Energy Demand
- Wind and Solar Primary Energy Demand
- Total Final Consumption of Energy Sources
- Final Electric Energy Consumption
- Electric Transport Final Energy Consumption
- Electric Buildings & Industry Final Energy Consumption
- Energy Intensity of GDP
- Emissions Comparisons
- Carbon Sequestration Comparisons
1. Preliminary Findings🔗
Our analysis shows that En-ROADS aligns well with future climate and energy scenarios modeled by major integrated assessment models (IAMs), including those used by the Network for Greening the Financial System (NGFS). When tested under similar conditions, En-ROADS’ results are, on average, statistically as close to those of other IAMs as the IAMs are to one another. This suggests that En-ROADS reliably captures the dynamics of the energy, land, forest, and climate systems in alignment with widely accepted scientific models, while maintaining its strengths of speed, transparency, and ease of use.
Statistical measures, including high correlation (R2) and low error metrics like RMSE, demonstrate a strong alignment between En-ROADS and other IAMs. To explore these metrics, click the three dots in the upper-right corner of a comparison graph on this page and select “Show Statistics.”
The following sections outline our methodology and scenario comparisons in greater detail.
2. Exceptions and Caveats🔗
While we present a broad range of comparisons, not all variables and scenarios are included here. Additionally, it is important to note that these comparisons are not fully independent validation tests. When designing the model, we do not optimize our parameters to match the output of other IAMs, but we do conduct plausibility checks by comparing En-ROADS results to projections from other models. This approach ensures that En-ROADS remains aligned with broader scientific expectations while preserving its independent formulation.
One notable difference between En-ROADS and the other IAMs is in the modeling of bioenergy: IAMs used by NGFS typically assume lower net emissions from bioenergy than En-ROADS does. Our approach to bioenergy, which accounts for system-wide impacts, is described in Sterman, Siegel, & Rooney-Varga (2018).
3. Testing En-ROADS Against Other More Disaggregated Integrated Assessment Models🔗
En-ROADS belongs to a category of more aggregated, decision-maker-oriented integrated assessment models (IAMs), complementing larger, more disaggregated models such as GCAM, MESSAGEix-GLOBIOM, and REMIND-MAgPIE. The larger models provide richer detail in many areas but take a significant amount of computational power to run and return results after a delay, sometimes in hours or days. En-ROADS, in contrast, returns results in less than a second, enabling real-time policy experimentation by decision-makers, and is designed for simplicity of use and transparency.
The diagram below illustrates these dimensions, with more scope and detail higher on the y-axis and more speed, simplicity of use, and transparency farther along the x-axis. More-aggregated IAMs such as En-ROADS enable users to gain insights that can be refined by more disaggregated models. In turn, the insights of more disaggregated models can inform the design and improve the performance of more-aggregated climate models. These feedbacks are depicted by the two arrows.
The sections below compare En-ROADS scenarios to scenarios generated by models used by four organizations: the Network for Greening the Financial System (NGFS), the International Energy Agency (IEA), the International Atomic Energy Agency (IAEA), and the Production Gap Report.
Click the arrows to reveal more information about these scenarios.
NGFS: Network for Greening the Financial System. (2023). NGFS Phase 4 Scenario Explorer.
An international consortium of central banks and financial institutions, the NGFS contributed to the most recent IPCC Assessment Report (AR6 2022). Three different integrated assessment modeling teams contributed to the NGFS scenarios and generated their own versions of each of the NGFS scenarios. For simplicity, only four of the seven NGFS scenarios are shown below, although we also compare En-ROADS behavior to the NGFS Low Demand, Delayed Transition, and Fragmented World scenarios in our testing.
The NGFS’s Current Policies Scenario assumes that the only policies that continue into the future are policies that were implemented by Fall 2022. This is similar to the assumptions in the En-ROADS Baseline Scenario, making it a useful comparison.
Models:
- GCAM (JGCRI/PNNL, USA)
- REMIND-MAgPIE (PIK, Germany)
- MESSAGEix-GLOBIOM (IIASA, Austria)
Scenarios (descriptions are from the NGFS Climate Scenarios Technical Documentation v4.2):
- Current Policies: “Only currently implemented policies are preserved. Existing climate policies remain in place but there is no strengthening of ambition level of these policies.”
- NDCs: “All pledged targets are assumed to be implemented, even if they are not yet backed up by effective policies. Countries implement pledged policies in addition to current policies and keep their level of ambition beyond the NDC horizon.”
- Below 2°C: “The stringency of climate policies is gradually increased, giving a 67% chance of limiting global warming to below 2°C by the end of the century. Global CO2 emissions evolve such that the end-of-century temperature goal of 2°C warming is reached (with a 67% chance). Countries who have net zero targets follow through on 80% of them, others follow less ambitious trajectories.”
- Net Zero 2050: “Global warming is limited to 1.5°C (with a 50% chance) through stringent climate policies and innovation, reaching global net zero CO2 emissions around 2050. Global CO2 emissions reach or approach zero in 2050. Countries with a political commitment to a net zero target defined before end of March 2023 reach net zero at their target year or earlier. Some jurisdictions such as the US, EU, UK, Canada, Australia, and Japan reach net zero for all GHGs.”
IEA WEO: International Energy Agency. (2024). World Energy Outlook 2024.
Scenario (description is from the IEA):
- Stated Policies (STEPS): “This scenario provides a sense of the prevailing direction of travel for the energy sector based on a detailed reading of the latest policy settings in countries around the world. It accounts for energy, climate and related industrial policies that are in place or that have been announced. The aims of these policies are not automatically assumed to be met; they are incorporated in the scenario only to the extent that they are underpinned by adequate provisions for their implementation.”
IAEA: International Atomic Energy Agency. (2024). Energy, Electricity and Nuclear Power Estimates for the Period up to 2050.
Scenarios (descriptions are from the IAEA):
- Low: “current market, technology and resource trends continue and there are few additional changes in explicit laws, policies and regulations affecting nuclear power. This case was designed to produce a conservative and plausible projection. Additionally, the low case does not assume that targets for nuclear power in a particular country will necessarily be achieved.”
- High: “more ambitious than the low case, while remaining plausible and technically feasible, and it is possible that capacity could increase beyond that projected in the high case. However, enabling factors would be necessary to help facilitate reaching — or exceeding — the high case, including national policies and strategies, supporting investment, demonstration projects for new reactor technologies, investment in grids, supply chain management for reactor construction, regulatory collaboration and global harmonization (particularly for SMR development), and work force development.”
Production Gap Report: SEI, Climate Analytics, E3G, IISD, and UNEP. (2023). The Production Gap: Phasing down or phasing up? Top fossil fuel producers plan even more extraction despite climate promises.
Scenario (description is from the Production Gap Report):
- Government plans and projections: “A global pathway of future fossil fuel production estimated in this report, based on the compilation and assessment of recent national energy plans, strategy documents, and outlooks published by governments and affiliated institutions.”
4. Understanding En-ROADS Scenario Comparisons🔗
Baseline Scenario Comparisons🔗
The first graph in each grid below compares the En-ROADS Baseline Scenario to low-climate-policy scenarios from the NGFS, IEA, IAEA, and Production Gap Report. Note that the En-ROADS Baseline Scenario represents the state of the world if societal and technological changes were to continue at their current rate of progress, without additional policies or action. Learn more in the En-ROADS Baseline Scenario chapter in the En-ROADS User Guide.
The En-ROADS Baseline Scenario uses different assumptions than the NGFS IAMs—for example, population in the En-ROADS Baseline Scenario is higher because it follows United Nations population projections, and the carbon prices in the NGFS Current Policies Scenario grow higher than the carbon price in the En-ROADS Baseline Scenario. The variation between the En-ROADS Baseline Scenario and the IAMs producing the NGFS Current Policies Scenario is similar to the variation among the IAMs themselves within the NGFS Current Policies Scenario.
Simulating NGFS Scenarios Using En-ROADS🔗
Another test of En-ROADS is to determine if En-ROADS behaves similarly to the other IAMs when run under similar conditions, including population and economic growth assumptions. The remaining graphs in each grid below compare En-ROADS versions of NGFS scenarios to results from the IAMs used by NGFS. To perform this test, we adjust key settings—such as carbon price and deforestation—to align as closely as possible with each NGFS scenario. The exact inputs to the other IAMs are not published, so we used scenario descriptions and output comparisons to adjust En-ROADS settings to match the overall trends of the NGFS scenarios for these tests.
Note, atmospheric concentration data for NGFS Phase 5 (November 2024) has not been released, so the graphs here compare with NGFS Phase 4, which includes this data.
To view the En-ROADS versions of the NGFS scenarios in the En-ROADS app, click on the three dots on the top right of the graph and select “Open Scenario in En-ROADS.”
Click the arrow to display the En-ROADS settings used to create the En-ROADS version of each NGFS scenario.
En-ROADS Settings for Approximating NGFS Scenarios
Definitions of Statistical Measures🔗
For each output variable under each scenario, we calculate statistical error measures to assess how close results from two models are to each other. Click on the three dots on the top right of the graph and select “Show Statistics” to open the statistics pane on a given graph.
Statistical measures of closeness included here:
- R2 (coefficient of determination) shows how closely the results from one model match another. Higher R2 values are better, as they mean the two models produce similar results.
- Symmetric Mean Absolute Percentage Error (SMAPE) measures the difference between two sets of results, adjusted for the size of the values. Lower SMAPE is better, as it means less error between the datasets. It's especially useful when values are very small, like near-zero emissions.
- Root Mean Square Error (RMSE) shows, on average, how much the two datasets differ. Lower RMSE is better, since it indicates smaller differences.
- Mean Squared Error (MSE) can be broken into three parts:
- Bias (lower is better) shows a systematic gap between the datasets.
- Unequal variance (lower is better) shows systematic differences in trends or direction of changes in response to policy.
- Unequal covariance (lower is better, but usually less concerning in long-term models) reflects random, short-term differences.
5. Economic Input Assumptions Comparisons🔗
The graphs below show global GDP ($US 2021 purchasing power parity) and carbon price in the En‑ROADS Baseline and En-ROADS versions of NGFS scenarios, alongside the NGFS IAMs. Unlike the other graphs on this page—which display model outputs—these two graphs represent two of the inputs that drive the simulations.
Economic Growth🔗
Economic growth in the En-ROADS Baseline Scenario is affected by the economic impact of climate change. Learn more in the Explainer: Economic Impact of Climate Change in En-ROADS.
Carbon Price🔗
The En-ROADS Baseline Scenario includes a carbon price rising up to $5/ton CO2 and continuing throughout the century. Learn more in this FAQ: How is the current global carbon price calculated?
6. Energy Comparisons🔗
Total Primary Energy🔗
En-ROADS, as well as many other sources, assumes that nuclear energy has an efficiency of 100% conversion of primary energy into electricity generated. Some sources, like the IEA WEO, assume that the primary energy equivalent from the electricity generation has an efficiency of 33%. To compare En-ROADS output to the IEA WEO, we multiply the primary energy from nuclear in En-ROADS by 3.
Primary Energy from Coal🔗
Primary Energy from Coal with CCS🔗
Primary Energy from Oil🔗
Primary Energy from Natural Gas🔗
Primary Energy from Natural Gas with CCS🔗
Primary Energy from Bioenergy🔗
Primary Energy from Bioenergy with CCS (BECCS)🔗
Primary Equivalent Energy from Nuclear🔗
En-ROADS, as well as many other sources, assumes that nuclear energy has an efficiency of 100% conversion of primary energy into electricity generated. Some sources, like the IEA WEO, assume that the primary energy equivalent from the electricity generation has an efficiency of 33%. To compare En-ROADS output to the IEA WEO, we multiply the primary energy from nuclear in En-ROADS by 3.
Primary Energy from Wind and Solar🔗
Total Final Consumption of Energy Sources🔗
Final Electric Energy Consumption🔗
Total Final Energy Consumption - Electric Transport🔗
Total Final Energy Consumption - Electric Buildings & Industry🔗
Energy Intensity of GDP🔗
7. Emissions Comparisons🔗
Greenhouse Gas Net Emissions🔗
Greenhouse gas net emissions in En-ROADS include CO2 emissions from bioenergy. In contrast, the IAMs modeling the NGFS scenarios appear to exclude bioenergy emissions from their accounting of greenhouse gas emissions. The graphs below compare En-ROADS to the NGFS IAMs for (1) greenhouse gas net emissions and (2) greenhouse gas net emissions excluding CO2 from bioenergy.
CO2 Net Emissions🔗
CO2 net emissions in En-ROADS include CO2 emissions from bioenergy. In contrast, the IAMs modeling the NGFS scenarios appear to exclude bioenergy emissions from their accounting of CO2 emissions. The graphs below compare En-ROADS to the NGFS IAMs for (1) CO2 net emissions and (2) CO2 net emissions excluding CO2 from bioenergy.
CO2 Net Emissions from Land Use, Land Use Change, & Forestry (LULUCF)🔗
CO2 net emissions from land use, land use change, and forestry (LULUCF) in En-ROADS include CO2 emissions from bioenergy. In contrast, the IAMs modeling the NGFS scenarios appear to exclude bioenergy emissions from their accounting of CO2 emissions. The graphs below compare En-ROADS to the NGFS IAMs for (1) CO2 net emissions from LULUCF and (2) CO2 net emissions from LULUCF excluding CO2 from bioenergy.
N2O Emissions🔗
CH4 Emissions🔗
F-Gas Emissions🔗
The Baseline Scenario in En-ROADS does not assume that the Kigali Amendment to the Montreal Protocol, which specifies 80% HFC phase out by 2047, is fully effective.