ESG Ratings: The Good, the Bad, the Ugly

Agencies, Methodologies

Headline ratings no longer enough

Prof. Dr. Nils Stieglitz gave a welcome address to the conference “ESG Ratings: The Good, the Bad, the Ugly” of the Corporate Governance Institute (Prof. Dr. Julia Redenius-Hövermann) at the Frankfurt School of Finance & Management, followed by Prof. Dr. Zacharias Sautner, showing data of Hartzmark and Sussman, 2019, proving that investors value sustainability. He introduces the subject “ESG Ratings in the Investment Process”.

“Corporate ESG ratings are the most unseful source of information”, says Zacharias Sautner. This is confirmed by various studies. ESG ratings provide data material to investment performance, supplement organization’s other research of corporate ESG performance/risk.

SustainAbility, an ERM Group company, is a think tank and advisory firm that works to inspire and enable business to lead the way to a sustainable economy. In 2010, SustainAbility undertook its first “Rate the Raters” project to better understand the environmental, social, and governance (ESG) ratings landscape and provide perspectives to help companies, investors and other stakeholders make sense of and derive more value from ESG ratings.

In general, investors recognize that ESG ratings and rankings are not going away any time soon. When
asked what changes and solutions they would like to see in the next five years the leading responses
from the survey were the following:

  • Improved quality and disclosure of methodology
  • Greater focus on relevant/material issues
  • Better linkage to company financial performance
  • Greater consistency and comparability across
  • rating methodologies
  • Greater engagement of rated companies in the
  • evaluation process
  • Consolidation of ratings

These expectations were reiterated in the interviews along with a desire for more timely coverage, more data, integration of ESG into financial reporting and the ability to evaluate corporate societal impact vs. just operational performance.

Dr. Florian Berg, Massachusetts Institute of Technology, spoke about the divergence of ESG ratings. Correlations of the varios raters’ ratings range from 38% to 71%. Based on data from six prominent rating agencies namely, KLD (MSCI Stats), Sustainalytics, Vigeo Eiris (Moody’s), RobecoSAM (S&P Global) the divergence into three sources. “We do not even know the truth, therefore we only can compare”, argues Florian Berg. “We describe ESG ratings in three elements.

  • Scope: which attributes are included?
  • Measuremaent: how ar e these attributes measured?
  • Weights: how are indicators aggreagted into one score?

Aggregation and mesurement are the biggest sources of differences. See Aggregate Confusion: The Divergence of ESG Ratings, 2019.

What are the implications for investors? According to Florian Berg, the following two tasks have to be accomplished: Clarify ESG preferences, scope and weights, and investors have to answer the question: What measurement methodology do you agree with most?

Ingo Speich, Head of Sustainability and Corporate Governance, Deka Investment, points to the fact that investors no longer look only at headline ratings, but dig deeply into the data. He outlines the problem that the regulator requires financial service providers to report on ESG criteria. However, the data required for this are not sufficiently reported by the companies concerned, so that the financial service providers are faced with the difficulty of collecting, processing and passing on this data.

Rating Agency Accused of Using Original, Then Rewritten Data

Agencies, Data

Refinitiv ESG’s rewriting not a one-time event

Prof. Dr. Kornelia Fabisik, Assistant Professor of Finance, Frankfurt School, reports on some discoveries at one of the ESG rating agencies at the conference “ESG Ratings: The Good, the Bad, the Ugly” of the Corporate Governance Institute (Prof. Dr. Julia Redenius-Hövermann) at the Frankfurt School of Finance & Management.

Refinitiv ESG is a key Environmental, Social, and Corporate Governance rating provider offering “one of the most comprehensive ESG databases in the industry”, and its ESG scores have been used (or referenced) in more than 1,500 academic articles since 2003. The scores were initially constructed by ASSET4, a company acquired in 2009 by Thomson Reuters, which became Refinitiv in 2018. Refinitiv ESG data are employed by major asset managers, such as BlackRock, to manage ESG-related investment risks.

“We document large rewriting of ESG ratings of Refinitiv ESG.” The same set of firm-year observations downloaded September 2018 and September 2020 provides evidence. For the full sample Kornelia Fabisik observerd a rewriting of 21% on average, 18 % median.

“Rewriting is not a one-time event”, warns Kornelia Fabisik. “Refinitv ESG continues to adjust the data ex-post, unannounced.”

“In April 2020, Refnitiv ESG changed the methodology used to determine the ratings.” There were two key changes: Treatment of boolean metrics and introduction of a propietary materiality matrix.

The ESG score deviations are related to firm characteristics, especially past stock returns. Firms that performed better in the past experienced rating upgrades. The ESG score deviations strongly affect the classification of firms into different ESG quantities.

“Has data rewriting stopped post methodology change? No,” ist the answer of Kornelia Fabisik, “the database changes on a weekly basis.” She shows concrete examples.

“ESG ratings industry follows an investor-pay model, whereby the data vendors compete on how useful their ratings are for ESG investments.”

“44% of carbon emission obsesrvations (Scope 1 CO2 emissions) have in some way been altered.” Specifically, 23.6% firm-years were added (i.e., data were missing in the 11/2019 download, but filled in for the 02/2021 download), 1.6% were deleted, and 18.4% were modified. The data rewriting affects all years and not just those closest to the end of the sample period.”

Prof. Dr. Kornelia Fabisik’s conclusions, as presented at the Frankfurt School conference:

  • The large differences in results that we document have implications for empirical test strategies using Refinitiv ESG data. Moving forward, researchers and investment professionals need to verify whether the original or rewritten ESG scores are needed to perform their tests.
  • For example, if the practitioners are unaware of the changes, asset managers could erroneously be benchmarked against the rewritten data that were unavailable at the time of portfolio formation (look-ahead bias).
  • She argues that the results reflect the incentive of the data provider to introduce a positive relationship between ESG scores and returns in the data, in order to demonstrate that their ESG scores are useful for data users developing ESG-related investing strategies.
  • Given that ESG research and ESG-related investment strategies are likely to grow even furhter, this is an important caveat for adhering to the status-quo.

Finance Working Paper N° 708/2020, August 2021, Abstract:

The explosion in ESG research has led to a strong reliance on ESG rating providers. The article documents widespread changes to the historical ratings of a key rating provider, Refinitiv ESG (formerly ASSET4). Depending on whether the original or rewritten data are used, ESG-based classifications of firms into ESG quantiles and tests that relate ESG scores to returns change. While there is a positive link between ESG scores and firms’ stock market performance in the rewritten data, the authors fail to observe such a relationship in the initial data. The ESG data rewriting is an ongoing rather than a one-off phenomenon.