AI Decoded - Random Forest
Random Forest, a machine learning technique, helps uncover patterns in financial data, reduce noise, and improve equity signals.
Today’s rapidly evolving technological landscape offers a variety of advanced Artificial Intelligence (AI) techniques with the potential to enhance investment strategies. The Systematic Equity team at Allianz Global Investors has been at the forefront of this innovation. Among the many approaches we have explored and researched, we have identified several promising AI-driven signals. One of the key techniques we use to enhance stock selection is Random Forest, a machine learning method renowned for its ability to model non-linear relationships between input variables (e.g., company characteristics) and the outcomes (e.g., future relative returns). After testing various machine learning methods, Random Forest consistently emerged as a standout, delivering stronger, more reliable results.
But what exactly is a Random Forest? The name is actually quite intuitive: as the name “forest” suggests, it’s a group of trees – specifically, decision trees. A decision tree functions like a flowchart, branching through a series of questions to reach a conclusion or decision. For example, as illustrated in the exhibit below, determining whether a person is fit might start by assessing their age and then their habits: people under 30 who eat a lot of pizza are generally categorized as unfit, while those over 30 who exercise regularly are more likely to be fit.
However, in practice, one key challenge with decision trees is their tendency to overfit data, meaning they can become overly tailored to historical specifics, capturing individual events rather than the broader underlying patterns (see Exhibit 2 on the following page for an illustration of overfitting). While this may enhance performance on past data, it often undermines the model’s ability to generalize to future scenarios, weakening predictive power with new or unseen data. This issue is particularly critical in finance, where data is inherently noisy, and market dynamics are in constant flux. Therefore, for a model to be effective, it must strike a balance – filtering out noise while adapting to these ongoing changes as effectively as possible.
Exhibit 1: Decision tree on fitness – a mock example
Source: https://chirag-sehra.medium.com/ decision-trees-explained-easily-28f23241248
Exhibit 2: What is overfitting?
Source: Allianz Global Investors, Systematic Equity team. For illustrative purposes only.
For example, consider a decision tree designed to time investments in the equity market. During the period from 1990 to 2005, indicators such as Russian bond spreads proved useful in “predicting” specific market events like the LTCM’s demise1 in 1998, as illustrated below (Exhibit 3). However, beyond this isolated event, this indicator lost much of its predictive power, as market dynamics shifted and the specific conditions that once made the signal relevant no longer applied.
To reduce overfitting, it is important to conduct a diverse “forest” of decision trees – this is where the Random Forest truly excels. Each tree is trained on different sub-periods of data and subsets of company characteristics, offering a unique perspective by capturing distinct patterns from various historical periods, each shaped by different market regimes. For example, relying on a single decision tree based on indicators such as Russian bond spreads might lead to a short signal as illustrated above. However, other trees may focus on different periods or market conditions, such as one dominated by higher ISM (Institute for Supply Management Manufacturing Index) and consumer confidence (as in the second decision tree in Exhibit 4 on the following page), while another emphasizes periods with higher economic surprises and earnings upgrades (as in the third decision tree in Exhibit 4). By leveraging a “wisdom-of-the-crowd” approach and aggregating predictions through majority voting, the model arrives at a more reliable outcome. In this example, since both the second and third decision trees yielded to the same result, the majority vote leads to a long signal as well, as illustrated in the exhibit below. In summary, the diversity of decision trees is fundamental to success, as it mitigates the risk of over-reliance on any single event or condition, enhancing resilience against challenging market dynamics, unprecedented events and new data.
The Systematic Equity team does not try to time the market but tries to evaluate the attractiveness of all investable stocks at a given point in time. Our database contains long history of company data such as balance sheet and income statement items, prices and returns, and a plethora of alternative data items across a broad investment universe, creating a strong foundation for AI applications like Random Forest. This focus on data quality and breadth is crucial in the training process of any AI model.
The Systematic Equity team’s random forest model, applied to approximately 60 billion USD in assets, leverages hundreds of these established data fields we collected over the past 25 years. With ample data available, our random forest model can grow enough decision trees to ensure stable and reproducible predictions.
1LTCM’s demise: Long-Term Capital Management L.P. (LCTM) was a highly leveraged hedge fund. In 1998, it received a USD 3.6 billion bailout from a
group of 14 banks, in a deal brokered and put together by the Federal Reserve Bank of New York. LTCM was initially successful, with annualized return
of around 21% in its first year, 43% in its second year and 41% in its third year. However, in 1998 it lost USD 4.6 billion in less than four months due to a
combination of high leverage and exposure to the 1997 Asian financial crisis and 1998 Russian financial crisis. Source: https://en.wikipedia.org/wiki/
Long-Term_Capital_Management
Exhibit 3: Relying on one decision tree might lead to bias
Source: Allianz Global Investors, Systematic Equity team. Securities mentioned in this document are for illustrative purposes only and do not constitute a recommendation or solicitation to buy or sell any particular security.
Exhibit 4: Diversity of the decision trees is key to success: 2:1 vote for long
Source: Allianz Global Investors, Systematic Equity team. For illustrative purposes only.
Exhibit 5: An example of a decision tree at the Systematic Equity team
Source: Allianz Global Investors, Systematic Equity team. For illustrative purposes only.
Exhibit 6: Enhancing the information ratio of Value with AI
Source: Allianz Global Investors, Systematic Equity team. Valuation period: 31/12/1998 – 30/09/2024. For illustrative purposes only. The hypothetical performance and simulations shown are for illustrative purposes only and do not represent actual performance; they do not predict future returns. Please see important information regarding back-testings and hypothetical or simulated performance data at the end of this document. For illustrative purposes only and does not represent actual performance of any client account. The information should not be relied upon as an indicator of future results.
