Finding an optimized portfolio of machine learning models using Sklearn, LazyPredict, and Precise Packages

In this post, I will provide an example of the use of the precise Python package (and PyPortfolioOpt) to create a diversified portfolio of scikit-learn models fitted using LazyPredict.

Background

For now, I’m content to draw the reader’s attention to the close parallels between financial portfolio construction and model ensembles. A linear combination of models is certainly akin to a portfolio of assets, insofar as the assessment is mathematically similar and sometimes, identical.

In the case of models assumed to be unbiased, we take an interest in the squared error of the combined model (or some similar metric). Likewise, in finance, we are sometimes interested in minimizing the variance of the value of a linear combination of securities (or, again, some proximate objective).

Interestingly both the forecasting literature and also the financial literature include the same stern empirical warning in different terminology. Over-concentration can hurt the long-term returns of a portfolio, just as over-reliance on a single model can hurt out-of-sample performance.

We also find empirical warnings in the literature about getting too fancy in portfolio construction (analogously, the combining of expert opinions, forecast models, or model outputs). The surprising difficulty in improving on what would seem to be a low bar (equal weights) is a refrain you’ll hear quite often.

It is a reasonable concern, and as an example, the dangers of point estimates of covariance are well appreciated — a topic you can spend hours reading about (here’s a list of papers) should you wish.

Yet theory and practice advance so the notion that cleverness does more harm than good is probably somewhat overwrought, in my opinion, and certainly should not dissuade people working in data-rich domains. It is my intent, with the creation of the precise package and some interoperability with AutoML, to gently push back against that perception.

In particular, I suspect that the “blending” stage of automated machine learning will start to borrow more from finance.

The Plan

My modest goal here is to take a standard sklearn dataset off the shelf and see if we can easily construct a “pretty good” autonomously created combination of models that serves you better than choosing the best one. A more careful comparison against various varieties of cross-validation will have to wait, but I’m providing you with the ingredients to charge ahead with that, should you choose to do so.

To this end, we’ll break the data into three partitions called “train”, “test” and “validate”. Here “test” is something of a misnomer as we’ll be doing the following:

1. Train on X_train, y_train
2. Select the best model based on X_test, y_test out of sample performance
3. Also estimate model portfolio weights using model residuals for X_test, y_test, and standard portfolio construction technique.
4. Retrain models on X_train+X_test
5. Compare the performance of the best model against a weighted combination, with weights from step 3.

(As a semantic aside one notes that choosing “best” model as determined above is, of course, a special case of a portfolio — one where all the mass is on one model. I should probably add it to the precise package!)

Preliminaries

Getting to nuts and bolts, we’ll need LazyPredict and precise packages. If you are using colab or an environment with a pre-provided pandas, save yourself possible annoyance. For instance on colab:

!pip install lazypredict
!pip install precise
!pip install --upgrade pandas 

then restart the environment. We’ll be using:

from sklearn import datasets
from sklearn.utils import shuffle
import numpy as np
from pprint import pprint
from lazypredict.Supervised import LazyRegressor

and the very same sklearn example as you’ll find in the LazyPredict README, the Boston dataset.

boston = datasets.load_boston()
X, y = shuffle(boston.data, boston.target)
X = X.astype(np.float32)

A careful reading of the sklearn documentation reminds us that this dataset has ethical problems and should not be used unless the purpose is to educate people about data ethics. Of course, of course, that is my intent. You should all buy and read my fabulous upcoming book “Microprediction: Building an Open AI Network” to be published by MIT Press in the fall. There you will find a long discussion about model fairness and why it can only be solved by a prediction web.

Anyway, back to our half-day effort to defeat enterprise AutoML…

The LazyPredict Package

It’s called LazyPredict for a reason, I guess. This nifty little package saves you quite a few lines of code marshaling scikit-learn models and their predictions en masse. For instance, the use of LazyRegressor below creates a performance table for models trained on X_train and tested on X_test:

reg1 = LazyRegressor(verbose=0, ignore_warnings=False, custom_metric=None, predictions=True)
models1, predictions1 = reg1.fit(np.copy(X_train), np.copy(X_test), np.copy(y_train), np.copy(y_test))

(You could do the same with pycaret or autoviml or similar packages and I started down that path. However, the interactivity assumption of pycaret slowed me down just enough to switch over, for the purpose of this particular experiment. I’ll write some adaptors for the popular choices soon enough).

The predictions1 output gives us the data we need, and now we’ll turn to the precise package to infer a combination of models.

The Precise Package

The precise package comprises a collection of “managers”, among other things. A manager is someone who receives a vector (which might be a collection of contemporaneous returns for several stocks but here comprises model residuals) and updates portfolio weights.

The package contains some novel approaches to this task, and also calls down to the excellent PyPortfolioOpt and Riskfolio-Lib packages. There is a full listing of managers in the file LISTING_OF_MANAGERS, would you believe?

In the Boston dataset, the rows are, I assume, non-temporal. The precise package is aimed mostly at real-time temporal problems, but that doesn’t really matter here.

For some managers we let Riskfolio-Lib do all the work, starting with an accumulated buffer of historical data. For others, a portfolio manager might be a combination of two choices:

1. A method of estimating covariance in online fashion, combined with
2. A way of determining a portfolio from a covariance forecast (sometimes using PyPortfolioOpt).

The explosion of possibilities leads to rather long names for fully autonomous managers, I’m afraid. If you squint at the import below and then refer to the precise README you might infer I’m using PyPortfolioOpt’s implementation of a minimum volatility portfolio and Graphical Lasso with cross-validation for covariance estimation.

from precise.skaters.managers.ppomanagers import ppo_sk_glcv_pcov_d0_n100_t0_vol_long_manager as mgr
s = {}
yhat_test = np.copy(predictions1.values)
n_test = len(yhat_test)
es = [-1]*(n_test-1)+[1]
for y, y_target,e in zip(yhat_test, y_test,es):
    y_error = np.copy(y-y_target)
    w, s = mgr(s=s, y=y_error, e=e)

you might also observe the style of calling the manager. I’m feeding it one vector of model residuals at a time, and I’m saving the state s on its behalf from one invocation to the next. Once you get that part of the joke, the use of the precise package is trivial.

(Don’t worry about the use of e here. That’s a little performance hack. It tells the portfolio manager that we only care about the final w. I refer you to the documentation, such as it is, for more explanation of “skating”.)

The Portfolio and Combined Model

The covariance of model residuals on the test data set suggests to the manager a diversified portfolio as follows:

Creating portfolio ...
[(0.07594000000000001, 'DecisionTreeRegressor'),
 (0.059660000000000005, 'GeneralizedLinearRegressor'),
 (0.055560000000000005, 'KNeighborsRegressor'),
 (0.05206000000000001, 'NuSVR'),
 (0.04846000000000001, 'TransformedTargetRegressor'),
 (0.04558000000000001, 'PoissonRegressor'),
 (0.044770000000000004, 'LarsCV'),
 (0.04420000000000001, 'PassiveAggressiveRegressor'),
 (0.043030000000000006, 'LinearSVR'),
 (0.04299000000000001, 'XGBRegressor'),
 (0.04263000000000001, 'SVR'),
 (0.04241000000000001, 'LinearRegression'),
 (0.04241000000000001, 'GradientBoostingRegressor'),
 (0.04241000000000001, 'DummyRegressor'),
 (0.03783000000000001, 'BayesianRidge'),
 (0.029790000000000004, 'RandomForestRegressor'),
 (0.029210000000000003, 'Ridge'),
 (0.026990000000000004, 'TweedieRegressor'),
 (0.025740000000000002, 'KernelRidge'),
 (0.024570000000000005, 'OrthogonalMatchingPursuitCV'),
 (0.022060000000000003, 'Lars'),
 (0.019960000000000002, 'ElasticNet'),
 (0.019730000000000004, 'GaussianProcessRegressor'),
 (0.018320000000000003, 'LassoLarsCV'),
 (0.014460000000000002, 'RANSACRegressor'),
 (0.014240000000000001, 'BaggingRegressor'),
 (0.009740000000000002, 'GammaRegressor'),
 (0.009420000000000001, 'HuberRegressor'),
 (0.007860000000000002, 'LassoLarsIC'),
 (0.005240000000000001, 'SGDRegressor'),
 (0.0027300000000000002, 'LassoCV')]

Once we have generated out-of-sample predictions for these models, we simply use a linear combination.

yhat_weighted = np.dot( yhat_val, w )

As an aside, I’m using a “long-only” manager, in financial parlance, for stability. In some cases, one might wish to relax that constraint, with due care.

Out-of-sample performance

Obviously, we don’t have a crystal ball to tell us which model will perform best out of the sample, but I’ve produced that ex-post squared-error leaderboard anyway as a point of reference.

I’ve inserted into that impossible leaderboard our weighted portfolio of models, and also the best model (which in this example was OrthogonalMatchingPursuit). As you can see, the portfolio was better on this occasion.

If you look at the script version in precise/examples_ensembles_lazypredict there is a little Monte Carlo study. It strongly suggests that using a portfolio of models is superior to choosing the best one, at least for the Boston dataset. (I’ve also included the best model not retrained on the test set, to anticipate that comment.)

Full Code

from sklearn import datasets
from sklearn.utils import shuffle
import numpy as np
from pprint import pprint
from lazypredict.Supervised import LazyRegressor

boston = datasets.load_boston()
X, y = shuffle(boston.data, boston.target)
X = X.astype(np.float32)
n_train = 100
n_test = 50
X_train, y_train = X[:n_train], y[:n_train]
X_test, y_test = X[n_train:(n_train+n_test)], y[n_train:(n_train+n_test)]
X_val, y_val = X[(n_train+n_test):], y[(n_train+n_test):]
X_train_and_test = X[:(n_train+n_test)]
y_train_and_test = y[:(n_train+n_test)]

# Train on some, predict test
reg1 = LazyRegressor(verbose=0, ignore_warnings=False, custom_metric=None, predictions=True)
models1, predictions1 = reg1.fit(np.copy(X_train), np.copy(X_test), np.copy(y_train), np.copy(y_test))
print(models1[:5])

# Train on some, predict validation
reg2 = LazyRegressor(verbose=0, ignore_warnings=False, custom_metric=None, predictions=True)
X_train_and_test_copy = np.copy(X_train_and_test)
X_val_copy = np.copy(X_val)
models2, predictions2 = reg2.fit(X_train_and_test_copy, X_val_copy, np.copy(y_train_and_test), np.copy(y_val))
yhat_val = predictions2.values
print(models2[:5])

# In-sample performance on train
reg3 = LazyRegressor(verbose=0, ignore_warnings=False, custom_metric=None, predictions=True)
models3, predictions3 = reg3.fit(np.copy(X_train), np.copy(X_train), np.copy(y_train), np.copy(y_train))

# In-sample performance on train + test
reg4 = LazyRegressor(verbose=0, ignore_warnings=False, custom_metric=None, predictions=True)
models4, predictions4 = reg4.fit(np.copy(X_train_and_test), np.copy(X_train_and_test), np.copy(y_train_and_test), np.copy(y_train_and_test))

best_model_1 = models1.index[0]  # <-- Best out of sample on test
best_model_2 = models3.index[0]  # <-- Best in sample on train
best_model_3 = models4.index[0]  # <-- Best in sample on train+test

# Train cov on out of sample prediction errors
print('Creating portfolio ...')
from precise.skaters.managers.ppomanagers import ppo_sk_glcv_pcov_d0_n100_t0_vol_long_manager as mgr
s = {}
yhat_test = np.copy(predictions1.values)
n_test = len(yhat_test)
es = [-1]*(n_test-1)+[1]
for y, y_target,e in zip(yhat_test, y_test,es):
    y_error = np.copy(y-y_target)
    w, s = mgr(s=s, y=y_error, e=e)

w_dict = sorted([(wi,mi) for (wi,mi) in zip(w, models1.index) if wi>0], reverse=True)
pprint(w_dict)

# Refit models using all the train+test data, and combine
sum_w = sum(w)
yhat_weighted = np.dot( yhat_val, w )
predictions2['>> weighted portfolio of models '] = yhat_weighted
predictions2['>> best out of sample model  (' + best_model_1 + ')'] = predictions2[best_model_1]
predictions2['>> best in sample i (' + best_model_2 + ')'] = predictions2[best_model_2]
predictions2['>> best in sample ii (' + best_model_3 + ')'] = predictions2[best_model_3]

val_errors = predictions2.copy()
for col in predictions2.columns:
    val_errors[col] = predictions2[col] - y_val

sq_errors = val_errors**2
print(sq_errors.mean().sort_values())
print('done')

Choosing a Model Portfolio: An Open Problem?

There is a healthy stacking, mixture of experts, and ensembling literature and yet I think the question of the best use of financial portfolio theory in this context is a mostly open question.

One might push back and suggest that it may not be well answered by the empirical financial literature since portfolio methods and their out-of-sample performance depend on the nature of the data-generating process quite strongly — and there are some characteristics of financial portfolios that don’t necessarily carry over to model residuals.

We shall find out soon enough, however. You can expect to see more Elo rating categories in the precise GitHub repo. These will try to get to the bottom of which types of portfolio construction are favored in different model combination contexts.

Source:

https://medium.com/geekculture/optimizing-a-portfolio-of-models-f1ed432d728b