LightGBM

LightGBM is a gradient boosting library from Microsoft. Like XGBoost, it builds decision trees one after another, each correcting the errors of the ones before. It sorts feature values into bins before training, limits trees by their number of leaves rather than their depth, and splits pandas category columns directly. People use it on tabular data for regression, classification and ranking. On this page it runs in your browser, so you can try it without installing anything. Run the example first, then paste any snippet below into a new cell to try it.

What the example does

The data is the same 1,000 made-up houses as in the XGBoost example: rooms, floor area and age, with a price from a fixed formula plus noise. lgb.Dataset wraps each set, and the test set gets reference=dtrain, as LightGBM expects for validation data. In params, num_leaves: 31 caps the leaves per tree and learning_rate: 0.05 scales down each tree's correction. feature_fraction: 0.9 gives each tree a random 90% of the columns, but with three columns every tree still gets all three. lgb.train() builds 10 trees. valid_sets=[dtest] scores the test set after each round, but prints nothing without the lgb.log_evaluation() callback. With no early stopping, model.best_iteration is 0, and num_iteration=0 makes predict() use all 10 trees.

The [LightGBM] lines are the library's log, and "verbose": -1 in params hides them. Ten rounds at a learning rate of 0.05 are far too few: the RMSE is about 114,400 and the predictions only span about 369,000 to 666,000, so the chart's points lie on a line much flatter than the diagonal. With 100 rounds the RMSE falls to about 12,900.

Classify with LGBMClassifier

LGBMClassifier and LGBMRegressor follow the scikit-learn interface. This one sorts the wine dataset into its three classes:

import lightgbm as lgb
from sklearn.datasets import load_wine
from sklearn.metrics import accuracy_score
from sklearn.model_selection import train_test_split

X, y = load_wine(return_X_y=True)
X_train, X_test, y_train, y_test = train_test_split(X, y, random_state=0, stratify=y)

clf = lgb.LGBMClassifier(n_estimators=100, verbose=-1)
clf.fit(X_train, y_train)

print("accuracy:", round(accuracy_score(y_test, clf.predict(X_test)), 3))
print(clf.predict_proba(X_test[:3]).round(3))   # one column per class

Stop early on a validation set

lgb.early_stopping(20) ends training once the validation score has not improved for 20 rounds, and lgb.log_evaluation(100) prints the score every 100 rounds:

import lightgbm as lgb
from sklearn.datasets import make_regression
from sklearn.model_selection import train_test_split

X, y = make_regression(n_samples=1000, n_features=5, noise=10, random_state=0)
X_train, X_valid, y_train, y_valid = train_test_split(X, y, random_state=0)

dtrain = lgb.Dataset(X_train, label=y_train)
dvalid = lgb.Dataset(X_valid, label=y_valid, reference=dtrain)
params = {"objective": "regression", "metric": "rmse", "learning_rate": 0.1, "verbose": -1}

model = lgb.train(params, dtrain, num_boost_round=1000, valid_sets=[dvalid],
                  callbacks=[lgb.early_stopping(20, verbose=False), lgb.log_evaluation(100)])
print("best round:", model.best_iteration)
print("validation RMSE:", round(model.best_score["valid_0"]["rmse"], 2))

The returned model keeps only the trees up to the best round (counted from 1), so predict() needs no extra argument.

Use category columns directly

LightGBM splits a pandas category column on its values, such as city == Boston, with no one-hot encoding. Convert text columns with .astype("category") first: an object column raises ValueError: pandas dtypes must be int, float or bool.

import numpy as np
import pandas as pd
import lightgbm as lgb
from sklearn.metrics import mean_absolute_error
from sklearn.model_selection import train_test_split

rng = np.random.default_rng(0)
df = pd.DataFrame({
    "size": rng.uniform(50, 200, 500),
    "city": pd.Categorical(rng.choice(["Austin", "Boston", "Denver"], 500)),
})
price = 3 * df["size"] + df["city"].map({"Austin": 100, "Boston": 300, "Denver": 200}).astype(float)

X_train, X_test, y_train, y_test = train_test_split(df, price, random_state=0)
model = lgb.LGBMRegressor(n_estimators=200, verbose=-1)
model.fit(X_train, y_train)
print("MAE:", round(mean_absolute_error(y_test, model.predict(X_test)), 1))

See which features matter

plot_importance() counts by default how many splits use each feature. With importance_type="gain", it adds up how much those splits reduced the error:

import matplotlib.pyplot as plt
import pandas as pd
import lightgbm as lgb
from sklearn.datasets import load_diabetes

data = load_diabetes()
X = pd.DataFrame(data.data, columns=data.feature_names)
model = lgb.LGBMRegressor(n_estimators=100, verbose=-1)
model.fit(X, data.target)

gain = pd.Series(model.booster_.feature_importance(importance_type="gain"), index=X.columns)
print(gain.sort_values(ascending=False).head(3).round())

lgb.plot_importance(model, importance_type="gain", precision=0)
plt.tight_layout()
plt.show()

Good to know

• In this build, the scikit-learn style classes print FutureWarning: 'force_all_finite' was renamed to 'ensure_all_finite'. It does not change the results, and warnings.filterwarnings("ignore", message=".*force_all_finite") hides it.
• Each leaf needs at least 20 rows by default (min_child_samples, or min_data_in_leaf in params). On 30 rows no split is possible, so every prediction is the training mean. Lower it for tiny datasets.