Full Pipeline Outputs

Legacy low-level tutorial: this page documents the pre-clusterer intermediate pipeline. For the current public contract use class-guided regions, regions_, region_metrics_, labels_, and assign_regions().

This tutorial documents the intermediate artifacts worth saving when you run InsideForest with get_detail=True. Each stage records dimensions, key columns, and up to 10 sample rows so the analysis can be audited later.

Optional dependencies

pip install InsideForest==0.3.9
pip install MetaCraft==2025.10.6

import pandas as pd

pd.set_option("display.max_columns", None)

def audit_df(name, df, n=10):
    print(f"{name}: shape={df.shape}")
    print("columns:", df.columns.tolist())
    return df.head(n) if len(df) >= n else df

Output Map

Iris: fit, labels, and regions

from InsideForest import InsideForest
from sklearn.datasets import load_iris

iris = load_iris()
df = pd.DataFrame(iris.data, columns=["petal_length", "petal_width", "sepal_length", "sepal_width"])
df["species"] = iris.target.astype(float)

in_f = InsideForest(
    rf_params={"random_state": 15},
    tree_params={"lang": "py", "n_sample_multiplier": 0.05, "ef_sample_multiplier": 10},
    leaf_percentile=95,
    low_leaf_fraction=0.25,
    get_detail=True,
    var_obj="species",
)
in_f.fit(df)

pred_labels = in_f.predict(df.drop(columns=["species"]))
training_labels = in_f.labels_

Prepared data output: shape=(150, 5). Keep at least 10 rows to review units and target encoding.

Labels and regions

df_plot = pd.DataFrame({"species": iris.target, "training_labels": training_labels})
audit_df("df_plot", df_plot)
audit_df("in_f.df_reres_[0]", in_f.df_reres_[0])

The label table stays aligned to the original target. Records that do not match any rule can receive -1.

df_reres_ is a list of DataFrames. Each table combines lower bounds, upper bounds, and region metrics. If the first table has fewer than 10 rows, show it entirely.

Descriptions, explanation, and frontiers

from InsideForest.descrip import get_frontiers, generate_descriptions

df_datos_explain, frontiers = get_frontiers(in_f.df_clusters_description_, df, divide=5)

audit_df("in_f.df_clusters_description_", in_f.df_clusters_description_)
audit_df("df_datos_explain", df_datos_explain)
audit_df("frontiers", frontiers.sort_values("score", ascending=False))

df_clusters_description_ contains one row per candidate cluster, with the original readable rule and its metrics.

df_datos_explain adds relative percentile columns for each variable used in the rules.

frontiers ranks cluster pairs. The score favors similar pairs with larger metric deltas.

Generated descriptions

descrip_generales = [
    x for x in in_f.df_clusters_description_["cluster_description"].unique().tolist()
    if isinstance(x, str)
]

# Optional when an API key is available.
descrip_gpt = generate_descriptions(descrip_generales[:10], "english", api_k)
audit_df("descriptions", pd.DataFrame({"cluster_description": descrip_generales[:10]}))

Titanic: metadata and experiments

import seaborn as sns
from metacraft import Metadata
from InsideForest.metadata import MetaExtractor, run_experiments

df = sns.load_dataset("titanic")
var_obj = "survived"

metadata = Metadata()
titanic_meta = "https://raw.githubusercontent.com/jcval94/Metadata_files/refs/heads/main/titanic.yaml"
metadata.update(df, titanic_meta, inplace=False, output="titanic.yaml")
metadata.validate(df)

mx = MetaExtractor(metadata.df, var_obj)
audit_df("metadata.df", metadata.df)

metadata.df.head(10): this is the minimum 10-row metadata review. It includes identity, logical type, domain, and actionability.

Experiment construction

percentile_75 = frontiers["score"].quantile(0.75)
frontiers_flt = frontiers[frontiers["score"] > percentile_75]

mis_df2s = {}
for vals in range(len(frontiers_flt) - 1):
    quick_wins = frontiers_flt[["cluster_1", "cluster_2"]].values[vals]
    df_QW_ = df_datos_explain[df_datos_explain["cluster"].isin(quick_wins)]
    mis_df2s[f"experiment_{vals}"] = df_QW_

top_experiments = run_experiments(mx, mis_df2s)
audit_df("top_experiments", top_experiments)

mis_df2s["experiment_0"] usually contains two rows because each experiment compares two clusters.

top_experiments.head(10) ranks contrasts by effectiveness delta, size, intersection, and action difficulty.

df_exper, df_meta_sub, and hypothesis

n_experimento = 0
experimento_ = top_experiments.loc[n_experimento, "dataset"]
cols_act = [
    "cluster_ef_a", "cluster_ef_b", "delta_ef", "variables_intersection",
    "variables_a", "variables_b", "intersection", "only_cluster_a",
    "only_cluster_b", "score",
]
df_exper = top_experiments[top_experiments["dataset"] == experimento_][cols_act]

cols_mtd = [
    "metadata_row", "rule_token", "identity.label_i18n.en",
    "identity.description_i18n.en", "type.logical_type",
    "domain.categorical.codes", "actionability.increase_difficulty",
    "actionability.decrease_difficulty", "actionability.side_effects",
]
df_meta_sub = mx.extract(mis_df2s[experimento_])[cols_mtd]

from InsideForest.descrip import generate_hypothesis

hypothesis = generate_hypothesis(df_meta_sub, df_exper, target="survived", lang="en")
print(hypothesis)

Example local output: Group A has an effectiveness of 0.28 versus 0.72 in group B, a difference of 0.44 for Survived. The lower-effectiveness group is distinguished by Sex and Ticket class; the higher-effectiveness group is distinguished by Sex and Fare. Treat this as a descriptive hypothesis, not a direct recommendation, because it includes sensitive attributes and socioeconomic proxies.

For reproducible reports, store df_clusters_description_, df_datos_explain, frontiers, metadata.df, top_experiments, df_exper, and df_meta_sub next to the notebook.