Subject: Information Systems Science
Title: Generalization Capabilities & Robustness of Autoregressive Language Models Fine Tuned for Explainable Tabular Classification: A Case Study in Credit Approval Decisions
Abstract:
Algorithm-made credit approval decisions are subject to strict requirements: they must be predictively stable and robust and the factors driving them must be faithfully explainable to regulators and affected customers alike. Autoregressive language models (LMs) appear increasingly attractive for this task, as they can generate a classification together with a natural language explanation in a single step and recent research reports promising results for LMs fine-tuned on serialized credit application datasets. Yet, an LM’s explanation is itself a generative text prediction rather than a computed account of its decision and evaluation frameworks applied in previous research do not match the rigor a high-risk decision domain such as credit approval demands. This thesis confronts the reported capabilities with the requirements of the domain and investigates whether the task-specific inductive bias of an inherently interpretable machine learning model can be distilled into an autoregressive LM such that its classifications and explanations withstand a domain-appropriate evaluation.
Three EBM teacher models were trained on publicly available credit-risk datasets (German, Australian, Lending Club) and their classifications and rank-ordered local feature attributions were serialized into a fine-tuned Llama 2 7b chat student model. The evaluation combines four safeguards absent from prior work: class-imbalance-aware metrics, a pre-training corpus contamination control based on distributionally similar synthetic datasets, a comparative analysis of explanation diversity between student and teacher model and systematic perturbation of the input feature order at inference time due to concerns stemming from the autoregressive nature of decoder-only language models for applications on tabular data. The results challenge the previously reported feasibility of this distillation paradigm.
Genuine distillation of the classification function succeeded in only one of three evaluated datasets, while seemingly strong performance elsewhere could be shown to stem from majority-class guessing and target-variable memorization from pre-training corpus contamination. Generated explanations showed systematically reduced diversity and proved to be textual artifacts decoupled from the associated classification function in an autoregressive language model. Reordering mathematically identical inputs at inference time resulted in decision flips of up to 25% of classification decisions in an LM.
These findings indicate that autoregressive language models, in their current form, do not meet the robustness and explainability requirements imposed on high-risk credit scoring applications and that conclusions about their capabilities for tabular prediction depend materially on the evaluation safeguards applied. Future research should prioritize encoder-only LM architectures, where the connection between prediction and explanation can be structurally enforced.
Keywords: Tabular Classification, Large Language Model, autoregressive Language Model, Cross Modal Distillation, Explainable Boosting Machine (EBM), Explainable AI (XAI), Credit Scoring, Self Explaining Models