Let’s talk

Blog

AUROC comparison across models on 108 datasets.
Fig. 1 Mean AUROC of each model across 108 small datasets.
Benchmarks 5 min read

Which Machine Learning Classifiers are Best for Small Datasets?

Testing common classifiers on 108 small datasets.

Read the featured post
Schematic slope chart: six models with nearly equal held-out scores spread far apart once deployed.
Fig. 2 Schematic: models with equal held-out scores can diverge once deployed.
Reliability 3 min read

Underspecification in Machine Learning

Why two models with the same held-out score can still behave very differently in production.

A/B test chart of average clicks per query.
Fig. 3 A/B test of the new ranker: clicks on the results page.
Search & ranking 18 min read

Building a Better Search Engine for Ai2

Lessons from using three years of search logs to improve Semantic Scholar’s ranking system.

Sum of absolute SHAP values vs. Bernoulli probability p.
Fig. 4 Total SHAP importance of a Bernoulli feature as p varies.
Interpretability 4 min read

SHAP Values and Feature Variance

How a feature’s variance shrinks or inflates its SHAP importance, and how to explain that to collaborators.