https://gael-varoquaux.info/programming/stepping-up-as-probabls-cso-to-supercharge-scikit-learn-and-its-ecosystem.html<div class="admonition note"> <p class="first admonition-title">Note</p> <div class="figure align-right"> <img alt="" src="../programming/attachments/probabl_team_2025.png" style="width: 400px;" /> <p class="caption">Probabl’s get together, in falls 2025</p> </div> <p class="last">I’m thrilled to announce that I’m stepping up as <a class="reference external" href="https://probabl.ai/?utm_source=employee_blog&amp;utm_medium=social_employee&amp;utm_campaign=202601_probabl_awareness_post">Probabl</a>’s CSO (Chief Science Officer) to supercharge scikit-learn and its ecosystem, pursuing my dreams of tools that help go from data to impact.</p> </div> <div class="section" id="scikit-learn-a-central-tool"> <h2>Scikit-learn, a central tool</h2> <p>Scikit-learn is central …</p></div><div class="content" morss_own_score="4.455445544554456" morss_score="10.009702521463367"> <h1>Stepping up as probabl’s CSO to supercharge scikit-learn and its ecosystem</h1> <p> under <a href="https://gael-varoquaux.info/tag/open-source.html">open source</a> <a href="https://gael-varoquaux.info/tag/growth.html">growth</a> <a href="https://gael-varoquaux.info/tag/communities.html">communities</a> <a href="https://gael-varoquaux.info/tag/scikit-learn.html">scikit-learn</a> <a href="https://gael-varoquaux.info/tag/inria.html">inria</a> <a href="https://gael-varoquaux.info/tag/probabl.html">probabl</a> <span> &amp;nbsp Read time: 3 min. </span> </p> <div class="admonition note" morss_own_score="2.6923076923076925" morss_score="7.7548076923076925"> <p>Note</p> <img src="https://gael-varoquaux.info/programming/attachments/probabl_team_2025.png"> <p>Probabl’s get together, in falls 2025</p> <p>I’m thrilled to announce that I’m stepping up as <a href="https://probabl.ai/?utm_source=employee_blog&amp;utm_medium=social_employee&amp;utm_campaign=202601_probabl_awareness_post">Probabl</a>’s CSO (Chief Science Officer) to supercharge scikit-learn and its ecosystem, pursuing my dreams of tools that help go from data to impact.</p> </div> <div class="section" id="scikit-learn-a-central-tool" morss_own_score="1.3796791443850267" morss_score="12.379679144385026"> <h2>Scikit-learn, a central tool</h2> <p morss_own_score="7.0" morss_score="9.0">Scikit-learn is central to data-scientists’ work: it is <strong>the most used machine-learning package</strong>. It has grown over more than a decade, supported by volunteers’ time, donations, and grant funding, with a central role of Inria.</p> <img src="https://gael-varoquaux.info/programming/attachments/scikit-learn_clickpy_2025.png"> <p>Scikit-learn download numbers; <a href="https://clickpy.clickhouse.com/dashboard/scikit-learn">reproduce and explore on clickpy</a></p> <p>And the usage numbers keep going up…</p> <p>Scikit-learn keeps growing because it enables crucial applications: machine-learning that can be easily adapted to a given application. This type of AI does not make the headlines, but it is central to the value brought by data science. It is used across the board to extract insights from data and automate business-specific processes, thus ensuring function and efficiency of a wide variety of activities.</p> <br> <p>And scikit-learn is quietly but steadily advancing. The recent releases bring progress in all directions: computational foundations (<a href="https://scikit-learn.org/stable/auto_examples/release_highlights/plot_release_highlights_1_8_0.html#array-api-support-enables-gpu-computations">the array API enabling GPU support</a>), user interface (<a href="https://scikit-learn.org/stable/auto_examples/release_highlights/plot_release_highlights_1_8_0.html#html-representation-of-estimators">rich HTML displays</a>), new models (eg <a href="https://scikit-learn.org/stable/modules/generated/sklearn.cluster.HDBSCAN.html">HDBSCAN</a>, <a href="https://scikit-learn.org/stable/auto_examples/release_highlights/plot_release_highlights_1_8_0.html#temperature-scaling-in-calibratedclassifiercv">temperature-scaling recalibration</a> …), and always algorithmic improvements (release 1.8 brought <a href="https://scikit-learn.org/stable/auto_examples/release_highlights/plot_release_highlights_1_8_0.html#efficiency-improvements-in-linear-models">marked speed ups to linear models</a> or <a href="https://scikit-learn.org/stable/auto_examples/release_highlights/plot_release_highlights_1_8_0.html#decisiontreeregressor-with-criterion-absolute-error">trees with MAE</a>).</p> </div> <div class="section" id="a-new-opportunity-to-boost-scikit-learn-and-its-ecosystem" morss_own_score="3.082872928176796" morss_score="9.587220754263752"> <h2>A new opportunity to boost scikit-learn and its ecosystem</h2> <p morss_own_score="6.0" morss_score="8.5">Probabl recently raised a <a href="https://blog.probabl.ai/probabl-raises-a-13m-in-seed-to-accelerate-enterprise-grade-ai?utm_source=employee_blog&amp;utm_medium=social_employee&amp;utm_campaign=202601_blog_awareness_post">beautiful seed funding</a> from investors who really understand the value and perspective of scikit-learn. We have a unique opportunity to accelerate scikit-learn’s development. Our analysis is that <strong>enterprises need dedicated tooling and partners to build best on scikit-learn</strong>, and we’re hard at work to provide this.</p> <p>2/3rd of probabl’s founders are scikit-learn contributors and we have been investing in all aspects of scikit-learn: features, releases, communication, documentation, and training. In addition, part of scikit-learn’s success has always been to nurture an ecosystem, for instance via its simple API that has become a standard. Thus Probabl is not only consolidating scikit-learn, but also this ecosystem: the <a href="https://skops.readthedocs.io/en/stable/">skops project, to put scikit-learn based models in production</a>, the <a href="https://skrub-data.org">skrub project, that facilitates data preparation</a>, the <a href="https://skore.probabl.ai/?utm_source=employee_blog&amp;utm_medium=social_employee&amp;utm_campaign=202601_skore_awareness_post">young skore project to track data science</a>, <a href="https://fairlearn.org/">fairlearn to help avoiding machine learning that discriminates</a>, and more upstream projects, such as <a href="https://joblib.readthedocs.io/en/stable/">joblib for parallel computing</a>.</p> </div> <div class="section" id="my-obsession-as-probabl-cso-serving-the-data-scientists" morss_own_score="3.0" morss_score="16.0"> <h2>My obsession as Probabl CSO: serving the data scientists</h2> <p morss_own_score="7.0" morss_score="11.0">As CSO (Chief Science Officer) at Probabl, my role is to nourish our development strategy with understanding of machine learning, data science, and open source. Making sure that <strong>scikit-learn and its ecosystem are enterprise ready</strong> will bring resources for scikit-learn’s sustainability, enabling its ecosystem to grow into a standard-setting platform for the industry, that continues <strong>to serve data scientists</strong>. This mission will require consolidating the existing tools and patterns, and inventing new ones.</p> <br> <p>Probabl is in a unique position for this endeavor: Our core is an amazing team of engineers with deep knowledge of data science. Working directly with businesses gives us an acute understanding of where the ecosystem can be improved. On this topic, I also profoundly enjoy working with people who have a different DNA than the historical DNA of scikit-learn, with product research, marketing, and business mindsets. I believe that the union of our different cultures will make the scikit-learn ecosystem better.</p> <p>Beyond the Probabl team, we have an amazing community, with a broader group of scikit-learn contributors who do an amazing job bringing together what makes scikit-learn so versatile, with a deep ecosystem of Python data tools enriched by so many different actors. I’m deeply greatful to the many scikit-learn and pydata contributors. At Probabl, we are very attuned to enabling the open-source contributor community. Such a community is what enables a single tool, scikit-learn, to serve a long tail of diverse usages.</p> </div> <a href="https://gael-varoquaux.info/programming/stepping-up-as-probabls-cso-to-supercharge-scikit-learn-and-its-ecosystem.html">Go Top</a> </div> Wed, 14 Jan 2026 00:00:00 https://gael-varoquaux.info/personnal/maic-you-lived-100-years-what-changed.html<p>At Maïc’s 100th birthday, I asked her “you lived 100 years, what was the most important change for you?”. She mentioned “Internet”. I asked, why was the Internet important to her eyes? Because this is how she kept close contact with her loved ones, sharing travels or discussing everyday …</p><div class="content" morss_own_score="5.702031602708804" morss_score="34.39049976105648"> <h1>Maïc, you lived 100 years, what changed?</h1> <p> under <a href="https://gael-varoquaux.info/tag/family.html">family</a> <a href="https://gael-varoquaux.info/tag/people.html">people</a> <span> &amp;nbsp Read time: 3 min. </span> </p> <p>At Maïc’s 100th birthday, I asked her “you lived 100 years, what was the most important change for you?”. She mentioned “Internet”. I asked, why was the Internet important to her eyes? Because this is how she kept close contact with her loved ones, sharing travels or discussing everyday life on her phone, her tablet…</p> <br> <p>Born in 1925, she was of a generation sometimes called the silent one. And indeed, she was often low-key. Her father was an administrator in the countryside, and she arrived in Paris in her youth. She studied maths, joining the prestigious “Ecole Normale Supérieure”, which provided her with an income and led her to become a maths teacher. After meeting and marrying <a href="https://gael-varoquaux.info/personnal/jean-dechoux-june-13rd-1923-feb-9th-2020.html">Jean Dechoux</a>, she used her income to fund his medical studies. The story goes that, living in a tiny room, she had to cook on the balcony.</p> <p>Maïc was a teacher, one of those unsung heroes that have educated the masses. Nowadays, this is not a job title that is much acclaimed, unlike say “start-up founder”. But the only reason we have good computer scientists that create start-ups, the only reason we have researchers to build computer science, is because they had great teachers. Maïc was also a mother, a foster mother, a grandmother, a great grandmother. She was kind, humble, tireless, always positive. Her life philosophy was focused on doing the best with what she got.</p> <p>Maïc never seemed left behind by the transformations of our world. Turning 100-years old, she was as sharp as ever, reading book after book and using her phone, her tablet, her computer. Whenever I hear how technology changes the world, I cannot help thinking of her, a 100-year-old geek. The world went through many transformations during her lifetime. But what she saw in these transformations, in Internet technology, is a way to stay in contact with others, a way to bring more humanity into our lives.</p> <img src="https://gael-varoquaux.info/personnal/attachments/nicole_dechoux.jpg"> <br><p><em>Remembering Nicole Dechoux, May 03rd 1925 - October 22nd 2025</em></p> <br> <br> <div class="poem docutils container" morss_own_score="3.0" morss_score="27.0"> <p>Il restera de toi ce que tu as donné</p> <p>Au lieu de le garder dans des coffres rouillés…</p> <p>Ce que tu as donné en d’autres fleurira…</p> <br> <p>Il restera de toi ce que tu as offert</p> <p>Entre tes bras ouverts un matin au soleil…</p> <p>Ce que tu as offert en d’autres revivra…</p> <br> <p>Il restera de toi un sourire épanoui</p> <p>Aux bords de tes lèvres comme au bord de ton cœur…</p> <p>Ce que tu as ouvert en d’autres grandira…</p> <br> <p>Il restera de toi ce que tu as semé</p> <p>Que tu as partagé aux mendiants du bonheur…</p> <p>Ce que tu as semé en d’autres germera…</p> <br> </div> <p><em>Adapted from Simone Weil and Michel Scouarnec</em></p> <a href="https://gael-varoquaux.info/personnal/maic-you-lived-100-years-what-changed.html">Go Top</a> </div> Wed, 29 Oct 2025 00:00:00 https://gael-varoquaux.info/personnal/a-national-recognition-but-science-and-open-source-are-bitter-victories.html<img alt="" class="align-right" src="../personnal/attachments/gael_speech.jpg" style="width: 400px;" /> <p>I have recently been awarded <a class="reference external" href="https://en.wikipedia.org/wiki/Ordre_national_du_M%C3%A9rite">France’s national order of merit</a>, for my career, in science, in open source, and around AI.</p> <p>The speech that I gave carries messages important to me (French below; it flows better).</p> <div class="contents topic" id="contents"> <p class="topic-title"><a class="reference internal" href="#top">Contents</a></p> <ul class="simple"> <li><a class="reference internal" href="#speech-translated-to-english" id="toc-entry-1">Speech translated to English</a></li> <li><a class="reference internal" href="#le-texte-d-origine-en-francais" id="toc-entry-2">Le texte d’origine, en Français</a></li> </ul> </div> <style> .content p …</style><div class="content" morss_own_score="5.592334494773519" morss_score="16.151050313116667"> <h1>A national recognition; but science and open source are bitter victories</h1> <p> under <a href="https://gael-varoquaux.info/tag/award.html">award</a> <a href="https://gael-varoquaux.info/tag/open-source.html">open source</a> <a href="https://gael-varoquaux.info/tag/science.html">science</a> <span> &amp;nbsp Read time: 6 min. </span> </p> <img src="https://gael-varoquaux.info/personnal/attachments/gael_speech.jpg"> <p>I have recently been awarded <a href="https://en.wikipedia.org/wiki/Ordre_national_du_M%C3%A9rite">France’s national order of merit</a>, for my career, in science, in open source, and around AI.</p> <p>The speech that I gave carries messages important to me (French below; it flows better).</p> <p><a href="https://gael-varoquaux.info/personnal/a-national-recognition-but-science-and-open-source-are-bitter-victories.html#top">Contents</a></p> <div class="section" id="speech-translated-to-english" morss_own_score="2.9387755102040813" morss_score="44.93877551020408"> <h2><a href="https://gael-varoquaux.info/personnal/a-national-recognition-but-science-and-open-source-are-bitter-victories.html#toc-entry-1">Speech translated to English</a></h2> <br> <p>Receiving such a medal is a powerful symbol. But what battles does it honor?</p> <p>My first battle, my first dream, was that of science, with the hope of understanding and improving the world. I probably turned to computers because they were simpler, less frightening, than society.</p> <p>This led me to my second battle: the dream of democratizing this science and these digital tools, thanks to open source, also in the hope of making a better world.</p> <p>The freedom I enjoyed, a privilege of researchers, allowed me to devote my time to these dreams. And many people helped on this journey, such as my colleagues at Inria and elsewhere –science is a team sport–, or free software developers from all over the world.</p> <br> <p>And two decades later, we have won. Open source is everywhere. Statistical algorithms raise billions of dollars. But what good will this free software, these algorithms, have been if an Elon Musk can buy their vector of action and transform it into a fascist machine. This victory is bitter.</p> <br> <p>Science, open source, come to play within a societal context, mediated by norms and means of action. These means of action are rooted in economic rationality, and I find myself, to my great surprise, interested in commercial and financial logics.</p> <p>Money is power. It is the ability to build, to buy Twitter or to finance Wikipedia. For science or open source to be successful, we need economic ambitions.</p> <p>But I do not want to reduce the world to economic motivations. Science and free software result from the work of individuals who believe in what they are doing. With scikit-learn, as with many other open source projects, humble developers with few resources have created incredible wealth.</p> <p>And it is these battles that today’s medal rewards. I have always been wary of individual distinctions. Success is rarely the work of a single person. We need more collective effort and fewer heroes, less ego.</p> <p>And yet, I hope that this medal, this symbol, can be useful. Indeed, symbols create the collective narrative, and control the choices we make, individually or as a society. For both science and free software, the risk is to be invisible, unheard, and powerless.</p> <br> <p>Neither lines of code nor equations will be enough to make a better world. The privilege of a researcher is the independence of thoughts necessary for the consolidation of knowledge. The unique strength of open source software is to offer independence to the user. Beyond independence, this knowledge and these software are only useful if society embraces them. And for that, we must win the battle of the narrative.</p> <br> <p>Today, I have only one dream: that our children live in the best possible world. Between the global rise of fascism and climate warming, this dream faces many challenges. But we can fight for it. For this, as always, we need to gather people and unite around the right causes. And thus, I thank you all for the support and help you have given me across the years, for today’s recognition.</p> <p>✶ ✶ ✶</p> <br> </div> <div class="section" id="le-texte-d-origine-en-francais" morss_own_score="5.869565217391305" morss_score="48.369565217391305"> <h2><a href="https://gael-varoquaux.info/personnal/a-national-recognition-but-science-and-open-source-are-bitter-victories.html#toc-entry-2">Le texte d’origine, en Français</a></h2> <br> <p>Recevoir un tel insigne est un symbole puissant. Mais quels combats décore-t-il?</p> <p>Mon premier combat, mon premier rêve a été celui de la science, avec l’espoir de comprendre et d’améliorer le monde. Je me suis probablement tourné vers les ordinateurs car ils étaient plus simples, moins effrayants, que la société.</p> <p>Un deuxième combat est né en moi: le rêve de démocratiser cette science et ces outils numériques, grâce au logiciel libre, toujours dans l’espoir de faire un monde meilleur.</p> <p>La liberté dont j’ai joui, privilège inouï des chercheurs, m’a permis de me consacrer à ces rêves. Et beaucoup m’ont aidé: mes collègues à Inria et ailleurs, car la science est un sport d’équipe; les développeurs logiciels libres partout dans le monde; mes parents, qui m’ont donné l’amour de la science même lorsque j’étais en échec scolaire.</p> <br> <p>Et deux décennies plus tard, nous avons gagné. Les logiciels libres sont partout. Les algorithmes statistiques font des levées de fonds de plusieurs milliards. Mais à quoi auront servi ces logiciels libres, ces algorithmes, si un Elon Musk peut racheter leur vecteur d’action et le transformer en machine à fascisme. Cette victoire est amère.</p> <br> <p>La science, le logiciel libre, se réalisent dans un contexte sociétal, médié par des normes et des moyens d’actions. Ces moyens d’actions sont ancrés dans le rationnel économique, et je me trouve, à ma grande surprise, à m’intéresser à des logiques commerciales et financières.</p> <p>L’argent, c’est le pouvoir. C’est la capacité de réaliser, de racheter twitter ou de financer wikipedia. Pour le succès de la science ou du logiciel libre, nous avons besoin d’une ambition économique.</p> <p>Mais je ne voudrais réduire le monde aux motivations économiques. La science et le logiciel libre résultent du travail d’individus qui croient à ce qu’ils font. Avec scikit-learn, comme avec beaucoup d’autres logiciels libres, des développeurs humbles et avec peu de moyens ont créé une richesse incroyable.</p> <p>Et c’est ces combats que récompense aujourd’hui l’insigne que je reçois. Je me suis toujours méfié des distinctions individuelles. Un succès est rarement l’œuvre d’un seul. Nous avons besoin de plus de collectif et de moins de héros, de moins d’égo.</p> <p>Et pourtant, j’espère que cette médaille, ce symbole, peut être utile. En effet, les symboles créent le récit collectif, et contrôlent les choix que nous faisons, individuellement ou en tant que société. Science comme logiciel libre, le risque est d’être invisibles, inaudibles, et impuissants.</p> <br> <p>La ligne de code, ou l’équation, ne suffiront à faire un meilleur monde. Le privilège du chercheur, c’est l’indépendance de pensée nécessaire à la consolidation de la connaissance. L’atout du logiciel libre, c’est d’offrir une indépendance à l’utilisateur. Au-delà de l’indépendance, cette connaissance et ces logiciels ne sont utiles que si la société s’en empare. Et pour cela, il nous faut gagner la bataille du récit.</p> <br> <p>Aujourd’hui, je n’ai plus qu’un rêve: que nos enfants vivent dans le meilleur monde possible. Entre montée mondiale du fascisme et réchauffement climatique, j’ai la détermination que ce rêve ne soit pas une chimère. Pour ce rêve, il nous faut encore réunir, rassembler, et je vous remercie tous des soutiens et des aides que vous m’avez apportés, de cet honneur que vous me faites aujourd’hui.</p> <img src="https://gael-varoquaux.info/personnal/attachments/gael_knight_monty_python.jpg"> <p>Technically, I might be a knight now</p> <p>✶ ✶ ✶</p> </div> <a href="https://gael-varoquaux.info/personnal/a-national-recognition-but-science-and-open-source-are-bitter-victories.html">Go Top</a> </div> Fri, 10 Oct 2025 00:00:00 https://gael-varoquaux.info/science/tabicl-pretraining-the-best-tabular-learner.html<div class="admonition note"> <p class="first admonition-title">Note</p> <p class="last">TabICL is a state-of-the-art tabular learner <a class="reference external" href="https://arxiv.org/abs/2502.05564">[Qu et al 2025]</a>. The key is its very rich prior, that is baked in a pre-trained architecture -a table foundation model-, and leveraged by in-context-learning. Thanks to clever choices, it is fast and scalable, efficient even without a GPU.</p> </div> <div class="contents topic" id="contents"> <p class="topic-title"><a class="reference internal" href="#top">Contents</a></p> <ul class="simple"> <li><a class="reference internal" href="#recent-progress-in-tabular-learning-in-context-learning" id="toc-entry-1">Recent progress …</a></li></ul></div><div class="content" morss_own_score="4.460000000000001" morss_score="14.13625898698366"> <h1>TabICL: Pretraining the best tabular learner</h1> <p> under <a href="https://gael-varoquaux.info/tag/machine-learning.html">machine learning</a> <a href="https://gael-varoquaux.info/tag/tabular-learning.html">tabular learning</a> <a href="https://gael-varoquaux.info/tag/foundation-models.html">foundation models</a> <span> &amp;nbsp Read time: 5 min. </span> </p> <p>Note</p> <p>TabICL is a state-of-the-art tabular learner <a href="https://arxiv.org/abs/2502.05564">[Qu et al 2025]</a>. The key is its very rich prior, that is baked in a pre-trained architecture -a table foundation model-, and leveraged by in-context-learning. Thanks to clever choices, it is fast and scalable, efficient even without a GPU.</p> <p><a href="https://gael-varoquaux.info/science/tabicl-pretraining-the-best-tabular-learner.html#top">Contents</a></p> <p>This note is about the research behind TabICL <a href="https://arxiv.org/abs/2502.05564">[Qu et al 2025]</a>, work by Jingang Qu, David Holzmüller, myself, and Marine Le Morvan, published at ICML 2025, and available as <a href="https://tabicl.readthedocs.io/en/latest/">open-source software</a>.</p> <br> <div class="section" id="recent-progress-in-tabular-learning-in-context-learning" morss_own_score="4.822784810126583" morss_score="14.253598763614955"> <h2><a href="https://gael-varoquaux.info/science/tabicl-pretraining-the-best-tabular-learner.html#toc-entry-1">Recent progress in tabular learning: In-Context Learning</a></h2> <p>Describing the statistical structure of tables in general is very subtle. They do have some unique statistical features. For instance, each column is typically meaningful by itself, more meaningful than linear combinations of columns (data <em>non rotationally invariant</em>, cf <a href="https://proceedings.neurips.cc/paper_files/paper/2022/hash/0378c7692da36807bdec87ab043cdadc-Abstract-Datasets_and_Benchmarks.html">[Grinsztajn et al, 2022]</a>). For long, tree-based models, in particular gradient-boosted trees, were the models that best captured this statistical structure.</p> <p>The question is indeed: <strong>how to build complex and rich inductive biases into statistical models</strong>?</p> <p>A pioneering contribution to this question was made with the TabPFN approach <a href="https://www.nature.com/articles/s41586-024-08328-6">[Hollmann et al, 2025]</a>.</p> <h3><a href="https://gael-varoquaux.info/science/tabicl-pretraining-the-best-tabular-learner.html#toc-entry-2">Tabular learning as a completion problem</a></h3> <img src="https://gael-varoquaux.info/science/attachments/tabicl/table_in_context_learning.png"> <p>Prediction by table completion using across-row transformers</p> <p>The key idea behind this line of work is that tabular learning can be seen as completing a table where one column has a missing entry. Transformer-based large-language models are very good at completing sequences, in particular in the few-shot regime. Hence the idea to use a transformer architecture for this table-completion task.</p> <p>More specifically, this is a <em>meta-learning</em> setting (learning to learn), using transformers.</p> <div class="section" id="sophisticated-prior-via-data-generation" morss_own_score="2.65" morss_score="15.65"> <h3><a href="https://gael-varoquaux.info/science/tabicl-pretraining-the-best-tabular-learner.html#toc-entry-3">Sophisticated prior via data generation</a></h3> <p>Teaching transformers to predict well requires showing them many many prediction problems.</p> <p>The benefit of this approach is that these prediction problems can be chosen to reflect well the downstream task. In particular, it becomes now easy to bake in any form of inductive bias by simulating data.</p> <p>TabPFN simulates data by cascading a series of simple transformations combining very few columns. The data-generative processes are actually more subtle, but the idea being that they are plausible for data tables.</p> <p>Experience (from us and others) shows that pretraining on a quality data-generation process is crucial to produce a good tabular learner, alike foundation models in other settings.</p> <br> </div> </div> <h2><a href="https://gael-varoquaux.info/science/tabicl-pretraining-the-best-tabular-learner.html#toc-entry-4">TabICL: improved architecture</a></h2> <h3><a href="https://gael-varoquaux.info/science/tabicl-pretraining-the-best-tabular-learner.html#toc-entry-5">The challenge: accounting for the structure of tables</a></h3> <img src="https://gael-varoquaux.info/science/attachments/tabicl/tabpfn_architecture.png"> <p>Tables are 2D objects, and the TabPFNv2 architecture alternates attentions across row and across columns</p> <p>In practice, a table is not a 1D structure, like sentences. It is closer to a 2D structure, with rows and columns. A good architecture will account for this structure, and the TabPFNv2 architecture uses transformers with alternating across-row and across-column attention.</p> <p morss_own_score="7.0" morss_score="13.0">One problem is the computational complexity: attention is quadratic in the number of entries, and the bi-directional transform of TabPFNv2 leads to a cost in <em>O(n p² + p n²)</em> for a table with <em>n</em> rows and <em>p</em> columns.</p> <h3><a href="https://gael-varoquaux.info/science/tabicl-pretraining-the-best-tabular-learner.html#toc-entry-6">TabICL’s solution</a></h3> <h4>Row-wise encoding</h4> <img src="https://gael-varoquaux.info/science/attachments/tabicl/tabicl_architecture.png"> <p>To break the quadratic cost, TabICL first encodes the rows to a smaller, fixed-sized, represention, before performing across-row in-context learning.</p> <p morss_own_score="7.0" morss_score="13.0">For more scalability and better inductive bias, our model, TabICL, first embeds the rows (using a first transformer) and then does in-context learning across rows (with a second transformer). The resulting computational complexity is <em>O(n p² + n²)</em>, which is more scalable, though still quadratic in <em>n</em> and <em>p</em>.</p> <p>Scalability is important because it enables us to pretrain TabICL on both small <em>and</em> large datasets, and as a consquence TabICL is a good predictor for large datasets.</p> <br> <div class="section" id="column-specific-embeddings" morss_own_score="6.0" morss_score="14.5"> <h4>Column-specific embeddings</h4> <img src="https://gael-varoquaux.info/science/attachments/tabicl/tabicl_embeddings.png"> <p>To apply different transformations on columns depending on their statistical properties, TabICL builds positional embeddings for columns that capture aspects of their distribution.</p> <p>Another important innovation of TabICL is that it inputs the entries in the transformer with column-specific embeddings. These column embeddings are computed to be a function of the distribution of the column. For this, we use a set transformer, which is a scalable transformer-like way of building a function on sets (but without the quadratic complexity).</p> <p>After pretraining, we find that the column embeddings have learned a mapping that implicitly captures statistical aspects of the data distribution in the column, as the kurtosis or the skewness.</p> </div> <div class="section" id="the-result-a-powerful-and-easy-to-use-tabular-learner" morss_own_score="4.920634920634921" morss_score="25.702453102453102"> <h2><a href="https://gael-varoquaux.info/science/tabicl-pretraining-the-best-tabular-learner.html#toc-entry-7">The result: a powerful and easy to use tabular learner</a></h2> <p>After a lot of pretraining on synthetic data, TabICL is a state-of-the-art tabular learner. Pretraining gave it the right inductive bias, as visible from the classifier-comparison plot below:</p> <img src="https://gael-varoquaux.info/science/attachments/tabicl/tabicl_comparison.png"> <p>A classic classification comparison plot that shows the decision boundaries on very simple toy data. It is useful to get a feeling of how classifiers behave.</p> <p>It is interesting to see that while TabICL forms very flexible decision boundaries, they do extend along the horizontal and vertical axes, as the decision tree and random forest. These axis-aligned features are a very important aspect of the inductive bias.</p> <p>At the end of the day, TabICL is an excellent tabular learner, as visible on benchmarks:</p> <img src="https://gael-varoquaux.info/science/attachments/tabicl/result_comparison.png"> <p>TabICL is a great predictor: Comparison of many predictors.</p> <img src="https://gael-varoquaux.info/science/attachments/tabicl/tabarena.png"> <p>Experimental results, from a benchmark paper independent of the TabICL paper: TabArena <a href="https://arxiv.org/abs/2506.16791">[Erickson et al, 2025]</a></p> <br> <p>The benefit of TabICL over TabPFNv2 becomes more marked for larger datasets:</p> <img src="https://gael-varoquaux.info/science/attachments/tabicl/tabicl_scale_bench.png"> <p>Rank (lower is best) as a function of dataset size.</p> <p>However, one limitation to keep in mind is that with in-context learners, as TabICL or TabPFN, inference (prediction on new datapoint) ican be costly.</p> <br> <p>All in all, TabICL is an excellent tabular predictor, and a push forward for tabular foundation models. From a fundamental standpoint, it shows that in-context learning is not only for few-shot learning, but that it can be very beneficial for sample sizes as large as <em>n=100 000</em>.</p> <br> <p>More about TabICL</p> <p>There is a lot more in TabICL: the details of pretraining are crucial, implementation uses memory offloading (which is facilitated by the architecture, which dissociates the train X from the test y for most of the operations). To learn more about TabICL:</p> <ul> <li>The paper: <a href="https://arxiv.org/abs/2502.05564">https://arxiv.org/abs/2502.05564</a></li> <li>The github code: <strong>TabICL is 100% open source</strong> <a href="https://github.com/soda-inria/tabicl">https://github.com/soda-inria/tabicl</a></li> <li>Install the Python package, TabICL is just one pip install away <a href="https://pypi.org/project/tabicl/">https://pypi.org/project/tabicl/</a></li> </ul> <br> <p>Other topics in table foundation models: leveraging strings</p> <p>TabICL is only one aspect of table foundation models. We are pursuing also another line of research that focuses on using strings (in entries and column names) to bring knowledge about the real world in table foundation models, see <a href="https://gael-varoquaux.info/science/carte-toward-table-foundation-models.html">CARTE</a> and more recently <a href="https://arxiv.org/abs/2505.14415">[Kim et al, 2025]</a>.</p> </div> <a href="https://gael-varoquaux.info/science/tabicl-pretraining-the-best-tabular-learner.html">Go Top</a> </div> Wed, 09 Jul 2025 00:00:00 https://gael-varoquaux.info/science/2024-highlights-of-computer-science-and-society.html<div class="admonition note"> <p class="first admonition-title">Note</p> <p class="last">For me, 2024 was full of back and forth between research, software, and connecting these to society. Here, I lay out some highlights on AI and society, as well as research and software, around tabular AI and language models.</p> </div> <p>As 2025 starts, I’m looking back on 2024. It …</p><div class="content" morss_own_score="4.936842105263158" morss_score="15.50140934080698"> <h1>2024 highlights: of computer science and society</h1> <p> under <a href="https://gael-varoquaux.info/tag/science.html">science</a> <a href="https://gael-varoquaux.info/tag/research.html">research</a> <a href="https://gael-varoquaux.info/tag/machine-learning.html">machine learning</a> <a href="https://gael-varoquaux.info/tag/statistics.html">statistics</a> <a href="https://gael-varoquaux.info/tag/yearly-report.html">yearly report</a> <span> &amp;nbsp Read time: 10 min. </span> </p> <p>Note</p> <p>For me, 2024 was full of back and forth between research, software, and connecting these to society. Here, I lay out some highlights on AI and society, as well as research and software, around tabular AI and language models.</p> <p>As 2025 starts, I’m looking back on 2024. It was an interesting professional year, as the research in the <a href="https://team.inria.fr/soda/">soda team</a> on machine learning for health and social science nourished reflection on society.</p> <p><a href="https://gael-varoquaux.info/science/2024-highlights-of-computer-science-and-society.html#top">Highlights</a></p> <h2><a href="https://gael-varoquaux.info/science/2024-highlights-of-computer-science-and-society.html#toc-entry-1">Thoughts from the national AI committee</a></h2> <p>Early 2024, I was serving in the French national AI committee. Our final write up can be found <a href="https://www.info.gouv.fr/actualite/25-recommandations-pour-lia-en-france">here</a>.</p> <p>It was a ton of work, a very interesting experience, and I learned a lot on many aspects of the interfaces between technology, policy, and society. A few things that stood out for me, some partly obvious but worth saying:</p> <ul> <li><p><strong>Digital services are a growing economy.</strong> The share of the economy that is digital keeps growing, whether we like it or not (IMHO, most of us spent too much time on our phones…). For France, or Europe, there is no question: we must produce our share of digital services and innovation, else our economic balance suffers.</p> </li> <li><p><strong>Privacy is erroding.</strong> Whether it is social network, information leaking into search engines or training of large language models, or people uploading private information to chatGPT, private information is more and more available. History has shown us the dangers behind loss of privacy, which the powerful (governing or economical elites) typically leverage to assert more power. Europe has had a long stance of trying to mitigate this loss of privacy via regulation (GDPR). But regulating services that we don’t control is hard, and it ends up being a geo-political and economical battle.</p> </li> <li><p><strong>Big AI is huge.</strong> The size of investments in AI is huge (dozens of billions yearly, comparable to a sizeable fraction of the state expenditures of a rich country like Switzerland). Data centers are having significant impacts on the electric grid of modern countries, running in competition with other usage. The cost of large models have ballooned (training a large language model is in the hundreds of millions of cost, which is comparable to a sizeable fraction of the budget of the national research institute that I work in (<a href="https://inria.fr/fr">inria</a>). Training costs are just the visible part of the iceberg, operational costs are huge and are everywhere.</p> <p>Not all in tech are worried about rising costs. Indeed, they go hand in hand with more money in tech, making us, tech bros, richer, as long as investments keep pouring in. But <a href="https://www.goldmansachs.com/images/migrated/insights/pages/gs-research/gen-ai--too-much-spend%2C-too-little-benefit-/TOM_AI%202.0_ForRedaction.pdf">bubble dynamics are at play</a>, and explain part of the conversation around AI.</p> </li> <li><p><strong>Concentration of power.</strong> Many factors in today’s AI lead to concentration into the hands of large actors. Training and operation costs, of course. But also limited access to the correspond skills, platform effect on the data and the users. The most striking bottleneck is the compute hardware. Only one company makes the chips that we all need. Few actors can afford buying them; and as a result most of the world lives from renting out to big landlords.</p> </li> <li><p><strong>AI neither good nor bad, but what we do of it.</strong> The above may paint a gloomy picture. But this is not how I see it. AI does have a lot of potential for good, as all general purpose technology. It all depends how society uses it. And here the future is open: we, as actors of democratic societies, as innovators, in tech but in every aspects of society, we can determine what the future of AI is. I look forward to technology that empowers each and everybody, to act for their own benefit. Key to this future is enabling and bringing in every stakeholder.</p> </li> </ul> <br> <h2><a href="https://gael-varoquaux.info/science/2024-highlights-of-computer-science-and-society.html#toc-entry-2">Adventures in software land</a></h2> <p>With the growing importance of data and artificial intelligence in shaping society, I believe more than ever in the importance of open source and commons for data science, making tools accessible to as many as possible.</p> <h3><a href="https://gael-varoquaux.info/science/2024-highlights-of-computer-science-and-society.html#toc-entry-3">probabl to supercharge scikit-learn</a></h3> <p>Early 2024, Inria span off the scikit-learn development to a new structure, <a href="https://probabl.ai">probabl</a>, to supercharge the development of the broader ecosystem. I detailed the motivation and the goals in <a href="https://gael-varoquaux.info/programming/promoting-open-source-from-inria-to-probabl.html">a previous article</a>. In a nutshell:</p> <ul> <li>Scikit-learn is <a href="https://gael-varoquaux.info/science/programming/people-underestimate-how-impactful-scikit-learn-continues-to-be.html">a key component of the machine-learning ecosystem</a>, but its development require funding.</li> <li>Probabl is there to foster a broader open data-science ecosystem, as scikit-learn can be sustainable only when used in such ecosystem. Probabl focus on delivering value to enterprises, and thus makes sure that there is a seamless solution to their needs.</li> <li>I have 10% of my time allocated from Inria to Probabl.</li> </ul> <p>Some of our successes are already publicly visible:</p> <ul> <li>The open-source team at probabl is maintaining and improving <a href="https://probabl.ai/open-source">a range of software libraries</a>: scikit-learn, joblib, imbalanced-learn, fairlearn, skops, skrub… Our priorities are openly discussed <a href="https://papers.probabl.ai/open-source-priorities-chapter-2">here</a>.</li> <li>We have launched <a href="https://papers.probabl.ai/official-scikit-learn-certification-launch">an official certification program for scikit-learn</a>. I’m very excited about these certifications (there are three levels), to grow recognition in the scikit-learn skills, and thus make sure that it is a dependable stack for the industry.</li> </ul> <div class="section" id="skrub-machine-learning-on-tables-made-easy" morss_own_score="2.6280991735537187" morss_score="27.234159779614323"> <h3><a href="https://gael-varoquaux.info/science/2024-highlights-of-computer-science-and-society.html#toc-entry-4">Skrub: machine learning on tables made easy</a></h3> <p><a href="https://skrub-data.org/">skrub</a> is a software project that I am very excited about. Many crucial applications of machine learning are on tables. Skrub facilitates the corresponding patterns. We are designing it with the insights of years of research and practice on the topic. It does not always look impressive, but it’s little things that add up for productivity.</p> <p>A typical dataset is the employees one:</p> <pre> &gt;&gt;&gt; from skrub.datasets import fetch_employee_salaries &gt;&gt;&gt; dataset = fetch_employee_salaries() &gt;&gt;&gt; employees_df, y = dataset.X, dataset.y </pre> <p>Skrub’s <a href="https://skrub-data.org/stable/reference/generated/skrub.TableReport.html">TableReport</a> makes it really easy to interactively visualize and explore such table:</p> <img src="https://gael-varoquaux.info/science/attachments/2024_highlights/table_report_vscode.png"> <p>The dataframe <cite>employees_df</cite> has plenty of non numerical columns, as visible above. Skrub’s <a href="https://skrub-data.org/stable/reference/generated/skrub.TableVectorizer.html">TableVectorizer</a> turns it into a numerical array suitable for machine learning, taking care of dates, categories, strings…</p> <pre> &gt;&gt;&gt; from skrub import TableVectorizer &gt;&gt;&gt; X = TableVectorizer().fit_transform(employees_df) </pre> <p>If you want to use deep-learning language models for the string categories, skrub’s <a href="https://skrub-data.org/stable/reference/generated/skrub.TextEncoder.html">TextEncoder</a> can download pre-trained models from hugginface:</p> <pre> &gt;&gt;&gt; from skrub import TextEncoder &gt;&gt;&gt; text_encoder = TextEncoder( "sentence-transformers/paraphrase-albert-small-v2", device="cpu", ) &gt;&gt;&gt; tab_vec = TableVectorizer(high_cardinality=text_encoder) &gt;&gt;&gt; X = tab_vec.fit_transform(employees_df) </pre> <p>With this, the latest artificial intelligent developments are easily brought to drive decisions on the data that matters.</p> <br> </div> <div class="section" id="research-better-ai-tools-more-understanding" morss_own_score="4.44" morss_score="12.40468577200906"> <h2><a href="https://gael-varoquaux.info/science/2024-highlights-of-computer-science-and-society.html#toc-entry-5">Research: better AI tools, more understanding</a></h2> <p>Software or thoughts on AI and society, are best built on solid understanding of AI, which calls for research.</p> <div class="section" id="table-foundation-models" morss_own_score="2.7507987220447285" morss_score="17.494762787176676"> <h3><a href="https://gael-varoquaux.info/science/2024-highlights-of-computer-science-and-society.html#toc-entry-6">Table foundation models</a></h3> <p><em>Modeling data semantics enable pretaining for tables</em></p> <p>I have been working on machine-learning for tables for more than a decade. These data are crucial for many applications, but they have so far not witnessed the breakthroughs of deep learning seen <em>eg</em> in vision or text. Much of these success of <strong>deep learning as been driven by the ability to reused pretrained models</strong>, fitted on very large datasets. Foundation models pushed this idea very far with models that provide background information useful for a wide variety of downstream tasks. But pretraining is challenging for tables.</p> <p>A crucial part of foundation models for text and images is the attention mechanism, stacked in a transformer architecture, that bring associative memory to the inputs by contextualizing them. We had a breakthough with the <a href="https://openreview.net/forum?id=9kArQnKLDp">CARTE model</a>: we managed to adapt these ideas to tables. The strings –tables entries and column names– give the information that enables transfer from one table to another: data semantics. Here, key is to have an architecture that 1) models both strings and numerical values 2) applies to any set of tables while using the column names to route the information. For this purpose, CARTE uses a new dedicated attention mechanism that accounts for column names. It is pre-trained on a very large knowledge base. As a result, it outperform the best models (including tree-based models) in small sample settings (up to n=2000).</p> <p>The pretrained CARTE model is available for download as <a href="https://pypi.org/project/carte-ai">a Python package</a>.</p> <p>This result is very significant as it opens the door to <strong>foundation models for tables</strong>: models that embark much background knowledge and can be specialized to many tabular-learning tasks.</p> <a href="https://openreview.net/forum?id=9kArQnKLDp"> <img src="https://gael-varoquaux.info/science/attachments/2024_highlights/carte_comparisons.png"> </a> <p>Extensive empirical results show that CARTE brings benefits to very broad set of baselines. The relative performance of baselines also contains interesting results.</p> <p>See also</p> <p>I wrote a longer <a href="https://gael-varoquaux.info/science/carte-toward-table-foundation-models.html">high-level post on CARTE</a>.</p> <br> </div> <div class="section" id="disparities-of-confidence-of-large-language-models" morss_own_score="5.6011080332409975" morss_score="18.849825981958947"> <h3><a href="https://gael-varoquaux.info/science/2024-highlights-of-computer-science-and-society.html#toc-entry-7">Disparities of confidence of large language models</a></h3> <a href="https://hal.science/hal-04750567"> <img src="https://gael-varoquaux.info/science/attachments/2024_highlights/hallucination_probability.png"> </a> <p>A good confidence assessment on replies of an LLM would separate out correct from incorrect statements: Einstein was not born on Jan 14th 1879 (close call, it was March 14th); his PhD was in Zurich.</p> <p>Large language models (LLMs), such as chatGPT, may produce answers that are plausible but not factually correct, the so-called “hallucinations”. A variety of approach try to assess how likely a statement is to be true, for instance by sampling multiple responses from the language model. Ideally, we would like to use these confidence assessments to flag the wrong statements in an LLM’s answer. For this, a challenge is to threshold them, or assign a probability of correctness.</p> <br> <a href="https://hal.science/hal-04750567"> <img src="https://gael-varoquaux.info/science/attachments/2024_highlights/llm_confidence_nationality.png"> </a> <p>Observed error rate and a function predicted probability of correctness For the birth date, when a large language model (here Mistral 7B) gives information on a given notable individual. The different curves give the corresponding calibration for different nationalities of the individuals, revealing that <strong>the probability is much more trustworthy for a citizen of the United States than for other countries</strong>, and particularly poor for people that originate from South-East Asia.</p> <p>In <a href="https://hal.science/hal-04750567/">Chen et al</a>, we investigate the confidence of LLMs in their answers. We show that the probabilities computed are not only overconfident, but also that there is heterogeneity (grouping loss): on some groups of queries the overconfidence is more pronounced than on others. For instance, for an answer on a notable individual, the LLMs’ confidence is reasonably calibrated if the individual is from the United States, but severely overconfident for individuals from South East Asia (fig:llmconfidencenationality). Characterizing the corresponding groups opens the door to correcting the corresponding bias, a “reconfidencing” procedure.</p> <p>This study is an application of our earlier, more theoretical, <a href="https://openreview.net/forum?id=6w1k-IixnL8">work</a> that contributed the first estimator grouping loss, a mathematically-solid concept behind hidden heterogeneity in classifier calibration. I am very happy to see that these fairly abstract ideas are useful to probe very concrete problems such as the disparity in LLM confidence across nationalities.</p> <br> </div> <div class="section" id="a-straggler-consistency-of-supervised-learning-with-missing-values" morss_own_score="2.577464788732394" morss_score="26.346686810272296"> <h3><a href="https://gael-varoquaux.info/science/2024-highlights-of-computer-science-and-society.html#toc-entry-8">A straggler: Consistency of supervised learning with missing values</a></h3> <p><em>A</em> <a href="https://link.springer.com/article/10.1007/s00362-024-01550-4">paper</a> <em>on the fundamentals of machine-learning with missing values</em></p> <p>In 2018, <a href="https://juliejosse.com">Julie Josse</a>, <a href="https://erwanscornet.github.io">Erwan Scornet</a>, and myself started working on the theory of how supervised learning works with missing values (learning theory). Working an intern, Nicolas Prost, we quickly realized that there was a gap between the statistical thinking around missing values, which was focused on enabling inference in parametric models as if their were no missing values, and the needs for prediction with missing values.</p> <p>We wrote <a href="https://link.springer.com/article/10.1007/s00362-024-01550-4">a paper</a> to lay out the theory cleanly, summarizing both elements of learning theory and the fundamentals of statistics with missing values. Beyond this didactic aspects, the paper gives a series of formal results, such as the need for multiple imputations to be able to use the <em>complete case</em> predictor (the optimal predictor without missing values), the optimal way to model missing values in trees (which was already used in XGBoost :) ), and the fact that asymptotically, constant imputation of missing values could work well for predictor.</p> <p><em>Frustrations of the academic game</em></p> <p><a href="https://hal.science/hal-02024202">The preprint</a> got a lot of success (more than a hundred citations), probably because it laid out fundamentals. But it took 5 years to publish it. The machine learning community did not like the absence of new methods (we only gave theoretical results on existing practice, such as imputation). The statistics literature really did not like our messages that imputation was not always important. In one journal, a reviewer rejected the paper on the basis that it was giving bad messages to the community, but not arguing that anything was wrong in our proofs or our experiments. Of course, there is a lot to say about the difficulties of doing data analysis with missing values, but the conversation did not go in these details. This is a good illustration that <strong>progress in science is social</strong>, and is as much about shifting norms than accumulating knowledge (actually, knowledge is social too, as put forward by <a href="https://en.wikipedia.org/wiki/Social_epistemology">social epistemology</a>).</p> <p>As time went by, my colleague <a href="https://marinelm.github.io">Marine Le Morvan</a> has published <a href="https://proceedings.mlr.press/v108/morvan20a.html">more</a> <a href="https://proceedings.neurips.cc/paper/2021/hash/5fe8fdc79ce292c39c5f209d734b7206-Abstract.html">and</a> <a href="https://academic.oup.com/gigascience/article/doi/10.1093/gigascience/giac013/6568998">more</a> <a href="https://arxiv.org/abs/2407.19804">results</a> that push deeper understanding of prediction with missing values. But I still see value in our original paper, as it lays the foundations.</p> <p>The paper is now out, thanks to my coauthors who kept replying to reviewers, improving the manuscripts, and resubmitting. Read <a href="https://link.springer.com/article/10.1007/s00362-024-01550-4">it</a>, I think that it is a good read.</p> <br> <hr> <p>Well, this article ended up longer than I had expected. Thanks for reading. Taking a step back to figure out what is important is always a good exercise for me.</p> </div> </div> <a href="https://gael-varoquaux.info/science/2024-highlights-of-computer-science-and-society.html">Go Top</a> </div> Wed, 01 Jan 2025 00:00:00