Paper Collections

Reading lists organized by the research areas referenced in the Data Counterfactuals memo.

Active Learning 4 papers

Representative approaches for choosing which data to label or include during training.

  • Active Learning Literature Survey
    Burr Settles
    University of Wisconsin-Madison, Computer Sciences Technical Report 1648 (2009)
  • Active Learning for Convolutional Neural Networks: A Core-Set Approach
    Ozan Sener, Silvio Savarese
    International Conference on Learning Representations (ICLR) (2018)
  • Coresets for Data-efficient Training of Machine Learning Models
    Baharan Mirzasoleiman, Jeff Bilmes, Jure Leskovec
    International Conference on Machine Learning (ICML) (2020)
  • DeepCore: A Comprehensive Library for Coreset Selection in Deep Learning
    Chengcheng Guo, Bo Zhao, Yanbing Bai
    arXiv preprint (2022)

Collective Action & Data Leverage 6 papers

Strategic data withholding, leverage, and collective action in data-centric systems.

  • "Data Strikes": Evaluating the Effectiveness of a New Form of Collective Action Against Technology Companies
    Nicholas Vincent, Brent Hecht, Shilad Sen
    The World Wide Web Conference (WWW) (2019)
  • Data Leverage: A Framework for Empowering the Public in its Relationship with Technology Companies
    Vincent, Nicholas and Li, Hanlin and Tilly, Nicole and Chancellor, Stevie and Hecht, Brent
    Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency (2021)
  • Algorithmic Collective Action in Machine Learning
    Moritz Hardt, Eric Mazumdar, Celestine Mendler-Dünner, Tijana Zrnic
    International Conference on Machine Learning (ICML) (2023)
  • Should We Treat Data as Labor? Moving Beyond 'Free'
    Imanol Arrieta-Ibarra, Leonard Goff, Diego Jimenez-Hernandez, Jaron Lanier, E. Glen Weyl
    AEA Papers and Proceedings (2018)
  • Mapping the Potential and Pitfalls of "Data Dividends" as a Means of Sharing the Profits of Artificial Intelligence
    Nicholas Vincent, Yichun Li, Renee Zha, Brent Hecht
    arXiv preprint arXiv:1912.00757 (2019)
  • The Dimensions of Data Labor: A Road Map for Researchers, Activists, and Policymakers to Empower Data Producers
    Proceedings of the 2023 ACM Conference on Fairness, Accountability, and Transparency (2023)

Data Augmentation & Curriculum Learning 5 papers

Foundational work on augmentation strategies and curriculum learning.

  • Curriculum Learning
    Yoshua Bengio, Jerome Louradour, Ronan Collobert, Jason Weston
    ICML 2009 (2009)
  • mixup: Beyond Empirical Risk Minimization
    Hongyi Zhang, Moustapha Cisse, Yann N. Dauphin, David Lopez-Paz
    ICLR 2018 (2018)
  • Improved Regularization of Convolutional Neural Networks with Cutout
    Terrance DeVries, Graham W. Taylor
    arXiv preprint (2017)
  • CutMix: Regularization Strategy to Train Strong Classifiers with Localizable Features
    Sangdoo Yun, Dongyoon Han, Seong Joon Oh, Sanghyuk Chun, Junsuk Choe, Youngjoon Yoo
    ICCV 2019 (2019)
  • A Survey on Image Data Augmentation for Deep Learning
    Connor Shorten, Taghi M. Khoshgoftaar
    Journal of Big Data (2019)

Data Counterfactuals Bibliography 14 papers

Collection of papers referenced by the Data Counterfactuals project.

  • Poisoning Attacks against Support Vector Machines
    Battista Biggio, Blaine Nelson, Pavel Laskov
    Proceedings of the 29th International Conference on Machine Learning (ICML) (2012)
  • Data Shapley: Equitable Valuation of Data for Machine Learning
    Amirata Ghorbani, James Zou
    Proceedings of the 36th International Conference on Machine Learning (ICML) (2019)
  • Dataset Security for Machine Learning: Data Poisoning, Backdoor Attacks, and Defenses
    Micah Goldblum, Dimitris Tsipras, Chulin Xie, Xinyun Chen, Avi Schwarzschild, Dawn Song, Aleksander Madry, Bo Li, Tom Goldstein
    IEEE Transactions on Pattern Analysis and Machine Intelligence (2022)
  • DeepCore: A Comprehensive Library for Coreset Selection in Deep Learning
    Chengcheng Guo, Bo Zhao, Yanbing Bai
    arXiv preprint (2022)
  • Algorithmic Collective Action in Machine Learning
    Moritz Hardt, Eric Mazumdar, Celestine Mendler-Dünner, Tijana Zrnic
    International Conference on Machine Learning (ICML) (2023)
  • Datamodels: Predicting Predictions from Training Data
    Andrew Ilyas, Sung Min Park, Logan Engstrom, Guillaume Leclerc, Aleksander Madry
    International Conference on Machine Learning (ICML) (2022)
  • Scaling Laws for Neural Language Models
    Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B. Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, Dario Amodei
    arXiv preprint (2020)
  • Understanding Black-box Predictions via Influence Functions
    Pang Wei Koh, Percy Liang
    Proceedings of the 34th International Conference on Machine Learning (ICML) (2017)
  • Beta Shapley: a Unified and Noise-reduced Data Valuation Framework for Machine Learning
    Yongchan Kwon, James Zou
    International Conference on Artificial Intelligence and Statistics (AISTATS) (2022)
  • Coresets for Data-efficient Training of Machine Learning Models
    Baharan Mirzasoleiman, Jeff Bilmes, Jure Leskovec
    International Conference on Machine Learning (ICML) (2020)
  • TRAK: Attributing Model Behavior at Scale
    Sung Min Park, Kristian Georgiev, Andrew Ilyas, Guillaume Leclerc, Aleksander Madry
    International Conference on Machine Learning (ICML) (2023)
  • Estimating Training Data Influence by Tracing Gradient Descent
    Garima Pruthi, Frederick Liu, Mukund Sundararajan, Satyen Kale
    Advances in Neural Information Processing Systems (NeurIPS) (2020)
  • Active Learning for Convolutional Neural Networks: A Core-Set Approach
    Ozan Sener, Silvio Savarese
    International Conference on Learning Representations (ICLR) (2018)
  • "Data Strikes": Evaluating the Effectiveness of a New Form of Collective Action Against Technology Companies
    Nicholas Vincent, Brent Hecht, Shilad Sen
    The World Wide Web Conference (WWW) (2019)

Data Poisoning & Adversarial Training 4 papers

Adversarial data manipulation, poisoning attacks, and training-time threats.

  • Poisoning Attacks against Support Vector Machines
    Battista Biggio, Blaine Nelson, Pavel Laskov
    Proceedings of the 29th International Conference on Machine Learning (ICML) (2012)
  • BadNets: Identifying Vulnerabilities in the Machine Learning Model Supply Chain
    Tianyu Gu, Brendan Dolan-Gavitt, Siddharth Garg
    IEEE Access (2019)
  • Poisoning Web-Scale Training Datasets is Practical
    Nicholas Carlini, Matthew Jagielski, Christopher A. Choquette-Choo, Daniel Paleka, Will Pearce, Hyrum Anderson, Andreas Terzis, Kurt Thomas, Florian Tramèr
    2024
  • Sleeper Agents: Training Deceptive LLMs that Persist Through Safety Training
    Evan Hubinger, Carson Denison, Jesse Mu, Mike Lambert, Meg Tong, Monte MacDiarmid, Tamera Lanham, Daniel M. Ziegler, Tim Maxwell, Newton Cheng, Adam Jermyn, Amanda Askell, Ansh Radhakrishnan, Cem Anil, David Duvenaud, Deep Ganguli, Fazl Barez, Jack Clark, Kamal Ndousse, Kshitij Sachan, Michael Sellitto, Mrinank Sharma, Nova DasSarma, Roger Grosse, Shauna Kravec, Yuntao Bai, Zachary Witten, Marina Favaro, Jan Brauner, Holden Karnofsky, Paul Christiano, Samuel R. Bowman, Logan Graham, Jared Kaplan, Sören Mindermann, Ryan Greenblatt, Buck Shlegeris, Nicholas Schiefer, Ethan Perez
    2024

Data Scaling Laws 7 papers

Work on scaling laws, data size regimes, and predictable performance curves.

  • Scaling Laws for Neural Language Models
    Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B. Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, Dario Amodei
    arXiv preprint (2020)
  • Training Compute-Optimal Large Language Models
    Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, Tom Hennigan, Eric Noland, Katie Millican, George van den Driessche, Bogdan Damoc, Aurelia Guy, Simon Osindero, Karen Simonyan, Erich Elsen, Jack W. Rae, Oriol Vinyals, Laurent Sifre
    NeurIPS 2022 (2022)
  • Deep learning scaling is predictable, empirically
    Hestness, Joel, Narang, Sharan, Ardalani, Newsha, Diamos, Gregory, Jun, Heewoo, Kianinejad, Hassan, Patwary, Md, Ali, Mostofa, Yang, Yang, Zhou, Yanqi
    arXiv preprint arXiv:1712.00409 (2017)
  • Beyond neural scaling laws: beating power law scaling via data pruning
    Sorscher, Ben, Geirhos, Robert, Shekhar, Shashank, Ganguli, Surya, Morcos, Ari
    Advances in Neural Information Processing Systems (2022)
  • Reconciling modern machine-learning practice and the classical bias–variance trade-off
    Belkin, Mikhail, Hsu, Daniel, Ma, Siyuan, Mandal, Soumik
    Proceedings of the National Academy of Sciences (2019)
  • Deep Double Descent: Where Bigger Models and More Data Hurt
    Preetum Nakkiran, Gal Kaplun, Yamini Bansal, Tristan Yang, Boaz Barak, Ilya Sutskever
    ICLR 2020 (2020)
  • Language Models are Few-Shot Learners
    Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, Dario Amodei
    arXiv preprint arXiv:2005.14165 (2020)

Data Selection & Coresets 4 papers

Coreset construction, selection strategies, and training-efficient data subset methods.

  • Active Learning for Convolutional Neural Networks: A Core-Set Approach
    Ozan Sener, Silvio Savarese
    International Conference on Learning Representations (ICLR) (2018)
  • Coresets for Data-efficient Training of Machine Learning Models
    Baharan Mirzasoleiman, Jeff Bilmes, Jure Leskovec
    International Conference on Machine Learning (ICML) (2020)
  • DeepCore: A Comprehensive Library for Coreset Selection in Deep Learning
    Chengcheng Guo, Bo Zhao, Yanbing Bai
    arXiv preprint (2022)
  • Dataset Security for Machine Learning: Data Poisoning, Backdoor Attacks, and Defenses
    Micah Goldblum, Dimitris Tsipras, Chulin Xie, Xinyun Chen, Avi Schwarzschild, Dawn Song, Aleksander Madry, Bo Li, Tom Goldstein
    IEEE Transactions on Pattern Analysis and Machine Intelligence (2022)

Data Valuation & Shapley 6 papers

Foundational work on valuing data points, Shapley-style attribution, and related estimators.

  • Towards Efficient Data Valuation Based on the Shapley Value
    Ruoxi Jia, Dah-Yuan Dao, Boxin Wang, Frances Ann Hubis, Nick Hynes, Neslihan M. Gurel, Carl J. Spanos
    International Conference on Artificial Intelligence and Statistics (2019)
  • Data Shapley: Equitable Valuation of Data for Machine Learning
    International Conference on Machine Learning (2019)
  • Beta Shapley: A Unified and Noise-Reduced Data Valuation Framework for Machine Learning
    arXiv preprint arXiv:2110.14049 (2021)
  • Beta Shapley: a Unified and Noise-reduced Data Valuation Framework for Machine Learning
    Yongchan Kwon, James Zou
    International Conference on Artificial Intelligence and Statistics (AISTATS) (2022)
  • Data Shapley: Equitable Valuation of Data for Machine Learning
    Amirata Ghorbani, James Zou
    Proceedings of the 36th International Conference on Machine Learning (ICML) (2019)
  • Datamodels: Predicting Predictions from Training Data
    Andrew Ilyas, Sung Min Park, Logan Engstrom, Guillaume Leclerc, Aleksander Madry
    International Conference on Machine Learning (ICML) (2022)

Experimental Design & Causal Inference 3 papers

Foundational work on experimental design, causal inference, and counterfactual estimation.

  • Causality: Models, Reasoning, and Inference
    Judea Pearl
    Cambridge University Press (2009)
  • Causal Inference Using Potential Outcomes: Design, Modeling, Decisions
    Donald B. Rubin
    Journal of the American Statistical Association (2005)
  • Causal Inference in Statistics, Social, and Biomedical Sciences
    Guido W. Imbens, Donald B. Rubin
    Cambridge University Press (2015)

Fairness via Data Interventions 9 papers

Fairness research that highlights data-based interventions and measurement.

  • A Reductions Approach to Fair Classification
    Alekh Agarwal, Alina Beygelzimer, Miroslav Dudik, John Langford, Hanna Wallach
    International Conference on Machine Learning (2018)
  • Big Data's Disparate Impact
    Barocas, Solon, Selbst, Andrew D.
    California Law Review (2016)
  • Fairness and Abstraction in Sociotechnical Systems
    Selbst, Andrew D., Boyd, Danah, Friedler, Sorelle A., Venkatasubramanian, Suresh, Vertesi, Janet
    Proceedings of the ACM Conference on Fairness, Accountability, and Transparency (FAccT) (2019)
  • Dissecting Racial Bias in an Algorithm Used to Manage the Health of Populations
    Obermeyer, Ziad, Powers, Brian, Vogeli, Christine, Mullainathan, Sendhil
    Science (2019)
  • Gender Shades: Intersectional Accuracy Disparities in Commercial Gender Classification
    Buolamwini, Joy, Gebru, Timnit
    Proceedings of the Conference on Fairness, Accountability and Transparency (FAT*) (2018)
  • Datasheets for Datasets
    Gebru, Timnit, Morgenstern, Jamie, Vecchione, Briana, Vaughan, Jennifer Wortman, Wallach, Hanna, Daumé III, Hal, Crawford, Kate
    arXiv:1803.09010 (2018)
  • Data Statements for Natural Language Processing: Toward Mitigating System Bias and Enabling Better Science
    Bender, Emily M., Friedman, Batya
    Transactions of the ACL (2018)
  • Excavating AI: The Politics of Images in Machine Learning Training Sets
    Crawford, Kate, Paglen, Trevor
    2019
  • The Dataset Nutrition Label: A Framework To Drive Higher Data Quality Standards
    Sarah Holland, Ahmed Hosny, Sarah Newman, Joshua Joseph, Kasia Chmielinski
    2018

Influence Functions & Data Attribution 9 papers

Tracing model behavior back to training data through influence, attribution, and tracing methods.

  • Understanding Black-box Predictions via Influence Functions
    Pang Wei Koh, Percy Liang
    Proceedings of the 34th International Conference on Machine Learning (ICML) (2017)
  • On the Accuracy of Influence Functions for Measuring Group Effects
    Pang Wei Koh, Kai-Siang Ang, Hubert H. K. Teo, Percy Liang
    Advances in Neural Information Processing Systems (2019)
  • Estimating Training Data Influence by Tracing Gradient Descent
    Garima Pruthi, Frederick Liu, Mukund Sundararajan, Satyen Kale
    Advances in Neural Information Processing Systems (NeurIPS) (2020)
  • TRAK: Attributing Model Behavior at Scale
    Sung Min Park, Kristian Georgiev, Andrew Ilyas, Guillaume Leclerc, Aleksander Madry
    International Conference on Machine Learning (ICML) (2023)
  • If open source is to win, it must go public
    Tan, Joshua, Vincent, Nicholas, Elkins, Katherine, Sahlgren, Magnus
    arXiv preprint arXiv:2507.09296 (2025)
  • What is Your Data Worth to GPT? LLM-Scale Data Valuation with Influence Functions
    arXiv preprint arXiv:2405.13954 (2024)
  • Training Data Influence Analysis and Estimation: A Survey
    Zayd Hammoudeh, Daniel Lowd
    arXiv preprint arXiv:2212.04612 (2022)
  • OLMoTrace: Tracing Language Model Outputs Back to Trillions of Training Tokens
    Liu, Jiacheng, Blanton, Taylor, Elazar, Yanai, Min, Sewon, Chen, YenSung, Chheda-Kothary, Arnavi, Tran, Huy, Bischoff, Byron, Marsh, Eric, Schmitz, Michael, others
    arXiv preprint arXiv:2504.07096 (2025)
  • LLM Dataset Inference: Did you train on my dataset?
    Pratyush Maini, Hengrui Jia, Nicolas Papernot, Adam Dziedzic
    arXiv preprint arXiv:2406.06443 (2024)

Machine Unlearning 3 papers

Foundational work on data deletion, unlearning, and model update guarantees.

  • Towards Making Systems Forget with Machine Unlearning
    Yinzhi Cao, Junfeng Yang
    IEEE Symposium on Security and Privacy (S&P) (2015)
  • Machine Unlearning
    Lucas Bourtoule, Varun Chandrasekaran, Christopher A. Choquette-Choo, Hengrui Jia, Adelin Travers, Baiwu Zhang, David Lie, Nicolas Papernot
    IEEE Symposium on Security and Privacy (S&P) (2021)
  • Machine Unlearning: A Survey
    Heng Xu, Tianqing Zhu, Lefeng Zhang, Wanlei Zhou, Philip S. Yu
    ACM Computing Surveys (2024)

Privacy, Memorization & Unlearning 9 papers

Differential privacy, memorization risks, and related data removal or anonymization concerns.

  • The Algorithmic Foundations of Differential Privacy
    Cynthia Dwork, Aaron Roth
    Foundations and Trends in Theoretical Computer Science (2014)
  • Privacy, anonymity, and perceived risk in open collaboration: A study of service providers
    McDonald, Nora, Hill, Benjamin Mako, Greenstadt, Rachel, Forte, Andrea
    Proceedings of the 2019 CHI conference on human factors in computing systems (2019)
  • Simple Demographics Often Identify People Uniquely
    Sweeney, Latanya
    Carnegie Mellon University, Data Privacy Working Paper (2000)
  • Robust De-anonymization of Large Sparse Datasets
    Narayanan, Arvind, Shmatikov, Vitaly
    Proceedings of the IEEE Symposium on Security and Privacy (2008)
  • Extracting Training Data from Large Language Models
    Carlini, Nicholas, Tramer, Florian, Wallace, Eric, Jagielski, Matthew, Herbert-Voss, Ariel, Lee, Katherine, Roberts, Adam, Brown, Tom B., Song, Dawn, Erlingsson, {\'U}lfar, Oprea, Alina, Papernot, Nicolas
    Proceedings of USENIX Security Symposium (2021)
  • Quantifying Memorization Across Neural Language Models
    Nicholas Carlini, Daphne Ippolito, Matthew Jagielski, Katherine Lee, Florian Tramer, Chiyuan Zhang
    International Conference on Learning Representations (2023)
  • Exploring the limits of strong membership inference attacks on large language models
    Jamie Hayes, Ilia Shumailov, Christopher A. Choquette-Choo, Matthew Jagielski, George Kaissis, Milad Nasr, Sahra Ghalebikesabi, Meenatchi Sundaram Mutu Selva Annamalai, Niloofar Mireshghallah, Igor Shilov, Matthieu Meeus, Yves-Alexandre de Montjoye, Katherine Lee, Franziska Boenisch, Adam Dziedzic, A. Feder Cooper
    arXiv preprint arXiv:2505.18773 (2025)
  • Are anonymity-seekers just like everybody else? An analysis of contributions to Wikipedia from Tor
    Tran, Chau, Champion, Kaylea, Forte, Andrea, Hill, Benjamin Mako, Greenstadt, Rachel
    2020 IEEE Symposium on Security and Privacy (SP) (2020)
  • Consent in Crisis: The Rapid Decline of the AI Data Commons
    arXiv preprint arXiv:2407.14933 (2024)