Research¶
ModelBump is also a research project aiming at a peer-reviewed paper.
Research questions (draft)¶
| RQ | Question |
|---|---|
| RQ1 | How accurately and how early can static analysis detect LLM-model dependencies that are about to be retired? |
| RQ2 | Can sequential statistical testing detect real behavioural regressions between model versions at bounded cost, with a controlled false-alarm rate? |
| RQ3 | How effective are automated regression-suite synthesis and prompt repair at restoring behaviour after a migration? |
| RQ4 | Is the end-to-end output useful to developers? |
Protocols¶
- Mining candidate migration commits
- Filtering: which candidates are migrations
- Validation: labelling a sample by hand
- Timing: when did projects migrate?
- Lag: how long before or after the shutdown?
- Targets: what do projects migrate to?
- Changed files: do migrations come with tests?
- Robustness: do the timing results hold?
- Paper numbers, tables and figures
- Releasing the dataset
- Scanner evaluation: precision and recall of
modelbump scan - Stats engine simulation study: error rates, power and cost
- Stats engine on real models: Qwen 2.5 against Qwen 3
- Dataset card: what the migration dataset holds, how it was built, its quality and limits
Key prior work¶
- H. L. Kim. When the Model Retires: An Empirical Study of LLM Migration in Open-Source Applications. arXiv:2609.31288, 2026. arXiv
- W. Ma, C. Yang, C. Kästner. (Why) Is My Prompt Getting Worse? Rethinking Regression Testing for Evolving LLM APIs. CAIN 2024. arXiv
- L. Chen, M. Zaharia, J. Zou. How Is ChatGPT's Behavior Changing over Time? Harvard Data Science Review, 2024. arXiv
The full related-work survey is written in Research Break 1.