"Microservices scale better than monoliths.""Code review catches more defects than testing alone.""Quantising the model cuts its energy use in half."
Each of these is a claim about the world. Some are true. Some are true only in particular contexts. Some are folklore repeated until it started to sound like fact. Empirical Software Engineering (EMSE) is the part of our field that asks one awkward question about every such sentence: HOW DO YOU KNOW?
WHAT it is?
Empirical Software Engineering studies how software, developers, and organizations work in the real world. Instead of relying on intuition or authority, it bases its conclusions on systematically gathered evidence. This leads to two surprising takeaways for newcomers:
- Code doesn't write itself. Software engineering involves human decisions, team dynamics, and tight deadlines. To get the full picture, empirical research studies both the technical systems (experiments, measurement) and the people building them (interviews, surveys).
- Building something is not evaluating it. A new tool is one contribution. Showing that it works, for whom, and under which conditions is a separate contribution with its own design, its own methods, and its own ways of going wrong.
WHY did the community form?
It all started in the 1970s when pioneering teams like NASA's Software Engineering Lab realized you couldn't just guess what made software good. Instead, you had to measure real teams in the wild using structured frameworks like the Goal-Question-Metric approach. By the 1990s, the field faced a serious wake-up call: studies revealed that nearly half of all software engineering papers were making bold technical claims without presenting a single piece of experimental proof. To fix this "credibility gap," researchers built dedicated journals, research networks, and standardized guidelines to bring rigor to the discipline. Fast-forward to today, and the community has borrowed lessons from evidence-based medicine, shifting away from one-off studies toward systematic reviews and shared quality standards so we can base software practices on solid, repeatable data instead of pure hype.
HOW is it done?
There is no single "empirical method." You don't pick a method based on personal taste. You pick it based on the specific question you're trying to answer.
| If your question is... | The right method | What it looks like in practice |
|---|---|---|
| Does Technique A work better than Technique B? | Controlled Experiment | Testing two approaches in a controlled environment to prove cause and effect. |
| How does this actually play out inside a real engineering team? | Case Study | In-depth observation of a real company, project, or workflow in its natural state. |
| What do developers actually believe, do, or experience at scale? | Survey | Gathering broad data from hundreds or thousands of practitioners across the industry. |
| What hidden patterns exist in the code, and artefacts we already create? | Repository Mining | Analyzing historical data from GitHub, GitLab, or issue trackers. |
| What does the entire existing body of research say about this topic? | Systematic Literature Review | Aggregating and synthesizing findings from dozens of prior studies. |
| What happens under conditions we can't safely test in real life? | Simulation | Modeling software architectures and environments computationally. |
| Can we help a real team improve while actively studying the process? | Action Research | Partnering directly with a team to introduce a change and observe the results together. |
The Gold Standard Rule: Control and realism are a trade-off. A controlled experiment gives you a clean answer in an artificial environment. A case study gives you a realistic answer in a messy, real-world environment. You can rarely have both at the same time!
A Worked Example: Does INT8 Model Quantization Actually Save Energy?
Imagine your team decides to quantize a Machine Learning model down to INT8 to cut energy costs. Before you run off to write a quick benchmark, let's see how Empirical Software Engineering turns a vague guess into a rock-solid finding.
Step 1: The GQM Framework (Goal → Question → Metric)
Instead of just running a quick script and grabbing a random graph, you structure your test top-down:
- Goal: Evaluate INT8 quantization on an edge device to see if the energy savings justify any drop in accuracy.
- Questions: How much energy do we save per 1,000 inferences? What accuracy do we lose in return?
- Metrics: Joules per 1,000 inferences (using a real hardware meter), Top-1 accuracy, and p95 latency.
Step 2: The Setup
To keep the test fair, keep the hardware, workload, and environment identical. Run both model variants in a random mixed order across 30 runs, discard the initial warm-up data, and keep background tasks off.
Step 3: The 4 Rules That Separate "Data" from "Noise"
This is where true empirical engineering happens. Before you publish your findings, ask yourself these four critical questions:
- Construct Validity (Are you measuring what you think you're measuring?): Software estimation models let us analyze complex systems, but physical hardware meters provide the ultimate ground truth. A strong study ensures that any software-based power estimation is rooted in, and validated by, real hardware measurements.
- Internal Validity (Did outside factors taint the result?): If your quantized tests ran on a hot afternoon, you might be measuring thermal throttling or air conditioning, rather than your code! That's why alternating and randomizing runs is mandatory.
- External Validity (Will this work for anyone else?): Be honest about scope. You measured one model on one device. Testing a second model family turns an isolated anecdote into real evidence.
- Conclusion Validity (Is the difference actually real?): A 2% energy saving with 5% variance isn't a victory. It's just noise. Always measure spread and effect size, not just averages.
The Big Mindset Shift: In empirical work, your hypotheses are locked in before you run the test. If INT8 shows "no significant difference," that is still a successful result. Proving something didn't work saves future teams from wasting time, which is just as valuable as a massive performance win.
Six things to keep in mind
- Goal first, Metric last: It's always tempting to measure whatever is easy to grab. Don't fall into that trap. Define the problem first, formulate your questions second, and then pick your metrics.
- Context is Everything: There are almost no universal laws in software. "It depends" is a completely valid answer, as long as you can explicitly state what it depends on.
- Validity is a Design Step, not an Afterthought: Thinking about risks and limitations before you run the experiment lets you prevent them. Naming a major flaw in your conclusion section just means you forgot to design it away!
- Developers Are Humans, Not Compilers: When your research involves people, human factors matter. How you recruit participants, request consent, protect anonymity, and incentivize them directly shapes the data you collect.
- Statistical Significance is not Practical Significance: A 0.5% optimization might be statistically real, but if it doesn't change a developer's daily workflow or business outcome, it doesn't really matter. Always look for practical impact.
- Make It Reproducible: Share your scripts, protocol, and raw dataset. In modern research and engineering, an experiment that can't be reproduced by someone else is just a story you told.
References & Resources
- V. R. Basili, G. Caldiera, and H. D. Rombach. "The Goal Question Metric Approach." Encyclopedia of Software Engineering, Wiley, 1994. PDF
- W. F. Tichy, P. Lukowicz, L. Prechelt, and E. A. Heinz. "Experimental Evaluation in Computer Science: A Quantitative Study." Journal of Systems and Software, 1995. doi:10.1016/0164-1212(94)00111-Y
- C. Wohlin, P. Runeson, M. Höst, M. C. Ohlsson, B. Regnell, and A. Wesslén. Experimentation in Software Engineering. Springer, 2024 edition (1st ed. Kluwer 2000; 2nd ed. Springer 2012). doi:10.1007/978-3-662-69306-3
- P. Runeson and M. Höst. "Guidelines for Conducting and Reporting Case Study Research in Software Engineering." Empirical Software Engineering, 2009. doi:10.1007/s10664-008-9102-8
- B. A. Kitchenham, T. Dybå, and M. Jørgensen. "Evidence-Based Software Engineering." Proc. 26th International Conference on Software Engineering (ICSE '04), 2004. doi:10.1109/ICSE.2004.1317449
- D. Mendez, P. Avgeriou, M. Kalinowski, and N. Bin Ali (eds.). Handbook on Teaching Empirical Software Engineering. Springer, 2024. doi:10.1007/978-3-031-71769-7 · editorial introduction preprint: arXiv:2501.07195 · open companion materials: emse.education
- Esposito, Robredo, Sridharan, Travassos, Peñaloza & Lenarduzzi. A Critical Reflection on the State of Data Analysis in Empirical Software Engineering. Published in ACM TOSEM, 2025. arXiv:2501.12728
- K.-J. Stol and B. Fitzgerald. "The ABC of Software Engineering Research." ACM Transactions on Software Engineering and Methodology, 2018. ACM Digital Library