Impacts of Bad ESP (Early Size Predictions) on Software Effort Estimation
Authors:
George Mathew,
Tim Menzies,
Jairus Hihn
Abstract:
Context: Early size predictions (ESP) can lead to errors in effort predictions for software projects. This problem is particular acute in parametric effort models that give extra weight to size factors (for example, the COCOMO model assumes that effort is exponentially proportional to project size). Objective: To test if effort estimates are crippled by bad ESP. Method: Document inaccuracies in ea…
▽ More
Context: Early size predictions (ESP) can lead to errors in effort predictions for software projects. This problem is particular acute in parametric effort models that give extra weight to size factors (for example, the COCOMO model assumes that effort is exponentially proportional to project size). Objective: To test if effort estimates are crippled by bad ESP. Method: Document inaccuracies in early size estimates. Use those error sizes to determine the implications of those inaccuracies via a Monte Carlo perturbation analysis of effort models and an analysis of the equations used in those effort models. Results: While many projects have errors in ESP of up to +/- 100%, those errors add very little to the overall effort estimate error. Specifically, we find no statistically significant difference in the estimation errors seen after increasing ESP errors from 0 to +/- 100%. An analysis of effort estimation models explains why this is so: the net additional impact of ESP error is relatively small compared to the other sources of error associated with in estimation models. Conclusion: ESP errors effect effort estimates by a relatively minor amount. As soon as a model uses a size estimate and other factors to predict project effort, then ESP errors are not crippling to the process of estimation
△ Less
Submitted 19 February, 2018; v1 submitted 9 December, 2016;
originally announced December 2016.
Negative Results for Software Effort Estimation
Authors:
Tim Menzies,
Ye Yang,
George Mathew,
Barry Boehm,
Jairus Hihn
Abstract:
Context:More than half the literature on software effort estimation (SEE) focuses on comparisons of new estimation methods. Surprisingly, there are no studies comparing state of the art latest methods with decades-old approaches. Objective:To check if new SEE methods generated better estimates than older methods. Method: Firstly, collect effort estimation methods ranging from "classical" COCOMO (p…
▽ More
Context:More than half the literature on software effort estimation (SEE) focuses on comparisons of new estimation methods. Surprisingly, there are no studies comparing state of the art latest methods with decades-old approaches. Objective:To check if new SEE methods generated better estimates than older methods. Method: Firstly, collect effort estimation methods ranging from "classical" COCOMO (parametric estimation over a pre-determined set of attributes) to "modern" (reasoning via analogy using spectral-based clustering plus instance and feature selection, and a recent "baseline method" proposed in ACM Transactions on Software Engineering).Secondly, catalog the list of objections that lead to the development of post-COCOMO estimation methods.Thirdly, characterize each of those objections as a comparison between newer and older estimation methods.Fourthly, using four COCOMO-style data sets (from 1991, 2000, 2005, 2010) and run those comparisons experiments.Fifthly, compare the performance of the different estimators using a Scott-Knott procedure using (i) the A12 effect size to rule out "small" differences and (ii) a 99% confident bootstrap procedure to check for statistically different grou**s of treatments). Results: The major negative results of this paper are that for the COCOMO data sets, nothing we studied did any better than Boehm's original procedure. Conclusions: When COCOMO-style attributes are available, we strongly recommend (i) using that data and (ii) use COCOMO to generate predictions. We say this since the experiments of this paper show that, at least for effort estimation,how data is collected is more important than what learner is applied to that data.
△ Less
Submitted 29 September, 2016; v1 submitted 18 September, 2016;
originally announced September 2016.
