Software Cost Estimation – Will the industry ever learn?

Software Cost Estimation is often considered the most difficult part of an integral cost estimate. Unrealistic software estimates often result in disasters. Examples are new tunnels, new metro lines or new factories that can’t start their operation because everything is ready, except for the software.

One of the main reasons for this is often an inaccurate and optimistic software cost (and schedule) estimate. Even professional software development organizations rely on their ‘experts’, when estimating large multi-million projects. However, human estimates are usually optimistic. Experts are often not able to oversee the complete project and forget to estimate certain parts. They usually don’t know the size of the software that needs to be developed. Furthermore, they don’t learn from the past and have no data of similar completed projects. So, expert estimates can’t be accurate.

Even in a time-boxed agile project, you need to estimate the total number of sprints per given team size to understand the total cost of the software to be developed and the moment when certain parts of functionality are ready. Humans simply can’t estimate this in a realistic way. So, most projects start with optimistic expectations and with too few people. These projects often end up in disaster. I just witnessed a large project recently that was estimated to last 12 sprints and are now in their 28th sprint, without anything useful already in production.

Software is gaining importance all the time and is often part of a greater object. Accurate software cost (and schedule) estimates are becoming extremely important for organizations, as they won’t be able to start operations without the software.

Unfortunately, Software Cost Estimation is still not regarded as a specialist job in the market. Most software cost estimates, even in CMMI level 5 system integrator companies, are carried out using Estimation Processes on Level 1 or 2, which means human expert estimates.

Especially in agile software development, estimation methods are of very low maturity and very low accuracy. So many organizations only use story point metrics, which can work quite well in one team, but can’t be used to compare between teams. Also recent research shows that even in experienced agile teams, story point estimates are not very accurate and predictability is low.

Research published by professor Abran[1] shows that sprint estimates in COSMIC function points show larger predictability and less variance than story point estimates, and as COSMIC is standardized, it becomes possible to compare between teams and with the industry data of ISBSG. Probably the same results can be achieved when using one of the other ISO standardized methods for functional size measurement, e.g. Nesma, IFPUG and FiSMA.

The International Cost Estimation and Analysis Association (ICEAA) and Nesma, supported by many industry experts and organizations (e.g. ISBSG, IFPUG, COSMIC and others), are creating the Software Cost Estimation Body of Knowledge with an associated certification program. This will result in certified Software Cost Estimators that will help the industry in creating more accurate estimates, improved decision making and increased business success.

ISBSG strongly supports this initiative and hopes that the industry will finally learn to estimate in a more realistic and accurate way, using mature estimation models and relevant data, like the ISBSG data repositories,  instead of expert opinions.

Like W. Edwards Deming said: ‘Without data, you are just another person with an opinions’.