In one of my previous blogs I mentioned Moneyball: The Art of Winning an Unfair Game by Michael Lewis as an inspiration for my job as a software analytics specialist. I got the book from one of my customers when I finished the implementation of a measurement approach and the setup of a performance dashboard for his software engineering projects. “You must read this. It’s a great example of how statistics are used in decision making”.
Off course he was right. Customers are always right.
Later on, in 2011, a film based on the book starring Brad Pitt was released. Brad Pitt plays the role of Billy Beane, a general manager of the Oakland Athletics, a baseball team that adopted an analytical, evidence-based, sabermetric approach to assemble a competitive baseball team, despite Oakland’s worrying financial situation. What I’d liked most in the book is the fact that Lewis suggests that where expert judgment is not always a good starting point for decision making, statistics might make the difference. And a second thing that I like is the idea that a team that is certainly not a big spender, the Oakland Athletics had the third-lowest team payroll in the league in 2002 (about $40 million), can make big wins.
See the following excerpt of Moneyball and experience how Brad Pitt tries to understand “What’s the problem?” when talking to his team of experts.
The impact of Moneyball is huge.
As Wikipedia states “Moneyball has entered baseball's lexicon; teams that appear to value the concepts of sabermetrics are often said to be playing ‘Moneyball’. Baseball traditionalists, in particular some scouts and media members, decry the sabermetric revolution and have disparaged Moneyball for emphasizing concepts of sabermetrics over more traditional methods of player evaluation. Nevertheless, Moneyball changed the way many major league front offices do business. In its wake, teams such as the New York Mets, New York Yankees, San Diego Padres, St. Louis Cardinals, Boston Red Sox, Washington Nationals, Arizona Diamondbacks, Cleveland Indians, and the Toronto Blue Jays have hired full-time sabermetric analysts. When the New York Mets hired Sandy Alderson – Beane's predecessor and mentor with the A's – as their general manager after the 2010 season, and hired Beane's former associates Paul DePodesta and J.P. Ricciardi to the front office, the team was jokingly referred to as the ‘Moneyball Mets’.”
Statistically the best coach for Brentford
In a recent article in a Dutch newspaper, NRC Handelsblad, titled “Statistically the best coach for Brentford”, Marinus Dijkhuizen was interviewed.
“With difficulty he kept Excelsior (a rather low ranked Dutch football club) in the major league, yet he forced a job as coach of a British football club, Brentford FC. How? Statistics. Like everything at the club is determined by statistics. Tomorrow awaits the first league match in London”. Apparently when big money is in stake statistics can make a difference.
Football is about millions of Euros. But nowadays software engineering is too.
Okay. Football is about millions of Euros. But nowadays software engineering is about millions too. Companies in banking and telecommunications and large government agencies spend tens, or even hundreds of millions each year on software engineering activities. And portfolios are growing over time. So why are these software companies not making more use of statistics in their decision making?
Within the Software Engineering Research Group TU Delft of the Delft University of Technology we perform research on the subject of how to quantify and qualify software engineering activities in order to help software companies to deliver the most value out of their efforts on building, enhancing and maintaining software. A subject that we dived into recently is pricing of software projects.
Pricing via Functional Size - A Case Study of a Company’s Portfolio of 77 Outsourced Projects
In a research paper “Pricing via Functional Size” we describe a case study on 77 outsourced software engineering projects that are performed in a medium-sized west-European telecom company. The organization experienced a worsening trend in performance, indicating that the organization did not learn from history, in combination with much time and energy spent on preparation and review of project proposals by the main Indian supplier that was responsible for design, build and test activities in the projects.
In order to create more transparency in the supplier proposal process a pilot was started on Functional Size Measurement pricing (FSM-pricing). Both the telecom company itself and the main Indian supplier joined efforts in the pilot to prepare project proposals that were priced (fixed price) based on function size solely. The function point analysis was performed as a part of the project proposal process where the Indian supplier performed the actual function point count and the telecom company reviewed results.
In our research paper we evaluate the implementation of FSM-pricing in the software engineering domain of the company, as an instrument useful in the context of software management and supplier proposal pricing. We analyzed 77 finalized software engineering projects, covering 14 million Euro project cost and a project portfolio size of more than 5.000 function points. We found that a statistical, evidence-based pricing approach for software engineering, as a single instrument (without a connection with expert judgment), can be used in the subject companies to create cost transparency and performance management of software project portfolios.
A Summary of the Results of the Survey Analysis
This list summarizes the findings from our analysis of the survey results:
- Interactions, communications, people
- Improved proposal transparency
- Improve knowledge of Function Point Analysis and FSM-pricing
- Discussion on size when lower price is expected or on waivers
- Organization, processes
- Uniform, standard and simplified process
- Too small projects; no focus on release-based working
- Delay due to search for clarity and review
- Improve pricing tables (e.g. benchmarking, more realistic figs.)
- Promote release-based working based on size
- Promote pricing tables based on applications (technology)
- Perform gap-analysis on FSM-price versus actual effort spent
- FSM-pricing does not cover non-functional requirements
- Low reliability of FSM-pricing when compared to actual effort
- Improved Requirement Management
- Good quality of Function Point Analysis process and products
Happy stakeholders and an operational approach
The telecom company used the pricing approach for more than a year, with happy stakeholders in both the telecom company and the Indian supplier that performed the projects. A survey among twenty-five stakeholders from both the telecom company and the Indian supplier revealed that overall people experienced the transparency of project proposals to be improved, almost everyone wanted the approach to become an operational practice. We found some things to be improved too. So, developers from the Indian supplier told us that the approach that was based on functional size solely, did not enough took non-functional requirements into account. And on the contrary, management of the telecom company asked for a higher coverage of projects that were in scope of the approach.
In a way we experienced a win-win here. The telecom company got more grip on the decision process of software engineering costs and the Indian supplier was triggered to deliver the highest possible value (measured in function points) to its customer as possible for a fixed amount of money per function point. But, as always, good things can end too.
Decision makers come and go. And the operational practices follow.
Decision makers come and go. And new decisions are made on how to proceed with innovation of software engineering. Finally the management team of the telecom company stopped the pricing approach and went for the agile highway. And with that decision making based on statistics changed.
Agile coaches don't like statistics. They see it as waste.
Where at first statistics were used for decision making on project prices without intervention of expert judgment, now agile coaches advise the company where to go. And, as in many other software companies that go agile, agile coaches don’t like statistics. They see it as waste. The best they came up with was badly measured velocity, in my eyes a sort of prehistoric variant of a productivity measure.
Okay, this is a blog so I can state blunt things.
With the risk that people might judge me as one of those elderly they-know-what’s-good-for-you experts as depicted in Moneyball’s “What’s the problem?” scene as shown above, I still do regret that the management team of the telecom company did not opt for a smashing combination of the pricing approach based on functional size and an agile approach on shortening time-to-market and improve process quality. In this way the price of projects and the functionality to be delivered would be fixed, leaving all management attention to go for shortening durations and improving quality, without the often supposed trade-off effects between cost, time and quality.
Especially in today’s agile world we do need decision models that do not solely rely on beautifully drawn, but often somewhat naive frameworks, a deluge of agile gurus that know how everything in the whole world is connected, and the foolish idea that making proper designs and collecting data for statistical analysis purposes is waist. Wow, that was a nice sentence to write down!
Software engineering is not always about big data. But it sure is about big money.
Simply stated: contemporary software engineering is not always about big data. But is sure is about big money. We need to start treating our decision making that way too. We need skilled software analytics experts. But maybe more we need managers that think like Billy Beane.