Quantifying financial risk

The federal government’s Accountability Act, combined with the Modern Comptrollership initiative, have necessitated a new approach to estimating costs for future expenditures. As CMA Michael Lionais explains, Agriculture and Agri-Food Canada has implemented a very successful process to do just this.

By Michael Lionais, CMA

Due to cost overruns across government in the 1990s, the Treasury Board of Canada instituted an initiative called Modern Comptrollership, which has evolved to include a requirement that Chief Financial Officers (CFO) of departments must personally attest to the reasonableness of financial information presented to the Government for decision making. Recently, the Government passed the Accountability Act, which makes the Deputy Minister (senior public servant) the Accounting Officer of a government department, thereby making this executive personally accountable for financial information presented to Government.

These changes underlined the need for an arms-length process by which CFOs can confirm that cost estimates in business cases are reasonable before they are presented by the Department. Agriculture and Agri-Food Canada (AAFC), for instance, has adopted a very successful arms-length process, which will be described in this article.

The validation process

All validations start with the expectation that the cost estimate developed by the project team is reasonable given the information available. The validation is carried out to review the proposal for consistency in applying departmental standards, and to ensure conformity with the spirit or intent of applicable statutes and policies. An arms-length review process also ensures that all cost elements have been properly identified and considered. Through the risk assessment process and the use of simulated risk-adjusted costs, AAFC is able to develop a stable planning figure for use by planning staff in the development of longer-term strategic considerations.

One of the main challenges in assessing financial risk is ensuring that the costs presented are a reasonable estimate at a given time in the approval process. The ultimate objective in performing this analysis is to enable the Director General of Finance to provide an objective recommendation to the CFO as to whether financial attestation should be provided with or without observation.

Potential conflicts

There are well documented human behavioural tendencies in management. Typically, research has indicated that managers are optimistic and tend to overestimate revenues, underestimate costs and think short-term. These tendencies are usually attributed to group think, bandwagon effects and consequential behaviour when one is committed to a project. This can be exacerbated in a government environment that is not constrained by profit. Additionally, there is usually a great deal of pressure to create programs for Canadians as quickly as possible for the lowest incremental cost possible. As such, project managers operate under the pressure of a cost ceiling and delivery deadlines.

Further, managers, in all industries, typically pad their numbers to allow some contingency to address the uncertainty of their cost estimate. Despite this padding, cost overruns still occur routinely. In large part, this has to do with the quality of the financial information used to build the cost estimate of a program under development. Typically, the program structure and delivery mechanisms are refined as the funding estimate is developed concurrently. As such, the funding proposal may not match perfectly with the program design. Further compounding this issue is that the financial estimate will be based on the best information available, but the reliability of information underlying the financial estimate isn’t always identified. Rather, it’s usually assessed intuitively by the reader of the business case.

To address these issues, Treasury Board has mandated CFOs to develop the “most reasonable” estimate so that the most likely project costs can be identified to reduce the likelihood of cost overruns. This is significant because cost overruns typically have to be absorbed within a department’s assigned resources, which means that reduced services (internal or external) or compensation must be provided to Canadians should cost overruns occur. As a last resort, a department can request additional funding from the Government. Remember, it’s illegal to exceed the appropriation (budget) that Parliament has assigned to a government department and that by personally attesting to the reasonableness of the financial information the Deputy Minister and CFO assume a personal accountability to Parliament.

Probabilistic model

The probabilistic model is based on providing a reasonably independent assessment of the quality of the information underlying the financial estimate. This is the foundation of the procedure because if the data underlying a decision is potentially flawed, executives need to factor this into their decision. Individuals preparing the cost estimate aren’t in a position to provide an arms-length assessment of the validity of the data underlying the financial information as it is difficult to challenge a program that you have been championing for the previously mentioned reasons. As such, the data integrity assessment is the foundation of the model.

The model itself is derived from documentation produced by the RAND institute, NASA and the U.S. Department of Defense (U.S. Naval Post Graduate School and Defence Acquisition University). The data integrity scale is similar to those in the NASA Cost Estimating Handbook, Cost Readiness Levels (2004) and in a published article by the Association for the Advancement of Cost Estimating (R.B. Lorance and R.V. Wending. “Basic Techniques for Analyzing and Presentation of Cost Risk Analysis,” 1999).

The costing validation procedure starts with a check to ensure that all appropriate cost components have been included in each cost element. If so, this is deemed to be the most likely cost. If not, the cost element is adjusted as required to determine the most likely cost. The data underlying the most likely cost estimate is fit against the data integrity scale in Table 1. Data is categorized into three broad categories: rough order of magnitude (ROM), indicative or substantive. An appropriate contingency range has been assigned for each category. The analyst will make an assessment of an appropriate contingency rating considering the categorization of the costing data, the type of costing information being considered and the information provided by the Business Case sponsor to substantiate the cost estimate in the business case.

This is a subjective assessment however, it’s common in cost estimating techniques. All data integrity assessments are fully identified, described and defended in the report of the findings, which is factored into the managerial approval process.

Figure 1 depicts a practical scale of how various cost data is rated from highest to lowest data quality. Again, with this information a model is created which identifies lower, most likely and upper bounds for each cost element. The lower boundary range is typically 20% of the appropriate contingency range from the table above. This accounts for the tendency to overestimate revenue and underestimate costs. Also, a triangular distribution that is skewed to the right is typically used because the model’s purpose is to ascertain the uncertainty associated with the financial estimate, not to recreate the financial estimate itself. The distribution is skewed to the right to account for the behavioural tendencies previously identified.

Identifying the appropriate bounds is the most difficult part of the process and requires collaboration with the team who developed the case. It’s essential that the contingency amounts imbedded in the funding requirements identified by the project sponsor be removed to prevent overstating the potential financial risk by double counting uncertainty. Once the base value is determined, the data sources for determining it are identified and matched to the scale in Figure 2. This step can only be accomplished through discussion with the team who developed the case. It is the foundation for determining an appropriate contingency range for the cost estimate. If, for some reason, the business case sponsor is game planning, there are other techniques that can be used to model the costs. One example is to have the sponsor identify a cost for which there is only a 1 in 10 chance that the actual cost will be lower, and then identify a cost for which there is only a 10% chance that the actual cost will be higher.

The model is then simulated using a Microsoft Office Excel Add-In. Several Excel Add-Ins exist to enable the application of stochastic techniques to an Excel spreadsheet. Two examples of these Excel Add-ins are Crystal Ball and @risk. The model is run using Monte Carlo simulation to a 95% confidence level to determine the probability of exceeding the identified cost estimate and the amount of funding required at a specific confidence interval.

For programs in which public funding is disbursed to Canadians, a 95% confidence interval is used. A lower confidence interval, typically 80%, is used for internal purposes. The higher confidence interval is used for disbursements to Canadians because the department must ensure that it’s able to deliver on the Government’s commitments to Canadians; whereas, the department is willing to accept more risk on internal decisions. This is because the data sources underlying internal decisions are typically more refined than the data sources supporting the financial estimate of external programs.

Results

To demonstrate this technique, a simple spreadsheet model is provided below (Figure 2). This spreadsheet is an example of a fictitious program in which the Government will provide $1,000 to each eligible Canadian. In this case, based on Statistics Canada data, it is anticipated that there will be 60,000 eligible recipients. Using standard cost metrics, it will cost $1,345,980 for Direct Program Administration Costs and another $2,430,490 in Indirect Program Delivery Costs for a Total Program Cost of $63,776,470.

In this model, tan squares are links or standard ratios, yellow are data integrity factors, green are assumptions and blue are forecasts. The key to the model are the data integrity factors. In order to better understand the model, several of the data integrity factors will be briefly expanded:

• Amount: This amount is fixed — as such, a data integrity factor of 0% is applied;
• Number of claimants: This estimate is based on Statistics Canada information, which is considered to be very good. As such, the highest data integrity factor of 5% is applied;
• Generic full-time salary: This estimate is considered to be a weak substantive estimate. As a result a data integrity factor of 10% is applied;
• Training: This amount is based on a benchmark that is considered indicative, and a data integrity factor of 20% is applied; and
• Travel: The number of trips and locations were known, but the timing and duration were not, so the estimate is considered to be rough order of magnitude and a data integrity factor of 40% is applied.

When the model is simulated, the distribution in Figure 3 is produced.

This distribution indicates that there is an 85% likelihood that the identified funding of $63,776,470 will be insufficient to deliver the program based on the data integrity assumptions modeled. Moreover, it indicates that there is a potential unfunded liability of $2,391,750 at the 95% confidence interval. This allows CFOs to study the results to determine whether they are willing to accept the indicated financial risk. If so, the CFO is in a position to develop cash management strategies, as the amount of potential financial risk has been quantified. The level of risk should be reassessed as the information underlying the financial estimate is improved over time.

If the CFO isn’t willing to accept the financial risk as modeled, the simulation identifies the relative importance of the cost elements from a risk perspective. The model shows which elements have the most influence on the cost estimate. In this example, it is obvious that the number of eligible recipients presents the most financial risk; yet the data integrity factor indicates the source of the estimate is considered high quality. Consequently, it is likely prohibitively expensive from both cost and time perspectives to refine this source of data further.

However, within the direct program costs, the cost elements can be broken down in a sensitivity chart (Figure 4) to determine whether the data underlying the financial estimate can be improved to reduce the financial risk associated with the program. Furthermore, an estimate can be made of the cost of improving the data integrity to determine if the benefit of refining the information further outweighs the costs of doing so.

In this example, moving from generic to position-specific salary estimates would address 99.5% of the uncertainty in the direct program cost category. This, in itself, shouldn’t be either an overly onerous or expensive process, and the reduced financial risk that would result would likely be worthwhile.

The cost elements that represent the greatest financial risk can be specifically identified and targeted to reduce the financial uncertainty associated with the data source used to create the financial estimate. As such, in the event that the CFO refuses to attest to the reasonableness of the financial estimate, the sponsoring executive is informed of the reasons why and what specific cost elements need to be refined to reduce the financial risk to a more acceptable level. Moreover, this analysis can serve as the basis for the development of risk mitigation strategies as the relative importance of the cost elements from a financial perspective has been quantified.

Impact on business

This procedure has been in place at AAFC since October 2006. Several significant proposals have been reviewed, ranging in value from $100 million to $10 billion. In several instances, the funding requested in the business case was adjusted (up and down) following validation to reflect the most reasonable estimate of the cost implications, considering the reliability of the information upon which the financial estimate is based. Cash management strategies are also being developed to address the potential unfunded liability of proposals that have been attested to as reasonable in situations in which the department is able to accept the potential unfunded liability.

The procedure is reaping intangible benefits as well. Financial risk is being concretely considered in the development of the business case, whereas previously this type of risk wasn’t necessarily a focus during a proposal’s development. This has led to more fiscally informed decision making at AAFC. Also, by quantifying the financial risk in proposals, the fiscal environment is stabilized, which permits executives to focus on strategic outcomes rather than urgent cash management issues. This increased fiscal awareness increases the chances that the business case will be accepted as proposed and reduces the likelihood that Treasury Board will impose financial terms and conditions on business cases submitted.

In turn, increasing Treasury Board’s confidence in the stewardship of a government department may lead it to increase the authority delegated to the department, which will increase the department’s autonomy and enable it to better serve Canadians by allowing it to be more responsive to Canadians’ needs. Additionally, greater fiscal stability in government departments will enable the Government to better project its financial situation, thereby facilitating better governance of Canada.

Conclusion

The procedure described predicts the chance of exceeding an identified budget and quantifies the potential financial obligation between the identified project cost and the predicted cost of the business case at a specific confidence interval. This process is iterative; it can be updated as the quality of the data underlying the cost estimates is updated. This determination of the potential unfunded liability supports the Government of Canada’s Modern Comptrollership initiative and permits the CFO to make an informed attestation to the Deputy Minister of the department that the financial information presented is a reasonable estimate of the projected costs of the business case at the time of the analysis.

In closing, the procedure itself:

• is a scientific methodology based on identified and defensible assumptions;
• is scalable to fit information and time available;
• addresses the optimistic nature of program managers;
• provides a meaningful assessment of the total cost implications of the business case and any potential unfunded liability associated with the proposal;
• demonstrates due diligence as required under Modern Comptrollership; and
• reduces executives’ reliance on intuition.

This procedure was reviewed by a panel of judges and deemed to be the best financial application of the nominations received for the inaugural 2007 Monte Carlo Simulation Award sponsored by Decisoneering, a subsidiary of Oracle, and the makers of Crystal Ball. 

REFERENCES

Randal B Lorance & Robert V. Wending, “Basic Techniques for Analyzing and Presentation of Cost Risk Analysis”, AACE International Transactions, Association for the Advancement of Cost Engineering, 1999.

NASA Cost Estimating Handbook, “Cost Readiness Levels”, NASA, 2004 (http://ceh.nasa.gov/webhelpfiles/Cost_Readiness_Leves_(CRL)s.htm)

DRAFT Agriculture and Agri-Food Canada Costing Handbook, October 2006

Michael Lionais, CMA, joined the Department of Agriculture and Agri-Food Canada as the Manager of Strategic Costing after 20 years of military service. 

Top