Critical study of the parametric model development process
Abstract:
Complex parametric models may consist of many interrelated cost estimating relationships (CERs), as well as other equations, ground rules, assumptions, and variables that describe and define the situation being studied. Models generate estimates based upon certain input parameters, or cost drivers. Parametric models can generally be classified as commercial or company-developed. This review provides practical information about developing, deploying, and maintaining company-developed parametric models. Company-developed models – also referred to as company-owned, in-house, or proprietary models – differ from cost estimating relationships (CERs) because of their higher level of complexity, and the range of costs they estimate. Commercial parametric estimating models, available in the public domain, use generic algorithms and estimating methods which are based on a database that contains a broad spectrum of industry-wide data. Unlike commercial models, company-developed models are designed for the specific estimating needs of an organization or to describe a particular product. A proprietary model offers an alternative to trying to use a commercial model to meet an organization’s unique estimating requirements.
JEL classification: C50, C51
Key words: equation, parameter, parametric model, commercial model, proprietary model
Introduction
A parametric cost model can be viewed as the collection of databases, cost estimating relationships (CERs) [1], cost factors and algorithms, which together are used to estimate the costs of a system and its components. A parametric cost model uses known values to estimate unknown ones. Industry use parametric models to support conceptual estimating, design-to-cost analyses, life-cycle cost estimates, risk analyses, budget planning and analyses. Parametric models can also be used as the basis of a cost estimate in preparation of firm business proposals, or in the independent assessment of cost estimates prepared using a traditional estimating approach.
Models generate estimates based upon certain input parameters, or cost drivers. Parameters “drive the cost” of the end product or service being estimated. Some examples are weight, size, efficiency, quantity, and time. Some models can develop estimates with only a limited set of descriptive program inputs; others, however, require the user to provide many detailed input values before the model can compute a total cost estimate. A model can utilize a mix of estimating methods, and it may allow as inputs estimates from other pricing models (or information systems) or quotes from external sources, such as subcontracts.
Several companies implemented commercial parametric estimating hardware models, which can rapidly compute development and design costs, manufacturing costs of prototypes, and production unit/manufacturing support costs.
Commercial parametric estimating models use generic algorithms and estimating methods which are based on a database that contains a broad spectrum of industry-wide data. Because this data encompasses many different products, a company working with a commercial parametric model must calibrate it before using it as a base of estimate for proposals submitted to the higher-tier contractors. Calibration tailors the commercial model so it reflects the products, estimating environment, and business culture of that particular company.
A proprietary model offers an alternative to trying to use a commercial model to meet an organization’s unique estimating requirements. Proprietary models are developed for an organization’s own product and cost estimating needs and are, in effect, self-calibrated.
Proprietary models can be implemented for a variety of estimating purposes, and have a wide range of complexity, completeness, and application.
The proprietary model development process
The major activities involved in developing a proprietary model are:
Step 1: Identifying the Parametric Model Opportunity
One of the most critical steps in the proprietary model development process is the identification of a good opportunity for implementing a parametric model. This involves two points. First, it is important to investigate the feasibility of developing the model, which entails an evaluation of both its technical feasibility and cost effectiveness. Technical feasibility refers to the ability of the model to meet the estimating needs of the organization, and examines whether the organization has the resources to develop the model within a reasonable timeframe. This includes performing a cost-benefit analysis to decide whether a proprietary model would be cost-effective to implement and maintain. All potential benefits should be considered in the cost-benefit analysis; for example, contractors have achieved significant savings in proposal preparation, evaluation, and negotiation through the implementation of proprietary parametric estimating models. Other contractors have achieved additional benefits through multiple applications of the same model, such as for design studies, target costing, and contract risk management as well as basic estimating.
The second critical point involves gaining the support of internal upper-level (including program) management and key customer management. If the model then meets the acceptance criteria provided by these groups, they agree to support its proper application in subsequent proposals. Little good comes from implementing a proprietary model if there is no internal management buy-in, or no support from the key customers on the estimating technique. Also, the firm’s management will want to understand the results of the feasibility study so it can properly assess the financial investment required to support model development and on-going maintenance activities, such as training, model enhancements, and software corrections. On receiving approval to begin development from internal and external management, the contractor establishes an implementation team to guide the creation of a valid proprietary model. This team should include representatives from the company and key customers.
Step 2: Information Systems Needs
When implementing a proprietary model, the organization should commit and obtain the necessary resources for information systems development and support activities. Information systems support is required for a variety of functions:
- defining the formal system requirements needed to support the cost estimating model (e.g., hardware, software, interfaces with other systems);
- testing the model to ensure it adequately satisfies all end-user requirements;
- maintaining the integrity of the model throughout its life p by establishing procedures to manage and control all changes (i.e., configuration management);
- providing software support services once the model is deployed to keep it operational (e.g., corrections, revisions, miscellaneous enhancements).
When simpler models are implemented (e.g., spreadsheet models), the degree of support is smaller, but the configuration management and long-term maintenance issues still must be addressed.
Step 3: Data Collection and Analysis
Historical costs should be used, with the development team ensuring that they are relevant to the firm’s current operating procedures.
In an effort to include as much relevant cost data as possible, analysts normalize it as it is incorporated into the database. They adjust data so it is as homogeneous as possible (e.g., similar in content, time value of money, quantity), and does not contain anomalies. Programmatic, noncost data may also require normalization. The analyst must assess the condition of each program’s data and make appropriate adjustments as required.
When developing a model, the team identifies the main characteristics, called the primary cost drivers, that are responsible for, and have the greatest impact on, the product or services cost to be estimated.
Step 4: Model Development
The development of a proprietary model incorporates many anticipated uses and goals – such as estimating/users’ requirements, availability of credible data, life-cycle costs, systems engineering costs, forward pricing rates – and it must integrate these into the parametric estimating approach. The modeling process, in particular, focuses on these tasks:
- specifying the estimating methods for accomplishing the estimating goals;
- identifying the job functions and other elements of cost that will be estimated;
- defining data input structures;
Proprietary models may contain a number of different estimating techniques.
Step 5: Calibration and Validation
Parametric models are calibrated and validated before they are used to develop estimates for proposals. Since proprietary models are based on an organization’s historical data, they are considered to be self-calibrated.
Validation is the process, or act, of demonstrating the proprietary model’s ability to function as a credible estimating tool . Validation ensures:
- estimating system policies and procedures are established and enforced;
- key personnel have proper experience and are adequately trained;
- proper information system controls are established to monitor system development and maintenance activities in order to ensure the model’s continued integrity;
- the model is a good predictor of costs.
Models should be validated and periodically updated to ensure they are based on current, accurate, and complete data, and that they remain good cost predictors.
The purpose of validation is the demonstration of a model’s ability to reliably predict costs. This can be done in a number of ways. For example, if a company has sufficient historical data, data points can be withheld from the model building process and then used as test points to assess the model’s estimating accuracy. Unfortunately, data sets available are often extremely small, and withholding a few points from the model’s development may affect the precision of its parameters. This trade-off between accuracy and testability is an issue model developers always consider.
When sufficient historical data are not available for testing, accuracy assessments can be performed using other techniques.
Another testing methodology compares a commercial program’s final cost to the proprietary model’s estimate of it. However, it may be months, or years, before this approach can be applied to a given program. The model team may use this method when a program is near completion, or is at a point where a meaningful earned value performance index for it can be determined.
Step 6: Estimating System Policies and Procedures
After validation, the company must modify its estimating system policies and procedures to explain the appropriate use and application of the model for reviewers and company users. In particular, the model’s developers need to document its proper use as a valid bidding tool.
Companies should also explain the model’s design, development, and use. For example, the contractor, as part of its support for the follow-on production model and estimating tool, developed a detailed manual containing information about the mechanics of the model, its estimating methodologies, and the timing of updates. The company also amended its Estimating System Manual to include a section on the model, and to refer the reader to the model’s own manual.
Step 7: Internal Approval Process
Model developers need to assure company representatives that the model relies on the firm’s historical data and, therefore, captures how the company executed similar projects in the past. Any departmental budget allocations produced by the model should reflect the average budgetary split the firm has historically experienced. Developers should also consider the fact that a model, if approved, might change the way the company anticipates executing an existing (or planned) program (e.g., the project director may need to shift work and modify the budget). This obviously affects the circumstances under which other company personnel would approve the model.
A best practice from contractor experience involves the integration of the company representatives into the model implementation team. As an example, when implementing the follow-on production model, the model designers, from the beginning, solicited the participation of key internal representatives. During the development of each module, the team incorporated the inputs of the functional department primarily responsible for executing that portion of the project which the module was designed to estimate. Although the Finance Department led the model building effort, it continuously reviewed its progress with representatives from the Engineering and Manufacturing Departments. These representatives were responsible for coordinating and obtaining any necessary information from their organization, and keeping management informed.
Step 8: External Approval Process
Although a company may internally approve a model, the customer must also be shown that the estimating approach is valid. The involving of customers in up-front decision facilitates their acceptance of parametric techniques.
In seeking acceptance of a proprietary model, the company formed a Continuous Improvement Process (CIP) team [4]. The team’s composition included company representatives from various departments. All team members participated in establishing selection criteria for the model’s database. Based on the selection criteria, the contractor personnel collected actual cost data from many contracts. When using the model for the first time with a buying organization, the CIP team invites the buying organization to the company for a joint review and explanation of the model.
Immediately after obtaining funding to develop the model, the developing company discussed it with other contractors, additional government organizations, to ensure widespread support in data collection and model validation.
Including customers on the development team does not guarantee a model’s acceptance, of course. It does ensure that the customer has a voice in the model’s design and usage, but the model’s ability to reasonably predict costs is the ultimate basis for acceptance. No person, internal or external to the company, can prove this before final development and testing.
Step 9: Model Maintenance
Through the development process, the team develops a sense of how often the model needs updating. Maintenance activities include not only the incorporation of new data into the model, but also an evaluation of the mathematical relationships between the technical parameters and the costs the model estimates. Periodic evaluation of the model is required to ensure the estimates are relevant and the contractor is using the most current, accurate, and complete data.
New data is contributed as programs mature and, occasionally, from non-company sources. In some situations, the cost modelers develop new CERs, based on a subset of the original database, in order to better match a new estimating requirement.
The process of maintaining a model involves keeping an audit trail of the CERs developed, the data points used, and their statistical effectiveness.
Conclusions
Company-developed parametric models – also known as proprietary models offers an alternative to use a commercial model regarding organization’s own product and cost estimating needs.
No company or individual can develop a valid model without the participation of a number of key people which include the customers, all interested company personnel, and government representatives.
Some concepts should be considered by all implementation teams as follows:
- establish a process flow and target development dates to ensure all team members provide their inputs to the model’s design;
- consider the costs and benefits of model development;
- evaluate commercial models as an alternative to proprietary development;
- remember that the goal is to establish a more efficient and reliable estimating system, not just create a model.
References
- Stuparu D., Vasile T., Stanciu M.
- The Cost Estimating Relationships (CER’s) – modern method for predicting cost, Revista Academiei Fortelor Terestre, nr. 1/2010.
- Vasile T., Stuparu D., Daniasa C.I.
- Collection and Normalization of Parametric Data, Analele Universitatii din Oradea, Tom XVIII, vol. II, 2009, pp. 703-708.
- Stewart R.D., Wyskida R.M., Johannes J.D.
- Cost Estimator’s Reference Manual, 2nd Edition, New York, Wiley, 1995, pp. 57-67. http://www.ceh.nasa.gov/webhelpfiles/Cost_Estimating_Handbook_NASA_2004.htm
