Apr 19, 2013

How Reliable Is The Expert Evaluation Of Your Forecast?

A human evaluation is an essential part of the life of a forecasting project. This happens at least at the presentation of the results. In this evaluation kick in factors that are hard or impossible to measure or include in a math or stat model. The list of these is long - it could be political situation, attitude toward a brand name, expected changes of any kind, a conversation on a party, gut feeling, etc. The evaluation is also heavily influenced by biases and an the agenda of the reviewer. Dealing with these is a matter of countless research papers. A modeling and forecasting practitioner is regularly involved in discussions and opinions on the outputs and the question "How much can I trust this evaluation?"is a constant companion. This is a vast topic but please allow me to put my 10 cents in it.

There is no straight answer. It has lot to do with the art of interviewing, presenting, asking the proper questions and building relation with the customer. These are foundations of not only good customer and personal relations but also of building good, reliable and well accepted models. I would not touch on these and focus on another side of the problem.

A good practitioner knows what is happening in their line of business at the moment and have very good idea about the development in near future. The near future is limited by the nature of the business itself. I would like to call it "a time-horizon of good estimation" or just "time horizon". The time horizon depends on factors like the purchase life cycle, market volatility, number of deals for a period, price of goods or services, commodity of the goods, some segment specifics, etc. In my opinion that the higher the price and lower the number of customers per year, the longer the time-horizon. The higher the customer flow, the lower the prices, the shorter the time-horizon. For an illustration, the salesmen of MRI equipment knows very well what the market will be in next 2, 3 even 5 years. The owner of the local convenience store would be very precise about her sales next week, next month even this year. This is also related to the natural cycle in purchasing the goods or services. The MRI equipment is expensive, big and its purchase goes through careful analysis, tenders, considerable amount of paper work. Bread sales are on the other pole - many customers, low prices and virtually no decision time. Large markets of massively used goods are subject to stochastic variations but still experts are usually very good in estimating the average.

The time horizon naturally depends on the "mileage" of the person on the specific line of business covered by the model to evaluate. It is not only the time spent but also the intensity of work that matters. I very much like the "effective year" term that comes from the nuclear reactor science - the basis idea is two years work on 50% of the capacity means one effective year. It is very rough but conveys the idea. The more precise phrasing would be the effective years spent that matter. Usually, few years are enough for a hard and smart working professional to become an expert you could rely on.

Beyond the natural time horizon people are not that confident and their forecast success is limited. This should not cast a shadow on their qualities, expertize and insight in any way. It does not mean their opinion should be neglected. By design, we humans do not have the ability to proceed large amounts of data and properly identify inter-dependencies in it or other properties. Here come the forecasting/analytical models to tame the complexity and the ocean of data and options coming with it. However, we should not put our trust on the models going into the future right away. I have covered some ideas how to treat the forecasting models in a previous post that you might find valuable.

The experts have great insight within the natural time horizon of their line of work. I would put under review any model that is too far apart from their opinion. However, always account for people's biases and agenda that heavily influences their evaluation on a forecast model. Beyond the natural time horizon opinions should matter less but still have to be deeply analyzed.

I am far from exhausting the topic and covering all the angles and specifics. I hope I shared some ideas to keep in mind during the next discussions of a model output.

No comments:

Post a Comment