Who cares if it’s reliable? It only needs to be “credible” – believable.
Credible: “Worthy of Belief”
Reliable: “Repeatability of findings”
Credibility comprises both trustworthiness and expertise. It has both objective and subjective elements. One can also increase credibility by being a good salesperson by convincing the reader of your viewpoint. You form an opinion, then support that opinion. Credibility is not objectively measurable.
Reliability is the degree of consistency of a measure. An appraisal is a measure of value. One can increase reliability by being more accurate and more precise (trueness and sureness). Reliability is measurable.
A measurably reliable, repeatable valuation will always be more credible!
A well-proffered traditional appraisal may or may not be more reliable.
I believe that trustworthiness and expertise are important. But this blog is about reliability. How can the valuation profession deliver more reliable work?
The first requirement is that analysis and results be repeatable. The second requirement is that the accuracy of the results be measurable.
How do we achieve repeatability? Reproducible work means two analysts can take on the same task, and get the same results. What are the steps?
- The problem must be stated in identical terms. For most appraisal work, this is not difficult.
- The data and predictors (comps and elements of comparison), must be the same.
- The analytics must be the same (both classification and association).
In line with the above three steps, some calculations may entail small differences of decision when it comes to handling outliers. These must be clearly set out. Outliers may be deleted, corrected, included, or explained, possibly with a sensitivity analysis, giving the reader/client options. Either way the workflow breadcrumbs, the audit trail, is clear. It’s that simple.
Why doesn’t the valuation profession provide these superior results possible today? Why not results that are simpler and more meaningful? Simply, it’s inertia. Personal resistance and social compliance.
- Our education. Appraiser education substantially repeats approaches developed in the 1930’s. Some progress has been made, such as the use of the accountant’s spreadsheet, the discounted cash flow model, and more recently, use of graphs and Geographical analytics (GIS). However, no progress has been made on data selection. It’s still, “trust me, I know a good comp when I see a good comp.” It’s still – use no more than 7 comps in a report because “it’s too hard to understand” more than that.
- Our standards. While standards require the use of all available information – the universal violation of those standards by peers, by clients, and by regulators is strongly established. “Everybody does it.” “We expect it.” “We enforce it.” The written word is overwritten by the belief system – is it “worthy of belief.”
Today. We have the software tools. We have the data. We have the straightforward models. We even have some appraisers who know how to develop evidence-based, measurably reliable valuations. What we need is one or two major clients who genuinely want a measurably reliable valuation. It is possible. It can be done.
January 24, 2018 @ 8:45 am
I would still like to see some justification or additional reasoning for including an overwhelming number of comps. In a cookiecutter subdivision with activity-its reasonable to assume that beyond 4-6 comps, you are just repeating the obvious.
In a rural market with super limited activity and nothing conforming to each other and WILD variance in apparent value based solely on stats, I believe you are inducing doubt in a desk-jockey who lives in a cubicle 2,000 miles away who cant believe that values can be that different.
Explain the comps you chose, explain why/where/how, attempt to reconcile value differences while acknowledging the “wife loves the drapes” factor and 4 sales, 2 listings should be good especially in a “rule 2” situation on a purchase. 6-9 can be used in very unusual situations where the client just HAS to have bracketing even if comps are truly otherwise irrelevant.
Please explain why I am wrong so I can be better.
January 24, 2018 @ 9:56 am
Mike,
In a word, bias Explain why using all of the and forget the desk-jockey. who I assume is a reviewer of some kind.
At the prices we have allowed our work to sink to I understand the impetus to be brief and efficient; however, our job regardless of what we are paid, as George advises, is to be objectively reliable, always.
Edd
January 24, 2018 @ 11:58 am
George,
Thanks for your keen perceptions and honest comments. USPAP does need a meaningful overhaul.
I asked an ASB member, “Who decides credibility?” The answer was, “The client.” Believe it or not that is what I was told.
Every con-man knows that if you want ’em to believe just tell ’em what they believe, then whatever you say will be “worthy of belief” and as an appraiser you can claim membership in the ranks of snake-oil salesmen. No wonder we are supposed to read the purchase contracts. How else can we keep up our averages of hitting the sale price and maintaining our credibility? Of course it does beg the legitimate question of why use an appraiser when the broker already told you what it is worth.
On the other hand you can do the work of being reliable and independent from walking the treacherous line of people-pleasing. Our job really is to figure out what is going on and then to tell it like it is. That is rough to start with and then according to USPAP we have to deal with being credible. Something has to give and it should be USPAP. I’ll go with reliable. Tell ’em I said so, would you?
Hope to take your class this summer. Keep up the good work.
Edd
January 26, 2018 @ 8:03 am
Good article. It has always been an objective of mine to explain reliability, as well as credibility, within the appraisal report. In my viewpoint, an appraisal cannot be done unless it can be done credibly. But, within the range of credible appraisals, there can be a fairly wide range of reliability. Putting aside different appraiser qualifications and abilities, the usual difference in reliability is caused by differences in available data. The quantity of data, similarity to the subject, validity of verification, types of analysis used, and pertinence to the appraisal problem to be solved should be explained in the report. Big data never even attempts to explain that three different people involved in a sale may give three different answers as to motivation and other conditions of sale… and sometimes even different answers as to price. Statistical analysis methods work better when a lot of data is available, but that data must also be reliable. In the face of data that may be questionable, it is axiomatic that the appraiser must use some degree of subjective reasoning.
The Standards requirement does not require use of all available data. The Standards only require consideration of available data. To say otherwise is a red herring, as your article states within that outliers might be dealt with by being discarded. What if you are appraising a property that has only outliers as comparable sales? How reliable is an appraisal that analyzes every single sale, when only two of the available sales in the data set are very comparable to the subject? Statistical analysis should be a part of every appraisal and works well as a method of valuation when there is a large enough data set with close enough similarity (or easily enough determined adjustment for dissimilarities). Otherwise, it is simply GIGO– garbage in garbage out.
The main problem with some attempts to measure reliability is that there has been a tendency to equate reliability with results. The appraisal could be credible as determined by the reader, and reliable as indicated by review of the work, and still not indicate the future transaction price. The reason is simple: markets are not perfect. No statistical analysis that I have ever seen can account for emotional attachment to real estate, i.e. a seller who is obstinate about price combined with a buyer who is willing to pay more than market, or conversely a more knowledgeable buyer combined with a more motivated seller.
My only real quibble with the article, however, is that there seems to be an assumption that repeatability equates reliability. History is rife with examples of people, scientists, economists, and others, who got the same results over and over again, but were proven wrong when new data or new methods became available. The simple fact that someone can repeat your steps and get the same result does not necessarily mean that result is reliable and that should be clarified.
January 26, 2018 @ 8:20 am
EXACTLY!
I had a subject in a gated super rural mountain community where 95% was occupied 10% of the time seasonally. My entire market MLS went back 10 years. The client was so sure I was wrong, in order to shut them up I had to spend an inordinate amount of time detailing the sales year by year for the past 10 years. It did not increase the reliability of my report as that market is based solely on availability and not a whit on a data point. People with too much money and no information buying because the drapes are blue “…and I love that color.”
Oversimplified but-agreed.
January 29, 2018 @ 8:11 am
If the mortgage lending industry wanted “reliable” appraisals, USPAP would refer to reliable conclusions, not “credible” conclusions. The lending industry wants credible appraisals since it then can pick and choose the appraisals it deems credible, or harass the appraiser until s/he renders the appraisal “credible” in the lender’s eyes. The way around this is fairly straightforward, but those of us who advocate it are subject to ad homonym attacks as “elitists”. Therefore, the demise of the residential end of our industry continues. Lenders are not interested in a property’s market value. They merely want us to confirm the contract price so they can close the loan and book their fees. That we have chosen to be complicit in this does not speak well for us. That another housing crash is on the horizon should be clear to any critically-thinking appraiser. We were given FIRREA after the S&L debacle on the late 80s. We got HVCC and Dodd-Frank after the last crash. After the next crash, who do you think will take the blame for it? Hint: It won’t the lenders, the builders, or the brokers. Now, what are we going to do about all of this so our industry benefits us, not the lenders, builders, and brokers?
January 29, 2018 @ 2:58 pm
I know from previous discussion re: “what will we do about this?” means “what will we the appraisers do about this?”
It should be patently obvious to everyone by now that no one, not Dodd, not Frank, not Cuomo is going to do anything on our behalf. We will simply not be taken care of.
Hope to see you in the next iteration of appraisal where we must have developed leadership with you know whats and concern for the boots on the ground appraiser who has taken the time to become educated.
February 3, 2018 @ 3:10 pm
Thank you all for the continuing comments. I am pleased to provide a place for those who care to pose excellent questions and suggest possible answers.
—
The reason for using the complete market segment is a simple statistical law. It is called the Fisher information function. Briefly it says that the trueness and sureness of your answer goes up by the square root of the amount of information you put in. More information = better answer.
The key is that the goal of EBA (Evidence Based Appraisal©) focuses on getting the ideal data set, not on scratching together ‘enough’ comps. Yes, 4-6 comps is easier to understand. But the predictor associations and contrasts (adjustments) are much, much more reliable if developed on the entire data set, not on just a subjectively selected set of comps.
It is true that in cookie cutter places, the improvement in trueness and sureness is small. But in markets with high variation and limited activity, optimizing the size of the data set is especially important.
The EBA© approach does require some change in thinking. We have become so inculcated in the old ways, it is hard to mentally integrate the new technology. At one time we passed laws against those noisy, stinky motorcars — to protect the obvious established advantages of horses. Technology.
For us, the current technology means we must select the optimal data set. It must include all directly competitive sales. USPAP has it right in this way. (See SR 1-4). To get the data right is the foundation of data science, and Evidence Based Valuation.© Remember the Dell Data Impossibility Theorem:
You can’t get objective results from subjectively selected data.