Rafael Aguayo
A master carpenter after a long and fruitful career decided to retire and take up a lifelong ambition to become an economist. No longer needing his tools he sold them on eBay. With these tools he had built marvelous cabinets of sublime beauty; produced toys, musical pieces and whole houses.
The entrepreneur who bought his tools on eBay knew the reputation of the Master and had heard people rave about the beauty, utility and overall workmanship of his products. He rented out the tools to others so they could boast of using the tools of the master. He then had the tools duplicated and sold them with a certificate. Anyone who bought those tools could claim to be using the tools of the master. And the expectation was that they would produce masterpieces on a par with the Master’s. Alas, it did not work out that way.
Anyone reading this knows that a carpenter no matter how good his tools cannot produce great work unless he spends years training with another master. Yet in management it is assumed that a certificate, a degree or a black belt somehow confers that magic ingredient.
Some Special Masters
Some people do not believe there is any special knowledge, skill or smarts beyond the tools and some kind of mastery of the tools. One historical example clearly elucidates the difference that profound mastery can make. The best violins are generally considered those made by two eighteenth-century masters of Cremona, Italy: Antonio Stradivari and Guiseppe Guarneri del Gesu. Violins made by these two masters are considered the best in the world and violinist pay tens of thousands of dollars to own and play one. Others have tried to duplicate their efforts yet they still fall short.
Some people make an excellent livelihood just cleaning and tuning these masterworks. In one case, however, a craftsman noticed a small bump in the wood on the inside of one of these violins and sanded it down—thinking that would improve the sound. The violin never sounded the same again. Try as he might to restore the richness of the sound to the instrument he could not.
If you had all of their tools, no matter how hard you tried you could not duplicate the rich sounds of the violins created by the two eighteenth century masters. You could build violins that looked identical; you can build violins that sound quite good, but not as good. Their violins have been taken apart and examined for over two hundred years, and no one yet knows how to make violins that match their richness of sound.
Beyond Tools
If you had trained with one of these masters you would be able to build a violin like they had already made. But these two masters were constantly experimenting and improving throughout their lives. They did not start out as preeminent violin masters, they became. And to keep growing and keep improving requires more than supreme craftsmanship and knowledge of what worked in the past. What are some of the areas of knowledge that these two had mastered? They must have understood and had an appreciation for music, the theory of music, and what we today call acoustics and vibrations. They understood materials such as wood, egg yolks and various others. They must have had some understanding of chemistry, tools and probably other areas that I would be hard pressed to name. And they were able to understand how all of these work together as a system to create a great violin. With this systemic appreciation of the various areas of knowledge they not only created great violins but kept experimenting and producing different and unique, timeless violins that lasted for centuries, which even today cannot be matched.
Tools by themselves, of course, are important. One difference between humans and the great apes is our ability to build tools of unmatched utility, complexity and specificity. And if you are competing against chimpanzees, tools by themselves may be sufficient. But for those who wish to improve the condition of humankind something more is required.
What to call this Beyond-Tools quality
Deming seemed to have struggled to find the proper name for this extra ingredient. I heard him refer to it as brains, or smarts. “Profound knowledge, call it unusual knowledge if you prefer,” I heard him say. And the term most used today is Profound Knowledge or, A System of Profound Knowledge.
But what are some of the aspects of Profound Knowledge (PK)? I would like to deal with a few parts that probably have not been well enough understood.
Data: “Show me the data,” is a common refrain. Some people even attribute this to Deming along with “In God we Trust, all others must use data.” In seven years with him I never heard either and I doubt that he said them, although it might be possible. Data is important but by itself it is meaningless.
There is no such thing as raw data. When you undertake to measure something you already have a theory in mind concerning what is important. The data you generate is a function of your theory. Some people are so attached to their theory that they will even change the data to conform to the theory. So-called scientists are caught doing this all the time.
Data at the Beginning of the Scientific Revolution
One fascinating example of data usage comes from the 17th century astronomer and mathematician Johannes Kepler. At the time there was a scientific controversy as to the nature of the solar system. Did the sun and planets revolve around the earth, the accepted Ptolemic model, or did the earth and planets revolve around the sun, the Copernican model. In addition there was a third view, that of Tycho Brahe which posited that the planets revolved around the sun and all of these revolved around the earth.
To resolve the issue of course required much better data and Tycho Brahe had spent a lifetime designing and building better instruments and then used these to gather data over several decades. Kepler fortuitously was given access to this data. Consider the nature of this data. At a given time on a given day at a given place on the earth a measurement is made on the position of a given planet. Time of course is measured using a clock of questionable accuracy; day is determined using a calendar that makes several assumptions such as the length of the year; position assumes the earth has a particular shape and is really two measurements, longitude and latitude; and the position of the planet are also two measurements, its height in the sky relative to two horizons.
Further Adjustments of Data
Already the data is loaded with assumptions and theory. But in order to make it useful it needs to be adjusted. Light from the sun and the planets bends as it enters the atmosphere. This is called the refraction of light. This refraction leads to an illusion that each planet is in a different position in the heavens than it really is. To compensate for refraction Kepler had to adjust the data to reflect the actual position of each planet at any given point in time. So he had to determine the amount of the bending of light, which varies with the angle that light enters the atmosphere. He had to invent Mathematical Optics, develop the theory for the refraction of light and then adjust the position of the planets accordingly.
But if the earth was moving through space as per the Copernican theory then the data had to be further adjusted to reflect that movement. Phew! And this is just a sample of what was involved in making sense of the “raw data.”
The Results
Was it worth it? While the Copernican model may have been an improvement over the Ptolemic model neither was very good at predicting. After a couple of years their predictions for astronomical events were off by a month or more and the tables of future events had to be recomputed based on the current position of the planets. With Kepler’s Rudolphine Tables astronomical events could be computed hundreds of years into the future and hundreds of years into the past with great accuracy. His mathematical model of the solar system is essentially what we use today.
And all this was done without a telescope, without a computer and without calculus.
Deming used to say, “Experience without theory teaches nothing.” Let me state Aguayo’s first rule, “Data without theory is meaningless.” A corollary is, “There is no raw data.”
I heard a scientist once describing what the mathematician in her group did as: “Making sense out of nonsense.” Most scientific data is a jumble of numbers and to get it to make sense requires knowledge and theory. The group’s mathematician was also a scientist who understood the science that had generated the data. She could not be replaced with a computer, the knowledge required for her to do her job was both deep and wide.
In management there seems to be an inherent assumption that the numbers give a clear picture of what is going on and therefore with a few tools and clear thinking major issues can readily be resolved. Not only is this naïve and incorrect, it is a dangerous and expensive mistake.
An Example from Business and Education
Today everyone is an expert in quality and an expert in management and the use of numbers. Politicians and pundits say, “Look at the numbers, those below average are no good. Get rid of them.” In education the chorus of experts who want to fire the bottom 10% includes everyone from the Mayor of New York, the President of the United States and Nicholas Kristof a normally sane Op-Ed writer for the New York Times.
I recently heard one statistician from a major university say that 70% of people he surveyed believe they are in the top 10%. Of course “it is not possible for 70% of any category of people to be in the top 10%.” Many people would agree with that and the next step for these ‘would-be’ statisticians and quality consultants is that there is something wrong with the 60% who think they are in the top 10% but are not.
I wonder how they would feel if the people vying to tell half the people they are below average were to be told the same thing. The statistician being interviewed may be in the bottom half. Nicholas Kristof may be in the bottom half of New York Times journalists. I suspect they would protest.
Of course this kind of thinking is simplistic and fallacious. While it is not possible for 70% to be in the top 10% it is possible for 70% (or even higher) to be excellent at what they do. This is a function of the system. How do you create excellent teachers? It is by supporting them, giving them valid feedback, and helping them grow. The feedback should not be of the sort “hey you are 23 out of 50, just average,” that is debilitating and assures the loss of morale. Great teaching requires high morale and a sense that you are helping students. By defining a great teacher as the top 16% as Kristof did, based on a study he was quoting, you limit your society. You assure mediocrity.
Is it possible that everyone in a group is competent and doing a good job? Yes it is. If each has been properly trained, is motivated and gets good coaching and continual training then each one can be an excellent teacher. Once you tell half of them that they are below average you are destroying at least half, maybe much more. Pride in their work and in their profession is destroyed.
There are two extremes views. One extreme point of view takes the position that every outcome is due to the individual. If the test improvement score of one teacher is 10% and the other teacher’s score is 12% the latter is clearly a superior teacher. You cannot argue with the numbers. Of course as we have seen from numerous examples teachers and principals who operate under this system will take extreme measures to have better test scores. When all else fails they change the test answers of their students. They cheat. This has occurred repeatedly and will continue to occur as long as this stupid and naïve view of results pervades our society. And this view destroys it does not help improvement.
The other extreme point of view is that everyone is trying their best, therefore they are all the same. In Profound Knowledge these are known as type I and Type II mistakes.
If teachers are properly trained, motivated and coached it is very likely that all the differences are due to chance. But we do not have to assume this. We have tools that will let us determine if someone’s performance is outside the system so that their performance is not due to chance. They might be really better than the system, or they might be really worse. This kind of analysis of course takes some smarts, or profound understanding. But it will lead to much better results. Rather than destroying half the workforce or more, we should be building up our teachers and workers and allowing them to experience pride in their profession and become constantly better. We should be identifying who are really performing below the system and helping them become better if possible. We should identify those who are truly above the system and finding out what we can learn from them and teach to others.
In the study quoted by Kristof the top teachers were defined as those being in the top 16%. In all probability most who were in the top 16% are there because of chance. They may be better in some sense, but they may not be. But once they get praise that they are good, they have a chance of becoming better. Those who happen to be below average in whatever measure you make will perform worse. In corporations that have merit reviews the quarter after the reviews suffers the worse productivity as everyone, even those who receive positive appraisals are depressed and demotivated.
Using numbers this way, is not just irrational, it is destructive. What we need in the educational and business dialog today is something else: smarts, knowledge, yes, Profound Knowledge
Objective measures of competence versus relative performance or grades
We have to distinguish between excellence and relative position based on measured numbers. Suppose we hire the very best teachers in the world and bring them to a low performing school. Now we rate them against each other and have them compete. The great teacher who happens to have below average test score improvements will get his merit rating and told that he is below average. So what? What possible good can this do? It is possible that he will be demotivated, rightfully annoyed and leave. Will this action of giving this ridiculous feedback do any good? Some will say, yes now they know where they stand. So What? This will not do any good whatsoever. It will make things worse.
Will having these great teachers in this underperforming school improve the test results of that school? It is possible. But it is much more likely that the results will not improve in any measurable way for the school as a whole. Even if there is some improvement much more improvement would occur if these teachers and the administration worked together. If they learned from each other, came up with better ways to support the students and help each other become even better teachers. Even excellent teams are destroyed when internal politics and the need to come out on top undermine teamwork. President Barack Obama recently mentioned in his State of the Union speech that teamwork and not personal glory is what makes Seal Team 6 so effective. But he completely ignores this when he pushes for his Race To The Top program in education.
Much of the results in the underperforming school may be due to factors outside of the school itself. Students in a family who are having financial problems, whose parents are worried about paying the rent or losing their homes will not be able to perform well. Only by recognizing that many things impinge on students learning can we begin to understand how to improve learning and the learning experience.
It is also possible that all the teachers in a system are incompetent in some sense. If teachers are teaching the wrong thing society will stagnate or even worsen. In some societies rote learning is just about all that is taught. In the US we are increasingly moving to teaching to the test, making students capable at taking tests but incapable of thinking critically and independently.
None of this is meant to justify behavior among teachers or administrators that we would all find unacceptable or unethical. Teachers with inappropriate relationships with students, for example should be removed at once. And at the same time administrators who change test scores should be prosecuted for fraud. But our pundits who advocate for a system that will only increase the pressure for unethical behavior such as changing test scores should also be held responsible for their ignorance. Deming said that fear leads to wrong numbers and that is what we are seeing throughout the US educational system. And this fear, the wrong numbers and ignorance of variation prevents us from improving our educational system. As long as ignorance reigns we will not improve.
Sub-Optimization: Imagine a young nation that wishes to get a functioning railroad working as soon as possible. The government hires two very good companies to lay down tracks, one starts on the east coast of the nation while the other starts on the west coast. They both lay down excellent track. But when they meet in the middle they find that the tracks are of different sizes. They cannot complete their mission of creating a rail network that runs from coast to coast.
Your boss asks you to build the best car in the world so you investigate the best parts. You get the best brakes, they come from BMW, the best engine is from Ferrari, the best crash protection is Mercedes and so on. You bring all the parts together and assemble your car and even after banging to get it all assembled it does not even run.
Misuse of numbers or data, and misunderstanding of the nature of a system, even being ignorant of the existence of systems are two major problems afflicting business throughout the world and education especially in the US.
We need Profound Knowledge to be universally understood. If not now—When?
