Introduction: You get what you measure

Well, the talk today is on, you get what you measure, how you're likely to think, that means if you measure accurately, you have accurate answers. If you don't, we don't. But it means something quite different. It means that the kind of a measurement you make will affect the organization. Now, the reason for us talking to you on this is because many of you will find yourself even installing or okaying the insulation of some further measuring system. And you should start thinking further the average person does. What happens when you put in a measuring system? So the whole talk is really about that. Or if you want. How do you manage? Because you're going out to find out a great deal of information. But it's not now generally collected. Because times are changing. Well, I would go back to the story I told you twice before, I think, adding a story about the people who went fishing with a net. They examined the fish they caught and each [side] was a minimum size fish in the sea. You see, the instrument they used affected what they got. It affected the conclusions they drew. Had they used a different size net, they would have come down to different minimum size, but they still [would come] down to a minimum size. If they used to hook sinker, it might have been somewhat different. The way you'll go about making the measurement will affect what you see and what conclusions you draw. That's the simplest example I can give you of the phenomenon. Now, another example, well-known is a business of recent years of measuring a profit by the bottom line every quarter. But I'm going to look at the profit every quarter and want to maximize it. It's not going to have [any] long range planning and it's not going do well in long run. Optimization is small. Quarter by quarter by quarter does not produce optimization in the large. You must do long term planning as well as short term planning. And these too often conflict. You think about how you're going to measure progress in your organization. Is it by short term goals or long term goals? Now, if everyone starts out already exists where everybody is 95 percent efficient, then there are a lot of things you can do to ruin your rating, but not much that you can do to improve it. Therefore, your strategy in that situation is to be quiet and don't rock the boat. The result is that some years later, as you start looking out for who you can make head, you have a large number of people, most of whom are conservatives and not rock the boat at all. They play things conservatively. Your method of promotion along the way has affected what you can get when you want them [to fight to make a] final choice. It may well be you want a risk taker. Your organization may be going down the pipe slowly and heading for ruin. [And you're rather a risk taker] who either ruins it rapidly or saves the organization. Well, [as a] huge promoter for conservatism you won't have that choice. There isn't one out there to choose from. On the other hand, you could do the opposite. You give everybody a 20 percent rating, then almost everybody wants to set out trying to show they got something [in your brew they're ready] and you'll have a larger risk takers at the top end when you want the company runs smoothly, all you've got a bunch of guys who take risks, and that's not good either. You have to think about how the system which you use for selection and promotion, affects the whole organization. Your rating system affects what can happen.

The things you chose to measure

Douse of a thing you choose to measure, some are easy to measure and some are hard. For example, it's easy to measure your height and your weight. It's hard to measure your morale. It's hard to measure your intelligence. Well, there's a confusion between what is reliably measured and what is relevant. To give a simple illustration: suppose we decide promotion was going to be based upon height. It would be a highly reliable, consistent method of promotion. There is no quarrel you will always get the right guy because you can measure accurately. But height apparently has very little [due ability] to manage at the top. Therefore, an accurate measure may be totally irrelevant, but because it's accurate, it's used a difficult measurement, like morale may not be used because it's hard to measure, but [maybe] the thing you want. And that's a very, very difficult thing. There's a constant tendency to go for something you measure easily rather than go over something which is relevant. Furthermore, they want to reproduce small measure, but that may not be what you truly want. Now, let me give you an example of how things work. I'll take the topic of IQ, which is a generally interesting topic. Let's consider how it was done. [???] made of a bunch of questions. Ask quite a few people the questions, looked at the grades and decided if some of the questions were relevant and correlate [well, the ones that some were not]. So he threw out the ones that did not correlate, he finally came down to a larger number of questions which produced consistency. Then he measured. And now we'll take the score. We run across. Now I'm going to take the [cumulative] amount, how many people got least a score? How many got the score? I'll divide by the total number each time, so I will get a curve. That's one, it will always be rising since I'm counting the cumulative number. Now I want to calibrate the exam. Well, here is the place where 50 percent of people are above and 50 percent are below. If I drop down to 34 units down and 34 above, I'm within one segment - 68 percent. [To segment and so on]. Now, what do I do when you get a score? I grew up here, costs there, and I give you the IQ. Now, you discover, of course, what I've done. IQs are normally distributed. I made it that way. I made it that way[ by my calibration]. So when you are told that the IQs are [normally distributed], you have two questions. Did the guy measure the intelligence? Now, what they want to do is get a measure which was such that for age, for the score divided by the age,[ it remained fairly constant ]for about the first 20 years. So the IQ of a child at six and the IQ of a child at twelve would be the same. You divide by 12, so by six, [they had no other things they want] to accomplish. They want the IQ to be independent of a lot of things. Well, if they got it or not or whether they should have tried that is another question. We are now stuck with IQ designed to have a normal distribution. If you think intelligence is not normally distributed. All right you're entitled to your belief. If you think the IQ tests don't measure intelligence, you're entitled to your belief. They haven't got aproved that it does. The assertion and the use doesn't mean a thing. And it consistently, which a person has the same IQ, is not a proof that you measure what you wanted to measure. [Now, this is terror calculus. A great many things] we do in our society. We have methods of measurement which get the kind of result we want. [00:08:41] Take, for example, why used to teach calculus classes. Expose on the final exam I could ask every question exactly equally difficult. [This grades zero would tend to be like this]. Either you knew what it was or you didn't. Because they were all about same hard. You either got em all or you didn't get them all. On the other hand, if I ask a fair number of easy questions predicted at the beginning, you get the students settle down and I ask a couple very difficult ones. I would expect to get a distribution like this. They all got some. Some got more and a few got the very best. Very few got all - high score.

Distributions

Well, it's evident that if I know the class, I can get any distribution I want. I have no [other classes. We will.] Now, what normally happens, at least to me, is when it comes along to a final while, I have some pretty good ideas. I am really worried about the people are going to flunk or not. Those are the ones I worry about so I ask an exam so I get a very sharp cut there. I'm not too worried. Furthermore, the evidence, if contested, can be shown to other people, I'll say, yeah, he deserved to flunk or she deserved to flunk. So I'm giving you a final exam. I have several goals, one of which is to make sure I've identified the flunkeys. I'd like to identify the best grades, but I also have to investigate and have real good evidence that the flunks are flunks because it's a serious matter to flunk a student. Now, another thing which you should think about is a dynamic range. Suppose I ask you to [score] something. One to nine with five being the average. Often happens, right, this score one to nine with five being the average. Or zero to 10. You've had them? Well, most people will pick fours and sixes. They like that. They didn't like that. Suppose I have the same likes and dislikes you do. But instead of doing four six, I use nines and ones and twos. So if you like a person and you score six and I dislike them, equally as much as you like them, and I score two, the effect is that mine will dominate the rating of the person. I use a dynamic range as a whole new cluster down here. Very common tendency is to cluster down here and not use a dynamic range as a result you can see how most things are measured later on, you have very little effect against the other person who has got a stronger use of the total dynamic range. In fact, you remember from information theory that if I've got a scoring system, I'm trying to communicate information the maximum entropy occurs when all the symbols are used equally. If I regard grading as communication to other people how well these students are doing, I should have all the grades straight across the line pretty uniform. We don't do this. Because flunking is very serious and some other ones. [He] said that predicting graduate school in universities, we give A's and B's only and don't give lower grades as a result. We do a very poor job communicating much information. We should do the other. Now there's another method, which is use a U.S. Naval Academy, among other places and that is you are rated in a class of seven you are five out of seven or you're 17 out of 25. This is a method of avoiding grade inflation, and it's a way that the total dynamic range is used. The fall is mainly the following as not too bad a fall there may be a chance that that five very capable people are in the same class and one of to has to be the dummy. The bottom of five. So being close to five doesn't mean you're dumb, but it may. It has that weakness. On the other hand, it probably is much better in our present methods because in rating people is a surprising thing. It applies in many areas. You can rate people better or worse without assigning a score. And let me tell you about the experience of Bell Laboratory. Bell Labs had at those days two times when they measured the people. One was for promoting [what were rating] and the other was for salary. These were done completely separately.

A group rating themselves always rates themselves high

I was often involved in the rating because, among other things, a group rating themselves always rate themselves high and the company has to get some [bearer's] mean. How good is Joe over in [. town A]. With Mary, who is over in [town B]. How can I compare these two people? Because every group always thinks their people are the best. And since I had a wide range of experience meeting people, working with them, the computers covered a whole range I had frequently been invited to these rating ones as asked to attend. So I saw how it worked. I wanted to get slightly now and say something capac to that, you tend not to like to rate people as a child. [ EMU's young Ph.D. instructor]. I didn't like to rate people. I remember the Bible. He, without sin, cast the first stone or judge [not she'd ] be judged and so on. I didn't like it, but I came to realize it must be done. Everybody cannot be the head of the organization. We don't have organization where everybody the same level very often. Even in entertainment a bunch of skits of an [evening] someone has to be first and someone has to be last. Among lectures, giving a series lectures and meeting some speakers first and some speakers last. And it makes a difference. Therefore, since you're going to have to rate, you might as well study it. [ I began studying the writing while I was involved in writing ones]. And the best [way I know is a guy named Hank McGartland me]. Now, in my opinion, Hank is a genius and I think Hank thought I was pretty good, too, but we didn't agree. Hank was a builder of things. He could put together central offices and all kinds of things he built with his hands. I was a theoretician. I thought I [ went to a differently]. He and I go to lunch in a restaurant and it wouldn't be ten minutes before we're yelling at each other because we didn't agree on anything. Well, we both [terpene] rating so long as you asked us was Joe better than Pete? Hank and I would agree. The moment you said, why Hank and I [ show's throats] because we didn't agree on why. So this rating business, you want to be careful. The subjective judgments may be very good, but the verbal ones may be highly in disagreement. And my belief is that the fact that Hank and I rated Joe above Pete meant that Joe was probably better than Pete. But when you ask why, oh, you've got an argument between Hank and Hamming over what was going on, because we had different standards. But you see, looking at the person as a whole we saw the same capable person. So when I say you make these rating judgments, it's very true that a friend of mine who used to be involved with skiing said they would watch people coming down the slope and rate the skiers, not numerical value, just rate them in order. And again, he said, as long as I didn't say why skater / skier A is better than B, there was no trouble. Just when we got down to saying why there was trouble. So in making rating systems, one thing to do is just ask them to rate one above the other and don't ask for the reasons you'll be somewhat better off when you get down the reasons you're now putting intuitive judgments into words and you're getting in trouble.

The relative judgement is one of the best methods

My experience watching for a long while, I came to believe that the relative judgment is a very good one for people who are reasonably [ mature to business of study, it is ] one the best methods. [Now, there's another thing you want to watch]. The people you [initiataly] attracted are the people you will later have. You attract people by the appearance your feel has weathers. That's you know mathematics has a reputation among students. But let's take the psychologists as I look at students psychology they look to me like they were more mixed up than the average undergraduate. Furthermore, the professors were and what the psychology department did was not to unmake such kids but mix them up further so that psychology [graduates were really big, so people]. They attracted that kind of people. Every field attracts let's take mathematics and computer science both. [In math night to ] cauculus and certainly beginning computer science the essence of the beginning is the ability to cope with a sea of detail. You've got to do all kinds of long differentiations without mistakes in algebra and so on. Or you gotta write a program with no bits wrong. Every instruction right. It's a sea of detail. This is what they see when they come into the field and those who are great, I will call minutia people, people who pay attention to minutia. They are attractive. But at the top there's a bigger picture. And so you have quite a few people in computer science who come out. But there are still minutia people they ever get the bigger picture. Same way in mathematics is a great many mathematicians are technicians and really never learn mathematics. We attract in one class of people and we're stuck with them afterwards. The kind of people you attract originally will be the people who are leaders and next generation. They have to come from people you're now hiring. There is no Walter. So it's a problem. At Bell Labs I became, in a sense, part of public relations. I may have told you this. The Bell Labs let me travel wherever I wanted to practice where I wanted to and the universities knew that they'd get a lecture from Hamming with only paying local costs. Bell Labs will pay the transportation, they would take care of me on the ground, pay the hotel bill and feed me and such other things, and get me back on the airplane if necessary. Or Although if I had to rent a car Bell Labs paid that. Well, I went a lot of places. I saw a lot of things. My role was the following. I went out and gave an enthusiastic lecture. I didn't say Bell Labs was great. I just said the kinds of things we did and made it sound exciting. Three months later, with recruiters came around they didn't have as much trouble persuading people who worked for Bell Labs. In sure, I knew that I was a carrot being dangled in front of students say this is what does happen at Bell Labs. People do do exciting things. And I did. But, of course, most [weren't going to have exciting] lives like I did. You must pay attention to whom you attract at the beginning or at the back end you'll be stuck with those people you attracted the hired. There's nobody else available to you. Once in a while, you go outside the organization to hire, but generally, it's not too good a theme because it ruins internal morale if it's widely known the company or the organization goes outside to get the top. People I call, I hang around who are able they're gonna go someplace else where they succeed, rather somebody out outside running it.

A great problem; who you first attract

So it's a very great problem. Who you first attract. And it's not what actually is true. It's what the perceived feature of your field perception mathematics is it's very hard and it's full of detail. Well, I don't think so. I think mathematics is comparatively easy. All things considered. And it's not detail. It's organizational, whole ideas. It's the organization of bigger bodies of knowledge. I told you in systems engineering, it seemed to me the mathematician played the role of the system as you're the person who kept the whole together as a whole. He played the bigger role, the dominant role, in a sense, not the managerial role, but the conscious of big organizations. Mathematicians, some of them, have that ability to see the whole as a whole. That's very valuable, but that's one of the things. Now, personnel employment, if you send them out to hire people for research, you get yourself in trouble. What do you want in research is people original ideas. But a personalized original ideas is in one place, it's like to have other original ideas and other place, such as how to dress or whether to your hair or such things. Thus personnel doesn't like the looks of some of the best researchers and they would never hire. Bell Labs was forced to do what I told you. I went out and plumeri recruiting and recruiters who went out were not personnel people they were members, senior members of the research department who went out and recruited and could recognize somebody's originality in physics also original in his personal behavior pattern sometimes. If you're going to get well behaved people, you aren't going to get great researchers, eccentricity correlates well [ with features]. You can't have outstanding ability in one area not likely about setting abilities of the directions and setting beliefs of other things. So Bell Labs simply gave up personnel department when the people came in that was hired by research personnel department shuttered and generally there wasn't much they could do but at least one man that's a lot of us want to hire, I didn't but a lot of other did, they got him off on medical grounds. He weighed something like 400 pounds and they thought physical grounds they didn't want to get that heavy a person in and then be medically responsible to a great extent. So he was turned down by personnel because he was just too, too big. I had other reason for not wanting that most my friends wanted him.

Self selection

Now, there's another feature that's very vicious about organization. I told you I was on the board of directors. The board of directors has great power over who is going to be promoted at last stages and also who's gonna be on the board next time. In short, they self select. Now, how do they select? Generally, they will select people like themselves with whom they'll be comfortable or as I told you about my boss, Porter, who behaves the way they behave or they succeeded, rather than hiring people who are going to be good for the future. There hire people like they were when they were succeeding. So there tends to be a great homogeneity at the top level. They hire people with whom they'll be comfortable. Result is, you do have an organization's spirit. It's a uniform one. It's not someone this way, someone this way [so I'm all alone, but it's all pretty much the same]. But it's also not like to be highly innovative. That's true widely. If you do the other one to hire troublemakers at the top, you have a lot of trouble coming to decisions. So there's a problem about this. Now, when I was in college, it was a rule pretty strongly understood that math departments do not hire their own graduates. That is inbreeding is too bad. As I look now around what will I see apparently that's no longer to being observed. Many university hires its own graduates. Now, this extreme's centralization of one type or another has caused troubled universities and several times I know the top management [ Peter President, a couple of deans have stepped] in and economics department and said you're loaded with one school of economics we must have breath. We are doing a hiring until we get a balanced department. It's happened at law, has happened in several departments, psychology too where top management has stepped in a university and forced the hiring of the department to be broad rather than highly concentrated. Well, but if the top is concentrated, there's nobody above to do that. That is the problem. So when you are promoting people, think about the question, am I promoting somebody who I'm just going to be comfortable with? What am I producing too damn much homogeneity in the organization? What am I doing when I promote? How much am I shaping the company for better or worse? How much goes on? What rating system, what evaluation system, shall we use? Those which make us comfortable? Well, yes. You don't want chaos at the top. You don't want a board of directors that are squabbling all the time. You want a compatible one. But you want some independence of opinion. That's most boards of corporations have insiders and outsiders. The ratio changes enormously from company to company. When I was with, I think about three outsiders and about five insiders, something like that, maybe four and five. So we got our there were a fair number of outsiders. That means outside views are brought in and we temper them. But most of all, what we did, was simple. By our presence they had to stop and think how would look to outsiders when we do this, how will that look like a promotion look like two independent board members who are not involved in the company from moment to moment. The mere presence of the outsiders force the insiders to be much more conscious of how their behavior looked. And that is a good thing. So many places you have this, you have this in the government periodically by panels, outside panels are brought in. I've served on innumerable government panels one time or other to make a survey every other year such such a department, something like that. Well, what happens to great extent is this. If the recommendations of the panel are what the organizers and the guys who were running want to do they cite them. If they don't they ignore them. And I met a brilliant one in the [past]. We made a recommendation, but I being a bit of a bastard that I am, I asked, let's look at some of the previous recommendations back for the last ten or fifteen years. Every one of them had the same recommendation we were making. Our major recommendation was exactly the same for the last five pounds, but they had never acted on. That's what often happens when you try to get outsiders to break up the inbreeding that occurs because you tend to promote people like yourself or the people who you are comfortable with. It's a problem. And the outsiders people are the rule and you want to get some strong outsiders to serve and I'm afraid you have to pay them moderately well, although in my case, I would have better pricing for nothing because I had once said to myself sometime earlier before I accepted the appointment [to the host]. I was asked by the president to serve the of board directors. I didn't want to. What the hell? I got my own business to do. I'm a scientist, so I went to legal department and got a decision. You can't do it. Well, sometime passed, I realized my interests were getting narrower, narrower and narrower and some extrapolations suggested that when I finally retired IBS and absolutely nothing, that I'd better do something about broadening my interests. Yep. Hamming you got to accept new opportunities are different. The president calls me up again, says we're now a New York Stock Exchange listed. How about being on the board directors now? I just said it myself. I got it. Accept new oportunities. I said to myself, OK. I went down legal department, got decision I could. And so I was on the board of directors. I got it for a new experience. And I think that most outsiders do this for that reason, although I'm not sure serving on the panels in Washington it's no great trouble. You hop the early morning shuttle flight out of Newark and got down there in time for the meeting and you hop the shuttle when you came home, when it was over with. You learn a lot on the outside role and [I can see you again.] You want to study when you get these chances. How do these things work?

How does promotion actually work?

How does promotion actually work? How does information really go about? And what should you be doing? There are plenty of opportunities to serve outside, for example, school boards and so on. [Every once in a while]. They need some outsiders in the school board. That's an opportunity to look at the question of how to manage things successfully, how to prevent inbreeding too much. But how on the same time to keep a career cohered team working together instead of fighting? It's a very interesting problem. Now, let me tell you another story. There was a method. There is a method, Botha's phase game, a goal. If you give me the game frequency versus gain. There is a way of calculating the minimum amount of phase shift you must have at various frequencies if you've got that gain curve. [Let's go to the Bulls phase game and go]. Well, they had in the early days. Let's get the black one. They had done it by hand. What they had done is fitted this curve by a straight line there, a straight line there, a straight line there, straight line there, straight line there. And then looking up in [ Thomas tables], they get numbers out and[ everything would be fine]. Four voice channels. Well, they hit the problem of a television channel. Enormous bandwidth the curves have looking like that. Just incredible detail. They know perfectly well the arbitrary longer lines on that, two guys doing get different answers. Furthermore, while they could do it for voice channel for television channel, they couldn't. So I got involved in it and I produced Finate Machine Solution. I was talking to one time to the guys on the receiving end and one the shooter guys said to me, it's not that your answers are necessarily better. We could do by hand. He said, but they're consistent. Before, when there was done by hand, we could get no further insight because it was this big random choice of how the individual might have chosen to draw those lines. Now, since you're doing this systematically, we can have a chance of getting down further [, B, B, the service and understanding more subtle effects]. And you see why is true.

Human judgements may not be desirable

You see why human judgments, becaus[ they're variable], may not be desirable in some respects. On the other hand, I'm going to argue that, by and large, I have more faith in human judgments. I haven't measuring instruments to a great extent. Human animal is a complex thing. He's more complicated in a vector. He's probably more complicated in even tensors or matrices. Yet or promotion. They must be ordered in line to who is the best candidate. Or who are the three best candidates? If you're getting promoted to professorship or someone or who are the 10 best, if you're going to some some other level? The whole complicated human being, plus the complicated environment which they're going to operate, must be reduced to a scalar quantity, a pure number. You can tell me it can't be done. I agree with you. But it must be. Somehow you must choose the best or the best three or the best seven or something. You can't do anything else. Now they won't be the same. They're obviously difficult. Well, I can use machine methods or I use intuition. I tend to like intuition. Although I just told you a story why machine methods which are more systematic are sometimes better. Most of the time, human judgments are very good. All these things that are provided to people are mature. They have gotten to a point where they can recognize their prejudices and not overcompensate. Overcompensating, I tell you a good example. In my earliest days of teaching, when a student is flunking a course, you say calculus again [and you you don't like a bastard], what you do is you invite him into your office and say, look, Joe, you really ought to flunk. Could you have known as well? But if you want I'll pass you. You can bet every time he will take the pass. And as he walks out, he always says he's everything I read through that guy. He'll never really know mathematics now because I didn't learn the calculus class. He's dead. So I got my revenge. Well, that, you see, is not something I shall allow myself to do. I did it while I was just a young instructor. After a while, I realized the evil of that. That is not the proper way to behave. You have to know yourself. You have to be very careful. But when you get people who are sophisticated enough to know their prejudices, know other things, then judgments monitored by watching themselves are generally better than measuring instruments of so many questionnaires and so on. But sometimes you have to. In the AT&T running all the bell companies faced a problem since it was a monopoly. How do we run it inefficiently? Answer. We've broken up into a bunch of operating companies and we rate one against the other. And so they put out questionnaires covering[ a large room. I think everybody in the company filled them out.] And we process those things and we got the means. [ It varies]. And we found the morale was [low in such a such a Southern belle who's high in our bell or] something else. [ Then AT&T looked into matter. Why was one high, one low when they got after guy? The low ones. Hey, you're way down. I get better]. They did an enormous amount of internal rating to produce internal competition. This is thing frequently done, but not done as often, perhaps, as we up to. It's not too well. Now [I give you smuggling, which you're wrong]. In the US Navy, as I understand it inspection for this or that is scheduled well, well in advance. And you're inspected for cleanliness or readiness or this or that feature. Everybody knows you're coming. And so the skipper gets prepared for this test and he passes that. He probably gets ready for next one. The next one. Well, when we come to war games, we must use this data. But that data you see does not represent the true readiness of the fleet. By no means represents this one, at that moment, this one at that moment, this one at that moment. But not the total. And I'm objective that ever since I first got into say. Right. Yeah. What does data come from? Oh, that's how you measure. Well, that data doesn't measure readiness to the fleet. The Air Force had a different method, which a captain friend of mine explained to me one time. Their inspections are random, apparently. But as he said, any guy at air base who's got a rare going knows what's in the air. And if a inspection flight comes in and he does knows coming in, he deserves what he gets. But it's true, he hasn't got as long a time to prepair. So random inspection does some good. But you really,[ walbert, that random inspections sent] the grapevine, gets going and passes the velocity of light the knowledge is out to who is going to be inspected. It's remarkable how leakage occurs of that type from secondaries and other things that what is supposed to be kept dead secret is out almost immediately. So it's not too easy to do random inspections, but it's more likely to measure what you want rather than what you [do when you get nice form. The other ones, however, are much more polite. Oh my God, they're much more polite].

Earthquakes

Now, I want to talk about something else. Earthquakes, as you know, are rated on a Richter scale, which is some multiple of the log of the energy released. I think it's about three log. Which means if they differ by one in the Richter scale is something I factor a thousand more energy release. Well, the Richter scale shows you is six points sixes and six point five, six point sevens are rare. There are lots of twos and threes and fours. Richter Scale was convenient for one purpose, but I don't think nature uses Richter scale to decide which distribution of earthquakes is going to have. It's got some other distribution entirely. It doesn't use that kind of units scaling from meters to feet as no difference. But when you scale logarithmically or you do something else like that, you[ vastly] affect what you're going to see. Now in the human being experience has taught us for refined, discrimination's other senses the logarithmic scale is probably the best. You can just take very, very faint sounds, but out of high pit high intensities, you can't detect those small differences. It's roughly a log scale, your sound, sight and brightness, touch and so on. Basically, that way for many purposes is logarithmic scale. But you have to find these out. How to find the natural scale is not an obvious question and the statisticians won't do you a lot of good. Furthermore, I've got a terrible observation. What may be the right scale for making some types of decisions may not be for another. For example, you have a herd of cows, you have three and you get three more. Well, we use additive ones. It was hell, yeah, we got three more cows. You have a thousand cows. We had three more. Well, [added a scale to our cause if we walk out]. But I think you'll agree that if you have three, it had three or you have a thousand and three. The ethics here, there's really not the way to look at the problem. It's more the percentage change in the size of the herd. We often use a arithmetic scale because the Abels combined to herds together, but then we use it for judging other things when we shouldn't. And I suggest you the different scales are needed for different things and frequent Physic you inspect it closely, you will see that in fact the equations which you use the subsidy in are in some sense doing some scaling for you very frequently. Scaling is done for you by that method, but trickily not. Now, I mentioned earlier the fact that people try to optimize things. As top management you will find that lower management is always trying to make themselves look good. They're bending things slightly. They're not lying. They're just bending. The only thing that saves top management is it by and large, lower management are bending this way, this way, this way. So something due to a weak law of large numbers, something near the truth emerges at the top. So you have a basis for making decisions.

If the whole organization is bent one way you’d have trouble finding it out.

If the whole organization is bend one way, you will have trouble finding out how you can do what I did, which is not popular. I read about the legal situation on boards of directors before I started serving. I spent a week, two weeks at Chicago and talk the legal department and business schools and got quite a few copies of court decisions. I started going there one day early. The board meeting or say one day late, for example, when we had high inventory and inventory is a big problem. I say one day late, I'm walking down the hall deliberately and I turn something and go into the crib where supplies are held for inventory while I'm dressed properly. The guy in charge of crib can't really do anything. He knows very well. I better try throw me out, although he does. He's not sure who I am. And I look and I haven't repaired my mind for long while I look around this way. Yeah. That's about twelve million dollars for the goods I guess. OK, game or toy figures. Right. Probably. Another thing I did look a loading platform. Were the machines being shipped actually being shipped? Well, yes, in our case. But I found out something bad in order to meet the shipping date other machines coming down line were sometimes cannibalized for spare parts so you have enough parts to get this one done and get it shipped because the salesman got paid by the quarter. As a result, the next week or two, there were practically no shipments. We were busy trying to undo the cannibalizing we'd done on the machines. When I brought this top banishments attention, there was nothing you could do about it. It was a custom in the organization, highly counterproductive to meet the three months shipping dates. [ Now, when in the beginning of the next quarter something might be shipped and three days later a piece paper comes down]. But at the end of the quarter, the moment that thing leaves a loading dock, the shipping paper goes down. Shipped things like that with all the cannibalization and other things they've done. You will find the organization does not do what you think it's doing. It's got some idiotic habits, some of which you cannot prevent. Some of you can. Well, I measure is what I think is the only method you go out and look for yourself. I may be a theoretician. So on I went out and looked to see what was going on. I look at wether the machines were actually I did now. And a board of directors, you reap quietly show due prudence and I would say any ranking officer required to do so due prudence in any organization. Due prudence doesn't mean you can keep an independent accounting system. But it does mean that you are not a sucker for anything anybody ever tells you that if the Treasurer hands you all these figures, this is what's going on in the books, you make some independent choices. Look around here, there, John. So if you find your what? The sand you can say to judge, I shall do prudence this and this and this is what I did to try to verify those numbers were approximately right. One thing you need to do, you can't believe the underlings rating systems are not always accurate. You get what you measure and what you measure is a strange business. There was another thing that happens. It's the popularity of a form of measurement has nothing to do with reliability or its accuracy. Not at all. [ You need to get done to nothing]. What do you want to know is something a lower level cannot deliver? For example, they may give you what they think is a capability, but the next level will interpret that as a probability. They will change the meaning willfully because what you want cannot be obtained and various people along the way will bend what is meant until it fits what top management is asking for. But of course, the bending didn't improve the numbers at all. It left the same thing. I discussed your life testing, which is almost impossible. Now, for the summary I want to say this, I've told, you enouph, horror stories before you find yourself putting in a rating system, ask yourself what will be the long term global effects? You all know that when a rating system is change, you change your behavior pattern. So you'd rate well. You put a new rating system. What will happen? What kind of people will you attract 10 years later? Who will you have at these positions? What will happen? The rating system determines what your organization is going to be to a great extent. And it is so much[ neglected] that I'm appalled at how little attention [the average person pays go out. We can hustle]. The supreme example you would know is software. We started rating software programmers by a number of lines written. Is there any incentive for them to clean up a program? No. Is there incentive for putting bells and whistles? Yes. As a result, we now have tremendously bloaters software. Over generations we have not disciplined the people to be automatically clean it up and get it down and get the minimal instruction. Therefore, you have these programs which fill megabytes of memory. Well, probably more than half of it is sheer wasted. We adopted a system of measuring [NameA] by lines of code and we encourage of style behavior pattern and now we're stuck with it. And you see it clearly what it is. And so I leave that with you as an example of the dangers of this business. OK, see you tomorrow.