+1, especially to your last sentence. That right there is my biggest medical informatics pet peeve- a lot of hospitals and clinics have "analysts" who are good at using the tools, and maybe even have a good grasp of the data schema of their repositories... but have very limited clinical knowledge, and even more limited knowledge of the workflows that generated their data in the first place.

So, what happens is that they generate some numbers without fully understanding the "story" behind them. For example, they might get a request for data concerning the frequency with which patients with condition "X" are treated at their hospital. In most EHR systems, the way to answer questions like this is by using ICD codes... however, there is rarely a 1:1 relationship between what we might think of as a "diagnosis" and a code. Depending on how it's defined, even something seemingly simple such as "Asthma" might be represented in an EHR by (for example) a dozen different codes, and which codes are used can depend heavily on a wide variety of factors: how the EHR's designers implemented the diagnosis system and user interface, how the clinicians were trained to use the system, specifics of how the patient's symptoms presented themselves, how the billing department coded the clinicians' diagnoses, the phase of the moon, etc. etc. etc. As a result, instead of a simple query ("find all patients with ICD code A"), the query ends up looking like "find all patients with codes A, B, C, D, E .... or J; or code K, but only if it co-occurs with L, M, or N; or code O, if the patient was seen in clinic number 4 or 5 between such-and-such dates; etc. etc. etc." And that's for a simple and straightforward clinical question. Imagine if it was something more complex, like "how many patients with condition X also develop condition Y after having treatment Z".

Coming up with a query like that takes significant clinical knowledge, but, more importantly, it requires intimate knowledge of the organization that created the data in the first place. It also requires some pretty serious "people skills"- the clinicians that the analyst will be working with to formulate the question will know virtually nothing about computers or databases, and so it will fall on the analyst to work with the clinicians to elucidate the implications and edge cases of the original question. It's kind of like being a detective. This, by the way, is a big part of why it's so hard to get good--- as in, reliable, valid, and comparable--- quality measures from large health care organizations. The data's often way more complex and ambiguous than novices realize, and (speaking from personal experience, here) it often takes people who come from non-clinical backgrounds and are used to more straightforward analytical questions quite a while to realize just how far down the rabbit hole they've gone. What might seem like

Of course the fun doesn't stop once our analyst has finally generated some numbers. Whoever wanted the numbers in the first place usually doesn't think much about where they came from (cf. "automation bias"), and as such can go on to make ill-informed decisions as a result of some subtle mistake in the data (i.e., unbeknownst to anybody, the analyst's query missed a whole block of patients coming from a particular clinic, thereby underestimating the prevalence estimates of asthma). This is doubly true when the consumers of the data are generic statisticians (as in, not specialist biostatisticians who are experienced in clinical data analysis) The first commandment of statistics is "Know thy data", and medical data is one of those areas where that's a tricker problem than usual.