Part 4: Hypothesis Testing of frequency-based samples
Tags: datascience, python
Published: 2021-02-16 15:59:19

In [**part one of this series**](https://dev.to/sonalake/an-introduction-to-hypothesis-testing-41ne), we introduced the idea of hypothesis testing, along with a full description of the different elements that go into using these tools. It ended with a cheat-sheet to help you choose which test to use based on the kind of data you’re testing.

[**Part two**](https://dev.to/sonalake/part-2-hypothesis-testing-of-proportion-based-samples-1ik2) outlined some code samples for how to perform z-tests on proportion-based samples.

[**Part three**](https://dev.to/sonalake/part-3-hypothesis-testing-of-mean-based-samples-5c16) outlined some code samples for how to perform t-tests on mean-based samples.

This post will now go into more detail for **frequency-based** samples.

[](https://sonalake.com/wp-content/uploads/2019/12/hypothesis-testing.png)

If any of these terms – _Null Hypothesis, Alternative Hypothesis, p-value_ – are new to you, then I’d suggest reviewing the first part of this series before carrying on with this one..

## **What is a frequency-based sample?**

In these cases we’re interested in checking frequencies, e.g. I’m expecting my result set to have a given distribution: does it?

Are differences between the distributions of two samples big enough that we should notice it? Are the distributions between variables in a single sample enough to indicate that the variables might depend on each other?

## **Requirements for the quality of the sample**

For these tests the following sampling rules are required:

<table>

<tbody>

<tr>

<td align="left"><strong>Random</strong></td>

<td align="left">The sample must be a random sample from the entire population</td>

</tr>

<tr>

<td align="left"><strong>Normal</strong></td>

<td align="left">The sample must be normal, for these tests either:
<ul>
<li>The underlying population must be normal – this can be tricky – as a population might normally be normal, only to be non-normal the day you sample it 😉</li>
<li>If you can’t assume the underlying population is normal then you should use a sample size of at least 30 (as per the central limit theorem)</li>

</td>

</tr>

<tr>

<td align="left"><strong>Independent</strong></td>

<td align="left">The sample must be independent – for these tests a good rule of thumb is that the sample size be less than 10% of the total population.</td>

</tr>

</tbody>

</table>


## **Tests for mean-based samples**

All of these code samples are available in [**this git repository**](https://bitbucket.org/sonalake/blog-hypothesis-testing)

### **Chi-squared quality-of-fit**

> **Compare the counts for some variables in a sample to an expected distribution**

In this test we have an expected distribution of data across a category, and we want to check if the sample matches that.

For example, suppose a network was sized to have the expected distribution, and a sample observed the following counts

<table width="100%">

<tbody>

<tr>

<td  align="left"><strong>Class of Service</strong></td>

<td align="left"><strong>Expected Distribution</strong></td>

<td align="left"><strong>Observed Count in sample (size 650)</strong></td>

</tr>

<tr>

<td align="left">A</td>

<td align="left">5%</td>

<td align="left">27</td>

</tr>

<tr>

<td align="left">B</td>

<td align="left">10%</td>

<td align="left">73</td>

</tr>

<tr>

<td align="left">C</td>

<td align="left">15%</td>

<td align="left">82</td>

</tr>

<tr>

<td align="left">D</td>

<td align="left">70%</td>

<td align="left">468</td>

</tr>

</tbody>

</table>

Given a null hypothesis that the distribution is as expected, then the following python code would derive the probability that the sample fits into this expected distribution.

```python
from scipy.stats import chisquare

# can we assume anything from our sample
significance = 0.05

# what do we expect to see in proportions?
expected_proportions = [.05, .1, .15, .7]

# what counts did we see in our sample?
observed_counts = [27, 73, 82, 468]

########################
# how big was our sample
sample_size = sum(observed_counts)

# we derive our comparison counts here for  our expected proportions, based on the sample size
expected_counts = [float(sample_size) * x for x in expected_proportions]

# Get the stat data
(chi_stat, p_value) = chisquare(observed_counts, expected_counts)

# report
print('chi_stat: %0.5f, p_value: %0.5f' % (chi_stat, p_value))

if p_value > significance:
   print("Fail to reject the null hypothesis - we have nothing else to say")
else:
   print("Reject the null hypothesis - suggest the alternative hypothesis is true")

```


### **Chi-squared (homogeneity)**

> **Compare the counts for some variables between two samples**

In this case, the test is similar to the best fit (above) but rather than estimate the expected counts from the expected distribution, the test is comparing two sets of sampled counts to see if their frequencies are different enough to suggest that the underlying populations have different distributions.

This is, in effect, the same code as above – only in this case we have actual expected values to match, rather than having to estimate them from the sample.

```python

from scipy.stats import chisquare

# can we assume anything from our sample
significance = 0.05

# what counts did we see in our samples?
observed_counts_A = [32, 65, 97, 450]
observed_counts_B = [27, 73, 82, 468]

########################

# Get the stat data
(chi_stat, p_value) = chisquare(observed_counts_A, observed_counts_B)

# report
print('chi_stat: %0.5f, p_value: %0.5f' % (chi_stat, p_value))

if p_value > significance:
   print("Fail to reject the null hypothesis - we have nothing else to say")
else:
   print("Reject the null hypothesis - suggest the alternative hypothesis is true")

```

### **Chi-squared (independence)**

> **Check single sample to see if the discrete variables are independent**

In this case you have a sample from a population, over two discrete variables, and you want to tell if these two discrete variables have some kind of relationship – or if they are independent.

**NOTE:** this is for _discrete_ variables (i.e. categories). If you wanted to check if numeric variables are independent you’d want to consider using something like a linear regression.

Suppose we had a pivot to see how people from different area types (town/country) voted for three different political parties.

The question we are asking is whether or not we can say whether or not there is likely to be a connection between these two variables (i.e. do town/country people have a strong preference to vote for a given party).

<table width="100%">

<tbody>

<tr>

<td align="left"></td>

<td style="text-align: left;" align="left"><em>Party</em></td>

<td align="left"></td>

<td align="left"></td>

</tr>

<tr>

<td style="text-align: left;" align="left"></td>

<td align="left"><strong>Cocktail Party</strong></td>

<td align="left"><strong>Garden Party</strong></td>

<td align="left"><strong>Mouse Party</strong></td>

</tr>

<tr>

<td align="left"><em>Voter Type</em></td>

<td align="left"></td>

<td align="left"></td>

<td align="left"></td>

</tr>

<tr>

<td align="left"><strong>Town</strong></td>

<td align="left">200</td>

<td align="left">150</td>

<td align="left">50</td>

</tr>

<tr>

<td align="left"><strong>Country</strong></td>

<td align="left">250</td>

<td align="left">300</td>

<td align="left">50</td>

</tr>

</tbody>

</table>


The python code to check this is:

```python
from scipy.stats import chi2_contingency
import numpy as np

# can we assume anything from our sample
significance = 0.05

pivot = np.array([
  # town votes
  [200,150,50],
  # country votes
  [250,300,50]
])
########################
# Get the stat data
(chi_stat, p_value, degrees_of_freedom, expected) = chi2_contingency(pivot)

# report
print('chi_stat: %0.5f, p_value: %0.5f' % (chi_stat, p_value))

if p_value > significance:
  print("Fail to reject the null hypothesis - we have nothing else to say")
else:
  print("Reject the null hypothesis - suggest the alternative hypothesis is true")
```

## **Where do we go next?**

Thank you for reading the final part of our introduction into hypothesis testing. I hope you found it a useful introduction into the world of statistical analysis. If you would like to look deeper into this field, I’d suggest the following.

*   I’ve not touched on issues of [**power**](https://en.wikipedia.org/wiki/Power_of_a_test) or [**effect size**](https://en.wikipedia.org/wiki/Effect_size) in this series. For that I would direct you to Robert Coe’s always worth reading: [**It’s the effect size, stupid: what effect size is and why it is important**](https://www.leeds.ac.uk/educol/documents/00002182.htm)
*   If you have more complex types of data to examine, then I’d suggest reading more into
    *   [**Analysis Of Variance**](https://en.wikipedia.org/wiki/Analysis_of_variance) – for when you have means in more than two sets of groups to compare, and using multiple t-sets would waste your power.
    *   [**Linear Regression**](https://en.wikipedia.org/wiki/Linear_regression) – for when you want to predict the value of one continuous variable, based on the values of some other continuous value, or just want to see if different continuous variables are, in fact, related.
*   If our previous post – [**Quantitative analysis is as subjective as qualitative analysis**](https://sonalake.com/latest/quantitative-analysis-is-as-subjective-as-qualitative-analysis/) – is making you doubt whether you can trust stats at all, then check out how [**meta analysis**](https://en.wikipedia.org/wiki/Meta-analysis) can be used to collect the results of multiple different analyses, and produce a single overall measure as to whether the underlying tests show a significant interaction.

If you would like to know more or have any suggestions, please don’t hesitate to reach out to us!

* * *

*   **PART I: [An Introduction to Hypothesis Testing](https://dev.to/sonalake/an-introduction-to-hypothesis-testing-41ne)**
*   **PART II: [Hypothesis Testing of proportion-based samples](https://dev.to/sonalake/part-2-hypothesis-testing-of-proportion-based-samples-1ik2)**
*   **PART III: [Hypothesis Testing of mean-based samples](https://dev.to/sonalake/part-3-hypothesis-testing-of-mean-based-samples-5c16)**