spoc-trace-generator 0.9.12
Toolkit for creating traces for Statistical Process Control charts.
License: MIT
Keywords: Statistical Process Control Charts,Reliable Change Charts,Outcome Charts

# spoc-trace-generator
Statistical Process Control and Outcomes Charting Library - Trace Generator

# Getting started

    pip install spoc-trace-generator

# Usage
    from tracegen.control.spoc import SpocTraceGenerator
    spc_trace_params = {
        'chart_type': 'p',
        'source_data': source_df,
        'date_col': 'date',
        'datestr_format': '%Y-%m-%d,
        'val_col': 'success_or_failure',
        'time_unit': 'week',
        'y_floor': 0,
        'y_ceiling': 1,
        'rule_set_measure': ['basic'],
        'rule_set_dispersion': None,  # NOTE: No dispersion graphs for p and np charts
        'source_rows_are_single_binom_obs': False,
        'enforce_equal_n': False,
        'reference_series': None,
        'reference_source_col': 'sample_size',
        'reference_instructions': 'sum_source_col_in_group',
        'ref_is_sample_size': True
    }
    spoc = SpocTraceGenerator(**spc_trace_params)
    print(spoc.traces)
    print(spoc.violating_points)

DOCSTRING for tracegen.control.spoc.SpocTraceGenerator
    
    Main class that provides tracegen analysis, detects rule violations, and renders charts.

    Parameters
    ----------
    chart_type : str
        Must be an instance of the class ChartType defined in util.base

    source_data : DataFrame
        The data used to generate all traces. Different chart_types have different data requirements. Please read the
        README carefully to understand the assumptions that must be true to successfully use this package.

    date_col : str
        Column in the `source_data` that contains the date relevant to each row of the DataFrame.

    datestr_format : str
        Date string format that will be used to convert dates in `date_col` to datetime objects.

    val_col : str
        Column in the `source_data` that contains the measurement you are checking for process control.

    rule_set_measure : List[Rule]
        The list of rules that will be used to generate `violating_points` where the process is considered Out of Control
            This will either be used on 1) just the measure, or 2) both the measure and the dispersion charts, where applicable.

    time_unit : Optional[str]
        Which period length will be used to aggregate the `val_col` of `source_data` based on the `date_col`.

    control_limits_start_date : Optional[str]
        Date which complies with `datestr_format` used to determine the start of the window used to calculate
            the control limits for the series of values represented in `val_col`

    control_limits_end_date : Optional[str]
        Date which complies with `datestr_format` used to determine the end of the window used to calculate
            the control limits for the series of values represented in `val_col`

    rule_set_dispersion : Optional[List[Rule]]
        If set, these rules will override the generation of `violating_points` for the dispersion chart in the SPC.

    y_floor : Optional[float]
        Lowest value your measure can take.

    y_ceiling : Optional[float]
        Largest value your measure can take.

    enforce_equal_n : Optional[bool]
        Defaults to True. Referenced in calculation of traces for several chart_types.
        If False may weaken the statistical interpretability of your Rule violation results.

    source_rows_are_single_obs : Optional[bool]
        Default False. If this is True, each row in the `source_data` corresponds to a single observation
        (not already rolled_up to Day, for example).
            NOTE: This parameter is particularly important for The Attribute Control Charts:
            ChartType.P, ChartType.NP, ChartType.C, ChartType.U

    reference_series : Optional[Series], indexed by date at same `time_unit`... dates must match exactly
        Set of reference data to pass to SpocStats. This Series must have only 1 value per index date included,
          or pandas will raise `ValueError: cannot reindex from a duplicate axis`
          Values will only be added to the return df if they have the exact same date in their index.

    reference_source_col : Optional[str]
        Name of the column which contains the measurement to use as a reference

    reference_instructions : Optional[str]
        There may be instances where the data to use for the reference is already contained in the `source_data`,
         such as for NP charts where we'd like to display the total number of events possible as a reference.
         This parameter can be used to specify those special cases.

        Currently implemented options to set reference:
          'sum_source_col_in_group': Use the sample size as the reference (ex: in NP charts)

    ref_is_sample_size : Optional[bool]
        Flag to make explicit that either the `reference_series`, or
         the `reference_source_col` and `reference_instructions` together,
         result in the rollup() function to populate a column rolled_up_df['reference']
         that signifies the sample_size of each row in the rolled_up_df.

         IMPORTANT NOTE: IF any of `reference_series`,
         `reference_source_col`, or `reference_instructions` are set,
                    *** This parameter is Required. ***

## Understanding your data

There are specific data requirements that must be in place in order to interpret statistical process control charts.

These requirements are specific to the type of chart. A helpful study resource is www.sixsigmastudyguide.com.

Guidelines for the data fed into the `SpocTraceGenerator` for each `chart_type` described below.


### What should my data look like? How is it prepped?

Note: The reference_source_col is optional. See DOCSTRING for tracegen.control.spoc.SpocTraceGenerator copied above.

|  Date | Measurement  |  Reference_source_col | 
|---|---|---|
| Jan 1  |  5 |  6 |   
| Jan 2  | 10  | 4 |  
| Jan 8  |  4 |  5 |   
| Jan 15  | 5  | 6 |
| Jan 16  | 3  | 4  | 

<b> How is the data rolled-up? </b>

This depends on the `time_unit` set at which to rollup the data. An example at the weekly
time rollup below.

|  Index: Start of Week | Sum Msrmnt  | Mean Msrmnt | SD Msrmnt  | Median Msrmnt  | Min Msrmnt | Max Msrmnt | Range Msrmnt | Computed Reference | 
|---|---|---|---|---|---|---|---|---|
| Jan 1  |  15 |  7.5 | and  |  |  |  |  | 10: this is | 
| Jan 8  |  4 |  4 | so  |  |  |  |  | 5: based on |
| Jan 14  | 8  | 4 | on  |  |  |  |  | 10: some reference_instructions |

These statistics are used to create the traces for the specific chart type of interest.

If `time_unit` is set to None, the data output by tracegen.util.prepare_data.prep_df()
will not have the columns for "sum", "mean", "sd", etc. The data will look as in the first example, with an index 
composed of the `date_col`, and two columns with the `val_col`, and `reference`. This will be valid
for some charts and not for others. Details on this point are described below.

## Data Considerations for individual chart_types

<i>This library requires that you pass data in long form, and that you group data by time_unit
on the `date_col` passed as a parameter to the SpocTraceGenerator. It does not at this time
allow for creating sample averages based on a unique identifier (ex: batch number), opting instead for
only time based averaging.</i>

### Parameters shared across multiple chart_type prep functions

These parameters are referred to by number in the individual chart_type descriptions below under the heading 
<b> Shared Parameters </b>

1) `y_floor`: a float value which is used as a minimum for all generated traces. Any values in any trace 
lower than y_floor will be rounded up to y_floor.

2) `y_ceiling`: a float value which is used as a maximum for all generated traces. Any values in any trace 
higher than y_ceiling will be rounded down to y_ceiling.

3) `time_unit_is_none`: a boolean value to indicate whether no time aggregation has been provided.

4) `ref_is_sample_size`: a boolean value to indicate whether the reference information provided is to be interpreted as
the sample size for each row.

5) `enforce_equal_n`: a boolean value to indicate whether you wish to enforce restraints that sample
size be the same for each set of observations.

6) `source_rows_are_single_binom_obs`: a boolean value to indicate whether source rows represent single binomial
trials.

7) `source_rows_are_single_poiss_obs`: a boolean value to indicate whether source rows represent single observations
from a poisson distribution.

### Variable Control Charts

`x_bar_r charts:`

You collect a series of samples (2 <= `sample_size` < 10) of a certain quantitative measurement, 
for which you are calculating a mean measurement within that sample.

The XBar chart displays this mean measurement, over time. 

The R chart displays the range of measurements in your subgroup over time.

As of 8/2/21, data must be provided as individual measurements. 

<b> Shared Parameters </b>

1) y_floor                          : No restrictions
2) y_ceiling                        : No restrictions
3) time_unit_is_none         : if True, ValueError
4) ref_is_sample_size               : N/A
5) enforce_equal_n                  : May set to True, and will value error if constant_subgroup_size = False.
6) source_rows_are_single_binom_obs : N/A - data is assumed from skewnormal distribution.
7) source_rows_are_single_poiss_obs : N/A - data is assumed from skewnormal distribution.

<b>[constant vs. non-constant sample size]</b>

There are two possible situations, one where your sample size is constant, and one where you have non-constant sample size.

Traditionally it is required that each subgroup of samples must be the same # of samples (constant `sample_size`), 
otherwise the standard deviation of the distibution of sample means varies and the calculated control limits are not constant.

This requirement can be relaxed, but you will notice that the control limits now must be represented by
arrays of values instead of being represented by a single number. This increases the amount of data that
is passed from the `SpocTraceGenerator`.


`x_bar_s charts:`

You collect a series of samples (10 <= `sample_size`) of a certain quantitative measurement, 
for which you are calculating a mean measurement within that sample. 

The XBar chart displays this mean measurement, over time. 

The SD chart displays the standard deviation of measurements in your subgroup over time.

<b> Shared Parameters </b>

1) y_floor                          : No restrictions
2) y_ceiling                        : No restrictions
3) time_unit_is_none         : if True, ValueError
4) ref_is_sample_size               : N/A
5) enforce_equal_n                  : May set to True, and will value error if constant_subgroup_size = False.
6) source_rows_are_single_binom_obs : N/A - data is assumed from skewnormal distribution.
7) source_rows_are_single_poiss_obs : N/A - data is assumed from skewnormal distribution.

See section <b>[constant vs. non-constant sample size]</b> above.

`x_mr charts`:

Each sample is an individual measurement (no subgroup).

Option 1:

Data can be fed in with `time_unit` = None, and a required `val_col` and optional `reference_source_col`
parameters in order to use no time aggregation method and pass data as-is.

Option 2:

When a `time_unit` parameter is set, there will be a rollup performed on the source data,
but the values will not be averaged. It is good for counts of something per week or month.

The X chart plots the individual values over time.

The MR chart plots the moving range of successive measurements.

<b> Shared Parameters </b>

1) y_floor                          : No restrictions
2) y_ceiling                        : No restrictions
3) time_unit_is_none         : if True, pass through original values. If False, use aggregated values as single observations.
4) ref_is_sample_size               : N/A - sample size is always 1.
5) enforce_equal_n                  : N/A - sample size is always 1.
6) source_rows_are_single_binom_obs : N/A - data is assumed from skewnormal distribution.
7) source_rows_are_single_poiss_obs : N/A - data is assumed from skewnormal distribution.

### Attribute Control Charts
For a review of the effect of process Stability, Number of subgroups, Subgroup size, and Expected variation on the 
reliability of Attribute SPCs:
https://support.minitab.com/en-us/minitab/18/Assistant_Attribute_Control_Charts.pdf

`p charts:`
Percentage of defectives.
Input data can be either 1) a series of binomial trials marked as a success or failure, or 2) a number of successes
in a set of trials. If 2) is the case, a sample size must be provided as reference.

<b> Shared Parameters </b>

1) y_floor                          : must be >= 0
2) y_ceiling                        : must be <= 1
3) time_unit_is_none         : 
       
        if 6) is True:
            3) True: ValueError

4) ref_is_sample_size               : 
          
        if 6) is False:
            4) False: ValueError

5) enforce_equal_n                  : N/A
6) source_rows_are_single_binom_obs : May be True.
7) source_rows_are_single_poiss_obs : N/A - data is assumed from binomial distribution.

`np charts:`
Number of defectives. 
Input data can be either 1) a series of binomial trials marked as a success or failure, or 2) a number of successes
in a set of trials. If 2) is the case, a sample size must be provided as reference.

<b> Shared Parameters </b>

1) y_floor                          : must be >= 0
2) y_ceiling                        : N/A
3) time_unit_is_none         : 
       
        if 6) is True:
            3) True: ValueError

4) ref_is_sample_size               : 
          
        if 6) is False:
            4) False: ValueError

5) enforce_equal_n                  : May set to True, and will value error if constant_subgroup_size = False.
6) source_rows_are_single_binom_obs : May be True.
7) source_rows_are_single_poiss_obs : N/A - data is assumed from binomial distribution.

`u charts:`
Defects per unit.
Input data can be either 1) a number of defects recorded on a single item, or 2) a number of defects recorded
on a set of items. If 2) is the case, a sample size must be provided as reference.

<b> Shared Parameters </b>

1) y_floor                          : must be >= 0
2) y_ceiling                        : N/A
3) time_unit_is_none         : 
       
        if 7) is True:
            3) True: ValueError

4) ref_is_sample_size               : 
          
        if 7) is False:
            4) False: ValueError

5) enforce_equal_n                  : N/A
6) source_rows_are_single_binom_obs : N/A - data is assumed from poisson distribution.
7) source_rows_are_single_poiss_obs : May be True.

`c charts:`
Total number of defects.
Input data can be either 1) a number of defects recorded on a single item, or 2) a number of defects recorded
on a set of items. If 2) is the case, a sample size must be provided as reference.

<b> Shared Parameters </b>

1) y_floor                          : must be >= 0
2) y_ceiling                        : N/A
3) time_unit_is_none         : 
       
        if 7) is True:
            3) True: ValueError

4) ref_is_sample_size               : 
          
        if 7) is False:
            4) False: ValueError

5) enforce_equal_n                  : N/A
6) source_rows_are_single_binom_obs : N/A - data is assumed from poisson distribution.
7) source_rows_are_single_poiss_obs : May be True.

# Parameter definitions for SpocTraceGenerator
## chart_type
### Variable control charts
    Sampled Values (n>1):
      x_bar_r         => XBar and Range Plots
                         (2 plots: XBar and R).
                         Sample size <= 10.
      x_bar_s_x       => XBar and Std Dev Plots
                         (2 plots: XBar and SD)
                         Sample size > 10.
      three_way       => XBar, Moving Range Between, Range Within
                         (3 plots).
                         Sample size <= 10.
                         {NOT IMPLEMENTED: https://www.spcforexcel.com/help/control-chart/variable/between-within/xbar-mr-s}
      ewma            => Exponentially Weighted Moving Average.
                         Sample Size >= 1.
                         {NOT IMPLEMENTED}
      cusum           => Cumulative Sum.
                         Sample Size >= 1.
                         {NOT IMPLEMENTED}

    Individual Values (n=1):
      x_mr            => X and Moving Range Plots
                         Sample size == 1.


### Attributes Control Charts:
    Counts:
      c               => Counts with constant sample size.
      u               => Counts with varying sample size.
    Proportions:
      np              => Proportions with constant sample size >= 1.
      p               => Proportions with varying sample size >= 1.
      time_series     => {NOT IMPLEMENTED}


## time_unit
    Unit of time to use for series creation and x-axis of chart
    Default: day
    Options:
      minute  => NOT SUPPORTED
      hour    => NOT SUPPORTED
      day
      week_s  => week beginning Sunday
      week_m  => week beginning Monday
      month
      quarter
      year

## rule_set_measure / rule_set_dispersion
    Comma delimited list of rules defining aberrant events
    Default: basic
    Possible options (zero or more allowed):
      1_beyond_3sigma           
      2_of_3_beyond_2sigma      
      4_of_5_beyond_1sigma      
      7_on_one_side             
      8_on_one_side             
      9_on_one_side             
      6_trending                
      14_up_down                
      15_below_1sigma           
      8_beyond_1sigma_both_sides
      ge_benchmark {NOT IMPLEMENTED}
      le_benchmark {NOT IMPLEMENTED}
    Additional options (Maximum of one allowed, overrides individual options):
      basic:
          1_beyond_3sigma, 
          7_on_one_side
      pmi:
          1_beyond_3sigma,
          8_on_one_side
      weco:
          1_beyond_3sigma,
          2_of_3_beyond_2sigma, 
          4_of_5_beyond_1sigma, 
          8_on_one_side,
          6_trending, 
          14_up_down
      nelson:
          1_beyond_3sigma,
          9_on_one_side,
          6_trending,
          14_up_down,
          2_of_3_beyond_2sigma,
          4_of_5_beyond_1sigma,
          15_below_1sigma,
          8_beyond_1sigma_both_sides
      all: 
          1_beyond_3sigma,
          2_of_3_beyond_2sigma,
          4_of_5_beyond_1sigma,
          7_on_one_side,
          8_on_one_side,
          6_trending,
          14_up_down,
          15_below_1sigma,
          8_beyond_1sigma_both_sides