dpk-code-quality-transform-python 0.2.0
Code Quality Python Transform
License: Apache-2.0
Requires Python: >=3.10

# Code Quality 

Please see the set of
[transform project conventions](../../../README.md)
for details on general project conventions, transform configuration,
testing and IDE set up.

## Summary
This module captures code specific metrics of input data. The implementation is borrowed from the work done in [CodeParrot](https://huggingface.co/blog/codeparrot) and [StarCoder](https://arxiv.org/abs/2305.06161) projects. In the current implementation, the module includes the following metrics & reports each metrics in individual column:

* line specific metrics include mean & max line length
* character and token ratio - uses the input tokenizer to tokenize the input data & measure the ratio between the characters and tokens
* identifies the high occurrence of the keywords "test " or "config" and tags them as config or test samples
* tags the samples as autogenerated if the sample contains keywords like `auto-generated`, `autogenerated` or `automatically generated`
* programming language specific identification, where:
    * if the input sample is `python` programming language and sample has no reference to constructs like def, class, it is highlighted as `has_no_keywords` 

This module adds the following fields into the output file:
<ul>
       <li>line_mean</li>
       <li>line_max</li>
       <li>total_num_lines</li>
       <li>avg_longest_lines</li>
       <li>alphanum_frac</li>
       <li>char_token_ratio</li>
       <li>autogenerated</li>
       <li>config_or_test</li>
       <li>has_no_keywords</li>
       <li>has_few_assignments</li>
       <li>is_xml</li>
       <li>is_html</li>
</ul>

It uses a tokenizer to collect metrics specific to token ratio.  It is designed to download the tokenizer from the [Huggingface](https://huggingface.co/) if the input tokenizer is not found in the local cache. By default, it uses [codeparrot/codeparrot](https://huggingface.co/codeparrot/codeparrot) tokenizer.

## Running

### Launcher Command Line Options 

The following command line arguments are available in addition to 
the options provided by the [ray launcher](../../../../data-processing-lib/doc/ray-launcher-options.md)
and the [python launcher](../../../../data-processing-lib/doc/python-launcher-options.md).

* "--contents_column_name" - input a column name which contains data to process. The default column name: `contents`
* "--language_column_name" - input a column name which contains programming language details. The default column name: `language`
* "--tokenizer" - input a tokenizer to convert the data into tokens. The default tokenizer is `codeparrot/codeparrot`
* "--hf_token" - input the Hugging Face auth token to download the tokenizer. This option is only required for the tokenizer's whose access is restricted in Hugging Face.

### Running the samples
To run the samples, use the following `make` targets

* `run-cli-sample` - runs src/code_quality_transform_python.py using command line args
* `run-local-sample` - runs src/code_quality_local_python.py

These targets will activate the virtual environment and set up any configuration needed.
Use the `-n` option of `make` to see the detail of what is done to run the sample.

For example, 
```shell
make run-cli-sample
...
```
Then 
```shell
ls output
```
To see results of the transform.