ssxtd 1.1.0
semi structured xml to dict
License: MIT

# semi structured xml to dict

ssxtd is an xmlreader similar to xmltodict, but supporting semi structured xml, and providing a more flexible environnment.
ssxtd use either of :
  * the native xml package
  * the lxml package
  * the defusedxml package 

Globally, stick to native xml package. It's the faster for most situations.

Note : ssxtd was created to parse very big files, as a result, the default parsing depth is 2 and the parsing functions are generators. 

## Quickstart

  ```
  pip install ssxtd
  ```
  ```
  from ssxtd import parsers
  result = next(parsers.xml_parse(my_file, depth=0))
  ```



## What if ...



### ...i want to parse big files?

use :  
`parsers.xml_iterparse(my_file)`  

note : by default, it will return depth 2 elements.  



### ...my xml file has mixed tag and text?

ssxtd will automatically convert mixed tags and text to a string, keeping the right order.



### ...i want to use defusedxml to secure my app?

use :  
`parsers.dxml_parse(my_file)`  
or  
`parsers.dxml_iterparse(my_file)`  



### ...i want to use lxml?

use :  
`parsers.lxml_parse(my_file)`  
or  
`parsers.lxml_iterparse(my_file)`  

note : it will be slower than xml_parse and xml_iterparse




## Options

### depth

you can adjust the depth level of the returned objects, even when not using iterparse.

note : you can't use depth = 0 when using iterparse

### trim_spaces

will trim spaces for each value found. can be usefull when you have some ugly xml like:
```
<root>
  <text>we have
  some indentation
  problems
  </text>
</root>
```

### del_empty (default to True)

if set to False, will not remove empty tags

### cleanup_namespaces

if set to False, will not remove namespaces

### verbose

if set to True, will show a progression bar \o/

### recover

if set to True, will recover from malformed xml ( cf test_malformed_xml.py)

note : lxml_parse and lxml_iterparse will use the lxml abilities whereas the others will use a BeautifulSoup transformation

### compression

ssxtd can manage ZIP, GZIP, ByteIO, and path to files

for ZIP and GZIP, you must set the parameter "compression" to either "gz" or "zip"
```
parsers.xml_parse(my_file, compression="gz"):
```
you can also set the parameter to "auto", ssxtd will then auto detect the file type from the extension (.xml, .zip, or .gz)

note : atm, in a zip compression mode, only .xml files situated at the root of the zip file will be read

### object_processorr

if you specify the parameter "object_processor=my_function" when calling a parser, your function will be called for each object 

```
see bin/run_exemple.py
```


Allows to do special actions like merging tags directly during the parsing

### value_processor

if you specify the parameter "value_processor=my_function" when calling a parser, your function will be called for each value found 

e.g a simple type conversion :
```
def try_conversion(value):
        try:
            return int(value)
        except (ValueError, TypeError):
            pass
        try:
            return float(value)
        except (ValueError, TypeError):
            pass
        return value
```        



## Requirements

### Python

python >= v3.7.0b1  
due to https://github.com/python/cpython/commit/066df4fd454d6ff9be66e80b2a65995b10af174f  
you CAN use older version of pythons ( i tested up to 3.5) but you won't be able to read zip files  

### Libs

- bs4
- tqdm

## Run tests

install pytest:  
`pip install pytest`  
in the root directory, run :  
`pytest`  
for running a single file, place yourself at the root folder and run :  
`python -c "import ssxtd.tests.test_malformed_xml"`  

## Performances of the parsing functions

time to parse https://ftp.ncbi.nlm.nih.gov/pubmed/baseline/pubmed19n0001.xml.gz  
GZ file size : 19MB  
extracted size : 185MB  
proc : i7-7700HQ  

| Function  | XML file | GZ file | ZIP file (no compression)|
| ------------- | ------------- | ------------- | ------------- |
|xml_parse|32.76501545000065|36.07715339999959|33.419777400000385|
|xml_iterparse|37.56028480000168|42.16279835000023|39.137448499999664|
|lxml_parse|37.464776250000796|38.880011499999455|37.046347550000064|
|lxml_iterparse|47.04024449999997|45.959421049999946|45.05521540000154|
|dxml_parse|41.52063830000043|40.07632935000038|38.88691465000011|
|dxml_iterparse|45.195273199999065|44.895784000000276|44.13825424999959|

for much more details please see ssxtd\benchmarks\results\result.csv