pyconquest 0.1.8
Python code which partly mimics the conquest pacs
Requires Python: >=3.6

# PyConquest

## Introduction

Python code to partly mimic the functionality of the Conquest Pacs system ( http://www.natura-ingenium.nl/dicom.html ).
No (source) code of Conquest was used to write this python code. Program was optimised to be partly compatible (files and database), so queries etc. can 
be re-used.

## Description:
Class is used to organise and index sets of dicom files. The dicom files are stored in
a directory (default name : data), with the files of each patient stored in a 
subdirectory with the patientid as the name. The class will index all files in the 
directory and store information (tag information of the dicom files) 
in a sqlite database (default name : conquest.db).
Which tags are stored in the database can be defined by the dicom.sql file, this 
file is optional, a standard set is used when not present.

Standard file/directory layout is :
```
conquest.db                       database file
data                              data_directory
    1                             subdirectory with name of PatientID
      file1_of_pat1.dcm           files of patient1
      file2_of_pat1.dcm
      ..
      ..
    2                             subdirectory with name of PatientID
      file1_of_pat2               files of patient2
      file2_of_pat2
      ..
      ..
[dicom.sql]                       optional definition of the database
```

### Implemented features :
- indexer properties : read a dicom tree and build a sqlite database from this which conforms to the Conquest standard.
- use the conquest style dicom.sql file to define the columns
- add some special columns,with for instance the roinames in a RTSTRUCT file
- provide low level procedures to write and read to the database and helper files to create queries
- provide basic SCU and SCP functionality
- basic printing of data properties

### What it is not and when you should use Conquest
- no attempt made to mimic full dicom functionality (this is no PACS)
- lua scripting
- image visualisation

# Examples

Note : use the argument *loglevel='INFO'* when creating the instance to get logging output
### (Re)create database and rebuild database

```
from pyconquest import pyconquest

c=pyconquest(loglevel='INFO')
c.create_standard_dicom_tables()
nr_files=c.rebuild_database_from_dicom()
# only rebuild a single MRN/patient
c.rebuild_database_from_dicom(mrn='1234567')
# build for every MRN, even if the MRN already exists in Database
c.rebuild_database_from_dicom(compute_only_missing=False)
```
### Basic database summary
```
from pyconquest import pyconquest

c=pyconquest()
c.dicom_series_summary()
```

## Extracting extra data from the dicomfiles
### adds data concerning images to the dicomseries table
```
from pyconquest import pyconquest

c=pyconquest()
c.analyse_images()

```
### Saving database information to a csv file
```
from pyconquest import pyconquest

c=pyconquest()

# by table(s) or views, specification of filenames is optional
c.dump_data_to_csv(table='dicomstudies')
c.dump_data_to_csv(table=['dicomseries','dicompatients'],
                   filename_dict={'dicomseries': 'dicomseries_filename', 'dicompatients': 'dicompatients_filename'} )
# by query, since 0.1.8 you can define the delimiter (default is ,)
c.dump_data_to_csv(query='select * from dicomstudies',filename='query_filename.csv', delimiter=';')
```

### More advanced querying and extraction from the database:
```
from pyconquest import pyconquest

c=pyconquest()

# extract python list type of selected seriesuids 
query='select * from dicomseries where modality='CT'
list_of_seriesuid=c.execute_db_query(query,return_list_from_col='SeriesInst')
for seriesuid in list_of_seriesuid:
  print(seriesuid)
  
In dicomimages.ElementList information is saved in 'python like' nested lists as a string. 
You can use the following code to turn the string ( in Python ) into a real list:

beam_info =eval(' '.join([("'"+part+"'" if part not in ['[[', ']]', ',', ' ', '', ']', '[', '],'] else part)
                      for part in str(**ElementListString**).split(' ')]))
```
### Using non standard names and directories
```
from pyconquest import pyconquest

c=pyconquest(
    data_directory='data2',
    sql_inifile_name='dicom2.sql',
    database_filename='test.db',
    connect_and_read_sql=True)
    
nr_files=c.rebuild_database_from_dicom()   
c.dicom_series_summary()
```


## Adding and deleting files to the database
### Adding a single file specified by name (optionally remove original)
```
from pyconquest import pyconquest

c=pyconquest()
c.store_dicom_file('1.2.840.113704.1.111....931a31.dcm')
c.store_dicom_file('1.2.840.113704.1.111....931a31.dcm', remove_after_store=True)
```
### Enter all dicom files in a directory into the database
````
from pyconquest import pyconquest

c=pyconquest()
c.store_dicom_files_from_directory('input_directory_name')
c.store_dicom_files_from_directory('input_directory_name', remove_after_store=True)

# using below, the filename is made equal to the sopinstanceuid :
c.store_dicom_files_from_directory('input_directory_name', sopinstance_as_filename=True)

# using the unzip parameter you can define to unpack zip files or not (default True)
# you need to define an existing (empty) directory for temporary unpacking (default='./unpack')

c.store_dicom_files_from_directory('input_directory_name', unzip=True, unzipdir='./unpack)
````

### Deleting series or list of series from the database (optionally 'real' deletion of the file)
````
c.delete_series(seriesuid='2.16.840.1.11....152.20190524082318.537836')
c.delete_series(seriesuid=['2.16..7836','1.2.666...55'])
#or by query
c.delete_series(query='select * from dicomseries where modality="CT"')

# Really physically remove the files (otherwise only the database entries are removed)
c.delete_series('2.16.840.1.11....152.20190524082318.537836', delete_files=True)

# to remove all objects and database info from a patient :
c.delete_series(query="select SeriesInst from dicomseries where seriespat='{}'".format(patientid), delete_files=True)

````

## Sending and receiving files via dicom connectivity

### Send dicom files via dicom protocol, by patientid, seriesuid or query

```
from pyconquest import pyconquest

c=pyconquest()
c.connect_db()
c.send_dicom(addres='127.0.0.1', port=5678, patientid='1234567', ae_title=b'destinationAE',  sending_ae_title=b'MY_AE_TITLE')
c.send_dicom(seriesuid='2.16.840.1.113669.2.931128.880152.20190524082318.537836')

RTPLAN_query="select seriesinst from dicomseries where modality=\'RTPLAN\'"
c.send_dicom(query=RTPLAN_query)
```

### Start receiver (received files are stored in data directory and database is updated )

```
from pyconquest import pyconquest

c=pyconquest(data_directory='data2')
c.start_dicom_listener(port=5678)
# Below does not update the database which saves time
c.start_dicom_listener(port=104, write_to_database=False)
```

### DICOM query to a dicom node via C-FIND

```
from pyconquest import pyconquest

c=pyconquest(data_directory='data2')
response_list = c.query_dicom(patientid='1313131313', addres='10.11.12.133', port=5678, modality='RTPLAN')
print(response_list) #this is a list of dicts with various predefined tags included in the dict
```

### Get a dicom file from a dicom node using  C-MOVE

What this routine does is it 'spins up' a receiver process and then instructs the dicom node via a C-MOVE 
instruction to send data back to this receiver / the caller

Note1 : The transfer is slow, so preferably only use on single files ( not CT/MR )\
Note2 : The SCP should know the destination (to return the data to).\
The default destination when sending back to the 'caller' should be : (PYCONQUEST,[ip of current machine],5699)\
Note3 : You can set the AE title of the requesting node with parameter : requesting_ae_title

```
from pyconquest import pyconquest

c=pyconquest(data_directory='data2')
response_list = c.query_dicom(patientid='1313131313', addres='10.11.12.133', port=5678, modality='RTPLAN')
for response in response_list:
    c.get_dicom(addres='10.11.12.133', port=5678, series_uid=response['SeriesInstanceUID'])
```
#### Variations:

You can change the receiving portnumber by using : receiving_port=12345 in case of port conflicts\
You can also send to another destination than 'yourself' by defining receiving_ae_title='PACS_C'

```
# use another portnumber than 5699 in case of port conflicts
c.get_dicom(addres='10.11.12.133', port=5678, receiving_port=12345, series_uid=response['SeriesInstanceUID'])

# Let A (yourself) make B send data to C using C-MOVE
c.get_dicom(addres='10.11.12.133', port=5678, receiving_aetitle='PACS_C', series_uid=response['SeriesInstanceUID'])
# here 10.11.12.133/5678 = PACS_B and it sends to PACS_C. PACS_B should know the ip/port of PACS_C for this to work
```
## Copying files of a series on the disk to another directory
#### by seriesuid or list of seriesuids
```
from pyconquest import pyconquest

c.copy_dicom_files_to_dest(seriesuid='1.2...8024597.12',destination='out', UseSubDirectories=True)
c.copy_dicom_files_to_dest(seriesuid=['1.2.84..97.12','2.16.840....33'],destination='data2')
```

#### Or by query (query should return seriesinst)
```
from pyconquest import pyconquest

c=pyconquest()
CTquery="select seriesinst from dicomseries where modality=\'CT\'"
c.copy_dicom_files_to_dest(query=CTquery, destination='CTdata',UseSubDirectories=True)
```

## Changing tags of dicom files
Simple functionality to change a single tag of a dicom file. For
each change the file is opened and saved, so not suitable for many
changes per dicom file. 

**NOTE** : The tags are changed in the dicom file only, not in the database. 
So if needed, a rebuild database needs to be done to synchronise the database.
```
from pyconquest import pyconquest

c=pyconquest()
# single tag of single file
c.change_tag('test.dcm', tag='PatientBirthDate', new_value='19000101')

# all tags from a list of filenames
c.change_all_tags(filename=['test.dcm', 'test.dcm'], new_value='19000102')

# using a helper function, change by series, study or patientid
lst = c.get_list_of_filenames(seriesuid='2.16.840.1.113669.2.931128.880152.20190524082317.937233'))
c.change_all_tags(filename=lst, tag='PatientBirthDate', new_value='19000101')

```
## Deleting and changing ROI names in a RTSTRUCT file

It is possible to change or delete rois in a RTSTRUCT file using either of 3 modes:
- delete : delete the specific names 
- leave : leave only the given names and delete all others
- change : change the given names to alternatives

There are flags to actualy write the file or not.
Using **delete_points** you can indicate whether deletion of points is allowed

you can use **get_list_of_filenames()** to apply to multiple files.

**NOTE** : The tags are changed in the dicom file only, not in the database. 
So if needed, a rebuild database needs to be done to synchronise the database.
```
from pyconquest import pyconquest

c=pyconquest()
c.modify_rtstruct(filename='tst.dcm',mode='change',roiname=['Lung_R','Lung_L'],newname=['LR','LL'],write_file=True)
c.modify_rtstruct(filename='tst.dcm',mode='delete',roiname=['Lung_R','Lung_L'],write_file=True)
c.modify_rtstruct(filename='tst.dcm',mode='leave',roiname=['Lung_R','Lung_L'],write_file=True)
# standard mode is to NOT delete points, you can overrule
# and filename can be a list of filenames
c.modify_rtstruct(filename=['tst.dcm','tst2.dcm'],mode='leave',roiname=['Lung_R','Lung_L'],write_file=True,delete_points=True)
```

## Load data from a .csv file into a database table

A routine to load a csv file as a single table into the sqlite database

- First line of the csv file should hold the column names
- A database table is created if non-existing
- Column names can be converted before becoming database column names by adding the key_translation parameter
```
from pyconquest import pyconquest

c=pyconquest()
c.load_csv_to_table('Mosaiq.csv', 'Mosaiq',delimiter=';')

# below with conversion of column names
c.load_csv_to_table('Mosaiq.csv', 'Mosaiq', key_translation={'aantal fracties':'aantal_fracties','A':'B'})

```
## Definition of the database structure
The database can be defined using various ways
- If a file dicom.sql exist that file is used
- If an alternative file is specified on creation of the instance, that file is used
    - use sql_infilename='filename.sql' when creating 
- If no file is found, a hardcoded definition is taken
    - You can add columns to this definition as illustrated below:
    
````
from pyconquest import pyconquest

c=pyconquest(sql_inifile_name='dicom.sql',loglevel='INFO')
c.add_column_to_database(tablename='DICOMpatients',column_definition=['0x0020', '0x000d', 'StudyInst'])
c.create_standard_dicom_tables()
````
since version 0.1.6 it is possible to load nested tags, for instance the beamdose of the first two beams of a RTPLAN:
````
from pyconquest import pyconquest

c=pyconquest(sql_inifile_name='dicom.sql',loglevel='INFO')
c.add_column_to_database(tablename='DICOMimages',
    column_definition=['0x300a','0x0070','BeamDose0', 0, '0x300c', '0x0004',0,'0x300a','0x0084'])
c.add_column_to_database(tablename='DICOMimages',
    column_definition=['0x300a','0x0070','BeamDose1', 0, '0x300c', '0x0004',1,'0x300a','0x0084'])
c.create_standard_dicom_tables()
````

the .sql files are compatible with the original Conquest file format

### To install and upload to PyPi:
```
python setup.py sdist
twine upload dist/*
or:
twine upload --config-file ./.pypirc dist/pyconquest-0.1.5.tar.gz
or 
twine upload --config-file D:.pypirc dist/pyconquest-0.1.5.tar.gz
wherre .pypirc looks like: (where you should include pypi- with the password)

[pypi]
username = __token__
password = pypi-<a very long token you can get from the pypi page under account settings>

```

# CHANGELOG

### version 0.1.8

- Added (optional) delimeter parameter to dump_data_to_csv
- method : store_dicom_files_from_directory can now handle zip files, unpacks it and reads dicom in
- method : store_dicom_file/files can now force the patientid to a value using force_patientid option, the change is 
recorded in a private tag ['0x0011', '0x0020', 'OLDMRN'] that should be added to the database schema..
- method : start_dicom_listener now has an option 'blocking' to enable / disable blocking ( default True )
- when storing dicom files from directory the LAST seriesuid is stored in self.last_seriesuid_loaded property of the class
- Added the isococenter position of the first beam of an RTPLAN to the ElementList item of Dicom Images
- When Reading REG objects, the CBCT and CT seriesuid's are stored in the database, **dicomimages.ReferencedSeriesUID** refers to the CT .
**dicomimages.Elementlist** refers to the seriesuid of the CBCT (all for REG objects from the Elekta XVI system.)

### version 0.1.7

- in get_dicom() transfer is skipped (default) if object is already in database
- improved logging information when sending C_MOVE in get_dicom()
- get_dicom() now returns the error or NONE if no error is encountered
- fixed bug where single quotes in contournames lead to insertion error, quote is now removed

### version 0.1.6

- Fixed bug where when receiving data via dicom, the added columns were not filled
- Added option to read nested tags into the database
- Added option to define the AE title of the requesting node in get_dicom()
- load_csv_to_table() can now handle quotes in the data field
- Added method : read_single_tag() to read a single tag from a single file (returns string or a pydicom structure 
in case of a sequence)

### version 0.1.5

- For RTPLAN, extra beaminfo is saved to the column DicomImages.ElementList.  Full data is now:  
[BeamName, BeamType, RadType, #Wedges, #CP, Energy CP[0], Angle CP[0], BeamDose, BeamMU]
- Added method : query_dicom() where a C_FIND query can be done of a dicom node. The result is a list of dicts of predefined tags
- Added method : get_dicom() where a dicom object can be retrieved via C_MOVE protocol
- Added method : load_csv_to_table() to read a csv file direcly to a SQLite table

### version 0.1.4
- For RTPLAN, beam information is saved in the column DicomImages.ElementList

### version 0.1.3
- Added functionality **change_tag()** to change tags of DICOM files
- Added functionality **modify_rtstruct()** to change and delete roi's from files

### version 0.1.2
- Added method **analyse_images** to analyse CT/MR/PET images and add data to DicomSeries table. Data added is number of slices, min/max slicepositions and a check for slice consistency is done.
- Added postprocessing routine **database_postprocessing()** of the database to enrich the database with entries that can only be added when the database is
complete. For instance the Referenced seriesuid in DICOMseries table for RTDOSE. Routine is automatically
executed at the end of **rebuild_database_from_dicom()** 
- Added option called **sopinstance_as_filename** to **store_dicom_files_from_directory()** and **store_dicom_file()** to change the actual filename in the database from the original (default)
to  sopinstanceUID.dcm

### version 0.1.1 (skipped to 0.1.1 because some systems take 0.0.51 as default)
- Added option **dump_data_to_csv()** to dump data from the database to csv files
- Logging of errors to a (rotating) log file : "pyconquest_error.log"
- Added option **write_to_database** to dicom_listener, if FALSE, sqlite db is not updated (for speed)
- For RTDOSE the referenced SOP UID of the RTPLAN is saved to DicomImages table
- For RTPLAN the referenced SOP UID of the RTSTUCT is saved to DicomImages table
### version 0.0.7
- added option not to check for double entries when rebuilding the database (for speed)
- added fast option to delete a patient from the database
- database indices are now created when creating a new database
- made index tables non unique to improve robustness

### version 0.0.6
- **Rebuild_database_from_dicom()** now has option to rebuild only missing directories
- When closing the database, the time elapsed is printed
- Referenced seriesuid is now extracted from RTSTRUCT and placed in dicomseries and dicomimages table
- **Dicom_series_summary()** now prints result directly to stdout, so no need for pandas to make a readable table
- The view **v_series** now only combines series and study table (previous version was too slow due to complexity)
- New view : **v_seriesRT** now combines series,study and image tables for only RT dicom objects (so not for images)
### version 0.0.5
- a view (**v_series**) is added to the sqlite database that joins study,series and image table
- delete_series can now handle query to define the series to delete
- series argument of delete_series changed from positional to named
- added option to rebuild database for a single mrn  (mrn='1234567')
- added option sending_ae_title to send_dicom() and send_dicom_file()
- bugfix in name of file in store_dicom (wrong path was used)
- execute_db_query now has an option to return (only) a list of a single column

### version 0.0.4
- Added function **insert_dict** to class to easily insert your own data in the database. Also takes care of creating the
table when the first dict is inserted
- If a database has no tables, tables are created automatically on db opening (so on instance creation)
- In **create_buildquery**, now columns can be defined with formats deviating from default and the default can be defined
- Number of  fractions and number of beams saved from RTPLAN to database columns 
- For RTDOSE, RTPLAN and RTSTRUCT file hashes are calculated and saved in dicomimage table
- Added **delete_series** function to delete the series from database and optionally delete the .dcm files
- Added **filter_roinames()** and **set_roi_filter()** methods to facilitate filtering of roi name lists for child classes
### version 0.0.3
- Improved documentation and docstrings
- uniformity in calling send and copy routines