ga_irods is a Django+Celery -> iRODS connector. The idea is that you can use this module to write webapps that call a data grid (iRODS) in a web-scale manner. Every iCommand is a Celery task. All iRODS environments are stored as model instances in a database.
This is all very well and good, but how to you use it? Assuming you know a bit about iRODS, have or know someone who has an account, and are familiar with the icommands clients (these are commands analogous to the unix file system commands, but with an i- prepended), then usage of this app is quite simple. First, add ‘ga_irods’ to your list of INSTALLED_APPS in Django’s settings.py. Then run:
That will add the RodsEnvironment model to the admin tool. Now, assuming you’re an admin on your django installation (you should be if you can run manage.py at all), then you can add RodsEnvironments for each User in the system. I capitalize User, because the owner of every RodsEnvironment must be an actual User in the django.contrib.auth application. Finally, make sure that celeryd is running and that django-celery is installed and listed in INSTALLED_APPS.
Once environments are setup, you can write a webapp that manipulates iRODS. One might attach a RodsEnvironment to a session object, then use this to accomplish things in an example view like so in your views.py:
A set of methods to start, close and manage multiple iRODS client sessions at the same time, using icommands.
ifsck - check if local files/directories are consistent with the associated objects/collections in iRODS.
Bundle file operations. This command allows structured files such as tar files to be uploaded and downloaded to/from iRODS.
A tar file containing many small files can be created with normal unix tar command on the client and then uploaded to the iRODS server as a normal iRODS file. The ‘ibun -x’ command can then be used to extract/untar the uploaded tar file. The extracted subfiles and subdirectories will appeared as normal iRODS files and sub-collections. The ‘ibun -c’ command can be used to tar/bundle an iRODS collection into a tar file.
Note the use of -C option with the tar command which will tar the content of mydir but without including the directory mydir in the paths. The ‘ibun -x’ command extracts the tar file into the mydir collection. The target mydir collection does not have to exist nor be empty. If a subfile already exists in the target collection, the ingestion of this subfile will fail (unless the -f flag is set) but the process will continue.
If a copy of a file to be bundled does not exist on the target resource, a replica will automatically be made on the target resource. Again, if the -D flag is not use, the bundling will be done using tar.
The -b option when used with the -x option, specifies bulk registration which does up to 50 rgistrations at a time to reduce overhead.
does not exist. Currently only one dataType - ‘t’ which specifies a tar file type is supported. If -D is not specified, the default is a tar file type
If the destLocalFile is ‘-‘, the files read from the server will be written to the standard output (stdout). Similar to the UNIX ‘cat’ command, multiple source files can be specified.
The -X option specifies that the restart option is on and the restartFile input specifies a local file that contains the restart info. If the restartFile does not exist, it will be created and used for recording subsequent restart info. If it exists and is not empty, the restart info contained in this file will be used for restarting the operation. Note that the restart operation only works for uploading directories and the path input must be identical to the one that generated the restart file
The -Q option specifies the use of the RBUDP transfer mechanism which uses the UDP protocol for data transfer. The UDP protocol is very efficient if the network is very robust with few packet losses. Two environment variables - rbudpSendRate and rbudpPackSize are used to tune the RBUDP data transfer. rbudpSendRate is used to throttle the send rate in kbits/sec. The default rbudpSendRate is 600,000. rbudpPackSize is used to set the packet size. The dafault rbudpPackSize is 8192. The -V option can be used to show the loss rate of the transfer. If the lost rate is more than a few %, the sendrate should be reduced.
The -T option will renew the socket connection between the client and server after 10 minutes of connection. This gets around the problem of sockets getting timed out by the firewall as reported by some users.
Options are: * -f force - write local files even it they exist already (overwrite them) * -I redirect connection - redirect the connection to connect directly
numThreads - the number of thread to use for the transfer. A value of 0 means no threading. By default (-N option not used) the server decides the number of threads to use.
restartFile - specifies that the restart option is on and the restartFile input specifies a local file that contains the restart info.
callback – a registered Celery task that can be called as a subtask with the entire contents of the file that was gotten (file must fit in memory)
post – a URL to which the results of the iget can be POSTed. File can be larger than available memory.
Store a file into iRODS. If the destination data-object or collection are not provided, the current irods directory and the input file name are used. The -X option specifies that the restart option is on and the restartFile input specifies a local file that contains the restart info. If the restartFile does not exist, it will be created and used for recording subsequent restart info. If it exists and is not empty, the restart info contained in this file will be used for restarting the operation. Note that the restart operation only works for uploading directories and the path input must be identical to the one that generated the restart file
The -I option specifies the redirection of the connection so that it can be connected directly to the resource server. This option can improve the performance of uploading a large number of small (<32 Mbytes) files. This option is only effective if the source is a directory and the -f option is not used
The -b option specifies bulk upload operation which can do up to 50 uploads at a time to reduce overhead. If the -b is specified with the -f option to overwrite existing files, the operation will work only if there is no existing copy at all or if there is an existing copy in the target resource. The operation will fail if there are existing copies but not in the target resource because this type of operation requires a replication operation and bulk replication has not been implemented yet. The bulk option does work for mounted collections which may represent the quickest way to upload a large number of small files.
resource - specifies the resource to store to. This can also be specified in your environment or via a rule set up by the administrator.