semvec/pypi-packages huggingface dataset:
twisted.internet.processes 1.0b1
An implementation of deferToProcess using the multiprocessing package.
License: Apache 2.0
Keywords: twisted multiprocessing deferToProcess

Introduction
============

The Python Global Interpreter Lock (GIL) poses some limitations on multi 
processor performance by locking on all pure Python code. Although the
threading support in twisted is excellent, the GIL was holding back performance
by essentially keeping the process bound to a single CPU.

In order to get around the GIL, Python brought in the multiprocessing package
as of version 2.6. The multiprocessing package has been backported to both
Python 2.4 and 2.5 and is available from the Python Package Index
(http://pypi.python.org/pypi). The multiprocessing package provides a similar
interface to the threading libraries provided by Python except that it uses
processes instead of threads.

This package, twisted.internet.processes, implements a wrapper around the 
multiprocessing library and provides a method, deferToProcess, that works
in the same manner as deferToThread.

Implementation
==============

A process pool is created on the reactor in the same manner that the thread 
pool is created in the normal reactor when deferToThread is first called. 

Initially, the goal was to use the non-blocking code and callbacks from the
multiprocessing library to provide a deferToProcess method. Unfortunately, 
the multiprocessing.Pool.apply_async does not call the callback in the event 
of a failure. In order to retain failure callbacks, the get method from the 
asynchronous result set, multiprocessing.pool.AsyncResult, is used by wrapping
it in a call to deferToThread. The thread is blocked while waiting on the 
results from the call in a separate process.

Using the multiprocessing.Pool.apply_async to defer to a process imposes some 
additional requirements on the user code. All arguments and the function itself 
must be pickleable by the Python cPickle module.


License
=======
Copyright 2009 Texas A&M University
 
Licensed under the Apache License, Version 2.0 (the "License"); you may not use 
this file except in compliance with the License. You may obtain a copy of the 
License at

http://www.apache.org/licenses/LICENSE-2.0
  
Unless required by applicable law or agreed to in writing, software distributed 
under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR 
CONDITIONS OF ANY KIND, either express or implied. See the License for the 
specific language governing permissions and limitations under the License. 

Changelog
=========

1.0b1 2009/08/24
----------------

* Initial release
* Locked to multiprocessing 2.6.1.1 or below due to a bug in logging in 2.6.2.1
* No testing yet
* deferToProcess based on a blocking call wrapped in deferToThread