I'm writing this between moments @ my job so it is a bit rambling... I end up talking about similar problems I've worked on, suggest an interface and then in the end address the particular problem and what modifications of convex hull I think could be used to solve the original problem we were discussing including holes. Before I get into all that I want to point at this article on StackOverflow dealing with triangularization of a polygon with holes http://stackoverflow.com/questions/406301/polygon-triangulat... and point out the typical algorithm used in triangularization, http://en.wikipedia.org/wiki/Delaunay_triangulation This paper also looks promising but I haven't read it : http://www.sciencedirect.com/science?_ob=ArticleURL&_udi...

This is relatively similar to the problem I worked on where a dose curve had to be decomposed into rectangles to minimize dose delivery time. A paper on shape rectangularization from my advisor that addresses this is here http://www.springerlink.com/content/t2555274j2k16641/ but I suspect other resources will be more immediately useful. The constraints were more interesting in terms of here is a patient represented as a set of voxels and here is the prescription and tolerance of each so improper decomposition could lead to exceeding tolerance of a critical structure, but I digress...

My understanding of this is that what you would like to have in your system is a data structure representing an arbitrary, complex polygon derived from the intersection of a set of rectangles such that the data structure can be queried with some rectangle and answer with another set of rectangles that indicates that the unpainted portion of the original rectangle. So something like :

PaintHistory{

//returns a polygon or set of rectangles indicating
//regions to be painted
Rectangle[] findUnpaintedSubregion(Rectangle);

//adds the given rectangle to the internal representation
//of the painted area
void paintRectangle(Rectangle);

}

So, you would want to query findUnpaintedRectangle as fast as possible and receive your set of rectangles to pain which could then be added I assume you already have such an interface that you are currently backing with some naive scheme.

I think building such a structure could involve all kinds of clever data structures depending on your exact constraints. If the paint really "blows up" then it doesn't seem approximations are appropriate, but I am reminded of the barnes-hut algorithm (for computing the N-body problem, http://www.cs.berkeley.edu/~demmel/cs267/lecture26/lecture26... ) and its use of the quadtree data structure to divide up the plane http://en.wikipedia.org/wiki/Quadtree , but I am not sure if that is useful in this particular case.

Originally, we were talking about the related problem of decomposing the intersection of a set of rectangles into a set of non-overlapping rectangles that completely cover the area covered by the original set of rectangles and no more. I think solving this problem efficiently (n log n) would suggest a data structure that could give more efficient real time querying.

"...This rules out computing the convex hull and then decomposing, because that would fill in any holes..."

I think all this means is that the convex hull is not sufficient in itself. If you have the points in the convex hull and a set of points representing each hole, then rectangularization should be easy because the convex hull should be rectilinear.

What about this algorithm :

1) Compute the set of points which are intersections of rectangles or their corner points. Here is a fast algorithm for this : http://en.wikipedia.org/wiki/Bentley%E2%80%93Ottmann_algorit...

2) Compute the convex hull of these points to give a containing polygon.

3) Compute the set of all polygons that represent holes in the convex hull polygon. I think this is a well studied problem, but you can imagine a number of naive solutions and then optimize them. For example, any polygon representing such a hole must be composed of corner points or intersection points, which are in the set we already identified. Do a plane sweep in increasing x direction and parse through the source rectangles, discarding points covered by other rectangles and points on the convex hull. This could probably be done as a slight modification of step 1. Definitely a tricky part, but I think a solvable problem...

4) At this point you have a set of points representing the rectilinear polygon that is a convex hull and you have a set of points representing each rectilinear polygonal hole in the convex hull. So, now you just have to decompose this complex polygon into rectangles - I think if you just draw vertical and horizontal lines at each point and compute intersections, that should lead to a rectangularization with minimal cleanup and you are done.

I hope some of that is helpful, certainly not a link to a complete solution but all that comes to the top of my head now....