Repository: quantopian/zipline
File: zipline/data/fx/hdf5.py (Python)

Code before:
        # However, we also need to handle two important edge cases:
        #
        #   1. row_ixs contains -1 for any currencies we don't know about.
        #   2. col_ixs contains -1 for dts before the start of self.dts.
        #
        # If either of the above cases obtains, we want to return NaN for the
        # corresponding output locations.

        # We handle (1) by reading raw data into a buffer with one extra
        # column. When we then apply the column index to permute the raw data
        # into the correct order, any columnss with values of -1 will pull from
        # the extra column, which will always contain NaN>
        #
        # We handle (2) by overwriting rows with indices of -1 with NaN as a
        # postprocessing step.
        slice_begin = max(col_ixs[0], 0)
        slice_end = max(col_ixs[-1], 0) + 1  # +1 to be inclusive of end date.

        # Allocate a buffer full of NaNs with one extra column. See
        # OPTIMIZATION notes above.
        buf = np.full(
            (len(self.currencies), slice_end - slice_begin + 1),
            np.nan,
        )

        buf[:, :-1] = dataset[:, slice_begin:slice_end]

        # Permute the rows into place, pulling from the empty NaN locations for
        # column indices of -1.
        out = buf[:, col_ixs - slice_begin][row_ixs]

        # Fill missing rows with NaN. See OPTIMIZATION notes above.
        out[row_ixs == -1] = np.nan

        # Transpose everything to maintain dts as row labels, currencies as col
        # labels which is expected everywhere else.
        return out.transpose()


class HDF5FXRateWriter(object):
    """Writer class for HDF5 files consumed by HDF5FXRateReader.
    """
    def __init__(self, group, date_chunk_size):
        self._group = group
        self._date_chunk_size = date_chunk_size

    def write(self, dts, currencies, data):
        """Write data to the file.

        Parameters
        ----------

Review comment (suggestion):
If we're not using `read_direct` anymore, we might as well allocate both an extra row and an extra column and use the extra row/column trick for handling both cases rather than using it for just one of them.

Code after:
        #
        # However, we also need to handle two important edge cases:
        #
        #   1. row_ixs contains -1 for any currencies we don't know about.
        #   2. col_ixs contains -1 for dts before the start of self.dts.
        #
        # If either of the above cases obtains, we want to return NaN for the
        # corresponding output locations.

        # We handle each of these cases by reading raw data into a buffer with
        # one extra column and one extra row. When we then permute the raw data
        # into the correct order, any row or column indices with values of -1
        # will pull from the extra row/column, which will always contain NaN.

        slice_begin = max(col_ixs[0], 0)
        slice_end = max(col_ixs[-1], 0) + 1  # +1 to be inclusive of end date.

        # Allocate a buffer full of NaNs with one extra column and row. See
        # OPTIMIZATION notes above.
        buf = np.full(
            (len(self.currencies) + 1, slice_end - slice_begin + 1),
            np.nan,
        )

        buf[:-1, :-1] = dataset[:, slice_begin:slice_end]

        # Permute the rows into place, pulling from the empty NaN locations for
        # row and column indices of -1.
        out = buf[:, col_ixs - slice_begin][row_ixs]

        # Transpose everything to maintain dts as row labels, currencies as col
        # labels which is expected everywhere else.
        return out.transpose()


class HDF5FXRateWriter(object):
    """Writer class for HDF5 files consumed by HDF5FXRateReader.
    """
    def __init__(self, group, date_chunk_size=HDF5_FX_DEFAULT_CHUNK_SIZE):
        self._group = group
        self._date_chunk_size = date_chunk_size

    def write(self, dts, currencies, data):
        """Write data to the file.

        Parameters
        ----------
        dts : pd.DatetimeIndex
            Index of row labels for rates to be written.
        currencies : np.array[object]
            Array of column labels for rates to be written.