Repository: quantopian/zipline
File: zipline/data/fx/hdf5.py (Python)

Code before:

        col_ixs = self.dts.searchsorted(dts, side='right') - 1
        row_ixs = self.currencies.get_indexer(bases)

        try:
            dataset = self._group[DATA][rate][quote][RATES]
        except KeyError:
            raise ValueError(
                "FX rates not available for rate={}, quote_currency={}."
                .format(rate, quote)
            )

        # OPTIMIZATION: Column indices correspond to dates, which must be in
        # sorted order. Rather than reading the entire dataset from h5, we can
        # read just the interval from min_col to max_col inclusive
        #
        # However, we also need to handle two important edge cases:
        #
        #   1. row_ixs contains -1 for any currencies we don't know about.
        #   2. col_ixs contains -1 for dts before the start of self.dts.
        #
        # If either of the above cases obtains, we want to return NaN for the
        # corresponding output locations.

        # We handle (1) by reading raw data into a buffer with one extra
        # column. When we then apply the column index to permute the raw data
        # into the correct order, any columnss with values of -1 will pull from
        # the extra column, which will always contain NaN>
        #
        # We handle (2) by overwriting rows with indices of -1 with NaN as a
        # postprocessing step.
        slice_begin = max(col_ixs[0], 0)
        slice_end = max(col_ixs[-1], 0) + 1  # +1 to be inclusive of end date.

        # Allocate a buffer full of NaNs with one extra column. See
        # OPTIMIZATION notes above.
        buf = np.full(
            (len(self.currencies), slice_end - slice_begin + 1),
            np.nan,
        )

        buf[:, :-1] = dataset[:, slice_begin:slice_end]

        # Permute the rows into place, pulling from the empty NaN locations for
        # column indices of -1.
        out = buf[:, col_ixs - slice_begin][row_ixs]

        # Fill missing rows with NaN. See OPTIMIZATION notes above.
        out[row_ixs == -1] = np.nan

        # Transpose everything to maintain dts as row labels, currencies as col

Review comment (suggestion):
I think this comment got a bit scrambled? `(1)` still refers to the possibility of nonexistent rows, but then we talk about columns here.

Code after:
        check_dts(dts)

        col_ixs = self.dts.searchsorted(dts, side='right') - 1
        row_ixs = self.currencies.get_indexer(bases)

        try:
            dataset = self._group[DATA][rate][quote][RATES]
        except KeyError:
            raise ValueError(
                "FX rates not available for rate={}, quote_currency={}."
                .format(rate, quote)
            )

        # OPTIMIZATION: Column indices correspond to dates, which must be in
        # sorted order. Rather than reading the entire dataset from h5, we can
        # read just the interval from min_col to max_col inclusive
        #
        # However, we also need to handle two important edge cases:
        #
        #   1. row_ixs contains -1 for any currencies we don't know about.
        #   2. col_ixs contains -1 for dts before the start of self.dts.
        #
        # If either of the above cases obtains, we want to return NaN for the
        # corresponding output locations.

        # We handle each of these cases by reading raw data into a buffer with
        # one extra column and one extra row. When we then permute the raw data
        # into the correct order, any row or column indices with values of -1
        # will pull from the extra row/column, which will always contain NaN.

        slice_begin = max(col_ixs[0], 0)
        slice_end = max(col_ixs[-1], 0) + 1  # +1 to be inclusive of end date.

        # Allocate a buffer full of NaNs with one extra column and row. See
        # OPTIMIZATION notes above.
        buf = np.full(
            (len(self.currencies) + 1, slice_end - slice_begin + 1),
            np.nan,
        )

        buf[:-1, :-1] = dataset[:, slice_begin:slice_end]

        # Permute the rows into place, pulling from the empty NaN locations for
        # row and column indices of -1.
        out = buf[:, col_ixs - slice_begin][row_ixs]

        # Transpose everything to maintain dts as row labels, currencies as col
        # labels which is expected everywhere else.
        return out.transpose()