Spaces:
Runtime error
Runtime error
| Adding Processors | |
| ################################################ | |
| This is a tutorial on adding new processors using ``lavis.processors`` module. | |
| The LAVIS library includes a standard processor module that preprocesses data e.g. image transformation and sequence concatenation. | |
| The ``lavis.processors`` module is designed such that any processors can be added, specifically to the requirements of corresponding models of interest. | |
| In this tutorial, we will replicate the steps to add visual and textual processors specifically for `video-grounded dialogue tasks <https://arxiv.org/pdf/1901.09107.pdf>`_. | |
| In addition, we also want the processors to have processing features to make the data samples compatible with GPT-style models. | |
| Base Processor ``lavis.processors.base_processors`` | |
| ***************************************************** | |
| Note that any new processor definition should inherit the base processor class ``BaseProcessor``: | |
| .. code-block:: python | |
| from omegaconf import OmegaConf | |
| class BaseProcessor: | |
| def __init__(self): | |
| self.transform = lambda x: x | |
| return | |
| def __call__(self, item): | |
| return self.transform(item) | |
| @classmethod | |
| def from_config(cls, cfg=None): | |
| return cls() | |
| def build(self, **kwargs): | |
| cfg = OmegaConf.create(kwargs) | |
| return self.from_config(cfg) | |
| This allows us to standardize operations of processors across all processor classes while still allowing customization of processors specifically to data and model types. | |
| We encourage users not to modify the implementation of the base processor class as this will have an impact on all existing processor subclasses. | |
| GPT-style Processors ``lavis.processors.gpt_processors`` | |
| ************************************************************** | |
| In this step, we can define new processor classes, e.g. under ``lavis.processors.gpt_processors``, for GPT models designed specifically for video-grounded dialogues. | |
| First, we want to process video features by defining ``GPTVideoFeatureProcessor`` class. | |
| In this tutorial, we assume video features are extracted beforehand and this processor simply loads the features from ``npy`` files. | |
| Other methods that are specifically defined are ``padding`` (which is used by dataset instances to pad multiple video samples) and ``get_attention_mask`` (which creates an attention mask for Transformer attention in GPT models). | |
| .. code-block:: python | |
| SPECIAL_TOKENS_DICT = {'bos_token': "<bos>", 'eos_token': "<eos>", 'additional_special_tokens': ["<speaker1>", "<speaker2>", "<video>", "<cap>"], 'pad_token': "<pad>"} | |
| ... | |
| @registry.register_processor("gpt_video_ft") | |
| class GPTVideoFeatureProcessor(BaseProcessor): | |
| def __init__(self, visual_ft, audio_ft): | |
| self.visual_ft = visual_ft | |
| self.audio_ft = audio_ft | |
| self.tokenizer = GPT2Tokenizer.from_pretrained('gpt2') | |
| self.tokenizer.add_special_tokens(SPECIAL_TOKENS_DICT) | |
| def padding(self, seq): | |
| padded_seq = torch.nn.utils.rnn.pad_sequence(seq, batch_first=True, padding_value=1.0) | |
| return padded_seq | |
| def get_attention_mask(self, seq): | |
| return torch.sum(seq != 1, dim=2) != 0 | |
| def __call__(self, ft_root, vname): | |
| all_ft = [] | |
| for ft_name in self.visual_ft: | |
| ft_path = os.path.join(ft_root, ft_name, vname) | |
| all_ft.append(np.load(ft_path + '.npy')) | |
| for ft_name in self.audio_ft: | |
| ft_path = os.path.join(ft_root, ft_name, vname) | |
| all_ft.append(np.load(ft_path + '.npy')) | |
| min_len = min([len(ft) for ft in all_ft]) | |
| sampled_ft = [ft[:min_len] for ft in all_ft] | |
| sampled_ft = np.concatenate(sampled_ft, axis=1) | |
| item = {} | |
| item['video_fts'] = torch.Tensor(sampled_ft) | |
| video_type_token = self.tokenizer.convert_tokens_to_ids('<video>') | |
| item['token_type_ids'] = torch.Tensor([video_type_token] * len(sampled_ft)).long() | |
| return item | |
| @classmethod | |
| def from_config(cls, cfg=None): | |
| if cfg is None: | |
| cfg = OmegaConf.create() | |
| visual_ft = cfg.get("visual_ft", ["i3d_rgb"]) | |
| audio_ft = cfg.get("audio_ft", ["vggish"]) | |
| return cls( | |
| visual_ft=visual_ft, | |
| audio_ft=audio_ft | |
| ) | |
| Another processor class that will be useful to have is to process dialogue data. Here we can define a ``GPTDialogueProcessor`` class. | |
| This processor class receives raw annotations and constructs inputs as a concatenation of input sequences (questions, dialogue contexts, and responses) to facilitate application in GPT models. | |
| Other methods that are specifically defined are ``padding`` (which is used by dataset instances to pad multiple sequence samples) and ``get_attention_mask`` (which creates an attention mask for Transformer attention in GPT models). | |
| .. code-block:: python | |
| SPECIAL_TOKENS_DICT = {'bos_token': "<bos>", 'eos_token': "<eos>", 'additional_special_tokens': ["<speaker1>", "<speaker2>", "<video>", "<cap>"], 'pad_token': "<pad>"} | |
| ... | |
| @registry.register_processor("gpt_dialogue") | |
| class GPTDialogueProcessor(BaseProcessor): | |
| def __init__(self, max_turns=3, use_caption=True): | |
| self.max_turns = max_turns | |
| self.use_caption = use_caption | |
| self.tokenizer = GPT2Tokenizer.from_pretrained('gpt2') | |
| self.tokenizer.add_special_tokens(SPECIAL_TOKENS_DICT) | |
| def sample_sequence(self, caption, history, answer): | |
| bos, eos, speaker1, speaker2, cap = self.tokenizer.convert_tokens_to_ids(SPECIAL_TOKENS[:-2]) | |
| instance = {} | |
| sequence = [caption] + history + [answer] | |
| sequence = [s + [eos] for s in sequence] | |
| instance["input_ids"] = list(chain(*sequence)) | |
| instance["token_type_ids"] = [cap] * len(sequence[0]) + [speaker2 if i % 2 else speaker1 for i, s in enumerate(sequence[1:]) for _ in s] | |
| instance["labels"] = ([-1]*sum(len(s) for s in sequence[:-1])) + sequence[-1] | |
| assert len(instance["input_ids"])==len(instance["token_type_ids"]) | |
| assert len(instance["token_type_ids"])==len(instance["labels"]) | |
| for k,v in instance.items(): | |
| instance[k] = torch.Tensor(v).long() | |
| return instance | |
| def padding(self, seq, pad_token=-1): | |
| if pad_token==-1: pad_token = self.tokenizer.pad_token_id | |
| padded_seq = torch.nn.utils.rnn.pad_sequence(seq, batch_first=True, padding_value=pad_token) | |
| return padded_seq | |
| def get_attention_mask(self, seq, pad_token=-1): | |
| if pad_token==-1: pad_token = self.tokenizer.pad_token_id | |
| return seq != pad_token | |
| def __call__(self, ann): | |
| if self.use_caption: | |
| caption = ' '.join([ann['caption'], ann['summary']]) | |
| caption = self.tokenizer.encode(caption) | |
| else: | |
| caption = [] | |
| dial_history = [] | |
| for turn in ann['dialog'][-self.max_turns:]: | |
| dial_history.append(turn['question']) | |
| dial_history.append(turn['answer']) | |
| dial_history.append(ann['question']) | |
| dial_history = [self.tokenizer.encode(t) for t in dial_history] | |
| answer = self.tokenizer.encode(ann['answer']) | |
| item = self.sample_sequence(caption, dial_history, answer) | |
| return item | |
| @classmethod | |
| def from_config(cls, cfg=None): | |
| if cfg is None: | |
| cfg = OmegaConf.create() | |
| use_caption = cfg.get("use_caption", True) | |
| max_turns = cfg.get("max_turns", 3) | |
| return cls(max_turns=max_turns, use_caption=use_caption) | |
| Registering New Processors ``lavis.processors.__init__`` | |
| ************************************************************** | |
| Finally, any new processor must be officially registered as part of the ``lavis.processors`` module. | |
| For instance, to add processor classes for GPT-based dialogue models, including one for dialogue data ``GPTDialogueProcessor`` and one for video features ``GPTVideoFeatureProcessor``, we can modify the ``__init__.py`` as follows: | |
| .. code-block:: python | |
| from lavis.processors.gpt_processors import ( | |
| GPTVideoFeatureProcessor, | |
| GPTDialogueProcessor, | |
| ) | |
| __all__ = [ | |
| ... | |
| # GPT | |
| "GPTVideoFeatureProcessor", | |
| "GPTDialogueProcessor" | |
| ] | |
| Assigning Processors | |
| ************************************************************** | |
| From the above example of processor classes, note that we define a ``from_config`` method for each class. | |
| This method will process a configuration file and pass specific parameters e.g. ``max_turns``, ``visual_ft``, to initialize the processor classes properly. | |
| To do this, we can assign/ associate the correct registry of processor classes in a configuration file. | |
| For instance, the following should be specified in a configuration file e.g. ``dialogue_avsd_ft.yaml``: | |
| .. code-block:: yaml | |
| datasets: | |
| avsd_dialogue: # name of the dataset builder | |
| vis_processor: | |
| train: | |
| name: "gpt_video_ft" # name of the visual processor for training data | |
| visual_ft: ["i3d_flow", "i3d_rgb"] | |
| audio_ft: ["vggish"] | |
| eval: | |
| name: "gpt_video_ft" # name of the visual processor for evaluation data | |
| visual_ft: ["i3d_flow", "i3d_rgb"] | |
| audio_ft: ["vggish"] | |
| text_processor: | |
| train: | |
| name: "gpt_dialogue" # name of the textual processor for training data | |
| max_turns: 3 | |
| use_caption: True | |
| eval: | |
| name: "gpt_dialogue" # name of the textual processor for evaluation data | |
| max_turns: 3 | |
| use_caption: True | |
| Subsequently, any processes (e.g. training) should load this configuration file to assign the correct processors. | |
| .. code-block:: sh | |
| python train.py --cfg-path dialogue_avsd_ft.yaml | |