Uni-Perceiver-MoE: Learning Sparse Generalist Models with Conditional MoEs
Anonymous Author(s)
AffiliationAddress email
Abstract
To build an artificial neural network like the biological intelligence system, recentworks have unified numerous tasks into a generalist model, which can process various tasks with shared parameters and do not have any task-specific modules.
Whilegeneralist models achieve promising results on various benchmarks, they haveperformance degradation on some tasks compared with task-specialized models.
In this work, we find that interference among different tasks and modalities is themain factor to this phenomenon.
To mitigate such interference, we introduce the
Conditional Mixture-of-Experts (Conditional MoEs) to generalist models.
Routingstrategies under different levels of conditions are proposed to take both the training/inference cost and generalization ability into account.
By incorporating theproposed Conditional MoEs, the recently proposed generalist model Uni-Perceivercan effectively mitigate the interference across tasks and modalities, and achievesstate-of-the-art results on a series of downstream tasks via prompt tuning on 1% ofdownstream data.
Moreover, the introduction of Conditional MoEs still holds thegeneralization ability of generalist models to conduct zero-shot inference on newtasks, e.g., video-text retrieval and video caption.
Code and pre-trained generalistmodels shall be released.
Introduction
Generalist models that handle multiple modalities and numerous tasks have been long pursued bythe machine learning community.
However, previous researches [65, 90, 71] focus on developingspecialized models with task-specific modules.
When these models are applied to new tasks, thespecifically-designed components need to be redesigned on demand and fine-tuned on sufficientdownstream data.
As a result, their model size increases with the number of diverse downstreamtasks, conflicting with the goal of generalist models.
Recently, some pioneers [94, 80, 3, 85, 87, 62] have made preliminary attempts to build generalistmodels by modeling various tasks into a unified formulation.
With the unified modeling, large-scalepre-training on various datasets enables the generalist models to process different downstream tasksusing shared parameters.
These generalist models not only achieve competitive performance onpre-training tasks [80, 3, 85, 87], but also can perform zero-shot inference on novel tasks withoutintroducing additional parameters [94, 62].
However, compared to specialized models with specific parameters for each task, generalist modelswith shared parameters would suffer from the task-interference issue — different tasks with sharedparameters may conflict with each other [89].
The same issue is also observed in multilingual NLPmodels [4, 82, 84].
We argue that the task-interference issue is mainly caused by the inconsistentoptimization in multi-task learning.
As shown in Tab. 1, during the training phase of generalistmodels, the gradient directions of different tasks would be inconsistent or even opposite.
Thus, if multiple tasks share parameters, the optimal update direction of the shared parameters will beuncertain, resulting in sub-optimal performance.
Allowing conflicting modalities and tasks to use separate parameters should effectively mitigate theinterference issue in generalist models.
Mixture of Experts (MoEs) [43, 23] provides a potentialsolution, which learns to activate sub-networks dynamically without introducing any task-specificmodules.
Nevertheless, vanilla MoEs [67] select the experts according to token representations,which suffers from high training/inference cost and neglects the information of different tasks andmodalities.
In this work, we argue that routing strategies of MoEs require special design when appliedto generalist models for mitigating the task-interference issue.
To address the task-interference issue in generalist models, we propose Conditional Mixture-ofExperts (Conditional MoEs), which improve vanilla MoEs by introducing information under differentlevels of conditions, including token-level, context-level, modality-level, task-level, and predefinedtoken attributes.
In this case, vanilla MoEs is a token-level variant of our Conditional MoEs, whichcan be replaced by other-level variants to implement stronger generalist models.
We carefullydiscussed the training/inference cost and generalization ability of different variants, and ablated theirperformances in mitigating the interference issue of generalist models.
Notably, Conditional MoEswith predefined token attributes introduces 8-bit attribute embedding to describe the information ofcurrently processed task and modalities, which demonstrate excellent computational and memoryefficiency and good generalization ability.
To verify the effectiveness of Conditional MoEs, we incorporated it with the recently proposedgeneric perception model Uni-Perceiver [94] by replacing the linear projection in self-attentionand FFN blocks with conditional MoE layers.
Experiments demonstrate that, by mitigating taskinterference with our proposed Conditional MoEs, Uni-Perceiver can be pre-trained on various tasksjointly without performance degradation, while its generalization to other tasks can be maintainedsimultaneously.
Our main contributions are as follows:
• We carefully analyze the task-interference issue in generalist models, and provide an explanationfrom the gradient direction perspective as well as a metric to quantify the issue.•
We propose Conditional MoEs to address the task-interference issue in generalist models.
By introducing the information of currently processed task and modalities, Conditional MoEs effectivelymitigate the interference issue, while keeping low computational and memory cost.
• Compared with previous SOTAs, our generalist model with 1% downstream data prompt tuningachieves competitive performance, while only < 5% training data and < 10% training cost are used.
We hope this work can serve as a solid baseline for generalist models and motivate further research.
Related Works
Specialized Models.
Previous research focuses on building specialized models for specific tasks.
CNNs [47, 26, 70] and ViTs [20, 53, 76, 81] are developed for image classification.
Subsequentworks re-design them to adapt to diverse downstream visual tasks, e.g., object detection [63] andsegmentation [15, 48].
In NLP, different architectures are specifically designed for neural machinetranslation [78], natural language understanding [19], and natural language generation [51].
As forvision-language tasks, previous works usually combined modality-specific encoders and representation fusion modules together [13, 54].
Recently, [90, 65, 71] integrate several specialized models intoa single one to handle diverse tasks.
Such integrated specialized models are equipped with multipletask-specific modules to adapt to as many downstream tasks as possible.
However, these methodsstill follow the task-specific paradigm, which conflicts with the objective of generalist models.
Vanilla Generalist Models.
Vanilla generalist models handle different tasks and modalities withshared parameters.
Uni-Perceiver [94] formulates various perception tasks as finding the maximum likelihood target for each input through the similarity of their representations.
OFA [80],
Flamingo [3] and SimVLM [85] attempt to unify different tasks into sequence-to-sequence generation.
UniCORN [87] and Gato [62] further incorporate bounding box and reinforcement learningtasks into the unified formulation, respectively.
These generalist models not only achieve competitiveperformance on pre-training tasks with shared parameters, but also can perform zero-shot inferenceon new tasks [62, 94].
However, these methods rarely investigate the potential interference amongdifferent modalities and tasks, which could result in the performance degradation of generalist models.
Standard Encoder Block
Layer Norm
Attention
Layer Norm
FFN task1 tokens task2 tokens task1 tokens task2 tokens
Encoder Block with task- dependent parametrization
Layer Norm
Layer Norm
Task1 FFN task1 tokens task2 tokens task1 tokens task2 tokens
Task2 Attention Task1 Attention
Task1 FFN
Encoder Block with Mixture-of-Experts
Layer Norm
MoEAttention
Layer Norm
MoE FFN task1 tokens task2 tokens task1 tokens task2 tokens task1 tokens task2 tokens
Gating
L i nea r L i nea r
L i nea r E … weighted sum
L i nea rselect 𝑤𝑤 𝐸 replace linear with MoELinear
Multi-Task Learning.
Multi-task learning [8,
] has been widely studied in the communityof vision [27, 74, 72], language [25, 16, 50] and vision-language learning [10, 55, 29].
Whilemulti-task training enables collaboration between tasks, it may also introduce the task interferenceproblem [82, 84, 28, 36, 72].
To mitigate the task-interference issue, some works[14, 24, 38] proposeto dynamically adjust the loss weight for each task, while others [91, 49, 36] instead use task-dedicatedparameters.
However, methods with task-specific parameters are difficult to generalize to new tasksand do not meet the requirements of generalist models.
Mixture of Experts (MoEs).
MoEs has shown its remarkable ability to scale neural networks [67, 43,23, 64, 21].
[67] first proves the effectiveness of MoEs by stacking MoE layers in the LSTM models.[68, 43] further introduce this approach to Transformer architectures.
[23, 40] train language modelswith trillion parameters successfully by utilizing simplified MoE routing strategy and efficient trainingtechniques.
There are also some works applying MoEs to CNNs for computer vision tasks [1, 86, 83].
Recently, V-MoE [64] successfully employs MoEs to ViTs, showing promising performance onmany visual tasks.
Task-MoE [42] focuses on applying MoEs for multilingual translation to mitigatethe interference among different languages.
In this work, we aim to explore MoEs under differentconditions for general models.
Methodology
In this section, we first analyze the task-interference problem from the gradient direction perspective.
Based on the analysis, we propose Conditional Mixture-of-Experts (Conditional MoEs) for generalistmodels, which introduces parameters conditioned by information of different levels to mitigate thetask-interference issue with negligible overhead.
3.1 Task Interference
To quantify the interference of the j -th task on the i -th task, we estimate the change in loss L i of thei -th task, when optimizing the shared parameters θ according to the j -th task L j as:
[Equation]
where x i and x j are the sampled training batches of the i -th and j -th tasks, respectively.
Without lossof generality, we only consider the update direction ignoring the update norm.
Then, the interferenceof the j -th task on the i -th task can be quantified as:
[Equation]
where the denominator is used to normalize the loss change scale.
As reported in Tab. 1, we sample100 batches for each tasks, and record the gradients to calculate the average interference metric
I i,j ofthe j -th task on the i -th task at the 4-th/12-nd FFN blocks.
We see that, at shallow layers, the image
To calculate the interference metric, we sample 100 batches for each tasks, and record the gradients based on the pre-trained Uni-Perceriver
Ti.
The red value indicates that the task j has a negative impact and
[Table]
caption task has positive impacts on image classification and masked language modeling, suggestingthat cooperation between different tasks exists.
While at deep layers, tasks with different optimizationobjectives hardly enhance each other, and the gradient directions may even opposite.
Fig.
1 summarizes three mainstream architectures for multi-task models.
The first is the standardarchitecture [94, 31, 32] with parameters fully shared by different tasks, which suffers from taskinterference problem as analyzed above.
The second is task-specific parameterized architecture [90,65, 29, 71] equipped with dedicated parameters for each task.
Although this architecture address theinterference problem by task-specific parameters, it is difficult to generalize to new tasks that didnot emerge in the training phase.
Unlike the above two architectures, the Mixture-of-Experts (MoE)architecture [67, 43, 23, 40, 64] activates models sparsely according to different given inputs byselectively utilizing different subset of the model parameters.
The sparse routing mechanism makes itpossible to train very large generalist models, which maximizes the collaboration and meanwhilemitigates the interference problem.
In this work, we focus on exploring Conditional MoEs for generalmodels, whose experts are gated by conditions from different levels.
3.2 Conditional Mixture-of-Experts (Conditional MoEs)
We first describe the prototype of Conditional MoEs, and then provide its specific instantiations underdifferent conditions, as well as the application to generalist models.
Prototype.
Given any token x i in the input sequence X
= { x i } Li =1 , conditional MoEs with Eexperts firstly introduces a gate decision vector G ∈ R E that dispatches different input tokens todifferent experts, which is calculated as:
[Equation]
3) where R ( · ) defines a general routing strategy for gate decision, which is alternative under differentconditions.
W g is the trainable weights in gate decision and ϵ is the noise term.
The top k ( · ) operatorsets all values to be zero except the largest k values.
Since G only has k ≪ E non-zero values, thetoken x i is routed to only a small number of experts.
After getting the gate decision vector G , thecorresponding output y i is the weighted combination of each expert’s computation on x i as:
[Equation]
where W e is the linear projection weights of the e -th expert and gate decision G e determines howmuch the e -th expert contributes to the output
y i .
Note that, experts with G e = 0 does not need to becomputed for saving computation.
In Conditional MoEs, the routing strategy R ( · ) plays an important role in the multi-modality andmulti-task training of generalist models.
By sparsely activating experts according to differentconditions, Conditional MoEs can mitigate the interference issue while maintaining the generalityof the pretrained model.
Next, we introduce variants with specific routing strategies under differentconditions, as shown in Fig.
Token-Level Routing.
Similar to vanilla MoEs [67, 43, 23, 40, 64], the token-Level MoEs directly token representation the routing strategy, can be written as:
[Equation]
The routing strategy of token-level MoEs is an identical function, where the gate decision onlydepends on each token’s own representation.
[Figure]
[Table]
Context-Level Routing.
Tokens with similar representations may appear in conflicting tasks, whoseoptimal expert decisions should be different to mitigate the task interference.
Therefore, to help gatefunction making more reliable decisions, we explore the combination of global context and localtoken representation.
The routing strategy utilizing global context can be expressed as:
[Equation]
(6) where concat( · ) indicates the concatenation operation,
X = { x i } Li =1 is sequence of all tokens in thecurrent sample, and attnpool( · ) indicates the attention pooling operator [61].
Modality-Level Routing.
Most current practice uses modality-specific encoders with independentparameters for different modality inputs.
Inspired by this, we also explore to leverage the modality ofthe current token as a routing strategy:
[Equation]
Here, embed( · ) represent the embedding layer, and id modal ( · ) indicates the modality index of currenttoken x i .
The routing function will assign this token to experts according to its modality embedding.
Task-Level Routing.
In addition to modality information, task information can also be used to guidegate functions to make reliable decisions for mitigating the task interference.
Similar to Eqn.
(7), therouting strategy can be formulated as:
[Equation]
where id task ( · ) is the task index of current token x i .
The task embedding for this task will be usedto compute gate decision.
Since all tokens from one task have the same task embedding, all tokenscorresponding to this task will be routed to the same set of experts.
Attribute Routing.
Among the aforementioned variants, token-level, context-level, and modalitylevel routing strategies only focus on input tokens but omit information about the currently processed task.
While task-level routing strategy relies on task-specific ids, it limits the generalization abilityto new downstream tasks.
To introduce the information of currently processed task and modalitieswithout losing the generalization ability of generalist models, we propose to introduce token attributesto assist the gate decision.
As described in Tab. 2, the attributes of the current token are represented as an 8-dimensional binaryembedding, whose attributes include the modalities of current task and token (index 0 ∼ 5), thecausation type of the model (index 6), and the token source (index 7).
As a result, the designed tokenattributes provide comprehensive information of currently processed task meanwhile keeping the taskgeneralization ability.
Based on the attribute embedding, the routing strategy is expressed as:
[Equation]
Here, attr( x i ) is the 8-dimensional binary attribute embedding of the current token x i as describedin Tab. 2.
W attr is the learnable weights to transform the attribute embedding to latent representation,and layernorm( · ) denotes the layer normalization [6] for training stabilization.
Application to Generalist Models.
Without loss of generality, we explore the application of
Conditional MoEs to the generalist model Uni-Perceiver [94], which uses Transformers to handlevarious modalities and tasks with shared parameters.
We replace linear projection layers in bothself-attention and FFN blocks with Conditional-MoE layers (see Fig. 1).
3.3 Comparison of Conditional-MoE Variants
As illustrated in Fig.
2, among the variants of Conditional MoEs, token-level and context-level MoEsare data-dependent, while modality-level, task-level, and attribute MoEs are data-independent.
Training and Inference Cost.
Compared to dense models with the same number of parameters, all
Conditional MoE variants can significantly reduce the computational cost benefiting from the sparserouting mechanism.
Due to the dependence of input data, the memory consumption of token-leveland context-level MoEs is relatively high during model training, and model parallelism is required torelieve memory cost by partitioning experts across multiple devices, leading to heavy inter-devicecommunication overhead.
This problem persists when using pre-trained models for task-specificinference, where all experts need to be loaded into memory and might be activated by any token.
Different from data-dependent Conditional MoEs, data-independent variants such as modality-level,task-level, and attribute MoEs have excellent memory efficiency, since only top- k experts need tobe activated for all tokens with the same modality/task/attributes.
Moreover, in both training andinference phase, the experts in a data-independent MoE layer can be merged into a single linearprojection using reparameterization techniques.
In this case, the computation cost of the networkwith data-independent Conditional MoEs will be equivalent to a dense model without MoEs.
Generalization Ability.
We hope to mitigate the task-interference issue in generalist models, whilekeeping their generalization ability to new downstream tasks.
While token-level, context-level, andmodality-level MoEs without task-specific designs do not harm the generalization ability, they ignorethe task-level information which is essential to resolve the task interference.
Conversely, task-levelrouting strategy is tied to a specific task id, which is difficult to generalize to new downstream tasks.
Attribute MoEs introduce predefined token attributes to comprehensively describe the information ofcurrently processed task and modalities, which can be transferred to new downstream tasks withoutany task-specific modifications.
This gives attribute MoEs the potential to mitigate task interferencewithout losing generalization ability.
Experiments
In this section, we first describe our experimental setup.
Then, we confirm the task-interference issuein the generalist model Uni-Perceiver [94] and ablate the the ability of different Conditional MoEs tomitigate task interference.
Finally, large-scale training is conducted to verify the effectiveness of ourproposed Conditional-MoEs and its generalization ability to novel tasks.
[Table]
We use the same datasets in Uni-Perceiver [94] to pre-train our models 1 .
Specifically, ImageNet21k [18] is used for image classification pre-training.
Kinetics-700 [37] and Moments in Time [57]are used for video classification pre-training.
Language modeling task is trained on BookCorpus [95]& English Wikipedia (Books&Wiki).
For language modeling with image clues and image-textretrieval, we use a combination of image-text-pair datasets: SBU Captions (SBU) [58], Visual
Genome [41], COCO Caption [12], CC3M [66], CC12M [9] and YFCC [35].
Following UniPerceiver, Imagenet1K [18], Kinetics-400 [37], COCO Caption [12], and Flickr30k [59] are utilizedto evaluate the performance of generalist models on downstream tasks.
We also use two datasetsthat evaluate the generalization ability to novel tasks: MSVD [11] and GLUE [79].
Additionally, alldataset licenses are included in Appendix.
4.2 Implementation Details
We incorporate the vanilla generalist model Uni-Perceiver with Conditional MoEs for experimentswith three different variants: Uni-Perceiver-Ti (Tiny), Uni-Perceiver-B (Base), and Uni-Perceiver-L(Large).
Please refer to Appendix for architecture hyperparameters.
If not specified, the input imageresolution is set to 224 × 224.
In each training iteration, each GPU independently samples a single taskand dataset.
The gradients of different GPUs are synchronized after the gradient back-propagation.
We use the AdamW optimizer with a base learning rate of 0.0005 and a weight decay of 0.05.
Similarto [52, 61], we find setting β 2 = 0 .
98 and ϵ = 10 − 6 helps improve stability when large-scale training.
Besides, gradient clipping with 0.5 is used to stabilize training.
Uni-Perceiver-B and Uni-Perceiver-L are equipped with Conditional-MoEs layer for every otherlayers while Uni-Perception-Ti use Conditional MoEs in all layers.
A normal noise is also used onthe gate logits following [64] for a better exploration for new potential experts.
If not specialized,top-2 gate function is used.
For other hyper-parameters of MoE layers, please refer to Appendix.
4.3 Ablation Studies
This part explores whether Conditional MoEs can effectively mitigate task interference in generalistmodels and compares different routing strategies.
Tab. 3 summarizes the performance of Uni-Perceiverand its variants on three typical tasks.
Compared with task-specific parameterization, the performancedegradation of Uni-Perceiver confirms the existence of task interference.
Incorporating Conditional
MoEs with any routing strategy can mitigate the task-interference issue and significantly improvethe performance.
Among these five routing strategies, token-level, context-level, and modality-level
MoEs deliver slightly worse performance.
We argue the missed task information is critical forresolving the task interference.
Besides, the data-dependent MoEs, i.e., token-level and context-level,have relatively higher training and inference cost, while the other three MoEs have excellent efficiencyby using reparameterization techniques.
Although both task-level and attribute MoEs achieve goodperformance, the specialized task-id design in task-level MoEs makes it difficult to generalize to newtasks.
Therefore, the Conditional MoEs with attribute routing strategy will be used.
[Table]
4.4 Evaluation on Pre-training tasks
Large-scale training is conducted to verify the effectiveness of our method, we first evaluate it on tasksinvolved in pre-training.
Specifically, we use widely-used Imagenet-1k [18] and Kinetics-400 [37]to evaluate image and video classification respectively, and use popular Flickr30k [59] and COCO
Caption [12] to evaluate image caption and image-text retrieval.
Tab. 4 and Tab. 5a show the results on the four pre-training tasks.
We see that Uni-perceiver with our Conditional MoEs consistently outperforms vanilla Uni-perceiver by a large margin.
Without anytuning, our models achieve comparable performance with task-specific SOTAs trained with similarmodel size and training data size.
Note that, our approach is a generalist model pretrained on a unifiedtask formulation, while task-specific approaches are trained specifically for the target task.
When prompt tuned on only 1% downstream data, the performance of our models are boosted to alevel close to counterparts that use > 50 × training data sizes and > 10 × training cost.
For the prompttuning of our models, only a small amount of parameters are tuned, and the encoder is still fixed andshared among different tasks, indicating that generalist models with Conditional MoEs can handledifferent tasks with significant low cost than counterparts.
We further fine-tune our models with 100% of the downstream data.
In this case, our model achievesperformance on-par with or better than the SOTAs trained with similar data size on all these tasks,which proves generalist models with Conditional MoEs has learned high-quality representations.
[Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table]
4.5 Generalization to Novel Tasks
The generalization ability is the most attractive aspect of generalist models, while the dynamicsub-networks activation of Conditional MoEs should maintain this ability while mitigating taskinterference.
To verify this, we conduct experiments on video caption, video-text retrieval
[Table]
andnatural language understanding tasks, which did not appear in pre-training.
As shown in Tab. 5c, our Uni-Perceiver equipped with Conditional MoEs could generalize to video-related tasks very well.
They can obtain reasonable zero-shot performance on those tasks and also perform better than vanilla
Uni-Perceiver with a great margin.
Moreover, Uni-Perceiver-MoEs can achieve comparable results to
SOTA methods with similar training cost by further conducting prompt tuning with only 1% data.
Beyond that, Conditional MoEs can significantly boost the performance of Uni-Perceiver on GLUEbenchmarks (Tab. 5b), owing to its excellent ability to resolve task interference in generalist models.
Conclusion
In this paper, we propose Conditional to address the task-interference issue in generalist models.
By sparsely activate sub-networks without introducing any task-specific designs, generalist modelscan be pre-trained on multiple tasks jointly without performance degradation, while keeping thegeneralization ablity to noval tasks.
We incorporate Conditional MoEs with the recently proposedgeneralist model Uni-Perceiver.
With prompt tuning on 1% downstream data, the proposed sparsegeneralist model achieves competitive performance with previous SOTAs using only < 5% trainingdata and < 10% training cost.
We hope this work can motivate further research in generalist models.
Limitations.
Our method is currently verified on generalist models with millions of parameters.
Forgeneralist models with billions of parameters, whether the task-interference issue exists and whetherour method is still effective are questionable, which we leave them to future work.
Potential Negative Societal Impact.
This work shares the common negative impacts of large-scaletraining, which may consume lots of electricity and result in increased carbon emissions.
This methodalso learns from a large number of datasets that may contain data biases.
Checklist
For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper’s
contributions and scope?
[Yes]
(b) Did you describe the limitations of your work?
[Yes] See Section 5.(c)
Did you discuss any potential negative societal impacts of your work?
[Yes] See
Section 5.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to
them?
[Yes]
If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results?
[N/A]
(b) Did you include complete proofs of all theoretical results?
[N/A]
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experi-
mental results (either in the supplemental material or as a URL)?
[No] Our code will
be released once the paper is accepted, and the used data is publicly available.
(b) Did you specify all the training details (e.g., data splits, hyperparameters,
how they
were chosen)?
[Yes] See Section 4.
(c) Did you report error bars (e.g., with respect to the random seed after running experi-
ments multiple times)?
[No] Most experiments have stable results with little variance.
(d) Did you include the total amount of compute and the type of resources used (e.g., typeof GPUs, internal cluster, or cloud provider)?
[Yes] See Section 4.
If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators?
[Yes] We cite the opendatasets that we use.
(b) Did you mention the license of the assets?
[Yes] See Section 4.
(c) Did you include any new assets either in the supplemental material or as a URL?
[No]
Our code will be released once the paper is accepted.
(d) Did you discuss whether and how consent was obtained from people whose data you’re
using/curating?
[Yes] See Section 4.
(e) Did you discuss whether the data you are using/curating contains personally identifiable
information or offensive content?
[Yes] See Section
If
used crowdsourcing or conducted research with human subjects..
(a) Did
include the full text of instructions given to participants and screenshots, if
applicable?
[N/A
[Table] [Table]
[Table] [Table]