Repository: pytorch
Issue: Transformer with convolutional network 
Labels: module: nn,triaged,needs research

### 🚀 The feature, motivation and pitch

I calculated in non-autoregressive tts, asr, machine-translation using transformer with convolutional network.  Machine learning was well carried out. So, I would like you to consider a transformer function with convolutional network.

In my preliminary non-autoregressive machine learning with convolutional network transformer, 37% WER was obtained in Japanese English machine translation, 17% TER was obtained in ASR, good TTS result was obtained.

### Alternatives

example with convolutional network
```
class ConvolutionNetwork(nn.Module):
    def __init__(self, d_in, d_hid, kernel_size=[5,1], dropout=0.1):
        super().__init__()

        # Use Conv1D
        self.w_1 = nn.Conv1d(
            d_in,
            d_hid,
            kernel_size=kernel_size[0],
            padding=(kernel_size[0] - 1) // 2,
        )
        self.w_2 = nn.Conv1d(
            d_hid,
            d_in,
            kernel_size=kernel_size[1],
            padding=(kernel_size[1] - 1) // 2,
        )

        self.layer_norm = nn.LayerNorm(d_in)
        self.dropout = nn.Dropout(dropout)

    def forward(self, x):
        residual = x
        output = x.transpose(1, 2)
        output = self.w_2(F.relu(self.w_1(output)))
        output = output.transpose(1, 2)
        output = self.dropout(output)
        output = self.layer_norm(output + residual)

        return output
```
example post-ln  TransformerEncoderLayer
```
class TransformerEncoderLayer(nn.Module):
    def __init__(self, n_state: int, n_head: int, kernel_size = [5,1], filter_size = 2048):
        super().__init__()
        
        self.dropout1 = nn.Dropout(0.1)
        self.dropout2 = nn.Dropout(0.1)

        self.att = nn.MultiheadAttention(n_state, n_head, batch_first = True)

        self.layernorm1 = nn.LayerNorm(n_state)
        self.layernorm2 = nn.LayerNorm(n_state)

        self.cn = ConvolutionNetwork(n_state, d_hid = filter_size, kernel_size = kernel_size, dropout = 0.1  )


    def forward(
        self,
        inputs,
        mask = None,
    ):
        attn_output, _ = self.att(inputs, inputs, inputs, key_padding_mask=mask )
        attn_output = self.dropout1(attn_output)
        out1 = self.layernorm1(inputs + attn_output)
        cn_output = self.cn(out1)
        cn_output = self.dropout2(cn_output)
        return self.layernorm2(out1 + cn_output)
```
example post-ln TransformerDecoderLayer
```
class TransformerDecoderLayer(nn.Module):
    def __init__(self, n_state: int, n_head: int, kernel_size = [5,1], filter_size = 2048):
        super().__init__()
        
        self.self_dropout = nn.Dropout(0.5)
        self.enc_dropout = nn.Dropout(0.1)
        self.cn_dropout = nn.Dropout(0.1)

        self.self_att = nn.MultiheadAttention(n_state, n_head, batch_first = True)
        self.enc_att = nn.MultiheadAttention(n_state, n_head, batch_first = True )

        self.layernorm1 = nn.LayerNorm(n_state)
        self.layernorm2 = nn.LayerNorm(n_state)
        self.layernorm3 = nn.LayerNorm(n_state)

        self.cn = ConvolutionNetwork(n_state, d_hid = filter_size, kernel_size = kernel_size, dropout = 0.1  )

    def forward(
        self,
        x,
        xa,
        mask1,
        mask2,
    ):
        input_shape = x.size()
        batch_size = input_shape[0]
        seq_len = input_shape[1]
        target_att, _ = self.self_att(x, x, x, attn_mask=mask1)
        target_norm = self.layernorm1(x + self.self_dropout(target_att))
        enc_out, attn_weight= self.enc_att(target_norm, xa, xa, key_padding_mask=mask2)
        enc_out_norm = self.layernorm2(self.enc_dropout(enc_out) + target_norm)
        cn_out = self.cn(enc_out_norm)
        cn_out_norm = self.layernorm3(enc_out_norm + self.cn_dropout(cn_out))
        
        return cn_out_norm
```

### Additional context

Thank you.

cc @albanD @mruberry @jbschlosser @walterddr @mikaylagawarecki