Paper:
Multisensory Encoding, Decoding, and Identification
We investigate a spiking neuron model of multisensory integration. Multiple stimuli from different sensory modalities are encoded by a single neural circuit comprised of a multisensory bank of receptive fields in cascade with a population of biophysical spike generators. We demonstrate that stimuli of different dimensions can be faithfully multiplexed and encoded in the spike domain and derive tractable algorithms for decoding each stimulus from the common pool of spikes. We also show that the identification of multisensory processing in a single neuron is dual to the recovery of stimuli encoded with a population of multisensory neurons, and prove that only a projection of the circuit onto input stimuli can be identified. We provide an example of multisensory integration using natural audio and video and discuss the performance of the proposed decoding and identification algorithms.

Review:
Reviewer's response to the rebuttal: 
      '"Theorem 1 needs to be corrected/clarified." We 
      apologize for the confusion: part of the theorem was accidentally omitted. 
      The sufficient condition is to have a large enough population of 
      (different) neurons: N >= SUM_m[(2L_1+1)(2L_2+1)…(2L_{n_m-1} + 1)]. The 
      necessary condition is that the total number of spikes is greater than 
      (2L_{n_m}+1)*SUM_m[(2L_1+1)(2L_2+1)…(2L_{n_m-1} + 1)] +N" OK, 
      that seems more acceptable, even though I don't fully understand the 
      condition -- what prevents these spikes from being completely synchronized 
      across neurons, and hence being uninformative? '"…what would 
      happen if all filter kernels were the same? Then no matter how many 
      neurons you added, you would not gain any further information.” That 
      is not entirely correct. For example, if all filters are Dirac-delta 
      functions, the inputs are encoded directly without processing. However, 
      one can recover all inputs if the neurons have different parameters (e.g., 
      bias, capacitance, threshold of the IAF neuron). Imposing linear 
      independence between kernels is another way to guarantee signal recovery.' 
      True, but in your definition of the IAF-TEM (eqn3 and following) 
      it does not appear that the neurons have different parameters, hence my 
      remark. Please say that somewhere. Review for "Multisensory 
      Encoding, Decoding, and Identification" The paper describes an 
      extension of Time Encoding Machines (TEM) to the 
      multiple-input-multiple-output (MIMO) setting, using the formalism of 
      Reproducing Kernel Hilbert Spaces (RHKS). The approach is motivated from a 
      neuroscientific perspective: in the introduction, it is claimed 
      (correctly) that multisensory integration (and processing) provides many 
      benefits to organisms, but that it is still poorly understood. What 
      follows is a particular extension of TEMs to the MIMO setting, with 
      trignometric polynomials as basis functions. Necessary conditions for 
      perfect decoding and/or system identification are given. The feasibility 
      of the resulting algorithms are demonstrated on an audio/video decoding 
      task, and an audio/video encoder identification task. The paper concludes 
      with the remark that this work constitutes the first tractable 
      computational model for multisensory integration (which is wrong) and that 
      extensions to the noisy case would be straightforward. Clarity: 
      The paper is mostly well written, even though the notation in section 
      3 is cumbersome, but I do not see how that could be avoided. 
      Originality: the paper's most original contribution is the 
      extensions of TEMs to the MIMO setting, the connection to the brain is 
      tenuous. Quality: I believe some parts of the paper need to be 
      improved before publication. Significance: from a technical 
      perspective, the paper demonstrates the feasibility of TEMs for 
      multi-modal encoding, which might prove useful. There is no clear 
      contribution to neural coding, other than the motivation in the 
      introduction. Detailed comments: line 100: 
      "bandlmited" -> bandlimited line 137: j is the complex unit, 
      please say that somewhere. line 144: the u_n in eqn.(1) are not 
      the same as the u_n^i in figure 1, right? I propose to use a different 
      letter. line 166: "BIBO-stable" bounded-input-bounded-output? 
      please define. line 166: "...finite temporal support of lenghth 
      S_i \leq T_i...": would that be spatio-temporal support for the video 
      filters in example 2? line 177: Say here that the h_{l_1 \ldots 
      l_n} are the filter coefficients, to connect this formula to theorem 1. 
      line 182: "...IAF TEM...": IAF="integrate and fire"? Please 
      specify. line 185: eqn. 3: what is the index k? From other 
      literature about IAF-TEMs, I would infer that it's the spike index? If 
      that is correct, please also say that t_k is a spike time, and that these 
      are strictly monotonically increasing with k etc. In fact, it would be 
      very useful if there was a short introduction to IAF-TEMs: up to here, the 
      paper only talks about continuous signals, now spikes are introduced (if I 
      interpret everything correctly). line 195-195: "Then there exists 
      a number N ... can be perfectly recovered" this sounds like a sufficient 
      condition. But in the proof of this theorem 1, line 224ff: "A necessary 
      condition for the latter (solvability)...." so you only show a necessary 
      condition? Hence, theorem 1 remains unproven, I think. In fact, I wonder 
      what would happen if all filter kernels were the same? Then no matter how 
      many neurons you added, you would not gain any further information, thus 
      the signal recovery would not improve. To remedy this, I think the kernels 
      would have to be constrained by some sort of linear independence 
      condition, similar to [15]. figures 1,2,4: the v_i do not appear 
      anywhere else in the paper. Please clarify their purpose. section 
      5, examples. The examples are instructive. But I wonder why you did not 
      choose a real video+audio for the examples -- it would have strengthened 
      the connection to neuroscience if you showed that your approach works 
      on relevant datasets. line 417: "...the first tractable 
      computation model for multisensory integration..." wrong, see e.g. the 
      works by Wei & Koerding, or Ernst & Banks. Please remove this 
      sentence.