Influenza virus infection is a leading infectious cause of morbidity and mortality in developed countries, and is of considerable public health concern [1] [2] [3] [4] [5] [6] [7] . Many vaccine formulations have been developed to reduce the burden of influenza illness, but the continued evolution of the viruses through antigenic drift requires that the vaccines be reformulated each year [8] [9] [10] . Monitoring the epidemiology and burden associated with circulating influenza viruses is important to make informed recommendations on vaccine use [8] [9] [10] , and the currently circulating strains include influenza A virus subtypes H1N1 and H3N2 and the Yamagata and Victoria lineages of influenza B [11] .
Since 2009, the Serious Outcomes Surveillance Network of the Canadian Immunization Research Network (CIRN SOS) has been conducting active surveillance for acute respiratory illness in hospitalized adults to monitor the burden of influenza illness and assess the effectiveness of seasonal influenza vaccines against laboratory-confirmed influenza [1] [2] [3] [4] [5] [6] [7] . The CIRN SOS Network comprises 15 to 45 acute care (depending on the year) hospitals across Canada, and influenza testing is performed using real-time reverse-transcriptase polymerase chain reaction (RT-PCR) methods derived from the World Health Organization (WHO) [12] [13] [14] . These methods allow the identification of influenza A and B, the discrimination of influenza A viruses into H1N1 and H3N2 subtypes, and the characterization of influenza B viruses into Yamagata or Victoria lineages [12] [13] [14] . These viruses were all circulating in Canada at the time of this study.
While WHO-based real-time RT-PCRs methods are often viewed as the reference standard for influenza virus detection, diagnostic laboratories and surveillance studies often test for other viral aetiologies of respiratory illness [15] [16] [17] [18] [19] [20] [21] [22] [23] [24] [25] [26] [27] [28] . To avoid the high cost and labour associated with individual virus detection, multiplex RT-PCR technologies have been developed and are now commercially available [28] . Multiplex RT-PCR can detect influenza A and B, as well as non-influenza respiratory viruses (NIRVs), such as respiratory syncytial virus (RSV), coronaviruses, rhinoviruses, human metapneumovirus (hMPV), human parainfluenza viruses, adenovirus and enteroviruses [11, [15] [16] [17] [18] [19] [20] [21] [22] [23] [24] [25] [26] [27] [28] . While the focus often lies on influenza, NIRVs can also be a significant causes of morbidity and mortality [29] [30] [31] [32] [33] [34] [35] [36] . NIRVs can also co-circulate with influenza [11, 29] , and can mirror the clinical presentations of influenza, or other viral or bacterial respiratory tract infections [30] [31] [32] [33] [34] [35] [36] . From a diagnostic perspective, rapid identification of viral respiratory viruses has potential benefits, such as a decrease in antibiotic prescriptions, a decrease in laboratory investigations and more judicious use of oseltamivir, and allows for the implementation of infection control practices, such as patient cohorting [31, 37, 38] . From a surveillance perspective, understanding the epidemiology of NIRVs can inform guidelines for patient management [9] [10] [11] 38] and help guide the development of new vaccines or therapeutics [31] . This study focuses on molecular detection of influenza viruses for the purpose of surveillance.
The CIRN SOS Network has performed testing for NIRVs on a subset of nasopharyngeal swabs collected for influenza surveillance. NIRV testing was performed using a Health Canada approved test, the Seeplex RV15 One-step ACE Detection assay (Seegene, Inc., Seoul, Republic of Korea) (RV15). This conventional multiplex RT-PCR uses dual-priming oligonucleotide (DPO) technology [39, 40] for the detection of influenza A and B, as well as 13 other respiratory viruses. The performance of RV15 has previously been compared against cell culture, direct immunofluorescence, real-time RT-PCR and other multiplex RT-PCRs for respiratory viruses [16-25, 36, 41] ; however, to the best of our knowledge, the performance of RV15 for the detection of influenza A and B has yet to be compared against that of the WHO real-time RT-PCR reference methods. Given that RV15 showed variable performance for other viruses compared to other molecular methods [16, 17, 19, 36] , an assessment of RV15 against the WHO reference methods for influenza A and B is well justified. Both RV15 and the WHO real-time RT-PCRs were used by the CIRN SOS Network, allowing a direct comparison of these methods.
(Toronto, ON), le Comité d'éthique de la recherche de Trois-Rivières-Centre hospitalier affilié universitaire regional (Trois-Rivières, QC), the University of British Columbia Clinical Research Ethics Board (Vancouver, BC) and the University of Manitoba Health Research Ethics Board (Winnipeg, MB).
Active surveillance for influenza was performed across up to 45 hospitals in 5 Canadian provinces over consecutive influenza seasons starting in 2009, but the specimens from this study were collected between November 2011 and May 2013. On a daily basis, dedicated SOS Network surveillance monitors reviewed all adult admissions (aged ≥16 years) to identify patients with an acute respiratory illness, and patient demographics and outcomes were collected. Patient demographics and outcomes have been the subject of several CIRN SOS Network publications [1] [2] [3] [4] [5] [6] [7] .
Within 7 days of onset of illness, consenting patients were enrolled and tested for influenza viruses A and B from NP swabs collected in universal transport media (UTM) (Copan Diagnostics). All swabs were divided into aliquots and archived at −80 °C for batch shipment on dry ice to the CIRN SOS Reference Laboratory at the Canadian Center for Vaccinology (CCfV) (Halifax, NS).
The CIRN SOS Reference Laboratory retested all specimens from each CIRN site using validated real-time RT-PCR methods [12] [13] [14] . Total nucleic acids (TNAs) were extracted from 140 µl of NP swab material using a MagNaPure LC 2.0 instrument (Roche Diagnostics). The TNAs were eluted into 60 µl, and 5 µl served as a template for each 25 µl real-time RT-PCR reaction consisting of 0.5 µl SuperScript III RT/Platinum Taq Mix PCR enzyme mix (Invitrogen, Carlsbad, CA, USA), 1×PCR Master Mix, 0.8 µM primers and 0.2 µM probes (Table S1, available in the online version of this article). Initially, influenza strains were identified as A or B with a duplex real-time RT-PCR using primers targeting the matrix genes for each influenza type (Table  S1 ). Subsequently, TNA that was positive for influenza A was subjected to a real-time RT-PCR subtyping assay targeting the haemagglutinin (HA) genes specific to either H1 or H3. TNA that was positive for influenza B was subjected to realtime RT-PCR characterization into Victoria or Yamagata lineages. Amplifications of all real-time RT-PCR assays were performed on an Applied Biosystems 7500 Fast Instrument (Life Technologies), under the following conditions: reverse transcription at 50 °C for 30 min; activation of the Taq DNA polymerase at 95 °C for 2 min; and 45 cycles of 95 °C for 15 s (denaturation) and 55 °C for 30 s (combined annealing and extension). Threshold cycle (C t ) values were provided by the manufacturer's software, and the C t cutoff for positivity was determined using previously validated values at defined thresholds (Table S2) .
Following the same TNA extraction method as described for real-time RT-PCR analyses, TNA was extracted from a separate aliquot of NP swab material and 10 µl of TNA was used as a template for RV15 reactions, as recommended in the manufacturer's instructions. Amplification was performed in a 96-well plate in a C1000 Touch Thermocycler (BioRad Laboratories Ltd, Mississauga, ON, Canada). Amplicons were resolved using 1.2 % (w/v) agarose gel electrophoresis with staining using 1.0 µg ml −1 ethidium bromide (final concentration), and visualized on a GelDoc XR+instrument with ImageLab software (version 5.1) (BioRad Laboratories).
Influenza A and B results from the RV15 or real-time RT-PCR assays were classified as positive or negative, and compared to a composite reference standard where concordant results between two of three methods were considered a true positive or negative result. Discrepant analyses were performed using real-time RT-PCR and/or sequencing by the National Microbiology Laboratory (Winnipeg, MB, Canada). Statistical Analysis Software (SAS) version 9.4 (SAS Institute, Cary, NC, USA) was used to assess significant differences between methods using 2×2 contingency tables and McNemar's chi square test. A P value ≤0.05 was considered statistically significant. Sensitivity, specificity, accuracy, misclassification rates and kappa statistics were reported with 95 % confidence intervals (CIs).
Viral loads in NP swabs were assessed based on standard curves generated by amplification of the matrix gene targets for influenza A and B using conventional RT-PCR, cloning of each target into plasmids and performance of real-time RT-PCR on serial dilutions of the quantified plasmids.
To directly compare the limits of detection (LoD) of RV15 and the WHO real-time RT-PCRs, parallel testing was performed using 10-fold serially diluted viruses. The viruses used were reference strains of the CIRN SOS Network Triplicates values from three independent experiments were analysed, and the C t values obtained from real-time RT-PCR for influenza A and B were used to estimate viral concentrations (relative to the standard curves generated using plasmid controls). The LoD was estimated at a probability of 95 % by Probit analysis [36, 41] using StatPlus 2009 Professional version 5.7.8.
Laboratory data were available on 1111 adults hospitalized with acute respiratory illness, who had all 3 molecular tests of interest performed: RV15, influenza A/B duplex real-time RT-PCR for influenza A or B screening; and H1/H3 real-time RT-PCR subtyping for positive influenza A specimens or realtime RT-PCR Yamagata/Victoria lineage discrimination for positive influenza B. Of the 1111 patients, 151 were positive for influenza A (96 H1N1 and 55 H3N2) and 265 were positive for influenza B (184 Yamagata lineage and 81 Victoria lineage). Most of the results overlapped between testing methods, but some discrepant results were observed (Figs 1  and 2 ).
Compared to the composite reference standard, RV15 showed a sensitivity for influenza A of 98.0 % (148/151; 95 % CI: 95.8-98.0 %) and a specificity of 100.0 % (960/960; 95 % CI: 99.6-100.0 %). Discordant results were seen in three influenza A positive results (Table 1 ; Fig. 1 ). The accuracy of RV15 for influenza A was 99.7 % (1108/1111; 95 % CI: 99.1-99.7 %). Similar performance characteristics were noted for influenza A H1N1 and H3N2 ( 
In this study, the estimated LoD of RV15 for each influenza A subtype and influenza B lineage was higher than that of the real-time RT-PCR for influenza A/B (Table 2 ), but below the detection limits of the real-time RT-PCRs for influenza A subtyping or B lineage characterization (Tables 2 and  S3 ). This is consistent with previously validated cutoffs for positivity for the real-time RT-PCR methods (Table S2) . At viral concentrations that were below the detectable limit of RV15 but detected using influenza A/B real-time RT-PCR, While this was not statistically significant, RV15 was unable to detect influenza A or B in a small subset of specimens in which the viral loads were low. Conventional multiplex PCRs such as RV15 are often less sensitive than real-time RT-PCR, and this can be explained in part by the assay principle. Realtime RT-PCR detects fluorescence signals captured during PCR amplification cycles, and results are defined objectively using validated cutoffs for positivity. Conventional RT-PCR is an end-point detection of amplicons following electrophoresis and staining; a process where visualization of amplicons can be difficult and subjective when working with specimens with low concentrations of target [15-17, 21-23, 36, 41] .
Whether conventional or real-time RT-PCR is used, the performance of these molecular methods can also vary, depending on factors such as genetic mismatches in the PCR target region, which could arise in influenza over time via antigenic drift. This emphasizes the need to verify the performance for characterized and circulating subtypes and lineages of influenza. However, for RV15, only a limited number of studies have assessed its ability to detect influenza A and B viruses, and none have specifically looked at its performance for the detection of influenza A virus subtypes or influenza B lineages [15, 19, 20, 23, 35] . Cho et al. [23] compared RV15 to a composite reference standard that included culture and a commercial real-time RT-PCR and demonstrated that RV15 *The concentration that could reproducibly (at 95 % confidence) be detected by the test method is described as the LoD. The LoD was determined by Probit analysis by testing replicate aliquots of virus dilutions (see Table S3 ). For each virus aliquot, the C t values obtained by the WHO real-time RT-PCR were used to infer viral load. Concentrations of virus dilutions were estimated using standard curves generated with plasmids pFluA and pFluB from this study (see Supplementary Material) [19] assessed the performance of RV15 against direct fluorescent antibody testing, virus culture and isolation, and three additional multiplex PCR methods. The specificities for influenza A and B were 98.8 and 100 %, and the sensitivities were 96.9 % (95 % CI: 91.9-96.9 %) and 100 % (92.6-100 %), respectively. In the present study, the performance of RV15 was verified for the detection of recent influenza A subtypes (H1N1 and H3N2) and influenza B lineages (Yamagata and Victoria), and it showed similar performance characteristics to the WHO real-time RT-PCR. Since this study used specimen collected prospectively from 2011 to 2013, it could be argued that the performance of the molecular methods assessed could vary with more recent circulating influenza strains, if primer or probe mismatches occur in the assay gene targets. However, the CIRN SOS laboratory participates in yearly quality assurance programmes proficiency testing for influenza A and B detection and characterization using all the assays from this study (all real-time RT-PCRs and the RV15 assay), and no differences in assay performance have been observed over time (data not shown). In this study, only a small subset of results in specimens with low viral loads were not detected by RV15, with this representing 0.3 and 0.2 % of the total tests for influenza A and B, respectively. While the lower sensitivity of multiplex PCRs such as RV15 was expected, the possibility of target gene sequence mismatches cannot be excluded.
It should be noted that viral loads were estimated using quantification relative to a standard curve generated using plasmid DNA controls. With influenza virus being an RNA virus, comparison to plasmid DNA does not account for the reverse transcription step. Regardless, all viruses were subjected to direct method comparison in the analytical analyses, which includes the reverse transcription step. Subsequently, the resulting C t values were subjected to comparison against the standard curve derived from plasmid DNA results, as plasmid DNA was more readily quantifiable. Overall, with this approach it remains valid to infer that the specimens not detected by RV15 had low viral loads, but the absolute quantity of virus should be considered to be an estimate (Tables  2, S2 and S3) . Overall, this would have little impact on the conclusions from this study.
This study's strengths include prospectively collected specimens from a defined patient population (adults hospitalized with acute respiratory illness), comparison of results against reference methods for influenza A and B detection, and analyses performed on influenza viruses characterized by subtyping or lineage determination. The main limitation of the study was that it was focused solely on influenza virus. However, this is justified because influenza viruses are the only respiratory viruses for which vaccines are currently available, and the performance for the detection of other viruses has been assessed previously [17, [19] [20] [21] [22] [23] . While assessing the performance of RV15 against validated real-time RT-PCRs for influenza is well justified, this study focused on the use of RV15 for population-based surveillance in hospitalized adults. These findings should not be extrapolated to other patient populations or for applications in clinical diagnostic testing where individual-level results are prioritized. The patients tested in this study were hospitalized adults, often presenting with co-morbidities and severe outcomes [1] [2] [3] [4] [5] [6] [7] . Further, respiratory virus testing is often performed using testing algorithms following initial screening methods for influenza A and B. As such, nucleic acids extracted from clinical specimens are sometimes inadvertently subjected to a freeze/thaw cycle prior to testing. The impact of freeze/ thaw was not assessed in this study, as testing was performed following independent nucleic acid extraction on a different specimen aliquot. However, given that RV15 failed to detect a small subset of influenza A and B specimens at low viral loads, additional freeze/thaw cycles may further compromise influenza virus detection, and should thus be avoided. Finally, the workflow is relatively simple with RV15 for small numbers of specimens, but additional benefits could be afforded by using imaging software enabling automated amplicon detection if high-throughput specimen processing is required [15] [16] [17] [19] [20] [21] [22] [23] . Such automated analyses of RV15 amplicons could also reduce reduce result subjectivity compared to interpretations made from visual assessment of amplicons [15] [16] [17] [19] [20] [21] [22] [23] . Automated analyses were not evaluated in this study, but the technical staff performing RV15 testing were blinded to the real-time RT-PCR method results to avoid bias, and the RV15 results were remained unchanged with subsequent independent review by other blinded staff members.
Overall, the performance of RV15 was comparable to the WHO real-time RT-PCR standards for the detection of recently circulating subtypes of influenza A and lineages of influenza B, and it only missed a very small subset of influenza A and B results at low viral loads. Given that the performance characteristics of the RV15 multiplex PCR are not provided in the manufacturer kit insert, these data are of value for its users, which include several acute care hospitals and provincial public health laboratories in Canada [36, 41] . This study shows that the RV15 conventional multiplex PCR can be used for surveillance studies for respiratory viruses without significantly compromising detection of influenza A and B. The use of multiplex technologies such as RV15 can help better define the epidemiology of influenza and NIRVs, and these data are important for the development of novel therapeutics and vaccines.
Influenza virus surveillance was funded by the Public Health Agency of Canada (PHAC) and the Canadian Institutes of Health Research (CIHR), and through a Collaborative Research Agreement between GlaxoSmith-Kline Biologicals SA and the CIRN SOS Network. CIRN and SM provided funding for RV15 testing. The authors received no financial support or other form of compensation related to the development of the manuscript, and were solely responsible for final manuscript content and data interpretation. the many dedicated CIRN SOS Network surveillance monitors, the hospital staff and the CCfV staff, who were instrumental in the recruitment, collection, processing and archiving of specimens, and data collection.