Query: Residual vs. Inception vs. Classical Networks for Low-Resolution Face Recognition

Relevant passages:
DeepFace: Closing the Gap to Human-Level Performance in Face Verification <=SEP=> In modern face recognition, the conventional pipeline consists of four stages: detect => align => represent => classify. We revisit both the alignment step and the representation step by employing explicit 3D face modeling in order to apply a piecewise affine transformation, and derive a face representation from a nine-layer deep neural network. This deep network involves more than 120 million parameters using several locally connected layers without weight sharing, rather than the standard convolutional layers. Thus we trained it on the largest facial dataset to-date, an identity labeled dataset of four million facial images belonging to more than 4, 000 identities. The learned representations coupling the accurate model-based alignment with the large facial database generalize remarkably well to faces in unconstrained environments, even with a simple classifier. Our method reaches an accuracy of 97.35% on the Labeled Faces in the Wild (LFW) dataset, reducing the error of the current state of the art by more than 27%, closely approaching human-level performance.

Identity Mappings in Deep Residual Networks <=SEP=> Deep residual networks [1] have emerged as a family of extremely deep architectures showing compelling accuracy and nice convergence behaviors. In this paper, we analyze the propagation formulations behind the residual building blocks, which suggest that the forward and backward signals can be directly propagated from one block to any other block, when using identity mappings as the skip connections and after-addition activation. A series of ablation experiments support the importance of these identity mappings. This motivates us to propose a new residual unit, which makes training easier and improves generalization. We report improved results using a 1001-layer ResNet on CIFAR-10 (4.62% error) and CIFAR-100, and a 200-layer ResNet on ImageNet. Code is available at: https://github.com/KaimingHe/ resnet-1k-layers.

Face Description with Local Binary Patterns: Application to Face Recognition <=SEP=> This paper presents a novel and efficient facial image representation based on local binary pattern (LBP) texture features. The face image is divided into several regions from which the LBP feature distributions are extracted and concatenated into an enhanced feature vector to be used as a face descriptor. The performance of the proposed method is assessed in the face recognition problem under different challenges. Other applications and several extensions are also discussed

Low-resolution Convolutional Neural Networks for video face recognition <=SEP=> Security and safety applications such as surveillance or forensics demand face recognition in low-resolution video data. We propose a face recognition method based on a Convolutional Neural Network (CNN) with a manifold-based track comparison strategy for low-resolution video face recognition. The low-resolution domain is addressed by adjusting the network architecture to prevent bottlenecks or significant upscaling of face images. The CNN is trained with a combination of a large-scale self-collected video face dataset and large-scale public image face datasets resulting in about 1.4M training images. To handle large amounts of video data and for effective comparison, the CNN face descriptors are compared efficiently on track level by local patch means. Our setup achieves 80.3 percent accuracy on a 32×32 pixels low-resolution version of the YouTube Faces Database and outperforms local image descriptors as well as the state-of-the-art VGG-Face network [20] in this domain. The superior performance of the proposed method is confirmed on a self-collected in-the-wild surveillance dataset.

Caffe: Convolutional Architecture for Fast Feature Embedding <=SEP=> Caffe provides multimedia scientists and practitioners with a clean and modifiable framework for state-of-the-art deep learning algorithms and a collection of reference models. The framework is a BSD-licensed C++ library with Python and MATLAB bindings for training and deploying general-purpose convolutional neural networks and other deep models efficiently on commodity architectures. Caffe fits industry and internet-scale media needs by CUDA GPU computation, processing over 40 million images a day on a single K40 or Titan GPU (approx 2 ms per image). By separating model representation from actual implementation, Caffe allows experimentation and seamless switching among platforms for ease of development and deployment from prototyping machines to cloud environments.
 Caffe is maintained and developed by the Berkeley Vision and Learning Center (BVLC) with the help of an active community of contributors on GitHub. It powers ongoing research projects, large-scale industrial applications, and startup prototypes in vision, speech, and multimedia.

Learning From Semi-Supervised Weak-Label Data <=SEP=> Multi-label learning deals with data objects associated with multiple labels simultaneously. Previous studies typically assume that for each instance, the full set of relevant labels associated with each training instance is given. In many applications such as image annotation, however, it’s usually difficult to get the full label set for each instance and only a partial or even empty set of relevant labels is available. We call this kind of problem as ‘semi-supervised weak-label learning’ problem. In this work we propose the SSWL (Semi-Supervised Weak-Label) method to address this problem. Both instance similarity and label similarity are considered for the complement of missing labels. Ensemble of multiple models are utilized to improve the robustness when label information is insufficient. We formulate the objective as a bi-convex optimization problem with an efficient block coordinate descent algorithm. Experiments validate the effectiveness of SSWL.

A Future-Adaptable Password Scheme <=SEP=> Many authentication schemes depend on secret passwords. Unfortunately, the length and randomness of user-chosen passwords remain xed over time. In contrast, hardware improvements constantly give attackers increasing computational power. As a result, password schemes such as the traditional UNIX user-authentication system are failing with time. This paper discusses ways of building systems in which password security keeps up with hardware speeds. We formalize the properties desirable in a good password system, and show that the computational cost of any secure password scheme must increase as hardware improves. We present two algorithms with adaptable cost|eksblow sh, a block cipher with a purposefully expensive key schedule, and bcrypt, a related hash function. Failing a major breakthrough in complexity theory, these algorithms should allow password-based systems to adapt to hardware improvements and remain secure well into the future.

The importance of rehabilitation concerning upper extremity amputees: A Systematic review <=SEP=> OBJECTIVE
To evaluate and point out the importance of prosthetic rehabilitation of upper extremity.


METHODS
A systematic literature search was performed to identify studies concerning prosthetic rehabilitation in upper extremity. The PRISMA Statement 2009 was used to establish the study and the methodological quality was assessed.


RESULTS
The literature search identified 620 studies. Of these 620, 9 studies fulfilled the inclusion criteria and were included for data extraction. The studies pointed out the upper limb prosthetic rehabilitation protocols consist of general exercise programme, motor tasks, phantom exercises, Muscle Training System, edema control, functional activities, signal strengthening, prosthetic education exercises, neuromuscular reeducation, virtual image and virtual reality exercises.


CONCLUSIONS
The current systematic literature review has shown that the prosthetic rehabilitation seems promising especially for upper extremity amputees.

Comparing ability and self-report trait emotional intelligence , fluid intelligence , and personality traits in career decision <=SEP=> This study investigated the role of ability and trait emotional intelligence (EI), fluid intelligence, and personality traits in career decision-making self-efficacy, career indecision and indecisiveness. The Advanced Progressive Matrices, Big Five Questionnaire, Mayer-Salovey-Caruso Emotional Intelligence Test, Bar-On Emotional Intelligence Inventory, Trait Emotional Intelligence Questionnaire, Career Decision SelfEfficacy Scale: Short Form, Career Decision-Making Difficulties Questionnaire, and Indecisiveness Scale were administered to 194 Italian high school students. These results highlighted the potentially important role of self-reported EI in career decisions. 2014 Elsevier Ltd. All rights reserved.

Sleep deprivation-induced reduction in cortical functional response to serial subtraction. <=SEP=> Thirteen normal volunteers were studied with fMRI during arithmetic performance after a normal night of sleep and following sleep deprivation (SD). Aims included determining whether the prefrontal cortex (PFC) and the parietal lobe arithmetic areas are vulnerable to the effects of SD. After a normal night of sleep, activation localized to the bilateral PFC, parietal lobes and premotor areas. Following SD, activity in these regions decreased markedly, especially in the PFC. Performance also dropped. Data from the serial subtraction task are consistent with Horne's PFC vulnerability hypothesis but, based on this and other studies, we suggest the localized, functional effects of SD in the brain may vary, in part, with the specific cognitive task.

Efficient Discovery of Statistically Significant Association Rules <=SEP=> Searching statistically significant association rules is an important but neglected problem. Traditional association rules do not capture the idea of statistical dependence and the resulting rules can be spurious, while the most significant rules may be missing. This leads to erroneous models and predictions which often become expensive.The problem is computationally very difficult, because the significance is not a monotonic property. However, in this paper we prove several other properties, which can be used for pruning the search space. The properties are implemented in the StatApriori algorithm, which searches statistically significant, non-redundant association rules. Based on both theoretical and empirical observations, the resulting rules are very accurate compared to traditional association rules. In addition, StatApriori can work with extremely low frequencies, thus finding new interesting rules.

Four Text Classification Algorithms Compared on a Dutch Corpus <=SEP=> We describe an experiment in applying text classification algorithms to Dutch texts. Four well-known learning algorithms, Rocchio’s algorithm, the Simple Bayesian Classifier (SBC), the Sleeping Experts (SE) and Winnow were implemented. They were tested on a corpus of articles from the Dutch newspaper NRC, pre-classified into four categories. The algorithms are compared on learning speed and error rate. We also investigated the effect of discarding terms, using either a dynamic stoplist or the Winnow heuristic.

Domain Alignment with Triplets <=SEP=> Deep domain adaptation methods can reduce the distribution discrepancy by learning domain-invariant embedddings. However, these methods only focus on aligning the whole data distributions, without considering the classlevel relations among source and target images. Thus, a target embeddings of a bird might be aligned to source embeddings of an airplane. This semantic misalignment can directly degrade the classifier performance on the target dataset. To alleviate this problem, we present a similarity constrained alignment (SCA) method for unsupervised domain adaptation. When aligning the distributions in the embedding space, SCA enforces a similarity-preserving constraint to maintain class-level relations among the source and target images, i.e., if a source image and a target image are of the same class label, their corresponding embeddings are supposed to be aligned nearby, and vise versa. In the absence of target labels, we assign pseudo labels for target images. Given labeled source images and pseudo-labeled target images, the similarity-preserving constraint can be implemented by minimizing the triplet loss. With the joint supervision of domain alignment loss and similarity-preserving constraint, we train a network to obtain domain-invariant embeddings with two critical characteristics, intra-class compactness and inter-class separability. Extensive experiments conducted on the two datasets well demonstrate the effectiveness of SCA.

Adaptive Linked Data-Driven Web Components: Building Flexible and Reusable Semantic Web Interfaces - Building Flexible and Reusable Semantic Web Interfaces <=SEP=> Due to the increasing amount of Linked Data openly published on the Web, user-facing Linked Data Applications (LDAs) are gaining momentum. One of the major entrance barriers for Web developers to contribute to this wave of LDAs is the required knowledge of Semantic Web (SW) technologies such as the RDF data model and SPARQL query language. This paper presents an adaptive componentbased approach together with its open source implementation for creating flexible and reusable SW interfaces driven by Linked Data. Linked Data-driven (LD-R) Web components abstract the complexity of the underlying SW technologies in order to allow reuse of existing Web components in LDAs, enabling Web developers who are not experts in SW to develop interfaces that view, edit and browse Linked Data. In addition to the modularity provided by the LD-R components, the proposed RDF-based configuration method allows application assemblers to reshape their user interface for different use cases, by either reusing existing shared configurations or by creating their proprietary configurations.

Elimination of Residual Flux in Transformers by the Application of an Alternating Polarity DC Voltage Source <=SEP=> The purpose of core demagnetization is twofold: 1) to reduce the inrush currents when transformers are energized; and 2) to make sure that the frequency-response analysis (FRA) tests are consistent to avoid false diagnoses of damage during transportation. The significance of demagnetizing is presented on field measurements of an 80 MVA unit with FRA measurements. A new demagnetizer device with an alternating polarity dc voltage source is prototyped. Experimental verification of this prototype is presented for the demagnetization of transformers. A nearly complete demagnetization was observed in the laboratory for a small single-phase isolation transformer. The method proposed in this paper is applied to three-phase transformers with different core configurations and connections. Topologically correct modeling and numerical simulations confirm the full demagnetization of all branches of three-phase (three- and five-limb) transformer cores. Inrush current measurements and FRA plots before and after demagnetization confirm the effectiveness of the process.

SensoTube: A Scalable Hardware Design Architecture for Wireless Sensors and Actuators Networks Nodes in the Agricultural Domain <=SEP=> Wireless Sensor and Actuators Networks (WSANs) constitute one of the most challenging technologies with tremendous socio-economic impact for the next decade. Functionally and energy optimized hardware systems and development tools maybe is the most critical facet of this technology for the achievement of such prospects. Especially, in the area of agriculture, where the hostile operating environment comes to add to the general technological and technical issues, reliable and robust WSAN systems are mandatory. This paper focuses on the hardware design architectures of the WSANs for real-world agricultural applications. It presents the available alternatives in hardware design and identifies their difficulties and problems for real-life implementations. The paper introduces SensoTube, a new WSAN hardware architecture, which is proposed as a solution to the various existing design constraints of WSANs. The establishment of the proposed architecture is based, firstly on an abstraction approach in the functional requirements context, and secondly, on the standardization of the subsystems connectivity, in order to allow for an open, expandable, flexible, reconfigurable, energy optimized, reliable and robust hardware system. The SensoTube implementation reference model together with its encapsulation design and installation are analyzed and presented in details. Furthermore, as a proof of concept, certain use cases have been studied in order to demonstrate the benefits of migrating existing designs based on the available open-source hardware platforms to SensoTube architecture.

DivQ: diversification for keyword search over structured databases <=SEP=> Keyword queries over structured databases are notoriously ambiguous. No single interpretation of a keyword query can satisfy all users, and multiple interpretations may yield overlapping results. This paper proposes a scheme to balance the relevance and novelty of keyword search results over structured databases. Firstly, we present a probabilistic model which effectively ranks the possible interpretations of a keyword query over structured data. Then, we introduce a scheme to diversify the search results by re-ranking query interpretations, taking into account redundancy of query results. Finally, we propose α-nDCG-W and WS-recall, an adaptation of α-nDCG and S-recall metrics, taking into account graded relevance of subtopics. Our evaluation on two real-world datasets demonstrates that search results obtained using the proposed diversification algorithms better characterize possible answers available in the database than the results of the initial relevance ranking.

EMARS: Efficient Management and Allocation of Resources in Serverless <=SEP=> We introduce EMARS, an efficient resource management system for serverless cloud computing frameworks with the goal to enhance resource (focus on memory) allocation among containers. We have built our prototype on top of an open-source serverless platform, OpenLambda. It is based upon application workloads and serverless functions' memory needs. As a background motivation we analyzed the latencies and memory requirements of functions running on AWS lambda. The memory limits also lead to variations in number of containers spawned on OpenLambda. We use memory limit settings to propose a model of predictive efficient memory management.

Neural Embeddings of Graphs in Hyperbolic Space <=SEP=> ABSTRACT Neural embeddings have been used with great success in Natural Language Processing (NLP). They provide compact representations that encapsulate word similarity and attain state-of-the-art performance in a range of linguistic tasks. The success of neural embeddings has prompted signi cant amounts of research into applications in domains other than language. One such domain is graph-structured data, where embeddings of vertices can be learned that encapsulate vertex similarity and improve performance on tasks including edge prediction and vertex labelling. For both NLP and graph based tasks, embeddings have been learned in highdimensional Euclidean spaces. However, recent work has shown that the appropriate isometric space for embedding complex networks is not the at Euclidean space, but negatively curved, hyperbolic space. We present a new concept that exploits these recent insights and propose learning neural embeddings of graphs in hyperbolic space. We provide experimental evidence that embedding graphs in their natural geometry signi cantly improves performance on downstream tasks for several real-world public datasets.

Simulink based performance analysis of lead acid batteries with the variation of load current and temperature <=SEP=> Storage device in renewable energy applications plays an important role for efficient operation. For solar PV applications, storage device is an integral part during post sun shine hours to supply power to the load. For Solar Home System (SHS), approximately 60% of the total cost is due to storage battery and the battery needs replacement in every five years or so. If the performance of the battery can be enhanced, not only the cost of energy will come down appreciably but the reliability of the system will improve. Battery is needed in PV system to store the excess energy from the sun and release the power to the system again when it is needed. In other words, to enhance a system performance we have to increase the battery performance. In this paper, the parameters which play important role for optimum performance has been studied. The major two parameters that affect the battery performance are load current and temperature. It has been found that, when the discharge current is high, battery capacity reduces and the vice-versa. Again in case of temperature, when temperature increases more than the rated standard value of 25°C, capacity increases but the life time of the battery decreases. On the other hand, when the ambient temperature goes down, capacity of battery decreases but lifetime of the battery is increased. To analyse the effect of temperature and load current on battery performance, the standard equivalent circuit of battery is considered first and then the equivalent circuit is simulated using Simulink under MATLAB environment.

The Design and Implementation of CoGaDB: A Column-oriented GPU-accelerated DBMS <=SEP=> Nowadays, the performance of processors is primarily bound by a fixed energy budget, the power wall. This forces hardware vendors to optimize processors for specific tasks, which leads to an increasingly heterogeneous hardware landscape. Although efficient algorithms for modern processors such as GPUs are heavily investigated, we also need to prepare the database optimizer to handle computations on heterogeneous processors. GPUs are an interesting base for case studies, because they already offer many difficulties we will face tomorrow. In this paper, we present CoGaDB, a main-memory DBMS with built-in GPU acceleration, which is optimized for OLAP workloads. CoGaDB uses the self-tuning optimizer framework HyPE to build a hardware-oblivious optimizer, which learns cost models for database operators and efficiently distributes a workload on available processors. Furthermore, CoGaDB implements efficient algorithms on CPU and GPU and efficiently supports star joins. We show in this paper, how these novel techniques interact with each other in a single system. Our evaluation shows that CoGaDB quickly adapts to the underlying hardware by increasing the accuracy of its cost models at runtime.

Monte Carlo Tree Search for Asymmetric Trees <=SEP=> We present an extension of Monte Carlo Tree Search (MCTS) that strongly increases its efficiency for trees with asymmetry and/or loops. Asymmetric termination of search trees introduces a type of uncertainty for which the standard upper confidence bound (UCB) formula does not account. Our first algorithm (MCTS-T), which assumes a non-stochastic environment, backs-up tree structure uncertainty and leverages it for exploration in a modified UCB formula. Results show vastly improved efficiency in a well-known asymmetric domain in which MCTS performs arbitrarily bad. Next, we connect the ideas about asymmetric termination to the presence of loops in the tree, where the same state appears multiple times in a single trace. An extension to our algorithm (MCTS-T+), which in addition to nonstochasticity assumes full state observability, further increases search efficiency for domains with loops as well. Benchmark testing on a set of OpenAI Gym and Atari 2600 games indicates that our algorithms always perform better than or at least equivalent to standard MCTS, and could be first-choice tree search algorithms for non-stochastic, fully-observable environments.

Nationalism and patriotism: national identification and out-group rejection. <=SEP=> It is argued that the differentiation between nationalism and patriotism proposed in the literature can be seen as analogous to judgments based on different types of comparisons: intergroup comparisons with other nations are associated with intergroup behaviour that corresponds to nationalism, whereas temporal or standard comparisons are linked with behaviour that corresponds to patriotism. Four studies (N = 103, 107, 96 and 105) conducted in Germany and Britain examined the hypothesis that national identification and in-group evaluation only show a reliable relationship with out-group rejection under an intergroup comparison orientation. Participants were primed with either an intergroup comparison, a temporal comparison or no explicit comparison orientation. A subsequent questionnaire assessed in-group (own country) identification, in-group evaluation (i.e. national pride) and rejection of national out-groups. Across all four studies, both in-group identification and in-group evaluation show a stronger correlation with out-group derogation if participants were primed with an intergroup comparison orientation compared to temporal and control conditions. Results are discussed with regard to nationalism and patriotism as well as Hinkle and Brown's (1990) model on relational vs. autonomous orientations.

Open Logo Detection Challenge <=SEP=> Existing logo detection benchmarks consider artificial deployment scenarios by assuming that large training data with fine-grained bounding box annotations for each class are available for model training. Such assumptions are often invalid in realistic logo detection scenarios where new logo classes come progressively and require to be detected with little or none budget for exhaustively labelling fine-grained training data for every new class. Existing benchmarks are thus unable to evaluate the true performance of a logo detection method in realistic and open deployments. In this work, we introduce a more realistic and challenging logo detection setting, called Open Logo Detection. Specifically, this new setting assumes fine-grained labelling only on a small proportion of logo classes whilst the remaining classes have no labelled training data to simulate the open deployment. We further create an open logo detection benchmark, called QMUL-OpenLogo, to promote the investigation of this new challenge. QMUL-OpenLogo contains 27,083 images from 352 logo classes, built by aggregating/refining 7 existing datasets and establishing an open logo detection evaluation protocol. To address this challenge, we propose a Context Adversarial Learning (CAL) approach to synthesising training data with coherent logo instance appearance against diverse background context for enabling more effective optimisation of contemporary deep learning detection models. Experiments show the performance advantage of CAL over existing state-of-the-art alternative methods on the more realistic and challenging open logo benchmark. The QMUL-OpenLogo benchmark is publicly available at https://qmul-openlogo.github.io/.

Weighted Coverage based Reviewer Assignment <=SEP=> Peer reviewing is a standard process for assessing the quality of submissions at academic conferences and journals. A very important task in this process is the assignment of reviewers to papers. However, achieving an appropriate assignment is not easy, because all reviewers should have similar load and the subjects of the assigned papers should be consistent with the reviewers' expertise. In this paper, we propose a generalized framework for fair reviewer assignment. We first extract the domain knowledge from the reviewers' published papers and model this knowledge as a set of topics. Then, we perform a group assignment of reviewers to papers, which is a generalization of the classic Reviewer Assignment Problem (RAP), considering the relevance of the papers to topics as weights. We study a special case of the problem, where reviewers are to be found for just one paper (Journal Assignment Problem) and propose an exact algorithm which is fast in practice, as opposed to brute-force solutions. For the general case of having to assign multiple papers, which is too hard to be solved exactly, we propose a greedy algorithm that achieves a 1/2-approximation ratio compared to the exact solution. This is a great improvement compared to the 1/3-approximation solution proposed in previous work for the simpler coverage-based reviewer assignment problem, where there are no weights on topics. We theoretically prove the approximation bound of our solution and experimentally show that it is superior to the current state-of-the-art.

Fine-grained change detection in structured text documents <=SEP=> Detecting and understanding changes between document revisions is an important task. The acquired knowledge can be used to classify the nature of a new document revision or to support a human editor in the review process. While purely textual change detection algorithms offer fine-grained results, they do not understand the syntactic meaning of a change. By representing structured text documents as XML documents we can apply tree-to-tree correction algorithms to identify the syntactic nature of a change.
 Many algorithms for change detection in XML documents have been propsed but most of them focus on the intricacies of generic XML data and emphasize speed over the quality of the result. Structured text requires a change detection algorithm to pay close attention to the content in text nodes, however, recent algorithms treat text nodes as black boxes.
 We present an algorithm that combines the advantages of the purely textual approach with the advantages of tree-to-tree change detection by redistributing text from non-overlapping common substrings to the nodes of the trees. This allows us to not only spot changes in the structure but also in the text itself, thus achieving higher quality and a fine-grained result in linear time on average. The algorithm is evaluated by applying it to the corpus of structured text documents that can be found in the English Wikipedia.

Multimodal imaging of optic disc drusen. <=SEP=> PURPOSE
To evaluate optic disc drusen, extracellular protein deposits known to contain numerous aggregates of mitochondria, using multimodal modalities featuring optical coherence tomography (OCT) and autofluorescence imaging.


DESIGN
Retrospective observational case series.


METHODS
Eyes with optic nerve drusen were examined with enhanced depth imaging (EDI)-OCT, swept source OCT, and fundus autofluorescence using a fundus camera.


RESULTS
Twenty-six eyes of 15 patients with optic disc drusen were evaluated. EDI-OCT and swept source OCT showed multiple optic disc drusen at different levels; most were located immediately anterior to the lamina cribrosa. The drusen were ovoid regions of lower reflectivity that were bordered by hyperreflective material, and in 12 eyes (46.2%) there were internal hyperreflective foci. The mean diameter of the optic disc drusen as measured in OCT images was 686.8 (standard deviation ± 395.2) μm. There was a significant negative correlation between the diameter of the optic disc drusen and the global retinal nerve fiber layer thickness (r = -0.61, P = .001). There was a significant negative correlation between proportion of the optic disc drusen area occupied by optic nerve drusen as detected by autofluorescence imaging and the global retinal nerve fiber layer thickness (r = -0.63, P = .001).


CONCLUSIONS
Deeper-penetration OCT imaging demonstrated the internal characteristics of optic disc drusen and their relationship with the lamina cribrosa in vivo. This study also showed that both the larger the drusen and the more area of the optic canal occupied by drusen, the greater the associated retinal nerve fiber layer abnormalities.

Parallel Restarted SGD with Faster Convergence and Less Communication: Demystifying Why Model Averaging Works for Deep Learning <=SEP=> In distributed training of deep neural networks, parallel minibatch SGD is widely used to speed up the training process by using multiple workers. It uses multiple workers to sample local stochastic gradient in parallel, aggregates all gradients in a single server to obtain the average, and update each worker’s local model using a SGD update with the averaged gradient. Ideally, parallel mini-batch SGD can achieve a linear speed-up of the training time (with respect to the number of workers) compared with SGD over a single worker. However, such linear scalability in practice is significantly limited by the growing demand for gradient communication as more workers are involved. Model averaging, which periodically averages individual models trained over parallel workers, is another common practice used for distributed training of deep neural networks since (Zinkevich et al. 2010) (McDonald, Hall, and Mann 2010). Compared with parallel mini-batch SGD, the communication overhead of model averaging is significantly reduced. Impressively, tremendous experimental works have verified that model averaging can still achieve a good speed-up of the training time as long as the averaging interval is carefully controlled. However, it remains a mystery in theory why such a simple heuristic works so well. This paper provides a thorough and rigorous theoretical study on why model averaging can work as well as parallel mini-batch SGD with significantly less communication overhead. Introduction Consider the distributed training of deep neural networks over multiple workers (Dean et al. 2012), where all workers can access all or partial training data and aim to find a common model that yields the minimum training loss. Such a scenario can be modeled as the following distributed parallel non-convex optimization min x∈Rm f(x) ∆ = 1 N N ∑

Algorithmic Aspects of Vertex Elimination on Graphs <=SEP=> We consider a graph-theoretic elimination process which is related to performing Gaussian elimination on sparse symmetric positive definite systems of linear equations. We give a new linear-time algorithm to calculate the fill-in produced by any elimination ordering, and we give two new related algorithms for finding orderings with special properties. One algorithm, based on breadth-first search, finds a perfect elimination ordering, if any exists, in O(n + e) time, if the problem graph has n vertices and e edges. An extension of this algorithm finds a minimal (but not necessarily minimum) ordering in O(ne) time. We conjecture that the problem of finding a minimum ordering is

An Algorithm to Find Overlapping Community Structure in Networks <=SEP=> Abstract. Recent years have seen the development of many graph clustering algorithms, which can identify community structure in networks. The vast majority of these only find disjoint communities, but in many real-world networks communities overlap to some extent. We present a new algorithm for discovering overlapping communities in networks, by extending Girvan and Newman’s well-known algorithm based on the betweenness centrality measure. Like the original algorithm, ours performs hierarchical clustering — partitioning a network into any desired number of clusters — but allows them to overlap. Experiments confirm good performance on randomly generated networks based on a known overlapping community structure, and interesting results have also been obtained on a range of real-world networks.