pwc-archive/papers-with-abstracts huggingface dataset:
Fine-tuning Large Language Models for Adaptive Machine Translation
Authors: Yasmin Moslem, Rejwanul Haque, Andy Way
Date: 2023-12-20 00:00:00
Tasks: In-Context Learning, Language Modelling, Large Language Model, Machine Translation, Translation

This paper presents the outcomes of fine-tuning Mistral 7B, a general-purpose large language model (LLM), for adaptive machine translation (MT). The fine-tuning process involves utilising a combination of zero-shot and one-shot translation prompts within the medical domain. The primary objective is to enhance real-time adaptive MT capabilities of Mistral 7B, enabling it to adapt translations to the required domain at inference time. The results, particularly for Spanish-to-English MT, showcase the efficacy of the fine-tuned model, demonstrating quality improvements in both zero-shot and one-shot translation scenarios, surpassing Mistral 7B's baseline performance. Notably, the fine-tuned Mistral outperforms ChatGPT "gpt-3.5-turbo" in zero-shot translation while achieving comparable one-shot translation quality. Moreover, the zero-shot translation of the fine-tuned Mistral matches NLLB 3.3B's performance, and its one-shot translation quality surpasses that of NLLB 3.3B. These findings emphasise the significance of fine-tuning efficient LLMs like Mistral 7B to yield high-quality zero-shot translations comparable to task-oriented models like NLLB 3.3B. Additionally, the adaptive gains achieved in one-shot translation are comparable to those of commercial LLMs such as ChatGPT. Our experiments demonstrate that, with a relatively small dataset of 20,000 segments that incorporate a mix of zero-shot and one-shot prompts, fine-tuning significantly enhances Mistral's in-context learning ability, especially for real-time adaptive MT.
Turning English-centric LLMs Into Polyglots: How Much Multilinguality Is Needed?
Authors: Tannon Kew, Florian Schottmann, Rico Sennrich
Date: 2023-12-20 00:00:00
Tasks: Cross-Lingual Transfer

The vast majority of today's large language models (LLMs) are English-centric, having been pretrained predominantly on English text. Yet, in order to meet user expectations, models need to be able to respond appropriately in multiple languages once deployed in downstream applications. This requires strong cross-lingual transfer abilities. In this work, we investigate the minimal amount of multilinguality required during finetuning to elicit cross-lingual generalisation in English-centric LLMs. In experiments across four LLMs, we find that multilingual instruction tuning with as few as two to three languages is both necessary and sufficient to elicit effective cross-lingual generalisation, with the limiting factor being the degree to which a target language is seen during pretraining. Evaluations on five different tasks further reveal that multilingual instruction tuning is most beneficial for generative tasks that assume input/output language agreement, such as in chat settings, while being of less importance for highly structured classification-style tasks. Our code and data is available at https://github.com/ZurichNLP/multilingual-instruction-tuning.
Is post-editing really faster than human translation?
Authors: Silvia Terribile
Date: 2023-12-19 00:00:00
Tasks: Machine Translation, Translation

Time efficiency is paramount for the localisation industry, which demands ever-faster turnaround times. However, translation speed is largely underresearched, and there is a lack of clarity about how language service providers (LSPs) can evaluate the performance of their post-editing (PE) and human translation (HT) services. This study constitutes the first large-scale investigation of translation and revision speed in HT and in the PE of neural machine translation, based on real-world data from an LSP. It uses an exploratory data analysis approach to investigate data for 90 million words translated by 879 linguists across 11 language pairs, over 2.5 years. The results of this research indicate that (a) PE is usually but not always faster than HT; (b) average speed values may be misleading; (c) translation speed is highly variable; and (d) edit distance cannot be used as a proxy for post-editing productivity, because it does not correlate strongly with speed.
Building a Llama2-finetuned LLM for Odia Language Utilizing Domain Knowledge Instruction Set
Authors: Guneet Singh Kohli, Shantipriya Parida, Sambit Sekhar, Samirit Saha, Nipun B Nair, Parul Agarwal, Sonal Khosla, Kusumlata Patiyal, Debasish Dhal
Date: 2023-12-19 00:00:00

Building LLMs for languages other than English is in great demand due to the unavailability and performance of multilingual LLMs, such as understanding the local context. The problem is critical for low-resource languages due to the need for instruction sets. In a multilingual country like India, there is a need for LLMs supporting Indic languages to provide generative AI and LLM-based technologies and services to its citizens. This paper presents our approach of i) generating a large Odia instruction set, including domain knowledge data suitable for LLM fine-tuning, and ii) building a Llama2-finetuned model tailored for enhanced performance in the Odia domain. The proposed work will help researchers build an instruction set and LLM, particularly for Indic languages. We will release the model and instruction set for the public for research and noncommercial purposes.
An Empirical study of Unsupervised Neural Machine Translation: analyzing NMT output, model's behavior and sentences' contribution
Authors: Isidora Chara Tourni, Derry Wijaya
Date: 2023-12-19 00:00:00
Tasks: Low Resource Neural Machine Translation, Low-Resource Neural Machine Translation, Machine Translation, NMT, Semantic Similarity, Semantic Textual Similarity, Translation

Unsupervised Neural Machine Translation (UNMT) focuses on improving NMT results under the assumption there is no human translated parallel data, yet little work has been done so far in highlighting its advantages compared to supervised methods and analyzing its output in aspects other than translation accuracy. We focus on three very diverse languages, French, Gujarati, and Kazakh, and train bilingual NMT models, to and from English, with various levels of supervision, in high- and low- resource setups, measure quality of the NMT output and compare the generated sequences' word order and semantic similarity to source and reference sentences. We also use Layer-wise Relevance Propagation to evaluate the source and target sentences' contribution to the result, expanding the findings of previous works to the UNMT paradigm.
Users Approach on Providing Feedback for Smart Home Devices
Authors: Santhosh Pogaku
Date: 2023-12-19 00:00:00

Smart Home technology has accomplished extraordinary interest in making individuals' lives more straightforward and more relaxing as of late. Technology as of late brought about delivering numerous savvy and refined frameworks which advanced clever living innovation. In this paper, we will be investigating the behavioural intention of user's approach on providing feedback for smart home devices. We will be conducting an online survey for sample of three to five students selected by simple random sampling to study the user's motto for giving feedback on smart home devices and their expectations. We have observed that most users are ready to share their feedback on smart home devices actively to improvise the service and quality of the product to fulfill the user needs and make their lives easier.
Tuning the activation function to optimize the forecast horizon of a reservoir computer
Authors: Lauren A. Hurley, Juan G. Restrepo, Sean E. Shaheen
Date: 2023-12-20 00:00:00
Tasks: Hyperparameter Optimization

Reservoir computing is a machine learning framework where the readouts from a nonlinear system (the reservoir) are trained so that the output from the reservoir, when forced with an input signal, reproduces a desired output signal. A common implementation of reservoir computers is to use a recurrent neural network as the reservoir. The design of this network can have significant effects on the performance of the reservoir computer. In this paper we study the effect of the node activation function on the ability of reservoir computers to learn and predict chaotic time series. We find that the Forecast Horizon (FH), the time during which the reservoir's predictions remain accurate, can vary by an order of magnitude across a set of 16 activation functions used in machine learning. By using different functions from this set, and by modifying their parameters, we explore whether the entropy of node activation levels or the curvature of the activation functions determine the predictive ability of the reservoirs. We find that the FH is low when the activation function is used in a region where it has low curvature, and a positive correlation between curvature and FH. For the activation functions studied we find that the largest FH generally occurs at intermediate levels of the entropy of node activation levels. Our results show that the performance of reservoir computers is very sensitive to the activation function shape. Therefore, modifying this shape in hyperparameter optimization algorithms can lead to improvements in reservoir computer performance.
Partially factorized variational inference for high-dimensional mixed models
Authors: Max Goplerud, Omiros Papaspiliopoulos, Giacomo Zanella
Date: 2023-12-20 00:00:00
Tasks: Uncertainty Quantification, Variational Inference

While generalized linear mixed models are a fundamental tool in applied statistics, many specifications, such as those involving categorical factors with many levels or interaction terms, can be computationally challenging to estimate due to the need to compute or approximate high-dimensional integrals. Variational inference is a popular way to perform such computations, especially in the Bayesian context. However, naive use of such methods can provide unreliable uncertainty quantification. We show that this is indeed the case for mixed models, proving that standard mean-field variational inference dramatically underestimates posterior uncertainty in high-dimensions. We then show how appropriately relaxing the mean-field assumption leads to methods whose uncertainty quantification does not deteriorate in high-dimensions, and whose total computational cost scales linearly with the number of parameters and observations. Our theoretical and numerical results focus on mixed models with Gaussian or binomial likelihoods, and rely on connections to random graph theory to obtain sharp high-dimensional asymptotic analysis. We also provide generic results, which are of independent interest, relating the accuracy of variational inference to the convergence rate of the corresponding coordinate ascent algorithm that is used to find it. Our proposed methodology is implemented in the R package, see https://github.com/mgoplerud/vglmer . Numerical results with simulated and real data examples illustrate the favourable computation cost versus accuracy trade-off of our approach compared to various alternatives.
Heterogeneous Transfer Learning for Building High-Dimensional Generalized Linear Models with Disparate Datasets
Authors: Ruzhang Zhao, Prosenjit Kundu, Arkajyoti Saha, Nilanjan Chatterjee
Date: 2023-12-20 00:00:00
Tasks: Transfer Learning

Development of comprehensive prediction models are often of great interest in many disciplines of science, but datasets with information on all desired features often have small sample sizes. We describe a transfer learning approach for building high-dimensional generalized linear models using data from a main study with detailed information on all predictors and an external, potentially much larger, study that has ascertained a more limited set of predictors. We propose using the external dataset to build a reduced model and then "transfer" the information on underlying parameters for the analysis of the main study through a set of calibration equations which can account for the study-specific effects of design variables. We then propose a penalized generalized method of moment framework for inference and a one-step estimation method that could be implemented using standard glmnet package. We develop asymptotic theory and conduct extensive simulation studies to investigate both predictive performance and post-selection inference properties of the proposed method. Finally, we illustrate an application of the proposed method for the development of risk models for five common diseases using the UK Biobank study, combining information on low-dimensional risk factors and high throughout proteomic biomarkers.
Categorical, Ratio, and Professorial Data: The Case for Reciprocal Rank
Authors: Alistair Moffat
Date: 2023-12-20 00:00:00
Tasks: Retrieval

Search engine results pages are usually abstracted as binary relevance vectors and hence are categorical data, meaning that only a limited set of operations is permitted, most notably tabulation of occurrence frequencies, with determination of medians and averages not possible. To compare retrieval systems it is thus usual to make use of a categorical-to-numeric effectiveness mapping. A previous paper has argued that any desired categorical-to-numeric mapping may be used, provided only that there is an argued connection between each category of SERP and the score that is assigned to that category by the mapping. Further, once that plausible connection has been established, then the mapped values can be treated as real-valued observations on a ratio scale, allowing the computation of averages. This article is written in support of that point of view, and to respond to ongoing claims that SERP scores may only be averaged if very restrictive conditions are imposed on the effectiveness mapping.
Perception Test: A Diagnostic Benchmark for Multimodal Video Models
Authors: Viorica Pătrăucean, Lucas Smaira, Ankush Gupta, Adrià Recasens Continente, Larisa Markeeva, Dylan Banarse, Skanda Koppula, Joseph Heyward, Mateusz Malinowski, Yi Yang, Carl Doersch, Tatiana Matejovicova, Yury Sulsky, Antoine Miech, Alex Frechette, Hanna Klimczak, Raphael Koster, Junlin Zhang, Stephanie Winkler, Yusuf Aytar, Simon Osindero, Dima Damen, Andrew Zisserman, João Carreira
Date: 2023-05-23 00:00:00
Proceeding: NeurIPS 2023 11
Tasks: Diagnostic, Grounded Video Question Answering, Object Tracking, Point Tracking, Sound Event Localization and Detection, Temporal Action Localization, Video Question Answering

We propose a novel multimodal video benchmark - the Perception Test - to evaluate the perception and reasoning skills of pre-trained multimodal models (e.g. Flamingo, SeViLA, or GPT-4). Compared to existing benchmarks that focus on computational tasks (e.g. classification, detection or tracking), the Perception Test focuses on skills (Memory, Abstraction, Physics, Semantics) and types of reasoning (descriptive, explanatory, predictive, counterfactual) across video, audio, and text modalities, to provide a comprehensive and efficient evaluation tool. The benchmark probes pre-trained models for their transfer capabilities, in a zero-shot / few-shot or limited finetuning regime. For these purposes, the Perception Test introduces 11.6k real-world videos, 23s average length, designed to show perceptually interesting situations, filmed by around 100 participants worldwide. The videos are densely annotated with six types of labels (multiple-choice and grounded video question-answers, object and point tracks, temporal action and sound segments), enabling both language and non-language evaluations. The fine-tuning and validation splits of the benchmark are publicly available (CC-BY license), in addition to a challenge server with a held-out test split. Human baseline results compared to state-of-the-art video QA models show a substantial gap in performance (91.4% vs 46.2%), suggesting that there is significant room for improvement in multimodal video understanding. Dataset, baseline code, and challenge server are available at https://github.com/deepmind/perception_test
Think Locally, Act Globally: Perfectly Balanced Graph Partitioning
Authors: Peter Sanders, Christian Schulz
Date: 2012-10-01 00:00:00
Tasks: graph partitioning

We present a novel local improvement scheme for the perfectly balanced graph partitioning problem. This scheme encodes local searches that are not restricted to a balance constraint into a model allowing us to find combinations of these searches maintaining balance by applying a negative cycle detection algorithm. We combine this technique with an algorithm to balance unbalanced solutions and integrate it into a parallel multi-level evolutionary algorithm, KaFFPaE, to tackle the problem. Overall, we obtain a system that is fast on the one hand and on the other hand is able to improve or reproduce most of the best known perfectly balanced partitioning results ever reported in the literature.
Distributed Evolutionary Graph Partitioning
Authors: Peter Sanders, Christian Schulz
Date: 2011-10-03 00:00:00
Tasks: graph partitioning

We present a novel distributed evolutionary algorithm, KaFFPaE, to solve the Graph Partitioning Problem, which makes use of KaFFPa (Karlsruhe Fast Flow Partitioner). The use of our multilevel graph partitioner KaFFPa provides new effective crossover and mutation operators. By combining these with a scalable communication protocol we obtain a system that is able to improve the best known partitioning results for many inputs in a very short amount of time. For example, in Walshaw's well known benchmark tables we are able to improve or recompute 76% of entries for the tables with 1%, 3% and 5% imbalance.
Scene Text Recognition using Higher Order Language Priors
Authors: Anand Mishra, Karteek Alahari, C. V. Jawahar
Date: 2012-09-01 00:00:00
Proceeding: BMVC 2012 - Electronic Proceedings of the British Machine Vision Conference 2012 2012 9
Tasks: Scene Text Recognition

The problem of recognizing text in images taken in the wild has gained significant attention from the computer vision community in recent years. Contrary to recognition of printed documents, recognizing scene text is a challenging problem. We focus on the problem of recognizing text extracted from natural scene images and the web. Significant attempts have been made to address this problem in the recent past. However, many of these works benefit from the availability of strong context, which naturally limits their applicability. In this work we present a framework that uses a higher order prior computed from an English dictionary to recognize a word, which may or may not be a part of the dictionary. We show experimental results on publicly available datasets. Furthermore, we introduce a large challenging word dataset with five thousand words to evaluate various steps of our method exhaustively. The main contributions of this work are: (1) We present a framework, which incorporates higher order statistical language models to recognize words in an unconstrained manner (i.e. we overcome the need for restricted word lists, and instead use an English dictionary to compute the priors). (2) We achieve significant improvement (more than 20%) in word recognition accuracies without using a restricted word list. (3) We introduce a large word recognition dataset (atleast 5 times larger than other public datasets) with character level annotation and benchmark it.
Non-Parallel Training Approach for Emotional Voice Conversion Using CycleGAN
Authors: Mohamed Elsayed, Sama Hadhoud, Alaa Elsetohy, Menna Osman, Walid Gomaa
Date: 2023-11-01 00:00:00
Proceeding: 20th International Conference on Informatics in Control, Automation and Robotics 2023 11
Tasks: Voice Conversion

The focus of this research is proposing a nonparallel emotional voice conversion for Egyptian Arabic speech. This method aims to change emotion-related features of a speech signal without changing its lexical content or speaker identity. We relied on the assumption that any speech signal can be divided into content and style code and the conversion between different emotion domains is done by combining the target style code with the content code of the input speech signal. We evaluated the model using an Egyptian Arabic dataset covering two emotion domains and the conversion results were successful depending on a survey conducted on random people. Our purpose is to produce a state-of-the-art pre-trained model as it will be an unprecedented model in the Egyptian Arabic language as far as we are concerned.
Privacy-preserving transactive energy systems: Key topics and open research challenges
Authors: Daniel Gerbi Duguma, Juliana Zhang, Meysam Aboutalebi, Shiliang Zhang, Catherine Banet, Cato Bjørkli, Chinmayi Baramashetru, Frank Eliassen, HUI ZHANG, Jonathan Muringani, Josef Noll, Knut Inge Fostervold, Lars Böcker, Lee Andrew Bygrave, Matin Bagherpour, Maunya Doroudi Moghadam, Olaf Owe, Poushali Sengupta, Roman Vitenberg, Sabita Maharjan, Thiago Garrett, Yushuai Li, Zhengyu Shan
Date: 2023-12-17 00:00:00
Tasks: energy management, energy trading, Management, Privacy Preserving

This manuscript aims to formalize and conclude the discussions initiated during the PriTEM workshop 22-23 March 2023. We present important ideas and discussion topics in the context of transactive energy systems. Moreover, the conclusions from the discussions articulate potential aspects to be explored in future studies on transactive energy management. Particularly, these conclusions cover research topics in energy technology and energy informatics, energy law, data law, energy market and socio-psychology that are relevant to the seamless integration of renewable energy resources and the transactive energy systems-in smart microgrids-focusing on distributed frameworks such as peer-to-peer (P2P) energy trading. We clarify issues, identify barriers, and suggest possible solutions to open questions in diversified topics, such as block-chain interoperability, consumer privacy and data sharing, and participation incentivization. Furthermore, we also elaborate challenges associated with cross-disciplinary collaboration and coordination for transactive energy systems, and enumerate the lessons learned from our work so far.
Democratize with Care: The need for fairness specific features in user-interface based open source AutoML tools
Authors: Sundaraparipurnan Narayanan
Date: 2023-12-16 00:00:00
Tasks: AutoML, Fairness

AI is increasingly playing a pivotal role in businesses and organizations, impacting the outcomes and interests of human users. Automated Machine Learning (AutoML) streamlines the machine learning model development process by automating repetitive tasks and making data-driven decisions, enabling even non-experts to construct high-quality models efficiently. This democratization allows more users (including non-experts) to access and utilize state-of-the-art machine-learning expertise. However, AutoML tools may also propagate bias in the way these tools handle the data, model choices, and optimization approaches adopted. We conducted an experimental study of User-interface-based open source AutoML tools (DataRobot, H2O Studio, Dataiku, and Rapidminer Studio) to examine if they had features to assist users in developing fairness-aware machine learning models. The experiments covered the following considerations for the evaluation of features: understanding use case context, data representation, feature relevance and sensitivity, data bias and preprocessing techniques, data handling capabilities, training-testing split, hyperparameter handling, and constraints, fairness-oriented model development, explainability and ability to download and edit models by the user. The results revealed inadequacies in features that could support in fairness-aware model development. Further, the results also highlight the need to establish certain essential features for promoting fairness in AutoML tools.