wcy1122/Long-TTS-Eval huggingface dataset:
Subsequently, we carry out a detailed examination of specific capabilities, including general visual perception, document understanding, multilingual recognition in images, video comprehension, and agent abilities. Finally, we present an ablation study to investigate several key components of our approach. Compare to SOTAs We evaluate the visual capabilities of our model through various visual benchmarks, video tasks, and agent-based assessments. Qwen2-VL demonstrates highly competitive performance at the same scale, achieving new state-of-the-art (SoTA) results. Overall, our 72B model consistently delivers top-tier performance across most evaluation metrics, frequently surpassing even closed-source models such as GPT-4o and Claude 3.5-Sonnet. Notably, it exhibits a significant advantage in document understanding tasks. However, in the MMMU benchmark, our model still lags behind GPT-4o to some extent, indicating that Qwen2-VL-72B has room for improvement when handling more complex and challenging problem sets. Quantitative Results In this section, we present an extensive evaluation of the Qwen2-VL series across an array of datasets, offering a comprehensive understanding of the model's capabilities in various aspects. General Visual Question Answering To rigorously assess our models' capabilities in general visual question answering tasks, we conduct extensive evaluations across a diverse array of state-of-the-art benchmarks: RealWorldQA, MMStar, MMVet, MMT-Bench, MMBench, MMbench-1.1, MME, and HallusionBench. The Qwen2-VL series exhibits exceptional performance across these benchmarks, with the 72B model consistently achieving or surpassing state-of-the-art results, while the 7B and 2B variants also demonstrate robust capabilities. On RealWorldQA, which evaluates real-world spatial comprehension, Qwen2-VL-72B achieves a score of 77.8, surpassing both the previous state-of-the-art (72.2) and formidable baselines such as GPT-4o (75.4), thus demonstrating superior understanding of physical environments. For MMStar, a benchmark designed to assess genuine multimodal capabilities through visually indispensable samples, Qwen2-VL-72B attains 68.3, outperforming the previous best of 67.1 and highlighting its proficiency in integrating visual and textual information. On MMVet, which evaluates the integration of core vision-language capabilities across 16 complex multimodal tasks, Qwen2-VL-72B achieves a remarkable 74.0, significantly outperforming strong competitors including GPT-4V (67.5) and showcasing its versatility in addressing diverse multimodal challenges.
Denoising Diffusion Probabilistic Models  Abstract  We present high quality image synthesis results using diffusion probabilistic models, a class of latent variable models inspired by considerations from nonequilibrium thermodynamics. Our best results are obtained by training on a weighted variational bound designed according to a novel connection between diffusion probabilistic models and denoising score matching with Langevin dynamics, and our models naturally admit a progressive lossy decompression scheme that can be interpreted as a generalization of autoregressive decoding. On the unconditional CIFAR10 dataset, we obtain an Inception score of 9.46 and a state-of-the-art FID score of 3.17. On 256x256 LSUN, we obtain sample quality similar to ProgressiveGAN. Section 1: Introduction  Deep generative models of all kinds have recently exhibited high quality samples in a wide variety of data modalities.
Regarding evaluations, we find that the win rates computed by GPT-4 are impacted by the prompt; future work may study the best way to elicit high-quality judgments from automated systems. Finally, many possible applications of DPO exist beyond training language models from human preferences, including training generative models in other modalities. Acknowledgements  EM gratefully acknowledges funding from a Knight-Hennessy Graduate Fellowship. CF and CM are CIFAR Fellows. This work was supported in part by the Stanford Accelerator for Learning (SAL) and Stanford Institute for Human-Centered Artificial Intelligence (HAI) Generative AI for the Future of Learning seed grant program. The Stanford Center for Research on Foundation Models (CRFM) provided part of the compute resources used for the experiments in this work. This work was supported in part by ONR grant N00014-20-1-2675.
This GapKids campaign highlights true stories of talented girls who celebrate creative self-expression and share messages of empowerment. The company is replacing the image with a different shot from the campaign, which encourages girls (and boys) everywhere to be themselves and feel pride in what makes them unique. It’s the right move in an unfortunate and complicated situation. As filmmaker Matthew A. Cherry pointed out, it’s not the first time a child has been used as an armrest in a Gap ad. Further, the two girls in the controversial pose are adopted sisters. Their mother is actor Brooke Smith (Ray Donovan, Grey’s Anatomy), who also went public on social media to provide some context. The campaign follows last year’s popular spotlight on the Pink Helmet Posse, a group of young girl skateboarders.
Travis Hancock and his wife Holly alternate shifts watching their 1-year-old daughter, Margo, when they work from home. While the norm in the board-game industry is to churn out more titles faster, Travis says they're intentionally growing their company slowly. He has no plans to hire employees or increase output beyond one game a year. "If you bring on employees, then you gotta manage them and then that kind of ties you down," he says. "We like it just being us." For now that's working: despite having plenty to do, they trade off shifts—one handles day-to-day business tasks while the other watches the baby—and they still find time to travel by working remotely when needed. The two hope their daughter will grow up to like board games as much as they do. "We're curious if she will grow up loving board games or hating them because we're going to make her play so many," he says. "But hopefully she'll like them."
More specifically, for each question, GRPO samples a group of outputs from the old policy and then optimizes the policy model by maximizing its objective. The advantage is calculated based on relative rewards of the outputs inside each group only. The group relative way that GRPO leverages to calculate the advantages, aligns well with the comparative nature of rewards models, as reward models are typically trained on datasets of comparisons between outputs on the same question. Also note that, instead of adding KL penalty in the reward, GRPO regularizes by directly adding the KL divergence between the trained policy and the reference policy to the loss, avoiding complicating the calculation of the advantage. And different from the KL penalty term used in PPO, we estimate the KL divergence with an unbiased estimator which is guaranteed to be positive. Outcome Supervision RL with GRPO Formally, for each question, a group of outputs are sampled from the old policy model. A reward model is then used to score the outputs, yielding corresponding rewards. Subsequently, these rewards are normalized by subtracting the group average and dividing by the group standard deviation. Outcome supervision provides the normalized reward at the end of each output and sets the advantages of all tokens in the output as the normalized reward, and then optimizes the policy by maximizing the objective. Process Supervision RL with GRPO Outcome supervision only provides a reward at the end of each output, which may not be sufficient and efficient to supervise the policy in complex mathematical tasks. Following others, we also explore process supervision, which provides a reward at the end of each reasoning step. Formally, given the question and sampled outputs, a process reward model is used to score each step of the outputs, yielding corresponding rewards. We also normalize these rewards with the average and the standard deviation. Subsequently, the process supervision calculates the advantage of each token as the sum of the normalized rewards from the following steps, and then optimizes the policy by maximizing the objective. Iterative RL with GRPO As the reinforcement learning training process progresses, the old reward model may not be sufficient to supervise the current policy model. Therefore, we also explore the iterative RL with GRPO. In iterative GRPO, we generate new training sets for the reward model based on the sampling results from the policy model and continually train the old reward model using a replay mechanism that incorporates 10% of historical data. Then, we set the reference model as the policy model, and continually train the policy model with the new reward model. Training and Evaluating DeepSeekMath-RL We conduct RL based on DeepSeekMath-Instruct 7B. The training data of RL are chain-of-thought-format questions related to GSM8K and MATH from the SFT data, which consists of around 144K questions. We exclude other SFT questions to investigate the impact of RL on benchmarks that lack data throughout the RL phase. We construct the training set of reward models following others.
We set a parameter budget of 18M (roughly 35MB if stored in FP16) on GPT-3 175B, which corresponds to r=8 if we adapt one type of attention weights or r=4 if we adapt two types, for all 96 layers. Note that putting all the parameters in delta Wq or delta Wk results in significantly lower performance, while adapting both Wq and Wv yields the best result. This suggests that even a rank of four captures enough information in delta W such that it is preferable to adapt more weight matrices than adapting a single type of weights with a larger rank. What is the Optimal Rank r for LoRA? We turn our attention to the effect of rank r on model performance. We adapt the set of Wq and Wv, the set of Wq, Wk, Wv, and Wc, and just Wq for a comparison. The results show that, surprisingly, LoRA already performs competitively with a very small r (more so for the set of Wq and Wv than just Wq). This suggests the update matrix delta W could have a very small "intrinsic rank". To further support this finding, we check the overlap of the subspaces learned by different choices of r and by different random seeds. We argue that increasing r does not cover a more meaningful subspace, which suggests that a low-rank adaptation matrix is sufficient. Subspace similarity between different r. Given A with subscript r equals 8 and A with subscript r equals 64 which are the learned adaptation matrices with rank r equals 8 and 64 using the same pre-trained model, we perform singular value decomposition and obtain the right-singular unitary matrices U with subscript A r equals 8 and U with subscript A r equals 64. We hope to answer: how much of the subspace spanned by the top i singular vectors in U with subscript A r equals 8 (for i between 1 and 8) is contained in the subspace spanned by top j singular vectors of U with subscript A r equals 64 (for j between 1 and 64)? We measure this quantity with a normalized subspace similarity based on the Grassmann distance, phi of A r equals 8, A r equals 64, i, j, which equals the squared Frobenius norm of the transpose of U A r equals 8 i times U A r equals 64 j, all divided by the minimum of i and j. The result is between 0 and 1. The similarity has a range of, where 1 represents a complete overlap of subspaces and 0 a complete separation. We make an important observation. Directions corresponding to the top singular vector overlap significantly between A with subscript r equals 8 and A with subscript r equals 64, while others do not. Specifically, delta Wv (respectively, delta Wq) of A with subscript r equals 8 and delta Wv (respectively, delta Wq) of A with subscript r equals 64 share a subspace of dimension 1 with normalized similarity greater than 0.5, providing an explanation of why r equals 1 performs quite well in our downstream tasks for GPT-3.
Notably, on the competition-level MATH dataset, our model surpasses all open-source models and the majority of proprietary models (e.g., Inflection-2 and Gemini Pro) by at least 9% absolute. This is true even for models that are substantially larger (e.g., Qwen 72B) or have been specifically enhanced through math-focused reinforcement learning (e.g., WizardMath-v1.1 7B). While DeepSeekMath-Instruct rivals the Chinese proprietary models GLM-4 and Baichuan-3 on MATH, it still underperforms GPT-4 and Gemini Ultra. Under the evaluation setting where models are allowed to integrate natural language reasoning and program-based tool use for problem solving, DeepSeekMath-Instruct 7B approaches an accuracy of 60% on MATH, surpassing all existing open-source models. On the other benchmarks, our model is competitive with DeepSeek-LLM-Chat 67B, the prior state-of-the-art that is 10 times larger. Reinforcement Learning  Group Relative Policy Optimization Reinforcement learning (RL) has been proven to be effective in further improving the mathematical reasoning ability of LLMs after the Supervised Fine-Tuning (SFT) stage. In this section, we introduce our efficient and effective RL algorithm, Group Relative Policy Optimization (GRPO). From PPO to GRPO Proximal Policy Optimization (PPO) is an actor-critic RL algorithm that is widely used in the RL fine-tuning stage of LLMs. In particular, it optimizes LLMs by maximizing a surrogate objective. In PPO, a value function needs to be trained alongside the policy model and to mitigate over-optimization of the reward model, the standard approach is to add a per-token KL penalty from a reference model in the reward at each token. As the value function employed in PPO is typically another model of comparable size as the policy model, it brings a substantial memory and computational burden. Additionally, during RL training, the value function is treated as a baseline in the calculation of the advantage for variance reduction. While in the LLM context, usually only the last token is assigned a reward score by the reward model, which may complicate the training of a value function that is accurate at each token. To address this, we propose Group Relative Policy Optimization (GRPO), which obviates the need for additional value function approximation as in PPO, and instead uses the average reward of multiple sampled outputs, produced in response to the same question, as the baseline.
DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models  Abstract Mathematical reasoning poses a significant challenge for language models due to its complex and structured nature. In this paper, we introduce DeepSeekMath 7B, which continues pre-training DeepSeek-Coder-Base-v1.5 7B with 120B math-related tokens sourced from Common Crawl, together with natural language and code data. DeepSeekMath 7B has achieved an impressive score of 51.7% on the competition-level MATH benchmark without relying on external toolkits and voting techniques, approaching the performance level of Gemini-Ultra and GPT-4. Self-consistency over 64 samples from DeepSeekMath 7B achieves 60.9% on MATH. The mathematical reasoning capability of DeepSeekMath is attributed to two key factors: First, we harness the significant potential of publicly available web data through a meticulously engineered data selection pipeline. Second, we introduce Group Relative Policy Optimization (GRPO), a variant of Proximal Policy Optimization (PPO), that enhances mathematical reasoning abilities while concurrently optimizing the memory usage of PPO. Introduction Large language models (LLM) have revolutionized the approach to mathematical reasoning in artificial intelligence, spurring significant advancements in both the quantitative reasoning benchmark and the geometry reasoning benchmark. Moreover, these models have proven instrumental in assisting humans in solving complex mathematical problems. However, cutting-edge models such as GPT-4 and Gemini-Ultra are not publicly available, and the currently accessible open-source models considerably trail behind in performance. In this study, we introduce DeepSeekMath, a domain-specific language model that significantly outperforms the mathematical capabilities of open-source models and approaches the performance level of GPT-4 on academic benchmarks. To achieve this, we create the DeepSeekMath Corpus, a large-scale high-quality pre-training corpus comprising 120B math tokens. This dataset is extracted from the Common Crawl (CC) using a fastText-based classifier.
March 28, 2018 — For Mohammed Saleh Ahmed and his family, returning to Mosul after more than a year away was bittersweet. Ahmed, 31, fled his home in March 2017 with his wife, parents and five children when U.S.-led coalition forces began their advance on western Mosul to rout Islamic State militants who had overrun the city in 2014. The family lived in the Hammam al-Alil refugee camp and returned home to Mosul in February 2018. He was happy to regain a semblance of his old life, but leaving behind the friends he had made in the camp weighed on him. “It’s so hard saying goodbye,” Mohammed said as he finished boxing up his belongings in his tent in Hammam al-Alil. Since he fled, the community of friends and relatives—fellow survivors of the battle for Mosul—had made life bearable. Carrying whatever he could, Mohammed Saleh Ahmad fled his home in the city with his wife and five children; his parents and siblings were not far behind. They left in March 2017 when US-led coalition forces began their advance on western Mosul to rout Islamic State militants who had overrun the city in 2014. The family lived in Hamam al-Alil refugee camp and returned to Mosul in February 2018.

A year earlier they had not expected to come back so soon, but when the opportunity arose they seized it. A few days after deciding to leave, neighbours and relatives helped load their belongings onto a small truck. Mohammed’s parents went too. As they said goodbye, his father, Saleh, played old Iraqi folk songs on an MP3 player amid tearful farewells and promises to return and visit soon. In Mosul they were greeted by Mohammed’s older brother, Ahmed, who had found them a modest two-room house to rent. As the men carried things inside the house, Mohammed’s wife Iman immediately set about preparing the family’s first meal in their new home. His parents, who had moved in with Mohammed’s family, celebrated their return over tea. Though rudimentary, their new house was a step up from life in the camps, with a separate kitchen and a makeshift bathroom, and the Ahmed family quickly settled into a new routine. Before finding a job at his uncle’s construction company, Mohammed busied himself going to the market, getting a haircut, taking his children to a recently reopened amusement park, and buying new clothes for Iman. "New clothes for our new life," he said as the couple wandered through the market picking out dresses. While the Ahmed family was away from Mosul, a nine-month campaign to rout the militants culminated in a brutal battle in the heart of the Old City, the centuries-old historic district at the center of Mosul. Though the Islamic State was defeated, the western half of the city, where Mohammed grew up, was largely flattened. Seeing it for the first time, Mohammed was shocked by what it had become as he walked around with his older brother Ahmed. Reporting by Khaled al-Mosuly; writing by Raya Jalabi; editing by Larry King.
Kevin Smith revealed he suffered a "massive" heart attack and posted a selfie from his hospital bed shortly after midnight. The photo showed him wearing a hospital gown and attached to tubes. Smith said the doctor who saved him told him he had 100% blockage of his LAD artery — the "widow-maker." He was shooting a new standup special, Kevin Smith Live!, at the Alex Theatre in Glendale; two shows were scheduled that night, and he canceled the second performance to go to the hospital. "If I hadn’t canceled show 2 to go to the hospital, I would’ve died tonight. But for now, I’m still above ground!" he tweeted. The doctor who saved my life told me I had 100% blockage of my LAD artery (aka "the Widow-Maker"). If I hadn’t canceled show 2 to go to the hospital, I would’ve died tonight. But for now, I’m still above ground!

Smith currently hosts and produces AMC’s Comic Book Men, a reality series set at his comics shop Jay and Silent Bob’s Secret Stash in Red Bank, New Jersey. It has run for seven seasons and features Smith and his pals talking about comic books. He recently finished directing an episode of The CW’s The Flash and last year appeared as himself in James Franco’s The Disaster Artist. His last feature as director was 2016’s Yoga Hosers. The Kevin Smith Live! special is produced by The Nacelle Company Presents. The logline for the show says it sees the Mallrats director tell "long-ass stories that aim to amuse. If you like your comedy by the pound, you can’t go wrong with this yapping fool from TV’s Comic Book Men as he talks about life, comic books, movies and more!" This was Smith’s first live TV comedy special in some years; he previously fronted a number of Q&A specials including An Evening With Kevin Smith — released on DVD — and his Kevin Smith: Too Fat for 40 and Kevin Smith: Burn in Hell, which aired on Epix in 2010 and 2012, respectively. Smith had earlier posted to Facebook:

Friends and fans were quick to take to Twitter upon Smith’s news today:

Kevin, we don’t know each other too well, but I have loved you since Clerks and I’m praying my ass off for you because I believe in the healing power of prayer. Can you please pray with me, people? — Chris Pratt

Praying for you. I will continue to. You inspired me with Clerks when I was a senior in high school. I’m tagging my LB/RB football coach who showed me the movie because he believed in me and knew I’d be inspired. — Chris Pratt

Scary! Glad you're OK. — Patricia Arquette

Oh my God. Wishing you the speediest and most painless recovery. Love you, buddy. — Brian Lynch

I keep reading this in shock. Damn. It’s an understatement to say, but myself and so many others are glad you cancelled and are recovering. Sending love to you and yours. — Sean Francis

Glad to still have you this side of the lawn. Cheers. — Michael McKean