Search Results - "Fei, Zhaoye"

Balanced Data Sampling for Language Model Training with Clustering by Shao, Yunfan, Li, Linyang, Fei, Zhaoye, Yan, Hang, Lin, Dahua, Qiu, Xipeng

“…Data plays a fundamental role in the training of Large Language Models (LLMs). While attention has been paid to the collection and composition of datasets,…”
Get full text

Query of CC: Unearthing Large Scale Domain-Specific Knowledge from Public Corpora by Fei, Zhaoye, Shao, Yunfan, Li, Linyang, Zeng, Zhiyuan, He, Conghui, Yan, Hang, Lin, Dahua, Qiu, Xipeng

“…Large language models have demonstrated remarkable potential in various tasks, however, there remains a significant scarcity of open-source models and data for…”
Get full text

Turn Waste into Worth: Rectifying Top-$k$ Router of MoE by Zeng, Zhiyuan, Guo, Qipeng, Fei, Zhaoye, Yin, Zhangyue, Zhou, Yunhua, Li, Linyang, Sun, Tianxiang, Yan, Hang, Lin, Dahua, Qiu, Xipeng

“…Sparse Mixture of Experts (MoE) models are popular for training large language models due to their computational efficiency. However, the commonly used top-$k$…”
Get full text

Pre-training for Information Retrieval: Are Hyperlinks Fully Explored? by Wu, Jiawen, Zhang, Xinyu, Zhu, Yutao, Liu, Zheng, Guo, Zikai, Fei, Zhaoye, Lai, Ruofei, Wu, Yongkang, Cao, Zhao, Dou, Zhicheng

“…Recent years have witnessed great progress on applying pre-trained language models, e.g., BERT, to information retrieval (IR) tasks. Hyperlinks, which are…”
Get full text

Coarse-to-Fine: Hierarchical Multi-task Learning for Natural Language Understanding by Fei, Zhaoye, Tian, Yu, Wu, Yongkang, Zhang, Xinyu, Zhu, Yutao, Liu, Zheng, Wu, Jiawen, Kong, Dejiang, Lai, Ruofei, Cao, Zhao, Dou, Zhicheng, Qiu, Xipeng

“…Generalized text representations are the foundation of many natural language understanding tasks. To fully utilize the different corpus, it is inevitable that…”
Get full text

WanJuan-CC: A Safe and High-Quality Open-sourced English Webtext Dataset by Qiu, Jiantao, Lv, Haijun, Jin, Zhenjiang, Wang, Rui, Ning, Wenchang, Yu, Jia, Zhang, ChaoBin, Li, Zhenxiang, Chu, Pei, Qu, Yuan, Shi, Jin, Lu, Lindong, Peng, Runyu, Zeng, Zhiyuan, Tang, Huanze, Lei, Zhikai, Hong, Jiawei, Chen, Keyu, Fei, Zhaoye, Xu, Ruiliang, Li, Wei, Tu, Zhongying, Dahua, Lin, Qiao, Yu, Yan, Hang, He, Conghui

“…This paper presents WanJuan-CC, a safe and high-quality open-sourced English webtext dataset derived from Common Crawl data. The study addresses the challenges…”
Get full text

InternLM-Math: Open Math Large Language Models Toward Verifiable Reasoning by Ying, Huaiyuan, Zhang, Shuo, Li, Linyang, Zhou, Zhejian, Shao, Yunfan, Fei, Zhaoye, Ma, Yichuan, Hong, Jiawei, Liu, Kuikun, Wang, Ziyi, Wang, Yudong, Wu, Zijian, Li, Shuaibin, Zhou, Fengzhe, Liu, Hongwei, Zhang, Songyang, Zhang, Wenwei, Yan, Hang, Qiu, Xipeng, Wang, Jiayu, Chen, Kai, Lin, Dahua

“…The math abilities of large language models can represent their abstract reasoning ability. In this paper, we introduce and open-source our math reasoning LLMs…”
Get full text

Towards More Effective and Economic Sparsely-Activated Model by Jiang, Hao, Zhan, Ke, Qu, Jianwei, Wu, Yongkang, Fei, Zhaoye, Zhang, Xinyu, Chen, Lei, Dou, Zhicheng, Qiu, Xipeng, Guo, Zikai, Lai, Ruofei, Wu, Jiawen, Hu, Enrui, Zhang, Yinxia, Jia, Yantao, Yu, Fan, Cao, Zhao

“…The sparsely-activated models have achieved great success in natural language processing through large-scale parameters and relatively low computational cost,…”
Get full text

InternLM2 Technical Report by Cai, Zheng, Cao, Maosong, Chen, Haojiong, Chen, Kai, Chen, Keyu, Chen, Xin, Chen, Xun, Chen, Zehui, Chen, Zhi, Chu, Pei, Dong, Xiaoyi, Duan, Haodong, Fan, Qi, Fei, Zhaoye, Gao, Yang, Ge, Jiaye, Gu, Chenya, Gu, Yuzhe, Gui, Tao, Guo, Aijia, Guo, Qipeng, He, Conghui, Hu, Yingfan, Huang, Ting, Jiang, Tao, Jiao, Penglong, Jin, Zhenjiang, Lei, Zhikai, Li, Jiaxing, Li, Jingwen, Li, Linyang, Li, Shuaibin, Li, Wei, Li, Yining, Liu, Hongwei, Liu, Jiangning, Hong, Jiawei, Liu, Kaiwen, Liu, Kuikun, Liu, Xiaoran, Lv, Chengqi, Lv, Haijun, Lv, Kai, Ma, Li, Ma, Runyuan, Ma, Zerun, Ning, Wenchang, Ouyang, Linke, Qiu, Jiantao, Qu, Yuan, Shang, Fukai, Shao, Yunfan, Song, Demin, Song, Zifan, Sui, Zhihao, Sun, Peng, Sun, Yu, Tang, Huanze, Wang, Bin, Wang, Guoteng, Wang, Jiaqi, Wang, Jiayu, Wang, Rui, Wang, Yudong, Wang, Ziyi, Wei, Xingjian, Weng, Qizhen, Wu, Fan, Xiong, Yingtong, Xu, Chao, Xu, Ruiliang, Yan, Hang, Yan, Yirong, Yang, Xiaogui, Ye, Haochen, Ying, Huaiyuan, Yu, Jia, Yu, Jing, Zang, Yuhang, Zhang, Chuyu, Zhang, Li, Zhang, Pan, Zhang, Peng, Zhang, Ruijie, Zhang, Shuo, Zhang, Songyang, Zhang, Wenjian, Zhang, Wenwei, Zhang, Xingcheng, Zhang, Xinyue, Zhao, Hui, Zhao, Qian, Zhao, Xiaomeng, Zhou, Fengzhe, Zhou, Zaida, Zhuo, Jingming, Zou, Yicheng, Qiu, Xipeng, Qiao, Yu, Lin, Dahua

“…The evolution of Large Language Models (LLMs) like ChatGPT and GPT-4 has sparked discussions on the advent of Artificial General Intelligence (AGI). However,…”
Get full text