(Google Scholar, github, slideshare)

Research Scientist at CyberAgent

e-mail: takase_sho [aatt] cyberagent.co.jp

Research Interest

Neural Language Model

Efficient Neural Model

Publications

Journal Papers

• Ayana Niwa, Sho Takase, Naoaki Okazaki. Nearest Neighbor Non-autoregressive Text Generation. Journal of Information Processing. vol. 31, pp. 344–352, 2023. pdf
• Tatsuya Hiraoka, Sho Takase, Kei Uchiumi, Atsushi Keyaki, Naoaki Okazaki. Joint Optimization of Word Segmentation and Downstream Model using Downstream Loss. Journal of Natural Language Processing, vol. 29, No. 1, pp. 112–143, March 2022. pdf
• Tatsuya Hiraoka, Sho Takase, Kei Uchiumi, Atsushi Keyaki, Naoaki Okazaki. Recurrent Neural Hidden Markov Model for High-Order Transition. ACM Transactions on Asian and Low-Resource Language Information Processing (TALLIP), vol. 21, issue 2, pp. 1–-15, March 2022. pdf
• Tatsuya Hiraoka, Sho Takase, Kei Uchiumi, Atsushi Keyaki, Naoaki Okazaki. Optimizing Word Segmentation for Downstream Tasks by Weighting Text Vector (in Japanese). Journal of Natural Language Processing, vol. 28, No. 2, pp. 479–507, June 2021. pdf
• Sho Takase, Naoaki Okazaki, Kentaro Inui. Learning to Compose Distributed Representations of Relational Patterns (in Japanese). Transactions of the Japanese Society for Artificial Intelligence, vol. 32, No. 4., July 2017. pdf
• Sho Takase, Naoaki Okazaki, Kentaro Inui. Modeling semantic compositionality of relational patterns. Engineering Applications of Artificial Intelligence, vol. 50, pp.256–264, April 2016. pdf
• Sho Takase, Naoaki Okazaki, Kentaro Inui. Set Expansion Using Sibling Relationships Between Semantic Categories (in Japanese). Journal of Natural Language Processing, vol. 20, No. 2, June 2013. pdf

(Refereed) International Conference and Workshop Papers

• Sho Takase, Shun Kiyono, Sosuke Kobayashi, Jun Suzuki. Spike No More: Stabilizing the Pre-training of Large Language Models. In Proceedings of 2nd Conference on Language Modeling (COLM). 2025.
• Kazuki Yano, Sho Takase, Sosuke Kobayashi, Shun Kiyono, Jun Suzuki. Efficient Construction of Model Family through Progressive Training Using Model Expansion. In Proceedings of 2nd Conference on Language Modeling (COLM). 2025.
• Sho Takase, Ryokan Ri, Shun Kiyono, Takuya Kato. Large Vocabulary Size Improves Large Language Models. In Proceedings of Findings of the Association for Computational Linguistics: ACL 2025. pp. 1015–-1026, 2025. pdf
• Seng Pei Liew, Takuya Kato, Sho Takase. Scaling Laws for Upcycling Mixture-of-Experts Language Models. In Proceedings of 42nd International Conference on Machine Learning (ICML). 2025. pdf
• Shun Kiyono, Sho Takase, Shengzhe Li, Toshinori Sato. Bridging the Gap between Subword and Character Segmentation in Pretrained Language Models. In Proceedings of the 14th International Conference on Recent Advances in Natural Language Processing (RANLP). pp. 568–-577, 2023. pdf
• Sho Takase, Shun Kiyono. Lessons on Parameter Sharing across Layers in Transformers. In Proceedings of The Fourth Workshop on Simple and Efficient Natural Language Processing (SustaiNLP). pp. 78-–90, 2023. pdf, code
• Mengsay Loem, Masahiro Kaneko, Sho Takase, Naoaki Okazaki. Exploring Effectiveness of GPT-3 in Grammatical Error Correction: A Study on Performance and Controllability in Prompt-Based Methods. In Proceedings of the 18th Workshop on Innovative Use of NLP for Building Educational Applications (BEA 2023). pp. 205–-219, 2023. pdf
• Nozomu Miyamoto, Masaru Isonuma, Sho Takase Junichiro Mori, Ichiro Sakata. Dynamic Structured Neural Topic Model with Self-Attention Mechanism. In Findings of the Association for Computational Linguistics: ACL 2023.
• Sho Takase, Shun Kiyono, Sosuke Kobayashi, Jun Suzuki. B2T Connection: Serving Stability and Performance in Deep Transformers. In Findings of the Association for Computational Linguistics: ACL 2023. pdf, code.
• Mengsay Loem, Sho Takase, Masahiro Kaneko, Naoaki Okazaki. ExtraPhrase: Efficient Data Augmentation for Abstractive Summarization. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies: Student Research Workshop. pp. 16–-24, 2022. pdf
• Sho Takase, Naoaki Okazaki. Multi-Task Learning for Cross-Lingual Abstractive Summarization. In Proceedings of the Thirteenth Language Resources and Evaluation Conference (LREC). pp. 3008–3016, 2022. pdf
• Tatsuya Hiraoka, Sho Takase, Kei Uchiumi, Atsushi Keyaki, Naoaki Okazaki. Word-level Perturbation Considering Word Length and Compositional Subwords. pp. 3268-–3275, 2022. pdf
• Sho Takase, Tatsuya Hiraoka, Naoaki Okazaki. Single Model Ensemble for Subword Regularized Models in Low-Resource Machine Translation. In Proceedings of Findings of the Association for Computational Linguistics: ACL 2022. pp. 2536–2541, 2022. pdf
• Masahiro Kaneko, Sho Takase, Ayana Niwa, Naoaki Okazaki. Interpretability for Language Learners Using Example-Based Grammatical Error Correction. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics. pp. 7176–-7187, 2022. pdf
• Tatsuya Hiraoka, Sho Takase, Kei Uchiumi, Atsushi Keyaki, Naoaki Okazaki. Joint Optimization of Tokenization and Downstream Model. In Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021. pp. 244–255, 2021. pdf
• Sho Takase, Shun Kiyono. Rethinking Perturbations in Encoder-Decoders for Fast Training. In Proceedings of 2021 Annual Conference of the North American Chapter of the Association for Computational Linguistics (NAACL-HLT 2021), pp. 5767–-5780, 2021. pdf, code, slide.
• Sho Takase, Sosuke Kobayashi. All Word Embeddings from One Embedding. In Proceedings of the thirty-fourth Conference on Neural Information Processing Systems (NeurIPS), 2020. pdf, code.
• Tatsuya Hiraoka, Sho Takase, Kei Uchiumi, Atsushi Keyaki, Naoaki Okazaki. Optimizing Word Segmentation for Downstream Task. In Findings of the Association for Computational Linguistics: EMNLP 2020. pp. 1341–1351, 2020.
• Kazuki Matsumaru, Sho Takase, Naoaki Okazaki. Improving Truthfulness of Headline Generation. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics (ACL 2020).
• Sho Shimazu, Sho Takase, Toshiaki Nakazawa, Naoaki Okazaki. Evaluation Dataset for Zero Pronoun in Japanese to English Translation. In Proceedings of The 12th Language Resources and Evaluation Conference (LREC 2020), pp. 3630–3634, 2020.
• Masaaki Nishino, Sho Takase, Tsutomu Hirao, Masaaki Nagata. Generating Natural Anagrams:Towards Language Generation Under Hard Combinatorial Constraints. In Proceedings of 2019 Conference on Empirical Methods in Natural Language Processing and 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP 2019).
• Yuichi Sasazawa, Sho Takase, Naoaki Okazaki. Neural Question Generation using Interrogative Phrases. In Proceedings of the 12th International Conference on Natural Language Generation (INLG 2019), pp. 106–111, 2019.
• Sho Takase, Naoaki Okazaki. Positional Encoding to Control Output Sequence Length. In Proceedings of 2019 Annual Conference of the North American Chapter of the Association for Computational Linguistics (NAACL-HLT 2019), pp. 3999–4004, 2019. pdf, code.
• Sho Takase, Jun Suzuki, Masaaki Nagata. Character n-gram Embeddings to Improve RNN Language Models. In Proceedings of the Thirty-Third AAAI Conference on Artificial Intelligence (AAAI 2019), pp. 5074–5082. 2019. pdf.
• Shun Kiyono, Sho Takase, Jun Suzuki, Naoaki Okazaki, Kentaro Inui and Masaaki Nagata. Reducing Odd Generation from Neural Headline Generation. In the Proceedings of the 32nd Pacific Asia Conference on Language, Information and Computing (PACLIC 32), 2018.
• Shun Kiyono, Sho Takase, Jun Suzuki, Naoaki Okazaki, Kentaro Inui and Masaaki Nagata. Unsupervised Token-wise Alignment to Improve Interpretation of Encoder-Decoder models. In the Proceedings of the Analyzing and interpreting neural networks for NLP (BlackboxNLP), 2018.
• Sho Takase, Jun Suzuki, Masaaki Nagata. Direct Output Connection for a High-Rank Language Model. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing (EMNLP 2018), pp. 4599–4609, 2018. pdf, code, score.
• Jun Suzuki, Sho Takase, Hidetaka Kamigaito, Makoto Morishita, Masaaki Nagata. An Empirical Study of Building a Strong Baseline for Constituency Parsing. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (ACL 2018), pp.612–618, 2018.
• Sho Takase, Jun Suzuki, Masaaki Nagata. Input-to-Output Gate to Improve RNN Language Models. In Proceedings of the 8th International Joint Conference on Natural Language Processing (IJCNLP 2017), pp.43–48, November 2017. pdf, code, score.
• Shota Sasaki, Sho Takase, Naoya Inoue, Naoaki Okazaki and Kentaro Inui. Handling Multiword Expressions in Causality Estimation. In proceedings of the 12th International Conference on Computational Semantics (IWCS 2017), Septembre 2017.
• Sho Takase, Jun Suzuki, Naoaki Okazaki, Tsutomu Hirao and Masaaki Nagata. Neural Headline Generation on Abstractive Meaning Representation. In Proceedings of the Conference on Empirical Methods in Natural Language Processing (EMNLP 2016), pp. 1054–1059, November 2016. pdf
• Sho Takase, Naoaki Okazaki and Kentaro Inui. Composing Distributed Representations of Relational Patterns. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (ACL 2016), pp.2276–2286, August 2016. pdf, poster, dataset, code
• Sho Takase, Naoaki Okazaki and Kentaro Inui. Fast and Large-scale Unsupervised Relation Extraction. In Proceedings of the 29th Pacific Asia Conference on Language Information and Computing (PACLIC29), pp.96–105, October 2015. pdf
• Sho Takase, Akiko Murakami, Miki Enoki, Naoaki Okazaki and Kentaro Inui. Detecting Chronic Critics Based on Sentiment Polarity and User’s Behavio in Social Media. In Proceedings of the ACL student research workshop 2013, pp.110–116, 2013. pdf
• Sho Takase, Naoaki Okazaki and Kentaro Inui. Set Expansion using Sibling Relations between Semantic Categories. In Proceedings of the 26th Pacific Asia Conference on Language Information and Computing (PACLIC26), pp.525–534, November 2012. pdf

Misc

• 高瀬 翔. NLP若手の会 (YANS) 第 16 回シンポジウム ―オンライン開催における施策について―. Journal of Natural Language Processing, vol. 28, No. 4, Dec 2021.
• 高瀬 翔. 実験を通して既存手法を整理する ―系列変換タスクにおける摂動の効率の調査―. Journal of Natural Language Processing, vol. 28, No. 3, Sep 2021.
• 高瀬 翔. NLP若手の会（YANS）シンポジウム. Journal of Natural Language Processing, vol. 28, No. 1, Mar 2021.
• Sho Takase, Naoaki Okazaki. Construction of Relation Knowledge Base from Natural Language Text. Journal of Japan Society for Fuzzy Theory and Intelligent Informatics, vol. 29, No. 2, pp.55–64, April 2017, in Japanese.

Talks

• SB Intuitionsにおける大規模言語モデルの事前学習. W&Bミートアップ, Dec, 2024. pdf, video.
• 少量の計算資源でも利用可能な自然言語処理モデルを目指して. 第27回ステアラボ人工知能セミナー, May, 2021. pdf, video.
• NeurIPS2020の概観とニューラルモデルのパラメータ数削減. 第79回人工知能セミナー, Mar, 2021.
• 自然言語処理におけるニューラルモデルのパラメータ数削減. 第5回愛媛大学DSセミナー, Feb 2021.
• ニューラル言語モデルの研究動向. IPSJ-SIGNL 240, June 2019.
• 国際会議 EMNLP 2016 参加報告 (2). 第18回音声言語シンポジウム & 第3回自然言語処理シンポジウム, December 2016.
• ACL 2013 参加報告. NLP若手の会第8回シンポジウム, September 2013.

Awards

• The 29th Annual Meeting of The Association for Natural Language Processing Special Committee Award (2023).
• The Association for Natural Language Processing Best paper award (first place) (2022).
• The 28th Annual Meeting of The Association for Natural Language Processing Special Committee Award (2022).
• The 28th Annual Meeting of the Association for Natural Language Processing Best Paper Award (2022).
• Outstanding reviewer, EMNLP-IJCNLP 2019 (2019).
• Information Processing Society of Japan, Special Interest Group of Natural Language Processing (IPSJ-SIGNL) Excellent Research Award (2019).
• The 25th Annual Meeting of the Association for Natural Language Processing Poster Award (2019)
• JSAI Best Paper Award (2018).
• The 24th Annual Meeting of the Association for Natural Language Processing Best Paper Award (2018).
• Dean’s Award for Excellence in Graduate School of Information Sciences at Tohoku University (2017).
• The 29th Annual Meeting of the JSAI Conference Student Incentive Award (2015).
• The 21th Annual Meeting of the Association for Natural Language Processing Excellent Paper Award (2015).

Program commitee member

• 言語処理学会会誌「自然言語処理」編集委員 (2020 - ).
• NLP若手の会（YANS）運営委員長 (2020 - 2022).
• Local committee and Workshop Chair of the 12th International Conference on Natural Language Generation (2019).
• NLP若手の会（YANS）運営委員 (2017 - 2018).

Lecture

• 手続き型プログラミング（発展） (2020 - 2022).
• 知能情報フォーラム (2020 - 2022).
• 高次ヒューマンインタフェース（非常勤講師） (2020 - 2021).

Biography

• Research Scientist, CyberAgent (2025. 8 - ).
• Research Scientist, SB Intuitions (2024. 7 - 2025. 7).
• Senior Researcher, LINE Corporation (2022. 10 - 2024. 6).
• Assistant Professor, Tokyo Institute of Technology (2020. 4 - 2022. 9).
• Researcher, Tokyo Institute of Technology (2018.10 - 2020. 3).
• Research associate, NTT Communication Science Laboratories (2017.4 - 2018.9).
• Ph.D. Graduate School of Information Sciences, Tohoku University (2014. 4 - 2017. 3).
• M.Sc. Graduate School of Information Sciences, Tohoku University (2012. 4 - 2014. 3).
• B.Eng. Tohoku University (2008. 4 - 2012. 3).