We are pleased to announce that we have 3 papers accepted to The Sixth Arabic Natural Language Processing Workshop (WANLP 2021) co-located with EACL 2021. Authored by our talented team members: Tarek Naous, Wissam Antoun, Reem Mahmoud, Fady Baly under the supervision of Prof. Hazem Hajj. The papers target Arabic empathetic conversational agents, generative language models, and language understanding models.
Empathetic BERT2BERT Conversational Model:
Learning Arabic Language Generation with Little Data
Our latest contribution to Arabic Conversational AI leverages knowledge transfer from AraBERT in a BERT2BERT architecture. We address the low resource challenges and achieve sota results in open domain empathetic response generation.
Paper: https://arxiv.org/abs/2103.04353
Abstract: Enabling empathetic behavior in Arabic dialogue agents is an important aspect of building human-like conversational models. While Arabic Natural Language Processing has seen significant advances in Natural Language Understanding (NLU) with language models such as AraBERT, Natural Language Generation (NLG) remains a challenge. The shortcomings of NLG encoder-decoder models are primarily due to the lack of Arabic datasets suitable to train NLG models such as conversational agents. To overcome this issue, we propose a transformer-based encoder-decoder initialized with AraBERT parameters. By initializing the weights of the encoder and decoder with AraBERT pre-trained weights, our model was able to leverage knowledge transfer and boost performance in response generation. To enable empathy in our conversational model, we train it using the ArabicEmpatheticDialogues dataset and achieve high performance in empathetic response generation. Specifically, our model achieved a low perplexity value of 17.0 and an increase in 5 BLEU points compared to the previous state-of-the-art model. Also, our proposed model was rated highly by 85 human evaluators, validating its high capability in exhibiting empathy while generating relevant and fluent responses in open-domain settings.
AraGPT2:
Pre-Trained Transformer for Arabic Language Generation
AraGPT2 is a 1.5B transformer model, the largest for Arabic, trained on 77GB of text for 9 days with a TPUv3-128. The model can generate news articles that are difficult to distinguish from human-written articles. AraGPT2 shows impressive Zero-shot performance on trivia QA.
Paper: arxiv.org/abs/2012.15520
GitHub: https://github.com/aub-mind/arabert/tree/master/aragpt2
Abstract: Recently, pre-trained transformer-based architectures have proven to be very efficient at language modeling and understanding, given that they are trained on a large enough corpus. Applications in language generation for Arabic are still lagging in comparison to other NLP advances primarily due to the lack of advanced Arabic language generation models. In this paper, we develop the first advanced Arabic language generation model, AraGPT2, trained from scratch on a large Arabic corpus of internet text and news articles. Our largest model, AraGPT2-mega, has 1.46 billion parameters, which makes it the largest Arabic language model available. The Mega model was evaluated and showed success on different tasks including synthetic news generation, and zero-shot question answering. For text generation, our best model achieves a perplexity of 29.8 on held-out Wikipedia articles. A study conducted with human evaluators showed the significant success of AraGPT2-mega in generating news articles that are difficult to distinguish from articles written by humans. We thus develop and release an automatic discriminator model with a 98% percent accuracy in detecting model-generated text. The models are also publicly available, hoping to encourage new research directions and applications for Arabic NLP.
AraELECTRA:
Pre-Training Text Discriminators for Arabic Language Understanding
AraELECTRA is our latest advancements in Arabic Language Understanding. The model was trained on 77GB of Arabic text for 24 days. AraELECTRA achieves impressive performance, especially on Question Answering tasks.
Paper: https://arxiv.org/abs/2012.15516
Github: https://github.com/aub-mind/arabert/tree/master/araelectra
Abstract: Advances in English language representation enabled a more sample-efficient pre-training task by Efficiently Learning an Encoder that Classifies Token Replacements Accurately (ELECTRA). Which, instead of training a model to recover masked tokens, it trains a discriminator model to distinguish true input tokens from corrupted tokens that were replaced by a generator network. On the other hand, current Arabic language representation approaches rely only on pretraining via masked language modeling. In this paper, we develop an Arabic language representation model, which we name AraELECTRA. Our model is pretrained using the replaced token detection objective on large Arabic text corpora. We evaluate our model on multiple Arabic NLP tasks, including reading comprehension, sentiment analysis, and named-entity recognition and we show that AraELECTRA outperforms current state-of-the-art Arabic language representation models, given the same pretraining data and with even a smaller model size.
Acknowledgments:
This research was supported by the University Research Board (URB) at the American University of Beirut (AUB), and by the TFRC program, which we thank for the free access to cloud TPUs. We also thank As-Safir newspaper for the data access.
I definitely enjoying every little bit of it and I have you bookmarked to check out new stuff you post. Zonnepanelen
I really enjoy simply reading all of your weblogs. Simply wanted to inform you that you have people like me who appreciate your work. Definitely a great post. Hats off to you! The information that you have provided is very helpful. Thuisbatterij
Great Information sharing .. I am very happy to read this article .. thanks for giving us go through info.Fantastic nice. I appreciate this post. eps isolatie
I have to search sites with relevant information on given topic and provide them to teacher our opinion and the article. Thuisbatterij 5 kwh prijs
This post is very simple to read and appreciate without leaving any details out. Great work! Installateur zonnepanelen Limburg
This is such a great resource that you are providing and you give it away for free. Thuisbatterij kopen
I’m happy I located this blog! From time to time, students want to cognitive the keys of productive literary essays composing. Your first-class knowledge about this good post can become a proper basis for such people. nice one Installateur zonnepanelen Antwerpen
Hello! I just wish to give an enormous thumbs up for the nice info you’ve got right here on this post. I will probably be coming back to your weblog for more soon! REC zonnepanelen
This article gives the light in which we can observe the reality. This is very nice one and gives indepth information. Thanks for this nice article. Warmtepompen
Glad to chat your blog, I seem to be forward to more reliable articles and I think we all wish to thank so many good articles, blog to share with us. Airco
I read your post, very nice post. w88 bắn cá
Wonderful article, thanks for putting this together! This is obviously one great post. Thanks for the valuable information and insights you have so provided here. Twitch stream downloader
What do you do for relaxation and the most convenient time for you? 33WIN.CH
Let is keep in touch and call it a day. ww88
I recently came across this fantastic content and felt compelled to share my excitement! As I’m about to celebrate my engagement, I wanted to gift myself some new and exciting experiences, and SEVlaser was introduced to me. Reading others’ glowing reviews here, I couldn’t resist booking a session. It’s been an absolute joy, and I’m thrilled to be part of this positive community. Don’t miss out—check out Laser Hair Removal in Brooklyn for an amazing experience!
789winbee.com
Thanks for watching
You understand what the teacher said and I go out with dinner. hướng dẫn e2bet
如果之前提交的作业是由自己独立完成的,而此次代写作业 https://www.essayghostwriting.com/ 的风格和质量突然发生明显变化,教授很可能会对此产生怀疑。特别是在同一门课程中,教授通常会对每位学生的写作风格和学术水平有一定的了解和判断。如果学生突然提交了一篇风格迥异或质量显著提高的作业,很容易引起教授的注意。因此,选择题目时应尽量避免与之前的作业内容相似,或者确保代写作业的风格和水平与之前的作业保持一致。
Thankyou so much for valueable information. You are requested to please upgrade your post with more visuals or reference videos. I am waiting for your new post with more videos. Thanks
#Thanks for sharing this article to us. I gathered more useful information from your blog. Keep sharing more related blogs.bsc.news
I want you to thank for your time of this wonderful read!!! I definately enjoy every little bit of it and I have you bookmarked to check out new stuff of your blog a must read blog! Zwembad aanleggen
Please give some advice on how to achieve this kind of posts. Airco
This is very educational content and written well for a change. It’s nice to see that some people still understand how to write a quality post! Polypropyleen zwembad
A great website with interesting and unique material what else would you need. Zonnepanelen Hasselt
This was among the best posts and episode from your team it let me learn many new things. Monoblok zwembad
I want you to thank for your time of this wonderful read!!! I definately enjoy every little bit of it and I have you bookmarked to check out new stuff of your blog a must read blog! Warmtepompen
Please give some advice on how to achieve this kind of posts. Zwembad plaatsen
This is very useful post for me. This will absolutely going to help me in my project. Zonnepanelen
Super site! I am Loving it!! Will return once more, Im taking your food likewise, Thanks. Zwembad plaatsen
Thanks for the best blog. it was very useful for me.keep sharing such ideas in the future as well. Zonnepanelen
Thumbs up guys your doing a really good job. Luxe zwembad
What a really awesome post this is. Truly, one of the best posts I’ve ever witnessed to see in my whole life. Wow, just keep it up. Zonnepanelen
Thanks for the best blog. it was very useful for me.keep sharing such ideas in the future as well. Monoblok zwembad
This is a smart blog. I mean it. You have so much knowledge about this issue, and so much passion. You also know how to make people rally behind it, obviously from the responses. Webdesign
Going to graduate school was a positive decision for me. I enjoyed the coursework, the presentations, the fellow students, and the professors. And since my company reimbursed 100% of the tuition, the only cost that I had to pay on my own was for books and supplies. Otherwise, I received a free master’s degree. All that I had to invest was my time. Buitenzwembad
I was just browsing through the internet looking for some information and came across your blog. I am impressed by the information that you have on this blog. It shows how well you understand this subject. Bookmarked this page, will come back for more. Zonnepanelen
Just saying thanks will not just be sufficient, for the fantasti c lucidity in your writing. I will instantly grab your rss feed to stay informed of any updates. Prijs zwembad
I will be interested in more similar topics. i see you got really very useful topics , i will be always checking your blog thanks Zwembad aanleggen
Such a very useful article. Very interesting to read this article.I would like to thank you for the efforts you had made for writing this awesome article. Zwembadbouw
I have express a few of the articles on your website now, and I really like your style of blogging. I added it to my favorite’s blog site list and will be checking back soon… Monoblock zwembad
Thank you so much for this insightful article! It’s been incredibly informative. ใลน์หวยบี
Information is fruitful and helpful. But I do want to highlight that we need to include quality games and apps in our lives. Apps like Wync music and InShot Pro make our tasks easy. The games like Shadow Fight 2 and Shadow Fight 3 keeps us busy and entertain as well. You can find and download them from APKs universe