We are pleased to announce that we have 3 papers accepted to The Sixth Arabic Natural Language Processing Workshop (WANLP 2021) co-located with EACL 2021. Authored by our talented team members: Tarek Naous, Wissam Antoun, Reem Mahmoud, Fady Baly under the supervision of Prof. Hazem Hajj. The papers target Arabic empathetic conversational agents, generative language models, and language understanding models.
Empathetic BERT2BERT Conversational Model:
Learning Arabic Language Generation with Little Data
Our latest contribution to Arabic Conversational AI leverages knowledge transfer from AraBERT in a BERT2BERT architecture. We address the low resource challenges and achieve sota results in open domain empathetic response generation.
Paper: https://arxiv.org/abs/2103.04353
Abstract: Enabling empathetic behavior in Arabic dialogue agents is an important aspect of building human-like conversational models. While Arabic Natural Language Processing has seen significant advances in Natural Language Understanding (NLU) with language models such as AraBERT, Natural Language Generation (NLG) remains a challenge. The shortcomings of NLG encoder-decoder models are primarily due to the lack of Arabic datasets suitable to train NLG models such as conversational agents. To overcome this issue, we propose a transformer-based encoder-decoder initialized with AraBERT parameters. By initializing the weights of the encoder and decoder with AraBERT pre-trained weights, our model was able to leverage knowledge transfer and boost performance in response generation. To enable empathy in our conversational model, we train it using the ArabicEmpatheticDialogues dataset and achieve high performance in empathetic response generation. Specifically, our model achieved a low perplexity value of 17.0 and an increase in 5 BLEU points compared to the previous state-of-the-art model. Also, our proposed model was rated highly by 85 human evaluators, validating its high capability in exhibiting empathy while generating relevant and fluent responses in open-domain settings.
AraGPT2:
Pre-Trained Transformer for Arabic Language Generation
AraGPT2 is a 1.5B transformer model, the largest for Arabic, trained on 77GB of text for 9 days with a TPUv3-128. The model can generate news articles that are difficult to distinguish from human-written articles. AraGPT2 shows impressive Zero-shot performance on trivia QA.
Paper: arxiv.org/abs/2012.15520
GitHub: https://github.com/aub-mind/arabert/tree/master/aragpt2
Abstract: Recently, pre-trained transformer-based architectures have proven to be very efficient at language modeling and understanding, given that they are trained on a large enough corpus. Applications in language generation for Arabic are still lagging in comparison to other NLP advances primarily due to the lack of advanced Arabic language generation models. In this paper, we develop the first advanced Arabic language generation model, AraGPT2, trained from scratch on a large Arabic corpus of internet text and news articles. Our largest model, AraGPT2-mega, has 1.46 billion parameters, which makes it the largest Arabic language model available. The Mega model was evaluated and showed success on different tasks including synthetic news generation, and zero-shot question answering. For text generation, our best model achieves a perplexity of 29.8 on held-out Wikipedia articles. A study conducted with human evaluators showed the significant success of AraGPT2-mega in generating news articles that are difficult to distinguish from articles written by humans. We thus develop and release an automatic discriminator model with a 98% percent accuracy in detecting model-generated text. The models are also publicly available, hoping to encourage new research directions and applications for Arabic NLP.
AraELECTRA:
Pre-Training Text Discriminators for Arabic Language Understanding
AraELECTRA is our latest advancements in Arabic Language Understanding. The model was trained on 77GB of Arabic text for 24 days. AraELECTRA achieves impressive performance, especially on Question Answering tasks.
Paper: https://arxiv.org/abs/2012.15516
Github: https://github.com/aub-mind/arabert/tree/master/araelectra
Abstract: Advances in English language representation enabled a more sample-efficient pre-training task by Efficiently Learning an Encoder that Classifies Token Replacements Accurately (ELECTRA). Which, instead of training a model to recover masked tokens, it trains a discriminator model to distinguish true input tokens from corrupted tokens that were replaced by a generator network. On the other hand, current Arabic language representation approaches rely only on pretraining via masked language modeling. In this paper, we develop an Arabic language representation model, which we name AraELECTRA. Our model is pretrained using the replaced token detection objective on large Arabic text corpora. We evaluate our model on multiple Arabic NLP tasks, including reading comprehension, sentiment analysis, and named-entity recognition and we show that AraELECTRA outperforms current state-of-the-art Arabic language representation models, given the same pretraining data and with even a smaller model size.
Acknowledgments:
This research was supported by the University Research Board (URB) at the American University of Beirut (AUB), and by the TFRC program, which we thank for the free access to cloud TPUs. We also thank As-Safir newspaper for the data access.
I’m impressed, I must say. Very rarely do I come across a blog thats both informative and entertaining, and let me tell you, you ve hit the nail on the head. Your blog is important.. Breakdown recovery high wycombe
Your platform has been a huge help to me – thank you! ทาง เข้า หวย ลาว
Good to become visiting your weblog again, it has been months for me. Nicely this article that i’ve been waited for so long. I will need this post to total my assignment in the college, and it has exact same topic together with your write-up. Thanks, good share. Zwembadbouwers Limburg
Thanks for the nice blog. It was very useful for me. I’m happy I found this blog. Thank you for sharing with us,I too always learn something new from your post. Zwembadbouwers Limburg
Writing with style and getting good compliments on the article is quite hard, to be honest.But you’ve done it so calmly and with so cool feeling and you’ve nailed the job. This article is possessed with style and I am giving good compliment. Best! Zwembaden polypropyleen
I recently came across your article and have been reading along. I want to express my admiration of your writing skill and ability to make readers read from the beginning to the end. I would like to read newer posts and to share my thoughts with you. Monoblock zwembaden
I would also motivate just about every person to save this web page for any favorite assistance to assist posted the appearance. Zwembad laten aanleggen
You completed certain reliable points there. I did a search on the subject and found nearly all persons will agree with your blog. Zwembad keramisch
This is a great inspiring article.I am pretty much pleased with your good work.You put really very helpful information… Zwembad laten bouwen
Very useful post. This is my first time i visit here. I found so many interesting stuff in your blog especially its discussion. Really its great article. Keep it up. Zwembaden polyester
I am overwhelmed by your post with such a nice topic. Usually I visit your blogs and get updated through the information you include but today’s blog would be the most appreciable. Well done! Zwembad laten plaatsen
Your blog provided us with valuable information to work with. Each & every tips of your post are awesome. Thanks a lot for sharing. Keep blogging, Zwembaden inox
I would also motivate just about every person to save this web page for any favorite assistance to assist posted the appearance. Zwembaden vinylester
I really loved reading your blog. It was very well authored and easy to understand. Unlike other blogs I have read which are really not that good.Thanks alot! Zwembaden vinylester
Nice post! This is a very nice blog that I will definitively come back to more times this year! Thanks for informative post. Monoblock zwembaden
Thanks for such a great post and the review, I am totally impressed! Keep stuff like this coming. Zwembaden polyester
I have read your article; it is very informative and helpful for me. I admire the valuable information you offer in your articles. Thanks for posting it. Zwembaden vinylester
Dream Home Riverside là một dự án bất động sản cao cấp tại Quận 8, TP.HCM, do Công ty TNHH Lý Khương phát triển. Với diện tích 2,2 ha, dự án bao gồm 3 tòa tháp 25 tầng, 2.096 căn hộ và 100 shophouse, mang đến không gian sống xanh mát, tiện nghi hiện đại. Điểm nhấn là hệ thống tiện ích đa dạng như hồ bơi, công viên ven sông, trung tâm thương mại và khu vui chơi.
Website: https://dreamhomeriverside.com.vn/
Thanks for offering such a clear and concise explanation. ยี่ กี lotto
I was very impressed by this post, this site has always been pleasant news Thank you very much for such an interesting post, and I meet them more often then I visited this site. anonymous bitcoin casinos
Your article not only provided valuable information but also sparked introspection and reflection on my part. nỏ hu
A must-visit site for Wisconsin residents! Their Medicare supplement plan options are well-explained, making it easy to pick the best one for your situation. Wisconsin Medicare Supplement Plans
I admire your ability to distill complex ideas into clear and concise prose. Nhà cái biggaming
This is a fascinating area of research! Developing Arabic empathetic conversational agents has the potential to bridge significant gaps in human-computer interaction, especially in Arabic-speaking communities. Leveraging pre-trained language models for this task could not only improve conversational fluency but also enhance the emotional intelligence of these agents.
I’m particularly curious about the challenges faced in adapting pre-trained language models to capture the nuances of Arabic dialects and cultural contexts. Were there specific techniques or datasets that proved particularly effective in training these models for empathetic responses?
Looking forward to seeing how this work evolves and contributes to both NLP and practical applications in customer service, mental health support, and more. Excellent work! try thiswolf cut straight hair
The menu truly comes to life with its variety of toppings and salsas. From the mild, refreshing tomato salsa to the spicy red-chili tomatillo salsa, there’s a level of heat for everyone. Other toppings include sautéed fajita veggies, crisp lettuce, creamy guacamole, and shredded cheese, giving you endless combinations to explore. With most toppings included in the base price (except guacamole), Chipotle encourages creativity without breaking the bank https://chipotlemenuus.com/
Weed.com is redefining cannabis shopping. Founded in 2021, their dedicated team delivers quality products and expert service. Check out their top-rated selection now! DELTA 8 Carts
Looking for fun and reliable party rentals in Tampa? Bounce Genie Tampa has you covered with the best options for any celebration. Book now! Bounce house Tampa
Very informative article.Really looking forward to read more. Really Cool. devops consultancy uae
You are very talented in your writing. I went through all of your articles, they were very interesting. ebet โปรโมชั่น
Great! It sounds good. Thanks for sharing.. Salt Trick
Thank you for your unwavering commitment to excellence. microgaming
I am grateful for the opportunity to learn from someone as knowledgeable as you.ติดต่อสอบถาม
Treat yourself to a luxurious experience with Miami’s leading male massage therapists. They specialize in customized massages for relaxation and overall wellness. male massage Miami
I think this is an informative post and it is very useful and knowledgeable. therefore, I would like to thank you for the efforts you have made in writing this article. Prijs van een zwembad
Thanks for sharing this information. I really like your blog post very much. You have really shared a informative and interesting blog post with people.. Prijs zwembad tuin
wow, great, I was wondering how to cure acne naturally. and found your site by google, learned a lot, now i’m a bit clear. I’ve bookmark your site and also add rss. keep us updated. Monoblock zwembad plaatsen
I finally found great post here.I will get back here. I just added your blog to my bookmark sites. thanks.Quality posts is the crucial to invite the visitors to visit the web page, that’s what this web page is providing. Monoblok zwembaden
Great Information sharing .. I am very happy to read this article .. thanks for giving us go through info.Fantastic nice. I appreciate this post. Polypropyleen zwembad aanleggen
Took me time to understand all of the comments, but I seriously enjoyed the write-up. It proved being really helpful to me and Im positive to all of the commenters right here! Its constantly nice when you can not only be informed, but also entertained! I am certain you had enjoyable writing this write-up. Monoblok zwembaden
The next time I read a blog, I hope that it doesnt disappoint me as much as this one. I mean, I know it was my choice to read, but I actually thought you have something interesting to say. All I hear is a bunch of whining about something that you could fix if you werent too busy looking for attention. Inbouw zwembad kopen
Interesting post. I Have Been wondering about this issue, so thanks for posting. Pretty cool post.It ‘s really very nice and Useful post.Thanks Wat kost een monoblock zwembad
I definitely enjoying every little bit of it and I have you bookmarked to check out new stuff you post. Aanleggen zwembad
I really loved reading your blog. It was very well authored and easy to understand. Unlike other blogs I have read which are really not that good.Thanks alot! Zwembad laten plaatsen
Interesting post. I Have Been wondering about this issue, so thanks for posting. Pretty cool post.It ‘s really very nice and Useful post.Thanks Zwembadbouwer
Really nice and interesting post. I was looking for this kind of information and enjoyed reading this one. Zwembad bouw
I think this is an informative post and it is very useful and knowledgeable. therefore, I would like to thank you for the efforts you have made in writing this article. Eigen zwembad aanleggen
I was surfing net and fortunately came across this site and found very interesting stuff here. Its really fun to read. I enjoyed a lot. Thanks for sharing this wonderful information. Zwembaden
Writing with style and getting good compliments on the article is quite hard, to be honest.But you’ve done it so calmly and with so cool feeling and you’ve nailed the job. This article is possessed with style and I am giving good compliment. Best! Zwembaden
Take the first step toward your dream smile at Aliso Creek Dental. Our Aliso Viejo dentist offers Invisalign braces to create straight, beautiful teeth. Call now! orthodontist
You guardians do an astounding web diary, and have some unfathomable substance. Continue doing extraordinary. Finance Legend