We are pleased to announce that we have 3 papers accepted to The Sixth Arabic Natural Language Processing Workshop (WANLP 2021) co-located with EACL 2021. Authored by our talented team members: Tarek Naous, Wissam Antoun, Reem Mahmoud, Fady Baly under the supervision of Prof. Hazem Hajj. The papers target Arabic empathetic conversational agents, generative language models, and language understanding models.
Empathetic BERT2BERT Conversational Model:
Learning Arabic Language Generation with Little Data
Our latest contribution to Arabic Conversational AI leverages knowledge transfer from AraBERT in a BERT2BERT architecture. We address the low resource challenges and achieve sota results in open domain empathetic response generation.
Paper: https://arxiv.org/abs/2103.04353
Abstract: Enabling empathetic behavior in Arabic dialogue agents is an important aspect of building human-like conversational models. While Arabic Natural Language Processing has seen significant advances in Natural Language Understanding (NLU) with language models such as AraBERT, Natural Language Generation (NLG) remains a challenge. The shortcomings of NLG encoder-decoder models are primarily due to the lack of Arabic datasets suitable to train NLG models such as conversational agents. To overcome this issue, we propose a transformer-based encoder-decoder initialized with AraBERT parameters. By initializing the weights of the encoder and decoder with AraBERT pre-trained weights, our model was able to leverage knowledge transfer and boost performance in response generation. To enable empathy in our conversational model, we train it using the ArabicEmpatheticDialogues dataset and achieve high performance in empathetic response generation. Specifically, our model achieved a low perplexity value of 17.0 and an increase in 5 BLEU points compared to the previous state-of-the-art model. Also, our proposed model was rated highly by 85 human evaluators, validating its high capability in exhibiting empathy while generating relevant and fluent responses in open-domain settings.
AraGPT2:
Pre-Trained Transformer for Arabic Language Generation
AraGPT2 is a 1.5B transformer model, the largest for Arabic, trained on 77GB of text for 9 days with a TPUv3-128. The model can generate news articles that are difficult to distinguish from human-written articles. AraGPT2 shows impressive Zero-shot performance on trivia QA.
Paper: arxiv.org/abs/2012.15520
GitHub: https://github.com/aub-mind/arabert/tree/master/aragpt2
Abstract: Recently, pre-trained transformer-based architectures have proven to be very efficient at language modeling and understanding, given that they are trained on a large enough corpus. Applications in language generation for Arabic are still lagging in comparison to other NLP advances primarily due to the lack of advanced Arabic language generation models. In this paper, we develop the first advanced Arabic language generation model, AraGPT2, trained from scratch on a large Arabic corpus of internet text and news articles. Our largest model, AraGPT2-mega, has 1.46 billion parameters, which makes it the largest Arabic language model available. The Mega model was evaluated and showed success on different tasks including synthetic news generation, and zero-shot question answering. For text generation, our best model achieves a perplexity of 29.8 on held-out Wikipedia articles. A study conducted with human evaluators showed the significant success of AraGPT2-mega in generating news articles that are difficult to distinguish from articles written by humans. We thus develop and release an automatic discriminator model with a 98% percent accuracy in detecting model-generated text. The models are also publicly available, hoping to encourage new research directions and applications for Arabic NLP.
AraELECTRA:
Pre-Training Text Discriminators for Arabic Language Understanding
AraELECTRA is our latest advancements in Arabic Language Understanding. The model was trained on 77GB of Arabic text for 24 days. AraELECTRA achieves impressive performance, especially on Question Answering tasks.
Paper: https://arxiv.org/abs/2012.15516
Github: https://github.com/aub-mind/arabert/tree/master/araelectra
Abstract: Advances in English language representation enabled a more sample-efficient pre-training task by Efficiently Learning an Encoder that Classifies Token Replacements Accurately (ELECTRA). Which, instead of training a model to recover masked tokens, it trains a discriminator model to distinguish true input tokens from corrupted tokens that were replaced by a generator network. On the other hand, current Arabic language representation approaches rely only on pretraining via masked language modeling. In this paper, we develop an Arabic language representation model, which we name AraELECTRA. Our model is pretrained using the replaced token detection objective on large Arabic text corpora. We evaluate our model on multiple Arabic NLP tasks, including reading comprehension, sentiment analysis, and named-entity recognition and we show that AraELECTRA outperforms current state-of-the-art Arabic language representation models, given the same pretraining data and with even a smaller model size.
Acknowledgments:
This research was supported by the University Research Board (URB) at the American University of Beirut (AUB), and by the TFRC program, which we thank for the free access to cloud TPUs. We also thank As-Safir newspaper for the data access.
I admire your ability to distill complex ideas into clear and concise prose.สมัคร หวย หุ้น เกาหลี
Your article has sparked a conversation that I believe is important and necessary.ลิงค์สมัครหวยเกาหลี
Thank you for your tireless pursuit of excellence in your writing.สอบถามหวยเกาหลี
In today’s digital age, having reliable cloud storage is essential for both personal and professional use. With numerous options available, Terabox APK download stands out as an innovative solution that provides users with a secure and efficient way to store and manage their files.
Thank you for your post, I look for such article along time, today i find it finally. this post give me lots of advise it is very useful for me. Aanleg zwembad tuin
If you don”t mind proceed with this extraordinary work and I anticipate a greater amount of your magnificent blog entries Zwembadaanleg tuin
I can see that you are an expert at your field! I am launching a website soon, and your information will be very useful for me.. Thanks for all your help and wishing you all the success in your business. Prijs spouwmuurisolatie
It’s late finding this act. At least, it’s a thing to be familiar with that there are such events exist. I agree with your Blog and I will be back to inspect it more in the future so please keep up your act. Prijs zwembad
Mmm.. good to be here in your article or post, whatever, I think I should also work hard for my own website like I see some good and updated working in your site. Spouwmuurisolatie
Its as if you had a great grasp on the subject matter, but you forgot to include your readers. Perhaps you should think about this from more than one angle. Kostprijs monoblock zwembad
It’s very informative and you are obviously very knowledgeable in this area. You have opened my eyes to varying views on this topic with interesting and solid content. Prijs zwembad
I would like to say that this blog really convinced me to do it! Thanks, very good post. Spouwisolatie
Awesome blog. I enjoyed reading your articles. This is truly a great read for me. I have bookmarked it and I am looking forward to reading new articles. Keep up the good work! Aanleg zwembaden
I curious more interest in some of them hope you will give more information on this topics in your next articles. Zonnepanelen Kopen: Vergelijk Prijzen en Bespaar
I can set up my new idea from this post. It gives in depth information. Thanks for this valuable information for all,.. Zwembadaanleg
If your looking for Online Illinois license plate sticker renewals then you have need to come to the right place.We offer the fastest Illinois license plate sticker renewals in the state. Muurisolatie
This is a great post. I like this topic.This site has lots of advantage.I found many interesting things from this site. It helps me in many ways.Thanks for posting this again. De complete gids voor het installeren van zonnepanelen – alles wat je moet weten
Very good points you wrote here..Great stuff…I think you’ve made some truly interesting points.Keep up the good work. Spouwmuurisolatie met EPS-parels
Spotify is ideal for making road trips more fun, with curated playlists and song suggestions that keep the drive exciting. It’s like having a DJ on the go who knows just what you need! For anyone wanting a reliable music app for travel, check out descargarspotapp.com.
The specialist in any field once started as an amateur. thabet
Intellect combined with moral qualities. helo88
That is the objective of genuine learning. 188bet
The intellect is not an empty container. thabet
To be loaded but a flame to be ignited. 188bet
Create and customize QR codes effortlessly! Our generator lets you share links and information in an instant, making engagement simple and funVisit mzad QR code generator
Null Brawl provides an exciting twist on the popular Brawl Stars game, allowing players to enjoy unlimited resources and unlocked characters. Available on various platforms, including the Apple Store, it enhances the gaming experience significantly.
“I really enjoyed this article! It offered some valuable insights into [topic]. For anyone interested in exploring related resources, I recommend checking out R&M vape for more information.”
Your article has left a lasting impression on me, and I am grateful for the opportunity to learn from your work. game nổ hũ uy tín
Gracefulness is not about attracting attention. 12bet
It’s about leaving a lasting impression. RR88 link
The post is excellent and provides numerous benefits, as it offers interesting content and aids in various ways. federal embezzlement lawyer Selecting the right federal embezzlement attorney is crucial for a successful case, as charges can lead to severe consequences like fines and jail time, and requires legal advice and emotional support.
There are countless awesome features of Pikashow mod Apk that make it superior to use. Check the highlighting characteristics of Pikashow here
Excellent Blog! I would like to thank for the efforts you have made in writing this post. I am hoping the same best work from you in the future as well. I wanted to thank you for this websites! Thanks for sharing. Great websites! Chapes.be
I think about it is most required for making more on this get engaged Seozoekmachineoptimalisatie.be
I am very enjoyed for this blog. Its an informative topic. It help me very much to solve some problems. Its opportunity are so fantastic and working style so speedy. Luchtluchtwarmtepompen.be
You completed certain reliable points there. I did a search on the subject and found nearly all persons will agree with your blog. Luchtwaterwarmtepompen.be
Thanks for such a great post and the review, I am totally impressed! Keep stuff like this coming. Warmtepompen-centrale.be
Going to graduate school was a positive decision for me. I enjoyed the coursework, the presentations, the fellow students, and the professors. And since my company reimbursed 100% of the tuition, the only cost that I had to pay on my own was for books and supplies. Otherwise, I received a free master’s degree. All that I had to invest was my time. Airco-airconditioning.be
Actually I read it yesterday but I had some thoughts about it and today I wanted to read it again because it is very well written. Zonnepanelenbatterij.be
Nice post! This is a very nice blog that I will definitively come back to more times this year! Thanks for informative post. Seowebdesign.be
After reading your article I was amazed. I know that you explain it very well. And I hope that other readers will also experience how I feel after reading your article. Massagesalon-limburg.be
I curious more interest in some of them hope you will give more information on this topics in your next articles. Seoleadgeneration.be
I can set up my new idea from this post. It gives in depth information. Thanks for this valuable information for all,.. A-aircool.be
I would like to thank you for the efforts you have made in writing this article. I am hoping the same best work from you in the future as well. In fact your creative writing abilities has inspired me to start my own Blog Engine blog now. Really the blogging is spreading its wings rapidly. Your write up is a fine example of it. Seo-zoekmachine-optimalisatie.be
Thank you for the post. I will definitely comeback. Zwembad-bouwen-tuin.nl
Its a great pleasure reading your post.Its full of information I am looking for and I love to post a comment that “The content of your post is awesome” Great work. Zwembad-plaatsen-tuin.nl
Going to graduate school was a positive decision for me. I enjoyed the coursework, the presentations, the fellow students, and the professors. And since my company reimbursed 100% of the tuition, the only cost that I had to pay on my own was for books and supplies. Otherwise, I received a free master’s degree. All that I had to invest was my time. Zwembad-aanleggen-tuin.nl
I’m constantly searching on the internet for posts that will help me. Too much is clearly to learn about this. I believe you created good quality items in Functions also. Keep working, congrats! 실시간무료스포츠중계보는곳
You’ve completed in excellent work. t suggest to my frtends ind personilly wtll certitnly dtgtt. t’m conftdent they’ll be gitned from thts webstte. 끈김없는 무료스포츠중계채널 홈런티비
The dynamic area of Hamra is known for its vibrant energy, cultural diversity, and nightlife. With BeiTeck, it’s simple to Rent an apartment in Hamra , where you’ll be at the center of exciting urban life.