We are pleased to announce that we have 3 papers accepted to The Sixth Arabic Natural Language Processing Workshop (WANLP 2021) co-located with EACL 2021. Authored by our talented team members: Tarek Naous, Wissam Antoun, Reem Mahmoud, Fady Baly under the supervision of Prof. Hazem Hajj. The papers target Arabic empathetic conversational agents, generative language models, and language understanding models.
Empathetic BERT2BERT Conversational Model:
Learning Arabic Language Generation with Little Data
Our latest contribution to Arabic Conversational AI leverages knowledge transfer from AraBERT in a BERT2BERT architecture. We address the low resource challenges and achieve sota results in open domain empathetic response generation.
Paper: https://arxiv.org/abs/2103.04353
Abstract: Enabling empathetic behavior in Arabic dialogue agents is an important aspect of building human-like conversational models. While Arabic Natural Language Processing has seen significant advances in Natural Language Understanding (NLU) with language models such as AraBERT, Natural Language Generation (NLG) remains a challenge. The shortcomings of NLG encoder-decoder models are primarily due to the lack of Arabic datasets suitable to train NLG models such as conversational agents. To overcome this issue, we propose a transformer-based encoder-decoder initialized with AraBERT parameters. By initializing the weights of the encoder and decoder with AraBERT pre-trained weights, our model was able to leverage knowledge transfer and boost performance in response generation. To enable empathy in our conversational model, we train it using the ArabicEmpatheticDialogues dataset and achieve high performance in empathetic response generation. Specifically, our model achieved a low perplexity value of 17.0 and an increase in 5 BLEU points compared to the previous state-of-the-art model. Also, our proposed model was rated highly by 85 human evaluators, validating its high capability in exhibiting empathy while generating relevant and fluent responses in open-domain settings.
AraGPT2:
Pre-Trained Transformer for Arabic Language Generation
AraGPT2 is a 1.5B transformer model, the largest for Arabic, trained on 77GB of text for 9 days with a TPUv3-128. The model can generate news articles that are difficult to distinguish from human-written articles. AraGPT2 shows impressive Zero-shot performance on trivia QA.
Paper: arxiv.org/abs/2012.15520
GitHub: https://github.com/aub-mind/arabert/tree/master/aragpt2
Abstract: Recently, pre-trained transformer-based architectures have proven to be very efficient at language modeling and understanding, given that they are trained on a large enough corpus. Applications in language generation for Arabic are still lagging in comparison to other NLP advances primarily due to the lack of advanced Arabic language generation models. In this paper, we develop the first advanced Arabic language generation model, AraGPT2, trained from scratch on a large Arabic corpus of internet text and news articles. Our largest model, AraGPT2-mega, has 1.46 billion parameters, which makes it the largest Arabic language model available. The Mega model was evaluated and showed success on different tasks including synthetic news generation, and zero-shot question answering. For text generation, our best model achieves a perplexity of 29.8 on held-out Wikipedia articles. A study conducted with human evaluators showed the significant success of AraGPT2-mega in generating news articles that are difficult to distinguish from articles written by humans. We thus develop and release an automatic discriminator model with a 98% percent accuracy in detecting model-generated text. The models are also publicly available, hoping to encourage new research directions and applications for Arabic NLP.
AraELECTRA:
Pre-Training Text Discriminators for Arabic Language Understanding
AraELECTRA is our latest advancements in Arabic Language Understanding. The model was trained on 77GB of Arabic text for 24 days. AraELECTRA achieves impressive performance, especially on Question Answering tasks.
Paper: https://arxiv.org/abs/2012.15516
Github: https://github.com/aub-mind/arabert/tree/master/araelectra
Abstract: Advances in English language representation enabled a more sample-efficient pre-training task by Efficiently Learning an Encoder that Classifies Token Replacements Accurately (ELECTRA). Which, instead of training a model to recover masked tokens, it trains a discriminator model to distinguish true input tokens from corrupted tokens that were replaced by a generator network. On the other hand, current Arabic language representation approaches rely only on pretraining via masked language modeling. In this paper, we develop an Arabic language representation model, which we name AraELECTRA. Our model is pretrained using the replaced token detection objective on large Arabic text corpora. We evaluate our model on multiple Arabic NLP tasks, including reading comprehension, sentiment analysis, and named-entity recognition and we show that AraELECTRA outperforms current state-of-the-art Arabic language representation models, given the same pretraining data and with even a smaller model size.
Acknowledgments:
This research was supported by the University Research Board (URB) at the American University of Beirut (AUB), and by the TFRC program, which we thank for the free access to cloud TPUs. We also thank As-Safir newspaper for the data access.
An orthodontist plays a crucial role in transforming smiles. It’s fascinating to learn how their expertise in teeth alignment and jaw structure can improve both appearance and function. Consulting an orthodontist can make all the difference in achieving a confident, healthy smile!
The Arabic NLP advancements discussed at WANLP 2021 highlight exciting progress in empathetic conversational agents. Such AI models could potentially assist industries, including accounting services in Dubai, by enhancing customer interactions and support efficiency.
Wazeslot slot depo 5k
Wazeslot slot depo 5k
Just wrapped up our move in London, Ontario with a great moving company! They were efficient, polite, and careful. Definitely worth hiring professionals! movers
Wendy’s menu features a variety of fresh options, including burgers, chicken sandwiches, salads, and frosties. Menu insights reveal a focus on quality ingredients and customization to cater to diverse tastes. Get prices now.
Not to individuals or possessions. ABC8
Mpo700 memberikan game slot yang seru dan penuh kemenangan! slot5000
Scheduling emails to be sent at optimal times can increase visibility and response rates, especially in email marketing campaigns. email list
Once you possess a dream. fun88
You must seize it and retain it unwaveringly. red88
Nice post! This is a very nice blog that I will definitively come back to more times this year! Thanks for informative post. Zwembad plaatsen tuin
Thanks for such a great post and the review, I am totally impressed! Keep stuff like this coming. Zonnepanelen
I really enjoy reading and also appreciate your work. Zwembad bouwen tuin
This is a great inspiring article.I am pretty much pleased with your good work.You put really very helpful information… Zonnepanelen installateur
Hey There. I found your blog using msn. This is a very well written article. I’ll be sure to bookmark it and come back to read more of your useful info. Thanks for the post. I’ll definitely return. Zwembadbouw
Truly, this article is really one of the very best in the history of articles. I am a antique ’Article’ collector and I sometimes read some new articles if I find them interesting. And I found this one pretty fascinating and it should go into my collection. Very good work! Zwembad aanleggen
Nice post! This is a very nice blog that I will definitively come back to more times this year! Thanks for informative post. Zwembad plaatsen tuin
Thanks for such a great post and the review, I am totally impressed! Keep stuff like this coming. Zonnepanelen Leuven
Always so interesting to visit your site.What a great info, thank you for sharing. this will help me so much in my learning Zwembad plaatsen tuin
This is a great inspiring article.I am pretty much pleased with your good work.You put really very helpful information… Zwembad plaatsen tuin
Very useful post. This is my first time i visit here. I found so many interesting stuff in your blog especially its discussion. Really its great article. Keep it up. Zonnepanelen
I am overwhelmed by your post with such a nice topic. Usually I visit your blogs and get updated through the information you include but today’s blog would be the most appreciable. Well done! Zonnepanelen Limburg
I’m happy I located this blog! From time to time, students want to cognitive the keys of productive literary essays composing. Your first-class knowledge about this good post can become a proper basis for such people. nice one Zonnepanelen Antwerpen
Thanks for such a great post and the review, I am totally impressed! Keep stuff like this coming. Zwembad bouwen tuin
Really impressed! Everything is very open and very clear clarification of issues. It contains truly facts. Your website is very valuable. Thanks for sharing. Zwembad aanleggen
Explore MBBS scholarships for Pakistani students in UK to pursue medical studies at top UK universities. These scholarships offer financial support for Pakistani students aspiring to study medicine in the UK.
Play Eggy Car Classroom 6x and enjoy this fun, addictive game while navigating a car with an egg on top. Overcome obstacles and complete levels in this exciting classroom-friendly version!
Your article is a shining example of the power of well-crafted writing. nỗ hu
thanks for sharing this post. virginia personal injury law All things considered, a favorable evaluation would highlight their commitment to customers, strong advocacy, and helpful demeanor under trying conditions.
Your article has sparked a conversation that I believe is important and necessary. ลาว สามัคคี
I’m grateful for your insights EdFinancial
It’s very informative and you are obviously very knowledgeable in this area. You have opened my eyes to varying views on this topic with interesting and solid content. Prijzen monoblock zwembad
Admiring the time and effort you put into your blog and detailed information you offer!.. Zwembad aanleggen
wow, great, I was wondering how to cure acne naturally. and found your site by google, learned a lot, now i’m a bit clear. I’ve bookmark your site and also add rss. keep us updated. Spouwmuur isoleren
I’m happy I located this blog! From time to time, students want to cognitive the keys of productive literary essays composing. Your first-class knowledge about this good post can become a proper basis for such people. nice one Zwembadbouw Limburg
Thanks for such a great post and the review, I am totally impressed! Keep stuff like this coming. Zwembad tuin
Its as if you had a great grasp on the subject matter, but you forgot to include your readers. Perhaps you should think about this from more than one angle. Spouwmuursiolatie kostprijs
This is a great inspiring article.I am pretty much pleased with your good work.You put really very helpful information… Kostprijs zwembad in de tuin
I was surfing net and fortunately came across this site and found very interesting stuff here. Its really fun to read. I enjoyed a lot. Thanks for sharing this wonderful information. Zwembad tuin
I am genuinely thankful to the holder of this web page who has shared this wonderful paragraph at at this place Plaatsen van een monoblock zwembad
Nice post! This is a very nice blog that I will definitively come back to more times this year! Thanks for informative post. Gevel in crepi
Thanks for such a great post and the review, I am totally impressed! Keep stuff like this coming. Plaatsen van een monoblock zwembad
This is a great inspiring article.I am pretty much pleased with your good work.You put really very helpful information… Crepi gevel
This is a great inspiring article.I am pretty much pleased with your good work.You put really very helpful information… Aanleggen zwembad
There’s no doubt i would fully rate it after i read what is the idea about this article. You did a nice job.. Zwembadbouw Limburg
I recently came across your article and have been reading along. I want to express my admiration of your writing skill and ability to make readers read from the beginning to the end. I would like to read newer posts and to share my thoughts with you. Zwembad aanleggen
The way you articulate your ideas demonstrates a deep understanding of the subject matter.หวย ขูด เกาหลี
I appreciate several from the Information which has been composed, and especially the remarks posted I will visit once more. 라이브스포츠중계
good information download this apkr