We are pleased to announce that we have 3 papers accepted to The Sixth Arabic Natural Language Processing Workshop (WANLP 2021) co-located with EACL 2021. Authored by our talented team members: Tarek Naous, Wissam Antoun, Reem Mahmoud, Fady Baly under the supervision of Prof. Hazem Hajj. The papers target Arabic empathetic conversational agents, generative language models, and language understanding models.
Empathetic BERT2BERT Conversational Model:
Learning Arabic Language Generation with Little Data
Our latest contribution to Arabic Conversational AI leverages knowledge transfer from AraBERT in a BERT2BERT architecture. We address the low resource challenges and achieve sota results in open domain empathetic response generation.
Paper: https://arxiv.org/abs/2103.04353
Abstract: Enabling empathetic behavior in Arabic dialogue agents is an important aspect of building human-like conversational models. While Arabic Natural Language Processing has seen significant advances in Natural Language Understanding (NLU) with language models such as AraBERT, Natural Language Generation (NLG) remains a challenge. The shortcomings of NLG encoder-decoder models are primarily due to the lack of Arabic datasets suitable to train NLG models such as conversational agents. To overcome this issue, we propose a transformer-based encoder-decoder initialized with AraBERT parameters. By initializing the weights of the encoder and decoder with AraBERT pre-trained weights, our model was able to leverage knowledge transfer and boost performance in response generation. To enable empathy in our conversational model, we train it using the ArabicEmpatheticDialogues dataset and achieve high performance in empathetic response generation. Specifically, our model achieved a low perplexity value of 17.0 and an increase in 5 BLEU points compared to the previous state-of-the-art model. Also, our proposed model was rated highly by 85 human evaluators, validating its high capability in exhibiting empathy while generating relevant and fluent responses in open-domain settings.
AraGPT2:
Pre-Trained Transformer for Arabic Language Generation
AraGPT2 is a 1.5B transformer model, the largest for Arabic, trained on 77GB of text for 9 days with a TPUv3-128. The model can generate news articles that are difficult to distinguish from human-written articles. AraGPT2 shows impressive Zero-shot performance on trivia QA.
Paper: arxiv.org/abs/2012.15520
GitHub: https://github.com/aub-mind/arabert/tree/master/aragpt2
Abstract: Recently, pre-trained transformer-based architectures have proven to be very efficient at language modeling and understanding, given that they are trained on a large enough corpus. Applications in language generation for Arabic are still lagging in comparison to other NLP advances primarily due to the lack of advanced Arabic language generation models. In this paper, we develop the first advanced Arabic language generation model, AraGPT2, trained from scratch on a large Arabic corpus of internet text and news articles. Our largest model, AraGPT2-mega, has 1.46 billion parameters, which makes it the largest Arabic language model available. The Mega model was evaluated and showed success on different tasks including synthetic news generation, and zero-shot question answering. For text generation, our best model achieves a perplexity of 29.8 on held-out Wikipedia articles. A study conducted with human evaluators showed the significant success of AraGPT2-mega in generating news articles that are difficult to distinguish from articles written by humans. We thus develop and release an automatic discriminator model with a 98% percent accuracy in detecting model-generated text. The models are also publicly available, hoping to encourage new research directions and applications for Arabic NLP.
AraELECTRA:
Pre-Training Text Discriminators for Arabic Language Understanding
AraELECTRA is our latest advancements in Arabic Language Understanding. The model was trained on 77GB of Arabic text for 24 days. AraELECTRA achieves impressive performance, especially on Question Answering tasks.
Paper: https://arxiv.org/abs/2012.15516
Github: https://github.com/aub-mind/arabert/tree/master/araelectra
Abstract: Advances in English language representation enabled a more sample-efficient pre-training task by Efficiently Learning an Encoder that Classifies Token Replacements Accurately (ELECTRA). Which, instead of training a model to recover masked tokens, it trains a discriminator model to distinguish true input tokens from corrupted tokens that were replaced by a generator network. On the other hand, current Arabic language representation approaches rely only on pretraining via masked language modeling. In this paper, we develop an Arabic language representation model, which we name AraELECTRA. Our model is pretrained using the replaced token detection objective on large Arabic text corpora. We evaluate our model on multiple Arabic NLP tasks, including reading comprehension, sentiment analysis, and named-entity recognition and we show that AraELECTRA outperforms current state-of-the-art Arabic language representation models, given the same pretraining data and with even a smaller model size.
Acknowledgments:
This research was supported by the University Research Board (URB) at the American University of Beirut (AUB), and by the TFRC program, which we thank for the free access to cloud TPUs. We also thank As-Safir newspaper for the data access.
This is a great inspiring article. สมัคร แทง หวย บี
Great service from Diamond Pest Control. They quickly resolved our pest issue, and their team was polite and efficient. Highly recommend them for reliable pest solutions. Diamond Pest Control
Going to graduate school was a positive decision for me. I enjoyed the coursework, the presentations, the fellow students, and the professors. And since my company reimbursed 100% of the tuition, the only cost that I had to pay on my own was for books and supplies. Otherwise, I received a free master’s degree. All that I had to invest was my time. Buitenzwembad
Your blog provided us with valuable information to work with. Each & every tips of your post are awesome. Thanks a lot for sharing. Keep blogging, Prijs polypropyleen zwembad aanleggen
I am overwhelmed by your post with such a nice topic. Usually I visit your blogs and get updated through the information you include but today’s blog would be the most appreciable. Well done! Polypropyleen zwembad voordelen
A debt of gratitude is in order for giving late reports with respect to the worry, I anticipate read more. Zonnepanelen Zero Power Leuven
Always so interesting to visit your site.What a great info, thank you for sharing. this will help me so much in my learning Aqua Pura zwembaden
I wanted to leave a little comment to support you and wish you a good continuation. Wishing you the best of luck for all your blogging efforts. Poolfab
This is a great inspiring article.I am pretty much pleased with your good work.You put really very helpful information… Aquapura.nl zwembaden
I was surfing net and fortunately came across this site and found very interesting stuff here. Its really fun to read. I enjoyed a lot. Thanks for sharing this wonderful information. Prijs crepi gevel
Writing with style and getting good compliments on the article is quite hard, to be honest.But you’ve done it so calmly and with so cool feeling and you’ve nailed the job. This article is possessed with style and I am giving good compliment. Best! Zwembad Uniek zwembaden
Superbly written article, if only all bloggers offered the same content as you, the internet would be a far better place.. Zwembad laten bouwen
Always so interesting to visit your site.What a great info, thank you for sharing. this will help me so much in my learning Batterij voor zonnepanelen
This is very useful post for me. This will absolutely going to help me in my project. Zwembadaanleg
This is a great inspiring article.I am pretty much pleased with your good work.You put really very helpful information… Eigen zwembad aanleggen
Hey There. I found your blog using msn. This is a very well written article. I’ll be sure to bookmark it and come back to read more of your useful info. Thanks for the post. I’ll definitely return. Zwembad op maat
Truly, this article is really one of the very best in the history of articles. I am a antique ’Article’ collector and I sometimes read some new articles if I find them interesting. And I found this one pretty fascinating and it should go into my collection. Very good work! Installateur zonnepanelen Leuven
Koitoto adalah tempat di mana keberuntungan Anda menunggu! koitoto
This article highlights the pressing challenges uninsured individualsindividuals encounter, especially when it comes to accessing essential services like healthcare, transportation, and mental health support. It’s encouraging to see services like Surya Immediate Medical Care and Aspen Dental in Albany stepping up to provide much-needed assistance and alleviate some of these burdens.
For those interested in understanding land value assessments, the platform landvalue.au offers valuable insights, helping individuals make informed decisions in real estate investments.
If you’re considering both healthcare access and property investments, staying informed about these issues can make a significant difference in your overall well-being and financial planning. Would you like help with any further details on either topic?
This is my first time i visit here. I found so many interesting stuff in your blog especially its discussion. From the tons of comments on your articles, I guess I am not the only one having all the enjoyment here! keep up the good work koitoto
I am grateful for the depth of thought and research that went into your article. kiếm thêm thu nhập
I value the expertise and wisdom you bring to the table.
หวย มาเล ย์ วัน นี้ 4d เมื่อ วาน
Wisconsin Medicare Supplement Plans helped me narrow down my choices for a Medicare supplement plan in WI. Medicare supplement plans in Wisconsin
It’s really interesting to hear how you’ve built relationships with your guests. It seems like you’ve created a truly welcoming and memorable experience for them ZYN Nicotine Pouches.
Null’s Brawl’s Mod APK offers Brawl Stars players unlimited resources, custom skins, and exclusive features. It enhances the game experience with fast progression, no ads, and access to special modes and events. View here!
I value the expertise and wisdom you bring to the table.มาเล 4d
Great post I would like to thank you for the efforts you have made in writing this interesting and knowledgeable article. bokep
This article is good. I will come to read it again
สมัคร 123bet
I look forward to reading more of your work in the future. Máy quay hũ
Nice post! This is a very nice blog that I will definitively come back to more times this year! Thanks for informative post.
3 cây
Packaging Castle provides high-quality packaging box, tailored to securely protect your products while enhancing your brand’s presentation. These durable, customizable boxes ensure a professional delivery and memorable unboxing experience.
Its as if you had a great grasp on the subject matter, but you forgot to include your readers. Perhaps you should think about this from more than one angle. Aanleggen zwembad tuin
wow… what a great blog, this writter who wrote this article it’s realy a great blogger, this article so inspiring me to be a better person Aquapura zwembadbouw
I’m excited to uncover this page. I need to to thank you for ones time for this particularly fantastic read!! I definitely really liked every part of it and i also have you saved to fav to look at new information in your site. Zwembadaanleg Zwembaduniek.nl
I’m happy I located this blog! From time to time, students want to cognitive the keys of productive literary essays composing. Your first-class knowledge about this good post can become a proper basis for such people. nice one Lebaindor.be zwembaden
I am genuinely thankful to the holder of this web page who has shared this wonderful paragraph at at this place Monoblock zwembaden
Going to graduate school was a positive decision for me. I enjoyed the coursework, the presentations, the fellow students, and the professors. And since my company reimbursed 100% of the tuition, the only cost that I had to pay on my own was for books and supplies. Otherwise, I received a free master’s degree. All that I had to invest was my time. Monoblock zwembad
This is a great inspiring article.I am pretty much pleased with your good work.You put really very helpful information… Monoblock zwembad
Nice blog and absolutely outstanding. You can do something much better but i still say this perfect.Keep trying for the best. Prijs zwembad
Wow! Such an amazing and helpful post this is. I really really love it. It’s so good and so awesome. I am just amazed. I hope that you continue to do your work like this in the future also. Zonnepanelen van de Zonnepanelen-centrale.be
Nice post! This is a very nice blog that I will definitively come back to more times this year! Thanks for informative post. Zwembaden polypropyleen
Thanks for such a great post and the review, I am totally impressed! Keep stuff like this coming. Inbouw zwembad
I really loved reading your blog. It was very well authored and easy to understand. Unlike other blogs I have read which are really not that good.Thanks alot! Inbouw zwembad
This is a great inspiring article.I am pretty much pleased with your good work.You put really very helpful information… Zwembad tuin prijs
Thank you for helping people get the information they need. Great stuff as usual. Keep up the great work!!! Aanleggen zwembad tuin
I am always searching online for articles that can help me. There is obviously a lot to know about this. I think you made some good points in Features also. Keep working, great job ! Zwembaden polypropyleen
Thank you for your unwavering commitment to excellence.
sòng bạc l68
Great article! This is the kind of high-quality content that keeps visitors coming back. Please keep up the great work! Dubai’s investor-friendly policies, world-class infrastructure, and business-friendly ecosystem make it one of the top destinations for company formation. Whether you’re launching a startup or expanding an established business, Dubai offers numerous incentives that can significantly benefit your enterprise.
This blog post was a great read! The advantages of incorporating a company in Singapore. With its strong support for businesses, favorable tax incentives, and strategic location, Singapore is a thriving hub for entrepreneurs and investors. The insights into the tax policies and business-friendly environment were particularly helpful. I’m definitely considering Singapore for my next venture. Thanks for sharing such valuable information!
Excellent article! This type of excellent material is what attracts and retains visitors. Keep up the fantastic work, please! Dubai’s investor-friendly policies, world-class infrastructure, and business-friendly ecosystem make it one of the top destinations. Fairfax Domestic Violence Lawyer A domestic violence attorney in Fairfax, Virginia, it’s critical to consider the attorney’s reputation, experience, and client testimonials in managing these delicate and grave matters. an attorney who focuses on issues involving domestic abuse. Protection orders, criminal charges, child custody, and delicate emotional dynamics are just a few of the particular legal issues associated with domestic abuse. These intricacies will be easier for a specialist attorney to manage.