AraBERT : Pre-training BERT for Arabic Language Understanding

by | Feb 28, 2020 | NLP | 47,020 comments

Authors: Wissam Antoun, Fady Baly, Hazem Hajj

 

AraBERT is an Arabic pretrained language model based on Google’s BERT architecture. AraBERT uses the same BERT-Base config. More details are available in the AraBERT PAPER and in the AraBERT Meetup

There is two versions of the model AraBERTv0.1 and AraBERTv1, with the difference being that AraBERTv1 uses pre-segmented text where prefixes and suffixes were split using the Farasa Segmenter.

The model was trained on ~70M sentences or ~23GB of Arabic text with ~3B words.

Source Code Repository: https://github.com/aub-mind/arabert
Paper: https://www.aclweb.org/anthology/2020.osact-1.2.pdf

 

Results (Accuracy)

We evaluate both AraBERT models on different downstream tasks and compare it to mBERT, and other state of the art models (To the extent of our knowledge). The Tasks were Sentiment Analysis on 6 different datasets (HARD, ASTD-Balanced, ArsenTD-Lev, LABR, ArSaS), Named Entity Recognition with the ANERcorp, and Arabic Question Answering on Arabic-SQuAD and ARCD

Task prev. SOTA mBERT AraBERTv0.1 AraBERTv1
HARD 95.7 ElJundi et.al. 95.7 96.2 96.1
ASTD 86.5 ElJundi et.al. 80.1 92.2 92.6
ArsenTD-Lev 52.4 ElJundi et.al. 51 58.9 59.4
AJGT 93 Dahou et.al. 83.6 93.1 93.8
LABR 87.5 Dahou et.al. 83 85.9 86.7
ANERcorp 81.7 (BiLSTM-CRF) 78.4 84.2 81.9
ARCD mBERT EM:34.2 F1: 61.3 EM:51.14 F1:82.13 EM:54.84 F1: 82.15

Model Weights and Vocab Download

Models AraBERTv0.1 AraBERTv1
TensorFlow Drive Link Drive Link
PyTorch Drive_Link Drive_Link

You can find the PyTorch models in HuggingFace’s Transformer Library under the aubmindlab username

If you used this model please cite us as:

@inproceedings{antoun2020arabert,
  title={AraBERT: Transformer-based Model for Arabic Language Understanding},
  author={Antoun, Wissam and Baly, Fady and Hajj, Hazem},
  booktitle={LREC 2020 Workshop Language Resources and Evaluation Conference 11--16 May 2020},
  pages={9}
}

Acknowledgments

Thanks to TensorFlow Research Cloud (TFRC) for the free access to Cloud TPUs, couldn’t have done it without this program, and to the AUB MIND Lab Members for the continous support. Also thanks to Yakshof and Assafir for data and storage access. Another thanks for Habib Rahal (https://www.behance.net/rahalhabib), for putting a face to AraBERT.

Contacts

Wissam Antoun: Linkedin | Twitter | Github | wfa07@mail.aub.edu | wissam.antoun@gmail.com

Fady Baly: Linkedin | Twitter | Github | fgb06@mail.aub.edu | baly.fady@gmail.com

We are looking for sponsors to train BERT-Large and other Transformer models, the sponsor only needs to cover to data storage and compute cost of the generating the pretraining data

47,020 Comments

← Older Comments
Newer Comments →
  1. tayyaba khan on March 9, 2025 at 8:25 am

    Very informative post! There is a lot of information here that can help any business get started with a successful social networking campaign. olxtoto

    Reply
  2. ed sheeran details the lovestruck jitters in sweet new single … on March 9, 2025 at 8:30 am

    This is my first time i visit here. I found so many entertaining stuff in your blog, especially its discussion. From the tons of comments on your articles, I guess I am not the only one having all the leisure here! Keep up the good work. I have been meaning to write something like this on my website and you have given me an idea. ed sheeran details the lovestruck jitters in sweet new single …

    Reply
  3. toto macau on March 9, 2025 at 8:31 am

    Hello, you used to write great, but the last few posts have been kinda boring¡K I miss your tremendous writings. Past few posts are just a little out of track! come on! toto macau

    Reply
  4. mytyles - wall and f on March 9, 2025 at 8:34 am

    Thanks for the nice blog. It was very useful for me. I m happy I found this blog. Thank you for sharing with us,I too always learn something new from your post. mytyles – wall and f

    Reply
  5. ed sheeran details the lovestruck jitters in sweet new single … on March 9, 2025 at 8:37 am

    This is my first time i visit here. I found so many entertaining stuff in your blog, especially its discussion. From the tons of comments on your articles, I guess I am not the only one having all the leisure here! Keep up the good work. I have been meaning to write something like this on my website and you have given me an idea. crypto 30x .com

    Reply
  7. tayyaba khan on March 9, 2025 at 8:42 am

    Thanks for taking the time to discuss this, I feel strongly about it and love learning more on this topic. If possible, as you gain expertise, would you mind updating your blog with extra information? It is extremely helpful for me. olxtoto resmi

    Reply
  9. ularpanama on March 9, 2025 at 8:47 am

    An interesting discussion may be worth comment. I’m sure that you can write read more about this topic, it will not be described as a taboo subject but typically everyone is insufficient to chat on such topics. To another. Cheers ularpanama

    Reply
  10. Software activation keys on March 9, 2025 at 8:51 am

    Outstanding post, you have pointed out some superb points , I as well believe this s a very superb website. Software activation keys

    Reply
  11. bandar slot on March 9, 2025 at 8:54 am

    A bandar slot is an online platform that offers various slot games for players to enjoy. These platforms are popular for their fun gameplay and chances to win big prizes. With many themes and features, a bandar slot provides a thrilling experience. Players can spin the reels from the comfort of their homes and have the opportunity to win exciting rewards.

    Reply
  12. mwe on March 9, 2025 at 8:54 am

    I appreciate everything you have added to my knowledge base.Admiring the time and effort you put into your blog and detailed information you offer.Thanks. 신용카드현금화

    Reply
  13. اسید مخصوص ساروج on March 9, 2025 at 8:56 am

    Hello! I would wish to give a enormous thumbs up to the wonderful information you may have here about this post. I will be returning to your blog site to get more detailed soon. اسید مخصوص ساروج

    Reply
  15. mwe on March 9, 2025 at 8:56 am

    Wow! Such an amazing and helpful post this is. I really really love it. It’s so good and so awesome. I am just amazed. I hope that you continue to do your work like this in the future also cemaratoto

    Reply
  16. mwe on March 9, 2025 at 9:01 am

    This is such a great resource that you are providing and you give it away for free. I love seeing blog that understand the value of providing a quality resource for free. cemaratoto

    Reply
  17. mwe on March 9, 2025 at 9:02 am

    Very efficiently written information. It will be beneficial to anybody who utilizes it, including me. Keep up the good work. For sure i will check out more posts. This site seems to get a good amount of visitors. situs toto slot

    Reply
  19. shanu on March 9, 2025 at 9:05 am

    フィリピン・セブ留学の無料手配を行う留学エージェント。定期的な現地視察で学校内外のリアルな情報提供が可能です。 セブ留学

    Reply
  20. mwe on March 9, 2025 at 9:07 am

    We are really grateful for your blog post. You will find a lot of approaches after visiting your post. I was exactly searching for. Thanks for such post and please keep it up. Great work. situs toto slot

    Reply
  22. https://pbv88.city/ on March 9, 2025 at 9:09 am

    PBN88 – https://pbv88.city/ là nhà cái cá cược trực tuyến uy tín, cung cấp đa dạng các sản phẩm giải trí như cá cược thể thao, game bài trực tuyến, slot game, bắn cá và nhiều trò chơi hấp dẫn khác. Với nền tảng công nghệ hiện đại, PBN88 mang đến trải nghiệm cá cược an toàn, minh bạch và tiện lợi.

    Reply
  23. crypto 30x .com on March 9, 2025 at 9:11 am

    Thanks for the nice blog. It was very useful for me. I m happy I found this blog. Thank you for sharing with us,I too always learn something new from your post. crypto 30x .com

    Reply
  24. Fatima Javaid on March 9, 2025 at 9:13 am

    Slot Thailand menjadi salah satu pilihan favorit bagi pecinta permainan slot online karena menawarkan berbagai jenis permainan menarik dengan fitur unggulan. Dengan perkembangan teknologi, slot Thailand kini dapat dimainkan dengan mudah melalui perangkat mobile maupun desktop, memberikan pengalaman bermain yang praktis dan menyenangkan. Selain itu, banyak situs slot Thailand menyediakan berbagai bonus dan promo menarik yang membuat permainan semakin seru.

    Reply
  25. michujobs on March 9, 2025 at 9:17 am

    I have express a few of the articles on your website now, and I really like your style of blogging. I added it to my favorite’s blog site list and will be checking back soon… michujobs

    Reply
  26. ayikatourbali on March 9, 2025 at 9:17 am

    SOV777 adalah situs games slot online yang menyediakan beragam jenis link slot gacor terutama pada kalangan slot777 yang bertujuan memberikan kesempatan bermain game slot gampang menang. Kunjungi Disini: Situs slot online

    Reply
  28. crypto30x.com tnt on March 9, 2025 at 9:23 am

    I have express a few of the articles on your website now, and I really like your style of blogging. I added it to my favorite’s blog site list and will be checking back soon… crypto30x.com tnt

    Reply
  29. mwe on March 9, 2025 at 9:25 am

    This is my first time visit here. From the tons of comments on your articles,I guess I am not only one having all the enjoyment right here! 카드 현금화

    Reply
  30. shanu on March 9, 2025 at 9:28 am

    フィリピン・セブ留学の無料手配を行う留学エージェント。定期的な現地視察で学校内外のリアルな情報提供が可能です。 セブ留学

    Reply
  31. how to say kiolopobgofit on March 9, 2025 at 9:28 am

    Thanks for the nice blog. It was very useful for me. I m happy I found this blog. Thank you for sharing with us,I too always learn something new from your post. how to say kiolopobgofit

    Reply
  32. mwe on March 9, 2025 at 9:32 am

    Thanks for taking the time to discuss this, I feel strongly about it and love learning more on this topic. If possible, as you gain expertise, would you mind updating your blog with extra information? It is extremely helpful for me. cemaratoto

    Reply
  33. mejor minero ASIC on March 9, 2025 at 9:33 am

    Want to mine Alephium profitably? OneMiners.es offers tailored hosting solutions! mejor minero ASIC

    Reply
  34. mwe on March 9, 2025 at 9:37 am

    I wanted to thank you for this excellent read!! I definitely loved every little bit of it. I have you bookmarked your site to check out the new stuff you post. toto togel 4d

    Reply
  35. money6x.com save money on March 9, 2025 at 9:37 am

    Thanks for the nice blog. It was very useful for me. I m happy I found this blog. Thank you for sharing with us,I too always learn something new from your post. money6x.com save money

    Reply
  36. financial updates aggr8finance on March 9, 2025 at 9:41 am

    I have express a few of the articles on your website now, and I really like your style of blogging. I added it to my favorite’s blog site list and will be checking back soon… financial updates aggr8finance

    Reply
  37. icryptox.com future on March 9, 2025 at 9:44 am

    Thanks for the nice blog. It was very useful for me. I m happy I found this blog. Thank you for sharing with us,I too always learn something new from your post. icryptox.com future

    Reply
  38. mwe on March 9, 2025 at 9:45 am

    You have a good point here!I totally agree with what you have said!!Thanks for sharing your views…hope more people will read this article!!! 신용카드현금화

    Reply
  39. shanu on March 9, 2025 at 9:47 am

    フィリピン・セブ留学の無料手配を行う留学エージェント。定期的な現地視察で学校内外のリアルな情報提供が可能です。 セブ留学

    Reply
  40. 5starsstocks.com staples on March 9, 2025 at 9:48 am

    I have express a few of the articles on your website now, and I really like your style of blogging. I added it to my favorite’s blog site list and will be checking back soon… 5starsstocks.com staples

    Reply
  41. addd on March 9, 2025 at 9:50 am

    Awesome blog. I enjoyed reading your articles. This is truly a great read for me. I have bookmarked it and I am looking forward to reading new articles. Keep up the good work!NowCollege 1v1

    Reply
  42. techyflavors on March 9, 2025 at 9:50 am

    I’ve got a comparable fascination it is the webpage understand every little thing thoroughly in addition to well then, i’ll really know what you think that. techyflavors

    Reply
  43. mwe on March 9, 2025 at 9:50 am

    thanks for this usefull article, waiting for this article like this again. cemaratoto

    Reply
  44. adddsada on March 9, 2025 at 9:51 am

    Awesome blog. I enjoyed reading your articles. This is truly a great read for me. I have bookmarked it and I am looking forward to reading new articles. Keep up the good work!NowCollege 1v1

    Reply
  45. DADDD on March 9, 2025 at 9:53 am

    I just got to this amazing site not long ago. I was actually captured with the piece of resources you have got here. Big thumbs up for making such wonderful blog page!bageltechnews social media news

    Reply
  46. well health organic skin care on March 9, 2025 at 9:56 am

    Acknowledges intended for penmanship a really valuable column, When i happened adjacent to your site other than estimate a couple aid. I’d prefer ones develop connected with manuscript… well health organic skin care

    Reply
  47. success100x.com factors on March 9, 2025 at 9:57 am

    I have express a few of the articles on your website now, and I really like your style of blogging. I added it to my favorite’s blog site list and will be checking back soon… success100x.com factors

    Reply
  48. mwe on March 9, 2025 at 9:57 am

    This is such a great resource that you are providing and you give it away for free. I love seeing blog that understand the value of providing a quality resource for free. situs toto slot

    Reply
  49. agen on March 9, 2025 at 9:58 am

    This is very educational content and written well for a change. It’s nice to see that some people still understand how to write a quality post.! agen ipototo

    Reply
  50. DADD on March 9, 2025 at 9:58 am

    Nice information, valuable and excellent design, as share good stuff with good ideas and concepts, lots of great information and inspiration, both of which I need, thanks to offer such a helpful information here.Crew CloudySocial

    Reply
← Older Comments
Newer Comments →

Submit a Comment

Your email address will not be published. Required fields are marked *