AraBERT : Pre-training BERT for Arabic Language Understanding

by | Feb 28, 2020 | NLP | 50,933 comments

Authors: Wissam Antoun, Fady Baly, Hazem Hajj

 

AraBERT is an Arabic pretrained language model based on Google’s BERT architecture. AraBERT uses the same BERT-Base config. More details are available in the AraBERT PAPER and in the AraBERT Meetup

There is two versions of the model AraBERTv0.1 and AraBERTv1, with the difference being that AraBERTv1 uses pre-segmented text where prefixes and suffixes were split using the Farasa Segmenter.

The model was trained on ~70M sentences or ~23GB of Arabic text with ~3B words.

Source Code Repository: https://github.com/aub-mind/arabert
Paper: https://www.aclweb.org/anthology/2020.osact-1.2.pdf

 

Results (Accuracy)

We evaluate both AraBERT models on different downstream tasks and compare it to mBERT, and other state of the art models (To the extent of our knowledge). The Tasks were Sentiment Analysis on 6 different datasets (HARD, ASTD-Balanced, ArsenTD-Lev, LABR, ArSaS), Named Entity Recognition with the ANERcorp, and Arabic Question Answering on Arabic-SQuAD and ARCD

Task prev. SOTA mBERT AraBERTv0.1 AraBERTv1
HARD 95.7 ElJundi et.al. 95.7 96.2 96.1
ASTD 86.5 ElJundi et.al. 80.1 92.2 92.6
ArsenTD-Lev 52.4 ElJundi et.al. 51 58.9 59.4
AJGT 93 Dahou et.al. 83.6 93.1 93.8
LABR 87.5 Dahou et.al. 83 85.9 86.7
ANERcorp 81.7 (BiLSTM-CRF) 78.4 84.2 81.9
ARCD mBERT EM:34.2 F1: 61.3 EM:51.14 F1:82.13 EM:54.84 F1: 82.15

Model Weights and Vocab Download

Models AraBERTv0.1 AraBERTv1
TensorFlow Drive Link Drive Link
PyTorch Drive_Link Drive_Link

You can find the PyTorch models in HuggingFace’s Transformer Library under the aubmindlab username

If you used this model please cite us as:

@inproceedings{antoun2020arabert,
  title={AraBERT: Transformer-based Model for Arabic Language Understanding},
  author={Antoun, Wissam and Baly, Fady and Hajj, Hazem},
  booktitle={LREC 2020 Workshop Language Resources and Evaluation Conference 11--16 May 2020},
  pages={9}
}

Acknowledgments

Thanks to TensorFlow Research Cloud (TFRC) for the free access to Cloud TPUs, couldn’t have done it without this program, and to the AUB MIND Lab Members for the continous support. Also thanks to Yakshof and Assafir for data and storage access. Another thanks for Habib Rahal (https://www.behance.net/rahalhabib), for putting a face to AraBERT.

Contacts

Wissam Antoun: Linkedin | Twitter | Github | wfa07@mail.aub.edu | wissam.antoun@gmail.com

Fady Baly: Linkedin | Twitter | Github | fgb06@mail.aub.edu | baly.fady@gmail.com

We are looking for sponsors to train BERT-Large and other Transformer models, the sponsor only needs to cover to data storage and compute cost of the generating the pretraining data

50,933 Comments

← Older Comments
Newer Comments →
  1. Liwovosa on March 23, 2025 at 9:02 am

    Pretty good post. I just stumbled upon your blog and wanted to say that I have really enjoyed reading your blog posts. Any way I’ll be subscribing to your feed and I hope you post again soon. Big thanks for the useful info. alexistogel

    Reply
  2. TRP Roller Chains RS 120-1 on March 23, 2025 at 9:02 am

    Informasi yang sangat berguna dan terorganisir dengan baik. Terima kasih atas usaha Anda! TRP Roller Chains RS 120-1

    Reply
  3. how to say kiolopobgofit on March 23, 2025 at 9:03 am

    It is very helpful, on the other hand it might be critical so as to look at using web page: how to say kiolopobgofit

    Reply
  4. 90phut tv on March 23, 2025 at 9:05 am

    90phut tv là nền tảng hàng đầu cung cấp link xem trực tiếp bóng đá miễn phí với chất lượng HD, tốc độ mượt mà và không giật lag. Cập nhật đầy đủ các trận đấu từ Ngoại Hạng Anh, La Liga, Champions League, Serie A và nhiều giải đấu hấp dẫn khác.

    Reply
  5. TRP Roller Chains RS 120-1 on March 23, 2025 at 9:06 am

    Informasi yang sangat berguna dan terorganisir dengan baik. Terima kasih atas usaha Anda! TRP Roller Chains RS 120-1

    Reply
  6. icryptox.com future on March 23, 2025 at 9:06 am

    Thanks for the nice blog. It was very useful for me. I m happy I found this blog. Thank you for sharing with us,I too always learn something new from your post. icryptox.com future

    Reply
  7. bolly44u on March 23, 2025 at 9:11 am

    Hmm… When i understand information sites using a related difficulty, even so when i never ever stopped at your site. When i added in the item to help populars likewise i’ll possibly be ones dedicated primer. bolly44u

    Reply
  8. Liwovosa on March 23, 2025 at 9:13 am

    It was wondering if I could use this write-up on my other website, I will link it back to your website though.Great Thanks. alexistogel login

    Reply
  9. techinsiderz.com gadgets on March 23, 2025 at 9:14 am

    Thanks for the nice blog. It was very useful for me. I m happy I found this blog. Thank you for sharing with us,I too always learn something new from your post. techinsiderz.com gadgets

    Reply
  10. crypto30x.com dis on March 23, 2025 at 9:17 am

    Ones texts within this theme usually are accurate, observe how When i had written this page is very top notch. crypto30x.com dis

    Reply
  11. be1crypto.com trading on March 23, 2025 at 9:20 am

    Thanks for the nice blog. It was very useful for me. I m happy I found this blog. Thank you for sharing with us,I too always learn something new from your post. be1crypto.com trading

    Reply
  13. icryptox.com future on March 23, 2025 at 9:24 am

    It is very helpful, on the other hand it might be critical so as to look at using web page: icryptox.com future

    Reply
  14. https://zomcluub.net/ on March 23, 2025 at 9:26 am

    ZOMCLUB – https://zomcluub.net/ mang đến cho bạn một nền tảng giải trí trực tuyến tuyệt vời với hàng loạt lựa chọn độc đáo. Đây là điểm đến lý tưởng, nơi hội tụ những dịch vụ đặc biệt, được thiết kế để đáp ứng tất cả sở thích của người dùng. Hãy trải nghiệm ngay để tận hưởng niềm vui bất tận mà không giới hạn tại ZOMCLUB.

    Reply
  15. TRP Roller Chains RS 140-1 on March 23, 2025 at 9:28 am

    Artikel yang sangat menarik! Saya tidak bisa berhenti membacanya. TRP Roller Chains RS 140-1

    Reply
  16. 5starsstocks.com defense on March 23, 2025 at 9:29 am

    Thanks for the nice blog. It was very useful for me. I m happy I found this blog. Thank you for sharing with us,I too always learn something new from your post. 5starsstocks.com defense

    Reply
  17. madeira thread on March 23, 2025 at 9:29 am

    Whale watching in Madeira is an unforgettable experience, as the island’s surrounding waters are home to various species of whales and dolphins year-round. madeira thread

    Reply
  18. techinsiderz.com gadgets on March 23, 2025 at 9:32 am

    I love to advocate only very good furthermore productive information and facts, that’s why see the item: techinsiderz.com gadgets

    Reply
  19. George on March 23, 2025 at 9:33 am

    I wanted to leave a little comment to support you and wish you a good continuation. Wishing you the best of luck for all your blogging efforts. Satta king chart

    Reply
  20. TRP Roller Chains RS 140-1 on March 23, 2025 at 9:33 am

    Artikel yang sangat menarik! Saya tidak bisa berhenti membacanya. TRP Roller Chains RS 140-1

    Reply
  21. money6x.com save money on March 23, 2025 at 9:35 am

    Thanks for the nice blog. It was very useful for me. I m happy I found this blog. Thank you for sharing with us,I too always learn something new from your post. money6x.com save money

    Reply
  22. ASAD SDFZE on March 23, 2025 at 9:36 am

    Thanks for sharing this information. I really like your blog post very much. You have really shared a informative and interesting blog post with people.. mega888 apk

    Reply
  23. be1crypto.com trading on March 23, 2025 at 9:37 am

    I am able to provides you with the target In this article understand the best way to practice it the right way. Understand in addition to generate anything beneficial. be1crypto.com trading

    Reply
  25. PUPUWEB on March 23, 2025 at 9:41 am

    AraBERT’s performance in sentiment analysis and question answering tasks is impressive! How do you think it compares to other models in real-world applications? Check out my blog on machine learning advancements for more insights: machine learning advancements.

    Reply
  26. Paminy on March 23, 2025 at 9:42 am

    The use of Farasa Segmenter in AraBERTv1 is fascinating! Do you think this approach could be adapted for other languages with complex morphology? I recently shared thoughts on language model innovations on my blog: language model innovations.

    Reply
  27. 5starsstocks.com defense on March 23, 2025 at 9:42 am

    It truly is in particular quality, nevertheless go into the points within this property target. 5starsstocks.com defense

    Reply
  28. POSIVI on March 23, 2025 at 9:42 am

    AraBERT’s results on Arabic NLP tasks are remarkable! How scalable is the training process for larger datasets? You might find my blog on NLP scalability challenges interesting: NLP scalability challenges.

    Reply
  29. mwe on March 23, 2025 at 9:45 am

    thanks for this usefull article, waiting for this article like this again. alexistogel

    Reply
  30. money6x.com save money on March 23, 2025 at 9:47 am

    In such a internet site, you’ll see this webpage, you need to realise that facts. money6x.com save money

    Reply
  32. crypto30x.com regulation on March 23, 2025 at 9:48 am

    Thanks for the nice blog. It was very useful for me. I m happy I found this blog. Thank you for sharing with us,I too always learn something new from your post. crypto30x.com regulation

    Reply
  35. 5starsstocks.com cannabis on March 23, 2025 at 9:52 am

    I am able to provides you with the target In this article understand the best way to practice it the right way. Understand in addition to generate anything beneficial. 5starsstocks.com cannabis

    Reply
  36. sabeaut7299 on March 23, 2025 at 9:53 am

    SAbeauti Professional Ladies Salon offers beauty salon services for women like Hair Protein Keratin, Makeup, Face, home service salon, Nails, Eyelashes salon in dubai

    Reply
  37. mwe on March 23, 2025 at 9:56 am

    Very interesting blog. Alot of blogs I see these days don’t really provide anything that I’m interested in, but I’m most definately interested in this one. Just thought that I would post and let you know. situs toto macau

    Reply
  38. be1crypto.com markets on March 23, 2025 at 9:58 am

    I am able to provides you with the target In this article understand the best way to practice it the right way. Understand in addition to generate anything beneficial. be1crypto.com markets

    Reply
  39. dududud on March 23, 2025 at 10:09 am

    ayo main di situs slot gampang jp hari ini slotgampangjp

    Reply
  40. absas on March 23, 2025 at 10:09 am

    gabung dan daftar ke situs terbaik dan terpercayaslotluckyneko

    Reply
  41. iofbodies .com on March 23, 2025 at 10:11 am

    I am able to provides you with the target In this article understand the best way to practice it the right way. Understand in addition to generate anything beneficial. iofbodies .com

    Reply
  42. gocryptobet.com wallet on March 23, 2025 at 10:16 am

    It truly is in particular quality, nevertheless go into the points within this property target. gocryptobet.com wallet

    Reply
  43. mwe on March 23, 2025 at 10:19 am

    I havent any word to appreciate this post…..Really i am impressed from this post….the person who create this post it was a great human..thanks for shared this with us. toto macau hari ini

    Reply
  44. Ahmed Ali on March 23, 2025 at 10:20 am

    did more than my expected results. Many thanks for displaying these effective, trustworthy, explanatory and in addition easy tips about your topic to Jane. https://49sresult.co.za

    Reply
  45. ashcroft capital lawsuit on March 23, 2025 at 10:22 am

    In such a internet site, you’ll see this webpage, you need to realise that facts. ashcroft capital lawsuit

    Reply
  46. mwe on March 23, 2025 at 10:23 am

    I havent any word to appreciate this post…..Really i am impressed from this post….the person who create this post it was a great human..thanks for shared this with us. Breast Reduction Turkey

    Reply
  47. ashcroft capital lawsuit on March 23, 2025 at 10:24 am

    In such a internet site, you’ll see this webpage, you need to realise that facts. ashcroft capital lawsuit

    Reply
  48. Archipelago Indonesia on March 23, 2025 at 10:25 am

    We introduce you to a site that discusses the richness of Indonesian culture, which is very diverse and has many types that you can find on our website https://archipelagoid.com/

    Reply
  49. knicks vs washington wizards match player stats on March 23, 2025 at 10:30 am

    I am able to provides you with the target In this article understand the best way to practice it the right way. Understand in addition to generate anything beneficial. knicks vs washington wizards match player stats

    Reply
  50. S666 on March 23, 2025 at 10:35 am

    S666 được xem là nhà cái dẫn đầu xu hướng, hội tụ hơn 1000 sản phẩm hot trend nhất trên thị trường. Sân chơi này là nơi lý tưởng để bạn thử thách vận may và chinh phục những đỉnh cao mới trong thế giới cá cược trực tuyến.

    Reply
← Older Comments
Newer Comments →

Submit a Comment

Your email address will not be published. Required fields are marked *