AraBERT : Pre-training BERT for Arabic Language Understanding

by | Feb 28, 2020 | NLP | 47,620 comments

Authors: Wissam Antoun, Fady Baly, Hazem Hajj

 

AraBERT is an Arabic pretrained language model based on Google’s BERT architecture. AraBERT uses the same BERT-Base config. More details are available in the AraBERT PAPER and in the AraBERT Meetup

There is two versions of the model AraBERTv0.1 and AraBERTv1, with the difference being that AraBERTv1 uses pre-segmented text where prefixes and suffixes were split using the Farasa Segmenter.

The model was trained on ~70M sentences or ~23GB of Arabic text with ~3B words.

Source Code Repository: https://github.com/aub-mind/arabert
Paper: https://www.aclweb.org/anthology/2020.osact-1.2.pdf

 

Results (Accuracy)

We evaluate both AraBERT models on different downstream tasks and compare it to mBERT, and other state of the art models (To the extent of our knowledge). The Tasks were Sentiment Analysis on 6 different datasets (HARD, ASTD-Balanced, ArsenTD-Lev, LABR, ArSaS), Named Entity Recognition with the ANERcorp, and Arabic Question Answering on Arabic-SQuAD and ARCD

Task prev. SOTA mBERT AraBERTv0.1 AraBERTv1
HARD 95.7 ElJundi et.al. 95.7 96.2 96.1
ASTD 86.5 ElJundi et.al. 80.1 92.2 92.6
ArsenTD-Lev 52.4 ElJundi et.al. 51 58.9 59.4
AJGT 93 Dahou et.al. 83.6 93.1 93.8
LABR 87.5 Dahou et.al. 83 85.9 86.7
ANERcorp 81.7 (BiLSTM-CRF) 78.4 84.2 81.9
ARCD mBERT EM:34.2 F1: 61.3 EM:51.14 F1:82.13 EM:54.84 F1: 82.15

Model Weights and Vocab Download

Models AraBERTv0.1 AraBERTv1
TensorFlow Drive Link Drive Link
PyTorch Drive_Link Drive_Link

You can find the PyTorch models in HuggingFace’s Transformer Library under the aubmindlab username

If you used this model please cite us as:

@inproceedings{antoun2020arabert,
  title={AraBERT: Transformer-based Model for Arabic Language Understanding},
  author={Antoun, Wissam and Baly, Fady and Hajj, Hazem},
  booktitle={LREC 2020 Workshop Language Resources and Evaluation Conference 11--16 May 2020},
  pages={9}
}

Acknowledgments

Thanks to TensorFlow Research Cloud (TFRC) for the free access to Cloud TPUs, couldn’t have done it without this program, and to the AUB MIND Lab Members for the continous support. Also thanks to Yakshof and Assafir for data and storage access. Another thanks for Habib Rahal (https://www.behance.net/rahalhabib), for putting a face to AraBERT.

Contacts

Wissam Antoun: Linkedin | Twitter | Github | wfa07@mail.aub.edu | wissam.antoun@gmail.com

Fady Baly: Linkedin | Twitter | Github | fgb06@mail.aub.edu | baly.fady@gmail.com

We are looking for sponsors to train BERT-Large and other Transformer models, the sponsor only needs to cover to data storage and compute cost of the generating the pretraining data

47,620 Comments

← Older Comments
Newer Comments →
  1. Fatima Javaid on March 12, 2025 at 2:48 am

    https://aj8gaming.com/ AJ8 adalah platform judi online multi negara yang menyediakan tiga jenis permainan unggulan: slot online, live casiino, dan Sbobet. Dengan dukungan teknologi canggih dan sistem keamanan yang terjamin, kami menawarkan pengalaman bermain yang adil dan menyenangkan bagi para pemain di seluruh Asia.

    Reply
  2. DURALINK Roller Chains RS 120-2 on March 12, 2025 at 2:50 am

    Sangat menyukai pendekatan Anda dalam menangani masalah ini. Terima kasih atas perspektif yang baru. DURALINK Roller Chains RS 120-2

    Reply
  3. DURALINK Roller Chains RS 120-2 on March 12, 2025 at 3:00 am

    Sangat menyukai pendekatan Anda dalam menangani masalah ini. Terima kasih atas perspektif yang baru. DURALINK Roller Chains RS 120-2

    Reply
  6. DURALINK Roller Chains RS 140-2 on March 12, 2025 at 3:30 am

    Saya suka cara Anda menjelaskan topik yang rumit menjadi sesuatu yang mudah dimengerti. DURALINK Roller Chains RS 140-2

    Reply
  7. DURALINK Roller Chains RS 140-2 on March 12, 2025 at 3:37 am

    Saya suka cara Anda menjelaskan topik yang rumit menjadi sesuatu yang mudah dimengerti. DURALINK Roller Chains RS 140-2

    Reply
  11. situs backlink terbaik on March 12, 2025 at 4:03 am

    Kontennya sangat berkualitas, saya akan merekomendasikannya ke teman-teman saya. situs backlink terbaik

    Reply
  12. jasa follow on March 12, 2025 at 4:08 am

    Terima kasih atas artikel yang luar biasa ini! Sangat membantu bagi saya. jasa follow

    Reply
  13. DURALINK Roller Chains RS 160-2 on March 12, 2025 at 4:10 am

    Saya selalu menantikan posting baru dari blog ini. Terima kasih atas konten yang konsisten dan berkualitas. DURALINK Roller Chains RS 160-2

    Reply
  14. universitas indonesia on March 12, 2025 at 4:13 am

    Penjelasan yang luar biasa! Saya sangat menikmati membaca blog ini. universitas indonesia

    Reply
  15. Dramanice on March 12, 2025 at 4:19 am

    I’ve read some good s here. Definitely worth bookmarking for revisiting. I surprise how much effort you put to create such a great informative website. Dramanice

    Reply
  16. DURALINK Roller Chains RS 160-2 on March 12, 2025 at 4:20 am

    Saya selalu menantikan posting baru dari blog ini. Terima kasih atas konten yang konsisten dan berkualitas. DURALINK Roller Chains RS 160-2

    Reply
  19. Way to Enliven on March 12, 2025 at 4:57 am

    This is my first visit to your web journal! We are a group of volunteers and new activities in the same specialty. Website gave us helpful data to work. http://www.tomandjerrys.net

    Reply
  20. ufabets on March 12, 2025 at 5:19 am

    UFABET เป็นเว็บพนันออนไลน์ที่ให้บริการแทงบอล คาสิโน และสล็อตครบวงจร ด้วยระบบที่มั่นคง ปลอดภัย และบริการตลอด 24 ชั่วโมง รองรับผู้เล่นทุกระดับ พร้อมโปรโมชั่นสุดคุ้มสำหรับสมาชิกใหม่ https://ufabets.top/

    Reply
  21. TRP Roller Chains RS 35-1 on March 12, 2025 at 5:21 am

    Sudah lama saya mencari informasi seperti ini. Terima kasih banyak! TRP Roller Chains RS 35-1

    Reply
  22. TRP Roller Chains RS 35-1 on March 12, 2025 at 5:29 am

    Sudah lama saya mencari informasi seperti ini. Terima kasih banyak! TRP Roller Chains RS 35-1

    Reply
  23. aa on March 12, 2025 at 5:31 am

    Yes i am totally agreed with this article and i just want say that this article is very nice and very informative article.I will make sure to be reading your blog more. You made a good point but I can’t help but wonder, what about the other side? !!!!!!Thanks law-momma.com

    Reply
  24. https://rwin.solutions/ on March 12, 2025 at 5:40 am

    RWIN – https://rwin.solutions/ tự hào là một trong những nhà cái cá cược trực tuyến hàng đầu, mang đến cho người chơi những trải nghiệm giải trí đỉnh cao. Đăng ký ngay để tận hưởng những ưu đãi độc quyền và khám phá thế giới cá cược chuyên nghiệp tại RWIN!

    Reply
  25. 45678 on March 12, 2025 at 5:48 am

    https://45678t.com/ – 45678.com là một trong những trang game trực tuyến được nhiều người tin tưởng và tham gia nhất năm 2025, với giao diện bắt mắt và dễ hiểu, cùng kho game phong phú và dịch vụ chăm sóc khách hàng tận tâm. Qua đó mang đến trải nghiệm tốt nhất cho hầu hết mọi người và mở ra nhiều cơ hội chiến thắng cho người chơi.

    Reply
  26. nhà cái uy tín dev on March 12, 2025 at 5:48 am

    I think your publications are informative and I am glad to have a look at these websites. Because it adds meaning to my thoughts. Continue to inspire your audience. nhà cái uy tín dev

    Reply
  27. anonymous on March 12, 2025 at 5:56 am

    Lung cancer is a type of cancer that occurs when cells in the lung tissue grow and multiply uncontrollably. This uncontrolled growth can lead to the formation of masses called tumors, which can spread (metastasize) to surrounding tissues and other parts of the body over time. Akciğer Kanserinin En Sinsi 9 Belirtisi

    Reply
  28. Fatima Javaid on March 12, 2025 at 6:10 am

    Slot asia adalah pilihan populer bagi para pecinta permainan kasino online di berbagai negara. Dengan berbagai jenis permainan menarik, Slot Asia menawarkan pengalaman bermain yang seru dengan grafis berkualitas tinggi dan fitur bonus menguntungkan. Banyak platform menyediakan layanan ini dengan berbagai pilihan slot bertema Asia, mulai dari mitologi Tiongkok hingga budaya Jepang.

    Reply
  30. Mominul Islam Sagor on March 12, 2025 at 6:19 am

    Thanks for this great post, i find it very interesting and very well thought out and put together. I look forward to reading your work in the future. 4imprint

    Reply
  32. TRP Roller Chains RS 40-1 on March 12, 2025 at 6:31 am

    Tulisan yang luar biasa. Saya akan membagikannya dengan teman-teman saya. TRP Roller Chains RS 40-1

    Reply
  33. crude palm oil refining machine on March 12, 2025 at 6:36 am

    The **crude palm oil refining machine ** is designed to process raw palm oil by removing impurities, free fatty acids, and unwanted odors, ensuring a high-quality final product. It plays a crucial role in enhancing the oil’s stability, clarity, and shelf life, making it suitable for cooking and industrial applications. This machine is essential for palm oil processing plants aiming for efficient and large-scale production.

    Reply
  34. Taha shaikh on March 12, 2025 at 6:41 am

    Appreciates created for penmanship quite a important line, While i transpired next to your blog aside from calculate one or two assist. I might like versions acquire linked with manuscript… Click here

    Reply
  35. TRP Roller Chains RS 40-1 on March 12, 2025 at 6:48 am

    Tulisan yang luar biasa. Saya akan membagikannya dengan teman-teman saya. TRP Roller Chains RS 40-1

    Reply
  39. mwe on March 12, 2025 at 7:07 am

    I definitely enjoying every little bit of it. It is a great website and nice share. I want to thank you. Good job! You guys do a great blog, and have some great contents. Keep up the good work. slot

    Reply
  40. HAMMAD on March 12, 2025 at 7:08 am

    I enjoy the variety of blogposts, I actually certainly favored, I like more knowledge about them, considering it is rather excellent., All the best designed for revealing. mitolyn

    Reply
  41. TRP Roller Chains RS 50-1 on March 12, 2025 at 7:11 am

    Saya menemukan jawaban atas banyak pertanyaan saya di sini. Terima kasih banyak! TRP Roller Chains RS 50-1

    Reply
  42. mwe on March 12, 2025 at 7:14 am

    Alexistogel merupakan situs slot online gacor terpercaya dengan provider zeus terbaik pada platform Pragmatic Play. Scatter kakek zeus menjadi incaran para pemain, karena ada diperingkat 1 permainan kemenangan gacor paling tinggi. slot online

    Reply
  43. seo on March 12, 2025 at 7:14 am

    I beloved as much as you will receive performed right here. The comic strip is attractive, your authored material stylish. nonetheless, you command get bought an nervousness over that you would like be handing over the following. unwell without a doubt come more before once more since precisely the same nearly very steadily inside of case you defend this increase. رویس بت

    Reply
  44. John Johns on March 12, 2025 at 7:14 am

    AraBERT is a powerful pre-trained model designed to enhance Arabic language understanding using deep learning techniques. Just as AI improves linguistic accuracy, seamless access through BDG Game URL ensures a smooth gaming experience. Both leverage advanced technology to enhance user interaction and engagement.

    Reply
  45. TRP Roller Chains RS 50-1 on March 12, 2025 at 7:16 am

    Saya menemukan jawaban atas banyak pertanyaan saya di sini. Terima kasih banyak! TRP Roller Chains RS 50-1

    Reply
  46. seo on March 12, 2025 at 7:20 am

    Free online games… […]With havin so much content do you ever run into any problems of plagorism or copyright infringement? My website has a lot of completely unique content I’ve either authored myself or outsourced but it looks like a lot of it is popping it up all ov… toto slot

    Reply
  47. SEO Brain on March 12, 2025 at 7:20 am

    Arabert is an impressive tool that combines cutting-edge AI technology with natural language processing, making it incredibly effective for Arabic language tasks. Its ability to understand and generate text in Arabic with high accuracy is a game-changer for businesses and developers working with Arabic content. Whether you’re looking to analyze text or create content, Arabert offers powerful solutions….. read more

    Reply
  50. Ali ladla on March 12, 2025 at 7:31 am

    Hey, this day is too much good for me, since this time I am reading this enormous informative article here at my home. Thanks a lot for massive hard work. Thabet

    Reply
← Older Comments
Newer Comments →

Submit a Comment

Your email address will not be published. Required fields are marked *