AraBERT : Pre-training BERT for Arabic Language Understanding

by | Feb 28, 2020 | NLP | 49,218 comments

Authors: Wissam Antoun, Fady Baly, Hazem Hajj

 

AraBERT is an Arabic pretrained language model based on Google’s BERT architecture. AraBERT uses the same BERT-Base config. More details are available in the AraBERT PAPER and in the AraBERT Meetup

There is two versions of the model AraBERTv0.1 and AraBERTv1, with the difference being that AraBERTv1 uses pre-segmented text where prefixes and suffixes were split using the Farasa Segmenter.

The model was trained on ~70M sentences or ~23GB of Arabic text with ~3B words.

Source Code Repository: https://github.com/aub-mind/arabert
Paper: https://www.aclweb.org/anthology/2020.osact-1.2.pdf

 

Results (Accuracy)

We evaluate both AraBERT models on different downstream tasks and compare it to mBERT, and other state of the art models (To the extent of our knowledge). The Tasks were Sentiment Analysis on 6 different datasets (HARD, ASTD-Balanced, ArsenTD-Lev, LABR, ArSaS), Named Entity Recognition with the ANERcorp, and Arabic Question Answering on Arabic-SQuAD and ARCD

Task prev. SOTA mBERT AraBERTv0.1 AraBERTv1
HARD 95.7 ElJundi et.al. 95.7 96.2 96.1
ASTD 86.5 ElJundi et.al. 80.1 92.2 92.6
ArsenTD-Lev 52.4 ElJundi et.al. 51 58.9 59.4
AJGT 93 Dahou et.al. 83.6 93.1 93.8
LABR 87.5 Dahou et.al. 83 85.9 86.7
ANERcorp 81.7 (BiLSTM-CRF) 78.4 84.2 81.9
ARCD mBERT EM:34.2 F1: 61.3 EM:51.14 F1:82.13 EM:54.84 F1: 82.15

Model Weights and Vocab Download

Models AraBERTv0.1 AraBERTv1
TensorFlow Drive Link Drive Link
PyTorch Drive_Link Drive_Link

You can find the PyTorch models in HuggingFace’s Transformer Library under the aubmindlab username

If you used this model please cite us as:

@inproceedings{antoun2020arabert,
  title={AraBERT: Transformer-based Model for Arabic Language Understanding},
  author={Antoun, Wissam and Baly, Fady and Hajj, Hazem},
  booktitle={LREC 2020 Workshop Language Resources and Evaluation Conference 11--16 May 2020},
  pages={9}
}

Acknowledgments

Thanks to TensorFlow Research Cloud (TFRC) for the free access to Cloud TPUs, couldn’t have done it without this program, and to the AUB MIND Lab Members for the continous support. Also thanks to Yakshof and Assafir for data and storage access. Another thanks for Habib Rahal (https://www.behance.net/rahalhabib), for putting a face to AraBERT.

Contacts

Wissam Antoun: Linkedin | Twitter | Github | wfa07@mail.aub.edu | wissam.antoun@gmail.com

Fady Baly: Linkedin | Twitter | Github | fgb06@mail.aub.edu | baly.fady@gmail.com

We are looking for sponsors to train BERT-Large and other Transformer models, the sponsor only needs to cover to data storage and compute cost of the generating the pretraining data

49,218 Comments

← Older Comments
Newer Comments →
  2. Jamaal on March 1, 2025 at 2:46 am

    Прежде всего, нужно определиться с тематикой
    сайта. Это во многом предопределит как набор инструментов,
    с помощью которых будет создаваться
    Ваш сайт, так и его внешний вид. Относительно программной части, можно сказать, что лучше всего обращаться за услугами профессиональных веб мастеров, цены на создание сайта в краснодаре цена
    – это достаточно сложный процесс, в котором требуется
    уделять внимание множеству нюансов.

    Reply
  3. สมัคร huaybee on March 1, 2025 at 2:48 am

    Every time I find new information I will always enter your blog first. to receive new news. สมัคร huaybee

    Reply
  4. DURALINK Roller Chains RS 40-2 on March 1, 2025 at 2:48 am

    Sangat senang menemukan artikel ini. Saya akan kembali untuk membaca lebih banyak lagi. DURALINK Roller Chains RS 40-2

    Reply
  5. DURALINK Roller Chains RS 40-2 on March 1, 2025 at 2:58 am

    Sangat senang menemukan artikel ini. Saya akan kembali untuk membaca lebih banyak lagi. DURALINK Roller Chains RS 40-2

    Reply
  8. DURALINK Roller Chains RS 50-2 on March 1, 2025 at 3:15 am

    Terima kasih telah berbagi perspektif yang berbeda tentang topik ini. Sangat menginspirasi! DURALINK Roller Chains RS 50-2

    Reply
  9. DURALINK Roller Chains RS 50-2 on March 1, 2025 at 3:27 am

    Terima kasih telah berbagi perspektif yang berbeda tentang topik ini. Sangat menginspirasi! DURALINK Roller Chains RS 50-2

    Reply
  15. adrian cryptopronetwork on March 1, 2025 at 3:51 am

    I have bookmarked your website because this site contains valuable information in it. I am really happy with articles quality and presentation. Thanks a lot for keeping great stuff. I am very much thankful for this site. adrian cryptopronetwork

    Reply
  16. J88 on March 1, 2025 at 3:56 am

    Khuyen mai tang 100% gia tri nap lan dau khi nap the tai J88.

    Reply
  18. Shenzhen Jewellok Technology Co., Ltd on March 1, 2025 at 4:02 am

    Shenzhen Jewellok Technology Co., Ltd Is A Ultra High Purity (UHP) Gas And Chemical Delivery Systems Manufacturer And Supplier In China, Manufacturing Ultra High Purity Regulators, Ultra High Purity Diaphragm Valves, High Purity Regulators For Use With Specialty, Flammable, And Industrial Gas Flows With Compact Construction And High Flow Capacity.

    Jewellok Design Ultra High Purity Gas Delivery System In Semiconductor Industry, Manufacturing Ultra High Purity (UHP) Gas And Chemical Delivery Systems, Ultra High Purity Regulators, Ultra High Purity Diaphragm Valves, High Purity Regulators For Use With Specialty, Flammable, And Industrial Gas Flows.
    Ultra High Purity Gas Delivery System

    Reply
  19. DURALINK Roller Chains RS 80-2 on March 1, 2025 at 4:05 am

    Saya sangat terinspirasi oleh artikel ini. Akan menjadi bahan renungan yang bagus bagi saya. DURALINK Roller Chains RS 80-2

    Reply
  20. general news logicalshout on March 1, 2025 at 4:07 am

    I have bookmarked your website because this site contains valuable information in it. I am really happy with articles quality and presentation. Thanks a lot for keeping great stuff. I am very much thankful for this site. general news logicalshout

    Reply
  21. DURALINK Roller Chains RS 80-2 on March 1, 2025 at 4:22 am

    Saya sangat terinspirasi oleh artikel ini. Akan menjadi bahan renungan yang bagus bagi saya. DURALINK Roller Chains RS 80-2

    Reply
  22. Kudasakti168 on March 1, 2025 at 4:23 am

    Selamat Menunaikan ibadah puasa bagi yang menjalankan. Perlu diketahui bahwa puasa adalah salah satu ibada wajib bagi umat muslim dan juga menjaga kesehatan tubuh seperti dikutip dari Kudasakti168 bahwa puasa yang dijalankan umat muslim dalam kurun waktu 1 Tahun sekali dalam sebulan adalah menjaga kebersihan lambung.

    Reply
  23. gaming news etruesports on March 1, 2025 at 4:32 am

    This is my first time i visit here. I found so many entertaining stuff in your blog, especially its discussion. From the tons of comments on your articles, I guess I am not the only one having all the leisure here! Keep up the good work. I have been meaning to write something like this on my website and you have given me an idea. gaming news etruesports

    Reply
  24. Packaged Wastewater Systems on March 1, 2025 at 4:47 am

    Packaged Wastewater Systems offer cost-effective, modular solutions for wastewater treatment, ensuring sustainability, water reuse, and regulatory compliance. Designed for durability, they reduce infrastructure costs and energy consumption while delivering high-quality water treatment.

    Reply
  26. crypto30x on March 1, 2025 at 4:50 am

    The next time I read a blog, I hope that it doesnt disappoint me as much as this one. I mean, I know it was my choice to read, but I actually thought you have something interesting to say. All I hear is a bunch of whining about something that you could fix if you werent too busy looking for attention. crypto30x

    Reply
  28. chrisplia8479 on March 1, 2025 at 4:57 am

    Gabung di Diana4d sekarang dan rasakan serunya main santai dengan peluang menang gede! Bonus melimpah dan game seru siap menemani harimu. Yuk, mulai petualanganmu di Diana4d! diana4d

    Reply
  29. Fauxjacket on March 1, 2025 at 4:59 am

    Our Fauxjacket website offers unbeatable prices on all products, with the added benefit of free worldwide shipping on every purchase! So check our website now and buy the product of your choice. white faux leather jacket

    Reply
  30. megawin138 on March 1, 2025 at 4:59 am

    This article gives me the idea to think more on this topic. Keep up the good work megawin138

    Reply
  31. updated always on March 1, 2025 at 5:13 am

    This is my first time i visit here. I found so many entertaining stuff in your blog, especially its discussion. From the tons of comments on your articles, I guess I am not the only one having all the leisure here! Keep up the good work. I have been meaning to write something like this on my website and you have given me an idea. updated always

    Reply
  32. tgaslot on March 1, 2025 at 5:20 am

    TGASLOT เป็นเว็บเกมสล็อตออนไลน์ที่รวมค่ายเกมกว่า 40 ค่ายในที่เดียว ทุกเกมเป็นเว็บตรงไม่ผ่านเอเย่นต์ แจกเครดิตฟรียอดนิยม อัปเดตเกมใหม่ทุกเดือน เล่นง่าย ปลอดภัย ระบบออโต้ 100% https://tgaslot.bio/

    Reply
  33. tga slot on March 1, 2025 at 5:36 am

    tga slot เป็นแพลตฟอร์มเกมสล็อตออนไลน์ที่รวมค่ายเกมดังมากกว่า 40 ค่าย มาพร้อมระบบออโต้ที่ใช้งานง่าย แจกฟรีเครดิต 50 ทุกวัน เล่นได้ทุกอุปกรณ์โดยไม่ต้องดาวน์โหลด ปลอดภัย ไม่มีโกง!

    Reply
  34. Mominul Islam Sagor on March 1, 2025 at 5:39 am

    Great article with excellent idea!Thank you for such a valuable article. I really appreciate for this great information. tik tok watch

    Reply
  35. australian men’s cricket team vs india national cricket team match scorecard on March 1, 2025 at 6:00 am

    This is my first time i visit here. I found so many entertaining stuff in your blog, especially its discussion. From the tons of comments on your articles, I guess I am not the only one having all the leisure here! Keep up the good work. I have been meaning to write something like this on my website and you have given me an idea. australian men’s cricket team vs india national cricket team match scorecard

    Reply
  36. Fatima Javaid on March 1, 2025 at 6:08 am

    Olxtoto Slot adalah salah satu platform permainan slot online yang menawarkan berbagai jenis permainan menarik dengan tampilan grafis yang modern dan sistem permainan yang mudah dipahami. Sebagai bagian dari industri iGaming yang terus berkembang, Olxtoto Slot menyediakan pengalaman bermain yang seru dengan berbagai pilihan tema dan fitur bonus yang menguntungkan.

    Reply
  39. DURALINK Roller Chains RS 100-2 on March 1, 2025 at 6:23 am

    Wow, saya tidak pernah berpikir tentang itu sebelumnya. Artikel yang sangat informatif! DURALINK Roller Chains RS 100-2

    Reply
  40. DURALINK Roller Chains RS 100-2 on March 1, 2025 at 6:27 am

    Wow, saya tidak pernah berpikir tentang itu sebelumnya. Artikel yang sangat informatif! DURALINK Roller Chains RS 100-2

    Reply
  41. Slot Mahjong on March 1, 2025 at 6:27 am

    This was such an insightful read! I never thought about it from that perspective before. Thanks for sharing! Slot Mahjong

    Reply
  42. Slot Mahjong on March 1, 2025 at 6:28 am

    This was such an insightful read! I never thought about it from that perspective before. Thanks for sharing! Slot Mahjong

    Reply
  45. Slot Mahjong on March 1, 2025 at 6:39 am

    Wow, this article really opened my eyes. I’ll definitely be sharing this with my friends and family! Slot Mahjong

    Reply
  46. Fatima Javaid on March 1, 2025 at 6:39 am

    Olxtoto slot login memberikan kemudahan bagi para pemain untuk mengakses berbagai permainan slot online dengan cepat dan aman. Dengan sistem login yang sederhana, pengguna dapat masuk ke akun mereka dan menikmati berbagai pilihan game berkualitas tinggi yang disediakan oleh Olxtoto. Selain itu, platform ini menawarkan keamanan data yang terjamin, memastikan bahwa informasi pribadi dan transaksi pemain tetap aman.

    Reply
  47. india national cricket team vs australian men’s cricket team match scorecard on March 1, 2025 at 6:41 am

    This is my first time i visit here. I found so many entertaining stuff in your blog, especially its discussion. From the tons of comments on your articles, I guess I am not the only one having all the leisure here! Keep up the good work. I have been meaning to write something like this on my website and you have given me an idea. india national cricket team vs australian men’s cricket team match scorecard

    Reply
  48. DURALINK Roller Chains RS 120-2 on March 1, 2025 at 6:41 am

    Informasi yang sangat berguna. Saya akan segera menerapkannya dalam kehidupan sehari-hari saya. DURALINK Roller Chains RS 120-2

    Reply
  49. Slot88 on March 1, 2025 at 6:42 am

    I love how well-researched this is. The data and examples really drive the point home. Great job! Slot88

    Reply
  50. Slot Gacor on March 1, 2025 at 6:43 am

    This is so relevant to what’s happening right now. It’s refreshing to see such a thoughtful take on the issue. Slot Gacor

    Reply
← Older Comments
Newer Comments →

Submit a Comment

Your email address will not be published. Required fields are marked *