AraBERT : Pre-training BERT for Arabic Language Understanding

by | Feb 28, 2020 | NLP | 49,215 comments

Authors: Wissam Antoun, Fady Baly, Hazem Hajj

 

AraBERT is an Arabic pretrained language model based on Google’s BERT architecture. AraBERT uses the same BERT-Base config. More details are available in the AraBERT PAPER and in the AraBERT Meetup

There is two versions of the model AraBERTv0.1 and AraBERTv1, with the difference being that AraBERTv1 uses pre-segmented text where prefixes and suffixes were split using the Farasa Segmenter.

The model was trained on ~70M sentences or ~23GB of Arabic text with ~3B words.

Source Code Repository: https://github.com/aub-mind/arabert
Paper: https://www.aclweb.org/anthology/2020.osact-1.2.pdf

 

Results (Accuracy)

We evaluate both AraBERT models on different downstream tasks and compare it to mBERT, and other state of the art models (To the extent of our knowledge). The Tasks were Sentiment Analysis on 6 different datasets (HARD, ASTD-Balanced, ArsenTD-Lev, LABR, ArSaS), Named Entity Recognition with the ANERcorp, and Arabic Question Answering on Arabic-SQuAD and ARCD

Task prev. SOTA mBERT AraBERTv0.1 AraBERTv1
HARD 95.7 ElJundi et.al. 95.7 96.2 96.1
ASTD 86.5 ElJundi et.al. 80.1 92.2 92.6
ArsenTD-Lev 52.4 ElJundi et.al. 51 58.9 59.4
AJGT 93 Dahou et.al. 83.6 93.1 93.8
LABR 87.5 Dahou et.al. 83 85.9 86.7
ANERcorp 81.7 (BiLSTM-CRF) 78.4 84.2 81.9
ARCD mBERT EM:34.2 F1: 61.3 EM:51.14 F1:82.13 EM:54.84 F1: 82.15

Model Weights and Vocab Download

Models AraBERTv0.1 AraBERTv1
TensorFlow Drive Link Drive Link
PyTorch Drive_Link Drive_Link

You can find the PyTorch models in HuggingFace’s Transformer Library under the aubmindlab username

If you used this model please cite us as:

@inproceedings{antoun2020arabert,
  title={AraBERT: Transformer-based Model for Arabic Language Understanding},
  author={Antoun, Wissam and Baly, Fady and Hajj, Hazem},
  booktitle={LREC 2020 Workshop Language Resources and Evaluation Conference 11--16 May 2020},
  pages={9}
}

Acknowledgments

Thanks to TensorFlow Research Cloud (TFRC) for the free access to Cloud TPUs, couldn’t have done it without this program, and to the AUB MIND Lab Members for the continous support. Also thanks to Yakshof and Assafir for data and storage access. Another thanks for Habib Rahal (https://www.behance.net/rahalhabib), for putting a face to AraBERT.

Contacts

Wissam Antoun: Linkedin | Twitter | Github | wfa07@mail.aub.edu | wissam.antoun@gmail.com

Fady Baly: Linkedin | Twitter | Github | fgb06@mail.aub.edu | baly.fady@gmail.com

We are looking for sponsors to train BERT-Large and other Transformer models, the sponsor only needs to cover to data storage and compute cost of the generating the pretraining data

49,215 Comments

← Older Comments
Newer Comments →
  2. v on March 1, 2025 at 9:20 pm

    Không chỉ dừng lại ở những giây phút giải trí, For88 còn mang đến nhiều cơ hội để bạn thể hiện bản lĩnh và sự nhạy bén của mình. Các chương trình ưu đãi hấp dẫn và quà tặng giá trị luôn sẵn sàng dành cho những người chơi xứng đáng. For88

    Reply
  3. Andre Hakkak Net Worth on March 1, 2025 at 9:29 pm

    I have bookmarked your website because this site contains valuable information in it. I am really happy with articles quality and presentation. Thanks a lot for keeping great stuff. I am very much thankful for this site. Andre Hakkak Net Worth

    Reply
  4. Dönergrills on March 1, 2025 at 9:30 pm

    Dönergrills sind spezialisierte Grillgeräte, die speziell für die Zubereitung von Döner Kebab entwickelt wurden. Diese Grills zeichnen sich durch ihre vertikale Bauweise aus, bei der das Fleisch senkrecht an einem Spieß gegrillt wird. Die Hitzequelle befindet sich seitlich, sodass das Fleisch gleichmäßig gegart wird, während überschüssiges Fett abtropfen kann. Dönergrills sind in verschiedenen Größen und Ausführungen erhältlich, von kompakten Modellen für kleine Imbissbuden bis hin zu großen, professionellen Geräten für Restaurants und Catering-Services. Einige Dönergrills verfügen über zusätzliche Funktionen wie integrierte Beleuchtung, Temperaturregler und automatische Spießdrehung, die die Bedienung erleichtern und eine konsistente Qualität gewährleisten. Die Reinigung von Dönergrills ist ebenfalls einfach, da viele Modelle über abnehmbare Teile und antihaftbeschichtete Oberflächen verfügen. Für Döner-Liebhaber, die das authentische Geschmackserlebnis zu Hause genießen möchten, gibt es auch kleinere, tragbare Dönergrills, die sich ideal für den privaten Gebrauch eignen.

    Reply
  5. xmarti on March 1, 2025 at 10:02 pm

    After study a few of the blog posts on your own internet site now, i genuinely as if your strategy for blogging. I bookmarked it to my bookmark site list and will be checking back soon. Pls check out my site also and let me know what you think. Audio Bible

    Reply
  6. seo on March 2, 2025 at 1:12 am

    Xocdia88 – Sân chơi giải trí đỉnh cao, nơi bạn thử thách sự may mắn! Với Xocdia88, bạn sẽ bước vào một thế giới đầy kịch tính và hấp dẫn, nơi trải nghiệm những phút giây thư giãn tuyệt vời chỉ với một cú click.Xocdia88

    Reply
  7. seo on March 2, 2025 at 1:24 am

    BK8 tự tin khẳng định vị thế của mình trên thị trường giải trí trực tuyến với các sản phẩm trải nghiệm đa dạng như bóng đá, slot game, v.v. Cùng với đó, nền tảng liên tục cập nhật các chương trình khuyến mãi hấp dẫn dành cho người dùng.BK8

    Reply
  9. seo on March 2, 2025 at 1:32 am

    MB66 – Nền tảng giải trí trực tuyến hàng đầu với tốc độ vượt trội và độ tin cậy cao. Với giao diện hiện đại, mượt mà cùng những tính năng tối ưu, MB66 mang đến trải nghiệm giải trí đỉnh cao cho mọi người dùng. MB66

    Reply
  10. TRP Roller Chains RS 40-1 on March 2, 2025 at 1:37 am

    Saya sangat setuju dengan pandangan Anda tentang topik ini. Sangat terperinci dan berbobot. TRP Roller Chains RS 40-1

    Reply
  11. TRP Roller Chains RS 40-1 on March 2, 2025 at 1:49 am

    Saya sangat setuju dengan pandangan Anda tentang topik ini. Sangat terperinci dan berbobot. TRP Roller Chains RS 40-1

    Reply
  14. TRP Roller Chains RS 50-1 on March 2, 2025 at 2:28 am

    Penjelasan yang jelas dan terperinci. Saya sangat menghargainya. TRP Roller Chains RS 50-1

    Reply
  15. TRP Roller Chains RS 50-1 on March 2, 2025 at 2:31 am

    Penjelasan yang jelas dan terperinci. Saya sangat menghargainya. TRP Roller Chains RS 50-1

    Reply
  16. king88 on March 2, 2025 at 2:33 am

    king88 luon di dau trong top nha cai the thao uy tin xanh chin 2025

    Reply
  19. TRP Roller Chains RS 60-1 on March 2, 2025 at 3:04 am

    Artikel yang sangat membantu! Saya akan menyimpannya untuk referensi di masa depan. TRP Roller Chains RS 60-1

    Reply
  20. TRP Roller Chains RS 60-1 on March 2, 2025 at 3:11 am

    Artikel yang sangat membantu! Saya akan menyimpannya untuk referensi di masa depan. TRP Roller Chains RS 60-1

    Reply
  23. TRP Roller Chains RS 80-1 on March 2, 2025 at 3:44 am

    Ini adalah bacaan yang sangat menginspirasi. Saya merasa lebih termotivasi sekarang. TRP Roller Chains RS 80-1

    Reply
  24. TRP Roller Chains RS 80-1 on March 2, 2025 at 3:50 am

    Ini adalah bacaan yang sangat menginspirasi. Saya merasa lebih termotivasi sekarang. TRP Roller Chains RS 80-1

    Reply
  27. TRP Roller Chains RS 120-1 on March 2, 2025 at 4:22 am

    Saya sangat menghargai sumber daya yang Anda berikan di sini. Sangat membantu saya memahami topik ini. TRP Roller Chains RS 120-1

    Reply
  28. TRP Roller Chains RS 120-1 on March 2, 2025 at 4:28 am

    Saya sangat menghargai sumber daya yang Anda berikan di sini. Sangat membantu saya memahami topik ini. TRP Roller Chains RS 120-1

    Reply
  31. TRP Roller Chains RS 140-1 on March 2, 2025 at 4:46 am

    Saya sangat menyukai cara Anda menghadirkan topik yang kompleks dalam bahasa yang mudah dimengerti. TRP Roller Chains RS 140-1

    Reply
  32. TRP Roller Chains RS 140-1 on March 2, 2025 at 4:52 am

    Saya sangat menyukai cara Anda menghadirkan topik yang kompleks dalam bahasa yang mudah dimengerti. TRP Roller Chains RS 140-1

    Reply
  34. Fatima Javaid on March 2, 2025 at 5:13 am

    ثبت نام در وان ایکس بت فرآیندی ساده و سریع است که به کاربران امکان می‌دهد به دنیای گسترده شرط‌بندی آنلاین دسترسی داشته باشند. برای ثبت نام، کافی است به وب‌سایت رسمی وان ایکس بت مراجعه کنید و گزینه ثبت نام را انتخاب نمایید. سپس می‌توانید یکی از روش‌های ثبت نام از جمله ایمیل، شماره تلفن، یا شبکه‌های اجتماعی را انتخاب کنید. پس از وارد کردن اطلاعات مورد نیاز و تأیید آن، حساب کاربری شما فعال خواهد شد و می‌توانید با شارژ حساب خود از امکانات مختلف سایت مانند پیش‌بینی‌های ورزشی، بازی‌های کازینویی و سایر خدمات بهره ببرید.

    Reply
  35. giftcardmall mygift on March 2, 2025 at 5:30 am

    I am impressed by this article. It is really engaging. Nice work giftcardmall mygift

    Reply
  36. Saqrun on March 2, 2025 at 5:30 am

    Hi! People call me Saqrun. So i’m a legislated rules related content writer.
    Take the time to find my personal a short article.

    Here is my blog: Colorado Springs Speeding Ticket
    Lawyer (Hamool)

    Reply
  37. Trang chủ ABC8 on March 2, 2025 at 5:44 am

    Trang chủ ABC8 tự hào là một trong những nền tảng cá cược trực tuyến uy tín hàng đầu tại châu Á. Với cam kết mang đến dịch vụ chất lượng cao, hệ thống bảo mật vượt trội và kho tàng trò chơi đa dạng, ABC8 luôn là lựa chọn hàng đầu của người chơi. Đến với ABC8, bạn sẽ được trải nghiệm giải trí đỉnh cao cùng nhiều chương trình khuyến mãi hấp dẫn!

    Reply
  38. TRP Roller Chains RS 160-1 on March 2, 2025 at 6:10 am

    Tulisan yang sangat memikat! Saya tidak sabar untuk membaca posting selanjutnya. TRP Roller Chains RS 160-1

    Reply
  39. سایت پوکر بدون فیلتر on March 2, 2025 at 6:11 am

    سایت ریور پوکر River (دانلود اپلیکیشن پوکر ریور اندروید) سایتی که مخصوص پوکر است. با داشتن علاقه به بازی پوکر میتوانید به بهترین سایت پوکر یعنی سایت ریور پوکر مراجعه کنید و با پوکربازان بزرگ پوکر بازی کنید. همچنین اگر با مشکل فیلترینگ رو به رو هستید می توانید برای ورود به river سایت از لینک ریور پوکر بدون فیلتر استفاده کنید. کلیک کنید (سایت رسمی بازی انفجار).سایت پوکر بدون فیلتر

    Reply
  41. سایت پوکر بدون فیلتر on March 2, 2025 at 6:22 am

    سایت ریور پوکر River (دانلود اپلیکیشن پوکر ریور اندروید) سایتی که مخصوص پوکر است. با داشتن علاقه به بازی پوکر میتوانید به بهترین سایت پوکر یعنی سایت ریور پوکر مراجعه کنید و با پوکربازان بزرگ پوکر بازی کنید. همچنین اگر با مشکل فیلترینگ رو به رو هستید می توانید برای ورود به river سایت از لینک ریور پوکر بدون فیلتر استفاده کنید. کلیک کنید (سایت رسمی بازی انفجار).سایت پوکر بدون فیلتر

    Reply
  43. TRP Roller Chains RS 200-1 on March 2, 2025 at 6:39 am

    Saya sangat terkesan dengan kualitas konten di sini. Artikel ini jelas dibuat dengan hati-hati. TRP Roller Chains RS 200-1

    Reply
  44. bouquet shop near me on March 2, 2025 at 6:42 am

    Your perspective adds bouquet shop near me so much value to the topic.

    Reply
  45. TRP Roller Chains RS 200-1 on March 2, 2025 at 6:47 am

    Saya sangat terkesan dengan kualitas konten di sini. Artikel ini jelas dibuat dengan hati-hati. TRP Roller Chains RS 200-1

    Reply
  47. 신용카드 현금화 on March 2, 2025 at 6:57 am

    It’s my belief mesothelioma can be the most lethal cancer. It contains unusual characteristics. The more I look at it the greater I am certain it does not respond like a true solid tissues cancer. In the event mesothelioma is often a rogue virus-like infection, therefore there is the chance for developing a vaccine along with offering vaccination for asbestos open people who are really at high risk associated with developing future asbestos associated malignancies. Thanks for sharing your ideas about this important health issue. 신용카드 현금화

    Reply
  48. 신용카드 현금화 on March 2, 2025 at 7:01 am

    Check out different facilities for beat making online and find out one that suits you best. Beat-making is a popular hobby today. There are a good number of options to combine so you only have to twist and turn. 신용카드 현금화

    Reply
  50. TRP Roller Chains British 06B-1 on March 2, 2025 at 7:13 am

    Anda melakukan pekerjaan yang luar biasa dengan menggali begitu dalam pada topik ini. Terima kasih banyak! TRP Roller Chains British 06B-1

    Reply
← Older Comments
Newer Comments →

Submit a Comment

Your email address will not be published. Required fields are marked *