AraBERT : Pre-training BERT for Arabic Language Understanding

by | Feb 28, 2020 | NLP | 47,992 comments

Authors: Wissam Antoun, Fady Baly, Hazem Hajj

 

AraBERT is an Arabic pretrained language model based on Google’s BERT architecture. AraBERT uses the same BERT-Base config. More details are available in the AraBERT PAPER and in the AraBERT Meetup

There is two versions of the model AraBERTv0.1 and AraBERTv1, with the difference being that AraBERTv1 uses pre-segmented text where prefixes and suffixes were split using the Farasa Segmenter.

The model was trained on ~70M sentences or ~23GB of Arabic text with ~3B words.

Source Code Repository: https://github.com/aub-mind/arabert
Paper: https://www.aclweb.org/anthology/2020.osact-1.2.pdf

 

Results (Accuracy)

We evaluate both AraBERT models on different downstream tasks and compare it to mBERT, and other state of the art models (To the extent of our knowledge). The Tasks were Sentiment Analysis on 6 different datasets (HARD, ASTD-Balanced, ArsenTD-Lev, LABR, ArSaS), Named Entity Recognition with the ANERcorp, and Arabic Question Answering on Arabic-SQuAD and ARCD

Task prev. SOTA mBERT AraBERTv0.1 AraBERTv1
HARD 95.7 ElJundi et.al. 95.7 96.2 96.1
ASTD 86.5 ElJundi et.al. 80.1 92.2 92.6
ArsenTD-Lev 52.4 ElJundi et.al. 51 58.9 59.4
AJGT 93 Dahou et.al. 83.6 93.1 93.8
LABR 87.5 Dahou et.al. 83 85.9 86.7
ANERcorp 81.7 (BiLSTM-CRF) 78.4 84.2 81.9
ARCD mBERT EM:34.2 F1: 61.3 EM:51.14 F1:82.13 EM:54.84 F1: 82.15

Model Weights and Vocab Download

Models AraBERTv0.1 AraBERTv1
TensorFlow Drive Link Drive Link
PyTorch Drive_Link Drive_Link

You can find the PyTorch models in HuggingFace’s Transformer Library under the aubmindlab username

If you used this model please cite us as:

@inproceedings{antoun2020arabert,
  title={AraBERT: Transformer-based Model for Arabic Language Understanding},
  author={Antoun, Wissam and Baly, Fady and Hajj, Hazem},
  booktitle={LREC 2020 Workshop Language Resources and Evaluation Conference 11--16 May 2020},
  pages={9}
}

Acknowledgments

Thanks to TensorFlow Research Cloud (TFRC) for the free access to Cloud TPUs, couldn’t have done it without this program, and to the AUB MIND Lab Members for the continous support. Also thanks to Yakshof and Assafir for data and storage access. Another thanks for Habib Rahal (https://www.behance.net/rahalhabib), for putting a face to AraBERT.

Contacts

Wissam Antoun: Linkedin | Twitter | Github | wfa07@mail.aub.edu | wissam.antoun@gmail.com

Fady Baly: Linkedin | Twitter | Github | fgb06@mail.aub.edu | baly.fady@gmail.com

We are looking for sponsors to train BERT-Large and other Transformer models, the sponsor only needs to cover to data storage and compute cost of the generating the pretraining data

47,992 Comments

← Older Comments
Newer Comments →
  1. 77WIN on March 13, 2025 at 5:40 am

    77WIN là nhà cái giải trí trực tuyến uy tín, mang đến nhiều cơ hội chiến thắng lớn với link đăng ký và đăng nhập mới nhất. Người chơi sẽ nhận ngay 58K khi tham gia, cùng với kho tàng game đa dạng từ thể thao, game bài đến các tựa game trực tuyến hấp dẫn khác.

    Reply
  3. Fatima Javaid on March 13, 2025 at 5:47 am

    Togel telah menjadi salah satu permainan yang populer di kalangan pecinta judi online, terutama di kawasan Asia. Salah satu pasaran yang paling diminati adalah Togel Singapore, yang dikenal dengan sistem permainan yang transparan dan peluang menang yang tinggi.

    Reply
  4. 500AE on March 13, 2025 at 5:52 am

    500AE là trang tin tức bóng đá uy tín, cập nhật 24/7 với những thông tin nhanh chóng, chính xác và hấp dẫn về các giải đấu, trận đấu và mọi chuyển động trong thế giới bóng đá.

    Reply
  6. F168 on March 13, 2025 at 6:06 am

    F168 là một trong những nhà cái cá cược trực tuyến hàng đầu tại Việt Nam, nổi bật nhờ độ uy tín và hệ thống bảo mật cao cấp. Nhà cái này mang đến đa dạng trò chơi giải trí trực tuyến như Thể thao, Bài, Xổ số, Slots, Bắn cá và nhiều lựa chọn hấp dẫn khác, đáp ứng mọi nhu cầu của người chơi.

    Reply
  7. TRP Roller Chains British 10B-1 on March 13, 2025 at 6:13 am

    Artikel yang sangat inspiratif! Saya merasa lebih termotivasi untuk mencapai tujuan saya sekarang. TRP Roller Chains British 10B-1

    Reply
  8. TRP Roller Chains British 10B-1 on March 13, 2025 at 6:18 am

    Artikel yang sangat inspiratif! Saya merasa lebih termotivasi untuk mencapai tujuan saya sekarang. TRP Roller Chains British 10B-1

    Reply
  9. BET88 on March 13, 2025 at 6:18 am

    BET88 là một trong những trang cược trực tuyến hàng đầu tại Châu Á, đặc biệt nổi bật trong lĩnh vực cá độ thể thao. Với hệ thống kèo đa dạng, tỷ lệ trả thưởng hấp dẫn và công nghệ tiên tiến, Bet88 cam kết mang đến trải nghiệm cá cược an toàn, minh bạch cùng nhiều ưu đãi giá trị dành cho người chơi.

    Reply
  10. Modular Wastewater Treatment Plants on March 13, 2025 at 6:18 am

    Modular Wastewater Treatment Plants offer scalable, energy-efficient solutions for commercial and industrial needs. Using advanced membrane bioreactors, they ensure compliance, sustainability, and cost-efficiency while promoting water conservation and minimizing environmental impact.

    Reply
  11. Modular Wastewater Treatment Plants on March 13, 2025 at 6:18 am

    Modular Wastewater Treatment Plants offer scalable, energy-efficient solutions for commercial and industrial needs. Using advanced membrane bioreactors, they ensure compliance, sustainability, and cost-efficiency while promoting water conservation and minimizing environmental impact.

    Reply
  12. nha cai on March 13, 2025 at 6:19 am

    BET88 là một trong những trang cược trực tuyến hàng đầu tại Châu Á, đặc biệt nổi bật trong lĩnh vực cá độ thể thao. Với hệ thống kèo đa dạng, tỷ lệ trả thưởng hấp dẫn và công nghệ tiên tiến, Bet88 cam kết mang đến trải nghiệm cá cược an toàn, minh bạch cùng nhiều ưu đãi giá trị dành cho người chơi.

    Reply
  15. sllengraving on March 13, 2025 at 6:33 am

    Celebrate life’s milestones with our unique laser engraving services. From photo-engraved glassware to custom cutting boards, we craft personalized gifts that last a lifetime.

    https://sllengraving.com/

    Reply
  16. TRP Roller Chains British 12B-1 on March 13, 2025 at 6:34 am

    Saya sangat terkesan dengan kedalaman pengetahuan Anda tentang topik ini. Sangat mengesankan! TRP Roller Chains British 12B-1

    Reply
  17. TRP Roller Chains British 12B-1 on March 13, 2025 at 6:45 am

    Saya sangat terkesan dengan kedalaman pengetahuan Anda tentang topik ini. Sangat mengesankan! TRP Roller Chains British 12B-1

    Reply
  19. kharizaffar on March 13, 2025 at 6:54 am

    I’d must talk with you here. Which isn’t some thing I usually do! I love to reading an article which will make people feel. Also, thank you allowing me to comment! alexistogel

    Reply
  20. univers0349 on March 13, 2025 at 6:58 am

    Jerukbet adalah situs game online gampang menang dan banyak menawarkan bonus harian yang sudah tersedia di akun anda. jerukbet login

    Reply
  21. HAMMAD on March 13, 2025 at 6:59 am

    I require the majority of the records, Have a preference for seriously favored, We really wish for addiitional information and facts over it, while it is definitely rather great., L8rs used only for documenting. bet120x

    Reply
  23. mwe on March 13, 2025 at 7:03 am

    If more people that write articles really concerned themselves with writing great content like you, more readers would be interested in their writings. Thank you for caring about your content. macau

    Reply
  24. sureman10 on March 13, 2025 at 7:04 am

    sureman10은 먹튀 피해를 최소화하기 위해 안전놀이터와 메이저놀이터를 검증하는 신뢰할 수 있는 먹튀검증 커뮤니티입니다. 철저한 검증과 보증금 시스템으로 안전한 베팅 환경을 제공합니다.

    Reply
  25. zipaworld on March 13, 2025 at 7:04 am

    This post really helped me understand [topic] better. I appreciate the way you structured the information. Inland Transport

    Reply
  26. Betyeklive241 on March 13, 2025 at 7:05 am

    Amazing post, thanks for sharing this article. I am truly motivated by you for blogging.Thank You betyekbet

    Reply
  27. Maya on March 13, 2025 at 7:06 am

    I am sure this paragraph has touched all the internet users, its really really fastidious article on building
    up new weblog.

    My webpage خرید بک لینک

    Reply
  28. zipaworld on March 13, 2025 at 7:08 am

    You made some excellent points. I wonder how this would apply to [related topic]. What are your thoughts? EximGPT

    Reply
  29. mwe on March 13, 2025 at 7:08 am

    Thanks for this great post, i find it very interesting and very well thought out and put together. I look forward to reading your work in the future. slot hoki

    Reply
  30. TRP Roller Chains British 16B-1 on March 13, 2025 at 7:09 am

    Saya akan merekomendasikan blog ini kepada siapa pun yang mencari informasi berkualitas tentang topik ini. TRP Roller Chains British 16B-1

    Reply
  31. mwe on March 13, 2025 at 7:14 am

    Hi there! Nice post! Please tell us when I will see a follow up! macau

    Reply
  32. TRP Roller Chains British 16B-1 on March 13, 2025 at 7:25 am

    Saya akan merekomendasikan blog ini kepada siapa pun yang mencari informasi berkualitas tentang topik ini. TRP Roller Chains British 16B-1

    Reply
  33. aa on March 13, 2025 at 7:28 am

    Hey There. I found your blog using msn. This is a very well written article. I’ll be sure to bookmark it and come back to read more of your useful info. Thanks for the post. I’ll definitely return. ARONBET88

    Reply
  34. mwe on March 13, 2025 at 7:34 am

    If you set out to make me think today; mission accomplished! I really like your writing style and how you express your ideas. Thank you. live macau

    Reply
  36. mwe on March 13, 2025 at 7:36 am

    Great post, you have pointed out some excellent points, I as well believe this is a very superb website. live macau

    Reply
  37. mwe on March 13, 2025 at 7:40 am

    I really impressed after read this because of some quality work and informative thoughts . I just wanna say thanks for the writer and wish you all the best for coming!. Alexistogel

    Reply
  38. mwe on March 13, 2025 at 7:41 am

    I felt very happy while reading this site. This was really very informative site for me. I really liked it. This was really a cordial post. Thanks a lot!. situs slot hoki

    Reply
  39. mwe on March 13, 2025 at 7:42 am

    That is the excellent mindset, nonetheless is just not help to make every sence whatsoever preaching about that mather. Virtually any method many thanks in addition to i had endeavor to promote your own article in to delicius nevertheless it is apparently a dilemma using your information sites can you please recheck the idea. thanks once more. live macau

    Reply
  40. POWER BANK on March 13, 2025 at 7:43 am

    Get the best POWER BANK at The Brand Store for reliable charging on the go! Shop now for top-quality power solutions.

    POWER BANK

    Reply
  41. mwe on March 13, 2025 at 7:44 am

    This article was written by a real thinking writer. I agree many of the with the solid points made by the writer. I’ll be back. live macau

    Reply
  42. hubetr.bar on March 13, 2025 at 7:45 am

    hubetr.bar ngay từ khi ra mắt đã trở thành nhà cái giải trí thu hút sự chú ý lớn từ người chơi khắp châu lục. Không chỉ dẫn đầu về danh mục game cá cược đa dạng, thương hiệu này còn nổi bật với hệ thống bảo mật, thanh toán an toàn và dịch vụ chăm sóc khách hàng chuyên nghiệp.

    Reply
  43. ten kainamu on March 13, 2025 at 7:49 am

    789Club mang đến hàng loạt sự kiện khuyến mãi đặc biệt, giúp người chơi có thêm cơ hội nhận quà tặng giá trị mỗi ngày.

    Reply
  45. Baby James on March 13, 2025 at 7:58 am

    Took me time to read all the comments, but I really enjoyed the article. It proved to be Very helpful to me and I am sure to all the commenters here! It’s always nice when you can not only be informed, but also entertained! https://svila-lina.net/

    Reply
  46. DURAJOINT Jaw Couplings on March 13, 2025 at 8:03 am

    Artikel yang sangat bagus! Saya tidak sabar untuk membaca konten lainnya dari blog ini. DURAJOINT Jaw Couplings

    Reply
  47. mytyles - wall and f on March 13, 2025 at 8:15 am

    Thanks for the nice blog. It was very useful for me. I m happy I found this blog. Thank you for sharing with us,I too always learn something new from your post. mytyles – wall and f

    Reply
  48. Way to Enliven on March 13, 2025 at 8:18 am

    I admire this article for the well-researched content and excellent wording. I got so involved in this material that I couldn’t stop reading. I am impressed with your work and skill. Thank you so much. crf2.com

    Reply
  49. HUZAIFA on March 13, 2025 at 8:20 am

    I spend time many of the subject material, I have to express i actually prized, I actually wants more information about the following, just because this can be very superb., Thanks a ton pertaining to discovering. art and craft

    Reply
  50. Bersih4D on March 13, 2025 at 8:22 am

    Banyak platform judi slot online menawarkan bonus besar untuk menarik pemain Seperti daftar Bersih4D

    Reply
← Older Comments
Newer Comments →

Leave a Reply to Genset Genpac Open / Silent Perkins 1700 kVA GP1700 Cancel reply

Your email address will not be published. Required fields are marked *