AraBERT : Pre-training BERT for Arabic Language Understanding

by | Feb 28, 2020 | NLP | 49,200 comments

Authors: Wissam Antoun, Fady Baly, Hazem Hajj

 

AraBERT is an Arabic pretrained language model based on Google’s BERT architecture. AraBERT uses the same BERT-Base config. More details are available in the AraBERT PAPER and in the AraBERT Meetup

There is two versions of the model AraBERTv0.1 and AraBERTv1, with the difference being that AraBERTv1 uses pre-segmented text where prefixes and suffixes were split using the Farasa Segmenter.

The model was trained on ~70M sentences or ~23GB of Arabic text with ~3B words.

Source Code Repository: https://github.com/aub-mind/arabert
Paper: https://www.aclweb.org/anthology/2020.osact-1.2.pdf

 

Results (Accuracy)

We evaluate both AraBERT models on different downstream tasks and compare it to mBERT, and other state of the art models (To the extent of our knowledge). The Tasks were Sentiment Analysis on 6 different datasets (HARD, ASTD-Balanced, ArsenTD-Lev, LABR, ArSaS), Named Entity Recognition with the ANERcorp, and Arabic Question Answering on Arabic-SQuAD and ARCD

Task prev. SOTA mBERT AraBERTv0.1 AraBERTv1
HARD 95.7 ElJundi et.al. 95.7 96.2 96.1
ASTD 86.5 ElJundi et.al. 80.1 92.2 92.6
ArsenTD-Lev 52.4 ElJundi et.al. 51 58.9 59.4
AJGT 93 Dahou et.al. 83.6 93.1 93.8
LABR 87.5 Dahou et.al. 83 85.9 86.7
ANERcorp 81.7 (BiLSTM-CRF) 78.4 84.2 81.9
ARCD mBERT EM:34.2 F1: 61.3 EM:51.14 F1:82.13 EM:54.84 F1: 82.15

Model Weights and Vocab Download

Models AraBERTv0.1 AraBERTv1
TensorFlow Drive Link Drive Link
PyTorch Drive_Link Drive_Link

You can find the PyTorch models in HuggingFace’s Transformer Library under the aubmindlab username

If you used this model please cite us as:

@inproceedings{antoun2020arabert,
  title={AraBERT: Transformer-based Model for Arabic Language Understanding},
  author={Antoun, Wissam and Baly, Fady and Hajj, Hazem},
  booktitle={LREC 2020 Workshop Language Resources and Evaluation Conference 11--16 May 2020},
  pages={9}
}

Acknowledgments

Thanks to TensorFlow Research Cloud (TFRC) for the free access to Cloud TPUs, couldn’t have done it without this program, and to the AUB MIND Lab Members for the continous support. Also thanks to Yakshof and Assafir for data and storage access. Another thanks for Habib Rahal (https://www.behance.net/rahalhabib), for putting a face to AraBERT.

Contacts

Wissam Antoun: Linkedin | Twitter | Github | wfa07@mail.aub.edu | wissam.antoun@gmail.com

Fady Baly: Linkedin | Twitter | Github | fgb06@mail.aub.edu | baly.fady@gmail.com

We are looking for sponsors to train BERT-Large and other Transformer models, the sponsor only needs to cover to data storage and compute cost of the generating the pretraining data

49,200 Comments

← Older Comments
Newer Comments →
  1. TRP Roller Chains British 06B-1 on March 2, 2025 at 7:19 am

    Anda melakukan pekerjaan yang luar biasa dengan menggali begitu dalam pada topik ini. Terima kasih banyak! TRP Roller Chains British 06B-1

    Reply
  3. Ahmed brohi on March 2, 2025 at 7:25 am

    Puppet Soccer 2014 Mod APK 3.1.8 là phiên bản nâng cấp của tựa game bóng đá vui nhộn với các nhân vật hoạt hình dễ thương. Người chơi sẽ được trải nghiệm lối chơi đơn giản nhưng đầy kịch tính, kết hợp với đồ họa đẹp mắt và âm thanh sống động. Phiên bản Mod Khumod này mở khóa nhiều tính năng hấp dẫn, giúp bạn thỏa sức khám phá và tận hưởng trận đấu đỉnh cao game hot khumod

    Reply
  4. Fatima Javaid on March 2, 2025 at 7:28 am

    Situs Hoki178 adalah platform permainan online yang menawarkan berbagai jenis permainan menarik, termasuk slot online, sportsbook, dan permainan kasino lainnya. Dikenal dengan tampilan yang ramah pengguna dan sistem keamanan yang andal, Hoki178 memberikan pengalaman bermain yang nyaman dan aman bagi para penggunanya. Selain itu, situs ini juga sering menawarkan berbagai promo dan bonus menarik yang membuat para pemain semakin betah.

    Reply
  6. TRP Roller Chains British 08B-1 on March 2, 2025 at 7:42 am

    Terima kasih atas wawasan yang berharga yang Anda berikan di sini. Saya merasa lebih terinformasi sekarang. TRP Roller Chains British 08B-1

    Reply
  7. Martin on March 2, 2025 at 7:44 am

    I’m not that much of a internet reader
    to be honest but your sites really nice, keep it up!
    I’ll go ahead and bookmark your website to come back later on. Many thanks

    My blog: vario terbaru

    Reply
  8. 신용카드 현금화 on March 2, 2025 at 7:44 am

    In this great pattern of things you actually secure a B+ with regard to hard work. Where exactly you lost me was first in your specifics. You know, people say, details make or break the argument.. And that couldn’t be much more true right here. Having said that, let me reveal to you precisely what did give good results. Your writing is actually highly powerful and that is probably the reason why I am taking the effort in order to comment. I do not really make it a regular habit of doing that. Second, even though I can see a leaps in reasoning you come up with, I am not necessarily confident of exactly how you seem to unite the points which produce the actual final result. For now I will, no doubt subscribe to your issue but hope in the near future you connect the dots better. 신용카드 현금화

    Reply
  9. TRP Roller Chains British 08B-1 on March 2, 2025 at 7:48 am

    Terima kasih atas wawasan yang berharga yang Anda berikan di sini. Saya merasa lebih terinformasi sekarang. TRP Roller Chains British 08B-1

    Reply
  10. NON on March 2, 2025 at 7:48 am

    PG88 – Sân chơi đỉnh cao dành cho những ai yêu thích cảm giác phấn khích và thử thách. Với giao diện hiện đại, các tính năng cạnh tranh và hỗ trợ khách hàng tận tình, PG88 mang đến trải nghiệm thú vị và cơ hội thắng lớn đáng kinh ngạc. PG88

    Reply
  12. nổ hũ 79king on March 2, 2025 at 7:55 am

    nổ hũ 79king – sân chơi cá cược uy tín, đa dạng trò chơi hấp dẫn, tỷ lệ thưởng cao, giao dịch nạp rút nhanh chóng, bảo mật an toàn. Tham gia ngay để trải nghiệm giải trí đỉnh cao và chinh phục những phần thưởng giá trị!

    Reply
  14. site on March 2, 2025 at 8:06 am

    Instead of spending money upfront , DANAGG

    Reply
  16. 5starsstocks.com staples on March 2, 2025 at 8:17 am

    This is my first time i visit here. I found so many entertaining stuff in your blog, especially its discussion. From the tons of comments on your articles, I guess I am not the only one having all the leisure here! Keep up the good work. I have been meaning to write something like this on my website and you have given me an idea. 5starsstocks.com staples

    Reply
  17. rs 125 only on thespark shop men jackets on March 2, 2025 at 8:18 am

    Thanks for the nice blog. It was very useful for me. I m happy I found this blog. Thank you for sharing with us,I too always learn something new from your post. rs 125 only on thespark shop men jackets

    Reply
  18. mytyles - wall and f on March 2, 2025 at 8:29 am

    Thanks for the nice blog. It was very useful for me. I m happy I found this blog. Thank you for sharing with us,I too always learn something new from your post. mytyles – wall and f

    Reply
  19. TRP Roller Chains British 10B-1 on March 2, 2025 at 8:30 am

    Saya suka bagaimana Anda memperhatikan detail dan memberikan solusi yang praktis. Sangat membantu! TRP Roller Chains British 10B-1

    Reply
  20. NON on March 2, 2025 at 8:32 am

    Win55 – Điểm đến thú vị cho những nhà thám hiểm! Chơi tại Win55 không chỉ mang tính chất giải trí mà còn là cảm giác phấn khích khi săn lùng những phần thưởng giá trị và trải nghiệm rút tiền nhanh như chớp. Win55

    Reply
  21. techinsiderz.com gadgets on March 2, 2025 at 8:33 am

    The next time I read a blog, I hope that it doesnt disappoint me as much as this one. I mean, I know it was my choice to read, but I actually thought you have something interesting to say. All I hear is a bunch of whining about something that you could fix if you werent too busy looking for attention. techinsiderz.com gadgets

    Reply
  22. olxtoto login on March 2, 2025 at 8:34 am

    Wow! This could be one particular of the most beneficial blogs We’ve ever arrive across on this subject. Actually Wonderful. I’m also an expert in this topic so I can understand your effort. olxtoto login

    Reply
  23. TRP Roller Chains British 10B-1 on March 2, 2025 at 8:35 am

    Saya suka bagaimana Anda memperhatikan detail dan memberikan solusi yang praktis. Sangat membantu! TRP Roller Chains British 10B-1

    Reply
  24. Bisma shakir on March 2, 2025 at 8:35 am

    They’re produced by the very best degree developers who will be distinguished for your polo dress creating. You’ll find polo Ron Lauren inside exclusive array which include particular classes for men, women. Prince Group Investments

    Reply
  25. thespark shop clothing men on March 2, 2025 at 8:37 am

    Thanks for the nice blog. It was very useful for me. I m happy I found this blog. Thank you for sharing with us,I too always learn something new from your post. thespark shop clothing men

    Reply
  26. yucatan5954 on March 2, 2025 at 8:38 am

    HANTU777 adalah situs game online resmi dengan RTP tertinggi, siap membayar berapapun kemenanganmu tanpa ribet! Nikmati ratusan permainan dari provider ternama dengan kemenangan terbesar dan bonus melimpah setiap hari! ???? Proses deposit dan withdraw super cepat, layanan customer service 24 jam, serta sistem keamanan terjamin! ???? Main sekarang di HANTU777 dan rasakan sensasi kemenangan tanpa batas! HANTU 777

    Reply
  27. luuxly.com on March 2, 2025 at 8:39 am

    The next time I read a blog, I hope that it doesnt disappoint me as much as this one. I mean, I know it was my choice to read, but I actually thought you have something interesting to say. All I hear is a bunch of whining about something that you could fix if you werent too busy looking for attention. luuxly.com

    Reply
  30. Zulfiqar Ali on March 2, 2025 at 8:45 am

    58WIN là nền tảng giải trí trực tuyến hàng đầu, cung cấp nhiều dịch vụ như dự đoán thể thao, trò chơi slot và các trò chơi trực tuyến tương tác khác. Nó nổi bật với giao diện thân thiện với người dùng, các tính năng cạnh tranh và các chương trình khuyến mãi hấp dẫn, khiến nó trở thành lựa chọn phổ biến của người chơi. 58WIN

    Reply
  31. malliesgrill4419 on March 2, 2025 at 8:46 am

    Sloto88 adalah situs judi slot online gacor dan situs judi slot88 terpercaya Indonesia yang menghadirkan game slot gacor hari ini terbaik dengan berbagai promo menarik slot88

    Reply
  32. Fatima Javaid on March 2, 2025 at 8:48 am

    Bandar slot adalah platform yang menyediakan berbagai jenis permainan slot online bagi para penggemar judi daring. Dengan perkembangan teknologi, bandar slot semakin populer karena menawarkan kemudahan akses dan beragam pilihan permainan yang menarik. Selain itu, banyak bandar slot yang menyediakan berbagai bonus dan promo untuk menarik pemain baru serta mempertahankan loyalitas pelanggan lama.

    Reply
  34. ddaad on March 2, 2025 at 9:06 am

    Such a very useful article. Very interesting to read this article.I would like to thank you for the efforts you had made for writing this awesome article.Crypto Trading

    Reply
  35. Apigacor55 on March 2, 2025 at 9:07 am

    Very nice really amazing post thank for this api55

    Reply
  36. wet dry vacuum on March 2, 2025 at 9:07 am

    Good – I should certainly pronounce, impressed with your site. I had no trouble navigating through all the tabs and related information ended up being truly easy to do to access. I recently found what I hoped for before you know it in the least. Quite unusual. Is likely to appreciate it for those who add forums or anything, website theme . a tones way for your client to communicate. Nice task. wet dry vacuum

    Reply
  38. the spark shop men winter jacket sportswear gym fitness on March 2, 2025 at 9:09 am

    Thanks for the nice blog. It was very useful for me. I m happy I found this blog. Thank you for sharing with us,I too always learn something new from your post. the spark shop men winter jacket sportswear gym fitness

    Reply
  39. bolly44u on March 2, 2025 at 9:09 am

    It is best to largely remarkable combined with well-performing stuff, so find it: bolly44u

    Reply
  40. add on March 2, 2025 at 9:10 am

    I just got to this amazing site not long ago. I was actually captured with the piece of resources you have got here. Big thumbs up for Cryptocurrency and Investmentmaking such wonderful blog page!

    Reply
  41. dadad on March 2, 2025 at 9:14 am

    Nice information, valuable and excellent design, as share good stuff with good ideas and concepts, lots of great information and inspiration, both of which I need, thanks to offer such a helpful information here.pedrovazpaulo business consultant

    Reply
  43. codeprinters.com on March 2, 2025 at 9:15 am

    The next time I read a blog, I hope that it doesnt disappoint me as much as this one. I mean, I know it was my choice to read, but I actually thought you have something interesting to say. All I hear is a bunch of whining about something that you could fix if you werent too busy looking for attention. codeprinters.com

    Reply
  44. crypto30x.com avalanche on March 2, 2025 at 9:15 am

    I should insist seldom of which it is amazing! The blog is usually info likewise generally fabricate wonderful entitys. crypto30x.com avalanche

    Reply
  46. crypto30x.com tnt on March 2, 2025 at 9:17 am

    In such a internet site, you’ll see this webpage, you need to realise that facts. crypto30x.com tnt

    Reply
  47. Harry on March 2, 2025 at 9:18 am

    First, let’s kill off a genuine fabrication: You can not use a reverse cellular phone number lookup for totally free anywhere, anytime. agen judi bola sbobet

    Reply
  48. report4284 on March 2, 2025 at 9:18 am

    Recover money lost to crypto scam, dating scam, investment scam by reporting to ReportScammedFunds. Report scammed crypto

    Reply
  49. dadd on March 2, 2025 at 9:18 am

    Awesome blog. I enjoyed reading your articles. This is truly a great read for me. I have bookmarked it and I am looking forward to reading new articles. Keep up the good work!millionaire life

    Reply
  50. ABDUL RAUF on March 2, 2025 at 9:20 am

    HB88 – Sân chơi đẳng cấp, nơi bạn làm chủ ván cược! HB88 giúp bạn chiếm lĩnh mọi thách thức và nhận về phần thưởng khủng qua mỗi trận đấu kịch tính. HB88

    Reply
← Older Comments
Newer Comments →

Submit a Comment

Your email address will not be published. Required fields are marked *