AraBERT : Pre-training BERT for Arabic Language Understanding

by | Feb 28, 2020 | NLP | 50,319 comments

Authors: Wissam Antoun, Fady Baly, Hazem Hajj

 

AraBERT is an Arabic pretrained language model based on Google’s BERT architecture. AraBERT uses the same BERT-Base config. More details are available in the AraBERT PAPER and in the AraBERT Meetup

There is two versions of the model AraBERTv0.1 and AraBERTv1, with the difference being that AraBERTv1 uses pre-segmented text where prefixes and suffixes were split using the Farasa Segmenter.

The model was trained on ~70M sentences or ~23GB of Arabic text with ~3B words.

Source Code Repository: https://github.com/aub-mind/arabert
Paper: https://www.aclweb.org/anthology/2020.osact-1.2.pdf

 

Results (Accuracy)

We evaluate both AraBERT models on different downstream tasks and compare it to mBERT, and other state of the art models (To the extent of our knowledge). The Tasks were Sentiment Analysis on 6 different datasets (HARD, ASTD-Balanced, ArsenTD-Lev, LABR, ArSaS), Named Entity Recognition with the ANERcorp, and Arabic Question Answering on Arabic-SQuAD and ARCD

Task prev. SOTA mBERT AraBERTv0.1 AraBERTv1
HARD 95.7 ElJundi et.al. 95.7 96.2 96.1
ASTD 86.5 ElJundi et.al. 80.1 92.2 92.6
ArsenTD-Lev 52.4 ElJundi et.al. 51 58.9 59.4
AJGT 93 Dahou et.al. 83.6 93.1 93.8
LABR 87.5 Dahou et.al. 83 85.9 86.7
ANERcorp 81.7 (BiLSTM-CRF) 78.4 84.2 81.9
ARCD mBERT EM:34.2 F1: 61.3 EM:51.14 F1:82.13 EM:54.84 F1: 82.15

Model Weights and Vocab Download

Models AraBERTv0.1 AraBERTv1
TensorFlow Drive Link Drive Link
PyTorch Drive_Link Drive_Link

You can find the PyTorch models in HuggingFace’s Transformer Library under the aubmindlab username

If you used this model please cite us as:

@inproceedings{antoun2020arabert,
  title={AraBERT: Transformer-based Model for Arabic Language Understanding},
  author={Antoun, Wissam and Baly, Fady and Hajj, Hazem},
  booktitle={LREC 2020 Workshop Language Resources and Evaluation Conference 11--16 May 2020},
  pages={9}
}

Acknowledgments

Thanks to TensorFlow Research Cloud (TFRC) for the free access to Cloud TPUs, couldn’t have done it without this program, and to the AUB MIND Lab Members for the continous support. Also thanks to Yakshof and Assafir for data and storage access. Another thanks for Habib Rahal (https://www.behance.net/rahalhabib), for putting a face to AraBERT.

Contacts

Wissam Antoun: Linkedin | Twitter | Github | wfa07@mail.aub.edu | wissam.antoun@gmail.com

Fady Baly: Linkedin | Twitter | Github | fgb06@mail.aub.edu | baly.fady@gmail.com

We are looking for sponsors to train BERT-Large and other Transformer models, the sponsor only needs to cover to data storage and compute cost of the generating the pretraining data

50,319 Comments

← Older Comments
Newer Comments →
  1. tg777 on March 21, 2025 at 8:35 pm

    Excellent website! I adore how it is easy on my eyes it is. I am questioning how I might be notified whenever a new post has been made. Looking for more new updates. Have a great day! tg777

    Reply
  2. RAJACABE88 on March 21, 2025 at 8:51 pm

    I once found some relevant information about this on rajacabe88. They have an article that may help clarify this point
    RAJACABE88

    Reply
  3. RAJACABE88 on March 21, 2025 at 9:38 pm

    What you discussed was very interesting. Check out rajacabe88 for similar articles that cover more of [related topic]
    RAJACABE88

    Reply
  4. Party bus rental in nyc on March 21, 2025 at 9:58 pm

    If you’re looking for a stylish and comfortable ride, ExoticLimoBus delivers the ultimate limo experience Party bus rental in nyc

    Reply
  5. Hiram on March 21, 2025 at 10:05 pm

    Hello, i think that i saw you visited my weblog so
    i got here to return the choose?.I am trying to find issues to enhance my web site!I assume its ok to use some of your concepts!!

    Also visit my web page :: 789p

    Reply
  6. sarim on March 21, 2025 at 10:17 pm

    MB66 – Nền tảng chơi game cao cấp với hệ thống giải trí đa dạng. Tham gia MB66 ngay hôm nay để tận hưởng các chương trình khuyến mãi hấp dẫn và trải nghiệm niềm vui đỉnh cao. MB66

    Reply
  7. Patrick on March 21, 2025 at 10:21 pm

    It’s remarkable to pay a quick visit this site
    and reading the views of all colleagues regarding this post,
    while I am also eager of getting familiarity.

    Also visit my blog :: important source

    Reply
  8. Webraseo on March 21, 2025 at 10:22 pm

    Spot on with this article, I really think this web site needs much more consideration. I’ll probably be again to read more, thanks for that information. pamanslot

    Reply
  9. Webra on March 21, 2025 at 10:28 pm

    We still cannot quite believe that It was not respectable often be those checking important points located on yuor web blog. Our kids and that i are sincerely thankful for your personal generosity also giving me possibility pursue our chosen profession path. Published information Manged to get with your web-site. pamanslot

    Reply
  10. jugydnxys on March 21, 2025 at 10:33 pm

    One more issue is video games are typically serious as the name indicated with the main focus on studying rather than enjoyment. Although, it has an entertainment feature to keep your young ones engaged, every single game is generally designed to work towards a specific experience or programs, such as math or research. Thanks for your publication. pamanslot

    Reply
  11. eco friendly wallpaper on March 21, 2025 at 10:35 pm

    Thank you for your information and respond to you. bad credit auto loans hawaiieco friendly wallpaper

    Reply
  12. Claudia on March 21, 2025 at 10:42 pm

    Hello to every one, it’s genuinely a good for me to pay
    a visit this site, it contains precious Information.

    My web page Накрутка отзывов

    Reply
  13. IgAnony on March 22, 2025 at 12:37 am

    This particular papers fabulous, and My spouse and i enjoy each of the perform that you have placed into this. I’m sure that you will be making a really useful place. I has been additionally pleased. Good perform!IgAnony

    Reply
  14. RAJACABE88 on March 22, 2025 at 12:52 am

    This is valuable information. On rajacabe88, I found an article with a different view on this topic
    RAJACABE88

    Reply
  15. RAJACABE88 on March 22, 2025 at 12:53 am

    This is valuable information. On rajacabe88, I found an article with a different view on this topic
    RAJACABE88

    Reply
  16. bóng đá tỷ lệ cá cược kèo nhà cái on March 22, 2025 at 1:02 am

    bóng đá tỷ lệ cá cược kèo nhà cái – tỷ lệ thưởng hấp dẫn và giao dịch siêu tốc mang đến trải nghiệm giải trí đỉnh cao, an toàn và minh bạch. Tham gia ngay để chinh phục những phần thưởng giá trị!

    Reply
  17. Muhammad Zubair on March 22, 2025 at 1:43 am

    Neo108 is an emerging trend in the world of fashion and accessories, offering a unique blend of modern aesthetics and timeless elegance. Whether you’re looking for stylish apparel, innovative designs, or premium-quality products, neo108 brings a fresh perspective to contemporary fashion. With a focus on creativity and sophistication, this brand stands out for its commitment to excellence and customer satisfaction. Explore the latest collections and redefine your style with the exclusive offerings from neo108 today!

    Reply
  18. Muhammad Zubair on March 22, 2025 at 1:48 am

    Jika Anda mencari pengalaman bermain yang seru dan menguntungkan, slot resmi gacor adalah pilihan terbaik. Dengan tingkat kemenangan tinggi dan fitur menarik, permainan ini memberikan peluang besar bagi pemain untuk meraih jackpot. Pastikan Anda memilih platform terpercaya agar mendapatkan pengalaman bermain yang aman dan fair. Bergabunglah sekarang dan nikmati sensasi bermain dengan berbagai tema menarik serta bonus melimpah!

    Reply
  19. Victorina on March 22, 2025 at 1:56 am

    Brilliant insights! As an Albany Park roofer, I’ve bed 50 smart seo prompts for chatgpt; http://www.itranslate.co.kr,
    Chicago’s brutal “emergency HVAC” SERPs through 773-area
    schema clusters + AI-trained SILO slayers. Burn cookie-cutter audits – Chi-Town demands two-fisted review
    syndication from Wicker Park mom blogs. Proof? 27%↑ Edgewater calls.
    PS: O’Hare proximity schema = rank rocket.
    – ChicagoSEO

    Reply
  22. DURALINK Roller Chains RS 40-2 on March 22, 2025 at 2:24 am

    Sangat senang menemukan artikel ini. Saya akan kembali untuk membaca lebih banyak lagi. DURALINK Roller Chains RS 40-2

    Reply
  23. DURALINK Roller Chains RS 40-2 on March 22, 2025 at 2:47 am

    Sangat senang menemukan artikel ini. Saya akan kembali untuk membaca lebih banyak lagi. DURALINK Roller Chains RS 40-2

    Reply
  24. RR88 on March 22, 2025 at 2:48 am

    RR88 nhà cái nổi bật trong lĩnh vực game giải trí trực tuyến, cung cấp nhiều trò chơi phong phú đa dạng

    Reply
  25. Muhammad Zubair on March 22, 2025 at 3:01 am

    Jika Anda mencari pengalaman bermain slot online yang menguntungkan, maka situs judi slot gacor adalah pilihan terbaik. Dengan berbagai pilihan permainan berkualitas tinggi, RTP tinggi, dan bonus menarik, situs ini memberikan peluang kemenangan yang lebih besar bagi para pemain. Selain itu, fitur keamanan yang canggih memastikan transaksi dan data pribadi Anda tetap aman. Jangan lewatkan kesempatan untuk merasakan sensasi bermain slot dengan peluang jackpot besar hanya di situs terbaik ini!

    Reply
  27. Muhammad Zubair on March 22, 2025 at 3:05 am

    Bagi para penggemar permainan slot online, menemukan link slot gacor hari ini adalah kunci untuk mendapatkan kemenangan besar. Dengan memilih situs yang terpercaya dan memiliki tingkat kemenangan tinggi, peluang untuk meraih jackpot semakin besar. Selain itu, menggunakan strategi yang tepat serta memahami pola permainan dapat membantu meningkatkan keberhasilan dalam bermain slot. Jangan lupa untuk selalu bermain secara bertanggung jawab dan menikmati keseruannya!

    Reply
  29. DURALINK Roller Chains RS 50-2 on March 22, 2025 at 3:17 am

    Terima kasih telah berbagi perspektif yang berbeda tentang topik ini. Sangat menginspirasi! DURALINK Roller Chains RS 50-2

    Reply
  30. DURALINK Roller Chains RS 50-2 on March 22, 2025 at 3:29 am

    Terima kasih telah berbagi perspektif yang berbeda tentang topik ini. Sangat menginspirasi! DURALINK Roller Chains RS 50-2

    Reply
  31. Tỷ Lệ Kèo Nhà Cái on March 22, 2025 at 3:37 am

    Đừng để lỡ cơ hội chiến thắng! Xem ngay Tỷ Lệ Kèo Nhà Cái mới nhất, chuẩn xác từ các chuyên gia hàng đầu trong lĩnh vực cá cược.

    Reply
  35. Amelia on March 22, 2025 at 4:16 am

    Your writing style is so engaging, and I found the insights in this blog truly enlightening! We’re currently exploring the dynamic between “are aquarius and leo compatible
    ” and I’d love to hear your perspective on it. It would be great if you could check out our website for a glimpse of our work. Looking forward to your thoughts and your upcoming content!

    Reply
  37. Amelia on March 22, 2025 at 4:18 am

    What an informative and engaging read! I gained so much knowledge from your perspective. As I’m currently focusing on “february stone” I’d love to hear any suggestions or insights you may have. Please feel free to visit our website to explore our work. Can’t wait to read your next post and hear your feedback!

    Reply
  42. Willaum on March 22, 2025 at 4:55 am

    I recently discovered Observateurturf.net, and it’s a great website for daily turf betting tips and predictions. The site delivers expert analysis, race updates, and well-researched strategies perfect for turf lovers. Observateurturf.net helps both beginners and experienced players make better betting decisions. If you’re serious about horse racing, Observateurturf.net is a reliable platform to follow for accurate predictions and consistent updates that improve your winning potential every day.

    Reply
  43. DURALINK Roller Chains RS 80-2 on March 22, 2025 at 4:58 am

    Saya sangat terinspirasi oleh artikel ini. Akan menjadi bahan renungan yang bagus bagi saya. DURALINK Roller Chains RS 80-2

    Reply
  44. DURALINK Roller Chains RS 80-2 on March 22, 2025 at 5:00 am

    Saya sangat terinspirasi oleh artikel ini. Akan menjadi bahan renungan yang bagus bagi saya. DURALINK Roller Chains RS 80-2

    Reply
  45. Ilana on March 22, 2025 at 5:05 am

    Admiring the commitment you put into your site and detailed information you provide.
    It’s nice to come across a blog every once in a while that isn’t
    the same unwanted rehashed material. Excellent read! I’ve saved your site and I’m including your
    RSS feeds to my Google account.

    Have a look at my blog – find this

    Reply
  46. Atlanta, GA on March 22, 2025 at 5:13 am

    Atlanta, GA homeowners can benefit from pool removal by reclaiming outdoor space, reducing maintenance costs, improving safety, and enhancing property value. Explore alternative outdoor uses through the Atlanta Pool Demolition Blog.

    Reply
  47. Baby James on March 22, 2025 at 5:15 am

    O Plano de Saúde Porto Seguro empresarial possui um dos melhores atendimentos de todo o Brasil. plano de saúde SJC

    Reply
  48. williamseo on March 22, 2025 at 5:22 am

    Keonhacai cung cấp tỷ lệ kèo nhà cái trực tuyến chính xác, cập nhật liên tục cho các trận đấu, kèo bóng đá 5 hôm nay tại https://keobongda.pro/. Nền tảng dễ sử dụng, giúp người chơi đưa ra quyết định cá cược hiệu quả.

    Reply
  49. Light Blue Earrings on March 22, 2025 at 5:34 am

    Light Blue Earrings, especially those featuring turquoise, are believed to promote inner calm, soothe sore throats, and balance emotions. They symbolize protection, energy cleansing, and help stabilize mood swings.

    Reply
  50. locksmith in NYC Midtown? on March 22, 2025 at 5:42 am

    locksmith in NYC Midtown?Need a Whether it’s an emergency lockout, key duplication, or security system installation, Midtown locksmiths offer fast, reliable services. Available 24/7, they provide professional solutions for residential, commercial, and automotive needs. With years of experience, you can trust them to ensure your property’s safety and security whenever you need assistance in Midtown NYC.

    Reply
← Older Comments
Newer Comments →

Submit a Comment

Your email address will not be published. Required fields are marked *