AraBERT : Pre-training BERT for Arabic Language Understanding

by | Feb 28, 2020 | NLP | 49,376 comments

Authors: Wissam Antoun, Fady Baly, Hazem Hajj

 

AraBERT is an Arabic pretrained language model based on Google’s BERT architecture. AraBERT uses the same BERT-Base config. More details are available in the AraBERT PAPER and in the AraBERT Meetup

There is two versions of the model AraBERTv0.1 and AraBERTv1, with the difference being that AraBERTv1 uses pre-segmented text where prefixes and suffixes were split using the Farasa Segmenter.

The model was trained on ~70M sentences or ~23GB of Arabic text with ~3B words.

Source Code Repository: https://github.com/aub-mind/arabert
Paper: https://www.aclweb.org/anthology/2020.osact-1.2.pdf

 

Results (Accuracy)

We evaluate both AraBERT models on different downstream tasks and compare it to mBERT, and other state of the art models (To the extent of our knowledge). The Tasks were Sentiment Analysis on 6 different datasets (HARD, ASTD-Balanced, ArsenTD-Lev, LABR, ArSaS), Named Entity Recognition with the ANERcorp, and Arabic Question Answering on Arabic-SQuAD and ARCD

Task prev. SOTA mBERT AraBERTv0.1 AraBERTv1
HARD 95.7 ElJundi et.al. 95.7 96.2 96.1
ASTD 86.5 ElJundi et.al. 80.1 92.2 92.6
ArsenTD-Lev 52.4 ElJundi et.al. 51 58.9 59.4
AJGT 93 Dahou et.al. 83.6 93.1 93.8
LABR 87.5 Dahou et.al. 83 85.9 86.7
ANERcorp 81.7 (BiLSTM-CRF) 78.4 84.2 81.9
ARCD mBERT EM:34.2 F1: 61.3 EM:51.14 F1:82.13 EM:54.84 F1: 82.15

Model Weights and Vocab Download

Models AraBERTv0.1 AraBERTv1
TensorFlow Drive Link Drive Link
PyTorch Drive_Link Drive_Link

You can find the PyTorch models in HuggingFace’s Transformer Library under the aubmindlab username

If you used this model please cite us as:

@inproceedings{antoun2020arabert,
  title={AraBERT: Transformer-based Model for Arabic Language Understanding},
  author={Antoun, Wissam and Baly, Fady and Hajj, Hazem},
  booktitle={LREC 2020 Workshop Language Resources and Evaluation Conference 11--16 May 2020},
  pages={9}
}

Acknowledgments

Thanks to TensorFlow Research Cloud (TFRC) for the free access to Cloud TPUs, couldn’t have done it without this program, and to the AUB MIND Lab Members for the continous support. Also thanks to Yakshof and Assafir for data and storage access. Another thanks for Habib Rahal (https://www.behance.net/rahalhabib), for putting a face to AraBERT.

Contacts

Wissam Antoun: Linkedin | Twitter | Github | wfa07@mail.aub.edu | wissam.antoun@gmail.com

Fady Baly: Linkedin | Twitter | Github | fgb06@mail.aub.edu | baly.fady@gmail.com

We are looking for sponsors to train BERT-Large and other Transformer models, the sponsor only needs to cover to data storage and compute cost of the generating the pretraining data

49,376 Comments

← Older Comments
Newer Comments →
  1. TRP Roller Chains British 12B-1 on March 3, 2025 at 1:27 am

    Saya akan merekomendasikan blog ini kepada siapa pun yang mencari informasi berkualitas tentang topik ini. TRP Roller Chains British 12B-1

    Reply
  6. seo on March 3, 2025 at 2:06 am

    At Agenda Cover Life, we believe every home has a unique story waiting to be told.agendacoverlife.org

    Reply
  7. DURAJOINT Jaw Couplings on March 3, 2025 at 2:11 am

    Sangat senang menemukan sumber daya yang berkualitas seperti ini. Terima kasih banyak! DURAJOINT Jaw Couplings

    Reply
  9. DURAJOINT Jaw Couplings on March 3, 2025 at 2:24 am

    Sangat senang menemukan sumber daya yang berkualitas seperti ini. Terima kasih banyak! DURAJOINT Jaw Couplings

    Reply
  10. Evonne on March 3, 2025 at 2:29 am

    After going over a few of the articles on your website, I truly
    appreciate your technique of writing a blog. I saved it
    to my bookmark webpage list and will be checking back in the near future.
    Take a look at my website as well and tell
    me how you feel.

    Also visit my web site :: 888b

    Reply
  13. DURAJOINT Rubber FCL Couplings on March 3, 2025 at 2:48 am

    Saya suka bagaimana Anda menyajikan fakta dengan cara yang menarik. Sangat menghibur! DURAJOINT Rubber FCL Couplings

    Reply
  14. https://wangw88.com/ on March 3, 2025 at 2:50 am

    WW88 – https://wangw88.com/ là một trong những thương hiệu nổi bật và được nhắc đến nhiều nhất trên các diễn đàn cá cược trực tuyến. Với hơn 10 triệu hội viên, W88 luôn đứng đầu trong danh sách các nền tảng uy tín hàng đầu. Nhờ sự minh bạch và chất lượng dịch vụ đỉnh cao, W88 đã trở thành biểu tượng trong lĩnh vực cá cược trực tuyến.

    Reply
  15. DURAJOINT Rubber FCL Couplings on March 3, 2025 at 2:58 am

    Saya suka bagaimana Anda menyajikan fakta dengan cara yang menarik. Sangat menghibur! DURAJOINT Rubber FCL Couplings

    Reply
  18. Gravity Rollers on March 3, 2025 at 3:17 am

    Saya suka bagaimana Anda merangkum informasi kompleks menjadi sesuatu yang mudah dipahami. Gravity Rollers

    Reply
  19. Gravity Rollers on March 3, 2025 at 3:29 am

    Saya suka bagaimana Anda merangkum informasi kompleks menjadi sesuatu yang mudah dipahami. Gravity Rollers

    Reply
  22. DGD on March 3, 2025 at 3:49 am

    Erfahre alles über Business, Lifestyle, Promi-News & Trends! Celeb’s Lifestyle hält dich mit Artikeln, Tipps & News zu Themen des modernen Lebens Pamela Reif Vermögen

    Reply
  23. king88 on March 3, 2025 at 4:10 am

    king88 luon di dau trong top nha cai the thao uy tin xanh chin 2025

    Reply
  25. FLEXCO Belt Fasteners on March 3, 2025 at 4:21 am

    Terima kasih atas pemikiran yang dalam dan terperinci. Saya sangat menghargainya. FLEXCO Belt Fasteners

    Reply
  26. FLEXCO Belt Fasteners on March 3, 2025 at 4:34 am

    Terima kasih atas pemikiran yang dalam dan terperinci. Saya sangat menghargainya. FLEXCO Belt Fasteners

    Reply
  27. web cờ bạc online on March 3, 2025 at 4:40 am

    Your writing style is engaging and accessible, making complex concepts easy to understand.

    web cờ bạc online

    Reply
  29. Della Garry on March 3, 2025 at 4:46 am

    Advancements in NLP for Arabic, particularly through AraBERT, showcase the power of specialized models in improving language understanding. The detailed comparison of various models highlights how effective training on a well-structured dataset can significantly enhance performance. Just as AraBERT revolutionizes Arabic NLP, CapCut Mod APK unlocks new possibilities for video editing by providing premium features for free. If you’re looking to elevate your editing experience, visit to download CapCut Mod APK and explore its enhanced capabilities today!

    Reply
  31. WHALE Belt Fasteners on March 3, 2025 at 4:52 am

    Artikel yang sangat bagus! Saya merasa lebih terinformasi sekarang. WHALE Belt Fasteners

    Reply
  32. WHALE Belt Fasteners on March 3, 2025 at 4:58 am

    Artikel yang sangat bagus! Saya merasa lebih terinformasi sekarang. WHALE Belt Fasteners

    Reply
  33. 789Win Gratis on March 3, 2025 at 5:01 am

    789Win thường xuyên tổ chức các sự kiện, giải đấu với giải thưởng lớn, tạo cơ hội cho người chơi tranh tài và nhận về những phần thưởng giá trị.

    Reply
  34. slot on March 3, 2025 at 5:05 am

    NAGA508 adalah situs slot gacor teratas yang menawarkan permainan RTP tinggi terbaru untuk kemenangan maksimal. Dengan akses mudah ke tautan slot gacor hari ini, pemain menikmati permainan yang lancar dan menguntungkan.
    https://www.cortinasyhogaraurora.es/

    Reply
  35. taixiuonline on March 3, 2025 at 5:10 am

    Tài xỉu online – taixiuonline là một hình thức cá cược phổ biến trên nền tảng trực tuyến, nơi người chơi có thể tham gia đặt cược bằng tiền xu (có thể quy đổi thành tiền mặt). Với sự phát triển mạnh mẽ của internet và smartphone, trò chơi này ngày càng trở nên phổ biến, thu hút đông đảo người chơi nhờ tính giải trí cao, dễ tiếp cận và cơ hội trúng thưởng hấp dẫn.

    Reply
  36. Cataract surgery on March 3, 2025 at 5:11 am

    Blue Fin Vision® offers advanced Cataract surgery using cutting-edge technology for precise, minimally invasive treatment. Led by Mr. Mfazo Hove, the clinic ensures fast recovery, optimal vision restoration, and exceptional patient care.

    Reply
  37. köp percocet online utan recept on March 3, 2025 at 5:14 am

    We love our customers, so we have created a store that allows you to get the highest quality products at a low cost köp percocet online utan recept

    Reply
  38. johan smith on March 3, 2025 at 5:16 am

    Discover Gimkit, the gamified learning platform that transforms education with live quizzes, virtual rewards, and student engagement tools.

    Reply
  39. baji on March 3, 2025 at 5:38 am

    Great insights here! If you enjoy online games, visit Baji.

    Reply
  40. baji on March 3, 2025 at 5:39 am

    Great insights here! If you enjoy online games, visit Baji.

    Reply
  41. LASIK on March 3, 2025 at 5:40 am

    Blue Fin Vision® offers advanced LASIK surgery on Harley Street, London, using cutting-edge technology for precise vision correction. Led by Mr. Mfazo Hove, the clinic ensures safe, minimally invasive procedures with fast recovery.

    Reply
  42. ahr147 on March 3, 2025 at 5:49 am

    其中，Nike Phoenix Waffle 和 nike sacai fragment 聯名款無疑是備受矚目的焦點。sacai哪裡買最便宜

    Reply
  44. matinezlopez36 on March 3, 2025 at 6:00 am

    Buy Romanian Driving License Online
    Buy Romanian Driving License Online
    Understanding the Difference Between a Genuine and Fake Romanian Driver’s License
    Buy Real & Authentic Romanian Driving License Online – Fast, Secure, and Legal
    Obtain a Registered Romanian Driver’s License Without the Driving License Exam Online – Fast & Legal
    For those seeking a Romanian driving license without the exam, there are specific online services available that cater to your needs. The process has been streamlined, enabling you to easily buy a registered Romanian driver’s license online. This ensures you can obtain your official Romanian driver’s license swiftly and without the stress of traditional driving tests and exam procedures.
    https://buyexpressdocuments.com/buy-romanian-driving-license-online/

    Contact E-mail:===(support@buyexpressdocuments.com)

    Visit website……buyexpressdocuments.com

    WhatsApp:WhatsApp:+17077330164

    Reply
  46. matinezlopez on March 3, 2025 at 6:03 am

    Buy IELTS Certificate Online
    Buy IELTS Certificate Online without Exam
    Are you looking to enhance your career prospects or gain admission to a prestigious university? The IELTS exam is often a hurdle many individuals face on their journey. What if there was a way to bypass the stress of preparation and testing altogether? Imagine being able to buy an IELTS certificate online, opening doors without the usual barriers. Whether you’re based in Australia, the UK, or elsewhere, this option might be within reach. Let’s explore how to obtain that coveted IELTS qualification without entering an examination hall. https://buyexpressdocuments.com/buy-ielts-certificate-online-fast-verified-ielts-certificates-buy-express-documents/
    How to Get Duplicate IELTS Certificate British Council
    Buy IELTS Certificate in Australia
    Buy IELTS Certificate UK
    The demand for IELTS certificates in the UK is ever-growing. Many individuals seek to enhance their career prospects or pursue education abroad. However, only some have the time or resources to take the test.
    Buy IELTS certificate without exam online
    Many individuals seek ways to streamline their academic and professional journeys in today’s fast-paced world. One option that has gained attention is the ability to buy an IELTS certificate without taking the exam.

    Reply
  47. Blue fin vision on March 3, 2025 at 6:10 am

    Blue fin vision® is a premier eye clinic on Harley Street, London, offering expert cataract surgery, laser eye treatments, and lens replacement. Led by Mr. Mfazo Hove, it ensures advanced, minimally invasive eye care.

    Reply
← Older Comments
Newer Comments →

Submit a Comment

Your email address will not be published. Required fields are marked *