AraBERT : Pre-training BERT for Arabic Language Understanding

by | Feb 28, 2020 | NLP | 49,329 comments

Authors: Wissam Antoun, Fady Baly, Hazem Hajj

 

AraBERT is an Arabic pretrained language model based on Google’s BERT architecture. AraBERT uses the same BERT-Base config. More details are available in the AraBERT PAPER and in the AraBERT Meetup

There is two versions of the model AraBERTv0.1 and AraBERTv1, with the difference being that AraBERTv1 uses pre-segmented text where prefixes and suffixes were split using the Farasa Segmenter.

The model was trained on ~70M sentences or ~23GB of Arabic text with ~3B words.

Source Code Repository: https://github.com/aub-mind/arabert
Paper: https://www.aclweb.org/anthology/2020.osact-1.2.pdf

 

Results (Accuracy)

We evaluate both AraBERT models on different downstream tasks and compare it to mBERT, and other state of the art models (To the extent of our knowledge). The Tasks were Sentiment Analysis on 6 different datasets (HARD, ASTD-Balanced, ArsenTD-Lev, LABR, ArSaS), Named Entity Recognition with the ANERcorp, and Arabic Question Answering on Arabic-SQuAD and ARCD

Task prev. SOTA mBERT AraBERTv0.1 AraBERTv1
HARD 95.7 ElJundi et.al. 95.7 96.2 96.1
ASTD 86.5 ElJundi et.al. 80.1 92.2 92.6
ArsenTD-Lev 52.4 ElJundi et.al. 51 58.9 59.4
AJGT 93 Dahou et.al. 83.6 93.1 93.8
LABR 87.5 Dahou et.al. 83 85.9 86.7
ANERcorp 81.7 (BiLSTM-CRF) 78.4 84.2 81.9
ARCD mBERT EM:34.2 F1: 61.3 EM:51.14 F1:82.13 EM:54.84 F1: 82.15

Model Weights and Vocab Download

Models AraBERTv0.1 AraBERTv1
TensorFlow Drive Link Drive Link
PyTorch Drive_Link Drive_Link

You can find the PyTorch models in HuggingFace’s Transformer Library under the aubmindlab username

If you used this model please cite us as:

@inproceedings{antoun2020arabert,
  title={AraBERT: Transformer-based Model for Arabic Language Understanding},
  author={Antoun, Wissam and Baly, Fady and Hajj, Hazem},
  booktitle={LREC 2020 Workshop Language Resources and Evaluation Conference 11--16 May 2020},
  pages={9}
}

Acknowledgments

Thanks to TensorFlow Research Cloud (TFRC) for the free access to Cloud TPUs, couldn’t have done it without this program, and to the AUB MIND Lab Members for the continous support. Also thanks to Yakshof and Assafir for data and storage access. Another thanks for Habib Rahal (https://www.behance.net/rahalhabib), for putting a face to AraBERT.

Contacts

Wissam Antoun: Linkedin | Twitter | Github | wfa07@mail.aub.edu | wissam.antoun@gmail.com

Fady Baly: Linkedin | Twitter | Github | fgb06@mail.aub.edu | baly.fady@gmail.com

We are looking for sponsors to train BERT-Large and other Transformer models, the sponsor only needs to cover to data storage and compute cost of the generating the pretraining data

49,329 Comments

← Older Comments
Newer Comments →
  1. RAJACABE88 on March 17, 2025 at 11:45 pm

    Social Media Sharing: Promote your content on social media platforms to increase visibility and drive traffic to your website
    RAJACABE88

    Reply
  2. RAJACABE88 on March 17, 2025 at 11:50 pm

    Building Relationships: Engage with other website owners and influencers in your niche to build natural and meaningful backlinks
    RAJACABE88

    Reply
  3. RAJACABE88 on March 17, 2025 at 11:53 pm

    Building Relationships: Engage with other website owners and influencers in your niche to build natural and meaningful backlinks
    RAJACABE88

    Reply
  4. RAJACABE88 on March 17, 2025 at 11:58 pm

    Local Listings: Ensure your business is listed on relevant local directories and websites
    RAJACABE88

    Reply
  5. RAJACABE88 on March 17, 2025 at 11:59 pm

    Local Listings: Ensure your business is listed on relevant local directories and websites
    RAJACABE88

    Reply
  6. RAJACABE88 on March 18, 2025 at 12:02 am

    Content Marketing: Create valuable, shareable content that attracts natural backlinks
    RAJACABE88

    Reply
  7. RAJACABE88 on March 18, 2025 at 12:04 am

    Content Marketing: Create valuable, shareable content that attracts natural backlinks
    RAJACABE88

    Reply
  8. TRP Roller Chains RS 140-2 on March 18, 2025 at 12:11 am

    Informasi yang sangat berguna dan terorganisir dengan baik. Terima kasih atas usaha Anda! TRP Roller Chains RS 140-2

    Reply
  9. RAJACABE88 on March 18, 2025 at 12:11 am

    Guest Blogging: Write articles for reputable sites in your niche
    RAJACABE88

    Reply
  10. RAJACABE88 on March 18, 2025 at 12:13 am

    Guest Blogging: Write articles for reputable sites in your niche
    RAJACABE88

    Reply
  11. RAJACABE88 on March 18, 2025 at 12:16 am

    Networking: Build relationships with influencers or industry leaders who might link to your content
    RAJACABE88

    Reply
  12. RAJACABE88 on March 18, 2025 at 12:17 am

    Networking: Build relationships with influencers or industry leaders who might link to your content
    RAJACABE88

    Reply
  13. basit khan on March 18, 2025 at 12:21 am

    Buy the best selection of HQD vape at good prices! Find premium disposable vapes, fast shipping, and top flavors. Buy now for a smooth vaping experience! hqd vape

    Reply
  14. RAJACABE88 on March 18, 2025 at 12:24 am

    Social Media: Share content on social media to generate traffic and awareness
    RAJACABE88

    Reply
  15. RAJACABE88 on March 18, 2025 at 12:26 am

    Social Media: Share content on social media to generate traffic and awareness
    RAJACABE88

    Reply
  16. TRP Roller Chains RS 140-2 on March 18, 2025 at 12:26 am

    Informasi yang sangat berguna dan terorganisir dengan baik. Terima kasih atas usaha Anda! TRP Roller Chains RS 140-2

    Reply
  17. RAJACABE88 on March 18, 2025 at 12:29 am

    On-site SEO: Optimize your website for better performance and user experience
    RAJACABE88

    Reply
  20. TRP Roller Chains RS 80-2 on March 18, 2025 at 12:40 am

    Artikel yang sangat menarik! Saya tidak bisa berhenti membacanya. TRP Roller Chains RS 80-2

    Reply
  21. TRP Roller Chains RS 80-2 on March 18, 2025 at 12:49 am

    Artikel yang sangat menarik! Saya tidak bisa berhenti membacanya. TRP Roller Chains RS 80-2

    Reply
  24. TRP Roller Chains RS 160-2 on March 18, 2025 at 1:12 am

    Terima kasih telah berbagi pandangan Anda yang berharga. Saya akan merenungkannya lebih dalam. TRP Roller Chains RS 160-2

    Reply
  25. TRP Roller Chains RS 160-2 on March 18, 2025 at 1:16 am

    Terima kasih telah berbagi pandangan Anda yang berharga. Saya akan merenungkannya lebih dalam. TRP Roller Chains RS 160-2

    Reply
  26. Honda EZ6500 CXS on March 18, 2025 at 1:24 am

    Tulisan yang sangat memikat! Saya tidak sabar untuk membaca posting selanjutnya. Honda EZ6500 CXS

    Reply
  27. Starla on March 18, 2025 at 1:35 am

    Unquestionably consider that that you stated. Your favorite justification appeared to
    be at the internet the easiest thing to understand of.
    I say to you, I definitely get irked even as people consider issues that they plainly don’t
    recognize about. You controlled to hit the nail upon the top and
    also defined out the entire thing without having side-effects ,
    other folks could take a signal. Will likely be
    back to get more. Thanks

    My blog – Mt blog indonesia

    Reply
  28. Honda EZ6500 CXS on March 18, 2025 at 1:39 am

    Tulisan yang sangat memikat! Saya tidak sabar untuk membaca posting selanjutnya. Honda EZ6500 CXS

    Reply
  29. TRP Roller Chains RS 35-2 on March 18, 2025 at 1:49 am

    Saya selalu senang ketika saya menemukan blog yang memberikan informasi yang relevan dan akurat seperti ini. TRP Roller Chains RS 35-2

    Reply
  30. TRP Roller Chains RS 35-2 on March 18, 2025 at 1:58 am

    Saya selalu senang ketika saya menemukan blog yang memberikan informasi yang relevan dan akurat seperti ini. TRP Roller Chains RS 35-2

    Reply
  32. Jerold on March 18, 2025 at 2:12 am

    Hi there, I read your blogs regularly. Your humoristic style is
    awesome, keep doing what you’re doing!

    Feel free to visit my blog post; useful source

    Reply
  34. Study Ark on March 18, 2025 at 2:16 am

    Great insights! Studying abroad is a life-changing experience, but the application process can be overwhelming. Professional admission consultants in India, like those at Study Ark, help students choose the right universities, prepare applications, and secure scholarships. Their expert guidance can make the entire journey seamless!

    Reply
  36. wither be on March 18, 2025 at 2:24 am

    Many students make application errors that can cost them their dream university admission. From missing deadlines to incomplete SOPs, small mistakes can have big consequences. Study abroad consultants, like Study Ark, help applicants craft compelling SOPs, LORs, and resumes, ensuring a strong profile. Thorough guidance is key to success!

    Reply
  39. Study Ark on March 18, 2025 at 2:42 am

    Selecting the right country and university is crucial for a successful international education. Factors like tuition fees, career opportunities, and cultural fit matter a lot. Admission consultants in India, such as Study Ark, offer personalized advice to help students make informed decisions. Have you considered seeking professional guidance for your study abroad journey?

    Reply
  42. Tonya on March 18, 2025 at 2:58 am

    Peculiar article, exactly what I needed.

    Stop by my blog post; Discover More Here

    Reply
  45. totositesug on March 18, 2025 at 3:19 am

    This article is a great reminder of why we need to take action on this 토토사이트 추천

    Reply
  46. nbettingsites on March 18, 2025 at 3:20 am

    No doubt it will be very useful for my future projects. Would like to see some other posts on the same subject! 토토사이트

    Reply
  47. fabotosite on March 18, 2025 at 3:21 am

    This blog is so nice to me. I will keep on coming here again and again. Visit my link as well. 스포츠 토토사이트

    Reply
  48. fabotosite on March 18, 2025 at 3:22 am

    This blog is so nice to me. I will keep on coming here again and again. Visit my link as well.
    스포츠 토토사이트

    Reply
  49. fabotosite on March 18, 2025 at 3:23 am

    This blog is so nice to me. I will keep on coming here again and again. Visit my link as well.
    스포츠 토토사이트

    Reply
← Older Comments
Newer Comments →

Submit a Comment

Your email address will not be published. Required fields are marked *