AraBERT : Pre-training BERT for Arabic Language Understanding

by | Feb 28, 2020 | NLP | 28 comments

Authors: Wissam Antoun, Fady Baly, Hazem Hajj

 

AraBERT is an Arabic pretrained language model based on Google’s BERT architecture. AraBERT uses the same BERT-Base config. More details are available in the AraBERT PAPER and in the AraBERT Meetup

There is two versions of the model AraBERTv0.1 and AraBERTv1, with the difference being that AraBERTv1 uses pre-segmented text where prefixes and suffixes were split using the Farasa Segmenter.

The model was trained on ~70M sentences or ~23GB of Arabic text with ~3B words.

Source Code Repository: https://github.com/aub-mind/arabert
Paper: https://www.aclweb.org/anthology/2020.osact-1.2.pdf

 

Results (Accuracy)

We evaluate both AraBERT models on different downstream tasks and compare it to mBERT, and other state of the art models (To the extent of our knowledge). The Tasks were Sentiment Analysis on 6 different datasets (HARD, ASTD-Balanced, ArsenTD-Lev, LABR, ArSaS), Named Entity Recognition with the ANERcorp, and Arabic Question Answering on Arabic-SQuAD and ARCD

Task prev. SOTA mBERT AraBERTv0.1 AraBERTv1
HARD 95.7 ElJundi et.al. 95.7 96.2 96.1
ASTD 86.5 ElJundi et.al. 80.1 92.2 92.6
ArsenTD-Lev 52.4 ElJundi et.al. 51 58.9 59.4
AJGT 93 Dahou et.al. 83.6 93.1 93.8
LABR 87.5 Dahou et.al. 83 85.9 86.7
ANERcorp 81.7 (BiLSTM-CRF) 78.4 84.2 81.9
ARCD mBERT EM:34.2 F1: 61.3 EM:51.14 F1:82.13 EM:54.84 F1: 82.15

Model Weights and Vocab Download

Models AraBERTv0.1 AraBERTv1
TensorFlow Drive Link Drive Link
PyTorch Drive_Link Drive_Link

You can find the PyTorch models in HuggingFace’s Transformer Library under the aubmindlab username

If you used this model please cite us as:

@inproceedings{antoun2020arabert,
  title={AraBERT: Transformer-based Model for Arabic Language Understanding},
  author={Antoun, Wissam and Baly, Fady and Hajj, Hazem},
  booktitle={LREC 2020 Workshop Language Resources and Evaluation Conference 11--16 May 2020},
  pages={9}
}

Acknowledgments

Thanks to TensorFlow Research Cloud (TFRC) for the free access to Cloud TPUs, couldn’t have done it without this program, and to the AUB MIND Lab Members for the continous support. Also thanks to Yakshof and Assafir for data and storage access. Another thanks for Habib Rahal (https://www.behance.net/rahalhabib), for putting a face to AraBERT.

Contacts

Wissam Antoun: Linkedin | Twitter | Github | wfa07@mail.aub.edu | wissam.antoun@gmail.com

Fady Baly: Linkedin | Twitter | Github | fgb06@mail.aub.edu | baly.fady@gmail.com

We are looking for sponsors to train BERT-Large and other Transformer models, the sponsor only needs to cover to data storage and compute cost of the generating the pretraining data

28 Comments

  2. Hdhd on March 22, 2024 at 8:39 am

    or loved ones, “สล็อตฝากถอนวอเล็ท”

  5. 카지노사이트 on May 20, 2024 at 5:24 am

    대부분의 카지노사이트와 바카라사이트에서는 해외 유명 게임사들의 게임을 카피하여 게임을 제작해서 자체적으로 운영하고 있습니다. 하지만 해외 게임사들의 게임과 계약을 진행하여 회원님들이 플레이하는 게임을 더욱더 투명하고 클린하게 게임을 운영하고 있습니다.

  6. WEAVER on May 29, 2024 at 2:25 pm

    testy1testy1testy1testy1

  9. falisha manpower on August 19, 2024 at 12:05 pm

    get free services

  10. true religion sweatpants on August 30, 2024 at 10:51 am

    very good work, keep it up

  11. with vegas on August 31, 2024 at 12:26 pm

    최신 카지노사이트 먹튀검증 시스템으로 안전하고 신뢰할 수 있는 게임을 즐기세요!
    위드베가스는 최신 카지노사이트 먹튀검증 시스템을 통해 안전한 온라인 게임 환경을 보장합니다. 철저한 검증을 거친 카지노사이트 먹튀검증에서 안심하고 게임을 즐기실 수 있으며, 슬롯, 토토, 홀덤 등 다양한 인기 게임과 함께 최신 뉴스와 프로모션도 제공하여 더욱 신뢰받는 플랫폼으로 자리 잡고 있습니다.
    위드베가스는 최신 카지노사이트 먹튀검증 시스템을 기반으로 카지노사이트 먹튀검증을 철저히 실시하여 먹튀 사례로부터 사용자를 보호하기 위해 강력한 보안 시스템을 운영하고 있으며, 사용자 안전을 최우선으로 하는 서비스로 많은 이용자들 사이에서 큰 호평을 받고 있습니다. 검증된 온라인 게임 사이트를 통해 다양한 혜택과 함께 안전한 게임 환경을 경험해 보세요.
    지금 바로 위드베가스를 방문하여 최신 카지노사이트 먹튀검증 시스템과 카지노사이트 먹튀검증을 직접 확인하고, 안심하고 게임을 즐겨보세요!

    visit for more :

  12. 소액결제 현금화 업체 on September 18, 2024 at 7:43 am

    소액결제 현금화 업체는 소액 결제된 금액을 현금으로 변환해주는 전문 서비스입니다. 친구티켓은 정식 사업자 등록을 마치고, 연중무휴 고객센터를 운영하며 안전하고 신뢰할 수 있는 서비스를 제공합니다. 최근 사기 업체에 주의하고, 검증된 업체를 선택하는 것이 중요합니다.

  25. aliuamir aliuad on December 25, 2024 at 8:42 am

    sdfdsffdsfsd

  26. ซื้อหวยลาว 4 ตัว เว็บไหนดี on December 26, 2024 at 9:43 am

    แนะนำเว็บซื้อหวยลาวที่ดีที่สุด ควรเลือกเว็บที่มีระบบการเล่นที่เข้าใจง่าย รองรับทุกอุปกรณ์และให้บริการลูกค้าอย่างมืออาชีพ

  27. หวยวิ่ง on December 26, 2024 at 9:59 am

    การใช้สูตรหวยวิ่งบน เซียนหวยหลายคนใช้สูตรหวยวิ่งบนที่มีการคำนวณจากสถิติและการวิเคราะห์ผลรางวัลที่ผ่านมา ซึ่งจะช่วยให้ผู้เล่นสามารถเลือกเลขได้แม่นยำขึ้น และเพิ่มโอกาสถูกรางวัลในแต่ละงวด

Submit a Comment