AraBERT : Pre-training BERT for Arabic Language Understanding

by | Feb 28, 2020 | NLP | 49,367 comments

Authors: Wissam Antoun, Fady Baly, Hazem Hajj

 

AraBERT is an Arabic pretrained language model based on Google’s BERT architecture. AraBERT uses the same BERT-Base config. More details are available in the AraBERT PAPER and in the AraBERT Meetup

There is two versions of the model AraBERTv0.1 and AraBERTv1, with the difference being that AraBERTv1 uses pre-segmented text where prefixes and suffixes were split using the Farasa Segmenter.

The model was trained on ~70M sentences or ~23GB of Arabic text with ~3B words.

Source Code Repository: https://github.com/aub-mind/arabert
Paper: https://www.aclweb.org/anthology/2020.osact-1.2.pdf

 

Results (Accuracy)

We evaluate both AraBERT models on different downstream tasks and compare it to mBERT, and other state of the art models (To the extent of our knowledge). The Tasks were Sentiment Analysis on 6 different datasets (HARD, ASTD-Balanced, ArsenTD-Lev, LABR, ArSaS), Named Entity Recognition with the ANERcorp, and Arabic Question Answering on Arabic-SQuAD and ARCD

Task prev. SOTA mBERT AraBERTv0.1 AraBERTv1
HARD 95.7 ElJundi et.al. 95.7 96.2 96.1
ASTD 86.5 ElJundi et.al. 80.1 92.2 92.6
ArsenTD-Lev 52.4 ElJundi et.al. 51 58.9 59.4
AJGT 93 Dahou et.al. 83.6 93.1 93.8
LABR 87.5 Dahou et.al. 83 85.9 86.7
ANERcorp 81.7 (BiLSTM-CRF) 78.4 84.2 81.9
ARCD mBERT EM:34.2 F1: 61.3 EM:51.14 F1:82.13 EM:54.84 F1: 82.15

Model Weights and Vocab Download

Models AraBERTv0.1 AraBERTv1
TensorFlow Drive Link Drive Link
PyTorch Drive_Link Drive_Link

You can find the PyTorch models in HuggingFace’s Transformer Library under the aubmindlab username

If you used this model please cite us as:

@inproceedings{antoun2020arabert,
  title={AraBERT: Transformer-based Model for Arabic Language Understanding},
  author={Antoun, Wissam and Baly, Fady and Hajj, Hazem},
  booktitle={LREC 2020 Workshop Language Resources and Evaluation Conference 11--16 May 2020},
  pages={9}
}

Acknowledgments

Thanks to TensorFlow Research Cloud (TFRC) for the free access to Cloud TPUs, couldn’t have done it without this program, and to the AUB MIND Lab Members for the continous support. Also thanks to Yakshof and Assafir for data and storage access. Another thanks for Habib Rahal (https://www.behance.net/rahalhabib), for putting a face to AraBERT.

Contacts

Wissam Antoun: Linkedin | Twitter | Github | wfa07@mail.aub.edu | wissam.antoun@gmail.com

Fady Baly: Linkedin | Twitter | Github | fgb06@mail.aub.edu | baly.fady@gmail.com

We are looking for sponsors to train BERT-Large and other Transformer models, the sponsor only needs to cover to data storage and compute cost of the generating the pretraining data

49,367 Comments

← Older Comments
Newer Comments →
  2. Frankie on March 4, 2025 at 5:43 am

    Good article! We are linking to this great content on our site.
    Keep up the good writing.

    Look into my blog click this site

    Reply
  3. rs 125 only on thespark shop men jackets on March 4, 2025 at 5:54 am

    Thanks for the nice blog. It was very useful for me. I m happy I found this blog. Thank you for sharing with us,I too always learn something new from your post. rs 125 only on thespark shop men jackets

    Reply
  5. max baer age 103 on March 4, 2025 at 6:06 am

    Thanks for the nice blog. It was very useful for me. I m happy I found this blog. Thank you for sharing with us,I too always learn something new from your post. max baer age 103

    Reply
  6. https://k8kbet.org/ on March 4, 2025 at 6:07 am

    8KBET – https://k8kbet.org/ là sân chơi cá cược trực tuyến hàng đầu, hội tụ đa dạng các loại hình giải trí từ thể thao, sòng bài trực tuyến đến slot nổ hũ đầy kịch tính, hấp dẫn. Sở hữu giao diện website trực quan, thân thiện, dễ tiếp cận cùng hệ thống bảo mật hiện đại, mang đến trải nghiệm an toàn, mượt mà cho người chơi.

    Reply
  7. the spark shop men winter jacket sportswear gym fitness on March 4, 2025 at 6:14 am

    Thanks for the nice blog. It was very useful for me. I m happy I found this blog. Thank you for sharing with us,I too always learn something new from your post. the spark shop men winter jacket sportswear gym fitness

    Reply
  8. DURAJOINT FCL Couplings 160 on March 4, 2025 at 6:22 am

    Sangat menyukai pendekatan Anda dalam menangani masalah ini. Sangat praktis! DURAJOINT FCL Couplings 160

    Reply
  9. BootezumVerkauf on March 4, 2025 at 6:27 am

    “Boote zum Verkauf bieten eine fantastische Möglichkeit, das Abenteuer auf dem Wasser zu erleben! Egal ob für ruhige Fahrten auf dem See, sportliches Segeln oder entspannte Ausflüge mit einem Motorboot – hier findet jeder das passende Modell. Schauen Sie sich die verschiedenen Angebote an und finden Sie das Boot, das perfekt zu Ihren Bedürfnissen passt!”

    Reply
  10. rs 119 wireless earbuds for gaming thespark shop on March 4, 2025 at 6:27 am

    Thanks for the nice blog. It was very useful for me. I m happy I found this blog. Thank you for sharing with us,I too always learn something new from your post. rs 119 wireless earbuds for gaming thespark shop

    Reply
  12. thespark shop batman style wireless bluetooth earbuds on March 4, 2025 at 6:36 am

    Thanks for the nice blog. It was very useful for me. I m happy I found this blog. Thank you for sharing with us,I too always learn something new from your post. thespark shop batman style wireless bluetooth earbuds

    Reply
  13. mytyles - wall and f on March 4, 2025 at 6:49 am

    Thanks for the nice blog. It was very useful for me. I m happy I found this blog. Thank you for sharing with us,I too always learn something new from your post. mytyles – wall and f

    Reply
  15. Adnan on March 4, 2025 at 6:53 am

    Looking for the latest trending video editing templates? Discover the Ical Template on CapCut, a perfect choice for creating stunning and viral videos effortlessly. With a variety of customizable options, you can enhance your content and stay ahead of social media trends. Check out the latest templates now: Ical Template .

    Reply
  16. https://sam868.com/ on March 4, 2025 at 6:53 am

    SAM86 – https://sam868.com/ là nhà cái uy tín mang đến trải nghiệm cá cược đẳng cấp với đa dạng sản phẩm hấp dẫn như thể thao, đánh bài, slot game và hơn thế nữa. Giao diện hiện đại, bảo mật cao cùng tỷ lệ cược cạnh tranh giúp đây trở thành lựa chọn hàng đầu cho những ai đam mê cá cược trực tuyến.

    Reply
  17. Mominul Islam Sagor on March 4, 2025 at 6:55 am

    If more people that write articles really concerned themselves with writing great content like you, more readers would be interested in their writings.Thank you for caring about your content. numlooker

    Reply
  18. Globesimregistration on March 4, 2025 at 6:56 am

    Globe SIM registration is a required process for all Globe users in the Philippines to enhance mobile security and prevent fraud. In compliance with the SIM Registration Act (Republic Act 11934), all users must register their SIM cards using valid identification to continue using Globe services. This ensures seamless access to calls, texts, data, and financial platforms like GCash.
    Globesimregistration

    Reply
  20. DURAJOINT FCL Couplings 180 on March 4, 2025 at 7:03 am

    Terima kasih telah berbagi sumber daya yang berkualitas seperti ini. Sangat dihargai! DURAJOINT FCL Couplings 180

    Reply
  21. login cr7vip on March 4, 2025 at 7:03 am

    It is very rare these days to find sites that provide information someone is watching for. I am glad to see that your site share valued information that can help to many readers. nice one and keep writing! login cr7vip

    Reply
  22. https://ee-88.it.com/ on March 4, 2025 at 7:07 am

    EE88 – https://ee-88.it.com/ không chỉ cung cấp dịch vụ cá cược đa dạng mà còn mang đến nhiều chương trình khuyến mãi hấp dẫn cho thành viên mới và người chơi lâu năm. Từ thưởng chào mừng, hoàn tiền cược đến các chương trình ưu đãi VIP, EE88 luôn tạo điều kiện tốt nhất để người chơi gia tăng lợi nhuận và có trải nghiệm cá cược thú vị.

    Reply
  24. login okebos on March 4, 2025 at 7:11 am

    Im glad to have found this post as its such an interesting one! I am always on the watchout for quality posts and articles so i suppose im lucky to have found this! I hope you will be adding more in the future… login okebos

    Reply
  25. ubaid on March 4, 2025 at 7:13 am

    “Desert Sound offers top-notch audio solutions with exceptional quality and professionalism. Their expertise in sound engineering truly sets them apart, making them the go-to choice for anyone looking for the best sound system installation in Pakistan. Highly recommended for both residential and commercial setups!”

    Reply
  26. DURAJOINT FCL Couplings 180 on March 4, 2025 at 7:17 am

    Terima kasih telah berbagi sumber daya yang berkualitas seperti ini. Sangat dihargai! DURAJOINT FCL Couplings 180

    Reply
  28. Men’s vitality product deals on March 4, 2025 at 7:20 am

    I was recommended this website through my cousin. I am now not positive whether this put up is written via him as nobody else understand such detailed Men’s vitality product deals

    Reply
  29. Ali ladla on March 4, 2025 at 7:24 am

    It is actually all at once a great put which i quite savored surfing. Is not really routine that give the outlook to see items. Tinychat

    Reply
  31. Crew CloudySocial on March 4, 2025 at 7:27 am

    Nice information, valuable and excellent design, as share good stuff with good ideas and concepts, lots of great information and inspiration, both of which I need, thanks to offer such a helpful information here. Crew CloudySocial

    Reply
  33. ChosenViber on March 4, 2025 at 7:47 am

    Really nice and interesting post. I was looking for this kind of information and enjoyed reading this one. Keep posting. Thanks for sharing. ChosenViber

    Reply
  34. DURAJOINT FCL Couplings 200 on March 4, 2025 at 7:54 am

    Saya suka cara Anda menyajikan informasi dalam cara yang menarik dan mudah dimengerti. DURAJOINT FCL Couplings 200

    Reply
  36. DURAJOINT FCL Couplings 200 on March 4, 2025 at 8:00 am

    Saya suka cara Anda menyajikan informasi dalam cara yang menarik dan mudah dimengerti. DURAJOINT FCL Couplings 200

    Reply
  37. AbitHelp Tablets on March 4, 2025 at 8:00 am

    I really enjoy simply reading all of your weblogs. Simply wanted to inform you that you have people like me who appreciate your work. Definitely a great post. Hats off to you! The information that you have provided is very helpful. AbitHelp Tablets

    Reply
  38. kharizaffar on March 4, 2025 at 8:04 am

    I am so happy to read this. This is the kind of manual that needs to be given and not the random misinformation that is at the other blogs. Appreciate your sharing this best doc. bandar togel

    Reply
  39. net worth the boring magazine on March 4, 2025 at 8:14 am

    I really enjoy simply reading all of your weblogs. Simply wanted to inform you that you have people like me who appreciate your work. Definitely a great post. Hats off to you! The information that you have provided is very helpful. net worth the boring magazine

    Reply
  40. Cakhia TV on March 4, 2025 at 8:21 am

    Hãy đón xem Cakhia TV – kênh truyền hình trực tuyến hàng đầu, phát sóng trực tiếp các giải đấu danh giá như Ngoại Hạng Anh, Bundesliga, Ligue 1, Serie A, VFF Cup và nhiều sự kiện bóng đá hấp dẫn khác. Kênh cung cấp lịch thi đấu chi tiết, tin tức cập nhật, bảng xếp hạng chính xác, livescore liên tục và tỷ lệ kèo uy tín, đáp ứng nhu cầu của người hâm mộ và nhà cá cược. Truy cập garcimunoz.com để thưởng thức những trận cầu đỉnh cao, cùng trải nghiệm công nghệ số tiên tiến và những pha bóng “làm mưa làm gió” như một bữa tiệc giải trí không giới hạn!

    Reply
  41. daytimestar on March 4, 2025 at 8:22 am

    I wanted to leave a little comment to support you and wish you a good continuation. Wishing you the best of luck for all your blogging efforts. daytimestar

    Reply
  42. upcoming fixtures sffareboxing on March 4, 2025 at 8:32 am

    Really nice and interesting post. I was looking for this kind of information and enjoyed reading this one. Keep posting. Thanks for sharing. upcoming fixtures sffareboxing

    Reply
  43. Mominul Islam Sagor on March 4, 2025 at 8:34 am

    Most of the time I don’t make comments on websites, but I’d like to say that this article really forced me to do so. Really nice post! numlooker

    Reply
  45. MS SUFIYAN on March 4, 2025 at 8:43 am

    That may be additionally a fantastic post i truly loved understanding. It isn’t really on a daily basis we carry the opportunity to uncover another thing. কুরআন শিক্ষা

    Reply
  47. NowCollege 1v1 on March 4, 2025 at 8:43 am

    Its as if you had a great grasp on the subject matter, but you forgot to include your readers. Perhaps you should think about this from more than one angle. NowCollege 1v1

    Reply
  48. seo on March 4, 2025 at 8:47 am

    Hello, Neat post. There is an issue along with your site in web explorer, could test this¡K IE still is the marketplace leader and a huge portion of other people will miss your magnificent writing because of this problem. togel china

    Reply
  49. seoi on March 4, 2025 at 8:48 am

    Ultimately, a problem that I’m passionate about. I have looked for data of this caliber for the previous various hours. Your site is greatly appreciated. data china terlengkap

    Reply
  50. Ahmed Raza on March 4, 2025 at 8:50 am

    Gli occhiali da sole Levi’s x Isabel Marant si distinguono per le loro montature raffinate, spesso caratterizzate da linee sottili e forme eleganti che richiamano un’estetica vintage. Le lenti offrono una protezione UV eccellente, rendendoli non solo un accessorio di stile, ma anche una scelta pratica per la protezione degli occhi. Le tonalità disponibili spaziano dai classici colori neutri a sfumature più audaci, in linea con la visione creativa di Isabel Marant. occhiali da sole michael kors

    Reply
← Older Comments
Newer Comments →

Submit a Comment

Your email address will not be published. Required fields are marked *