AraBERT : Pre-training BERT for Arabic Language Understanding

by | Feb 28, 2020 | NLP | 37,674 comments

Authors: Wissam Antoun, Fady Baly, Hazem Hajj

 

AraBERT is an Arabic pretrained language model based on Google’s BERT architecture. AraBERT uses the same BERT-Base config. More details are available in the AraBERT PAPER and in the AraBERT Meetup

There is two versions of the model AraBERTv0.1 and AraBERTv1, with the difference being that AraBERTv1 uses pre-segmented text where prefixes and suffixes were split using the Farasa Segmenter.

The model was trained on ~70M sentences or ~23GB of Arabic text with ~3B words.

Source Code Repository: https://github.com/aub-mind/arabert
Paper: https://www.aclweb.org/anthology/2020.osact-1.2.pdf

 

Results (Accuracy)

We evaluate both AraBERT models on different downstream tasks and compare it to mBERT, and other state of the art models (To the extent of our knowledge). The Tasks were Sentiment Analysis on 6 different datasets (HARD, ASTD-Balanced, ArsenTD-Lev, LABR, ArSaS), Named Entity Recognition with the ANERcorp, and Arabic Question Answering on Arabic-SQuAD and ARCD

Task prev. SOTA mBERT AraBERTv0.1 AraBERTv1
HARD 95.7 ElJundi et.al. 95.7 96.2 96.1
ASTD 86.5 ElJundi et.al. 80.1 92.2 92.6
ArsenTD-Lev 52.4 ElJundi et.al. 51 58.9 59.4
AJGT 93 Dahou et.al. 83.6 93.1 93.8
LABR 87.5 Dahou et.al. 83 85.9 86.7
ANERcorp 81.7 (BiLSTM-CRF) 78.4 84.2 81.9
ARCD mBERT EM:34.2 F1: 61.3 EM:51.14 F1:82.13 EM:54.84 F1: 82.15

Model Weights and Vocab Download

Models AraBERTv0.1 AraBERTv1
TensorFlow Drive Link Drive Link
PyTorch Drive_Link Drive_Link

You can find the PyTorch models in HuggingFace’s Transformer Library under the aubmindlab username

If you used this model please cite us as:

@inproceedings{antoun2020arabert,
  title={AraBERT: Transformer-based Model for Arabic Language Understanding},
  author={Antoun, Wissam and Baly, Fady and Hajj, Hazem},
  booktitle={LREC 2020 Workshop Language Resources and Evaluation Conference 11--16 May 2020},
  pages={9}
}

Acknowledgments

Thanks to TensorFlow Research Cloud (TFRC) for the free access to Cloud TPUs, couldn’t have done it without this program, and to the AUB MIND Lab Members for the continous support. Also thanks to Yakshof and Assafir for data and storage access. Another thanks for Habib Rahal (https://www.behance.net/rahalhabib), for putting a face to AraBERT.

Contacts

Wissam Antoun: Linkedin | Twitter | Github | wfa07@mail.aub.edu | wissam.antoun@gmail.com

Fady Baly: Linkedin | Twitter | Github | fgb06@mail.aub.edu | baly.fady@gmail.com

We are looking for sponsors to train BERT-Large and other Transformer models, the sponsor only needs to cover to data storage and compute cost of the generating the pretraining data

37,674 Comments

← Older Comments
Newer Comments →
  1. MBlive on January 24, 2025 at 11:05 pm

    MBlive mang đến trải nghiệm livestream độc đáo, từ những buổi phát sóng đầy hấp dẫn đến các nội dung đa dạng như chia sẻ kỹ năng hay giải trí phòng the đỉnh cao. Bên cạnh đó, ứng dụng còn tích hợp hàng loạt trò chơi thú vị, từ baccarat, xóc đĩa đến các game hiện đại, giúp người chơi thử vận may không giới hạn. MBlive

    Reply
  2. W88 on January 24, 2025 at 11:15 pm

    W88 – W88 TTNET là một thương hiệu cá cược trực tuyến quốc tế nổi bật, cung cấp đa dạng dịch vụ và sản phẩm giải trí. Với tính năng bảo mật tiên tiến, hệ thống đổi thưởng chuyên nghiệp, W88 mang đến cho người chơi trải nghiệm cá cược chất lượng và an toàn.

    Reply
  3. professional tire services in chicago heights on January 25, 2025 at 12:05 am

    For reliable and professional tire services in chicago heights, trust our experienced team to handle all your tire needs. From tire installation and repair to balancing and rotation, we offer top-quality service to ensure your vehicle performs safely and efficiently on the road. Visit us today!

    Reply
  4. Corine on January 25, 2025 at 12:23 am

    Do you have a spam problem on this website; I also am a blogger, and I
    was wondering your situation; we have created some nice methods and we are looking to exchange
    solutions with other folks, why not shoot me an e-mail if interested.

    my web blog … chat gpt try

    Reply
  5. Lasonya on January 25, 2025 at 12:23 am

    I am sure this article has touched all the internet viewers, its really really good article on building up new website.

    Here is my webpage … متابعين تيك توك

    Reply
  6. TRP Roller Chains RS 80-1 on January 25, 2025 at 1:14 am

    saya berterimakasih karena website ini selalu memberikan hal menarik TRP Roller Chains RS 80-1

    Reply
  13. sòng bạc trực tuyến on January 25, 2025 at 1:47 am

    I wanted to take a moment to express my heartfelt gratitude for your outstanding article. Your insights have truly enlightened me.
    sòng bạc trực tuyến

    Reply
  19. 789Win on January 25, 2025 at 2:27 am

    If you can’t fly then run, if you can’t run then walk. https://789wins.bet/

    Reply
  23. 888B on January 25, 2025 at 3:09 am

    https://888bb.sbs/ tự tin là nhà cái uy tín đẳng cấp hàng đầu Châu Á. Đăng ký tài khoản tại 888bb.sbs để nhận 88k vốn cược miễn phí với vô số ưu đãi!

    Reply
  26. SHBET on January 25, 2025 at 3:57 am

    SHBETkhông chỉ là một nền tảng cá cược trực tuyến mà còn là điểm đến tin cậy dành cho những ai đam mê giải trí và muốn tìm kiếm cơ hội thắng lớn. Với hệ thống trò chơi đa dạng, tỷ lệ cược hấp dẫn và dịch vụ chuyên nghiệp, SHBET cam kết mang đến trải nghiệm an toàn, minh bạch và đầy kịch tính cho người chơi.

    Reply
  28. Ulrike on January 25, 2025 at 4:04 am

    I have been browsing online more than three hours today, but I never found any fascinating article like yours.
    It’s pretty price sufficient for me. In my opinion, if all web owners and bloggers made good
    content as you probably did, the net will likely be a lot more useful than ever before.

    My web site ручка перьевая аврора

    Reply
  29. rr88 on January 25, 2025 at 4:07 am

    If you can’t walk then crawl, but whatever you do you have to keep moving forward. link vào RR88

    Reply
  32. HUBET on January 25, 2025 at 4:31 am

    HUBET tự hào là một trong những nền tảng giải trí hiện đại nhất, mang đến không gian thư giãn sôi động và đầy cuốn hút. Với cam kết mang lại trải nghiệm giải trí tuyệt vời, HUBET luôn đáp ứng mọi nhu cầu của người chơi, từ dịch vụ chất lượng đến đa dạng sản phẩm hấp dẫn. Tham gia ngay để tận hưởng những phút giây giải trí đỉnh cao!

    Reply
  34. TK88 on January 25, 2025 at 4:40 am

    https://tk88.pw/ là nhà cái cá cược uy tín nhất tại Việt Nam với hệ thống bảo mật cao, đa dạng sản phẩm và thường xuyên có các sự kiện ưu đãi lớn.

    Reply
  36. aa on January 25, 2025 at 5:41 am

    Hey what a brilliant post I have come across and believe me I have been searching out for this similar kind of post for past a week and hardly came across this. Thank you very much and will look for more postings from you. Private BTech College in Rajasthan

    Reply
  37. jung hong on January 25, 2025 at 5:43 am

    Discover new opportunities and insights with 오피, your trusted source for business information.

    Reply
  38. boundhub on January 25, 2025 at 6:01 am

    Boundhub is a versatile platform that offers users a wide range of features tailored for specific interests and needs. Whether you’re a casual user or someone looking for niche content, boundhub provides a robust solution to explore, share, and connect with a global audience. In this blog post, we’ll dive deep into what makes Boundhub unique, its key features, and tips on how to use it effectively.

    Reply
  39. innov8equipment1609 on January 25, 2025 at 6:02 am

    Innov8 VB is making waves with the release of its highly anticipated full-colour video trailer, showcasing the brand’s cutting-edge technology and innovation in a vibrant and captivating display. The trailer brings to life the essence of the Innov8 VB product, highlighting its versatile functionality, dynamic visuals, and advanced features. Viewers are treated to a stunning array of colors, seamlessly integrated into a sleek design that embodies the forward-thinking vision of Innov8.

    https://innov8equipment.com.au/innov8-vb-full-colour-video-trailer/

    Reply
  41. Baby James on January 25, 2025 at 6:04 am

    You made such an interesting piece to read, giving every subject enlightenment for us to gain knowledge. Thanks for sharing the such information with us to read this… best university in jaipur

    Reply
  44. Nickolas on January 25, 2025 at 6:16 am

    Excellent post. Keep posting such kind of information on your page.
    Im really impressed by your site.
    Hey there, You’ve performed an incredible job.
    I will definitely digg it and in my view recommend to my friends.

    I’m confident they will be benefited from this web site.

    My web site خرید بک لینک

    Reply
  45. kasif sofeyn kasan on January 25, 2025 at 6:21 am

    I usually cant find it in me to care enough to leaves a comment for articles on the web but this was actually pretty good, thanks and keep it up, Ill check back again net88

    Reply
  50. asie on January 25, 2025 at 6:30 am

    فرصت مطالعاتی دانشگاه پیام نور به اساتید این امکان را می‌دهد تا با اخذ مرخصی مطالعاتی، در دانشگاه‌های معتبر داخل یا خارج از کشور، به ادامه تحصیل در مقاطع بالاتر یا انجام فعالیت‌های پژوهشی بپردازند.

    Reply
← Older Comments
Newer Comments →

Submit a Comment

Your email address will not be published. Required fields are marked *