AraBERT : Pre-training BERT for Arabic Language Understanding

by | Feb 28, 2020 | NLP | 46,940 comments

Authors: Wissam Antoun, Fady Baly, Hazem Hajj

 

AraBERT is an Arabic pretrained language model based on Google’s BERT architecture. AraBERT uses the same BERT-Base config. More details are available in the AraBERT PAPER and in the AraBERT Meetup

There is two versions of the model AraBERTv0.1 and AraBERTv1, with the difference being that AraBERTv1 uses pre-segmented text where prefixes and suffixes were split using the Farasa Segmenter.

The model was trained on ~70M sentences or ~23GB of Arabic text with ~3B words.

Source Code Repository: https://github.com/aub-mind/arabert
Paper: https://www.aclweb.org/anthology/2020.osact-1.2.pdf

 

Results (Accuracy)

We evaluate both AraBERT models on different downstream tasks and compare it to mBERT, and other state of the art models (To the extent of our knowledge). The Tasks were Sentiment Analysis on 6 different datasets (HARD, ASTD-Balanced, ArsenTD-Lev, LABR, ArSaS), Named Entity Recognition with the ANERcorp, and Arabic Question Answering on Arabic-SQuAD and ARCD

Task prev. SOTA mBERT AraBERTv0.1 AraBERTv1
HARD 95.7 ElJundi et.al. 95.7 96.2 96.1
ASTD 86.5 ElJundi et.al. 80.1 92.2 92.6
ArsenTD-Lev 52.4 ElJundi et.al. 51 58.9 59.4
AJGT 93 Dahou et.al. 83.6 93.1 93.8
LABR 87.5 Dahou et.al. 83 85.9 86.7
ANERcorp 81.7 (BiLSTM-CRF) 78.4 84.2 81.9
ARCD mBERT EM:34.2 F1: 61.3 EM:51.14 F1:82.13 EM:54.84 F1: 82.15

Model Weights and Vocab Download

Models AraBERTv0.1 AraBERTv1
TensorFlow Drive Link Drive Link
PyTorch Drive_Link Drive_Link

You can find the PyTorch models in HuggingFace’s Transformer Library under the aubmindlab username

If you used this model please cite us as:

@inproceedings{antoun2020arabert,
  title={AraBERT: Transformer-based Model for Arabic Language Understanding},
  author={Antoun, Wissam and Baly, Fady and Hajj, Hazem},
  booktitle={LREC 2020 Workshop Language Resources and Evaluation Conference 11--16 May 2020},
  pages={9}
}

Acknowledgments

Thanks to TensorFlow Research Cloud (TFRC) for the free access to Cloud TPUs, couldn’t have done it without this program, and to the AUB MIND Lab Members for the continous support. Also thanks to Yakshof and Assafir for data and storage access. Another thanks for Habib Rahal (https://www.behance.net/rahalhabib), for putting a face to AraBERT.

Contacts

Wissam Antoun: Linkedin | Twitter | Github | wfa07@mail.aub.edu | wissam.antoun@gmail.com

Fady Baly: Linkedin | Twitter | Github | fgb06@mail.aub.edu | baly.fady@gmail.com

We are looking for sponsors to train BERT-Large and other Transformer models, the sponsor only needs to cover to data storage and compute cost of the generating the pretraining data

46,940 Comments

← Older Comments
  1. Regulation on March 9, 2025 at 7:05 pm

    Going to graduate school was a positive decision for me. I enjoyed the coursework, the presentations, the fellow students, and the professors. And since my company reimbursed 100% of the tuition, the only cost that I had to pay on my own was for books and supplies. Otherwise, I received a free master’s degree. All that I had to invest was my time. Regulation

    Reply
  2. Be1Crypto on March 9, 2025 at 7:15 pm

    This post is very simple to read and appreciate without leaving any details out. Great work! Be1Crypto

    Reply
  3. ssssssssss on March 9, 2025 at 7:16 pm

    I need to verify with you here. Which isn’t one thing I often do! I get pleasure from reading a publish that can make people think. Additionally, thanks for allowing me to remark! S666

    Reply
  4. Grover on March 9, 2025 at 7:28 pm

    A testnet faucet provides web3 developers with free tokens for deploying, testing, and optimizing smart contracts on test blockchains such as Sepolia, Goerli, and Mumbai.

    Because smart contracts on public, mainnet blockchains like Ethereum and Polygon require gas fees to run smart contracts,
    testnets provide blockchain developers with a network that mirrors production blockchain environments without requiring gas fees that cost real money.

    Get testnet node Tokens!

    Reply
  5. Cryptocurrency Traders on March 9, 2025 at 7:29 pm

    It’s very informative and you are obviously very knowledgeable in this area. You have opened my eyes to varying views on this topic with interesting and solid content. Cryptocurrency Traders

    Reply
  7. ek rupee coin ka manufacturing cost kitna hoga? on March 9, 2025 at 7:53 pm

    This is my first time i visit here. I found so many entertaining stuff in your blog, especially its discussion. From the tons of comments on your articles, I guess I am not the only one having all the leisure here! Keep up the good work. I have been meaning to write something like this on my website and you have given me an idea. ek rupee coin ka manufacturing cost kitna hoga?

    Reply
  8. 123bet on March 9, 2025 at 8:03 pm

    123bet: Nơi hội tụ những game bài thú vị! Tham gia ngay, chiến thắng dễ dàng và nhận thưởng cực hấp dẫn!

    Reply
  9. thesparkshop on March 9, 2025 at 8:04 pm

    Going to graduate school was a positive decision for me. I enjoyed the coursework, the presentations, the fellow students, and the professors. And since my company reimbursed 100% of the tuition, the only cost that I had to pay on my own was for books and supplies. Otherwise, I received a free master’s degree. All that I had to invest was my time. thesparkshop

    Reply
  10. WellHealthOrganic Protein on March 9, 2025 at 8:15 pm

    Took me time to understand all of the comments, but I seriously enjoyed the write-up. It proved being really helpful to me and Im positive to all of the commenters right here! Its constantly nice when you can not only be informed, but also entertained! I am certain you had enjoyable writing this write-up. WellHealthOrganic Protein

    Reply
  11. Kajer Subidha on March 9, 2025 at 8:29 pm

    I will really appreciate the writer’s choice for choosing this excellent article appropriate to my matter.Here is deep description about the article matter which helped me more. Kajer Subidha

    Reply
  12. wheon gta 5 on March 9, 2025 at 8:41 pm

    It’s late finding this act. At least, it’s a thing to be familiar with that there are such events exist. I agree with your Blog and I will be back to inspect it more in the future so please keep up your act. wheon gta 5

    Reply
  13. thespark shop clothing men on March 9, 2025 at 8:59 pm

    Good to become visiting your weblog again, it has been months for me. Nicely this article that i’ve been waited for so long. I will need this post to total my assignment in the college, and it has exact same topic together with your write-up. Thanks, good share. thespark shop clothing men

    Reply
  16. whitemagz on March 9, 2025 at 9:43 pm

    You completed certain reliable points there. I did a search on the subject and found nearly all persons will agree with your blog. whitemagz

    Reply
  17. Book Now – Luxury Villa in Kalkan on March 9, 2025 at 9:46 pm

    Looking for a Villa in kalkan for your next getaway? This stunning coastal town offers a range of luxury villas with private pools, breathtaking sea views, and modern amenities. Whether you’re planning a romantic retreat, family holiday, or group vacation, Kalkan has the perfect stay for you. Enjoy sun-soaked terraces, infinity pools, and easy access to beautiful beaches, fine dining, and vibrant nightlife. Book now to secure the best deals on holiday villas in Kalkan and experience the ultimate Mediterranean escape!

    #VillaInKalkan #BookNow #LuxuryVillas #KalkanHolidays #PrivatePool

    Reply
  18. www betterthisworld com on March 9, 2025 at 9:57 pm

    Such a very useful article. Very interesting to read this article.I would like to thank you for the efforts you had made for writing this awesome article. www betterthisworld com

    Reply
  19. art thunderonthegulf crafts on March 9, 2025 at 10:10 pm

    I was just browsing through the internet looking for some information and came across your blog. I am impressed by the information that you have on this blog. It shows how well you understand this subject. Bookmarked this page, will come back for more. art thunderonthegulf crafts

    Reply
  20. CallScroll on March 9, 2025 at 10:25 pm

    Awesome blog. I enjoyed reading your articles. This is truly a great read for me. I have bookmarked it and I am looking forward to reading new articles. Keep up the good work! CallScroll

    Reply
  21. www on thisveryspot .com on March 9, 2025 at 10:34 pm

    It’s very informative and you are obviously very knowledgeable in this area. You have opened my eyes to varying views on this topic with interesting and solid content. www on thisveryspot .com

    Reply
  22. Lanora on March 9, 2025 at 11:58 pm

    In recent years, the demand for effective
    air and surface purification solutions has surged, driven by heightened awareness of environmental pollutants
    and pathogens.

    My site :: Activpure CBD

    Reply
  23. In N Out Menu Prices on March 10, 2025 at 12:43 am

    In-N-Out menu prices are affordable, offering quality burgers, fries, and shakes. The classic Double-Double costs around $4.50, while a Cheeseburger is about $3.50. Fries are around $2, and shakes cost $2.50–$3. Secret menu items like Animal Style add extra flavor for a small price increase. Prices vary slightly by location. Check deals and menu prices here in-n-outprices.com

    Reply
← Older Comments

Submit a Comment

Your email address will not be published. Required fields are marked *