AraBERT : Pre-training BERT for Arabic Language Understanding

by | Feb 28, 2020 | NLP | 49,350 comments

Authors: Wissam Antoun, Fady Baly, Hazem Hajj

 

AraBERT is an Arabic pretrained language model based on Google’s BERT architecture. AraBERT uses the same BERT-Base config. More details are available in the AraBERT PAPER and in the AraBERT Meetup

There is two versions of the model AraBERTv0.1 and AraBERTv1, with the difference being that AraBERTv1 uses pre-segmented text where prefixes and suffixes were split using the Farasa Segmenter.

The model was trained on ~70M sentences or ~23GB of Arabic text with ~3B words.

Source Code Repository: https://github.com/aub-mind/arabert
Paper: https://www.aclweb.org/anthology/2020.osact-1.2.pdf

 

Results (Accuracy)

We evaluate both AraBERT models on different downstream tasks and compare it to mBERT, and other state of the art models (To the extent of our knowledge). The Tasks were Sentiment Analysis on 6 different datasets (HARD, ASTD-Balanced, ArsenTD-Lev, LABR, ArSaS), Named Entity Recognition with the ANERcorp, and Arabic Question Answering on Arabic-SQuAD and ARCD

Task prev. SOTA mBERT AraBERTv0.1 AraBERTv1
HARD 95.7 ElJundi et.al. 95.7 96.2 96.1
ASTD 86.5 ElJundi et.al. 80.1 92.2 92.6
ArsenTD-Lev 52.4 ElJundi et.al. 51 58.9 59.4
AJGT 93 Dahou et.al. 83.6 93.1 93.8
LABR 87.5 Dahou et.al. 83 85.9 86.7
ANERcorp 81.7 (BiLSTM-CRF) 78.4 84.2 81.9
ARCD mBERT EM:34.2 F1: 61.3 EM:51.14 F1:82.13 EM:54.84 F1: 82.15

Model Weights and Vocab Download

Models AraBERTv0.1 AraBERTv1
TensorFlow Drive Link Drive Link
PyTorch Drive_Link Drive_Link

You can find the PyTorch models in HuggingFace’s Transformer Library under the aubmindlab username

If you used this model please cite us as:

@inproceedings{antoun2020arabert,
  title={AraBERT: Transformer-based Model for Arabic Language Understanding},
  author={Antoun, Wissam and Baly, Fady and Hajj, Hazem},
  booktitle={LREC 2020 Workshop Language Resources and Evaluation Conference 11--16 May 2020},
  pages={9}
}

Acknowledgments

Thanks to TensorFlow Research Cloud (TFRC) for the free access to Cloud TPUs, couldn’t have done it without this program, and to the AUB MIND Lab Members for the continous support. Also thanks to Yakshof and Assafir for data and storage access. Another thanks for Habib Rahal (https://www.behance.net/rahalhabib), for putting a face to AraBERT.

Contacts

Wissam Antoun: Linkedin | Twitter | Github | wfa07@mail.aub.edu | wissam.antoun@gmail.com

Fady Baly: Linkedin | Twitter | Github | fgb06@mail.aub.edu | baly.fady@gmail.com

We are looking for sponsors to train BERT-Large and other Transformer models, the sponsor only needs to cover to data storage and compute cost of the generating the pretraining data

49,350 Comments

← Older Comments
Newer Comments →
  1. Crypto30x on February 24, 2025 at 4:48 pm

    It is perfect time to make some plans for the future and it is time to be happy. I’ve read this post and if I could I desire to suggest you some interesting things or suggestions. Perhaps you could write next articles referring to this article. I want to read more things about it! Crypto30x

    Reply
  2. what not to say to your wife on February 24, 2025 at 5:03 pm

    This article gives me the satifaction after reading it whole keep up the great things what not to say to your wife

    Reply
  3. vaporgenz on February 24, 2025 at 5:08 pm

    The IQOS Iluma in Dubai offers a cutting-edge, smoke-free alternative to traditional cigarettes. Featuring advanced heating technology, it delivers a smooth and flavorful tobacco experience without combustion, reducing harmful chemicals. Compact, discreet, and convenient, the IQOS Iluma provides a cleaner and healthier option for smokers in Dubai. Perfect for those seeking a modern, smoke-free lifestyle with minimal odor and maximum satisfaction.

    https://vaporgenz.com/product/new-iqos-iluma-one-kit/

    Reply
  4. amna seo on February 24, 2025 at 5:15 pm

    We are really grateful for your blog post. You will find a lot of approaches after visiting your post. Great work appoloiptv

    Reply
  5. Fatima Javaid on February 24, 2025 at 5:29 pm

    Alexistogel adalah salah satu platform permainan online yang semakin populer di kalangan pecinta togel. Dengan menyediakan berbagai jenis permainan dan pasaran togel yang lengkap, Alexistogel menjadi pilihan utama bagi para pemain yang mencari pengalaman bermain yang aman dan nyaman.

    Reply
  6. Terea Cat Santorini on February 24, 2025 at 5:40 pm

    This is genuinely well-researched content with excellent wording. I became so engaged with the material that I couldn’t stop reading. I’m truly impressed with your work and skill. Thank you! catamaran Santorini

    Reply
  7. Fatima Javaid on February 24, 2025 at 5:46 pm

    Slot games have become one of the most popular forms of entertainment in the online gaming industry, offering players an exciting and immersive experience. With a variety of themes, graphics, and bonus features, slot machines attract both beginners and seasoned players alike. The convenience of playing online slot games has made them even more accessible, allowing users to enjoy their favorite games anytime and anywhere.

    Reply
  8. Terea Cat Santorini on February 24, 2025 at 5:48 pm

    The content is remarkably well-researched and excellently written. I was so captivated by the material that I couldn’t wait to read more. I admire your work and skills. Thanks for sharing! Santorini sunset cruise

    Reply
  9. Fatima Javaid on February 24, 2025 at 6:31 pm

    เว็บปั้มไลค์เป็นเครื่องมือที่ช่วยให้เพจหรือโพสต์ของคุณได้รับความสนใจมากขึ้น และช่วยสร้างความน่าเชื่อถือได้อย่างรวดเร็ว อย่างไรก็ตาม การเลือกใช้บริการจากแหล่งที่มีความปลอดภัยและคุณภาพเป็นสิ่งสำคัญ เพื่อลดความเสี่ยงในการถูกจำกัดการเข้าถึงจากแพลตฟอร์มโซเชียลมีเดีย นอกจากนี้ การสร้างเนื้อหาที่มีคุณภาพและการสร้างปฏิสัมพันธ์กับผู้ติดตามก็เป็นสิ่งจำเป็นที่ช่วยให้บัญชีโซเชียลมีเดียของคุณเติบโตอย่างมั่นคงและมีประสิทธิภาพ

    Reply
  10. Fatima Javaid on February 24, 2025 at 6:38 pm

    เว็บปั้มไลค์เป็นเครื่องมือที่ช่วยให้ผู้ใช้สามารถเพิ่มยอดไลค์ให้กับโพสต์หรือเพจของตนเองได้โดยอัตโนมัติ ระบบของเว็บเหล่านี้สามารถเพิ่มไลค์ได้จากบัญชีผู้ใช้จริง หรือใช้เทคนิคอื่น ๆ เพื่อเพิ่มจำนวนไลค์อย่างรวดเร็ว ทั้งนี้ การทำงานของเว็บปั้มไลค์อาจแตกต่างกันไปตามแพลตฟอร์มที่ให้บริการ บางระบบอาจใช้เครือข่ายแลกเปลี่ยนไลค์จากผู้ใช้ที่ลงทะเบียน ในขณะที่บางระบบอาจใช้วิธีการอื่นเพื่อให้ได้ผลลัพธ์ที่รวดเร็วและมีประสิทธิภาพ

    Reply
  11. sunder translate hello in hindi on February 24, 2025 at 6:52 pm

    I have bookmarked your website because this site contains valuable information in it. I am really happy with articles quality and presentation. Thanks a lot for keeping great stuff. I am very much thankful for this site. sunder translate hello in hindi

    Reply
  12. vapor genz on February 24, 2025 at 6:52 pm

    The IQOS Iluma in Dubai offers a cutting-edge, smoke-free alternative to traditional cigarettes. Featuring advanced heating technology, it delivers a smooth and flavorful tobacco experience without combustion, reducing harmful chemicals. Compact, discreet, and convenient, the IQOS Iluma provides a cleaner and healthier option for smokers in Dubai. Perfect for those seeking a modern, smoke-free lifestyle with minimal odor and maximum satisfaction.

    https://vaporgenz.com/product/new-iqos-iluma-one-kit/

    Reply
  13. trainingsplan fussball on February 24, 2025 at 7:08 pm

    Fußball ist eine der beliebtesten Sportarten weltweit, und um auf dem Platz erfolgreich zu sein, ist ein strukturierter Trainingsplan unerlässlich. Ein gut durchdachter Trainingsplan hilft nicht nur dabei, die individuellen Fähigkeiten der Spieler zu verbessern, sondern stärkt auch das Teamplay und die taktische Ausrichtung. In diesem Artikel erfahren Sie, wie Sie einen effektiven trainingsplan fussball erstellen und welche Elemente dabei besonders wichtig sind. Bevor Sie mit der Erstellung eines Trainingsplans beginnen, ist es wichtig, klare Ziele zu definieren. Möchten Sie die Kondition der Spieler verbessern, die technischen Fähigkeiten schulen oder das taktische Verständnis des Teams stärken? Die Ziele bestimmen den Schwerpunkt des Trainings und helfen dabei, die richtigen Übungen auszuwählen.

    Reply
  14. Finance Tips on February 24, 2025 at 7:22 pm

    Really nice and interesting post. I was looking for this kind of information and enjoyed reading this one. Keep posting. Thanks for sharing. Finance Tips

    Reply
  15. wheon subway surfers for pc on February 24, 2025 at 7:45 pm

    This is my first time i visit here. I found so many entertaining stuff in your blog, especially its discussion. From the tons of comments on your articles, I guess I am not the only one having all the leisure here! Keep up the good work. I have been meaning to write something like this on my website and you have given me an idea. wheon subway surfers for pc

    Reply
  16. Wyatt Carter on February 24, 2025 at 8:30 pm

    Wow, What a Excellent post. I really found this to much informatics. It is what i was searching for.I would like to suggest you that please keep sharing such type of info.Thanks typescript key value pair types guide

    Reply
  17. Enzo on February 24, 2025 at 8:34 pm

    The embroidery industry is evolving with trends like AI-driven precision faster turnaround times and demand for personalized designs. Businesses and individuals seek high-quality digitized embroidery for jackets. If you’re wondering how to How to Get Jacket Embroidery Digitizing Services choose experts who offer accuracy affordability and quick delivery for the best results.

    Reply
  18. Amma on February 24, 2025 at 8:37 pm

    Forex white label software/ is a ready-made trading platform that allows entrepreneurs and financial institutions to launch their own branded forex brokerage without developing software from scratch.

    Reply
  19. Jenny Mod Unlocked on February 24, 2025 at 8:58 pm

    Jenny Mod Unlocked lets players enjoy the full version of the mod with no restrictions. Get Jenny Mod Unlocked today and take your game to the next level!

    Reply
  20. Kiera on February 24, 2025 at 9:50 pm

    Awesome post.

    my web site … 31231

    Reply
  21. ch on February 25, 2025 at 12:17 am

    Đừng bỏ lỡ cơ hội thắng lớn tại fb88wt.com! Đăng ký ngay để nhận nhiều ưu đãi cực khủng! 🔥 https://fb88wt.com/

    Reply
  22. sòng bạc trực tuyến on February 25, 2025 at 1:14 am

    The depth of research and clarity of presentation in your writing are remarkable. sòng bạc trực tuyến

    Reply
  24. Deanne on February 25, 2025 at 1:35 am

    A testnet faucet provides web3 developers with free tokens for deploying, testing, and optimizing smart contracts on test
    blockchains such as hemi sepolia eth faucet,
    Goerli, and Mumbai.

    Because smart contracts on public, mainnet blockchains like Ethereum and Polygon require gas fees
    to run smart contracts, testnets provide blockchain developers with a network
    that mirrors production blockchain environments without
    requiring gas fees that cost real money.

    Get Testnet Tokens!

    Reply
  25. DURAJOINT Chain Couplings 8022 on February 25, 2025 at 2:11 am

    Saya suka cara Anda menjelaskan topik yang rumit menjadi sesuatu yang mudah dimengerti. DURAJOINT Chain Couplings 8022

    Reply
  26. DURAJOINT Chain Couplings 8022 on February 25, 2025 at 2:19 am

    Saya suka cara Anda menjelaskan topik yang rumit menjadi sesuatu yang mudah dimengerti. DURAJOINT Chain Couplings 8022

    Reply
  27. DURALINK Roller Chains RS 50-1 on February 25, 2025 at 2:28 am

    Anda benar-benar tahu bagaimana menyampaikan informasi dengan cara yang menarik! DURALINK Roller Chains RS 50-1

    Reply
  30. DURALINK Roller Chains RS 60-1 on February 25, 2025 at 2:42 am

    Saya setuju dengan poin ini, dan sangat menghargai cara Anda menyajikannya. DURALINK Roller Chains RS 60-1

    Reply
  31. DURALINK Roller Chains RS 80-1 on February 25, 2025 at 2:47 am

    Terima kasih sudah mengulas topik ini secara mendetail, sangat bermanfaat! DURALINK Roller Chains RS 80-1

    Reply
  32. DURALINK Roller Chains RS 80-1 on February 25, 2025 at 2:49 am

    Terima kasih sudah mengulas topik ini secara mendetail, sangat bermanfaat! DURALINK Roller Chains RS 80-1

    Reply
  33. Heavy Duty Roller Standard on February 25, 2025 at 2:51 am

    Saya selalu menantikan posting baru dari blog ini. Terima kasih atas konten yang konsisten dan berkualitas. Heavy Duty Roller Standard

    Reply
  34. u888 on February 25, 2025 at 2:51 am

    Nha cai u888.com khong chi noi bat voi su da dang ve san pham ca cuoc ma con hap dan nguoi choi boi nhung chuong trinh khuyen mai cuc ky phong phu va thu vi.

    Reply
  35. Heavy Duty Roller Standard on February 25, 2025 at 2:58 am

    Saya selalu menantikan posting baru dari blog ini. Terima kasih atas konten yang konsisten dan berkualitas. Heavy Duty Roller Standard

    Reply
  37. Wire Screen Ayakan/Mesh Besar on February 25, 2025 at 3:12 am

    Artikel yang sangat informatif! Saya belajar banyak hal baru hari ini. Wire Screen Ayakan/Mesh Besar

    Reply
  39. Wire Screen Ayakan/Mesh Besar on February 25, 2025 at 3:19 am

    Artikel yang sangat informatif! Saya belajar banyak hal baru hari ini. Wire Screen Ayakan/Mesh Besar

    Reply
  46. abc8 on February 25, 2025 at 3:46 am

    nhà cái abc8 chao mung thanh vien moi, sieu tien thuong nap dau len toi 888k.

    Reply
  47. 58Win on February 25, 2025 at 3:46 am

    https://58win.bond/ là nền tảng cá cược trực tuyến uy tín hàng đầu. Chào đón với tỷ lệ thưởng hấp dẫn, dịch vụ giao dịch nhanh chóng và nhiều chương trình khuyến mãi đặc biệt.

    Reply
  48. Geneva on February 25, 2025 at 3:46 am

    Your means of describing everything in this article
    is in fact good, every one can simply understand it, Thanks
    a lot.

    my web site free crazy games

    Reply
  50. Naomi on February 25, 2025 at 3:47 am

    When I initially commented I clicked the “Notify me when new comments are added” checkbox and now each time a comment is added I
    get several e-mails with the same comment. Is there any way you can remove me from that service?
    Cheers!

    Here is my blog post; canadian pharmacy

    Reply
← Older Comments
Newer Comments →

Submit a Comment

Your email address will not be published. Required fields are marked *