AraBERT : Pre-training BERT for Arabic Language Understanding

by | Feb 28, 2020 | NLP | 48,502 comments

Authors: Wissam Antoun, Fady Baly, Hazem Hajj

 

AraBERT is an Arabic pretrained language model based on Google’s BERT architecture. AraBERT uses the same BERT-Base config. More details are available in the AraBERT PAPER and in the AraBERT Meetup

There is two versions of the model AraBERTv0.1 and AraBERTv1, with the difference being that AraBERTv1 uses pre-segmented text where prefixes and suffixes were split using the Farasa Segmenter.

The model was trained on ~70M sentences or ~23GB of Arabic text with ~3B words.

Source Code Repository: https://github.com/aub-mind/arabert
Paper: https://www.aclweb.org/anthology/2020.osact-1.2.pdf

 

Results (Accuracy)

We evaluate both AraBERT models on different downstream tasks and compare it to mBERT, and other state of the art models (To the extent of our knowledge). The Tasks were Sentiment Analysis on 6 different datasets (HARD, ASTD-Balanced, ArsenTD-Lev, LABR, ArSaS), Named Entity Recognition with the ANERcorp, and Arabic Question Answering on Arabic-SQuAD and ARCD

Task prev. SOTA mBERT AraBERTv0.1 AraBERTv1
HARD 95.7 ElJundi et.al. 95.7 96.2 96.1
ASTD 86.5 ElJundi et.al. 80.1 92.2 92.6
ArsenTD-Lev 52.4 ElJundi et.al. 51 58.9 59.4
AJGT 93 Dahou et.al. 83.6 93.1 93.8
LABR 87.5 Dahou et.al. 83 85.9 86.7
ANERcorp 81.7 (BiLSTM-CRF) 78.4 84.2 81.9
ARCD mBERT EM:34.2 F1: 61.3 EM:51.14 F1:82.13 EM:54.84 F1: 82.15

Model Weights and Vocab Download

Models AraBERTv0.1 AraBERTv1
TensorFlow Drive Link Drive Link
PyTorch Drive_Link Drive_Link

You can find the PyTorch models in HuggingFace’s Transformer Library under the aubmindlab username

If you used this model please cite us as:

@inproceedings{antoun2020arabert,
  title={AraBERT: Transformer-based Model for Arabic Language Understanding},
  author={Antoun, Wissam and Baly, Fady and Hajj, Hazem},
  booktitle={LREC 2020 Workshop Language Resources and Evaluation Conference 11--16 May 2020},
  pages={9}
}

Acknowledgments

Thanks to TensorFlow Research Cloud (TFRC) for the free access to Cloud TPUs, couldn’t have done it without this program, and to the AUB MIND Lab Members for the continous support. Also thanks to Yakshof and Assafir for data and storage access. Another thanks for Habib Rahal (https://www.behance.net/rahalhabib), for putting a face to AraBERT.

Contacts

Wissam Antoun: Linkedin | Twitter | Github | wfa07@mail.aub.edu | wissam.antoun@gmail.com

Fady Baly: Linkedin | Twitter | Github | fgb06@mail.aub.edu | baly.fady@gmail.com

We are looking for sponsors to train BERT-Large and other Transformer models, the sponsor only needs to cover to data storage and compute cost of the generating the pretraining data

48,502 Comments

← Older Comments
Newer Comments →
  1. 농구중계 on March 15, 2025 at 6:38 am

    농구중계를 찾고 있다면 복구티비가 최고의 선택입니다. NBA, 유로리그 등 다양한 농구 경기를 실시간으로 시청할 수 있으며, 무료로 고화질 중계를 제공합니다.

    Reply
  2. Willaum on March 15, 2025 at 6:48 am

    Visit Herosturf.net for a wide range of high-quality lawn care solutions. Whether you’re dealing with weeds, pests, or poor grass health, Herosturf.net offers eco-friendly products to meet your needs. The website features detailed guides and expert advice to help you achieve a vibrant, healthy lawn. From natural treatments to innovative tools, Herosturf.net is your trusted source for effective lawn care. Explore Herosturf.net today and keep your turf in perfect condition all year long!

    Reply
  3. Baby James on March 15, 2025 at 6:49 am

    This is a good post. This post gives truly quality information. I’m definitely going to look into it. Really very useful tips are provided here. Thank you so much. Keep up the good works. markwalstrom.com

    Reply
  5. Daiane Bobbin Lace on March 15, 2025 at 6:53 am

    Scopri l’arte del merletto a tombolo con Daiane Bobbin Lace, il punto di riferimento per gli appassionati di lavorazione artigianale. Offriamo una vasta selezione di schemi per merletti a tombolo, perfetti per creare design unici e dettagliati con un tocco di eleganza.

    Che tu sia un principiante o un esperto, troverai modelli ispiratori, consigli per l’uso dei filati e idee creative per personalizzare le tue creazioni con colori e forme diverse. Ogni schema è pensato per stimolare la fantasia e permetterti di dare vita a meravigliosi lavori fatti a mano.

    Esplora Daiane Bobbin Lace oggi e lasciati ispirare dalla bellezza del merletto artigianale!

    Reply
  6. Islandpaws Com on March 15, 2025 at 6:56 am

    https://islandpaws.com/
    situs slot gacor Alexistogel selalu menjadi pilihan pemain slot karena memiliki banyak koleksi game slot gacor dan sudah terbukti gampang menang

    Reply
  8. Fatima Javaid on March 15, 2025 at 7:01 am

    Kebuntoto adalah salah satu platform yang banyak dicari oleh pecinta permainan online karena menawarkan berbagai jenis hiburan menarik. Dengan sistem yang mudah digunakan dan layanan yang responsif, kebuntoto menjadi pilihan yang tepat bagi pengguna yang menginginkan pengalaman bermain yang nyaman dan aman. Selain itu, keamanan transaksi dan berbagai bonus yang ditawarkan membuat platform ini semakin diminati oleh banyak orang.

    Reply
  10. aa on March 15, 2025 at 7:08 am

    Wow! Such an amazing and helpful post this is. I really really love it. It’s so good and so awesome. I am just amazed. I hope that you continue to do your work like this in the future also GARASI189

    Reply
  11. Islandpaws Com on March 15, 2025 at 7:09 am

    https://islandpaws.com/
    situs slot gacor Alexistogel selalu menjadi pilihan pemain slot karena memiliki banyak koleksi game slot gacor dan sudah terbukti gampang menang

    Reply
  12. HAMMAD on March 15, 2025 at 7:09 am

    I favor an entire couple of merchandise, Dreamed about seriously experienced, I require specifics. during this, taking into account that must be to some extent really good., With thanks considerably with regards to expressing. Quietum Plus Review

    Reply
  13. mientrungrealty on March 15, 2025 at 7:10 am

    “Dự án căn hộ căn hộ ThaiGrand Casa Riva Đà Nẵng là cơ hội đầu tư tuyệt vời cho những ai tìm kiếm một kênh sinh lời ổn định. Với vị trí đắc địa tại trung tâm Đà Nẵng, căn hộ mặt tiền sông Hàn và các tiện ích quan trọng, căn hộ tại Đà Nẵng này không chỉ mang lại không gian sống sang trọng, hiện đại mà còn tiềm năng tăng giá mạnh mẽ trong tương lai. Thiết kế tinh tế, tiện nghi đầy đủ, và môi trường sống thoải mái sẽ là lựa chọn lý tưởng cho cả nhà đầu tư và cư dân. Hãy đầu tư ngay hôm nay để nắm bắt cơ hội vàng!
    Website: https://mientrungrealty.com/
    ĐC: 709, Nguyễn Văn Linh, Hải Châu, Đà Nẵng
    SĐT: 0944 811 143
    hastags: #mientrung #mientrungrealty #BấtĐộngSảnĐàNẵng #CănHộĐàNẵng #ChungCưĐàNẵng #MiềnTrungRealty #ĐầuTưBấtĐộngSản #CănHộMới #ĐầuTưĐàNẵng #CănHộSangTrọng #ChungCưViewBiển”

    Reply
  14. khan zada on March 15, 2025 at 7:10 am

    The platform’s real-time customer support via phone, chat, and email ensures users receive prompt assistance. loto188

    Reply
  15. khan zada on March 15, 2025 at 7:10 am

    The platform’s well-organized FAQ section helps users find answers to common questions without needing support assistance. loto188

    Reply
  16. AALIYAN KHAN on March 15, 2025 at 7:12 am

    Loto188’s advanced reporting tools help users analyze their performance and improve their strategies. loto188

    Reply
  17. jack on March 15, 2025 at 7:15 am

    White wine is highly known for getting very sensual and classy alcoholic drink. It is mainly accessible in yellow or golden color. White wine is widely appreciated all across the continents because of its delicious flavour. One can anytime pair white wine with white meat so as to produce a light meal looks delicious. But, to support all you starters for better wine merging we’ve organized this white wine chef for starters. olxtoto login toge

    Reply
  18. DURAJOINT FCL Couplings 250 on March 15, 2025 at 7:16 am

    Saya sangat terkesan dengan kedalaman pengetahuan Anda tentang topik ini. Sangat mengesankan! DURAJOINT FCL Couplings 250

    Reply
  19. mwe on March 15, 2025 at 7:19 am

    Hello I am so delighted I located your blog, I really located you by mistake, while I was watching on google for something else, Anyways I am here now and could just like to say thank for a tremendous post and a all round entertaining website. Please do keep up the great work. alexistoto

    Reply
  20. MS SUFYAN on March 15, 2025 at 7:19 am

    Tersedia fitur live chat untuk komunikasi langsung dengan tim support Ultra88. Link Ultra88 Resmi

    Reply
  21. DURAJOINT FCL Couplings 250 on March 15, 2025 at 7:22 am

    Saya sangat terkesan dengan kedalaman pengetahuan Anda tentang topik ini. Sangat mengesankan! DURAJOINT FCL Couplings 250

    Reply
  22. Taha shaikh on March 15, 2025 at 7:24 am

    I realize of their line. After i understand Individuals match some kind of nearly all think it is difficult in order to situated which statement. After i coveted by types exercise. https://summersmiledental.com/

    Reply
  23. mwe on March 15, 2025 at 7:24 am

    I am impressed. I don’t think Ive met anyone who knows as much about this subject as you do. You are truly well informed and very intelligent. You wrote something that people could understand and made the subject intriguing for everyone. Really, great blog you have got here. alexistogel

    Reply
  24. 스포츠중계 on March 15, 2025 at 7:25 am

    스포츠중계는 팬들이 실시간으로 경기를 시청할 수 있도록 돕는 필수 서비스입니다. 복구티비는 프리미어리그, 라리가, 세리에 A, 분데스리가 등 주요 리그뿐만 아니라 챔피언스리그, 유로파리그 같은 대회도 제공합니다. 어디서든 무료로 고화질 스포츠 중계를 즐겨보세요!

    Reply
  25. nexus market link on March 15, 2025 at 7:26 am

    Hello There. I found your blog using msn. This is a really well written article. I’ll make sure to bookmark it and return to read more of your useful information. Thanks for the post. I will definitely comeback. nexus market link

    Reply
  26. Web Solution Centre on March 15, 2025 at 7:28 am

    Recognized as the Best Website Designing Company In Delhi, Web Solution Centre provides cutting-edge, SEO-friendly websites. Our designs ensure exceptional user experience, mobile responsiveness, and innovative features to enhance your online presence effectively.

    Reply
  27. mwe on March 15, 2025 at 7:33 am

    Wow, cool post. I’d like to write like this too – taking time and real hard work to make a great article… but I put things off too much and never seem to get started. Thanks though. situs slot gacor

    Reply
  28. Fatima Javaid on March 15, 2025 at 7:34 am

    Gambliing enthusiasts are always on the lookout for the best gaming experience, and one of the most popular choices today is gacor slot. This term refers to slot games that are known for their high payout rates and frequent wins, making them a favorite among players.

    Reply
  29. Snow rider 3d on March 15, 2025 at 7:34 am

    This is such a great resource that you are providing and you give it away for free. I love seeing websites that understand the value of providing a quality resource for free. It’s the old what goes around comes around routine. Snow rider 3d

    Reply
  30. HUZAIFA on March 15, 2025 at 7:35 am

    Hey there! I’m at work browsing your blog from my new iphone! Just wanted to say I love reading your blog and look forward to all your posts! Carry on the outstanding work! loto188

    Reply
  31. Rwin on March 15, 2025 at 7:35 am

    Rwin – Nhà cái cá cược trực tuyến hàng đầu với đa dạng trò chơi từ thể thao, ca sino, đến game slot. Tham gia Rwin ngay để nhận thưởng hấp dẫn và trải nghiệm thế giới cá cược chuyên nghiệp.

    Reply
  33. mwe on March 15, 2025 at 7:37 am

    You make so many great points here that I read your article a couple of times. Your views are in accordance with my own for the most part. This is great content for your readers. togel online

    Reply
  35. aliuamir aliuad on March 15, 2025 at 7:44 am

    TESLATOTO situs slot online terpercaya yang mempunya scatter hitam Mahjong Ways 2 viral. Situs ini menawarkan berbagai keunggulan, termasuk permainan slot scatter hitam dengan hadiah besar, keamanan, layanan pelanggan 24 jam, dan proses transaksi cepat. juga memberikan panduan bermain dan tips untuk pemain baru TESLATOTO

    Reply
  36. DURAJOINT Chain Couplings 4012 on March 15, 2025 at 7:49 am

    Terima kasih atas penjelasan yang jelas dan terperinci. Saya sangat menghargainya. DURAJOINT Chain Couplings 4012

    Reply
  37. Baby James on March 15, 2025 at 7:49 am

    Nice post! This is a very nice blog that I will definitively come back to more times this year! Thanks for informative post. http://www.icsmagazine.com

    Reply
  38. DURAJOINT Chain Couplings 4012 on March 15, 2025 at 7:53 am

    Terima kasih atas penjelasan yang jelas dan terperinci. Saya sangat menghargainya. DURAJOINT Chain Couplings 4012

    Reply
  39. 온라인 카지노 on March 15, 2025 at 7:57 am

    온라인 카지노는 컴퓨터, 스마트폰, 태블릿 등 다양한 디지털 기기를 통해 언제 어디서나 접속할 수 있는 가상 게임 플랫폼입니다. 슬롯 머신, 포커, 블랙잭, 룰렛, 바카라 등 다양한 게임을 제공하며, 실시간 딜러와의 상호작용이 가능한 라이브 카지노 기능도 지원합니다.

    Reply
  40. Reiki on March 15, 2025 at 7:57 am

    The law of attraction has completely changed my perspective on life! I’ve learned that manifestation works when I stay aligned with my goals and emotions. I use visualization to mentally experience success and say positive affirmations to reinforce my mindset. When I feel stuck, I turn to healing techniques like working on my Sacral Chakra or practicing Reiki. This keeps my alignment and vibrations high, allowing me to attract my desires more easily. Subliminal messages have also helped shift my subconscious beliefs, making manifestation even more powerful! Reiki

    Reply
  41. hubet on March 15, 2025 at 7:57 am

    Voi muc thuong len den 50K cho lan nap tien dau tien tai Hubet com, ban se co them von de trai nghiem cac tro choi hap dan.

    Reply
  42. عوامل مؤثر بر رتبه‌بندی سایت on March 15, 2025 at 7:59 am

    Thanks for taking the time to discuss this, I feel strongly about it and love learning more on this topic. If possible, as you gain expertise, would you mind updating your blog with extra information? It is extremely helpful for me. عوامل مؤثر بر رتبه‌بندی سایت

    Reply
  43. SAM86 on March 15, 2025 at 8:00 am

    SAM86 là một trong những cổng game bài đổi thưởng trực tuyến uy tín và được nhiều người chơi yêu thích nhất hiện nay. Ra mắt năm 2021, cổng game nhanh chóng thu hút người chơi nhờ kho game đa dạng và giao diện đẹp mắt, âm thanh sống động, tạo cảm giác chân thực.

    Reply
  45. IndoBet on March 15, 2025 at 8:05 am

    Jika Anda mencari platform yang memberikan panduan terpercaya tentang perjudian online dan tips untuk meningkatkan peluang menang, IndoBet adalah sumber yang sangat berguna. Banyak artikel yang mendalam dan up-to-date yang bisa membantu pemain pemula maupun berpengalaman.

    Reply
  47. SlotVIP on March 15, 2025 at 8:06 am

    SlotVIP menawarkan pengalaman bermain slot yang sangat menarik dengan berbagai pilihan game inovatif. Selain itu, mereka juga memberikan panduan yang bermanfaat bagi pemain untuk memahami cara bermain dan meningkatkan peluang kemenangan, membuatnya menjadi pilihan yang tepat bagi para penggemar slot.

    Reply
  48. DURAJOINT Chain Couplings 4016 on March 15, 2025 at 8:12 am

    Artikel yang sangat informatif dan relevan! Saya merasa lebih terinformasi sekarang. DURAJOINT Chain Couplings 4016

    Reply
  49. aa on March 15, 2025 at 8:12 am

    I admire this article for the well-researched content and excellent wording. I got so involved in this material that I couldn’t stop reading. I am impressed with your work and skill. Thank you so much. mommycallblog.com

    Reply
  50. mwe on March 15, 2025 at 8:13 am

    I recently came across your blog and have been reading along. I thought I would leave my first comment. I don’t know what to say except that I have enjoyed reading. Nice blog. I will keep visiting this blog very often. rtp slot gacor

    Reply
← Older Comments
Newer Comments →

Submit a Comment

Your email address will not be published. Required fields are marked *