AraBERT : Pre-training BERT for Arabic Language Understanding

by | Feb 28, 2020 | NLP | 49,403 comments

Authors: Wissam Antoun, Fady Baly, Hazem Hajj

 

AraBERT is an Arabic pretrained language model based on Google’s BERT architecture. AraBERT uses the same BERT-Base config. More details are available in the AraBERT PAPER and in the AraBERT Meetup

There is two versions of the model AraBERTv0.1 and AraBERTv1, with the difference being that AraBERTv1 uses pre-segmented text where prefixes and suffixes were split using the Farasa Segmenter.

The model was trained on ~70M sentences or ~23GB of Arabic text with ~3B words.

Source Code Repository: https://github.com/aub-mind/arabert
Paper: https://www.aclweb.org/anthology/2020.osact-1.2.pdf

 

Results (Accuracy)

We evaluate both AraBERT models on different downstream tasks and compare it to mBERT, and other state of the art models (To the extent of our knowledge). The Tasks were Sentiment Analysis on 6 different datasets (HARD, ASTD-Balanced, ArsenTD-Lev, LABR, ArSaS), Named Entity Recognition with the ANERcorp, and Arabic Question Answering on Arabic-SQuAD and ARCD

Task prev. SOTA mBERT AraBERTv0.1 AraBERTv1
HARD 95.7 ElJundi et.al. 95.7 96.2 96.1
ASTD 86.5 ElJundi et.al. 80.1 92.2 92.6
ArsenTD-Lev 52.4 ElJundi et.al. 51 58.9 59.4
AJGT 93 Dahou et.al. 83.6 93.1 93.8
LABR 87.5 Dahou et.al. 83 85.9 86.7
ANERcorp 81.7 (BiLSTM-CRF) 78.4 84.2 81.9
ARCD mBERT EM:34.2 F1: 61.3 EM:51.14 F1:82.13 EM:54.84 F1: 82.15

Model Weights and Vocab Download

Models AraBERTv0.1 AraBERTv1
TensorFlow Drive Link Drive Link
PyTorch Drive_Link Drive_Link

You can find the PyTorch models in HuggingFace’s Transformer Library under the aubmindlab username

If you used this model please cite us as:

@inproceedings{antoun2020arabert,
  title={AraBERT: Transformer-based Model for Arabic Language Understanding},
  author={Antoun, Wissam and Baly, Fady and Hajj, Hazem},
  booktitle={LREC 2020 Workshop Language Resources and Evaluation Conference 11--16 May 2020},
  pages={9}
}

Acknowledgments

Thanks to TensorFlow Research Cloud (TFRC) for the free access to Cloud TPUs, couldn’t have done it without this program, and to the AUB MIND Lab Members for the continous support. Also thanks to Yakshof and Assafir for data and storage access. Another thanks for Habib Rahal (https://www.behance.net/rahalhabib), for putting a face to AraBERT.

Contacts

Wissam Antoun: Linkedin | Twitter | Github | wfa07@mail.aub.edu | wissam.antoun@gmail.com

Fady Baly: Linkedin | Twitter | Github | fgb06@mail.aub.edu | baly.fady@gmail.com

We are looking for sponsors to train BERT-Large and other Transformer models, the sponsor only needs to cover to data storage and compute cost of the generating the pretraining data

49,403 Comments

← Older Comments
Newer Comments →
  1. TRP Roller Chains RS 35-1 on March 1, 2025 at 9:16 am

    Saya suka cara Anda menyajikan fakta dan argumen dalam artikel ini. Sangat meyakinkan! TRP Roller Chains RS 35-1

    Reply
  2. fastupnow on March 1, 2025 at 9:18 am

    I have bookmarked your website because this site contains valuable information in it. I am really happy with articles quality and presentation. Thanks a lot for keeping great stuff. I am very much thankful for this site. fastupnow

    Reply
  3. racing games on March 1, 2025 at 9:19 am

    cheers, I thoroughly enjoyed scaning your post. I really appreciate your wonderful know-how and the time you put into educating the rest of us. racing games

    Reply
  5. rs 119 wireless earbuds for gaming thespark shop on March 1, 2025 at 9:24 am

    Thanks for the nice blog. It was very useful for me. I m happy I found this blog. Thank you for sharing with us,I too always learn something new from your post. rs 119 wireless earbuds for gaming thespark shop

    Reply
  7. mudstone1869 on March 1, 2025 at 9:26 am

    Slot Thailand merupakan agen slot yang di dukung dengan server thailand yang sudah tersertifikasi secara global Server Thailand

    Reply
  8. Chasunah LTD on March 1, 2025 at 9:29 am

    Howdy I’m thus excited I found your own web page, I really identified you accidentally, as i had been studying about Yahoo for something different, Anyhow I will be right here now and would likely prefer to point out cheers for any amazing publish plus a all-round thrilling blog (I also adore the actual theme/design), I can’t have enough time in order to browse everything in the moment but I have bookmarked that as well as added your own RSS feeds, then when We have moment I am to read far more, Please keep up the fantastic work. Chasunah LTD

    Reply
  9. luxury villas provence le collectionist on March 1, 2025 at 9:35 am

    A very awesome blog post. We are really grateful for your blog post. combat, law enforcement You will find a lot of approaches after visiting your post. I was exactly searching for. Thanks for such post and please keep it up. Great work. luxury villas provence le collectionist

    Reply
  11. slotxoauto on March 1, 2025 at 9:35 am

    XOSLOTZ คือเว็บเกมสล็อตออนไลน์ที่มีเกมให้เลือกมากกว่า 1,000 เกม กราฟฟิกสวยงาม เอฟเฟกต์ตระการตา สมัครสมาชิกได้ตลอด 24 ชั่วโมง พร้อมรับเครดิตฟรีและโบนัสพิเศษทุกวัน. https://slotxoauto.in.th/

    Reply
  12. ベイプ on March 1, 2025 at 9:36 am

    Nice post. I learn something more difficult on various blogs everyday. It will always be stimulating to read content off their writers and practice a little something there. I’d opt to use some with the content in my small weblog whether you do not mind. Natually I’ll supply you with a link on your web blog. Many thanks sharing. ベイプ

    Reply
  13. qwddwqwqd on March 1, 2025 at 9:39 am

    I am deeply grateful for the valuable assistance of mbiserviceslimited.org in helping me get my money back from this crypto scam company. Their guidance and … victims of digital cryptocurrency scams

    Reply
  14. giftcardmall1634 on March 1, 2025 at 9:41 am

    MyGift Mall has the Gift for all. Introducing our new collection at the Gift Card Mall. So go ahead, go wild, go Check Your Balance™. giftcardmall balance

    Reply
  15. Dry Hearing aids on March 1, 2025 at 9:43 am

    very good submit, i definitely love this web site, keep on it} Dry Hearing aids

    Reply
  16. wellhealthorganic surgery for wrist and ankle injuries on March 1, 2025 at 9:43 am

    Thanks for the nice blog. It was very useful for me. I m happy I found this blog. Thank you for sharing with us,I too always learn something new from your post. wellhealthorganic surgery for wrist and ankle injuries

    Reply
  17. netwyman blogs on March 1, 2025 at 9:46 am

    I have bookmarked your website because this site contains valuable information in it. I am really happy with articles quality and presentation. Thanks a lot for keeping great stuff. I am very much thankful for this site. netwyman blogs

    Reply
  18. ấu dâm on March 1, 2025 at 9:47 am

    Những trẻ em là nạn nhân của ấu dâm phải chịu những tổn hại nặng nề về sức khỏe và tinh thần, hãy trang bị cho trẻ những kiến thức cần thiết để phòng tránh và bảo vệ bản thân khỏi tội phạm ấu dâm.#audam #hanhviaudam
     

    Reply
  19. iptv reseller on March 1, 2025 at 9:49 am

    Thanks for taking the time to discuss this, I feel strongly about it and really like learning more on this matter. If possible, as you gain expertise, would you mind updating your blog page with more data? It’s extremely helpful for me. iptv reseller

    Reply
  20. the spark shop men winter jacket sportswear gym fitness on March 1, 2025 at 9:56 am

    Thanks for the nice blog. It was very useful for me. I m happy I found this blog. Thank you for sharing with us,I too always learn something new from your post. the spark shop men winter jacket sportswear gym fitness

    Reply
  21. agen togel on March 1, 2025 at 9:56 am

    Usually I do not read post on blogs, but I wish to say that this write-up very pressured me to take a look at and do it! Your writing style has been amazed me. Thanks, very nice post. agen togel

    Reply
  22. Mindfulnesscontent Com on March 1, 2025 at 9:59 am

    MindfulnessContent.com is the ultimate resource hub for mindfulness professionals, coaches, and meditation teachers seeking high-quality, done-for-you content. Whether you need guided meditation scripts, mindfulness infographics, self-help planners, or hypnosis scripts, our premium content is designed to help you grow your business, engage your audience, and create impactful experiences for your clients.

    From 600+ guided meditation scripts to ready-to-use self-development infographics, we provide everything you need to save time and deliver value effortlessly. Perfect for use in coaching programs, social media, online courses, and membership sites, MindfulnessContent.com empowers you to focus on what truly matters—helping your audience achieve inner peace, personal growth, and transformation.

    Visit MindfulnessContent.com and unlock the tools to elevate your mindfulness practice today!

    Reply
  23. procurementnation .com shipping on March 1, 2025 at 9:59 am

    I have bookmarked your website because this site contains valuable information in it. I am really happy with articles quality and presentation. Thanks a lot for keeping great stuff. I am very much thankful for this site. procurementnation .com shipping

    Reply
  24. Online banking platform on March 1, 2025 at 10:04 am

    you’re in reality a just right webmaster. The website loading velocity is incredible. It kind of feels that you are doing any distinctive trick. In addition, The contents are masterwork. you have performed a great job in this subject! Online banking platform

    Reply
  25. Toilet Plumber Mount Gravatt on March 1, 2025 at 10:05 am

    Looking for a reliable Toilet Plumber Mount Gravatt? Top Tier Plumbing offers expert plumbing services with a focus on quality and efficiency. Our experienced team provides prompt, professional solutions for all your toilet plumbing needs, ensuring lasting results and peace of mind. Trust us for reliable service every time.

    Reply
  26. crypto30x.com avalanche on March 1, 2025 at 10:05 am

    I have bookmarked your website because this site contains valuable information in it. I am really happy with articles quality and presentation. Thanks a lot for keeping great stuff. I am very much thankful for this site. crypto30x.com avalanche

    Reply
  27. which greblovz2004 should i start with on March 1, 2025 at 10:05 am

    Thanks for the nice blog. It was very useful for me. I m happy I found this blog. Thank you for sharing with us,I too always learn something new from your post. which greblovz2004 should i start with

    Reply
  28. mwe on March 1, 2025 at 10:10 am

    I was reading some of your content on this website and I conceive this internet site is really informative ! Keep on putting up. mawartoto

    Reply
  29. aaliyan on March 1, 2025 at 10:11 am

    GA179 has made my gaming life so much easier. The cloud gaming feature means I don’t need to worry about expensive hardware, and I can still enjoy top-tier games. The platform is intuitive, and the social features are perfect for connecting with other gamers. I definitely recommend it for anyone looking for a seamless gaming experience. GA179

    Reply
  30. aaliyan on March 1, 2025 at 10:11 am

    I’ve been using GA179 for a couple of months now, and it’s the best gaming platform I’ve tried. The cloud gaming option is incredibly convenient, and the performance is always top-notch. I also love the social aspect, which makes it easy to connect with other players and form teams. It’s definitely worth the investment. GA179

    Reply
  31. custom software development on March 1, 2025 at 10:13 am

    We specialize in custom software development, crafting tailored digital platforms, applications, and portals that empower enterprises. Our expertise ensures seamless innovation, helping you lead with scalable, efficient, and future-ready solutions.

    Reply
  32. ecryptobit.com tokens on March 1, 2025 at 10:13 am

    I have bookmarked your website because this site contains valuable information in it. I am really happy with articles quality and presentation. Thanks a lot for keeping great stuff. I am very much thankful for this site. ecryptobit.com tokens

    Reply
  33. aaliyan on March 1, 2025 at 10:14 am

    I’ve been using GA179 for a while now, and I’m never disappointed. The game library is diverse and always up to date, and the cloud gaming feature lets me play without needing to upgrade my hardware. The community features make it easy to connect with others, and the performance is always top-notch. GA179

    Reply
  34. Free consultation with RCIC on March 1, 2025 at 10:15 am

    I not to mention my friends were found to be going through the excellent tips from your web page and so all of a sudden came up with an awful feeling I had not expressed respect to the web blog owner for those strategies. Those boys had been so passionate to see all of them and now have unquestionably been making the most of them. Many thanks for truly being really accommodating as well as for deciding on these kinds of nice tips millions of individuals are really desirous to learn about. Our own sincere regret for not expressing gratitude to you sooner. click here – alternative medicine institute Free consultation with RCIC

    Reply
  35. crypto30x.com tnt on March 1, 2025 at 10:16 am

    Thanks for the nice blog. It was very useful for me. I m happy I found this blog. Thank you for sharing with us,I too always learn something new from your post. crypto30x.com tnt

    Reply
  36. ecryptobit.com tokens on March 1, 2025 at 10:17 am

    I recently came across your article and have been reading along. I want to express my admiration of your writing skill and ability to make readers read from the beginning to the end. I would like to read newer posts and to share my thoughts with you. ecryptobit.com tokens

    Reply
  37. Mindfulnesscontent Com on March 1, 2025 at 10:20 am

    MindfulnessContent.com is the ultimate resource hub for mindfulness professionals, coaches, and meditation teachers seeking high-quality, done-for-you content. Whether you need guided meditation scripts, mindfulness infographics, self-help planners, or hypnosis scripts, our premium content is designed to help you grow your business, engage your audience, and create impactful experiences for your clients.

    From 600+ guided meditation scripts to ready-to-use self-development infographics, we provide everything you need to save time and deliver value effortlessly. Perfect for use in coaching programs, social media, online courses, and membership sites, MindfulnessContent.com empowers you to focus on what truly matters—helping your audience achieve inner peace, personal growth, and transformation.

    Visit MindfulnessContent.com and unlock the tools to elevate your mindfulness practice today!

    Reply
  38. mwe on March 1, 2025 at 10:20 am

    I am impressed. I don’t think Ive met anyone who knows as much about this subject as you do. You are truly well informed and very intelligent. You wrote something that people could understand and made the subject intriguing for everyone. Really, great blog you have got here. mawartoto

    Reply
  39. valerie kratochvil obituary smithtown on March 1, 2025 at 10:22 am

    I have bookmarked your website because this site contains valuable information in it. I am really happy with articles quality and presentation. Thanks a lot for keeping great stuff. I am very much thankful for this site. valerie kratochvil obituary smithtown

    Reply
  40. https://snorkus.co.uk/ on March 1, 2025 at 10:22 am

    KUBET – https://snorkus.co.uk/ không chỉ nổi bật với tỷ lệ cược hấp dẫn mà còn thu hút người chơi nhờ các chương trình khuyến mãi phong phú như thưởng chào mừng, hoàn tiền cược và ưu đãi đặc biệt cho thành viên VIP. Các sự kiện ưu đãi diễn ra liên tục, mang đến nhiều cơ hội nhận thưởng giá trị.

    Reply
  41. bolly44u on March 1, 2025 at 10:25 am

    Thanks for the nice blog. It was very useful for me. I m happy I found this blog. Thank you for sharing with us,I too always learn something new from your post. bolly44u

    Reply
  42. taunting quotes in marathi on March 1, 2025 at 10:35 am

    I have bookmarked your website because this site contains valuable information in it. I am really happy with articles quality and presentation. Thanks a lot for keeping great stuff. I am very much thankful for this site. taunting quotes in marathi

    Reply
  43. servicii foto și video pentru nunta ta on March 1, 2025 at 10:36 am

    Superb read, I just passed this onto a friend who was doing a little analysis on that. And he really bought me lunch because I discovered it for him smile So let me rephrase that: Thanks for lunch! servicii foto și video pentru nunta ta

    Reply
  44. rs 125 only on thespark shop men jackets on March 1, 2025 at 10:36 am

    Thanks for the nice blog. It was very useful for me. I m happy I found this blog. Thank you for sharing with us,I too always learn something new from your post. rs 125 only on thespark shop men jackets

    Reply
  45. onbola on March 1, 2025 at 10:38 am

    ONBOLA adalah agen judi bola dan situs slot online terpercaya di Indonesia sejak 2013, menawarkan layanan terbaik, proses deposit dan withdrawal cepat, serta berbagai pasaran taruhan bola dan permainan slot populer. https://onbola.net/

    Reply
  46. eslot on March 1, 2025 at 10:38 am

    Eslot adalah sumber yang sangat berguna untuk informasi teknologi terbaru. Saya mendapatkan banyak wawasan berharga di sini, terutama untuk kebutuhan bisnis dan inovasi digital.

    Reply
  47. gemoy138 on March 1, 2025 at 10:39 am

    gemoy138 adalah situs yang sangat menyenangkan dan edukatif untuk anak-anak. Dengan berbagai gambar yang lucu dan kreatif, anak-anak dapat belajar sambil bermain. Ini adalah situs yang sangat baik untuk mendidik anak-anak dengan cara yang menyenangkan.

    Reply
  48. covishield side effects on March 1, 2025 at 10:43 am

    I have bookmarked your website because this site contains valuable information in it. I am really happy with articles quality and presentation. Thanks a lot for keeping great stuff. I am very much thankful for this site. covishield side effects

    Reply
  49. 5starsstocks.com staples on March 1, 2025 at 10:44 am

    Thanks for the nice blog. It was very useful for me. I m happy I found this blog. Thank you for sharing with us,I too always learn something new from your post. 5starsstocks.com staples

    Reply
← Older Comments
Newer Comments →

Submit a Comment

Your email address will not be published. Required fields are marked *