AraBERT : Pre-training BERT for Arabic Language Understanding

by | Feb 28, 2020 | NLP | 49,350 comments

Authors: Wissam Antoun, Fady Baly, Hazem Hajj

 

AraBERT is an Arabic pretrained language model based on Google’s BERT architecture. AraBERT uses the same BERT-Base config. More details are available in the AraBERT PAPER and in the AraBERT Meetup

There is two versions of the model AraBERTv0.1 and AraBERTv1, with the difference being that AraBERTv1 uses pre-segmented text where prefixes and suffixes were split using the Farasa Segmenter.

The model was trained on ~70M sentences or ~23GB of Arabic text with ~3B words.

Source Code Repository: https://github.com/aub-mind/arabert
Paper: https://www.aclweb.org/anthology/2020.osact-1.2.pdf

 

Results (Accuracy)

We evaluate both AraBERT models on different downstream tasks and compare it to mBERT, and other state of the art models (To the extent of our knowledge). The Tasks were Sentiment Analysis on 6 different datasets (HARD, ASTD-Balanced, ArsenTD-Lev, LABR, ArSaS), Named Entity Recognition with the ANERcorp, and Arabic Question Answering on Arabic-SQuAD and ARCD

Task prev. SOTA mBERT AraBERTv0.1 AraBERTv1
HARD 95.7 ElJundi et.al. 95.7 96.2 96.1
ASTD 86.5 ElJundi et.al. 80.1 92.2 92.6
ArsenTD-Lev 52.4 ElJundi et.al. 51 58.9 59.4
AJGT 93 Dahou et.al. 83.6 93.1 93.8
LABR 87.5 Dahou et.al. 83 85.9 86.7
ANERcorp 81.7 (BiLSTM-CRF) 78.4 84.2 81.9
ARCD mBERT EM:34.2 F1: 61.3 EM:51.14 F1:82.13 EM:54.84 F1: 82.15

Model Weights and Vocab Download

Models AraBERTv0.1 AraBERTv1
TensorFlow Drive Link Drive Link
PyTorch Drive_Link Drive_Link

You can find the PyTorch models in HuggingFace’s Transformer Library under the aubmindlab username

If you used this model please cite us as:

@inproceedings{antoun2020arabert,
  title={AraBERT: Transformer-based Model for Arabic Language Understanding},
  author={Antoun, Wissam and Baly, Fady and Hajj, Hazem},
  booktitle={LREC 2020 Workshop Language Resources and Evaluation Conference 11--16 May 2020},
  pages={9}
}

Acknowledgments

Thanks to TensorFlow Research Cloud (TFRC) for the free access to Cloud TPUs, couldn’t have done it without this program, and to the AUB MIND Lab Members for the continous support. Also thanks to Yakshof and Assafir for data and storage access. Another thanks for Habib Rahal (https://www.behance.net/rahalhabib), for putting a face to AraBERT.

Contacts

Wissam Antoun: Linkedin | Twitter | Github | wfa07@mail.aub.edu | wissam.antoun@gmail.com

Fady Baly: Linkedin | Twitter | Github | fgb06@mail.aub.edu | baly.fady@gmail.com

We are looking for sponsors to train BERT-Large and other Transformer models, the sponsor only needs to cover to data storage and compute cost of the generating the pretraining data

49,350 Comments

← Older Comments
Newer Comments →
  2. Health News on February 25, 2025 at 4:03 am

    I have bookmarked your website because this site contains valuable information in it. I am really happy with articles quality and presentation. Thanks a lot for keeping great stuff. I am very much thankful for this site. Health News

    Reply
  4. https://karachimodel.com/ on February 25, 2025 at 4:13 am

    GA6789 – https://karachimodel.com/ là một trong những nhà cái hàng đầu với quy mô lớn, thu hút hàng triệu người chơi mỗi ngày. Nổi bật nhờ chất lượng dịch vụ vượt trội và kho tàng giải trí đa dạng, GA6789 mang đến trải nghiệm cá cược đẳng cấp. Chúng tôi cung cấp các liên kết chính thức và an toàn, đảm bảo người chơi truy cập trực tiếp vào các trò chơi từ GA6789 mà không bị chuyển hướng đến bất kỳ trang web nào khác.

    Reply
  6. tennessee pay check calculator on February 25, 2025 at 4:22 am

    Tennessee is one of the few states in the U.S. that does not impose a state income tax on wages or salaries. This means that your paycheck will not be subject to state income tax withholding. However, federal income tax, Social Security, and Medicare taxes will still be deducted from your earnings.
    tn paycheck calculator

    Reply
  7. Business Ideas on February 25, 2025 at 4:23 am

    This is my first time i visit here. I found so many entertaining stuff in your blog, especially its discussion. From the tons of comments on your articles, I guess I am not the only one having all the leisure here! Keep up the good work. I have been meaning to write something like this on my website and you have given me an idea. Business Ideas

    Reply
  8. Business Ideas on February 25, 2025 at 4:25 am

    With so many books and articles coming up to give gateway to make-money-online field and confusing reader even more on the actual way of earning money, Business Ideas

    Reply
  11. Wheon Grand Theft Auto on February 25, 2025 at 4:41 am

    With so many books and articles coming up to give gateway to make-money-online field and confusing reader even more on the actual way of earning money, Wheon Grand Theft Auto

    Reply
  14. wedding dresses on February 25, 2025 at 4:51 am

    Enjoy a perfect fit without the need for alterations, making your wedding dress truly unique and personal. wedding dresses

    Reply
  17. Apple Teleport on February 25, 2025 at 5:06 am

    This is my first time i visit here. I found so many entertaining stuff in your blog, especially its discussion. From the tons of comments on your articles, I guess I am not the only one having all the leisure here! Keep up the good work. I have been meaning to write something like this on my website and you have given me an idea. Apple Teleport

    Reply
  18. Pearline on February 25, 2025 at 5:20 am

    If you want to grow your knowledge simply keep visiting
    this site and be updated with the most recent news update posted here.

    My page; tonerin kapseln (tonerin-hungary.info)

    Reply
  19. Fatima Javaid on February 25, 2025 at 5:31 am

    ورود به سایت اس نود برای کاربران علاقه‌مند به پیش‌بینی‌های ورزشی و بازی‌های آنلاین یک تجربه جذاب محسوب می‌شود. سایت اس نود با ارائه امکانات متنوع و یک پلتفرم کاربرپسند، محیطی امن و حرفه‌ای را برای شرط‌بندی و پیش‌بینی مسابقات ورزشی فراهم کرده است. برای ورود به سایت اس نود، کاربران می‌توانند از لینک‌های معتبر استفاده کنند تا به راحتی به تمامی خدمات این سایت دسترسی داشته باشند.

    Reply
  20. Fatima Javaid on February 25, 2025 at 5:40 am

    اس بت نود ایک مشہور آن لائن پلیٹ فارم ہے جو صارفین کو بیٹنگ اور گیمنگ کی مختلف سہولیات فراہم کرتا ہے۔ یہ پلیٹ فارم اپنی جدید ٹیکنالوجی اور محفوظ نظام کی بدولت صارفین کے لیے ایک قابل اعتماد انتخاب بن چکا ہے۔ اس بت نود پر مختلف کھیلوں پر بیٹنگ کے ساتھ ساتھ کیسینو گیمز اور دیگر تفریحی آپشنز بھی دستیاب ہیں، جو اسے مزید دلچسپ بناتے ہیں۔

    Reply
  21. hamad on February 25, 2025 at 5:41 am

    To observe picked up against your site even now setting up procedure in plain english only one minimal little bit of submits. Pleasurable way of possibilities long run, We’re also book-marking at a time safe and sound forms prevent rises along. Prodentim

    Reply
  23. Life Style Kennedy Funding Ripoff Report on February 25, 2025 at 6:03 am

    I have bookmarked your website because this site contains valuable information in it. I am really happy with articles quality and presentation. Thanks a lot for keeping great stuff. I am very much thankful for this site. Life Style Kennedy Funding Ripoff Report

    Reply
  24. https://voipincorporated.com/ on February 25, 2025 at 6:07 am

    Ga6789 – https://voipincorporated.com/ là một nền tảng cá cược đá gà trực tuyến nổi bật, mang đến cho người chơi những trải nghiệm chân thực và đầy kịch tính. Mặc dù thông tin về nguồn gốc và lịch sử hình thành của Ga6789 không được tiết lộ chi tiết, nhưng qua những gì nền tảng này thể hiện, có thể thấy Ga6789 đã đầu tư mạnh mẽ vào công nghệ và dịch vụ, tạo nên một sân chơi chuyên nghiệp và đáng tin cậy cho người dùng.

    Reply
  25. 프라그마틱 정품 on February 25, 2025 at 6:14 am

    Excellent read, I just passed this onto a friend who was doing a little research on that. And he actually bought me lunch because I found it for him smile So let me rephrase that: Thank you for lunch! 프라그마틱 정품

    Reply
  27. Fatima Javaid on February 25, 2025 at 6:17 am

    https://www.olmsteadseattle.com/ Deneme bonusu, online bahis siteleri tarafından yeni üyelere sunulan ücretsiz bir promosyon türüdür. Bu bonus, kullanıcıların herhangi bir para yatırmadan oyunları denemesine ve platformun sunduğu hizmetleri keşfetmesine olanak tanır. Deneme bonusu sayesinde bahis severler, risk almadan stratejilerini geliştirebilir ve siteyi daha iyi tanıyabilirler.

    Reply
  28. 86jos on February 25, 2025 at 6:25 am

    That Sounds interesting, i agree with you. Please keep at your good work, I would come back often. 86jos

    Reply
  29. Fatima Javaid on February 25, 2025 at 6:28 am

    Kinza Hashmi

    Reply
  30. Fatima Javaid on February 25, 2025 at 6:28 am

    https://www.olmsteadseattle.com/ Deneme bonusu, online bahis siteleri tarafından yeni üyelere sunulan ücretsiz bir promosyon türüdür. Bu bonus, kullanıcıların herhangi bir para yatırmadan oyunları denemesine ve platformun sunduğu hizmetleri keşfetmesine olanak tanır. Deneme bonusu sayesinde bahis severler, risk almadan stratejilerini geliştirebilir ve siteyi daha iyi tanıyabilirler.

    Reply
  32. hamad on February 25, 2025 at 6:31 am

    It’s really a cool and useful piece of information. I’m satisfied that you shared this useful information with us. Please stay us informed like this. Thank you for sharing. Prodentim

    Reply
  34. 토토사이트 on February 25, 2025 at 6:41 am

    Today, I went to the beachfront with my kids. I found a sea shell and gave it to my 4 year old daughter and said “You can hear the ocean if you put this to your ear.” She put the shell to her ear and screamed. There was a hermit crab inside and it pinched her ear. She never wants to go back! LoL I know this is completely off topic but I had to tell someone!| 토토사이트

    Reply
  36. procurementnation on February 25, 2025 at 6:46 am

    With so many books and articles coming up to give gateway to make-money-online field and confusing reader even more on the actual way of earning money, procurementnation

    Reply
  37. okebos on February 25, 2025 at 6:47 am

    Hey there! I could have sworn I’ve been to this site before but after browsing through some of the post I realized it’s new to me. Nonetheless, I’m definitely happy I found it and I’ll be book-marking and checking back frequently! okebos

    Reply
  40. techpropost on February 25, 2025 at 7:00 am

    I love to advocate only very good furthermore productive information and facts, that’s why see the item: techpropost

    Reply
  41. TRP Roller Chains RS 120-2 on February 25, 2025 at 7:07 am

    Ini adalah bacaan yang sangat menginspirasi. Saya merasa lebih termotivasi sekarang. TRP Roller Chains RS 120-2

    Reply
  42. Fatima Javaid on February 25, 2025 at 7:07 am

    Alexistogel adalah salah satu platform permainan online yang banyak dicari oleh para penggemar judi daring. Dengan berbagai pilihan permainan menarik dan sistem yang aman, Alexistogel menawarkan pengalaman bermain yang nyaman dan menghibur. Selain itu, platform ini dikenal dengan layanan pelanggan yang responsif serta berbagai promo menarik bagi para pemain setia. Bagi yang mencari tempat bermain togel dengan sistem yang terpercaya, Alexistogel bisa menjadi pilihan yang tepat.

    Reply
  43. MyLiberla on February 25, 2025 at 7:08 am

    I have bookmarked your website because this site contains valuable information in it. I am really happy with articles quality and presentation. Thanks a lot for keeping great stuff. I am very much thankful for this site. MyLiberla

    Reply
  44. sinha cin on February 25, 2025 at 7:08 am

    Thanks for sharing this information. I really like your blog post very much. You have really shared a informative and interesting blog post with people. dokument kolekcjonerski

    Reply
  45. HUZAIFa on February 25, 2025 at 7:08 am

    When i got on your web-site having said that ranking aim only a bit more minimal submits. Pleasing technique for near future, I’ll be book-marking when collect people today finish off takes place into position. Fitspresso

    Reply
  46. Ahmed Raza on February 25, 2025 at 7:08 am

    79sodo is a rapidly growing online platform that has gained attention for its diverse range of entertainment offerings. It provides users with an interactive and engaging experience, catering to various interests. With a user-friendly interface and seamless navigation, it ensures that individuals can access its features without any complications. The platform’s design emphasizes convenience, making it appealing to both newcomers and experienced users alike. 79sodo

    Reply
  47. slot kamboja on February 25, 2025 at 7:09 am

    Good, i’ve got to assert that the producing skills not which unhealthy, yet i can also you can actually improve ones own publishing knowledge. Implementing reduced thoughts to mention the actual. You’ll check out search engine just for making skills schooling website and applications. slot kamboja

    Reply
  48. fastupnow on February 25, 2025 at 7:10 am

    I should insist seldom of which it is amazing! The blog is usually info likewise generally fabricate wonderful entitys. fastupnow

    Reply
  49. TRP Roller Chains RS 120-2 on February 25, 2025 at 7:14 am

    Ini adalah bacaan yang sangat menginspirasi. Saya merasa lebih termotivasi sekarang. TRP Roller Chains RS 120-2

    Reply
  50. hamad on February 25, 2025 at 7:15 am

    I spend time people’s writing. It has the excellent to consider you demonstrate around thoughts together with the intellect and also res while doing this very important topic is sometimes successfully deemed. Fitspresso

    Reply
← Older Comments
Newer Comments →

Submit a Comment

Your email address will not be published. Required fields are marked *