AraBERT : Pre-training BERT for Arabic Language Understanding

by | Feb 28, 2020 | NLP | 47,585 comments

Authors: Wissam Antoun, Fady Baly, Hazem Hajj

 

AraBERT is an Arabic pretrained language model based on Google’s BERT architecture. AraBERT uses the same BERT-Base config. More details are available in the AraBERT PAPER and in the AraBERT Meetup

There is two versions of the model AraBERTv0.1 and AraBERTv1, with the difference being that AraBERTv1 uses pre-segmented text where prefixes and suffixes were split using the Farasa Segmenter.

The model was trained on ~70M sentences or ~23GB of Arabic text with ~3B words.

Source Code Repository: https://github.com/aub-mind/arabert
Paper: https://www.aclweb.org/anthology/2020.osact-1.2.pdf

 

Results (Accuracy)

We evaluate both AraBERT models on different downstream tasks and compare it to mBERT, and other state of the art models (To the extent of our knowledge). The Tasks were Sentiment Analysis on 6 different datasets (HARD, ASTD-Balanced, ArsenTD-Lev, LABR, ArSaS), Named Entity Recognition with the ANERcorp, and Arabic Question Answering on Arabic-SQuAD and ARCD

Task prev. SOTA mBERT AraBERTv0.1 AraBERTv1
HARD 95.7 ElJundi et.al. 95.7 96.2 96.1
ASTD 86.5 ElJundi et.al. 80.1 92.2 92.6
ArsenTD-Lev 52.4 ElJundi et.al. 51 58.9 59.4
AJGT 93 Dahou et.al. 83.6 93.1 93.8
LABR 87.5 Dahou et.al. 83 85.9 86.7
ANERcorp 81.7 (BiLSTM-CRF) 78.4 84.2 81.9
ARCD mBERT EM:34.2 F1: 61.3 EM:51.14 F1:82.13 EM:54.84 F1: 82.15

Model Weights and Vocab Download

Models AraBERTv0.1 AraBERTv1
TensorFlow Drive Link Drive Link
PyTorch Drive_Link Drive_Link

You can find the PyTorch models in HuggingFace’s Transformer Library under the aubmindlab username

If you used this model please cite us as:

@inproceedings{antoun2020arabert,
  title={AraBERT: Transformer-based Model for Arabic Language Understanding},
  author={Antoun, Wissam and Baly, Fady and Hajj, Hazem},
  booktitle={LREC 2020 Workshop Language Resources and Evaluation Conference 11--16 May 2020},
  pages={9}
}

Acknowledgments

Thanks to TensorFlow Research Cloud (TFRC) for the free access to Cloud TPUs, couldn’t have done it without this program, and to the AUB MIND Lab Members for the continous support. Also thanks to Yakshof and Assafir for data and storage access. Another thanks for Habib Rahal (https://www.behance.net/rahalhabib), for putting a face to AraBERT.

Contacts

Wissam Antoun: Linkedin | Twitter | Github | wfa07@mail.aub.edu | wissam.antoun@gmail.com

Fady Baly: Linkedin | Twitter | Github | fgb06@mail.aub.edu | baly.fady@gmail.com

We are looking for sponsors to train BERT-Large and other Transformer models, the sponsor only needs to cover to data storage and compute cost of the generating the pretraining data

47,585 Comments

← Older Comments
Newer Comments →
  2. Cryptocurrency on March 11, 2025 at 6:58 pm

    Such a very useful article. Very interesting to read this article.I would like to thank you for the efforts you had made for writing this awesome article. Cryptocurrency

    Reply
  3. Loguytren Problems on March 11, 2025 at 7:10 pm

    I am really enjoying reading your well written articles. It looks like you spend a lot of effort and time on your blog. I have bookmarked it and I am looking forward to reading new articles. Keep up the good work. Loguytren Problems

    Reply
  4. asas on March 11, 2025 at 7:16 pm

    I’m happy to see the considerable subtle element here!. وانیکس

    Reply
  6. i fear no one, but respect everyone. – tymoff on March 11, 2025 at 7:29 pm

    They’re produced by the very best degree developers who will be distinguished for your polo dress creating. You’ll find polo Ron Lauren inside exclusive array which include particular classes for men, women. i fear no one, but respect everyone. – tymoff

    Reply
  7. سایت شرط بندی با درگاه مستقیم on March 11, 2025 at 7:37 pm

    در سایت شرط بندی با درگاه مستقیم، هیچ‌گاه برای پرداخت‌های خود معطل نخواهید شد. این سایت‌ها با استفاده از سیستم‌های پیشرفته و پرداخت‌های سریع، تجربه‌ای امن و راحت را برای شما فراهم می‌کنند. از آنجایی که واسطه‌ها حذف شده‌اند، تمامی تراکنش‌ها به‌سرعت و با امنیت بالا انجام می‌شود.

    https://siteshartbandi-farsi.com/

    Reply
  9. Feedback on March 11, 2025 at 7:53 pm

    I am impressed by the article you have chosen for the readers. Keep up nice work mcdvoice.com survey

    Reply
  10. Florence on March 11, 2025 at 7:59 pm

    It is not my first time to pay a visit this
    website, i am browsing this site dailly and take fastidious facts from here all the time.

    my web site – خرید بک لینک

    Reply
  11. card game techinsiderz on March 11, 2025 at 8:08 pm

    Really impressed! Everything is very open and very clear clarification of issues. It contains truly facts. Your website is very valuable. Thanks for sharing. card game techinsiderz

    Reply
  12. GamerXyt on March 11, 2025 at 8:19 pm

    I have bookmarked your website because this site contains valuable information in it. I am really happy with articles quality and presentation. Thanks a lot for keeping great stuff. I am very much thankful for this site. GamerXyt

    Reply
  13. ProgramGeeks Social Media on March 11, 2025 at 8:24 pm

    I really enjoy simply reading all of your weblogs. Simply wanted to inform you that you have people like me who appreciate your work. Definitely a great post. Hats off to you! The information that you have provided is very helpful. ProgramGeeks Social Media

    Reply
  14. anonymous on March 11, 2025 at 8:25 pm

    Lung cancer is a type of cancer that occurs when cells in the lung tissue grow and multiply uncontrollably. This uncontrolled growth can lead to the formation of masses called tumors, which can spread (metastasize) to surrounding tissues and other parts of the body over time. Akciğer Kanseri 4. Evre Ölüm Belirtileri

    Reply
  15. Business Ideas on March 11, 2025 at 8:26 pm

    I have bookmarked your website because this site contains valuable information in it. I am really happy with articles quality and presentation. Thanks a lot for keeping great stuff. I am very much thankful for this site. Business Ideas

    Reply
  16. Health News on March 11, 2025 at 8:33 pm

    I have bookmarked your website because this site contains valuable information in it. I am really happy with articles quality and presentation. Thanks a lot for keeping great stuff. I am very much thankful for this site. Health News

    Reply
  17. Online Connectivity on March 11, 2025 at 8:34 pm

    They’re produced by the very best degree developers who will be distinguished for your polo dress creating. You’ll find polo Ron Lauren inside exclusive array which include particular classes for men, women. Online Connectivity

    Reply
  18. zimbabwe national cricket team vs india national cricket team timeline on March 11, 2025 at 8:40 pm

    I have bookmarked your website because this site contains valuable information in it. I am really happy with articles quality and presentation. Thanks a lot for keeping great stuff. I am very much thankful for this site. zimbabwe national cricket team vs india national cricket team timeline

    Reply
  19. WellHealthOrganic Organic Pet Food Benefits on March 11, 2025 at 8:47 pm

    I have bookmarked your website because this site contains valuable information in it. I am really happy with articles quality and presentation. Thanks a lot for keeping great stuff. I am very much thankful for this site. WellHealthOrganic Organic Pet Food Benefits

    Reply
  20. Juliet Lagrone on March 11, 2025 at 8:48 pm

    Writing a dissertation is one of the most challenging academic tasks, requiring extensive research, critical thinking, and strong writing skills. Many students struggle with meeting deadlines while maintaining quality, leading them to seek professional assistance. If you’re looking for expert guidance, choosing a reliable service can make a huge difference in achieving academic success. However, it’s important to ensure originality and ethical practices when using such services. Have you ever considered professional help for your dissertation? What are your thoughts on balancing academic integrity and external assistance? If you ever find yourself thinking, Write My Dissertation For Me, make sure to choose a trustworthy service that values quality and authenticity.

    Reply
  21. Latest News LogicalShout on March 11, 2025 at 8:50 pm

    Going to graduate school was a positive decision for me. I enjoyed the coursework, the presentations, the fellow students, and the professors. And since my company reimbursed 100% of the tuition, the only cost that I had to pay on my own was for books and supplies. Otherwise, I received a free master’s degree. All that I had to invest was my time. Latest News LogicalShout

    Reply
  22. Blogsternation .com on March 11, 2025 at 9:06 pm

    It has fully emerged to crown Singapore’s southern shores and undoubtedly placed her on the global map of residential landmarks. I still scored the more points than I ever have in a season for GS. I think you would be hard pressed to find somebody with the same consistency I have had over the years so I am happy with that. Blogsternation .com

    Reply
  23. blogs on March 11, 2025 at 10:00 pm

    The variety of genres on 뉴토끼
    is incredible.

    Reply
  24. Women’s Shearling Leather Hooded Jacket on March 11, 2025 at 10:09 pm

    Stay warm without sacrificing style in the Women’s Shearling Leather Hooded Jacket by Leather Outfitter. Featuring high-quality leather with a soft shearling lining, this jacket combines luxurious comfort with a casual, sporty vibe. The hooded design adds an extra layer of warmth, making it ideal for chilly weather. Whether you’re lounging around or on the go, this jacket ensures that you stay cozy and chic. Women’s Shearling Leather Hooded Jacket

    Reply
  25. Arnold on March 11, 2025 at 11:17 pm

    Hi, i read your blog from time to time and i own a similar one and i was
    just curious if you get a lot of spam responses? If so how do
    you stop it, any plugin or anything you can suggest?
    I get so much lately it’s driving me mad so any assistance is very much appreciated.

    Feel free to visit my web page … BIOTIC CREAM

    Reply
  26. john on March 11, 2025 at 11:21 pm

    “Minecraft APK is still my favorite game! Whether you play survival or creative, the possibilities are endless. Mods make it even more exciting!” minecraft apk

    Reply
  27. togel178 on March 11, 2025 at 11:53 pm

    of course like your web-site however you need to check the spelling on quite a few of your posts. A number of them are rife with spelling issues and I to find it very bothersome to inform the reality however I’ll surely come back again. togel 178

    Reply
  28. Raquel Bednar on March 11, 2025 at 11:58 pm

    “Buy Cash App Accounts” refers to the process of purchasing pre-existing Cash App accounts, which are used for sending and receiving money, making payments, and handling financial transactions. These accounts may come with varying levels of setup, such as having existing balances, linked cards, or a verified status. While some may seek to buy buy Cash App Accounts for convenience, this practice is often against Cash App’s terms of service, and could lead to security risks, fraud, or account suspension. Users are generally advised to create their own accounts directly through the Cash App platform to ensure safety and compliance with the app’s rules.

    Reply
  29. Angelika on March 12, 2025 at 12:27 am

    Hmm is anyone else experiencing problems with the images on this blog loading?
    I’m trying to figure out if its a problem on my
    end or if it’s the blog. Any feedback would
    be greatly appreciated.

    Here is my blog post – mama-forum-1.ru

    Reply
  30. anonymous on March 12, 2025 at 12:35 am

    Lung cancer is a type of cancer that occurs when cells in the lung tissue grow and multiply uncontrollably. This uncontrolled growth can lead to the formation of masses called tumors, which can spread (metastasize) to surrounding tissues and other parts of the body over time. Akciğer Kanseri 1. Evre Belirtileri

    Reply
  31. Feedback on March 12, 2025 at 12:41 am

    Lung cancer is a type of cancer that occurs when cells in the lung tissue grow and multiply uncontrollably. This uncontrolled growth can lead to the formation of masses called tumors, which can spread (metastasize) to surrounding tissues and other parts of the body over time. Akciğer Kanseri 1. Evre Belirtileri

    Reply
  32. asabet on March 12, 2025 at 1:11 am

    سایت آس بت 90 (Asabet 90) یک پلتفرم پیشرفته برای شرط بندی آنلاین است که تجربه ‌ای جذاب و امن برای کاربران فراهم می ‌کند. این ‌ نام

    asabet

    Reply
  33. alexistogel on March 12, 2025 at 1:12 am

    Selamat datang di situs slot gacor Alexistogel pasti maxwin, platform situs slot resmi terpercaya kami menawarkan berbagai macam
    alexistogel

    Reply
  34. DURALINK Roller Chains RS 80-2 on March 12, 2025 at 1:50 am

    Saya baru menemukan blog ini, dan saya sangat senang mengetahui ada begitu banyak konten berkualitas di sini. DURALINK Roller Chains RS 80-2

    Reply
  35. DURALINK Roller Chains RS 80-2 on March 12, 2025 at 1:56 am

    Saya baru menemukan blog ini, dan saya sangat senang mengetahui ada begitu banyak konten berkualitas di sini. DURALINK Roller Chains RS 80-2

    Reply
  36. 신용카드 현금화 on March 12, 2025 at 2:07 am

    I think this is among the most important information for me. D. 신용카드 현금화

    Reply
  38. 바카라사이트 on March 12, 2025 at 2:08 am

    The website style is ideal, the articles is really nice : D.
    바카라사이트

    Reply
  40. 카지노사이트 on March 12, 2025 at 2:10 am

    Hi there very cool web site!! Guy .. Excellent .. D. 카지노사이트

    Reply
  41. 토토사이트 on March 12, 2025 at 2:11 am

    I’m definitely loving the information. D. 토토사이트

    Reply
  42. 안전놀이터 on March 12, 2025 at 2:12 am

    Exceptional blog and great
    style and design. D.
    안전놀이터

    Reply
  43. 신용카드 현금화 on March 12, 2025 at 2:13 am

    Assume you did a lot of your very own html coding. D.
    신용카드 현금화

    Reply
  46. DURALINK Roller Chains RS 100-2 on March 12, 2025 at 2:28 am

    Ini benar-benar membuka mata saya tentang topik ini. Saya akan membacanya lagi untuk lebih memahami. DURALINK Roller Chains RS 100-2

    Reply
  47. j88 on March 12, 2025 at 2:35 am

    Tat ca tai khoan moi dang ky hoi vien https://j88z.club/ trong vong 24 gio va xac minh tai khoan thanh cong se duoc nhan 50k vao tai khoan.

    Reply
  48. DURALINK Roller Chains RS 100-2 on March 12, 2025 at 2:35 am

    Ini benar-benar membuka mata saya tentang topik ini. Saya akan membacanya lagi untuk lebih memahami. DURALINK Roller Chains RS 100-2

    Reply
← Older Comments
Newer Comments →

Submit a Comment

Your email address will not be published. Required fields are marked *