AraBERT : Pre-training BERT for Arabic Language Understanding

by | Feb 28, 2020 | NLP | 49,381 comments

Authors: Wissam Antoun, Fady Baly, Hazem Hajj

 

AraBERT is an Arabic pretrained language model based on Google’s BERT architecture. AraBERT uses the same BERT-Base config. More details are available in the AraBERT PAPER and in the AraBERT Meetup

There is two versions of the model AraBERTv0.1 and AraBERTv1, with the difference being that AraBERTv1 uses pre-segmented text where prefixes and suffixes were split using the Farasa Segmenter.

The model was trained on ~70M sentences or ~23GB of Arabic text with ~3B words.

Source Code Repository: https://github.com/aub-mind/arabert
Paper: https://www.aclweb.org/anthology/2020.osact-1.2.pdf

 

Results (Accuracy)

We evaluate both AraBERT models on different downstream tasks and compare it to mBERT, and other state of the art models (To the extent of our knowledge). The Tasks were Sentiment Analysis on 6 different datasets (HARD, ASTD-Balanced, ArsenTD-Lev, LABR, ArSaS), Named Entity Recognition with the ANERcorp, and Arabic Question Answering on Arabic-SQuAD and ARCD

Task prev. SOTA mBERT AraBERTv0.1 AraBERTv1
HARD 95.7 ElJundi et.al. 95.7 96.2 96.1
ASTD 86.5 ElJundi et.al. 80.1 92.2 92.6
ArsenTD-Lev 52.4 ElJundi et.al. 51 58.9 59.4
AJGT 93 Dahou et.al. 83.6 93.1 93.8
LABR 87.5 Dahou et.al. 83 85.9 86.7
ANERcorp 81.7 (BiLSTM-CRF) 78.4 84.2 81.9
ARCD mBERT EM:34.2 F1: 61.3 EM:51.14 F1:82.13 EM:54.84 F1: 82.15

Model Weights and Vocab Download

Models AraBERTv0.1 AraBERTv1
TensorFlow Drive Link Drive Link
PyTorch Drive_Link Drive_Link

You can find the PyTorch models in HuggingFace’s Transformer Library under the aubmindlab username

If you used this model please cite us as:

@inproceedings{antoun2020arabert,
  title={AraBERT: Transformer-based Model for Arabic Language Understanding},
  author={Antoun, Wissam and Baly, Fady and Hajj, Hazem},
  booktitle={LREC 2020 Workshop Language Resources and Evaluation Conference 11--16 May 2020},
  pages={9}
}

Acknowledgments

Thanks to TensorFlow Research Cloud (TFRC) for the free access to Cloud TPUs, couldn’t have done it without this program, and to the AUB MIND Lab Members for the continous support. Also thanks to Yakshof and Assafir for data and storage access. Another thanks for Habib Rahal (https://www.behance.net/rahalhabib), for putting a face to AraBERT.

Contacts

Wissam Antoun: Linkedin | Twitter | Github | wfa07@mail.aub.edu | wissam.antoun@gmail.com

Fady Baly: Linkedin | Twitter | Github | fgb06@mail.aub.edu | baly.fady@gmail.com

We are looking for sponsors to train BERT-Large and other Transformer models, the sponsor only needs to cover to data storage and compute cost of the generating the pretraining data

49,381 Comments

← Older Comments
Newer Comments →
  1. fabbasketballtoto on March 18, 2025 at 3:25 am

    Thank you for sharing your info. I truly appreciate your efforts and I am waiting for your further post thanks once again. 농구토토 하는법

    Reply
  2. basketballbet on March 18, 2025 at 3:26 am

    I like the helpful info you provide on your articles.
    I will bookmark your weblog and test again right here frequently. 농구 토토

    Reply
  3. 축구토토 승무패 on March 18, 2025 at 3:28 am

    You could definitely see your enthusiasm in the work you write. The sector hopes for even more passionate writers such as you who are not afraid to say how they believe.
    축구토토 승무패

    Reply
  4. handicaptoto on March 18, 2025 at 3:29 am

    I would like to thank you for the efforts you have put in penning this site.
    I’m hoping to view the same high-grade blog posts from you later on as well. 핸디캡 토토사이트

    Reply
  5. fabtotobetting on March 18, 2025 at 3:30 am

    I am actually delighted to read this website posts which consists of tons of helpful information, thanks for providing such data. 토토 베팅

    Reply
  6. fabwagskr on March 18, 2025 at 3:30 am

    Excellent weblog right here! Additionally your website loads up very fast! What host are you the usage of? 토토사이트

    Reply
  7. Conveyor Belt Sersan 3 Ply BW 500 on March 18, 2025 at 3:31 am

    Artikel yang sangat inspiratif! Saya merasa lebih termotivasi untuk mencapai tujuan saya sekarang. Conveyor Belt Sersan 3 Ply BW 500

    Reply
  8. GOOD88 on March 18, 2025 at 3:42 am

    Ngoai khuyen mai chao mung, GOOD88 con thuong xuyen to chuc cac chuong trinh khuyen mai nap tien hang thang.

    Reply
  9. Ok365 on March 18, 2025 at 3:42 am

    Ok365 cung cấp link đăng ký chính thức mới nhất năm 2025, giúp bạn truy cập nền tảng một cách an toàn và nhanh chóng. Với hướng dẫn chi tiết, bạn sẽ dễ dàng tham gia và tận hưởng các dịch vụ tiện ích hấp dẫn. OK365 đảm bảo an toàn và mang đến trải nghiệm tuyệt vời với quy trình đăng ký đơn giản, chính thức và nhanh gọn.

    Reply
  12. Graduate coaching programs on March 18, 2025 at 4:22 am

    Graduate coaching programsGraduate Coach is about providing support and career advice to young students graduating from the engineering, environmental science, IT graduation and business faculties in Melbourne. Our programs are tailored to help graduates identify their preferred industry segment based around their own personality types and strengths.We also provide support for newly immigrating engineers wanting to find work. We specialise in marketing the person correctly and aligning their overseas experience with compatible companies here in Australia

    Reply
  13. Jens on March 18, 2025 at 4:24 am

    Hello everyone,

    SEO and digital marketing are absolutely crucial in today’s fast-paced business world.
    Living in Illinois, I’ve observed how difference between local seo and techniical seo (https://bd.Cane-Recruitment.com/companies/nfinitelimits-top-tier-digital-marketing-and-professional-web-presence) SEO transform online visibility—especially foor competitive
    industries. Nfinitelimits, a premier agency in Mundelein,
    truly stands out with customized solutions.

    Their specialization in local search optimization and PPC campaigns greatly benefits businesses
    to dominate in regional markets. Take their work|For instance successful campaigns: theey refine Google
    Business Profiles and employ state-of-the-art technology to keep ahead of search engine updates.
    Paired with social media integration, they
    develop well-rounded approaches.

    Should you be aiming to revitalize your online reach,
    their team delivers measurable results. The agency’s bespoke solutions ignite sustainable growth,
    not just generic solutions.

    Reply
  14. Fatima Javaid on March 18, 2025 at 4:26 am

    Football is more than just a sport; it’s a passion that unites fans worldwide, and one of the best ways to show support for your favorite team is by wearing football shirts. These shirts are not just about style; they represent team spirit, history, and identity, making them a must-have for any football enthusiast. Whether you’re playing on the field or cheering from the stands, a high-quality football shirt provides comfort and durability, ensuring you stay cool and confident.

    Reply
  15. sòng bạc trực tuyến on March 18, 2025 at 4:31 am

    Your article is a testament to the depth of your expertise and the breadth of your knowledge.

    sòng bạc trực tuyến

    Reply
  16. Conveyor Belts Sersan 3 Ply BW 600 on March 18, 2025 at 4:31 am

    Saya sangat terkesan dengan kedalaman pengetahuan Anda tentang topik ini. Sangat mengesankan! Conveyor Belts Sersan 3 Ply BW 600

    Reply
  17. Muhammad Zubair on March 18, 2025 at 4:32 am

    Ipototo is an innovative platform that brings unique and stylish products to consumers who appreciate quality and creativity. Whether you are looking for fashion accessories, home decor, or trendy gadgets, ipototo offers a diverse selection to meet your needs. With a focus on modern aesthetics and functionality, this brand ensures that every item is carefully curated to provide the best value.

    Reply
  18. Max on March 18, 2025 at 4:38 am

    This is the right blog for anybody who wishes to find out about this topic. Legsgo Holidays offers unforgettable travel experiences with customized itineraries and expert planning. Explore with Best Travel Agency world hassle-free with our trusted travel service.

    Reply
  19. Conveyor Belts Sersan 3 Ply BW 600 on March 18, 2025 at 4:39 am

    Saya sangat terkesan dengan kedalaman pengetahuan Anda tentang topik ini. Sangat mengesankan! Conveyor Belts Sersan 3 Ply BW 600

    Reply
  21. anonymous on March 18, 2025 at 4:50 am

    The morning-after pill is a method of emergency contraception used to prevent unwanted pregnancy after unprotected sex or when birth control has failed.  Ertesi Gün Hapı

    Reply
  23. family law firm sydney on March 18, 2025 at 5:10 am

    Consort is a boutique family law firm sydney, assisting clients with divorce, separation, property settlements, spousal support, and co-parenting. Expert solicitors provide legal guidance and represent clients in court.

    Reply
  24. Willaum on March 18, 2025 at 5:12 am

    For all the essential details about the event, visit Bordeaucourse.org. The official website, Bordeaucourse.org, offers easy access to registration, race schedules, and tips for participants. Whether you’re a first-time runner or a seasoned marathoner, Bordeaucourse.org is the best place to prepare for the event. Get all the information you need, from race day instructions to helpful guides, by visiting Bordeaucourse.org today and secure your spot in this exciting race!

    Reply
  25. bb on March 18, 2025 at 5:13 am

    A good blog always comes-up with new and exciting information and while reading I have feel that this blog is really have all those quality that qualify a blog to be a one. Nenek188

    Reply
  26. HUZAIFA on March 18, 2025 at 5:15 am

    I spend time many of the subject material, I have to express i actually prized, I actually wants more information about the following, just because this can be very superb., Thanks a ton pertaining to discovering. سایت تاک تیک بدون فیلتر

    Reply
  27. 88vv on March 18, 2025 at 5:15 am

    88vv được mệnh danh là điểm cược hàng đầu, nơi mọi người có thể thỏa sức tận hưởng niềm đam mê và nhận những phần thưởng giá trị. Với chất lượng dịch vụ đẳng cấp và tỷ lệ trả thưởng siêu hấp dẫn, thương hiệu cam kết mang đến những giây phút cá cược đỉnh cao cùng vô số cơ hội làm giàu cho người chơi.

    Reply
  28. HUZAIFA on March 18, 2025 at 5:38 am

    I enjoy the variety of blogposts, I actually certainly favored, I like more knowledge about them, considering it is rather excellent., All the best designed for revealing. سایت شرط بندی فوتبال سبد

    Reply
  29. aliuamir aliuad on March 18, 2025 at 5:41 am

    신용카드 현금화란 신용카드의 한도를 이용하여 현금을 마련하는 서비스입니다. 상품을 결제하고, 일정 수수료를 제외한 뒤 계좌로 현금을 입금받는 금융 서비스입니다. 도프티켓은 정식등록사업자로 안전하고 빠른 결제와 즉시 입금 서비스를 제공하고 있습니다. 급하게 현금이 필요할 때 간편하고 신용 조회 없이 이용할 수 있어 많은 사람들이 찾고 있습니다. 업계 최저 수수료인 신용카드 현금화 90%를 제공하고 있으니 자세한 내용은 도프티켓 24시간 고객센터에 문의하여 주세요. 신용카드 현금화

    Reply
  30. aliuamir aliuad on March 18, 2025 at 5:44 am

    I am thankful to you for sharing this plethora of useful information. I found this resource utmost beneficial for me. Thanks a lot for hard work. olxtoto

    Reply
  32. Safaga on March 18, 2025 at 5:51 am

    Safaga ist eine ruhige Hafenstadt am Roten Meer in Ägypten, bekannt für ihre kristallklaren Gewässer, unberührten Strände und hervorragenden Tauch- und Surfspots. Die Stadt zieht sowohl Abenteurer als auch Erholungssuchende an, besonders wegen ihrer heilenden schwarzen Sandstrände, die für therapeutische Behandlungen genutzt werden. Safaga bietet eine entspannte Alternative zu den belebteren Urlaubszielen wie Hurghada.

    Reply
  33. aliuamir aliuad on March 18, 2025 at 5:53 am

    신용카드 현금화란 신용카드의 한도를 이용하여 현금을 마련하는 서비스입니다. 상품을 결제하고, 일정 수수료를 제외한 뒤 계좌로 현금을 입금받는 금융 서비스입니다. 도프티켓은 정식등록사업자로 안전하고 빠른 결제와 즉시 입금 서비스를 제공하고 있습니다. 급하게 현금이 필요할 때 간편하고 신용 조회 없이 이용할 수 있어 많은 사람들이 찾고 있습니다. 업계 최저 수수료인 신용카드 현금화 90%를 제공하고 있으니 자세한 내용은 도프티켓 24시간 고객센터에 문의하여 주세요. 신용카드 현금화

    Reply
  34. aliuamiraliuad on March 18, 2025 at 5:57 am

    I think this is an informative post and it is very beneficial and knowledgeable. Therefore, I would like to thank you for the endeavors that you have made in writing this article. All the content is absolutely well-researched. Thanks… olxtoto link

    Reply
  35. HUZAIFA on March 18, 2025 at 6:07 am

    I spend time many of the subject material, I have to express i actually prized, I actually wants more information about the following, just because this can be very superb., Thanks a ton pertaining to discovering. بت برو

    Reply
  36. aliuamir aliuad on March 18, 2025 at 6:16 am

    신용카드 현금화란 신용카드의 한도를 이용하여 현금을 마련하는 서비스입니다. 상품을 결제하고, 일정 수수료를 제외한 뒤 계좌로 현금을 입금받는 금융 서비스입니다. 도프티켓은 정식등록사업자로 안전하고 빠른 결제와 즉시 입금 서비스를 제공하고 있습니다. 급하게 현금이 필요할 때 간편하고 신용 조회 없이 이용할 수 있어 많은 사람들이 찾고 있습니다. 업계 최저 수수료인 신용카드 현금화 90%를 제공하고 있으니 자세한 내용은 도프티켓 24시간 고객센터에 문의하여 주세요. https://www.dopeticket.com

    Reply
  37. https://andrewfashion.com/ on March 18, 2025 at 6:20 am

    https://andrewfashion.com/ mang lại muôn vàn ưu đãi hấp dẫn, với nền tảng giải trí công bằng minh bạch.

    Reply
  38. aliuamir aliuad on March 18, 2025 at 6:20 am

    Excellent website! I adore how it is easy on my eyes it is. I am questioning how I might be notified whenever a new post has been made. Looking for more new updates. Have a great day! link alternatif olxtoto

    Reply
  39. HUZAIFA on March 18, 2025 at 6:21 am

    I spend time many of the subject material, I have to express i actually prized, I actually wants more information about the following, just because this can be very superb., Thanks a ton pertaining to discovering. بازی انفجار 2

    Reply
  40. Conveyor Belts Sersan 4 Ply BW 500 on March 18, 2025 at 6:25 am

    Saya akan merekomendasikan blog ini kepada siapa pun yang mencari informasi berkualitas tentang topik ini. Conveyor Belts Sersan 4 Ply BW 500

    Reply
  42. HUZAIFA on March 18, 2025 at 6:36 am

    I spend time many of the subject material, I have to express i actually prized, I actually wants more information about the following, just because this can be very superb., Thanks a ton pertaining to discovering. بازی اسلات

    Reply
  43. Conveyor Belts Sersan 4 Ply BW 500 on March 18, 2025 at 6:39 am

    Saya akan merekomendasikan blog ini kepada siapa pun yang mencari informasi berkualitas tentang topik ini. Conveyor Belts Sersan 4 Ply BW 500

    Reply
  44. RR88 on March 18, 2025 at 6:45 am

    Nhà Cái RR88 chinh thuc ap dung chuong trinh hoan tien len den 10% vao moi thu hai dau tuan.

    Reply
  45. clini chelsa on March 18, 2025 at 6:45 am

    فیلربادی
    فیلر بادی یا فیلر بدن موادی هستند که برای حجیم‌سازی و شکل‌دهی به بخش‌های مختلف بدن مانند باسن، ران، و سینه‌ها استفاده می‌شوند. این مواد به منظور بهبود ظاهر بدن و افزایش حجم نواحی خاص تزریق می‌شوند. فیلرهای مورد استفاده در بدن معمولاً شامل ژل‌های هیالورونیک اسید یا ترکیبات شبه‌چربی هستند که قابلیت جذب توسط بدن را دارند.

    Reply
  46. HUZAIFA on March 18, 2025 at 6:48 am

    I favor an entire couple of merchandise, Dreamed about seriously experienced, I require specifics. during this, taking into account that must be to some extent really good., With thanks considerably with regards to expressing. بازی ۲۱ ایرانی

    Reply
  48. Kina on March 18, 2025 at 6:54 am

    I wanted to thank you for this wonderful read!!

    I certainly loved every little bit of it. I have got you book-marked to look at new things you post…

    Feel free to visit my webpage: imp source

    Reply
  49. antminer s19 on March 18, 2025 at 6:57 am

    As I watched the stars of yesteryear light up the screen, I couldn’t help but feel a sense of awe and admiration for the artistry and talent that defined a generation. antminer s19

    Reply
← Older Comments
Newer Comments →

Submit a Comment

Your email address will not be published. Required fields are marked *