AraBERT : Pre-training BERT for Arabic Language Understanding

by | Feb 28, 2020 | NLP | 49,613 comments

Authors: Wissam Antoun, Fady Baly, Hazem Hajj

 

AraBERT is an Arabic pretrained language model based on Google’s BERT architecture. AraBERT uses the same BERT-Base config. More details are available in the AraBERT PAPER and in the AraBERT Meetup

There is two versions of the model AraBERTv0.1 and AraBERTv1, with the difference being that AraBERTv1 uses pre-segmented text where prefixes and suffixes were split using the Farasa Segmenter.

The model was trained on ~70M sentences or ~23GB of Arabic text with ~3B words.

Source Code Repository: https://github.com/aub-mind/arabert
Paper: https://www.aclweb.org/anthology/2020.osact-1.2.pdf

 

Results (Accuracy)

We evaluate both AraBERT models on different downstream tasks and compare it to mBERT, and other state of the art models (To the extent of our knowledge). The Tasks were Sentiment Analysis on 6 different datasets (HARD, ASTD-Balanced, ArsenTD-Lev, LABR, ArSaS), Named Entity Recognition with the ANERcorp, and Arabic Question Answering on Arabic-SQuAD and ARCD

Task prev. SOTA mBERT AraBERTv0.1 AraBERTv1
HARD 95.7 ElJundi et.al. 95.7 96.2 96.1
ASTD 86.5 ElJundi et.al. 80.1 92.2 92.6
ArsenTD-Lev 52.4 ElJundi et.al. 51 58.9 59.4
AJGT 93 Dahou et.al. 83.6 93.1 93.8
LABR 87.5 Dahou et.al. 83 85.9 86.7
ANERcorp 81.7 (BiLSTM-CRF) 78.4 84.2 81.9
ARCD mBERT EM:34.2 F1: 61.3 EM:51.14 F1:82.13 EM:54.84 F1: 82.15

Model Weights and Vocab Download

Models AraBERTv0.1 AraBERTv1
TensorFlow Drive Link Drive Link
PyTorch Drive_Link Drive_Link

You can find the PyTorch models in HuggingFace’s Transformer Library under the aubmindlab username

If you used this model please cite us as:

@inproceedings{antoun2020arabert,
  title={AraBERT: Transformer-based Model for Arabic Language Understanding},
  author={Antoun, Wissam and Baly, Fady and Hajj, Hazem},
  booktitle={LREC 2020 Workshop Language Resources and Evaluation Conference 11--16 May 2020},
  pages={9}
}

Acknowledgments

Thanks to TensorFlow Research Cloud (TFRC) for the free access to Cloud TPUs, couldn’t have done it without this program, and to the AUB MIND Lab Members for the continous support. Also thanks to Yakshof and Assafir for data and storage access. Another thanks for Habib Rahal (https://www.behance.net/rahalhabib), for putting a face to AraBERT.

Contacts

Wissam Antoun: Linkedin | Twitter | Github | wfa07@mail.aub.edu | wissam.antoun@gmail.com

Fady Baly: Linkedin | Twitter | Github | fgb06@mail.aub.edu | baly.fady@gmail.com

We are looking for sponsors to train BERT-Large and other Transformer models, the sponsor only needs to cover to data storage and compute cost of the generating the pretraining data

49,613 Comments

← Older Comments
Newer Comments →
  2. 8kbet on March 19, 2025 at 3:26 am

    De chao don anh em gia nhap san cuoc, 8kbet nap dau gui tang 500k ngay sau khi giao dich hoan tat tu 50k.

    Reply
  4. Leah on March 19, 2025 at 3:39 am

    I do consider all the ideas you have presented for your post.
    They are really convincing and will definitely work. Still, the posts are very brief for newbies.
    May you please lengthen them a bit from next time? Thanks for the post.

    My page: link

    Reply
  7. Detroit Letterpress on March 19, 2025 at 3:52 am

    Experience the timeless beauty of Detroit Letterpress with Hearty Greetings, a brand dedicated to preserving the artistry of traditional printing. Specializing in custom stationery, invitations, and letterpress designs, Hearty Greetings brings vintage craftsmanship into the modern world.

    Using movable type and century-old printing presses, each piece is meticulously crafted to capture the authenticity and charm of classic letterpress printing. Whether you’re looking for personalized invitations, elegant greeting cards, or bespoke stationery, our handcrafted designs make every occasion unforgettable.

    Explore the finest in Detroit Letterpress with Hearty Greetings—where history, craftsmanship, and creativity come to life. Shop now and add a touch of vintage elegance to your correspondence.

    Reply
  9. Cruz on March 19, 2025 at 3:59 am

    Hello! I know this is kinda off topic but I was wondering which blog platform are you using for
    this website? I’m getting fed up of WordPress because I’ve had issues with hackers and I’m looking at options for
    another platform. I would be great if you could point me in the direction of a good platform.

    Look at my website … pg99

    Reply
  14. JOYSTREAM on March 19, 2025 at 4:46 am

    Discover JOYSTREAM, the premier brand for high-quality gymnastics apparel that blends performance, comfort, and style. Designed for gymnasts of all levels, from beginners to elite athletes, our collections feature innovative fabrics and ergonomic designs to enhance flexibility and durability.

    Known for stylish and resilient leotards, unitards, and biketards, JOYSTREAM supports athletes’ movements with confidence and flair. Whether for practice or competition, our gymnastics wear ensures you look and feel your best.

    Elevate your performance with JOYSTREAM—where style meets function in every move.

    Reply
  15. Muhammad Zubair on March 19, 2025 at 4:49 am

    Jika Anda mencari pengalaman bermain slot online yang seru dan menguntungkan, slot777 adalah pilihan terbaik. Dengan berbagai pilihan permainan menarik, fitur bonus besar, dan sistem keamanan tinggi, slot777 memberikan pengalaman bermain yang nyaman dan menyenangkan. Jangan lewatkan kesempatan untuk meraih jackpot besar dan nikmati sensasi permainan slot online dengan layanan terbaik!

    Reply
  17. https://commodityonlinetips.in/ on March 19, 2025 at 5:04 am

    https://commodityonlinetips.in/ đấu trường đỉnh cao, sân chơi công bằng với tỷ lệ thắng cao

    Reply
  19. wholesalepeptide on March 19, 2025 at 5:15 am

    Wholesale peptides delivers premium peptides with over 13 years of expertise, ensuring high-quality products, competitive pricing, and outstanding customer service, fostering partnerships that drive innovation and scientific progress in research.

    Reply
  23. hha school near me on March 19, 2025 at 5:37 am

    hha school near meIn Lahore, Pakistan, the Holistic Care Academy (HCA) is a leading institution offering comprehensive caregiver training programs, including Home Health Aide (HHA) courses. Established as the first dedicated caregiver training academy in Lahore, HCA provides both theoretical and practical training to equip students with the necessary skills for caregiving roles. Their curriculum emphasizes a holistic approach, addressing the mental, emotional, and physical well-being of patients. The academy is located at 6-A, Zafar Ali Road, Gulberg V, Lahore

    Reply
  26. Grazyna on March 19, 2025 at 5:50 am

    I am really delighted to read this web site posts which
    carries lots of valuable information, thanks for providing these
    information.

    Feel free to visit my web page 우리카지노 에볼루션

    Reply
  27. wsparcieizmiana on March 19, 2025 at 5:51 am

    Psycholog online at “Wsparcie i Zmiana” offers personalized therapy for Polish-speaking clients, addressing anxiety, depression, life crises, and personal development, with experienced therapists providing confidential, evidence-based support remotely.

    Reply
  28. qwddqwqwd on March 19, 2025 at 5:55 am

    Skrota bilen Göteborg få 1 300 – 8000 kr, hämtning ingår. Mottagningsbevis erhålls. Har tillstånd från Länsstyrelsen för bärgning av bilskrot Skrota bilen Göteborg

    Reply
  30. dwqqdwwdq on March 19, 2025 at 6:02 am

    OLXTOTO adalah situs togel terpercaya di indonesia dengan prediksi jitu terbaik. Didukung dengan akses Login tercepat menggunakan link alternatif berkualitas. olxtoto togel

    Reply
  31. wqdqwdqwd on March 19, 2025 at 6:08 am

    OLXTOTO adalah situs login resmi yang menawarkan pengalaman bermain Bandar Toto Macau dengan diskon terbesar. Nikmati berbagai permainan menarik dengan peluang menang yang tinggi dan bonus menarik. olxtoto macau

    Reply
  35. slot gacor on March 19, 2025 at 6:27 am

    Looking for the best slot gacor site in 2025? This is where the BIG WINS happen! Join now!

    Reply
  36. sortoto viral on March 19, 2025 at 6:28 am

    The #1 sortoto viral site of 2025 is here! High RTP, fast payouts, and non-stop JACKPOTS!

    Reply
  37. sortoto on March 19, 2025 at 6:28 am

    Want to win real money? Play at the most trusted Sortoto site of 2025 – register now!

    Reply
  38. Fatima Javaid on March 19, 2025 at 6:29 am

    Bagi para penggemar permainan angka, menemukan situs toto yang terpercaya adalah hal yang sangat penting. Dengan akses ke berbagai jenis permainan menarik dan peluang kemenangan besar, situs toto menjadi pilihan utama bagi mereka yang ingin merasakan sensasi taruhan online. Selain itu, platform yang aman dan sistem transaksi yang cepat membuat pengalaman bermain semakin nyaman dan menyenangkan.

    Reply
  40. youtube to mp4 on March 19, 2025 at 6:33 am

    youtube to mp4 Great for saving YouTube videos.

    Reply
  41. aliuamir aliuad on March 19, 2025 at 6:37 am

    Thank you for some other informative website. The place else may just I get that kind of information written in such a perfect method? I have a venture that I am simply now running on, and I’ve been at the glance out for such info. https://www.jungheinrich.co.nz/

    Reply
  42. AlasaDogo on March 19, 2025 at 6:40 am

    Elevate your space with the serene presence of a Fibre Buddha Statue, blending spiritual essence with timeless elegance. These delicately crafted statues represent mindfulness and tranquility and make ideal spiritual elements for either meditation sessions or decorative purposes.
    While they differ from stone items fibre statues demonstrate combination of ancient artistic value with extended lifespan and reduced weight. The cultural heritage foundation of these items delivers serene positive energy to your environment. Establish an atmosphere of peace through a Buddha statue which expresses both skilled construction and serenity and refined elegance. Search for the ideal enlightenment symbol that will beautify your spiritual area.

    Reply
  45. Permanent Hair Removal in Dubai on March 19, 2025 at 7:01 am

    There are some attention-grabbing points in time in this article but I dont know if I see all of them middle to heart. There may be some validity however I will take hold opinion till I look into it further. Good article , thanks and we wish extra! Added to FeedBurner as well Permanent Hair Removal in Dubai

    Reply
  47. 8KBET on March 19, 2025 at 7:05 am

    Thanh vien mo tai khoan trong 72h va cung cap du thong tin cho Tải app 8KBET, se duoc tang qua free.

    Reply
  48. situs liga 2000 on March 19, 2025 at 7:07 am

    Thank you so much regarding giving me personally an update on this theme on your web page. Please realize that if a new post becomes available or if any modifications occur to the current publication, I would be considering reading a lot more and focusing on how to make good usage of those approaches you reveal. Thanks for your efforts and consideration of other individuals by making this site available. situs liga 2000

    Reply
  49. mwe on March 19, 2025 at 7:10 am

    hi was just seeing if you minded a comment. i like your website and the thme you picked is super. I will be back. olxtoto

    Reply
← Older Comments
Newer Comments →

Submit a Comment

Your email address will not be published. Required fields are marked *