AraBERT : Pre-training BERT for Arabic Language Understanding

by | Feb 28, 2020 | NLP | 49,350 comments

Authors: Wissam Antoun, Fady Baly, Hazem Hajj

 

AraBERT is an Arabic pretrained language model based on Google’s BERT architecture. AraBERT uses the same BERT-Base config. More details are available in the AraBERT PAPER and in the AraBERT Meetup

There is two versions of the model AraBERTv0.1 and AraBERTv1, with the difference being that AraBERTv1 uses pre-segmented text where prefixes and suffixes were split using the Farasa Segmenter.

The model was trained on ~70M sentences or ~23GB of Arabic text with ~3B words.

Source Code Repository: https://github.com/aub-mind/arabert
Paper: https://www.aclweb.org/anthology/2020.osact-1.2.pdf

 

Results (Accuracy)

We evaluate both AraBERT models on different downstream tasks and compare it to mBERT, and other state of the art models (To the extent of our knowledge). The Tasks were Sentiment Analysis on 6 different datasets (HARD, ASTD-Balanced, ArsenTD-Lev, LABR, ArSaS), Named Entity Recognition with the ANERcorp, and Arabic Question Answering on Arabic-SQuAD and ARCD

Task prev. SOTA mBERT AraBERTv0.1 AraBERTv1
HARD 95.7 ElJundi et.al. 95.7 96.2 96.1
ASTD 86.5 ElJundi et.al. 80.1 92.2 92.6
ArsenTD-Lev 52.4 ElJundi et.al. 51 58.9 59.4
AJGT 93 Dahou et.al. 83.6 93.1 93.8
LABR 87.5 Dahou et.al. 83 85.9 86.7
ANERcorp 81.7 (BiLSTM-CRF) 78.4 84.2 81.9
ARCD mBERT EM:34.2 F1: 61.3 EM:51.14 F1:82.13 EM:54.84 F1: 82.15

Model Weights and Vocab Download

Models AraBERTv0.1 AraBERTv1
TensorFlow Drive Link Drive Link
PyTorch Drive_Link Drive_Link

You can find the PyTorch models in HuggingFace’s Transformer Library under the aubmindlab username

If you used this model please cite us as:

@inproceedings{antoun2020arabert,
  title={AraBERT: Transformer-based Model for Arabic Language Understanding},
  author={Antoun, Wissam and Baly, Fady and Hajj, Hazem},
  booktitle={LREC 2020 Workshop Language Resources and Evaluation Conference 11--16 May 2020},
  pages={9}
}

Acknowledgments

Thanks to TensorFlow Research Cloud (TFRC) for the free access to Cloud TPUs, couldn’t have done it without this program, and to the AUB MIND Lab Members for the continous support. Also thanks to Yakshof and Assafir for data and storage access. Another thanks for Habib Rahal (https://www.behance.net/rahalhabib), for putting a face to AraBERT.

Contacts

Wissam Antoun: Linkedin | Twitter | Github | wfa07@mail.aub.edu | wissam.antoun@gmail.com

Fady Baly: Linkedin | Twitter | Github | fgb06@mail.aub.edu | baly.fady@gmail.com

We are looking for sponsors to train BERT-Large and other Transformer models, the sponsor only needs to cover to data storage and compute cost of the generating the pretraining data

49,350 Comments

← Older Comments
Newer Comments →
  1. https://thanbai8.com/ on March 8, 2025 at 5:43 am

    THANBAI88 – https://thanbai8.com/ mang đến thế giới game bài online đẳng cấp, với trải nghiệm thú vị, giao diện đẹp mắt, hệ thống bảo mật an toàn và tỷ lệ thưởng cực kỳ hấp dẫn. Hãy đăng ký ngay để tận hưởng những giây phút giải trí bất tận và có cơ hội săn những giải thưởng giá trị!

    Reply
  2. aliuamir aliuad on March 8, 2025 at 5:45 am

    Great job for publishing such a beneficial web site. Your web log isn’t only useful but it is additionally really creative too. There tend to be not many people who can certainly write not so simple posts that artistically. Continue the nice writing KAWI777

    Reply
  3. giftgenerator.info on March 8, 2025 at 5:46 am

    giftgenerator.info is a fun way to discover unique gift ideas! Spin the roulette and get personalized suggestions like a custom star map, making gift shopping exciting and effortless. Try it now! 🎁✨

    Reply
  4. aliuamiraliuad on March 8, 2025 at 6:07 am

    Just admiring your work and wondering how you managed this blog so well. It’s so remarkable that I can’t afford to not go through this valuable information whenever I surf the internet! KAWI777

    Reply
  5. tayyaba khan on March 8, 2025 at 6:07 am

    Very efficiently written information. It will be beneficial to anybody who utilizes it, including me. Keep up the good work. For sure i will check out more posts. This site seems to get a good amount of visitors. TESLATOTO

    Reply
  7. fariha on March 8, 2025 at 6:14 am

    I definitely enjoying every little bit of it. It is a great website and nice share. I want to thank you. Good job! You guys do a great blog, and have some great contents. Keep up the good work. black scatter slot

    Reply
  8. Sport Physio near me on March 8, 2025 at 6:17 am

    Sport Physio near me MyoFitness provides expert physiotherapy services, grounded in evidence-based practice, to address a wide range of musculoskeletal and neurological conditions. Located in Yarraville and North Melbourne, our experienced physiotherapists are dedicated to improving physical function, supporting recovery, and promoting long-term health.

    Reply
  10. order tesla cybertruck accessories online on March 8, 2025 at 6:18 am

    order tesla cybertruck accessories online from our premium selection! Enhance performance, style, and comfort with custom-fit floor mats, tech gadgets, and sleek upgrades—crafted for durability and a perfect fit.

    Reply
  11. HAMMAD on March 8, 2025 at 6:22 am

    I require the majority of the records, Have a preference for seriously favored, We really wish for addiitional information and facts over it, while it is definitely rather great., L8rs used only for documenting. ufa100 ทางเข้า

    Reply
  14. CEO Nguyễn Khổng Kìu on March 8, 2025 at 6:35 am

    Author: Nguyễn Khổng Kìu Ngày sinh: 13/05/1992 Chức vụ: CEO tại 12bet.bz Phone: 0985764712 Email: ceonguyenkhongkiu@gmail. com Địa chỉ: 60 Huỳnh Tấn Phát, Quận 7, Hồ Chí Minh, Vietnam. Kinh nghiệm: Với hơn 13 năm hoạt động trong lĩnh vực cá cược trực tuyến, Nguyễn Khổng Kìu hiện là Giám đốc Điều hành của 12bet.bz. Trang web này là cổng đăng ký và đăng nhập chính thức của nhà cái 12bet tại thị trường Việt Nam.

    https://12bet.bz/author/ceo-nguyen-khong-kiu/

    Reply
  16. byirfanusta8309 on March 8, 2025 at 6:45 am

    EDM88 adalah platform game online terbaik yang membawa Anda ke dunia penuh keseruan dan peluang besar! Di sini, setiap detik yang Anda habiskan tidak hanya akan membawa pengalaman bermain yang tak terlupakan, tetapi juga peluang untuk meraih cuan besar! EDM88

    Reply
  18. alloy wheels on March 8, 2025 at 6:50 am

    Upgrade your vehicle with high-quality alloy wheels for a sleek look and superior performance. Shop online by model, enjoy real-time stock availability, discounted prices, and a full fitment guarantee.

    Reply
  22. tayyaba khan on March 8, 2025 at 7:00 am

    I think this is an informative post and it is very useful and knowledgeable. therefore, I would like to thank you for the efforts you have made in writing this article. togel4d

    Reply
  23. tayyaba khan on March 8, 2025 at 7:01 am

    We have sell some products of different custom boxes.it is very useful and very low price please visits this site thanks and please share this post with your friends. alexistogel

    Reply
  24. .vfbvfvfv on March 8, 2025 at 7:01 am

    hi!,I love your writing very much! share we keep in touch extra approximately your post on AOL? I need an expert in this space to resolve my problem. May be that is you! Taking a look forward to look you. https://ifilmyoga.com

    Reply
  26. fariha on March 8, 2025 at 7:06 am

    It was wondering if I could use this write-up on my other website, I will link it back to your website though.Great Thanks. alexistogel

    Reply
  27. fariha on March 8, 2025 at 7:08 am

    Yes i am totally agreed with this article and i just want say that this article is very nice and very informative article.I will make sure to be reading your blog more. You made a good point but I can’t help but wonder, what about the other side? !!!!!!Thanks situs slot gacor

    Reply
  28. tayyaba khan on March 8, 2025 at 7:10 am

    Very informative post! There is a lot of information here that can help any business get started with a successful social networking campaign. bandar slot gacor

    Reply
  30. 에볼루션 카지노 on March 8, 2025 at 7:16 am

    에볼루션 카지노는 다양한 기능과 서비스를 통해 2024년에도 최고의 라이브 카지노 플랫폼으로 자리잡고 있습니다. 다양한 게임과 매끄러운 사용자 경험, 신뢰할 수 있는 보안으로 누구나 즐길 수 있는 카지노의 새로운 패러다임을 제시합니다.

    Reply
  31. CEO Trần Hủi on March 8, 2025 at 7:17 am

    Tên: CEO Trần Hủi Ngày sinh: 16/04/1991 Chức vụ: Giám đốc Điều hành tại typhu88.promo Số điện thoại: 0976523613 Email: ceotranhui@gmail. com Địa chỉ: 55 Nguyễn Thị Thập, Quận 7, Hồ Chí Minh, Việt Nam. Kinh nghiệm: CEO Trần Hủi có hơn 12 năm kinh nghiệm trong lĩnh vực cá cược trực tuyến, và hiện đang là Giám đốc Điều hành của typhu88.promo, một trang web cung cấp đường dẫn đăng ký và đăng nhập chính thức của nhà cái typhu88 tại thị trường Việt Nam. https://typhu88.promo/author/ceo-tran-hui/

    Reply
  32. alloys on March 8, 2025 at 7:18 am

    Discover high-quality alloys with guaranteed fitment for your vehicle. Shop alloy wheels and tyre packages online with real-time stock updates, great discounts, and fast, secure delivery. Upgrade your ride effortlessly today!

    Reply
  33. Conveyor Belts 2 Ply BW 400 on March 8, 2025 at 7:25 am

    Saya suka bagaimana Anda menyajikan fakta dengan cara yang menarik. Sangat menghibur! Conveyor Belts 2 Ply BW 400

    Reply
  34. Classychoice.pk on March 8, 2025 at 7:35 am

    Discover the finest imported skincare products in Pakistan at ClassyChoice.pk. Shop our curated collection of top global brands, offering high-quality, authentic skincare solutions tailored to your skin’s needs. Elevate your skincare routine with products that deliver visible results. Classychoice.pk

    Reply
  35. tayyaba khan on March 8, 2025 at 7:38 am

    I was reading some of your content on this website and I conceive this internet site is really informative ! Keep on putting up. togel4d register

    Reply
  36. rims on March 8, 2025 at 7:52 am

    Upgrade your vehicle with high-quality rims designed for perfect fitment. Browse our wide selection of alloy wheels and tyre packages, ensuring compatibility, great prices, and secure fast delivery. Shop confidently online!

    Reply
  37. Conveyor Belts 2 Ply BW 400 on March 8, 2025 at 7:58 am

    Saya suka bagaimana Anda menyajikan fakta dengan cara yang menarik. Sangat menghibur! Conveyor Belts 2 Ply BW 400

    Reply
  38. tayyaba khan on March 8, 2025 at 8:05 am

    Very efficiently written information. It will be beneficial to anybody who utilizes it, including me. Keep up the good work. For sure i will check out more posts. This site seems to get a good amount of visitors. olxtoto

    Reply
  39. fariha on March 8, 2025 at 8:05 am

    I definitely enjoying every little bit of it. It is a great website and nice share. I want to thank you. Good job! You guys do a great blog, and have some great contents. Keep up the good work. olxtoto resmi

    Reply
  40. Japanese on March 8, 2025 at 8:08 am

    Hey! Do you know if they make any plugins to help with SEO? I’m trying to get my blog to rank for some targeted keywords but I’m not seeing very good results. If you know of any please share. Thanks! BUY ZOPICLONE TABLETS UK

    Reply
  41. tayyaba khan on March 8, 2025 at 8:10 am

    We have sell some products of different custom boxes.it is very useful and very low price please visits this site thanks and please share this post with your friends. セブ留学

    Reply
  42. tayyaba khan on March 8, 2025 at 8:12 am

    We have sell some products of different custom boxes.it is very useful and very low price please visits this site thanks and please share this post with your friends. olxtoto link

    Reply
  44. Hubet on March 8, 2025 at 8:14 am

    Thuong nap dau mang den co hoi nhan uu dai len den 8888K danh cho thanh vien moi tại https://hubet.bar/ tuyet voi nay.

    Reply
  45. asupre8294 on March 8, 2025 at 8:17 am

    HARUM189 hadir sebagai platform hiburan online terpercaya se-Asia, menawarkan berbagai permainan menarik, sistem keamanan tinggi, serta layanan profesional yang memastikan kenyamanan dan kepuasan para pemain. HARUM189

    Reply
  46. icryptox.com future on March 8, 2025 at 8:19 am

    Thanks for the nice blog. It was very useful for me. I m happy I found this blog. Thank you for sharing with us,I too always learn something new from your post. icryptox.com future

    Reply
  47. hometown8189 on March 8, 2025 at 8:25 am

    BOLABET189 adalah situs gaming berlisensi resmi yang aman dan terpercaya. Dikenal sebagai platform terfavorit karena situs ini menawarkan pengalaman bermain terbaik dengan sistem keamanan canggih, transaksi super cepat, dan pelayanan nonstop 24 jam. Daftar dan login sekarang untuk merasakan sensasi bermain di situs gaming aman dan berlisensi resmi serta nikmati keuntungan maksimal dengan jaminan keamanan dan kenyamanan terbaik disini! BOLABET189

    Reply
  48. how to say kiolopobgofit on March 8, 2025 at 8:28 am

    Thanks for the nice blog. It was very useful for me. I m happy I found this blog. Thank you for sharing with us,I too always learn something new from your post. how to say kiolopobgofit

    Reply
  50. money6x.com save money on March 8, 2025 at 8:37 am

    Thanks for the nice blog. It was very useful for me. I m happy I found this blog. Thank you for sharing with us,I too always learn something new from your post. money6x.com save money

    Reply
← Older Comments
Newer Comments →

Submit a Comment

Your email address will not be published. Required fields are marked *