AraBERT : Pre-training BERT for Arabic Language Understanding

by | Feb 28, 2020 | NLP | 49,215 comments

Authors: Wissam Antoun, Fady Baly, Hazem Hajj

 

AraBERT is an Arabic pretrained language model based on Google’s BERT architecture. AraBERT uses the same BERT-Base config. More details are available in the AraBERT PAPER and in the AraBERT Meetup

There is two versions of the model AraBERTv0.1 and AraBERTv1, with the difference being that AraBERTv1 uses pre-segmented text where prefixes and suffixes were split using the Farasa Segmenter.

The model was trained on ~70M sentences or ~23GB of Arabic text with ~3B words.

Source Code Repository: https://github.com/aub-mind/arabert
Paper: https://www.aclweb.org/anthology/2020.osact-1.2.pdf

 

Results (Accuracy)

We evaluate both AraBERT models on different downstream tasks and compare it to mBERT, and other state of the art models (To the extent of our knowledge). The Tasks were Sentiment Analysis on 6 different datasets (HARD, ASTD-Balanced, ArsenTD-Lev, LABR, ArSaS), Named Entity Recognition with the ANERcorp, and Arabic Question Answering on Arabic-SQuAD and ARCD

Task prev. SOTA mBERT AraBERTv0.1 AraBERTv1
HARD 95.7 ElJundi et.al. 95.7 96.2 96.1
ASTD 86.5 ElJundi et.al. 80.1 92.2 92.6
ArsenTD-Lev 52.4 ElJundi et.al. 51 58.9 59.4
AJGT 93 Dahou et.al. 83.6 93.1 93.8
LABR 87.5 Dahou et.al. 83 85.9 86.7
ANERcorp 81.7 (BiLSTM-CRF) 78.4 84.2 81.9
ARCD mBERT EM:34.2 F1: 61.3 EM:51.14 F1:82.13 EM:54.84 F1: 82.15

Model Weights and Vocab Download

Models AraBERTv0.1 AraBERTv1
TensorFlow Drive Link Drive Link
PyTorch Drive_Link Drive_Link

You can find the PyTorch models in HuggingFace’s Transformer Library under the aubmindlab username

If you used this model please cite us as:

@inproceedings{antoun2020arabert,
  title={AraBERT: Transformer-based Model for Arabic Language Understanding},
  author={Antoun, Wissam and Baly, Fady and Hajj, Hazem},
  booktitle={LREC 2020 Workshop Language Resources and Evaluation Conference 11--16 May 2020},
  pages={9}
}

Acknowledgments

Thanks to TensorFlow Research Cloud (TFRC) for the free access to Cloud TPUs, couldn’t have done it without this program, and to the AUB MIND Lab Members for the continous support. Also thanks to Yakshof and Assafir for data and storage access. Another thanks for Habib Rahal (https://www.behance.net/rahalhabib), for putting a face to AraBERT.

Contacts

Wissam Antoun: Linkedin | Twitter | Github | wfa07@mail.aub.edu | wissam.antoun@gmail.com

Fady Baly: Linkedin | Twitter | Github | fgb06@mail.aub.edu | baly.fady@gmail.com

We are looking for sponsors to train BERT-Large and other Transformer models, the sponsor only needs to cover to data storage and compute cost of the generating the pretraining data

49,215 Comments

← Older Comments
Newer Comments →
  1. primenewssite on March 3, 2025 at 8:18 pm

    Nice information, valuable and excellent design, as share good stuff with good ideas and concepts, lots of great information and inspiration, both of which I need, thanks to offer such a helpful information here. primenewssite

    Reply
  2. prousamagazine on March 3, 2025 at 8:33 pm

    I just got to this amazing site not long ago. I was actually captured with the piece of resources you have got here. Big thumbs up for making such wonderful blog page! prousamagazine

    Reply
  3. Amya Hansen on March 3, 2025 at 9:00 pm

    “FSIBlogs: The Ultimate Platform for Bloggers and Content Creators” is the go-to destination for individuals looking to enhance their blogging and content creation skills. This platform offers valuable resources, expert tips, and tools to help creators produce high-quality content, grow their audience, and stay ahead of industry trends. Whether you’re a beginner or an experienced blogger, fsiblogs provides the support and insights you need to succeed in the digital world and elevate your creative journey.

    Reply
  7. Wyatt Carter on March 3, 2025 at 10:44 pm

    강남 최고의 노래방, 강남가라오케! 고급스러운 시설과 다양한 음료 서비스로 잊지 못할 경험을 제공합니다. 강남가라오케

    Reply
  8. E-Signature Compliance on March 4, 2025 at 12:00 am

    E-Signature Compliance refers to the adherence to legal and regulatory requirements related to the use of electronic signatures. Different jurisdictions have different laws governing the use of e-signatures, such as the Electronic Signatures in Global and National Commerce Act (E-SIGN Act) in the United States and the eIDAS regulation in the European Union. E-Signature Compliance ensures that electronic signatures are legally binding and enforceable. This involves using secure e-signature solutions that provide authentication, encryption, and audit trails. Compliance is crucial for businesses to avoid legal disputes and ensure that their electronic transactions are valid and recognized by law.

    Reply
  9. praktiken der grünen produktion on March 4, 2025 at 12:21 am

    In einer Zeit, in der der technologische Fortschritt Branchen und Unternehmen verändert, ist es von entscheidender Bedeutung, durch den Einsatz der richtigen Software und kompetenten Beratungsdienste immer einen Schritt voraus zu sein. Innopulse, ein führender Anbieter innovativer Lösungen, revolutioniert den Geschäftsbetrieb mit seiner hochmodernen Software wie SignNTrack und Tresorly, kombiniert mit fachkundiger Beratung in den Bereichen digitale Transformation, Compliance, Cybersicherheit und vielem mehr. Diese Innovationen zielen nicht nur darauf ab, Geschäftsprozesse zu rationalisieren, sondern auch die ökologische Nachhaltigkeit durch Praktiken wie „praktiken der grünen produktion“ sicherzustellen.

    Reply
  10. seo on March 4, 2025 at 1:29 am

    U888 – Nền tảng giải trí hàng đầu nơi hội tụ những trải nghiệm thú vị và cơ hội chiến thắng lớn. Hãy tham gia U888 ngay để khám phá nhiều trò chơi đa dạng, tận hưởng tỷ lệ cược cạnh tranh và mở khóa những phần thưởng giá trị!U888

    Reply
  11. seo on March 4, 2025 at 1:37 am

    OK365 mang đến cho bạn một nền tảng giải trí trực tuyến với công nghệ hiện đại, giúp kết nối nhanh chóng và liền mạch. Dịch vụ tối ưu đảm bảo trải nghiệm mượt mà, không gián đoạn, giúp bạn tận hưởng những giây phút thư giãn trọn vẹn.OK365

    Reply
  15. https://79kinggg.com/ on March 4, 2025 at 1:58 am

    79King – https://79kinggg.com/ đã và đang khẳng định vị thế là một trong những nhà cái cá cược trực tuyến hàng đầu tại Việt Nam cũng như khu vực Châu Á. Nhằm tạo ra một môi trường giải trí đẳng cấp và phù hợp với xu hướng hiện đại, nền tảng này liên tục cải tiến và nâng cấp để đáp ứng trọn vẹn mọi nhu cầu của người chơi. Sự kết hợp giữa công nghệ tiên tiến, dịch vụ chuyên nghiệp và kho game đa dạng đã giúp thu hút hàng triệu thành viên tham gia, tạo nên một cộng đồng cá cược sôi động.

    Reply
  20. 23win on March 4, 2025 at 2:40 am

    Nguoi choi thanh cong mo tai khoan, chi can tap trung thiet lap giao dich toi thieu 50K la da du dieu kien rinh qua tai https://23win.broker/.

    Reply
  23. https://79kinggg.net/ on March 4, 2025 at 2:57 am

    79King – https://79kinggg.net/ nổi bật với hệ thống giao dịch nhanh chóng, hỗ trợ nạp rút tiền linh hoạt qua nhiều phương thức thanh toán. Đội ngũ chăm sóc khách hàng hoạt động 24/7, luôn sẵn sàng giải đáp thắc mắc và hỗ trợ người chơi một cách tận tình, chuyên nghiệp.

    Reply
  25. DURAJOINT FCL Couplings 125 on March 4, 2025 at 3:14 am

    Sangat menyukai cara Anda menyajikan informasi. Sangat mudah dipahami dan menarik. DURAJOINT FCL Couplings 125

    Reply
  26. DURAJOINT FCL Couplings 125 on March 4, 2025 at 3:22 am

    Sangat menyukai cara Anda menyajikan informasi. Sangat mudah dipahami dan menarik. DURAJOINT FCL Couplings 125

    Reply
  29. Garland on March 4, 2025 at 3:37 am

    With havin so much written content do you ever run into any problems of plagorism or copyright violation? My blog
    has a lot of completely unique content I’ve either created myself or outsourced but it appears a lot of it is popping it pin up all over the web without
    my permission. Do you know any solutions to help protect against content from being stolen? I’d definitely appreciate it.

    Reply
  30. DURAJOINT FCL Couplings 140 on March 4, 2025 at 3:53 am

    Terima kasih atas wawasan yang berharga yang Anda berikan di sini. Sangat membantu! DURAJOINT FCL Couplings 140

    Reply
  31. https://win5566.net/ on March 4, 2025 at 3:54 am

    WIN55 – https://win5566.net/ là lựa chọn hàng đầu dành cho những tín đồ cá cược trực tuyến, nơi mang đến thế giới giải trí đỉnh cao với hàng loạt trò chơi phong phú, tỷ lệ kèo hấp dẫn và tốc độ giao dịch nhanh chóng. Nhờ hệ thống bảo mật tiên tiến cùng dịch vụ khách hàng tận tâm, nền tảng này hứa hẹn sẽ tạo nên dấu ấn mạnh mẽ trong năm 2025.

    Reply
  32. Eduardo Harding on March 4, 2025 at 3:57 am

    Worms Zone Mod APK is a modified version of Worms Zone.io that offers unlimited money, unlocked skins, and an ad-free experience. It allows faster growth, enhanced controls, and special abilities to help players dominate the game. With this mod, you can enjoy a more exciting and competitive gameplay experience. wormzoneapk.com

    Reply
  33. DURAJOINT FCL Couplings 140 on March 4, 2025 at 4:02 am

    Terima kasih atas wawasan yang berharga yang Anda berikan di sini. Sangat membantu! DURAJOINT FCL Couplings 140

    Reply
  34. Bisma shakir on March 4, 2025 at 4:09 am

    I have a mission that I’m just now working on, and I have been at the look out for such information tinyzone

    Reply
  36. DURAJOINT FCL Couplings 160 on March 4, 2025 at 4:37 am

    Sangat menyukai pendekatan Anda dalam menangani masalah ini. Sangat praktis! DURAJOINT FCL Couplings 160

    Reply
  37. https://win5566.org/ on March 4, 2025 at 4:37 am

    WIN55 – https://win5566.org/ sở hữu kho trò chơi phong phú từ các nhà cung cấp hàng đầu thế giới, mang đến trải nghiệm cá cược đẳng cấp với tỷ lệ trả thưởng cao. Bên cạnh đó, các chương trình khuyến mãi liên tục được cập nhật, giúp người chơi có thêm nhiều cơ hội gia tăng lợi nhuận.

    Reply
  38. tara matka on March 4, 2025 at 4:39 am

    tara matkaMatka 420 is a term used in India to describe an illegal form of lottery or gaming associated with the Matka betting system. The “420” refers to a section of the Indian Penal Code dealing with fraud, highlighting its unlawful nature. Players bet on numbers, hoping to win large sums, but it remains banned in many areas due to its gaming risks.

    Reply
  39. pinkpalmpuff on March 4, 2025 at 4:51 am

    The Pink Palm Puff Hoodie is a cozy and trendy piece of outerwear featuring a soft, pastel pink color and playful puffed palm details. It combines comfort with a touch of whimsy, making it a stylish choice for casual outfits. Perfect for adding a fun, laid-back vibe to your wardrobe.
    Pink Palm Puff Hoddies

    Reply
  40. pinkpalmpuff on March 4, 2025 at 4:52 am

    The Pink Palm Puff Hoodie is a cozy and trendy piece of outerwear featuring a soft, pastel pink color and playful puffed palm details. It combines comfort with a touch of whimsy, making it a stylish choice for casual outfits. Perfect for adding a fun, laid-back vibe to your wardrobe.
    Pink Palm Puff Hoddies

    Reply
  41. https://ee-88.org/ on March 4, 2025 at 4:56 am

    EE88 – https://ee-88.org/ khẳng định vị thế nhà cái trực tuyến đỉnh cao năm 2025 với kho game đa dạng, tỷ lệ cược hấp dẫn và dịch vụ nạp rút siêu tốc. Nền tảng hiện đại, bảo mật tối ưu cùng ưu đãi cực khủng giúp người chơi tận hưởng trải nghiệm cá cược chuyên nghiệp và đẳng cấp nhất.

    Reply
  42. Tara Matka on March 4, 2025 at 4:58 am

    tara matka is an illegal lottery-style gaming game popular in India. Players place bets on numbers, hoping to win large sums through random draws. The game is part of the broader Matka gaming system, which originated in the 1960s. Though widely played, Tara Matka is banned in many regions due to its association with unregulated gaming activities.

    Reply
  43. conciertos en Madrid on March 4, 2025 at 5:06 am

    Its like you read my mind! You seem to know so much about this, like you wrote the book in it or something. I think that you can do with some pics to drive the message home a bit, but instead of that, this is magnificent blog. An excellent read. I will certainly be back. conciertos en Madrid

    Reply
  44. standing desks on March 4, 2025 at 5:23 am

    Well a lot of them sample isaac hayes and Maceo Parker. . Try the link below in the sources to see a list of a few that sample isaac hayes standing desks

    Reply
  45. shbet on March 4, 2025 at 5:27 am

    shbet – sân chơi cá cược uy tín, đa dạng trò chơi hấp dẫn, tỷ lệ thưởng cao, giao dịch nạp rút nhanh chóng, bảo mật an toàn. Tham gia ngay để trải nghiệm giải trí đỉnh cao và chinh phục những phần thưởng giá trị!

    Reply
  46. standing desks on March 4, 2025 at 5:34 am

    Well a lot of them sample isaac hayes and Maceo Parker. . Try the link below in the sources to see a list of a few that sample isaac hayes standing desks

    Reply
  47. aaliyan on March 4, 2025 at 5:34 am

    The detailed layout of the platform ensures that users can easily locate the information they need, improving overall usability. SV368

    Reply
  48. Cakhia TV on March 4, 2025 at 5:34 am

    Hãy đón xem Cakhia TV – kênh truyền hình trực tuyến hàng đầu, phát sóng trực tiếp các giải đấu danh giá như Ngoại Hạng Anh, Bundesliga, Ligue 1, Serie A, VFF Cup và nhiều sự kiện bóng đá hấp dẫn khác. Kênh cung cấp lịch thi đấu chi tiết, tin tức cập nhật, bảng xếp hạng chính xác, livescore liên tục và tỷ lệ kèo uy tín, đáp ứng nhu cầu của người hâm mộ và nhà cá cược. Truy cập garcimunoz.com để thưởng thức những trận cầu đỉnh cao, cùng trải nghiệm công nghệ số tiên tiến và những pha bóng “làm mưa làm gió” như một bữa tiệc giải trí không giới hạn!

    Reply
  49. aaliyan on March 4, 2025 at 5:35 am

    The engaging atmosphere of SV368 creates an enjoyable experience that encourages users to return and continue exploring its various features. SV368

    Reply
  50. AALIYAN KHAN on March 4, 2025 at 5:36 am

    SV368 features a visually appealing design with smooth navigation. The layout is structured in a way that makes it easy to find exactly what you’re looking for without unnecessary complications. SV368

    Reply
← Older Comments
Newer Comments →

Submit a Comment

Your email address will not be published. Required fields are marked *