Authors: Wissam Antoun, Fady Baly, Hazem Hajj
AraBERT is an Arabic pretrained language model based on Google’s BERT architecture. AraBERT uses the same BERT-Base config. More details are available in the AraBERT PAPER and in the AraBERT Meetup
There is two versions of the model AraBERTv0.1 and AraBERTv1, with the difference being that AraBERTv1 uses pre-segmented text where prefixes and suffixes were split using the Farasa Segmenter.
The model was trained on ~70M sentences or ~23GB of Arabic text with ~3B words.
Source Code Repository: https://github.com/aub-mind/arabert
Paper: https://www.aclweb.org/anthology/2020.osact-1.2.pdf
Results (Accuracy)
We evaluate both AraBERT models on different downstream tasks and compare it to mBERT, and other state of the art models (To the extent of our knowledge). The Tasks were Sentiment Analysis on 6 different datasets (HARD, ASTD-Balanced, ArsenTD-Lev, LABR, ArSaS), Named Entity Recognition with the ANERcorp, and Arabic Question Answering on Arabic-SQuAD and ARCD
| Task | prev. SOTA | mBERT | AraBERTv0.1 | AraBERTv1 |
|---|---|---|---|---|
| HARD | 95.7 ElJundi et.al. | 95.7 | 96.2 | 96.1 |
| ASTD | 86.5 ElJundi et.al. | 80.1 | 92.2 | 92.6 |
| ArsenTD-Lev | 52.4 ElJundi et.al. | 51 | 58.9 | 59.4 |
| AJGT | 93 Dahou et.al. | 83.6 | 93.1 | 93.8 |
| LABR | 87.5 Dahou et.al. | 83 | 85.9 | 86.7 |
| ANERcorp | 81.7 (BiLSTM-CRF) | 78.4 | 84.2 | 81.9 |
| ARCD | mBERT | EM:34.2 F1: 61.3 | EM:51.14 F1:82.13 | EM:54.84 F1: 82.15 |
Model Weights and Vocab Download
| Models | AraBERTv0.1 | AraBERTv1 |
|---|---|---|
| TensorFlow | Drive Link | Drive Link |
| PyTorch | Drive_Link | Drive_Link |
You can find the PyTorch models in HuggingFace’s Transformer Library under the aubmindlab username
If you used this model please cite us as:
@inproceedings{antoun2020arabert,
title={AraBERT: Transformer-based Model for Arabic Language Understanding},
author={Antoun, Wissam and Baly, Fady and Hajj, Hazem},
booktitle={LREC 2020 Workshop Language Resources and Evaluation Conference 11--16 May 2020},
pages={9}
}
Acknowledgments
Thanks to TensorFlow Research Cloud (TFRC) for the free access to Cloud TPUs, couldn’t have done it without this program, and to the AUB MIND Lab Members for the continous support. Also thanks to Yakshof and Assafir for data and storage access. Another thanks for Habib Rahal (https://www.behance.net/rahalhabib), for putting a face to AraBERT.
Contacts
Wissam Antoun: Linkedin | Twitter | Github | wfa07@mail.aub.edu | wissam.antoun@gmail.com
Fady Baly: Linkedin | Twitter | Github | fgb06@mail.aub.edu | baly.fady@gmail.com
We are looking for sponsors to train BERT-Large and other Transformer models, the sponsor only needs to cover to data storage and compute cost of the generating the pretraining data
This post is very simple to read and appreciate without leaving any details out. Great work!Modsterra
This is my first time i visit here. I found so many entertaining stuff in your blog, especially its discussion. From the tons of comments on your articles, I guess I am not the only one having all the leisure here! Keep up the good work. I have been meaning to write something like this on my website and you have given me an idea. mommyandlove
Managing multiple Reddit accounts or handling frequent posts can be time-consuming, but with a Reddit post scheduler, this process becomes much more efficient. A Reddit post scheduler allows you to plan and schedule your posts in advance, ensuring that your content is shared at the right times to reach your target audience. This tool is perfect for businesses, social media managers, or individuals who want to maintain consistent activity on Reddit without having to manually post every time. By automating your posts, you can focus more on engagement and less on the logistics.
You completed certain reliable points there. I did a search on the subject and found nearly all persons will agree with your blog.work
I am a new user of this site so here i saw multiple articles and posts posted by this site,I curious more interest in some of them hope you will give more information on this topics in your next articles.good
This is my first time i visit here. I found so many entertaining stuff in your blog, especially its discussion. From the tons of comments on your articles, I guess I am not the only one having all the leisure here! Keep up the good work. I have been meaning to write something like this on my website and you have given me an idea. crypto fintechzoom
I am a new user of this site so here i saw multiple articles and posts posted by this site,I curious more interest in some of them hope you will give more information on this topics in your next articles.Gramhir
I absolutely love your website.. Great colors & theme. Did you
create this amazing site yourself? Please reply back as I’m
planning to create my own website and would love to
know where you got this from or what the theme is called.
Thanks!
Also visit my web-site – bolagila asia
Glad to chat your blog, I seem to be forward to more reliable articles and I think we all wish to thank so many good articles, blog to share with us.Recurbate
Glad to chat your blog, I seem to be forward to more reliable articles and I think we all wish to thank so many good articles, blog to share with us.youtube to mp4
I recently came across your article and have been reading along. I want to express my admiration of your writing skill and ability to make readers read from the beginning to the end. I would like to read newer posts and to share my thoughts with you. how i sleep at night knowing l’m failing all my cl – tymoff
This is my first time i visit here. I found so many entertaining stuff in your blog, especially its discussion. From the tons of comments on your articles, I guess I am not the only one having all the leisure here! Keep up the good work. I have been meaning to write something like this on my website and you have given me an idea. poorvika mobiles gul
I have express a few of the articles on your website now, and I really like your style of blogging. I added it to my favorite’s blog site list and will be checking back soon…Hypackel
With so many books and articles coming up to give gateway to make-money-online field and confusing reader even more on the actual way of earning money, wellhealthorganic.com protein
This is my first time i visit here. I found so many entertaining stuff in your blog, especially its discussion. From the tons of comments on your articles, I guess I am not the only one having all the leisure here! Keep up the good work. I have been meaning to write something like this on my website and you have given me an idea. get_ready_bell:client_pulse
Discover exclusive savings of up to 10 on Always Do What You Should Do Joggers at official Store ADWYSD Get a discount on this product adwysd
kingsbet app – Suggests users looking for a Kingsbet mobile application. kingsbet app
I have express a few of the articles on your website now, and I really like your style of blogging. I added it to my favorite’s blog site list and will be checking back soon… thespark shop batman style wireless bluetooth earbuds
You actually make it look so easy with your performance but I find this matter to be actually something which I think I would never comprehend. It seems too complicated and extremely broad for me. I’m looking forward for your next post, I’ll try to get the hang of it! wellhealthorganic.com : remove dark spots on face tang – lemon juice
I have express a few of the articles on your website now, and I really like your style of blogging. I added it to my favorite’s blog site list and will be checking back soon… chosenviber.net
I agree with your points , great post. judi dadu