Authors: Obeida ElJundi, Wissam Antoun, Nour El Droubi, Hazem Hajj, Wassim El-Hajj, Khaled Shaban
Paper: https://www.aclweb.org/anthology/W19-4608
Source Code Repository: https://github.com/aub-mind/hULMonA
Project: http://oma-project.com/
Introduction
Recent state-of-the-art models in NLP (e.g., BERT, GPT, ULMFiT) utilize transfer learning by pre-training a language model on large corpus and then fine-tuning it on any downstream task. We developed the first Arabic specific universal language model, hULMonA, that can be fine-tuned for almost any Arabic text classification task. We evaluated hULMonA on Sentiment Analysis and achieved state-of-the-art on 4 Arabic datasets. hULMonA consists of three main stages:
1. General domain hULMonA pretraining
To capture the various properties of the Arabic language, we train the SOTA-ish language model AWD-LSTM on all Arabic Wikipedia.
This step is time-consuming, but it should be done only once. We publish our pre-trained model, and it is available in the models directory. To check the implementation details, or to pre-train your own LM, check build_arabic_language_model.ipynb
2. Target task hULMonA fine-tuning
The target task data (e.g., Twitter) will likely come from a different distribution than the general-domain data (Wikipedia). Therefore, fine-tuning the pretrained general-domain LM on the target task data is necessary for the LM to adapt to the new textual properties (e.g., dialects).
To fine-tune the pre-trained hULMonA on your own dataset, please check fine_tune_LM.ipynb
3. Target task classification
Finally, for downstream task classification, we augment the fine-tuned hULMonA with two fully connected layers with ReLU and Softmax activations respectively. Implementation details can be found here: fine_tune_LM.ipynb
How do I cite hULMonA?
Please cite this paper:
@inproceedings{eljundi2019hulmona,
title={hULMonA: The Universal Language Model in Arabic},
author={ElJundi, Obeida and Antoun, Wissam and El Droubi, Nour and Hajj, Hazem and El-Hajj, Wassim and Shaban, Khaled},
booktitle={Proceedings of the Fourth Arabic Natural Language Processing Workshop},
pages={68--77},
year={2019}
}
Contact information
For help, issues, or personal communication related to using hULMonA, please contact Obeida ElJundi (oae15@mail.aub.edu), Wissam Antoun (wfa07@mail.aub.edu), or Nour El Droubi (ngd02@mail.aub.edu)
Nice post . Thanks for, writing on my blog page mate. I will message you some time! I didnt know that! halloween zentai suit
I’m absolutely thrilled to share my excitement about this wonderful content! After reading so many glowing comments and personal stories here, I felt compelled to add my own. Following a recent engagement, I decided to explore SEVlaser, thanks to a recommendation from a friend. Reserving time for this was the best gift I could give myself. The experience has been truly fantastic. For those interested, check out Laser Hair Removal in Glendale at this link.
I’m impressed, I have to admit. Really rarely will i encounter a blog that’s both educative and entertaining, and without a doubt, you may have hit the nail to the head. Your concept is outstanding; the thing is something too few consumers are speaking intelligently about. We are very happy i always stumbled across this in my seek out something in regards to this. venta de flejes en mexico
Youre so cool! I dont suppose Ive read anything this way before. So nice to locate somebody by original applying for grants this subject. realy thank you for beginning this up. this web site is one area that is needed over the internet, someone if we do originality. beneficial problem for bringing interesting things towards the internet! erotic massage palma
I am lucky that I discovered this web site, just the right info that I was looking for! SOFT DRINK MANUFACTIRNG PLANT
Perfect work you have done, this web site is really cool with good information. catering services in London
I rattling glad to find this website on bing, just what I was looking for as well saved to fav. opp袋
Pretty nice post. I just stumbled upon your blog and wanted to say that I have really enjoyed browsing your blog posts. In any case I’ll be subscribing to your feed and I hope you write again soon. 背心袋
Hello! I just want to give a enormous thumbs up for any wonderful information you have here during this post. I’ll be returning to your blog site for much more soon. 真空袋
This is a right blog for would like to find out about this topic. You realize a lot its almost challenging to argue along (not that I personally would want…HaHa). You actually put the latest spin with a topic thats been discussed for a long time. Wonderful stuff, just fantastic! shopping bag
Hello! I simply would like to give a huge thumbs up for the great info you’ve here on this post. I might be coming back to your weblog for more soon. 二手電腦回收
If you happen to acquiring a substantial repayment, you have to keep in mind what quantity of money you could be deducting coming from paydays or maybe spending money on for quarterly income taxes. 手機回收價格表
Hmm it seems like your site ate my first comment (it was extremely long) so I guess I’ll just sum it up what I submitted and say, I’m thoroughly enjoying your blog. I too am an aspiring blog writer but I’m still new to everything. Do you have any tips for rookie blog writers? I’d certainly appreciate it. 手機回收
Hi my loved one! I want to say that this article is amazing, great written and include almost all vital infos. I would like to peer more posts like this . 魔術表演
I really want to thank you for yet another great informative post, I’m a loyal visitor to this blog and I can’t say how much valuable tips I’ve learned from reading your content. I really appreciate all the hard work you put into this great blog. 到校興趣班
hey you should get social media plugin. was looking for the ‘like’ button but couldn’t find it. 魔術
I absolutely agree with everything you have explained. In reality, I browsed throughout your additional content articles and I do think that you’re completely correct. Great job with this website. 魔術課程
I do not even know how I ended up here, but I believed this submit used to be great. I don’t recognize who you might be but certainly you’re going to a well-known blogger should you are not already Cheers! power screwdriver
I believe, the salesperson need: a great outer photo, routine; keep in mind be happy; usage of big eyes; get experience to speak to your manners. mug warmer for desk
Hello, you used to write fantastic articles, but the last several posts have been kinda lackluster… I miss your great articles. Past several posts are just a little out of track! furniture rissers
What a fantabulous post this has been. Never seen this kind of useful post. I am grateful to you and expect more number of posts like these. Thank you very much. INSTASLOT88
Judging by the way you write, you seem like a professional writer . Halloween Pumpkin Night Light
Pretty section of content. I just stumbled upon your blog and in accession capital to assert that I acquire actually enjoyed account your blog posts. Anyway I’ll be subscribing to your augment and even I achievement you access consistently fast. MINERAL WATER BOTTLING PLANT MANUFACTURER
I enjoy you because of all of your labor on this blog. My niece loves going through investigations and it is easy to understand why. We know all relating to the dynamic form you offer sensible things through the blog and in addition inspire participation from others on this concept plus our favorite child is actually becoming educated so much. Take pleasure in the rest of the new year. You’re the one performing a useful job. 減肥藥
I know your expertise on this. I must say we should have an online discussion on this. Writing only comments will close the discussion straight away! And will restrict the benefits from this information. togel singapore
I impressed, I need to say. Really not often do I encounter a blog that each educative and entertaining, and let me tell you, you have hit the nail on the head. Your concept is outstanding; the problem is something that not sufficient persons are speaking intelligently about. I’m very comfortable that I stumbled across this in my seek for one thing referring to this. Roofers Albany NY
Recently, I didnt give plenty of consideration to making comments on site page articles and have placed comments even much less. Reading via your enjoyable post, will help me to do so sometimes. Team Headshot photographers Miami, FL
Just a fast hello and also to thank you for discussing your ideas on this page. I wound up in your weblog right after researching physical fitness connected issues on Yahoo… guess I lost track of what I had been performing! Anyway I’ll be back once again inside the future to examine out your blogposts down the road. Thanks! Additions Renovation Company Lakeway TX
Pretty nice post. I recently stumbled upon your blog and wished to state that I’ve actually appreciated searching your blog content. After all I will be registering to your own nourish and i also we imagine you compose once more quickly! 千葉県 塗装業者
glad to find this website on bing, just what I was looking for as well saved California Healthcare
This information is beneficial for those who need it. I hope you will make many more posts like this.
FlyingTogether UAL
Such a fascinating article on a lovely day like today; thank you for your efforts in creating such a beneficial article starbucks secret menu frappuccino
I would really like you to become a guest poster on my blog.–;’- Journey
Hey dude” what kind of wordpress theme are you using? i want it to use on my blog too . 羅氏鮮
very good put up, i certainly love this web site, keep on it Plumber Durham
Hi there, Can I download this post photograph and employ it on my personal blogging site? website development services
Your writing has a distinct combination of professionalism and relatability. ยี่ กี รวย
I undoubtedly did not know that. Learnt some thing new today! Thanks for that. NUSTABET
Very valuable information, so i can search for things more easily here หวย ลาว สมัคร
Thank you for your tireless pursuit of excellence in your writing.magnum 4d วัน นี้
Congratulations on having one of the most sophisticated blogs Ive come across in some time! Its just incredible how much you can take away from something simply because of how visually beautiful it is. Youve put together a great blog space –great graphics, videos, layout. This is definitely a must-see blog. Moroccan Atlas carpets
Your writing resonated with me on a personal level, and I am thankful for the connection it fostered.
ติดต่อสอบหวยบี
Classical music is always the best, it is relaxing and very rich in melody;; Big Blue Restoration
Just want to say your article is as astonishing. The clarity in your post is simply nice and i could assume you’re an expert on this subject. Well with your permission let me to grab your feed to keep updated with forthcoming post. Thanks a million and please keep up the gratifying work. Ironbound Containers
I do believe this is an excellent website. I stumbled upon it I may revisit once again since I saved as a favorite it.
Spotifystats
Oh my goodness! a tremendous article dude. Thank you Nevertheless I’m experiencing concern with ur rss . Don’t know why Unable to subscribe to it. Is there anyone getting equivalent rss downside? Anybody who is aware of kindly respond. Thnkx FintechZoom
I’d have to seek advice from you here. Which is not some thing Which i do! I like reading a post that can make people feel. Also, thanks for allowing me to comment! champion generator covers
WellStar MyChart Login is a very easy, safe and time saving way to manage your healthcare needs online. With MyChart WellStar login, you have the ability to access your health record anytime, anyplace.
Wellstar hospital patient portal
This guide has offered everything you need to get around the platform effortlessly whether you are signing up for the first time, logging in or troubleshooting common issues. Make the most of online medical care and improve your healthcare by using WellStar MyChart Login today.
I like the way you made this article. Short but juicy. Thanks for sharing!
ProsperidadSocial
Obviously like your web-site but you have to check the spelling on quite a few of your posts. Several of them are rife with spelling issues and I find it very bothersome to tell the truth nevertheless I’ll certainly come back again. rebirth ro download