{"647819":{"#nid":"647819","#data":{"type":"news","title":"New Training Algorithm for Faster Deployment","body":[{"value":"\u003Cp\u003EMachine learning (ML) is the future of computing, but it\u0026rsquo;s still fairly inaccessible to many developers without ML expertise or deep pockets. Usually, developers need to train ML models for a wide variety of deployment targets while keeping in mind the hardware constraints. Training multiple models this way is a slow, expensive task, but Georgia Tech researchers have come up with a way to make it faster and cheaper.\u003C\/p\u003E\r\n\r\n\u003Cp\u003ECompOFA is an algorithm that trains hundreds of models simultaneously and makes this process inexpensive by identifying and focusing on the most efficient possible models.\u003C\/p\u003E\r\n\r\n\u003Cp\u003E\u0026ldquo;This research is definitely in the spirit of democratizing ML,\u0026rdquo; School of Computer Science (SCS) Assistant Professor \u003Ca href=\u0022https:\/\/www.cc.gatech.edu\/~atumanov\/\u0022\u003E\u003Cstrong\u003EAlexey Tumanov\u003C\/strong\u003E\u003C\/a\u003E said. \u0026ldquo;Only large companies with resources can afford to do research like this. Usually, for state of the art accuracy, you need money, a lot of GPUs at your disposal, or a lot of time.\u0026rdquo;\u003C\/p\u003E\r\n\r\n\u003Cp\u003E\u003Cstrong\u003EChallenges with Training Multiple Models\u003C\/strong\u003E\u003C\/p\u003E\r\n\r\n\u003Cp\u003ETraditionally, developers have to design and train ML models for each platform on which they want to deploy their application. This is a slow, costly process, requiring ML experts to put in several days\u0026rsquo; worth of computation on expensive hardware. Techniques like Neural Architecture Search (NAS) aim to automatically find good ML models but are even more resource intensive.\u003C\/p\u003E\r\n\r\n\u003Cp\u003EMore recently, weight-sharing training algorithms were proposed to produce trillions of models during a query. However, only some models in this large search space can be \u0026ldquo;good\u0026rdquo; \u0026ndash; the vast majority of remaining models are inefficient and thus waste computation.\u003C\/p\u003E\r\n\r\n\u003Cp\u003EThe researchers realized it\u0026rsquo;s possible to extract just the optimal models from this search space, \u0026nbsp;giving the most accurate models at a given latency target.\u003C\/p\u003E\r\n\r\n\u003Cp\u003E\u0026ldquo;Our observation is it\u0026rsquo;s not really necessary for specialists to spend time, resources, and effort to train this many architecture candidates if there is a high probability they won\u0026rsquo;t be optimal,\u0026rdquo; Tumanov said. \u0026ldquo;If we reduce the suboptimal space, we can come up with a faster training procedure.\u0026rdquo;\u003C\/p\u003E\r\n\r\n\u003Cp\u003ETheir second key observation is that the design space, or a collection of hundreds of ML models, doesn\u0026rsquo;t need to be as dense. Rather, the researchers only need to pick the models that are sufficiently different in their size \u0026mdash;any smaller differences are too fine-grained to be distinguishable \u0026mdash; a systems insight leveraged to further reduce computation.\u003C\/p\u003E\r\n\r\n\u003Cp\u003E\u003Cstrong\u003EPruning the Architecture\u003C\/strong\u003E\u003C\/p\u003E\r\n\r\n\u003Cp\u003EOnce optimal models are identified and extracted, the architecture space is pruned. This makes the training speed 2x faster and search speed 200x faster. What originally would take four hours now takes 70 seconds. CompOFA also halves the cost and CO\u003Csub\u003E2\u003C\/sub\u003E emissions over previous state-of-the-art methods.\u003C\/p\u003E\r\n\r\n\u003Cp\u003E\u0026ldquo;You can remove models safely while keeping performance objectives,\u0026rdquo; SCS master\u0026rsquo;s student \u003Ca href=\u0022https:\/\/sahnimanas.github.io\/\u0022\u003E\u003Cstrong\u003EManas Sahni\u003C\/strong\u003E\u003C\/a\u003E said. \u0026ldquo;We receive results that are just as good at half the time to train the model.\u0026rdquo;\u003C\/p\u003E\r\n\r\n\u003Cp\u003ECompOFA can produce a family of models. While a lot of research in this area focuses on searching for and training a single architecture, the researchers wanted to produce a family of models that can run simultaneously, saving costs.\u003C\/p\u003E\r\n\r\n\u003Cp\u003EThe researchers presented at the International Conference on Learning Representations. Tumanov and Sahni wrote the paper, \u003Ca href=\u0022https:\/\/openreview.net\/pdf?id=IgIk8RRT-Z\u0022\u003E\u003Cstrong\u003E\u003Cem\u003ECompOFA \u0026ndash; Compound Once-For-All Networks for Faster Multi-Platform Deployment\u003C\/em\u003E\u003C\/strong\u003E\u003C\/a\u003E\u003Cstrong\u003E\u003Cem\u003E, \u003C\/em\u003E\u003C\/strong\u003Ewith SCS master\u0026rsquo;s student \u003Ca href=\u0022https:\/\/github.com\/shreyavarshini\u0022\u003E\u003Cstrong\u003EShreya Varshini\u003C\/strong\u003E\u003C\/a\u003E and SCS Ph.D. student \u003Ca href=\u0022https:\/\/www.cc.gatech.edu\/~akhare39\u0022\u003E\u003Cstrong\u003EAlind Khare\u003C\/strong\u003E\u003C\/a\u003E.\u003C\/p\u003E\r\n","summary":null,"format":"limited_html"}],"field_subtitle":"","field_summary":"","field_summary_sentence":[{"value":"CompOFA is an algorithm that trains hundreds of models simultaneously and makes this process inexpensive by identifying and focusing on the most efficient possible models."}],"uid":"34541","created_gmt":"2021-05-28 15:50:05","changed_gmt":"2021-05-28 16:53:06","author":"Tess Malone","boilerplate_text":"","field_publication":"","field_article_url":"","dateline":{"date":"2021-05-28T00:00:00-04:00","iso_date":"2021-05-28T00:00:00-04:00","tz":"America\/New_York"},"extras":[],"hg_media":{"647821":{"id":"647821","type":"image","title":"CompOFA","body":null,"created":"1622220561","gmt_created":"2021-05-28 16:49:21","changed":"1622220561","gmt_changed":"2021-05-28 16:49:21","alt":"Compofa","file":{"fid":"245916","name":"compofa-overview.png","image_path":"\/sites\/default\/files\/images\/compofa-overview.png","image_full_path":"http:\/\/www.tlwarc.hg.gatech.edu\/\/sites\/default\/files\/images\/compofa-overview.png","mime":"image\/png","size":202358,"path_740":"http:\/\/www.tlwarc.hg.gatech.edu\/sites\/default\/files\/styles\/740xx_scale\/public\/images\/compofa-overview.png?itok=UQ_lnigN"}}},"media_ids":["647821"],"groups":[{"id":"47223","name":"College of Computing"},{"id":"50875","name":"School of Computer Science"}],"categories":[],"keywords":[],"core_research_areas":[],"news_room_topics":[],"event_categories":[],"invited_audience":[],"affiliations":[],"classification":[],"areas_of_expertise":[],"news_and_recent_appearances":[],"phone":[],"contact":[{"value":"\u003Cp\u003ETess Malone, Communications Officer\u003C\/p\u003E\r\n\r\n\u003Cp\u003E\u003Ca href=\u0022mailto:tess.malone@cc.gatech.edu\u0022\u003Etess.malone@cc.gatech.edu\u003C\/a\u003E\u003C\/p\u003E\r\n","format":"limited_html"}],"email":[],"slides":[],"orientation":[],"userdata":""}}}