Understanding and mitigating instability in low-precision training of large machine learning models

In the last decade, advancements in machine learning have been driven by the development of purpose-built hardware accelerators. In the last couple of years, however, the size of models has increased at a pace that hardware progress alone has not been able to match: since 2022, the number of parameters in the largest models has increased by one order of magnitude, and the number of operations needed to train them by at least two.<br /> <br /> Scientific computations typically require a 64-bit format to yield meaningful results, and fewer-than-32-bit formats were traditionally considered adequate only for data storage. This changed in December 2017, when NVIDIA released the first machine learning accelerators explicitly supporting high-throughput 16-bit floating-point computation.<br /> <br /> Today, 8-bit formats are routinely used to train the largest neural networks. These formats have extremely low accuracy and can make the training process unstable, causing spikes in the loss function and potentially compromising irrecoverably the training process. Currently, this problem is mitigated by relying on checkpointing: the training algorithm takes snapshots of the model's parameter at regular intervals, and rolls back to the last valid checkpoint if instability is detected. This solution is costly to implement, but is not guaranteed to work.<br /> <br /> Evidence suggests that these instabilities are a numerical phenomenon due to the use of low precision, and this project aims to investigate this hypothesis and propose solutions that do not require the use of higher precision.<br /> <br /> First, we will seek to understand whether loss spikes can be predicted by analysing the training data. The goal is to identify the conditions that make instabilities likely to occur and use this knowledge to inform a model's design. This will help us establish what numerical formats that are safe to use in specific contexts, such as particular nodes in a deep neural network or specific blocks within a transformer architecture.<br /> <br /> Next, we will explore how existing floating-point formats can be modified to enhance their reliability in the context of training. This could involve, for example, the use of different rounding modes or the development of mixed-precision approaches that employ higher precision only where necessary.<br /> <br /> The outcomes of this project can impact the design and training of large models, where computational efficiency is crucial. Understanding the relationships between numerical precision and training stability on a deeper level will contribute to the development of more robust and efficient training strategies for large neural networks.

<p>Formal applications for research degree study should be made online through the <a href="https://www.leeds.ac.uk/research-applying/doc/applying-research-degrees">University's website</a>. Please state clearly in the Planned Course of Study section that you are applying for <strong><em>PHD Computer Science FT</em></strong>, in the research information section, please state clearly that the research project you wish to be considered for is <strong><em>Computer arithmetic for the next generation of integrated circuits</em></strong>, and mention <em><a href="https://eps.leeds.ac.uk/computing/staff/14034/massimiliano-fasi">Massimiliano Fasi</a></em> as your proposed supervisor and in the finance section, please state clearly <em><strong>the funding that you are applying for, if you are self-funding or externally sponsored</strong></em>.</p> <p>If English is not your first language, you must provide evidence that you meet the University's minimum English language requirements (below).</p> <p style="text-align:start; margin-bottom:11px"><em>As an international research-intensive university, we welcome students from all walks of life and from across the world. We foster an inclusive environment where all can flourish and prosper, and we are proud of our strong commitment to student education. Across all Faculties we are dedicated to diversifying our community and we welcome the unique contributions that individuals can bring, and particularly encourage applications from, but not limited to Black, Asian, people who belong to a minority ethnic community, people who identify as LGBT+ and people with disabilities. Applicants will always be selected based on merit and ability.</em></p> <p>Applications will be considered on an ongoing basis.  Potential applicants are strongly encouraged to contact the supervisors for an informal discussion before making a formal application.  We also advise that you apply at the earliest opportunity as the application and selection process may close early, should we receive a sufficient number of applications or that a suitable candidate is appointed.</p> <p><strong>Please ensure you provide your supporting documents by the closing date of Friday 3 January 2025: </strong></p> <ul> <li>Full Transcripts of all degree study or if in final year of study, full transcripts to date</li> <li>Personal Statement outlining your interest in the project</li> <li>CV</li> </ul>

Applicants to research degree programmes should normally have at least a first class or an upper second class British Bachelors Honours degree (or equivalent) in an appropriate discipline. The criteria for entry for some research degrees may be higher, for example, several faculties, also require a Masters degree. Applicants are advised to check with the relevant School prior to making an application. Applicants who are uncertain about the requirements for a particular research degree are advised to contact the School or Graduate School prior to making an application.

The minimum English language entry requirement for research postgraduate research study in the School of Computer Science is an IELTS of 6.5 overall with at least 6.5 in writing and at least 6.0 in reading, listening and speaking or equivalent. The test must be dated within two years of the start date of the course in order to be valid.

<p>A highly competitive School of Computer Science Studentship providing the award of full academic fees, together with a tax-free maintenance grant at the standard UKRI rate (£19,237 in academic session 2024/25) for 3.5 years. There are no additional allowances for travel, research expenses, conference attendance or any other costs.</p> <p>You will be responsible for paying the overtime fee in full in your writing up/overtime year (£320 in Session 2024/25), but the scholarship maintenance allowance will continue to be paid for up to 6 months in the final year of award.</p> <p>Please refer to the <a href="https://www.ukcisa.org.uk/">UKCISA</a> website for information regarding Fee Status for Non-UK Nationals.</p> <p><strong>Self-Funded or externally sponsored students are welcome to apply.</strong></p> <p><strong>Competition Funding</strong></p> <p><strong>UK</strong> – The <a href="https://phd.leeds.ac.uk/funding/209-leeds-doctoral-scholarships-2022">Leeds Doctoral Scholarships</a> <strong>(closing date: Monday 3 February 2025)</strong> and <a href="https://phd.leeds.ac.uk/funding/234-leeds-opportunity-research-scholarship-2022">Leeds Opportunity Research Scholarship</a> <strong>(closing date: Tuesday 1 April 2025)</strong> are available to UK applicants.  <a href="https://phd.leeds.ac.uk/funding/60-alumni-bursary">Alumni Bursary</a> is available to graduates of the University of Leeds.</p> <p><strong>Non-UK</strong> – The <a href="https://phd.leeds.ac.uk/funding/48-china-scholarship-council-university-of-leeds-scholarships-2021">China Scholarship Council - University of Leeds Scholarship</a> is available to nationals of China <strong>(closing date: Monday 6 January 2025)</strong>. The <a href="https://phd.leeds.ac.uk/funding/73-leeds-marshall-scholarship">Leeds Marshall Scholarship</a> is available to support US citizens. <a href="https://phd.leeds.ac.uk/funding/60-alumni-bursary">Alumni Bursary</a> is available to graduates of the University of Leeds.</p> <p>You will be responsible for paying the overtime fee in full in your writing up/overtime year (£320 in Session 2024/25), but the scholarship maintenance allowance will continue to be paid for up to 6 months in the final year of award.</p> <p><strong>Important:</strong> Please note that that the award does <em><strong>not </strong></em>cover the costs associated with moving to the UK.  All such costs (<a href="https://www.leeds.ac.uk/international-visas-immigration/doc/applying-student-visa">visa, Immigration Health Surcharge</a>, flights etc) would have to be met by yourself, or you will need to find an alternative funding source. </p> <p>Please refer to the <a href="https://www.ukcisa.org.uk/">UKCISA</a> website for information regarding Fee Status for Non-UK Nationals.</p>

<p>For further information about this project, please contact Massimiliano Fasi at <a href="mailto:m.fasi@leeds.ac.uk?subject=%5BPhD%20contact%5D%20Computer%20arithmetic%20for%20the%20next%20generation%20of%20integrated%20circuits">M.Fasi@leeds.ac.uk</a>.</p> <p>For further information about your application, please contact the Postgraduate Research Admissions team at <a href="mailto:phd@engineering.leeds.ac.uk">phd@engineering.leeds.ac.uk</a>.</p>