AI-Digital Assessment: Machine Learning of exam questions and answers from lecture transcripts

This project will explore and evaluate approaches to machine learning of exam questions and answers from texts, such as lecture transcripts, textbooks, and online sources of learning materials in specific disciplines. Possible approaches include:<br /> <br /> - scanning the text for statements of fact as candidate answers, then generating corresponding questions; see for example the JISC exploreAI question-generation demo https://exploreai.jisc.ac.uk/tool/question-generation<br /> <br /> - devising a knowledge base or ontology for the subject to be learnt, for example by applying methods for ontology extraction from text; then generating questions and candidate answers via the ontology and text.<br /> <br /> - web search for existing sources of exam questions and answers in specific disciplines, to collate an exam corpus, to use as a training corpus for subsequent machine learning experiments to extend the corpus<br /> <br /> - apply a knowledge-based approach: observe lecturers in devising example exams and review the education literature on exam question and answer design, to elicit human expert strategies and methods, and then encode these in AI software to directly emulate human practices<br /> <br /> The project should begin with a broad survey of the field, to devise an initial list of methods to explore and evaluate. The project will aim to deliver a practical software tool to suggest exam questions and answers for lecturers in a selection of case-study taught modules at Leeds University, to include Computing taught modules.<br /> <br /> Online education can reach very large classes, as online learning resources and online lectures can be created once and then delivered at scale. The manpower bottleneck is assessment: traditional assignments and exams are marked by teachers, and marking time increases with number of students. A response is to semi-automate digital assessment; for example tools like Minerva Tests and GradeScope can help to organise, manage and speed up the marking process. Also a switch from open-ended essay-style exam questions to Multiple Choice Questions and Multiple Answer Questions allows semi-automated grading at scale. An MCQ has a question, one correct answer, and several incorrect answers; a MAQ has a question and several possible answers of which none, some or all are correct. <br /> However, GradeScope is limited to MCQ and MAQ questions and answers already devised and written by the Lecturer; GradeScope cannot help the Lecturer in formulating appropriate exam questions and sets of MCQ/MAQ answers. Devising the questions and answers takes a lot of thought and time, so automating this could significantly improve efficiency of large-scale digital assessment<br />

<p>REFERENCES</p> <p>Das, B., Majumder, M., Phadikar, S. and Sekh, A.A., 2021. Automatic question generation and answer assessment: a survey. Research and Practice in Technology Enhanced Learning, 16(1), pp.1-15.</p> <p>Indurthi, S.R., Raghu, D., Khapra, M.M. and Joshi, S., 2017. Generating natural language question-answer pairs from a knowledge graph using a RNN based question generation model. In Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics. pp. 376-385.</p> <p>Kurdi, G., Leo, J., Matentzoglu, N., Parsia, B., Sattler, U., Forge, S., Donato, G. and Dowling, W., 2019. A comparative study of methods for a priori prediction of MCQ difficulty. Semantic Web, pp.1-17.</p> <p>Liu, B., Wei, H., Niu, D., Chen, H. and He, Y., 2020. Asking questions the human way: Scalable question-answer generation from text corpus. In&nbsp;<em>Proceedings of The Web Conference 2020</em>. pp. 2032-2043.</p> <p>Pho, V.M., Andr&eacute;, T., Ligozat, A.L., Grau, B., Illouz, G. and Fran&ccedil;ois, T., 2014. Multiple choice question corpus analysis for distractor characterization. In Proceedings of LREC&rsquo;2014 International Conference on Language Resources and Evaluation.</p> <p>Song, L., Wang, Z., Hamza, W., Zhang, Y. and Gildea, D., 2018. Leveraging context information for natural question generation. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. pp. 569-574.</p> <p>Wang, B., Wang, X., Tao, T., Zhang, Q. and Xu, J., 2020. Neural question generation with answer pivot. In Proceedings of the AAAI Conference on Artificial Intelligence. Vol. 34, No. 05, pp. 9138-9145.</p> <p>Wang, L., Xu, Z., Lin, Z., Zheng, H. and Shen, Y., 2020. Answer-driven Deep Question Generation based on Reinforcement Learning. In Proceedings of the 28th International Conference on Computational Linguistics. pp. 5159-5170.</p>

<p>Formal applications for research degree study should be made online through the&nbsp;<a href="https://www.leeds.ac.uk/research-applying/doc/applying-research-degrees">University&#39;s website</a>. Please state clearly in the research information section&nbsp;that the research degree you wish to be considered for is <em>AI-Digital Assessment: Machine Learning of exam questions and answers from lecture transcripts</em> as well as<a href="https://eps.leeds.ac.uk/computing/staff/33/professor-eric-atwell">&nbsp;Professor Eric Atwell</a> as your proposed supervisor.</p> <p>If English is not your first language, you must provide evidence that you meet the School of Computing&#39;s minimum English language requirements (below).</p> <p>&nbsp;</p>

Applicants to research degree programmes should normally have at least a first class or an upper second class British Bachelors Honours degree (or equivalent) in an appropriate discipline. The criteria for entry for some research degrees may be higher, for example, several faculties, also require a Masters degree. Applicants are advised to check with the relevant School prior to making an application. Applicants who are uncertain about the requirements for a particular research degree are advised to contact the School or Graduate School prior to making an application.

The minimum English language entry requirement for research postgraduate research study is an IELTS of 6.5 overall with at least 6.5 in writing and at 6.0 in reading, listening and speaking) or equivalent. The test must be dated within two years of the start date of the course in order to be valid. Some schools and faculties have a higher requirement.

<p><strong>Self-Funded or externally sponsored students are welcome to apply.</strong></p> <p><strong>UK</strong>&nbsp;&ndash;&nbsp;The&nbsp;<a href="https://phd.leeds.ac.uk/funding/209-leeds-doctoral-scholarships-2022">Leeds Doctoral Scholarships</a>, <a href="https://phd.leeds.ac.uk/funding/53-school-of-computing-scholarship">School of Computing Scholarship&nbsp;</a>, <a href="https://phd.leeds.ac.uk/funding/198-akroyd-and-brown-scholarship-2022">Akroyd &amp; Brown</a>, <a href="https://phd.leeds.ac.uk/funding/199-frank-parkinson-scholarship-2022">Frank Parkinson</a> and <a href="https://phd.leeds.ac.uk/funding/204-boothman-reynolds-and-smithells-scholarship-2022">Boothman, Reynolds &amp; Smithells</a> Scholarships are available to UK applicants. &nbsp;<a href="https://phd.leeds.ac.uk/funding/60-alumni-bursary">Alumni Bursary</a> is available to graduates of the University of Leeds.&nbsp;</p> <p><strong>Non-UK</strong>&nbsp;&ndash; The&nbsp;<a href="https://phd.leeds.ac.uk/funding/53-school-of-computing-scholarship">School of Computing Scholarship&nbsp;</a>&nbsp;is available to support the additional academic fees of Non-UK applicants. The&nbsp;<a href="https://phd.leeds.ac.uk/funding/48-china-scholarship-council-university-of-leeds-scholarships-2021">China Scholarship Council - University of Leeds Scholarship</a>&nbsp;is available to nationals of China. The&nbsp;<a href="https://phd.leeds.ac.uk/funding/73-leeds-marshall-scholarship">Leeds Marshall Scholarship</a>&nbsp;is available to support US citizens. <a href="https://phd.leeds.ac.uk/funding/60-alumni-bursary">Alumni Bursary</a> is available to graduates of the University of Leeds.</p>

<p>For further information regarding your application, please contact Doctoral College Admissions<br /> e:&nbsp;<a href="mailto:phd@engineering.leeds.ac.uk">phd@engineering.leeds.ac.uk</a>, t: +44 (0)113 343 5057.</p> <p>For further information regarding the project, please contact Professor Eric Atwell<br /> e:&nbsp;<a href="mailto:E.S.Atwell@leeds.ac.uk">E.S.Atwell@leeds.ac.uk</a></p>