Over the past few months, we have laid the groundwork for the ReproHum project (summarised in the 'pre-project' column in the Work Plan document) with (i) a study of 20 years of human evaluation in NLG which reviewed and labelled 171 papers in detail, (ii) the development of a classification system for NLP evaluations, (iii) a proposal for a shared task for reproducibility of human evaluation in NLG, and (iv) a proposal for a workshop on human evaluation in NLP. We have built an international network of 20 research teams currently working on human evaluation who will actively contribute to this project (see Track Record section), making combined contributions in kind of over £80,000. This pre-project activity has created an advantageous starting position for the proposed work, and means we can 'hit the ground running' with the scientifically interesting core of the work. In this foundational project, our key goals are the development of a methodological framework for testing the reproducibility of human evaluations in NLP, and of a multi-lab paradigm for carrying out such tests in practice, carrying out the first study of this kind in NLP. We will (i) systematically diagnose the extent of the human evaluation reproducibility problem in NLP and survey related current work to address it (WP1); (ii) develop the theoretical and methodological underpinnings for reproducibility testing in NLP (WP2); (iii) test the suitability of the shared-task paradigm (uniformly popular across NLP fields) for reproducibility testing (WP3); (iv) create a design for multi-test reproducibility studies, and run the ReproHum study, an international large-scale multi-lab effort conducting 50+ individual, coordinated reproduction attempts on human evaluations in NLP from the past 10 years (WP4); and (v) nurture and build international consensus regarding how to address the reproducibility crisis, via technical meetings and growing our international network of researchers (WP5).

Since the beginning of humanity our societies have been based on commerce, i.e. we make things and we sell them to other people. Relatively simple beginnings led to the Industrial Revolution and now to the technological age. Over-generalising, the Far East are currently the masters of mass manufacture and the West are (or wish to be) the masters of advanced manufacture - the production of high-value goods, often involving a significant degree of innovation. To be able to manufacture goods in a cost-effective, environmentally-sustainable manner, quality control procedures are required. And quality control in turn requires an appropriate measurement infrastructure to be in place. It is a sub-set of this measurement infrastructure that is the subject of this fellowship. The UK government has been investing heavily in advanced manufacturing. In the academic arena, there are the sixteen EPSRC Centres of Innovative Manufacturing. To ease the pain of transferring academic research to the manufacturing sector, there are the seven High-Value Manufacturing Catapults (the Manufacturing Technology Centre being the main one of note here). For industry, there are a number of funding initiatives and tax breaks. To support this burgeoning UK advanced manufacturing infrastructure, there are a small number of academic centres for metrology - those based at Huddersfield and Bath are the main players. And, at the top of the measurement tree, there is the world-class National Physical Laboratory - a centre of excellence in metrology. But, there are still some gaps in the manufacturing metrology research jigsaw, and the aim of this fellowship is to plug those gaps. Coordinate metrology has been used for decades in the manufacturing industry as the most dominant form of process control, usually employing tactile coordinate measuring machines (CMMs). However, due to the slow speed of tactile systems and the fact that they can only take a limited amount of points, optical CMMs are starting to flourish. On the smaller scale, there are many optical surface measuring devices that tend to be used off-line in industry. When making small, high-precision, complex components, with difficult to access geometries, it is a combination of the surface measurement systems and the CMMs that is required. This requirement is one of the main aims of the fellowship - to develop a suite of fast, high-accuracy, non-contact measurement systems, which can be employed in industry. These principles will also be applied to the field of additive manufacturing - a new paradigm in manufacturing which is seeing significant government support and, in some cases, media hype. As with high-precision components, a coordinate metrology infrastructure for additive manufacturing is required, in many cases in-line to allow direct feedback to the manufacturing process. This is the second field of metrology that the fellowship will address. The outputs of the fellowship will be in the form of academic publications; new measurement instruments, along with new ways to use existing instruments; methods to allow manufacturers to verify the performance of their instruments; and the necessary pre-normative work that will lead to specification standards in the two fields (currently lacking). The academic world will benefit from the fundamental nature of elements of the research, and the industrial manufacturing world will benefit from the techniques developed and routes to standardisation. But, ultimately, it will be the UK citizens that will reap the greatest benefit in terms of new and enhanced products, and the wealth creation potential from precision and additive manufacturing.

The vision of the Hub is to create ground-breaking embedded metrology and universal metrology informatics systems to be applied across the manufacturing value chain. This encompasses a paradigm shift in measurement technologies, embedded sensors/instrumentation and metrology solutions. A unified approach to creating new, scientifically-validated measurement technologies in manufacturing will lead to critical underpinning solutions to stimulate significant growth in the UK's productivity and facilitate future factories. Global manufacturing is evolving through disruptive technologies towards a goal of autonomous production, with manufacturing value-chains increasingly digitised. Future factories must be faster, more responsive and closer to customers as manufacturing is driven towards mass customisation of lower-cost products on demand. Metrology is crucial in underpinning quality, productivity and efficiency gains under these new manufacturing paradigms. The Advanced Metrology Hub brings together a multi-disciplinary team from Huddersfield with spokes at Loughborough, Bath and Sheffield universities, with fundamental support from NPL. Expertise in Engineering, Mathematics, Physics and Computer Science will address the grand challenges in advanced metrology and the Hub's vision through two key research themes and parallel platform activities: Theme I - Embedded Metrology will build sound technological foundations by bridging four formidable gaps in process- and component-embedded metrology. This covers: physical limits on the depth of field; high dynamic range measurement; real-time dynamic data acquisition in optical sensor/instruments; and robust, adaptive, scalable models for real-time control systems using sensor networks with different physical properties under time-discontinuous conditions. Theme II - Metrology Data analytics will create a smart knowledge system to unify metrology language, understanding, and usage between design, production and verification for geometrical products manufacturing; Establishment of data analytics systems to extract maximal information from measurement data going beyond state-of-the-art for optimisation of the manufacturing process to include system validation and product monitoring. Platform research activities will underpin the Hub's vision and core research programmes, stimulate new areas of research and support the progression of fundamental and early-stage research towards deployment and impact activities over the Hub's lifetime. In the early stage of the Hub, the core research programme will focus on four categories (Next generation of surface metrology; Metrology technologies and applications; In-process metrology and Machine-tool and large volume metrology) to meet UK industry's strategic agenda and facilitate their new products. The resulting pervasive embedding and integration of manufacturing metrology by the Hub will have far reaching implications for UK manufacturing as maximum improvements in product quality, minimization of waste/rework, and minimum lead-times will ultimately deliver direct productivity benefits and improved competitiveness. These benefits will be achieved by significantly reducing (by 50% to 75%) verification cost across a wide swathe of manufacture sectors (e.g. aerospace, automotive, electronics, energy, medical devices, optics, precision engineering) where the current cost of verification is high (up to 20% of total costs) and where product quality and performance is critical.