## [Is your OCR good enough?](/dataset/your-ocr-good-enough) IntroductionBody ### Description This webpage contains information about the datasets used and code developed as part of the KB research in residency project: "Is your OCR good enough? A comprehensive assessment of the impact of OCR quality on downstream tasks." This project focuses on a comprehensive assessment of the impact of OCR quality in Dutch newspaper, journal and book collections, comparing it with published results for English and French. This is be done via *extrinsic evaluation*: assessing results from a set of representative downstream tasks, such as text classification or clustering. The ultimate goal of the project is to contribute guidelines detailing when OCR quality is to be considered good enough, in order to inform the development and use of textual collections. We considered the following downstream tasks which are relevant to the digital humanities and the library and information communities. In particular, this research focused on: 1. Document classification: the goal is to assign one or more labels to each document. The model learns from a manually annotated dataset; this is therefore a supervised learning task. 2. Document clustering: the goal is to group documents into clusters, by similarity. This task also includes topic modelling, a widely used technique in the digital humanities. These are therefore unsupervised learning tasks because they are performed in the absence of annotated data. 3. Post-OCR correction: the goal is to post hoc improve the quality of the OCR. This can be done using either supervised, unsupervised and rule-based techniques. This task is somewhat different from the previous three, as its purpose is to improve on the quality of OCRed texts, independently of their subsequent use. ### Data used Results for this research are based on the following datasets: 1. Historical newspapers from the KB OCR research project. 2. A selection of books from DBNL. 3. ICDAR 2019 post-OCR correction challenge. A brief description of the datasets can be found on [this Wiki page](https://github.com/Giovanni1085/KB_OCR_impact/wiki/Datasets). ### Webinar On March the 18th Dr. Giovanni Colavizza gave a webinar about the results of his Researcher in Residence project. [The recording of his presentation](https://youtu.be/i2YVRK-o4SM) is available online. ### Acknowlegdement Mirjam Cuper (KB) and Konstantin Todorov (UvA) have also contributed to this work as follows: Mirjam has contributed data, including the evaluation of OCR quality, and provided invaluable support throughout the project; Konstantin is fully responsible for the post-OCR correction task. Citaat When using this dataset we request you to cite it as follows: Giovanni Colavizza, & Mirjam Cuper. (2021). Is your OCR good enough? A comprehensive assessment of the impact of OCR quality on downstream tasks \[Data set\]. Zenodo. Schermafbeelding Image ![Courante_uyt_Italien](/sites/default/files/styles/max_1300x1300/public/images/Courante_uyt_Italien.jpg?itok=rg5XGKIU) Developed by [Dr Giovanni Colavizza](/person/dr-giovanni-colavizza) [Mirjam Cuper](/person/mirjam-cuper) Content [Book](/taxonomy/term/28) [Journal](/taxonomy/term/29) [Newspaper](/taxonomy/term/26) Category [Text analysis](/taxonomy/term/23) File format [CSV](/taxonomy/term/42) Related [Is your OCR good enough? A comprehensive assessment of the impact of OCR qualit…](/about-us/blog/your-ocr-good-enough-comprehensive-assessment-impact-ocr-quality-downstream-tasks) [Is your OCR good enough? Probably so. Results from an assessment of the impact …](/about-us/blog/your-ocr-good-enough-probably-so-results-assessment-impact-ocr-quality-downstream) [Is your OCR good enough? Results from an assessment of the impact of OCR qualit…](https://docs.google.com/presentation/d/1N4wvB6vRTFO_CYmCcWLGj2JRrkQmi_77vaBrB2krDBE/edit?usp=sharing) Contact [Giovanni Colavizza](/person/dr-giovanni-colavizza) AccessToegang ### Data & code for this project The code used for this project can be found [on the github page](https://github.com/Giovanni1085/KB_OCR_impact). The dataset that was generated for this project and used to evaluate the OCR quality can be found [on the zenodo page](http://doi.org/10.5281/zenodo.4498186). ### Post-OCR correction For more information about the post-OCR correction see [https://zenodo.org/record/4033104 ](http://doi.org/10.5281/zenodo.4033104) Or read more about the post-OCR correction dataset in 'Transfer learning for historical corpora: an assessment on Post\_OCR Correction and Name Entity Recognition' by Konstantin Todorov en Giovanni Colavizza.