This post is a set of resources I am compiling to create a scrape of a language archive to create a Static OLAC feed.

https://www.youtube.com/watch?v=RvCBzhhydNk

https://www.kdnuggets.com/2022/02/build-web-scraper-python-5-minutes.html

The archive: http://roa.rutgers.edu/article/browse

https://www.geeksforgeeks.org/how-to-build-web-scraping-bot-in-python/

https://www.edureka.co/blog/web-scraping-with-python/

https://www.webscrapingapi.com/python-web-scraping

Text Extraction
https://www.geeksforgeeks.org/extract-text-from-pdf-file-using-python/
https://towardsdatascience.com/how-to-extract-text-from-pdf-245482a96de7
https://betterprogramming.pub/how-to-convert-pdfs-into-searchable-key-words-with-python-85aab86c544f

OCR
https://www.javatpoint.com/how-to-read-contents-of-pdf-using-ocr-in-python
https://www.geeksforgeeks.org/python-reading-contents-of-pdf-using-ocr-optical-character-recognition/
https://pypi.org/project/ocrmypdf/
https://towardsdatascience.com/extracting-text-from-scanned-pdf-using-pytesseract-open-cv-cd670ee38052
https://stackabuse.com/applying-ocr-to-a-scanned-pdf-in-python-using-borb/

NER

Named Entity Recognition (NER) in Python with Spacy

File Type Detection
https://github.com/ahupp/python-magic
https://www.geeksforgeeks.org/determining-file-format-using-python/
https://stackoverflow.com/questions/10937350/how-to-check-type-of-files-without-extensions

XML parsing

https://pypi.org/project/defusedxml/
https://www.tutorialspoint.com/python/python_xml_processing.htm

Python + Mysql

https://realpython.com/python-mysql/ https://dev.mysql.com/doc/connector-python/en/preface.html https://www.youtube.com/watch?v=3vsC05rxZ8c https://www.youtube.com/watch?v=91iNR0eG8kE&ab_channel=TechWithTim https://www.youtube.com/watch?v=xiUTqnI6xk8&ab_channel=NetworkChuck https://www.geeksforgeeks.org/working-with-pdf-files-in-python/ https://www.geeksforgeeks.org/extract-text-from-pdf-file-using-python/ https://qxf2.com/blog/extracting-data-from-pdfs-python/ https://www.youtube.com/watch?v=Ohz1f-e0ick&ab_channel=AdrianDolinay https://www.youtube.com/watch?v=RULkvM7AdzY&ab_channel=Indently https://www.metachris.com/pdfx/ https://softwarerecs.stackexchange.com/questions/76210/software-to-extract-the-list-of-references-and-title-from-a-pdf-of-a-research-pa https://pypi.org/project/refextract/ https://stackoverflow.com/questions/62365767/extract-references-from-pdf-python https://discuss.python.org/t/pdf-extraction-with-python-wrappers/40384

March 3, 2024

In "Other Journals"

OCR solutions for python toolchains

https://github.com/Calamari-OCR/calamari https://builtin.com/data-science/python-ocr https://www.geeksforgeeks.org/python-reading-contents-of-pdf-using-ocr-optical-character-recognition/# https://www.analyticsvidhya.com/blog/2022/09/extract-text-from-images-quickly-using-keras-ocr-pipeline/ https://github.com/usnistgov/ocr-pipeline https://www.youtube.com/watch?v=bcmEMcEzV9M

May 4, 2023

In "Other Journals"

Topic Modeling in Python

What if my import pd data array was the OLAC metadata schema? https://www.semanticscholar.org/paper/A-Gentle-Introduction-to-Topic-Modeling-Using-Saxton/38742c56eadfdf11fb7218f7702c8fccfc78bd95 https://gist.github.com/umbertogriffo/5041b9e4ec6c3478cef99b8653530032 https://towardsdatascience.com/contextualized-topic-modeling-with-python-eacl2021-eacf6dfa576 https://www.analyticsvidhya.com/blog/2016/08/beginners-guide-to-topic-modeling-in-python/ https://www.holisticseo.digital/python-seo/topic-modeling/ https://melaniewalsh.github.io/Intro-Cultural-Analytics/05-Text-Analysis/08-Topic-Modeling-Text-Files.html https://asandeepc.bitbucket.io/courses/inls613_summer2019/lectures/08-lda_topic_modeling.pdf http://derekgreene.com/slides/topic-modelling-with-scikitlearn.pdf https://ourcodingclub.github.io/tutorials/topic-modelling-python/ https://stackabuse.com/python-for-nlp-topic-modeling/ https://www.toptal.com/python/topic-modeling-python https://towardsdatascience.com/end-to-end-topic-modeling-in-python-latent-dirichlet-allocation-lda-35ce4ed6b3e0

November 19, 2022

In "Other Journals"