Tag Archives: OLAC

Django modules and links

Posted on by
Reply

Django application to collect submitted DOIs, acquire their API provided metadata (Bibliographic metadata and citation graph metadata), allow limited (specified) annotation, and then make those records harvestable via OAI-PMH. Language Resource tagger—Adding a layer of language related metadata to published resources.

Some Django modules for OAI-PMH
https://github.com/saw-leipzig/foaipmh
https://github.com/jnphilipp/django_oai_pmh

https://pypi.org/user/jnphilipp/ his topic extraction module looks interesting.

Also look at the xsd schema here https://github.com/saw-leipzig/foaipmh/blob/5b15d5cc4700a3cccf497c47218c2fba6b3421d5/entrypoint.prod.sh#L5

Metadata utility for OAI-PMH

https://combine.readthedocs.io/en/master/configuration.html

User Authentication
https://github.com/ubffm/django-orcid
https://django.fun/en/docs/social-docs/0.1/backends/orcid/

Crossref
https://github.com/fabiobatalha/crossrefapi

Introducing Crossref, the basics

Database Versioning
This depends on how the DB is set up. If we only have one record per item or one record per state... This needs more definition.
https://djangopackages.org/grids/g/versioning/
https://www.wpbeginner.com/beginners-guide/complete-guide-to-wordpress-post-revisions/

Form Builders
https://djangopackages.org/grids/g/form-builder/

Some Javascript tools for creating the specific forms needed:
https://github.com/HughP/dublin-core-generator
https://nsteffel.github.io/dublin_core_generator/generator.html

Markdown for documentation
https://neutronx.github.io/django-markdownx/

Bibtex
https://bibtexparser.readthedocs.io/en/master/
https://github.com/sciunto-org/python-bibtexparser
https://github.com/jnphilipp/bibliothek
https://github.com/lucastheis/django-publications <-- also check the network as "improvements" are all over the place.
Other names include:
* Babybib
* Pybtex
* Pybibliographer

APIs

ORCID
https://github.com/ORCID/python-orcid

API Tutorial: Searching the ORCID registry

Crossref API doc
https://github.com/CrossRef/rest-api-doc/blob/master/demos/crossref-api-demo.ipynb
Crossref types: https://www.crossref.org/documentation/register-maintain-records/
https://api.crossref.org/swagger-ui/index.html#/Types/get_types__id__works

Others — Mostly citation and references
http://www.scholix.org/
https://scholexplorer.openaire.eu/#/query/page=5/q=language
https://crossref.gitlab.io/knowledge_base/products/event-data/
FatCat https://fatcat.wiki/
InternetArchive Scholar https://scholar.archive.org/
Thor project https://project-thor.readme.io/docs/introduction-for-integrators
Corsscite.org
Semantic Scholar API https://api.semanticscholar.org/api-docs/graph
https://core.ac.uk/
https://opencitations.net/
https://unpaywall.org/ --> see: http://musingsaboutlibrarianship.blogspot.com/2017/11/using-oadoi-crossref-event-data-api-to.html
https://openalex.org/
https://arxiv.org/help/api/index
https://www.aminer.org/citation
https://www.aminer.org/download
https://open.aminer.cn/
https://analytics.hathitrust.org/datasets#top
https://pro.dp.la/developers/api-codex
https://pro.europeana.eu/page/apis

LCSH
https://github.com/edsu/id

MARC
For generating an ingesting MARC records
https://pymarc.readthedocs.io/en/latest/

Zotero
https://github.com/urschrei/pyzotero

Overview see: https://researchguides.smu.edu.sg/api-list/scholarly-metadata-api

ISSNs
ISSN.org is supposed to have an API.. but not sure if they do.
https://portal.issn.org/resource/ISSN/1904-0008
Any request to the portal may be automated thanks to the use of REST protocol. The download of results is also automated. This service is restricted to subscribing users. Please contact sales [at] issn.org for more information.
https://portal.issn.org/node/170

https://portal.issn.org/resource/ISSN/2549-5089#
https://portal.issn.org/resource/ISSN/2549-5089?format=json
We could also slurp the HTML for the sameAs links to other DBs if needed.

JATS
https://pypi.org/project/jatsgenerator/
https://stackoverflow.com/questions/42084165/extracting-text-from-jats-xml-file-using-python
https://github.com/sibils/jats-parser

Pandas
https://pypi.org/project/django-pandas/

Beautiful Soup

There is the issue of how do we add to a Dublin Core OAI record how it was changed over time.... I need to architect this out.

Record Provenance:
[]Explore
https://www.w3.org/TR/prov-dc/
https://www.w3.org/2011/prov/track/issues/607?changelog
http://www.ukoln.ac.uk/metadata/dcmi/collection-provenance/
https://edoc.hu-berlin.de/bitstream/handle/18452/2727/332.pdf?sequence=1&isAllowed=y
https://www.ncbi.nlm.nih.gov/pmc/articles/PMC4177195/
https://www.loc.gov/standards/mods/userguide/recordinfo.html
https://tsl.access.preservica.com/tslac-digital-preservation-framework/qualified-dublin-core-schema/
https://dl.acm.org/doi/10.5555/2770897.2770924
https://blog.datacite.org/exposing-doi-metadata-provenance/
https://dgarijo.com/papers/dc2011.pdf
https://ceur-ws.org/Vol-670/paper_3.pdf
https://ecommons.cornell.edu/bitstream/handle/1813/55327/Encoding%20Provenance%20for%20Social%20Science%20Data-final.pdf?sequence=3&isAllowed=y

Views:
1. login with ORCID
2. query APIs (DOIs, ISBNs, ISSNs, ORCID, WikiData, etc.)
3. results display and annotation
4. submission
5. List of past submissions
6. update past submission screen (same as #3?)

If we ran a module like this:
https://pybliometrics.readthedocs.io/en/latest/classes/SerialTitle.html

Then we could take a reading on where the least spoken languages appear in the most highly ranked journals and determine if there was a bias or a loss to science.

Data Examples:

Have been moved to:
https://github.com/HughP/CrossRef-to-OLAC-data-examples

PDF Extraction:
https://levelup.gitconnected.com/scrap-data-from-website-and-pdf-document-for-django-app-fa8f37010085
https://towardsdatascience.com/how-to-extract-pdf-data-in-python-876e3d0c288
https://stackoverflow.com/questions/71850349/download-a-pdf-from-url-edit-it-an-render-it-in-django
https://stackoverflow.com/questions/48882768/django-reading-pdf-files-content
https://www.geeksforgeeks.org/working-with-pdf-files-in-python/

PDF Creation:
https://docs.djangoproject.com/en/4.1/howto/outputting-pdf/
https://jeltef.github.io/PyLaTeX/current/examples/header.html

NER:




https://johnfraney.github.io/django-ner-trainer/settings/

Named Entity Recognition (NER) in Python with Spacy

Other:
https://prodi.gy/

https://realpython.com/testing-in-django-part-1-best-practices-and-examples/

here is a django app for controlling URIs for linked data vocabularies.
https://github.com/unt-libraries/django-controlled-vocabularies
as seen here https://digital2.library.unt.edu/vocabularies/agent-qualifiers/

And here is a one for source authority records.
https://github.com/unt-libraries/django-name
as seen here: https://digital2.library.unt.edu/name/nm0000001/

Link Checker
https://github.com/Kaltsoon/dead-link-checker
https://pypi.org/project/django-linkcheck/
https://github.com/bartdag/pylinkvalidator
https://stackoverflow.com/questions/43264291/in-django-how-can-i-unit-test-all-links-recursively-every-view-check-for-200-o

Abstract and Table of Contents

Posted on by
Reply

If abstract is a sample of about-ness, then a table of contents is sample if is-ness. Some have said that journal articles should not have table of contents (instructional staff at the UNT program teaching the Metadata I course). I disagree, but so does Habing, et al (2001). Sometimes more than an abstract a table of contents can deliver a substantial understanding of what an article is and is about by displaying its structure. In fact many law review articles actually include a table of contents prior to the main part of the article. Law review articles can be over 70 pages in length. An outline offers useful information to the potential reader.

An example of an outline from a linguistics article.

Roberts, David. 2011. “A Tone Orthography Typology.” Written Language & Literacy 14 (1): 82–108. doi:10.1075/wll.14.1.05rob.

  1. Introduction
  2. The six parameters
    2.1 First parameter: Domain
    2.2 Second parameter: Target
    2.2.1 Tones
    2.2.2 Grammar
    2.2.3 Lexicon
    2.2.4 Dual strategies
    2.3 Third parameter: Symbol
    2.3.1 Phonographic representations
    2.3.2 Semiographic representations
    2.4 Fourth parameter: Position
    2.5 Fifth parameter: Density
    2.5.1 Introduction
    2.5.2 Zero density
    2.5.3 Partial density
    2.5.4 Exhaustive density
    2.6 Sixth parameter: Depth
    2.6.1 Introduction
    2.6.2 Surface representation
    2.6.3 Deep representation
    2.6.4 Shallow (transparent) representation
  3. Conclusion
    Abbreviations
    Notes
    Bibliographical references

References

Thomas G. Habing, Timothy W. Cole, and William H. Mischo. 2001. Qualified Dublin Core using RDF for Sci-Tech Journal Articles. https://dli.grainger.uiuc.edu/Publications/metadatacasestudy/HabingDC2001.pdf

Opinions on OCLC and metadata ownership

Posted on by
Reply

https://librarytechnology.org/document/7266
https://librarytechnology.org/document/7266/ownership-of-machine-readable-records-a-neglected-consideration-in-retrospective-conversion

Landgrab For Ownership Of Library Catalog Data

https://www.oclc.org/en/worldcat/cooperative-quality/policy.html

https://repository.law.uic.edu/cgi/viewcontent.cgi?article=1557&context=jitpl

https://dltj.org/article/oclc-records-use-policy-1/

https://wiki.harvard.edu/confluence/display/LibraryStaffDoc/OCLC+Institution+records+discontinuation

Matching algorithms

https://www.oclc.org/en/news/announcements/2022/worldcat-quality-enhancements.html
https://www.ohiolink.edu/content/matching_bibliographic_records_central_site

Scraping archives for OLAC

Posted on by
Reply

This post is a set of resources I am compiling to create a scrape of a language archive to create a Static OLAC feed.

https://www.youtube.com/watch?v=RvCBzhhydNk

https://www.kdnuggets.com/2022/02/build-web-scraper-python-5-minutes.html

The archive: http://roa.rutgers.edu/article/browse

https://www.geeksforgeeks.org/how-to-build-web-scraping-bot-in-python/

https://www.edureka.co/blog/web-scraping-with-python/

https://www.webscrapingapi.com/python-web-scraping

Text Extraction
https://www.geeksforgeeks.org/extract-text-from-pdf-file-using-python/
https://towardsdatascience.com/how-to-extract-text-from-pdf-245482a96de7
https://betterprogramming.pub/how-to-convert-pdfs-into-searchable-key-words-with-python-85aab86c544f

OCR
https://www.javatpoint.com/how-to-read-contents-of-pdf-using-ocr-in-python
https://www.geeksforgeeks.org/python-reading-contents-of-pdf-using-ocr-optical-character-recognition/
https://pypi.org/project/ocrmypdf/
https://towardsdatascience.com/extracting-text-from-scanned-pdf-using-pytesseract-open-cv-cd670ee38052
https://stackabuse.com/applying-ocr-to-a-scanned-pdf-in-python-using-borb/

NER

Named Entity Recognition (NER) in Python with Spacy

File Type Detection
https://github.com/ahupp/python-magic
https://www.geeksforgeeks.org/determining-file-format-using-python/
https://stackoverflow.com/questions/10937350/how-to-check-type-of-files-without-extensions

XML parsing

https://pypi.org/project/defusedxml/
https://www.tutorialspoint.com/python/python_xml_processing.htm

MARC profiles for Language Archives

Posted on by
Reply

AACR2 and RDA both constitute application profiles using the same database structure known as MARC. MARC defines the fields and the expected values within those fields (type control) while AACR2 and RDA compose definitions of cognitive models and data fingerprinting (not the LIS terms for these concepts). By cognitive model I mean the mental representation of entities and their relationships and by fingerprinting of data I mean that some artifacts are "well described" when various fields are employed. E.g., a book description needs a publisher, while a manuscript does not.

AACR2 and RDA both constitute application profiles to which the documentation is only provided on a subscription basis. This is a pay-to-play game. This sort of game is not well received by the language documentation community. These facts do no mean that preservation organizations need to avoid MARC, rather a MARC profile could be established and documented in the open.

When considering the future of OLAC and language resource archiving an outstanding question emerges, is this sort of profile something that is of interest within the community?

Dublin Core Subject field

Posted on by
Reply

Dublin Core has a subject element. But what constitutes a subject?

Two points on this:

  1. Subject-hood is a complex notion. As pointed out by Birger Hjørland included in this concept can be both is-ness and about-ness. LIS theory can say to divide these concepts, but if Dublin Core as a descriptive framework does not allow this, then the notion of subjecthood should be assumed to include both notions.
  2. Pictures (still images, including paintings) are complex when evaluating their subject hood. First, when a picture depicts something then it is reasonable to say that the picture is about that thing, as well as the picture is something...

I am suggesting that Dublin Core as a standard does not distinguish between about-ness and is-ness with regard to subject. And to further make matters complicated about-ness and is-ness merge more in visual media than in other types of print based media.

The following articles indirectly address the distinction of about-ness and is-ness or address about-ness in visual media.

Rushton, M. Public Funding of Controversial Art. Journal of Cultural Economics 24, 267–282 (2000). https://doi.org/10.1023/A:1007682121108

Wall, J. M. (2005). The Medium & the Message: Theology and Film. Theology Today, 62(1), 74–77. https://doi.org/10.1177/004057360506200109

Wanda Klenczon & Paweł Rygiel (2014) Librarian Cornered by Images, or How to Index Visual Resources, Cataloging & Classification Quarterly, 52:1, 42-61, DOI: 10.1080/01639374.2013.848123

in a book
Emerging Frameworks and Methods: CoLIS 4 : Proceedings of the Fourth

Andrea Witcomb (1997) On the Side of the Object: an Alternative Approach to Debates About Ideas, Objects and Museums, Museum Management and Curatorship, 16:4, 383-399, DOI: 10.1080/09647779700501604

Wang, X., Song, N., Liu, X. and Xu, L. (2021), "Data modeling and evaluation of deep semantic annotation for cultural heritage images", Journal of Documentation, Vol. 77 No. 4, pp. 906-925. https://doi.org/10.1108/JD-06-2020-0102

Platform tools for OAI harvesting

Posted on by
Reply

So, in recent OLAC presentation I talked about enabling Omeka or Drupal via recipes for OAI harvesting. Here is some links to internet chatter on these issues.

Koha

enable Items in KOHA OAI Harvesting

https://koha-community.org/manual/18.05/en/html/webservices.html

https://forums.zotero.org/discussion/38956/export-of-zotero-citation-to-marc-format-for-import-into-koha-lms

wordpress

Day 13: Harvest data with OAI-PMH

WordPress and Drupal

https://acrl.ala.org/techconnect/post/creating-an-oai-pmh-feed-from-your-website/

eHive

https://developers.ehive.com/

Catmandu

https://librecatproject.wordpress.com/tutorial/

Omeka
https://omeka.org/classic/docs/Plugins/OaiPmhRepository/

MOAI
https://pypi.org/project/MOAI/

PyOAI
https://pypi.org/project/pyoai/

OLAC and PREMIS

Posted on by
Reply

During the presentation yesterday on PREMIS, which I liked very much, it seemed that a multi-author survey of archival institutions which conduct media transference and also participate in OLAC, would be an interesting group project which would have publishable results.

Such a project might have three components:

  1. A “crosswalk style” mapping between PREMIS data values or fields and where those may appear in the OLAC profile. (How would one know if this information were available? or put another way, if an institution were to use this framework how could they publish that metadata via the OLAC profile?)
  2. A review of current practice as evidenced by what we can see in OLAC records. (Are institutions contributing this metadata? and in what capacitiy?)
  3. A questionnaire or interview with specific institutions conducting digitization. Offhand I think the following do digitization: Alaska Native Language Archive, Kaipuleohone,  SIL L&CA, PARIDESIC, Oxford Text Archive, and the COllections de COrpus Oraux Numeriques (CoCoON ex-CRDO). (In what ways are institutions solving the problems that PREMIS helps to solve?) 
https://www.loc.gov/standards/premis/