Working in an archive, one can imagine that letting go of materials is a real challenge. Both in that it is hard to do becasue of policy, but also because it is hard to do because of the emotional “pack-rat” nature of archivist. This is no less the case of the archive where I work. We were recently working through a set of items and getting rid of the duplicates. (Physical space has its price; and the work should soon be available via JASOR.) However, one of the items we were getting rid of was a journal issue on a people group/language. The journal has three articles, of these, only one of them article was written by someone who worked for the same organization I am working for now. So the “employer” and owner-operator of the archive only has rights to one of the three works. (Rights by virtue of “work-for-hire” laws.) We have the the off-print, which is what we have rights to share, so we keep and share that. It all makes sense. However, what we keep is catalogued and inventoried. Our catalogue is shared with the world via OLAC. With this tool someone can search for a resource on a language, by language. It occurs to me that the other two articles on this people group/language will not show in the aggregation of results of OLAC. This is a shame as it would be really helpful in many ways. I wish there was a groundswell, open source, grassroots web facilitated effort where various researchers can go and put metadata (citations) of articles and then they would be added to the OLAC search.
In the course of my experience I have been asked about PDFs and OCR several times. The questions usually follow the main two questions of this post.
So is OCR built into PDFs? or is there a need for independent OCR?
In particular an image based PDF, is it searchable?
The Short answer is Yes. Adobe Acrobat Pro has an OCR function built in. And to the second part: No, an image is not searchable. But what can happen is that Adobe Acrobat Pro can perform an OCR function to an image such as a .tiff file and then add a layer of text, (the out put of the OCR process) behind the image. Then when the PDF is searched it actually searches the text layer which is behind the image and tries to find the match. The OCR process is usually between 80-90% accurate on texts in english. This is usually good enough for finding words or partial words.
The Data Conversion Laboratory has a really nice and detailed write up on the process of converting from images to text with Adobe Acrobat Pro.
University Illinois Chicago explains how to do use Adobe Acrobat Pro and OCR with a scanner using a TWAIN driver.
The better OCR option
Since I work in an industry where we are dealing with multiple languages and the need to professionally OCR thousands of documents I thought I would provide a few links on the comparison of OCR software on the market.
Lifehacker has short write up of the top five OCR tools.
Of those top 5, in this article, two, ABBYY Fine Reader and Adobe Acrobat are compared side by side on both OS X and Windows.
Are all files used to create an orignal PDF included in the PDF?
The Short answer is No. But the long answer is Yes. Depending on the settings of the PDF creator the original files might be altered before they are wrapped in a PDF wrapper.
So the objection, usually in the form of a question sometimes comes up:
Is the PDF file just using the PDF framework as a wrapper around the original content? Therefore, to archive things “properly” do I still need to keep the .tiff images if they are included in the PDF document?
The answer is: “it depends”. It depends on several things, one of which is, what program created the PDF and how it created the PDF. – Did it send the document through PostScript first? Another thing that it depends on is what else might one want to do with the .tiff files?
In an archiving mentality, the real question is: “Should the .tiff files also be saved?” The best practice answer is Yes. The reason is that the PDF is viewed as a presentation version and the .tiff files are views as the digital “originals”.
The company I work for has an archive for many kinds of materials. In recent times this company has moved to start a digital repository using DSpace. To facilitate contributions to the repository the company has built an Adobe AIR app which allows for the uploading of metadata to the metadata elements of DSpace as well as the attachement of the digital item to the proper bitstream. Totally Awesome.
However, one of the challenges is that just because the metadata is curated, collected and properly filed, it does not mean that the metadata is embedded in the digital items uploaded to the repository. PDFs are still being uploaded with the PDF’s author attribute set to Microsoft-WordMore about the metadata attributes of PDF/A can be read about on pdfa.org. Not only is the correct metadata and the wrong metadata in the same place at the same time (and being uploaded at the same time) later, when a consumer of the digital file downloads the file, only the wrong metadata will travel with the file. This is not just happening with PDFs but also with .mp3, .wav, .docx, .mov, .jpg and a slew of other file types. This saga of bad metadata in PDFs has been recognized since at least 2004 by James Howison & Abby Goodrum. 2004. Why can’t I manage academic papers like MP3s? The evolution and intent of Metadata standards.
So, today I was looking around to see if Adobe AIR can indeed use some of the available tools to propagate the correct metadata in the files before upload so that when the files arrive in DSpace that they will have the correct metadata.
- The first step is to retrieve metadata from files. It seems that Adobe AIR can do this with PDFs. (One would hope so as they are both brain children of the geeks at Adobe.) However, what is needed in this particular set up is a two way street with a check in between. We would need to overwrite what was there with the data we want there.
- However, as of 2009, there were no tools in AIR which could manipulate exif Data (for photos).
- But it does look like the situation is more hopeful for working with audio metadata.
Three Lingering Thoughts
- Even if the Resource and Metadata Packager has the abilities to embed the metadata in the files themselves, it does not mean that the submitters would know about how to use them or why to use them. This is not, however, a valid reason to not include functionality in a development project. All marketing aside, an archive does have a responsibility to consumers of the digital content, that the content will be functional. Part of today’s “functional” is the interoperability of metadata. Consumers do appreciate – even expect – that the metadata will be interoperable. The extra effort taken on the submitting end of the process, pays dividends as consumers use the files with programs like Picasa, iPhoto, PhotoShop, iTunes, Mendeley, Papers, etc.
- Another thought that comes to mind is that When one is dealing with large files (over 1 GB) It occurs to me that there is a reason for making a “preview” version of a couple of MB. That is if I have a 2 GB audio file, why not make 4 MB .mp3 file for rapid assessment of the file to see if it is worth downloading the .wav file. It seems that a metadata packager could also create a presentation file on the fly too. This is no-less true with photos or images. If a command-line tool could be used like imagemagick, that would be awesome.
- This problem has been addressed in the open source library science world. In fact a nice piece of software does live out there. It is called the Metadata Extraction Tool. It is not an end-all for all of this archive’s needs but it is a solution for some needs of this type.
As part of my job I work with materials created by the company I work for, that is the archived materials. We have several collections of photos by people from around the world. In fact we might have as many as 40,000 photos, slides, and Negatives. Unfortunately most of these images have no Meta-data associated with them. It just happens to be the case that many of the retirees from our company still live around or volunteer in the offices. Much of the meta-data for these images lives in the minds of these retirees. Each image tells a story. As an archivist I want to be able to tell that story to many people. I do not know what that story is. I need to be able to sit down and listen to that story and make the notes on each photo. This is time consuming. More time consuming than I have.
Here is the Data I need to minimally collect:
Photo ID Number: ______________________________
Who (photographer): ____________________________
Who (subject): ________________________________
When (was the photo taken): _______________________
Where (Country): _______________________________
Where (City): _________________________________
Where (Place): ________________________________
What is in the Photo: ____________________________
Why was the photo taken (At what event):_________________________
Photo Description:__short story or caption___
Who (provided the Meta-data): _________________________
Here is my idea: Have 2 volunteers with iPads sit down with the retirees and show these pictures on the iPads to the retirees and then start collecting the data. The iPad app needs to be able to display the photos and then be able to allow the user to answer the above questions quickly and easily.
The iPad is only the first step though. The iPad works in one-on-one sessions working with one person at a time. Part of the overall strategy needs to be a cloud sourcing effort of meta-data collection. To implement this there needs to be central point of access where interested parties can have a many to one relationship with the content. This community added meta-data may have to be kept in a separate taxonomy until it can be verified by a curator, but there should be no reason that this community added meta-data can not be expected to be valid.
However, what the app needs to do is more inline with MetaEditor 3.0. MetaEditor actually edits the IPTC tags in the photos – Allowing the meta-data to travel with the images.In one sense adding meta-data to an image is annotating an image. But this is something completely different than what Photo Annotate does to images.
Photosmith seems to be a move in the right direction, but it is focused on working with Lightroom. Not with a social media platform like Gallery2 & Gallery3, Flickr or CopperMine.While looking at open source photo CMS’s one of the things we have to be aware of is that meta-data needs to come back to the archive in a doublin core “markup”. That is it needs to be mapped and integrated with our current DC aware meta-data scehma. So I looked into modules that make Gallery and Drupal “DC aware”. One of the challenges is that there are many photo management modules for drupal. None of them will do all we want and some of them will do what we want more elegantly (in a Code is Poetry sense). In drupal it is possible that several modules might do what we want. But what is still needed is a theme which elegantly, and intuitively pulls together the users, the content, the questions and the answers. No theme will do what we want out of the box. This is where Form, Function, Design and Development all come together – and each case, especially ours is unique.
- Adding Dublin Core Metadata to Drupal
- Dublin Core to Gallery2 Image Mapping
- Galleries in Drupal
- A Potential Gallery module for drupal – Node Gallery
- Embedding Gallery 3 into Drupal
- Embedding Gallery 2 into Drupal
This, cloud sourcing of meta-data model has been implemented by the Library of Congress in the Chronicling America project. Where the Library of Congress is putting images out on Flickr and the public is annotating (or “enriching” or “tagging” ) them. Flickr has something called Machine Tags, which are also used to enrich the content.
There are two challenges though which still remain:
- How do we sync offline iPad enriched photos with online hosted images?
- How do we sync the public face of the hosted images to the authoritative source for the images in the archive’s files?
This post is a open draft! It might be updated at any time… But was last updated on at .
Meta-data is not just for Archives
Bringing the usefulness of meta-data to the language project workflow
It has recently come to my attention that there is a challenge when considering the need for a network accessible file management solution during a language documentation project. This comes with my first introduction to linguistic field experience and my first field setting for a language documentation project.The project I was involved with was documenting 4 Languages in the same language family. The Location was in Mexico. We had high-speed Internet, and a Local Area Network. Stable electric (more than not). The heart of the language communities were a 2-3 hour drive from where we were staying, so we could make trips to different villages in the language community, and there were language consultants coming to us from various villages. Those consultants who came to us were computer literate and were capable of writing in their language. The methods of the documentation project was motivated along the lines of: “we want to know ‘xyz’ so we can write a paper about ‘xyz’ so lets elicit things about ‘xyz'”. In a sense, the project was product oriented rather than (anthropological) framework oriented. We had a recording booth. Our consultants could log into a Google-doc and fill out a paradigm, we could run the list of words given to us through the Google-doc to a word processor and create a list to be recorded. Give that list to the recording technician and then produce a recorded list. Our consultants could also create a story, and often did and then we would help them to revise it and record it. We had Geo-Social data from the Mexican government census. We had Geo-spacial data from our own GPS units. During the corse of the project massive amounts of data were created in a wide variety of formats. Additionally, in the case of this project language description is happening concurrently with language documentation. The result is that additional data is desired and generated. That is, language documentation and language description feed each other in a symbiotic relationship. Description helps us understand why this language is so important to document and which data to get, documenting it gives us the data for doing analysis to describe the language. The challenge has been how do we organize the data in meaningful and useful ways for current work and future work (archiving)?People are evidently doing it, all over the world… maybe I just need to know how they are doing it. In our project there were two opposing needs for the data:
- Data organization for archiving.
- Data organization for current use in analysis and evaluation of what else to document.It could be argued that a well planned corpus would eliminate, or reduce the need for flexibility to decide what else there is to document. This line of thought does have its merits. But flexibility is needed by those people who do not try to implement detailed plans.
Steven Egland, Doris Bartholomew, Saúl Cruz Ramos (1978) La inteligibilidad interdialectal de las lenguas indígenas de México: Resultado de algunos sondeos, Instituto Lingüístico de Verano, p. 58-59, Mexico City: Instituto Lingüístico de Verano, url, url[mendeley type="groups" id="899061" groupby="year" grouporder="desc"]
One of the most popular Citation Management software applications among academics is the application Endnote. Endnote has a long history is published by a reputable company, and has some pretty cool features. I use it (version X4) primarily because it is the only citation softwareThere is other citation management software for OS X which claims integration with pages but none of these solutions are endorsed or supported by Apple. Some of the other applications which claim integration with Pages are:
- Papers – This is according to Wikipedia, but I own and use Papers 1.9.7 and have not seen how to integrate it with Pages. (However, Papers2, released March 8th, 2011 does say that it supports citation integration with Pages.)
which integrates natively with the word processor Pages, by Apple, Inc. The software boasts a bit of flexibility and quite a few useful features. Some of the really useful features I use are below.
- Customizing the output style of the bibliographies.There are several Linguistics Journals with style sheets on Endnote’s Website. Among them are:
- Linguistic Inquiry
- Journal of the International Phonetic Association
- Journal of Phonetics
- Language: The Journal of the Linguistic Society of America
Additionally there is a version of the Unified Linguistics Style Sheet available for Endnote. This is available from Manchester UK. http://www.llc.manchester.ac.uk/intranet/ug/useful-links/computing-resources/wordprocessing/. [.ens file]
- Looking for PDF files.
- Attaching additional meta-data to each citation. (Like ISO 639-3 Language Codes)
- Adding additional types of resources like Rutgers Optimality Archive Documents with an ROA number.
- Smart Groups of files based on desired criteria.
- Integration with Apple’s word processor Pages.
- Research Notes section in the citation’s file for creating an annotated bibliography.
- Copy our all the selected works, so that they can be pasted as a bibliography in another document.
- XML Output of Citation DataThe XML Support of Endnote has not been hailed as the greatest implementation of XML but there are tools out there to work with it.
However, regardless of how many good features I find and use in Endnote there are several things about it which irk me to no end. This is sort of a laundry list of these problematic areas.
Can not sort by resource type:
For instance if I wanted to sort or create a smart list of all my Book references, or just Journal Articles. This can be done, one just has to create a smart list and then set Reference Type to Contains: “Book Section”. There is not a drop down list of reference types invoked by the user.
Can not sort by custom field:
I think you can do this in the interface. Though it was not obvious on how to do it.
- Can not view all the custom fields for a resource type across all resources.
This seems to be limited to eight fileds in the sorting viewer at a time.
Can not view all entries without content in a specified field.
This would be especially nice to create a smart list for this.
- No exports of PDFs or exports of PDFs with .ris files.
- There is no keyboard short-cut to bring up the Import function (or Export) under File Menu
- Does not rename PDFs based on metadata of the resource.
This is possible with Papers and Mendeley. The user has the option to rename the file based on things like Author, Date of publication, etc.
- Can not create a smart list based on a constant in the Issue data part.
I have Volume and Issue Data. Some of the citation data pulled in for some items has the issue set as 02, 03, etc. I want to be able to find all the issues which start with a zero so I can remove the zeros. Most stylesheets do not remove the zeros and also do not allow for them.
- Can not export PDFs with embedded metadata in the PDF.
- Can not open the folder which contains a PDF included in an Endnote Library.
- Modifying Resource type does not accept |Language| Subject Language|
There is no guide in any of Endnote’s documentation for how to create an export style sheet.
This is in the Help Menus I was expecting it on the producers website or in a book.
- When editing an entry’s meta-data i.e. the author, or the title of a work, pressing TAB does not move the cursor to the next field.
At least some times it does not continue to TAB. If I do a new entry as a Journal article, then it will tab till the issue field, but not beyond. It gets stuck.
- There is no LAN collaboration or sharing feature for a local network solution.
- There is no Cloud based collaborative solution.
- There is no way to create a smart group based off of a subset of items in a normal group.
i.e. I want to create a smart group of all the references with a PDF attached but I only want it to pull from the items in a particular group (or set of groups).
- There is no PDF Preview within the application. The existing Preview is for seeing the current citation in the selected citation style. (Preview of the output.) It would be helpful if there was also a preview pane for viewing the PDF or the attached file.
I have been looking for a way to create Posts with both Footnotes and a Bibliography section. I have wanted to make my post a little more professional looking, and let the information flow more easily with the way I write. What I have come to realize, is that Footnotes and Endnotes are different and function differently in respect to information processing. Traditionally, in print media Endnotes have occurred at the end of the article, whereas Footnotes have occurred at the end of the page on which the footnote is mentioned. This leads to a three way breakdown:
The purpose of footnotes is to facilitate quick information processing without breaking the flow of reading or information processing of the consumer of information. On web-based media, the end of the article and the end of the page is the same if pagination is not enabled. So this creates a sort of syncretism between Endnotes and Footnotes. However, the greater principal of quick reference to additional information still applies on the web. There are several strategies which have tried to fill this information processing nitch, these include things like:
- Tooltips (The pop-up text which appears when your mouse cursor hovers over a link or some other text.)
- Lightbox (The darker shading of the background and the high-lighting of the content in focus.)
- Pop-up windows (which have been phased out of popular "good web design").
- Information (Text) balloons (an example of this is Wikipop Wikipop is really a combination of the above mentioned effects above to create an inline experience for the user. But some web-sites have a similar effect which is dependent on the mouse hovering over the "trigger".).
With strategies for conveying information like Tooltips it is possible to meet the same information communication and information processing goals which were formerly achieved through footnotes. For Web-based information, which is intended to be consumed through a web medium Wiki-pop makes a lot of sense. However, if the goal is a good print out of content then footnotes are still needed, that is why I am using footnotes on this particular web presentationA solution which does both, tooltips or solutions like Wikipop, and footnotes when the content is printed, would be ideal. .
So here is a quick post on how I am doing it.
I am using two different "endnotes" plugins. One for the Bibliography section and the other for the Notes section.
Creating the Footnotes section:
To create the notes section I have elected to use a plugin called FootnotesEven though there are other options for Footnote Plugins. One other option I know about is FD-Footnotes. by Rob Miller. (Big surprise on the name of the plugin...) Footnotes allows for me to put what I want to show up as footnotes in
<ref>something</ref>In order to get these tags to display inside of
</code> tags I had to use HTML codes for the greater than sign, less than sign and slash. There is some additional good information about character encoding in HTML on Wikipedia. tags.
Additionally I can set a tag
<reference /> anywhere in the post and produce a list of footnotes.
Creating the Bibliography:
To create the Bibliography Section I am using WP-Footnotes (in the WP plugins repository) by Simon Elvery. More information can be read about his plugin here. What this plugin allows me to do is to craft the citation of the item I want to cite. I have to figure out how I want to "code" the citation and then present the citation.
This will produce a citation marker (a number) as a super script inline with the text. Like this Nikolaus P. Himmelmann. 1998. Documentary and Descriptive Linguistics. Linguistics vol. 36:161-195. [PDF] [Accessed 24 Dec. 2010] :
And that will produce a citation in the bibliography section like the following:
One interesting thing that occurs on the admin side of WordPress is that the plugin WP-Footnotes has an options page which shows up in the Settings menu, however what is interesting is that in that in the menu it is called Footnotes, not WP-Footnotes.
The options for WP-Footnotes really make it flexible, it is these setting which have allowed me to rename the section from Notes to Bibliography.
Is this my final solution? No. One thing I really don't like is that the bibliography is not orderd in alphabetical order of the last names, and then in order of the year of publication. Rather, citations are ordered in the order of appearance (as footnotes generally are). The plugin does not have any options for changing the order that thing appear in (though the headings on the ordered list can be changed). There is also no way to structure the data in the bibliography for reuse (even if it is just within this site), so each use of each citation must be hand-crafted with love. There are some other solutions which I am looking at integrating with this one but have not had time to really explore. One options is to integrate with Mendeley and aggregate bibliography data from a Mendeley collection. Another option is to create bibliographies as bibtex files and then use those to display the bibliography.
I am looking to re-skin Wikindex. I thought that I would add some CSS classes that would embed the meta-data in a manner that the citations could be picked up by Zotero quite easily. It seems to be a bit more difficult than I first anticipated. As a Microformat for citations is not yet been fully fleshed out. Obviously one way to go would be to embed everything in a
span element as COinS does but that is not really what I am looking for. (Mostly because I don’t have a way to generate the
Attributes in the
span element automatically.) I have thought of using RDFa. But I still need to do some more research and see what can be gleaned in terms of which controlled vocabularies to use. I am hoping that this Lesson On RDFa will really help me out here. Finally I do need to know something about OAI so that once the Resources are put into Wikindex I can then tell OLAC what language they belong to.
One of the projects I have been involved with published a paper this week in JIPA. It is a first for me; being published. Being the thoughtful person I am, I was considering how this paper will be categorized by librarians. For the most part papers themselves are not catalogued. Rather journals are catalogued. In a sense this is reasonable considering all the additional meta-data librarians would have to create in their meta-data tracking systems. However, in today’s world of computer catalogues it is really a shame that a user can’t go to a library catalogue and say what resources are related to German [deu]? As a language and linguistics researcher I would like to quickly reference all the titles in a library or collection which reference a particular language. The use of the ISO 639-3 standard can and does help with this. OLAC also tires to help with this resource location problem by aggregating the tagged contents of participating libraries. But in our case the paper makes reference to over 15 languages via ISO 639-3 codes. So our paper should have at least those 15 codes in its meta-data entry. Furthermore, there is no way for independent researchers to list their resource in the OLAC aggregation of resources. That is, I can not go to the OLAC website and add my citation and connect it to a particular language code.
There is one more twist which I noticed today too. One of the ISO codes is already out of date. This could be conceived of as a publication error. But even if the ISO had made its change after our paper was published then this issue would still be persistent.
During the course of the research and publication process of our paper, change request 2009-78 was accepted by the ISO 639-3 Registrar. This is actually a good thing. (I really am pro ISO 639-3.)
Basically, Buhi’non Bikol is now considered a distinct language and has been assigned the code [ubl]. It was formerly considered to be a variety of Albay Bicolano [bhk]. As a result of this change [bhk] has now been retired.
Here is where we use the old code, on page 208 we say:
voiced velar fricative [ɣ]
- Aklanon [AKL] (Scheerer 1920, Ryder 1940, de la Cruz & Zorc 1968, Payne 1978, Zorc 1995) (Zorc 1995: 344 considers the sound a velar approximant)
- Buhi’non [BHK] (McFarland 1974)
In reality McFarland did not reference the ISO code in 1974. (ISO 639-3 didn’t exist yet!) So the persistent information is that it was the language Buhi’non. I am not so concerned with errata or getting the publication to be corrected. What I want is for people to be able to find this resource when they are looking for it. (And that includes searches which are looking for a resource based on the languages which that resource references.)
The bottom line is that the ISO does change. And when it does change we can start referencing our new publications and data to the current codes. But there are going to be thousands of libraries out there with out-dated language codes referencing older publications. A librarian’s perspective might say that they need to add both the old and the new codes to the card catalogues. This is probably the best way to go about this. But who will notice that the catalogues need to be updated with the new codes? What this change makes me think is that there needs to be an Open Source vehicle where linguists and language researchers can give their knowledge about a language resources a community. Then librarians can pull that meta-data from that community. The community needs to be able to vet the meta-data so that the librarians feel like it is credible meta-data. In this way the quality and relevance of Meta-data can always be improved upon.