Author Archives: Hugh Paterson III

TIFFs, PDFs and OCR

Posted on by
Reply

In the course of my experience I have been asked about PDFs and OCR several times. The questions usually follow the main two questions of this post.

So is OCR built into PDFs? or is there a need for independent OCR?

In particular an image based PDF, is it searchable?

The Short answer is Yes. Adobe Acrobat Pro has an OCR function built in. And to the second part: No, an image is not searchable. But what can happen is that Adobe Acrobat Pro can perform an OCR function to an image such as a .tiff file and then add a layer of text, (the out put of the OCR process) behind the image. Then when the PDF is searched it actually searches the text layer which is behind the image and tries to find the match. The OCR process is usually between 80-90% accurate on texts in english. This is usually good enough for finding words or partial words.

The Data Conversion Laboratory has a really nice and detailed write up on the process of converting from images to text with Adobe Acrobat Pro.

Daily Designer has a tutorial on how to do it on OS X.
David R. Mankin explains on his blog what the process looks like using Windows.

One of the beauties of Adobe Acrobat Pro is that this process can be scripted and the TIFFs processed in batches.
[On Windows] :: [On OS X using AppleScript] :: [Cross platform help from Adobe]

University Illinois Chicago explains how to do use Adobe Acrobat Pro and OCR with a scanner using a TWAIN driver.

The better OCR option

Since I work in an industry where we are dealing with multiple languages and the need to professionally OCR thousands of documents I thought I would provide a few links on the comparison of OCR software on the market.
Lifehacker has short write up of the top five OCR tools.

Of those top 5, in this article, two, ABBYY Fine Reader and Adobe Acrobat are compared side by side on both OS X and Windows.

Are all files used to create an orignal PDF included in the PDF?

One thing to remember, Which I have said before, is that not all PDFs are created equal. This Manual talks a bit about different settings inside of PDFs when using Adobe’s PDF printer.

The Short answer is No. But the long answer is Yes. Depending on the settings of the PDF creator the original files might be altered before they are wrapped in a PDF wrapper.

So the objection, usually in the form of a question sometimes comes up:

Is the PDF file just using the PDF framework as a wrapper around the original content? Therefore, to archive things “properly” do I still need to keep the .tiff images if they are included in the PDF document?

The answer is: “it depends”. It depends on several things, one of which is, what program created the PDF and how it created the PDF. – Did it send the document through PostScript first? Another thing that it depends on is what else might one want to do with the .tiff files?

In an archiving mentality, the real question is: “Should the .tiff files also be saved?” The best practice answer is Yes. The reason is that the PDF is viewed as a presentation version and the .tiff files are views as the digital “originals”.

Agricola

Posted on by
Reply

This past weekend I had the opportunity to play Agricola for the first time. It is an interesting game. I have played it a few times now. Twice with 4 people and twice with 2 people. It takes a while to wrap one’s head around the game play. But once I got it I had to evaluate it. The verdict is in. I would rather play Ticket to Ride than Agricola. Agricola is a great game, don’t get me wrong, it has many intricate plays, and a lot of variety. It is completly a different game as a two player game than it is as a four player game. – it is just that it takes too long to put the pieces back in the box.

Seriously though for the brain strain that it causes (and I enjoy brain strain) I would rather play Ticket to Ride.

Interoperability of online dictionary data: 
 A test case using WordPress as a CMS

Posted on by
Reply

Linked data is an effort to enhance applications and thereby lives with structured knowledge. This structure at its core is developed by human interaction. The challenge to consumers of linked data is to convince holders of unstructured data to structure it into actionable, manipulatable knowledge. Continue reading →

Metadata Magic

Posted on by
Reply

The company I work for has an archive for many kinds of materials. In recent times this company has moved to start a digital repository using DSpace. To facilitate contributions to the repository the company has built an Adobe AIR app which allows for the uploading of metadata to the metadata elements of DSpace as well as the attachement of the digital item to the proper bitstream. Totally Awesome.

However, one of the challenges is that just because the metadata is curated, collected and properly filed, it does not mean that the metadata is embedded in the digital items uploaded to the repository. PDFs are still being uploaded with the PDF’s author attribute set to Microsoft-WordMore about the metadata attributes of PDF/A can be read about on pdfa.org. Not only is the correct metadata and the wrong metadata in the same place at the same time (and being uploaded at the same time) later, when a consumer of the digital file downloads the file, only the wrong metadata will travel with the file. This is not just happening with PDFs but also with .mp3, .wav, .docx, .mov, .jpg and a slew of other file types. This saga of bad metadata in PDFs has been recognized since at least 2004 by James Howison & Abby Goodrum. 2004. Why can’t I manage academic papers like MP3s? The evolution and intent of Metadata standards.

So, today I was looking around to see if Adobe AIR can indeed use some of the available tools to propagate the correct metadata in the files before upload so that when the files arrive in DSpace that they will have the correct metadata.

  • The first step is to retrieve metadata from files. It seems that Adobe AIR can do this with PDFs. (One would hope so as they are both brain children of the geeks at Adobe.) However, what is needed in this particular set up is a two way street with a check in between. We would need to overwrite what was there with the data we want there.
  • However, as of 2009, there were no tools in AIR which could manipulate exif Data (for photos).
  • But it does look like the situation is more hopeful for working with audio metadata.

One way around the limitations of JavaScript itself might be to use JavaScript to call a command-line tool or execute a python, perl, or shell script, or even use a library. There are some technical challenges which need bridged when using these kinds of tools in a cross-platform environment. (Anything from flavors of Linux to, OS X 10.4-10.7 and Windows XP – Current.) This is mostly because of the various ways of implementing scripts on differnt platforms.

The technical challenge is that Adobe AIR is basically a JavaScript environment. As such there are certain technical challenges around implementation of command-line tools like Xpdf from fooLabs and Coherent PDF Tools or Phil Harvey’s ExifTool, Exifv2, pdftk, or even TagLib. One of the things that Adobe AIR can do is call an executable via something called actionscript. There are even examples of how to do this with PDF Metadata. This method uses PurePDF, a complete actionscript PDF library. Actionscript is powerful in and of itself, it can be used to call the XMP metadata of a PDF, Though one could use it to call on Java to do the same “work”.

Three Lingering Thoughts

  1. Even if the Resource and Metadata Packager has the abilities to embed the metadata in the files themselves, it does not mean that the submitters would know about how to use them or why to use them. This is not, however, a valid reason to not include functionality in a development project. All marketing aside, an archive does have a responsibility to consumers of the digital content, that the content will be functional. Part of today’s “functional” is the interoperability of metadata. Consumers do appreciate – even expect – that the metadata will be interoperable. The extra effort taken on the submitting end of the process, pays dividends as consumers use the files with programs like Picasa, iPhoto, PhotoShop, iTunes, Mendeley, Papers, etc.
  2. Another thought that comes to mind is that When one is dealing with large files (over 1 GB) It occurs to me that there is a reason for making a “preview” version of a couple of MB. That is if I have a 2 GB audio file, why not make 4 MB .mp3 file for rapid assessment of the file to see if it is worth downloading the .wav file. It seems that a metadata packager could also create a presentation file on the fly too. This is no-less true with photos or images. If a command-line tool could be used like imagemagick, that would be awesome.
  3. This problem has been addressed in the open source library science world. In fact a nice piece of software does live out there. It is called the Metadata Extraction Tool. It is not an end-all for all of this archive’s needs but it is a solution for some needs of this type.

The British Kerfuffle

Posted on by
Reply

I was rather shocked when I heard about the Riots in the UK. The UK‽ Really‽ That’s different. Kind of disturbing really.Jamie Oliver’s Restaurant was burnt. Why? This is perhaps the saddest news out of England in the last 48 hours. (Not that the BBC’s Photographs don’t make the situation look like a scene from V for Vendetta.)

BBC Map of UK Riots

UK Riots

I got to thinking about some of the implications of the reporting coming out of the BBC.

Officers believe some rioters have used BlackBerry Messenger – a service allowing users to send free real-time messages – to organise violence.I often see the words gang and masked youths used in reporting. One must keep in mind that gangs in the UK do not mean the Crips and Bloods, rather a more generic idea of swarm of people. Also keep in mind that the UK has more street video cameras and video surveillance than any other country, so wearing a mask is logical for this kind of activity.

And I was told that there was some use of social media platforms like FaceBook and Twitter. In particular, Facebook groups. Allegedly, locals on Facebook were reporting the groups as fast as they were being created. An interesting social dynamic as opposing social forces collide. I got to thinking about these alleged locals using social media and wondered: What if someone who was not in the UK was organizing or creating the Facebook groups. That would be something. It would be a way that sympathizers who were not local or in the UK could lend a hand to people on the ground. This lead me to think about how would law enforcement react to this sort of help to the rioters.

Law in the U.S.A. is different than in the U.K. But none the less if it were in the U.S. and people were to lend a hand then those lending a hand might be considered to be Terrorist. Using the definition preferred by the state department, terrorism is: “Premeditated, politically motivated violence perpetrated against noncombatant* targets by subnational groups or clandestine agents, usually intended to influence an audience.” [Quote taken from The definition of terrorism]It seems though that this definition agrees with what I read in the Patriot Act some years ago. It also agrees with Wikipedia and About.com. One thing that I see in the current news is that there is no political motivation. So some might say that this is not terrorism because it lacks motive.

But if it were terrorism, and the individuals were known through FaceBook, would the UK ask for their extradition to the UK? – to put them on trial as terrorists? Would UK law apply outside of the UK? If the person is creating a FaceBook group and is doing from outside of the UK and several blokes join a group does that make the creator of the group a terrorist? and also guilty of breaking UK law? Can someone break another country’s law without leaving their current country – even country of citizenship? What gives the UK legal rights prosecute?

The other thing which is disturbing in this social media business is the global social reaction to the difference between what is happening in the UK right now and what happened in Libya, Morocco, Egypt, and Syria earlier this year. Social media was part of that unrest as well, however, political motivation was definitely part of that. Yet the rioting was deemed to be for a “worthy” cause. Yet if the same actions were taken in the U.S. we would call it Terrorism. I think The Guardian really points out some of the idiosyncrasy of the U.S. definition and the application of the term Terrorism as it relates to crime.

AKAI GX-220D

Posted on by
Reply

This past weekend I walked across the street from our house with Becky and went to an estate sale. In one of the rooms there was a stack of electronics. There on a dresser was a Reel-to-Reel machine in almost perfect condition.

AKAI GX-220D

AKAI GX-220D

The week prior I had been working with some Reels of language data in the archive. We have two machines but one of them does not work completely. Some amazing stuff. So now it sits on my desk and I am using my Marantz PMD 661 to digitize all sorts of Reel to Reels. Today I was looking up this model online and found it on Ebay for $300 and in Germany on Ebay for 500€, not bad as I paid $35 for mine. I finally found the manuals online for Free! It seems that somebody is trying to sell print offs on ebay for $11 each. But here is what I found.

Specifications

Track system: 4 track, 2 channel stereo/monaural system

Maximum reel capacity: 7″ reel

Wow and flutter: 0.08% at 7.5 ips

Frequency response: 30Hz to 24kHz at 7.5ips

Distortion: less than 1.5%

Signal to noise ratio: >50dB

Heads: 3 heads

Motors: 3 motors

Dimensions: 430 x 425 x 230mm

Weight: 19kg

One of my concerns is with the 30Hz playback.

Was this standard for reel-to-reels or was there better recordings done but I will never know because of the model of playback machine I have?

Another concern I have is about the heads:

If they are magnetized will they erase my tape? How do I tell if they are magnetized?

Here is a listing of Reel-to-Reel machines by AKAI.

Here is a youtube video of the same model I have.

Many Facets to Planning a Website

Posted on by
Reply

I found this image which I think explains many of the kinds of things that Website builders need to think through. For some in the industry this is like.. Duh! but for others this kind of layout really helps us see the complexity and the parts we need to be thinking through to implement the website.

Many of the different kinds of things that website planners have to think through.

Cars

Posted on by
Reply

Who would have thunk I that I would do a post on cars? Mostly, I avoid car discussions. I have avoided these kinds of discussions since high-school, when all my friends raved about NASCAR and North American muscle cars, I have regularly reminded them that the best cars in the world prove their excellence in a race by negotiating turns in multiple directions, as well as negotiating changes in track elevation. By-the-way, RACECAR is a palindrome.

This posture has landed me quite a bit of criticism over the years. Aside from this, my practical side has suggested that a good car is one that allows one to get from ‘point A’ to ‘point B’.

My first car was a Renault, followed by a 1983 Honda Prelude, which in turn was followed by 1996 Honda Accord LX. I currently drive a 2005 Kia Spectra, which my wife bought before we were married.

Renault

1982 Renault - Mine was all light blue with a grey interior.

Honda Prelude

1982 Honda Prelude - Mine was this exact color scheme.

1996 Honda Accord LX

Hugh’s Honda Front
Interior of Hugh’s Honda

2005 Kia Spectra

Driving from 2007 through 2011

But recently Becky and I have been looking at cars, not to buy one, just to look at them… and so I have been looking at some cars with more of an artistic eye rather than a functional or economical (esp. fuel efficiency) eye.

Several cars have caught my eye. I have long been a fan of the Porsche 911 and Porsche Boxter. But here are some other cars that look rather appealing. Most of these cars I have never even ridden in. Concerning color, I really like yellow. However, Bi-tonal color schemes like is common with the Toyota FJ and the Mini Cooper are also really cool. Of course Blue and white, Blue and golden (yellow), or Black and White all go nicely together.

Porsche Boxter

Porsche Boxster

Mini Cooper

Blue and White Mini Cooper
Mini Cooper with Union Jack
Mini Cooper Clubman
Mini Cooper White with Black Stripes

The Union Jack looks really cool on either the roof or the hood of this car. But being that my family color is blue, I thought that perhaps a Scottish flag might fit better.

Scottish Flag

Dodge Challenger

I always find the Dodge Challenger an agressive and smart looking car.

Grey Dodge Challenger
Silver and Black

Toyota FJ

Black and White Toyata FJ
On vehicle where it needs a roof rack to look good.
Blue and White 2007 Toyota FJ Cruiser
Yellow and White 2012 Toyota FJ Cruiser
Silver 2010 Toyota FJ
Blue and White 2010 Toyota FJ

Mazda Protege P5

Red Mazda Protege P5
Yellow Mazda P5

Subaru Baja

The Subaru Baja is also a nice car.

Black Subaru Baja
Subaru Baja
Yellow on Silver Subaru Baja 2005
2003 Subaru Baja with Roof Rack

How this metadata stuff is stored

Posted on by
Reply

This is an introduction to how this metadata is stored.

http://wiki.gbif.org/gbif/wikka.php?wakka=MMMetaData

XMP Sidecar Files
A sidecar file is an alternative to storing the metadata directly in the image file itself by instead storing the data in a separate .xmp file with the same base name as the photo. Sidecars are typically used in cases where the file format of the photo doesn't directly support embedding metadata or in cases when the image file should not be edited directly. It should be noted that very few programs support the reading the xmp sidecar files, most will default to reading and writing to the photo directly.

Gracefully copied from http://www.earlyinnovations.com/photolinker/xmp-sidecar-files.html

Note: that side car files are separate files from the photos so if that photo were to be archived it would need to form a package of two files a sidecar file and the main photo image.

Embedded Metadata
Exif
http://www.digicamhelp.com/glossary/exif-data/
http://www.opanda.com/en/iexif/
http://www.sno.phy.queensu.ca/~phil/exiftool/TagNames/EXIF.html
http://www.stuffware.co.uk/cheese/

IPTC
http://www.iptc.org/std/photometadata/2008/specification/IPTC-PhotoMetadata-2008.pdf

The funky Dublin Core Metadata Schema
http://www.iptc.org/std/Iptc4xmpCore/1.0/specification/Iptc4xmpCore_1.0-spec-XMPSchema_8.pdf
http://www.prismstandard.org/about/relationships.asp

Ok, so the items which can be recorded in Exif data spots, might not be able to be recorded in ITCP spots and vise versa. That is that some elements like copyright holder or photographer can be recorded in the ITCP data but not in the Exif data. This means that there is not 100% correspondence between the two sets. We can not choose to use one and ignore the other. When we throw XMP into the mix there are attentional things which can be recorded in XMP but not in either of the other sets. Additionally, XMP is in its own file, not embedded. Dublin Core (DC) is also a set of options for metadata. They are not embedded in the photo itself, rather in a way it is a way to organize a database of metadata about objects. REAP uses DC. DC is extensible, that is we can move embedded metadata (or sidecars) from photos into the REAP database's metadata structure. But then what happens when the photo is removed from the REAP container. Does the metadata travel with it?

Here is a clip gracefully copied from http://www.earlyinnovations.com/photolinker/annotation-philosophy.html

Many popular websites and applications allow you to annotate your photos by adding keywords, a description, a title, a location, a list of the people in the photograph and many other tags. These websites and applications generally suffer from two major deficiencies:

Annotations are often exclusively added to a propriety database, and not written back to the photo. This means that unless the software or website is still available in, say, 50 years, the annotations will be completely lost.
Programs that do write the annotations directly to the image file usually corrupt existing tags or write partial information.
PhotoLinker solves both of these issues.

  1. PhotoLinker write the annotations directly to the photo so that your annotations stay with the photo forever. After annotating with PhotoLinker you can use other programs or upload to popular websites with the knowledge that your annotations will stay with your copy of the photos.
  2. PhotoLinker is one of first application to adhere to Metadata Working Group Guidelines for Handling Image Metadata. These guidelines ensure that annotations are handled correctly. In addition, PhotoLinker maintains transparency about how it handles the metadata by using open source tool ExifTool and showing exactly which tags are between read and written.