A Metadata Exercise:
Created by Hugh
Phillips
Wisconsin
State Cartographer's Office and the University
of Wisconsin-Madison General Library System
for the Federal
Geographic Data Committee
adapted by LAGIC, August, 1999
The original exercise is available On-Line @ http://badger.state.wi.us/agencies/wlib/sco/metex/psylabus.htm
- PC, DOS/Windows
Web browser, mp, cns, and Metamaker 2.0 or a word processor or text editor (download instructions for these tools given later in exercise).- NOTE: This exercise uses Metamaker 2.0. However, CorpsMet95 is also an acceptable application.
- Metadata Element or Element
- One of the specific named parts of the Content Standards for Digital Geospatial Metadata, e.g. Identification_Information or Abscissa_Resolution
- Value
- What is entered into a Metadata Element that is not a Container element. For example: Wisconsin could be a Value for the Place_Keyword element.
- Parent
- The Metadata Element which is one hierarchical level above the currently considered element and which is a 'Container' for it. For example: Citation is the Parent of Citation_Information.
- Children
- The Metadata Elements which are one or more hierarchical levels below the currently considered Parent element. For example: Series_Name and Issue_Identification are Children of Series_Information.
- Compound or Container Element
- A Metadata Element that does not itself explicitly contain metadata information, but that acts as a folder for Children elements below it hierarchically. Elements that are Parents are always Container elements. For example: Spatial Domain is a Container element for Bounding_Coordinates and Data_Set_G-Polygon, but does not itself take a Value.
- Production Rules
- The definition of the CSDGM which encodes which elements are mandatory, optional, parents, or children, what values certain elements may take, and how many times each particular element may repeat.
Introducing the Metadata Format:
Specific formatting for metadata is not a part of or required by the federal metadata standard. Unfortunately, the freedom afforded by the current lack of formatting rules in the CSDGM has cost us dearly. An entire generation of metadata tools has grown up that cannot accept or export a generic form of metadata. Individual agencies produce metadata which is CSDGM compliant (for content) but it is formatted in different ways, thus it cannot be easily incorporated into a searchable Clearinghouse or exchanged. The FGDC is currently considering an encoding standard for metadata which will address this problem. It is based on the use of SGML.
At the present time the closest thing to exchangeable metadata is
metadata that can pass Peter Schweitzer's metadata parser, mp. This
tool can produce SGML output. So by insuring that metadata can pass his tool
now, one safeguards the work invested in metadata production for the time
when more sophisticated metadata tools are available. We will make extensive
use of the metadata parser in this workshop.
Installing your Metadata Tools
When the extraction is complete you will have a directory structure like so:
The tools\bin\dos directory contains the executable versions of cns and mp. The tools\doc directory contains documentation and help files.
You should now have a more complicated directory structure like so:
The tools\metadata directory contains metadata files for the exercise. The tools\cfg directory contains a configuration file for mp and an alias file for cns.
setws c:\tools\bin\dos
If they are in d:\junk\tools\bin\dos, then type:
setws d:\junk\tools\bin\dos
Now test the operation of mp and cns. From the DOS prompt change directory to tools\metadata. Try mp by typing:
mp
You should get a message back announcing the use of DOS4GW, and the possible parameters to use on the command line with mp. If you don't, and get an error message like 'Command not found,' then the path to mp is incorrect, and you may need to redo the above. Try out cns the same way, by typing:
cns
If you need to restore your path to what it was before running 'setws,'
then run the batch file 'unsetws' from the tools\metadata directory. You
may find that going back and forth between Windows and DOS causes your
path to be reset. You can rerun 'setws' if necessary. I recommend
keeping a DOS window open in Windows that you can use for mp and cns
operations when needed.
- apbad
- apbad2
- apgood
- csdgmtab.txt
- gmgb93
- maratax.asc
- setws.bat
- unsetws.bat
At this point, we are really ready to start the exercise for the workshop. It isn't my intention that workshop time be devoted to explaining the software installation and checkout; I hope this is all done before the workshop.
mp is very picky about two things:
These are not petty eccentricities of mp, it uses them to create a map of your metadata. To meet the two above requirements, mp's companion tool cns is often useful. We will use cns later.
You should be in the tools\metadata directory, at the DOS prompt to continue...
Type mp without parameters to see its command line options.
If you got a 'Command not found' error, then look back at the setup procedure for this tool.mp
mp -e maratax.err -h maratax.htm -s maratax.sgm maratax.ascError Report: Take a look at the error report, maratax.err with a text editor. The warnings are just that - in the identification of the Marathon County coordinate system under Other_Grid_System's_Definition, mp has detected some valid metadata element names that seem out of place (they follow a free text domain element). The errors for Abscissa_Resolution and Ordinate_Resolution are more serious - these are mandatory elements that expect a real number, not the text string 'unknown.'
HTML File: Take a look at the HTML output, maratax.htm in \tools\metadata with your Web browser.
SGML File: Take a look at the SGML (Standardized General Markup Language) output, maratax.sgm in \tools\metadata with a text editor. It looks pretty ugly, no? It wasn't meant for people to read or work with directly, but it makes it much easier to index the metadata. To those familiar with HTML tags from Web pages, SGML markup will look similar, with start and end tags around elements. This is the last we will look at SGML.
mp -e apbad.err apbad
The error report should look like this. Try to figure out what caused these errors, fix them, and run apbad through mp again. Less errors this time, I hope? Remember that often errors are related, and that by fixing one key problem, many others may disappear. Look for blocks of errors, and usually the culprit will be revealed. Here is the explanation of the error report for the apbad file.
Did your repair work actually cause the reported number of errors to increase? If so, you are a natural for this stuff! It means you actually tried to do something, instead of saying "Yeah, yeah this is easy," and just skipping on ahead. Probably what happened was that you messed up the hierarchical indentation. cns, which we will look at next, can deal with that kind of problem. Don't spend a lot of time slaving on trying to get this data to pass mp perfectly, we are just sort of fooling around with these tools here, not too serious, and there is a bunch of other stuff to do!
Get a directory:mp -c ..\cfg\ws.cfg apgoodTake a look at the HTML output file, tools\metadata\apgood.htm with your Web Browserdir apgood.*
You should still be in the tools\metadata directory, at the DOS prompt, to continue...
Type cns without parameters to see its command line options.
If you got a 'Command not found' error, then look back at the setup procedure for this tool.cns
That error report should look familiar.mp -e apbad2.err apbad2Now, let's pre-clean apbad2 with cns, produce a leftovers file, and an output file. Type:
Now run the output from cns through mp:cns -e apbad2.lo -o apbad2.out apbad2Look at apbad2.err. Not bad, huh?mp -e apbad2.err apbad2.out
As we showed above, cns can clean up minor 'glitches' in metadata. However, it really shines when the formatting of the source metadata has no hope of getting through mp without hundreds of errors. We are talking about metadata files that would take hours of editing with a word processor to make acceptable to mp. The output from Metamaker is a prime example.
First, let's try to jam some CorpsMet95 output, gmgb93, directly through mp. Type:
Look at the error report, tools\metadata\gmgb93.err.mp -e gmgb93.err gmgb93
315 errors!
Okay, now let's try again after passing gmgb93 through cns.
Take a look at the formatting of the output (\tools\metadata\gmgb93.out) from cns. It's quite a bit different from the source gmgb93, no?cns -e gmgb93.lo -o gmgb93.out gmgb93Pass the output from cns through mp.
Now look at the error report from mp for the output from cns. It looks much better now, only 37 errors, most of them due to a characteristic of Metamaker 2.0 to pass through some empty container elements to the output report. Metamaker 2.x, which you may use in the next part of this exercise, has improved in that respect.mp -e gmgb93.err gmgb93.outNote: Corpsmet95 does a pretty good job keeping the errors to a minimum. However, there are some instances when a <tab> or a misplaced <space> in text-box will cause syntax errors. These errors are often corrected by re-typing the values or <copy><paste> the values into the text box from a word processing editor.
How
To: Using
Metamaker
Click
here for Corpsmet95
Introduction
MetaMaker is a Microsoft
Access based metadata tool developed at the Environmental
Management Technical Center in Onalaska, Wisconsin.
Because this tool makes some use of its foundational database, it is possible to:
This tool is designed to support documentation of biological type data
sets so you may notice that there are some sections possible to fill in
which are not part of the CSDGM (these are NBII
extensions to the CSDGM). Additionally, some element names have been
changed to improve the direct interpretation of what they mean. This alias
file will help cns make the translation of MetaMaker
element names to the CSDGM standard name for MetaMaker's
output reports.
From the MetaMaker's main menu screen, select Template Menu.
From the Template Menu, select Citation Section 8.
In the white box under Citation Template Name, where it currently says 'Name and Save template,' type a name for your template (with an awareness that only the first 36 characters will be visible on pick lists in the seven major forms), then select the 'Save Template' button. Metamaker will inform you it has completed the save.
You may question why the Citation_Information for the data set you are documenting is placed into a template, because 'template' implies that this Citation_Information might be used again. Well, it could be, if other data sets in the future are derived from the one you are documenting now, in which case this information would be used in a Source_Citation on the Lineage description for that future data set.
You now need to enter your Citation Information. Click on the arrow button to the right of the 'Select Citation to Edit' button, then drag down until you find your new Template name, then release the mouse button.
Click on the 'Add' button on the 8.1 Originator line in the form. A white box will open to the right. Type in an Originator name for the data set you are documenting, then click the 'Done' button when you have finished entering the Originator.
Based on the logic of Susan Stitt's graphical representation of the standard (including the color), you may be able to discern that the only other elements of this section (Citation_Information) that you need to fill in are the Publication_Date and the Title. Mouse click in the white box to the right of these elements, then type in the appropriate information.
Pick the 'Save Template' button from the top of this form, then the 'Go Back to Menu' button.
Select Contact Section 10.
As for the Citation Template, give this template a name, then select 'Save Template.'
Use the the arrow to the bottom right of the 'Select Contact to Edit' button to find your new Contact template name, then release the mouse when you template name is highlighted.
Go down into the yellow part of the form and select the Add button for either Contact_Person_Primary or Contact_Organization_Primary. You need only fill out the elements which are yellow, then select the 'Done' button.
Under Contact Address you need to fill in at least Adress_Type, City, State_or_Province, although some address lines would be normal as well. Notice that clicking the arrows to the right of Adress_Type and State_or_Province will reveal a 'pick list' of choices.
If you have more than one Address_Type, then when you have completed the first one, go down to the bottom of the Contact_Adress part of the form where it should now say 'Record 1 of 1' and click the right arrow once so the bottom of the form now says 'Record 2 of 2.' Complete the second address information.
You need to enter at least one 10.5 Contact_Voice_Telephone, so pick the Add button and go to it.
Go to the top of the form, pick the 'Save Template' button, then
select the 'Go Back to Menu' button.
Scroll down through the list until you find 1.6.1.2 (comes well after 1.12.2.3!), Theme Keyword, Air Temperature, Archaeological etc. We are going to add a new value to the pick list for Theme Keyword, Wilderness Area. Scroll on through the Theme Keywords and see that there isn't such a pick already.
From the top of the form, select 'Add New Keyword.' This will take you clear to the bottom of the list, where there is one empty row. Click your mouse in the left-most empty box and enter the code number '1.6.1.2,' tab to the next box and enter 'Theme Keyword,' and then tab to the last box and enter 'Wilderness Area.' Notice that another row has opened up in case you want to add additional pick list values.
After completing the Wilderness Area entry, select 'Save Keyword' from the top of the form, then 'Go Back to Menu.' From the Template Menu, select Keyword Pick List again, and scroll down into 1.6.1.2 Theme Keyword and see that Wilderness Area has been moved into that section in the proper alphabetical order.
Select 'Go Back to Menu' from the top of the form.
From the top of the Template Menu form, pick 'Go Back to Main Menu.'
At mid-form should be listed the existing datasets in Metamaker's metadata database. Handily, it comes with some examples.
For a Good Time click the 'Tip for Adding a New Metadata' button.
Type a name (descriptive, not just a file name) for the data set you will document into the white box. Click the 'Save Metadata Datset' button at the top of the form, then select 'Go Back to Edit Menu.'
Click on the arrow at the right end of the grey box under 'Select Metadata Dataset to View or Edit.' Highlight your newly named dataset.
Now under the 'Selected Metadata Dataset' it should say your new dataset name.
Underneath this is a big empty box labeled 'Abstract.' Don't try to type anything into the empty box. It won't accept entry. Abstract Information *appears* in this box after you have completed it through the Identification_Information form.
Click the 'Go to Identification 1' button below the Abstract box.
At the top of the yellow part of the form is 1.1 Citation Information. You can create a new Citation_Information section by selecting Open 8 Citation_Information_Form. We already did that though, so click the grey arrow to the right of the grey box to the right of the Citation_Information element to display the existing Citation_Information templates. You will see from this display why a short name was suggested for the Citation_Information template, long names are truncated in this display.
Select your Citation_Information template. The list will disappear and in the white box to the right of Citation_Information, your Citation_Information template name will appear.
Move on down to Abstract, and click in the white box to start typing.
Lines will wrap automatically, but just hitting the 'return' key does
nothing (nothing useful, anyway).
We're doing this by the book now, so type something meaningful into the Pupose box also.
Work your way down to Time_Period_of_Content, then select 'Open 9 Time Period Information.'
Is it me? I just can't seem to name these Time_Period templates so the names have much meaning, and so I can remember what they relate to. Maybe it doesn't matter if you use them immediately while you still remember what they are for.
Okay, we're back from that random thought. Using the arrow to the lower right of the 'Select Time Period to Edit' button, select the name you just gave your Time_Period_Information template.
Fill in one of: Single Date and Time, Multiple Dates and Times, or Range of Dates and Times.
Select 'Save Template,' then 'Go Back to Menu.'
1.3.1 Time Period of Content Date Explanation is the CSDGM 'Currentness_Reference' element, so fill it in appropriately (pick list).
Under 1.4 Status, both Progress and Maintenance_and_Update_Frequency have pick lists.
Under Geographic Extent, you need only fill the West, East, North, and South Bounding Coordinate (decimal degrees!).
Proceed down to 1.6 Keywords, and for the Theme Keyword Thesaurus enter 'none.'
Select the 'Add' button by Theme Keywords. Click the small down arrow to the right of the grey box there and observe the keywords - did you see the 'Wilderness Area' choice that you added to the pick list earlier? Use one (or more) of the pick list keywords if appropriate for your data set or else type your own into the white boxes.
The Keywords section is one of several places where Metamaker is still deficient. For example, the CSDGM allows you as many Keyword Thesaurus as you want; Metamaker only allows you one.
You also nee to fill in 1.8 Access_Constraints and 1.9
Use_Constraints. Let's do this by cutting and pasting in from another
Windows window.
This same copy and paste method could be used to enter data into Metamaker from any file you may have open.
After completing both of these elements, go to the top of the form, select 'Save Metadata,' and then select 'Go Back to Edit Menu.'
On this 'Select Metadata Dataset to View or Edit' page, notice that now, the Abstract box for your data set now has some information in it (you did write an Abstract for your data set didn't you?).
Select the 'Go the Metadata Ref 7' button under the Abstract box.
For Metadata_Contact, use the grey arrow to expose the pick list containing the Contact_Information template we filled in earlier.
From the 7.5 Metadata_Standard_Name pick list select 'FGDC Content Standards for Digital Geospatial Metadata.'
From the 7.6 Metadata Standard Version pick list, select June 8, 1994.
Click on 'Save Metadata' at the top of the form, then 'Go Back to *Main* Menu.'
We have now completed the minimum mandatory sections of metadata so
it is time to produce a report.
Select 'Metadata Dataset Output Menu' from the Report Menu.
The numbered list below the 'Metadata Reports' bar gives the steps necessary to produce an output report.
First, use the arrow button by the Select Dataset Name box to expose and select the name of your dataset.
Select the 'Build Report' button.
Select the 'To File' button under 'Report Output To,' then supply a file name and destination for the output file. Do not use the default output filename of 'dataset.txt.' For a destination directory, I suggest your tools/metadata directory where you have been doing all your other work.
If you have lots of memory in your computer, you can probably toggle to Program Manager and open a DOS window to complete processing of the file with cns and mp, leaving Metamaker idle in the background, but let's just shut MetaMaker down anyway.
Select 'Go Back to Main Menu' from the Report Menu page.
Select 'Exit NBII Metamaker' from the Main Menu.
Introduction
As mentioned earlier, CorpsMet95
(metadata development software created by the US Army Corps of Engineers) is
not the primary metadata tool for this exercise. However, Corpsmet is
a complete metadata tool that will work with this exercise but will not
match the example responses.
For more information about CorpsMet95
Get the Update!
If you wish to follow this exercise using Metamaker, download and configure Metamaker from the Exercise Configuration.
Format and Validate (parse) Your Record
Introduction
Now that you have created a metadata
record, it is time to validate it against the CSDGM. Correcting the
errors and ensuring a successful parse is the last step for creating
metadata.
Start
Your DOS...
Get a DOS window open and change directory to your tools\metadata
directory. Make sure your path still includes the one to where cns
and mp are.
Check your metadata data record
Open the metadata file you produced with a text editor. If your
file looks like mine did after this exercise, then you will see that the
section headings for the major sections 2-6 (Data_Quality_Information -
Distribution_Information) are showing up in the output report, but they
have no children. Expect mp to complain about that! You could fix
that up now by deleting them, but lets don't - we know they are there.
Pre-Parse the record
Okay, now let's jam this output through cns. For
example let's say our file is called acme.txt.
Look to see what is in acme.lo with a text editor. I didn't see anything serious, just some header information that Metamaker slapped on the top of the output report. We don't need it.cns -e acme.lo -o acme.out acme.txt
CorpsMet95 generated documents typically do not require much editing.
For an updated version of this exercise, visit the Metadata
WorkBook Web Page.
The contents of acme.err are interesting. As expected, those major sections without children generated some errors. Of those five sections only three showed errors because only they have mandatory child elements. The first error is the interesting one, that is 'Spatial Domain is required in Identification_Information.' In Metamaker, the Spatial_Domain element is replaced by 'Geographic Extent,' and Bounding_Coordinates is replaced by 'Bounding Rectangle Coordinates.' If you look at the HTML output file, you will see that it is a little messed up in that section...mp -e acme.err -h acme.htm acme.out
This is where an alias file can help cns do its job better. Let's try again with an alias file.
cns -a ..\cfg\mm21ales.txt -e acme.lo -o acme.out acme.txt
mp -e acme.err -h acme.htm acme.out
The first error is now gone from mp's error report and the HTML
output looks better also!Congratulations on producing your first complete metadata document and validating it!
Previous | Beginning | Metadata WorkBook