Annotation Group

• Jim Croft, Paul Morris, Austin Mast, Steve Winner, Debbie Paul
  • What follows is each of our notes from the 2-day Annotation Working Group meeting.
• Scope:
  • to discuss the current issues surrounding sharing annotations about images (and other objects),
  • suggest areas needing further investigation as well as...
  • point out the need to review other standards that already exist about making comments about digital images (so as to not re-invent...).

Jim Croft's notes for Annotation working group

• e.g. getting FSU specimen images into MB, issues associated with annotations and taxonomy
• names vs identifications
• higher taxonomy a problem
• annotation history
• name string problems
  • orthographic variants
  • typographic errorr
• c. Fl. of the SE US
  • displays results coloured WRT mapping to concepts
• Annotations enables user contribution to data quality
  • verification of annotator
  • authorization of annotator
• Entomologists interested in descriptive annotations
  • e.g arrows, boxes, circles, tags, etc.
  • annotation of the shapes in the image
• Tagging with flickr
  • simple annotation
• Association of collection notes/methods with insect specimens
• Potential for student projects
  • search for characters state, taxa, etc.
  • free for all, vs controlled ontology
  • phenotypic barcode of life
• Image upload to Morphbank
  • bare minimum specs
  • user information
    • MB user
    • MB group
    • password
  • specimen data
    • DC institution code
    • DC collection code
    • DC specimen number
    • DC secimen name
  • MB will return MB number
    • Basis to LSID
• Search API
  • DC fields
  • Harvesters?
  • How do ontologies work into this?

• Use cases
  • Annotating images
    • determination annotation
    • general annotation
    • Personal notes
    • can apply to:
      • Specimen image
        • in collection
          • in field
      • Taxon image
      • Locality image
      • Person image
    • SDD
• Problem with definition of specimen
  • gathering, units, items (incl. derived objects)

Governance

• Morphbank Needs to become part of the community
  • c.f GenBank?
• wider management structure
• steering committee
• Annotations need to be structured
  • link to a schema
  • controlled concepts, vocabulary
  • Darwin Core
  • Can use a subset of the ontology
  • Annotation to track agreement or disagreement

Names a big issue

• name/identification as published
• annotation expressing confirmation/difference
• revised name/identification(s)
• need application to manage annotations and name changes
• need to include dhistory of annotations and effected changes
• Need a circuit breaker to prevent flip-flop

Queries

• to show records with annotations
• to show history of annotations
• agreement and disagreement
• need to be able to limit by taxon, region other area of interest

Positioning annotations

• absolutely (pixels)
  • need a transformation when resized
• relatively (%)
  • need to convert to pixels with each display
• transparency

Graphical standards

• SVG
  • 2D and 3D
• 2D or 3D mapping displays
  • treat as a map
  • OGSC

Sharing annotations

• between institutions/annotators
• between collections

Workfow issues

• ? TDWG
  • nothing specific yet
• Annotations to be seen in the context of an activity ***
• c.f. Kepler
  • based on experimental research
  • probably not appropriate in this context
• c.f. business processes
  • how to get something done
  • rule based systems
  • business rules, objects, precesses
  • Off the shelf applications

Check out 'Herbaria at Home'

• UK herbarium specimen annotation project - small herbaria working together to image specimens. Using volunteers to enter the data from the label.

An annotation standard

• a mixture of standards
  • a standard for annotating digital objects?
  • the google earth model
  • (see dojo - javascript animation toolkit)
• Annotation
  • Type of annotation
    • Determination
    • General
    • Tag
    • Handwriting identification
  • Workflow context of annotation
    • determination
    • character sets
    • collecton
    • deaccession, etc.
  • Schema/ontology
    • TCS
    • Darwin Core
    • etc.
  • Annotator
    • ID
    • Name
    • Authority
    • Authenticity, Validation
    • Confidentiality
    • user's level of expertise
  • Date
  • Digital Object
    • LSID
      • ? Darwin Core ID triplet
    • Subject area(s)
  • Where does annotation apply on image
    • Area of the image of interest
    • Point(s) region(s) of interest
    • graphical elements
      • arrows
      • circles
      • boxes
      • areas
    • Display settings
      • state - colour, contrast, brightness, etc.
  • Annotation content
    • text
    • relationships
      • grouping of digital objects
        • is-part-of, includes, overlaps, etc.
        • derived objects
    • identification
      • confirmation
      • correction
      • conflict
    • resources used
    • references
  • Meta annotation
    • appropriate use
    • applicability

• 1) Taxon Name Issues
  • Important to track determination history
  • Important to tag names as published for the specimen
  • Pace of (online) determination may be higher than that for collection visits, more chance for disagreement among experts on determination, need for a structure for resolving disputes.
  • Audit trail of actions resulting from annotations (generalized to all sorts of annotations (structure is not an antenna...).
  • Record annotations
  • Record response of system to the annotation
    • Name applied to image
    • Person disagrees and offers annotation with different name
    • System (automated or human) approves application of new name as current name
    • All three above need to be tracked.
  • DB to watch for annotations that flip flop repeatedly.
  • Recommendation – governance structure to include experts able to resolve disputes about the content.
    • Graphically highlight issues in dispute
    • Allow structured comments on issues
    • Allow easy thumbs up/thumbs down
• 2) Control of annotations at entry point:
  • Tags – quick, easy, uncontrolled single input type=text box.
    • General Annotations.
      • Useful for "new" species.
      • Useful for ad hoc groupings of images.
  • Schema level control – text entry fields linked to concepts = text control over atomic field in a normalized database.
  • Ontology level control – picklist/autocomplete/validated data entry control with data entry limited by a controlled vocabulary.
  • What to capture?
    • Text (like field notes), marked-up images (graphic/vector), ontologies?
  • Generalize – Annotate using controlled concepts using fields based on existing schemas, ontologies, etc.
• 3) Likely scope of concepts in annotations:
  • Largely or entirely covered by existing schemas, much less so by ontologies:
    • Dublin Core – authorship/creation date of annotation
    • DarwinCore?/ABCD - specimen data
    • GML – locality polygons
    • StructuredDescriptiveData? – character data
    • TaxonomicConceptSchema? – determinations
    • Graphical presentation elements ? SVG
    • ?3d/video/audio, annotation of non 2d image digital media files
    • Agree or disagree with another annotation.
• 4)Presentation and scaling
  • Capture data early.
  • Client side – css positioning, javascript positioning
  • Server side – scripted image editing (imagemagick), serving images with mapping tools (mapserver, geoserver etc).
• 5) Related items discussed: Interface of the Flora of the SE US (Austin), Florabase (Jim), image/shape evaluation: MapServe?, OpenGIS?, Flickr, Photosynth, Open Street Map
• 6) Specimen issues: (from Deb) Jim pointed out the importance of being "clear" and "accurate" about what one is annotating with his example of "gathering", "item," and "?" His group avoids using the term "specimen."
  • (from Paul): See history of collections database information models starting with Stan Blum's ACS model in the US and the European history behind ABCD – both use a generalized “collection object” concept to handle both specimen based and lot based collections and differing practices in the management of material in collections – concrete cases: herbarium sheet with specimens of more than one taxon affixed, multiple parts of the same plant spread over several herbarium sheets, multiple parts of the same mammal (skin, skeleton, tissue samples) having different preparation histories and storage locations, derived objects such as casts related to the collection objects from which they were made, bulk samples that may get picked and sorted into lots by higher taxonomic groups, which may be later sorted into lots determined to the species level. Note not to confuse gathering with ABCD gathering event.
  • What is being annotated? Specimen? Image? Group of images?
    • Annotation is of a Set of one or more digital objects.
    • Annotating set of images as having same view – annotation user interface as means of creating an annotation and as a means of adding data to other parts of database.
  • Are there information standards for comments on digital objects?
• 7) Things involved in an Annotation ( a summary from the discussion process )
  • Annotator, Date, Time
    • Authentication of annotator is required.
    • Validation of annotators rights to make annotation is required.
  • Digital Object being annotated
    • LSID of digital object
  • Area of interest
    • Graphical region of 2d/3d image
    • Field/portion of field in textural record.
    • Image modifications desired by user in area of interest – contrast etc.
      • Conditions of observation under which the observation was made.
  • Points/Regions pointed to / discussed in the annotations
    • Point on specimen, set of points on specimen. etc
  • Type of annotation
    • Schemas/ontologies related to annotation.
    • Much content has appropriate standards .
  • Content of annotation
    • Determination
      • TaxonConceptSchema?
    • Morphology
      • StandardForDescriptive? Data
    • Relationships of digital objects
      • Image of plant
      • Drawing of leaf
      • Relationship: this image is an example of the leaf shape in this drawing.
      • Relationships between digital objects not directly captured by database structure e.g. (institution,collectioncode,catalognumber for specimen)?
    • References supporting the decisions/inferences made by the annotator.
    • Expert statements about the subject of the image that can't be otherwise recorded (observations, data) e.g. taste/smell of leaf. I fed this specimen to this mouse and the mouse died.
    • Location
      • GML, other OGC standards
    • Ecology
      • EML
    • Handwriting
      • Whose handwriting is on the label.
  • Graphical presentation elements layered on the area of interest by the annotator
    • Arrows, polygons, polylines, text
      • SVG, OGC map standards.
  • Scope of display of annotation and annotator
    • Keep annotation private/group/public
    • ?desire for anonymous annotations/distrust of anonymous sources lack of comfort with anonymity – use case – annotations by reviewers in review process.
    • Issue for governance structures involving user community.
  • Subsequent history/context of annotation
    • Meta-annotation – this annotation is appropriate/inappropriate for the following purposes. Treatment of adhomonym arguments between scientists.
    • Issue for governance structures involving user community.
    • Workflow – place of image in larger workflow – consequences and notifications generated by annotation, requirements produced by workflow for creation of annotation.
    • Thumbs up/thumbs down on annotation.
    • Congruence to other annotations
  • Annotations feeding data into other parts of the database.
    • Actions taken as the result of annotations.

Deb's notes:

• Taxon Name Issues
  • Annotations with taxon names not matching local authority files. How to handle?
  • Capture old "legacy" (Determination) annotations as well as "online" annotations in such a way that the data can be shared & associated with the identification.
  • Capturing the "Taxon Concept" if possible.
    • What fields would be needed for this?
    • What format?
    • Capture original name/s on specimen
      • Relate names to Taxon Concept, as in name=, name is unresolvable, name is ambiguous, name includes, name is included...
  • U-Bio separates the Name Classification from the Taxon ID String. Everyone can use the classification that best fits their world view.
  • Taxon Concept needs to be tied to fields in the Determination Annotation (ABCD, SDD, ...)
  • Ask the person creating the Identification to identify the resources used to make the Identification (Image, DNA, Key, ...)
  • Some Authentication process: Name of the Annotator and Time Stamp
• Need a way to change the name associated with a specimen is the expert says the specimen is something other than the most recent determination.
  • Must keep the history of name changes
• Possible solution to Schema Issues for Annotations is to allow each user to choose the Schema (and therefore the most relevant fields for their purpose).
• How to store the Annotations made to/on images?
  • Mapping and Map Layers / 2-3D SVG Standards
    • Graphics vs. Vectoring
  • Store annotation as SVG version or map (graphic) version
    • Allow user to choose
• Are there Standards for graphical display of information? Are there Standards about making comments about digital images?
  • Is there a subset of standards specific to graphics?

• Check out DOJO (Java Script Tool Kit)

Attributes of an Annotation

• Type of Annotation
  • General
  • Determination
  • Handwriting
• Schema or Ontology Used (e.g. TCSB)
• Annotator
  • Validation of right to make the annotation (Authentication Process)
  • Might the Annotator want to remain anonymous? What if they are a reviewer?
  • If someone annotates, should anonymity even be an option? How could the community rate/weight the value of the annotation if the annotator's name is not present.
  • Could use a "flag" to "Display Annotator" or not. The system would capture the name of the annotator, but not display it?
• Date & Time of Annotation
• Digital Object being Annotated
  • Using LSID? (to reference digital object in Morphbank)
    • Yes, an annotation could be added to this object
• Area of interest
  • point/s of interest highlighted
• Annotation Conditions: e.g. The annotation event took place while the researcher was coding a character or perhaps creating a collection...
  • helps researcher and future researchers to understand the context of the event and why a particular comment is made or conclusion is reached.
• Content of Annotation: allowing users to have flexibility; reaching out to use as many different standards are possible (GML, SDD, ...)
• Relationship/s between objects: is a part of, is included in...
  • examples include: protein derived from, DNA derived from, illustration to image to a pollen image
• Resource/s the Annotator uses to make an annotation (a microscope, a key, DNA, ...)
• Graphical Elements added to the Object (circles, squares, arrows) by annotation
• Meta-annotation: a review process in place (by Morphbank or...) to prevent spurious annotations (ex. "Viagra").

Work Flow Issues of Annotations

• 3rd Party viewer, does original person accept the annotation, who is the "original person" if there are multiple annotations, what happens after an annotation is made?
  • Project Kepler mentioned but Kepler is experiment-based
  • USE CASE may be sharing annotations between Harvard Herbaria and Morphbank, SDD being captured, what concepts are available for exchange are known

Look at

• Specify and fields present there for Determination Annotations
• MX System

Idea that a Specimen has a "curator"

GBIF - may help to set up standards to oversee governance of Morphbank resolving issues like how to handle annotations

-- DebPaul - 25 Oct 2007