Jump to content


Photo

GEDCOM Source Citation export anomalies

GEDCOM PAGE Certainty QUAY Citations for notes

  • Please log in to reply
24 replies to this topic

#1 TomH

TomH
  • Members
  • 25 posts
  • Gender:Male

Posted 07 August 2014 - 01:59 PM

I have found these anomalies in assisting a TMG user migrate data to RootsMagic:

  1. TMG outputs the unlimited length Citation Detail field to +1 PAGE using CONC|CONT for overflow of the max linewidth setting. GEDCOM states {0:1} lines, i.e., no CONC|CONT. There does not seem to be a field within TMG to store the page value corresponding to WHERE_WITHIN_SOURCE that should go to PAGE value separate from the lengthy transcription text or TEXT_FROM_SOURCE that should  go to +2 TEXT. RootsMagic accepts the PAGE CONC|CONT into its Page field up to some length beyond which it puts it all into its Research Notes field (+1 NOTE). Is there a way to enter data in TMG that does separate WHERE_WITHIN_SOURCE from TEXT_FROM_SOURCE into the correct tags, PAGE and TEXT respectively?
  2. TMG outputs a synthesis of its Surety values to +0 QUAY [0-3]  which RootsMagic does not recognise either because it is on the wrong level or RM has no way to map a single digit to a 3 value quality indicator. GEDCOM specifies that it should be output to +QUAY, not the same level as n SOUR. Simply changing the line to "n NOTE {TMG QUAY 3}" for example, puts the note into the note of the fact for which the source is cited. That's really awkward given that it is the quality of the citation that is being described, and really confusing if there are multiple sources cited.
  3. TMG outputs source citations for Notes, unsupported by GEDCOM, by RootsMagic and by others, I'm sure. It would avoid loss of these citations with their transcriptions and analyses if TMG had an export option to "Move citations for Notes to citations for corresponding Facts" or words to that effect where "fact" means event and attribute in GEDCOM. 

​Because of these GEDCOM errors, I am guessing that other software has similar problems with importing from TMG. Is there any prospect of these being corrected?

 

My apology if my use of TMG terminology and understanding of how it works is flawed.



#2 Jim Byram

Jim Byram
  • Moderators
  • 7,340 posts
  • Gender:Male
  • Location:Framingham, MA

Posted 07 August 2014 - 02:17 PM

As you have noted elsewhere, when exporting for RM, it is important to set the GEDCOM export line length to the maximum 246 characters. That will help with the CD issue.

 

1. Citation Detail

 

Michael covers this below and talks about various strategies in using this field.

 

As he says, the Citation Detail field is the standard PAGE field but in TMG is a memo of unlimited length.

 

The Citation Memo field is the 'transcription' field and is exported as a NOTE field subordinate to the SOUR tag.

 

2. QUAY 

 

from TMG Help...

 

Sureties

When this option is selected, surety values are exported to the extent supported by GEDCOM. Date, place, and memo sureties are exported with a QUAY tag at one level higher (usually 4) than the source citation from which they are referenced.

 

SOUR level

When selected, surety values (QUAY) are exported at the same level as the source citation from which they are referenced in order to accommodate certain importing programs.

 

By default, TMG exports...

n SOUR

+1 QUAY

The [  ] SOUR sub-selection should not be selected (and isn't by default).

 

2 SOUR @S6@
3 PAGE MS, Attala Co., Attalaville, p. 467, Sh 207 Ln 39 (Nat. Arch. Film #M653-577)
3 QUAY 2

 

That is what you want, right?

 

TMG has up to 5 surety.values per citation (each being -/0/1/2/3). GEDCOM has 1 QUAY value (being 0/1/2/3). TMG Surety and GEDCOM QUAY are really apples and oranges and I see little point to exporting Sureties.

 

3. NOTE

 

TMG treats Note tags just like any other event. They have principals and witnesses, date and sort date, place fields, sentences, citations, exhibits, etc. They are exported like any other event tag. This puts them at odds with GEDCOM and with other programs that follow the GEDCOM pattern. That isn't going to change. The dates are not exported in v9.03 but will be exported as 2 _DATE tags in a later build to get the data into the GEDCOM. The NOTE data will require cleanup after import to other programs.

 

A more promising strategy is to set the TMG Note tag type GEDCOM export value to EVEN TYPE. Note tags will then export as custom events and will import into RM with all of their data.

 

(I would also follow this strategy with the TMG Address and Telephone tag types.)


Edited by Jim Byram, 08 August 2014 - 10:48 AM.


#3 Michael Hannah

Michael Hannah
  • Moderators
  • 2,733 posts
  • Gender:Male
  • Location:Los Ranchos, New Mexico, USA
  • Interests:Genealogy, Computers

Posted 08 August 2014 - 08:23 AM

I am sure Jim will have more to say, but as someone with some experience in GEDCOM I thought I might chime in on your first question.
 

TMG outputs the unlimited length Citation Detail field to +1 PAGE... There does not seem to be a field within TMG to store the page value... separate from the lengthy transcription text...


You are correct, Tom, that the default of putting all text into TMG's Citation Detail field might produce GEDCOM continuation lines that are non-standard. I would guess that WhollyGenes decided that it was better not to truncate and lose anything the user put in the CD at the price of non-standard continuation lines. However, output of the CD data into the GEDCOM PAGE element makes sense to me in that the most common examples of use of TMG's CD is to put the page value there.

But there are separate TMG fields for the page value and transcription text. These can be used by a careful TMG user to ensure output of "standard" GEDCOM source structures. The user can put the page value in the Citation Detail, and discipline themselves to limit any CD data to about 7 characters less than their defined GEDCOM max line length. The remaining transcription text, to an unlimited length, can be put in TMG's separate Citation Memo field. The CM will be output as the standard, and unlimited length, GEDCOM NOTE structure as part of the source citation. As this NOTE structure is standard, I would presume that RootsMagic would import it.

There have been several discussions and posts over the years where users have noted that with disciplined and restrictive data-entry standards, avoiding many TMG constructs which have no GEDCOM equivalent construct, and imposing limits on how one uses some of the TMG constructs, a TMG project can be constructed which will output cleanly to GEDCOM. This restriction on how one uses CD and CM is one more such example.


Michael
See my book on how I customize TMG My Way.
My website.

#4 TomH

TomH
  • Members
  • 25 posts
  • Gender:Male

Posted 08 August 2014 - 12:15 PM

Thanks, Jim and Michael, for confirming the relationship of the two TMG Citation fields to the two GEDCOM fields. I wondered if the || separator had some effect on parsing the Citation Detail into the GEDCOM PAGE and TEXT tags, the latter being where the transcription should go but I suspect it just passes through literally to the PAGE field only, because 8 or 9 separators are allowed. Putting the transcript in the Citation Memo field along with the analysis and commentary causes both to end up in the NOTE field and in RootsMagic's Citation Comments from which the transcript needs to be parsed out and put into either TEXT in GEDCOM or Research Notes in RM.

It has now been stated twice today in the RootsMagic forums that The RootsMagician is working on a direct import from TMG but there is no ETA. Whether direct or by GEDCOM, mapping from two fields in TMG to three in RM is not on and I think the same strategies should be used to prepare the TMG database for export. Move all but the key navigation reference from Citation Detail to the beginning of Citation Memo, bumping what was there down and separating them with the || separator or some other distinctive one, e.g., {} which can be suppressed in RootsMagic reports. These separators could be used with custom scripts operating either on the GEDCOM or on the SQLite database to move the stuff ahead of them to the correct place.

#5 TomH

TomH
  • Members
  • 25 posts
  • Gender:Male

Posted 08 August 2014 - 01:51 PM

2. QUAY 
 
from TMG Help...
 
Sureties
When this option is selected, surety values are exported to the extent supported by GEDCOM. Date, place, and memo sureties are exported with a QUAY tag at one level higher (usually 4) than the source citation from which they are referenced.
 
SOUR level
When selected, surety values (QUAY) are exported at the same level as the source citation from which they are referenced in order to accommodate certain importing programs.
 
By default, TMG exports...
n SOUR
+1 QUAY
The [  ] SOUR sub-selection should not be selected (and isn't by default).
 
2 SOUR @S6@
3 PAGE MS, Attala Co., Attalaville, p. 467, Sh 207 Ln 39 (Nat. Arch. Film #M653-577)
3 QUAY 2
 
That is what you want, right?

 Uhh, TMG sureties are more complicated than RM, for sure, which has a three-dimension Quality rating based on Elizabeth Shown Mills' advocacy but applying only to the citation itself, not to Prin1, Prin2, Date, Place, Memo... I cannot speak for other software. The only place that the GEDCOM standard shows QUAY being used is for a SOURCE_CITATION, at the same level as PAGE, so, yes, your example is what I would want to see in the GEDCOM. The QUAY tag can be readily replaced with NOTE and the value surrounded by something useful such as a label and privacy braces:
 
2 SOUR @S6@
3 PAGE MS, Attala Co., Attalaville, p. 467, Sh 207 Ln 39 (Nat. Arch. Film #M653-577)
3 NOTE {TMG QUAY 2}
 
The end-user can search for the "{TMG QUAY" string to find citations for which they may wish to edit the destination software's rating system, perhaps looking back at the TMG data to interpret.
 
Alternatively, a direct read or a TMG utility could populate a note with something like "{TMG Surety vwxyz}" to facilitate the manual translation.
 

TMG has up to 5 surety.values per citation (each being -/0/1/2/3). GEDCOM has 1 QUAY value (being 0/1/2/3). TMG Surety and GEDCOM QUAY are really apples and oranges and I see little point to exporting Sureties.

I think I have given a rationale above for doing something with them as a convenience for the emigre but I agree that it is debatable. 

3. NOTE
 
TMG treats Note tags just like any other event. They have principals and witnesses, date and sort date, place fields, sentences, citations, exhibits, etc. They are exported like any other event tag. This puts them at odds with GEDCOM and with other programs that follow the GEDCOM pattern. That isn't going to change. The dates are not exported in v9.03 but will be exported as 2 _DATE tags in a later build to get the data into the GEDCOM. The NOTE data will require cleanup after import to other programs.
 
A more promising strategy is to set the TMG Note tag type GEDCOM export value to EVEN TYPE. Note tags will then export as custom events and will import into RM with all of their data.
 
(I would also follow this strategy with the TMG Address and Telephone tag types.)

I think that makes sense. Just be sure to use one or few consistent values for TYPE as each different one becomes a new Fact Type in RootsMagic.

#6 Jim Byram

Jim Byram
  • Moderators
  • 7,340 posts
  • Gender:Male
  • Location:Framingham, MA

Posted 08 August 2014 - 02:03 PM

You don't set the type. TMG will make the type the tag type name. So you would be importing three custom facts - Address, Telephone, Note. I've tested this and it is a good solution since it gets all of the data for those TMG tags into RM as custom facts.



#7 TomH

TomH
  • Members
  • 25 posts
  • Gender:Male

Posted 08 August 2014 - 03:49 PM

Perfect.

#8 TomH

TomH
  • Members
  • 25 posts
  • Gender:Male

Posted 09 August 2014 - 07:25 AM

Reviewing the GEDCOM again:

 

1 RESI
2 DATE 1892
2 PLAC , Chicago, Cook, Illinois,
2 SOUR @S18@
3 PAGE p1637, 1892,
2 QUAY 2

 

Is TMG attempting to ascribe QUAY to the RESI or is it attempting to ascribe it to the citation, erroneously? Because, from the same database, I see:

 

2 NOTE or Slown, or Slowie
3 SOUR @S3@
4 PAGE p273
4 QUAY 2

 

which correctly ascribes QUAY to the citation.

 

I think there is one class of events or attributes or things for which TMG outputs the correct level of QUAY for their SOURs and another class for which it is off by 1.



#9 Jim Byram

Jim Byram
  • Moderators
  • 7,340 posts
  • Gender:Male
  • Location:Framingham, MA

Posted 09 August 2014 - 09:30 AM

Again, for this GEDCOM, SOUR should not have been selected.

 

If it was not selected, the 2 QUAY 2 is in error and should be 3 QUAY 2.

 

I'll test a Residence tag. And got this with SOUR unselected,

 

1 RESI
2 DATE 01 JAN 1852
2 _SDATE 01 JAN 1852
2 PLAC Attalaville, Attala County, Mississippi
2 SOUR @S6@

3 QUAY 3

 

My suspicion is still that the user selected QUAY.

 

btw... The QUAY value comes from the first of the 5 possible surety values for the citation and is normally the TMG surety value for the citation for P1. A tag almost always has a P1.



#10 Michael Hannah

Michael Hannah
  • Moderators
  • 2,733 posts
  • Gender:Male
  • Location:Los Ranchos, New Mexico, USA
  • Interests:Genealogy, Computers

Posted 11 August 2014 - 08:23 AM

Jim commented: "You don't set the type. TMG will make the type the tag type name."

 

Well, yes and no.  (You probably know the following, but for the sake of others reading this thread...)  The GEDCOM tag type which will be output from a TMG tag type is under control of (and possible misuse :ohno: by) the user.  In the TMG Tag Type definition the user can set an option to either enter (what they believe is) a specific 4-character alphabetic GEDCOM tag type, or set it to use the GEDCOM EVEN/TYPE structure with the TMG tag name automatically output as the TYPE.  TMG sets the default for this option to a standard GEDCOM tag type if that seems appropriate and equivalent for the given TMG tag type, or to the EVEN/TYPE structure for TMG tag types which do not have an equivalent GEDCOM tag type.  But the user "can" override these defaults to be something more (or less) meaningful to the GEDCOM they wish to produce.

 

Hope this clarifies rather than muddies the water :D


Michael
See my book on how I customize TMG My Way.
My website.

#11 Jim Byram

Jim Byram
  • Moderators
  • 7,340 posts
  • Gender:Male
  • Location:Framingham, MA

Posted 11 August 2014 - 09:30 AM

You can't override the EVEN TYPE tag name (which is the TMG tag type name) and there would be no point in doing so. You can always do it by renaming the tag type but why would you do that?



#12 Michael Hannah

Michael Hannah
  • Moderators
  • 2,733 posts
  • Gender:Male
  • Location:Los Ranchos, New Mexico, USA
  • Interests:Genealogy, Computers

Posted 11 August 2014 - 11:32 AM

Jim commented: "You can't override the EVEN TYPE tag name (which is the TMG tag type name)".  Yes, that is why I even emphasized that the TMG tag type name is automatically output as the TYPE.   The point I was trying to make is that if a user wants to create a special TYPE for some reason, such as it might be recognized by some other software, this is easily accomplished simply by creating a TMG tag type of that name.


Michael
See my book on how I customize TMG My Way.
My website.

#13 TomH

TomH
  • Members
  • 25 posts
  • Gender:Male

Posted 11 August 2014 - 07:18 PM

Jim, I finally got the reason why I am seeing QUAY at the same level as SOUR. You have probably said why a few times about not checking SOUR. I now see that it is the option SOUR level in the Export Wizard which should be left unchecked. ("When selected, surety values (QUAY) are exported at the same level as the source citation from which they are referenced in order to accommodate certain importing programs.")

 

I also see in Help "NOTE: If the citation detail is split, only CD1 is exported". Then, I think that a TMG user emigrating via GEDCOM to RootsMagic should edit Citations so that Citation Detail contains only the "where in the source" information and no || delimiters, moving all else to Citation Memo where the || delimiter might be used to separate Transcription from Comment. One might also enclose the transcription in sensitivity markers {same as RootsMagic's privacy markers} as that may facilitate moving the transcription to the correct field in RootsMagic. I believe this strategy would also be advisable to prepare for RootsMagic's future direct import because the migration is from two fields (Citation Detail and Memo) to three (Page, Research Notes and Comments).



#14 TomH

TomH
  • Members
  • 25 posts
  • Gender:Male

Posted 11 August 2014 - 07:28 PM

How do you search for Citations for which Citation Details contains a delimiter, i.e., CD1, CD2...?



#15 Jim Byram

Jim Byram
  • Moderators
  • 7,340 posts
  • Gender:Male
  • Location:Framingham, MA

Posted 11 August 2014 - 10:59 PM

How do you search for Citations for which Citation Details contains a delimiter, i.e., CD1, CD2...?

 

List of Citations

filter

Citation Detail Contains || END

 

I also see in Help "NOTE: If the citation detail is split, only CD1 is exported".

Things are more complicated than you might image. Someone might use this technique...

 

Each Citation Detail part is a fixed piece of the citation.

<CD1>||<CD2>||<CD3>||<CD4>||<CD5>||<CD6>||<CD7>||<CD8>||<CD9>

 

And the footnote template looks like this...

Footnote

[CD2] household, [TITLE], [COUNTY], [STATE], population schedule<, [CD3]>, enumeration district (ED) [CD4], page/sheet [CD5]<, dwelling [CD6]><, family [CD7]><, line [CD8]>; National Archives micropublication [FILM NUMBER]<; ([CD9])>.

 

CD1 is reserved for the full footnote for GEDCOM export

 

The generated output...

Alonzo Byram household, 1880 U.S. Census, Bossier Parish, Louisiana, population schedule, Ward 5, enumeration district (ED) 10, page/sheet 41a/27, dwelling 3, family 3, line 10; National Archives micropublication T9-448.

 

So the Citation Detail field would look like this... 

Alonzo Byram household, 1880 U.S. Census, Bossier Parish, Louisiana, population schedule, Ward 5, enumeration district (ED) 10, page/sheet 41a/27, dwelling 3, family 3, line 10; National Archives micropublication T9-448||Alonzo Byram||Ward 5||10||41a/25||3||3||10||T9-448

 

:D 

 

If this were done, then GEDCOM export is covered.



#16 Michael Hannah

Michael Hannah
  • Moderators
  • 2,733 posts
  • Gender:Male
  • Location:Los Ranchos, New Mexico, USA
  • Interests:Genealogy, Computers

Posted 12 August 2014 - 12:53 PM

I also see in Help "NOTE: If the citation detail is split, only CD1 is exported".

Although not mentioned in HELP, I believe this is also true for the Citation Memo: only CM1 is exported.

Thus Jim's clever trick of always reserving CD1 (and CM1) for the entire text of what you want output to GEDCOM, and only using the higher numbered split parts for TMG template use, will give you the best of both worlds.

 

Further, with the new feature in V9.03 on the Citation edit screen of previewing the full footnote output produced by the TMG template, the user could first just enter the split parts from 2 thru 9, then preview the Citation, and copy/paste the full footnote preview output into CD1.


Michael
See my book on how I customize TMG My Way.
My website.

#17 TomH

TomH
  • Members
  • 25 posts
  • Gender:Male

Posted 12 August 2014 - 01:08 PM

If CM1 is all that is exported, then one should use something other than || to delimit Transcription from Comments. I would suggest {} or %%. (the whole Citation Memo is exported c/w the delimiters || )

Interesting to me is that Jim's exploit of this TMG feature is something I have been beseeching for RootsMagic. That is, the ability to utilize source sentence templates internally for superior footnotes and for the drafting of a coexisting Free Form footnote for export to standard GEDCOM. Currently, you can have one but not the other. Its source templates do not export well to standard GEDCOM.


Edited by TomH, 12 August 2014 - 02:22 PM.


#18 TomH

TomH
  • Members
  • 25 posts
  • Gender:Male

Posted 12 August 2014 - 02:20 PM

Jim, thanks for the clue to searching. I had looked at List of Citations but misinterpreted the Subject(s) of the report to be equivalent to RM's Select People.

 

Michael, I see || in the export of Citation Memo to GEDCOM NOTE so it looks like it is only Citation Detail that filters out all but the first segment.



#19 beetle3247

beetle3247

    Beetle Bailey

  • Members
  • 58 posts
  • Gender:Male
  • Location:Conover, WI, USA

Posted 13 August 2014 - 07:51 AM

Jim stated much earlier in discussing NOTE tags to "set the TMG Not tag type GEDCOM export value to EVEN TYPE ... will import into RM with all their data." How do you do that?



#20 Jim Byram

Jim Byram
  • Moderators
  • 7,340 posts
  • Gender:Male
  • Location:Framingham, MA

Posted 13 August 2014 - 07:54 AM

Open the master tag type list. Select the Note tag type and click [Edit]. Select the Other tab. Click the radio button next to 1 EVEN 2 TYPE. Click [OK] to save the change.







Also tagged with one or more of these keywords: GEDCOM, PAGE, Certainty, QUAY, Citations for notes

0 user(s) are reading this topic

0 members, 0 guests, 0 anonymous users