Jump to content
TomH

GEDCOM Source Citation export anomalies

Recommended Posts

I have found these anomalies in assisting a TMG user migrate data to RootsMagic:

  1. TMG outputs the unlimited length Citation Detail field to +1 PAGE using CONC|CONT for overflow of the max linewidth setting. GEDCOM states {0:1} lines, i.e., no CONC|CONT. There does not seem to be a field within TMG to store the page value corresponding to WHERE_WITHIN_SOURCE that should go to PAGE value separate from the lengthy transcription text or TEXT_FROM_SOURCE that should go to +2 TEXT. RootsMagic accepts the PAGE CONC|CONT into its Page field up to some length beyond which it puts it all into its Research Notes field (+1 NOTE). Is there a way to enter data in TMG that does separate WHERE_WITHIN_SOURCE from TEXT_FROM_SOURCE into the correct tags, PAGE and TEXT respectively?
  2. TMG outputs a synthesis of its Surety values to +0 QUAY [0-3] which RootsMagic does not recognise either because it is on the wrong level or RM has no way to map a single digit to a 3 value quality indicator. GEDCOM specifies that it should be output to +1 QUAY, not the same level as n SOUR. Simply changing the line to "n NOTE {TMG QUAY 3}" for example, puts the note into the note of the fact for which the source is cited. That's really awkward given that it is the quality of the citation that is being described, and really confusing if there are multiple sources cited.
  3. TMG outputs source citations for Notes, unsupported by GEDCOM, by RootsMagic and by others, I'm sure. It would avoid loss of these citations with their transcriptions and analyses if TMG had an export option to "Move citations for Notes to citations for corresponding Facts" or words to that effect where "fact" means event and attribute in GEDCOM.

​Because of these GEDCOM errors, I am guessing that other software has similar problems with importing from TMG. Is there any prospect of these being corrected?

 

My apology if my use of TMG terminology and understanding of how it works is flawed.

Share this post


Link to post
Share on other sites

As you have noted elsewhere, when exporting for RM, it is important to set the GEDCOM export line length to the maximum 246 characters. That will help with the CD issue.

 

1. Citation Detail

 

Michael covers this below and talks about various strategies in using this field.

 

As he says, the Citation Detail field is the standard PAGE field but in TMG is a memo of unlimited length.

 

The Citation Memo field is the 'transcription' field and is exported as a NOTE field subordinate to the SOUR tag.

 

2. QUAY

 

from TMG Help...

 

Sureties

When this option is selected, surety values are exported to the extent supported by GEDCOM. Date, place, and memo sureties are exported with a QUAY tag at one level higher (usually 4) than the source citation from which they are referenced.

SOUR level

When selected, surety values (QUAY) are exported at the same level as the source citation from which they are referenced in order to accommodate certain importing programs.

 

By default, TMG exports...

n SOUR

+1 QUAY

The [ ] SOUR sub-selection should not be selected (and isn't by default).

 

2 SOUR @S6@
3 PAGE MS, Attala Co., Attalaville, p. 467, Sh 207 Ln 39 (Nat. Arch. Film #M653-577)
3 QUAY 2

 

That is what you want, right?

 

TMG has up to 5 surety.values per citation (each being -/0/1/2/3). GEDCOM has 1 QUAY value (being 0/1/2/3). TMG Surety and GEDCOM QUAY are really apples and oranges and I see little point to exporting Sureties.

 

3. NOTE

 

TMG treats Note tags just like any other event. They have principals and witnesses, date and sort date, place fields, sentences, citations, exhibits, etc. They are exported like any other event tag. This puts them at odds with GEDCOM and with other programs that follow the GEDCOM pattern. That isn't going to change. The dates are not exported in v9.03 but will be exported as 2 _DATE tags in a later build to get the data into the GEDCOM. The NOTE data will require cleanup after import to other programs.

 

A more promising strategy is to set the TMG Note tag type GEDCOM export value to EVEN TYPE. Note tags will then export as custom events and will import into RM with all of their data.

 

(I would also follow this strategy with the TMG Address and Telephone tag types.)

Edited by Jim Byram

Share this post


Link to post
Share on other sites

I am sure Jim will have more to say, but as someone with some experience in GEDCOM I thought I might chime in on your first question.

TMG outputs the unlimited length Citation Detail field to +1 PAGE... There does not seem to be a field within TMG to store the page value... separate from the lengthy transcription text...


You are correct, Tom, that the default of putting all text into TMG's Citation Detail field might produce GEDCOM continuation lines that are non-standard. I would guess that WhollyGenes decided that it was better not to truncate and lose anything the user put in the CD at the price of non-standard continuation lines. However, output of the CD data into the GEDCOM PAGE element makes sense to me in that the most common examples of use of TMG's CD is to put the page value there.

But there are separate TMG fields for the page value and transcription text. These can be used by a careful TMG user to ensure output of "standard" GEDCOM source structures. The user can put the page value in the Citation Detail, and discipline themselves to limit any CD data to about 7 characters less than their defined GEDCOM max line length. The remaining transcription text, to an unlimited length, can be put in TMG's separate Citation Memo field. The CM will be output as the standard, and unlimited length, GEDCOM NOTE structure as part of the source citation. As this NOTE structure is standard, I would presume that RootsMagic would import it.

There have been several discussions and posts over the years where users have noted that with disciplined and restrictive data-entry standards, avoiding many TMG constructs which have no GEDCOM equivalent construct, and imposing limits on how one uses some of the TMG constructs, a TMG project can be constructed which will output cleanly to GEDCOM. This restriction on how one uses CD and CM is one more such example.

Share this post


Link to post
Share on other sites

Thanks, Jim and Michael, for confirming the relationship of the two TMG Citation fields to the two GEDCOM fields. I wondered if the || separator had some effect on parsing the Citation Detail into the GEDCOM PAGE and TEXT tags, the latter being where the transcription should go but I suspect it just passes through literally to the PAGE field only, because 8 or 9 separators are allowed. Putting the transcript in the Citation Memo field along with the analysis and commentary causes both to end up in the NOTE field and in RootsMagic's Citation Comments from which the transcript needs to be parsed out and put into either TEXT in GEDCOM or Research Notes in RM.

 

It has now been stated twice today in the RootsMagic forums that The RootsMagician is working on a direct import from TMG but there is no ETA. Whether direct or by GEDCOM, mapping from two fields in TMG to three in RM is not on and I think the same strategies should be used to prepare the TMG database for export. Move all but the key navigation reference from Citation Detail to the beginning of Citation Memo, bumping what was there down and separating them with the || separator or some other distinctive one, e.g., {} which can be suppressed in RootsMagic reports. These separators could be used with custom scripts operating either on the GEDCOM or on the SQLite database to move the stuff ahead of them to the correct place.

Share this post


Link to post
Share on other sites

2. QUAY

 

from TMG Help...

 

Sureties

When this option is selected, surety values are exported to the extent supported by GEDCOM. Date, place, and memo sureties are exported with a QUAY tag at one level higher (usually 4) than the source citation from which they are referenced.

 

SOUR level

When selected, surety values (QUAY) are exported at the same level as the source citation from which they are referenced in order to accommodate certain importing programs.

 

By default, TMG exports...

n SOUR

+1 QUAY

The [ ] SOUR sub-selection should not be selected (and isn't by default).

 

2 SOUR @S6@

3 PAGE MS, Attala Co., Attalaville, p. 467, Sh 207 Ln 39 (Nat. Arch. Film #M653-577)

3 QUAY 2

 

That is what you want, right?

Uhh, TMG sureties are more complicated than RM, for sure, which has a three-dimension Quality rating based on Elizabeth Shown Mills' advocacy but applying only to the citation itself, not to Prin1, Prin2, Date, Place, Memo... I cannot speak for other software. The only place that the GEDCOM standard shows QUAY being used is for a SOURCE_CITATION, at the same level as PAGE, so, yes, your example is what I would want to see in the GEDCOM. The QUAY tag can be readily replaced with NOTE and the value surrounded by something useful such as a label and privacy braces:

 

2 SOUR @S6@

3 PAGE MS, Attala Co., Attalaville, p. 467, Sh 207 Ln 39 (Nat. Arch. Film #M653-577)

3 NOTE {TMG QUAY 2}

 

The end-user can search for the "{TMG QUAY" string to find citations for which they may wish to edit the destination software's rating system, perhaps looking back at the TMG data to interpret.

 

Alternatively, a direct read or a TMG utility could populate a note with something like "{TMG Surety vwxyz}" to facilitate the manual translation.

TMG has up to 5 surety.values per citation (each being -/0/1/2/3). GEDCOM has 1 QUAY value (being 0/1/2/3). TMG Surety and GEDCOM QUAY are really apples and oranges and I see little point to exporting Sureties.

I think I have given a rationale above for doing something with them as a convenience for the emigre but I agree that it is debatable.

3. NOTE

 

TMG treats Note tags just like any other event. They have principals and witnesses, date and sort date, place fields, sentences, citations, exhibits, etc. They are exported like any other event tag. This puts them at odds with GEDCOM and with other programs that follow the GEDCOM pattern. That isn't going to change. The dates are not exported in v9.03 but will be exported as 2 _DATE tags in a later build to get the data into the GEDCOM. The NOTE data will require cleanup after import to other programs.

 

A more promising strategy is to set the TMG Note tag type GEDCOM export value to EVEN TYPE. Note tags will then export as custom events and will import into RM with all of their data.

 

(I would also follow this strategy with the TMG Address and Telephone tag types.)

I think that makes sense. Just be sure to use one or few consistent values for TYPE as each different one becomes a new Fact Type in RootsMagic.

Share this post


Link to post
Share on other sites

You don't set the type. TMG will make the type the tag type name. So you would be importing three custom facts - Address, Telephone, Note. I've tested this and it is a good solution since it gets all of the data for those TMG tags into RM as custom facts.

Share this post


Link to post
Share on other sites

Reviewing the GEDCOM again:

 

1 RESI
2 DATE 1892
2 PLAC , Chicago, Cook, Illinois,
2 SOUR @S18@
3 PAGE p1637, 1892,
2 QUAY 2

 

Is TMG attempting to ascribe QUAY to the RESI or is it attempting to ascribe it to the citation, erroneously? Because, from the same database, I see:

 

2 NOTE or Slown, or Slowie
3 SOUR @S3@
4 PAGE p273
4 QUAY 2

 

which correctly ascribes QUAY to the citation.

 

I think there is one class of events or attributes or things for which TMG outputs the correct level of QUAY for their SOURs and another class for which it is off by 1.

Share this post


Link to post
Share on other sites

Again, for this GEDCOM, SOUR should not have been selected.

 

If it was not selected, the 2 QUAY 2 is in error and should be 3 QUAY 2.

 

I'll test a Residence tag. And got this with SOUR unselected,

 

1 RESI
2 DATE 01 JAN 1852
2 _SDATE 01 JAN 1852
2 PLAC Attalaville, Attala County, Mississippi
2 SOUR @S6@

3 QUAY 3

 

My suspicion is still that the user selected QUAY.

 

btw... The QUAY value comes from the first of the 5 possible surety values for the citation and is normally the TMG surety value for the citation for P1. A tag almost always has a P1.

Share this post


Link to post
Share on other sites

Jim commented: "You don't set the type. TMG will make the type the tag type name."

 

Well, yes and no. (You probably know the following, but for the sake of others reading this thread...) The GEDCOM tag type which will be output from a TMG tag type is under control of (and possible misuse :ohno: by) the user. In the TMG Tag Type definition the user can set an option to either enter (what they believe is) a specific 4-character alphabetic GEDCOM tag type, or set it to use the GEDCOM EVEN/TYPE structure with the TMG tag name automatically output as the TYPE. TMG sets the default for this option to a standard GEDCOM tag type if that seems appropriate and equivalent for the given TMG tag type, or to the EVEN/TYPE structure for TMG tag types which do not have an equivalent GEDCOM tag type. But the user "can" override these defaults to be something more (or less) meaningful to the GEDCOM they wish to produce.

 

Hope this clarifies rather than muddies the water :D

Share this post


Link to post
Share on other sites

You can't override the EVEN TYPE tag name (which is the TMG tag type name) and there would be no point in doing so. You can always do it by renaming the tag type but why would you do that?

Share this post


Link to post
Share on other sites

Jim commented: "You can't override the EVEN TYPE tag name (which is the TMG tag type name)". Yes, that is why I even emphasized that the TMG tag type name is automatically output as the TYPE. The point I was trying to make is that if a user wants to create a special TYPE for some reason, such as it might be recognized by some other software, this is easily accomplished simply by creating a TMG tag type of that name.

Share this post


Link to post
Share on other sites

Jim, I finally got the reason why I am seeing QUAY at the same level as SOUR. You have probably said why a few times about not checking SOUR. I now see that it is the option SOUR level in the Export Wizard which should be left unchecked. ("When selected, surety values (QUAY) are exported at the same level as the source citation from which they are referenced in order to accommodate certain importing programs.")

 

I also see in Help "NOTE: If the citation detail is split, only CD1 is exported". Then, I think that a TMG user emigrating via GEDCOM to RootsMagic should edit Citations so that Citation Detail contains only the "where in the source" information and no || delimiters, moving all else to Citation Memo where the || delimiter might be used to separate Transcription from Comment. One might also enclose the transcription in sensitivity markers {same as RootsMagic's privacy markers} as that may facilitate moving the transcription to the correct field in RootsMagic. I believe this strategy would also be advisable to prepare for RootsMagic's future direct import because the migration is from two fields (Citation Detail and Memo) to three (Page, Research Notes and Comments).

Share this post


Link to post
Share on other sites
How do you search for Citations for which Citation Details contains a delimiter, i.e., CD1, CD2...?

 

List of Citations

filter

Citation Detail Contains || END

 

I also see in Help "NOTE: If the citation detail is split, only CD1 is exported".

Things are more complicated than you might image. Someone might use this technique...

 

Each Citation Detail part is a fixed piece of the citation.

||||||||||||||||

 

And the footnote template looks like this...

Footnote

[CD2] household, [TITLE], [COUNTY], [sTATE], population schedule, enumeration district (ED) [CD4], page/sheet [CD5]; National Archives micropublication [FILM NUMBER].

 

CD1 is reserved for the full footnote for GEDCOM export

 

The generated output...

Alonzo Byram household, 1880 U.S. Census, Bossier Parish, Louisiana, population schedule, Ward 5, enumeration district (ED) 10, page/sheet 41a/27, dwelling 3, family 3, line 10; National Archives micropublication T9-448.

 

So the Citation Detail field would look like this...

Alonzo Byram household, 1880 U.S. Census, Bossier Parish, Louisiana, population schedule, Ward 5, enumeration district (ED) 10, page/sheet 41a/27, dwelling 3, family 3, line 10; National Archives micropublication T9-448||Alonzo Byram||Ward 5||10||41a/25||3||3||10||T9-448

 

:D

 

If this were done, then GEDCOM export is covered.

Share this post


Link to post
Share on other sites

I also see in Help "NOTE: If the citation detail is split, only CD1 is exported".

Although not mentioned in HELP, I believe this is also true for the Citation Memo: only CM1 is exported.

 

Thus Jim's clever trick of always reserving CD1 (and CM1) for the entire text of what you want output to GEDCOM, and only using the higher numbered split parts for TMG template use, will give you the best of both worlds.

 

Further, with the new feature in V9.03 on the Citation edit screen of previewing the full footnote output produced by the TMG template, the user could first just enter the split parts from 2 thru 9, then preview the Citation, and copy/paste the full footnote preview output into CD1.

Share this post


Link to post
Share on other sites

If CM1 is all that is exported, then one should use something other than || to delimit Transcription from Comments. I would suggest {} or %%. (the whole Citation Memo is exported c/w the delimiters || )

Interesting to me is that Jim's exploit of this TMG feature is something I have been beseeching for RootsMagic. That is, the ability to utilize source sentence templates internally for superior footnotes and for the drafting of a coexisting Free Form footnote for export to standard GEDCOM. Currently, you can have one but not the other. Its source templates do not export well to standard GEDCOM.

Edited by TomH

Share this post


Link to post
Share on other sites

Jim, thanks for the clue to searching. I had looked at List of Citations but misinterpreted the Subject(s) of the report to be equivalent to RM's Select People.

 

Michael, I see || in the export of Citation Memo to GEDCOM NOTE so it looks like it is only Citation Detail that filters out all but the first segment.

Share this post


Link to post
Share on other sites

Jim stated much earlier in discussing NOTE tags to "set the TMG Not tag type GEDCOM export value to EVEN TYPE ... will import into RM with all their data." How do you do that?

Share this post


Link to post
Share on other sites

Open the master tag type list. Select the Note tag type and click [Edit]. Select the Other tab. Click the radio button next to 1 EVEN 2 TYPE. Click [OK] to save the change.

Share this post


Link to post
Share on other sites

Embedded citations don't export.

 

This is the Marriage Tag Memo:

ceremony performed by Rev. D. B. Cooper.[CIT:]1:3;this is a citation detail;this is citation memo[:CIT] Now is the time for all good men to come to the aid of the party.

This is the narrative:

...on 29 Sep 1863 at Brookfield, Linn County, Missouri; ceremony performed by Rev. D. B. Cooper. (Family data, Alexander/Keebler Family Bible, Holy Bible (New York: American Bible Society, 1854); original owned in 2000 by Lissa Soergel, (504 North Quaker Lane, Alexandria, Virginia 22304-1827). The Frank Alexander Family Bible passed from Frank to his daughter, Carrie (Alexander) Barns; to her nephew, Allen Long; to his cousin, Lissa Soergel (great-grandaughter of Frank Alexander).) Now is the time for all good men to come to the aid of the party.vi,vii

The embedded citation can be seen between the blue blocks of text that came from the Memo.

 

Here is the GEDCOM:

1 MARR
2 DATE 29 SEP 1863
2 PLAC Brookfield, Linn County, Missouri
2 NOTE ceremony performed by Rev. D. B. Cooper. Now is the time for all good me
3 CONC n to come to the aid of the party.
2 SOUR @S2@
3 QUAY 2
2 SOUR @S12@
3 PAGE Uncle Sam & Aunt Bell married Sept. 1863.
3 QUAY 2

Source #1 was cited; it's not listed as a source nor is it expanded by TMG to be exported in the NOTE.

 

Suggests that users wishing to migrate should make sure that embedded citations are also normal citations.

Share this post


Link to post
Share on other sites
Embedded citations don't export

That's true. Embedded citations are just text and do not link to records in the citation table.

 

The structure of the embedded 'citation' is used to create footnotes or endnotes in reports during report generation. They might not even involve a source and could be just be a comment.

Share this post


Link to post
Share on other sites

 

Embedded citations don't export

That's true. Embedded citations are just text and do not link to records in the citation table.

I understand that the citation is not in a table but the master source it cites (#1 in this case) is in a source table. I don't understand why the phrases between the code tags are not part of the resulting parenthetical note in the narrative output but that is beside the point which is that the source description is retrieved from that table. I'm not surprised that there is no citation SOUR tag exported for an embedded citation but what does seem reasonable to expect is that the parenthetical citation as it appears in the narrative should also appear in the exported NOTE.

Share this post


Link to post
Share on other sites

I found some previous discussion about the failure to export anything from embedded citations and a suggested alternative:

John,

 

Two points:

 

1. Are you sure that the rest of your data is being exported? Do you have people entered as witnesses, say in census, will, or residence tags? Do you have two- person tags, especially for people who are not married to each other? Have you used Sentence or Roles to convey information? Do you use split memos or split citation details? If the answer to any of these is yes, the embedded citations may be the least of your worries, and perhaps you should explore other ways of sharing your data.

 

2. You can solve the issue of embedded citations by breaking the text into separate tags, ending each tag where you want the citation to appear. Then use conventional rather than embedded citations, and order the tags with sort dates.

Share this post


Link to post
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now

×