Truncation of memo fields in Tags & Embedded Citation Problems

[Leslie1899 0]

I'm hoping somebody has an answer as to why occasional memo fields are being either truncated or are not shown at all on the Details screen and, occasionally, text from these fields is not exported into my word processing program.

 

Naturally, I'm in a deadline situation with several reports due to a respected genealogist so I really, really care that my information is accurate. In one case, I have a Will Tag memo field which, as soon as I type in the text and click OK, the following happens: first the county and state no longer show on the Details Screen and are replaced by the name of the focus person; secondly, although I can see that the memo field is full of text, it does not output into word processing.

 

After repeatedly creating new Will Tags where the same thing happened, I tried using subsequent Note Tags to contain the information. At first this worked, and then it began exhibiting the same behavior as the Will Tag. After many repeat efforts, I managed to cobble together a work-around using Note Tags, but it's extremely awkward. I have no idea why this happened. There must be some sort of invisible-to-the-user control codes in these Tags.

 

There's more: in several Note Tags, I have laboriously typed out transcriptions of Revolutionary War audited accounts from SC. In the output report, only random words (sometimes the beginning of the memo field and sometimes only the last few words representing signatures) are output.

 

This is bad enough, but there's even more. I am probably the heaviest user of embedded citations in the WhollyGenes community. Most work perfectly (although, I'm unable to produce them in a List of Citations Report despite using TMGv7.04), but, yesterday some didn't. An example is a Note Tag that contains census information on three men. In the memo field, I typed the data I think applies to the focus person followed by an embedded citation. I then listed data for the second man with another embedded citation. For the third man's data, I used the regular citation screen. In the report, the sentences printed out, but no footnotes were produced from the two embedded citations so that all three men were shown with a citation source that applied only to the last man. In the past, if the citation wasn't properly formatted, the printout produced the flawed citation detail. It didn't this time. I've double-checked the citation format and it looks okay.

 

Does anyone have any suggestions? I've sent in a Trouble Report to tech support, but as it's the weekend I know I won't hear back soon enough to help me over the weekend. I really have to get this material into the mail by Monday morning at the latest, so I'm sort of desperate.

Leslie

[Jim Byram 0]

Embedded 'citations' are just structured text that is processed during report output to produce footnotes. Only real citations that are recorded in the citation data table are output by the List of Citations report.

 

Since embedded citations are just text in memo fields, you can identify events using a particular embedded citation with the List of Events report and a filter searching for a text pattern contained in the memo field. You want to keep the pattern to as few characters as possible to speed the search.

 

You need to show us the complete text of a memo that is causing an output problem so that it can be examined. The sentence structure for the tag would also help.

 

From some of your other comments, you need to refresh your program installation and I suspect that your project will need to be evaluated.

[Leslie1899 0]

As I went through about 30 problem Tags this morning, I noticed that in some of the ones that truncate memo data, I'd used print control codes to try to separate stuff during output to word processing. The code I was using was [:TAB:]. I've used it in earlier versions, but not only does the control code NOT initiate a Tab Stop, the code itself doesn't print out in word processing files (as it did in earlier versions of TMG). But, this control code seems to be causing data not to show in the Details window and NOT TO OUTPUT to wordprocessing. Only a few words here and there print out.

 

I've got a real disaster on my hands here, because even after removing every single [:TAB:] code, material iin the memo field still doesn't show up on Details screen as it should. I haven't tried to re-output the file because I've spent too much time painstakingly adding the "lost" data, as well as footnotes, to the word processing file.

 

I will try your suggestion about List of Events report and see if that helps. However, when TMG gives the option of searching for embedded citations for a List Report, one would think TMG meant it. I assume it tries to do what you're suggesting?

 

I agree that something is seriously wrong. Here is an example from a report I was working on yesterday:

there are three James Watsons in Pike County. I believe this man to be James Sr who is between the ages of 60 and 70.[CIT:]683:;James Watson Sr||Pike County||Missouri||10||Roll 72, page 258, image 59||;[:CIT] Another James Sr is between 70 and 80.[CIT:]683:;James Watson Sr||Pike County||Missouri||24||Roll 72, page 257, image 57||;[:CIT] James M. Watson is between 30 and 40.

 

Only the citation for James M. Watson was output and showed up in the word processing file.

 

I can't provide an example of the truncation because I don't know how to do a screen shot to show the Details screen and I don't think you could tell anything from the scrambled words that were output.

 

Leslie

[Jim Byram 0]

You cannot use split memo parts in embedded citations.

 

From the Memo help topic...

"NOTE: Split citation detail elements and split citation memo elements are not supported in embedded citations. Any ‘||’ delimiters within embedded citations would be interpreted as delimiters for split memo elements for the memo field."

 

I suspect that is the root of your problems. The memo output will be truncated at the first '||' encountered in an embedded citation (depending on the sentence structure).

 

I will try your suggestion about List of Events report and see if that helps. However, when TMG gives the option of searching for embedded citations for a List Report, one would think TMG meant it. I assume it tries to do what you're suggesting?

I have no idea what this means. I just explained how to use the List of Events report to locate embedded citations.

[Leslie1899 0]

You cannot use split memo parts in embedded citations.

 

This is terrible. The only split citations I use are for census and most are not embedded, but sometimes . . . . .

 

However, this isn't the MAIN PROBLEM. These are not the memo fields that are truncating. Those do NOT have any split citations at all.

 

Here is an example of a Note Tag that has a problem:

"for 302 days Duty as adjutant under the command of Colo. Watson from 6 February to 4 Dec 1779 131-00-0

State of South Carolina

To Jonathan Fitchet for Service Done in the Militia as Adjutant under Colo Watson three Hundrerd and two Days at three Pounds St pr Day South

In Sterling 131/00/0

This is to certify that the above Service was Done pr me

Samuel Watson Colo

 

Lost in Said Service

one gun

one Saddle and Wallat

one Blanket & Sidecoat

Shot Bag, Bell, Tomhawke & Hand Kerchief

above as appraised

State of Carolina} this day Jonathan

Camden Deztrick} Fitchet came before me

and made oath that the above account is Just and that he never Received any part thereof

Sworn and Subscribed before me this

20th day of Agst 1784

Samuel Watson J. P."

 

27 Sept 1785

The within is one of the accounts [unreadable] of in the [unreadable] that I left in Mr. Parts office Janry ye 11th 1780 certified by me,

Samuel Watson

The above Jonathan Fitchett was in Georgia

with me and Sent upon a Command with Major

Ross to Brier Creek and there taken prisoner by the inglish Certified by me

Samuel Watson"

 

This document, much folded, was next certified by Joseph Howe (paymaster to Col. Watson's Regt.) and also by Wm. Bratton and Fitchett was recompensed for his service and losses in the total amount of L131.

End of example.

 

The only thing that shows on the Details screen is Fichet came before me which was absolutely all that was output to word processing. The CD printed out perfectly.

Leslie

[John Cardinal 0]

There are a couple right-braces ("}") in the example above. Are there left-braces in there, too? They would cause a problem: they are the "sensitive text" markers and all text between { and } is suppressed.

[Leslie1899 0]

There are a couple right-braces ("}") in the example above. Are there left-braces in there, too? They would cause a problem: they are the "sensitive text" markers and all text between { and } is suppressed.

 

No, there are never { left braces. I use them when I'm typing data from legal sources that use right braces in front of legal descriptions.

 

I've found some memo fields that include the [:TAB:] codes which have exported perfectly. I am beyond puzzled.

Leslie

[Jim Byram 0]

Braces and brackets are reserved characters. Any special character that TMG uses in sentence construction or footnote construction is a reserved character.

 

If you use braces as part of the text, you need to escape them. If you use brackets as part of the text, you need to escape them. This is true whether using single characters or matched pairs.

 

In help, read the Escape Character topic.

 

I seriously doubt there is any issue with the [:TAB:] print code but a loose unescaped bracket somewhere as part of the text will completely blow the TAB print code.

 

However, this isn't the MAIN PROBLEM. These are not the memo fields that are truncating. Those do NOT have any split citations at all.

That makes no sense.

 

First, I asked for an example of text being truncated and you provide the example with '||' characters which will surely result in truncated output. Yet you say that output is not truncated?

 

Please deal with one example at a time. Provide an example that gives a problem with only the necessary commentary. Let's deal with that example. After that is resolved, let's move to the next example. That way, we'll resolve all of your output issues.

 

You provided your first example and I've explained the problems with that example.

 

You provided your second example and we've explained the problems with that example.

[Leslie1899 0]

Jim:

There were two separate problems occurring. I posted an example of the easiest one to show which had the split CD embedded footnotes. You've explained that. I may not be happy about it, but it's easy to work around using two separate Note Tags to contain the data on the two different men.

 

The second problem occurred in much lengthier memo fields and it took a long time to find one short enough to post. There were no "loose" brackets floating around in these memo fields. Only, rarely, right braces, but often the [:TAB:] code had been there.

 

I have gone through and removed all Tab codes. I have double-checked the right braces (there never were any left braces) and looked for errant ] or ['s of which there were none. I haven't tried to export a another word processing report, but I can tell you from looking at the Details window on TMG that the problem hasn't been fixed. It is only occurring on random Note Tags, but I can tell by looking at the screen, that something is wrong . . . . . . if no text shows up on the Note tag line, that's been a sure sign that the text was not going to be exported.

 

If I click on the tag, the text is there which is why I can verify that there are no printer codes left or anything else that shouldn't be there. But only a few words, which can occur at the beginning, middle, or end, are being exported.

 

My routine maintenance is as follows: twice a day, at the least, I Reindex, Optimize, Validate File Integrity, and Backup the Project on which I'm working. If I'm doing really heavy editing or adding a lot of new people, I go through this procedure more frequently. And, I've never merged a Gedcom with my project file.

Leslie

[Jim Byram 0]

Did you escape the right braces and brackets? (which you must do) ...and test the above memo example?

 

Please post a memo using [:TAB:] that truncates so that I can test it.

[Leslie1899 0]

Did you escape the right braces and brackets? (which you must do) ...and test the above memo example?

 

Please note the following memo field:

 

"Moorhead & Watson

vs.

the Publick

 

employed by Col. Saml

Watson to carry Cloath &

money to 24 men of his

Regimt at Briar Creek

the 3d March 1779

Four hundred &

thirteen Pounds 12/6

South Carolina/} William Moorhead & Wm. Watson

New Aquisition/} appeared before me and made oath

that the written account is Just & true

and [?] Stated

August ye 1st 1779

William Moorehead

William Watson

 

April 8th 1779

State of S. Carolina [?] To Wm Moorhead & Wm Watson

Being Employed by Col. Samuel Watson To carry

Cloathing and money for Twenty four men of Sd Regimt

was Taken Prisoners by the English @ Bryer creek

the 3d of March

To 20 Days of OurSelves and four horses @ L3..5} L91..0..0

pr Day To Each man/} L91..0..0

To cash Paid for ourselves & horses 231..12..6

L413..12.6

 

State of South Carolina

The above William Moorhead and Watson

was imployed by me to take Necesarys to

Sd. prisoners and I Say it before the Honourable Hous for their approbation

pr me Samuel Watson Colo"

 

 

On the Details screen all that shows on the Note line is L91..0..0.

 

On the jounal report output from this Tag, all I get follows:

 

L91..0..0

To cash Paid for ourselves & horses 231..12..6

L413..12.6

 

State of South Carolina

The above William Moorhead and Watson

was imployed by me to take Necesarys to

Sd. prisoners and I Say it before the Honourable Hous for their approbation

pr me Samuel Watson Colo."

 

 

Addinig the escape character before the right braces hasn't helped. There are repeated instances of this, but I don't think I need to fill up this post with them. Clearly you are correct that the braces are causing the problem, but I don't know how else to copy "legal" documents which use them. I use [ ] brackets to enclose my comments and parens as used in the documents. Yesterday, I deleted all the printer codes in these memos in case that was partly to blame so can't append any examples.

 

I didn't have time to test all this yesterday. I had to re-type everything I could find that hadn't been output in order to get the reports into the mail this morning.

Leslie

[Jim Byram 0]

The escape character is '\' (backslash), not '/' (slash). One '}' wasn't escaped at all. And you didn't escape the brackets. Testing your memo after repairing it.

 

The corrected memo (attached) displayed in the Tag Box correctly and output correctly. For testing, I used an Anecdote tag, added '[:CR:][:CR:]' to the beginning to force it to a new paragraph, and dated it to make it the last tag for an individual so that it would stand alone in the report.

memo.txt

Edited March 9, 2009 by Jim Byram

[Leslie1899 0]

I need to add that I found a copy of a Journal Report on the same individual using basically the same data produced 16 Sep 2007, before I upgraded to version 7, in which absolutely NO data was truncated. No escape characters were used in front of any of the right braces prior to generating this report.

 

Therefore, my surprise at having difficulties with a report on the same man using the same, albeit updated, data isn't out of line. Prior to this week, I hadn't output anything of significance since upgrading to version 7 last fall. I really like 7.04, but, evidently, it must be more sensitive.

 

With respect to the fact that one cannot use split CDs within embedded citations, I emailed Terry Reigel (my hero), whose suggestions for split census CDs I'd followed, to suggest he post a warning on his census page about this.

Leslie

[Jim Byram 0]

I see the difference between the v6.12 output and the v7.04 output when the reserved characters are not escaped. I have no recollection of anything being done that would result in that. There was no major change in report output between v6 and v7.

 

However, that said, the reserved characters always have been reserved and the recommendation always has been not to use them. The addition of the escape character as a feature provides a means to use the reserved characters with no concern about side-effects.

[Leslie1899 0]

Okay, I see that I misread the Help section on escape characters. I'd never seen any post that explained v7 was this sensitive and had no reason to expect such serious problems in my large database. I deliberately waited to update to 7 until the fall of 2008, waiting for initial software issues to be resolved.

 

After loading TMG and running a test, I find that adding the escape character ONLY in front of the right brace characters solves the problem on the Details screen and, I expect, on the output as well. As long as I don't have to add escape characters to all the [ ] brackets found in my memos, I can live with this.

 

Had I needed to add escape characters to all the square brackets, I would've despaired because there are too many of them in too many memo fields throughout my large database. I was feeling deeply discouraged (and beginning to regret updating to v7 in spite of loving the split CDS and just finishing correcting about 10,500 census citations to utiliize them). Now that I find only the right } are affected, it's not such in insurmountable task. These are used only when trying to reproduce court documents and, consequently, shouldn't be too hard to find.

 

Thanks for you help in resolving this perplexing issue.

Leslie

Edited March 10, 2009 by Leslie1899

[Jim Byram 0]

TMG Utility can take care of such changes on a global scale so you don't have to do this memo by memo.

http://www.johncardinal.com/tmgutil/

[Leslie1899 0]

I went to John Cardinal's utility immediately after posting my reply (before reading your response) and ran it. Because I'd already fixed two tags, I didn't want to automatically run the search and replace program which might have entered double escape characters on the fixed tags. Instead I ran search and replace to create a log file which I saved to word processing, printed, and used as a go-by. There were 28 instances that required escape characters on 14 different individuals, but most were on only two men. I've completed the fix. In two cases, the errant } was a typo where I'd meant to hit the ] key. Had I had more } to which to add an escape character, I would've let the utility do the replacing, but this seemed safer. I really love the ability to create a log file (thanks, John, as I've often discovered, your utility is just great).

Leslie