Is this a bug or a limitation of TMG?

[Guest Michael Dietz]

I am trying to import a GEDCOM file from Ancestry.com. There are 140,000 individuals in the file. What is unusual about this file is the quantity of NOTEs. The file size is approximately 90 mgb which is very large for that number of individuals. I looked in the .ged file and found there is about 30 mgb of 'data' and 60 mgb of NOTE, CONT, and CONCs. I use the advanced import function selecting the assume marriages and read NPX names options. I also unassign all the non-standard event tags.

 

When I perform the import it crashes in step 8 with the following displays:

Creating the necessary tables is 60%

Reading submitter info is 4%

The odometer reads 60.9413 1866504/3062792

 

The error message is:

filec:\ancestry input\jenkins 2582920\jenkins2582920_g.fpt is too large 0 TAGEVEN

 

Looking at the file properties gives:

 

Size = 2,145,481,472 bytes

Size on Disk = 2,147,483,648 bytes

 

Disk free space is 18.1 ggb

 

What should I do?

 

Thank you

Mike

[John Cardinal 0]

Mike,

 

First, I assume that "mgb" is megabytes; that's usually abbreviated "MB". For gigabytes, use "GB".

 

The error message indicates that the "_g.fpt" file is more than 2GB. I believe 2GB is the maximum size of a Visual FoxPro data file and thus for TMG. That is apparently the immediate cause of the issue.

 

It's not clear why the _g.fpt file is 2GB when the GEDCOM file was 90MB. It's possible some records in the GEDCOM file are attached to multiple TMG records and that essentially multiplies the storage used for the records. It's also possible that the GEDCOM file references external files that TMG stores in the _g.fpt file. (The _g.fpt file is where TMG stores the text from event memos, among other things.)

 

You may be able to inspect the GEDCOM file to determine what's amiss. You'll need a good text editor, one capable of opening a 90MB text file.

 

If that doesn't work, then you might try a process of elimination where you configure TMG to ignore some of the GEDCOM tags. If the problem is the NOTE tags (and their associated CONT and CONC tags), that may not be very helpful: the load will work when skipping NOTE tags but you probably can't skip all those. I am a little rusty on GEDCOM, but as I recall, NOTE tags can be used to describe an event, but they are also used to attach text to other record types.

 

You could try importing the GEDCOM file into some other genealogy program to see if any program can handle it. If you succeed, then you could try to load the data into TMG directly from the other program.

[Jim Byram 0]

Mike,

 

Using Help / Message Manager / Outgoing tab, browse to the GEDCOM, select 'Compress before sending' and upload the GEDCOM to the Wholly Genes server. Let me know the name of the uploaded file so that I can retrieve it.

 

Jim

[Guest Michael Dietz]

Thank you Jim and John.

 

It is the NOTE, CONC, and CONT tags. I used Word to delete these tags (they are in the last third of the file) and then imported the remainding data, the 30MB of stuff. It went fine except I did get several thousand error messages saying the NOTEs were missing.

 

As to why 60Mb of text grew to over 2GB I have no idea. I will send the compressed GEDCOM to you Jim but I doubt if there is anything really amiss. I do not know that much about FoxPro but I do remember that when I was setting up b-tree databases I would run into trouble where every item resulted in a separate branch on the tree. It would get just to big. So I would have a few thousand items and trees that were enormous. I have a feeling the same thing is happening here.

 

Again thank you

Mike

[Guest Michael Dietz]

Jim:

The file is uploaded It is named 2582920.zip and is from folder C:\Ancestry Input\Jenkins 2582920. The uploaded file size is 27,248,662.

 

Thank you.

Mike

[Jim Byram 0]

Got it. Thanks, Mike.

[Jim Byram 0]

The fundamental problem is that a single note will be linked to by many event tags. When the g-table is created, that note will be stored in the memo of each tag record that links to the note. So the total size of memos stored in the g-table balloons. There is not really any way around this.

 

I suspect that many of these 'notes' are really sources (thus the reason for being linked to by many events). If they had been recorded as citations and sources, we wouldn't be seeing the problem.

 

The GEDCOM contains:

140,641 individuals

58,593 families

62,968 notes

[Guest Michael Dietz]

Thank you Jim.

 

I figured something was amiss. I have two ways to deal with this. (Actually three, I can forget the whole darn thing). I can remove the NOTE lines from the GEDCOM and build the tree accepting the thousands of errors. I then can manually search the removed NOTE lines for each record I want from the file. Or I can go to Ancestry.com whenever I find somebody I am interested in and copy the NOTE or source from there. Either way it will be a lot of manual work.

 

Again thank you.

Mike

[Jim Byram 0]

You could break the GEDCOM into two or three GEDCOMs. There would be some people with missing relationship links. You could even overlap the breaks and might get most of those links.

 

For example... You edit the GEDCOM and remove 1/2 of the person records leaving the first half (with some overlap). Then import that. And do the same retaining the last half. Figure out the break point and overlap 1000 records. The imports will need to be into 2 different projects. If you try to combine them, you would hit the same g-table limit.