Jump to content


Photo

FTM Import taking a long time


  • Please log in to reply
22 replies to this topic

#1 DaveR

DaveR
  • Members
  • 11 posts

Posted 13 March 2014 - 06:04 AM

I bought TMG v9 yesterday, downloaded one of my trees (26,000+ people, 180,000+ citations) from Ancestry.com to FTM 2014, exported that in FTM 2008/2009 format and started the TMG v9 import process.  That was 18 hours ago.  It is roughly 45% of the way through the first step (importing people and names).  This is on a multi-core Intel i7 with 8GB RAM and a SSD running 64-bit Windows 7.  Is that normal???  Are there "best practices" for dealing with Ancestry.com data?  Should I think in terms of thousands of family twigs instead of several trees?



#2 Jim Byram

Jim Byram
  • Moderators
  • 7,340 posts
  • Gender:Male
  • Location:Framingham, MA

Posted 13 March 2014 - 06:16 AM

The FTM import can take a while. The GenBridge part of the import is not particularly well optimized. It helps to pack the FTM database (Tools / Compact file...) before doing the TMG import.



#3 DaveR

DaveR
  • Members
  • 11 posts

Posted 14 March 2014 - 03:55 AM

40 hours into the import it is 2% of the way through step 4 (finish) on the first tree. 

 

I use FTM just to clean up data in trees I maintain on Ancestry.com.  In this case I created the FTM tree by linking to Ancestry.com immediately before the export to FTM 2008/2009 which was immediately before I began the import to TMG.  It seems odd to pack the FTM database given no activity in it, but I try that with the next tree.  Thanks.

 

For the Ancestry.com - FTM 2012/2014 process to maintain clickable links to source images I needed to think of the Ancestry.com tree as primary and the FTM tree as expendable.

 

Given the time it takes to import, I likely need to think of the TMG tree as primary and look for another way to make the results available on the web. Just as well.



#4 Terry Reigel

Terry Reigel
  • Moderators
  • 4,643 posts
  • Gender:Male
  • Location:Boone, NC

Posted 14 March 2014 - 04:48 AM

Given the time it takes to import, I likely need to think of the TMG tree as primary and look for another way to make the results available on the web. Just as well.

Dave,

 

That's the course I've adopted. I think that publishing TMG data with Second Site offers much more robust opportunities than Ancestry trees do. See my articles on use of Second Site, and the illustrations linked to that starting page.

 

The one advantage to posting on Ancestry is the easy access by other Ancestry users. Ancestry does accept GEDCOM uploads, but pretty poorly - much is lost. However I have uploaded basic data on a couple of popular lines, and in over a year have had not a single contact from them.


Terry

See my Tutorials and Articles on using TMG at tmg.reigelridge.com

The Second Edition of my book, A Primer for The Master Genealogist, is still available
. For more information see my website.

#5 DaveR

DaveR
  • Members
  • 11 posts

Posted 14 March 2014 - 03:10 PM

Terry,,

 

I get fairly regular contact on three of my Ancestry trees.  The others not so much.  One problem with the Ancestry-FTM link is that the link sometimes becomes corrupt.  I susect it occurs when one person appears in two d ifferent places in the tree and I modify the relationships to eliminate one of them.  There may be other circumstances.  In any event one time when it became corrupt I deleted the Ancestry tree and reloaded it from FTM.  In the process, every last link to census records, etc.in the ttree stopped being clickable and 10,000+ "new" Ancestry hints popped up.  After that I would only delete the FTM copy so retain the clickable links to images of the source documents on Ancestry.  I have used GEDcom to uplad to Ancestry.  If I want clickable links to source documents it worked better to upload the GEDcom without source information then document it on Ancestry: It is still quite tedious on a large tree but it sames much of the clean up once it is documented.  I'd have much more flexibility with my own web site.

 

Second Site looks promising.  I used GedHTree for that purpose a number of years ago.

 

Dave



#6 Terry Reigel

Terry Reigel
  • Moderators
  • 4,643 posts
  • Gender:Male
  • Location:Boone, NC

Posted 14 March 2014 - 04:52 PM

Dave,

 

I've never uploaded a tree to Ancestry from FTM, but I know that when you do that from a GEDCOM the tree doesn't recognize any of the citations. So I just exclude most of my data when I do that and upload just BMDB data, and let Ancestry leave the citations unrecognized. That means I get tons of hints for sources I already have, but I've researched the folks I'm uploading pretty well so I'm not interested in that. I'm more interested in whether other trees have anything useful I've missed, and occasionally I find that. But mostly the hints from other trees are data entered differently or what they have is just wrong. :(

 

With my own tree I can get it just as I want it, and I like that a lot better.

 

Terry


Terry

See my Tutorials and Articles on using TMG at tmg.reigelridge.com

The Second Edition of my book, A Primer for The Master Genealogist, is still available
. For more information see my website.

#7 DaveR

DaveR
  • Members
  • 11 posts

Posted 15 March 2014 - 03:14 AM

64 hours into the import and it is now 27% of the way through step 4 (finish) on the first tree.

 

Until one can see the result it is hard to appreciate what a great job GenBridge does on the import.

 

Perhaps some future TMG update could look into GenBridge performance.

 

These are mostly descendant trees that start small in the late 1600s/early 1700s and grow geometrically larger toward the present. 

 

Current information in other trees tends to involve living individuals and shows only as "Private". Older information may offer clues to what happened to some child who "disappeared".  Sometimes they just made it up.  Sometimes they confused two people with similar names.  Sometimes it looks like they got it right.

 

Information between 1850 and 1940 tends to come from state and federal census information with some information from WW1/WW2 draft registrations, vital records, find-a-grave, city directories, etc. 

 

I hear from people who discover their parents or grandparents in one of the trees or see that I added one of the photos they posted.. In many cases I can connect them with other researchers working their lines.

 

Dave



#8 DaveR

DaveR
  • Members
  • 11 posts

Posted 16 March 2014 - 03:22 AM

88 hours into the import and it is now 52% of the way through step 4 (finish) "Importing other person information" on the first tree. <sigh>



#9 TPG

TPG
  • Members
  • 57 posts
  • Gender:Male
  • Location:Silver Spring, Maryland

Posted 16 March 2014 - 11:36 AM

If I might be excused for stating the obvious: 88 hours into an import with it being only 52% finished implies that something is seriously wrong! Jim Byram's comment that " The GenBridge part of the import is not particularly well optimized" is at best a gross understatement. Assuming that DaveR has not messed up the import parameters, then Wholey Genes should definitely treat it as a problem that needs to be addressed. If, on the other hand, DaveR has messed up the import parameters, then we should be asking him questions about how he has gone about the import - and not be discussing Second Site or other alternatives.

Although I maintain my primarily data on a TMG file, I keep copies of several of my trees at ancestry.com and get a fair number of contacts via that route. I have written a side program that scans GEDCOMs to be uploaded to ancestry.com, compares them to the TMG data files, and then adds "witness" and some other information extracted from the TMG files to the GEDCOM before uploading it. The "witness" part works fairly well, but the Event Notes have to be automatically broken up into parts due to the limitations in ancestry.com. In some cases, the narrative portion of some of the GEDCOM tags are truncated by ancestry.com. I am, however, not very happy with that whole process. But, in general, it serves the purpose of giving other ancestry.com folk a better picture of what I have in my TMG file. "http://trees.ancestr...rson/6327322746" and "http://trees.ancestr...rson/6327322048" will give those of you on ancestry.com some idea of what I'm talking about.

An aside to Terry Riegel -
My experience with uploading GEDCOMs to ancestry.com is that they do recognize source citations. They replicate whatever is in the citation itself. What they don't do is provide a direct link to the source document itself. But this is quite understandable, since the TMG source citation itself doesn't contain any such link.

#10 Terry Reigel

Terry Reigel
  • Moderators
  • 4,643 posts
  • Gender:Male
  • Location:Boone, NC

Posted 16 March 2014 - 07:33 PM

An aside to Terry Riegel -
My experience with uploading GEDCOMs to ancestry.com is that they do recognize source citations. 

Sorry, Tom, but you are simply wrong. Yes, the tree displays the citations included an uploaded GEDCOM, but it fails  to recognize those that come from Ancestry, as I said. As a result Ancestry will then provide you "hints" to the very source you cited. This renders the hints to Ancestry sources virtually unusable because the majority of them are already present in the data. That's what I described.


Terry

See my Tutorials and Articles on using TMG at tmg.reigelridge.com

The Second Edition of my book, A Primer for The Master Genealogist, is still available
. For more information see my website.

#11 DaveR

DaveR
  • Members
  • 11 posts

Posted 17 March 2014 - 03:17 AM

If I have somehow messed up the process, that would be good to know. 

 

This started as an Ancestry tree with 26,000+ people and 180,000+ citations (clickable links to the associated census recrods, find-a-grave, social security death index, etc.)

I created a new FTM 2014 tree by downloading the tree from Ancestry.

I exported the FTM 2014 data in FTM 2008/2009 format so that it could be imported by TMG v9.

I started the TMG v9 import specifying FTM 2008/2009 format and pointing it to the FTM 2008/2009 export.

That's it.

 

112 hours into the import and it is 76% of the way through step 4 (finish) "importing other person information"  The overall progress shown is 36%, which doesn't seem to have updated since the start of step 4, so I am hoping for something better.  This is the first of several trees.   Anything learned from this one might benefit later imports.

 

This is with a fast processor (4th generation Intel I7 - the import is consistently using 25% of the CPU) with 8GB of RAM (the import is currently using 218 MB which has grown slowly from about 65MB when it started).  The disk is solid state.  There should be nothing hardware-wise to slow it down.  It is all on one machine so there should be no network slowdown.

 

I expect a tree of this size to take some time.  I'd be OK with 3-4 hours for the import.  I'd be delighted with 1-2 hours.  The progress is very slow but it is still reporting progress: it hasn't locked up or stopped.

 

Is there a way to dissect the problem and better identify what is going on?  I want to speed up the process/fix the problem not jiust say "it's slow".



#12 TPG

TPG
  • Members
  • 57 posts
  • Gender:Male
  • Location:Silver Spring, Maryland

Posted 17 March 2014 - 07:24 AM

 
Yes, the tree displays the citations included an uploaded GEDCOM, but it fails  to recognize those that come from Ancestry, as I said.

 

I guess that I just didn't understand your use of the term "recognize". I took it to mean that ancestry.com didn't display any of the citations, but not that they didn't replicate the links to ancestry.com source records in some fashion or another. But, why then would you expect them to do so?

#13 Terry Reigel

Terry Reigel
  • Moderators
  • 4,643 posts
  • Gender:Male
  • Location:Boone, NC

Posted 17 March 2014 - 08:59 AM

Tom,

 

I'd "expect" them to recognize those citations to Ancestry sources because failure to do so renders the hints to Ancestry sources basically useless. That's really the reason I mention this issue.

 

Realistically, Ancestry's process for accepting trees from GEDCOM is seriously flawed, and is clearly not a priority in their development effort. As you point out, recognizing the citations to Ancestry sources is hard, and they don't even do easy things that would improve the process, so expecting them to do the obvious but difficult is not realistic. 

 

An example of poor handling of simple GEDCOM data is text from a Tag Memo, which is placed in a GEDCOM NOTE field on export, is not placed in the text field associated with events in the tree on import. Instead it is buried deep in a citation (even though it was not part of any citation in the GEDCOM). This means that any cautionary notes to add as Memo text are for practical purposes unfindable by readers.

 

Another major flaw, which is probably difficult to handle in practice, is that there is no means to provide updated data to a tree via GEDCOM. So far as I know, you can't even add new people via GEDCOM to an existing tree and manually connect them to the existing people. The only choice is to add a whole new tree, which may explain some of the duplicate trees that plague the system.

 

I'd really like to put more of my data up in Ancestry Trees to make it more accessible to others, but they currently make it very difficult. I would think that encouraging more advanced researchers to place date in their trees would make their system more valuable to all their users, but apparently they don't see it that way.


Terry

See my Tutorials and Articles on using TMG at tmg.reigelridge.com

The Second Edition of my book, A Primer for The Master Genealogist, is still available
. For more information see my website.

#14 DaveR

DaveR
  • Members
  • 11 posts

Posted 18 March 2014 - 02:17 AM

The import of the first tree finally completed after 133 hours.

 

Regarding the Ancestry trees, I was able to retain clickable links to Ancestry citations by treating the Ancestry tree as primary and the FTM tree as expendable.  It is possilbe to use FTM to clean up place names, though the FTM tools to do so aren't nearly as good as TMG's tools.  I was able to add branches to an existing Ancestry tree through GEDCOM import.and connect it manually to existing people in the tree.  I found this useful only in building a framework of birth, marriage, death, people, places and relationships to later document on Ancestry if Ancestry documentation was the intended result.



#15 Terry Reigel

Terry Reigel
  • Moderators
  • 4,643 posts
  • Gender:Male
  • Location:Boone, NC

Posted 18 March 2014 - 04:47 AM

I was able to add branches to an existing Ancestry tree through GEDCOM import.and connect it manually to existing people in the tree.  I found this useful only in building a framework of birth, marriage, death, people, places and relationships to later document on Ancestry if Ancestry documentation was the intended result.

Dave,

 

Could you tell me how you did that? I've not been able to figure it out.


Terry

See my Tutorials and Articles on using TMG at tmg.reigelridge.com

The Second Edition of my book, A Primer for The Master Genealogist, is still available
. For more information see my website.

#16 DaveR

DaveR
  • Members
  • 11 posts

Posted 18 March 2014 - 09:37 AM

Terry,

 

This works:

  • Download an Ancestry tree to a new FTM 2012/2014 tree or sync to an existing FTM 20012/2014 tree.
  • Import the GEDCOM for the branch you want to add into the FTM 2012/2014 tree
  • Synchronize the FTM tree with the Ancestry tree
  • manually add the relationships for the new people to their proper place in the Ancestry tree.  If the GEDCOM is "pruned" to what you want to add.  That might involve a father and mother on an ancestor tree or the children of one set of parents on a descendants tree.

It should also preserve clickable links to source images such as census records in the original part of the Ancestry tree.

 

Dave



#17 Terry Reigel

Terry Reigel
  • Moderators
  • 4,643 posts
  • Gender:Male
  • Location:Boone, NC

Posted 18 March 2014 - 10:06 AM

Thanks, Dave,

 

From this I gather you can't do this on the Ancestry website, but have to do it in FTM then upload from there. I don't know that I want to get FTM just to do that. :(


Terry

See my Tutorials and Articles on using TMG at tmg.reigelridge.com

The Second Edition of my book, A Primer for The Master Genealogist, is still available
. For more information see my website.

#18 DaveR

DaveR
  • Members
  • 11 posts

Posted 19 March 2014 - 01:55 AM

Terry,

 

I hear you.  I picked up FTM just because of the sync option.  Aside from import/export tasks, I only use FTM to clean up place names in Ancestry trees.  I could do that via GEDCOM import/export at the expense of losing 180,000+ clickable links to sourse images.  While I find those links useful and easy to create, I often wonder if they are worth maintaining given the primitive tools and givien that they are essentially limited to one repository   Even downloading a tree from Ancestry to FTM, deleting the Ancestry tree then uploading it again from FTM breaks 180,000+ clickable links.  If I want them back, I need to once more document 180,000+ citations AND delete the no longer clickable links to the same  citations uploaded from FTM.  No thank you: way too labor intensive.

 

I am early in the TMG v9 leaening curve.  I will look into Second Site to see what's involved in getting clickable links to source images that way and will also look into what changes I can make in TMG v9 and have them show up in an Ancestry tree without breaking any clickable links.  I'll pick a much smaller tree for such experiments though. 

 

Dave



#19 Terry Reigel

Terry Reigel
  • Moderators
  • 4,643 posts
  • Gender:Male
  • Location:Boone, NC

Posted 19 March 2014 - 04:50 AM

Dave,

 

With TMG and Second Site you have two ways available to create links to sources.

 

One is to save an image of it and attach it as an exhibit in TMG. Second Site will then let you include that image in the site. The beauty of that approach is it works for any source you can find as an image or can scan or photograph. On the other hand with 180,000 citations doing this is a huge task.

 

The other approach is to include the URL to online sources, and properly done Second Site will make that a link in your site. This works well for links to the main page of a site like Ancestry, but since many such sites revise their structure from time to time links to individual source images are more problematic. Because of that I provide URLs only to the "front page" and try to provide enough information about the source itself in the citation that a reader can find it.


Terry

See my Tutorials and Articles on using TMG at tmg.reigelridge.com

The Second Edition of my book, A Primer for The Master Genealogist, is still available
. For more information see my website.

#20 DaveR

DaveR
  • Members
  • 11 posts

Posted 20 March 2014 - 03:56 AM

Terry,

 

180,000 citations is a large number.  It is difficult to manage.  Even within TMG v9 the Master Source List  is so slow as to be unusable.

 

I tried exporting the GEDCOM for one person with two addresses that needed fixing from TMG v9.  I merged that into FTM 2014, specifiying the records to prefer, then did the sync to Ancestry.  That process retained the links to source images in the Ancestry tree.  The difficult part of the task (for me) was getting from the Master Place list in TMG v9 to the person whose records were affected.  I ended up takeing the place name as it appeared in the TMG v9 Master Place list and looking it up in FTM, whcih shows exactly which people are involved.  I took that information back to TMG v9 to do the GEDCOM export.  There's got to be a way to do that within TMG v9, but I didn't find it.

 

I suppose I need to see what this looks like in Second Site before I repeat the import process N times.

 

Dave






0 user(s) are reading this topic

0 members, 0 guests, 0 anonymous users