Random Data Loss After Data Set Merge

[Kevin Schultz 0]

I reported a problem via email where after a merge of 2 data sets, much data was lost. Names, married names, and many dates. Utilities (Reindex, optimize and check file integrity) had been run prior to the merge with no problems. After the merge and running the same utilities, 4365 problems were reported and "fixed". That's when the data was lost. Tech support confirmed the problem with my file and was told that the problem was forwarded to developers to determine the cause and try to repair the data sets. The interesting piece of information was that all the data appeared to be intact until the utilities were run after the merge.

 

I've never had any further questions or data requested. Last email I received was dated 9-30-05 with a promise to check with developers and get back to me. I've inquired twice since then (mid and late October) for any update and have had no further response.

 

Is there any hope for repair?

[Jim Byram 0]

Kevin,

 

You should have been advised to revert to a backup with your initial report. And to hold off merging until the source of the issue could be pinned down.

 

You should also update your installation to v6.07 if you have not done so.

 

I've replicated the data set merge issue with your project and am looking at it. One of the programmers is also looking and the issue appears to be damage in your original project but it's taking a while to understand exactly what is happening.

 

Jim

Edited November 23, 2005 by Jim Byram

[Kevin Schultz 0]

Jim,

 

Thanks for the response. I have updated my installation to 6.07. I have not added anything to that project since the problem was discovered. I had spent quite a bit of time cleaning up one of the data sets before the merge and have several backups of the previous few days. I have not tried to see how far back the problem exists since I'm hoping for a fix. (I probably had ~60 hours of work in the last week before the merge.)

 

I will check my other project to see if a similar problem exists.

 

Kevin

[Jim Byram 0]

Kevin,

 

The problem appears to be limited to the E-table (witness table). The problem is that an EPER has records in the wrong data set (DSID). All of the individuals with problems appear to be in data set #2.

 

Your example Anna Maria Schmitz (2:3674) has two records in the E-table... one with DSID 1 and one with DSID 2.

 

Ewen Castor (2:1) has three records in the E-table, all DSID 1.

 

Persons 2:3696, 2:3702, 2:3707, 2:3713 each have one E-table record in DSID 1 and another in DSID 2. These four person records are grouped together. Maybe you can see some sort of pattern?

 

We're trying to make sure that all the records in the $-table (person) have the correct DSID. From looking at the records towards the end of the table, you've been entering new people into both data sets.

 

We've never seen the E-table issue. Have you been doing any particular sort of unusual data manipulations that might be tied to this?

 

If the issue is only an E-table issue, these records can be cross-checked against the $-table and corrected. An addition will be made to VFI to check for this sort of data issue and make corrections.

 

Update: The same DSID issue is occurring in the NPV-table and this explains the name loss with merge.

 

Update: Code has been written to repair this and I've tested it on your data. I can now repair your project and sent you an email requesting your current project to fix it.

 

Jim

Edited November 25, 2005 by Jim Byram

[Kevin Schultz 0]

Jim,

 

Persons 2:3696, 2:3702, 2:3707, 2:3713 each have one E-table record in DSID 1 and another in DSID 2. These four person records are grouped together. Maybe you can see some sort of pattern?

 

Other than they probably all are the remaining record of a duplicate merge in the #2 data set. They were not all done at the same time and these people never existed in the #1 dataset. In fact there is very little overlap between the #1 and #2 data sets. Once I had a successful merge, I planned to eliminate the few remaining dulplicates that would link the 2 data sets.

 

For info, the #2 set was imported as a GEDCOM from another person and that file resulted from merging at least 3 different data sources. There were many duplicates that had to be reconciled and in some cases 3 records of the same person. There were also many (100-200) unlinked people that were apparently "leftovers" from his previous merge of the three sets.

 

Kevin

[Kevin Schultz 0]

To those who may have followed this thread, I want to let you know that Jim was able to fix my corrupted project and I am back in business. I completed the merge of the datasets and my data is intact. Thanks Jim.

 

Kevin