Before asking for help please read "Requesting Help and Suggestions" by clicking on that tab above here.
  • Page:
  • 1

TOPIC:

Error with Gedcom Export 6 days 3 hours ago #1

  • StuartG
  • StuartG's Avatar Topic Author
  • Offline
  • Junior Member
  • Junior Member
  • Posts: 150
On exporting a Gedcom file today I noticed that it was missing the final O TRLR line. I have now tried it several times and this line was missing each time. Each time I received the following message - "The family tree has been exported to mncp.ged." No error messages. All files prior to today have exported with the 0 TRLR line included.

4 May : 2,680,780 lines
19 May : 2,685,387 lines
21 May : 2,686,796 lines
Today : 2,727,153 lines - missing 0 TRLR line
Today : 2,726,840 lines - missing 0 TRLR line

From the above data, it would seem that some hundreds of lines could be missing. As well as the most important one!

This is a critical issue for me and any help would be very much appreciated. Thanks.
Stuart
webtrees 2.0.16
⚶ Vesta Modules
PHP 7.3.7
Mysqli

Please Log in or Create an account to join the conversation.

Error with Gedcom Export 5 days 20 hours ago #2

  • fisharebest
  • fisharebest's Avatar
  • Offline
  • Administrator
  • Administrator
  • Posts: 14861
Do you have any errors in your webserver (not webtrees) error log? In particular, any timeout errors.
Greg Roach - This email address is being protected from spambots. You need JavaScript enabled to view it. - fisharebest.webtrees.net

Please Log in or Create an account to join the conversation.

Error with Gedcom Export 5 days 16 hours ago #3

  • StuartG
  • StuartG's Avatar Topic Author
  • Offline
  • Junior Member
  • Junior Member
  • Posts: 150
Error logging on the webserver was established only a short time ago. Since then I have exported the Gedcom a number of times to the server and to a linked computer. The file to the computer has to be via a zip file, as by not selecting this, the download fails to send anything. No error message. The Export a GEDCOM file page is simply refreshed. Although one browser creates a 0 byte file.

Each file exported were missing lines. Each time there were no errors logged on the webserver either for PHP or Apache. Webtrees simply advised that all exports were completed.

The php.ini has max_execution_time set for 1800. The exported files were all completed in a matter of seconds.

The exported files were about 400 lines longer than the zip files and both had been truncated mid line.

Any advice or assistance will be most welcome.
Stuart
webtrees 2.0.16
⚶ Vesta Modules
PHP 7.3.7
Mysqli

Please Log in or Create an account to join the conversation.

Last edit: by StuartG.

Error with Gedcom Export 5 days 15 hours ago #4

  • Sir Peter
  • Sir Peter's Avatar
  • Offline
  • Junior Member
  • Junior Member
  • Posts: 231
How big in bytes are those truncated files? Are they close to any php limits which you can find in the webtrees Control Panel under Server Information, e. g. post_max_size, upload_max_filesize. Apache may have its own limits as well.
Peter

Please Log in or Create an account to join the conversation.

Error with Gedcom Export 5 days 6 hours ago #5

  • StuartG
  • StuartG's Avatar Topic Author
  • Offline
  • Junior Member
  • Junior Member
  • Posts: 150
Thank you for your response. Here is some relevant data:

Gedcom zipped file downloaded today is 6,280,180 bytes
Gedcom unzipped file downloaded today is 35,217,408 bytes
Gedcom file saved to server today is 35,221,504 bytes
Gedcom file saved to server yesterday is 42,856,448 bytes
Gedcom file saved to server on 4 May is 62,940,227 bytes

From the server information:
post_max_size is set to 2047M
upload_max_filesize is set to 2047M

At the moment, I am unable to assess any limits that Apache may have. I will need to investigate how I can do this.

I am happy to provide any further data that I am able to and I welcome any further assistance. Thank you.
Stuart
webtrees 2.0.16
⚶ Vesta Modules
PHP 7.3.7
Mysqli

Please Log in or Create an account to join the conversation.

Error with Gedcom Export 4 days 20 hours ago #6

  • Sir Peter
  • Sir Peter's Avatar
  • Offline
  • Junior Member
  • Junior Member
  • Posts: 231
There might be other php parameters that are relevant for exporting files, e. g. the memory_limit - or Apache parameters.

Do you export to the server or to your computer? In case you export to your computer, which switches do you activate?
Do you have sufficient diskspace available?
Do all the truncated files end with the same GEDCOM line? That could point to some unusual/invisible character which sometimes enter your GEDCOM data with copy and paste.

Sorry, I am just poking around.
Peter

Please Log in or Create an account to join the conversation.

Error with Gedcom Export 4 days 19 hours ago #7

  • fisharebest
  • fisharebest's Avatar
  • Offline
  • Administrator
  • Administrator
  • Posts: 14861
> The file to the computer has to be via a zip file, as by not selecting this, the download fails to send anything. No error message. The Export a GEDCOM file page is simply refreshed.

Can you expand on this a little?

You are saying it works fine when you select a ZIP file?

Strange - as this runs the same export process - but to a temporary file first, which is then added to the ZIP.
If there are resources to create the temporary file (and presumably no bugs), then there should be resources to create the download response.

Perhaps it is a timeout issue - and sucess/failure depends on the server load which will be different each time you run the export.

Is it a busy server? What is your PHP execution limit?
Greg Roach - This email address is being protected from spambots. You need JavaScript enabled to view it. - fisharebest.webtrees.net

Please Log in or Create an account to join the conversation.

Error with Gedcom Export 4 days 5 hours ago #8

  • StuartG
  • StuartG's Avatar Topic Author
  • Offline
  • Junior Member
  • Junior Member
  • Posts: 150
Thank you both very much for your interest and assistance.

My son (Andrew) has again stepped up (as usual) to assist me and he thinks he has resolved the major issue. It relates to the size of the temp file/folder on my QNAP server. It was not big enough to contain the created Gedcom file. There seems to be other minor issues relating to the way webtrees exports the files. There are also some other issues relating to how the QNAP handles these files.

QNAP internally sets a temp file/folder size of 64 Megs. Changing this is not straight forward. It has now been changed to 128 Megs and all the Gedcom exports are thankfully currently functioning. This would be OK, except on the next reboot/restart of the QNAP, it resets this back to 64 Megs. Andrew has written a script that runs following a reboot/restart of the QNAP to set it back to 128 Megs. Hopefully this will continue to be of value into the future.

The Gedcom file created by the Export function in the temp file/folder is deleted from there once the file has been exported to either the server or a file on my computer. However, if the zip function is used, the zip file is not deleted following the export and therefore the size of the temp file/folder continues to grow.

We carried out a series of zip file downloads and the size of the zip file continued to diminish each time. There were no error messages displayed until we got to exporting a zip file which conmtained 0 bytes. Then an error was displayed.

Andrew will probably want to add some more technical comments for the benefit of other users. For me the core issue has now be resolved.

Thanks again for your support.
Stuart
webtrees 2.0.16
⚶ Vesta Modules
PHP 7.3.7
Mysqli

Please Log in or Create an account to join the conversation.

Do you need a web hosting solution for your webtrees site?
If you prefer a host that specialises in webtrees, the following page lists some suppliers able to provide one for you: 

Error with Gedcom Export 4 days 5 hours ago #9

  • StuartG
  • StuartG's Avatar Topic Author
  • Offline
  • Junior Member
  • Junior Member
  • Posts: 150
Perhaps I should have added that my Gedcom file is now over 62 Megs.
Stuart
webtrees 2.0.16
⚶ Vesta Modules
PHP 7.3.7
Mysqli

Please Log in or Create an account to join the conversation.

Error with Gedcom Export 3 days 23 hours ago #10

  • Sir Peter
  • Sir Peter's Avatar
  • Offline
  • Junior Member
  • Junior Member
  • Posts: 231

... if the zip function is used, the zip file is not deleted following the export and therefore the size of the temp file/folder continues to grow.

We carried out a series of zip file downloads and the size of the zip file continued to diminish each time. There were no error messages displayed until we got to exporting a zip file which conmtained 0 bytes. Then an error was displayed.


@Stuart, I am glad you were able to resolve the issue.
@Greg, can these two situations be handled better by webtrees? Should a respective issue be created on Github?
Peter

Please Log in or Create an account to join the conversation.

Error with Gedcom Export 3 days 14 hours ago #11

  • fisharebest
  • fisharebest's Avatar
  • Offline
  • Administrator
  • Administrator
  • Posts: 14861
> if the zip function is used, the zip file is not deleted following the export

We create temporary files using the PHP function tmpfile()
The documentation - www.php.net/tmpfile - says that this file is deleted automatically at the end of the script.

As far as I can tell, this happens. No temp files are left on my system.

Where do you find these files? In the system /tmp folder? In /data/tmp? Somewhere else?
Greg Roach - This email address is being protected from spambots. You need JavaScript enabled to view it. - fisharebest.webtrees.net

Please Log in or Create an account to join the conversation.

Error with Gedcom Export 3 days 14 hours ago #12

@greg

In the documentation it also says:
Caution If the script terminates unexpectedly, the temporary file may not be deleted.

It's more likely, that this might be the problem @StuartG is having. I stopped using my QNAP-device as a internal development-device because I had too many timing-issues on it.

Please Log in or Create an account to join the conversation.

Last edit: by Lars1963.

Error with Gedcom Export 3 days 3 hours ago #13

  • andrewg_oz
  • andrewg_oz's Avatar
  • Offline
  • New Member
  • New Member
  • Posts: 10
My Dad uses a Qnap TS-473 NAS to host his webtrees site.

As he described above, attempts to export the GEDCOM file resulted in files missing the final trailing "0 TRLR" record. This was exporting using "to a file on the server" and "to a file on your computer", but only when using the compression option. Exporting an uncompressed file "to your computer" would fail with no file data at all, and no error message.

By checking against a subsequent successful export, I found that the incorrectly exported files were formed of an initial successful export of data, up to a point. At that point a partially written record would stop mid-line and a new record from much further on would begin. Then after one or more lines of that record the export would again stop mid-line, this time for good. For example, part-way through writing an INDI record, a NOTE record might start, then the file end before that was completely written.

This was the same for the files saved to the server, and the files compressed in the ZIP.

While troubleshooting this I checked Apache and PHP error logs and found nothing. However, I did notice that over time the size of the exported files was decreasing. After about 30 exports the size reached zero and I finally got an error message out of PHP (shown in the browser). This error message indicated the /tmp storage was full.

I checked the /tmp folder and found a great number of "phpXXXXXX" files. These all turned out to be the ZIP files generated for the "to a compressed file on your computer" exports.

It turns out that QNAP create their /tmp storage using a RAM disk of a fixed size. 64MB in the case of the TS-473. Other models might only have 32MB. In my research for this I found mention of web hosts who limit their customers to 4MB of /tmp storage! In my Dad's case, his 62MB GEDCOM file basically filled the available /tmp size and it didn't take much extra /tmp usage to run out of space. Deleting the temporary PHP files reduced the /tmp usage to about 500KB, and restored the ability to export his GEDCOM. However, the limit of 64MB is a problem given how big his GEDCOM files are becoming.

To assist other Qnap users, I found instructions to resize the Qnap /tmp storage at: forum.qnap.com/viewtopic.php?f=25&t=44541&start=15 and more instructions to make the change persistent at: wiki.qnap.com/wiki/Running_Your_Own_Application_at_Startup . Other Qnap owners using webtrees may find those links useful.

For now I have increased my Dad's Qnap /tmp allocation to 128MB. That should be sufficient for a good while.

I can also say that even with this increased storage, the "export to a compressed file on your computer" option is still leaving temporary files behind. I know that Qnap are using "php-fpm", and it is possible that it is keeping PHP processes around to handle multiple requests, delaying the deletion of temporary files. It's also possible that the pool of php processes is ungracefully terminated, preventing deletion of temporary files.

While investigating all this, I became concerned that webtrees might use the system /tmp during GEDCOM upload, but that does not appear to be the case. I was able to successfully upload a 62MB GEDCOM on a test site where the /tmp storage was only 32MB.

My final concern was what would happen if I uploaded a truncated GEDCOM file to webtrees. In that case I found that the file was imported into webtrees without any errors or indications of any problems. This, combined with the ability of webtrees to export truncated GEDCOM files without warning, could result in silent data loss.

For webtrees, this raises several issues:

1. webtrees is exporting malformed GEDCOM files without reporting any problems.
2. webtrees is accepting truncated GEDCOM files without reporting any problems.
3. Orphan temporary ZIP files filling the system /tmp storage is a problem.

The issues with webtrees silently generating and accepting truncated GEDCOM files is obvious. The orphan /tmp files is less obvious. Perhaps webtrees should manage its own temporary files in its own data folder? That way they won't be subject to arbitrary system limits, and can likely be manually deleted by webtrees admins as necessary.

Should I create github issues for these?

Please Log in or Create an account to join the conversation.

Error with Gedcom Export 1 day 23 hours ago #14

  • fisharebest
  • fisharebest's Avatar
  • Offline
  • Administrator
  • Administrator
  • Posts: 14861
Clearing out /tmp is the responsibility of your operating system. Most linux based systems have a "house-keeping service" to do this. Typically this deletes files older than 24 hours.

Since the start of the project, we have been aware of problems exporting large GEDCOM files on servers with limited resources. This is why we have two export options.

* Export to the server does minimal processing on the data, and simply dumps it from the database to the disk, one record at a time. Since DB queries don't count against PHP time limits, and since there is no temporary storage needed, we can export files larger than the available memory.

* Export to your computer does additional processing, and requires temporary storage. e.g. we write to a memory buffer (which your server may/may not convert to a file buffer). This buffer can be sent directly or added to a zip file.

The first option should work on pretty much any server. You can then collect your files using SFTP.

> Perhaps webtrees should manage its own temporary files in its own data folder?

You can do this yourself. There is an example module included with 2.0 which shows how to set your own temp folder. Choose one in a real filesystem (not a ramdisk) and then add a cron-job to delete old files.

> webtrees is exporting malformed GEDCOM files without reporting any problems.

Looking at the code, we are not checking the return value for fwrite(). i.e. we write data and assume it was written.

Easy to fix. Can you create an issue for this.

> webtrees is accepting truncated GEDCOM files without reporting any problems.

I guess this ought to be flagged somehow. Can you create an issue for this.
Greg Roach - This email address is being protected from spambots. You need JavaScript enabled to view it. - fisharebest.webtrees.net

Please Log in or Create an account to join the conversation.

Error with Gedcom Export 1 day 17 hours ago #15

  • andrewg_oz
  • andrewg_oz's Avatar
  • Offline
  • New Member
  • New Member
  • Posts: 10

> webtrees is exporting malformed GEDCOM files without reporting any problems.

Looking at the code, we are not checking the return value for fwrite(). i.e. we write data and assume it was written.

Easy to fix. Can you create an issue for this.


Done: github.com/fisharebest/webtrees/issues/4043

> webtrees is accepting truncated GEDCOM files without reporting any problems.

I guess this ought to be flagged somehow. Can you create an issue for this.


and done: github.com/fisharebest/webtrees/issues/4044

Regarding the use of tmpfile(), would using the unlink() 'trick' be better?:
$tmpfname = tempnam(sys_get_temp_dir(), 'FOO');
$temp = fopen($tmpfname, "w");
if ($temp) {
  unlink($tmpfname);
  // .. do stuff ..
  fclose($temp);
}

I understand what you're saying about systems house-keeping their /tmp folder, but that isn't happening on the Qnap and there's no guarantee it will actually happen in general. It would be better if webtrees cleaned up after itself as much as possible. I can also confirm that /tmp running out of space on the Qnap isn't the cause of the leftover zips. I've both increased the Qnap /tmp to 128MB, but also exported a single-person tree to a Gedcom and the 500-byte ZIP is still left behind...

Please Log in or Create an account to join the conversation.

  • Page:
  • 1
Powered by Kunena Forum