This Help forum is for issues relates to the latest release (1.7.9). For issues related to beta or github version please use their own Help forum.
Before asking for help please read "How to request help" by clicking on that tab above here."

TOPIC: Import process hung-up

Import process hung-up 1 week 14 hours ago #1

  • ricklach
  • ricklach's Avatar
  • Offline
  • New
  • Posts: 11
I had to stop my import process because I had not converted my gedcom file to utf-8. I changed the file to utf-8 and then tried to import the file. webtrees import is hung-up on the import page and only provides the message "calculating." I have tried navigating to another page, closing the browser, restarting the server but nothing stops the application from going back to import page. How do I stop it so I can start over again?
The administrator has disabled public write access.

Import process hung-up 1 week 14 hours ago #2

  • fisharebest
  • fisharebest's Avatar
  • Online
  • Administrator
  • Posts: 10299
Delete all the rows in the table wt_gedcom_chunk
Greg Roach - This email address is being protected from spambots. You need JavaScript enabled to view it. - fisharebest.webtrees.net
The administrator has disabled public write access.

Import process hung-up 1 week 13 hours ago #3

  • ricklach
  • ricklach's Avatar
  • Offline
  • New
  • Posts: 11
Ok, did that. Retried the process and still stuck on the "calculating" hangup, nothing is being added to the chunk table other than the same 1118 rows all with nothing in them.
The administrator has disabled public write access.

Import process hung-up 1 week 10 hours ago #4

  • fisharebest
  • fisharebest's Avatar
  • Online
  • Administrator
  • Posts: 10299
> the same 1118 rows all with nothing in them.

All rows completely empty. Very strange. I've not seen this before.

You said you "changed the file to utf-8". Can you tell me exactly what you did?
Greg Roach - This email address is being protected from spambots. You need JavaScript enabled to view it. - fisharebest.webtrees.net
The administrator has disabled public write access.

Import process hung-up 1 week 9 hours ago #5

  • ricklach
  • ricklach's Avatar
  • Offline
  • New
  • Posts: 11
To change the file from ansi to utf-8 you open the file in notepad, change the gedcom file so that it reads UTF-8 instead of ansi. select save-as and in the bottom of the page there is the option to save the file as UTF-8, select that and save the file. This has worked for me for years. I have retried your suggestion several times by changing the file name and the most recent time it loaded absolutely nothing into the chunk table after several hours of waiting. My file is quite large (almost 100K people) and I have not experienced this behaviour in the past. I am re-running my gedcom export file as I type and will complete the process once I have a new file to see if it makes any difference.
The administrator has disabled public write access.

Import process hung-up 1 week 9 hours ago #6

  • fisharebest
  • fisharebest's Avatar
  • Online
  • Administrator
  • Posts: 10299
The "calculating" phase includes deleting all the old data. On slow servers, deleting all the data can take longer than the server timeout.

To eliminate this, you could try importing into a new/empty tree.

But your comment that "rows all with nothing in them." is puzzling. Off the top of my head, the only reason I can think that this would be empty is if the data is not valid UTF8, and mysql (or PHP) is ignoring it.

Just to be clear, can you clarify what you mean by this, and how you tested it.

> it loaded absolutely nothing into the chunk table after several hours of waiting

Most likely, there was either an error or timeout. So, if you've waited for the timeout period (lowest of webserver and PHP timeouts - typically 60 seconds), then there is no point waiting any longer.
Greg Roach - This email address is being protected from spambots. You need JavaScript enabled to view it. - fisharebest.webtrees.net
The administrator has disabled public write access.

Import process hung-up 1 week 8 hours ago #7

  • ricklach
  • ricklach's Avatar
  • Offline
  • New
  • Posts: 11
Re; Just to be clear, can you clarify what you mean by this, and how you tested it. I tried loading the gedcom file and then while it was supposedly loading went to the mysql database and had a look at what was being loaded - if that is what you mean? In the past I have loaded this same sized gedcom file without any problems and I still have a site that is working with much older data. Right now I want to load the data into a webserver/database on my local machine to test it out before going live. I am going to try a sample gedcom file of smaller size to see if it will load. Hope this helps. I have some programming experience so if you want to ask a more technical question I will do my best to find the answer.

As a follow-on I have tried to import a gedcom file that is only 21mb and it appears to be working. So my original file of 72mb could be the problem - you see size does matter. I think the setting for this is in the apache config setting and I will try to up the value unless you have any suggestions in the meantime.
Last Edit: 1 week 8 hours ago by ricklach. Reason: new data
The administrator has disabled public write access.

Import process hung-up 1 week 8 hours ago #8

  • fisharebest
  • fisharebest's Avatar
  • Online
  • Administrator
  • Posts: 10299
> while it was supposedly loading went to the mysql database and had a look at what was being loaded

The import scripts are wrapped in a transaction, so you shouldn't be able to see updated rows until it commits - and if there's an error then it won't commit. MySQL won't rollback until the transaction times out.

> I have some programming experience so if you want to ask a more technical question I will do my best to find the answer.

The import happens using lots of small AJAX/XHR requests. Each one is supposed to be small enough to run without exceeding CPU/memory limits.

One of these requests presumably failed, and there may be an error message concealed within it. Use the developer tools in firefox or chrome to view the responses and see if there are any errors.
Greg Roach - This email address is being protected from spambots. You need JavaScript enabled to view it. - fisharebest.webtrees.net
The administrator has disabled public write access.
Do you need a web hosting solution for your webtrees site?
If you prefer a host that specialises in webtrees, the following page lists some suppliers able to provide one for you: 

Import process hung-up 1 week 3 hours ago #9

  • ricklach
  • ricklach's Avatar
  • Offline
  • New
  • Posts: 11
This particular error message is repeated several times (the exact same error): Source map error: TypeError: NetworkError when attempting to fetch resource. Resource URL: moz-extension://5027d2e8-9b32-4a0d-8dad-527410e3b9f8/scripts/common.js Source Map URL: common.js.map[Learn More]

Is this related to a server time out problem perhaps? I imported a smaller file as I mentioned and it worked just fine.
The administrator has disabled public write access.

Import process hung-up 6 days 8 hours ago #10

  • ricklach
  • ricklach's Avatar
  • Offline
  • New
  • Posts: 11
I am not certain that I am making progress but here is the latest error:

( ! ) Fatal error: Allowed memory size of 134217728 bytes exhausted (tried to allocate 69632 bytes) in C:\wamp64\www\webtrees\app\Statement.php on line 82
Call Stack
# Time Memory Function Location
1 0.0014 403968 {main}( ) ...\import.php:0
2 0.1217 806984 Fisharebest\Webtrees\Statement->execute( ) ...\import.php:48
3 0.1217 807760 execute ( ) ...\Statement.php:82

I have set my php memory limit to 256M but that doesn't seem to make a difference. I then selected a file that was about 2/3rds the size and it worked beautifully. That file was about 48mb and the other one is about 72mb.
Last Edit: 6 days 8 hours ago by ricklach.
The administrator has disabled public write access.

Import process hung-up 6 days ago #11

  • fisharebest
  • fisharebest's Avatar
  • Online
  • Administrator
  • Posts: 10299
Your error is in import.php line 48:

Database::prepare("SELECT * FROM `##gedcom_chunk` WHERE gedcom_id=? FOR UPDATE")->execute(array($gedcom_id));


This line is not strictly required. It is simply creating a database lock to prevent two scripts running at the same time. For example, if you opened the manage trees page in two windows, then both would try to run the import.

You could try commenting out or removing this line
Greg Roach - This email address is being protected from spambots. You need JavaScript enabled to view it. - fisharebest.webtrees.net
The administrator has disabled public write access.

Import process hung-up 5 days 15 hours ago #12

  • ricklach
  • ricklach's Avatar
  • Offline
  • New
  • Posts: 11
Thank's Greg for your patience. I commented out the line and when I ran the import I received another fatal error related to a time-out. I doubled the max execution time in php.ini and the whole import process, once I got past these hurdles only took a few minutes. The file was 1.2 M lines long and I guess just needed the time to be read and processed. Thanks again for your help. It is too bad that we can't export to XML instead of a gedcom file which has too many limitations for genealogy programs.
The administrator has disabled public write access.
Powered by Kunena Forum