Question Periodic GEDCOM export

On my web server, I have a daily backup of the database and of the directory containing all the webtrees app files.

But I wold also like to be able to specify an automatic periodic export of a GEDCOM file to the …/data directory.

Can there be an option in control panel to set up a periodic job?

Or is there a command that could be put in a cron job? Would not want it to be anything that could be called remotely. Maybe a wget or curl command that includes a password inside a script that is not available to the web server?

WGroleau wrote:

But I wold also like to be able to specify an automatic periodic export of a GEDCOM file to the …/data directory.

Can there be an option in control panel to set up a periodic job?

Or is there a command that could be put in a cron job? Would not want it to be anything that could be called remotely. Maybe a wget or curl command that includes a password inside a script that is not available to the web server?

For automatic downloads of GEDCOM files from webtrees, I created a small custom module: DownloadGedcomWithURL

I am using it with webtrees on my NAS server and automatically download GEDCOM files to my PC with a python script. Afterwards, I run some further GEDCOM post-processing on it in order to eliminate some deficencies, before I save the final files as backups.

If you can run some kind of script on your web server, it should be possible to use the module for your backup purposes. You would need to direct the http download to some location of your choice.

Important note: Installing this module will enable everyone, who can reach the webtrees URL, to download the GEDCOM files from webtrees. Therefore, you would need to apply additional access restrictions, e.g. restrict download to certain IP addresses only. Maybe, you can restrict the access to the specific URL with a .htaccess file. It would also be possible to add 2 more lines of code to the module and check the ip address of the request.

Jefferson49 wrote:

For automatic downloads of GEDCOM files from webtrees, I created a small custom module: DownloadGedcomWithURL

I am using it with webtrees on my NAS server and automatically download GEDCOM files to my PC with a python script. Afterwards, I run some further GEDCOM post-processing on it in order to eliminate some deficencies, before I save the final files as backups.

If you can run some kind of script on your web server, it should be possible to use the module for your backup purposes. You would need to direct the http download to some location of your choice.

Important note: Installing this module will enable everyone, who can reach the webtrees URL, to download the GEDCOM files from webtrees. Therefore, you would need to apply additional access restrictions, e.g. restrict download to certain IP addresses only. Maybe, you can restrict the access to the specific URL with a .htaccess file. It would also be possible to add 2 more lines of code to the module and check the ip address of the request.

I'll take a look at this. The last paragraph makes it unwelcome to me, even if it implements the privacy settings. IP address isn't sufficient remotely, as that changes with DHCP. Not sure whether using the local address of the host would work, since many other people have accounts on the same web server.

Currently, I just go to the control panel and export every time (almost) I am logged in.

WGroleau wrote:

The last paragraph makes it unwelcome to me, even if it implements the privacy settings. IP address isn't sufficient remotely, as that changes with DHCP. Not sure whether using the local address of the host would work, since many other people have accounts on the same web server.

Well, I understand that IP address check is limited somehow.

Maybe, you can do some basic tests on the concept if it works with your idea of scripts and so on. If yes, I can add some lines of code to the module with a security feature. Maybe, I could introduce an additional parameter "security key", which could be read from a file and checked by the module.

As a quick and dirty solution, you could use some of the existing parameters (e.g. line_endings) and just substitute 3 lines of code in line 199-201 of the module code :

If you use this with https, it should already be an acceptable solution.

WGroleau wrote:

Jefferson49 wrote:

For automatic downloads of GEDCOM files from webtrees, I created a small custom module: DownloadGedcomWithURL

I am using it with webtrees on my NAS server and automatically download GEDCOM files to my PC with a python script. Afterwards, I run some further GEDCOM post-processing on it in order to eliminate some deficencies, before I save the final files as backups.

If you can run some kind of script on your web server, it should be possible to use the module for your backup purposes. You would need to direct the http download to some location of your choice.

Important note: Installing this module will enable everyone, who can reach the webtrees URL, to download the GEDCOM files from webtrees. Therefore, you would need to apply additional access restrictions, e.g. restrict download to certain IP addresses only. Maybe, you can restrict the access to the specific URL with a .htaccess file. It would also be possible to add 2 more lines of code to the module and check the ip address of the request.

I'll take a look at this. The last paragraph makes it unwelcome to me, even if it implements the privacy settings. IP address isn't sufficient remotely, as that changes with DHCP. Not sure whether using the local address of the host would work, since many other people have accounts on the same web server.

Currently, I just go to the control panel and export every time (almost) I am logged in.

I agree to your privacy concerns and don't like this solution too. I second your request for a possibility to use a cronjob.

I have attached two scripts that work together from the command line (and should work in cron) in my web server. Use at your own risk.

Your server must have /bin/bash — if not, other shells should work but I did not test. The shell you want must be invoked on the first line of the script.

You'll have to edit several details—see the comments within the two scripts.

ALSO: due to a typo, the attached script will leave out some records! To fix, change the column name for sources from m_file to s_file

Unpack the scripts into a directory that the web server cannot access but from which the script can access the database. Further protect them with (this step is important, since your DB password is in the script!)

Make all your edits and test
if the test passes, put it in your crontab.

If you don't own the server, check your provider's policies on cron jobs.

Note: in the mysql command, there must NOT be a space between -p and the password.

WGroleau wrote:

I have attached two scripts that work together from the command line (and should work in cron) in my web server. Use at your own risk.

Your server must have /bin/bash — if not, other shells should work but I did not test. The shell you want must be invoked on the first line of the script.

You'll have to edit several details—see the comments within the two scripts.

Unpack the scripts into a directory that the web server cannot access but from which the script can access the database. Further protect them with

Code:

chmod 700 (path)/gedcom.bash chmod 600 (path)/export.sql

(this step is important, since your DB password is in the script!)

Make all your edits and test
if the test passes, put it in your crontab.

If you don't own the server, check your provider's policies on cron jobs.

Note: in the mysql command, there must NOT be a space between -p and the password.

I deeply appreciate your script as my coding-abilities are limited to VB from the eighties, modern SQL und Excel-VBA und just startet to learn PHP.

All the best wishes for 2023!

Since it was discussed several times and makes sense in general, I created a new version of the DownloadGedcomWithURL custom module , which checks the authorization for downloads. An access key was added to the URL. Access to the download is only allowed if the provided access key in the URL is identical to a secret key in the webtrees database (module preferences/settings).

Maybe, this can also be the base for a second or enhanced cronjob to create and backup GEDCOM files.

WGroleau wrote:

I have attached two scripts that work together from the command line (and should work in cron) in my web server. Use at your own risk

ALSO: due to a typo, the attached script will leave out some records! To fix, change the column name for sources from m_file to s_file

perfect....
this idea can also be taken to create shell-scripts with curl or wget to research Find a Grave or others

WGroleau wrote:

Unpack the scripts into a directory that the web server cannot access but from which the script can access the database. Further protect them with

Code:

chmod 700 (path)/gedcom.bash chmod 600 (path)/export.sql

(this step is important, since your DB password is in the script!)

ALSO, recommend you put both scripts in a place the web server cannot access.  Some webserver configurations serve files as the owner of the files.  In the configuration, the permissions I stated would allow others to access the files in a web browser.

There is a custom module available for this function: github.com/Jefferson49/DownloadGedcomWithURL (there is a list of all custom modules in the German webtrees manual ).

hermann wrote:

There is a custom module available for this function: github.com/Jefferson49/DownloadGedcomWithURL (there is a list of all custom modules in the German webtrees manual ).

Yes, Jefferson49 mentioned it some time ago, but also explained why it would need additional scripting. So it seemed easier to me to make my own standalone script.

WGroleau wrote:

hermann wrote:

There is a custom module available for this function: github.com/Jefferson49/DownloadGedcomWithURL (there is a list of all custom modules in the German webtrees manual ).

Yes, Jefferson49 mentioned it some time ago, but also explained why it would need additional scripting. So it seemed easier to me to make my own standalone script.

Yes indeed, if you want to trigger an export with DownloadGedcomWithURL by using a cron job, a script is needed to be called by the cron job. However, the script might only contain a single wget command, e.g.:

DownloadGedcomWithURL also supports some additional features:

• The full set of options, which can be chosen for GEDCOM exports in the webtrees control panel, are also available in DownloadGedcomWithURL. Therefore, options like privacy, encoding, endings, export format can be selected for each download.
• Default export settings can be defined in the module settings in order to keep the download URLs more simple.
• A timestamp can be added as postfix or prefix to the file name.
• An option for a GEDCOM 7 export ist included as a beta version.
• All versions from v3.0.0 use an authorization key. Therefore, the authorization is much more secure than in the original module (which was only suitable for private networks)

Jefferson49 wrote:

WGroleau wrote:

hermann wrote:

There is a custom module available for this function: github.com/Jefferson49/DownloadGedcomWithURL (there is a list of all custom modules in the German webtrees manual ).

Yes, Jefferson49 mentioned it some time ago, but also explained why it would need additional scripting. So it seemed easier to me to make my own standalone script.

Yes indeed, if you want to trigger an export with DownloadGedcomWithURL by using a cron job, a script is needed to be called by the cron job. However, the script might only contain a single wget command, e.g.:

Code:

wget --timeout=60 --tries=1 "http://MY_URL/webtrees/index.php?route=/webtrees/DownloadGedcomWithURL&tree=tree1&file=export&time_stamp=postfix&privacy=gedadmin&action=save&key=hYHBiZM9odh7098ahd" &

DownloadGedcomWithURL also supports some additional features:

• The full set of options, which can be chosen for GEDCOM exports in the webtrees control panel, are also available in DownloadGedcomWithURL. Therefore, options like privacy, encoding, endings, export format can be selected for each download.
• Default export settings can be defined in the module settings in order to keep the download URLs more simple.
• A timestamp can be added as postfix or prefix to the file name.
• An option for a GEDCOM 7 export ist included as a beta version.
• All versions from v3.0.0 use an authorization key. Therefore, the authorization is much more secure than in the original module (which was only suitable for private networks)

Sounds pretty good. 

I may have misinterpreted the mention of python as requiring python. After so many years of retirement, I have a lot less motivation to learn a new programming language, and it wasn’t clear that it could only be a one-liner.

And then there’s the “additional post-processing” which probably is not necessary for most users.  In my case, though, I have additional post-processing which puts each level zero record in its own separate file.