Please do NOT post requests for help here. Use the Help forum for that.
  • Page:
  • 1
  • 2

TOPIC: Changes to media handling for webtrees-1.4.0

Changes to media handling for webtrees-1.4.0 4 years 1 month ago #1

  • fisharebest
  • fisharebest's Avatar
  • Offline
  • Administrator
  • Posts: 10290
I plan to fix lots of long-standing media issues for the next release of
webtrees. For most people, these changes should be transparanent and
automatic. Everything will "just work".

But for people who have tweaked their installation, use unusual configuration,
or who simply like to know exactly what is going on, here is the plan.



1) Changes to media-firewall

Previously, webtrees would generate media URLs containing filenames, and
rely on .htaccess to redirect to the media-firewall code.

Now, webtrees generates media URLs containing just the media ID (XREF).
Therefore we no longer need .htaccess files, and we no longer need to
expose the filenames.

This is complete in SVN. This also fixes many "file not found" errors.


2) Too many media folders.

Previously, webtrees looked for media files in three different locations

a) WT_DATA_DIR . $MEDIA_DIRECTORY (in the data directory)
b) WT_ROOT . $MEDIA_DIRECTORY (a.k.a. the "public" media folder)
c) $MEDIA_FIREWALL_ROOTDIR . $MEDIA_DIRECTORY (a.k.a. the "protected" media folder)

This causes lots of complications in the code and confusion for users.
So, I plan to stop supporting (b) and (c).
An upgrade script will move files from (b) and (c) to (a).

This and the other changes means that we can remove the restriction
that $MEDIA_DIRECTORY cannot contain "../" in the path. So, you could
set your media directory to "../../../foo/bar/".
Some people may find this useful.

3) Ambigious folders.

Previously, webtrees adds the $MEDIA_DIRECTORY to the filename when
you upload files.

So, if you upload file "foo.jpeg", webtrees will create a GEDCOM
record containing

1 FILE media/foo.jpeg

BUt, if you import a GEDCOM containing

1 FILE foo.jpeg

It will find the *same* file. The current code needs to check for, and
remove the "media/" before any calculations.

I plan to stop adding this folder. An upgrade sript will remove it from
the GEDCOM by performing a search/replace of "1 FILE media/" with "1 FILE "

With the above changes, we'll now be able to easily find the location
of a file using

WT_DATA_DIR . $MEDIA_DIRECTORY . $filename

Similarly, the thumbnail will be found at

WT_DATA_DIR . $MEDIA_DIRECTORY . 'thumbs/' . $filename


4) Filenames containing absolute paths.


If you use webtrees to display data generated by a desktop application, you
may find that your media files contain a full path. e.g.

1 FILE C:\users\greg\genealogy\pictures\certificates\birth\john-smith.jpeg

We need to ignore the prefix "C:\users\greg\genealogy\pictures\" so that
we can find the file at

WT_DATA_DIR . $MEDIA_DIRECTORY . "certificates\birth\john-smith.jpeg"


The current solution is the "directory levels to keep", which in this
example is 2 ("certificates", "birth").

But this assumes that all your media have the same depth. There is a
long-standing bug report about this. In fact, this is the oldest bug
report that is still open.

So, I plan to replace this with a new setting "media prefix to ignore".
In this example, we'd set it to "C:\users\greg\genealogy\pictures\"

Our file location now becomes

WT_DATA_DIR . $MEDIA_DIRECTORY . (trim $prefix from $filename)


We should also be able to use this setting when creating new media
objects, and/or when downloading GEDCOMs.


The "directory levels to keep" option is also used for a second
purpose - as a limit to the number of user-defined subfolders.
I'm not sure that we need such a limit, so unless someone has
a good reason to keep it, I plan to remove the limit.
Greg Roach - This email address is being protected from spambots. You need JavaScript enabled to view it. - fisharebest.webtrees.net
The administrator has disabled public write access.

Changes to media handling for webtrees-1.4.0 4 years 1 month ago #2

  • davistom
  • davistom's Avatar
  • Offline
  • Junior
  • davistom
  • Posts: 100
Greg:

I am delighted to see that some media handling change plans are in the works. Eliminating the "public" media tree and the "directory levels to keep" parameter/related code are welcome changes, at least to me.

It probably makes me sound too greedy but there are a couple of other media handling issues I would like to see addressed. First, I would like to see the thumbnails subtree eliminated and the thumbs incorporated into the DB media records, which would make it much easier to manage mismatched and orphaned thumbnails in large, multi-level media libraries. Second, I would sure like to see the webtrees media editing and administration features re-worked so that large, multi level media trees could be navigated and accessed via their directory structure, e.g. walking the tree via Ajax calls, rather than the current time consuming construction of an internal representation of the entire media tree (as for example the folder list construction in functions_mediadb.php).

Tom Davis
Live site (fotgp.com) - php 5.4, MySql 5.0.96, webtrees 1.7.9
Desktop - Win 7, Xampp 1.8.1, php 5.4.7, MySql 5.5.27, webtrees 1.7.9
The administrator has disabled public write access.

Changes to media handling for webtrees-1.4.0 4 years 1 month ago #3

  • WGroleau
  • WGroleau's Avatar
  • Offline
  • Gold
  • Posts: 1483
Since media are directly referenced in GEDCOM, this may not be possible, but I would like the media handling to be similar to other modules, so that we foolhardy souls can try replacing it with our own modules.

Mine would store thumbnails AND full images in the DB.
--
Wes Groleau
UniGen.us/
PHP 5.4.20; MySQL 5.1.66; Apache
The administrator has disabled public write access.

Changes to media handling for webtrees-1.4.0 4 years 1 month ago #4

  • makitso
  • makitso's Avatar
  • Offline
  • Administrator
  • Posts: 617
Greg,

Couple of things I noted on my test webtrees SVN

In the admin, clicking on "Upload media files" results in the error:
Uploading media files is not allowed because the media folder is not writable.

Edit: This problem exists if both the /media and /data/media folders coexist. If you delete the /media folder the problem goes away.
In the admin, clicking on Media results in "Permissions Not set" for all image files in the media/ folder.
Edit: This problem exists if both the /media and /data/media folders coexist. If you delete the /media folder the problem goes away.

Correct read/write/execute permissions
Permissions Not Set (666) (media/logo.jpg)
Permissions Not Set (666) (media/house.jpg)
Permissions Not Set (777) (media/thumbs)

If one tries to add a media for an individual AND generate the thumbnail, the base image is uploaded to the media folder but nothing goes into the thumbs folder and the error "Blank name or illegal characters in name" is issueed. If you do the same thing but upload a thumbnail it works correctly.
Edit: This problem was also corrected by deleting the /media folder and setting the gedcom database element MEDIA_FIREWALL_ROOTDIR to the value of media/

I had image files and thumbnails in the folder /media/media and they were not copied to /data/media folder.
Rob
webtrees.net site administrator
www.skatekey.net (version1.7.7)
PHP 7.07, MySQL 5.5.49, Apache 2.2.9,
Hosted at tigertech.net
Last Edit: 4 years 1 month ago by makitso.
The administrator has disabled public write access.

Changes to media handling for webtrees-1.4.0 4 years 1 month ago #5

  • fisharebest
  • fisharebest's Avatar
  • Offline
  • Administrator
  • Posts: 10290
I haven't (yet) changed any code relating to media uploading.

All I have changed is

1) links to media files are now made directly to mediafirewall.php, rather than via a 404 redirect
2) the "media firewall" configuration option has been removed from the settings page - although it still exists and is still used.

So, I'm pretty sure that these are pre-existing bugs.

Also, the fact that the "create thumbnail" logic gives this errorneous message, whatever problems it finds, is well known.
Greg Roach - This email address is being protected from spambots. You need JavaScript enabled to view it. - fisharebest.webtrees.net
The administrator has disabled public write access.

Changes to media handling for webtrees-1.4.0 4 years 1 month ago #6

  • makitso
  • makitso's Avatar
  • Offline
  • Administrator
  • Posts: 617
I haven't (yet) changed any code relating to media uploading.

OK, that covers it.
Also, the fact that the "create thumbnail" logic gives this errorneous message, whatever problems it finds, is well known.
This was fixed when I renamed the /media folder to something else.

One other point, you mentioned an "upgrade script". I presume this does not exist yet? So, this script would

1. Copy files from /media to /data/media and rename the old /media folder
2. Update the gedcom 1 FILE media/foo.jpeg to 1 FILE foo.jpeg
3. Change the MEDIA_FIREWALL_ROOTDIR for each gedcom to point to media/
4. and others?

I am asking this detail to be sure the wiki changes are correct for the 1.4 release.
Rob
webtrees.net site administrator
www.skatekey.net (version1.7.7)
PHP 7.07, MySQL 5.5.49, Apache 2.2.9,
Hosted at tigertech.net
Last Edit: 4 years 1 month ago by makitso.
The administrator has disabled public write access.

Changes to media handling for webtrees-1.4.0 4 years 1 month ago #7

  • wdm001
  • wdm001's Avatar
Greg

In another post you mentioned that you were going to retain the ability to reference the media at another location.

I have my media in another location (above the www root) that is also referenced by other programs and in particular tmg. I also sinc this with my own small NAS at home.

I think the wording in this section of your note helps me to retain this.
This and the other changes means that we can remove the restriction
that $MEDIA_DIRECTORY cannot contain "../" in the path. So, you could
set your media directory to "../../../foo/bar/".
Some people may find this useful.

So in my current path will just need an additional ../ to lift it up one more directory. Please confirm

Will you be retaining the ability to have "Folders" in the media directory
The administrator has disabled public write access.

Changes to media handling for webtrees-1.4.0 4 years 1 month ago #8

  • fisharebest
  • fisharebest's Avatar
  • Offline
  • Administrator
  • Posts: 10290
So in my current path will just need an additional ../ to lift it up one more directory. Please confirm

Yes, that is the plan.

Don't forget that you will still be able to move the entire /data folder (and hence also the media files), the same as previously.

On my development server, I have about 10 different webtrees installations. They all share the same data folder (../webtrees-data), which means they all share media files, gedcom files, etc. while using different code and databases.
Will you be retaining the ability to have "Folders" in the media directory

Yes.

All I plan to do in this release is to replace the current list of three possible media locations:

WT_ROOT / $MEDIA_DIRECTORY
WT_DATA_DIR / $MEDIA_DIRECTORY
$MEDIA_FIREWALL_ROOTDIR / $MEDIA_DIRECTORY

with just one:

WT_DATA_DIR / $MEDIA_DIRECTORY
Greg Roach - This email address is being protected from spambots. You need JavaScript enabled to view it. - fisharebest.webtrees.net
The administrator has disabled public write access.
Do you need a web hosting solution for your webtrees site?
If you prefer a host that specialises in webtrees, the following page lists some suppliers able to provide one for you: 

Changes to media handling for webtrees-1.4.0 4 years 1 month ago #9

  • jon48
  • jon48's Avatar
  • Offline
  • Junior
  • Posts: 135
Hello Greg,

fisharebest wrote:
So in my current path will just need an additional ../ to lift it up one more directory. Please confirm
[...] Don't forget that you will still be able to move the entire /data folder (and hence also the media files), the same as previously.

Just to clarify, from your description in post 1, we will still be able to have only the media folder moved to any location while keeping the data folder at the public root (I do not want to change the location of my custom translations)? So either we move the whole data folder (and the media folder will only be a subfolder of it, without ../ in the path), or we move only the media folder using ../ (or we leave it as it is of course!)

fisharebest wrote:
3) Ambigious folders.
[...] I plan to stop adding this folder. An upgrade sript will remove it from
the GEDCOM by performing a search/replace of "1 FILE media/" with "1 FILE "
I understand the logic behind it, and just want to make sure that it will still be possible when downloading a GEDCOM to add a prefix to the path. When I need to transfer it to another software, I am indeed usually storing my downloaded data under one folder, the gedcom at the root (so "./mygedcom.ged"), and a "./media/" folder at the root with the same structure as the website. So far, I had nothing to do as the media folder was part of the downloaded GEDCOM, but with the new handling, it will have to be added when I download one file. The only thing I will have to remember is not to reimport this file, as it will contain the path "media/filename.jpg" which will not be correct any more...

Regards.
Jonathan
https://genea.jaubart.com - Famille Jaubart-Rousset en Lozère - webtrees 1.7.9 (modifié - GitHub repo, thème Rural)
Local: Ubuntu 16.04 - Apache 2.4 - PHP 7.0 - MySQL 5.7 / Public: FastComet
The administrator has disabled public write access.

Changes to media handling for webtrees-1.4.0 4 years 1 month ago #10

  • fisharebest
  • fisharebest's Avatar
  • Offline
  • Administrator
  • Posts: 10290
So either we move the whole data folder (and the media folder will only be a subfolder of it, without ../ in the path), or we move only the media folder using ../ (or we leave it as it is of course!)

Yes. That is the plan.
still be possible when downloading a GEDCOM to add a prefix to the path

In my post, I described a new options - "media prefix to ignore".

This will be used on import, and it would remove the specified prefix. e.g. "C:\users\greg\documents\genealogy\".

I guess the export option would be able to insert it.

So, you could set this option to "media/", and it would be added when you download.
Greg Roach - This email address is being protected from spambots. You need JavaScript enabled to view it. - fisharebest.webtrees.net
The administrator has disabled public write access.

Changes to media handling for webtrees-1.4.0 4 years 1 month ago #11

  • fisharebest
  • fisharebest's Avatar
  • Offline
  • Administrator
  • Posts: 10290
FYI, I have moved the "create media object" link from the "media admin" page to the "add unlinked record" page.

It requires a "current family tree" - which is unavailable on the media admin page. At present, you cannot be certain to which tree you are adding a media object.

It also seems more consistent to have it here.
Greg Roach - This email address is being protected from spambots. You need JavaScript enabled to view it. - fisharebest.webtrees.net
The administrator has disabled public write access.

Changes to media handling for webtrees-1.4.0 4 years 1 month ago #12

  • ToyGuy
  • ToyGuy's Avatar
  • Offline
  • Moderator
  • Live like it's Christmas every day - Santa Stephen
  • Posts: 5404
I noticed when I updated. Let you know if I think that creates any issues, but so far - looks good.
Santa Stephen the Fabled Santa
Latest webtrees at MyArnolds.com
Hosted by webtreesonline.com, a division of GeneHosts LLC
MacOS 10.6.8, Apache 2.2+, PHP 5.4.16, MySQL 5.5.28
The administrator has disabled public write access.

Changes to media handling for webtrees-1.4.0 4 years 1 month ago #13

  • makitso
  • makitso's Avatar
  • Offline
  • Administrator
  • Posts: 617
Greg,

Like Stephen, I have been doing a lot of testing of the media changes -- they are working very well.
Rob
webtrees.net site administrator
www.skatekey.net (version1.7.7)
PHP 7.07, MySQL 5.5.49, Apache 2.2.9,
Hosted at tigertech.net
The administrator has disabled public write access.

Changes to media handling for webtrees-1.4.0 4 years 1 month ago #14

  • popcorn
  • popcorn's Avatar
Maybe this is not the right place to ask this: :? I could not find a developer section in the forum

Does webtrees run on its/a framework? And if not, are there any plans to go in that direction, especially surrounding templating?
The administrator has disabled public write access.

Changes to media handling for webtrees-1.4.0 4 years 1 month ago #15

  • fisharebest
  • fisharebest's Avatar
  • Offline
  • Administrator
  • Posts: 10290
Does webtrees run on its/a framework?

No. All the code is home grown.
And if not, are there any plans to go in that direction,

I like the ZendFramework. We use some components of it. I'd like to use the full MVC stack, and am slowly restructuring the code to fit the pattern.
especially surrounding templating?

Templating would be good - but it is quite some way off.
Greg Roach - This email address is being protected from spambots. You need JavaScript enabled to view it. - fisharebest.webtrees.net
The administrator has disabled public write access.

Changes to media handling for webtrees-1.4.0 4 years 1 month ago #16

  • popcorn
  • popcorn's Avatar
I agree, an MVC implementation would make webtrees more powerful to developers, easier to develop, extend and maintain.
Is there a branch where such development and changes are introduced?
I would like to follow and contribute where possible to such development.
The administrator has disabled public write access.

Changes to media handling for webtrees-1.4.0 4 years 1 month ago #17

  • fisharebest
  • fisharebest's Avatar
  • Offline
  • Administrator
  • Posts: 10290
Greg Roach - This email address is being protected from spambots. You need JavaScript enabled to view it. - fisharebest.webtrees.net
The administrator has disabled public write access.

Changes to media handling for webtrees-1.4.0 4 years 1 month ago #18

  • fisharebest
  • fisharebest's Avatar
  • Offline
  • Administrator
  • Posts: 10290
Over the next few days, I will submit these media changes to SVN.

If you use SVN on your live site (and you know the risks of doing this!), then note that

1) there have been lots of changes to the handling of media files. Much of the code is new. Please verify every media file upload/rename/edit.

I know that some people share their media folder between their live and test systems. If you do this, then beware that your test system has the potential to damage your live system.

2) there are changes to the "media firewall" configuration. i.e. there is no configuration and it is always enabled/configured. The update scripts should take care of every configuration that I've thought of - and update your settings automatically. If you are using a different configuration(!!!) and your media is missing, you may need to update your media-folder settings manually. Let me know any issues here.

3) there will almost certainly be bugs relating to the *display* of media. These should be harmless (in the sense that they won't change any data).

Make sure you have good backups. You probably want to try things out in a test environment before updating your live site.

Externally, there are few visible changes. The media-list and admin-media pages look a bit different. Missing images are now shown with CSS icons instead of image icons.

Internally, most of the code is new. It still needs a lot of tidying up, but it has reached the stage of "doing all the obvious things right", so it's good enough for SVN.

One final note. If your media files have full path names (e.g. c:\users\documents\foo\bar\...), then the "directory levels to keep" option has been removed - but the replacement code hasn't yet been written. You'll need to remove the prefix using a search/replace.

If this makes you nervous - and it should - then simply stick at the current SVN release for a week or two.
Greg Roach - This email address is being protected from spambots. You need JavaScript enabled to view it. - fisharebest.webtrees.net
The administrator has disabled public write access.

Changes to media handling for webtrees-1.4.0 4 years 1 month ago #19

  • ToyGuy
  • ToyGuy's Avatar
  • Offline
  • Moderator
  • Live like it's Christmas every day - Santa Stephen
  • Posts: 5404
Thanks for the extra warning. FYI, I intend to update the SVNTEST site with the new code, when posted, and prior to activating the update, I'll backup (more than usual) the media files and SQL. Then, the procedure will be available for you, Nigel, and I to test on my 110,000 INDI, 7000 media items site. (Stephen)
Santa Stephen the Fabled Santa
Latest webtrees at MyArnolds.com
Hosted by webtreesonline.com, a division of GeneHosts LLC
MacOS 10.6.8, Apache 2.2+, PHP 5.4.16, MySQL 5.5.28
The administrator has disabled public write access.

Changes to media handling for webtrees-1.4.0 3 years 3 months ago #20

  • lisaman
  • lisaman's Avatar
I need help please.. I am a Gedview user and changed.. still not live as media which was so simple in gedview is driving me crazy ;-)
This is my media file
/home/lisabrad/public_html/webtrees/data/media/
It is full of files.. jpegs and pdfs
If I click on media.. most files have this

media/Adrien3.JPG
This media file does not exist.

in cpanel I can click on the file in Media and it shows
IN webtrees when I click image it again tells me it does not exist..

I have just loaded iris3 into data/media.. in cpanel it is there but when I search in webtrees in media.. It can't find the file!!

I also need to link each media item to the person.. I don't mind doing it manually.. but how?

Thank you
Your help is much appreciated..
Please use simple language.. I'm not that clued up!! ;-)

I have just found this error when going to Iris individual record then clicking the media tab
Fatal error: Call to a member function getAllFactsByType() on a non-object in /home/lisabrad/public_html/webtrees/modules_v3/lightbox/functions/lightbox_print_media.php on line 369
must all the media be copied to lightbox!
Last Edit: 3 years 3 months ago by lisaman.
The administrator has disabled public write access.

Changes to media handling for webtrees-1.4.0 3 years 3 months ago #21

  • fisharebest
  • fisharebest's Avatar
  • Offline
  • Administrator
  • Posts: 10290
Each media object (a record in your family tree) contains a filename. It may also contain folders and subfolders, to help organise your media. You'll see entries like this:

1 FILE photo.jpg
1 FILE smith/john_smith.jpg
2 FILE places/london/st_pauls.jpg

Where does webtrees look for these? In the media folder specified in your family-tree settings. By default, this is data/media

So, webtrees would look for these three files in

data/media/photo.jpg
data/media/smith/john_smith.jpg
data/media/places/london/st_pauls.jpg

Now, there was a bug/feature in PGV (and early versions of webtrees) that let you *optionally* include the media folder twice - once in the config settings and once in the media record.

So, you could have

1 FILE photo.jpg
1 FILE media/photo.jpg

and PGV would look for *both* of these in the same place - media/photo.jpg.

webtrees removed this "flexibility", because it caused no end of problems.

So, what I guess is happening is that (some) of your data contains the superfluous media folder.

Hence webtrees is looking for

/home/lisabrad/public_html/webtrees/data/media/media/photo.jpg

instead of

/home/lisabrad/public_html/webtrees/data/media/photo.jpg

If this is the case, you need to remove the redundant folder from the GEDCOM file.

If there are just a few, edit these using the normal edit screen and remove the folder.

If there are hundreds, use a text editor (not word processor) and search/replace

"1 FILE media/"

with

"1 FILE "
Greg Roach - This email address is being protected from spambots. You need JavaScript enabled to view it. - fisharebest.webtrees.net
Last Edit: 3 years 3 months ago by fisharebest.
The administrator has disabled public write access.

Changes to media handling for webtrees-1.4.0 3 years 3 months ago #22

  • lisaman
  • lisaman's Avatar
Hi Greg.. Thanks so much
I had already got rid of that double media file.. it was there
so all media is in data/ media

How would I look for the individual path of a picture to see if it says media or media/media ;-)

I did right mouse click on a picture and copied the link.. This is what it gave me!!

lisabradshaw.com/webtrees/mediafirewall....%20Scott&cb=a22696f0
Last Edit: 3 years 3 months ago by lisaman.
The administrator has disabled public write access.

Changes to media handling for webtrees-1.4.0 3 years 3 months ago #23

  • fisharebest
  • fisharebest's Avatar
  • Offline
  • Administrator
  • Posts: 10290
How would I look for the individual path of a picture to see if it says media or media/media ;-)

Menu -> Edit -> Edit raw GEDCOM record
Greg Roach - This email address is being protected from spambots. You need JavaScript enabled to view it. - fisharebest.webtrees.net
The administrator has disabled public write access.

Changes to media handling for webtrees-1.4.0 3 years 3 months ago #24

  • nigelo
  • nigelo's Avatar
lisaman wrote:
I need help please.. I am a Gedview user and changed.. still not live as media which was so simple in gedview is driving me crazy ;-)

Don't worry Lisa, you're not alone. The webtrees media change (in v1.4+) has been the single most significant factor in the growth of my kiwitrees.net hosting business !! Far more people than ever before now feel they need help setting up and using webtrees, and in particular trying to upgrade webtrees or transfer from PGV. I'm loving the changes :-)
The administrator has disabled public write access.

Changes to media handling for webtrees-1.4.0 3 years 3 months ago #25

  • ToyGuy
  • ToyGuy's Avatar
  • Offline
  • Moderator
  • Live like it's Christmas every day - Santa Stephen
  • Posts: 5404
nigelo wrote:
lisaman wrote:
I need help please.. I am a Gedview user and changed.. still not live as media which was so simple in gedview is driving me crazy ;-)

Don't worry Lisa, you're not alone. The webtrees media change (in v1.4+) has been the single most significant factor in the growth of my kiwitrees.net hosting business !! Far more people than ever before now feel they need help setting up and using webtrees, and in particular trying to upgrade webtrees or transfer from PGV. I'm loving the changes :-)

And I can second that emotion, errrr...notion, uuhhhhh...motion. It seems that many, many more simply want a host like mine or nigel's to install it for them and then tell them how to keep it up. Trust me, with my 'regular' job a bit seasonal, I can use the extra income too.
Santa Stephen the Fabled Santa
Latest webtrees at MyArnolds.com
Hosted by webtreesonline.com, a division of GeneHosts LLC
MacOS 10.6.8, Apache 2.2+, PHP 5.4.16, MySQL 5.5.28
The administrator has disabled public write access.
  • Page:
  • 1
  • 2
Moderators: makitso
Powered by Kunena Forum