I’ve upgraded ImageMagick on iOcean to the latest release, 6.2.5. ImageMagick is a set of libraries for creating, manipulating, and converting images. There are some built-in command line tools that interface with these libraries, as well as interfaces for many common programming languages. In the process of installing ImageMagick I’ve also updated the jpeg and png libraries on iOcean. Additionally, I’ve installed the Image-Magick-Thumbnail module for Perl (the basic interface for Perl, PerlMagick, came bundled with the ImageMagick install). I’ll be installing a PHP interface in the near future - there are a few out there and I’m not sure yet which one I’ll be installing.

The reason I went through the trouble of installing these software packages is so that I could write an image gallery generator script. The last few galleries I’ve put up have been more time consuming than they really need to be, especially creating thumbnails for each image, so I’m going to write a helper Perl script. I’m curious to hear if anyone else has any other uses for scripted image manipulation.

In following up on a researcher request for supplemental material posting on a project web site in coordination with a published journal article, we’ve discussed establishing a simplified path such as (http://pal.lternet.edu/suppl) that can be created as a physical location initially and shifted to a virtual pointer as our web structure matures. The idea is to provide directories tied then to the related database (in this case the bibliographic database with its attendant unique identifier (or LTER contribution#, ie http://pal.lternet.edu/suppl/biblio279)

At the following link

(http://www.elsevier.com/wps/find/journaldescription.cws_home/601265/authorinstructions),
the use of the Digital Object Identifier is summarized as follows:

“The digital object identifier (DOI) may be used to cite and link to electronic documents. The DOI consists of a unique alpha-numeric character string which is assigned to a document by the publisher upon the initial electronic publication. The assigned DOI never changes. Therefore, it is an ideal medium for citing a document, particularly ?Articles in press? because they have not yet received their full bibliographic information. The correct format for citing a DOI is shown as follows (example taken from a document in the journal Physics Letters B): doi:10.1016/j.physletb.2003.10.071″

“When you use the DOI to create URL hyperlinks to documents on the web, they are guaranteed never to change. ”

The idea of ‘guaranteed never to change’ brings forward the question of the length of ‘forever’ in contemporary organizational life or in internet timeframes and prompts two thoughts: 1) the lternet virtual pointer has an advantage of stability in addition to the original strengths of network identity and geographic indendence; 2) it might be worthwhile inquiring at the sio library about their insights or plan with respect to this type request.

Last week, Shaun and I met with Robert Thombley to discuss website structure goals and best practices. Before this, Shaun and I had never had to explain our website structure to someone who was completely unfamiliar with it. This gave us a good chance to put into words the philosophies behind the structure of the various Ocean Informatics sites. Some of these ideas we have implemented more fully than others; below is a brief outline of the discussion and where I feel we stand on each point.

1) Website file system structure: We all agreed that the file system structure should mirror the site’s navigation. First, it makes the site more sensible to the designers, and easier to work with. Second, it obviates the need for a lot of absolute links or ‘..’-based relative links, which make sites less modular. Third, it allows you to cleanly automate navigation elements (such as the breadcrumbs on the Palmer site). The OI sites have stayed have stuck close to this design philosophy, with the exception of the inherited Palmer site, which is currently on the list for some file system restructuring.

2) Naming conventions: Strong naming conventions are worth the time to set and observe. If you need to sift through your site’s files - particularly useful for auto-generating content and navigation - good naming conventions will save a lot of time by making your actions scriptable. If you are auto-generating navigation elements based on the file system, it is also important to have easily readable filenames. You may know that 05aug_phmeet means ‘August 2005 Phytoplankton Meeting’, but your navigation-building script won’t know this unless it is explicitly told. The OI sites are fairly good in terms of naming conventions, but the conventions aren’t always consistent between subsections of a site. For example, files in different projects may use different date formats in their filenames. Also, some sites were set up without auto-generated navigation in mind, so user-unfriendly abbreviations are sometimes an issue.

3) Data storage: The main issue we talked about here is storing data in database software versus storing it in flat files. The pros and cons of each approach is a blog post of its own; I won’t go into detail on that here. The most important point that was raised was that, whichever approach you take, if you implement it cleanly migrating to a new approach should not be a difficult task. Well-formatted data is more important than the storage medium itself. On the Palmer site, where we are currently migrating from flat files to a database, we’re finding that individual data files are well structured and easily parsed. However, the data as whole lacks some coherency, because the datasets were not initially recorded with interoperability in mind.

4) Templating and inclusions: Templating is an area in which it is important to strike a balance. A rigid template can be frustrating from a design point of view, but a highly configurable template can become difficult to manage. The OI sites are currently lean more towards rigid designs, with little configuration on a per-page basis. We are implementing some PHP code, however, which allows us to override the default template for those few pages where the standard design is not workable.

5) Database-driven sites: We also discussed the merits of database-driven sites. For cases where being able to quickly and easily recreate the structure and navigation is more important than flexibility of design, DB-driven sites are very useful. Robert mentioned that the site he was designing would be re-implemented for each cruise - four sites a year, all using the same structure and navigation, just different content. In a case like this, the initial, higher overhead of setting up a DB-driven site pays off in the long run. Although we have DB-driven elements on most of our sites, none of the OI sites are fully database-driven (yet).

These five points constitute only a small fraction of the considerations one must make when setting up a website. As we continue to present our sites to others, more strengths, shortcomings, and goals will come to light.

Last month, we upgraded iOcean to Mac OS 10.4.2. Since then, a series of ‘ripples’ from the update have occurred that show how much impact an environment change has beyond the initial three hours of downtime.

1) After the upgrade, we needed to upgrade to a newer version of the server administration tools from Apple (the 10.3 tools can’t be used to administer a 10.4 server).

2) In order to run to 10.4 administration tools, the OS on the computers from which iOcean is administered had to be upgraded to 10.4 as well.

3) One of the workstations from which iOcean is administered (my workstation, in fact) did not have a DVD drive, and thus OS 10.4 had to be installed on it over the network

4) We did not have a NetBoot/NetInstall server, so the service need to be set up on one of our servers, and the NetInstall images created, a process that took several hours of reading, installation, and troubleshooting.

5) The workstation I was using turned out to be incompatible with NetBoot 2.0. Luckily, we had another, newer computer that was not in use, and became (after some hardware swapping) my new workstation.

In the end, the iOcean upgrade - which was fairly minor, as it didn’t involve a major version change to Apache, PHP, MySQL, or any other services on iOcean - resulted in a great deal of additional work. In this case, the additional work - the 10.4 upgrades, the NetInstall setup, and even the hardware swapping - are all things that were slated to happen soon anyway, so the effort was not really wasted.

However, it is also the case that this additional work could have been planned out ahead of time. For example, although I knew I would have to upgrade my workstation to 10.4, I didn’t think to check that we had compatible media until I sat down with the disk in my hand. Likewise with the NetBoot 2.0 incompatibility - it was discovered during the NetBoot setup process, not beforehand. Knowing that this incompatibility existed would have saved me a few hours in the long run. Having to deal with these issues as the after-effects of the iOcean upgrade, rather than projects in their own right, made them more pressing and meant there was less time for preventative research.

Of course, in hindsight it’s easy to see the problems one should have known about. In practice, there will often be issues that are overlooked, and as such there will often be ripples to deal with.

Starting last week, we’ve implemented many changes and features to the Palmer LTER development website. Though we have not yet completed our on-growing list of items, we are comfortable enough with the site that we aim to publish it at the public domain by the end of this week.

Our list of changes and features spans many areas. While some changes are architectural and “behind-the-scenes”, others are up front and purely visual. Some of the features are mainly used as development aids, making it easier to modify the site without breaking it.

A breakdown of some of the recent changes and features:

Normalizing the CSS
This is an architectural change. Our CSS files were decentralized and fragmented, making it virtually impossible to apply trivial style changes. By restructuring and modularizing the css code into one centralized file, we have a much stronger base to modify and add new styles for html elements.

Polishing the Theme
This is a visualization change. Using Photoshop, I changed the header and background images to spice up the site. I updated the css code (which was easy after the normalization) to reflect the new theme, particularly the two columns. The left column with the yellowish background is slightly bigger, and the right column with the blue gradient is smaller. I added more style to the sub-navigation lists in the right column. Checkout the snapshots page showing the previous theme and the new theme.

Using Cookies to Test the New Theme
I added this feature as a workaround while testing and developing the new theme. Using a bit of php logic, I enabled the use of web-browser cookies to locally change the stylesheet. By setting a cookie named “newcss” with a value of “1″, the new stylesheet would only be applied to my current web browser on my computer. Everywhere else in the world would still recieve the current stylesheet. This allowed me to test any visual changes to the site without breaking it (in case there are bugs).

This feature was needed because we have been treating our development site as a working site, relying on it for presentations and demos. This extra usage makes it less of a development site, especially when users expect it to always be stable. Thus, a workaround was needed to be able to develop the site without the worry of breaking it for everyone else’s use. One option would have been to move it into another local development area. However, I felt that not only would that require too much overhead (with code versioning and merging), but that it would also be a redundant effort since this site was already in a development area. The second option of using cookies to apply local settings was easier to implement (at the cost of a few lines of php), and it prevents source code and content from getting out of sync.

Converting Absolute Paths to Relative Paths
Almost every single php page used absolute paths when including other php files. This is an extremely bad practice and would break the site when porting the code over to a new physical location on the server. I went thru every page I could find, and changed all absoulte paths to relative paths. This is a good example of an architectural change that is “behind-the-scenes”.

Cleaning the HTML Markup
Some of the html pages have sloppy or invalid code. A common example is the lack of a closing tag. For instance, <p> is never followed by </p>. When testing the new theme and css, I cleaned up sloppy html wherever I found it.

Here’s a list of some of the To-Dos:

Develop a Best Practices
We need a best practices document for site development and content placement. Having a best-practices helps achieve consistency throughout the site, and it also provides a starting point when developing new site apps or modules.
A rough best-practices:
- Use the right column for sub-navigation links.
- Use the left column for header title and main content.
- Use the bottom column for application output.

Revisit Relative Paths
We may want to create an infrastructure for relative paths, so that it is easier to move files around freely without the need to edit them to reflect their new path. There are a few ways to do this. One option is to set a local php config variable that points to a specified include path. Another option is to create a root.inc.php file inside each directory that stores that directory’s relative path to the site root in a global variable, and have all other php files in any given directory include that directory’s root.inc.php file.

Searching Capability
One future idea for the site is to add a search bar, probably at the top of the right column, for all pages.

This morning, iOcean was upgraded from Mac OS 10.3.9 to 10.4.2. OS 10.4.2 brings with it improved support for setting up SSL certificates, which will allow us to enable the https protocol and start developing web apps with secure logins. Of course, there are several other new and improved features in 10.4, which you can read about on Apple’s OS X Server site.

The installation was relatively painless: we imaged the existing 10.3.9 system drive for safety’s sake, ran an Upgrade Install, and troubleshot a few issues that cropped up. Of our major services, file sharing and logon services transitioned without any issues. Apache and the firewall required some small reconfiguration. The subversion URL responsibilities have been moved from an instance of apache in /usr/local to an instance in /opt. The instance in /opt is part of the 10.4 install and will hopefully be supported in Apple software and security updates (as opposed to the /usr/local instance, which we had to maintain manually). MySQL and PHP both worked after the upgrade, but PHP could not connect to MySQL because the MySQL socket had moved from /tmp to /var/mysql. Changing this path in the php.ini file solved the problem.

The remaining work to be done on iOcean consists mainly of cleanup. The unused instance of apache in /usr/local should be removed. Also, during the upgrade process it came to light that there are two instances of PHP on iOcean. One is a standard install, which will be support by updates from Apple. The other is a custom install that was compiled with the domxml libraries (needed for some of our scripts) but which needs to be maintained manually. We are hoping to consolidate the two soon. We are also still considering whether or not to reinstall Fink, as it is questionable how much use it saw on iOcean under 10.3.

The oceaninformatics mounted directory (…/projects/oceaninformatics) is cluttered with unorganized, temporary, and obsolete data. This makes it difficult to filter out a needed file, or to determine where to save a given file. I feel it’s time we add some conventions on how we structure our shared working environment. We may also consider using conventions for versioning our data, or perhaps even using Subversion to do some dirty work.

Here’s my proposition: All data belonging to specific projects reside in a folder with the project’s name, first letter capitalized. Examples: PersonnelDirectory, Dictionary, LTER, etc.
I suggest using a CamelCase naming convention (no spaces between words) to make things easier. These folders can be referred to as Project folders.

We may also have a set of “data” folders, where each data folder contains a specified category of data. Examples: photos, schemas, files, etc.
All data folders names are lowercase.

Any Project folders can contain other Project folders and data folders. Data specific to a project is stored in a data folder under that Project folder. Example: The source photos used for making thumbnails for the personnel directory are stored in: oceaninformatics/PersonnelDirectory/photos/src. Likewise, the thumbs may be stored in: oceaninformatics/PersonnelDirectory/photos/thumb.

Project folders may contain “sub-projects”. Example: LTER may contain the Project folder “IM Meeting in Montreal Aug4-7 2005″.

This rearrangement of folders and files is fairly trivial. I am not interested in creating a strict convention for organizing project-specific data. Rather, I would like to invoke a simple working-space skeleton that organizes our projects and data in a logical sense. This approach works well for static data files (files and documents that never change); however, we need to determine a better way to store our dynamic documents (files we edit… a lot!).

Our existing method of storing dynamic documents is to put them into personalized tmp folders (e.g. temp_ksb/, temp_srh/). We also have personalized working folders (i.e. working_lry/) used for storing “up-to-date” files. I dislike the concept of using personalized folders in a collaborative working environment. That’s what our home directories are for. I feel we should dismiss personalized tmp/working folders in favor of keeping our documents organized by topic, not by author.

(On a side note, what’s the difference between a temp folder and a working folder? To me, they are equivalent. Both types of folders tend to get bloated with “up-to-date” documents and archived revisions of those documents.)

That being said, I still see the value of a single tmp folder. It is a useful place to store documents, only with the anticipation of moving them elsewhere or trashing them.

It may be a good idea to embed author names and revision numbers into the file names (of dynamic documents), particularly when taking turns editing files. For example, I create file A and save it as: A_srh01.doc. Karen edits it and saves it as: A_ksb02.doc.
Because we are sharing documents without the aid of a file management tool, we must adhere to some kind of naming convention to keep our revisions ordered.

We may also consider using Subversion to version some of our files. Though it comes at the expense of extra overhead in the workflow process (the advantage we have now is that we are using no tools!), it efficiently saves each revision we commit and enforces us to log comments for each revision. Even if we choose to bypass Subversion and continue with our tool-less approach, we should at least practice the conceptual ideas of file versioning that Subversion offers.