Multiuser Bibliography Project Journal

This journal documents the progress of the MultiBib project and the activities of Scrum Team 3 (Casey DeSmet, Jennifer Ismirle, and Michael Wojcik). It's organized in reverse chronological order (most recent first), with entries that correspond to "intervals", the periods between scrum meetings.

Wednesday 24 April 2008

Done in previous interval
  • The team peer-reviewed our prototypes and functional specs. There were some good comments all around.
  • A few minor glitches in the applications were tracked down and fixed.
  • I fixed an issue with updating entries: when an entry was updated, any comments attached to it stayed with the old version, rather than moving to the new one.
  • Finished the functional spec. Wrote the reflection. Packaged Module 3 for submission.
Planned for next interval

This is the release; there are no more intervals in this sprint. For the next sprint (if I return to this project — and I just might), I'll look at the remaining things that were never implemented, what's called the "backlog" in Scrum.

Here are some of the things on the backlog:

  • Possibly implement an add-user mechanism, or simplify the password mechanism (which doesn't really need salt, though without it it's vulnerable to brute forcing with a precomputed dictionary) to make it possible to add users easily with SQL or phpMyAdmin.
  • Implement Javascript pre-submit verification of form fields, as a convenience feature.
  • Possibly implement some of Mike's suggestions from the review of the design packet, such as not displaying rows with no value in the detail view, and cleaning up the add/edit view somewhat.
  • A nice convenience feature, time permitting, would be a Javascript-based hinting mechanism for add/edit. It would work by displaying a couple of sample bibliographic entries alongside (probably below) the current input field, with the appropriate data for that field highlighted. It would use onfocus events and spans around the various parts of the sample entries. The sample entries would be rendered in a div with absolute positioning, so it'd float as the form scrolled.
  • Currently, comments are only associated with the most recent version of an entry. When an entry with comments is updated, it'd be better to have it keep its association with the existing comments, and have them be associated with the new one as well. That would require a new relational table, so that comments could be associated with multiple entries. Not hard, just some database redesign and some work in the backend.
  • Adding a comment to an entry doesn't make that entry eligible for the "recently updated" view; it should. That's because recently-updated works off dateadded, which is changed when an entry is added or updated (because an update is an add, in this application), but not for comments. We should have an additional datechanged field, which could be initialized to the same as dateadded, but updated if a comment is created for that entry. Then that would be the sort key for recently-updated.
  • There's too much duplicated code, eg in formatting entries for display. That needs to be refactored into common functions.
Impediments
None, now that the sprint's finished. And we have a working prototype. Success!

Monday 21 April 2008

Done in previous interval
  • The team reviewed one another's functional specifications. This was a useful exercise — we compared organization, etc, and also made suggestions on what extraneous material (eg unnecessary screenshots) we could each drop.
  • The edit mechanism now works, including moving the old entry to the "oldentries" table. Edit uses the Add code (in do-add.php), with an additional hidden parameter that tells it that the new entry supersedes an old one. If the new entry is added successfully, do-add will find the old entry, retrieve its data, and build an INSERT for the oldentries table; and if that works, it will do a DELETE for the superseded entry from the entries table.
  • The comment mechanism is implemented.
  • There used to be an intermediate module named update-switch.php which handled the two buttons (Edit and Comment) on the Detail View page. They were both submit buttons for a single form, with update-switch as its action; and update-switch then would redirect to the edit page or the comment page. Now the buttons are each in their own form, and go directly to edit.php or comment.php. This required some style work to get the buttons rendered as I wanted.
  • A fix to the Add (now also Edit) mechanism: it was always setting all fields in the INSERT statement, including ones that had no data. Instead it should have been leaving those off (for string fields) or setting them to NULL (for integer fields), so that they wouldn't end up with empty or default-zero values.
  • Added tabindex attributes to input fields on multipart forms so that the submit button would be selected before the reset button.
  • Ensure the user is signed in before actually performing a controlled function in do-add and do-comment. The UI already checks this in add, edit, and comment, but it would be possible to bypass those checks by replaying or forging a POST request to the appropriate resource.
  • Some minor fixes to displayed text and the like.
  • Added a test user (test@foo.bar, password "test") for use in demonstration, etc. There's currently no user-add page, which is a bit of a problem, because the (perhaps overly-complicated) salted password scheme I'm using makes it difficult to add them manually.
  • Worked on the Functional Specification .
Planned for next interval
  • Finish the functional specification.
  • Possibly implement an add-user mechanism, or simplify the password mechanism (which doesn't really need salt, though without it it's vulnerable to brute forcing with a precomputed dictionary) to make it possible to add users easily with SQL or phpMyAdmin.
  • Implement Javascript pre-submit verification of form fields, as a convenience feature.
  • Possibly implement some of Mike's suggestions from the review of the design packet, such as not displaying rows with no value in the detail view, and cleaning up the add/edit view somewhat.
  • A nice convenience feature, time permitting, would be a Javascript-based hinting mechanism for add/edit. It would work by displaying a couple of sample bibliographic entries alongside (probably below) the current input field, with the appropriate data for that field highlighted. It would use onfocus events and spans around the various parts of the sample entries. The sample entries would be rendered in a div with absolute positioning, so it'd float as the form scrolled.
Impediments
There may not be time to do convenience functions and bells & whistles. If not, I'll mention them in the functional spec as possible future enhancements.

Wednesday 16 April 2008

Done in previous interval
  • Worked on the Functional Specification .
  • Fixed the display of non-ASCII characters, such as the c-cedilla in "Françes". This was not at all apparent from the PHP or MySQL documentation; I finally found the secret on some blog page. My MySQL tables were already declared as UTF-8, but to actually get MySQL to add or retrieve data as UTF-8, you have to submit the SQL statement SET NAMES 'utf8'.
Planned for next interval
  • Finish the functional specification.
  • Implement the comment mechanism.
  • Implement the edit mechanism, including moving the old entry to the superseded table.
Impediments
Too many other responsibilites...

Monday 14 April 2008

Done in previous interval
  • Superseded entries now live in a separate "oldentries" table. This simplifies queries that shouldn't find superseded entries (browse, recent, quicksearch, some advanced searches), without significantly complicating ones that should (detail view). oldentries is similar to entries, except that its id field is not autoinc — id values are unique across entries and oldentries, and an entry doesn't get a new id just because it's moved from one table to another. The detail view will now show a link to the superseding entry if there is one.
  • Advanced search works, and it's pretty cool. It builds a big ol' SQL query, with multiple WHERE phrases, and if you tell it you want to retrieve superseded entries as well as current ones, it does a UNION SELECT with that table as well. There's some monkeying to get the columns to align for the UNION. It's all in the code in do-search.php.
  • Set "overflow: auto" on the content div, so that any overflowing entries would cause it to become scrollable, rather than overflowing out of the container.
  • Password fields now use the "password" input type.
  • Made global nav consistent across all pages. Some were missing "Browse" or "Recent", and some had dead links for "Add" (a holdover from original stub pages).
  • The Add page now uses the real user ID, rather than having "0" hard-coded.
  • Added automatic emulation of magic_quotes_gpc if that feature isn't enabled in the PHP configuration. This code comes straight from the Yank book.
  • Minor tweak to the long-URL truncation mechanism to preserve a closing angle-bracket, for better appearance.
Planned for next interval
  • Implement the comment mechanism.
  • Implement the edit mechanism, including moving the old entry to the superseded table.
Impediments
Some problem with the MSU Wireless network in Olds today, so I haven't been able to get to my site to modify the tables or upload new pages yet.

Wednesday 9 April 2008

Done in previous interval
  • Implemented sign-in and sign-out. This includes the actual sign-in support (verifying credentials and setting user ID in session) and the UI components. I also now set a hidden "returnto" parameter in the sign-in form displayed in the main navigation bar on various pages, which tells the sign-in code where to go after a successful sign-in.
  • Sign-out returns the user to an appropriate place; for browse, that's browsing with the same starting point, while for add it's the "recent" (home) page, since once you've signed out you can't add a new entry. When returning to paged views (browse and quicksearch), sign-out also sets the query-string so that it goes back to the right point in the view; this requires escaping the second question mark and restoring it when generating the meta-refresh element.
  • Fixes to browse view: book titles are now formatted correctly (I was using the wrong name for the "entry type" field), and extra publication info is now shown, with long URLs abbreviated using a regex so they don't exceed the content area width. And containing work (if any) is now shown. Similar fixes to "recently updated" view; this code needs to be refactored into a common function.
  • Quick-search is implemented! This is identical to browse, except that it restricts the results using a WHERE clause that's a series of OR'd LIKE phrases against various fields, comparing them to the quicksearch string wrapped in wildcards (eg primaryauthor LIKE '%$q%'). Had to fix the declarations of those fields in the entries table using phpMyAdmin; they had been declared as "utf8_bin", which is case-sensitive, and they needed to be "utf8_general_ci", which isn't.
Planned for next interval
  • Implement the separate "superseded entries" table, as suggested by Mike. This should only take a few minutes. Some queries will be updated: browse and recent no longer need to filter out superseded entries, and detail-view should check both tables (it should be possible to detail-view a superseded entry), and indicate (with a link) if there's a superseding entry.
  • Implement the advanced search. This should be pretty easy, even with the separate "superseded" table. (Probably quick search shouldn't search the superseded table, and advanced search should have an option to include superseded entries.)
  • Sign-in is supported now, but new entries are still using a hard-coded user ID. Easy to fix.
  • Implement the comment mechanism.
Impediments
My academic-work laptop died on Tuesday morning — looks like a memory controller failure. After failing to repair it, I pulled the disk and put it in an external enclosure so I could commit Monday's changes to CVS. I didn't lose any work, but I lost some time.

Monday 7 April 2008

Done in previous interval
  • Accepted a good suggestion from Mike: when entries are superseded, move them to a different table. I can still use the supersedes / supersededby fields, but now they refer across tables. (If an entry is updated twice, it's ambiguous which table its supersededby refers to — but that's OK because the id values will be unique across both tables, so we can find the matching entry by searching both tables for that id.) This change gets rid of the extra WHERE clause (to remove superseded entries) in various queries, and lets me implement an actual DELETE.
  • Fixed the default page for the http://ideoplast.org/apps/multibib/ directory. My Tomcat configuration supports a default page named "index.php" — but it won't actually run the PHP processor on it. (One of these days I'll have to find and fix that bug.) So to make recent.php the initial page, I created an index.html with an http-equiv redirect and a startup message (with a link, in case anyone has redirects disabled).
  • Worked on browse view. Much of it is there. Need to implement advanced browses (eg entries starting with letter X, show all on one page, etc) and the advanced-browse menu shown on the project wireframe. Need to implement nicer previous / next links - these are just proof-of-concept. Also, some things aren't working correctly; book titles aren't showing up in "booktitle" spans, for example, and containing works aren't showing up at all.
  • Created new colors for the "odd" and "even" rows in the browse tables (in the browse and recently-updated views). Based on a suggestion from the team, I changed the colors (which I had picked more or less at random from color charts — always risky when you're colorblind) to lighter, less-saturated versions of colors already on the page. I did that using a Javascript color converter I wrote.
Planned for next interval
Implement login. I put a link to the app on the site's main page, so now it's indexed — and now I'm getting spam added to the database. Sigh.
Impediments
None.

Wednesday 2 April 2008

Done in previous interval
Implemented the number-updated displays on the Recent page. These were done with pure SQL - "SELECT COUNT (*) ... WHERE dateadded > SUBDATE(CURDATE(), INTERVAL 1 DAY)" and so on. Not much, but it's something.
Planned for next interval
Do the browse view. I'm also tinkering with visual elements, particularly colors.
Impediments
None.

Monday 31 March 2008

Done in previous interval
  • In workshop, resolved a number of SQL and PHP issues in members' projects, eg use of session variable to let a successful login return to the page where the login request was generated (by saving PHP path info).
  • Loaded a couple of dozen entries into the entries table, so that the browse view will have something to display:
    1. Export existing bibliographic data from Zotero in BibTeX form.
    2. Write gawk script to parse relevant data from BibTeX entries and output SQL INSERT statements.
    3. Run gawk script against BibTeX data using Cygwin.
    4. Use phpMyAdmin to import resulting SQL program and run it against my database.
  • Manually added a comment to the comment table so that the detail view can really retrieve and display it, rather than the mock-up in last interval.
  • Implement comment display in detail view.
  • Tweaks and enhancements:
    • Use consistent date/time formats throughout, using the date() function, a global variable with the format string for date(), and the strtotime() function to parse SQL DATETIME values.
    • Add a generic function to look up a user's information by ID, with caching to avoid redundant queries. (Should really cache in session data, though, rather than per-page.)
Planned for next interval
Work on browse view.
Impediments
None.

Wednesday 26 March 2008

Done in previous interval
  • In workshop, discussed our individual project and reviewed design packets. Discussed questions about the projects and implementation issues.
  • Noted that I omitted a place-of-publication field in my bibliographic-entry table. Altered the table to add that column, then updated the add page and backend to support it.
  • Created one real row in the entries table and one real row in the users table to bootstrap the detail-view page.
  • Converted the detail-view mockup into a real page (except comments, which are still mocked up).
Planned for next interval
Finish detail view (real comments view, etc). Work on browse view.
Impediments
None.

Monday 24 March 2008

Done in previous interval
Initial emails among team members. Circulated design packets. Formulated implementation strategy (see below).
Planned for next interval
Review comments from team (and record here, with responses). Work on detail view.
Impediments
The holiday weekend made it difficult for us to all get in touch for the initial scrum interval. (We probably should have anticipated that on Wednesday and tried to communicate before Friday.)

Implementation Strategy

Now that the input forms (at least some of them) are in place and working, the next step is probably to implement the browse and detail views. That means getting the SQL queries created with appropriate parameters in PHP, and formatting the output in a suitable fashion.

I'm comfortable with this technology; my questions for my team will probably be design and usability ones. Those are fields where they definitely have more expertise.

So, the next few steps:

  1. Workshop with team. When we get to my application, show them what I have so far, and ask for input on design (especially how comfortable / pleasant they find the look & feel) and usability (particularly what seems clear and what needs explanation).
  2. Implement detail view (which is actually simpler than browse view - only need to retrieve and format one record).
  3. Implement browse view, including subsetting (browse records X through Y).
  4. Implement edit-record functionality.
  5. Implement add-comment functionality. Add comment displays to detail view.
  6. Discuss possible improvements with team. (For example, here's one idea I had: for the add-entry page, have a couple of sample entries visible, maybe using absolute positioning so they're visible regardless of how the form is scrolled. Highlight the relevant parts of each sample based on what form field has focus. Should be easy enough w/ a bit of Javascript and CSS.) Worth doing? Other ideas?