CVS log for loncom/metadata_database/searchcat.pl

Up to [LON-CAPA] / loncom / metadata_database

Request diff between arbitrary revisions

Keyword substitution: kv
Default branch: MAIN

- Course Editor has "Standard Problem" item (Grading tab) for creation
  of new problem in user's Authoring Space, or in a Course "Authoring" Space.
- Course Editor has "Import from Course Resources" item (Import tab) to
  import published content from Course "Authoring" Space.
- Course "Authoring" Space
  - default.rights -- course-only access
  - quota is shared with content uploaded directly to course
  - content only browsable in course context
  - metadata not included in searchable meatdata MySQL table
CVs: ----------------------------------------------------------------------

- Score upload form supports identification of a user based on clicker ID,
  for Course Coordinators who prefer not to use LON-CAPA's in-built
  "Process Clicker" utility.
- clickers.db file on a library server contains key = value pairs, where key
  is (escaped) clicker ID, and value is (escaped) comma-separated list of
  usernames who registered that particular clicker ID.
- bi-nightly run of searchcat.pl will update clickers.db file.

- Accommodate sub-directory structure of user directories containing dot(s)
  within the first three characters of the username.

- Prevent "Can't use string as an ARRAY ref" error.

Corrected rev. 1.80:
Include loncommon to allow call of loncommon::build_recipient_list()

- Domain Configuration (Contacts)
  - Enable/Disable e-mail sent to LON-CAPA when:
    (a) error form is submitted
    (b) domain defaults (authentication type or language) are changed.
  - E-mail recipient can be specified when bi-nightly searchcat.pl detects
    situation where a student/employeeID is used for more than one user in
    server's domain.

- New routines to delete unwanted keys from ids.db GBDM file in a domain
  lond: &del_id_handler() called to process "iddel" request from lonc side
  -- &iddel() routine in lonnet.pm.
- searchcat.pl will compare id => username retrieved from ids.db with
  expected id => username retrieved from each user's environment.db file
  (loops over user directories in /home/httpd/lonUsers/$dom), and updates
  ids.db based on latest data in environment.db, and logs actions.

- When populating allusers table:
  - Use hash of courseIDs from single call to lonnet::courseiddump() for domain, instead of individual call to lonnet::is_course() to test if user is a course.
  - Check if directory in lonUsers/$dom/$1/$2/$3/$uname is for a user
    by testing for existence of passwd file.

searchcat.pl
- Additional MySQL table to build: allusers
- &descend_tree() arguments changed. $dom added as first arg, and reference to %allusers hash added as last arg.
- %allusers keys are usernames for all users in the domain which are not courses.
- user information put into allusers MySQL table.

LONCAPA/lonmetadata.pm
- description of allusers table
- &update_metadata(), &lookup_metadata() and &delete_metadata() modified to allow more flexibility in the WHERE condition in the SQL query (no longer forced to be url = ).
- &process_allusers_data() added to add/modify the contents of the allusers table.

- eliminate some copy/paste

- backport 1.74

- backport 1.73

- backport 1.72

portfolio_metadata, portfolio_addedfields and portfolio_access tables in MySQL updated when portfolio access controls are modified or when portfolio metadata is updated.

Some routines in searchcat.pl moved to LONCAPA::lonmetadata.pm so they are available to lonsql when performing MySQL table updates on the library server which houses the portfolio files.

metadata updates and access control updates from lonmeta and lonnet::modify_access_controls are routed through lonnet::update_portfolio_table() which in turn does a querysend to dispatch a query to lonsql (via lond).

Only access control settings for public and passphrase-protected controls are stored in the portfolio_access MySQL table.

If no metadata file exists, only include courserestricted in record in portfolio_metadata MySQL table if this is a real group.

Only store in MySQL portfolio_access table for searching if acl is for public or passphrase protected access.

Extra / in fullpath and a typo.

Moving &sqltime() from searchcat.pl to lonmetadata.pm so it can accessed elsewhere when usin Apache::lonmetadata().

- switch to is_course routine

Support for searching for portfolio files using metadata associated with a file. Three new MySQL tables added - portfolio_access, portfolio_metadata and portfolio_addedfields.  Both personal portfolio and course group portfolio files with currently active access controls are included.

- removing use Apache2::Compat that never really worked

- BUG#4645, since no_chdir is now being used need to specify the full path

- don't do the chdir since we use the full filename anyway (faster this way)

- Apache 2 compatability

- tturning off some old debugging bits

use Apache::lonnet instead of LONCAPA::LonConfiguration
multi-domain support is now handled on the same code path as single domain support,
    obiviating the need for a command line parameter.
No longer uses the nonexistant "mailto" command to send email about user id mismatches,,
    instead uses 'mail' to complain.

searchcat.pl:Bug 3961 - metadata only processed for default domain on
    multi-domain servers.  Now takes command line option -multi_domain.
    This causes a hostname lookup and the hosts.tab file to be parsed for
    matches of the hostname.  Added 'domain' to the dynamic metadata.
lonmetadata.pm:Added 'domain' to the metadata table and to the dynamic
    metadata;

&process_meta_file: If the creationdate and/or lastrevisiondate is not given
in the .meta file, use the last modified time for the target resource.

Append to the logfile instead of overwriting it.

Append PID to temporary table name to prevent collisions when multiple
searchcat run simultaniously.

The description does not have all fields.

Moved nohist_reseval parsing functionality from searchcat.pl to lonmetadata.pm.
searchcat.pl: stop dying on insert errors.
lonmetadata.pm: &store_metadata now translates 'nan' to 'NULL' because MySQL
won't take 'nan' in a numeric field.
Added &process_reseval_data and &process_dynamic_metadata routines.

Added command line option handling and logging routine.
Added &process_dynamic_metadata and &get_dynamic_metadata.  Removed
&dynamicmetadata subroutine.

Complete refactoring.

Currently does not attempt to get dynamic metadata.
Now uses lonmetadata routines to manage MySQL tables and inserts.
Appears to work fine, insert correct values into mysql, and run without
warnings or errors.
Added counting of copyright types.

* Do not catalog directory metadata under "default"
* Get rid of 1969 as "dates"

Bugs #789 and 1926: Trying to make new fields searchable, just saving my work.

Saving my work

Work on Bugs #789 and #1926: gradelevel and standards info

FLOATs working.

The FLOAT fields do not work yet.

Slowing getting there. Still needs subjective evaluation data included.

Saving my work - this produces and SQL error on Insert.

Setup for new table

Saving my work. Dynamic metadata now correctly generated, needs to be
put into database

Just saving my work on retrieving dynamic metadata - this does not work yet.

Does what the previous one did, but uses less memory. The next version needs
to address file locking, etc., which is going to be a major change.

Not yet sufficiently tested: reduce memory consumption of searchcat.pl by
replacing foreach with while.

lonmeta.pm has a display problem, but I ran several "before/after" tests
directly on nohist_resevaldata, and this version of searchcat.pl seems to
work. Need to make sure we start from the backup versions of nohist_resevaldata
and then run this version of searchcat.pl.

Start to respect "obsolete"

Comment out calls to dynamic meta function.

Crashed with

Illegal division by zero at searchcat.pl line 347.

when hitting the first author who had no resevaldata yet ($oldsize=0).

Discovered while adding MIME-type count.

Added check on the ratio of the sizes of the nohist_reseval.db and
nohist_new_reseval.db files.  This will hopefully prevent us from
overwriting data.

Simple one-line status for cluster-wide automated reporting

Part of bug #1638: Rollback of changes to dynamic_metadata routine to version
1.32. New routine had been corrupting the data file.

&dynamic_metadata
   while(each) instead of keys, so we only read the db file once.
   study the regular expression before we do a bunch of regex matches
metadata database update code:
   Use statement handlers to do quoting
   No longer bother to check if something is in the database before
       telling MySQL to delete it - we just tell it to delete it and let
       it do what must be done.

Indentation fix only.  No real code changes.  Move along.......

Start work on inserting dynamic metadata into MySQL database.

best wishes to all.

Trying to make sense of the diffs with previous versions, i.e., 1.27 (sniff)

- modifying header to be correct
- cleaning out unneccesary comments

beautified; BUG 1215 FIXED documentation that was horrible is now non-horrible
and
it now exists in every place that it should be; BUG 1216 FIXED stale records that
are not accounted for on the system are deleted; BUG 1217 FIXED, the
loncapa:metadata table can now change structure, and this script will
verify and alter the table structure if necessary; BUG 1218 PARTIALLY FIXED
(a small solid step toward The Great Metadata Overhaul); also using File::Find
now instead of find.pl

According to Scott's suggestion, it is better to check the user ID than
to allow running as root, and then fix ownerships.

Bug #1111
searchcat.pl is apparently running as "root" and changed ownership on
nohist_resevaldata.db to "root". Need to make "www" owner of that file.
This should go into 0.6.1

This took a lot of memory since it first stored ALL filenames in an array.
Now does it in author chunks.
Also tries to condense resevaldata files. First run will take a long time.

Bug 839

- Need to put () after all calls to GDBM_*

Does cleanup now (still disabled)

Can collect dynamical metadata.

loncapa_apache.conf does not need to be directly specified

Automatically create metadata database if it does not exist.  This happens
weekly during a cronjob run, not during interactive publishing.

Replaced access.conf with loncapa_apache.conf.

using LONCAPA::Configuration::read_conf
BUG 129

add in header

updating change information at beginning as well as
only running if machine is a library server -Scott

calculating the correct value for month -Scott

making a 'time' to sql-datetime conversion for entering into sql
database -Scott

fixing uri processing -Scott

making url to be syntactically correct uri as opposed to a truncated version..
also making sql search&destroy functionality to be case-sensitive

syntax error.. forgot a logical operator -Scott

not entering in non-current versions, and cleanly replacing existing
entries -Scott

should work now -Scott

minor syntax error -Scott

adding the very important DBI module -Scott

removing stupid bug -Scott

removing minor syntax error -Scott

making dbh variable accessible to the entire script.. stopping errors
-Scott

filtering resource directories to only look at directories
belonging to a given home user -Scott

a batch script for synchronizing the contents of a library server with
a SQL metadata database -Scott