CVS log for loncom/interface/loncoursedata.pm

Up to [LON-CAPA] / loncom / interface

Request diff between arbitrary revisions

Keyword substitution: kv
Default branch: MAIN

- For 2.11.4 (modified)
  Include changes in 1.209

- Bug 5273. Authors can show which unexpired co-author roles are managers via:
  People > "Manage Co-authors"

- For 2.11.4 (modified)
  Include changes in 1.202

- For 2.11
  Backport 1.208

- Support for Ubuntu 20 LTS.
  Explicitly include ORDER BY, as an implicit sort based on GROUP BY was
  eliminated in MySQL 8.

- For 2.11
  Backport 1.207

- Bug 6949
  - Store placeholder in place of actual submission in md5_*_responsedata table
    if length of submission > max_allowed_packet size for MySQL server.
  - Length of insert statement should not exceed max_allowed_packet size
    when making SQL inserts into md5_*_responsedata table.

- For 2.11
  Backport 1.206

- Bug 6949
  Store placeholder in place of actual submission in md5_*_parameter table if
  data length of submission > max_allowed_packet size for MySQL server.

- For 2.11
  Backport 1.205

- Bug 6949
  Length of insert statement should not exceed max_allowed_packet size
  when making SQL inserts into md5_*_performance or md5_*_parameters tables

- For 2.11
  Backport 1.204.

- Bug 6601

- For 2.11
  - Backport 1.203

- For &rank_students_by_scores_on_resources() include a secondary sort
  (arbitrarily chosen to be by username) so rank for users with the same score
  is invariant between calls.

- Bug 6754 LON-CAPA as LTI Consumer
  - Grading screens can be used to display passback transactions, and
    for override of scores etc., for external tools set to be "gradable".

- Preserve student_id (originally from auto increment) when ensuring course's
  "temporary" MySQL tables for student information are current.

- Ensure course's "temporary" MySQL tables for student information are
  current when Chart or Statistics web GUI is used.

- Bug 5464
  Can set failsafe value in domain (or override in Auto-enollment course
  configuration to avoid unwanted dropping of students when no data are
  retrieved for a particular inststitutional section.

- Only need to ensure course's "temporary" MySQL tables for groupnames and
  studentgroups are current if tables exist.
- Remove two debug lines.

- Ensure course's "temporary" MySQL tables for groupnames and studentgroups
  are current when Chart or Statistics web GUI are used in course with groups.

- For 2.10
  - Backport 1.196

Bug 5674

- Columns for disk usage (%) and disk quota (MB) available for selection
  when listing users with Author roles (domain context).
- User listing can be sorted by disk usage or disk quota (sorts numerically),
  whereas lexical sort used for other sortable fields.

- Records in classlist.db file can contain an additional item (credits), if
  credits are in use in the domain, and the credits earned by the particular
  student are different from the course default.

- Fix POD.

- Fix typo in POD.

Some desultory adding of POD comments.

Reworked a lot of the POD comments

Comment tweaks

Past tense of "occur" is "occurred" (rr!)
Also added some missing &mt() calls and corrected some other little typos

Modify calls to &lonnet::GetFileTimestamp() to us the updated routine, which makes no assumptions session-server side about the directory structure for userfiles on the homeserver.

- &section_group_filter(): move initialization of @options inside loop, in case additional filters are added.
- call to lonnet::retrievestudentphoto() for full size image occurs immediately before call for thumbnail, during initial parse of %userlist.
- Add additional index to array, for each userlist item, (for thumbnail of student photo).

bugs 5661, 5662.
- request for urls to access photos (both thumbnails and full size images) served from library server (via token mechanism), moved to occur during first pass of %userlist.
- eliminates timing problems caused when tokens were being requested as table display was being generated.

Bug 5422
More comprehensive listing of LON-CAPA users.  Available to Domain Coordinators, Authors and Course Coordinators. Lists displayable in HTML, CSV or Excel formats.
- filter by status and role.
- DCs can specify a subset of courses for display of course roles.

lonuserutils.pm
- Elimination of &domain_form() -- generates linked selectboxes for domain and server selection -- no longer required (as new users can now only be created in the domain of the user's current role).
- Added &domain_roles_select() - used for linked role type and role selection boxes for DC, when displaying user roles.
- Display of active advanced roles when viewing course classlist eliminated
  - replaced with selectable advanced roles in role filter for user lists.

loncommon.pm
- &linked_select_forms() can take an additional argument ($menuorder) which specifies order of items in the first select box.
  - %menuhash can include hash key,value pair: order = reference to array
    (order of contents of the second select box) as optional item.
- get_course_users() has new optional arg (statushash) to return information about status (Expired, Active or Future) for each section for user's roles.
  - studentID information now included in optional hash of user information for course users.

lonhtmlcommon.pm
- &course_selection() routine split from &course_select_row() to allow reuse when selecting course filter for DC's listing of user roles.

loncoursedata.pm
- Three subroutines added to return indices for permanentemail, role and extent, so user listings in lonuserutils:;show_user_lists() can sort by these using the existing sorting code.

loncreateuser.pm
- display of user lists included in menu on first page.
- some wording changes (as per bug 5506)
- switches on $env{'form.state'} for call to &print_userlist() eliminated (not needed).
- course selector included when in DC context

lonpickcourse.pm
- domain selector defaults to domain of user's role when course selector used for studentform.

Work in progress - role expiration and section switching still to be added to interface.

	Set group index to constant.

- switch Store to Save

- with - and . allowed in courseids (and much longer domain/lonids allowed)
  need to do something to keep under the 64 character limit on table names,
  reverting to md5ing them

- bug#5150 was looking in wrong columns for updatetimes

- dumping of lonnet::stored info has only the symb and the
  parameter escaped so need to push the unescaping off into
  the users of the dumped info (as a result resources with :
  in the symb work in STAT again)

- bug 5003 - STAT caches not noticing change in student status

- propagate back ourt any errors taht occur while trying to fetch
  data (and don't cache the failed get)

- switch from using static numbers to using the constants so as to not break
   when the table changes (BUG#5050)

- eliminating unneed group_id fetches (BUG#4958)

- BUG#4954 - adding future to the coursedata view of students status

&Apache::lonnet::unescape -> &unescape
&Apache::lonnet::escape -> &escape

- was bombing when no stdnet enrolled in the course

&coursegroups() and &get_group_settings() moved to longroup.pm, which contains general utility functions for asking about groups.  Also contains &group_changes() which is used to add/drop group memberships as a result of role changes, as determined by group settings for auto-add and auto-drop. 'none' and 'all' are now reserved words which may not be used as section or group names, so they can be used instead of _all and _none  when specifying auto-add and auto-drop settings for all sections or no section roles.

Include keylist as argument passed to loncoursedata::get_group_memberships().  Used to add comma-separated list of active groups to a students classlist data in $classlist->{$student}->[index], where index is set so as to add to the end of the existing array.

- changing loncommon::coursegroups to be easier to use

A couple of ANDs missing from the SQL WHERE conditions for group limits.

Adding groups filter to STATS functionality. New tables added (dom_course_groupnames and dom_course_studentgroups).

Functions to retrieve information about group affiliation of students. Groups for students are classified as previous, future, active or aftercourse.  The aftercourse category includes groups in which a student's membership has expired, but in which the expiration time was less than 24 hours before the default access end date for the course, for students for whom access to the course has expired.

- adding task support to the cached data tables for response level data

- .task 'version' is really tries for them
- 'correct_by_override' is the normal stat of affairs for a .task

- II'm just goign dumb today

- typo

- need to make sure awrded ends up in the awarded column and solved ends up in the solved column

- add the part award to the response_data_by_student

- adding a response_specific_2 column to the response data table for supporting tasks
- adding the response specific data to the result of the &get_response_data_by_student()

- doesn't need this

- switching two uses of dump to dump store

- removing some old unused code

- style police

Fix bug 4537; degree of discrimation should be much more useful and
accurate now.

- style police

* If the SQL fails to become a $sth, try to provide a better error
message.

* Correct an SQL query that needed an "AS a" in it. This should never
have worked; this may have failed on my system because I recently
upgraded to MySQL 5.

- BUG#4448

- remove course_id as an argument to loncoursedata::get_classlist

- stop tring to store data if there isn't any to store

- switching the logic so that it persists as long as ->next returns something

- more defensive

- I am' 90% cure this wants to be a 1, but I would be unable to explain why

Caching bug - loading chart or spreadsheet would update the cache with the
most current submissions but would pretend that a full update (all
submissions) was done.

&store_student_data: Bug 4025: store data in cache if awarded or solved is set.

- ENV -> env

Bug 3529: correct problems plot now shows scores instead of parts correct.

Removed the hideous blight that was &get_sequence_assessment_data.
Refactored &populate_weight_table to use navmaps instead of the afforemented
subroutine.  Minor documentation changes.

loncoursedata.pm: changed &get_sum_of_scores to take symb instead of a
'resource' hash.
lonproblemstatistics: navmapification is complete.  Required addition of
package variables $navmap and @sequences.

Changed &get_response_time_data inputs to be section & enrollment status
(like the rest of the statistics calls).  Added error logging.

consolidate some of the multitude of functions out there that try to format and display the student's 'fullname'

- fixes bug#3350 hopefully for good.
    - $navmaps objects never actaully got cleaned up due the resource objects in it holding a reference back to the navmap object, this meant the hashes didn't get untied and that we leaked memory
    - needs WeakRef

modified &get_sequence_assessment_data to use $res->compTitle

&get_sum_of_scores: Added checks on validity of $students input.
Modified error logging code in various subroutines to give unique error
messages so I can tell where the messages came from and what was going
on when they occured.  I do not believe this will lead to a signifignant
increase in the size of the log files.

Added error checks in &get_response_data.

AUtomated Enrollment Manager now provides automatic switching from 'manual' to 'auto' enrollment type when a manually enrolled student is detected in the institution's classlist.  Auto switching can be prevented on a per-student basis by 'locking' the type to 'manual'. Also changes to Enrollment.pm to make it more modular.

lonnavmaps: Added is_survey method
loncoursedata: get_sequence_assessment_data: checks for surveys.

&init_dbs now takes a parameter, $drop, which indicates if the system
should drop all the cache tables before trying to recreate them.  Modified
&populate_student_table to call &init_dbs with $drop = 0.  Modified
&ensure_tables_are_set_up to call &init_dbs with $drop = 1.  This means that
additions to the student table will not cause the caches to be cleared, but
the lack of any single table in a cache set will cause all the remaining
tables to be dropped and recreated.

Bug 3007: Degree of Difficulty should not be negative.  #YES (aka $Solved)
was being computed as the sum of the awards on the problem.  Now it is
computed as the sum of the awards on the problem minus the sum of the
awards given by override.

Bug 2954.  MySQL CHAR columns need to be BINARY to be case-sensitive.

Bug 2954: MySQL CHAR/VARCHAR are case insensitive by default, so use
'BINARY' to make them behave the way we expect.

Remove debug spew.

Moved storage of dynamic metadata from loncoursedata to lonproblemstatistics.
lonproblemstatistics now only stores the dynamic metadata when the stats are
computed for all sections.
Discrimination factor computations now respect start and end times.
Modified loncoursedata::rank_students_by_scores_on_resources and
loncoursedata::get_sum_of_scores to support time constraints.

Bug 539: Report KR-21 reliability statistic

loncoursedata.pm:
  Added &limit_by_start_end_time to consolidate $starttime and $endtime code.
  Added &score_stats and &count_stats to return summary data on the scores
    and problem correct counts.

lonproblemstatistics:
  Added description of @Fields data structures
  Added @SeqFields, &output_sequence_statistics, &sequence_html_header,
    &sequence_html_output, and &compute_sequence_statistics.

Reporting statistics on sequences (instead of problems) is still being
fleshed out, interface wise.  No Excel output is available for sequence
statistics at this time.

Modified &get_problem_statistics to return the number of students with the
problem wrong as well as the percent of students.

First stab at degree of discrimination computation.
loncoursedata:
    Added $weight_table, &populate_weight_table,
    &rank_students_by_scores_on_resources, &get_sum_of_scores, and
    &limit_by_section_and_status
lonproblemstatistics:
    Added degree of discrimination to @Fields,
    &compute_discrimination_factor, and modified &get_statistics to call
    the latter.

loncoursedata: reworked &get_response_data to take section and enrollment
as inputs instead of a list of students.
lonproblemanalysis: Rewrote calls to get_response_data to provide new inputs.
Modified numerical response analysis to analyzse an entire problem, not just
one response.  This will probably need to be changed in the future.
Added &text, which produces <drawimage> text elements.

Remove debugging spew.

- backport 1.116 and 1.117

POD update for get_problem_statistics

Added comments to &get_problem_statistics.

loncoursedata: &get_problem_statistics now supports limiting statistics
    by time.
lonproblemstatistics: Now supports limiting statistics by time.
    Made text of output options more concise to save real estate.

loncoursedata.pm: &get_student_scores modified to limit by time.
loncorrectproblemplot.pm: Added time limits, corrected "bin"ing bug which
    caused the last data set to be missed, plot title is now the sequence
    being analyzed or "Multiple Sequences", added "N", the number of students,
    to the title.  Added the maximum possible number of correct problems to
    the x-axis label.

Removed log spew by &get_response_data_by_student

Modified &get_response_data_by_student to return the 'awarded' value.

Added &get_response_data_by_student and associated routines.

When calling dbh->quote() on the submissions before insert, we have to
remember to unquote on retrieval.  This was messing up problem analysis.

Clean up do-it-yourself quoting.

Further changes reflecting new MySQL table structure.

Added: loncorrectproblems plot: produce a histogram of the number of students
having N problems correct.
loncoursedata: Modified &get_student_scores to accept enrollment status and
section as inputs.  No longer works at a student level.  At worst the
SQL queries generated are cleaner, at best the queries run faster.
lonstatistics: Modified to export the selected enrollment status.
    Added the correct problems plot to the statistics menu.

Added &get_student_scores(), which returns the sum of student scores in the
course.
Restructured tables:
   Removed $studentdata_table and merged its functionality into an expanded
   $student_table.
   $student_table now includes the students section and status.
   Modified get_student_id() to call the new function &populate_student_table
   which loads the results of &get_classlist into $student_table.
   Added &store_updatetime to encapsulate the table structure a little bit.

Properly quote student submission data when inserting into the response
table.

Added CL_TYPE subroutine to retrieve user type.
User type is either 'auto' (if added via automated enrollment or via the
'Update roster now' function in lonpopulate) or '' otherwise.

get_sequence_assessment_data: make sure we have a title for the resource
by using the filename if the title is not set.

Bug 2371 is fixed!

Taking a que from Jeremy, using subroutines to indicate the array entries
in which data resides.  Also renamed get_optionresponse_data to
get_response_data and added the students name:domain as a return value.

Overhaul of &get_response_time_data().

Added &get_response_time_data.

Excel output for optionresponse analysis.

loncoursedata.pm:
Added &get_student_data.
Modified return values of get_optionresionse_data it now logs errors and
(on error) no longer returns a reference to useless data.

lonproblemanalysis.pm:
Added 'Produce Excel Data Sheet' button.  Wording will probably change, as
will page which presents the spreadsheet for download.
Added a bunch of routines associated with excel output, including two
routines 'calc_serial' and '_adjustment' copied from Andrew Benham's
datecalc1.pl, part of the Spreadsheet::WriteExcel CPAN module (GPL).
Read through some of the comments on these to learn a little about date
handling in Excel.

&get_optionresponse_data modified to return 'awarddetail'.

Changes I forgot to commit so it may work now.  get_optionresponse_data
now returns a pointer, restricts by students properly, and returns
undef explicitly on errors.

We need to escape single quotes as these are used to enclose the strings
when inserted.

Removed 'tries' from the wrong insert list.  update_full_student_data now
works again.

- backport 1.99

Removed 'tries' from $fulldump_response_table as 'tries' only applies to
parts, not responses.
Added &get_optionresponse_data to retrieve option response data.  This
routine is evolving, which is my excuse for not documenting it futher
and continuing to log the MySQL request.
Added paranoid call to &setup_table_names in &get_problem_statistics which
is completely unneccessary and utterly benign.

Added loncoursedata::clear_internal_caches() and calls to it in
&lonstatistics::Gather_Student_Data (used by statistics),
lonstudentassessment.pm (chart), assesscalc.pm (spreadsheet).  Should
go into 1.0.2 if possible.

Use 'INSERT IGNORE INTO' instead of 'INSERT INTO' so we don't bail on
all of a students data because something is redundant.  Not for 1.0.2.

- backport 1.97

&get_student_data_from_performance_cache:
Pick up all parameters all the time when asked for student data, instead of
just returning the known relevant ones.
Include in 1.0.2

Should be 'okay', not simply not begin with 'error'.

Only store the update time if the updates were successful.

Store the performance table first since it has the highest priority.

Remove 'awarded' column from fulldump_response_table.

'correct_by_scantron' is now the same as 'correct_by_student'

Added count of response types to the part data stored in
&get_sequence_assessment_data().

In full updates we need to pick up and store the response specific parameters
'submissiongrading' and 'molecule'.

Implement storing of student performance data in the MySQL cache database.
Added 3 tables: $fulldump_part_table, $fulldump_response_table, and
$fulldump_timestamp_table.  Added &update_full_student_data to do a
lonnet::dump, load the data into the new tables, and then update the old
tables used for Chart, Spreadsheet, and the current implementation of
Statistics.  Reworked some existing code to avoid duplication.

Update to get_sequence_assessment_data to expect arrays from responseType
and responseIds.

1. get_sequence_assessment_data now stores away response id and type data.
2. Replaced MySQL table $courseid.'_updatetime' with $courseid.'_studentdata'.
   Added 2 columns, section and classification which are currently unused.
   Changed primary key from student (name:domain) to student_id (unsigned int).

Hiding symb-splitting behind &decode_symb

Added 'classification' field to student table in MySQL.  Very preliminary...

Bug 2065: colons in sequence names become :  This does not address
the problem sufficiently in the future, but it does address the problem
for existing data.

lonnavmaps are now maximally lazy; no processing is performed until it
is needed, except in cases where it makes sense to grab all data for the
course at a time.

As a result, the Apache::lonnavmaps::navmap->new() call no longer needs
any arguments; it assumes you're using the current user's navmaps and
generates the other data as needed.

Remove the requirement to pass in the nav hash and the param hash;
nobody ever does it and I'm pretty sure it will never be useful. (If it
is, it can be easily added back in as an optional parameter later;
meanwhile it just confuses things.)

Make sure get_sequence_assessment_data gives a title, even if there is none
returned by the navmaps resource object.
Trivial whitespace changes.

Bug 1638: Problem statistics stored in metadata again.

Bug 1497 - random crashes in spreadsheet caused by loncoursedata.pm.
The error was due to some unexpected return data being passed to
&get_current_state from &ensure_current_data.  Now explicitly return
undef for the second parameter if appropriate.  Extra code has been put in
&get_current_state to prevent it from trying to cast undef as a hash.

Address to some degree the bug we saw today with instructors getting the
white page of death when they view stats or chart.  Now the white page
of death does not appear but the chart and stats think there are no
resources in the course.

Comment out some debugging info.

Title for course level sequence was not being returned by the $top_level_map.
Set it explicitly to course description.

Towards Bug 1251 - get_sequence_assessment_data keeps track of randompick.

Fix asinine bug I wrote.  This bug caused an 'excused' problem status to
not be stored in the cache.

Address Bug 1538 (read it for a description) in two ways by
   a) deleting ALL the tables associated with a cache if any one is missing.
   b) storing the part as well as the part_id in the performance table.
Thus if the part_id table goes missing we should either remove all the other
tables and begin caching again or, if that does not occur, be able to serve
up all the data anyway.  This is an improvement over the current behavior.

Debugging info to help track down white screen of death from spreadsheet.

Added &GetUserName($name,$domain) to wrap around lonnet calls to find out
a users full name.
There probably should be a 'lonuserdata.pm' file for this sort of thing...

Added 'Clear Caches' button to statistics which results in
&Apache::loncoursedata::delete_caches() being called.  delete_caches()
executes a 'drop table' for every table used to do local caching for
the given course.

Fix for bug 1379 where not all sequences were found in
    &get_sequence_assessment_data().
Fixed bug where user submissions were not quoted properly when being stored
in the MySQL database.

Fix: was returning all data the first time data was requested regardless of
whether a specific symb was requested.

Fix get_current_state and get_student_performance_data to work when requesting
a specific symb.

Modified &get_problem_statistics to return a hash reference.

Take the request parameter out of the navmap->new call.

&get_problem_statistics
Added POD.
Implemented limiting of the stats to selected students.

Removed 'weight' from local caching.  If you have looked at the chart or
stats you will need to run

/home/httpd/perl/cleanup_database.pl --killtime 0

to clean out your old caches.

Removed debugging code from statistics subroutine.

Removed &DownloadClasslist, &DownloadCourseInformation, &ProcessTopResourceMap,
&ProcessClasslist, &ProcessStudentData, &ExtractStudentData,
&CheckDateStampError, &TestCacheData, &DownloadStudentCourseData,
&DownloadStudentCourseDataSeparate, and &CheckForResidualDownload

Local Caching:
Added flags to &get_part_id, &get_symb_id, and &get_student_id to fix a bug
where they automatically added the part/symb/student to the table the first
time they were called instead of reading the table to get the value.
&get_current_state was reduced in size by moving its initialization code
to &setup_table_names.  &setup_table_names was relocated to the tail of the
local caching code.

Statistics:
Added &get_problem_statistics.  It does not currently limit the statistics
to a set of students, but will soon.  Classification of students is also
unimplemented.

Added SQL helper function &execute_SQL_request to encapsulate the
'prepare request, execute request, get returned row' sequence and make
debugging easier.

Local caching:
    Added 'weight' to performance_table and EXT call to get the weight.
    Added timestamp checking for cache devalidation.

The addition of 'weight' (and the EXT call) seem to have slowed the caching
down considerably (it now takes 50% longer to store the data).  The timestamp
checking appears to have had a much more modest effect on data retrieval
( < 0.02 sec/student is my estimate).

get_classlist:
    Added slightly more verbose error logging.

Removed debugging logthis calls.

Fix the 'Cannot initialize course' error.

&get_current_state has been rewritten to use MySQL via lonmysql.pm.  Lots
of comments and POD documentation and a heck of a lot of new code.  Some
of it may even make sense!

&get_current_state() now uses a cache file per student per class instead
of one for the entire course.  Removed old lonnet::logthis() calls as well.

- adding timing info to the popwin, and adding new functionality,
- converted stats to use new interface

Corrects documentation for get_classlist, and adds constants that can
be used to future-proof clients against later changes to the order.

&get_sequence_assessment_data:
    Added 'num_assess_parts' to sequence data structure
    Added 'parts' and 'num_parts' to assessment data structure
    Cleaned up POD a little bit.

&get_sequence_assessment_data() handles top level map better:
   title is set to course description
   symb is set to 'top'
   src is set (which is new) to 'not applicable' (which is stupid)

Remove development code.

Fixed bugs in &get_sequence_assessment_data() which caused it to report the
data for the first item in a sequence instead of the sequence itself.
Fixed related bug which caused it to not report an assessment which appeared
as the first item in a sequence.
Minor indention change.
Some POD additions/subtractions/mangling occured as well.

&get_sequence_assessment_data() now additionally returns array refs holding
the sequences and assessments which appear in the course.

&get_current_state() now uses DIFFERENT keys to store different information
instead of using THE SAME KEY EVERY TIME it stores anything for a student.
Most errors seem to be caused by something between my chair and my keyboard.

Implemented caching scheme for &get_current_state().  This routine has not
been tested for retrieval of data for a single symb.  It has been tested
with the spreadsheet which requests all of the students current data.
The cache file is prepended with the users name and domain in order to
prevent collisions between concurrent instructors.  The use of this function
does not appreciably modify the performance of the spreadsheet.
Added helper function &make_into_hash.

Added loncoursedata::get_current_state, which is currently a simple
wrapper for currentdump.  Changed loncoursedata to use $Apache::lonnet::tmpdir
and changed lonspreadsheet to use loncoursedata::get_current_state.

Added &get_sequence_assessment_data(), which is entirely experimental and
should not be used at this time.

- looks like there are situations in which to_idnumber is supposed to be undefined?

Caching enabled version.

Backdoor disableing of caching, for a 0.6.2 release.
Also no longer treat the string 'Not downloaded' as a number....

Error logs should probably contain actual information instead of
variable names....

try to fix bug 1039
I am not sure it has benn fixed.

Fixed bug #752,
Added total points to each homework set total

Log failed attempts to get students environment.

Fix to use &Apache::lonnet::getsection.

changed &get_classlist to update the classlist if the users information is
available.  Tested on data and appears to work properly.  If the users
information is not found the file is not updated.

Added &get_classlist, which gets the classlist for the current class in
a simple manner and does not bother with caching at all.  It will get
data from a students environment if the classlist does not contain the
needed information.  The data is *not* rewritten to the classlist at this
time.

Rewrite of ProcessFullName to be a little easier to understand and to
strip off leading and trailing whitespace from name components.

- if there is an error getting the class list return nothing.

POD added to &ProcessClasslist describing the fields in the hash.

Should call 'clutter' (which adds /res/ if necessary) instead of adding
it by hand.

Incorrect parameters for CheckForResidualDownload.  This explains a couple
of oddities when using the stop button, which is why the status display
didn't always show up.  When that occurred, it was probably just skipping
the already downloaded data, but would just be downloaded again in the
next step so no data was missed.

- perl 5.005 desn't like the original line

First, fixed a typo in lonstudentassessment.  Also, fixed the updating of
the statistics information if the resource tree changes.  Note, in
student assessment it will show up as a blank space unless an attempt
was made for that problem.

Fixed the problem where lonstudentassessment would not update after the
data had been downloaded once, even though the student had done more
work.  Now it updates properly.  Simply set the value to false instead
of checking to see if it was defined, since I define it if it true.  I
noticed the columns getting off a little now, so I will look at it next

First, was getting a strange warning for doing scalar(split(...)) in
lonpercentage.  Second I think I fixed the updating of symbols in
chart after they have been changed.  I tried it out and it work for
my test case.  We will see if it works for everyone.  The problem
is as I suspected, and because I didn't remove the keys when the
students data was updated it was still using old data.  This
would occur if the data didn't overwrite, but changed supporting
information.

Now, when the refresh button is pressed or you switch to a report, the
student's information datestamp will be check.  It will be downloaded if
the datestamp is newer than the stored datestamp.  There may still be
issues with problem statistics due to the fact that there isn't a way
to signal the update of the statistical cached data.

Now I untie the hash before I try to download the data.

Update all students was not working because the untie that was performed
before the call to DownloadStudentCourseData did nothing because it
was not on the same level as its associated tie :(  Moved ties onto
the same level in DownloadStudentCourseData and lonstatistics::PrepareData.

Fixed some single quote problems in my documentation

First, I think there was a typo in the last submit for loncoursedata.  One
too many paranthesis.  Fixed that.  Moved GetDatestamp function to lonnet,
like Guy wanted a while ago.  Added documentation to loncoursedata.

A few comments added and check to see if $c is defined before using it.

Played a little bit more with the downloading regular expression and
changed error checking for lonnet::get.  There is a problem in lonnet::get
that needs to be fixed, and then statistics will properly catch get errors.

fixed status display for CheckForResidualDownload

Modified regular expression, still needs more work.  Added while each
instead of foreach for the keys in a hash.  That reduced the size
of the httpd to its normal size instead of ballooning.

Fixed the number not updating in the status window for update all students
button.  Also fixed some tieing to hash problems for lonproblemstatistics
and lonproblemanalysis.

Added section selection to problem statistics and problem analysis.
Fixed some divide by zeros in lonproblemstatistics.
Broke up the table of option response problems and gave them the
heading of the sequence in which they lie.

Fixed a part problem

Discriminant factors now work again.

First, added unescaping of $key in lond dump command.
Next, I added a new way to download student course data.  There are now
two functions for storing data, DownloadStudentCourseData and
DownloadStudentCourseDataSeparate.  These two functions base their running
on input parameters.  The option parameters are whether or not to check
the date for downloading, whether or not to store all the dumped data or
extract out the data you want, whether or not to display a status window.
The extracting data parameter will be best utilized if someone adds in the
ability to send a list of what parameters are desired and perhaps some simple
commands to affect how that data is processed, like tries, sum would
sum record the sum of all the tries for a student.  This is just an idea.

Currently, I have all the statistics modules using the extract ability.
This slightly increases in download time, but drastically reduces cache
size.  Possible ideas include pushing the extract to the lond side with a
list of parameter/commands, or even downloading everything to a temp cache,
then extract the necessary data into the cache then removing the temp
cache.  There are lots of other possibilities, which can change the download
time, cache size, and other factors.  Now, only loncoursedata handles the
downloading of data to a hash.

lonstudentassessment was changed slightly to remove ' ' as a link if the
student actually hadn't attempted the problem.

lonproblemanalysis was updated for the new str2hash type functions.  There
are a couple of (cludges/fixes) for it.  Depending on whether or not the
str2hash type functions are changed, these may or may not need to be
updated.

lonproblemstatistics was drastically overhauled.  Most of the processing
was removed.  Now, it just does its few statistics functions and outputs
the table.  Currently, I broke the graph, discussion column, and
discriminant factor columns.  These will be fixed on the next commit soon.
There is also no caching done.  This will also be remedied soon.  The
problem that will need attention with caching is to know when to update
the statistics cached data when a student's course data is updated.

Lastly, I plan to add perhaps a toggle legend display button, another graph
button(percentage correct), a button to send the CSV format(not just display),
and add a toggle button for sorting within a sequence and sorting all
the problems.

Also, I changed the look and feel to be the same as the class list page.
Also, the displaying of sequence headers and child sequences are not
working.  This will be fixed, but thought will be put into how best to
make it look and have a similiar fill for all the table combinations.

Fixed the problem where not all the problems were showing up for BS111.  The
problem was that sequences in the start resource were being skipped.  By
simply adding a next in the process sequence block, it will process a
sequence right away instead of trying to move to the next resource.

Was not handling branches and circle resource paths correctly.  Adding
in checks for already visited sequences and problems seemed to fix it.
I get all the sequences in student assessment for BS111 now.

First, added the parenthesis thing to the GDBM stuff.  Fixed the interface
problem statistics so that the buttons work correctly.  How the data
is interpretted is not finished.

Added some stuff to cache for make link to problem resource.

Fixed the problem gathering the correct data to be used for the analysis.
The problem stemmed mostly to the correct use of the result from ssi.

Added section selection.  I added a new multiselect box for sections.  All
students without a section number or a space or undefined will fall under
the none category.  A list of possible sections for a course will be displayed.
Only students with a section number matching one of the selected sections
will be selectable in the students menu and/or display their report.  If
the currently view student doesn't have the correct section when the section
selection changes, the student selection will revert to no student selected.

Note: To refresh the reports after changing which sections to display, press
the refresh button or any of the other onchange interface controls.

Updating lonstudentassessment to add in features of lonchart.  Now the
data display is like lonchart with formatted columns of data.  The next
step will be to add in the column removal check boxes and a multiselect
box for sections.

removing global variables

Broke lonstatistics up.  Now the bulk of the code that generate the
html pages for the different forms have there own module.  Some minor
modifications were necessary to accomplish this.

Also added default student selected on student assessment page is now
all students.  The full name links on the classlist page now take
you to the student assessment page with that student selected.  I
had forgotten to put the post data in the get_unprocessed_cgi
function call.

Lonchart is now part of lonstatistics.  When the CHRT button is pressed on
the remote control it will take you to the student assessment page.  In
the next commit or two, the student assessment page and lonchart will
gradually merge, or I will make chart its own report.

I added another perl module, lonhtmlcommon which will contain a series of
functions that create controls and data layouts for various different
pieces of information.  This is just a fledgling module.

The biggest feature of this commit is that the downloading of student course
data has changed slightly.  Now, the classlist will contain all students,
but will only display the appropriate ones.  Different reports require
different amounts of data, so some may only download student course data
when selected and some will download it all at once.  The downloading
will first grab the time stamp on the course.db file for the student and
compare it will the cached timestamp for that file.  It will only do a
dump if the data is not yet cached or the timestamp is newer than the one
stored.  Only when the data is processed is the timestamp update, not
when downloaded.  The Download/Process duo of function work hand in hand.
Calling both will be typical even if there was nothing actually downloaded.

The is now a new form, which is a class list with different information.
This list also contain the timestamp for each student, when their course
data was downloaded last.  Having this page is both informational, and
hopefully will mean that it is not necessary for all the other pages, at least
as defaults.

Each report now has a common header.  There is currently some white space,
but it can be filled.  The header has a refresh button, update all student
data button, a selection of forms and a selection of student status.  The
update all student data button will download all students (active and expired)
for the given course.  This allows someone to come to the entry page(
class list) update all the students while they do stuff and come back.
The two selection boxes are special in that they will update the
document when changed.  The onchange attribute was tested on mac,
linux and pc, for the different browsers.  The only noticed difference
was netscape 4.7 executes the javascript if it you select the same thing,
where all others do not.  Also, the same browser causes the first
button in the form to be pressed when the document.form.submit()
javascript is executed for the onchange of the selection box.

There was other cosmetic changes, but were minor.

Gather some extra data about the resource tree in ProcessTopLevelMap.  The
problem title and option response information are needed in lonstatistics.

Removed general functions for downloading and processing of student course
data from lonchart into loncoursedata.  loncoursedata will be usable in
lonstatistics and perhaps lonspreadsheet.