CVS log for loncom/interface/Attic/lonspreadsheet.pm

Up to [LON-CAPA] / loncom / interface

Request diff between arbitrary revisions

Keyword substitution: kv
Default branch: MAIN

Remove lonspreadsheet.pm from interface/.  Replacement is
interface/spreadsheet/lonspreadsheet.pm

Bug #1499: always call "LONCAPAreg" inline while loading, so that no extra
window gets opened if loading is aborted.

Bug #1496: long-running spreadsheet did not register with Remote before
completely loaded. Produces "extra windows".

Fixes bugs 1295 and 753 in unstable release (fixes have already been made
to the lonspreadsheet.pm for version 0.6.2 of LON-CAPA.

Fix two bugs - Bug 1321 (multipart problems don't work with default assess
spreadsheet) and a bug which caused the incorrect spreadsheet to be loaded
at the student level.

- correcting bacport

- matthews fix for sections info being dropped

- backport of 1.174, parameter cascading works now

- silly typo

Created loncommon::decode_user_agent() subroutine with code removed from
lonauth.pm.  The code was modified to not require $r be passed along
ad nauseum.

Created lonhtmlcommon::javascript_nothing which returns an appropriate value
of nothing for use as the first parameter for window.open.  Most notably,
IE on macs get a different version, "'javascript:void(0);'", than everyone
one else, "''".  Of course IE on PC appearantly used to require the former
but now requires the latter.

lonmenu.pm and lonspreadsheet.pm were modified to use javascript_nothing.
lonauth was modified to use decode_user_agent.

- spaces seem to work wel in part numbers, I need to do more testing of it, but it looks good so far

- cascading up the part list now works
- accepts parts with : and - in them

- converted lonspreadsheet to use new mechanism
- removed debugging sleep
- typo in lonproblemanalysis

Trivial changes to give the user just a little more information than
"Starting" while it waits for 1+ minutes (getting metadata) to do the
first students spreadsheet on a complete recalculation.

Added loncoursedata::get_current_state, which is currently a simple
wrapper for currentdump.  Changed loncoursedata to use $Apache::lonnet::tmpdir
and changed lonspreadsheet to use loncoursedata::get_current_state.

Now define temp directory in lonnet.

Reordering of parameters for currentdump.

Changed &Apache::lonnet::currentdump parameters to make a little more sense.

Debugging caching mechanisms.
%loadedcaches and %oldsheets in lonspreadsheet::Spreadsheet were not used,
so they are commented out for now.
Fixed a bug in Spreadsheet::rowlabels which caused it to NEVER return the
rowlabels for the current spreadsheet.  Frankly I am amazed it worked at all
with this bug.
Fixed cache calls in get_student_rowlabels and get_assess_rowlabels so they
actually work (I think is a new bug, not an old one, but I am not sure).
This has a noticable effect on the computation time of the spreadsheet.

$self->rows() no longer returns row '0', the export row.

Changed A_column() to rows() since that is what it was used for anyway.

BIG CHANGES (this commit is 3000 lines longer than the code is :) )
This code has been tested and I believe it produces identical results to
the lonspreadsheet.pm in 0.6.1.
The spreadsheet is now an object, accessed via methods and all that nifty
stuff.
&mask, being a pure function, now memoizes its results.
The HASH and related subroutines have temporarily been removed from the
safe space.  I would like to remove them permenantly but this is unlikely
to be feasable.
&expandnamed and &sett have been moved outside the safe space.
logging and accessor/setter functions have been added.  In
more than a few cases the previous code did
foreach my $cell (keys(%formulas)) {
    next if ($cell !~ /^A(\d+)/);
..
methods have been written to return just the needed cells for the 'A' column
and the template row.
A few debugging routines have been added ($sheet->dump_formulas_to_log() and
$sheet->dump_values_to_log() );
The code has been re-arranged and will be re-arranged further.

Rename dumpcurrent to currentdump
lond: fixed 1D10T bug in code.  $value was used instead of $v (version).
lonspreadsheet: now call lonnet::currentdump, removed debugging code and
                code that was commented out.
lonnet:renamed dumpcurrent to currentdump.  currentdump will now deal
       transparently with old lond's that do not support currentdump.

Added new lond command: dumpcurrent.
Added lonnet::dumpcurrent
Modified spreadsheet.pm to use dumpcurrent to retrieve user data.
Some logging information is active, some is not.  More testing required.

Removed code which appears verbatim in lonnet::restore.  No sense in doing
the same thing twice.

Bug 1086.  $_ contained a cell name ('A67','A94',...) but was not the
cell we needed to refer to here.

Fix another caching problem - the student level caches were not being read
in because the filename was incorrect.

Added a hack to display the row number of the export row.  It works!
Avoid unescaping of spreadsheet values if they contain    This is
to help me write a more visually appealing set of default spreadsheets.
Fix a bug in the output selection box - using 'onChange' instead of
'onClick' because NS4.7 dislikes the latter.

Added 'else' clause to check for successful loading of perviously saved
spreadsheet files.  If we are unable to load the proper spreadsheet file, we now load the default.  If we are unable to load the default, we tell the user.
This change is mostly to make life easier for me as I test new default
spreadsheets.

Stop automatic translation of ' to " on submissions.  This at least allows
the user to enter single quotes, even if they still get turned to " if they
attempt to edit them :(

Changes to fix bug 1031 - no longer print out spreadsheet cells as editable
    when the users role is that of a student.
Omit the Template row when displaying information to the student.
Rename the 'Export' row as 'Summary' because (from the students perspective)
    there is nothing to export the row to.

Multiple bugfixes.
Correct spreadsheet is now loaded for assessment sheets (not just the default)
Spreadsheet filename is now a part of the rowlabel data for 'symb' rowlabels
    and used by all the format_xxx_rowlabel routines.
The symb, title, and spreadsheet filename are now escaped before being
    stored internally for rowlabels of type 'symb'.
The order assessments were display in the student spreadsheet was borken and
    has now been fixed.
The code marked 'I do not think this works anymore' has been removed as has
    the comment because it was correct.
IMPORTANT CHANGE: Changing the default assessment spreadsheet or the default
    student spreadsheet will now expire all student spreadsheets.  This
    makes our caching work better.

Fix bug in storage of multi-line cells in temporary spreadsheets.

Redo encoding by calling HTML::Entities::encode().
Added encoding of all HTML cell output.  This effectively disallows the use
of HTML in the output of the spreadsheet.
Fixed a startup error caused by excessive use of 'my'....
Reworked the way a modified cell is described at the beginning of the
spreadsheet output.

Bugfix for escaping of special characters.  < and > must be translated to
&lt; and &gt; before & is translated to &amp; because the code gets unescaped
by the browser twice.  This current version works for cells like:

if (X0<Z0) {
   "Less";
} else {
   "Greater";
}

Disabled multi-sheet excel output.  Crashes on large classes after taking
an eternity to compute all the spreadsheets.

Fix a bug which caused the links in student B's spreadsheet to be to
student A's assessment spreadsheets.  This required adding the ubiquitous
$sheet as a parameter to all the format_????_rowlabel functions.

Fix - if the student did not have an name information specified the name
output as a link to their student sheet is $sname.'@'.$sdom.  This prevents
empty links.

Another each -> foreach conversion.

Fix bug which caused students spreadsheets to be written into the
recursive excel sheet in the wrong order.

Fix two bugs -

&exportsheet caused bad cache data to be stored away and cache data to be
improperly loaded.  This was caused by the programmer (who shall remain me)
not properly rewriting some of the old code.  I'd like to have fixed this
by making the caching more straightforward but that will have to wait.

&loadstudent - using while(my($key,$value) = each(%hash)) { ... } is
not a good idea if you modify %hash (specifically by adding more keys
to it) inside the loop.  Best to use foreach my $key (keys(%hash)) and
take the (minor) performance hit in this case.

That latter is a horribly insidious bug that will haunt me until the end
of my days.

Fix bug in caching mechanism which stored away calculation time as
'uname:udom:studentcalc:time' instead of 'uname:udom:studentcalc:.time'.
Made handling of non-existant expiration times more explicit.

Fixed bad parameter passing for connection interruption testing.
Fixed bug in sort_indicies which existed prior to my changes today - off
by one in my data.  Also added secondary sorting in student sheet by
title of assessment.
Fixed bug which prevented the display of empty rows despite form.showall
setting.
Implemented empty row skipping for the excel sheet so what appears in the
html version will appear in the excel version.

Fixes Bug 426 by implementing sorting the the student and course
spreadsheet views.  Students are sorted alphabetically by full name
(lastname generation, firstname middlename).
Resources are sorted by containing sequence or symb as required.

Bugfix: 'Hide empty rows' now works.

Added checks to determine connection status with browser so we can abort
the computations if no one will see the resulting spreadsheet.

Added &gettitle to get the title of a spreadsheet.  Changes to the rowlabel
handling to give the map id and the resource id of symbs.  Many changes to
&handler to reorganize the output.  excel spreadsheets now include the export
row of the spreadsheet (but still not the template row).
Many cosmetic changes.

Multi-line cell contents edit window.

Removed logging code.

Many minor changes.  This is a check-in before I start reworking the internal
cache mechanisms.
&updatestudentassesssheet in to three parts -
  &get_student_rowlabels, &get_assess_rowlabels, and what remains of
&updatestudentassesssheet.  May undo this change in the future....
Changed a few foreach (keys %hash) to while (.. each (%hash)).
Some extra logging is currently done, to be removed.
Comments added to &handler.
Changed (probably broke) the caching....

Added "Multi-Sheet Excel" output.  Tested on small class, not on large.
Broke the Excel handling routines into smaller pieces to facilitate this.
Multi-Sheet Excel (recursive excel as it is known internally) is only
available at the course sheet level.

Module now uses Spreadsheet::WriteExcel
csv download now writes to a temp file in the prtspool directory and serves
up a link to the user.
Excel output is now available - &outsheet_excel writes the currently
displayed spreadsheet to an excel file in the prtspool directory and gives the
user a link.
&format_plain_rowlabel has been renamed and rewritten to &format_excel_rowlabel
The output method is now specified with a <select /> form.  html is the
default.
The buttons to insert rows in the spreadsheet have been removed temporarily.

CSV download now works as it used to.
Removed debugging code from html_editable_cell.
Fixed bug which caused cells which had only whitespace as a value to be
uneditable.
Broke &format_rowlabel into
&format_html_rowlabel, &format_csv_rowlabel, and &format_plain_rowlabel
Added meat to &outsheet_csv (look at how short it is!)
Modified &outsheet to call &outsheet_csv or &outsheet_html

Major changes:
Outputting of the spreadsheet is beginning to be broken up into smaller
pieces.  HTML output works as best as I can test it.  The function
&outsheet was added to handle the calling of the proper output function for
the requested target.  Currently &outsheet_html is the only implemented option.
*** This means CSV output is broken in this commit ***  If this affects you,
you are me.
The function &rown has been removed and given a respectful burial.
&getrow was reworked to call &templaterow, &outrowassess, and &outrow.
Each of the functions &templaterow, &outrowassess, and &outrow have been
modified to return a row label (colon seperated string) and an array of
hash pointers which describe the cells (name,formula, and value).
Sorting of the rows of the spreadsheet has been moved to &sort_indicies.
&format_rowlabel was modified to deal with 'template' and 'export' rows
without incident.

Some debugging code has been added (and commented out).

Bug Fix: &mask had an error introduced in it during a minor change -
doing

($v1,$v2)=($f=~/(regexpA)(regexpB)/);
($v3,$v4)=($g=~/(regexpA)(regexpB)/);

is not the same as doing:

$f=~/(regexpA)(regexpB)/;
($v1,$v2)=($1,$2);
$g=~/(regexpA)(regexpB)/;
($v3,$v4)=($1,$2);

Because in the second case if $g is undefined its pattern match does not
occur and the variables $1 and $2 retain their values from the pattern match
on $f.  Perl is rather devious at times....

Changes to &exportsheet and &writesheet.
    Changed calls to lonnet::reply to lonnet::get and lonnet::put.
    Fixed a bug introduced in r 1.120 (not distributed to users) which
    passed a course id as a domain.

Fixes bug which caused some cells to not be exported up from the student
spreadsheet.  &exportdata was modified to not return values of undef.
&loadcourse was modified to use named parameters to pass values to
&exportsheet.  It was also modified to do a if (defined(*value*)) instead of
if (*value).  Some strings appearantly are false, despite being nonzero and
of non-zero length.

Reinstated warning user of bad parameters.

Many minor changes.
Never again should the message 'Bad parameter name' appear.
A few comment changes, some minor formatting changes.
Reworked portions of &updatestudentassesssheet.  Changed hash names from
    'allassess' to 'assesslist' and replaced 'allkeys' with 'parameter_labels'.
Reworked some of &exportrow, more work to come.
&expirationdates now uses lonnet::dump instead of reply.
&cachedsheets now uses lonnet::dump instead of reply.

Fixed bug in csv output of parameters (assessment sheet) which caused every
parameter to have the name '1'.  Someone might have complained....

The bug mentioned in previous commit was due to passing 'usmb' instead of
'usymb' from the student sheet to get to the assessment sheet.  Fixed.

Added &format_rowlabel.
Changed the calling of &setrowlabels
The row labels are no longer a part of the safe space.
The row labels now have formats that must be respected.  See &format_rowlabel
for the code which acts on them.
A few other minor changes.
There is a bug in here somewhere which causes the assessment sheets to not
have any data in them.  Not for general consumption.

Undo of previous changes.  Moving the calculation out of the safe space
caused it to take twice as long.

Moved calculation of the spreadsheet outside of the safe space, with calls
going in.
Changed calling structure of &setformulas slightly.
Removed calls to study in safe space.

Moved &templaterow, &outrow, and &outrowassess outside the safe space.
Rewrote part of &rown to make this work.  Changed the calling structure of
&setconstants to something a little more (Perl) object-like.

Removed safe space function exportrowa.

Now do csv output of student level spreadsheet.
This output is not complete yet since the sheet will sometimes end up
with links inside the csv output.  Doing a recalculation sets it properly
since the labels for each row are determined on loading/calculation and
not on display.

Module-wide changes: There are no longer seperate $safeeval and $sheetdata
entities.  There is only $sheet, which will eventually become an object of
sorts.  Most functions earlier took $safeeval and $sheetdata as parameters
but now only take $sheet.
Many comments added to the variables inside the safe space.  The safe space
has not been changed in this commit, so there are no speedups to show off.
No longer print a student status selection dialog on non-course spreadsheets.
Still have the bug in which not all student export data makes it into the
course sheet.

Cleanups and attempted speedups.
Moved &mask out of the safe space.  The other items probably cannot be moved
easily out of the safe space due to scoping issues.  More on this later.
Modified &updateclasssheet to call loncoursedata::get_classlist.

Fixed idiot bug which caused data to crash.

Note: this version has a lot of changes and has not been thoroughly tested.
Big changes:
    Removed many of the get/set functions.  The variables set are in
	%{$sheetdata}.
    &loadassessment was modified to cache unescaped, hashified data instead
       of caching the results of &reply.
Many functions were changed to use $sheetdata to get/store information about
    the sheets.
&handler was changed to use $sheet and $sheetdata instead of $asheet and
$asheetdata.  Also modified to not call the removed get/set functions.

- YEAH!!!!! Can have values of 0 for weight, I think I got every location
  that a had a version of parmval and modified it to used defined
- additionally I updated some docs in parmset
- Fixes BUG#796 and BUG#199

No longer need to ask for 'resourcedata.db' file every assessment sheet.
Now cache the fact that it did not exist.  This reduces the number of
calls to lonnet::dump to 1 instead of 1 per assessment.

By caching student section data we can reduce the number of calls to
lonnet::dump (via usection) by about 50%.

Backport of fix for 0 bug.

Can now enter 0 in spreadsheet cells and have it show up.

Backport of fix to disallow student editing of spreadsheet.
Pick up changes from 1.100.0.2 (caching bug fix, section = -1 misunderstanding)
because I chose the wrong revision to branch off of :(.

Fixed indentation on loaderror code for some reason.
Sorting is now case insensitive.
Students are no longer allowed to enter perl code in their spreadsheet.
If they attempt to fake it by 'post'ing values, we catch that and ignore it.

Overload protection

Backport of changes to deal with sections eq '-1' not indicating a lack of
enrollment but instead indicating a lack of a section number.

A few changes.
changed $cfn to $coursefilename
commented out code that thought &Apache::lonnet::usection returned -1 if
the user was not in the course.  It returns -1 if the user has no section
listed or if they are not in the course.
Minor reformatting where we get the user data, no code changes at all (yet).
A little more removing &gettype and other calls into ths safe space.

- adding a fix from matthew to get the student look up correct

Added $sheetdata goodness to &updateclasssheet and &updatestudentassesssheet.
Fixed a caching bug which caused caching to only happen on a sheet basis, not
a student basis.  Thus the instructor could only see one students assessment
sheets, regardless of which student she wanted to see.

Finishing up earlier work trying to reduce use of $safeeval->reval().  More
to do.

Indentation and extraordinarily minor logic changes.

Attempt to reduce calls into the safe space by keeping two sets of books.
Added use of $sheetdata and $asheetdata, which required changes to most
functions used outside of the safe space.  This code should be considered
beta (at best).  It runs, presumedly, but I wouldn't use it during a demo.

Many changes that should have been checked in one at a time....
Replaced %v with %sheet_values.
Many indentation and whitespace changes.
Some changes in logic.
No new features have been added, but tests have been done to exercise the
code.  Nevertheless, this should *not* be included in 0.5a.

Added MINPARM and MAXPARM, which return minimum and maximum values associated
with a parameter name.  Invoke as follows: localtime(MINPARM('duedate'));
Note, these functions do regular expression matches of their inputs on the
parameters available in a spreadsheet, not any string substitutioni, as in
the EXPANDSUM function.

Replaced %rl with %rowlabel...  Many attempts to get a grip on the code.
Rewrote &updateclasssheet to call lonnet::dump instead of lonnet::reply.
This pretty much precipitated a complete rewrite of the function.

Expect more of the same.

Starting to implement common header and color scheme for LON-CAPA handlers
(non-content pages).

Instead of <body bgcolor="#...."><h1>... call

   &Apache::loncommon::bodytag(title,[role],[add_body_parms]);

title: what it says in the header
role (OPTIONAL): override role choice
                 ('admin','coordinator','student','author')
add_body_parms: additional parameters to be put into the body tag, for
                example 'onLoad="init();" or stuff

Colors and layout will likely change in the future, including domain
customization, help function calls, (css?)

Fix dumb error - if a requested parameter name matched more than one parameter,
we should return one that matches exactly the requested parameter name.

- i think the last of bug#574

Added student status selector from lonhtmlcommon.pm.

Bug #507
Allows to insert extra lines on top and bottom of spreadsheet.

Internally marked "---n" and "~~~n".

Will always open up columns B-Z for calculations (changes to sett).

share('$@') magically activated the eval error checking in a test program
(thanks Matthew!), but apparently still not here.

Test program:

use strict;
use Safe;

my $stuff=(<<'ENDSTUFF');
my $a=5;
my $b=0;

my $c=eval '$a/$b';
return 'Oops '.$@;
ENDSTUFF

my $se=new Safe;
$se->permit("entereval");
$se->share('$@');

print $se->reval($stuff)."\n";
print eval($stuff)."\n";

Working on better debugging

f{'A'rownumber} has the row info. First characters

!!! message means "error" (for example, "obsolete")
--- 000nnn  to be added to the top
~~~ 000nnn  to be added to the bottom

Fixed countrows in &tmpread - was counting cells before.

Towards bug 507.
Has buttons now, but adds too many rows.

removed send_msg safehole wrapping because send_msg routines have been
removed from the spreadsheet.

Bug 472
Starting version 1.78, spreadsheet would only load default sheets in case of
error. Previous versions also loaded when no spreadsheet filename could
be found, i.e., unless (($fn) && ($fn!~/^error/))

Changed to use get_unprocessed_cgi.  BUG 90.

Added ability in &expandnamed to use [ModuleWeight] instead of
[parameter_0_ModuleWeights].  Added a few comments to &sett.

Removed subroutines to send internal messages.

Added send_crit_msg and send_msg to send critical and normal messages.
Don't use this code yet - it currently sends multiple messages every time
the spreadsheet is recomputed.

Remove debugging code.

A little more POD documentation for user-accessible spreadsheet functions.

Load spreadsheets from resource space now works properly.

Added code to load in a spreadsheet from a file.  This code has not been
successfully tested, but at things don't crash during normal operation.

reply to dump

hash, comma-deliminated list, and unit weight computation functions added,
along with nifty POD descriptions.

gpl header

Systematic replacement of 'map' with 'foreach'.

Default is "show all rows", respects course preferences.

Bad hack to get libraries working with stuff like "resource.13.3_45.7".
The spreadsheet is becoming more obfuscated with every version.

New items for feedback, discussion, etc

Discussion as assessment and _symb in exports

Read other default files

New general items: Evaluation, Feedback, Discussion

CSV output for course list

Hopefully works for "stores" now with response-ids.

Also restore stores correctly for part.responseid

Get parmvals out of response-id labelled stuff

Last change not so cool.

Spreadsheet needs \w-only names for internal calculations

Sort of sorts.

Even less silly, and still far from good.

Less silly, and still not good.

Somewhat silly version.

First attempt on hiding rows.

Not load non-existing parameters.

Experimental named variable expansion

Remove debug output

Loads specific sheets for assessments

Need to save after updatesheet

Part way there to integrate different assessment sheets into student sheet,
load of specific sheet possible.

Devalidate caches before complete recalc

Access control, forced recalc, and level up.

Timestamp incorporated into spreadsheet.

Spelling mistake ...

Correct username and domain from studentsheet to assessment sheets

Additional levels of control on expired sheets

Loads expiration dates

Expiration dates to cooperate with parmset

Now it actually should do all the caching, etc, but it needs major
testing now! Many possible sources for errors. Also, the caches need
to be invalidated.

Adds link to student sheet to class sheet

Caches calculated sheets

Window closed too early

Bugfixes and new default spreadsheets

Bugfix: import of text fields in student and class sheets

Symbolic field references (stores and parameter)

Uses dumped data for students, rather than individual assessments

Symbolic names, progress

Progress report

Memory leak found, might now run into endless loop on classes

Caches updated spreadsheet info

Use varglobs to assign and retrieve variables.

Memory leak - deal with it. Quick fix: at least terminate child when done.

With 200 students, the httpd child grows to 121 MB RAM. Need to find reason.

Undef some variables to maybe gain space

Far less resource consuming, but still eating memory. Could be result of
caching. Currently arbitrary limitation of classlist length.

Runs again on assessment level, or so it seems

Still kaputt, but getting there.

Better kaputt than dysfunctional

DO NOT RUN FOR COMPLETE COURSE - is too resource-intensive, slows down
machine. Need to modify to not calculate lower sheets by simple ssi. Make
sheets more local. Generally clean up.

Make spreadsheet default

Lots of new lines of code, little new functionality

Getting ready to display course lists

Save spreadsheets

Template row output, and workcopies fixed.

Now has row template. Workcopy seems like it is kaputt.

Reads and writes environment from course data

Progress on Student Calc Sheet

Small bug fixes with modified worksheet (reading from wrong file)

Started work on studentassess and ssi

Now gets data from homework problems

Rerouting of output possible

Cache expiration time 2 minutes

Imports parameters (partly tested). Should import resource data (not tested)

Starting to retrieve values in the input column

Can now change work copy of sheet

Table cell background color scheme to better find rows and columns

Now reads actual sheet and gets data

New columns with lower case chars, bug in mask fixed

More stuff, now produces a spreadsheet

Not much new ...

Turned inside out, so that v,t, and f are now local to safeeval. Handler
can now have more than one spreadsheet open.

Now runs as a handler, has routines to read spreadsheet and metadata

Now uses safehole

Spreadsheet/Grades Viewer