--- doc/build/Attic/loncapasqldatabase.html	2001/02/13 12:53:43	1.7
+++ doc/build/Attic/loncapasqldatabase.html	2001/02/20 22:24:45	1.15
@@ -8,7 +8,7 @@
 Scott Harrison
 </P>
 <P>
-Last updated: 02/12/2001
+Last updated: 02/15/2001
 </P>
 <P>
 This file describes issues associated with LON-CAPA
@@ -17,9 +17,11 @@ and a SQL database.
 <H2>Latest HOWTO</H2>
 <P>
 <UL>
+<LI>Important notes
 <LI>Current status of documentation</LI>
 <LI>Current status of implementation</LI>
 <LI>Purpose within LON-CAPA</LI>
+<LI>Dependencies</LI>
 <LI>Installation</LI>
 <LI>Installation from source</LI>
 <LI>Configuration (automated)</LI>
@@ -28,6 +30,77 @@ and a SQL database.
 <LI>Example sections of code relevant to LON-CAPA</LI>
 </UL>
 </P>
+<H2>Important notes</H2>
+<P>
+The current database is implemented assuming a non-adjustable
+architecture involving these data fields (specific to each version
+of a resource).
+<UL>
+<LI>title</LI>
+<LI>author</LI>
+<LI>subject</LI>
+<LI>notes</LI>
+<LI>abstract</LI>
+<LI>mime</LI>
+<LI>language</LI>
+<LI>creationdate</LI>
+<LI>lastrevisiondate</LI>
+<LI>owner</LI>
+<LI>copyright</LI>
+</UL>
+</P>
+<P>
+Security occurs as a function of the user 'www',
+and the permissions on the files in the /usr/local/mysql
+directory.  (These files and directories should
+be 700, 600, 500, 400, etc; not allow access to anyone
+but user 'www'.)
+</P>
+<P>
+These commands create the loncapameta database.
+<PRE>
+mysql> CREATE DATABASE IF NOT EXISTS loncapa;
+mysql> USE loncapa;
+mysql> CREATE TABLE IF NOT EXISTS metadata (title TEXT, author TEXT, subject TEXT, notes TEXT, abstract TEXT, mime TEXT, language TEXT, creationdate TEXT, lastrevisiondate TEXT, owner TEXT, copyright TEXT, FULLTEXT idx_title (title), FULLTEXT idx_author (author), FULLTEXT idx_subject (subject), FULLTEXT idx_notes (notes), FULLTEXT idx_abstract (abstract), FULLTEXT idx_mime (mime), FULLTEXT idx_language (language), FULLTEXT idx_creationdate (creationdate), FULLTEXT idx_lastrevisiondate (lastrevisiondate), FULLTEXT idx_owner (owner), FULLTEXT idx_copyright (copyright)) TYPE=MYISAM;
+mysql> INSERT INTO metadata VALUES ('1','2','3','4','5','6','7','8','9','10','11');
+mysql> SELECT * FROM metadata WHERE title REGEXP "1";
+</PRE>
+</P>
+<P>
+Current time values for things.
+<PRE>
+fenchurch.lite.msu.edu
+Mem:    46812K av,   45632K used,    1180K free,   14756K shrd,    4292K buff
+Swap: 1148608K av,   11260K used, 1137348K free                   13244K cached
+
+Red Hat Linux release 6.2 (Zoot)
+Kernel 2.2.16-3 on an i586
+</PRE>
+<BR>Q: How big are data records in test database? A: on average, 1000 bytes each, medline records from PubMed.
+<BR>Q: How big is the biggest field? A: on average, 838 bytes each
+<BR>Q: How much time to insert 5284 medline records into database?
+A: 600 seconds
+<BR>Q: What about when using "speed-technique" on page 130? A: 689 seconds (weird, eh?)
+<BR>Q: What about REGEXP searching? A: about 1-2 seconds for small fields;
+10 to 20 seconds for REGEXP search on "abstract" field
+<BR>Q: What about FULLTEXT indexing? A: about 6 seconds for abstract field.
+
+</P>
+<P>
+An important quote from the manual:
+<BLOCKQUOTE>
+In MySQL Version 3.23.23 or later, you can also create special FULLTEXT indexes. They are used for full-text search. Only the MyISAM table type supports FULLTEXT indexes. They can be created only from
+VARCHAR and TEXT columns. Indexing always happens over the entire column and partial indexing is not supported. See section 25.2 MySQL Full-text Search for details. 
+</BLOCKQUOTE>
+</P>
+<P>
+I plan on using a MyISAM table type with 11 metadata fields of column 
+type=TEXT.
+</P>
+<P>
+It might be worthwhile to look at /usr/local/mysql/manual.html.
+It is quite in depth.
+</P>
 <H2>Current status of documentation</H2>
 <P>
 I am going to begin documentation by inserting what notes
@@ -142,6 +215,23 @@ THE SOLUTION:
   processes (lonsql's) handle the MySQL database manipulations.
 </PRE>
 </P>
+<H2>Dependencies</H2>
+<P>
+I believe (but am not 100% confident) that the following
+RPMs are necessary (in addition to the current ones
+in rpm_list.txt) to run MySQL.  Basically I discovered these
+dependencies while trying to do external RPM based installs.
+I assume, and sometimes found, that these dependencies apply
+to tarball-based distributions too.  (So to play it on the
+safe side, I am going to include these RPMs as part of the
+core, minimal RPM set.)
+<UL>
+<LI>egcs-1.1.2-30</LI>
+<LI>cpp-1.1.2-30</LI>
+<LI>glibc-devel-2.1.3-15</LI>
+<LI>zlib-devel-1.1.3-6</LI>
+</UL>
+</P>
 <H2>Installation</H2>
 <P>
 Installation of the LON-CAPA SQL database normally occurs
@@ -174,6 +264,253 @@ actual MySQL functionality on the system
 </P>
 <H2>Installation from source</H2>
 <P>
+Note: the mysql site recommends that Linux users install by
+using the MySQL RPMs (MySQL-client, MySQL, MySQL-shared, etc).
+While these RPMs work, I was unsuccessful at integrating
+this RPM-installed database with perl modules from www.cpan.org.
+Hence, I <STRONG>strongly</STRONG> recommend that, when installing
+from "source", MySQL and the perl components be in fact installed
+from their tarballs (.tar.gz, .tgz).  (Perl components, when installed
+from RPMs, also wound up in incorrect locations on the disk.)
+Do not coordinate a source install with externally made RPMs!
+It is, of course, okay to use LON-CAPA RPMs such as LON-CAPA-systemperl
+and LON-CAPA-mysql since we, in fact, made these RPMs correctly :).
+<UL>
+<LI>http://www.cpan.org/authors/id/JWIED/Msql-Mysql-modules-1.2215.tar.gz
+<BR>This tarball Released 20th August 2000
+<LI>http://www.mysql.com/Downloads/MySQL-3.23/mysql-3.23.33-pc-linux-gnu-i686.tar.gz
+<BR>This tarball Last changed 2000-11-11
+<BR>This is actually a binary tarball (as opposed to source code
+that is subsequently compiled).
+<LI>http://www.cpan.org/authors/id/TIMB/DBI-1.14.tar.gz
+<BR>This tarball Released 14th June 2000
+</UL>
+</P>
+<P>So, here is exactly how I installed MySQL-3.23. (Note that all files
+wind up in /usr/local/mysql-3.23.33-pc-linux-gnu-i686 except for
+a link from /usr/local/mysql to /usr/local/mysql-3.23.33-pc-linux-gnu-i686
+and some files involved in system process handling (/etc/rc.d/*/*mysql).
+<PRE>
+(As user=root)
+cd /usr/local/;
+tar xzvf mysql-3.23.33-pc-linux-gnu-i686.tar.gz
+ln -s /usr/local/mysql-3.23.33-pc-linux-gnu-i686 mysql
+cd /usr/local/mysql
+chown -R www /usr/local/mysql/.
+chgrp -R users /usr/local/mysql/.
+chmod -R g-w,g-r,g-x /usr/local/mysql/.
+(probably also want chmod -R a-w,a-r,a-x /usr/local/mysql/.)
+
+Alter <TT>safe_mysqld</TT>
+and <TT>support-files/mysql.server</TT> to use 'localhosts' instead
+of `bin/hostname`.  Also, to use user 'www' instead of 'mysql'.
+
+(These changes could be done with /etc/my.cnf, but
+I think this approach makes sure the database NEVER
+gets screwed up due to somebody forgetting to install /etc/my.cnf).
+
+Change this line in mysql.server from
+  pid_file=$datadir/`@HOSTNAME@`.pid
+to
+  pid_file=$datadir/localhost.pid
+
+Change this line in safe_mysqld from
+user=root
+to
+user=www
+
+Change this line in safe_mysqld from
+  pid_file=$DATADIR/`/bin/hostname`.pid
+to
+  pid_file=$DATADIR/localhost.pid
+
+Change this line in safe_mysqld from
+test -z "$err_log"  && err_log=$DATADIR/`/bin/hostname`.err
+to
+test -z "$err_log"  && err_log=$DATADIR/localhost.err
+
+cp -p support-files/mysql.server /etc/rc.d/init.d/mysql
+chmod 755 /etc/rc.d/init.d/mysql
+/sbin/chkconfig --add mysql
+/etc/rc.d/init.d/mysql start
+
+</PRE>
+</P>
+<P>This is how I installed the Msql-Mysql-modules perl modules.
+<PRE>
+[root@fenchurch Msql-Mysql-modules-1.2215]# perl Makefile.PL 
+Which drivers do you want to install?
+
+    1)	MySQL only
+    2)	mSQL only (either of mSQL 1 or mSQL 2)
+    3)  MySQL and mSQL (either of mSQL 1 or mSQL 2)
+
+    4)  mSQL 1 and mSQL 2
+    5)  MySQL, mSQL 1 and mSQL 2
+
+Enter the appropriate number:  [3] 1
+
+
+Do you want to install the MysqlPerl emulation? You might keep your old
+Mysql module (to be distinguished from DBD::mysql!) if you are concerned
+about compatibility to existing applications! [y] n
+Where is your MySQL installed? Please tell me the directory that
+contains the subdir 'include'. [/usr/local/mysql] 
+Which database should I use for testing the MySQL drivers? [test] 
+On which host is database test running (hostname, ip address
+or host:port) [localhost] 
+User name for connecting to database test? [undef]
+Password for connecting to database test? [undef]
+[root@fenchurch Msql-Mysql-modules-1.2215]# make
+[root@fenchurch Msql-Mysql-modules-1.2215]# make test
+make[1]: Entering directory `/home/user/Msql-Mysql-modules-1.2215/mysql'
+make[1]: Leaving directory `/home/user/Msql-Mysql-modules-1.2215/mysql'
+make[1]: Entering directory `/home/user/Msql-Mysql-modules-1.2215/mysql'
+PERL_DL_NONLAZY=1 /usr/bin/perl -I../blib/arch -I../blib/lib -I/usr/lib/perl5/5.00503/i386-linux -I/usr/lib/perl5/5.00503 -e 'use Test::Harness qw(&runtests $verbose); $verbose=0; runtests @ARGV;' t/*.t
+t/00base............ok
+t/10dsnlist.........ok
+t/20createdrop......ok
+t/30insertfetch.....ok
+t/40bindparam.......ok
+t/40blobs...........ok
+t/40listfields......ok
+t/40nulls...........ok
+t/40numrows.........ok
+t/50chopblanks......ok
+t/50commit..........ok
+t/60leaks...........skipping test on this platform
+t/ak-dbd............ok
+t/akmisc............ok
+t/dbdadmin..........ok
+t/mysql.............ok
+t/mysql2............ok
+All tests successful, 1 test skipped.
+Files=17,  Tests=732, 40 wallclock secs (15.38 cusr +  1.30 csys = 16.68 CPU)
+[root@fenchurch Msql-Mysql-modules-1.2215]# make install
+
+These files are installed.
+/usr/bin/dbimon
+/usr/lib/perl5/man/man3/Bundle::DBD::mysql.3
+/usr/lib/perl5/man/man3/DBD::mysql.3
+/usr/lib/perl5/man/man3/Mysql.3
+/usr/lib/perl5/site_perl/5.005/i386-linux/Bundle/DBD/mysql.pm
+/usr/lib/perl5/site_perl/5.005/i386-linux/DBD/mysql.pm
+/usr/lib/perl5/site_perl/5.005/i386-linux/Mysql.pm
+/usr/lib/perl5/site_perl/5.005/i386-linux/Mysql/Statement.pm
+/usr/lib/perl5/site_perl/5.005/i386-linux/auto/DBD/mysql/mysql.bs
+/usr/lib/perl5/site_perl/5.005/i386-linux/auto/DBD/mysql/mysql.so
+/usr/man/man1/dbimon.1
+/usr/lib/perl5/site_perl/5.005/i386-linux/auto/Msql-Mysql-modules/.packlist
+</PRE>
+</P>
+<P>
+This is how I installed the DBI perl modules.
+<PRE>$dbh
+[root@fenchurch DBI-1.14]# perl Makefile.PL
+*** Note:
+    The optional PlRPC-modules (RPC::PlServer etc) are not installed.
+    If you want to use the DBD::Proxy driver and DBI::ProxyServer
+    modules, then you'll need to install the RPC::PlServer, RPC::PlClient,
+    Storable and Net::Daemon modules. The CPAN Bundle::DBI may help you.
+    You can install them any time after installing the DBI.
+    You do *not* need these modules for typical DBI usage.
+
+Optional modules are available from any CPAN mirror, in particular
+    http://www.perl.com/CPAN/modules/by-module
+    http://www.perl.org/CPAN/modules/by-module
+    ftp://ftp.funet.fi/pub/languages/perl/CPAN/modules/by-module
+
+Checking if your kit is complete...
+Looks good
+Writing Makefile for DBI
+
+    Remember to actually *read* the README file!
+    Use  'make' to build the software (dmake or nmake on Windows).
+    Then 'make test' to execute self tests.
+    Then 'make install' to install the DBI and then delete this working
+    directory before unpacking and building any DBD::* drivers.
+
+[root@fenchurch DBI-1.14]# make
+[root@fenchurch DBI-1.14]# make test
+PERL_DL_NONLAZY=1 /usr/bin/perl -Iblib/arch -Iblib/lib -I/usr/lib/perl5/5.00503/i386-linux -I/usr/lib/perl5/5.00503 -e 'use Test::Harness qw(&runtests $verbose); $verbose=0; runtests @ARGV;' t/*.t
+t/basics............ok
+t/dbidrv............ok
+t/examp.............ok
+t/meta..............ok
+t/proxy.............skipping test on this platform
+t/shell.............ok
+t/subclass..........ok
+All tests successful, 1 test skipped.
+Files=7,  Tests=179,  7 wallclock secs ( 6.46 cusr +  0.49 csys =  6.95 CPU)
+PERL_DL_NONLAZY=1 /usr/bin/perl -Iblib/arch -Iblib/lib -I/usr/lib/perl5/5.00503/i386-linux -I/usr/lib/perl5/5.00503 test.pl
+test.pl 
+DBI test application $Revision: 1.15 $
+Using /home/user/DBI-1.14/blib
+Switch: DBI 1.14 by Tim Bunce, 1.14
+Available Drivers: ADO, ExampleP, Multiplex, Proxy, mysql
+dbi:ExampleP:: testing 5 sets of 20 connections:
+Connecting... 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 
+Disconnecting...
+Connecting... 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 
+Disconnecting...
+Connecting... 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 
+Disconnecting...
+Connecting... 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 
+Disconnecting...
+Connecting... 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 
+Disconnecting...
+Made 100 connections in  0 wallclock secs ( 0.22 usr +  0.03 sys =  0.25 CPU)
+
+Testing handle creation speed...
+5000 NullP statement handles cycled in 6.6 cpu+sys seconds (762 per sec)
+
+test.pl done
+
+[root@fenchurch DBI-1.14]# make install
+These files are installed.
+/usr/lib/perl5/site_perl/5.005/i386-linux/auto/DBI/.packlist
+/usr/bin/dbiproxy
+/usr/bin/dbish
+/usr/lib/perl5/man/man3/Bundle::DBI.3
+/usr/lib/perl5/man/man3/DBD::ADO.3
+/usr/lib/perl5/man/man3/DBD::Multiplex.3
+/usr/lib/perl5/man/man3/DBD::Proxy.3
+/usr/lib/perl5/man/man3/DBI.3
+/usr/lib/perl5/man/man3/DBI::DBD.3
+/usr/lib/perl5/man/man3/DBI::FAQ.3
+/usr/lib/perl5/man/man3/DBI::Format.3
+/usr/lib/perl5/man/man3/DBI::ProxyServer.3
+/usr/lib/perl5/man/man3/DBI::Shell.3
+/usr/lib/perl5/man/man3/DBI::W32ODBC.3
+/usr/lib/perl5/man/man3/Win32::DBIODBC.3
+/usr/lib/perl5/site_perl/5.005/i386-linux/Bundle/DBI.pm
+/usr/lib/perl5/site_perl/5.005/i386-linux/DBD/ADO.pm
+/usr/lib/perl5/site_perl/5.005/i386-linux/DBD/ExampleP.pm
+/usr/lib/perl5/site_perl/5.005/i386-linux/DBD/Multiplex.pm
+/usr/lib/perl5/site_perl/5.005/i386-linux/DBD/NullP.pm
+/usr/lib/perl5/site_perl/5.005/i386-linux/DBD/Proxy.pm
+/usr/lib/perl5/site_perl/5.005/i386-linux/DBD/Sponge.pm
+/usr/lib/perl5/site_perl/5.005/i386-linux/DBI.pm
+/usr/lib/perl5/site_perl/5.005/i386-linux/DBI/DBD.pm
+/usr/lib/perl5/site_perl/5.005/i386-linux/DBI/FAQ.pm
+/usr/lib/perl5/site_perl/5.005/i386-linux/DBI/Format.pm
+/usr/lib/perl5/site_perl/5.005/i386-linux/DBI/ProxyServer.pm
+/usr/lib/perl5/site_perl/5.005/i386-linux/DBI/Shell.pm
+/usr/lib/perl5/site_perl/5.005/i386-linux/DBI/W32ODBC.pm
+/usr/lib/perl5/site_perl/5.005/i386-linux/Win32/DBIODBC.pm
+/usr/lib/perl5/site_perl/5.005/i386-linux/auto/DBI/DBI.bs
+/usr/lib/perl5/site_perl/5.005/i386-linux/auto/DBI/DBI.so
+/usr/lib/perl5/site_perl/5.005/i386-linux/auto/DBI/DBIXS.h
+/usr/lib/perl5/site_perl/5.005/i386-linux/auto/DBI/Driver.xst
+/usr/lib/perl5/site_perl/5.005/i386-linux/auto/DBI/dbd_xsh.h
+/usr/lib/perl5/site_perl/5.005/i386-linux/auto/DBI/dbi_sql.h
+/usr/lib/perl5/site_perl/5.005/i386-linux/auto/DBI/dbipport.h
+/usr/man/man1/dbiproxy.1
+/usr/man/man1/dbish.1
+</PRE>
+</P>
+<FONT COLOR="green"> old notes in green
+<P>
 The following set of tarballs was found to work together
 properly on a LON-CAPA RedHat 6.2 system:
 <UL>
@@ -186,6 +523,7 @@ properly on a LON-CAPA RedHat 6.2 system
 Installation was simply a matter of following the instructions
 and typing the several "make" commands for each 
 </P>
+</FONT>
 <H2>Configuration (automated)</H2>
 <P>
 Not yet developed.  This will be part of an interface
@@ -297,6 +635,21 @@ An up-to-date lond and lonsql.
 </P>
 <H2>Testing</H2>
 <P>
+To test the backend MySQL database, a number of commands should be
+run after installation.
+<UL>
+<LI><TT>cd /usr/local/mysql/sql-bench; ./run-all-tests --small-test</TT></LI>
+<BR>without the --small-test flag, this test can take more than 10 hours!
+<LI><TT>cd /usr/local/mysql; bin/mysqladmin version</TT></LI>
+<LI><TT>cd /usr/local/mysql; bin/mysqladmin variables</TT></LI>
+<LI><TT>cd /usr/local/mysql; bin/mysqlshow</TT></LI>
+<LI><TT>cd /usr/local/mysql; bin/mysqlshow mysql</TT></LI>
+<LI><TT>cd /usr/local/mysql; bin/mysql -e "select host,db,user from db" mysql</TT></LI>
+<LI><TT>cd /usr/local/mysql/mysql-test; ./test-run-all</TT></LI>
+
+</UL>
+<P>
+These are sections of perl code which helps test the LON-CAPA network.
 <PRE>
 <STRONG>** TEST the database connection with my current tester.pl code
 which mimics what command will eventually be sent through lonc.</STRONG>