[BACK]Return to loncapasqldatabase.html CVS log [TXT] [DIR] Up to [LON-CAPA] / doc / build

Diff for /doc/build/Attic/loncapasqldatabase.html between versions 1.2 and 1.18

version 1.2, 2001/02/07 13:02:38 version 1.18, 2002/09/23 01:25:09
Line 8 Line 8
 Scott Harrison  Scott Harrison
 </P>  </P>
 <P>  <P>
 Last updated: 02/07/2001  Last updated: 02/15/2001
 </P>  </P>
 <P>  <P>
 This file describes issues associated with LON-CAPA  This file describes issues associated with LON-CAPA
 and a SQL database.  and a SQL database.
 </P>  </P>
 <H3>Latest HOWTO</H3>  <H2>Latest HOWTO</H2>
   <P>
   <UL>
   <LI>Important notes
   <LI>Current status of documentation</LI>
   <LI>Current status of implementation</LI>
   <LI>Purpose within LON-CAPA</LI>
   <LI>Dependencies</LI>
   <LI>Installation</LI>
   <LI>Installation from source</LI>
   <LI>Configuration (automated)</LI>
   <LI>Manual configuration</LI>
   <LI>Testing</LI>
   <LI>Example sections of code relevant to LON-CAPA</LI>
   </UL>
   </P>
   <H2>Important notes</H2>
   <P>
   The current database is implemented assuming a non-adjustable
   architecture involving these data fields (specific to each version
   of a resource).
   <UL>
   <LI>title</LI>
   <LI>author</LI>
   <LI>subject</LI>
   <LI>notes</LI>
   <LI>abstract</LI>
   <LI>mime</LI>
   <LI>language</LI>
   <LI>creationdate</LI>
   <LI>lastrevisiondate</LI>
   <LI>owner</LI>
   <LI>copyright</LI>
   </UL>
   </P>
   <P>
   Security occurs as a function of the user 'www',
   and the permissions on the files in the /usr/local/mysql
   directory.  (These files and directories should
   be 700, 600, 500, 400, etc; not allow access to anyone
   but user 'www'.)
   </P>
   <P>
   These commands create the loncapameta database.
   <PRE>
   mysql> CREATE DATABASE IF NOT EXISTS loncapa;
   mysql> USE loncapa;
   mysql> CREATE TABLE IF NOT EXISTS metadata (title TEXT, author TEXT, subject TEXT, url TEXT, keywords TEXT, version TEXT, notes TEXT, abstract TEXT, mime TEXT, language TEXT, creationdate DATETIME, lastrevisiondate DATETIME, owner TEXT, copyright TEXT, FULLTEXT idx_title (title), FULLTEXT idx_author (author), FULLTEXT idx_subject (subject), FULLTEXT idx_url (url), FULLTEXT idx_keywords (keywords), FULLTEXT idx_version (version), FULLTEXT idx_notes (notes), FULLTEXT idx_abstract (abstract), FULLTEXT idx_mime (mime), FULLTEXT idx_language (language), FULLTEXT idx_owner (owner), FULLTEXT idx_copyright (copyright)) TYPE=MYISAM;
   mysql> INSERT INTO metadata VALUES ('The Structure of Scientific Revolutions','Thomas S. Kuhn','scientific philosophy','/res/msu/shh1/poobah2.html','aphorisms, theories, paradigm, revolution','current','still developing','This famous book stands out in contrast to the more rigid theories of Popper.','html','seniso','1999-03-03 12:34:56','1999-03-03 3:12:00','shh1@msu.edu','default');
   mysql> SELECT * FROM metadata WHERE title REGEXP "1";
   </PRE>
   </P>
   <P>
   Current time values for things.
   <PRE>
   fenchurch.lite.msu.edu
   Mem:    46812K av,   45632K used,    1180K free,   14756K shrd,    4292K buff
   Swap: 1148608K av,   11260K used, 1137348K free                   13244K cached
   
   Red Hat Linux release 6.2 (Zoot)
   Kernel 2.2.16-3 on an i586
   </PRE>
   <BR>Q: How big are data records in test database? A: on average, 1000 bytes each, medline records from PubMed.
   <BR>Q: How big is the biggest field? A: on average, 838 bytes each
   <BR>Q: How much time to insert 5284 medline records into database?
   A: 600 seconds
   <BR>Q: What about when using "speed-technique" on page 130? A: 689 seconds (weird, eh?)
   <BR>Q: What about REGEXP searching? A: about 1-2 seconds for small fields;
   10 to 20 seconds for REGEXP search on "abstract" field
   <BR>Q: What about FULLTEXT indexing? A: about 6 seconds for abstract field.
   
   </P>
   <P>
   An important quote from the manual:
   <BLOCKQUOTE>
   In MySQL Version 3.23.23 or later, you can also create special FULLTEXT indexes. They are used for full-text search. Only the MyISAM table type supports FULLTEXT indexes. They can be created only from
   VARCHAR and TEXT columns. Indexing always happens over the entire column and partial indexing is not supported. See section 25.2 MySQL Full-text Search for details. 
   </BLOCKQUOTE>
   </P>
   <P>
   I plan on using a MyISAM table type with 11 metadata fields of column 
   type=TEXT.
   </P>
   <P>
   It might be worthwhile to look at /usr/local/mysql/manual.html.
   It is quite in depth.
   </P>
   <H2>Current status of documentation</H2>
 <P>  <P>
 I am going to begin documentation by inserting what notes  I am going to begin documentation by inserting what notes
 I have into this file.  I will be subsequently rearranging  I have into this file.  I will be subsequently rearranging
 them and editting them based on the tests that I conduct.  them and editing them based on the tests that I conduct.
 I am trying to make sure that documentation, installation,  I am trying to make sure that documentation, installation,
 and run-time issues are all consistent and correct.  The  and run-time issues are all consistent and correct.  The
 current status of everything is that it works and has  current status of everything is that it works and has
 been minimally tested, but things need to be cleaned up  been minimally tested, but things need to be cleaned up
 and checked again!  and checked again!
 </P>  </P>
 <H3>How to add a user to the SQL database</H3>  <H2>Current status of implementation</H2>
   <P>
   Need to
   <UL>
   <LI>Installation: Fix binary file listings for user permissions and ownership.
   <LI>Installation: Make sure sql server starts, and if database does not
   exist, then create. (/etc/rc.d).
   <LI>Processes: Make sure loncron initiates lonsql on library machines.
   <LI>Read in metadata from right place periodically.
   <LI>Implement tested perl module handler.
   </UL>
   <P>
   Right now, a lot of "feasibility" work has been done.
   Recipes for manual installation and configuration have
   been gathered.  Network connectivity of lond->lonsql->lond->lonc
   type tests have been performed.  A binary installation
   has been compiled in an RPM (LON-CAPA-mysql, with perl components
   a part of LON-CAPA-systemperl).
   The most lacking test in terms of feasibility has
   been looking at benchmarks to analyze the load at which
   the SQL database can efficiently allow many users to
   make simultaneous requests of the metadata database.
   </P>
   <P>
   Documentation has been pieced together over time.  But,
   as mentioned in the previous section, it needs an
   overhaul.
   </P>
   <P>
   The binary installation has some quirks associated with it.
   Some of the user permissions are wrong, although this is
   benign.  Also, other options of binary installation (such
   as using binary RPMs put together by others) were dismissed
   given the difficulty of getting differing combinations of
   these external RPMs to work together.
   </P>
   <P>
   Most configuration questions have been initially worked out
   to the point of getting this SQL software component working,
   however there may be more optimal approaches than currently
   exist.
   </P>
   <H2>Purpose within LON-CAPA</H2>
   <P>
   LON-CAPA is meant to distribute A LOT of educational content
   to A LOT of people.  It is ineffective to directly rely on contents
   within the ext2 filesystem to be speedily scanned for 
   on-the-fly searches of content descriptions.  (Simply put,
   it takes a cumbersome amount of time to open, read, analyze, and
   close thousands of files.)
   </P>
   <P>
   The solution is to hash-index various data fields that are
   descriptive of the educational resources on a LON-CAPA server
   machine.  Descriptive data fields are referred to as
   "metadata".  The question then arises as to how this metadata
   is handled in terms of the rest of the LON-CAPA network
   without burdening client and daemon processes.  I now
   answer this question in the format of Problem and Solution
   below.
   </P>
 <P>  <P>
 <PRE>  <PRE>
 start the mysql daemon as /usr/local/bin/safe_mysqld &  PROBLEM SITUATION:
 Login as root: mysql -u root -p mysql  
 enter the password as newmysql  
 add the user www: grant all priveleges on *.* to www@localhost identified by 'newmysql' with grant option;  
   
 INSERT INTO user (Host, User, Password)    If Server A wants data from Server B, Server A uses a lonc process to
 VALUES ('localhost','www',password('newmysql'));    send a database command to a Server B lond process.
       lonc= loncapa client process    A-lonc= a lonc process on Server A
       lond= loncapa daemon process
   
 GRANT ALL PRIVILEGES ON *.* TO www@localhost;                   database command
       A-lonc  --------TCP/IP----------------> B-lond
   
 FLUSH PRIVILEGES;    The problem emerges that A-lonc and B-lond are kept waiting for the
     MySQL server to "do its stuff", or in other words, perform the conceivably
     sophisticated, data-intensive, time-sucking database transaction.  By tying
     up a lonc and lond process, this significantly cripples the capabilities
     of LON-CAPA servers. 
   
     While commercial databases have a variety of features that ATTEMPT to
     deal with this, freeware databases are still experimenting and exploring
     with different schemes with varying degrees of performance stability.
   
 Here the user www has the right to grant privileges to other users.  THE SOLUTION:
 This can be changed if required with a simple update command on the grant tables  
     A separate daemon process was created that B-lond works with to
     handle database requests.  This daemon process is called "lonsql".
   
     So,
                   database command
     A-lonc  ---------TCP/IP-----------------> B-lond =====> B-lonsql
            <---------------------------------/                |
              "ok, I'll get back to you..."                    |
                                                               |
                                                               /
     A-lond  <-------------------------------  B-lonc   <======
              "Guess what? I have the result!"
   
 /home/httpd/perl/perlsql/lonsql    Of course, depending on success or failure, the messages may vary,
 /usr/local/mysql/fakeclient    but the principle remains the same where a separate pool of children
     processes (lonsql's) handle the MySQL database manipulations.
 </PRE>  </PRE>
 </P>  </P>
 <H3>To do</H3>  <H2>Dependencies</H2>
   <P>
   I believe (but am not 100% confident) that the following
   RPMs are necessary (in addition to the current ones
   in rpm_list.txt) to run MySQL.  Basically I discovered these
   dependencies while trying to do external RPM based installs.
   I assume, and sometimes found, that these dependencies apply
   to tarball-based distributions too.  (So to play it on the
   safe side, I am going to include these RPMs as part of the
   core, minimal RPM set.)
   <UL>
   <LI>egcs-1.1.2-30</LI>
   <LI>cpp-1.1.2-30</LI>
   <LI>glibc-devel-2.1.3-15</LI>
   <LI>zlib-devel-1.1.3-6</LI>
   </UL>
   </P>
   <H2>Installation</H2>
 <P>  <P>
   Installation of the LON-CAPA SQL database normally occurs
   by default when using the LON-CAPA installation CD
   (see http://install.lon-capa.org).  It is installed
   as the LON-CAPA-mysql RPM.  This RPM encodes for the MySQL
   engine.  Related perl interfaces (Perl::DBI, Perl::Msql-Mysql)
   are encoded in the LON-CAPA-systemperl RPM.
   </P>
   <P>
   The three components of a MySQL installation for the
   LON-CAPA system are further described immediately below.
   <TABLE BORDER="0">
   <TR><TD COLSPAN="2"><STRONG>Perl::DBI module</STRONG>-
   the API "front-end"...</TD></TR>
   <TR><TD WIDTH="10%"></TD><TD>database interface module for organizing generic
   database commands which are independent of specific
   database implementation (such as MySQL, mSQL, Postgres, etc).
   </TD></TR>
   <TR><TD COLSPAN="2"><STRONG>Perl::MySQL module</STRONG>-
   the API "mid-section"...</TD></TR>
   <TR><TD WIDTH="10%"></TD><TD>the module to directly interface with the actual
   MySQL database engine</TD></TR>
   <TR><TD COLSPAN="2"><STRONG>MySQL database engine</STRONG>-
   the "back-end"...</TD></TR>
   <TR><TD WIDTH="10%"></TD><TD>the binary installation (compiled either
   from source or pre-compiled file listings) which provides the
   actual MySQL functionality on the system</TD></TR>
   </TABLE>
   </P>
   <H2>Installation from source</H2>
   <P>
   Note: the mysql site recommends that Linux users install by
   using the MySQL RPMs (MySQL-client, MySQL, MySQL-shared, etc).
   While these RPMs work, I was unsuccessful at integrating
   this RPM-installed database with perl modules from www.cpan.org.
   Hence, I <STRONG>strongly</STRONG> recommend that, when installing
   from "source", MySQL and the perl components be in fact installed
   from their tarballs (.tar.gz, .tgz).  (Perl components, when installed
   from RPMs, also wound up in incorrect locations on the disk.)
   Do not coordinate a source install with externally made RPMs!
   It is, of course, okay to use LON-CAPA RPMs such as LON-CAPA-systemperl
   and LON-CAPA-mysql since we, in fact, made these RPMs correctly :).
   <UL>
   <LI>http://www.cpan.org/authors/id/JWIED/Msql-Mysql-modules-1.2215.tar.gz
   <BR>This tarball Released 20th August 2000
   <LI>http://www.mysql.com/Downloads/MySQL-3.23/mysql-3.23.33-pc-linux-gnu-i686.tar.gz
   <BR>This tarball Last changed 2000-11-11
   <BR>This is actually a binary tarball (as opposed to source code
   that is subsequently compiled).
   <LI>http://www.cpan.org/authors/id/TIMB/DBI-1.14.tar.gz
   <BR>This tarball Released 14th June 2000
   </UL>
   </P>
   <P>So, here is exactly how I installed MySQL-3.23. (Note that all files
   wind up in /usr/local/mysql-3.23.33-pc-linux-gnu-i686 except for
   a link from /usr/local/mysql to /usr/local/mysql-3.23.33-pc-linux-gnu-i686
   and some files involved in system process handling (/etc/rc.d/*/*mysql).
 <PRE>  <PRE>
 This is the output from scripts/mysql_install_db...  (As user=root)
 still some todo things (like support-files/mysql.server)  cd /usr/local/;
   tar xzvf mysql-3.23.33-pc-linux-gnu-i686.tar.gz
 Creating db table  ln -s /usr/local/mysql-3.23.33-pc-linux-gnu-i686 mysql
 Creating host table  cd /usr/local/mysql
 Creating user table  chown -R www /usr/local/mysql/.
 Creating func table  chgrp -R users /usr/local/mysql/.
 Creating tables_priv table  chmod -R g-w,g-r,g-x /usr/local/mysql/.
 Creating columns_priv table  (probably also want chmod -R a-w,a-r,a-x /usr/local/mysql/.)
   
   Alter <TT>safe_mysqld</TT>
   and <TT>support-files/mysql.server</TT> to use 'localhosts' instead
   of `bin/hostname`.  Also, to use user 'www' instead of 'mysql'.
   
   (These changes could be done with /etc/my.cnf, but
   I think this approach makes sure the database NEVER
   gets screwed up due to somebody forgetting to install /etc/my.cnf).
   
   Change this line in mysql.server from
     pid_file=$datadir/`@HOSTNAME@`.pid
   to
     pid_file=$datadir/localhost.pid
   
   Change this line in safe_mysqld from
   user=root
   to
   user=www
   
   Change this line in safe_mysqld from
     pid_file=$DATADIR/`/bin/hostname`.pid
   to
     pid_file=$DATADIR/localhost.pid
   
   Change this line in safe_mysqld from
   test -z "$err_log"  && err_log=$DATADIR/`/bin/hostname`.err
   to
   test -z "$err_log"  && err_log=$DATADIR/localhost.err
   
   cp -p support-files/mysql.server /etc/rc.d/init.d/mysql
   chmod 755 /etc/rc.d/init.d/mysql
   /sbin/chkconfig --add mysql
   /etc/rc.d/init.d/mysql start
   
 To start mysqld at boot time you have to copy support-files/mysql.server  </PRE>
 to the right place for your system  </P>
   <P>This is how I installed the Msql-Mysql-modules perl modules.
 PLEASE REMEMBER TO SET A PASSWORD FOR THE MySQL root USER !  <PRE>
 This is done with:  [root@fenchurch Msql-Mysql-modules-1.2215]# perl Makefile.PL 
 /usr/local/bin/mysqladmin -u root password 'new-password'  Which drivers do you want to install?
 See the manual for more instructions.  
   
 Please report any problems with the /usr/local/bin/mysqlbug script!  
   
 The latest information about MySQL is available on the web at http://www.mysql.com      1) MySQL only
 Support MySQL by buying support/licenses at http://www.tcx.se/license.htmy.      2) mSQL only (either of mSQL 1 or mSQL 2)
       3)  MySQL and mSQL (either of mSQL 1 or mSQL 2)
   
       4)  mSQL 1 and mSQL 2
       5)  MySQL, mSQL 1 and mSQL 2
   
   Enter the appropriate number:  [3] 1
   
   
   Do you want to install the MysqlPerl emulation? You might keep your old
   Mysql module (to be distinguished from DBD::mysql!) if you are concerned
   about compatibility to existing applications! [y] n
   Where is your MySQL installed? Please tell me the directory that
   contains the subdir 'include'. [/usr/local/mysql] 
   Which database should I use for testing the MySQL drivers? [test] 
   On which host is database test running (hostname, ip address
   or host:port) [localhost] 
   User name for connecting to database test? [undef]
   Password for connecting to database test? [undef]
   [root@fenchurch Msql-Mysql-modules-1.2215]# make
   [root@fenchurch Msql-Mysql-modules-1.2215]# make test
   make[1]: Entering directory `/home/user/Msql-Mysql-modules-1.2215/mysql'
   make[1]: Leaving directory `/home/user/Msql-Mysql-modules-1.2215/mysql'
   make[1]: Entering directory `/home/user/Msql-Mysql-modules-1.2215/mysql'
   PERL_DL_NONLAZY=1 /usr/bin/perl -I../blib/arch -I../blib/lib -I/usr/lib/perl5/5.00503/i386-linux -I/usr/lib/perl5/5.00503 -e 'use Test::Harness qw(&runtests $verbose); $verbose=0; runtests @ARGV;' t/*.t
   t/00base............ok
   t/10dsnlist.........ok
   t/20createdrop......ok
   t/30insertfetch.....ok
   t/40bindparam.......ok
   t/40blobs...........ok
   t/40listfields......ok
   t/40nulls...........ok
   t/40numrows.........ok
   t/50chopblanks......ok
   t/50commit..........ok
   t/60leaks...........skipping test on this platform
   t/ak-dbd............ok
   t/akmisc............ok
   t/dbdadmin..........ok
   t/mysql.............ok
   t/mysql2............ok
   All tests successful, 1 test skipped.
   Files=17,  Tests=732, 40 wallclock secs (15.38 cusr +  1.30 csys = 16.68 CPU)
   [root@fenchurch Msql-Mysql-modules-1.2215]# make install
   
   These files are installed.
   /usr/bin/dbimon
   /usr/lib/perl5/man/man3/Bundle::DBD::mysql.3
   /usr/lib/perl5/man/man3/DBD::mysql.3
   /usr/lib/perl5/man/man3/Mysql.3
   /usr/lib/perl5/site_perl/5.005/i386-linux/Bundle/DBD/mysql.pm
   /usr/lib/perl5/site_perl/5.005/i386-linux/DBD/mysql.pm
   /usr/lib/perl5/site_perl/5.005/i386-linux/Mysql.pm
   /usr/lib/perl5/site_perl/5.005/i386-linux/Mysql/Statement.pm
   /usr/lib/perl5/site_perl/5.005/i386-linux/auto/DBD/mysql/mysql.bs
   /usr/lib/perl5/site_perl/5.005/i386-linux/auto/DBD/mysql/mysql.so
   /usr/man/man1/dbimon.1
   /usr/lib/perl5/site_perl/5.005/i386-linux/auto/Msql-Mysql-modules/.packlist
   </PRE>
   </P>
   <P>
   This is how I installed the DBI perl modules.
   <PRE>$dbh
   [root@fenchurch DBI-1.14]# perl Makefile.PL
   *** Note:
       The optional PlRPC-modules (RPC::PlServer etc) are not installed.
       If you want to use the DBD::Proxy driver and DBI::ProxyServer
       modules, then you'll need to install the RPC::PlServer, RPC::PlClient,
       Storable and Net::Daemon modules. The CPAN Bundle::DBI may help you.
       You can install them any time after installing the DBI.
       You do *not* need these modules for typical DBI usage.
   
   Optional modules are available from any CPAN mirror, in particular
       http://www.perl.com/CPAN/modules/by-module
       http://www.perl.org/CPAN/modules/by-module
       ftp://ftp.funet.fi/pub/languages/perl/CPAN/modules/by-module
   
   Checking if your kit is complete...
   Looks good
   Writing Makefile for DBI
   
       Remember to actually *read* the README file!
       Use  'make' to build the software (dmake or nmake on Windows).
       Then 'make test' to execute self tests.
       Then 'make install' to install the DBI and then delete this working
       directory before unpacking and building any DBD::* drivers.
   
   [root@fenchurch DBI-1.14]# make
   [root@fenchurch DBI-1.14]# make test
   PERL_DL_NONLAZY=1 /usr/bin/perl -Iblib/arch -Iblib/lib -I/usr/lib/perl5/5.00503/i386-linux -I/usr/lib/perl5/5.00503 -e 'use Test::Harness qw(&runtests $verbose); $verbose=0; runtests @ARGV;' t/*.t
   t/basics............ok
   t/dbidrv............ok
   t/examp.............ok
   t/meta..............ok
   t/proxy.............skipping test on this platform
   t/shell.............ok
   t/subclass..........ok
   All tests successful, 1 test skipped.
   Files=7,  Tests=179,  7 wallclock secs ( 6.46 cusr +  0.49 csys =  6.95 CPU)
   PERL_DL_NONLAZY=1 /usr/bin/perl -Iblib/arch -Iblib/lib -I/usr/lib/perl5/5.00503/i386-linux -I/usr/lib/perl5/5.00503 test.pl
   test.pl 
   DBI test application $Revision$
   Using /home/user/DBI-1.14/blib
   Switch: DBI 1.14 by Tim Bunce, 1.14
   Available Drivers: ADO, ExampleP, Multiplex, Proxy, mysql
   dbi:ExampleP:: testing 5 sets of 20 connections:
   Connecting... 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 
   Disconnecting...
   Connecting... 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 
   Disconnecting...
   Connecting... 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 
   Disconnecting...
   Connecting... 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 
   Disconnecting...
   Connecting... 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 
   Disconnecting...
   Made 100 connections in  0 wallclock secs ( 0.22 usr +  0.03 sys =  0.25 CPU)
   
   Testing handle creation speed...
   5000 NullP statement handles cycled in 6.6 cpu+sys seconds (762 per sec)
   
   test.pl done
   
   [root@fenchurch DBI-1.14]# make install
   These files are installed.
   /usr/lib/perl5/site_perl/5.005/i386-linux/auto/DBI/.packlist
   /usr/bin/dbiproxy
   /usr/bin/dbish
   /usr/lib/perl5/man/man3/Bundle::DBI.3
   /usr/lib/perl5/man/man3/DBD::ADO.3
   /usr/lib/perl5/man/man3/DBD::Multiplex.3
   /usr/lib/perl5/man/man3/DBD::Proxy.3
   /usr/lib/perl5/man/man3/DBI.3
   /usr/lib/perl5/man/man3/DBI::DBD.3
   /usr/lib/perl5/man/man3/DBI::FAQ.3
   /usr/lib/perl5/man/man3/DBI::Format.3
   /usr/lib/perl5/man/man3/DBI::ProxyServer.3
   /usr/lib/perl5/man/man3/DBI::Shell.3
   /usr/lib/perl5/man/man3/DBI::W32ODBC.3
   /usr/lib/perl5/man/man3/Win32::DBIODBC.3
   /usr/lib/perl5/site_perl/5.005/i386-linux/Bundle/DBI.pm
   /usr/lib/perl5/site_perl/5.005/i386-linux/DBD/ADO.pm
   /usr/lib/perl5/site_perl/5.005/i386-linux/DBD/ExampleP.pm
   /usr/lib/perl5/site_perl/5.005/i386-linux/DBD/Multiplex.pm
   /usr/lib/perl5/site_perl/5.005/i386-linux/DBD/NullP.pm
   /usr/lib/perl5/site_perl/5.005/i386-linux/DBD/Proxy.pm
   /usr/lib/perl5/site_perl/5.005/i386-linux/DBD/Sponge.pm
   /usr/lib/perl5/site_perl/5.005/i386-linux/DBI.pm
   /usr/lib/perl5/site_perl/5.005/i386-linux/DBI/DBD.pm
   /usr/lib/perl5/site_perl/5.005/i386-linux/DBI/FAQ.pm
   /usr/lib/perl5/site_perl/5.005/i386-linux/DBI/Format.pm
   /usr/lib/perl5/site_perl/5.005/i386-linux/DBI/ProxyServer.pm
   /usr/lib/perl5/site_perl/5.005/i386-linux/DBI/Shell.pm
   /usr/lib/perl5/site_perl/5.005/i386-linux/DBI/W32ODBC.pm
   /usr/lib/perl5/site_perl/5.005/i386-linux/Win32/DBIODBC.pm
   /usr/lib/perl5/site_perl/5.005/i386-linux/auto/DBI/DBI.bs
   /usr/lib/perl5/site_perl/5.005/i386-linux/auto/DBI/DBI.so
   /usr/lib/perl5/site_perl/5.005/i386-linux/auto/DBI/DBIXS.h
   /usr/lib/perl5/site_perl/5.005/i386-linux/auto/DBI/Driver.xst
   /usr/lib/perl5/site_perl/5.005/i386-linux/auto/DBI/dbd_xsh.h
   /usr/lib/perl5/site_perl/5.005/i386-linux/auto/DBI/dbi_sql.h
   /usr/lib/perl5/site_perl/5.005/i386-linux/auto/DBI/dbipport.h
   /usr/man/man1/dbiproxy.1
   /usr/man/man1/dbish.1
   </PRE>
   </P>
   <FONT COLOR="green"> old notes in green
   <P>
   The following set of tarballs was found to work together
   properly on a LON-CAPA RedHat 6.2 system:
   <UL>
   <LI>DBI-1.13.tar.gz
   <LI>Msql-Mysql-modules-1.2209.tar.gz
   <LI>mysql-3.22.32.tar.gz
   </UL>
   </P>
   <P>
   Installation was simply a matter of following the instructions
   and typing the several "make" commands for each 
   </P>
   </FONT>
   <H2>Configuration (automated)</H2>
   <P>
   Not yet stabilized.  We are looking at webmin
   and/or a text-UI tool with less software dependencies
   than the python-anaconda hunk represented by
   the prior loncapaconfig.
   </P>
   <H2>Manual configuration</H2>
   <P>
   This is not complete.
   </P>
   <P>
   <STRONG>Starting the mysql daemon</STRONG>: Login on the Linux
   system as user 'www'.  Enter the command
   <TT>/usr/local/bin/safe_mysqld &</TT>
   </P>
   <P>
   <STRONG>Set a password for 'root'</STRONG>:
   <TT>/usr/local/bin/mysqladmin -u root password 'new-password'</TT>
   </P>
   <P>
   <STRONG>Adding a user</STRONG>:  Start the mysql daemon.  Login to the
   mysql system as root (<TT>mysql -u root -p mysql</TT>)
   and enter the right password (for instance 'newmysql').  Add the user
   www
   <PRE>
   INSERT INTO user (Host, User, Password)
   VALUES ('localhost','www',password('newmysql'));
 </PRE>  </PRE>
 </P>  </P>
 <H3>Source Installation and Manual Configuration</H3>  
 <P>  <P>
   <STRONG>Granting privileges to user 'www'</STRONG>:
   <PRE>
   GRANT ALL PRIVILEGES ON loncapa.* TO www@localhost;
   FLUSH PRIVILEGES;
   </PRE>
   </P>
   <P>
   <STRONG>Set the SQL server to start upon system startup</STRONG>:
   Copy support-files/mysql.server to the right place on the system
   (/etc/rc.d/...).
   </P>
   <P>
   <STRONG>The Perl API</STRONG>
 <PRE>  <PRE>
 August, 29 2000; Scott Harrison; LON-CAPA  
   
 These are notes related to a Perl interface and MySQL server installation  
 on Redhat 6.1 and 6.2 boxes.  (Guy Albertelli and Harsha Jagasia  
 contributed significantly to this.)  
   
 ********************  
 * MySQL COMPONENTS *  
 ********************  
   
 There are three components to an effective MySQL installation for the  
 LON-CAPA system.  
   
 Perl::DBI module- the API "front-end"...  
                   database interface module for organizing generic  
                   database commands which are independent of specific  
                   database implementation (such as MySQL, mSQL, Postgres, etc).  
   
 Perl::MySQL module- the API "mid-section"...  
                     the module to directly interface with the actual  
                     MySQL database engine  
   
 MySQL database engine- the "back-end"...  
                        the binary installation (compiled either from source  
                        or pre-compiled file listings) which provides the  
                        actual MySQL functionality on the system  
   
 RedHat Installation-  
   
 Initially done from source:  
 DBI-1.13.tar.gz  Msql-Mysql-modules-1.2209.tar.gz  mysql-3.22.32.tar.gz  
   
 I am now using pre-compiled file listings.  
   
 There were problems with using the RedHat packages since the three  
 different RedHat packages were somewhat noncompatible with each other  
 in terms of expected file locations. (The Debian linux distribution,  
 on the other hand, has a working set of these packages).  
   
 Regardless of how we install these three components, there still remain  
 certain things which need to happen for the configuration.  
   
 *****************  
 * CONFIGURATION *  
 *****************  
   
 (Note: SOMEPASSWORD is actually set to another text string on the current  
 LON-CAPA systems.)  
   
 Configuration is needed to generate the necessary functionality for the  
 MySQL system with LON-CAPA.  
   
 The functionality needed can be understood from this example line  
 of perl code from "lonsql".  
   
    $dbh = DBI->connect( "DBI:mysql:loncapa",     $dbh = DBI->connect( "DBI:mysql:loncapa",
  "www",   "www",
  "SOMEPASSWORD",   "SOMEPASSWORD",
Line 182  As user www, run this command Line 618  As user www, run this command
 enter the password as SOMEPASSWORD  enter the password as SOMEPASSWORD
   
 To add the user www to the MySQL server, and grant all  To add the user www to the MySQL server, and grant all
 privileges on *.* to www@localhost identified by 'SOMEPASSWORD'  privileges on loncapa.* to www@localhost identified by 'SOMEPASSWORD'
 with grant option;  with grant option;
   
 INSERT INTO user (Host, User, Password)  INSERT INTO user (Host, User, Password)
 VALUES ('localhost','www',password('SOMEPASSWORD'));  VALUES ('localhost','www',password('SOMEPASSWORD'));
   
 GRANT ALL PRIVILEGES ON *.* TO www@localhost;  GRANT ALL PRIVILEGES ON loncapa.* TO www@localhost;
   
 FLUSH PRIVILEGES;  FLUSH PRIVILEGES;
   
 ** ABILITY for LON-CAPA machines to communicate with SQL databases on  ** ABILITY for LON-CAPA machines to communicate with SQL databases on
    other LON-CAPA machines     other LON-CAPA machines
   
 This is a little more intricate than might first be expected (and I probably  An up-to-date lond and lonsql.
 won't do a perfect job reciting everything in this short synopsis).  Because  </PRE>
 LON-CAPA machines will likely be handling many SQL requests at a time,  </P>
 there were some problems with current MySQL capabilities.  <H2>Testing</H2>
   <P>
 PROBLEM SITUATION:  To test the backend MySQL database, a number of commands should be
   run after installation.
   If Server A wants data from Server B, Server A uses a lonc process to  <UL>
   send a database command to a Server B lond process.  <LI><TT>cd /usr/local/mysql/sql-bench; ./run-all-tests --small-test</TT></LI>
     lonc= loncapa client process    A-lonc= a lonc process on Server A  <BR>without the --small-test flag, this test can take more than 10 hours!
     lond= loncapa daemon process  <LI><TT>cd /usr/local/mysql; bin/mysqladmin version</TT></LI>
   <LI><TT>cd /usr/local/mysql; bin/mysqladmin variables</TT></LI>
                  database command  <LI><TT>cd /usr/local/mysql; bin/mysqlshow</TT></LI>
     A-lonc  --------TCP/IP----------------> B-lond  <LI><TT>cd /usr/local/mysql; bin/mysqlshow mysql</TT></LI>
   <LI><TT>cd /usr/local/mysql; bin/mysql -e "select host,db,user from db" mysql</TT></LI>
   The problem emerges that A-lonc and B-lond are kept waiting for the  <LI><TT>cd /usr/local/mysql/mysql-test; ./test-run-all</TT></LI>
   MySQL server to "do its stuff", or in other words, perform the conceivably  
   sophisticated, data-intensive, time-sucking database transaction.  By tying  
   up a lonc and lond process, this significantly cripples the capabilities  
   of LON-CAPA servers.   
   
   While commercial databases have a variety of features that ATTEMPT to  
   deal with this, freeware databases are still experimenting and exploring  
   with different schemes with varying degrees of performance stability.  
   
 THE SOLUTION:  
   
   A separate daemon process was created that B-lond works with to  
   handle database requests.  This daemon process is called "lonsql".  
   
   So,  
                 database command  
   A-lonc  ---------TCP/IP-----------------> B-lond =====> B-lonsql  
          <---------------------------------/                |  
            "ok, I'll get back to you..."                    |  
                                                             |  
                                                             /  
   A-lond  <-------------------------------  B-lonc   <======  
            "Guess what? I have the result!"  
   
   Of course, depending on success or failure, the messages may vary,  </UL>
   but the principle remains the same where a separate pool of children  <P>
   processes (lonsql's) handle the MySQL database manipulations.  These are sections of perl code which helps test the LON-CAPA network.
   <PRE>
   <STRONG>** TEST the database connection with my current tester.pl code
   which mimics what command will eventually be sent through lonc.</STRONG>
   
   $reply=reply(
       "querysend:SELECT * FROM general_information WHERE Id='AAAAA'",$lonID);
   </PRE>
   </P>
   <H2>Example sections of code relevant to LON-CAPA</H2>
   <P>
 Here are excerpts of code which implement the above handling:  Here are excerpts of code which implement the above handling:
   </P>
 **LONSQL  <P>
   <PRE>
   <STRONG>**LONSQL
 A subroutine from "lonsql" which establishes a child process for handling  A subroutine from "lonsql" which establishes a child process for handling
 database interactions.  database interactions.</STRONG>
   
 sub make_new_child {  sub make_new_child {
     my $pid;      my $pid;
Line 336  sub make_new_child { Line 759  sub make_new_child {
         exit;          exit;
     }      }
 }  }
   </P>
 ** LOND enabling of MySQL requestsw  <P>
   <STRONG>** LOND enabling of MySQL requests</STRONG>
   This code is part of every lond child process in the way that it parses command request syntax  <BR />
   sent to it from lonc processes.  querysend corresponds to B-lonc sending the result of the query.  This code is part of every lond child process in the
   queryreply corresponds to B-lond indicating that it has received the request and will start the  way that it parses command request syntax sent to it
   database transaction (it returns "ok" to A-lonc ($client)).  from lonc processes.  Based on the diagram above, querysend
   corresponds to B-lonc sending the result of the query.
   queryreply corresponds to B-lond indicating that it has
   received the request and will start the database transaction
   (it returns "ok" to
   A-lonc ($client)).
   <PRE>
 # ------------------------------------------------------------------- querysend  # ------------------------------------------------------------------- querysend
                    } elsif ($userinput =~ /^querysend/) {                     } elsif ($userinput =~ /^querysend/) {
                        my ($cmd,$query)=split(/:/,$userinput);                         my ($cmd,$query)=split(/:/,$userinput);
Line 363  sub make_new_child { Line 791  sub make_new_child {
    print $client "error:$!\n";     print $client "error:$!\n";
        }         }
   
   
   
 ** TEST the database connection with my current tester.pl code which mimics what command will eventually be  
    sent through lonc.  
   
 $reply=reply(  
     "querysend:SELECT * FROM general_information WHERE Id='AAAAA'",$lonID);  
 </PRE>  </PRE>
   
 </P>  </P>
 </BODY>  </BODY>
 </HTML>  
   
   </HTML>
Removed from v.1.2  
changed lines
  Added in v.1.18

FreeBSD-CVSweb <freebsd-cvsweb@FreeBSD.org>