-Last updated: 02/10/2001 +Last updated: 02/15/2001
This file describes issues associated with LON-CAPA and a SQL database.
-
+The current database is implemented assuming a non-adjustable +architecture involving these data fields (specific to each version +of a resource). +
+These commands create the loncapameta database. +
+mysql> CREATE DATABASE IF NOT EXISTS loncapa;
+mysql> USE loncapa;
+mysql> CREATE TABLE IF NOT EXISTS metadata (title TEXT, author TEXT, subject TEXT, notes TEXT, abstract TEXT, mime TEXT, language TEXT, creationdate TEXT, lastrevisiondate TEXT, owner TEXT, copyright TEXT, FULLTEXT (title, author, subject, notes, abstract, mime, language, creationdate, lastrevisiondate, owner, copyright)) TYPE=MYISAM;
+mysql> INSERT INTO metadata VALUES ('1','2','3','4','5','6','7','8','9','10','11');
+mysql> SELECT * FROM metadata WHERE title REGEXP "1";
+
+
++Current time values for things. +
+fenchurch.lite.msu.edu +Mem: 46812K av, 45632K used, 1180K free, 14756K shrd, 4292K buff +Swap: 1148608K av, 11260K used, 1137348K free 13244K cached + +Red Hat Linux release 6.2 (Zoot) +Kernel 2.2.16-3 on an i586 ++
+An important quote from the manual: +
+In MySQL Version 3.23.23 or later, you can also create special FULLTEXT indexes. They are used for full-text search. Only the MyISAM table type supports FULLTEXT indexes. They can be created only from +VARCHAR and TEXT columns. Indexing always happens over the entire column and partial indexing is not supported. See section 25.2 MySQL Full-text Search for details. ++ +
+I plan on using a MyISAM table type with 11 metadata fields of column +type=TEXT. +
++It might be worthwhile to look at /usr/local/mysql/manual.html. +It is quite in depth. +
+I am going to begin documentation by inserting what notes I have into this file. I will be subsequently rearranging @@ -39,13 +105,24 @@ current status of everything is that it been minimally tested, but things need to be cleaned up and checked again!
-+Need to +
Right now, a lot of "feasibility" work has been done. Recipes for manual installation and configuration have been gathered. Network connectivity of lond->lonsql->lond->lonc type tests have been performed. A binary installation -has been compiled in an RPM (LON-CAPA-mysql). +has been compiled in an RPM (LON-CAPA-mysql, with perl components +a part of LON-CAPA-systemperl). The most lacking test in terms of feasibility has been looking at benchmarks to analyze the load at which the SQL database can efficiently allow many users to @@ -70,7 +147,7 @@ to the point of getting this SQL softwar however there may be more optimal approaches than currently exist.
-LON-CAPA is meant to distribute A LOT of educational content to A LOT of people. It is ineffective to directly rely on contents @@ -131,13 +208,31 @@ THE SOLUTION: processes (lonsql's) handle the MySQL database manipulations.
-+I believe (but am not 100% confident) that the following +RPMs are necessary (in addition to the current ones +in rpm_list.txt) to run MySQL. Basically I discovered these +dependencies while trying to do external RPM based installs. +I assume, and sometimes found, that these dependencies apply +to tarball-based distributions too. (So to play it on the +safe side, I am going to include these RPMs as part of the +core, minimal RPM set.) +
Installation of the LON-CAPA SQL database normally occurs by default when using the LON-CAPA installation CD (see http://install.lon-capa.org). It is installed as the LON-CAPA-mysql RPM. This RPM encodes for the MySQL -engine and related perl interfaces (Perl::DBI, Perl::Msql-Mysql). +engine. Related perl interfaces (Perl::DBI, Perl::Msql-Mysql) +are encoded in the LON-CAPA-systemperl RPM.
The three components of a MySQL installation for the @@ -160,7 +255,254 @@ from source or pre-compiled file listing actual MySQL functionality on the system
-+Note: the mysql site recommends that Linux users install by +using the MySQL RPMs (MySQL-client, MySQL, MySQL-shared, etc). +While these RPMs work, I was unsuccessful at integrating +this RPM-installed database with perl modules from www.cpan.org. +Hence, I strongly recommend that, when installing +from "source", MySQL and the perl components be in fact installed +from their tarballs (.tar.gz, .tgz). (Perl components, when installed +from RPMs, also wound up in incorrect locations on the disk.) +Do not coordinate a source install with externally made RPMs! +It is, of course, okay to use LON-CAPA RPMs such as LON-CAPA-systemperl +and LON-CAPA-mysql since we, in fact, made these RPMs correctly :). +
So, here is exactly how I installed MySQL-3.23. (Note that all files +wind up in /usr/local/mysql-3.23.33-pc-linux-gnu-i686 except for +a link from /usr/local/mysql to /usr/local/mysql-3.23.33-pc-linux-gnu-i686 +and some files involved in system process handling (/etc/rc.d/*/*mysql). +
+(As user=root) +cd /usr/local/; +tar xzvf mysql-3.23.33-pc-linux-gnu-i686.tar.gz +ln -s /usr/local/mysql-3.23.33-pc-linux-gnu-i686 mysql +cd /usr/local/mysql +chown -R www /usr/local/mysql/. +chgrp -R users /usr/local/mysql/. +chmod -R g-w,g-r,g-x /usr/local/mysql/. +(probably also want chmod -R a-w,a-r,a-x /usr/local/mysql/.) + +Alter safe_mysqld +and support-files/mysql.server to use 'localhosts' instead +of `bin/hostname`. Also, to use user 'www' instead of 'mysql'. + +(These changes could be done with /etc/my.cnf, but +I think this approach makes sure the database NEVER +gets screwed up due to somebody forgetting to install /etc/my.cnf). + +Change this line in mysql.server from + pid_file=$datadir/`@HOSTNAME@`.pid +to + pid_file=$datadir/localhost.pid + +Change this line in safe_mysqld from +user=root +to +user=www + +Change this line in safe_mysqld from + pid_file=$DATADIR/`/bin/hostname`.pid +to + pid_file=$DATADIR/localhost.pid + +Change this line in safe_mysqld from +test -z "$err_log" && err_log=$DATADIR/`/bin/hostname`.err +to +test -z "$err_log" && err_log=$DATADIR/localhost.err + +cp -p support-files/mysql.server /etc/rc.d/init.d/mysql +chmod 755 /etc/rc.d/init.d/mysql +/sbin/chkconfig --add mysql +/etc/rc.d/init.d/mysql start + ++ +
This is how I installed the Msql-Mysql-modules perl modules. +
+[root@fenchurch Msql-Mysql-modules-1.2215]# perl Makefile.PL +Which drivers do you want to install? + + 1) MySQL only + 2) mSQL only (either of mSQL 1 or mSQL 2) + 3) MySQL and mSQL (either of mSQL 1 or mSQL 2) + + 4) mSQL 1 and mSQL 2 + 5) MySQL, mSQL 1 and mSQL 2 + +Enter the appropriate number: [3] 1 + + +Do you want to install the MysqlPerl emulation? You might keep your old +Mysql module (to be distinguished from DBD::mysql!) if you are concerned +about compatibility to existing applications! [y] n +Where is your MySQL installed? Please tell me the directory that +contains the subdir 'include'. [/usr/local/mysql] +Which database should I use for testing the MySQL drivers? [test] +On which host is database test running (hostname, ip address +or host:port) [localhost] +User name for connecting to database test? [undef] +Password for connecting to database test? [undef] +[root@fenchurch Msql-Mysql-modules-1.2215]# make +[root@fenchurch Msql-Mysql-modules-1.2215]# make test +make[1]: Entering directory `/home/user/Msql-Mysql-modules-1.2215/mysql' +make[1]: Leaving directory `/home/user/Msql-Mysql-modules-1.2215/mysql' +make[1]: Entering directory `/home/user/Msql-Mysql-modules-1.2215/mysql' +PERL_DL_NONLAZY=1 /usr/bin/perl -I../blib/arch -I../blib/lib -I/usr/lib/perl5/5.00503/i386-linux -I/usr/lib/perl5/5.00503 -e 'use Test::Harness qw(&runtests $verbose); $verbose=0; runtests @ARGV;' t/*.t +t/00base............ok +t/10dsnlist.........ok +t/20createdrop......ok +t/30insertfetch.....ok +t/40bindparam.......ok +t/40blobs...........ok +t/40listfields......ok +t/40nulls...........ok +t/40numrows.........ok +t/50chopblanks......ok +t/50commit..........ok +t/60leaks...........skipping test on this platform +t/ak-dbd............ok +t/akmisc............ok +t/dbdadmin..........ok +t/mysql.............ok +t/mysql2............ok +All tests successful, 1 test skipped. +Files=17, Tests=732, 40 wallclock secs (15.38 cusr + 1.30 csys = 16.68 CPU) +[root@fenchurch Msql-Mysql-modules-1.2215]# make install + +These files are installed. +/usr/bin/dbimon +/usr/lib/perl5/man/man3/Bundle::DBD::mysql.3 +/usr/lib/perl5/man/man3/DBD::mysql.3 +/usr/lib/perl5/man/man3/Mysql.3 +/usr/lib/perl5/site_perl/5.005/i386-linux/Bundle/DBD/mysql.pm +/usr/lib/perl5/site_perl/5.005/i386-linux/DBD/mysql.pm +/usr/lib/perl5/site_perl/5.005/i386-linux/Mysql.pm +/usr/lib/perl5/site_perl/5.005/i386-linux/Mysql/Statement.pm +/usr/lib/perl5/site_perl/5.005/i386-linux/auto/DBD/mysql/mysql.bs +/usr/lib/perl5/site_perl/5.005/i386-linux/auto/DBD/mysql/mysql.so +/usr/man/man1/dbimon.1 +/usr/lib/perl5/site_perl/5.005/i386-linux/auto/Msql-Mysql-modules/.packlist ++ +
+This is how I installed the DBI perl modules. +
+[root@fenchurch DBI-1.14]# perl Makefile.PL +*** Note: + The optional PlRPC-modules (RPC::PlServer etc) are not installed. + If you want to use the DBD::Proxy driver and DBI::ProxyServer + modules, then you'll need to install the RPC::PlServer, RPC::PlClient, + Storable and Net::Daemon modules. The CPAN Bundle::DBI may help you. + You can install them any time after installing the DBI. + You do *not* need these modules for typical DBI usage. + +Optional modules are available from any CPAN mirror, in particular + http://www.perl.com/CPAN/modules/by-module + http://www.perl.org/CPAN/modules/by-module + ftp://ftp.funet.fi/pub/languages/perl/CPAN/modules/by-module + +Checking if your kit is complete... +Looks good +Writing Makefile for DBI + + Remember to actually *read* the README file! + Use 'make' to build the software (dmake or nmake on Windows). + Then 'make test' to execute self tests. + Then 'make install' to install the DBI and then delete this working + directory before unpacking and building any DBD::* drivers. + +[root@fenchurch DBI-1.14]# make +[root@fenchurch DBI-1.14]# make test +PERL_DL_NONLAZY=1 /usr/bin/perl -Iblib/arch -Iblib/lib -I/usr/lib/perl5/5.00503/i386-linux -I/usr/lib/perl5/5.00503 -e 'use Test::Harness qw(&runtests $verbose); $verbose=0; runtests @ARGV;' t/*.t +t/basics............ok +t/dbidrv............ok +t/examp.............ok +t/meta..............ok +t/proxy.............skipping test on this platform +t/shell.............ok +t/subclass..........ok +All tests successful, 1 test skipped. +Files=7, Tests=179, 7 wallclock secs ( 6.46 cusr + 0.49 csys = 6.95 CPU) +PERL_DL_NONLAZY=1 /usr/bin/perl -Iblib/arch -Iblib/lib -I/usr/lib/perl5/5.00503/i386-linux -I/usr/lib/perl5/5.00503 test.pl +test.pl +DBI test application $Revision: 1.13 $ +Using /home/user/DBI-1.14/blib +Switch: DBI 1.14 by Tim Bunce, 1.14 +Available Drivers: ADO, ExampleP, Multiplex, Proxy, mysql +dbi:ExampleP:: testing 5 sets of 20 connections: +Connecting... 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 +Disconnecting... +Connecting... 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 +Disconnecting... +Connecting... 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 +Disconnecting... +Connecting... 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 +Disconnecting... +Connecting... 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 +Disconnecting... +Made 100 connections in 0 wallclock secs ( 0.22 usr + 0.03 sys = 0.25 CPU) + +Testing handle creation speed... +5000 NullP statement handles cycled in 6.6 cpu+sys seconds (762 per sec) + +test.pl done + +[root@fenchurch DBI-1.14]# make install +These files are installed. +/usr/lib/perl5/site_perl/5.005/i386-linux/auto/DBI/.packlist +/usr/bin/dbiproxy +/usr/bin/dbish +/usr/lib/perl5/man/man3/Bundle::DBI.3 +/usr/lib/perl5/man/man3/DBD::ADO.3 +/usr/lib/perl5/man/man3/DBD::Multiplex.3 +/usr/lib/perl5/man/man3/DBD::Proxy.3 +/usr/lib/perl5/man/man3/DBI.3 +/usr/lib/perl5/man/man3/DBI::DBD.3 +/usr/lib/perl5/man/man3/DBI::FAQ.3 +/usr/lib/perl5/man/man3/DBI::Format.3 +/usr/lib/perl5/man/man3/DBI::ProxyServer.3 +/usr/lib/perl5/man/man3/DBI::Shell.3 +/usr/lib/perl5/man/man3/DBI::W32ODBC.3 +/usr/lib/perl5/man/man3/Win32::DBIODBC.3 +/usr/lib/perl5/site_perl/5.005/i386-linux/Bundle/DBI.pm +/usr/lib/perl5/site_perl/5.005/i386-linux/DBD/ADO.pm +/usr/lib/perl5/site_perl/5.005/i386-linux/DBD/ExampleP.pm +/usr/lib/perl5/site_perl/5.005/i386-linux/DBD/Multiplex.pm +/usr/lib/perl5/site_perl/5.005/i386-linux/DBD/NullP.pm +/usr/lib/perl5/site_perl/5.005/i386-linux/DBD/Proxy.pm +/usr/lib/perl5/site_perl/5.005/i386-linux/DBD/Sponge.pm +/usr/lib/perl5/site_perl/5.005/i386-linux/DBI.pm +/usr/lib/perl5/site_perl/5.005/i386-linux/DBI/DBD.pm +/usr/lib/perl5/site_perl/5.005/i386-linux/DBI/FAQ.pm +/usr/lib/perl5/site_perl/5.005/i386-linux/DBI/Format.pm +/usr/lib/perl5/site_perl/5.005/i386-linux/DBI/ProxyServer.pm +/usr/lib/perl5/site_perl/5.005/i386-linux/DBI/Shell.pm +/usr/lib/perl5/site_perl/5.005/i386-linux/DBI/W32ODBC.pm +/usr/lib/perl5/site_perl/5.005/i386-linux/Win32/DBIODBC.pm +/usr/lib/perl5/site_perl/5.005/i386-linux/auto/DBI/DBI.bs +/usr/lib/perl5/site_perl/5.005/i386-linux/auto/DBI/DBI.so +/usr/lib/perl5/site_perl/5.005/i386-linux/auto/DBI/DBIXS.h +/usr/lib/perl5/site_perl/5.005/i386-linux/auto/DBI/Driver.xst +/usr/lib/perl5/site_perl/5.005/i386-linux/auto/DBI/dbd_xsh.h +/usr/lib/perl5/site_perl/5.005/i386-linux/auto/DBI/dbi_sql.h +/usr/lib/perl5/site_perl/5.005/i386-linux/auto/DBI/dbipport.h +/usr/man/man1/dbiproxy.1 +/usr/man/man1/dbish.1 ++ + old notes in green
The following set of tarballs was found to work together properly on a LON-CAPA RedHat 6.2 system: @@ -174,13 +516,14 @@ properly on a LON-CAPA RedHat 6.2 system Installation was simply a matter of following the instructions and typing the several "make" commands for each
-Not yet developed. This will be part of an interface present on LON-CAPA systems that can be launched by entering the command /usr/sbin/loncapaconfig.
-This is not complete.
@@ -215,124 +558,9 @@ FLUSH PRIVILEGES; Copy support-files/mysql.server to the right place on the system (/etc/rc.d/...). --Not yet documented or formalized. -
--
- -+The Perl API
-start the mysql daemon as /usr/local/bin/safe_mysqld &
-Login as root: mysql -u root -p mysql
-enter the password as newmysql
-add the user www: grant all priveleges on *.* to www@localhost identified by 'newmysql' with grant option;
-
-INSERT INTO user (Host, User, Password)
-VALUES ('localhost','www',password('newmysql'));
-
-GRANT ALL PRIVILEGES ON *.* TO www@localhost;
-
-FLUSH PRIVILEGES;
-
-Here the user www has the right to grant privileges to other users.
-This can be changed if required with a simple update command on the grant tables
-
-
-/home/httpd/perl/perlsql/lonsql
-/usr/local/mysql/fakeclient
-
-
--
-This is the output from scripts/mysql_install_db... -still some todo things (like support-files/mysql.server) - -Creating db table -Creating host table -Creating user table -Creating func table -Creating tables_priv table -Creating columns_priv table - -To start mysqld at boot time you have to copy support-files/mysql.server -to the right place for your system - -PLEASE REMEMBER TO SET A PASSWORD FOR THE MySQL root USER ! -This is done with: -/usr/local/bin/mysqladmin -u root password 'new-password' -See the manual for more instructions. - -Please report any problems with the /usr/local/bin/mysqlbug script! - -The latest information about MySQL is available on the web at http://www.mysql.com -Support MySQL by buying support/licenses at http://www.tcx.se/license.htmy. -- -
-
-August, 29 2000; Scott Harrison; LON-CAPA
-
-These are notes related to a Perl interface and MySQL server installation
-on Redhat 6.1 and 6.2 boxes. (Guy Albertelli and Harsha Jagasia
-contributed significantly to this.)
-
-********************
-* MySQL COMPONENTS *
-********************
-
-There are three components to an effective MySQL installation for the
-LON-CAPA system.
-
-Perl::DBI module- the API "front-end"...
- database interface module for organizing generic
- database commands which are independent of specific
- database implementation (such as MySQL, mSQL, Postgres, etc).
-
-Perl::MySQL module- the API "mid-section"...
- the module to directly interface with the actual
- MySQL database engine
-
-MySQL database engine- the "back-end"...
- the binary installation (compiled either from source
- or pre-compiled file listings) which provides the
- actual MySQL functionality on the system
-
-RedHat Installation-
-
-Initially done from source:
-DBI-1.13.tar.gz Msql-Mysql-modules-1.2209.tar.gz mysql-3.22.32.tar.gz
-
-I am now using pre-compiled file listings.
-
-There were problems with using the RedHat packages since the three
-different RedHat packages were somewhat noncompatible with each other
-in terms of expected file locations. (The Debian linux distribution,
-on the other hand, has a working set of these packages).
-
-Regardless of how we install these three components, there still remain
-certain things which need to happen for the configuration.
-
-*****************
-* CONFIGURATION *
-*****************
-
-(Note: SOMEPASSWORD is actually set to another text string on the current
-LON-CAPA systems.)
-
-Configuration is needed to generate the necessary functionality for the
-MySQL system with LON-CAPA.
-
-The functionality needed can be understood from this example line
-of perl code from "lonsql".
-
$dbh = DBI->connect( "DBI:mysql:loncapa",
"www",
"SOMEPASSWORD",
@@ -395,56 +623,43 @@ FLUSH PRIVILEGES;
** ABILITY for LON-CAPA machines to communicate with SQL databases on
other LON-CAPA machines
-This is a little more intricate than might first be expected (and I probably
-won't do a perfect job reciting everything in this short synopsis). Because
-LON-CAPA machines will likely be handling many SQL requests at a time,
-there were some problems with current MySQL capabilities.
-
-PROBLEM SITUATION:
-
- If Server A wants data from Server B, Server A uses a lonc process to
- send a database command to a Server B lond process.
- lonc= loncapa client process A-lonc= a lonc process on Server A
- lond= loncapa daemon process
-
- database command
- A-lonc --------TCP/IP----------------> B-lond
-
- The problem emerges that A-lonc and B-lond are kept waiting for the
- MySQL server to "do its stuff", or in other words, perform the conceivably
- sophisticated, data-intensive, time-sucking database transaction. By tying
- up a lonc and lond process, this significantly cripples the capabilities
- of LON-CAPA servers.
-
- While commercial databases have a variety of features that ATTEMPT to
- deal with this, freeware databases are still experimenting and exploring
- with different schemes with varying degrees of performance stability.
-
-THE SOLUTION:
-
- A separate daemon process was created that B-lond works with to
- handle database requests. This daemon process is called "lonsql".
-
- So,
- database command
- A-lonc ---------TCP/IP-----------------> B-lond =====> B-lonsql
- <---------------------------------/ |
- "ok, I'll get back to you..." |
- |
- /
- A-lond <------------------------------- B-lonc <======
- "Guess what? I have the result!"
+An up-to-date lond and lonsql.
+
+
++To test the backend MySQL database, a number of commands should be +run after installation. +
+These are sections of perl code which helps test the LON-CAPA network. +
+** TEST the database connection with my current tester.pl code +which mimics what command will eventually be sent through lonc. +$reply=reply( + "querysend:SELECT * FROM general_information WHERE Id='AAAAA'",$lonID); ++ +
Here are excerpts of code which implement the above handling: - -**LONSQL - +
++
+**LONSQL
A subroutine from "lonsql" which establishes a child process for handling
-database interactions.
+database interactions.
sub make_new_child {
my $pid;
@@ -536,14 +751,19 @@ sub make_new_child {
exit;
}
}
-
-** LOND enabling of MySQL requestsw
-
- This code is part of every lond child process in the way that it parses command request syntax
- sent to it from lonc processes. querysend corresponds to B-lonc sending the result of the query.
- queryreply corresponds to B-lond indicating that it has received the request and will start the
- database transaction (it returns "ok" to A-lonc ($client)).
-
+
+
+** LOND enabling of MySQL requests
+
+This code is part of every lond child process in the
+way that it parses command request syntax sent to it
+from lonc processes. Based on the diagram above, querysend
+corresponds to B-lonc sending the result of the query.
+queryreply corresponds to B-lond indicating that it has
+received the request and will start the database transaction
+(it returns "ok" to
+A-lonc ($client)).
+
# ------------------------------------------------------------------- querysend
} elsif ($userinput =~ /^querysend/) {
my ($cmd,$query)=split(/:/,$userinput);
@@ -563,14 +783,8 @@ sub make_new_child {
print $client "error:$!\n";
}
-
-
-** TEST the database connection with my current tester.pl code which mimics what command will eventually be
- sent through lonc.
-
-$reply=reply(
- "querysend:SELECT * FROM general_information WHERE Id='AAAAA'",$lonID);
+