-Last updated: 02/10/2001 +Last updated: 02/15/2001
This file describes issues associated with LON-CAPA and a SQL database.
-
+The current database is implemented assuming a non-adjustable +architecture involving these data fields (specific to each version +of a resource). +
+Security occurs as a function of the user 'www', +and the permissions on the files in the /usr/local/mysql +directory. (These files and directories should +be 700, 600, 500, 400, etc; not allow access to anyone +but user 'www'.) +
++These commands create the loncapameta database. +
+mysql> CREATE DATABASE IF NOT EXISTS loncapa;
+mysql> USE loncapa;
+mysql> CREATE TABLE IF NOT EXISTS metadata (title TEXT, author TEXT, subject TEXT, notes TEXT, abstract TEXT, mime TEXT, language TEXT, creationdate TEXT, lastrevisiondate TEXT, owner TEXT, copyright TEXT, FULLTEXT idx_title (title), FULLTEXT idx_author (author), FULLTEXT idx_subject (subject), FULLTEXT idx_notes (notes), FULLTEXT idx_abstract (abstract), FULLTEXT idx_mime (mime), FULLTEXT idx_language (language), FULLTEXT idx_creationdate (creationdate), FULLTEXT idx_lastrevisiondate (lastrevisiondate), FULLTEXT idx_owner (owner), FULLTEXT idx_copyright (copyright)) TYPE=MYISAM;
+mysql> INSERT INTO metadata VALUES ('1','2','3','4','5','6','7','8','9','10','11');
+mysql> SELECT * FROM metadata WHERE title REGEXP "1";
+
+
++Current time values for things. +
+fenchurch.lite.msu.edu +Mem: 46812K av, 45632K used, 1180K free, 14756K shrd, 4292K buff +Swap: 1148608K av, 11260K used, 1137348K free 13244K cached + +Red Hat Linux release 6.2 (Zoot) +Kernel 2.2.16-3 on an i586 ++
+An important quote from the manual: +
+In MySQL Version 3.23.23 or later, you can also create special FULLTEXT indexes. They are used for full-text search. Only the MyISAM table type supports FULLTEXT indexes. They can be created only from +VARCHAR and TEXT columns. Indexing always happens over the entire column and partial indexing is not supported. See section 25.2 MySQL Full-text Search for details. ++ +
+I plan on using a MyISAM table type with 11 metadata fields of column +type=TEXT. +
++It might be worthwhile to look at /usr/local/mysql/manual.html. +It is quite in depth. +
+I am going to begin documentation by inserting what notes I have into this file. I will be subsequently rearranging @@ -39,13 +112,24 @@ current status of everything is that it been minimally tested, but things need to be cleaned up and checked again!
-+Need to +
Right now, a lot of "feasibility" work has been done. Recipes for manual installation and configuration have been gathered. Network connectivity of lond->lonsql->lond->lonc type tests have been performed. A binary installation -has been compiled in an RPM (LON-CAPA-mysql). +has been compiled in an RPM (LON-CAPA-mysql, with perl components +a part of LON-CAPA-systemperl). The most lacking test in terms of feasibility has been looking at benchmarks to analyze the load at which the SQL database can efficiently allow many users to @@ -70,7 +154,7 @@ to the point of getting this SQL softwar however there may be more optimal approaches than currently exist.
-LON-CAPA is meant to distribute A LOT of educational content to A LOT of people. It is ineffective to directly rely on contents @@ -131,13 +215,31 @@ THE SOLUTION: processes (lonsql's) handle the MySQL database manipulations.
-+I believe (but am not 100% confident) that the following +RPMs are necessary (in addition to the current ones +in rpm_list.txt) to run MySQL. Basically I discovered these +dependencies while trying to do external RPM based installs. +I assume, and sometimes found, that these dependencies apply +to tarball-based distributions too. (So to play it on the +safe side, I am going to include these RPMs as part of the +core, minimal RPM set.) +
Installation of the LON-CAPA SQL database normally occurs by default when using the LON-CAPA installation CD (see http://install.lon-capa.org). It is installed as the LON-CAPA-mysql RPM. This RPM encodes for the MySQL -engine and related perl interfaces (Perl::DBI, Perl::Msql-Mysql). +engine. Related perl interfaces (Perl::DBI, Perl::Msql-Mysql) +are encoded in the LON-CAPA-systemperl RPM.
The three components of a MySQL installation for the @@ -160,7 +262,254 @@ from source or pre-compiled file listing actual MySQL functionality on the system
-+Note: the mysql site recommends that Linux users install by +using the MySQL RPMs (MySQL-client, MySQL, MySQL-shared, etc). +While these RPMs work, I was unsuccessful at integrating +this RPM-installed database with perl modules from www.cpan.org. +Hence, I strongly recommend that, when installing +from "source", MySQL and the perl components be in fact installed +from their tarballs (.tar.gz, .tgz). (Perl components, when installed +from RPMs, also wound up in incorrect locations on the disk.) +Do not coordinate a source install with externally made RPMs! +It is, of course, okay to use LON-CAPA RPMs such as LON-CAPA-systemperl +and LON-CAPA-mysql since we, in fact, made these RPMs correctly :). +
So, here is exactly how I installed MySQL-3.23. (Note that all files +wind up in /usr/local/mysql-3.23.33-pc-linux-gnu-i686 except for +a link from /usr/local/mysql to /usr/local/mysql-3.23.33-pc-linux-gnu-i686 +and some files involved in system process handling (/etc/rc.d/*/*mysql). +
+(As user=root) +cd /usr/local/; +tar xzvf mysql-3.23.33-pc-linux-gnu-i686.tar.gz +ln -s /usr/local/mysql-3.23.33-pc-linux-gnu-i686 mysql +cd /usr/local/mysql +chown -R www /usr/local/mysql/. +chgrp -R users /usr/local/mysql/. +chmod -R g-w,g-r,g-x /usr/local/mysql/. +(probably also want chmod -R a-w,a-r,a-x /usr/local/mysql/.) + +Alter safe_mysqld +and support-files/mysql.server to use 'localhosts' instead +of `bin/hostname`. Also, to use user 'www' instead of 'mysql'. + +(These changes could be done with /etc/my.cnf, but +I think this approach makes sure the database NEVER +gets screwed up due to somebody forgetting to install /etc/my.cnf). + +Change this line in mysql.server from + pid_file=$datadir/`@HOSTNAME@`.pid +to + pid_file=$datadir/localhost.pid + +Change this line in safe_mysqld from +user=root +to +user=www + +Change this line in safe_mysqld from + pid_file=$DATADIR/`/bin/hostname`.pid +to + pid_file=$DATADIR/localhost.pid + +Change this line in safe_mysqld from +test -z "$err_log" && err_log=$DATADIR/`/bin/hostname`.err +to +test -z "$err_log" && err_log=$DATADIR/localhost.err + +cp -p support-files/mysql.server /etc/rc.d/init.d/mysql +chmod 755 /etc/rc.d/init.d/mysql +/sbin/chkconfig --add mysql +/etc/rc.d/init.d/mysql start + ++ +
This is how I installed the Msql-Mysql-modules perl modules. +
+[root@fenchurch Msql-Mysql-modules-1.2215]# perl Makefile.PL +Which drivers do you want to install? + + 1) MySQL only + 2) mSQL only (either of mSQL 1 or mSQL 2) + 3) MySQL and mSQL (either of mSQL 1 or mSQL 2) + + 4) mSQL 1 and mSQL 2 + 5) MySQL, mSQL 1 and mSQL 2 + +Enter the appropriate number: [3] 1 + + +Do you want to install the MysqlPerl emulation? You might keep your old +Mysql module (to be distinguished from DBD::mysql!) if you are concerned +about compatibility to existing applications! [y] n +Where is your MySQL installed? Please tell me the directory that +contains the subdir 'include'. [/usr/local/mysql] +Which database should I use for testing the MySQL drivers? [test] +On which host is database test running (hostname, ip address +or host:port) [localhost] +User name for connecting to database test? [undef] +Password for connecting to database test? [undef] +[root@fenchurch Msql-Mysql-modules-1.2215]# make +[root@fenchurch Msql-Mysql-modules-1.2215]# make test +make[1]: Entering directory `/home/user/Msql-Mysql-modules-1.2215/mysql' +make[1]: Leaving directory `/home/user/Msql-Mysql-modules-1.2215/mysql' +make[1]: Entering directory `/home/user/Msql-Mysql-modules-1.2215/mysql' +PERL_DL_NONLAZY=1 /usr/bin/perl -I../blib/arch -I../blib/lib -I/usr/lib/perl5/5.00503/i386-linux -I/usr/lib/perl5/5.00503 -e 'use Test::Harness qw(&runtests $verbose); $verbose=0; runtests @ARGV;' t/*.t +t/00base............ok +t/10dsnlist.........ok +t/20createdrop......ok +t/30insertfetch.....ok +t/40bindparam.......ok +t/40blobs...........ok +t/40listfields......ok +t/40nulls...........ok +t/40numrows.........ok +t/50chopblanks......ok +t/50commit..........ok +t/60leaks...........skipping test on this platform +t/ak-dbd............ok +t/akmisc............ok +t/dbdadmin..........ok +t/mysql.............ok +t/mysql2............ok +All tests successful, 1 test skipped. +Files=17, Tests=732, 40 wallclock secs (15.38 cusr + 1.30 csys = 16.68 CPU) +[root@fenchurch Msql-Mysql-modules-1.2215]# make install + +These files are installed. +/usr/bin/dbimon +/usr/lib/perl5/man/man3/Bundle::DBD::mysql.3 +/usr/lib/perl5/man/man3/DBD::mysql.3 +/usr/lib/perl5/man/man3/Mysql.3 +/usr/lib/perl5/site_perl/5.005/i386-linux/Bundle/DBD/mysql.pm +/usr/lib/perl5/site_perl/5.005/i386-linux/DBD/mysql.pm +/usr/lib/perl5/site_perl/5.005/i386-linux/Mysql.pm +/usr/lib/perl5/site_perl/5.005/i386-linux/Mysql/Statement.pm +/usr/lib/perl5/site_perl/5.005/i386-linux/auto/DBD/mysql/mysql.bs +/usr/lib/perl5/site_perl/5.005/i386-linux/auto/DBD/mysql/mysql.so +/usr/man/man1/dbimon.1 +/usr/lib/perl5/site_perl/5.005/i386-linux/auto/Msql-Mysql-modules/.packlist ++ +
+This is how I installed the DBI perl modules. +
$dbh +[root@fenchurch DBI-1.14]# perl Makefile.PL +*** Note: + The optional PlRPC-modules (RPC::PlServer etc) are not installed. + If you want to use the DBD::Proxy driver and DBI::ProxyServer + modules, then you'll need to install the RPC::PlServer, RPC::PlClient, + Storable and Net::Daemon modules. The CPAN Bundle::DBI may help you. + You can install them any time after installing the DBI. + You do *not* need these modules for typical DBI usage. + +Optional modules are available from any CPAN mirror, in particular + http://www.perl.com/CPAN/modules/by-module + http://www.perl.org/CPAN/modules/by-module + ftp://ftp.funet.fi/pub/languages/perl/CPAN/modules/by-module + +Checking if your kit is complete... +Looks good +Writing Makefile for DBI + + Remember to actually *read* the README file! + Use 'make' to build the software (dmake or nmake on Windows). + Then 'make test' to execute self tests. + Then 'make install' to install the DBI and then delete this working + directory before unpacking and building any DBD::* drivers. + +[root@fenchurch DBI-1.14]# make +[root@fenchurch DBI-1.14]# make test +PERL_DL_NONLAZY=1 /usr/bin/perl -Iblib/arch -Iblib/lib -I/usr/lib/perl5/5.00503/i386-linux -I/usr/lib/perl5/5.00503 -e 'use Test::Harness qw(&runtests $verbose); $verbose=0; runtests @ARGV;' t/*.t +t/basics............ok +t/dbidrv............ok +t/examp.............ok +t/meta..............ok +t/proxy.............skipping test on this platform +t/shell.............ok +t/subclass..........ok +All tests successful, 1 test skipped. +Files=7, Tests=179, 7 wallclock secs ( 6.46 cusr + 0.49 csys = 6.95 CPU) +PERL_DL_NONLAZY=1 /usr/bin/perl -Iblib/arch -Iblib/lib -I/usr/lib/perl5/5.00503/i386-linux -I/usr/lib/perl5/5.00503 test.pl +test.pl +DBI test application $Revision: 1.15 $ +Using /home/user/DBI-1.14/blib +Switch: DBI 1.14 by Tim Bunce, 1.14 +Available Drivers: ADO, ExampleP, Multiplex, Proxy, mysql +dbi:ExampleP:: testing 5 sets of 20 connections: +Connecting... 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 +Disconnecting... +Connecting... 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 +Disconnecting... +Connecting... 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 +Disconnecting... +Connecting... 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 +Disconnecting... +Connecting... 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 +Disconnecting... +Made 100 connections in 0 wallclock secs ( 0.22 usr + 0.03 sys = 0.25 CPU) + +Testing handle creation speed... +5000 NullP statement handles cycled in 6.6 cpu+sys seconds (762 per sec) + +test.pl done + +[root@fenchurch DBI-1.14]# make install +These files are installed. +/usr/lib/perl5/site_perl/5.005/i386-linux/auto/DBI/.packlist +/usr/bin/dbiproxy +/usr/bin/dbish +/usr/lib/perl5/man/man3/Bundle::DBI.3 +/usr/lib/perl5/man/man3/DBD::ADO.3 +/usr/lib/perl5/man/man3/DBD::Multiplex.3 +/usr/lib/perl5/man/man3/DBD::Proxy.3 +/usr/lib/perl5/man/man3/DBI.3 +/usr/lib/perl5/man/man3/DBI::DBD.3 +/usr/lib/perl5/man/man3/DBI::FAQ.3 +/usr/lib/perl5/man/man3/DBI::Format.3 +/usr/lib/perl5/man/man3/DBI::ProxyServer.3 +/usr/lib/perl5/man/man3/DBI::Shell.3 +/usr/lib/perl5/man/man3/DBI::W32ODBC.3 +/usr/lib/perl5/man/man3/Win32::DBIODBC.3 +/usr/lib/perl5/site_perl/5.005/i386-linux/Bundle/DBI.pm +/usr/lib/perl5/site_perl/5.005/i386-linux/DBD/ADO.pm +/usr/lib/perl5/site_perl/5.005/i386-linux/DBD/ExampleP.pm +/usr/lib/perl5/site_perl/5.005/i386-linux/DBD/Multiplex.pm +/usr/lib/perl5/site_perl/5.005/i386-linux/DBD/NullP.pm +/usr/lib/perl5/site_perl/5.005/i386-linux/DBD/Proxy.pm +/usr/lib/perl5/site_perl/5.005/i386-linux/DBD/Sponge.pm +/usr/lib/perl5/site_perl/5.005/i386-linux/DBI.pm +/usr/lib/perl5/site_perl/5.005/i386-linux/DBI/DBD.pm +/usr/lib/perl5/site_perl/5.005/i386-linux/DBI/FAQ.pm +/usr/lib/perl5/site_perl/5.005/i386-linux/DBI/Format.pm +/usr/lib/perl5/site_perl/5.005/i386-linux/DBI/ProxyServer.pm +/usr/lib/perl5/site_perl/5.005/i386-linux/DBI/Shell.pm +/usr/lib/perl5/site_perl/5.005/i386-linux/DBI/W32ODBC.pm +/usr/lib/perl5/site_perl/5.005/i386-linux/Win32/DBIODBC.pm +/usr/lib/perl5/site_perl/5.005/i386-linux/auto/DBI/DBI.bs +/usr/lib/perl5/site_perl/5.005/i386-linux/auto/DBI/DBI.so +/usr/lib/perl5/site_perl/5.005/i386-linux/auto/DBI/DBIXS.h +/usr/lib/perl5/site_perl/5.005/i386-linux/auto/DBI/Driver.xst +/usr/lib/perl5/site_perl/5.005/i386-linux/auto/DBI/dbd_xsh.h +/usr/lib/perl5/site_perl/5.005/i386-linux/auto/DBI/dbi_sql.h +/usr/lib/perl5/site_perl/5.005/i386-linux/auto/DBI/dbipport.h +/usr/man/man1/dbiproxy.1 +/usr/man/man1/dbish.1 ++ + old notes in green
The following set of tarballs was found to work together properly on a LON-CAPA RedHat 6.2 system: @@ -174,13 +523,14 @@ properly on a LON-CAPA RedHat 6.2 system Installation was simply a matter of following the instructions and typing the several "make" commands for each
-Not yet developed. This will be part of an interface present on LON-CAPA systems that can be launched by entering the command /usr/sbin/loncapaconfig.
-This is not complete.
@@ -215,124 +565,9 @@ FLUSH PRIVILEGES; Copy support-files/mysql.server to the right place on the system (/etc/rc.d/...). --Not yet documented or formalized. -
--
- -+The Perl API
-start the mysql daemon as /usr/local/bin/safe_mysqld &
-Login as root: mysql -u root -p mysql
-enter the password as newmysql
-add the user www: grant all priveleges on *.* to www@localhost identified by 'newmysql' with grant option;
-
-INSERT INTO user (Host, User, Password)
-VALUES ('localhost','www',password('newmysql'));
-
-GRANT ALL PRIVILEGES ON *.* TO www@localhost;
-
-FLUSH PRIVILEGES;
-
-Here the user www has the right to grant privileges to other users.
-This can be changed if required with a simple update command on the grant tables
-
-
-/home/httpd/perl/perlsql/lonsql
-/usr/local/mysql/fakeclient
-
-
--
-This is the output from scripts/mysql_install_db... -still some todo things (like support-files/mysql.server) - -Creating db table -Creating host table -Creating user table -Creating func table -Creating tables_priv table -Creating columns_priv table - -To start mysqld at boot time you have to copy support-files/mysql.server -to the right place for your system - -PLEASE REMEMBER TO SET A PASSWORD FOR THE MySQL root USER ! -This is done with: -/usr/local/bin/mysqladmin -u root password 'new-password' -See the manual for more instructions. - -Please report any problems with the /usr/local/bin/mysqlbug script! - -The latest information about MySQL is available on the web at http://www.mysql.com -Support MySQL by buying support/licenses at http://www.tcx.se/license.htmy. -- -
-
-August, 29 2000; Scott Harrison; LON-CAPA
-
-These are notes related to a Perl interface and MySQL server installation
-on Redhat 6.1 and 6.2 boxes. (Guy Albertelli and Harsha Jagasia
-contributed significantly to this.)
-
-********************
-* MySQL COMPONENTS *
-********************
-
-There are three components to an effective MySQL installation for the
-LON-CAPA system.
-
-Perl::DBI module- the API "front-end"...
- database interface module for organizing generic
- database commands which are independent of specific
- database implementation (such as MySQL, mSQL, Postgres, etc).
-
-Perl::MySQL module- the API "mid-section"...
- the module to directly interface with the actual
- MySQL database engine
-
-MySQL database engine- the "back-end"...
- the binary installation (compiled either from source
- or pre-compiled file listings) which provides the
- actual MySQL functionality on the system
-
-RedHat Installation-
-
-Initially done from source:
-DBI-1.13.tar.gz Msql-Mysql-modules-1.2209.tar.gz mysql-3.22.32.tar.gz
-
-I am now using pre-compiled file listings.
-
-There were problems with using the RedHat packages since the three
-different RedHat packages were somewhat noncompatible with each other
-in terms of expected file locations. (The Debian linux distribution,
-on the other hand, has a working set of these packages).
-
-Regardless of how we install these three components, there still remain
-certain things which need to happen for the configuration.
-
-*****************
-* CONFIGURATION *
-*****************
-
-(Note: SOMEPASSWORD is actually set to another text string on the current
-LON-CAPA systems.)
-
-Configuration is needed to generate the necessary functionality for the
-MySQL system with LON-CAPA.
-
-The functionality needed can be understood from this example line
-of perl code from "lonsql".
-
$dbh = DBI->connect( "DBI:mysql:loncapa",
"www",
"SOMEPASSWORD",
@@ -395,56 +630,43 @@ FLUSH PRIVILEGES;
** ABILITY for LON-CAPA machines to communicate with SQL databases on
other LON-CAPA machines
-This is a little more intricate than might first be expected (and I probably
-won't do a perfect job reciting everything in this short synopsis). Because
-LON-CAPA machines will likely be handling many SQL requests at a time,
-there were some problems with current MySQL capabilities.
-
-PROBLEM SITUATION:
-
- If Server A wants data from Server B, Server A uses a lonc process to
- send a database command to a Server B lond process.
- lonc= loncapa client process A-lonc= a lonc process on Server A
- lond= loncapa daemon process
-
- database command
- A-lonc --------TCP/IP----------------> B-lond
-
- The problem emerges that A-lonc and B-lond are kept waiting for the
- MySQL server to "do its stuff", or in other words, perform the conceivably
- sophisticated, data-intensive, time-sucking database transaction. By tying
- up a lonc and lond process, this significantly cripples the capabilities
- of LON-CAPA servers.
-
- While commercial databases have a variety of features that ATTEMPT to
- deal with this, freeware databases are still experimenting and exploring
- with different schemes with varying degrees of performance stability.
-
-THE SOLUTION:
-
- A separate daemon process was created that B-lond works with to
- handle database requests. This daemon process is called "lonsql".
-
- So,
- database command
- A-lonc ---------TCP/IP-----------------> B-lond =====> B-lonsql
- <---------------------------------/ |
- "ok, I'll get back to you..." |
- |
- /
- A-lond <------------------------------- B-lonc <======
- "Guess what? I have the result!"
+An up-to-date lond and lonsql.
+
+
++To test the backend MySQL database, a number of commands should be +run after installation. +
+These are sections of perl code which helps test the LON-CAPA network. +
+** TEST the database connection with my current tester.pl code +which mimics what command will eventually be sent through lonc. +$reply=reply( + "querysend:SELECT * FROM general_information WHERE Id='AAAAA'",$lonID); ++ +
Here are excerpts of code which implement the above handling: - -**LONSQL - +
++
+**LONSQL
A subroutine from "lonsql" which establishes a child process for handling
-database interactions.
+database interactions.
sub make_new_child {
my $pid;
@@ -536,14 +758,19 @@ sub make_new_child {
exit;
}
}
-
-** LOND enabling of MySQL requestsw
-
- This code is part of every lond child process in the way that it parses command request syntax
- sent to it from lonc processes. querysend corresponds to B-lonc sending the result of the query.
- queryreply corresponds to B-lond indicating that it has received the request and will start the
- database transaction (it returns "ok" to A-lonc ($client)).
-
+
+
+** LOND enabling of MySQL requests
+
+This code is part of every lond child process in the
+way that it parses command request syntax sent to it
+from lonc processes. Based on the diagram above, querysend
+corresponds to B-lonc sending the result of the query.
+queryreply corresponds to B-lond indicating that it has
+received the request and will start the database transaction
+(it returns "ok" to
+A-lonc ($client)).
+
# ------------------------------------------------------------------- querysend
} elsif ($userinput =~ /^querysend/) {
my ($cmd,$query)=split(/:/,$userinput);
@@ -563,14 +790,8 @@ sub make_new_child {
print $client "error:$!\n";
}
-
-
-** TEST the database connection with my current tester.pl code which mimics what command will eventually be
- sent through lonc.
-
-$reply=reply(
- "querysend:SELECT * FROM general_information WHERE Id='AAAAA'",$lonID);
+