-Last updated: 02/07/2001 +Last updated: 02/12/2001
This file describes issues associated with LON-CAPA and a SQL database.
-+
I am going to begin documentation by inserting what notes I have into this file. I will be subsequently rearranging -them and editting them based on the tests that I conduct. +them and editing them based on the tests that I conduct. I am trying to make sure that documentation, installation, and run-time issues are all consistent and correct. The current status of everything is that it works and has been minimally tested, but things need to be cleaned up and checked again!
-+Need to +
+Right now, a lot of "feasibility" work has been done. +Recipes for manual installation and configuration have +been gathered. Network connectivity of lond->lonsql->lond->lonc +type tests have been performed. A binary installation +has been compiled in an RPM (LON-CAPA-mysql). +The most lacking test in terms of feasibility has +been looking at benchmarks to analyze the load at which +the SQL database can efficiently allow many users to +make simultaneous requests of the metadata database. +
++Documentation has been pieced together over time. But, +as mentioned in the previous section, it needs an +overhaul. +
++The binary installation has some quirks associated with it. +Some of the user permissions are wrong, although this is +benign. Also, other options of binary installation (such +as using binary RPMs put together by others) were dismissed +given the difficulty of getting differing combinations of +these external RPMs to work together. +
++Most configuration questions have been initially worked out +to the point of getting this SQL software component working, +however there may be more optimal approaches than currently +exist. +
++LON-CAPA is meant to distribute A LOT of educational content +to A LOT of people. It is ineffective to directly rely on contents +within the ext2 filesystem to be speedily scanned for +on-the-fly searches of content descriptions. (Simply put, +it takes a cumbersome amount of time to open, read, analyze, and +close thousands of files.) +
++The solution is to hash-index various data fields that are +descriptive of the educational resources on a LON-CAPA server +machine. Descriptive data fields are referred to as +"metadata". The question then arises as to how this metadata +is handled in terms of the rest of the LON-CAPA network +without burdening client and daemon processes. I now +answer this question in the format of Problem and Solution +below. +
-start the mysql daemon as /usr/local/bin/safe_mysqld &
-Login as root: mysql -u root -p mysql
-enter the password as newmysql
-add the user www: grant all priveleges on *.* to www@localhost identified by 'newmysql' with grant option;
+PROBLEM SITUATION:
-INSERT INTO user (Host, User, Password)
-VALUES ('localhost','www',password('newmysql'));
+ If Server A wants data from Server B, Server A uses a lonc process to
+ send a database command to a Server B lond process.
+ lonc= loncapa client process A-lonc= a lonc process on Server A
+ lond= loncapa daemon process
-GRANT ALL PRIVILEGES ON *.* TO www@localhost;
+ database command
+ A-lonc --------TCP/IP----------------> B-lond
-FLUSH PRIVILEGES;
+ The problem emerges that A-lonc and B-lond are kept waiting for the
+ MySQL server to "do its stuff", or in other words, perform the conceivably
+ sophisticated, data-intensive, time-sucking database transaction. By tying
+ up a lonc and lond process, this significantly cripples the capabilities
+ of LON-CAPA servers.
-Here the user www has the right to grant privileges to other users.
-This can be changed if required with a simple update command on the grant tables
+ While commercial databases have a variety of features that ATTEMPT to
+ deal with this, freeware databases are still experimenting and exploring
+ with different schemes with varying degrees of performance stability.
+
+THE SOLUTION:
+ A separate daemon process was created that B-lond works with to
+ handle database requests. This daemon process is called "lonsql".
-/home/httpd/perl/perlsql/lonsql
-/usr/local/mysql/fakeclient
+ So,
+ database command
+ A-lonc ---------TCP/IP-----------------> B-lond =====> B-lonsql
+ <---------------------------------/ |
+ "ok, I'll get back to you..." |
+ |
+ /
+ A-lond <------------------------------- B-lonc <======
+ "Guess what? I have the result!"
+
+ Of course, depending on success or failure, the messages may vary,
+ but the principle remains the same where a separate pool of children
+ processes (lonsql's) handle the MySQL database manipulations.
-+Installation of the LON-CAPA SQL database normally occurs +by default when using the LON-CAPA installation CD +(see http://install.lon-capa.org). It is installed +as the LON-CAPA-mysql RPM. This RPM encodes for the MySQL +engine and related perl interfaces (Perl::DBI, Perl::Msql-Mysql). +
++The three components of a MySQL installation for the +LON-CAPA system are further described immediately below. +
| Perl::DBI module- +the API "front-end"... | |
| database interface module for organizing generic +database commands which are independent of specific +database implementation (such as MySQL, mSQL, Postgres, etc). + | |
| Perl::MySQL module- +the API "mid-section"... | |
| the module to directly interface with the actual +MySQL database engine | |
| MySQL database engine- +the "back-end"... | |
| the binary installation (compiled either +from source or pre-compiled file listings) which provides the +actual MySQL functionality on the system | |
+The following set of tarballs was found to work together +properly on a LON-CAPA RedHat 6.2 system: +
+Installation was simply a matter of following the instructions +and typing the several "make" commands for each +
++Not yet developed. This will be part of an interface +present on LON-CAPA systems that can be launched by +entering the command /usr/sbin/loncapaconfig. +
++This is not complete. +
++Starting the mysql daemon: Login on the Linux +system as user 'www'. Enter the command +/usr/local/bin/safe_mysqld & +
++Set a password for 'root': +/usr/local/bin/mysqladmin -u root password 'new-password' +
++Adding a user: Start the mysql daemon. Login to the +mysql system as root (mysql -u root -p mysql) +and enter the right password (for instance 'newmysql'). Add the user +www
-This is the output from scripts/mysql_install_db...
-still some todo things (like support-files/mysql.server)
-
-Creating db table
-Creating host table
-Creating user table
-Creating func table
-Creating tables_priv table
-Creating columns_priv table
-
-To start mysqld at boot time you have to copy support-files/mysql.server
-to the right place for your system
-
-PLEASE REMEMBER TO SET A PASSWORD FOR THE MySQL root USER !
-This is done with:
-/usr/local/bin/mysqladmin -u root password 'new-password'
-See the manual for more instructions.
-
-Please report any problems with the /usr/local/bin/mysqlbug script!
-
-The latest information about MySQL is available on the web at http://www.mysql.com
-Support MySQL by buying support/licenses at http://www.tcx.se/license.htmy.
+INSERT INTO user (Host, User, Password)
+VALUES ('localhost','www',password('newmysql'));
-+Granting privileges to user 'www': +
+GRANT ALL PRIVILEGES ON *.* TO www@localhost; +FLUSH PRIVILEGES; ++ +
+Set the SQL server to start upon system startup: +Copy support-files/mysql.server to the right place on the system +(/etc/rc.d/...). +
++The Perl API
-August, 29 2000; Scott Harrison; LON-CAPA
-
-These are notes related to a Perl interface and MySQL server installation
-on Redhat 6.1 and 6.2 boxes. (Guy Albertelli and Harsha Jagasia
-contributed significantly to this.)
-
-********************
-* MySQL COMPONENTS *
-********************
-
-There are three components to an effective MySQL installation for the
-LON-CAPA system.
-
-Perl::DBI module- the API "front-end"...
- database interface module for organizing generic
- database commands which are independent of specific
- database implementation (such as MySQL, mSQL, Postgres, etc).
-
-Perl::MySQL module- the API "mid-section"...
- the module to directly interface with the actual
- MySQL database engine
-
-MySQL database engine- the "back-end"...
- the binary installation (compiled either from source
- or pre-compiled file listings) which provides the
- actual MySQL functionality on the system
-
-RedHat Installation-
-
-Initially done from source:
-DBI-1.13.tar.gz Msql-Mysql-modules-1.2209.tar.gz mysql-3.22.32.tar.gz
-
-I am now using pre-compiled file listings.
-
-There were problems with using the RedHat packages since the three
-different RedHat packages were somewhat noncompatible with each other
-in terms of expected file locations. (The Debian linux distribution,
-on the other hand, has a working set of these packages).
-
-Regardless of how we install these three components, there still remain
-certain things which need to happen for the configuration.
-
-*****************
-* CONFIGURATION *
-*****************
-
-(Note: SOMEPASSWORD is actually set to another text string on the current
-LON-CAPA systems.)
-
-Configuration is needed to generate the necessary functionality for the
-MySQL system with LON-CAPA.
-
-The functionality needed can be understood from this example line
-of perl code from "lonsql".
-
$dbh = DBI->connect( "DBI:mysql:loncapa",
"www",
"SOMEPASSWORD",
@@ -195,56 +290,28 @@ FLUSH PRIVILEGES;
** ABILITY for LON-CAPA machines to communicate with SQL databases on
other LON-CAPA machines
-This is a little more intricate than might first be expected (and I probably
-won't do a perfect job reciting everything in this short synopsis). Because
-LON-CAPA machines will likely be handling many SQL requests at a time,
-there were some problems with current MySQL capabilities.
-
-PROBLEM SITUATION:
-
- If Server A wants data from Server B, Server A uses a lonc process to
- send a database command to a Server B lond process.
- lonc= loncapa client process A-lonc= a lonc process on Server A
- lond= loncapa daemon process
-
- database command
- A-lonc --------TCP/IP----------------> B-lond
-
- The problem emerges that A-lonc and B-lond are kept waiting for the
- MySQL server to "do its stuff", or in other words, perform the conceivably
- sophisticated, data-intensive, time-sucking database transaction. By tying
- up a lonc and lond process, this significantly cripples the capabilities
- of LON-CAPA servers.
-
- While commercial databases have a variety of features that ATTEMPT to
- deal with this, freeware databases are still experimenting and exploring
- with different schemes with varying degrees of performance stability.
-
-THE SOLUTION:
-
- A separate daemon process was created that B-lond works with to
- handle database requests. This daemon process is called "lonsql".
-
- So,
- database command
- A-lonc ---------TCP/IP-----------------> B-lond =====> B-lonsql
- <---------------------------------/ |
- "ok, I'll get back to you..." |
- |
- /
- A-lond <------------------------------- B-lonc <======
- "Guess what? I have the result!"
-
- Of course, depending on success or failure, the messages may vary,
- but the principle remains the same where a separate pool of children
- processes (lonsql's) handle the MySQL database manipulations.
+An up-to-date lond and lonsql.
+
+
++
+** TEST the database connection with my current tester.pl code +which mimics what command will eventually be sent through lonc. +$reply=reply( + "querysend:SELECT * FROM general_information WHERE Id='AAAAA'",$lonID); ++ +
Here are excerpts of code which implement the above handling: - -**LONSQL - +
++
+**LONSQL
A subroutine from "lonsql" which establishes a child process for handling
-database interactions.
+database interactions.
sub make_new_child {
my $pid;
@@ -336,14 +403,19 @@ sub make_new_child {
exit;
}
}
-
-** LOND enabling of MySQL requestsw
-
- This code is part of every lond child process in the way that it parses command request syntax
- sent to it from lonc processes. querysend corresponds to B-lonc sending the result of the query.
- queryreply corresponds to B-lond indicating that it has received the request and will start the
- database transaction (it returns "ok" to A-lonc ($client)).
-
+
+
+** LOND enabling of MySQL requests
+
+This code is part of every lond child process in the
+way that it parses command request syntax sent to it
+from lonc processes. Based on the diagram above, querysend
+corresponds to B-lonc sending the result of the query.
+queryreply corresponds to B-lond indicating that it has
+received the request and will start the database transaction
+(it returns "ok" to
+A-lonc ($client)).
+
# ------------------------------------------------------------------- querysend
} elsif ($userinput =~ /^querysend/) {
my ($cmd,$query)=split(/:/,$userinput);
@@ -363,14 +435,8 @@ sub make_new_child {
print $client "error:$!\n";
}
-
-
-** TEST the database connection with my current tester.pl code which mimics what command will eventually be
- sent through lonc.
-
-$reply=reply(
- "querysend:SELECT * FROM general_information WHERE Id='AAAAA'",$lonID);
+