[BACK]Return to loncapasqldatabase.html CVS log [TXT] [DIR] Up to [LON-CAPA] / doc / build
File:  [LON-CAPA] / doc / build / Attic / loncapasqldatabase.html
Revision 1.3: download - view: text, annotated - select for diffs
Sat Feb 10 17:07:47 2001 UTC (23 years, 8 months ago) by harris41
Branches: MAIN
CVS tags: HEAD
additions to documentation -Scott

<HTML>
<HEAD>
<TITLE>LON-CAPA SQL Database Documentation</TITLE>
</HEAD>
<BODY>
<H1>LON-CAPA SQL Database Documentation</H1>
<P>
Scott Harrison
</P>
<P>
Last updated: 02/07/2001
</P>
<P>
This file describes issues associated with LON-CAPA
and a SQL database.
</P>
<H3>Latest HOWTO</H3>
<P>
I am going to begin documentation by inserting what notes
I have into this file.  I will be subsequently rearranging
them and editting them based on the tests that I conduct.
I am trying to make sure that documentation, installation,
and run-time issues are all consistent and correct.  The
current status of everything is that it works and has
been minimally tested, but things need to be cleaned up
and checked again!
</P>
<H3>How to add a user to the SQL database</H3>
<P>
<PRE>
start the mysql daemon as /usr/local/bin/safe_mysqld &
Login as root: mysql -u root -p mysql
enter the password as newmysql
add the user www: grant all priveleges on *.* to www@localhost identified by 'newmysql' with grant option;

INSERT INTO user (Host, User, Password)
VALUES ('localhost','www',password('newmysql'));

GRANT ALL PRIVILEGES ON *.* TO www@localhost;

FLUSH PRIVILEGES;

Here the user www has the right to grant privileges to other users.
This can be changed if required with a simple update command on the grant tables


/home/httpd/perl/perlsql/lonsql
/usr/local/mysql/fakeclient
</PRE>
</P>
<H3>To do</H3>
<P>
<PRE>
This is the output from scripts/mysql_install_db...
still some todo things (like support-files/mysql.server)

Creating db table
Creating host table
Creating user table
Creating func table
Creating tables_priv table
Creating columns_priv table

To start mysqld at boot time you have to copy support-files/mysql.server
to the right place for your system

PLEASE REMEMBER TO SET A PASSWORD FOR THE MySQL root USER !
This is done with:
/usr/local/bin/mysqladmin -u root password 'new-password'
See the manual for more instructions.

Please report any problems with the /usr/local/bin/mysqlbug script!

The latest information about MySQL is available on the web at http://www.mysql.com
Support MySQL by buying support/licenses at http://www.tcx.se/license.htmy.
</PRE>
</P>
<H3>Source Installation and Manual Configuration</H3>
<P>
<PRE>
August, 29 2000; Scott Harrison; LON-CAPA

These are notes related to a Perl interface and MySQL server installation
on Redhat 6.1 and 6.2 boxes.  (Guy Albertelli and Harsha Jagasia
contributed significantly to this.)

********************
* MySQL COMPONENTS *
********************

There are three components to an effective MySQL installation for the
LON-CAPA system.

Perl::DBI module- the API "front-end"...
                  database interface module for organizing generic
                  database commands which are independent of specific
                  database implementation (such as MySQL, mSQL, Postgres, etc).

Perl::MySQL module- the API "mid-section"...
                    the module to directly interface with the actual
                    MySQL database engine

MySQL database engine- the "back-end"...
                       the binary installation (compiled either from source
                       or pre-compiled file listings) which provides the
                       actual MySQL functionality on the system

RedHat Installation-

Initially done from source:
DBI-1.13.tar.gz  Msql-Mysql-modules-1.2209.tar.gz  mysql-3.22.32.tar.gz

I am now using pre-compiled file listings.

There were problems with using the RedHat packages since the three
different RedHat packages were somewhat noncompatible with each other
in terms of expected file locations. (The Debian linux distribution,
on the other hand, has a working set of these packages).

Regardless of how we install these three components, there still remain
certain things which need to happen for the configuration.

*****************
* CONFIGURATION *
*****************

(Note: SOMEPASSWORD is actually set to another text string on the current
LON-CAPA systems.)

Configuration is needed to generate the necessary functionality for the
MySQL system with LON-CAPA.

The functionality needed can be understood from this example line
of perl code from "lonsql".

   $dbh = DBI->connect(	"DBI:mysql:loncapa",
			"www",
			"SOMEPASSWORD",
			{ RaiseError =>0,PrintError=>0});

There is an obvious need to CONNECT to the database, and in order to do
this, there must be:
  a RUNNING mysql daemon;
  a DATABASE named "loncapa";
  a USER named "www";
  and an ABILITY for LON-CAPA on one machine to access
       SQL database on another machine;
  
So, here are some notes on implementing these configurations.

** RUNNING mysql daemon (safe_mysqld method)

The recommended way to run the MySQL daemon is as a non-root user
(probably www)...

so, 1) login as user www on the linux machine
    2) start the mysql daemon as /usr/local/bin/safe_mysqld &

safe_mysqld only works if the local installation of MySQL is set to the
right directory permissions which I found to be:
chown www:users /usr/local/var/mysql
chown www:users /usr/local/lib/mysql
chown -R www:users /usr/local/mysql
chown www:users /usr/local/include/mysql
chown www:users /usr/local/var

** DATABASE named "loncapa"

As user www, run this command
    mysql -u root -p mysql
enter the password as SOMEPASSWORD

This allows you to manually enter MySQL commands.
The MySQL command to generate the loncapa DATABASE is:

CREATE DATABASE 'loncapa';

** USER named "www"

As user www, run this command
    mysql -u root -p mysql
enter the password as SOMEPASSWORD

To add the user www to the MySQL server, and grant all
privileges on *.* to www@localhost identified by 'SOMEPASSWORD'
with grant option;

INSERT INTO user (Host, User, Password)
VALUES ('localhost','www',password('SOMEPASSWORD'));

GRANT ALL PRIVILEGES ON *.* TO www@localhost;

FLUSH PRIVILEGES;

** ABILITY for LON-CAPA machines to communicate with SQL databases on
   other LON-CAPA machines

This is a little more intricate than might first be expected (and I probably
won't do a perfect job reciting everything in this short synopsis).  Because
LON-CAPA machines will likely be handling many SQL requests at a time,
there were some problems with current MySQL capabilities.

PROBLEM SITUATION:

  If Server A wants data from Server B, Server A uses a lonc process to
  send a database command to a Server B lond process.
    lonc= loncapa client process    A-lonc= a lonc process on Server A
    lond= loncapa daemon process

                 database command
    A-lonc  --------TCP/IP----------------> B-lond

  The problem emerges that A-lonc and B-lond are kept waiting for the
  MySQL server to "do its stuff", or in other words, perform the conceivably
  sophisticated, data-intensive, time-sucking database transaction.  By tying
  up a lonc and lond process, this significantly cripples the capabilities
  of LON-CAPA servers. 

  While commercial databases have a variety of features that ATTEMPT to
  deal with this, freeware databases are still experimenting and exploring
  with different schemes with varying degrees of performance stability.

THE SOLUTION:

  A separate daemon process was created that B-lond works with to
  handle database requests.  This daemon process is called "lonsql".

  So,
                database command
  A-lonc  ---------TCP/IP-----------------> B-lond =====> B-lonsql
         <---------------------------------/                |
           "ok, I'll get back to you..."                    |
                                                            |
                                                            /
  A-lond  <-------------------------------  B-lonc   <======
           "Guess what? I have the result!"

  Of course, depending on success or failure, the messages may vary,
  but the principle remains the same where a separate pool of children
  processes (lonsql's) handle the MySQL database manipulations.

Here are excerpts of code which implement the above handling:

**LONSQL

A subroutine from "lonsql" which establishes a child process for handling
database interactions.

sub make_new_child {
    my $pid;
    my $sigset;
    
    # block signal for fork
    $sigset = POSIX::SigSet->new(SIGINT);
    sigprocmask(SIG_BLOCK, $sigset)
        or die "Can't block SIGINT for fork: $!\n";
    
    die "fork: $!" unless defined ($pid = fork);
    
    if ($pid) {
        # Parent records the child's birth and returns.
        sigprocmask(SIG_UNBLOCK, $sigset)
            or die "Can't unblock SIGINT for fork: $!\n";
        $children{$pid} = 1;
        $children++;
        return;
    } else {
        # Child can *not* return from this subroutine.
        $SIG{INT} = 'DEFAULT';      # make SIGINT kill us as it did before
    
        # unblock signals
        sigprocmask(SIG_UNBLOCK, $sigset)
            or die "Can't unblock SIGINT for fork: $!\n";
	
	
        #open database handle
	# making dbh global to avoid garbage collector
	unless (
		$dbh = DBI->connect("DBI:mysql:loncapa","www","SOMEPASSWORD",{ RaiseError =>0,PrintError=>0})
		) { 
	            my $st=120+int(rand(240));
		    &logthis("<font color=blue>WARNING: Couldn't connect to database  ($st secs): $@</font>");
		    print "database handle error\n";
		    sleep($st);
		    exit;

	  };
	# make sure that a database disconnection occurs with ending kill signals
	$SIG{TERM}=$SIG{INT}=$SIG{QUIT}=$SIG{__DIE__}=\&DISCONNECT;

        # handle connections until we've reached $MAX_CLIENTS_PER_CHILD
        for ($i=0; $i < $MAX_CLIENTS_PER_CHILD; $i++) {
            $client = $server->accept()     or last;
            
            # do something with the connection
	    $run = $run+1;
	    my $userinput = <$client>;
	    chomp($userinput);
	    	    
	    my ($conserver,$querytmp)=split(/&/,$userinput);
	    my $query=unescape($querytmp);

            #send query id which is pid_unixdatetime_runningcounter
	    $queryid = $thisserver;
	    $queryid .="_".($$)."_";
	    $queryid .= time."_";
	    $queryid .= $run;
	    print $client "$queryid\n";
	    
            #prepare and execute the query
	    my $sth = $dbh->prepare($query);
	    my $result;
	    unless ($sth->execute())
	    {
		&logthis("<font color=blue>WARNING: Could not retrieve from database: $@</font>");
		$result="";
	    }
	    else {
		my $r1=$sth->fetchall_arrayref;
		my @r2; map {my $a=$_; my @b=map {escape($_)} @$a; push @r2,join(",", @b)} (@$r1);
		$result=join("&",@r2) . "\n";
	    }
            &reply("queryreply:$queryid:$result",$conserver);

        }
    
        # tidy up gracefully and finish
	
        #close the database handle
	$dbh->disconnect
	   or &logthis("<font color=blue>WARNING: Couldn't disconnect from database  $DBI::errstr ($st secs): $@</font>");
    
        # this exit is VERY important, otherwise the child will become
        # a producer of more and more children, forking yourself into
        # process death.
        exit;
    }
}

** LOND enabling of MySQL requestsw

  This code is part of every lond child process in the way that it parses command request syntax
  sent to it from lonc processes.  querysend corresponds to B-lonc sending the result of the query.
  queryreply corresponds to B-lond indicating that it has received the request and will start the
  database transaction (it returns "ok" to A-lonc ($client)).

# ------------------------------------------------------------------- querysend
                   } elsif ($userinput =~ /^querysend/) {
                       my ($cmd,$query)=split(/:/,$userinput);
		       $query=~s/\n*$//g;
                     print $client sqlreply("$hostid{$clientip}\&$query")."\n";
# ------------------------------------------------------------------ queryreply
                   } elsif ($userinput =~ /^queryreply/) {
                       my ($cmd,$id,$reply)=split(/:/,$userinput); 
		       my $store;
                       my $execdir=$perlvar{'lonDaemons'};
                       if ($store=IO::File->new(">$execdir/tmp/$id")) {
			   print $store $reply;
			   close $store;
			   print $client "ok\n";
		       }
		       else {
			   print $client "error:$!\n";
		       }



** TEST the database connection with my current tester.pl code which mimics what command will eventually be
   sent through lonc.

$reply=reply(
    "querysend:SELECT * FROM general_information WHERE Id='AAAAA'",$lonID);
</PRE>
</P>
</BODY>
</HTML>
FreeBSD-CVSweb <freebsd-cvsweb@FreeBSD.org>