Annotation of doc/build/loncapasqldatabase.html, revision 1.13
1.1 harris41 1: <HTML>
2: <HEAD>
3: <TITLE>LON-CAPA SQL Database Documentation</TITLE>
4: </HEAD>
5: <BODY>
6: <H1>LON-CAPA SQL Database Documentation</H1>
7: <P>
8: Scott Harrison
9: </P>
10: <P>
1.12 harris41 11: Last updated: 02/15/2001
1.1 harris41 12: </P>
1.2 harris41 13: <P>
1.1 harris41 14: This file describes issues associated with LON-CAPA
15: and a SQL database.
16: </P>
1.5 harris41 17: <H2>Latest HOWTO</H2>
1.2 harris41 18: <P>
1.4 harris41 19: <UL>
1.9 harris41 20: <LI>Important notes
1.4 harris41 21: <LI>Current status of documentation</LI>
22: <LI>Current status of implementation</LI>
23: <LI>Purpose within LON-CAPA</LI>
1.8 harris41 24: <LI>Dependencies</LI>
1.4 harris41 25: <LI>Installation</LI>
26: <LI>Installation from source</LI>
27: <LI>Configuration (automated)</LI>
28: <LI>Manual configuration</LI>
29: <LI>Testing</LI>
30: <LI>Example sections of code relevant to LON-CAPA</LI>
31: </UL>
32: </P>
1.9 harris41 33: <H2>Important notes</H2>
34: <P>
1.12 harris41 35: The current database is implemented assuming a non-adjustable
36: architecture involving these data fields (specific to each version
37: of a resource).
38: <UL>
39: <LI>title</LI>
40: <LI>author</LI>
41: <LI>subject</LI>
42: <LI>notes</LI>
43: <LI>abstract</LI>
44: <LI>mime</LI>
45: <LI>language</LI>
46: <LI>creationdate</LI>
47: <LI>lastrevisiondate</LI>
48: <LI>owner</LI>
49: <LI>copyright</LI>
50: </UL>
51: </P>
52: <P>
53: These commands create the loncapameta database.
54: <PRE>
1.13 ! harris41 55: mysql> CREATE DATABASE IF NOT EXISTS loncapa;
! 56: mysql> USE loncapa;
! 57: mysql> CREATE TABLE IF NOT EXISTS metadata (title TEXT, author TEXT, subject TEXT, notes TEXT, abstract TEXT, mime TEXT, language TEXT, creationdate TEXT, lastrevisiondate TEXT, owner TEXT, copyright TEXT, FULLTEXT (title, author, subject, notes, abstract, mime, language, creationdate, lastrevisiondate, owner, copyright)) TYPE=MYISAM;
! 58: mysql> INSERT INTO metadata VALUES ('1','2','3','4','5','6','7','8','9','10','11');
! 59: mysql> SELECT * FROM metadata WHERE title REGEXP "1";
! 60: </PRE>
! 61: </P>
! 62: <P>
! 63: Current time values for things.
! 64: <PRE>
! 65: fenchurch.lite.msu.edu
! 66: Mem: 46812K av, 45632K used, 1180K free, 14756K shrd, 4292K buff
! 67: Swap: 1148608K av, 11260K used, 1137348K free 13244K cached
1.12 harris41 68:
1.13 ! harris41 69: Red Hat Linux release 6.2 (Zoot)
! 70: Kernel 2.2.16-3 on an i586
1.12 harris41 71: </PRE>
1.13 ! harris41 72: <BR>Q: How big are data records in test database? A: on average, 1000 bytes each, medline records from PubMed.
! 73: <BR>Q: How big is the biggest field? A: on average, 838 bytes each
! 74: <BR>Q: How much time to insert 5284 medline records into database?
! 75: A: 600 seconds
! 76: <BR>Q: What about when using "speed-technique" on page 130? A: 689 seconds (weird, eh?)
! 77: <BR>Q: What about REGEXP searching? A: about 1-2 seconds for small fields;
! 78: 10 to 20 seconds for REGEXP search on "abstract" field
! 79: <BR>Q: What about FULLTEXT indexing? A: about 6 seconds for abstract field.
! 80:
1.12 harris41 81: </P>
82: <P>
83: An important quote from the manual:
84: <BLOCKQUOTE>
85: In MySQL Version 3.23.23 or later, you can also create special FULLTEXT indexes. They are used for full-text search. Only the MyISAM table type supports FULLTEXT indexes. They can be created only from
86: VARCHAR and TEXT columns. Indexing always happens over the entire column and partial indexing is not supported. See section 25.2 MySQL Full-text Search for details.
87: </BLOCKQUOTE>
88: </P>
89: <P>
90: I plan on using a MyISAM table type with 11 metadata fields of column
91: type=TEXT.
92: </P>
93: <P>
1.9 harris41 94: It might be worthwhile to look at /usr/local/mysql/manual.html.
95: It is quite in depth.
96: </P>
1.5 harris41 97: <H2>Current status of documentation</H2>
1.4 harris41 98: <P>
1.2 harris41 99: I am going to begin documentation by inserting what notes
100: I have into this file. I will be subsequently rearranging
1.4 harris41 101: them and editing them based on the tests that I conduct.
1.2 harris41 102: I am trying to make sure that documentation, installation,
103: and run-time issues are all consistent and correct. The
104: current status of everything is that it works and has
105: been minimally tested, but things need to be cleaned up
106: and checked again!
107: </P>
1.5 harris41 108: <H2>Current status of implementation</H2>
1.4 harris41 109: <P>
1.6 harris41 110: Need to
111: <UL>
112: <LI>Installation: Fix binary file listings for user permissions and ownership.
113: <LI>Installation: Make sure sql server starts, and if database does not
114: exist, then create. (/etc/rc.d).
115: <LI>Processes: Make sure loncron initiates lonsql on library machines.
116: <LI>Read in metadata from right place periodically.
117: <LI>Implement tested perl module handler.
118: </UL>
119: <P>
1.4 harris41 120: Right now, a lot of "feasibility" work has been done.
121: Recipes for manual installation and configuration have
122: been gathered. Network connectivity of lond->lonsql->lond->lonc
123: type tests have been performed. A binary installation
1.7 harris41 124: has been compiled in an RPM (LON-CAPA-mysql, with perl components
125: a part of LON-CAPA-systemperl).
1.4 harris41 126: The most lacking test in terms of feasibility has
127: been looking at benchmarks to analyze the load at which
128: the SQL database can efficiently allow many users to
129: make simultaneous requests of the metadata database.
130: </P>
131: <P>
132: Documentation has been pieced together over time. But,
133: as mentioned in the previous section, it needs an
134: overhaul.
135: </P>
136: <P>
137: The binary installation has some quirks associated with it.
138: Some of the user permissions are wrong, although this is
139: benign. Also, other options of binary installation (such
140: as using binary RPMs put together by others) were dismissed
141: given the difficulty of getting differing combinations of
142: these external RPMs to work together.
143: </P>
144: <P>
145: Most configuration questions have been initially worked out
146: to the point of getting this SQL software component working,
147: however there may be more optimal approaches than currently
148: exist.
149: </P>
1.5 harris41 150: <H2>Purpose within LON-CAPA</H2>
1.4 harris41 151: <P>
152: LON-CAPA is meant to distribute A LOT of educational content
153: to A LOT of people. It is ineffective to directly rely on contents
154: within the ext2 filesystem to be speedily scanned for
155: on-the-fly searches of content descriptions. (Simply put,
156: it takes a cumbersome amount of time to open, read, analyze, and
157: close thousands of files.)
158: </P>
159: <P>
160: The solution is to hash-index various data fields that are
161: descriptive of the educational resources on a LON-CAPA server
162: machine. Descriptive data fields are referred to as
163: "metadata". The question then arises as to how this metadata
164: is handled in terms of the rest of the LON-CAPA network
165: without burdening client and daemon processes. I now
166: answer this question in the format of Problem and Solution
167: below.
168: </P>
169: <P>
170: <PRE>
171: PROBLEM SITUATION:
172:
173: If Server A wants data from Server B, Server A uses a lonc process to
174: send a database command to a Server B lond process.
175: lonc= loncapa client process A-lonc= a lonc process on Server A
176: lond= loncapa daemon process
177:
178: database command
179: A-lonc --------TCP/IP----------------> B-lond
180:
181: The problem emerges that A-lonc and B-lond are kept waiting for the
182: MySQL server to "do its stuff", or in other words, perform the conceivably
183: sophisticated, data-intensive, time-sucking database transaction. By tying
184: up a lonc and lond process, this significantly cripples the capabilities
185: of LON-CAPA servers.
186:
187: While commercial databases have a variety of features that ATTEMPT to
188: deal with this, freeware databases are still experimenting and exploring
189: with different schemes with varying degrees of performance stability.
190:
191: THE SOLUTION:
192:
193: A separate daemon process was created that B-lond works with to
194: handle database requests. This daemon process is called "lonsql".
195:
196: So,
197: database command
198: A-lonc ---------TCP/IP-----------------> B-lond =====> B-lonsql
199: <---------------------------------/ |
200: "ok, I'll get back to you..." |
201: |
202: /
203: A-lond <------------------------------- B-lonc <======
204: "Guess what? I have the result!"
205:
206: Of course, depending on success or failure, the messages may vary,
207: but the principle remains the same where a separate pool of children
208: processes (lonsql's) handle the MySQL database manipulations.
209: </PRE>
210: </P>
1.8 harris41 211: <H2>Dependencies</H2>
212: <P>
213: I believe (but am not 100% confident) that the following
214: RPMs are necessary (in addition to the current ones
215: in rpm_list.txt) to run MySQL. Basically I discovered these
216: dependencies while trying to do external RPM based installs.
217: I assume, and sometimes found, that these dependencies apply
218: to tarball-based distributions too. (So to play it on the
219: safe side, I am going to include these RPMs as part of the
220: core, minimal RPM set.)
221: <UL>
222: <LI>egcs-1.1.2-30</LI>
223: <LI>cpp-1.1.2-30</LI>
224: <LI>glibc-devel-2.1.3-15</LI>
225: <LI>zlib-devel-1.1.3-6</LI>
226: </UL>
227: </P>
1.5 harris41 228: <H2>Installation</H2>
1.4 harris41 229: <P>
230: Installation of the LON-CAPA SQL database normally occurs
231: by default when using the LON-CAPA installation CD
232: (see http://install.lon-capa.org). It is installed
233: as the LON-CAPA-mysql RPM. This RPM encodes for the MySQL
1.7 harris41 234: engine. Related perl interfaces (Perl::DBI, Perl::Msql-Mysql)
235: are encoded in the LON-CAPA-systemperl RPM.
1.4 harris41 236: </P>
237: <P>
238: The three components of a MySQL installation for the
239: LON-CAPA system are further described immediately below.
240: <TABLE BORDER="0">
241: <TR><TD COLSPAN="2"><STRONG>Perl::DBI module</STRONG>-
242: the API "front-end"...</TD></TR>
243: <TR><TD WIDTH="10%"></TD><TD>database interface module for organizing generic
244: database commands which are independent of specific
245: database implementation (such as MySQL, mSQL, Postgres, etc).
246: </TD></TR>
247: <TR><TD COLSPAN="2"><STRONG>Perl::MySQL module</STRONG>-
248: the API "mid-section"...</TD></TR>
249: <TR><TD WIDTH="10%"></TD><TD>the module to directly interface with the actual
250: MySQL database engine</TD></TR>
251: <TR><TD COLSPAN="2"><STRONG>MySQL database engine</STRONG>-
252: the "back-end"...</TD></TR>
253: <TR><TD WIDTH="10%"></TD><TD>the binary installation (compiled either
254: from source or pre-compiled file listings) which provides the
255: actual MySQL functionality on the system</TD></TR>
256: </TABLE>
257: </P>
1.5 harris41 258: <H2>Installation from source</H2>
1.4 harris41 259: <P>
1.8 harris41 260: Note: the mysql site recommends that Linux users install by
261: using the MySQL RPMs (MySQL-client, MySQL, MySQL-shared, etc).
262: While these RPMs work, I was unsuccessful at integrating
263: this RPM-installed database with perl modules from www.cpan.org.
264: Hence, I <STRONG>strongly</STRONG> recommend that, when installing
265: from "source", MySQL and the perl components be in fact installed
266: from their tarballs (.tar.gz, .tgz). (Perl components, when installed
267: from RPMs, also wound up in incorrect locations on the disk.)
268: Do not coordinate a source install with externally made RPMs!
269: It is, of course, okay to use LON-CAPA RPMs such as LON-CAPA-systemperl
270: and LON-CAPA-mysql since we, in fact, made these RPMs correctly :).
271: <UL>
272: <LI>http://www.cpan.org/authors/id/JWIED/Msql-Mysql-modules-1.2215.tar.gz
273: <BR>This tarball Released 20th August 2000
274: <LI>http://www.mysql.com/Downloads/MySQL-3.23/mysql-3.23.33-pc-linux-gnu-i686.tar.gz
275: <BR>This tarball Last changed 2000-11-11
276: <BR>This is actually a binary tarball (as opposed to source code
277: that is subsequently compiled).
1.9 harris41 278: <LI>http://www.cpan.org/authors/id/TIMB/DBI-1.14.tar.gz
279: <BR>This tarball Released 14th June 2000
1.8 harris41 280: </UL>
1.9 harris41 281: </P>
282: <P>So, here is exactly how I installed MySQL-3.23. (Note that all files
283: wind up in /usr/local/mysql-3.23.33-pc-linux-gnu-i686 except for
284: a link from /usr/local/mysql to /usr/local/mysql-3.23.33-pc-linux-gnu-i686
285: and some files involved in system process handling (/etc/rc.d/*/*mysql).
1.11 harris41 286: <PRE>
287: (As user=root)
288: cd /usr/local/;
289: tar xzvf mysql-3.23.33-pc-linux-gnu-i686.tar.gz
290: ln -s /usr/local/mysql-3.23.33-pc-linux-gnu-i686 mysql
291: cd /usr/local/mysql
292: chown -R www /usr/local/mysql/.
293: chgrp -R users /usr/local/mysql/.
294: chmod -R g-w,g-r,g-x /usr/local/mysql/.
295: (probably also want chmod -R a-w,a-r,a-x /usr/local/mysql/.)
296:
297: Alter <TT>safe_mysqld</TT>
298: and <TT>support-files/mysql.server</TT> to use 'localhosts' instead
299: of `bin/hostname`. Also, to use user 'www' instead of 'mysql'.
300:
301: (These changes could be done with /etc/my.cnf, but
302: I think this approach makes sure the database NEVER
303: gets screwed up due to somebody forgetting to install /etc/my.cnf).
304:
305: Change this line in mysql.server from
306: pid_file=$datadir/`@HOSTNAME@`.pid
307: to
308: pid_file=$datadir/localhost.pid
309:
310: Change this line in safe_mysqld from
311: user=root
312: to
313: user=www
314:
315: Change this line in safe_mysqld from
316: pid_file=$DATADIR/`/bin/hostname`.pid
317: to
318: pid_file=$DATADIR/localhost.pid
319:
320: Change this line in safe_mysqld from
321: test -z "$err_log" && err_log=$DATADIR/`/bin/hostname`.err
322: to
323: test -z "$err_log" && err_log=$DATADIR/localhost.err
324:
325: cp -p support-files/mysql.server /etc/rc.d/init.d/mysql
326: chmod 755 /etc/rc.d/init.d/mysql
327: /sbin/chkconfig --add mysql
328: /etc/rc.d/init.d/mysql start
329:
330: </PRE>
1.9 harris41 331: </P>
332: <P>This is how I installed the Msql-Mysql-modules perl modules.
333: <PRE>
334: [root@fenchurch Msql-Mysql-modules-1.2215]# perl Makefile.PL
335: Which drivers do you want to install?
336:
337: 1) MySQL only
338: 2) mSQL only (either of mSQL 1 or mSQL 2)
339: 3) MySQL and mSQL (either of mSQL 1 or mSQL 2)
340:
341: 4) mSQL 1 and mSQL 2
342: 5) MySQL, mSQL 1 and mSQL 2
343:
344: Enter the appropriate number: [3] 1
345:
346:
347: Do you want to install the MysqlPerl emulation? You might keep your old
348: Mysql module (to be distinguished from DBD::mysql!) if you are concerned
349: about compatibility to existing applications! [y] n
1.10 harris41 350: Where is your MySQL installed? Please tell me the directory that
351: contains the subdir 'include'. [/usr/local/mysql]
352: Which database should I use for testing the MySQL drivers? [test]
353: On which host is database test running (hostname, ip address
354: or host:port) [localhost]
1.11 harris41 355: User name for connecting to database test? [undef]
356: Password for connecting to database test? [undef]
1.9 harris41 357: [root@fenchurch Msql-Mysql-modules-1.2215]# make
358: [root@fenchurch Msql-Mysql-modules-1.2215]# make test
359: make[1]: Entering directory `/home/user/Msql-Mysql-modules-1.2215/mysql'
360: make[1]: Leaving directory `/home/user/Msql-Mysql-modules-1.2215/mysql'
361: make[1]: Entering directory `/home/user/Msql-Mysql-modules-1.2215/mysql'
362: PERL_DL_NONLAZY=1 /usr/bin/perl -I../blib/arch -I../blib/lib -I/usr/lib/perl5/5.00503/i386-linux -I/usr/lib/perl5/5.00503 -e 'use Test::Harness qw(&runtests $verbose); $verbose=0; runtests @ARGV;' t/*.t
363: t/00base............ok
364: t/10dsnlist.........ok
365: t/20createdrop......ok
366: t/30insertfetch.....ok
367: t/40bindparam.......ok
368: t/40blobs...........ok
369: t/40listfields......ok
370: t/40nulls...........ok
371: t/40numrows.........ok
372: t/50chopblanks......ok
373: t/50commit..........ok
374: t/60leaks...........skipping test on this platform
375: t/ak-dbd............ok
376: t/akmisc............ok
377: t/dbdadmin..........ok
378: t/mysql.............ok
379: t/mysql2............ok
380: All tests successful, 1 test skipped.
381: Files=17, Tests=732, 40 wallclock secs (15.38 cusr + 1.30 csys = 16.68 CPU)
382: [root@fenchurch Msql-Mysql-modules-1.2215]# make install
383:
384: These files are installed.
385: /usr/bin/dbimon
386: /usr/lib/perl5/man/man3/Bundle::DBD::mysql.3
387: /usr/lib/perl5/man/man3/DBD::mysql.3
388: /usr/lib/perl5/man/man3/Mysql.3
389: /usr/lib/perl5/site_perl/5.005/i386-linux/Bundle/DBD/mysql.pm
390: /usr/lib/perl5/site_perl/5.005/i386-linux/DBD/mysql.pm
391: /usr/lib/perl5/site_perl/5.005/i386-linux/Mysql.pm
392: /usr/lib/perl5/site_perl/5.005/i386-linux/Mysql/Statement.pm
393: /usr/lib/perl5/site_perl/5.005/i386-linux/auto/DBD/mysql/mysql.bs
394: /usr/lib/perl5/site_perl/5.005/i386-linux/auto/DBD/mysql/mysql.so
395: /usr/man/man1/dbimon.1
396: /usr/lib/perl5/site_perl/5.005/i386-linux/auto/Msql-Mysql-modules/.packlist
397: </PRE>
398: </P>
399: <P>
400: This is how I installed the DBI perl modules.
401: <PRE>
402: [root@fenchurch DBI-1.14]# perl Makefile.PL
403: *** Note:
404: The optional PlRPC-modules (RPC::PlServer etc) are not installed.
405: If you want to use the DBD::Proxy driver and DBI::ProxyServer
406: modules, then you'll need to install the RPC::PlServer, RPC::PlClient,
407: Storable and Net::Daemon modules. The CPAN Bundle::DBI may help you.
408: You can install them any time after installing the DBI.
409: You do *not* need these modules for typical DBI usage.
410:
411: Optional modules are available from any CPAN mirror, in particular
412: http://www.perl.com/CPAN/modules/by-module
413: http://www.perl.org/CPAN/modules/by-module
414: ftp://ftp.funet.fi/pub/languages/perl/CPAN/modules/by-module
415:
416: Checking if your kit is complete...
417: Looks good
418: Writing Makefile for DBI
419:
420: Remember to actually *read* the README file!
421: Use 'make' to build the software (dmake or nmake on Windows).
422: Then 'make test' to execute self tests.
423: Then 'make install' to install the DBI and then delete this working
424: directory before unpacking and building any DBD::* drivers.
425:
426: [root@fenchurch DBI-1.14]# make
427: [root@fenchurch DBI-1.14]# make test
428: PERL_DL_NONLAZY=1 /usr/bin/perl -Iblib/arch -Iblib/lib -I/usr/lib/perl5/5.00503/i386-linux -I/usr/lib/perl5/5.00503 -e 'use Test::Harness qw(&runtests $verbose); $verbose=0; runtests @ARGV;' t/*.t
429: t/basics............ok
430: t/dbidrv............ok
431: t/examp.............ok
432: t/meta..............ok
433: t/proxy.............skipping test on this platform
434: t/shell.............ok
435: t/subclass..........ok
436: All tests successful, 1 test skipped.
437: Files=7, Tests=179, 7 wallclock secs ( 6.46 cusr + 0.49 csys = 6.95 CPU)
438: PERL_DL_NONLAZY=1 /usr/bin/perl -Iblib/arch -Iblib/lib -I/usr/lib/perl5/5.00503/i386-linux -I/usr/lib/perl5/5.00503 test.pl
439: test.pl
1.13 ! harris41 440: DBI test application $Revision: 1.12 $
1.9 harris41 441: Using /home/user/DBI-1.14/blib
442: Switch: DBI 1.14 by Tim Bunce, 1.14
443: Available Drivers: ADO, ExampleP, Multiplex, Proxy, mysql
444: dbi:ExampleP:: testing 5 sets of 20 connections:
445: Connecting... 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20
446: Disconnecting...
447: Connecting... 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20
448: Disconnecting...
449: Connecting... 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20
450: Disconnecting...
451: Connecting... 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20
452: Disconnecting...
453: Connecting... 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20
454: Disconnecting...
455: Made 100 connections in 0 wallclock secs ( 0.22 usr + 0.03 sys = 0.25 CPU)
456:
457: Testing handle creation speed...
458: 5000 NullP statement handles cycled in 6.6 cpu+sys seconds (762 per sec)
459:
460: test.pl done
461:
462: [root@fenchurch DBI-1.14]# make install
463: These files are installed.
464: /usr/lib/perl5/site_perl/5.005/i386-linux/auto/DBI/.packlist
465: /usr/bin/dbiproxy
466: /usr/bin/dbish
467: /usr/lib/perl5/man/man3/Bundle::DBI.3
468: /usr/lib/perl5/man/man3/DBD::ADO.3
469: /usr/lib/perl5/man/man3/DBD::Multiplex.3
470: /usr/lib/perl5/man/man3/DBD::Proxy.3
471: /usr/lib/perl5/man/man3/DBI.3
472: /usr/lib/perl5/man/man3/DBI::DBD.3
473: /usr/lib/perl5/man/man3/DBI::FAQ.3
474: /usr/lib/perl5/man/man3/DBI::Format.3
475: /usr/lib/perl5/man/man3/DBI::ProxyServer.3
476: /usr/lib/perl5/man/man3/DBI::Shell.3
477: /usr/lib/perl5/man/man3/DBI::W32ODBC.3
478: /usr/lib/perl5/man/man3/Win32::DBIODBC.3
479: /usr/lib/perl5/site_perl/5.005/i386-linux/Bundle/DBI.pm
480: /usr/lib/perl5/site_perl/5.005/i386-linux/DBD/ADO.pm
481: /usr/lib/perl5/site_perl/5.005/i386-linux/DBD/ExampleP.pm
482: /usr/lib/perl5/site_perl/5.005/i386-linux/DBD/Multiplex.pm
483: /usr/lib/perl5/site_perl/5.005/i386-linux/DBD/NullP.pm
484: /usr/lib/perl5/site_perl/5.005/i386-linux/DBD/Proxy.pm
485: /usr/lib/perl5/site_perl/5.005/i386-linux/DBD/Sponge.pm
486: /usr/lib/perl5/site_perl/5.005/i386-linux/DBI.pm
487: /usr/lib/perl5/site_perl/5.005/i386-linux/DBI/DBD.pm
488: /usr/lib/perl5/site_perl/5.005/i386-linux/DBI/FAQ.pm
489: /usr/lib/perl5/site_perl/5.005/i386-linux/DBI/Format.pm
490: /usr/lib/perl5/site_perl/5.005/i386-linux/DBI/ProxyServer.pm
491: /usr/lib/perl5/site_perl/5.005/i386-linux/DBI/Shell.pm
492: /usr/lib/perl5/site_perl/5.005/i386-linux/DBI/W32ODBC.pm
493: /usr/lib/perl5/site_perl/5.005/i386-linux/Win32/DBIODBC.pm
494: /usr/lib/perl5/site_perl/5.005/i386-linux/auto/DBI/DBI.bs
495: /usr/lib/perl5/site_perl/5.005/i386-linux/auto/DBI/DBI.so
496: /usr/lib/perl5/site_perl/5.005/i386-linux/auto/DBI/DBIXS.h
497: /usr/lib/perl5/site_perl/5.005/i386-linux/auto/DBI/Driver.xst
498: /usr/lib/perl5/site_perl/5.005/i386-linux/auto/DBI/dbd_xsh.h
499: /usr/lib/perl5/site_perl/5.005/i386-linux/auto/DBI/dbi_sql.h
500: /usr/lib/perl5/site_perl/5.005/i386-linux/auto/DBI/dbipport.h
501: /usr/man/man1/dbiproxy.1
502: /usr/man/man1/dbish.1
503: </PRE>
1.8 harris41 504: </P>
505: <FONT COLOR="green"> old notes in green
506: <P>
1.4 harris41 507: The following set of tarballs was found to work together
508: properly on a LON-CAPA RedHat 6.2 system:
509: <UL>
510: <LI>DBI-1.13.tar.gz
511: <LI>Msql-Mysql-modules-1.2209.tar.gz
512: <LI>mysql-3.22.32.tar.gz
513: </UL>
514: </P>
515: <P>
516: Installation was simply a matter of following the instructions
517: and typing the several "make" commands for each
518: </P>
1.8 harris41 519: </FONT>
1.5 harris41 520: <H2>Configuration (automated)</H2>
1.4 harris41 521: <P>
522: Not yet developed. This will be part of an interface
523: present on LON-CAPA systems that can be launched by
524: entering the command <TT>/usr/sbin/loncapaconfig</TT>.
525: </P>
1.5 harris41 526: <H2>Manual configuration</H2>
1.4 harris41 527: <P>
528: This is not complete.
529: </P>
530: <P>
531: <STRONG>Starting the mysql daemon</STRONG>: Login on the Linux
532: system as user 'www'. Enter the command
533: <TT>/usr/local/bin/safe_mysqld &</TT>
534: </P>
535: <P>
536: <STRONG>Set a password for 'root'</STRONG>:
537: <TT>/usr/local/bin/mysqladmin -u root password 'new-password'</TT>
538: </P>
539: <P>
540: <STRONG>Adding a user</STRONG>: Start the mysql daemon. Login to the
541: mysql system as root (<TT>mysql -u root -p mysql</TT>)
542: and enter the right password (for instance 'newmysql'). Add the user
543: www
544: <PRE>
545: INSERT INTO user (Host, User, Password)
546: VALUES ('localhost','www',password('newmysql'));
547: </PRE>
548: </P>
549: <P>
550: <STRONG>Granting privileges to user 'www'</STRONG>:
551: <PRE>
552: GRANT ALL PRIVILEGES ON *.* TO www@localhost;
553: FLUSH PRIVILEGES;
554: </PRE>
555: </P>
556: <P>
557: <STRONG>Set the SQL server to start upon system startup</STRONG>:
558: Copy support-files/mysql.server to the right place on the system
559: (/etc/rc.d/...).
560: </P>
1.1 harris41 561: <P>
1.5 harris41 562: <STRONG>The Perl API</STRONG>
1.2 harris41 563: <PRE>
564: $dbh = DBI->connect( "DBI:mysql:loncapa",
565: "www",
566: "SOMEPASSWORD",
567: { RaiseError =>0,PrintError=>0});
568:
569: There is an obvious need to CONNECT to the database, and in order to do
570: this, there must be:
571: a RUNNING mysql daemon;
572: a DATABASE named "loncapa";
573: a USER named "www";
574: and an ABILITY for LON-CAPA on one machine to access
575: SQL database on another machine;
576:
577: So, here are some notes on implementing these configurations.
578:
579: ** RUNNING mysql daemon (safe_mysqld method)
580:
581: The recommended way to run the MySQL daemon is as a non-root user
582: (probably www)...
583:
584: so, 1) login as user www on the linux machine
585: 2) start the mysql daemon as /usr/local/bin/safe_mysqld &
586:
587: safe_mysqld only works if the local installation of MySQL is set to the
588: right directory permissions which I found to be:
589: chown www:users /usr/local/var/mysql
590: chown www:users /usr/local/lib/mysql
591: chown -R www:users /usr/local/mysql
592: chown www:users /usr/local/include/mysql
593: chown www:users /usr/local/var
594:
595: ** DATABASE named "loncapa"
596:
597: As user www, run this command
598: mysql -u root -p mysql
599: enter the password as SOMEPASSWORD
600:
601: This allows you to manually enter MySQL commands.
602: The MySQL command to generate the loncapa DATABASE is:
603:
604: CREATE DATABASE 'loncapa';
605:
606: ** USER named "www"
607:
608: As user www, run this command
609: mysql -u root -p mysql
610: enter the password as SOMEPASSWORD
611:
612: To add the user www to the MySQL server, and grant all
613: privileges on *.* to www@localhost identified by 'SOMEPASSWORD'
614: with grant option;
615:
616: INSERT INTO user (Host, User, Password)
617: VALUES ('localhost','www',password('SOMEPASSWORD'));
618:
619: GRANT ALL PRIVILEGES ON *.* TO www@localhost;
620:
621: FLUSH PRIVILEGES;
622:
623: ** ABILITY for LON-CAPA machines to communicate with SQL databases on
624: other LON-CAPA machines
625:
1.5 harris41 626: An up-to-date lond and lonsql.
627: </PRE>
628: </P>
629: <H2>Testing</H2>
630: <P>
1.11 harris41 631: To test the backend MySQL database, a number of commands should be
632: run after installation.
633: <UL>
634: <LI><TT>cd /usr/local/mysql/sql-bench; ./run-all-tests --small-test</TT></LI>
635: <BR>without the --small-test flag, this test can take more than 10 hours!
636: <LI><TT>cd /usr/local/mysql; bin/mysqladmin version</TT></LI>
637: <LI><TT>cd /usr/local/mysql; bin/mysqladmin variables</TT></LI>
638: <LI><TT>cd /usr/local/mysql; bin/mysqlshow</TT></LI>
639: <LI><TT>cd /usr/local/mysql; bin/mysqlshow mysql</TT></LI>
640: <LI><TT>cd /usr/local/mysql; bin/mysql -e "select host,db,user from db" mysql</TT></LI>
641: <LI><TT>cd /usr/local/mysql/mysql-test; ./test-run-all</TT></LI>
642:
643: </UL>
644: <P>
645: These are sections of perl code which helps test the LON-CAPA network.
1.5 harris41 646: <PRE>
647: <STRONG>** TEST the database connection with my current tester.pl code
648: which mimics what command will eventually be sent through lonc.</STRONG>
1.2 harris41 649:
1.5 harris41 650: $reply=reply(
651: "querysend:SELECT * FROM general_information WHERE Id='AAAAA'",$lonID);
652: </PRE>
653: </P>
654: <H2>Example sections of code relevant to LON-CAPA</H2>
655: <P>
1.2 harris41 656: Here are excerpts of code which implement the above handling:
1.5 harris41 657: </P>
658: <P>
659: <PRE>
660: <STRONG>**LONSQL
1.2 harris41 661: A subroutine from "lonsql" which establishes a child process for handling
1.5 harris41 662: database interactions.</STRONG>
1.2 harris41 663:
664: sub make_new_child {
665: my $pid;
666: my $sigset;
667:
668: # block signal for fork
669: $sigset = POSIX::SigSet->new(SIGINT);
670: sigprocmask(SIG_BLOCK, $sigset)
671: or die "Can't block SIGINT for fork: $!\n";
672:
673: die "fork: $!" unless defined ($pid = fork);
674:
675: if ($pid) {
676: # Parent records the child's birth and returns.
677: sigprocmask(SIG_UNBLOCK, $sigset)
678: or die "Can't unblock SIGINT for fork: $!\n";
679: $children{$pid} = 1;
680: $children++;
681: return;
682: } else {
683: # Child can *not* return from this subroutine.
684: $SIG{INT} = 'DEFAULT'; # make SIGINT kill us as it did before
685:
686: # unblock signals
687: sigprocmask(SIG_UNBLOCK, $sigset)
688: or die "Can't unblock SIGINT for fork: $!\n";
689:
690:
691: #open database handle
692: # making dbh global to avoid garbage collector
693: unless (
694: $dbh = DBI->connect("DBI:mysql:loncapa","www","SOMEPASSWORD",{ RaiseError =>0,PrintError=>0})
695: ) {
696: my $st=120+int(rand(240));
697: &logthis("<font color=blue>WARNING: Couldn't connect to database ($st secs): $@</font>");
698: print "database handle error\n";
699: sleep($st);
700: exit;
701:
702: };
703: # make sure that a database disconnection occurs with ending kill signals
704: $SIG{TERM}=$SIG{INT}=$SIG{QUIT}=$SIG{__DIE__}=\&DISCONNECT;
705:
706: # handle connections until we've reached $MAX_CLIENTS_PER_CHILD
707: for ($i=0; $i < $MAX_CLIENTS_PER_CHILD; $i++) {
708: $client = $server->accept() or last;
709:
710: # do something with the connection
711: $run = $run+1;
712: my $userinput = <$client>;
713: chomp($userinput);
714:
715: my ($conserver,$querytmp)=split(/&/,$userinput);
716: my $query=unescape($querytmp);
717:
718: #send query id which is pid_unixdatetime_runningcounter
719: $queryid = $thisserver;
720: $queryid .="_".($$)."_";
721: $queryid .= time."_";
722: $queryid .= $run;
723: print $client "$queryid\n";
724:
725: #prepare and execute the query
726: my $sth = $dbh->prepare($query);
727: my $result;
728: unless ($sth->execute())
729: {
730: &logthis("<font color=blue>WARNING: Could not retrieve from database: $@</font>");
731: $result="";
732: }
733: else {
734: my $r1=$sth->fetchall_arrayref;
735: my @r2; map {my $a=$_; my @b=map {escape($_)} @$a; push @r2,join(",", @b)} (@$r1);
736: $result=join("&",@r2) . "\n";
737: }
738: &reply("queryreply:$queryid:$result",$conserver);
739:
740: }
741:
742: # tidy up gracefully and finish
743:
744: #close the database handle
745: $dbh->disconnect
746: or &logthis("<font color=blue>WARNING: Couldn't disconnect from database $DBI::errstr ($st secs): $@</font>");
747:
748: # this exit is VERY important, otherwise the child will become
749: # a producer of more and more children, forking yourself into
750: # process death.
751: exit;
752: }
753: }
1.5 harris41 754: </P>
755: <P>
756: <STRONG>** LOND enabling of MySQL requests</STRONG>
757: <BR />
758: This code is part of every lond child process in the
759: way that it parses command request syntax sent to it
760: from lonc processes. Based on the diagram above, querysend
761: corresponds to B-lonc sending the result of the query.
762: queryreply corresponds to B-lond indicating that it has
763: received the request and will start the database transaction
764: (it returns "ok" to
765: A-lonc ($client)).
766: <PRE>
1.2 harris41 767: # ------------------------------------------------------------------- querysend
768: } elsif ($userinput =~ /^querysend/) {
769: my ($cmd,$query)=split(/:/,$userinput);
770: $query=~s/\n*$//g;
771: print $client sqlreply("$hostid{$clientip}\&$query")."\n";
772: # ------------------------------------------------------------------ queryreply
773: } elsif ($userinput =~ /^queryreply/) {
774: my ($cmd,$id,$reply)=split(/:/,$userinput);
775: my $store;
776: my $execdir=$perlvar{'lonDaemons'};
777: if ($store=IO::File->new(">$execdir/tmp/$id")) {
778: print $store $reply;
779: close $store;
780: print $client "ok\n";
781: }
782: else {
783: print $client "error:$!\n";
784: }
785:
1.5 harris41 786: </PRE>
1.2 harris41 787:
1.1 harris41 788: </P>
789: </BODY>
1.3 harris41 790: </HTML>
FreeBSD-CVSweb <freebsd-cvsweb@FreeBSD.org>
![[BACK]](/icons/back.gif){kind=icon}
Return to loncapasqldatabase.html CVS log ![[TXT]](/icons/text.gif){kind=icon}
![[DIR]](/icons/dir.gif){kind=icon}
Up to [LON-CAPA] / doc / build