--- loncom/build/readme.html 2001/01/17 13:46:02 1.15 +++ loncom/build/readme.html 2002/04/27 16:31:39 1.18 @@ -1,204 +1,409 @@ - - - -LON-CAPA Software Developer Instructions - - -

LON-CAPA Software Developer Instructions

-
Written by Scott Harrison, January 17, 2001 -
Last updated, January 17, 2001 -
    -
  1. Using CVS - -
  2. Viewing the software (make HTML) -
  3. Compiling the software (make build) -
  4. Adding/removing files from the LON-CAPA installation (doc/loncapafiles/loncapafiles.html) -
  5. Configurable files versus non-configurable files -
  6. Updating the non-configurable files on your machine (make install) -
  7. Updating the configurable files on your machine (make configinstall) -
  8. Building RPMs (make RPM) -
- -
    - -
  1. Using CVS

    -
    -
  2. -

    Viewing the software (make HTML)

    -Commands - -
    +
    +
    +
    +
    +
    +
    +LON-CAPA Software Developer Instructions
    +
    +
    +

    LON-CAPA Software Developer Instructions

    +

    +
    Written by Scott Harrison, January 17, 2001 +
    Last updated, April 27, 2002 +

    +
      +
    1. Using CVS
    2. + +
    3. Viewing the software (make HTML)
    4. +
    5. Compiling the software (make build)
    6. +
    7. Adding/removing files from the LON-CAPA +installation (doc/loncapafiles/loncapafiles.lpml)
    8. +
    9. Configurable files versus +non-configurable files
    10. +
    11. Updating the non-configurable files on your +machine (make install)
    12. +
    13. Updating the configurable files on your +machine (make configinstall)
    14. +
    15. Building RPMs (make RPM)
    16. +
    + +
      + +
    1. Using CVS

    2. +These instructions assume that you are using a Linux or UNIX based +terminal. +
      +
    3. +

      Viewing the software (make HTML)

    4. +Commands + +
       cd loncom/build
       rm -Rf HTML (or alternatively, "make clean")
       make HTML
       cd HTML
       (look at the index.html file with a web browser such as Netscape)
      -
      -
      -General description of what happens -

      +

    +
    +General description of what happens +

    This is the actual make target code. - -

    +
    +
     
     HTML:
    -        install -d HTML
    -        cp ../../doc/loncapafiles/*.gif HTML
    -        perl parse.pl ../../doc/loncapafiles/loncapafiles.html HTML > HTML/index.html
    +	install -d HTML
    +	cp $(SOURCE)/doc/loncapafiles/*.gif HTML
    +	cat $(SOURCE)/doc/loncapafiles/loncapafiles.lpml | \
    +	perl lpml_parse.pl html development default "$(SOURCE)" '$(TARGET)' \
    +	> HTML/index.html
     
    -
    -
    +
    + What basically happens is that specially marked-up data in the LON-CAPA -cvs repository file doc/loncapafiles.html is parsed into a more -viewable format by loncom/build/parse.pl. The resulting +cvs repository file doc/loncapafiles.lpml is parsed into a more +viewable format by loncom/build/lpml_parse.pl. The resulting file gives a very well organized view of all the files, directories, links, ownerships, permissions, and brief documentation of what each file does. -

    -
  3. -

    Compiling the software (make build)

    -Commands - -
    +

    +
  4. +

    Compiling the software (make build)

    +Commands + +
     cd loncom/build
     make build
    -
    -
    -General description of what happens -

    +

  5. +
    +

    +General description of what happens +

    +

    This is the actual make target code. - -

    +
    +
     
    -build:
    -        perl parse.pl ../../doc/loncapafiles/loncapafiles.html build > Makefile.build
    -        make -f Makefile.build all
    +build: Makefile.build pod2html.sh pod2man.sh
    +	echo -n "" > WARNINGS
    +	make -f Makefile.build all
    +	make warningnote
    +
    +Makefile.build: $(SOURCE)/doc/loncapafiles/loncapafiles.lpml lpml_parse.pl
    +	cat $(SOURCE)/doc/loncapafiles/loncapafiles.lpml | \
    +	perl lpml_parse.pl build $(CATEGORY) $(DIST) "$(SOURCE)" "$(TARGET)" \
    +	> Makefile.build
     
    -
    -
    -loncom/build/parse.pl reads in all the build information out -of doc/loncapafiles/loncapafiles.html. A new Makefile named -loncom/build/Makefile.build is dynamically constructed. +
    + +loncom/build/lpml_parse.pl reads in all the build information out +of doc/loncapafiles/loncapafiles.lpml. A new Makefile named +loncom/build/Makefile.build is dynamically constructed. This dynamically generated Makefile is then run to build/compile all the software targets from source. This currently takes 10 minutes (depends on the speed of the machine you compile with). -

    -Example -

    -Here is information for one file tth.so provided in -doc/loncapafiles/loncapafiles.html. - -

    -
    <METAGROUP> -
    <LONCAPA TYPE=LOCATION DIST="redhat6.2" SOURCE="loncom/modules/TexConvert/tthperl/tth.so" TARGET="usr/lib/perl5/site_perl/5.005/tth.so" CATEGORY="system file"> -
    <DESCRIPTION> -
    shared library file for dynamic loading and unloading of TeX-to-HTML functionality -
    </DESCRIPTION> -
    <BUILD> -
    loncom/modules/TexConvert/tthperl/commands -
    </BUILD> -
    <DEPENDENCIES> -
    ../tthdynamic/tthfunc.c -
    ../ttmdynamic/ttmfunc.c -
    </DEPENDENCIES> -
    - -loncom/build/parse.pl sees the BUILD tags and sets up -a dynamic file Makefile.build to run the command inside the -BUILD tags (currently, DEPENDENCIES is not used for anything -besides documentation). -

    -

    -Here is an example of a dynamically generated Makefile.build -that builds two LON-CAPA files (one of which is tth.so). - -

    +

    +Example +

    +Here is information for one file tth.so provided in +doc/loncapafiles/loncapafiles.lpml. + +

    +<file>
    +<source>loncom/homework/caparesponse/capa.so</source>
    +<target dist='default'>usr/lib/perl5/site_perl/5.005/capa.so</target>
    +<target dist='redhat7 redhat7.1'>usr/lib/perl5/site_perl/5.6.0/capa.so</target>
    +<categoryname>system file</categoryname>
    +<description>
    +shared library file for dynamic loading and unloading
    +</description>
    +<build trigger='always run'>
    +loncom/homework/caparesponse/commands
    +</build>
    +<dependencies>
    +caparesponse.c;
    +caparesponse.pm;
    +README;
    +Makefile.PL;
    +capa.i;
    +commands
    +</dependencies>
    +</file>
    +
    + +loncom/build/lpml_parse.pl sees the build tags and sets up +a dynamic file Makefile.build to run the command inside the +build tags. The files listed inside the dependencies tags +are included in the Makefile.build so as to determine whether +or not there is a need to compile. +

    +

    +Here is an example of a dynamically generated Makefile.build +that builds two LON-CAPA files (one of which is tth.so). + +

     all: ../homework/caparesponse/capa.so ../modules/TexConvert/tthperl/tth.so 
     
    -../homework/caparesponse/capa.so:  ../homework/caparesponse/caparesponse.c ../ho
    -mework/caparesponse/caparesponse.pm alwaysrun
    +../homework/caparesponse/capa.so:  ../homework/caparesponse/caparesponse.c ../homework/caparesponse/caparesponse.pm alwaysrun
             cd ../homework/caparesponse/; sh ./commands
     
    -../modules/TexConvert/tthperl/tth.so:  ../modules/TexConvert/tthperl/../tthdynam
    -ic/tthfunc.c ../modules/TexConvert/tthperl/../ttmdynamic/ttmfunc.c
    +../modules/TexConvert/tthperl/tth.so:  ../modules/TexConvert/tthperl/../tthdynamic/tthfunc.c ../modules/TexConvert/tthperl/../ttmdynamic/ttmfunc.c
             cd ../modules/TexConvert/tthperl/; sh ./commands
     
     alwaysrun:
    -
    - -

    -
  6. -

    Adding/removing files from the LON-CAPA installation (doc/loncapafiles/loncapafiles.html)

    -To add and remove (and alter) -

    +

  7. + +

    +
  8. +

    Adding/removing files from the LON-CAPA installation (doc/loncapafiles/loncapafiles.html)

    +To add and remove (and alter) +

    All that you have to do to alter the behavior of the installation is -edit a single file (doc/loncapafiles/loncapafiles.html). +edit a single file (doc/loncapafiles/loncapafiles.lpml). Adding, removing, and altering files requires proper attention to the syntax of file format of course. -

    -File Format +

    +File Format

    -The preceding "make build" documentation -gives an example METAGROUP entry describing one particular file. -All data within loncapafiles.html is specified according -to markup tags. The format and syntax of loncapafiles.html +The preceding "make build" documentation +gives an example of a file entry describing one particular file. +All data within loncapafiles.lpml is specified according +to markup tags. The format and syntax of loncapafiles.lpml is currently best described by the HTML documentation code at the beginning of loncapafiles.html (as well as, by example, seeing how various information is coded). All in all, the syntax is quite simple. -

    -Philosophy and notes (the thing nobody reads) -

    +

    +Philosophy and notes (the thing nobody reads) +

    Packaging the software from CVS onto a machine file system requires many things: -

    +

    +

    I looked into, and tried, different ways of accomplishing the above including automake and recursive make. The automake system seemed quite complicated (and needlessly so in terms of this project since, by and large, @@ -209,50 +414,51 @@ is kept in one place, and there are sign between all the things that must be done to keep software packaging up to date. A particularly convincing article I found when looking into much of this was - -"Recursive Make Considered Harmful" by Peter Miller. Complicating + +"Recursive Make Considered Harmful" by Peter Miller. Complicating matters was, at the time, it was unclear as to what categories of software files we had, and whether or not the directory structure of CVS would remain constant. With an ever-developing directory structure to CVS, I preferred to organize the information on a per-file basis as opposed to a per-directory basis (although there is a successful -implementation of a standard big Makefile in loncom/Makefile). +implementation of a standard big Makefile in loncom/Makefile). Additionally, a standard big Makefile assumes certain "normalcy" to the directory structure of different potential operating system directories (RedHat vs. Debian). -

    -

    -If you take time to look at loncapafiles.html -(and perhaps run the make HTML command) +

    +

    +If you take time to look at loncapafiles.lpml +(and perhaps run the make HTML command) you will find that the organizing information according to the markup -syntax in loncapafiles.html is simple. Simple is good. -

    -

    -loncom/build/parse.pl is the script (invoked automatically -by the various targets in loncom/build/Makefile) that reads -doc/loncapafiles/loncapafiles.html. parse.pl +syntax in loncapafiles.lpml is simple. Simple is good. +

    +

    +loncom/build/lpml_parse.pl is the script (invoked automatically +by the various targets in loncom/build/Makefile) that reads +doc/loncapafiles/loncapafiles.lpml. lpml_parse.pl is capable of reading and returning different types of information -from loncapafiles.html depending on how parse.pl -is invoked. parse.pl has yet to have introduced new sources +from loncapafiles.lpml depending on how lpml_parse.pl +is invoked. lpml_parse.pl has yet to have introduced new sources of error, and has been tested in quite a number of ways. As with any parser however, I remain paranoid. -

    -

    -My regrets with the current system is that parse.pl is -slow (can take 1 minute to run) and includes a few tidbits of code, -specific to the make process, that probably should be in -loncom/build/Makefile. Additionally, loncapafiles.html -should have a DTD and all those other good SGML-ish things (and parsing -should be done with a real SGML-derived parser). -

    -

    -On the plus side, the parse.pl-loncapafiles.html +

    +

    +lpml_parse.pl is very fast and styled after a state-based SAX-like +approach. Additionally, loncapafiles.lpml has a +DTD (loncom/build/lpml.dtd) against which it is valid. +I would like to use more ENTITY's inside lpml.dtd but currently +Perl XML modules available at CPAN do not digest complex ENTITY's that well. +

    +

    +The lpml_parse.pl-loncapafiles.lpml combination has been working very efficiently and error-free. -

    -
  9. -

    Configurable files versus non-configurable files

    -Machine-specific information is the difference -

    +

    +
  10. +

    Configurable files versus non-configurable files

    +

    +Machine-specific information is the difference +

    +

    The current list of configurable files for the LON-CAPA system is /etc/httpd/access.conf, /etc/smb.conf, /etc/ntp.conf, /etc/krb.conf, /etc/atalk/config, /etc/ntp/step-tickers, @@ -260,51 +466,64 @@ The current list of configurable files f /home/httpd/html/res/adm/includes/un_keyword.tab, /home/httpd/hosts.tab, and /home/httpd/spare.tab. -

    -

    +

    +

    All of these configurable files contain machine-specific information. For instance, the LON-CAPA system relies on unique host IDs such as msua3, s1, s2, msul1, and 103a1 (specified as a "PerlSetVar lonHostID" field within /etc/httpd/access.conf). Non-configurable files simply do NOT have machine-specific information. -The impact on updating software -

    +The impact on updating software +

    What this means in terms of software updating is that -

      -
    • non-configurable files can be simply overwritten with newer versions -(without "anything" else to worry about), -
    • and configurable files must follow these steps to be safely overwritten -
        -
      1. have their machine specific information saved, -
      2. be overwritten, and then -
      3. have their machine specific information restored. -
      -
    -

    -
  11. -

    Updating the non-configurable files on your machine (make install)

    -Commands - -
    +
      +
    • non-configurable files can be simply overwritten with newer versions +(without "anything" else to worry about),
    • +
    • and configurable files must follow these steps to be safely +overwritten
    • +
        +
      1. have their machine specific information saved,
      2. +
      3. be overwritten, and then
      4. +
      5. have their machine specific information restored.
      6. +
      +
    +

    +
  12. +

    Updating the non-configurable files on your machine (make install)

    +Commands + +
     cd loncom/build
     make install
    -
    -
    -General description of what happens -

    +

  13. +
    +

    +General description of what happens +

    +

    This is the actual make target code. - -

    +
    +
     
    -install: build
    -        perl parse.pl ../../doc/loncapafiles/loncapafiles.html install > Makefil
    -e.install
    -        make -f Makefile.install SOURCE="../.." TARGET="" directories
    -        make -f Makefile.install SOURCE="../.." TARGET="" files
    -        make -f Makefile.install SOURCE="../.." TARGET="" links
    +install: TEST_hosts_tab Makefile.install Makefile
    +	echo -n "" > WARNINGS
    +	make -f Makefile.install SOURCE="$(SOURCE)" TARGET="$(TARGET)" \
    +	directories
    +	make -f Makefile.install SOURCE="$(SOURCE)" TARGET="$(TARGET)" files
    +	make -f Makefile.install SOURCE="$(SOURCE)" TARGET="$(TARGET)" links
    +	make SOURCE="$(SOURCE)" TARGET="$(TARGET)" \
    +	NORESTORECONF="$(NORESTORECONF)" configinstall
    +	make postinstall
    +	make warningnote
    +	echo "You can run 'make test' to see if your system is ready to go!"
    +
    +Makefile.install: $(SOURCE)/doc/loncapafiles/loncapafiles.lpml lpml_parse.pl
    +	cat $(SOURCE)/doc/loncapafiles/loncapafiles.lpml | \
    +	perl lpml_parse.pl install $(CATEGORY) $(DIST) "$(SOURCE)" \
    +	"$(TARGET)" > Makefile.install
     
    -
    -
    +
    + For safety reasons (so as to not mess up a machine's configuration), configuration files are NOT installed during this step. This means that files such as /etc/httpd/access.conf, /etc/smb.conf, /etc/atalk/config, @@ -312,98 +531,124 @@ that files such as /etc/httpd/access.con /home/httpd/spare.tab are not overwritten, but remain as old, non-updated copies. (To automatically update these files and save/restore their encoded machine configuration, you must run "make configinstall"). -

    -
  14. -

    Updating the configurable files on your machine (make configinstall)

    -Commands - -
    +

    +
  15. +

    Updating the configurable files on your machine (make configinstall)

    +Commands + +
     cd loncom/build
     make configinstall
    -
    -
    -General description of what happens -

    +

  16. +
    +General description of what happens +

    This is the actual make target code. - -

    +
    +
     
    -configinstall: 
    -        # there is a dependency on having directories in place, but oh well...
    -        perl parse.pl ../../doc/loncapafiles/loncapafiles.html configinstall > Makefile.configinstall
    -        make -f Makefile.configinstall SOURCE="../.." TARGET="" configfiles
    -        perl loncaparestoreconfigurations lasttimestamp
    -        make -f Makefile.configinstall TARGET="" configpermissions
    +configinstall: Makefile.configinstall
    +	make -f Makefile.configinstall SOURCE="$(SOURCE)" TARGET="$(TARGET)" \
    +	configfiles
    +	if (test "0" = $(NORESTORECONF)); then \
    +	perl loncaparestoreconfigurations suffix .lpmlnew; fi
    +
    +Makefile.configinstall: $(SOURCE)/doc/loncapafiles/loncapafiles.lpml lpml_parse.pl
    +	cat $(SOURCE)/doc/loncapafiles/loncapafiles.lpml | \
    +	perl lpml_parse.pl configinstall $(CATEGORY) $(DIST) "$(SOURCE)" \
    +	"$(TARGET)" > Makefile.configinstall
     
    -
    -
    +
    + Configuration files are installed during this step. This means that files such as /etc/httpd/access.conf, /etc/smb.conf, /etc/atalk/config, /home/httpd/html/res/adm/includes/copyright.tab, and /home/httpd/spare.tab are overwritten. Before being overwritten, a backup copy is made though. Information is read out of these backup copies and restored to the new files by the -loncaparestoreconfigurations script. To ensure that +loncaparestoreconfigurations script. To ensure that new file permissions and ownerships are installed, a final set of -chown and chmod commands are called for each of +chown and chmod commands are called for each of the configuration files. -

    -For the truly paranoid -

    +

    +

    +For the truly paranoid +

    +

    If you are truly paranoid, you can just make the -Makefile.configinstall file and then save, copy, +Makefile.configinstall file and then save, copy, and restore all the configuration values yourself. -loncaparestoreconfigurations is pretty smart though, has yet to -fail, and besides, a backup copy is always made (time-stamped so that backup -copies are not overwritten). -

    -
  17. -

    Building RPMs (make RPM)

    -Commands - -
    +loncaparestoreconfigurations is pretty smart though, has yet to
    +fail, and besides, when needed backup copies are made.
    +

    +
  18. +

    Building RPMs (make RPM)

    +

    +LON-CAPA is currently installed through "intelligent tarballs". This +is part of an earlier (and perhaps future) effort involving RPMs. +Commands + +

     cd loncom/build
    -rm -Rf BinaryRootL (or alternatively, "make clean")
    +rm -Rf BinaryRoot (or alternatively, "make clean")
     make RPM
    -(to subsequently install, you can type commands like
    -"rpm -Uvh --force LON-CAPA-base-3.1-1.i386.rpm")
    -
    - -

    -WARNING!!!!!!!!!!!!!! -

    + (to subsequently install, you can type commands like + "rpm -Uvh --force LON-CAPA-base-3.1-1.i386.rpm") + + +

    +WARNING!!!!!!!!!!!!!! +

    Never never never never never manually install the LON-CAPA-setup-3.1-1.i386.rpm. This RPM is meant to only be installed by the CD installation process (it wipes out the existing /etc/passwd file). -

    -Configuration files -

    +

    +

    +Configuration files +

    +

    Configuration files are automatically saved with the file suffix -".rpmsave". So /etc/httpd/conf/access.conf is saved as -/etc/httpd/conf/access.conf.rpmsave. You can restore +".rpmsave". So /etc/httpd/conf/access.conf is saved as +/etc/httpd/conf/access.conf.rpmsave. You can restore the machine-specific configuration information by running -the /usr/sbin/loncaparestoreconfigurations. However, -a warning is important here. If you install an RPM twice +the /usr/sbin/loncaparestoreconfigurations. However, +a warning is important here. If you install an RPM twice without restoring your configuration, you will overwrite the ".rpmsave" files. -

    -General description of what happens -

    +

    +

    +General description of what happens +

    +

    This is the actual make target code. - -

    +
    +
     
    +RPM: BinaryRoot base_rpm_file_list
    +	cat $(SOURCE)/doc/loncapafiles/loncapafiles.lpml | \
    +	perl lpml_parse.pl make_rpm $(CATEGORY) $(DIST) $(SOURCE) $(TARGET) \
    +	> base_customizerpm.xml
    +	cat base_rpm_file_list.txt | perl make_rpm.pl base 3.2 '' '' \
    +	BinaryRoot base_customizerpm.xml
    +
    +BinaryRoot: base_rpm_file_list
    +	make TARGET='BinaryRoot' NORESTORECONF='1' install
     
    -
    -
    -A BinaryRoot directory is generated that reflects the locations, +base_rpm_file_list: + cat $(SOURCE)/doc/loncapafiles/loncapafiles.lpml | \ + perl lpml_parse.pl rpm_file_list $(CATEGORY) $(DIST) $(SOURCE) \ + 'BinaryRoot' | sort > base_rpm_file_list.txt + +
    + +A BinaryRoot directory is generated that reflects the locations, ownerships, permissions, and contents for all the CVS source files, compiled binaries, directories, and links as they should eventually occur on the '/' filesystem location. -

    -

    -loncom/build/make_rpm.pl is robust (tested over the +

    +

    +loncom/build/make_rpm.pl is robust (tested over the span of months) and, unlike other automated RPM-builders, cleanly builds new RPMs without any after-effect of temporary files left on the system. (On the negative side, there are a number of @@ -411,10 +656,11 @@ LON-CAPA specific customizations inside the sake of reusability, should eventually be removed). Two new RPMs are generated: LON-CAPA-base-3.1-1.i386 and LON-CAPA-setup-3.1-1.i386.rpm (again, never manually install LON-CAPA-setup-3.1-1.i386.rpm). -

    -
- - +

+ + + +