Annotation of doc/packaging/index.html, revision 1.2
1.1 harris41 1: <HTML>
2: <HEAD>
3: <TITLE>LON-CAPA Packaging</TITLE>
4: </HEAD>
5: <BODY BGCOLOR=white>
6: <H1>LON-CAPA Packaging and Filesystem Management</H1>
7: <B>Functional Model Description</B> <I>(Scott Harrison, April 2000)</I>
8: <P>
9: <OL>
10: <LI>Synopsis
11: <LI>Summary of Data Store Objects <I>(e.g. CVS, RPM, LCIC)</I>
12: <LI>Summary of Processes <I>(e.g. making a CD image, upgrading a system, altering package composition)</I>
13: <LI>Summary of Actor Objects <I>(e.g. cron jobs, programmers, head packaging
14: developer)</I>
15: <LI>Functional Model Diagram
16: <LI>Description of the Functional Model Diagram
17: <OL>
18: <LI>Data Store Objects
19: <LI>Processes
20: <LI>Actor Objects
21: </OL>
22: <LI>Frequently Asked Questions
23: </OL>
24: <P>
25: <H1>1. Synopsis</H1>
26: Up to this point, dozens of scripts have been written
27: to automate various elements of the LON-CAPA <U>installation</U>, <U>upgrade</U>, and <U>software development</U> process. A number of significant issues
28: relate to this effort. Security on the system is largely a matter of <B>limiting</B>
29: unnecessary network services, <B>monitoring</B> the filesystem of LON-CAPA
30: servers to make full account for the presence of every file on the system,
31: and securely <B>communicating potential problems</B> to system administrators of a given
32: LON-CAPA server machine.
33: <P>
34: This document is the functional model description which describes the computations
1.2 ! harris41 35: and expected results necessary to manage packaging and filesystem management
1.1 harris41 36: for LON-CAPA both now and in the future in a manner that is <B>synchronous</B>
37: with software development, installations, and upgrades. Promulgation of coding
38: changes, version control, and maintaining a handle on security and network optimization
39: issues (such as introducing SAMBA and NetaTalk into the network architecture) have
40: been some of the reasons for introducing a more strongly defined characterization
41: of all of these processes involved with LON-CAPA software packaging and file system
42: management.
43: <P>
44: <H1>2. Summary of Data Store Objects</H1>
45: <UL>
46: <LI><B>CVS</B>
47: <BR>Concurrent Versions System. A central file repository is set up for LON-CAPA software development.
48: <LI><B>RPM</B>
49: <BR>Red Hat Package Manager. A system to manage software applications (packages). This system tracks package
50: interdependencies, configuration files, and file alterations. The RPM packaging system can (and probably should) reside on a different
51: computer than the central file repository computer. The RPM packaging system should probably be on a computer
1.2 ! harris41 52: accessible to programmers involved with system development.
1.1 harris41 53: <LI><B>LCIC</B>
54: <BR>LON-CAPA Integrity Check. A script-based system which analyzes a computer to determine the RPM composition
55: (missing or contrastingly extraneous RPMs), extraneous files which do not belong to any RPM, and RPM file
56: alterations.
57: </UL>
58: <P>
59: <H1>3. Summary of Processes</H1>
60: <UL>
61: <LI><B>Non-automated processes</B>
62: <UL>
63: <LI><B>implements change</B>
64: <BR>Programmer creates a successful change to the LON-CAPA software. To carry out this step, the programmer
65: must enter in bug-free code to the CVS file repository. The programmer must also inform the packaging developer
66: as to where and how the files need to be placed and configured on LON-CAPA computer systems.
67: <LI><B>alter package composition</B>
68: <BR>The packaging developer may choose to add/remove different sets of entire RPM packages from LON-CAPA computer
69: systems. The packaging developer may also choose to modify the file composition and dependency details of individual
70: RPM packages. <I>There are over 700 potential RedHat packages to select from on RedHat 6.1 disks. Selecting an appropriate
71: mix of RPMs can GREATLY improve security on LON-CAPA systems (by, for instance, removing unnecessary services) and is
72: considered the general start- and end- point of LON-CAPA security.</I>
73: <LI><B>change process programs</B>
74: <BR>The various automated processes are to be written and modified by the packaging developer. These processes should
75: reside on the packaging development system (RPM packaging system) as well the CVS file repository.
76: <LI><B>upgrade</B>
77: <BR>The packaging developer chooses when to upgrade the packaging system so that the packaging system upgrade passes
78: along to other computers in the software development cluster.
79: </UL>
80: <LI><B>Automated processes</B>
81: <UL>
82: <LI><B>LON_CAPA_update_RPM_build_tree</B>
83: <BR>Retrieves files out of the central software development CVS file repository. Builds any customized packages associated
84: with the LON-CAPA system.
85: <LI><B>LON_CAPA_make_CD_image</B>
86: <BR>Compiles a RedHat installation/upgrade CD image based on the current LON-CAPA RPM packaging descriptions.
87: <LI><B>LON_CAPA_post_CD_image</B>
88: <BR>Make CD image available for other LON-CAPA development/run-time systems to upgrade from.
89: <LI><B>LON_CAPA_upgrade_sys</B>
90: <BR>Check to see if a general LON-CAPA RedHat system upgrade should be performed based on the version available from the packaging
91: system. Also, monitor current status of system with LCIC (perhaps a "corrupt" system should be re-upgraded/re-installed). If upgrade
92: should be performed, then perform a network install/upgrade while preserving all LON-CAPA configuration files.
93: </UL>
94: </UL>
95: <P>
96: <H1>4. Summary of Actor Objects</H1>
97: <UL>
98: <LI><B>Head Packaging Developer</B>
99: <BR>Responsible for maintaining and writing scripts which automate the packaging and monitoring of distributed LON-CAPA systems. Also
100: responsible for altering overall package composition of LON-CAPA systems.
101: <LI><B>Programmer</B>
102: <BR>Responsible for introducing software improvements to the LON-CAPA system itself. Submits software additions and modifications to
103: the central CVS file repository.
104: <LI><B>Cron</B>
105: <BR>Cron jobs issue scheduled commands to facilitate the independent distributed processing of the various scripts associated with
106: LON-CAPA Packaging and Filesystem Management.
107: </UL>
108: <P>
109: <H1>5. Functional Model Diagram</H1>
110: <IMG BORDER=1 SRC=scheme.gif>
111: <P>
112: <H1>6. Description of the Functional Model Diagram</H1>
113: I will now describe a semi-chronological viewpoint of how all the different processes take place
114: so as to better understand both the specifics of the functional model, as well as the pertinent
115: dynamical and object-oriented characteristics of the algorithm.
116: <P>
117: Every week on Thursday at 3 in the morning, a <B>cron</B> job associated with the <B>RPM</B> packaging system on
118: the main packaging computer updates the build tree of all LON-CAPA source RPMs based on files in the central
119: <B>CVS</B> file repository. It does this by initiating the process <B>LON_CAPA_update_RPM_build_tree</B>. This process
120: requests and receives files from the central <B>CVS</B> file repository, and updates all the source RPM file trees. RPM packages
121: are then "bundled" by issuing build commands to the "Red Hat Package Manager" (<I>rpm -b ...</I>).
122: <P>
123: The <B>LON_CAPA_update_RPM_build_tree</B> process calls the <B>LON_CAPA_make_CD_image</B> process which writes a
124: CD image into a single file on the main packaging system. This is followed by the process <B>LON_CAPA_post_CD_image</B>
125: which mounts and posts the CD image file onto a web accessible location.
126: <P>
127: The <B>LON_CAPA_upgrade_sys</B> process is launched by a <B>cron</B> job specific to every individual
128: computer present in the software development cluster. Each individual computer checks the main packaging computer to see
129: if a system upgrade is needed. System upgrade information is passed to the <B>LCIC</B> on each specific computer (LON-CAPA
130: Integrity Checker) so that integrity checking is done as a function of the most recent system upgrade.
131: <P>
132: Finally, on Sunday at 3 in the morning, a <B>cron</B> job associated with each computer launches the <B>LCIC</B> integrity checker
133: system which resyncs itself based on the RPMs and files present on the system as well as information pertaining to
134: the expected status of the system (in terms of upgrade, RPM composition, etc). The results of the <B>LCIC</B> integrity checker
135: "re-syncing" are posted under a password controlled directory.
136: <P>
137: <H1>7. Frequently Asked Questions</H1>
138: <P>
139: <OL>
140: <LI><B>What does LON-CAPA stand for?</B>
141: <BR>The LearningOnline Network with a Computer-Assisted Personalized Approach
142: <P>
143: <LI><B>What is the general structure of the CVS repository?</B>
144: <BR>These are the current directories:
145: <BR><U>loncom</U> - server configuration and application files associated with LON-CAPA library and access servers
146: <BR><U>lonline</U> - lecture online
147: <BR><U>CAPA</U> - files associated with CAPA (homework processing, grading, homework generating, exam and quiz administration)
148: <BR><U>doc</U> - documentation
149: <BR><U>modules</U> - custom built software modules meant for use with the LON-CAPA project
150: <BR><U>rat</U> - the "RAT"; resource assembly tool
151: <BR><U>packaging</U> - files (including this one) specific to packaging and filesystem management of LON-CAPA
152: <P>
153: <LI><B>How does a programmer set himself up to use CVS?</B>
154: <BR>
155: <PRE>
156: you will need to have the CVSROOT enviroment variable set to
157: :pserver:yourname@zaphod.lite.msu.edu:/home/cvs
158: To do this if you are using csh or tcsh put in your .login:
159: setenv CVSROOT :pserver:yourname@zaphod.lite.msu.edu:/home/cvs
160: For bash put this in .bash_profile
161: export CVSROOT=:pserver:yourname@zaphod.lite.msu.edu:/home/cvs
162:
163: You can add the "cvs login" command to your .login/.bash_profile if you wish to make this simple.
164: And you can add "cvs logout" to your .logout/.bash_logout so you rember to logout.
165:
166: To use CVS, you must first login (cvs login) and enter in your zaphod password.
167: To leave CVS, you must logout (cvs logout).
168: </PRE>
169: <P>
170: <LI><B>How does a programmer retrieve files from the CVS repository for the first time?</B>
171: <BR><TT>cvs checkout [resourcename]</TT>
172: <BR><TT>[resourcename]</TT> is one of the directories above (loncom, rat, doc, and so on)
173: <P>
174: <LI><B>How does a programmer update his files from the CVS repository?</B>
175: <BR>(In other words, CVS server ----to----your---->> client computer.)
176: <BR><TT>cvs update -d [resourcename]</TT>
177: <BR><TT>[resourcename]</TT> is one of the directories above (loncom, rat, doc, and so on)
178: <P>
179: <LI><B>How does a programmer send his changes to the central CVS repository?</B>
180: <BR>(In other words, client computer ----to----our---->> CVS server.)
181: <BR><TT>cvs commit [resourcename]</TT>
182: <BR><TT>[resourcename]</TT> is one of the directories above (loncom, rat, doc, and so on)
183: <P>
184: <LI><B>How does a programmer add or remove files from the central CVS repository?</B>
185: <BR>Note: this is different from "updating" or "committing" changes. This is not the alteration
186: of existing files. This is the ADDITION or REMOVAL of files.
187: <BR>Within "the" directory, use these two commands to add a file:
188: <BR><TT>cvs add [filename]</TT>; <TT>cvs commit</TT>
189: <BR>Within "the" directory, use these two commands to remove a file:
190: <BR><TT>cvs remove [filename]</TT>; <TT>cvs commit</TT>
191: <P>
192: <LI><B>How does a programmer add or remove directories from the central CVS repository?</B>
193: <BR>Within the upper directory, use these two commands to add a directory:
194: <BR><TT>cvs add [directoryname]</TT>; <TT>cvs commit</TT>
195: <BR>Within "the" directory, use these two commands to remove a directory:
196: <BR><TT>cvs remove [directoryname]</TT>; <TT>cvs commit</TT>
197: <BR>There is a slight catch that you cannot directly "add" a new major directory like
198: loncom, lonline, rat, doc, and so on. You must first "cvs add" it as a subdirectory, then
199: move that subdirectory to the top directory structure and "cvs commit".
200: <P>
201: Better yet, you can <TT>cvs import</TT> your new directory into the top level area of the CVS repository.
202: <LI><B>How does a programmer "inform" the packaging developer of changes to LON-CAPA system file composition?</B>
203: <BR>By talking to the packaging developer in person.
204: <BR>By making well-formatted changes to the <TT>packaging/file_map.txt</TT> file in the CVS repository. See below
205: for more information on how to format the <TT>packaging/file_map.txt</TT> file.
206: <P>
207: <LI><B>What does "run any process" mean in the functional model?</B>
208: <BR>Most processes are meant to run <I>automatically</I> and are invoked by cron jobs on various machines.
209: It may be the case that, for instance, the processes need to be manually invoked so as to prepare
210: a specific CD release.
211: <P>
212: <LI><B>Where are the automated processes located?</B>
213: <BR>In the directory <TT>/usr/local/bin</TT>.
214: <BR>By typing <TT>ls /usr/local/bin/LON_CAPA_*</TT>, you can view all the automated LON-CAPA processes associated with the computer systems.
215: <P>
216: <LI><B>Where are the <U>cron</U> jobs located? What are their names and how reliable are they?</B>
217: <BR>On every LON-CAPA computer there is a file <TT>/etc/cron.d/LON_CAPA_filesystem</TT> which launches standard cron jobs.
218: <BR>On the LON-CAPA main packaging computer there is a file <TT>/etc/cron.d/LON_CAPA_packager</TT> which launches cron jobs specific to the
219: main packaging computer.
220: <BR>On LON-CAPA software development computers, there is a file <TT>/etc/cron.d/LON_CAPA_upgrader</TT> which upgrades all computers in the
221: LON-CAPA software development cluster.
222: <P>
223: <LI><B>Can automated processes be run manually? On the packaging computer? On other LON-CAPA computers?</B>
224: <BR>Yes. To correctly manually run an automated process, look to see how the process is invoked in the <TT>/etc/cron.d</TT> directory.
225: <P>
226: <LI><B>What are the specifics associated with the RPM packaging system on the main packaging computer?</B>
227: <BR>The RPM packaging system on the main packaging computer introduces the following differences to the main packaging computer compared to
228: other computers in the software development cluster:
229: <UL>
230: <LI>Presence of RPM build tree
231: <LI>Presence of CD image
232: <LI>More automated processes present on system (LON_CAPA_update_RPM_build_tree, LON_CAPA_make_CD_image, LON_CAPA_post_CD_image)
233: </UL>
234: <P>
235: <LI><B>What are the specifics associated with the process LON_CAPA_update_RPM_build_tree?</B>
1.2 ! harris41 236: <BR>This process checkouts LON-CAPA directories and files. This process reads from the <TT>packaging/file_map.txt</TT> and the
! 237: <TT>packaging/permissions.txt</TT> file. This process reads RPM <TT>.spec</TT> template files to set up the two LON-CAPA RPMs (LON-CAPA-base and LON-CAPA-auxiliary).
1.1 harris41 238: These updated RPMs are placed into a pool of RPMs from which to generate the customized RedHat installation/upgrade CD.
239: A 'comps' file is generated to customize the grouping of packages on the CD image. The CD image is written and made accessible
240: on the web.
241: <P>
242: <LI><B>What are the specifics associated with the writing and web posting of the LON-CAPA system CD image on the main packaging computer?</B>
243: <BR>There are a number of shell commands which need to be launched. (Details coming soon in this document.)
244: <P>
245: <LI><B>What are the specifics associated with the process LON_CAPA_make_CD_image?</B>
246: <BR>There are a number of shell commands which need to be launched. (Details coming soon in this document.)
247: <P>
248: <LI><B>How are upgrades performed on the packaging computer? Is there any way to upgrade/install LON-CAPA on a new machine?</B>
249: <BR>Upgrades are performed only based on package composition. New machines can be upgraded via network install or CD install. (More details coming soon.)
250: <P>
251: <LI><B>What are the specifics associated with the process LON_CAPA_post_CD_image?</B>
252: <BR>There are a number of shell commands which need to be launched. (Details coming soon in this document.)
253: <P>
254: <LI><B>What are the specifics associated with the process LON_CAPA_upgrade_sys?</B>
255: <BR>LON_CAPA_upgrade_sys does an automated network upgrade of any out-of-date packages based on the main packaging computer.
256: LON_CAPA_upgrade_sys initializes and invokes the LCIC (integrity checker) based on current upgrade information.
257: <P>
258: <LI><B>What output from the LCIC (integrity checker) warrants taking corrective action?</B>
259: <BR>Currently, the output from the LCIC consists of two files, TOTAL_FILE.txt and SUMMARY_FILE.txt. There are no immediate plans for
260: automating any form of response to this output. This does provide information as to whether critical system binaries or configuration
261: files are being tampered with. Perhaps the SUMMARY_FILE.txt should be e-mailed weekly to a system administrator at the responsible
262: institution.
263: <P>
264: <LI><B>What is the correct format for the <TT>packaging/file_map.txt</TT> file?</B>
265: <BR>Every line refers to a file or directory to install/upgrade on a LON-CAPA system. There are six fields to each line which are tab-separated.
266: <PRE>
267: [computer-type]
268: [file location on CVS repository]
269: [ultimate location on LON-CAPA system]
270: [package name]
271: [configuration settings that need to be saved?]
272: [description of file and what it does]
1.2 ! harris41 273: </PRE>
! 274: <P>
! 275: <LI><B>What is the correct format for the <TT>packaging/permissions.txt</TT> file?</B>
! 276: <BR>Every line refers to a file or directory to install/upgrade on a LON-CAPA system. There are four/five fields to each line which are space-separated.
! 277: File listing:
! 278: <PRE>
! 279: [octal mode]
! 280: [user to own file]
! 281: [group to own file]
! 282: [file name]
! 283: </PRE>
! 284: Directory listing:
! 285: <PRE>
! 286: [octal mode]
! 287: [user to own file]
! 288: [group to own file]
! 289: DIR
! 290: [file name]
1.1 harris41 291: </PRE>
292: <UL>
293: <LI><TT>[computer-type]</TT> - a list of capitalized letters indicating which computer-types this file is to be installed on. P=main packaging computer. A=LON-CAPA software development access server. L=LON-CAPA software development library server. C=central CVS file repository computer. PAL means this file is to be installed on the main packaging computer and the software development library and access servers. P means this file is to be installed on just the main packaging computer. AL means the file is to be installed on just the software development library and access servers.
294: <LI><TT>[file location on CVS repository]</TT> - the file location on the CVS repository. For example, <TT>/loncom/htpasswd</TT>, <TT>loncom/startup.pl</TT>, and
295: <TT>/rat/images/1.1.dt.gif</TT> are possible entries.
296: <LI><TT>[ultimate location on LON-CAPA system]</TT> - where the file (taken from the CVS repository) is to be installed on the LON-CAPA system. For example,
297: <TT>/home/httpd/lonTabs/htpasswd</TT>, <TT>/etc/httpd/conf/startup.pl</TT>, and <TT>/home/httpd/html/adm/rat/1.1.dt.gif</TT> are possible entries.
298: <LI><TT>[package name]</TT> - What RPM package should this file be incorporated into for the final RPM install? Current possible entries are
299: <TT>LON-CAPA-auxiliary</TT> and <TT>LON-CAPA-base</TT>.
300: <LI><TT>[configuration settings that need to be saved?]</TT> - Every machine is anticipated to have a unique identity in terms of the role it
301: fills at a given institution. For instance, /etc/httpd/conf/httpd.conf has settings like these:
302: <PRE>
303: PerlSetVar lonHostID msua3
304: # Role of this machine: library, access
305: PerlSetVar lonRole access
306: # Server Administration
307: PerlSetVar lonAdmEMail korte@lite.msu.edu
308: # Default domain
309: PerlSetVar lonDefDomain msu
310: </PRE>
311: At some point, these settings need to be saved and used during upgrade processes.
312: <LI><TT>[description of file and what it does]</TT> - a brief sentence.
313: </UL>
314: <P>
315: <LI><B>What controls version naming in terms of system upgrades?</B>
316: <BR>This information is present in <TT>packaging/version.txt</TT>. This file consists of three tab-separated fields (version number, release number, and version
317: name). The release number is automatically incremented each week as a function of LON_CAPA_post_CD_image. The version number and release numbers are used
318: to label the LON-CAPA-base and LON-CAPA-auxiliary RPMs as well as the customized RedHat CD release.
319: </OL>
320: </BODY>
321: </HTML>
FreeBSD-CVSweb <freebsd-cvsweb@FreeBSD.org>
![[BACK]](/icons/back.gif){kind=icon}
Return to index.html CVS log ![[TXT]](/icons/text.gif){kind=icon}
![[DIR]](/icons/dir.gif){kind=icon}
Up to [LON-CAPA] / doc / packaging