CVS log for loncom/loncnew

Up to [LON-CAPA] / loncom

Request diff between arbitrary revisions

Keyword substitution: kv
Default branch: MAIN

- Changes from loncnew rev. 1.108 when creating a new connection between
  LON-CAPA nodes to (a) support verification of common name when creating
  SSL tunnel between 2.12 nodes, while (b) supporting continued queuing of
  any delayed transactions when reconnecting (both 2.12 and pre-2.12).

- For 2.11
  Backport 1.109

- Reverse change in rev. 1.102.
 - the HostID of the peer is included in the string returned by:
   $Transaction->getRequest().
 - for a deferred transaction, the value of $Transaction->getClient() will
   be undef.

- Include verification of common name when creating SSL tunnel unless
  connecting to/from pre-2.12 node.
- $IO::Socket::SSL::DEBUG is set to current $DEBUG value so debugging
  from IO/Socket/SSL.pm is written to lond_errors or lonc_errors.

- Verification of CN when creating SSL tunnel on multi-domain servers.
  CN is default lonid, i.e., one listed last for the node's hostname in
  hosts.tab and/or dns_hosts.tab.

- Remove unused assignment added in rev. 1.105

- For 2.11
  Backport 1.102, 1.103, 1.104

- Support Certificate Revocation List checking when using SSL channel
  for key exchange during negotiation of connection to remote lond.

- Support "local" mode in LondConnection.pm when a server's lonc is connecting
  to its own lond. Sys::Hostname::FQDN::fqdn() preferred with Ubuntu 18.04
- Need to add dependency on Sys::Hostname::FQDN to LONCAPA-prerequisites.

- Support "local" mode in LondConnection.pm when a server's lonc is connecting
  to its own lond in distros where &hostname() returns only the short hostname
  (e.g., Ubuntu) by using Net::Domain::hostfqdn() in loncnew::my_hostname().
  (Net::Domain is a core perl module).
  -  On Ubuntu, also need to edit /etc/resolvconf/resolv.conf.d/tail
  to include: domain <domain name>  , and also run resolvconf -u.

- Log hostID for completed deferred transactions in lonnet.perm.log.

- Use of LON-CAPA SSL certificates to create secure channel for exchange of
  encryption key between LON-CAPA nodes can be set to: "no", "yes", "req"
  (i.e., required) separately for outbound (i.e., lonc) and inbound connections
  (i.e., lond) in domain config (default domain only on a multi-domain server).
- Can differentiate between connections between nodes in same domain, nodes
  in same "internet" domain. and nodes in neither.
- Supercedes use of loncAllowInsecure and londAllowInsecure from loncapa.conf
  on each node.
- Run /home/httpd/perl/loncron --justreload (as www) after changing domain config

- Eliminate lone single quotes in POD which could cause problems with
  syntax highlighting for some editors.

- Fix typos in POD.

Reset londconnecting if the connection count becomes 0 in KillSocket

- Fix typos in documentation.

Improve comment that describes why no action is taken on errors in write
to client socket.

Add a 'ClientData' member to LondConnection and have loncnew use it to
properly keep track of which connection is which.
(Bug 6377 comment 43).

Decrease the idle timeout to 5min...improve comment on how to set that
value.

Insulate child processes from SIGPIPE that will get thrown on a
disconnect.

Ensure that ticks are relative to last tick calls so they can't fall on the
heels of each other.

Ensure that time on flock keeping messages (forking server) in logs are
updated...note that this will make the ps axuww message include the time
of the last fork if I'm reading the code correctly..rather than the
start time.

- Cancel watcher on fd when killing off a socket...just in case we'll
  get 'readable' events on fds that are dead.
- When killing a socket update the ps axuww status that is displayed for the
  loncnew process.

Workaround for Bug #6304, until we have something better that is sufficiently tested.

Added documentation of the log messages to the pod.

- when doing reconnects, be more targeted, just reset the retries on the
  one loncnew child (if it even exists)

- delayed message flush log message to perm log was incorrectly formatted

- when handling the reload need to clear out the children preccesses after they have exited

- vcu was seeing this error condition occur, catch and exit in this case as
  somethign has gone awry

- when there is a race to create a connection make sure only one child gets forked off

- eliminate need for lonnet, by having lonnet send all known lonids over the launch channel
- need to quotemeta on the re now since lonids can have . and -

- eliminate any reading of hosts.tab from loncnew
- launch of lonc connection requires passing the lonid through the common
  launch socket
- dealyed messages getting sent again (requires loncnew using lonnet....)

- eliminate the non-DIeWhenIdle option

- switch on the dynamic hosts.tab support in loncnew by default

- switch lond to use new multiplex lonc communication route
- swithc to making the common socket a config option

- add ability for loncnew to work without having to read a hosts.tab

- typo and redeclaration

- BUG#4927 - delayed messages were not being noticed after the host name transition

- connections are now be hostname not lon machine id
- socket files are by hostname
- lonnet.pm prepends sethost:<lonid>: to each command
- LondConnection takes care of tracking with machine id the lond thinks it
    - it strips the sethost command if it's unneeded
    - it splits it into two seperate commands that it sends in order
      if lond needs to be told
    - eventually for lond of late enoyugh heritage it will just
      send the whole command

- typo

- upping the default number of connection attempts made

- BUg#4128

- after reseting the ConnectionRetries should update the status indicator

POD fixups.  It no longer throws errors, at least.

  some of BUG 3996

- backport sleep removal, and uppdatestatus fixes 1.66 1.67

- getting rid of the sleep 1, byy using locking instead

- the status might change throught the course of a Tick in a way that causes Tick()s to stop
  so need to update also at the end

Full signal handling in loncnew.  I believe this now works so I have set
DieWhenIdle to true.   If there are problems in an emergency, set DieWhenIdle
to 0 and old style behavior (no process trimming) will occur.

Factor out all the paths in the child that can exit and ensure they
create lock files for the unix domain socket if die when idle is turned
on.  Still need signal handling so DO NOT turn on DieWhenIdle unless
you don't care about that.

Add child exit handling to server trim branch of the develoment.  Note:
The other signals are still not appropriately handled.  Therefore do not
yet set DieWhenIdle true unless you are just playing around... and certainly
not in production servers.

Get the subprocess forking to work.  A lot of stuff still to do (Handling
child exit for one, signals etc. for another), so pleases still leave
DieWhenIdle false.

Start adding logic for parent listen.  All this is still conditionalized on
the DieWhenIdle variable, and is incomplete.  Do not set DieWhenIdle to true
as the current effect is DieBeforeStarting. (RF).

Add toplevel flow control for switching loncnew to monitoring the
lonnet socket and forking off kids that then die when their connection
count gets trimmed back to 0.  All this is conditionalized on the
DieWhenIdle variable which MUST NOT BE SET NONZERO as the support is
incomplete.  The variable is there so that I can commit my changes
and continue to test but not affect the current working loncnew
SO KEEP YOUR HANDS OFF IT:

Oh, and by the way, did I mention that you should not modify the
DieWhenIdle variable?

Remove policy decision that unix domain sockets for lonnet clients must
be connected to an IO event that dispatches to NewClient.  This is only
appropriate for the child process.  Come the revolution, when the parent
is taking over truly idle lonnet client sockets, things will be different.

Continue adding latent support for child exit when connection count
trimmed to zero: Allow default host for GetLoncSocketPath and
SetupLoncListener to be optionally parameterized with a loncapa
hostname.  The actual server processes will typically not parameterize
these, but the parent/master will.

- Factor out link status display into UpdateStatus
- Add a call to UpdateStatus when the connection count goes to zero.
  Otherwise, since the Tick clock gets disabled, the ps status never
  shows that the process has 0 connections.
- Add/test latent support in Tick for exiting the child process with
  a lock file on the unix domain socket when the number of connections
  has been trimmed to zero.  This is currently disabled (controlled via
  the DieWhenIdle variable).   Note that since parent logic for fork on
  accept has not yet been written, some transactions may fail if this
  support is enabled at this time, because there will be a time window
  during which the child is being restarted while there is no listen on
  the unix domain socket.

Only allow new connections if there are no connection requests in progress.
This should prevent connection explosions due to short term load spikes.
I believe that connection explosion may also be responsible for transaction
failures due to resource issues.

- up the number of people we can put on hold
- enable ron's connection increase code

- Fix a bit of an oops where I forgot to put a $ in front of a var.
- Set up to only increase connection count if there are no connections
  in the process of being established... just need change one line to get there.

  Fix the issue where loncnew would rapidly oscillate connections. The problem
was that once it started trimmning connections. they'd all get trimmed out
at the rate of one/sec, rather than at the rate of one /idle timeout
period... needed to reset the idle timeout each time a connection got
trimmed.

- Completely turn off ticking when there are no connections left.
- Add a bit more foreceful kill logic since there have been 'rumors' of cases where
  lonc's don't die appropriately.
- Remove some of the less useful log spew.

- Decrease Tick length on Idle servers

- many firewalls out there drop idle tcp connections that have been idl for 10 minutes or more.

Add connection authmode type to status.

Turn down the logging volume now that the ssl stuff looks like it's functional.

Make parameter passing meet standard using @_ not multiple shifts
or English @ARG.

- don't always log Connection status, only when debugging

- more xhtmlized

- Yeah, finally figured out the loncnew goes on a holiday bug

- retabinate

- Clean up write logic and factor out common code.
- Fix up some error logic to ensure that:
  o All failed transactions fromthe lonc point of view close the
    associated connection
  o All closures of a lond connection will fail any associated transaction
  o A bit of defensive programming when closing a connection that should be
    idle to ensure that any bad book-keeping gets cleaned up by failing
    the associaed transaction.

- fix tabination

Handle the additional states in LondConnection during connection setup.
Get and log the version of our peer.

Added capability for transaction logging. To toggle between 'normal' operation
and transaction logging and back, send USR2 at a specific lonc daemon.
Transactions are logged in the /home/httpd/perl/lonc.log file with success
status, as normal log messages.  Two types of messages are generated:
Transaction: yadayada
Reply from lond: yadayada

Transaction is a request issued to lonc from a local client.
Reply from lond:  is a transaction reply.

Note that no effort is made, at this time, to match up transactions and replies or even to tag them, so in a heavily used access server, this logging will be confusing as multiple simultaneous transactions and replies fly back and forth.

For later work, it may be useful to assign transactions a monotonically incrementing serial number and add that serial number to log messages that are relevant
to that transaction (sort of like the mail id you see in an smtp log file).

Add additional status output for SIGUSR1:
- In the status file give the status of each of the connections as well
  as (implicitly) the connection count.
- In the error file /home/httpd/perl/logs/lonc_error dump the state of each
  connection (depends on update to LondConnection.pm to redirect Dump to STDERR)


The idea is that if you have some weird state in loncnew perhaps this will give enough information to diagnose this or at least get started.

- better fix for handling the connection Count issue

- I am seeing negative Connection Counts, (BUG#2502) Forcing it to zero if this happens.

Simplify the connection timeout logic... iterate over all active connections
that are not idle.

- connections could get frozen in the connectiong state if the remote end accepted the connection but never got back to us on any of our commands BUG#2488

Remove debug flag from event timer setup.

- removing Debugging use lib
- removing change log

Call LondConnection::ReadConfig explicitly so that the configuration is loaded
prior to getting the hosts table hashiterator.

Tighten up a few compares to eq and flip a few debug levels around... nothing
critical

Fix little typo that may explain growth of connections

- making it easier to run loncnew,
   /etc/init.d/loncontrol startnew
   /etc/init.d/loncontrol restartnew
  will now start loncnew in place of lonc

Installed and tested code to process reinit in parent server.

Add book-keeping hashes to support the re-init procedure.

Tested ability to receive sigusr2  This is now logged and must be
properly implemented as a re-read of hosts and re-init of appropriate
children.

Added skeletal infrastructure to support SIGUSR2 update hosts request.

Add use strict and fix all the fallout from that.

- Fix errors in host dead detection logic (too many cases where the
  retries left were not getting incremented or just not checked).
- Added some additional status to the ps axuww display:
  o Remaining retries on a host.
  o >>> DEAD <<< indicator if I've given up on a host.
- Tested the SIGHUP will reset the retries remaining count (thanks to
  the above status stuff, and get allow the loncnew to re-try again
  on the host (thanks to the log).

How embarrassing... put in the SocketTimeout function in loncnew and forgot
to actually hook it into the LondTransaction.  Added this to MakeLondConnection
where it belongs... hopefully transactions (not just connection attempts) will
timeout more speedily than the socket errors will catch it.

- fixing a forgotten ;

Get socket directory from configuration rather than the old hard coded test
way that I forgot to un-hard code.

Also needed to remember to fail in-flight transactions if their sends fail.

1. Correct handling of connection failure: Assume it means the host is
   unreachable and fail all of the queued transactions.  Note that the
   inflight transactions should fail on their own time due either to timeout
   or send/receive failures.
2. Correct handling of logs for forced death signals.  Pull the signal
   from the event watcher.

Add SIGINT processing to child processes to toggle annoying trace mode
on/off.. will try to use this to isolate the compute boud process issue.

Added code for lonc/lond transaction timeouts.  Who knows if it works right.
The intent is for a timeout to fail any transaction in progress and kill
off the sockt that timed out.

Get all of the signals to work correctly.

Added kill -HUP logic (restart).

- Add some debugging to killthemall
- Add better error handling to LondReadable
- Remove tick logging in the timer handler.

Fix more problems with transaction failure.

Put a limit on  the number of times we'll retry a connection.
Start getting the signal stuff put in as well...note that need to get signals
going or else 6the client will permanently give up on dead servers.

Add logging in 'expected format'

Support delayed transactions... this is done uniformly by encapsulating
transactions in an object ... a LondTransaction that is implemented by
LondTransaction.pm

complete coding to support deferred transactions.

>>>>UNTESTED<<<<<
  Add beginnings of code for delayed transactions...
  Add code to fork off into separate session...
>>>>UNTESTED<<<<
  Comitting prior to ripping guts out to re-cast the transaction as an object
so that state/history/etc. can be more easily carried with it (need to do that
to know when to retire a delayed transaction).

- Add a bit more error checking on write fails.
- Write pid file.
- If a tick goes by and there's work to do but no connections attempt
  to start connections to process the work (can happen on initial
  connection fail.. this retries once per second).

Added code to roll back connection counts.  Also added code to show
status of daemon on ps -axuww: show who 'I'm connected to and current connection
count

- correcting pod documentation so that 'perldoc loncnew' runs and generates something that looks reasonable

- add GNU GPL headers

loncnew - lonc daemon the next generation.  This version of lonc is capable
of running multiple connections to lond's in the target hosts; This commit
moves loncnew from the experimental sandbox to the mainline development area,
as it has achieved minimally useful functionality