Add an include dir feature to condor mapfiles
For HTCondor-CE, we'd like the ability to add lines to our condor_mapfiles to include directories containing additional mapfiles. We were imagining a syntax like the following:
Where lines in the mapfile would be parsed in order, then when an include dir line is encountered, HTCondor would parse files in the directory in lexicographical order.
We'd also like the ability to have multiple include directories in a single mapfile for default mappings and admin overrides (and therefore should be able to handle empty directories).
From the discussion in https://htcondor-wiki.cs.wisc.edu/index.cgi/tktview?tn=7592,4:
2020-Apr-10 10:38:23 by zmiller:
That all seems reasonable. Seems like admin overrides would just be an include at the top of the file, since first match wins. And defaults could just go last. So something like this?
SCITOKENS https://demo.scitokens.org firstname.lastname@example.org
SCITOKENS https://iam-escape.cloud.cnaf.infn.it/ email@example.com
SCITOKENS https://scitokens.org/dteam firstname.lastname@example.org
Multiple @include <thing> statements are allowed, and <thing> can be either a file or a directory. Include files can include additional files. (Up to a default MAX_MAP_INCLUDE_DEPTH=20 deep in case someone decides to make a cycle.)
2020-Apr-10 10:45:07 by blin:
Yeah, that all sounds great!
Additionally, CanonicalMapEntry::dump() makes me sad. We’re casting to pointers based on a type field rather than using a virtual function. Are we really so short on memory that we can’t do this right? Given that entry_type is totally redundant with a vtable, and the presence of spare, are we actually reducing memory usage at all?
Barring that, the rest of the code looks good.
ToddT, TJ, and TimT briefly discussed this ticket. Plan is to allow in the map:
an include that points to a file (no wildcards or regex)
an include that points to a directory (no wildcards or regex), in which case all files in that directory are included in lexicographic order while ignoring files specified with LOCAL_CONFIG_DIR_EXCLUDE_REGEXP
In both cases, the file can be an absolute path, or a path that is relative to the location of the map file containing the include. In the initial implementation, only the top level map file may have includes; in other words, MAX_MAP_INCLUDE_DEPTH=1 is hard-coded and attempts to include in a map already included will result in a specific error message.
The use of an exclusion list for the config dir was not a good design decision. it was clearly an after the fact patch to solve a problem that came from not using a glob in the first place and then discovering that RMP and various editors would leave behind files in the config dir.
As a said before, there is no reason at all to believe that a config dir exclusion regex is designed for config files is sufficient to solve the problem of over-inclusion. It is a perfectly reasonable design to put your .map files into your config dir and then adding them to the exclusion regex so that they don’t get parsed as config files.
I'm with Brian here. We should just reuse the code that is used to select files from configuration directories to pick up the map files.
To also mention here –
I think there’s power in using the same approach as the configuration subsystem. While I agree that these are technically two different parts of HTCondor, I wonder whether most admins would appreciate the difference. To the sysadmin who has never written a submit file, all the files they dump in /etc/condor are probably “the configuration”.
Particularly, what’s the benefit from having two distinct, incompatible mechanisms for “including directories” in HTCondor? Is there a specific benefit from introducing a unique mechanism for mapfiles that is different from the configuration directory? Is there a case where the existing config directory has been insufficient and users have asked for more fine-grained globs?
Similarly, while reusing LOCAL_CONFIG_DIR_EXCLUDE_REGEXP perhaps abuses the name, is it really worthwhile to have another configuration knob so the admin can define transient files (like emacs temp files) differently depending on which condor configuration directory is being processed? Have we seen anyone change this configuration knob in practice?
As has been saying recently, this feels like an opportunity to be a bit more “Steve Jobs”-like and less “infinitely configurable”.