Add an include dir feature to condor mapfiles

Description

For HTCondor-CE, we'd like the ability to add lines to our condor_mapfiles to include directories containing additional mapfiles. We were imagining a syntax like the following:

Where lines in the mapfile would be parsed in order, then when an include dir line is encountered, HTCondor would parse files in the directory in lexicographical order.

We'd also like the ability to have multiple include directories in a single mapfile for default mappings and admin overrides (and therefore should be able to handle empty directories).

From the discussion in https://htcondor-wiki.cs.wisc.edu/index.cgi/tktview?tn=7592,4:

2020-Apr-10 10:38:23 by zmiller:
That all seems reasonable. Seems like admin overrides would just be an include at the top of the file, since first match wins. And defaults could just go last. So something like this?

@include /path/to/admin_override_file
SCITOKENS https://demo.scitokens.org bbockelm@users.opensciencegrid.org
SCITOKENS https://iam-escape.cloud.cnaf.infn.it/ bbockelm@users.opensciencegrid.org
SCITOKENS https://scitokens.org/dteam dteam@users.opensciencegrid.org
@include /path/to/specific/map_file
@include /path/to/a/directory/mapfiles.d/
@include /path/to/specific/map_file_defaults_if_nothing_else_matched

Multiple @include <thing> statements are allowed, and <thing> can be either a file or a directory. Include files can include additional files. (Up to a default MAX_MAP_INCLUDE_DEPTH=20 deep in case someone decides to make a cycle.)

Sound good?

2020-Apr-10 10:45:07 by blin:
Yeah, that all sounds great!

Activity

Show:
Todd L Miller
January 13, 2021, 7:26 PM

Code Review

Needs documentation.

Additionally, CanonicalMapEntry::dump() makes me sad. We’re casting to pointers based on a type field rather than using a virtual function. Are we really so short on memory that we can’t do this right? Given that entry_type is totally redundant with a vtable, and the presence of spare[], are we actually reducing memory usage at all?

Barring that, the rest of the code looks good.

Todd Tannenbaum
January 4, 2021, 10:52 PM

ToddT, TJ, and TimT briefly discussed this ticket. Plan is to allow in the map:

  1. an include that points to a file (no wildcards or regex)

  2. an include that points to a directory (no wildcards or regex), in which case all files in that directory are included in lexicographic order while ignoring files specified with LOCAL_CONFIG_DIR_EXCLUDE_REGEXP

In both cases, the file can be an absolute path, or a path that is relative to the location of the map file containing the include. In the initial implementation, only the top level map file may have includes; in other words, MAX_MAP_INCLUDE_DEPTH=1 is hard-coded and attempts to include in a map already included will result in a specific error message.

John (TJ) Knoeller
December 28, 2020, 8:40 PM

The use of an exclusion list for the config dir was not a good design decision. it was clearly an after the fact patch to solve a problem that came from not using a glob in the first place and then discovering that RMP and various editors would leave behind files in the config dir.

As a said before, there is no reason at all to believe that a config dir exclusion regex is designed for config files is sufficient to solve the problem of over-inclusion. It is a perfectly reasonable design to put your .map files into your config dir and then adding them to the exclusion regex so that they don’t get parsed as config files.

Tim Theisen
December 20, 2020, 6:22 PM

I'm with Brian here. We should just reuse the code that is used to select files from configuration directories to pick up the map files.

Brian Bockelman
December 20, 2020, 3:00 PM

To also mention here –

I think there’s power in using the same approach as the configuration subsystem. While I agree that these are technically two different parts of HTCondor, I wonder whether most admins would appreciate the difference. To the sysadmin who has never written a submit file, all the files they dump in /etc/condor are probably “the configuration”.

Particularly, what’s the benefit from having two distinct, incompatible mechanisms for “including directories” in HTCondor? Is there a specific benefit from introducing a unique mechanism for mapfiles that is different from the configuration directory? Is there a case where the existing config directory has been insufficient and users have asked for more fine-grained globs?

Similarly, while reusing LOCAL_CONFIG_DIR_EXCLUDE_REGEXP perhaps abuses the name, is it really worthwhile to have another configuration knob so the admin can define transient files (like emacs temp files) differently depending on which condor configuration directory is being processed? Have we seen anyone change this configuration knob in practice?

As has been saying recently, this feels like an opportunity to be a bit more “Steve Jobs”-like and less “infinitely configurable”.

Due date

2021/01/15

Time remaining

0m

Assignee

John (TJ) Knoeller