Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

The Great LDAPification #33

Merged
merged 164 commits into from
Feb 14, 2025
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
164 commits
Select commit Hold shift + click to select a range
8e7ad31
Move notebooks to their own directory
camillescottatwork Jul 18, 2024
5b82740
Add conda environment file
camillescottatwork Jul 19, 2024
40ba268
Vendor xdg-base-dirs single-file dependency (https://github.com/srste…
camillescottatwork Jul 19, 2024
3b81b46
Add basic config file support and subcommands
camillescottatwork Jul 19, 2024
e74f948
Update deprecated NewType marshmallow types to Annotated
camillescottatwork Jul 24, 2024
52ec54e
Fix deprecated ruamel.safe_load usage
camillescottatwork Jul 24, 2024
29ea542
Sketch out testing code, add basic 'cheeto puppet validate' tests
camillescottatwork Jul 24, 2024
17b728c
Merge branch 'main' into ldap-refactor
camillescottatwork Jul 25, 2024
5473201
Remove pyyaml entirely in favor of using ruamel for both loading and …
camillescottatwork Jul 25, 2024
c438594
Update dependencies: particularly, updated marshmallow and marshmallo…
camillescottatwork Jul 25, 2024
bf1197f
Add tests for overriding values and bad shell
camillescottatwork Jul 25, 2024
8c20cfb
TestUserRecord load
camillescottatwork Jul 25, 2024
2520c80
Remove old tests directory
camillescott Jul 25, 2024
e4fc792
Test some basic user and group record loads
camillescott Jul 29, 2024
7390eba
Add tests for parse_yaml and puppet_merge
camillescott Jul 29, 2024
f05a78c
Tests for parse_yaml_forest
camillescott Jul 29, 2024
2101306
Basic LDAPManager with user querying, tests
camillescott Jul 30, 2024
4d88253
gssapi dev package for gssapi package
camillescott Jul 30, 2024
34fb353
ADd ipython as dev dependency
camillescottatwork Jul 31, 2024
5e85c7a
Add user and group classes and attrs to LDAP config
camillescottatwork Jul 31, 2024
a02f58d
Add testing config.yaml and UCD schemas
camillescottatwork Jul 31, 2024
0708d5c
Parse test config.yaml for testing config rather than creating and in…
camillescottatwork Jul 31, 2024
3cfaa50
user querying and adding, group adding
camillescottatwork Aug 3, 2024
5aa5da7
Basic ldap-sync for puppet YAML tree
camillescottatwork Aug 9, 2024
4b71c3f
Initial schemas, functions for database
camillescottatwork Aug 14, 2024
cdb4cc5
Update test config.yaml, remove unused collection field from mongo co…
camillescottatwork Aug 15, 2024
71bbbf2
Merged Mongo UserRecord type
camillescottatwork Aug 19, 2024
e28d0bb
site user query tweaks
camillescott Aug 19, 2024
352b438
User status updates, CLI for it
camillescottatwork Aug 20, 2024
f725fe0
Add YAML string representers to make sure long strings do not get wei…
camillescottatwork Aug 20, 2024
abb92e6
Add database index command; finish implementing full site-level user …
camillescottatwork Aug 20, 2024
58aca08
Add group querying CLI
camillescottatwork Aug 20, 2024
a589ba4
Fix querying for full records; fix param types with query params usin…
camillescottatwork Aug 20, 2024
9dcbdb7
Major group membership functionality
camillescottatwork Aug 22, 2024
86a6b76
Add convenience properties to puppet UserRecord
camillescottatwork Aug 22, 2024
054fdec
Generate HiPPO openapi client
camillescottatwork Aug 28, 2024
3f99e53
Generate UCD IAM openapi client
camillescottatwork Aug 28, 2024
20d03dd
Migrate to mongoengine.
camillescottatwork Aug 28, 2024
330ab03
Remove openapi geneator metadata files
camillescottatwork Aug 28, 2024
c9c0b6d
hippoapi event handling
camillescottatwork Aug 28, 2024
2d1eda2
Full hippoapi event processing, basic ldap load
camillescottatwork Aug 28, 2024
98b64d6
new-system subcommand for generating new system groups
camillescottatwork Aug 29, 2024
b4a6a49
Implement Slurm information in database
camillescottatwork Aug 30, 2024
0fbf723
Implement slurmers - ie slurm account membership without group member…
camillescottatwork Aug 30, 2024
dffa360
Add guarded database purge function
camillescottatwork Aug 30, 2024
0e4b465
Add sitename index field to slurm qos in database
camillescottatwork Aug 30, 2024
cd85a1b
Slurm sync adapter for database source
camillescottatwork Aug 30, 2024
9e7c97b
implement signal handlers for global groups and slurmers
camillescottatwork Aug 30, 2024
b964b57
Convenience properties for parent dereferencing; user portion of pupp…
camillescottatwork Aug 31, 2024
00fbbf6
Storage schemas, loading; some refactoring
camillescottatwork Sep 5, 2024
6613395
Add user groups to database on load
camillescottatwork Sep 6, 2024
0296e41
Full Storage implementation, puppet yaml from database
camillescottatwork Sep 10, 2024
90e6738
storage syncing, storage ldap
camillescottatwork Sep 13, 2024
a8a25cf
Finalize storage schemas
camillescottatwork Sep 14, 2024
bd61383
storage to-puppet
camillescottatwork Sep 14, 2024
f9a7a87
Represent ordereddict in yaml dump, separate user and group storages …
camillescottatwork Sep 15, 2024
55d917a
Add sshkey syncing; ldap force; and sympa dump
camillescottatwork Sep 16, 2024
2250979
Add root-key generation command
camillescottatwork Sep 16, 2024
58c9bf5
Make sure the special groups still get created during ldap sync
camillescottatwork Sep 16, 2024
c917aac
Remove defaults channel from environment yaml
camillescottatwork Sep 16, 2024
8302d9a
Add site list, don't use rich print
camillescottatwork Sep 16, 2024
f79b9a9
add TLS mongo config option, have cheeto print some basic info to std…
camillescottatwork Sep 16, 2024
81a82d8
implement git syncing for new puppet standard
camillescottatwork Sep 16, 2024
9801b01
bump version
camillescottatwork Sep 16, 2024
7b78c12
Handle empty commit properly
camillescottatwork Sep 16, 2024
2f55aa8
Give SiteSlurmAccount slurm attribute on SiteGroup a default
camillescottatwork Sep 16, 2024
65af2ce
Allow using domain name for --site argument in cheeto slurm sync
camillescottatwork Sep 16, 2024
1091056
bump version
camillescottatwork Sep 16, 2024
f3c52e9
Make defaults for PuppetSlurmTRES -1 instead of None
camillescottatwork Sep 16, 2024
534fe65
Add group type field; add slurm as access type; add sync cmd for old …
camillescottatwork Sep 17, 2024
e66d8d7
bump version
camillescottatwork Sep 17, 2024
d4a998b
add additional account level slurm params; optimize association handl…
camillescottatwork Sep 18, 2024
120c176
bump version
camillescottatwork Sep 18, 2024
a12cd0d
fix sitename bugs in event processing; fix bugs in home storage querying
camillescottatwork Sep 18, 2024
362a4aa
bump version
camillescottatwork Sep 18, 2024
273a50e
add class user type, fix some bugs with puppet dumping
camillescottatwork Sep 18, 2024
dabb69b
Fix slurm sacctmgr account add
camillescottatwork Sep 19, 2024
dd4e0fb
Implement add and remove for all SiteGroup listfields; fix repeated c…
camillescottatwork Sep 20, 2024
8eb9900
Implement ldap_synced update in custom QuerySet; add user access meth…
camillescottatwork Sep 21, 2024
e465331
Add help text to database user subcommands
camillescottatwork Sep 21, 2024
4f77321
Change some puppet.py dataclass types to List to preserve sorting
camillescottatwork Sep 25, 2024
86cc374
Fix ordering on a few more fields
camillescottatwork Sep 25, 2024
5abd939
Fix bad .only() call in query_user_exists
camillescottatwork Sep 26, 2024
a5ffa41
Add class group type; refactor command method names; add some more pr…
camillescottatwork Oct 4, 2024
1029c4c
Regenerate hippoapi
camillescottatwork Oct 4, 2024
4366051
Start reorg for mail
camillescottatwork Oct 8, 2024
9660fe5
Upgrade dependencies
camillescottatwork Oct 8, 2024
efc2124
Add xkcdpass and pyescrypt to poetry conf
camillescottatwork Oct 12, 2024
934636c
Add config profiles
camillescottatwork Oct 12, 2024
426a807
Implement set-password and generate-passwords subcommands
camillescottatwork Oct 12, 2024
63654e7
bit of formatting
camillescottatwork Oct 12, 2024
deecac8
More rearrangment
camillescottatwork Oct 14, 2024
84aad67
Storage collection commands; user show queries; command aliases; etc
camillescottatwork Oct 24, 2024
7c08582
Slurm command group; subcommands for new QOSes and associations
camillescottatwork Oct 25, 2024
40b2882
Add --status filtering on user show
camillescottatwork Oct 25, 2024
4ceee92
Implement basic n-gram search for users
camillescottatwork Oct 28, 2024
c08cf3e
Remove deprecated user query and group query subcommands
camillescottatwork Oct 28, 2024
a672ff8
Bring in CmdTree and related code from Cryptic
camillescottatwork Nov 1, 2024
5559827
config command and postprocessors
camillescottatwork Nov 1, 2024
c8d5d36
Move EnumAction to args
camillescottatwork Nov 1, 2024
ab976d1
Convert cheeto subcommands to new CmdTree API
camillescottatwork Nov 1, 2024
0fb19a2
Remove a bunch of the old command registration
camillescottatwork Nov 1, 2024
21ae791
Fixes to handle_site_ functions
camillescottatwork Nov 1, 2024
7309256
Merge branch '_tmp' into ldap-refactor
camillescottatwork Nov 1, 2024
55e6366
more arg changes
camillescottatwork Nov 1, 2024
2d5b1d0
Refactor out CmdTree code and use ponderosa; convert database command…
camillescottatwork Nov 4, 2024
59b7cd3
Move all commands to cmds submodule, remove some deprecated commands
camillescottatwork Nov 4, 2024
b6ca841
arg cleanup, ponderosa version bump
camillescottatwork Nov 5, 2024
2011172
Remove double logging handler with defaulted /dev/null; instead do no…
camillescottatwork Nov 5, 2024
225314e
Properly enforce uniqueness constraint on GlobalUser and GlobalGroup …
camillescottatwork Nov 5, 2024
9551a85
Fix edge cases in get_next_system_id and related funcs
camillescottatwork Nov 5, 2024
148316a
Improved Document pretty-printing and improved new-class command
camillescottatwork Nov 5, 2024
8fcdde9
Email templates and impl, updated mongoengine with transaction suppor…
camillescottatwork Nov 20, 2024
e26465f
decompose some subcommands, add commands for adding existing global u…
camillescottatwork Nov 20, 2024
22b278c
Allow setting gid in create_user
camillescottatwork Nov 20, 2024
16287f9
force add banner.txt
camillescottatwork Nov 21, 2024
d80facf
bump version: 1.0
camillescottatwork Nov 21, 2024
6d98a1c
Fix missing imports; bump version to 1.0.1
camillescottatwork Nov 21, 2024
6cfc5c3
Fix subtle bug with site_arg postprocessor
camillescottatwork Nov 21, 2024
019750b
Add site creation subcommand
camillescottatwork Dec 18, 2024
b031e7b
Have sync-new-puppet dump 'share' table as well
camillescottatwork Dec 18, 2024
f7a93f9
bump: v1.0.2 -> v1.1.0
camillescottatwork Dec 18, 2024
b0145e9
Export /share as 2775
camillescottatwork Dec 23, 2024
a98d5e3
v1.1.1
camillescottatwork Dec 23, 2024
cf0c510
mark ldap_synced False in hippo updatesshkey event
camillescottatwork Jan 16, 2025
f8aaecb
1.1.1 => 1.1.2
camillescottatwork Jan 16, 2025
bd565eb
generate hippoapi script
camillescottatwork Jan 17, 2025
19c55c5
Update gitignore
camillescottatwork Jan 17, 2025
c2381a5
Add poetry-bumpversion
camillescottatwork Jan 21, 2025
1d16421
Add marshmallow import to types.py
camillescottatwork Jan 21, 2025
77839d7
Implement loading of sub-sites
camillescottatwork Jan 21, 2025
156b939
v1.1.3
camillescottatwork Jan 22, 2025
cb66b2f
Normalize QOS representation between cheeto DB and slurmdb
camillescottatwork Jan 23, 2025
828f37e
v1.1.4
camillescottatwork Jan 23, 2025
a359fa9
More help info for core subcommands
camillescottatwork Jan 23, 2025
374256b
Implement storage source edit command
camillescottatwork Jan 24, 2025
755a89e
v1.1.5
camillescottatwork Jan 24, 2025
5fdc1f7
Improve storage show output
camillescottatwork Jan 24, 2025
3d6f069
v1.1.6
camillescottatwork Jan 24, 2025
5ad6d0a
Reload StorageSource after updating before displaying in storage sour…
camillescottatwork Jan 24, 2025
4b043ec
v1.1.7
camillescottatwork Jan 24, 2025
a28b4d1
sync home dirs to ldap
camillescottatwork Jan 29, 2025
b5c8c62
Break down database.py into a submodule with seperate files
camillescottatwork Jan 31, 2025
a90400d
implement db user set shell cmd
camillescottatwork Jan 31, 2025
5d1e010
regenerate iamapi
camillescottatwork Feb 4, 2025
1b13738
Implement basic IAM data sync and IAM fields on User
camillescottatwork Feb 4, 2025
2375098
fix statistics import
camillescottatwork Feb 4, 2025
dd441ae
v1.2
camillescottatwork Feb 4, 2025
ea9de66
Fix imports.
camillescottatwork Feb 5, 2025
d650991
v1.2.1
camillescottatwork Feb 5, 2025
64e0efc
Add mongod temp instance setup functionality
camillescottatwork Feb 6, 2025
4f550d9
DB tests, error handling
camillescottatwork Feb 8, 2025
6cad5e3
Install mongodb on github actions
camillescottatwork Feb 8, 2025
132e4da
bump poetry version in actions
camillescottatwork Feb 8, 2025
1dc0f5b
query_user_home_storage: check against mount site as well; use owner …
camillescottatwork Feb 11, 2025
81c4643
v1.2.2
camillescottatwork Feb 11, 2025
377368f
actions: bump cache to v4
camillescottatwork Feb 11, 2025
b120e5a
actions: bump second cache action to v4
camillescottatwork Feb 11, 2025
af5f49b
handle_createaccount_event: reenable creation of home storages
camillescottatwork Feb 12, 2025
6507c97
v1.2.3
camillescottatwork Feb 12, 2025
07afae4
handle_createaccount_event: rewrite with improved error handling; add…
camillescottatwork Feb 13, 2025
1a4afaa
test_db: ensure_indexes() when doing collection reset
camillescottatwork Feb 13, 2025
8956447
test_ldap: update tests and test-data
camillescottatwork Feb 14, 2025
93a2bca
user add site: fix invalid variable
camillescottatwork Feb 14, 2025
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
14 changes: 8 additions & 6 deletions .github/workflows/test.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -10,29 +10,31 @@ on:

jobs:
test:
runs-on: ubuntu-latest
runs-on: ubuntu-22.04
steps:
- name: Install apt dependencies
run: |
curl -fsSL https://www.mongodb.org/static/pgp/server-8.0.asc | sudo gpg -o /usr/share/keyrings/mongodb-server-8.0.gpg --dearmor
echo "deb [ arch=amd64,arm64 signed-by=/usr/share/keyrings/mongodb-server-8.0.gpg ] https://repo.mongodb.org/apt/ubuntu jammy/mongodb-org/8.0 multiverse" | sudo tee /etc/apt/sources.list.d/mongodb-org-8.0.list
sudo apt update
sudo apt install libldap-2.5-0 libldap-dev libsasl2-dev
sudo apt install -y libldap-2.5-0 libldap-dev libsasl2-dev libkrb5-dev gnupg curl mongodb-org
- uses: actions/checkout@v2
- uses: actions/setup-python@v2
with:
python-version: 3.12
- name: cache poetry install
uses: actions/cache@v2
uses: actions/cache@v4
with:
path: ~/.local
key: poetry-1.8.3-0
key: poetry-2.0.1-0
- uses: snok/install-poetry@v1
with:
version: 1.8.3
version: 2.0.1
virtualenvs-create: true
virtualenvs-in-project: true
- name: cache deps
id: cache-deps
uses: actions/cache@v2
uses: actions/cache@v4
with:
path: .venv
key: pydeps-${{ hashFiles('**/poetry.lock') }}
Expand Down
1 change: 1 addition & 0 deletions .gitignore
Original file line number Diff line number Diff line change
@@ -1,3 +1,4 @@
cheeto/**/__pycache__
.ipynb_checkpoints
*.txt
.vscode
2 changes: 1 addition & 1 deletion cheeto/__init__.py
Original file line number Diff line number Diff line change
@@ -1 +1 @@
__version__ = '0.9.8'
__version__ = '1.2.3'
61 changes: 0 additions & 61 deletions cheeto/__main__.py

This file was deleted.

20 changes: 0 additions & 20 deletions cheeto/_yaml.py

This file was deleted.

78 changes: 49 additions & 29 deletions cheeto/args.py
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
#!/usr/bin/env python3
# -*- coding: utf-8 -*-
# (c) Camille Scott, 2023
# (c) Camille Scott, 2023-2024
# (c) The Regents of the University of California, Davis, 2023
# File : __main__.py
# License: Modified BSD
Expand All @@ -10,39 +10,59 @@
# pyright: reportMissingTypeArgument=true


from argparse import ArgumentParser, Namespace, _SubParsersAction
import os
from functools import wraps
from pathlib import Path
from typing import Callable
from typing_extensions import Concatenate, ParamSpec
from argparse import (Action,
ArgumentTypeError)
from enum import Enum
import re

P = ParamSpec('P')
Subparsers = _SubParsersAction
NS = Namespace
NamespaceFunc = Callable[Concatenate[NS, P], None]
SubCommandFunc = Callable[Concatenate[Subparsers, P], None]
from . import __version__
from .templating import PKG_TEMPLATES


def add_common_args(parser):
parser.add_argument('--log', type=Path, default=Path(os.devnull),
help='Log to file.')
parser.add_argument('--quiet', default=False, action='store_true')

splash = (PKG_TEMPLATES / 'banner.txt').read_text().rstrip('\n').format(version=f'v{__version__}')
banner = f'''
\b
{splash}
'''

def subcommand(subcommand_name: str, *arg_adders: Callable[[ArgumentParser], None]) \
-> Callable[[NamespaceFunc[P]], SubCommandFunc[P]]:
def regex_argtype(pattern: re.Pattern[str] | str):
_pattern = pattern
def inner(value: str | None):
if value is None:
return value
if not isinstance(_pattern, re.Pattern):
pattern = re.compile(_pattern)
else:
pattern = _pattern
if not pattern.match(value):
raise ArgumentTypeError(f'Invalid value, should match: {pattern.pattern}')
return value
return inner

def wrapper(func: NamespaceFunc[P]) -> SubCommandFunc[P]:

@wraps(func)
def wrapped(parent_parser: Subparsers, *args: P.args, **kwargs: P.kwargs) -> None:
parser = parent_parser.add_parser(subcommand_name)
add_common_args(parser)
parser.set_defaults(func=func)
for adder in arg_adders:
adder(parser)

return wrapped
class EnumAction(Action):
"""
Argparse action for handling Enums
"""
def __init__(self, **kwargs):
# Pop off the type value
enum = kwargs.pop("type", None)

return wrapper
# Ensure an Enum subclass is provided
if enum is None:
raise ValueError("type must be assigned an Enum when using EnumAction")
if not issubclass(enum, Enum):
raise TypeError("type must be an Enum when using EnumAction")

# Generate choices from the Enum
kwargs.setdefault("choices", tuple(e.name for e in enum))

super(EnumAction, self).__init__(**kwargs)

self._enum = enum

def __call__(self, parser, namespace, values, option_string=None):
# Convert value back into an Enum
enum = self._enum[values]
setattr(namespace, self.dest, enum)
22 changes: 22 additions & 0 deletions cheeto/cmds/__init__.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,22 @@
from argparse import Namespace, RawDescriptionHelpFormatter
import sys

from ponderosa import CmdTree

from .. import __version__
from ..args import banner
from ..log import Console


commands = CmdTree(description=banner,
formatter_class=RawDescriptionHelpFormatter)
commands._root.add_argument('--version', action='version', version=f'cheeto {__version__}')


@commands.register('help',
help='Show the full subcomamnd tree')
def help(args: Namespace):
console = Console()
print(banner, file=sys.stderr)
console.print(commands.format_help())
return 0
27 changes: 27 additions & 0 deletions cheeto/cmds/__main__.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,27 @@
#!/usr/bin/env python3
# -*- coding: utf-8 -*-
# (c) Camille Scott, 2023
# (c) The Regents of the University of California, Davis, 2023
# File : __main__.py
# License: Modified BSD
# Author : Camille Scott <[email protected]>
# Date : 17.02.2023

import sys

from . import (commands,
config,
database,
hippo,
monitor,
nocloud,
puppet,
slurm)


def main():
return commands.run()


if __name__ == '__main__':
sys.exit(main())
Loading
Loading