poiesis.core package

Subpackages

• poiesis.core.adaptors package
  • Subpackages
    • poiesis.core.adaptors.kubernetes package
      • Submodules
      • poiesis.core.adaptors.kubernetes.kubernetes module
      • poiesis.core.adaptors.kubernetes.models module
      • Module contents
    • poiesis.core.adaptors.message_broker package
      • Submodules
      • poiesis.core.adaptors.message_broker.redis_adaptor module
      • Module contents
  • Module contents
• poiesis.core.ports package
  • Submodules
  • poiesis.core.ports.kubernetes module
    • KubernetesPort
      • KubernetesPort.create_job()
      • KubernetesPort.create_pod()
      • KubernetesPort.create_pvc()
      • KubernetesPort.delete_pvc()
      • KubernetesPort.get_job()
      • KubernetesPort.get_pod()
      • KubernetesPort.get_pod_log()
      • KubernetesPort.get_pods()
  • poiesis.core.ports.message_broker module
    • Message
      • Message.message
      • Message.status
      • Message.timestamp
      • Message.to_json()
    • MessageBroker
      • MessageBroker.close()
      • MessageBroker.publish()
      • MessageBroker.subscribe()
    • MessageStatus
      • MessageStatus.ERROR
      • MessageStatus.SUCCESS
  • Module contents
• poiesis.core.services package
  • Subpackages
    • poiesis.core.services.filer package
      • Subpackages
      • Submodules
      • poiesis.core.services.filer.filer module
      • poiesis.core.services.filer.filer_strategy_factory module
      • poiesis.core.services.filer.tif module
      • poiesis.core.services.filer.tof module
      • Module contents
    • poiesis.core.services.texam package
      • Submodules
      • poiesis.core.services.texam.texam module
      • Module contents
    • poiesis.core.services.torc package
      • Submodules
      • poiesis.core.services.torc.torc module
      • poiesis.core.services.torc.torc_execution_template module
      • poiesis.core.services.torc.torc_texam_execution module
      • poiesis.core.services.torc.torc_tif_execution module
      • poiesis.core.services.torc.torc_tof_execution module
      • Module contents
  • Submodules
  • poiesis.core.services.models module
    • PodPhase
      • PodPhase.FAILED
      • PodPhase.PENDING
      • PodPhase.RUNNING
      • PodPhase.SUCCEEDED
      • PodPhase.UNKNOWN
  • Module contents

Submodules

poiesis.core.constants module

Constants used in core services.

class poiesis.core.constants.PoiesisCoreConstants

Bases: object

Constants used in core services.

K8s

Constants used in Kubernetes.

MessageBroker

Constants used in message broker.

Texam

Constants used in Texam.

class K8s

Bases: object

Constants used in Kubernetes.

K8S_NAMESPACE

The namespace in Kubernetes.

TES_TASK_PREFIX

The prefix for configmap holding the tes task request.

TORC_PREFIX

The prefix for the Task Orchestrator job name.

TIF_PREFIX

The prefix for the Task Input Filer job name.

TE_PREFIX

The prefix for the Task Executor pod name.

TOF_PREFIX

The prefix for the Task Output Filer job name.

PVC_PREFIX

The prefix for the Persistent Volume Claim name.

TES_TASK_PREFIX

The prefix for configmap holding the tes task request.

TES_TASK_CONFIGMAP_KEY

The key for configmap holding the tes task request.

TES_TASK_REQUEST_MOUNT_PATH

The path in the pods where the tes task request is mounted.

TEXAM_PREFIX

The prefix for the Texam job name.

PVC_DEFAULT_DISK_SIZE

The default disk size for the Persistent Volume Claim.

PVC_ACCESS_MODE

The access mode for PVCs (e.g., ReadWriteOnce, ReadWriteMany). Defaults to ReadWriteOnce for compatibility with most storage providers.

PVC_STORAGE_CLASS

The storage class name for PVCs. Defaults to ‘standard’ which is commonly available across different K8s distributions.

POIESIS_IMAGE

The Poiesis image.

COMMON_PVC_VOLUME_NAME

The common PVC volume name.

FILER_PVC_PATH

The path in the PVC for the filer.

S3_SECRET_NAME

The S3 K8s secret name.

REDIS_SECRET_NAME

The redis K8s secret name.

MONGODB_SECRET_NAME

The mongo K8s secret name.

MONGODB_URI_SECRET_KEY

The mongo K8s secret key.

SERVICE_ACCOUNT_NAME

The K8s service account name that allows core component to interact with K8s API and create, list and delete pods.

BACKOFF_LIMIT

The backoff limit for Job.

CONFIGMAP_NAME

The configmap name for the core services.

RESTART_POLICY

Restart policy for pods.

IMAGE_PULL_POLICY

Image pull policy.

JOB_TTL

Time in seconds after which the completed or failed job will be removed.

BACKOFF_LIMIT = 0

COMMON_PVC_VOLUME_NAME = 'task-pvc-volume'

CONFIGMAP_NAME = None

EXECUTOR_SECURITY_CONTEXT_ENABLED = True

FILER_PVC_PATH = '/transfer'

IMAGE_PULL_POLICY = 'IfNotPresent'

INFRASTRUCTURE_SECURITY_CONTEXT_ENABLED = True

JOB_TTL = None

K8S_NAMESPACE = 'poiesis'

MONGODB_SECRET_NAME = None

MONGODB_URI_SECRET_KEY = 'MONGODB_URI'

POIESIS_IMAGE = 'docker.io/jaeaeich/poiesis:latest'

PVC_ACCESS_MODE = None

PVC_DEFAULT_DISK_SIZE = '1Gi'

PVC_PREFIX = 'pvc'

PVC_STORAGE_CLASS = None

REDIS_SECRET_NAME = None

RESTART_POLICY = 'Never'

S3_SECRET_NAME = None

SECURITY_CONTEXT_CONFIGMAP_NAME = None

SECURITY_CONTEXT_PATH = None

SERVICE_ACCOUNT_NAME = None

TES_TASK_CONFIGMAP_KEY = 'task.json'

TES_TASK_PREFIX = 'tes-task'

TES_TASK_REQUEST_MOUNT_PATH = '/mnt/poiesis/tes'

TEXAM_PREFIX = 'texam'

TE_PREFIX = 'te'

TIF_PREFIX = 'tif'

TOF_PREFIX = 'tof'

TORC_PREFIX = 'torc'

class Texam

Bases: object

Constants used in Texam.

BACKOFF_LIMIT

The maximum time is second to wait in exponential backoff. starting from 1 second for a failing executor pod.

POLL_INTERVAL

The interval in seconds to poll the executor pod status as a fallback strategy if watch is not available.

MONITOR_TIMEOUT_SECONDS

The timeout in seconds to monitor the executor pod status. Default to 0, which means infinity.

BACKOFF_LIMIT = 60

MONITOR_TIMEOUT_SECONDS = 0

POLL_INTERVAL = 10

poiesis.core.constants.get_configmap_names()

Get names of the configmap.

Return type:

tuple[V1EnvVar, ...]

poiesis.core.constants.get_executor_container_security_context()

Returns a V1SecurityContext for task executor containers.

Returns:

The security context for task executor containers. None: If security context is disabled.

Return type:

V1SecurityContext

poiesis.core.constants.get_executor_pod_security_context()

Returns a V1PodSecurityContext for task executor pods.

Returns:

The security context for task executor pods. None: If security context is disabled.

Return type:

V1PodSecurityContext

poiesis.core.constants.get_executor_security_volume()

Returns the name of the security context configmap.

Return type:

list[V1Volume]

poiesis.core.constants.get_executor_security_volume_mount()

Returns the name of the security context configmap.

Return type:

list[V1VolumeMount]

poiesis.core.constants.get_infrastructure_container_security_context()

Returns a V1SecurityContext for infrastructure containers.

Returns:

The security context for infrastructure containers. None: If security context is disabled.

Return type:

V1SecurityContext

poiesis.core.constants.get_infrastructure_pod_security_context()

Returns a V1PodSecurityContext for infrastructure components.

Returns:

The security context for infrastructure components. None: If security context is disabled.

Return type:

V1PodSecurityContext

poiesis.core.constants.get_infrastructure_security_volume()

Returns the name of the security context configmap.

Return type:

list[V1Volume]

poiesis.core.constants.get_infrastructure_security_volume_mount()

Returns the name of the security context configmap.

Return type:

list[V1VolumeMount]

poiesis.core.constants.get_labels(component, task_id, name=None, parent=None)

Get the labels for a job or a PVC.

Parameters:

• component (str) – The component that is creating the resource.

• task_id (str) – The id of the task.

• name (str | None) – The name of the resource.

• parent (str | None) – The parent of the resource.

Return type:

dict[str, str]

Returns:

The labels for the resource.

poiesis.core.constants.get_message_broker_envs()

Get the env vars for redis.

Used in k8s manifest for tif, torc etc.

Return type:

tuple[V1EnvVar, ...]

poiesis.core.constants.get_mongo_envs()

Get the env vars for mongo.

Used in k8s manifest for tif, torc etc.

Return type:

tuple[V1EnvVar, ...]

poiesis.core.constants.get_poiesis_core_constants()

Get the Poiesis core constants.

Returns:

The Poiesis core constants.

Return type:

PoesisCoreConstants

poiesis.core.constants.get_s3_envs()

Get the env vars for s3.

Used in k8s manifest for tif, tof.

Return type:

tuple[V1EnvVar, ...]

poiesis.core.constants.get_secret_names()

Returns name of the secrets as env.

Return type:

tuple[V1EnvVar, ...]

poiesis.core.constants.get_security_context_envs()

Get the env vars for security context.

Return type:

tuple[V1EnvVar, ...]

poiesis.core.constants.get_tes_task_request_path()

Returns the full path to the TES task request file.

Returns:

The complete file path combining the mount path and config key.

Return type:

Path

poiesis.core.constants.get_tes_task_request_volume(tes_task_id)

Returns the volume for the TES task request.

Return type:

list[V1Volume]

poiesis.core.constants.get_tes_task_request_volume_mounts()

Returns the volume mounts for the TES task request.

Return type:

list[V1VolumeMount]

poiesis.core.logging_config module

Logging configuration for the core module.

class poiesis.core.logging_config.ColorFormatter(fmt=None, datefmt=None, style='%', validate=True, *, defaults=None)

Bases: Formatter

Color formatter for logging.

Parameters:

logging.Formatter – The formatter to use.

format(record)

Format the log message.

Parameters:

record (LogRecord) – The log record to format.

Return type:

str

Returns:

The formatted log message.

poiesis.core.logging_config.setup_logging(level=None)

Set up colorful logging configuration for the core module.

Parameters:

level (str | None) – Optional logging level. If not provided, defaults to INFO.

Return type:

None

Module contents

poiesis code package.

This package contains the core component of poiesis like Torc, Tif, Texam etc.

This file is also used to initialize the logging configuration.