Skip to content

Jobs Management API (2)

The Job Management API allows for the management and execution of asynchronous jobs within CDQ services. This API enables the creation, monitoring, and management of jobs, allowing developers to automate and streamline complex workflows and tasks.

Languages
Servers
Mock server

https://idp.cdq.com/_mock/apis/jobs-management-api/api-v2/

Production SOAP

https://job-management-api.reverse-proxy.prod.k8s.production.cdq-cloud-engine.com/soap/v2/

Production

https://job-management-api.reverse-proxy.prod.k8s.production.cdq-cloud-engine.com/

Jobs

The Jobs resource represents a job executed asynchronously. Jobs are used to perform long-running tasks that can be monitored and managed through the API. Jobs can be created, updated, and deleted, and their status can be monitored and queried.

Operations

List Jobs

Request

List jobs that meet the querying criteria.

Query
userstring(UserId)

The user that started the job.

Example: user=johndoe
organizationIdstring(OrganizationId)

The organizationId parameter set in the job.

Example: organizationId=cdq_monitor
storageIdstring(BusinessPartnerStorageId)

The storageId parameter set in the job.

Example: storageId=72d6900fce6b326088f5d9d91049e3e6
jobTypeArray of strings<= 100 items

The kind of job described as JobType.

Example: jobType=DATA_CURATION_JOB, DATA_VALIDATION_JOB
sortstring

Define the attributes to sort by. Supported are in (createdAt), separated by comma (,). Sort "+" means ascending, "-" means descending. Default sort (when no + or - set) is ascending.

Example: sort=-createdAt
jobStatusArray of strings(JobStatus)<= 50 items

The kind of job described as JobStatus.

Items Enum ValueDescription
UNKNOWN

Job becomes in unknown status.

CREATED

Job has been created.

PERSISTED

Job metadata has been persisted.

SCHEDULED

Job has been scheduled for execution.

WAITING

Job is waiting for being scheduled.

RUNNING

Job is being executed.

FINISHED

Job has finished.

DIED

Job was scheduled and started running but died unexpectedly.

CANCELED

Job has been canceled due to invalid input configuration.

FAILED

Job has failed.

Example: jobStatus=RUNNING
startAfterstring(StartAfter)

Indicator for the next page.

Example: startAfter=5712566172571652
limitinteger(int32)[ 1 .. 100 ]

Number of results that should be fetched. Maximum 100 results can be returned in one page.

Default 50
Example: limit=50
tagArray of strings(JobTagResult)<= 100 items

Filter jobs by tag.

Example: tag=tag
tagsConjunctionboolean

Treats tags as AND operation. By default, tags are treated as OR operation.

Default false
Example: tagsConjunction=false
featuresOnArray of strings(JobsReadFeatureParam)

Enable additional features when fetching jobs. Supported features:

  • NUMBER_OF_TOTAL - sends in response number of total jobs meeting the querying criteria
  • TAGS_CONJUNCTION - Treats tags as AND operation. By default, tags are treated as OR operation.
  • IGNORE_ORGANIZATION_RESTRICTIONS - allows to read jobs from any organization.
Items Enum ValueDescription
NUMBER_OF_TOTAL

Number of totals.

TAGS_CONJUNCTION

Tags conjunction.

IGNORE_ORGANIZATION_RESTRICTIONS

Ignore organization restrictions.

Example: featuresOn=NUMBER_OF_TOTAL
dependentJobIdstring(JobId)

The ID of a job that it depends on. Requires storageId and jobType parameters to be provided.

Example: dependentJobId=35f23c03-1c22-45fe-9484-3ffe769325de
curl -i -X GET \
  https://idp.cdq.com/_mock/apis/jobs-management-api/api-v2/v2/jobs

Responses

OK

Bodyapplication/json
jobsArray of objects(Job)

List of jobs.

nextStartAfterstring(NextStartAfter)

Provides a value to be used as a startAfter in next page request.

Example: "5712566172571652"
totalinteger(int64)

Total number of items, which can be paged.

Example: "67"
Response
application/json
{
  "jobs": [
    { … }
  ],
  "nextStartAfter": "5712566172571652",
  "total": "67"
}

List Job Types

Request

List all available job types.

Query
featureOnArray of strings(JobTypeFeatureParam)

Features to select subset of Job Types.

Items ValueDescription
CANCELLABLE

Job can be canceled.

Example: featureOn=CANCELLABLE
startAfterstring(StartAfter)

Indicator for the next page.

Example: startAfter=5712566172571652
limitinteger(int32)[ 1 .. 100 ]

Number of results that should be fetched. Maximum 100 results can be returned in one page.

Default 50
Example: limit=50
curl -i -X GET \
  https://idp.cdq.com/_mock/apis/jobs-management-api/api-v2/v2/jobs/types

Responses

OK

Bodyapplication/json
jobsArray of objects(JobTypeSearchResult)

List of job types.

nextStartAfterstring(NextStartAfter)

Provides a value to be used as a startAfter in next page request.

Example: "5712566172571652"
Response
application/json
{
  "jobs": [
    { … }
  ],
  "nextStartAfter": "5712566172571652"
}

Update Job Fields

Request

Update specific fields in job.

Path
jobIdstring(JobId)required

Unique identifier of the job.

Example: 35f23c03-1c22-45fe-9484-3ffe769325de
Bodyapplication/jsonrequired

Update specific fields in job.

namestring<= 50 characters

The name of the job.

Example: "job-name"
descriptionstring<= 200 characters

The description of the job.

Example: "job-description"
curl -i -X PATCH \
  https://idp.cdq.com/_mock/apis/jobs-management-api/api-v2/v2/jobs/35f23c03-1c22-45fe-9484-3ffe769325de \
  -H 'Content-Type: application/json' \
  -d '{
    "name": "job-name",
    "description": "job-description"
  }'

Responses

OK

Bodyapplication/json
commandsArray of strings

List of commands to be executed.

Example: ["command"]
startedAtstring

The time when the job was started.

Example: "2019-09-07T00:00:00Z"
createdAtstring(CreatedAt)

Date of creation (ISO 8601-compliant).

Example: "2020-08-31T16:47+00:00"
modifiedAtstring(ModifiedAt)

Date of modification (ISO 8601-compliant).

Example: "2020-08-31T16:47+00:00"
finishedAtstring

The time when the job was finished.

Example: "2019-09-07T00:00:00Z"
expiresAtstring

The time when the job expires.

Example: "2019-09-07T00:00:00Z"
dockerImageUrlstring

The URL of the Docker image.

Example: "data-curation"
domainstring

The domain of the job.

Example: "cdqag/data-curation-job"
environmentVariablesArray of objects(EnvironmentVariable)

Max 512 kB allowed.

idstring(JobId)

Unique identifier of a job.

Example: "35f23c03-1c22-45fe-9484-3ffe769325de"
inputUristring

The URI of the input.

Example: "input-uri"
jobDependenciesArray of objects(JobDependency)

List of job dependencies.

jobStatusstring(JobStatus)

Job execution status.

Enum ValueDescription
UNKNOWN

Job becomes in unknown status.

CREATED

Job has been created.

PERSISTED

Job metadata has been persisted.

SCHEDULED

Job has been scheduled for execution.

WAITING

Job is waiting for being scheduled.

RUNNING

Job is being executed.

FINISHED

Job has finished.

DIED

Job was scheduled and started running but died unexpectedly.

CANCELED

Job has been canceled due to invalid input configuration.

FAILED

Job has failed.

Example: "RUNNING"
messagestring(JobStatusMessage)

Additional information to explain the status.

Example: "The job failed because storage is empty."
labelsArray of objects(Label)

List of labels.

tagsArray of strings(JobTagResult)

List of tags.

Example: ["tag"]
lastJobStatusstring(JobStatus)

Job execution status.

Enum ValueDescription
UNKNOWN

Job becomes in unknown status.

CREATED

Job has been created.

PERSISTED

Job metadata has been persisted.

SCHEDULED

Job has been scheduled for execution.

WAITING

Job is waiting for being scheduled.

RUNNING

Job is being executed.

FINISHED

Job has finished.

DIED

Job was scheduled and started running but died unexpectedly.

CANCELED

Job has been canceled due to invalid input configuration.

FAILED

Job has failed.

Example: "RUNNING"
progressinteger(int32)

The progress of the job.

Example: "50"
totalItemsinteger(int64)

The total number of items.

Example: "100"
processedItemsinteger(int64)

The number of processed items.

Example: "50"
failedItemsinteger(int64)

The number of failed items.

Example: "10"
reportUristring

The URI of the report.

Example: "https://link-to-my-report"
resourceRequestobject(ResourceRequest)

Resource request.

userstring(JobUser)

ID of (human) user or API key.

Example: "742429-234242-4343-232323"
storageIdstring(BusinessPartnerStorageId)

Unique identifier of the Storage.

Example: "72d6900fce6b326088f5d9d91049e3e6"
attachmentsArray of objects(Attachment)

List of attachments.

resultDatastring

The result data of the job.

Example: "{'param1': 'value1'}"
metricsArray of objects(JobMetric)

List of metrics.

locksArray of objects(Lock)

List of locks.

typestring

The type of the job.

Example: "DATA_CURATION_JOB"
volumesArray of objects(Volume)

List of volumes.

namestring(JobName)

Name of a Job.

Example: "Process vendor data."
descriptionstring(JobDescription)

Detailed description of a Job.

Example: "I started this job to improve quality of our data."
organizationIdstring(OrganizationId)

Uniquely identifying ID of the organization.

Example: "cdq_monitor"
workspaceIdstring(WorkspaceId)

Uniquely identifying ID of the workspace.

Example: "c074b9f3-abf0-4f8e-9a20-74deb6cfa2a4"
mainLabelstringDeprecated

The main label of the job. Deprecated, https://cdqcom.atlassian.net/browse/CDLD-50931.

Example: "My job."
Response
application/json
{
  "commands": [
    "command"
  ],
  "startedAt": "2019-09-07T00:00:00Z",
  "createdAt": "2020-08-31T16:47+00:00",
  "modifiedAt": "2020-08-31T16:47+00:00",
  "finishedAt": "2019-09-07T00:00:00Z",
  "expiresAt": "2019-09-07T00:00:00Z",
  "dockerImageUrl": "data-curation",
  "domain": "cdqag/data-curation-job",
  "environmentVariables": [
    { … }
  ],
  "id": "35f23c03-1c22-45fe-9484-3ffe769325de",
  "inputUri": "input-uri",
  "jobDependencies": [
    { … }
  ],
  "jobStatus": "RUNNING",
  "message": "The job failed because storage is empty.",
  "mainLabel": "My job.",
  "labels": [
    { … }
  ],
  "tags": [
    "tag"
  ],
  "lastJobStatus": "RUNNING",
  "progress": "50",
  "totalItems": "100",
  "processedItems": "50",
  "failedItems": "10",
  "reportUri": "https://link-to-my-report",
  "resourceRequest": {
    "memoryLimit": 1024,
    "memoryRequest": 1024,
    "cpuLimit": 0.2,
    "cpuRequest": 0.1,
    "xmxLimit": 512
  },
  "user": "742429-234242-4343-232323",
  "storageId": "72d6900fce6b326088f5d9d91049e3e6",
  "attachments": [
    { … }
  ],
  "resultData": "{'param1': 'value1'}",
  "metrics": [
    { … }
  ],
  "locks": [
    { … }
  ],
  "type": "DATA_CURATION_JOB",
  "volumes": [
    { … }
  ],
  "name": "Process vendor data.",
  "description": "I started this job to improve quality of our data.",
  "organizationId": "cdq_monitor",
  "workspaceId": "c074b9f3-abf0-4f8e-9a20-74deb6cfa2a4"
}

Job Results

The Job Results resource represents the results of a job that has been executed. Job results can be created, updated, and deleted, and their status can be monitored and queried.

Operations

Processing logs

The Job Results resource represents the results of data monitors that has been executed. Processing log results can be created, updated, and deleted, and their status can be monitored and queried.

Operations

Scheduled Jobs

Operations