Jobs

The Jobs resource represents a job executed asynchronously. Jobs are used to perform long-running tasks that can be monitored and managed through the API. Jobs can be created, updated, and deleted, and their status can be monitored and queried.

Operations

Start Job

Request

Start a new job. The job is created with the status PENDING and is executed asynchronously.

Bodyapplication/jsonrequired

commandsArray of strings<= 100 items

List of commands to be executed.

Example: ["command"]

dockerImageUrlstring(JobDockerImageUrl)non-emptyrequired

The URL of the Docker image.

Example: "docker-image-url"

resultStorageIdstring(JobResultStorageId)

Initializes job result storage named resultStorageId. When passed, enables filtering by storageId in job listing. When using job-results-starter shared library, it is obligatory. to pass this param, that must be equal to JobResultsWriteRequest#storageId. When used only to filter jobs, then during job creation, must be set together with feature SKIP_RESULTS_INITIALIZATION.

Example: "35f23c03-1c22-45fe-9484-3ffe769325de"

domainstring

The domain of the job.

Example: "cdqag/data-curation-job"

environmentVariablesArray of objects(EnvironmentVariable)<= 100 items

Max 512 kB allowed.

jobDependenciesArray of objects(JobDependencyParam)<= 8 items

List of job dependencies.

labelsArray of objects(Label)<= 50 items

List of labels.

tagsArray of strings(JobTag)<= 15 items

List of tags.

Example: ["tag"]

typestringrequired

The type of the job.

Enum Value	Description
AML_MAINTENANCE_JOB	AML maintenance job
BANK_DOCUMENT_DATA_INGESTION_JOB	Bank document data ingestion job
BANK_TABULAR_DATA_INGESTION_JOB	Bank tabular data ingestion job
BANK_TREE_DATA_INGESTION_JOB	Bank tree data ingestion job
BANKACCOUNT_DATA_CONFIRMATION_JOB	Bankaccount data confirmation job
BANKACCOUNT_DATA_IMPORT_JOB	Bankaccount data import job
BANKACCOUNT_DATA_STATISTICS_JOB	Bankaccount data statistics job
BANKACCOUNT_DATA_WHITELIST_CURATION_JOB	Bankaccount data whitelist curation job
BANKACCOUNT_DATA_WHITELIST_IMPORT_JOB	Bankaccount data whitelist import job
BATCH_LOOKUP_JOB	Batch lookup job

Example: "BANKACCOUNT_DATA_CONFIRMATION_JOB"

memoryLimitinteger(int32)(ResourceRequestMemoryLimit)

The memory limit. The value should be in megabytes.

Example: 1024

xmxLimitinteger(int32)(ResourceRequestXmxLimit)

The xmx option in java if null then memoryLimit - 200m. The value should be in megabytes.

Example: 512

memoryRequestinteger(int32)(ResourceRequestMemoryRequest)

The memory request. The value should be in megabytes.

Example: 1024

cpuLimitnumber(float)(ResourceRequestCpuLimit)

The CPU limit. The value should be in cores.

Example: 0.2

cpuRequestnumber(float)(ResourceRequestCpuRequest)

The CPU request. The value should be in cores.

Example: 0.1

userstring(JobUser)required

ID of (human) user or API key.

Example: "742429-234242-4343-232323"

volumesArray of objects(Volume)<= 50 items

List of volumes.

locksArray of objects(Locks)<= 5 items

Enables job locking. Prevents from running duplicated jobs of the same locks set.

Checks against NOT FINISHED, CANCELED nor FAILED jobs.

When tag=DATA_MONITOR is provided together with locks, then job is locked only against another DATA_MONITOR jobs.

Example: [ {"key": "jobType", "value": "DATA_CURATION_JOB"}, {"key": "storageId", "value": "87621345678976543245678965432"}, {"key": "dataSourceIds", "value": hash(["dataSourceId1", "dataSourceId2", "dataSourceId3", "dataSourceId4", "dataSourceId5", "dataSourceId6"])} ]

expiresAtstring

ISO 8601 dates since when the job will become unreachable after the job has finished execution.

Default: 6 months.
Min: 7 days from now.
Max: 2 years from now.

Example: "2020-09-07T00:00:00Z"

namestring<= 50 characters

The name of the job.

Example: "job-name"

descriptionstring<= 200 characters

The description of the job.

Example: "job-description"

organizationIdstring(OrganizationId)

Uniquely identifying ID of the organization.

Example: "cdq_monitor"

workspaceIdstring(WorkspaceId)

Uniquely identifying ID of the workspace.

Example: "c074b9f3-abf0-4f8e-9a20-74deb6cfa2a4"

featuresArray of strings(JobCreateFeatureEnum)

List of features to be activated.

Items Enum Value	Description
FORCE_DATADOG	Force Datadog.
SKIP_RESULTS_INITIALIZATION	Skip results initialization when job is not going to use it. Temporary, until initialization is event-based.

Example: ["FORCE_DATADOG"]

mainLabelstringDeprecated

The main label of the job. Deprecated, to be removed in https://cdqcom.atlassian.net/browse/CDLD-50931.

Example: "My job."

curl -i -X POST \
  https://idp.cdq.com/_mock/apis/jobs-management-api/api-v2/v2/jobs \
  -H 'Content-Type: application/json' \
  -d '{
    "commands": [
      "command"
    ],
    "dockerImageUrl": "docker-image-url",
    "resultStorageId": "35f23c03-1c22-45fe-9484-3ffe769325de",
    "domain": "cdqag/data-curation-job",
    "environmentVariables": [
      {
        "key": "STORAGE_ID",
        "value": "123456789"
      }
    ],
    "jobDependencies": [
      {
        "id": "35f43c33-1c22-42fe-9984-3ffe750325de",
        "strategy": "SCHEDULE_NEXT_WHEN_FINISHED"
      }
    ],
    "mainLabel": "My job.",
    "labels": [
      {
        "key": "my label",
        "value": "my job"
      }
    ],
    "tags": [
      "tag"
    ],
    "type": "BANKACCOUNT_DATA_CONFIRMATION_JOB",
    "memoryLimit": 1024,
    "xmxLimit": 512,
    "memoryRequest": 1024,
    "cpuLimit": 0.2,
    "cpuRequest": 0.1,
    "user": "742429-234242-4343-232323",
    "volumes": [
      {
        "mountPath": "/path",
        "subPath": "subpath"
      }
    ],
    "locks": [
      {
        "key": "jobType",
        "value": "SUBSCRIPTION_FAT_JOB"
      }
    ],
    "expiresAt": "2020-09-07T00:00:00Z",
    "name": "job-name",
    "description": "job-description",
    "organizationId": "cdq_monitor",
    "workspaceId": "c074b9f3-abf0-4f8e-9a20-74deb6cfa2a4",
    "features": [
      "FORCE_DATADOG"
    ]
  }'

Responses

OK

Bodyapplication/json

commandsArray of strings

List of commands to be executed.

Example: ["command"]

startedAtstring

The time when the job was started.

Example: "2019-09-07T00:00:00Z"

createdAtstring(CreatedAt)

Date of creation (ISO 8601-compliant).

Example: "2020-08-31T16:47+00:00"

modifiedAtstring(ModifiedAt)

Date of modification (ISO 8601-compliant).

Example: "2020-08-31T16:47+00:00"

finishedAtstring

The time when the job was finished.

Example: "2019-09-07T00:00:00Z"

expiresAtstring

The time when the job expires.

Example: "2019-09-07T00:00:00Z"

dockerImageUrlstring

The URL of the Docker image.

Example: "data-curation"

domainstring

The domain of the job.

Example: "cdqag/data-curation-job"

environmentVariablesArray of objects(EnvironmentVariable)

Max 512 kB allowed.

idstring(JobId)

Unique identifier of a job.

Example: "35f23c03-1c22-45fe-9484-3ffe769325de"

inputUristring

The URI of the input.

Example: "input-uri"

jobDependenciesArray of objects(JobDependency)

List of job dependencies.

jobStatusstring(JobStatus)

Job execution status.

Enum Value	Description
UNKNOWN	Job becomes in unknown status.
CREATED	Job has been created.
PERSISTED	Job metadata has been persisted.
SCHEDULED	Job has been scheduled for execution.
WAITING	Job is waiting for being scheduled.
RUNNING	Job is being executed.
FINISHED	Job has finished.
DIED	Job was scheduled and started running but died unexpectedly.
CANCELED	Job has been canceled due to invalid input configuration.
FAILED	Job has failed.

Example: "RUNNING"

messagestring(JobStatusMessage)

Additional information to explain the status.

Example: "The job failed because storage is empty."

labelsArray of objects(Label)

List of labels.

tagsArray of strings(JobTagResult)

List of tags.

Example: ["tag"]

lastJobStatusstring(JobStatus)

Job execution status.

Enum Value	Description
UNKNOWN	Job becomes in unknown status.
CREATED	Job has been created.
PERSISTED	Job metadata has been persisted.
SCHEDULED	Job has been scheduled for execution.
WAITING	Job is waiting for being scheduled.
RUNNING	Job is being executed.
FINISHED	Job has finished.
DIED	Job was scheduled and started running but died unexpectedly.
CANCELED	Job has been canceled due to invalid input configuration.
FAILED	Job has failed.

Example: "RUNNING"

progressinteger(int32)

The progress of the job.

Example: "50"

totalItemsinteger(int64)

The total number of items.

Example: "100"

processedItemsinteger(int64)

The number of processed items.

Example: "50"

failedItemsinteger(int64)

The number of failed items.

Example: "10"

reportUristring

The URI of the report.

Example: "https://link-to-my-report"

resourceRequestobject(ResourceRequest)

Resource request.

userstring(JobUser)

ID of (human) user or API key.

Example: "742429-234242-4343-232323"

storageIdstring(BusinessPartnerStorageId)

Unique identifier of the Storage.

Example: "72d6900fce6b326088f5d9d91049e3e6"

attachmentsArray of objects(Attachment)

List of attachments.

resultDatastring

The result data of the job.

Example: "{'param1': 'value1'}"

metricsArray of objects(JobMetric)

List of metrics.

locksArray of objects(Lock)

List of locks.

typestring

The type of the job.

Example: "DATA_CURATION_JOB"

volumesArray of objects(Volume)

List of volumes.

namestring(JobName)

Name of a Job.

Example: "Process vendor data."

descriptionstring(JobDescription)

Detailed description of a Job.

Example: "I started this job to improve quality of our data."

organizationIdstring(OrganizationId)

Uniquely identifying ID of the organization.

Example: "cdq_monitor"

workspaceIdstring(WorkspaceId)

Uniquely identifying ID of the workspace.

Example: "c074b9f3-abf0-4f8e-9a20-74deb6cfa2a4"

mainLabelstringDeprecated

The main label of the job. Deprecated, https://cdqcom.atlassian.net/browse/CDLD-50931.

Example: "My job."

Response

application/json

{
  "commands": [
    "command"
  ],
  "startedAt": "2019-09-07T00:00:00Z",
  "createdAt": "2020-08-31T16:47+00:00",
  "modifiedAt": "2020-08-31T16:47+00:00",
  "finishedAt": "2019-09-07T00:00:00Z",
  "expiresAt": "2019-09-07T00:00:00Z",
  "dockerImageUrl": "data-curation",
  "domain": "cdqag/data-curation-job",
  "environmentVariables": [
    { … }
  ],
  "id": "35f23c03-1c22-45fe-9484-3ffe769325de",
  "inputUri": "input-uri",
  "jobDependencies": [
    { … }
  ],
  "jobStatus": "RUNNING",
  "message": "The job failed because storage is empty.",
  "mainLabel": "My job.",
  "labels": [
    { … }
  ],
  "tags": [
    "tag"
  ],
  "lastJobStatus": "RUNNING",
  "progress": "50",
  "totalItems": "100",
  "processedItems": "50",
  "failedItems": "10",
  "reportUri": "https://link-to-my-report",
  "resourceRequest": {
    "memoryLimit": 1024,
    "memoryRequest": 1024,
    "cpuLimit": 0.2,
    "cpuRequest": 0.1,
    "xmxLimit": 512
  },
  "user": "742429-234242-4343-232323",
  "storageId": "72d6900fce6b326088f5d9d91049e3e6",
  "attachments": [
    { … }
  ],
  "resultData": "{'param1': 'value1'}",
  "metrics": [
    { … }
  ],
  "locks": [
    { … }
  ],
  "type": "DATA_CURATION_JOB",
  "volumes": [
    { … }
  ],
  "name": "Process vendor data.",
  "description": "I started this job to improve quality of our data.",
  "organizationId": "cdq_monitor",
  "workspaceId": "c074b9f3-abf0-4f8e-9a20-74deb6cfa2a4"
}

List Jobs

Request

List jobs that meet the querying criteria.

Query

userstring(UserId)

The user that started the job.

Example: user=johndoe

organizationIdstring(OrganizationId)

The organizationId parameter set in the job.

Example: organizationId=cdq_monitor

storageIdstring(BusinessPartnerStorageId)

The storageId parameter set in the job.

Example: storageId=72d6900fce6b326088f5d9d91049e3e6

jobTypeArray of strings<= 100 items

The kind of job described as JobType.

Example: jobType=DATA_CURATION_JOB, DATA_VALIDATION_JOB

sortstring

Define the attributes to sort by. Supported are in (createdAt), separated by comma (,). Sort "+" means ascending, "-" means descending. Default sort (when no + or - set) is ascending.

Example: sort=-createdAt

jobStatusArray of strings(JobStatus)<= 50 items

The kind of job described as JobStatus.

Items Enum Value	Description
UNKNOWN	Job becomes in unknown status.
CREATED	Job has been created.
PERSISTED	Job metadata has been persisted.
SCHEDULED	Job has been scheduled for execution.
WAITING	Job is waiting for being scheduled.
RUNNING	Job is being executed.
FINISHED	Job has finished.
DIED	Job was scheduled and started running but died unexpectedly.
CANCELED	Job has been canceled due to invalid input configuration.
FAILED	Job has failed.

Example: jobStatus=RUNNING

startAfterstring(StartAfter)

Indicator for the next page.

Example: startAfter=5712566172571652

limitinteger(int32)[ 1 .. 100 ]

Number of results that should be fetched. Maximum 100 results can be returned in one page.

Default 50

Example: limit=50

tagArray of strings(JobTagResult)<= 100 items

Filter jobs by tag.

Example: tag=tag

tagsConjunctionboolean

Treats tags as AND operation. By default, tags are treated as OR operation.

Default false

Example: tagsConjunction=false

featuresOnArray of strings(JobsReadFeatureParam)

Enable additional features when fetching jobs. Supported features:

NUMBER_OF_TOTAL - sends in response number of total jobs meeting the querying criteria
TAGS_CONJUNCTION - Treats tags as AND operation. By default, tags are treated as OR operation.
IGNORE_ORGANIZATION_RESTRICTIONS - allows to read jobs from any organization.

Items Enum Value	Description
NUMBER_OF_TOTAL	Number of totals.
TAGS_CONJUNCTION	Tags conjunction.
IGNORE_ORGANIZATION_RESTRICTIONS	Ignore organization restrictions.

Example: featuresOn=NUMBER_OF_TOTAL

dependentJobIdstring(JobId)

The ID of a job that it depends on. Requires storageId and jobType parameters to be provided.

Example: dependentJobId=35f23c03-1c22-45fe-9484-3ffe769325de

curl -i -X GET \
  https://idp.cdq.com/_mock/apis/jobs-management-api/api-v2/v2/jobs

Responses

OK

Bodyapplication/json

jobsArray of objects(Job)

List of jobs.

nextStartAfterstring(NextStartAfter)

Provides a value to be used as a startAfter in next page request.

Example: "5712566172571652"

totalinteger(int64)

Total number of items, which can be paged.

Example: "67"

Response

application/json

{
  "jobs": [
    { … }
  ],
  "nextStartAfter": "5712566172571652",
  "total": "67"
}

List Job Types

Request

List all available job types.

Query

featureOnArray of strings(JobTypeFeatureParam)

Features to select subset of Job Types.

Items Value	Description
CANCELLABLE	Job can be canceled.

Example: featureOn=CANCELLABLE

startAfterstring(StartAfter)

Indicator for the next page.

Example: startAfter=5712566172571652

limitinteger(int32)[ 1 .. 100 ]

Number of results that should be fetched. Maximum 100 results can be returned in one page.

Default 50

Example: limit=50

curl -i -X GET \
  https://idp.cdq.com/_mock/apis/jobs-management-api/api-v2/v2/jobs/types

Responses

OK

Bodyapplication/json

jobsArray of objects(JobTypeSearchResult)

List of job types.

nextStartAfterstring(NextStartAfter)

Provides a value to be used as a startAfter in next page request.

Example: "5712566172571652"

Response

application/json

{
  "jobs": [
    { … }
  ],
  "nextStartAfter": "5712566172571652"
}

Job Results

The Job Results resource represents the results of a job that has been executed. Job results can be created, updated, and deleted, and their status can be monitored and queried.

Operations

Processing logs

The Job Results resource represents the results of data monitors that has been executed. Processing log results can be created, updated, and deleted, and their status can be monitored and queried.

Operations

Scheduled Jobs

Operations

Jobs Management API (2)

Jobs

Start Job

Request

Responses

Share the feedback!

List Jobs

Request

Responses

Share the feedback!

List Job Types

Request

Responses

Share the feedback!

Job Results

Processing logs

Scheduled Jobs

Jobs Management API (2)

Jobs

Start Job

RequestExpand all

Responses

Share the feedback!

List Jobs

Request

Responses

Share the feedback!

List Job Types

Request

Responses

Share the feedback!

Job Results

Processing logs

Scheduled Jobs

Request