Jobs Management API (2)

The Job Management API allows for the management and execution of asynchronous jobs within CDQ services. This API enables the creation, monitoring, and management of jobs, allowing developers to automate and streamline complex workflows and tasks.

Servers

Mock server

https://idp.cdq.com/_mock/apis/jobs-management-api/api-v2/

Production SOAP

https://job-management-api.reverse-proxy.prod.k8s.production.cdq-cloud-engine.com/soap/v2/

Production

https://job-management-api.reverse-proxy.prod.k8s.production.cdq-cloud-engine.com/

Jobs

The Jobs resource represents a job executed asynchronously. Jobs are used to perform long-running tasks that can be monitored and managed through the API. Jobs can be created, updated, and deleted, and their status can be monitored and queried.

Operations

Update Result Data

Request

Update the result data of a job.

Path

jobIdstring(JobId)required

Unique identifier of the job.

Example: 35f23c03-1c22-45fe-9484-3ffe769325de

Bodyapplication/jsonrequired

cmd

resultDatastring<= 65535 characters

Stringified JSON representation of the job result created for the record identified by the businessPartnerId and the external ID.

Example: "{'param1': 'value1'}"

curl -i -X PUT \
  https://idp.cdq.com/_mock/apis/jobs-management-api/api-v2/jobs/35f23c03-1c22-45fe-9484-3ffe769325de/resultdata \
  -H 'Content-Type: application/json' \
  -d '{
    "resultData": "{'\''param1'\'': '\''value1'\''}"
  }'

Responses

Bodyapplication/json

commandsArray of strings

List of commands to be executed.

Example: ["command"]

startedAtstring

The time when the job was started.

Example: "2019-09-07T00:00:00Z"

createdAtstring(CreatedAt)

Date of creation (ISO 8601-compliant).

Example: "2020-08-31T16:47+00:00"

modifiedAtstring(ModifiedAt)

Date of modification (ISO 8601-compliant).

Example: "2020-08-31T16:47+00:00"

finishedAtstring

The time when the job was finished.

Example: "2019-09-07T00:00:00Z"

expiresAtstring

The time when the job expires.

Example: "2019-09-07T00:00:00Z"

dockerImageUrlstring

The URL of the Docker image.

Example: "data-curation"

domainstring

The domain of the job.

Example: "cdqag/data-curation-job"

environmentVariablesArray of objects(EnvironmentVariable)

Max 512 kB allowed.

idstring(JobId)

Unique identifier of a job.

Example: "35f23c03-1c22-45fe-9484-3ffe769325de"

inputUristring

The URI of the input.

Example: "input-uri"

jobDependenciesArray of objects(JobDependency)

List of job dependencies.

jobStatusstring(JobStatus)

Job execution status.

Enum Value	Description
UNKNOWN	Job becomes in unknown status.
CREATED	Job has been created.
PERSISTED	Job metadata has been persisted.
SCHEDULED	Job has been scheduled for execution.
WAITING	Job is waiting for being scheduled.
RUNNING	Job is being executed.
FINISHED	Job has finished.
DIED	Job was scheduled and started running but died unexpectedly.
CANCELED	Job has been canceled due to invalid input configuration.
FAILED	Job has failed.

Example: "RUNNING"

messagestring(JobStatusMessage)

Additional information to explain the status.

Example: "The job failed because storage is empty."

labelsArray of objects(Label)

List of labels.

tagsArray of strings(JobTagResult)

List of tags.

Example: ["tag"]

lastJobStatusstring(JobStatus)

Job execution status.

Enum Value	Description
UNKNOWN	Job becomes in unknown status.
CREATED	Job has been created.
PERSISTED	Job metadata has been persisted.
SCHEDULED	Job has been scheduled for execution.
WAITING	Job is waiting for being scheduled.
RUNNING	Job is being executed.
FINISHED	Job has finished.
DIED	Job was scheduled and started running but died unexpectedly.
CANCELED	Job has been canceled due to invalid input configuration.
FAILED	Job has failed.

Example: "RUNNING"

progressinteger(int32)

The progress of the job.

Example: "50"

totalItemsinteger(int64)

The total number of items.

Example: "100"

processedItemsinteger(int64)

The number of processed items.

Example: "50"

failedItemsinteger(int64)

The number of failed items.

Example: "10"

reportUristring

The URI of the report.

Example: "https://link-to-my-report"

resourceRequestobject(ResourceRequest)

Resource request.

userstring(JobUser)

ID of (human) user or API key.

Example: "742429-234242-4343-232323"

storageIdstring(BusinessPartnerStorageId)

Unique identifier of the Storage.

Example: "72d6900fce6b326088f5d9d91049e3e6"

attachmentsArray of objects(Attachment)

List of attachments.

resultDatastring

The result data of the job.

Example: "{'param1': 'value1'}"

metricsArray of objects(JobMetric)

List of metrics.

locksArray of objects(Lock)

List of locks.

typestring

The type of the job.

Example: "DATA_CURATION_JOB"

volumesArray of objects(Volume)

List of volumes.

namestring(JobName)

Name of a Job.

Example: "Process vendor data."

descriptionstring(JobDescription)

Detailed description of a Job.

Example: "I started this job to improve quality of our data."

organizationIdstring(OrganizationId)

Uniquely identifying ID of the organization.

Example: "cdq_monitor"

workspaceIdstring(WorkspaceId)

Uniquely identifying ID of the workspace.

Example: "c074b9f3-abf0-4f8e-9a20-74deb6cfa2a4"

mainLabelstringDeprecated

The main label of the job. Deprecated, https://cdqcom.atlassian.net/browse/CDLD-50931.

Example: "My job."

Response

application/json

{
  "commands": [
    "command"
  ],
  "startedAt": "2019-09-07T00:00:00Z",
  "createdAt": "2020-08-31T16:47+00:00",
  "modifiedAt": "2020-08-31T16:47+00:00",
  "finishedAt": "2019-09-07T00:00:00Z",
  "expiresAt": "2019-09-07T00:00:00Z",
  "dockerImageUrl": "data-curation",
  "domain": "cdqag/data-curation-job",
  "environmentVariables": [
    { … }
  ],
  "id": "35f23c03-1c22-45fe-9484-3ffe769325de",
  "inputUri": "input-uri",
  "jobDependencies": [
    { … }
  ],
  "jobStatus": "RUNNING",
  "message": "The job failed because storage is empty.",
  "mainLabel": "My job.",
  "labels": [
    { … }
  ],
  "tags": [
    "tag"
  ],
  "lastJobStatus": "RUNNING",
  "progress": "50",
  "totalItems": "100",
  "processedItems": "50",
  "failedItems": "10",
  "reportUri": "https://link-to-my-report",
  "resourceRequest": {
    "memoryLimit": 1024,
    "memoryRequest": 1024,
    "cpuLimit": 0.2,
    "cpuRequest": 0.1,
    "xmxLimit": 512
  },
  "user": "742429-234242-4343-232323",
  "storageId": "72d6900fce6b326088f5d9d91049e3e6",
  "attachments": [
    { … }
  ],
  "resultData": "{'param1': 'value1'}",
  "metrics": [
    { … }
  ],
  "locks": [
    { … }
  ],
  "type": "DATA_CURATION_JOB",
  "volumes": [
    { … }
  ],
  "name": "Process vendor data.",
  "description": "I started this job to improve quality of our data.",
  "organizationId": "cdq_monitor",
  "workspaceId": "c074b9f3-abf0-4f8e-9a20-74deb6cfa2a4"
}

Fetch Last Processed ID.

Request

Fetch the last processed ID of a job.

Bodyapplication/jsonrequired

jobTypestring

The type of job.

Example: "DATA_CURATION_JOB"

storageIdstring(BusinessPartnerStorageId)

Unique identifier of the Storage.

Example: "72d6900fce6b326088f5d9d91049e3e6"

curl -i -X POST \
  https://idp.cdq.com/_mock/apis/jobs-management-api/api-v2/jobs/results/fetchLast \
  -H 'Content-Type: application/json' \
  -d '{
    "jobType": "DATA_CURATION_JOB",
    "storageId": "72d6900fce6b326088f5d9d91049e3e6"
  }'

Responses

Bodyapplication/json

idstring

ID of the Job Result,

Example: "5daef6fe65bafb00012901e6"

jobIdstring(JobId)

Unique identifier of a job.

Example: "35f23c03-1c22-45fe-9484-3ffe769325de"

dataSourceIdstring(BusinessPartnerStorageDataSourceId)

Unique identifier for a Data Source of the Storage.

Example: "648824a691d8d2503d65103e"

businessPartnerIdstring(BusinessPartnerId)

A CDQ ID identifies a business partner uniquely in the context of the Corporate Data League.

Example: "63e635235c06b7396330fe40"

addressIdstring(AddressId)

ID of the address in the storage or data mirror.

Example: "address-id"

externalIdstring(BusinessPartnerExternalId)

Arbitrary identifier type to mark customer IDs that are "external" from CDQ's perspective. This is the identifier a customer provides to identify its records.

Example: "The ID managed in the customer's SAP systems."

addressExternalIdstring(AddressExternalId)

ID the address has in the external system where the record originates from.

Example: "1"

addressVersionobject(AddressVersion)

CDQ supports multiple versions of addresses comprising different charsets and languages. An address version is a unique combination of a character set and a language.

jobTypestring

The type of job.

Example: "DATA_CURATION_JOB"

resultDatastring

Stringified JSON representation of the job result created for the record identified by the businessPartnerId and the external ID.

Example: "{'param1': 'value1'}"

statusstring(ResultDataStatus)

Can be OK or FAILED - default is OK.

Default "OK"

Example: "OK"

messagestring

Additional explanations for status - especially for the FAILED case.

Example: "Missing storageId."

countryShortNamestring(CountryShortName)

Country code (ISO 3166-1 alpha-2).

Example: "CH"

executedAtstring(ExecutedAt)

ISO 8601 representation of the timestamp when the result data was created.

Example: "2019-09-07T00:00:00Z"

Response

application/json

{
  "id": "5daef6fe65bafb00012901e6",
  "jobId": "35f23c03-1c22-45fe-9484-3ffe769325de",
  "dataSourceId": "648824a691d8d2503d65103e",
  "businessPartnerId": "63e635235c06b7396330fe40",
  "addressId": "address-id",
  "externalId": "The ID managed in the customer's SAP systems.",
  "addressExternalId": "1",
  "addressVersion": {
    "characterSet": { … },
    "language": { … }
  },
  "jobType": "DATA_CURATION_JOB",
  "resultData": "{'param1': 'value1'}",
  "status": "OK",
  "message": "Missing storageId.",
  "countryShortName": "CH",
  "executedAt": "2019-09-07T00:00:00Z"
}

Job Results

The Job Results resource represents the results of a job that has been executed. Job results can be created, updated, and deleted, and their status can be monitored and queried.

Operations

Processing logs

The Job Results resource represents the results of data monitors that has been executed. Processing log results can be created, updated, and deleted, and their status can be monitored and queried.

Operations

Scheduled Jobs

Operations

Jobs Management API (2)

Jobs

Update Result Data

Request

Responses

Share the feedback!

Fetch Last Processed ID.

Request

Responses

Share the feedback!

Job Results

Processing logs

Scheduled Jobs