De-identifying DICOM Files

DICOM files may contain sensitive and personally identifiable information (PII) about patients, including their name, date of birth, medical record number. In these cases it is essential to anonymize each file, to protect patient privacy and comply with legal and ethical regulations related to healthcare data.

This is a premium feature and running it incurs additional costs. Contact us at support@encord.com to learn about pricing.

In this tutorial you will learn how to anonymize / de-identify DICOM files in two steps:

Setting up the de-identification function.
Using the de-identification function.

Finally, you will learn how to interpret the JSON output of the de-identification process.

Redacted tag values are completely removed.

Set up the de-identification function

Adjust the de-identification function found here to suit your needs.

De-identifying

The Python code below is used to add criteria and call the de-identification function.

An SSH public / private key pair is required to use the sample script below. To learn how to generate one, see our documentation here.

Make sure to edit the criteria used to evaluate each file to suit your needs - any number of criteria can be used.

import multiprocessing
import time
from pathlib import Path
from typing import List
import glob
import os
from encord import EncordUserClient
from encord.objects.common import (
    DeidentifyRedactTextMode,
    SaveDeidentifiedDicomConditionNotSubstr,
    SaveDeidentifiedDicomConditionIn
)

# Replace s3://EXAMPLE-BUCKET/raw/ with the path to the file storage you're using 
def filelist_helper(dir,prefix='s3://EXAMPLE-BUCKET/raw/'):
    fl = [prefix+os.path.basename(f) for f in glob.glob(dir+"*.dcm")]
    return fl

# Add criteria to evaluate each file. See the 'Setting evaluation criteria' section below for more info. 
criteria = [
    SaveDeidentifiedDicomConditionNotSubstr("PRIMARY","ImageType"),              
    SaveDeidentifiedDicomConditionIn(["ct","pt","nm","mr","mg","pt"],"Modality") 
]

# Authenticate with Encord using the path to your private key
user_client = EncordUserClient.create_with_ssh_private_key(
    ssh_private_key_path="<private_key_path>",
    # For US platform users use "https://api.us.encord.com"
    domain="https://api.encord.com",
)

def deidentify(
    integration_title: str,
    dicom_urls: List[str],
) -> List[str]:

    integration_hash = None

    # Find integration_hash for requested integration_title
    for integration in user_client.get_cloud_integrations():
        if integration.title == integration_title:
            integration_hash = integration.id

    if not integration_hash:
        raise Exception(f"Integration with integration_title={integration_title} not found")

    deidentified_dicom_urls = []

    # 'dicom_urls' should be a a single list containing the URLs of all instances of a series to be de-identified. Splitting a series into multiple lists might lead to inaccurate results and is therefore not recommended
    deidentified_dicom_url = user_client.deidentify_dicom_files(
        dicom_urls=dicom_urls,
        integration_hash=integration_hash,
        redact_dicom_tags = True,
        redact_pixels_mode = DeidentifyRedactTextMode.REDACT_ALL_TEXT,
        save_conditions = criteria,
        upload_dir = "s3://EXAMPLE-BUCKETt/output"
    )
    print(f"Deidentified url: {deidentified_dicom_url}")
    deidentified_dicom_urls += deidentified_dicom_url

    return deidentified_dicom_urls

# Replace MY-ENCORD-INTEGRATION with the title of your private cloud integration
_integration_title = "MY-ENCORD-INTEGRATION"

# Replace <path-to-local-dicom-folder>/ with the local folder that mirrors the DICOM files in your cloud storage
_dicom_urls = filelist_helper("<path-to-local-dicom-folder>/")

_deidentified_dicom_urls = deidentify(
    _integration_title,
    _dicom_urls,
)

De-identify without blocking your script

deidentify_dicom_files() waits until the de-identification job finishes. To start a job and collect the results later, use deidentify_dicom_files_start(), which returns a job UUID, and pass that UUID to deidentify_dicom_files_get_result(). Both methods take the same arguments as the script above.

from encord import EncordUserClient
from encord.objects.common import DeidentifyRedactTextMode

# Authenticate with Encord using the path to your private key
user_client = EncordUserClient.create_with_ssh_private_key(
    ssh_private_key_path="<private_key_path>",
    # For US platform users use "https://api.us.encord.com"
    domain="https://api.encord.com",
)

# Replace MY-ENCORD-INTEGRATION with the title of your private cloud integration
integration = user_client.get_cloud_integrations(filter_integration_titles=["MY-ENCORD-INTEGRATION"])[0]

# Start the de-identification job. This returns immediately with the job UUID.
job_uuid = user_client.deidentify_dicom_files_start(
    dicom_urls=["s3://EXAMPLE-BUCKET/raw/file-1.dcm", "s3://EXAMPLE-BUCKET/raw/file-2.dcm"],
    integration_hash=integration.id,
    redact_dicom_tags=True,
    redact_pixels_mode=DeidentifyRedactTextMode.REDACT_ALL_TEXT,
    upload_dir="s3://EXAMPLE-BUCKET/output",
)
print(f"Started de-identification job {job_uuid}")

# ... do other work ...

# Wait for the job to finish (up to 1 hour here) and get the URLs of the de-identified files
deidentified_dicom_urls = user_client.deidentify_dicom_files_get_result(job_uuid, timeout_seconds=60 * 60)
print(deidentified_dicom_urls)

Setting evaluation criteria

Evaluation criteria are conditions that determine whether a file will be de-identified or not. Criteria can take many forms, but will always return either ‘true’ or ‘false’.

All strings and inputs into the criteria functions are case-insensitive.

There are two distinct criteria functions:

“SaveDeidentifiedDicomConditionNotSubstr” will return ‘true’ if the first argument (PRIMARY in the example above), is not contained in the second argument (ImageType in the example above). In plain English, the example above checks whether the file’s ImageType doesn’t contain the word ‘Primary’, and returns ‘true’ if this condition is fulfilled.
“SaveDeidentifiedDicomConditionIn” will return ‘true’ if the first argument ([“ct”,“pt”,“nm”,“mr”,“mg”,“pt”] in the example above) is contained in the second argument Modality. In plain English, the example above checks whether any of the strings contained in the list are contained within the file’s Modality. If any one of them is, the function returns ‘true’.

Output

This section will explain the output of the de-identification function. Click the dropdown below to see a sample output file

Sample JSON

{
"url": "s3://EXAMPLE-BUCKETt/raw/should_fail_1.2.840.113619.2.283.6945.3146400.16119.1391477777.833.dcm",
"StudyInstanceUID": "1.2.840.113619.6.283.4.679947340.8065.1391798290.307",
"StudyInstanceUID_deid": "1.2.840.113619.6.283.4.679947340.8065.1391798290.307",
"SeriesInstanceUID": "1.2.840.113619.2.283.6945.3146400.21673.1391477673.97",
"SeriesInstanceUID_deid": "1.2.840.113619.2.283.6945.3146400.21673.1391477673.97",
"save_conditions_evaluations":
[
    {
        "condition":
        {
            "value": "PRIMARY",
            "condition_type": "NOT_SUBSTR",
            "dicom_tag": "ImageType"
        },
        "tag_value": "DEMOGRAPHICDATA",
        "evaluated_value": false
    },
    {
        "condition":
        {
            "value":
            [
                "ct",
                "pt",
                "nm",
                "mr",
                "xr",
                "mg"
            ],
            "condition_type": "IN",
            "dicom_tag": "Modality"
        },
        "tag_value": "MR",
        "evaluated_value": true
    }
],
"save_condition": false,
"save_condition_series_agg": false,
"save_disabling_urls_series_agg":
[
    "s3://EXAMPLE-BUCKETt/raw/should_fail_1.2.840.113619.2.283.6945.3146400.16119.1391477777.833.dcm"
],
"url_deid": "s3://EXAMPLE-BUCKETt/Pixel-redaction-ready/deid_169089614143326592_should_fail_1.2.840.113619.2.283.6945.3146400.16119.1391477777.833.dcm"
}

Key	Description	Notes
`url`	A URL to the DICOM file
`StudyInstanceUID`	The study’s ID
`StudyInstanceUID_deid`	A converted version of the study’s ID used by Encord	Identical to `StudyInstanceUID`, unless `StudyInstanceUID` was invalid
`SeriesInstanceUID`	The series’ ID
`SeriesInstanceUID_deid`	A converted version of the series’ ID used by Encord	Identical to `SeriesInstanceUID`, unless `SeriesInstanceUID` was invalid
`save_conditions_evaluations`	Contains a list of conditions	These need to be met in order for a file to be de-identified
`condition`	Contains a given condition’s details	A `condition` is satisfied when the `value` fulfills the `condition_type`
`value`	The elements being evaluated by the `condition_type`	Can be thought of as the ‘answer’ to a `condition`
`condition_type`	The type of condition being evaluated	Can be thought of as the ‘question’ to a `condition`
`dicom_tag`	The DICOM tag on which a condition is being evaluated
`save_condition`	Does the condition have to be `true` or `false` for the file to pass
`save_condition_series_agg`	The series as a whole evaluated as `true` or `false`	For this to be `true` all `save_condition`s need to be `true` For this to be `false` all `save_conditions`s need to be `false`
`save_disabling_urls_series_agg`	A list of files which did not meet the required `save_condition`	Only relevant if the series didn’t pass
`url_deid`	The URL of the new, de-identified file.

Get Started

General

Data

Ontologies

Projects

Labels

DICOM

De-identifying DICOM Files

Set up the de-identification function

De-identifying

De-identify without blocking your script

Setting evaluation criteria

Output

​Set up the de-identification function

​De-identifying

​De-identify without blocking your script

​Setting evaluation criteria

​Output

Set up the de-identification function

De-identifying

De-identify without blocking your script

Setting evaluation criteria

Output