Skip to content

[redcap] bulk importer #9808

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 30 commits into
base: 27.0-release
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
30 commits
Select commit Hold shift + click to select a range
1a6dc39
redcap config - load all config file
regisoc May 23, 2025
df7d062
redcap tool - bulk importer - draft
regisoc May 23, 2025
a7842ea
lint
regisoc May 23, 2025
f28abd7
lint
regisoc May 23, 2025
0be46d1
lint
regisoc May 23, 2025
c49657a
generic include path
regisoc May 23, 2025
7016b29
arg parse
regisoc May 23, 2025
b152c5a
candid str
regisoc May 23, 2025
bfaf8cb
db select query iterator to array
regisoc May 23, 2025
046d2d7
array to query object
regisoc May 23, 2025
9cc984e
lint
regisoc May 23, 2025
1fc591b
mv tools to redcap module
regisoc May 27, 2025
6e31301
query - loris instrument/commentIDs to be imported
regisoc May 28, 2025
a6b334a
queries
regisoc May 28, 2025
0db9994
loris instrument to import to queries
regisoc May 28, 2025
f03fb51
trigger notifications fn
regisoc May 28, 2025
1b662f7
rm override mention
regisoc May 28, 2025
e95a7e8
url path
regisoc May 29, 2025
96d052b
guzzle exception mgmt
regisoc May 29, 2025
549024b
lint
regisoc Jul 9, 2025
44a3dda
queries method/parameters renaming
regisoc Jul 9, 2025
4f6d65c
renaming
regisoc Jul 9, 2025
6a6f500
importer - docstring types
regisoc Jul 9, 2025
e1fcb3a
lint - docstring test
regisoc Jul 9, 2025
72f4ca1
lint - docstring test
regisoc Jul 9, 2025
3ed3cc2
multiline docblock longtype test
regisoc Jul 10, 2025
7fcc61f
multiline docblock longtype simplified types
regisoc Jul 10, 2025
be17705
phan neon redcap update
regisoc Jul 10, 2025
07e09c7
phan neon redcap update
regisoc Jul 10, 2025
50176a5
phan neon redcap update
regisoc Jul 10, 2025
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
81 changes: 81 additions & 0 deletions modules/redcap/php/config/redcapconfigparser.class.inc
Original file line number Diff line number Diff line change
Expand Up @@ -448,4 +448,85 @@ class RedcapConfigParser
. " (redcap-project-id: {$this->_redcap_project_id}): $message"
);
}

/**
* Get the full REDCap configuration as described in 'config.xml' file.
* Returns a struct ordered by REDCap instance and project.
* Expected structure is:
* [REDCap instance URL => [REDCap Project ID => REDCapConfig object]]
* else null if no configurations are found.
*
* @param \LORIS\LorisInstance $loris the loris instance
*
* @throws \LorisException
*
* @return array<mixed|RedcapConfig|null>[] a REDCap configuration with the form
* [REDCap instance URL => [REDCap Project ID => REDCapConfig object]]
* else null
Comment on lines +463 to +465
Copy link
Contributor

@MaximeBICMTL MaximeBICMTL Jun 8, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This return type is too complex IMO. I would prefer to return a simple list of RedcapConfig. The caller can handle special cases like the empty list itself, and the instance URLs and project IDs are available in each RedcapConfig if needed.

*/
public static function getConfiguration(
\LORIS\LorisInstance $loris
): array {
// final REDCap configuration struct
$redcapConfig = [];

// list of REDCap instances and projects
$rawRedcapConfig = $loris->getConfiguration()->getSetting('redcap');
if ($rawRedcapConfig === null || empty($rawRedcapConfig)) {
throw new \LorisException("No REDCap configuration.");
}

// instances, must at least have one
$redcapInstances = $rawRedcapConfig['instance'] ?? null;
if ($redcapInstances === null || empty($redcapInstances)) {
throw new \LorisException(
"No REDCap instance defined in configuration file."
);
}

// only one instance, wrap it in an array to be able to iterate
if (!array_key_exists(0, $redcapInstances)) {
$redcapInstances = [$redcapInstances];
}

// going throught REDCap defined instances
foreach ($redcapInstances as $instanceStruct) {
// instance URL
$redcapInstanceURL = $instanceStruct['redcap-url'];

// project of this instance
$redcapProjects = $instanceStruct['project'] ?? null;
if ($redcapProjects === null || empty($redcapProjects)) {
// no project for that instance, next REDCap instance
continue;
}

// only one project, wrap it in an array to be able to iterate
if (!array_key_exists(0, $redcapProjects)) {
$redcapProjects = [$redcapProjects];
}

// add REDCap configuration
foreach ($redcapProjects as $redcapProject) {
// project ID
$redcapProjectID = $redcapProject['redcap-project-id'];

// parser object
$configParser = new RedcapConfigParser(
$loris,
$redcapInstanceURL,
$redcapProjectID
);
Comment on lines +515 to +519
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ideally I guess I should refactor the parser to more elegantly handle the "get all project configurations" and "get single project configuration" cases, but this is out of scope for this PR, your code is fine 👍


// add the parsed REDCap configuration to the final struct
$redcapConfig[$redcapInstanceURL] = [
...$redcapConfig[$redcapInstanceURL] ?? [],
$redcapProjectID => $configParser->parse()
];
}
}

// return struct
return empty($redcapConfig) ? null : $redcapConfig;
}
}
62 changes: 62 additions & 0 deletions modules/redcap/php/queries.class.inc
Original file line number Diff line number Diff line change
Expand Up @@ -16,6 +16,7 @@
namespace LORIS\redcap;

use \LORIS\LorisInstance;
use \LORIS\Database\Query;
use \LORIS\StudyEntities\Candidate\CandID;

/**
Expand Down Expand Up @@ -283,4 +284,65 @@ class Queries
{
$this->_db->run("UNLOCK TABLES");
}

/**
* Get the list of all instrument records to import from REDCap.
*
* @param array $allowedRedcapInstrumentNames authorized redcap instruments
* @param bool $forceUpdate do force update?
*
* @return Query
*/
public function getLORISInstrumentsToImport(
array $allowedRedcapInstrumentNames,
bool $forceUpdate = false
): Query {
// extract first importable instrument name
$firstInstrumentName = $allowedRedcapInstrumentNames[0];
$allOtherInstrumentNames = array_slice(
$allowedRedcapInstrumentNames,
1
);

// build query for exact instrument name match
$selectedInstruments = "AND (";
$selectedInstruments .= " BINARY f.test_name = '{$firstInstrumentName}'";
foreach ($allOtherInstrumentNames as $instrumentName) {
$selectedInstruments .= " OR BINARY f.test_name = '{$instrumentName}'";
}
$selectedInstruments .= ")";

// if forced, then we do not filter out administration/data entry already
// set up, they will be overridden
$forceUpdateCondition = "";
if (!$forceUpdate) {
$forceUpdateCondition = "
AND (f.Data_entry = 'In Progress' OR f.Data_entry IS NULL)
AND f.Administration IS NULL
";
}

$query = "SELECT c.PSCID as pscid,
c.CandID as candid,
s.Visit_label as visitLabel,
f.test_name as instrument,
f.CommentID as commentID,
f.Data_entry as dataEntry,
f.Administration as administration
FROM flag f
JOIN session s ON (s.ID = f.sessionID)
JOIN candidate c ON (c.candid = s.candid)
WHERE s.Active = 'Y'
AND c.Active = 'Y'
AND f.CommentID NOT LIKE 'DDE%'
{$forceUpdateCondition}
{$selectedInstruments}
";

//
return $this->_db->pselect(
$query,
[]
);
}
}
Loading
Loading