Skip to content

MEPTS-207: Implement Script that can identify which metadata are being used in forms #2

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 21 commits into
base: master
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
21 commits
Select commit Hold shift + click to select a range
dba8358
MEPTS-207: Exctracting concept usage details from HTML forms and prep…
kaweesi May 27, 2019
4183841
MEPTS-207: Added programs usage from forms
kaweesi May 28, 2019
ea96b8d
MEPTS-207: Added indentifier types and person attribute types
kaweesi May 30, 2019
7290acc
MEPTS-207: Supported used and not used roles
kaweesi May 31, 2019
c7cdcab
MEPTS-207: Added used global properties form usages
kaweesi May 31, 2019
0ccdd57
MEPTS-207: accepting uuids as ids for concepts
kaweesi Jun 6, 2019
70b356f
MEPTS-207: added locations usage and introduced failed logging to all…
kaweesi Jun 11, 2019
07bbc7b
MEPTS-207: added metadataUsage readme
kaweesi Jun 11, 2019
8e429ac
MEPTS-207: added external dependencies to readme
kaweesi Jun 13, 2019
a8faa08
MEPTS-207: preparing for running for more than one database
kaweesi Jun 17, 2019
7a733e4
MEPTS-207: added metadataUsage/params.json to gitignore
kaweesi Jun 17, 2019
c802383
un tracking ignored metadataUsage/params.json
kaweesi Jun 17, 2019
5cd30de
MEPTS-207: fixed previous commit of supporting running for more than …
kaweesi Jun 17, 2019
f54d4c1
MEPTS-207: making mysql password optional and adding sample params.js…
kaweesi Jun 17, 2019
8ccb144
MEPTS-207: formatted json in readme
kaweesi Jun 17, 2019
51c4b90
MEPTS-207: making mysql options optional and accepting properties as …
kaweesi Jun 17, 2019
8598466
MEPTS-207: Added both console and log.out file loggings
kaweesi Jun 18, 2019
cff808b
MEPTS-207: fixed --secure-file-priv error etc
kaweesi Jun 19, 2019
381dd2a
MEPTS-207: Fixed using system variables in params.json
kaweesi Jun 19, 2019
0b533ab
MEPTS-207: added troubleshoot section for readme.md file
cfaife Sep 3, 2019
b3a6605
MEPTS-207: formting README.md file
cfaife Sep 3, 2019
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 6 additions & 0 deletions .gitignore
Original file line number Diff line number Diff line change
@@ -0,0 +1,6 @@
metadataUsage/data/
metadataUsage/usage/
metadataUsage/failed/
metadataUsage/test.sh
metadataUsage/params.json
metadataUsage/log.out
63 changes: 63 additions & 0 deletions metadataUsage/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,63 @@
# Generate Metadata usage statistics
> This OpenMRS Script helps to generate csv files including used metadata objects currently used in eSaude environments; concepts, programs, patient identifier types, person attribute types, roles, global properties, locations. The script can be pointed at-least one database through params.json which contains MySQL connection details from which the separated usages are obtained


### Setup
> This script has been written against OS X El Capitan 10.11.6 (15G22010), in any case running against another OS or version complains about any external dependencies such as `xmllint`, `jq`, `bc` etc, please manually install those and proceed.
Configure mysql instances to extract metadata usages within `params.json`, here's a sample content and the this file needs to be in the same folder as the script
```json
{
"mysqlDbConnections": [
{
"binaryLocation": "mysql",
"options": "",
"database": "openmrs",
"username": "root",
"password": ""
},
{
"binaryLocation": "mysql",
"options": "",
"database": "esaude",
"username": "root",
"password": ""
},
{
"binaryLocation": "/Users/u/Programs/openmrs-standalone/database/./mysql",
"options": "--socket=/tmp/esaude3.sock --max_allowed_packet=96M",
"database": "openmrs",
"username": "openmrs",
"password": "test"
},
{
"binaryLocation": "mysql",
"options": "",
"database": "openmrs",
"username": "root",
"password": "$MYSQL_ROOT_PASSWORD"
}
]
}
```

### Running the script
> Run; `./openmrsMetadataUsage.sh`

### Results
> If running it returned no errors (logged into `log.out`) while quitting execution, then three folders should have been created 3 folders; `data, failed, usage`
```
drwxr-xr-x ... data
drwxr-xr-x ... failed
-rw-r--r-- ... log.out
-rwxr-xr-x@ ... openmrsMetadataUsage.sh
drwxr-xr-x ... usage
```
`data`: contains xml forms export from the database as well as metadata field extracts
`failed`: contains failed xml forms which need human intervention, the details of what failed in text files such as; `*_form_failed.txt`
`usage`: contains csv output per exported metadata objects

### Troubleshooting
> if genarated `usage` folder is empty while `/var/lib/mysql-files/` contains generated csv files, you can move it manually as `root` or with `sudo` typing:
```
`mv /var/lib/mysql-files/*.csv usage/`
```
Loading