qa:kerberos:cascadingLoadData fails with missing file

[talevy]

:qa:kerberos:cascadingLoadData task fails with snapshot file missing:

/var/lib/jenkins/workspace/elastic elasticsearch-hadoop 6.7 periodic/qa/kerberos/build/libs/kerberos-6.7.0-SNAPSHOT-cascading-testing.jar

full stacktrace:

19/02/13 19:52:19 INFO flow.Flow: [] stopping all jobs
19/02/13 19:52:19 INFO flow.FlowStep: [] stopping: (1/1) ...beros_cascading_data/_doc
19/02/13 19:52:19 INFO flow.Flow: [] stopped all jobs
Exception in thread "main" cascading.flow.FlowException: unhandled exception
	at cascading.flow.BaseFlow.complete(BaseFlow.java:918)
	at org.elasticsearch.hadoop.qa.kerberos.cascading.LoadToES.run(LoadToES.java:81)
	at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
	at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:84)
	at org.elasticsearch.hadoop.qa.kerberos.cascading.LoadToES$1.run(LoadToES.java:53)
	at org.elasticsearch.hadoop.qa.kerberos.cascading.LoadToES$1.run(LoadToES.java:50)
	at java.security.AccessController.doPrivileged(Native Method)
	at javax.security.auth.Subject.doAs(Subject.java:422)
	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1762)
	at org.elasticsearch.hadoop.qa.kerberos.security.KeytabLogin.doAfterLogin(KeytabLogin.java:52)
	at org.elasticsearch.hadoop.qa.kerberos.cascading.LoadToES.main(LoadToES.java:50)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.apache.hadoop.util.RunJar.run(RunJar.java:226)
	at org.apache.hadoop.util.RunJar.main(RunJar.java:141)
Caused by: java.io.FileNotFoundException: File /var/lib/jenkins/workspace/elastic elasticsearch-hadoop 6.7 periodic/qa/kerberos/build/libs/kerberos-6.7.0-SNAPSHOT-cascading-testing.jar does not exist
	at org.apache.hadoop.fs.RawLocalFileSystem.deprecatedGetFileStatus(RawLocalFileSystem.java:611)
	at org.apache.hadoop.fs.RawLocalFileSystem.getFileLinkStatusInternal(RawLocalFileSystem.java:824)
	at org.apache.hadoop.fs.RawLocalFileSystem.getFileStatus(RawLocalFileSystem.java:601)
	at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:428)
	at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:337)
	at org.apache.hadoop.fs.FileSystem.copyFromLocalFile(FileSystem.java:1968)
	at org.apache.hadoop.fs.FileSystem.copyFromLocalFile(FileSystem.java:1936)
	at org.apache.hadoop.fs.FileSystem.copyFromLocalFile(FileSystem.java:1901)
	at org.apache.hadoop.mapreduce.JobResourceUploader.copyJar(JobResourceUploader.java:246)
	at org.apache.hadoop.mapreduce.JobResourceUploader.uploadFiles(JobResourceUploader.java:166)
	at org.apache.hadoop.mapreduce.JobSubmitter.copyAndConfigureFiles(JobSubmitter.java:95)
	at org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.java:190)
	at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1290)
	at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1287)
	at java.security.AccessController.doPrivileged(Native Method)
	at javax.security.auth.Subject.doAs(Subject.java:422)
	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1762)
	at org.apache.hadoop.mapreduce.Job.submit(Job.java:1287)
	at org.apache.hadoop.mapred.JobClient$1.run(JobClient.java:575)
	at org.apache.hadoop.mapred.JobClient$1.run(JobClient.java:570)
	at java.security.AccessController.doPrivileged(Native Method)
	at javax.security.auth.Subject.doAs(Subject.java:422)
	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1762)
	at org.apache.hadoop.mapred.JobClient.submitJobInternal(JobClient.java:570)
	at org.apache.hadoop.mapred.JobClient.submitJob(JobClient.java:561)
	at cascading.flow.hadoop.planner.HadoopFlowStepJob.internalNonBlockingStart(HadoopFlowStepJob.java:108)
	at cascading.flow.planner.FlowStepJob.blockOnJob(FlowStepJob.java:217)
	at cascading.flow.planner.FlowStepJob.start(FlowStepJob.java:160)
	at cascading.flow.planner.FlowStepJob.call(FlowStepJob.java:124)
	at cascading.flow.planner.FlowStepJob.call(FlowStepJob.java:43)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)
:qa:kerberos:cascadingLoadData FAILED```

CI run: https://elasticsearch-ci.elastic.co/job/elastic+elasticsearch-hadoop+6.7+periodic/8/console

[jbaiera]

I have been looking into this for a couple days. The issue here is a little deeper than Cascading, but rather Hadoop URL Decodes file paths for jar files before trying to load them to the local HDFS for job submission. Our CI builds are run underneath a directory with plus signs (+) in them: elastic+elasticsearch-hadoop+6.7+periodic, which are decoded into spaces by Hadoop and results in Hadoop not being able to find the files on the local FS anymore.

[jbaiera]

These tests are disabled while we figure out the best way forward for running them on CI.