collectstatic with compiling and remote storage is broken

[skirsdeda]

I'm using django-pipeline with S3 as storage. Less compiler is used for css.

When running collectstatic this happens:

1. Existing static files are copied to S3
   For every less file:
2. less file is compiled and it's compiled version appears in source directory (same directory as less file)
3. after compiling pipeline tries to compress the css file and cannot find it because it's not uploaded to S3 yet

Maybe static files finders could be used in compressing stage instead of trying to open compiled file path on storage.

[skirsdeda]

Even if a compiled css is found in storage when compressing, it might be outdated! In case it was changed and compilation happened just before compressing, a newly compiled version is not yet uploaded to storage!

[jplehmann]

@skirsdeda I reported #486 but after analyzing it more realize it's the same issue you were having. Did #475 fix your problem? I see it's failing the test.

I would also love to have our expert @cyberdelia speak into this. Surely we are configuring something wrong here -- we can't be the first people who've ever used pipeline+s3+less?!

[jplehmann]

And is this related to the fact that our compiled CSS is showing up in same directory as our source tree? That may be an indication that we've configured something wrong which is causing this problem. This was discussed in 202 but it was unclear what the resolution was.

[skirsdeda]

@jplehmann Well yes, #475 fixed this problem and I'm using it in production. The only problem is compiled CSS showing up in same directory.
When I have some more time for this I'll look into #202 as well as build errors in my pull request but I'm pretty certain this is a bug and not a configuration issue.

[skirsdeda]

@jplehmann While #475 might not be the best solution, it works because compressors look for local files instead of in remote storage. And compilers (like lessjs) output files to local directories, that's why compressors cannot find compiled js or css in remote storage.

But in case no compressors were used, compiled files wouldn't appear in remote storage (at least not until collectstatic is run for a consecutive time and compiled files are found in local directories).

So the best solution would be to fix compiler code to output compiled files straight to remote storage while compressors should remain unchanged (look for compiled files in remote storage as they do now). I'll fix the code in #475 (hopefully next week).

[jplehmann]

Thanks @skirsdeda! The fix works for me too. Hopefully we can get it passing the test and merged, or get input from @cyberdelia on how this was supposed to work.

[jcalazan]

+1 Fix worked for me as well.

[steinbachr]

+1 fix worked perfectly

[peec]

having this problem as well, atm deployment of our app is stuck here.

Stack trace:

  Traceback (most recent call last):
  File "markant_app/manage.py", line 10, in <module>
  execute_from_command_line(sys.argv)
  File "/opt/python/run/venv/local/lib/python2.7/site-packages/django/core/management/__init__.py", line 338, in execute_from_command_line
  utility.execute()
  File "/opt/python/run/venv/local/lib/python2.7/site-packages/django/core/management/__init__.py", line 330, in execute
  self.fetch_command(subcommand).run_from_argv(self.argv)
  File "/opt/python/run/venv/local/lib/python2.7/site-packages/django/core/management/base.py", line 393, in run_from_argv
  self.execute(*args, **cmd_options)
  File "/opt/python/run/venv/local/lib/python2.7/site-packages/django/core/management/base.py", line 444, in execute
  output = self.handle(*args, **options)
  File "/opt/python/run/venv/local/lib/python2.7/site-packages/django/contrib/staticfiles/management/commands/collectstatic.py", line 168, in handle
  collected = self.collect()
  File "/opt/python/run/venv/local/lib/python2.7/site-packages/django/contrib/staticfiles/management/commands/collectstatic.py", line 114, in collect
  for original_path, processed_path, processed in processor:
  File "/opt/python/run/venv/local/lib/python2.7/site-packages/pipeline/storage.py", line 26, in post_process
  packager.pack_stylesheets(package)
  File "/opt/python/run/venv/local/lib/python2.7/site-packages/pipeline/packager.py", line 96, in pack_stylesheets
  variant=package.variant, **kwargs)
  File "/opt/python/run/venv/local/lib/python2.7/site-packages/pipeline/packager.py", line 106, in pack
  content = compress(paths, **kwargs)
  File "/opt/python/run/venv/local/lib/python2.7/site-packages/pipeline/compressors/__init__.py", line 73, in compress_css
  css = self.concatenate_and_rewrite(paths, output_filename, variant)
  File "/opt/python/run/venv/local/lib/python2.7/site-packages/pipeline/compressors/__init__.py", line 137, in concatenate_and_rewrite
  content = self.read_text(path)
  File "/opt/python/run/venv/local/lib/python2.7/site-packages/pipeline/compressors/__init__.py", line 220, in read_text
  content = self.read_bytes(path)
  File "/opt/python/run/venv/local/lib/python2.7/site-packages/pipeline/compressors/__init__.py", line 214, in read_bytes
  file = staticfiles_storage.open(path)
  File "/opt/python/run/venv/local/lib/python2.7/site-packages/django/core/files/storage.py", line 37, in open
  return self._open(name, mode)
  File "/opt/python/run/venv/local/lib/python2.7/site-packages/storages/backends/s3boto.py", line 366, in _open
  raise IOError('File does not exist: %s' % name)
  IOError: File does not exist: static/scss/main.css
   (ElasticBeanstalk::ExternalInvocationError)

[thomasyip]

I wish I had discovered this thread before I have the patch. Regardless, here is my PR: #502.

I came to the same conclusion as @skirsdeda regarding the problem. But, I was trying to solve the problem by copying the file to s3. If you would like the intermediate file or if you are not compressing the css, my patch might be helpful.

[jplehmann]

I wonder why our leader @cyberdelia would not share wisdom on this issue and the PR contribution after 2 months? Surely he must be delayed be real-life.

[thomasyip]

@jplehmann, he is probably not actively working in a related area in the last few weeks.

He might want to look at the code before making comments and it is very easy to be carry away by day-to-day busi-ness.

[peec]

Also tried many other previous versions (that did not work). 1.4.0-1.5.4, didn't get pipeline to work on any of the versions with S3.