Feature/hybrid rag pipeline #2
Open
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
- Add hybrid_rag_pipeline.py: Main pipeline with S3 PDFs and Elasticsearch sales data - Add Elasticsearch setup scripts for consolidated/non-consolidated indices - Add example data files showing both data structures - Add comprehensive test_setup.py for environment validation - Update .gitignore to exclude cache files and credentials - Update .env.example with all required configuration variables - Add requirements.txt with necessary dependencies - Add development plan documentation
- Added verify_customer_support_index() function to hybrid_rag_pipeline.py - Function analyzes metadata to verify both S3 and Elasticsearch sources are present - Generates file type distribution metrics (PDF vs Text files) - Integrated as Step 7 in main pipeline execution - Added standalone verify_customer_support_index.py script for manual verification - Provides source distribution analysis and verification status
…i_ner), verify customer-support index; update notebook structure to only remaining tasks; update deps
…aml and enrich_and_convert.py; generate enriched .py and .ipynb; update structure doc to remaining tasks only
- Implemented custom Elasticsearch preprocessing with index validation - Added smart index management (validates sales-records-consolidated, recreates customer-support) - Updated all connector functions to use proper context managers and best practices - Migrated to modern Unstructured API patterns (CreateWorkflowRequest, CreateWorkflow) - Enhanced error handling with clear emoji indicators and detailed feedback - Removed global client initialization in favor of context managers - Updated README with comprehensive documentation, architecture diagram, and usage guide - Added troubleshooting section with common issues and solutions - Improved notebook generation pipeline with enrich_and_convert.py - All functions now follow current Unstructured SDK best practices
- Updated INTRO markdown block with comprehensive parallel workflow diagram - Added detailed 7-stage Unstructured API pipeline explanation - Enhanced CREATE_WORKFLOWS section with parallel processing benefits - Improved ES_PREPROCESSING section with smart index management details - Updated README.md architecture section with new parallel workflow diagram - Regenerated enriched notebook with improved documentation - All diagrams now clearly show two independent workflows converging at destination
- Added verify_customer_support_results() function to analyze processed data - Function identifies document sources (S3 PDF vs Elasticsearch) and shows examples - Added verification as separate code block using # %% for jupytext compatibility - Simplified parallel workflow architecture diagram for better clarity - Cleaned up markdown documentation to be more matter-of-fact - Added direct main() call for proper pipeline execution - Pipeline now automatically verifies results after processing completes
![cursor[bot]](https://avatars.githubusercontent.com/in/1210556?s=60&v=4){kind=small}
This comment was marked as outdated.
This comment was marked as outdated.
Sorry, something went wrong.
This comment was marked as outdated.
This comment was marked as outdated.
Sorry, something went wrong.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Ready for review. Please review the file
hybrid_rag_pipeline_enriched.ipynb.