feature: AI-Enabled Food Search #2325

taylorpatterson-T1D · 2025-07-05T22:31:32Z

Initial commit

taylorpatterson-T1D · 2025-07-05T23:44:39Z

Could I please get a review from the maintainers? @ps2, @marionbarker, @loudnate

This is my submission for a new AI-enabled food search capability for Loop. Loop has helped me become a better diabetic and I would love to give back to the community by contributing this new feature for Loop, which I believe will benefit all Loop users.

PROBLEM WE'RE SOLVING: Diabetics lack consistent nutrition knowledge when estimating carb intake. And we need that knowledge right in the moment, when we’re making dosing decisions - this knowledge is the single most important tool we have to improve our Time in Range and A1C.

NEW FEATURE'S IMPACT: The Food Search system is a comprehensive food analysis and nutrition tracking solution integrated into Loop for improved diabetes management. It provides multiple search options including text, barcode scanning, voice, and AI-powered image analysis that produces comprehensive nutrition data and dosing suggestions via diabetic notes. This let's us enter a meal bolus with MUCH greater accuracy and improve Time in Range and A1C. The AI-enabled analysis engine can even read menu items from an image captured, and provide guidance in multiple languages!!!

Video Demo: https://youtu.be/L0LD8AxNX0Q

Review the docs in “//LoopWorkspace/Loop/Documentation/FoodSearch Docs” for details. API setup guides are provided so the user can enter their your own AI API keys for AI image analysis.

New Search for Food section on the Add Carb Entry view:

Text search:

Barcode scan search for high accuracy:

AI-enabled food image analysis engine in action:

AI-Analysis results with comprehensive analysis details including an item by item breakdown of the meal:

Even more detail about how Portions and Servings were calculated and diabetic specific notes to advise during dosing:

New Setup screen. Pick the AI service you prefer, enter your own API keys:

☐ Added network quality detection ☐ Implement safer timeout handling ☐ Add fast mode fallback logic 1. On Good WiFi: Fast parallel processing (unchanged performance) 2. On Restaurant WiFi: Sequential processing prevents freezing 3. On Cellular: Conservative mode with longer timeouts 4. On Poor Networks: Automatic fallback to safer processing

Improved AI prompt to include glycemic index analysis and strengthen diabetic notes section. Removed excessive DEBUG logging for better overall performance.

marionbarker

This is the first of several comments. It includes 2 specific lines that need to be removed from Loop.xcodeproj/project.pbxproj. The 2 lines contain Taylor's Developer ID. They break the automatic signing configuration for Loop and are not required - they can be deleted.

In addition to those 2 lines in Loop.xcodeproj/project.pbxproj, I do not know why the LibreTransmitter information was touched. I think that is a mistake. (It might not harm anything, but we prefer not to make changes unless they are necessary.)

Loop.xcodeproj/project.pbxproj

marionbarker · 2025-07-08T05:24:21Z

More general comments:

This was pretty slick. The bar code reader was the one I played with the most. The general food search seemed to spend a long time loading, but maybe my network was having an issue.

I will do more testing and report back.

Regarding this addition, I'm not sure of the best way to incorporate it. I'm sure it is user specific whether they want this level of detail or if they go with a few Favorite Food listings and use a SWAG for the carbs to enter (which is my approach). We might want to move the position of the food search/service lower on the screen into its own FOOD SEARCH area (like FAVORITE FOODS).

You might want to upload the images in your comment as png files - they are distorted and hard to read on my phone and distorted and impossible to read on my computer browser.

The following graphic was made as a composite from my test phone (an SE 2nd gen) - the display is taller than will fit on the phone with the keyboard deployed.

taylorpatterson-T1D · 2025-07-09T18:52:23Z

@marionbarker - Thank you for your kind help and suggestion. I have repaired the Taylor-specific issues and reverted the version uptick.

If we make this functionality a new Customization option from the main Loop build script then the user is saying that they want to prioritize the value delivered by making food search results their new priority (versus anything they may have used in the past). So keeping the functionality at the top of the Add Carb Entry view seems appropriate.

If they don't like the functionality then it can be removed returning Add Carb Entry to its default.

If they make the choice to add Food Search, then my POV is that it should remain at the top of the Add Carb Entry section and not be moved below (near Favorite Foods) where it may go overlooked and therefore unused.

Your thoughts?

Also reverted object version to 54

…o dev

marionbarker

I did another review and there are still some changes that need to be removed.

Please note: this only builds with Mac-Xcode. I am unable to build with GitHub Actions (at this time). I tested both the version in PR 2325, commit feb6f26, and the version with the items removed as described below.

For GitHub errors, please see: https://github.com/docs-test/LoopWorkspace/actions/runs/16239378975

Modification Summary

I removed these items and tested that Loop with food search still builds correctly with Mac (macOS 15.5) and Xcode (16.3):

I removed all references to LibreTransmitter from Loop.xcodeproj/project.pbxproj
I removed Loop.xcodeproj/project.xcworkspace/xcshareddata/swiftpm/Package.resolved
I removed the entire folder Loop/Loop.xcodeproj (the Loop.xcodeproj folder should only be at the top level, this was duplicated at a lower level)

I pushed my version to the loopandlearn github account so I could link some of the changes in the details below and to support testing the build with GitHub actions.

Modification Details

Before making these changes

Be sure to clean build folder and close the workspace in Xcode
Only open Xcode workspace again after changes are made

Item 1:

see this commit: loopandlearn@f2874d2

Item 2:

command line to take this action:

% git rm Loop.xcodeproj/project.xcworkspace/xcshareddata/swiftpm/Package.resolved
rm 'Loop.xcodeproj/project.xcworkspace/xcshareddata/swiftpm/Package.resolved'

see this commit: loopandlearn@aa67ab8

Item 3:

command line to take this action:

% git rm -r Loop/Loop.xcodeproj 
rm 'Loop/Loop.xcodeproj/project.pbxproj'
rm 'Loop/Loop.xcodeproj/project.xcworkspace/contents.xcworkspacedata'
rm 'Loop/Loop.xcodeproj/project.xcworkspace/xcshareddata/IDEWorkspaceChecks.plist'
rm 'Loop/Loop.xcodeproj/project.xcworkspace/xcshareddata/WorkspaceSettings.xcsettings'
rm 'Loop/Loop.xcodeproj/xcshareddata/xcschemes/DoseMathTests.xcscheme'
rm 'Loop/Loop.xcodeproj/xcshareddata/xcschemes/Loop Intent Extension.xcscheme'
rm 'Loop/Loop.xcodeproj/xcshareddata/xcschemes/Loop Status Extension.xcscheme'
rm 'Loop/Loop.xcodeproj/xcshareddata/xcschemes/Loop.xcscheme'
rm 'Loop/Loop.xcodeproj/xcshareddata/xcschemes/LoopTests.xcscheme'
rm 'Loop/Loop.xcodeproj/xcshareddata/xcschemes/SmallStatusWidgetExtension.xcscheme'
rm 'Loop/Loop.xcodeproj/xcshareddata/xcschemes/WatchApp.xcscheme'

see this commit: loopandlearn@b40a220

marionbarker · 2025-07-12T18:38:07Z

Regarding not being able to build with GitHub actions. The modification in this link fixed the error.

loopandlearn@b40cb3c

Required change:

add missing arguments as indicated in the error message

Other change:

be consistent using imageUrl instead of imageURL in all arguments and labels
- the imageURL version was found in some places with imageUrl in most places

Error reported using Xcode: Product->Archive was:

/Users/marion/Downloads/ManualClones/Loop_dev_updates/LoopWorkspace/
 Loop/Loop/Services/FoodSearchRouter.swift:
 240:58: error: missing arguments for parameters
 'servingQuantity', 'imageFrontUrl', 'code' in call

            let placeholderProduct = OpenFoodFactsProduct(

marionbarker · 2025-07-13T20:13:15Z

In order for this to be considered for adding to Loop (instead of as a customization - which is a different process), I think the Loop, Settings, AI Food Analysis row would need an addition for showing or not showing the AI tool in the carb screen. Following other additions to the UI, the default would be for this to be disabled.

taylorpatterson-T1D · 2025-07-13T21:07:39Z

@marionbarker
Easy peasy. I will add that and send another pull request tonight or tomorrow.

On the subject of the project files stuff, I was unable to build by following your directions of removing those sub contents. It wouldn't see the Loop project as valid without them, unless I'm doing something wrong.

Move the Food Search behind a feature flag found in Settings Improved AI prompt to prevent overdose recommendations Streamlined Diabetic Notes, Portion Details and other UX improvements

taylorpatterson-T1D · 2025-07-14T00:47:07Z

Food Search Feature Flag Implementation Done!
Files Modified:
UserDefaults+Loop.swift:

Added foodSearchEnabled key to enum
Added computed property with default value false (disabled by default)
Settings are persistent - when disabled, all AI/provider settings remain saved
AISettingsView.swift:
Added feature toggle at the top of the settings screen
Wrapped all configuration sections in if foodSearchEnabled { }
Medical disclaimer remains always visible
Clear messaging about feature control
CarbEntryView.swift:
Wrapped entire Food Search UI section with feature flag check
Properly handles CardSectionDivider logic for both enabled/disabled states
Only shows when isNewEntry && UserDefaults.standard.foodSearchEnabled
SettingsView.swift:
Updated descriptive text to clarify this enables/configures Food Search
User Experience:
Default State: Food Search is disabled by default for new users Settings Persistence: After the user enables Food Search and adds any provider choices or API keys for AI Analysis, all AI API keys and configurations remain saved when later disabled again. Easy Toggle: Users can enable/disable via toggle in Food Search settings. Clean UI: When disabled, the Search for Food UI section completely disappears. No Data Loss: Re-enabling restores all previous settings

I also improved the AI prompt and context to add Glycemic Index inputs, and refined how the food is presented in the UI to make it easier to read.

@marionbarker , @ps2
Since it's now hidden behind a feature flag configurable by the user, can we add this functionality to the main branch of Loop???

marionbarker · 2025-07-14T15:15:10Z

General comments about this commit: 2adc9a3:

I have not finished my detailed review and testing yet, but wanted to get this general comment out.

I already mentioned the changes to Loop.xcodeproj/project.pbxproj need to be reverted.

You did not implement some of my earlier suggestions because you "were unable to build" - we can talk about those on zulipchat via DM if you prefer.

The other general comment is that it is better to do atomic commits that change one thing at a time. Please consider this for future commits - I'm not asking you to change this one.

I see 3 "changes" in this one commit:

You modified imageUrl to be imageURL
- This is opposite from the choice I made but I'm not sure if it matters one way or the other, I will confirm that it is now consistent in all places in a later review
You added features to speed up the process
You added the requested feature to add an enable/disable switch to display the AI tool on the Carb screen

marionbarker · 2025-07-14T16:47:43Z

Summary

The modifications I previously requested are still required before this can be considered.

With those modifications made, this successfully builds both with Mac-Xcode and with Browser Build.
Thank you for adding the enable/disable flag.

Once you make the required modifications, I will test again and then we can see if @ps2 will approve it.

Be sure the modifications are made with Xcode closed. Then when you open Xcode again, do a clean build folder, close the workspace and then open it again. If you do not take those steps, you will get build failures.

Modifications

I started with a fresh branch (modify-taylor-dev-pr2325_vs3) using your latest commit, 2adc9a3, and repeated most of the changes I had previously made (in earlier testing) using cherry-pick. Some had to be handled differently as described below. (The first 3 commits were cherry-pick, the last 3 commits were manual.)

You can see the commits I applied to your version:

https://github.com/loopandlearn/Loop/commits/modify-taylor-dev-pr2325_vs3/

The successful Browser Build is:

https://github.com/docs-test/LoopWorkspace/actions/runs/16272023435

Modification Details

One of the things you did not add was the extra parameters required to enable Browser Build (or Product -> Archive) in Loop/Services/FoodSearchRouter.swift.

The missing arguments I did manually (instead of cherry-pick) because of the case you chose for imageURL.

It turns out that imageFrontUrl must be spelled with Url in title case, so it took me 2 tries to get that modification correct.

Details about case

Initially, you had some instances of imageUrl and others that used imageURL:

You chose to change all instances to imageURL
I had chosen the other way because imageUrl was more frequent in the code at the time
So now all imageURL instances use upper case (capital letters for URL) and all imageFrontUrl use title case
This does not prevent the code from building and operating as expected but it can lead to someone typing the wrong thing

taylorpatterson-T1D · 2025-07-14T21:10:58Z

I did another review and there are still some changes that need to be removed.

Please note: this only builds with Mac-Xcode. I am unable to build with GitHub Actions (at this time). I tested both the version in PR 2325, commit feb6f26, and the version with the items removed as described below.

For GitHub errors, please see: https://github.com/docs-test/LoopWorkspace/actions/runs/16239378975

Modification Summary

I removed these items and tested that Loop with food search still builds correctly with Mac (macOS 15.5) and Xcode (16.3):

I removed all references to LibreTransmitter from Loop.xcodeproj/project.pbxproj

I removed Loop.xcodeproj/project.xcworkspace/xcshareddata/swiftpm/Package.resolved

I removed the entire folder Loop/Loop.xcodeproj (the Loop.xcodeproj folder should only be at the top level, this was duplicated at a lower level)

I pushed my version to the loopandlearn github account so I could link some of the changes in the details below and to support testing the build with GitHub actions.

Modification Details

Before making these changes

Be sure to clean build folder and close the workspace in Xcode

Only open Xcode workspace again after changes are made

Item 1:

see this commit: loopandlearn@f2874d2

Item 2:

command line to take this action:
% git rm Loop.xcodeproj/project.xcworkspace/xcshareddata/swiftpm/Package.resolved
rm 'Loop.xcodeproj/project.xcworkspace/xcshareddata/swiftpm/Package.resolved'
see this commit: loopandlearn@aa67ab8

Item 3:

command line to take this action:
% git rm -r Loop/Loop.xcodeproj 
rm 'Loop/Loop.xcodeproj/project.pbxproj'
rm 'Loop/Loop.xcodeproj/project.xcworkspace/contents.xcworkspacedata'
rm 'Loop/Loop.xcodeproj/project.xcworkspace/xcshareddata/IDEWorkspaceChecks.plist'
rm 'Loop/Loop.xcodeproj/project.xcworkspace/xcshareddata/WorkspaceSettings.xcsettings'
rm 'Loop/Loop.xcodeproj/xcshareddata/xcschemes/DoseMathTests.xcscheme'
rm 'Loop/Loop.xcodeproj/xcshareddata/xcschemes/Loop Intent Extension.xcscheme'
rm 'Loop/Loop.xcodeproj/xcshareddata/xcschemes/Loop Status Extension.xcscheme'
rm 'Loop/Loop.xcodeproj/xcshareddata/xcschemes/Loop.xcscheme'
rm 'Loop/Loop.xcodeproj/xcshareddata/xcschemes/LoopTests.xcscheme'
rm 'Loop/Loop.xcodeproj/xcshareddata/xcschemes/SmallStatusWidgetExtension.xcscheme'
rm 'Loop/Loop.xcodeproj/xcshareddata/xcschemes/WatchApp.xcscheme'
see this commit: loopandlearn@b40a220

…aceChecks.plist

…Settings.xcsettings

…on.xcscheme

…tension.xcscheme

taylorpatterson-T1D · 2025-07-14T23:11:00Z

rm 'Loop/Loop.xcodeproj/project.pbxproj'

@marionbarker
I did all the reset of the deletions but before I do this one - I wanted to clarify, won't this result in Loop not having all the additional Food Search related files registered in the project? Would that force the builder to have to re-add them to their own project.pbxproj?

taylorpatterson-T1D · 2025-07-14T23:50:02Z

Regarding not being able to build with GitHub actions. The modification in this link fixed the error.

loopandlearn@b40cb3c

Required change:

add missing arguments as indicated in the error message

Other change:

be consistent using imageUrl instead of imageURL in all arguments and labels

the imageURL version was found in some places with imageUrl in most places

Error reported using Xcode: Product->Archive was:
/Users/marion/Downloads/ManualClones/Loop_dev_updates/LoopWorkspace/
 Loop/Loop/Services/FoodSearchRouter.swift:
 240:58: error: missing arguments for parameters
 'servingQuantity', 'imageFrontUrl', 'code' in call

            let placeholderProduct = OpenFoodFactsProduct(

I chose imageURL as the standard for consistency because URL is an acronym and therefore more likely to be more future proof if others contribute to the feature. This was tested as working on my end so it should now be resolved with my last commit.

marionbarker · 2025-07-15T15:24:50Z

@taylorpatterson-T1D

Please do not be discouraged. But please read carefully.

commit you need to revert

You labeled this commit as "Delete redundant Loop.xcodeproj directory"

48e88e0

But in fact you deleted the one directory that was correct, i.e., you deleted the Loop.xcodeproj folder you were supposed to keep. So of course it won't build, in fact you get an error just by opening the workspace.

I tried fixing just that commit with this command:

git revert 48e88e07

And in my clone it now builds fine.

atomic commits

When I said use atomic commits - I meant all changes in a single commit should solve a problem. It might be a lot of files. Doing this is not required but it is a good idea because then reviewers can look at the changes made per commit. It makes it easier to review.

test build before push

Please do not push any commits until you have a working build. Then push them to GitHub at one time.

When you pushed following every commit of deleting single files (all for one purpose) yesterday, that sent a separate email to every one who watches the Loop repository.

You deleted (one file at a time) items found in the Loop/Loop.xcodeproj directory that was truly redundant. But you did not delete the directory itself.

missing arguments

You have not fixed the missing arguments that will prevent this from working for a browser build.

Test this yourself by taking these actions in Xcode:

Product, Clean Build Folder
Product, Archive

The build will fail. Click on the Report Navigator icon
Click on the error:

Showing Recent Messages
/Users/marion/Downloads/ManualClones/Loop_dev_updates/LoopWorkspace/Loop/Loop/Services/FoodSearchRouter.swift:240:58: Missing arguments for parameters 'servingQuantity', 'imageFrontUrl', 'code' in call

Initial Commit

1dff182

taylorpatterson-T1D mentioned this pull request Jul 6, 2025

Integrate OpenFoodFacts food search API into Loop's "Add Carb Entry" screen (+barcode scan +voice dictation) #1939

Closed

taylorpatterson-T1D changed the title ~~AI-enabled food search~~ AI-Enabled Food Search Jul 6, 2025

taylorpatterson-T1D added 2 commits July 6, 2025 14:28

AI Prompt Improvements

9287c97

Improved AI prompt to include glycemic index analysis and strengthen diabetic notes section. Removed excessive DEBUG logging for better overall performance.

taylorpatterson-T1D changed the title ~~AI-Enabled Food Search~~ feature: AI-Enabled Food Search Jul 7, 2025

marionbarker requested changes Jul 8, 2025

View reviewed changes

Loop.xcodeproj/project.pbxproj Outdated Show resolved Hide resolved

Loop.xcodeproj/project.pbxproj Outdated Show resolved Hide resolved

Loop.xcodeproj/project.pbxproj Outdated Show resolved Hide resolved

taylorpatterson-T1D added 3 commits July 9, 2025 11:58

Removed developer-specific team refs in Loop.xcodeproj/project.pbxproj

e78c3a1

Also reverted object version to 54

Merge branch 'dev' of https://github.com/taylorpatterson-T1D/Loop int…

82348a6

…o dev

Repaired object version (again)

feb6f26

marionbarker requested changes Jul 12, 2025

View reviewed changes

Make Food Search a feature flag so it can be included in main branch.

2adc9a3

Move the Food Search behind a feature flag found in Settings Improved AI prompt to prevent overdose recommendations Streamlined Diabetic Notes, Portion Details and other UX improvements

taylorpatterson-T1D requested a review from marionbarker July 14, 2025 13:07

This was referenced Jul 14, 2025

Add AI Food Search to Loop Carb Interface as a Customization loopandlearn/customization#59

Open

Add AI Food Search to Loop Carb Interface as a Customization loopandlearn/lnl-scripts#82

Open

taylorpatterson-T1D added 5 commits July 14, 2025 15:32

Delete Loop/DefaultAssets.xcassets/AI-logo-master.png

6f7253b

Update project.pbxproj

54c64db

Update project.pbxproj to remove Taylor's team ID

ced051a

Update project.pbxproj - cleaned unnecessary LibreTransmitter code

a0a1f24

Update Package.resolved

a8d8804

taylorpatterson-T1D added 11 commits July 14, 2025 16:02

Delete redundant Loop.xcodeproj directory

48e88e0

Delete Loop/Loop.xcodeproj/project.xcworkspace/xcshareddata/IDEWorksp…

9a75864

…aceChecks.plist

Delete Loop/Loop.xcodeproj/project.xcworkspace/xcshareddata/Workspace…

603c694

…Settings.xcsettings

Delete Loop/Loop.xcodeproj/project.xcworkspace/contents.xcworkspacedata

a18e3d1

Delete Loop/Loop.xcodeproj/xcshareddata/xcschemes/DoseMathTests.xcscheme

14333fb

Delete Loop/Loop.xcodeproj/xcshareddata/xcschemes/Loop Intent Extensi…

61fc8ac

…on.xcscheme

Delete Loop/Loop.xcodeproj/xcshareddata/xcschemes/Loop Status Extensi…

74e1e93

…on.xcscheme

Delete Loop/Loop.xcodeproj/xcshareddata/xcschemes/WatchApp.xcscheme

9b26acb

Delete Loop/Loop.xcodeproj/xcshareddata/xcschemes/SmallStatusWidgetEx…

107754e

…tension.xcscheme

Delete Loop/Loop.xcodeproj/xcshareddata/xcschemes/LoopTests.xcscheme

3859746

Delete Loop/Loop.xcodeproj/xcshareddata/xcschemes/Loop.xcscheme

dd71480

feature: AI-Enabled Food Search #2325

Are you sure you want to change the base?

feature: AI-Enabled Food Search #2325

Conversation

taylorpatterson-T1D commented Jul 5, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

taylorpatterson-T1D commented Jul 5, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

marionbarker left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

marionbarker commented Jul 8, 2025

Uh oh!

taylorpatterson-T1D commented Jul 9, 2025

Uh oh!

marionbarker left a comment

Choose a reason for hiding this comment

Modification Summary

Modification Details

Item 1:

Item 2:

Item 3:

Uh oh!

marionbarker commented Jul 12, 2025

Uh oh!

marionbarker commented Jul 13, 2025

Uh oh!

taylorpatterson-T1D commented Jul 13, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

taylorpatterson-T1D commented Jul 14, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

marionbarker commented Jul 14, 2025

Uh oh!

marionbarker commented Jul 14, 2025

Summary

Modifications

Modification Details

Uh oh!

taylorpatterson-T1D commented Jul 14, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Modification Summary

Modification Details

Item 1:

Item 2:

Item 3:

Uh oh!

taylorpatterson-T1D commented Jul 14, 2025

Uh oh!

taylorpatterson-T1D commented Jul 14, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

marionbarker commented Jul 15, 2025

commit you need to revert

atomic commits

test build before push

missing arguments

Uh oh!

Uh oh!

taylorpatterson-T1D commented Jul 5, 2025 •

edited

Loading

taylorpatterson-T1D commented Jul 5, 2025 •

edited

Loading

taylorpatterson-T1D commented Jul 13, 2025 •

edited

Loading

taylorpatterson-T1D commented Jul 14, 2025 •

edited

Loading

taylorpatterson-T1D commented Jul 14, 2025 •

edited

Loading

taylorpatterson-T1D commented Jul 14, 2025 •

edited

Loading