Local LLM Chat

A Flutter chat application that integrates with multiple AI providers (Ollama, Llama.cpp, LM Studio) through Ngrok tunnels.

ScreenRecording_08-15-2025.15-03-52_1.MP4

Features

• Multi-Provider Support: Choose between Ollama, Llama.cpp, and LM Studio
• Modern UI: Clean, responsive design with Material 3
• State Management: BLoC pattern for predictable state management
• Navigation: GoRouter for type-safe navigation
• Real-time Chat: Smooth chat experience with automatic scrolling
• Environment Configuration: Configurable API endpoints through .env
• Extensible Architecture: Designed to support dynamic provider configuration in future iterations

Architecture

The app follows a feature-based architecture with clear separation of concerns:

lib/
├── core/                    # Core app functionality
│   ├── config/             # App configuration and routing
│   ├── constants/          # App constants and enums
│   ├── di/                 # Dependency injection setup
│   └── theme/              # App theming
└── features/               # Feature modules
    ├── home/               # Home screen with provider selection
    │   └── screens/
    └── chat/               # Chat functionality
        ├── bloc/           # BLoC state management
        ├── models/         # Data models
        ├── screens/        # Chat screen
        ├── services/       # AI service implementations
        └── widgets/        # Chat widgets

Getting Started

Prerequisites

• Flutter SDK 3.6.2 or higher
• Dart SDK
• One or more of the following AI providers running locally:
  • Ollama (default port: 11434)
  • Llama.cpp server (default port: 8080)
  • LM Studio (default port: 1234)

Installation

1. Clone the repository:

   git clone <repository-url>
   cd it_3265_ngrok

2. Install dependencies:

   flutter pub get

3. Configure your AI provider endpoints and models in the .env file:

   # OpenAI API Key 
   OPENAI_API_KEY=your_openai_api_key_here
   
   # Current supported providers (hardcoded for now)
   OLLAMA_URL=https://your-ngrok-url.ngrok.io
   OLLAMA_MODEL=llama3.1
   
   LLAMA_CPP_URL=https://your-ngrok-url.ngrok.io
   LLAMA_CPP_MODEL=llama3.2
   
   LMSTUDIO_URL=https://your-ngrok-url.ngrok.io
   LMSTUDIO_MODEL=llama3.2

4. Run the app:

   flutter run

Usage

1. Launch the app - You'll see the home screen with provider selection
2. Choose an AI provider - Select from the dropdown (Ollama, Llama.cpp, or LM Studio)
3. Start chatting - Tap "Start Chat" to begin a conversation
4. Send messages - Type your message and tap the send button
5. Return home - Use the back button to return to provider selection

Key Features

Provider Selection

• Dropdown interface for choosing AI providers
• Visual icons for each provider type
• Information panel about Ngrok tunneling
• Note: Currently supports 3 hardcoded providers, but architecture allows for dynamic expansion

Chat Interface

• Message bubbles with distinct styling for user vs AI messages
• Timestamp display for each message
• Auto-scroll to latest messages
• Loading states during AI response
• Error handling with user feedback

State Management

• BLoC pattern for predictable state management
• Reactive UI updates
• Proper loading and error states
• Message history management

Configuration

Environment Variables

The app uses the following environment variables (configured in .env):

Required Variables:

• OPENAI_API_KEY: Your OpenAI API key (if using OpenAI directly)

Current Supported Providers (Hardcoded):

• OLLAMA_URL: Ollama API endpoint
• OLLAMA_MODEL: Model name for Ollama (e.g., llama3.2, codellama)
• LLAMA_CPP_URL: Llama.cpp server endpoint
• LLAMA_CPP_MODEL: Model name for Llama.cpp (e.g., llama-2-7b-chat)
• LMSTUDIO_URL: LM Studio API endpoint
• LMSTUDIO_MODEL: Model file name for LM Studio (e.g., llama-2-7b-chat.Q4_K_M.gguf)

Future Enhancement: The app architecture is designed to support dynamic provider configuration, allowing users to add any number of AI providers through environment variables with minimal code changes.

AI Provider Integration

All AI providers are accessed through OpenAI-compatible APIs using the dart_openai package. The app automatically configures the base URL based on the selected provider.

Ngrok Configuration

Make sure all tunnels are configured - type

ngrok config edit

and make sure the content is similar to ( provide your own authtoken and proper ports)

version: "3"
agent:
  authtoken: {YOUR_AUTH_TOKEN}
  log_level: debug

endpoints:
  - name: llama
    upstream:
      url: 8080
    traffic_policy:
      inbound:
        - actions:
            - type: "add-headers"
              config:
                headers:
                  host: localhost

  - name: ollama
    upstream:
      url: 11434
    traffic_policy:
      inbound:
        - actions:
            - type: "add-headers"
              config:
                headers:
                  host: localhost

  - name: lmStudio
    upstream:
      url: http://localhost:1234
    traffic_policy:
      inbound:
        - actions:
            - type: "add-headers"
              config:
                headers:
                  host: localhost

Then run the

ngrok start --all

command to initiates all tunnels and to obtain public urls.

Development

Running Tests

flutter test

Code Analysis

flutter analyze

Running

flutter run

Architecture Decisions

Current Implementation

• Hardcoded Providers: Currently supports 3 specific providers (Ollama, Llama.cpp, LM Studio)
• Extensible Design: Architecture allows for easy addition of new providers

Future Enhancements

• Dynamic Provider Configuration: Planned feature to read provider list from environment variables
• Provider Agnostic Design: Framework supports any OpenAI-compatible API endpoint

BLoC Pattern

• Chosen for predictable state management
• Separates business logic from UI
• Easy testing and debugging

Feature-based Structure

• Modular organization
• Clear separation of concerns
• Scalable architecture

GoRouter

• Type-safe navigation
• Declarative routing
• Better deep linking support

GetIt for DI

• Simple service locator pattern
• Easy dependency management
• Testability

Contributing

1. Fork the repository
2. Create a feature branch
3. Make your changes
4. Add tests if applicable
5. Submit a pull request

📝 License

This project is licensed under the BSD-3-Clause license License - see the LICENSE file for details.

👥 Authors

Created by Oleksandr Kholiavko
Extrawest.com, 2025