|
| 1 | +--- |
| 2 | +layout: page |
| 3 | +title: Modeling indoor behaviors using deep-RL |
| 4 | +description: Modeling indoor behaviors of people using deep-RL and simulating virus transmissions to evaluate health interventions |
| 5 | +img: |
| 6 | +importance: 1 |
| 7 | +category: work |
| 8 | +related_publications: false |
| 9 | +--- |
| 10 | + |
| 11 | +> 📘 **This post is an accessible read of a work under publication** |
| 12 | +> |
| 13 | +> I am happy to share the code on request for any evaluation of skills, or discussion about the project. After the submission of the work, I will open-source the code, and the manuscript on arxiv. |
| 14 | +
|
| 15 | +## Simulating Indoor Behavior and Viral Transmission with Deep RL |
| 16 | + |
| 17 | +This project simulates human behavior in an indoor home environment using Deep Q-Learning to model rational decision-making based on physiological and routine-driven needs. Each agent (represented as a circle in the simulation) has internal state variables such as `needs_food`, `needs_bathroom`, and others that drive its movement across rooms (e.g., kitchen, bathroom, bedroom). |
| 18 | + |
| 19 | +The goal is to train agents to behave in a way that satisfies their needs efficiently while interacting with others in a shared space—capturing both individual rationality and population-level emergent behavior. |
| 20 | + |
| 21 | +### A video demo |
| 22 | + |
| 23 | +Below is a short clip describing the simulation. One of the agents is infected with covid-19 and the other is uninfected. They move about, occassionally going to the bathroom or kitchen depending on the next. |
| 24 | + |
| 25 | +<video width="100%" controls> |
| 26 | + <source src="{{ site.baseurl }}/assets/video/behavior_modeling.mp4" type="video/mp4"> |
| 27 | + Your browser does not support the video tag. |
| 28 | +</video> |
| 29 | + |
| 30 | +--- |
| 31 | + |
| 32 | +### Feature Representation |
| 33 | + |
| 34 | +Each agent’s state is encoded as a one-hot vector concatenating both **need states** and **room location**. Specifically: |
| 35 | + |
| 36 | +Let: |
| 37 | +- $$\mathbf{n} \in \{0,1\}^N $$ be a one-hot vector representing the current active need (e.g., hunger, bathroom). |
| 38 | +- $$\mathbf{r} \in \{0,1\}^R $$ be a one-hot vector representing the agent’s current room (e.g., kitchen, bathroom, bedroom, etc.). |
| 39 | + |
| 40 | +The full input to the Q-network is: |
| 41 | + |
| 42 | +$$\mathbf{s} = [\mathbf{n} | \mathbf{r}]$$ |
| 43 | + |
| 44 | +where $$ | $$ denotes concatenation. |
| 45 | + |
| 46 | +This vectorized state is passed through a neural network (Q-network) to produce Q-values for each possible room transition action. |
| 47 | + |
| 48 | +--- |
| 49 | + |
| 50 | +### Learning |
| 51 | + |
| 52 | +Agents are trained using the standard Deep Q-Learning algorithm. The objective is to approximate the optimal action-value function $$ Q^*(s, a) $$, which represents the expected cumulative reward from state $$ s $$, taking action $$ a $$, and following the optimal policy thereafter. |
| 53 | + |
| 54 | +At each step, we minimize the temporal difference loss: |
| 55 | + |
| 56 | +$$ |
| 57 | +\mathcal{L}(\theta) = \mathbb{E}_{(s, a, r, s')} \left[ \left( r + \gamma \max_{a'} Q_\theta^{-}(s', a') - Q_\theta(s, a) \right)^2 \right] |
| 58 | +$$ |
| 59 | + |
| 60 | +where: |
| 61 | +- $$ \theta $$ are the parameters of the current Q-network. |
| 62 | +- $$ Q_\theta^{-} $$ is a target network with periodically updated weights. |
| 63 | +- $$ \gamma \in [0,1] $$ is the discount factor. |
| 64 | +- $$ r $$ is the reward received for satisfying a need. |
| 65 | + |
| 66 | +Rewards are sparse and only provided when a need is satisfied (e.g., an agent eats when `needs_food` is active and they enter the kitchen). |
| 67 | + |
| 68 | + |
| 69 | +### Virus Transmission Dynamics |
| 70 | + |
| 71 | +In addition to behavioral modeling, the simulation includes a probabilistic model of airborne virus transmission, visualized as red particles that diffuse through space. |
| 72 | + |
| 73 | +Each infected agent emits viral particles following a Gaussian spatial distribution centered at their current location: |
| 74 | + |
| 75 | +$$ |
| 76 | +P(x, y) = \frac{1}{2\pi\sigma^2} \exp\left(-\frac{(x - x_0)^2 + (y - y_0)^2}{2\sigma^2}\right) |
| 77 | +$$ |
| 78 | + |
| 79 | +where: |
| 80 | +- $$ (x_0, y_0) $$ is the agent's current position, |
| 81 | +- $$ \sigma $$ is the spread parameter that increases with infection intensity, |
| 82 | +- $$ P(x, y) $$ is the probability density of viral concentration at point $$ (x, y) $$. |
| 83 | + |
| 84 | +The **intensity of infection** determines both the **frequency** of particle emission and the **spread** of the Gaussian. This intensity is a time-dependent function that peaks between 4–7 days post-infection and then decays: |
| 85 | + |
| 86 | +$$ |
| 87 | +I(t) = \exp\left(-\frac{(t - \mu)^2}{2\sigma_t^2} \right) |
| 88 | +$$ |
| 89 | + |
| 90 | +with: |
| 91 | +- $$ t $$ as the number of days since contact, |
| 92 | +- $$ \mu = 5.5 $$ (days), |
| 93 | +- $$ \sigma_t = 1.0 $$. |
| 94 | + |
| 95 | +Transmission risk is computed by integrating exposure over time and proximity to infected individuals. When multiple agents are in the same room, this allows the simulation to capture how behavioral patterns—like congregating in the kitchen—can influence disease spread at both the individual and population levels. |
| 96 | + |
| 97 | +--- |
| 98 | + |
| 99 | +### Applications |
| 100 | + |
| 101 | +This system provides a testbed for studying: |
| 102 | +- The effect of household layouts on transmission risk. |
| 103 | +- The impact of behavioral interventions (e.g., staggered routines). |
| 104 | +- Emergent coordination among rational agents in confined, shared spaces. |
| 105 | + |
| 106 | +By combining deep reinforcement learning with epidemiological modeling, this simulation allows for a rich analysis of decision-making under constraints, with applications in public health, smart environments, and human-AI interaction design. |
| 107 | + |
| 108 | +### Next Steps |
| 109 | + |
| 110 | +Future directions aim to enrich the simulation's realism and policy relevance by incorporating adaptive behavioral and environmental responses: |
| 111 | + |
| 112 | +- **Avoidance Behavior**: Extend agent behavior policies to include infection-aware actions, such as avoiding rooms recently visited by infected individuals or dynamically rerouting to meet needs while minimizing risk. These behaviors can be trained via modified reward structures that penalize risky proximity. |
| 113 | + |
| 114 | +- **Intervention Modeling**: Simulate public health interventions such as room-based ventilation improvements, masking, scheduled access to shared spaces (e.g., staggered kitchen use), or isolation protocols. These can be encoded as either constraints on agent actions or environmental parameters influencing transmission probability. |
| 115 | + |
| 116 | +- **Dynamic Policy Optimization**: Implement meta-RL or multi-agent coordination mechanisms to explore how global objectives (e.g., minimizing overall infection) can be balanced against individual needs through learned policies. |
| 117 | + |
| 118 | +These additions will allow the simulation to serve as a testbed for evaluating the effectiveness of behavioral and policy interventions in indoor spaces. |
0 commit comments