Skip to content

Add Spectacles-2-Unitree sample #5

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 3 commits into
base: main
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
The table of contents is too big for display.
Diff view
Diff view
  •  
  •  
  •  
25 changes: 25 additions & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -451,6 +451,31 @@ Extended Permissions mode on device must be enabled in some of the projects to a
</tr>
</table>

## Community Samples

<table>
<tr>
<td align="center" valign="top" width="33%" style="padding-top: 20px;" >
<a href="#">
<img src="./Spectacles-2-Unitree/README-ref/sample-list-unitree.gif" alt="spectacles-2-unitree" width="250px" />
</a>
<h3>Spectacles to Unitree Teleoperation</h3>
<p>
<a href="https://developers.snap.com/spectacles/about-spectacles-features/apis/gesture-module"><img src="https://img.shields.io/badge/Gesture%20Module-Light%20Gray?color=D3D3D3"/></a>
<a href="https://developers.snap.com/spectacles/about-spectacles-features/apis/web-socket"><img src="https://img.shields.io/badge/WebSocket-Light%20Gray?color=D3D3D3"/></a>
<a href="https://developers.snap.com/spectacles/spectacles-frameworks/spectacles-interaction-kit/features/overview"><img src="https://img.shields.io/badge/SIK-Light%20Gray?color=D3D3D3"/></a>
</p>
<p>Simulate or remote control a humanoid robot by copying your hand movements</p>
</td>
<td align="center" valign="top" width="33%" style="padding-top: 20px;" >

</td>
<td align="center" valign="top" width="33%">

</td>
</tr>
</table>

## Support

If you have any questions or need assistance, please don't hesitate to reach out. Our community is here to help, and you can connect with us and ask for support [here](https://www.reddit.com/r/Spectacles/). We look forward to hearing from you and are excited to assist you on your journey!
Expand Down
96 changes: 96 additions & 0 deletions Spectacles-2-Unitree/.devcontainer/Dockerfile
Original file line number Diff line number Diff line change
@@ -0,0 +1,96 @@
FROM ubuntu:noble AS dev

LABEL org.opencontainers.image.description="Spectacles-2-Unitree dev container"

ARG UV_VERSION=0.6.2

ENV CYCLONEDDS_HOME=/usr/local \
DEBIAN_FRONTEND=noninteractive \
PATH="/opt/conda/bin:${PATH}"

RUN apt-get update && apt-get install -y --no-install-recommends \
ca-certificates \
apt-transport-https \
curl \
gnupg \
# Needed for some Python dependencies that want the full non-headless version of OpenCV.
libgl1 \
libglib2.0-0 \
lsb-release \
python3.12 \
&& apt-get clean && rm -rf /var/lib/apt/lists/*

RUN . /etc/lsb-release \
# Git LFS
&& curl -fsSL https://packagecloud.io/github/git-lfs/gpgkey \
| gpg --dearmor -o /etc/apt/keyrings/github_git-lfs-archive-keyring.gpg \
&& curl -fsSL "https://packagecloud.io/install/repositories/github/git-lfs/config_file.list?os=${DISTRIB_ID}&dist=${DISTRIB_CODENAME}" > /etc/apt/sources.list.d/github_git-lfs.list \
# GitHub CLI
&& curl -fsSL https://cli.github.com/packages/githubcli-archive-keyring.gpg -o /usr/share/keyrings/githubcli.gpg \
&& echo "deb [arch=$(dpkg --print-architecture) signed-by=/usr/share/keyrings/githubcli.gpg] https://cli.github.com/packages stable main" > /etc/apt/sources.list.d/github-cli.list \
# Docker
&& curl -fsSL https://download.docker.com/linux/ubuntu/gpg -o /etc/apt/keyrings/docker.asc \
&& echo "deb [arch=$(dpkg --print-architecture) signed-by=/etc/apt/keyrings/docker.asc] https://download.docker.com/linux/ubuntu $(. /etc/os-release && echo "$VERSION_CODENAME") stable" > /etc/apt/sources.list.d/docker.list

# Additional dev packages
RUN apt-get update && apt-get install -y --no-install-recommends \
build-essential \
cmake \
docker-buildx-plugin \
docker-ce-cli \
gcc-aarch64-linux-gnu \
gh \
git \
git-lfs \
htop \
iputils-ping \
jq \
less \
linux-tools-common \
linux-tools-generic \
micro \
net-tools \
openssl \
pipx \
python3-pip \
ssh \
unzip \
&& apt-get clean && rm -rf /var/lib/apt/lists/*

# Install act
RUN --mount=type=cache,target=/root/.cache/act \
curl --proto '=https' --tlsv1.2 -sSf https://raw.githubusercontent.com/nektos/act/master/install.sh | bash

# Install uv
RUN pipx ensurepath && pipx install "uv==$UV_VERSION"

# Install conda
RUN curl -fsSL https://github.com/conda-forge/miniforge/releases/download/25.3.0-1/Miniforge3-25.3.0-1-Linux-aarch64.sh -o /tmp/miniforge.sh \
&& chmod +x /tmp/miniforge.sh \
&& /tmp/miniforge.sh -b -p /opt/conda \
&& rm /tmp/miniforge.sh

# Build CycloneDDS
RUN git clone https://github.com/eclipse-cyclonedds/cyclonedds /tmp/cyclonedds \
&& cd /tmp/cyclonedds \
# The unitree-sdk2py depends on CycloneDDS 0.10.2
&& git checkout 0.10.2 \
&& mkdir build && cd build \
&& cmake -DCMAKE_INSTALL_PREFIX=/usr/local .. \
&& cmake --build . --target install \
&& cd / && rm -rf /tmp/cyclonedds

SHELL ["/bin/bash", "-c"]

COPY unitree-client/environment.yml environment.yml

RUN source /opt/conda/etc/profile.d/conda.sh \
&& conda env create -f environment.yml \
&& conda clean -afy

RUN source /opt/conda/etc/profile.d/conda.sh \
&& conda activate unitree-client \
&& pip install --no-cache-dir \
meshcat@git+https://github.com/TastyDucks/meshcat-python.git \
unitree_sdk2py@git+https://github.com/TastyDucks/unitree_sdk2_python.git \
cyclonedds==0.10.2
24 changes: 24 additions & 0 deletions Spectacles-2-Unitree/.devcontainer/devcontainer.json
Original file line number Diff line number Diff line change
@@ -0,0 +1,24 @@
// For format details, see https://aka.ms/devcontainer.json.
{
"name": "Spectacles-2-Unitree dev",
"build": {
"dockerfile": "Dockerfile",
"context": "..",
"options": [
"--platform=linux/arm64"
]
},
"mounts": [
{
"source": "/var/run/docker.sock",
"target": "/var/run/docker.sock",
"type": "bind"
}
],
"runArgs": [
"--platform=linux/arm64"
],
"forwardPorts": [
8080
]
}
105 changes: 105 additions & 0 deletions Spectacles-2-Unitree/.gitattributes
Original file line number Diff line number Diff line change
@@ -0,0 +1,105 @@
# Shipped by Lens Studio
*.esproj text linguist-language=yaml

Assets/**/* filter=lfs diff=lfs merge=lfs -text

Assets/**/*.css !filter !diff !merge text
Assets/**/*.glsl !filter !diff !merge text
Assets/**/*.js !filter !diff !merge text
Assets/**/*.json !filter !diff !merge text
Assets/**/*.lensanim !filter !diff !merge text
Assets/**/*.md !filter !diff !merge text
Assets/**/*.rtf !filter !diff !merge text
Assets/**/*.svg !filter !diff !merge text
Assets/**/*.t3d !filter !diff !merge text
Assets/**/*.ts !filter !diff !merge text
Assets/**/*.txt !filter !diff !merge text
Assets/**/*.ui !filter !diff !merge text
Assets/**/*.yaml !filter !diff !merge text

Assets/**/*.meta !filter !diff !merge text linguist-language=yaml merge=lens-studio-yaml-merge

Assets/**/*.animationAsset !filter !diff !merge text linguist-language=yaml merge=lens-studio-yaml-merge
Assets/**/*.animationCurveTrack !filter !diff !merge text linguist-language=yaml merge=lens-studio-yaml-merge
Assets/**/*.animationLayer !filter !diff !merge text linguist-language=yaml merge=lens-studio-yaml-merge
Assets/**/*.animationMixerLayer !filter !diff !merge text linguist-language=yaml merge=lens-studio-yaml-merge
Assets/**/*.audioOutput !filter !diff !merge text linguist-language=yaml merge=lens-studio-yaml-merge
Assets/**/*.bitmojiModule !filter !diff !merge text linguist-language=yaml merge=lens-studio-yaml-merge
Assets/**/*.bodyDepthTexture !filter !diff !merge text linguist-language=yaml merge=lens-studio-yaml-merge
Assets/**/*.bodyInstanceSegmentationTexture !filter !diff !merge text linguist-language=yaml merge=lens-studio-yaml-merge
Assets/**/*.bodyMesh !filter !diff !merge text linguist-language=yaml merge=lens-studio-yaml-merge
Assets/**/*.bodyNormalsTexture !filter !diff !merge text linguist-language=yaml merge=lens-studio-yaml-merge
Assets/**/*.bodyTracking3D !filter !diff !merge text linguist-language=yaml merge=lens-studio-yaml-merge
Assets/**/*.cameraModule !filter !diff !merge text linguist-language=yaml merge=lens-studio-yaml-merge
Assets/**/*.cameraRawModule !filter !diff !merge text linguist-language=yaml merge=lens-studio-yaml-merge
Assets/**/*.cloudStorageModule !filter !diff !merge text linguist-language=yaml merge=lens-studio-yaml-merge
Assets/**/*.colocatedLandmarks2DMesh !filter !diff !merge text linguist-language=yaml merge=lens-studio-yaml-merge
Assets/**/*.colocatedLandmarks3DMesh !filter !diff !merge text linguist-language=yaml merge=lens-studio-yaml-merge
Assets/**/*.connectedLensModule !filter !diff !merge text linguist-language=yaml merge=lens-studio-yaml-merge
Assets/**/*.coreMLModule !filter !diff !merge text linguist-language=yaml merge=lens-studio-yaml-merge
Assets/**/*.depthStencilRenderTarget !filter !diff !merge text linguist-language=yaml merge=lens-studio-yaml-merge
Assets/**/*.depthTexture !filter !diff !merge text linguist-language=yaml merge=lens-studio-yaml-merge
Assets/**/*.deviceCameraTexture !filter !diff !merge text linguist-language=yaml merge=lens-studio-yaml-merge
Assets/**/*.deviceTrackingModule !filter !diff !merge text linguist-language=yaml merge=lens-studio-yaml-merge
Assets/**/*.dialogModule !filter !diff !merge text linguist-language=yaml merge=lens-studio-yaml-merge
Assets/**/*.faceCropTexture !filter !diff !merge text linguist-language=yaml merge=lens-studio-yaml-merge
Assets/**/*.faceImagePickerTexture !filter !diff !merge text linguist-language=yaml merge=lens-studio-yaml-merge
Assets/**/*.faceTexture !filter !diff !merge text linguist-language=yaml merge=lens-studio-yaml-merge
Assets/**/*.faceTrackingModule !filter !diff !merge text linguist-language=yaml merge=lens-studio-yaml-merge
Assets/**/*.facelessTexture !filter !diff !merge text linguist-language=yaml merge=lens-studio-yaml-merge
Assets/**/*.facemesh !filter !diff !merge text linguist-language=yaml merge=lens-studio-yaml-merge
Assets/**/*.filter !filter !diff !merge text linguist-language=yaml merge=lens-studio-yaml-merge
Assets/**/*.handTracking3D !filter !diff !merge text linguist-language=yaml merge=lens-studio-yaml-merge
Assets/**/*.imagePickerTexture !filter !diff !merge text linguist-language=yaml merge=lens-studio-yaml-merge
Assets/**/*.imgmarker !filter !diff !merge text linguist-language=yaml merge=lens-studio-yaml-merge
Assets/**/*.internetModule !filter !diff !merge text linguist-language=yaml merge=lens-studio-yaml-merge
Assets/**/*.leaderBoardModule !filter !diff !merge text linguist-language=yaml merge=lens-studio-yaml-merge
Assets/**/*.licensedAudio !filter !diff !merge text linguist-language=yaml merge=lens-studio-yaml-merge
Assets/**/*.location !filter !diff !merge text linguist-language=yaml merge=lens-studio-yaml-merge
Assets/**/*.locationCloudStorageModule !filter !diff !merge text linguist-language=yaml merge=lens-studio-yaml-merge
Assets/**/*.locationTexture !filter !diff !merge text linguist-language=yaml merge=lens-studio-yaml-merge
Assets/**/*.location_mesh !filter !diff !merge text linguist-language=yaml merge=lens-studio-yaml-merge
Assets/**/*.mapModule !filter !diff !merge text linguist-language=yaml merge=lens-studio-yaml-merge
Assets/**/*.mat !filter !diff !merge text linguist-language=yaml merge=lens-studio-yaml-merge
Assets/**/*.matter !filter !diff !merge text linguist-language=yaml merge=lens-studio-yaml-merge
Assets/**/*.mediaPickerTexture !filter !diff !merge text linguist-language=yaml merge=lens-studio-yaml-merge
Assets/**/*.micaudio !filter !diff !merge text linguist-language=yaml merge=lens-studio-yaml-merge
Assets/**/*.microphoneRawModule !filter !diff !merge text linguist-language=yaml merge=lens-studio-yaml-merge
Assets/**/*.motionControllerModule !filter !diff !merge text linguist-language=yaml merge=lens-studio-yaml-merge
Assets/**/*.native !filter !diff !merge text linguist-language=yaml merge=lens-studio-yaml-merge
Assets/**/*.objectTrackingTexture !filter !diff !merge text linguist-language=yaml merge=lens-studio-yaml-merge
Assets/**/*.prefab !filter !diff !merge text linguist-language=yaml merge=lens-studio-yaml-merge
Assets/**/*.processedLocationModule !filter !diff !merge text linguist-language=yaml merge=lens-studio-yaml-merge
Assets/**/*.proxyTexture !filter !diff !merge text linguist-language=yaml merge=lens-studio-yaml-merge
Assets/**/*.rawLocationModule !filter !diff !merge text linguist-language=yaml merge=lens-studio-yaml-merge
Assets/**/*.rayTracingSettings !filter !diff !merge text linguist-language=yaml merge=lens-studio-yaml-merge
Assets/**/*.remoteMLAsset !filter !diff !merge text linguist-language=yaml merge=lens-studio-yaml-merge
Assets/**/*.remoteMediaModule !filter !diff !merge text linguist-language=yaml merge=lens-studio-yaml-merge
Assets/**/*.remoteReferenceAsset !filter !diff !merge text linguist-language=yaml merge=lens-studio-yaml-merge
Assets/**/*.remoteServiceModule !filter !diff !merge text linguist-language=yaml merge=lens-studio-yaml-merge
Assets/**/*.renderTarget !filter !diff !merge text linguist-language=yaml merge=lens-studio-yaml-merge
Assets/**/*.reverseCameraTexture !filter !diff !merge text linguist-language=yaml merge=lens-studio-yaml-merge
Assets/**/*.scanModule !filter !diff !merge text linguist-language=yaml merge=lens-studio-yaml-merge
Assets/**/*.scene !filter !diff !merge text linguist-language=yaml merge=lens-studio-yaml-merge
Assets/**/*.scmarker !filter !diff !merge text linguist-language=yaml merge=lens-studio-yaml-merge
Assets/**/*.screenCropTexture !filter !diff !merge text linguist-language=yaml merge=lens-studio-yaml-merge
Assets/**/*.screenTexture !filter !diff !merge text linguist-language=yaml merge=lens-studio-yaml-merge
Assets/**/*.segmentationTexture !filter !diff !merge text linguist-language=yaml merge=lens-studio-yaml-merge
Assets/**/*.shoppingModule !filter !diff !merge text linguist-language=yaml merge=lens-studio-yaml-merge
Assets/**/*.textInputModule !filter !diff !merge text linguist-language=yaml merge=lens-studio-yaml-merge
Assets/**/*.textTexture !filter !diff !merge text linguist-language=yaml merge=lens-studio-yaml-merge
Assets/**/*.textToSpeechModule !filter !diff !merge text linguist-language=yaml merge=lens-studio-yaml-merge
Assets/**/*.upperBodyMesh !filter !diff !merge text linguist-language=yaml merge=lens-studio-yaml-merge
Assets/**/*.upperBodyTracking3D !filter !diff !merge text linguist-language=yaml merge=lens-studio-yaml-merge
Assets/**/*.voiceMLModule !filter !diff !merge text linguist-language=yaml merge=lens-studio-yaml-merge
Assets/**/*.worldQueryModule !filter !diff !merge text linguist-language=yaml merge=lens-studio-yaml-merge
Assets/**/*.worldSettings !filter !diff !merge text linguist-language=yaml merge=lens-studio-yaml-merge
Assets/**/*.worldUnderstandingModule !filter !diff !merge text linguist-language=yaml merge=lens-studio-yaml-merge
Assets/**/*.worldmesh !filter !diff !merge text linguist-language=yaml merge=lens-studio-yaml-merge
# Shipped by Lens Studio
.png filter=lfs diff=lfs merge=lfs -text
.jpg filter=lfs diff=lfs merge=lfs -text
.gif filter=lfs diff=lfs merge=lfs -text

# Unitree Client Meshes
*.STL filter=lfs diff=lfs merge=lfs -text
80 changes: 80 additions & 0 deletions Spectacles-2-Unitree/.github/workflows/build-and-push.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,80 @@
name: Build and Push Docker Images
on:
push:
pull_request:
workflow_dispatch:
permissions:
id-token: write
contents: read
packages: write
env:
URL: ghcr.io/${{ github.repository }}
jobs:
build-and-push:
permissions:
id-token: write
contents: read
packages: write
runs-on: ubuntu-latest
strategy:
matrix:
container:
- context: .
file: .devcontainer/Dockerfile
name: dev
platforms: linux/arm64
- context: unitree-client
file: unitree-client/Dockerfile
name: client
target: runtime
platforms: linux/arm64
- context: coordination-server
file: coordination-server/Dockerfile
name: server
platforms: linux/amd64
steps:
- name: Checkout code
uses: actions/checkout@v3
with:
lfs: true
- run: git lfs pull
- name: Log in to GitHub Container Registry
uses: docker/login-action@v3
with:
registry: ghcr.io
username: ${{ github.actor }}
password: ${{ secrets.GITHUB_TOKEN }}
- name: Docker meta
id: meta
uses: docker/metadata-action@v5
with:
images: ${{ env.URL }}.${{ matrix.container.name}}
flavor: |
latest=true
tags: |
type=ref,event=branch
type=ref,event=pr
type=ref,event=tag
type=sha
- name: QEMU
uses: docker/setup-qemu-action@v3
- name: Set up Docker Buildx
uses: docker/setup-buildx-action@v3
with:
version: latest
driver-opts: image=moby/buildkit:latest
- name: Build and Push
uses: docker/build-push-action@v6
with:
context: ${{ matrix.container.context }}
file: ${{ matrix.container.file }}
tags: ${{ steps.meta.outputs.tags }}
labels: ${{ steps.meta.outputs.labels }}
platforms: ${{ matrix.container.platforms }}
target: ${{ matrix.container.target }}
push: true
pull: true
cache-from: |
type=gha,scope=${{ matrix.container.name }}
type=registry,ref=${{ env.URL }}.${{ matrix.container.name }}:cache
cache-to: type=gha,mode=max,scope=${{ matrix.container.name }}
16 changes: 16 additions & 0 deletions Spectacles-2-Unitree/.gitignore
Original file line number Diff line number Diff line change
@@ -0,0 +1,16 @@
# Shipped by Lens Studio
Cache/**
PluginsUserPreferences/**
Support/**
BackUp/**
Workspaces/**
.vscode/**
jsconfig.json
tsconfig.json
.DS_Store
*.lock
# Shipped by Lens Studio
# Python
.venv
*.egg-info
__pycache__
23 changes: 23 additions & 0 deletions Spectacles-2-Unitree/ATTRIBUTION.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,23 @@
# Third Party License Information

- This project uses Python source, STL, and URDF files from [Unitree](https://github.com/unitreerobotics/avp_teleoperate/blob/main/LICENSE), originally licensed under the Apache 2.0 License:
- [unitree-client/src/ik/urdf/](./unitree-client/src/ik/urdf/)
- [unitree-client/src/ik/utils/](./unitree-client/src/ik/utils/)
- [unitree-client/src/ik/g1_controller.py](./unitree-client/src/ik/g1_controller.py)
- [unitree-client/src/ik/g1_solver.py](./unitree-client/src/ik/g1_solver.py), modifications licensed under the [MIT License](./LICENSE).
- Python dependencies are not included in the source code, and are gathered as part of the build process for the Docker images via PyPi and conda-forge:
- [aiohttp](https://github.com/aio-libs/aiohttp/blob/master/LICENSE.txt) (Apache 2.0 License)
- [aiohttp-jinja2](https://github.com/aio-libs/aiohttp-jinja2/blob/master/LICENSE) (Apache 2.0 License)
- [asyncio](https://github.com/python/cpython/blob/main/LICENSE) (Python Software Foundation License) - Part of Python Standard Library
- [casadi](https://github.com/casadi/casadi/blob/develop/LICENSE.txt) (LGPL v3)
- [ipython](https://github.com/ipython/ipython/blob/main/LICENSE) (BSD 3-Clause License)
- [jinja2](https://github.com/pallets/jinja/blob/main/LICENSE.txt) (BSD 3-Clause License)
- [mujoco](https://github.com/google-deepmind/mujoco/blob/main/LICENSE) (Apache 2.0 License)
- [numpy](https://github.com/numpy/numpy/blob/main/LICENSE.txt) (BSD 3-Clause License)
- [opencv-contrib-python-headless](https://github.com/opencv/opencv-python/blob/master/LICENSE.txt) (MIT)
- [pillow](https://github.com/python-pillow/Pillow/blob/main/LICENSE) (MIT-CMU License)
- [pinocchio](https://github.com/stack-of-tasks/pinocchio/blob/master/LICENSE) (BSD 2-Clause License)
- [pip](https://github.com/pypa/pip/blob/main/LICENSE.txt) (MIT License)
- [pyzmq](https://github.com/zeromq/pyzmq/blob/main/LICENSE.md) (BSD 3-Clause License)
- [scipy](https://github.com/scipy/scipy/blob/main/LICENSE.txt) (BSD 3-Clause License)
- [websockets](https://github.com/aaugustin/websockets/blob/main/LICENSE) (BSD 3-Clause License)
3 changes: 3 additions & 0 deletions Spectacles-2-Unitree/Assets/Base.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading