making ai learn visual HMI programming

node-based logic builder with embeddings + diffusion

Inspired by the diffusion-based code generation models, I started experimenting with building a model based on Incari Studio’s visual HMI programming environment as the flows are node-based, visually rich, distinct and composable.

Merging this with documentation-powered embeddings and a few-shot approach I managed to build a small MVP in a few days that can generate simple HMI flows based on text prompts. The results were promising, and it was a fun experiment.