.md .pdf

Learner

The Learner module is responsible for distilling knowledge generated from the interaction of the system and human designer in ways that allow later retrieval and use of that knowledge in downstream tasks with the goal to provide access to novel ways of doing search through a design space and to open up new avenues of understanding implications of decisions throughout the design process. To be more precise, the Learner focuses on two aspects of knowledge within the system: (i) knowledge related to the designs generated, and the design space that describes them; (ii) knowledge related to the human designer and her preferences. There are many ideas involved in knowledge distillation of this kind, however the two main domains of research that will be involved are representation learning, mostly involving Deep Learning (DL) models that learn the latent structure of the design data produced and preference learning which tries to induce predictive preference models from empirical data [FurnkranzHullermeier10], produced by the interaction of the human designer with the system.

Search is typically a process of applying operators that move between design states, points in the representation space [Woo91]. The Learner module can allow designers to conduct search in this way only in, mostly, lower dimensional representations. To accomplish this, self-supervised learning (SSL) models and techniques will be used to learn latent representations (embeddings) of the design space. SSL has made great strides in the last few years and has provided a tremendous improvement to the performance of many unsupervised learning (UL) models ([vdOLV18], [HenaffSF+19], [BCG+19], [MvdM19], [CKNH20], [HFW+20], and [SCK+20] among others) while at the same time also improving supervised learning (SL) models, becoming a strong component of almost all ML tasks and especially those running on low data regimes. SSL models utilize smart pretext tasks to learn useful representations from unlabeled data, making them an ideal methodology for a generative design system. Different embeddings will be learned according to the different types of inputs to these models. An input space embedding will be learned using data received from the Generator and a performance-space embedding will be learned using data received from Prophet. Latent space representations have the advantage of distilling the important aspects of a high-dimensional space, like architectural design, into a lower dimensional representation that the system and designer can use to search, analyze, and navigate design spaces along important and potentially interesting dimensions of the space.

On the other hand, considerable amount of information can be extracted from the preferences of the human design, as she operates the system. A preference can be consider as a form of relaxed constraint which can be violated to some degree [FurnkranzHullermeier10]. Preference Learning (PL) mainly involves the problem of ranking, i.e. learning a function that predicts the preferences for a set of items, in this case designs. PL methods can be categorized through the types of ranking problems and the learning techniques employed. There are generally three types of problems label ranking, instance ranking, and object ranking, with the latter possibly more relevant to a generative design system which by definition generates sets of objects (designs). Different techniques are also possible such as utility functions, preference relations, model-based PL, and local aggregation methods. PL has also led to new interactive alternatives of search and evolutionary algorithms [AF20] embedded within mixed-initiative co-creativity (MI-CC) design tools. These approaches focus on learning what is called Designer Preference models, which allows for adaptation and personalization of the design experience, while balancing between expressivity (of the Generator) and controllability by the human designer. Data-driven approaches of Designer Preference models [AF20] integrate the idea of preference learning within the evolutionary loop that is responsible for selection, evolution, generation, and evaluation of design alternatives within the system.

Fig. 3 Schematic representation of the Learner’s function and relationship with the other system modules.

The Learner module represents an unknown in relation to the field of Architectural design since neither the idea of SSL nor PL used to extract knowledge and inform the generative design process of the system in a mixed-initiative setting has been done before. It’s a practitioner’s challenge rather than a theoretical one, and trial and error across different embedding approaches and DL architectures in order to understand which one will work best, and in which specific design tasks, will be required. Another important issue is that of finding intelligent data augmentation techniques and pretext tasks – an important aspect of SSL– that do not invalidate input data (e.g. rotating a CFD simulation output invalidates the result given it is related to orientation of wind and building massing). Additionally, training these models can take time and is certainly something that will need to happen in an offline setting. However, as this process can be automated, the user could improve her own models by fine-tuning previous trained models, or train new specialized models, simply by using the system. On the Preference Learning side, it will be required to obtain labelled data regarding preference in relation to specific designs in an intuitive manner through the interface of the system and the accuracy and efficacy of these models and their interventions in the evolutionary process will need to be investigated thoroughly.

Despite these open questions, the Learner is an exciting addition to the system, as well as it is timely. The SSL revolution currently taking place, especially in deep metric learning, provides a great methodology of learning and constructing good embeddings of design spaces. The system can then use these representations in a multitude of ways and can allow transfer of information and knowledge across different design tasks (e.g. capturing design style of designer). The Learner module can provide valuable feedback to the Generator and guide generation in the latent space and can also inform the design process downstream by providing new ways to evaluate design performance to the Critic. The latter can be done by designing a neural approach to assessing novelty of designs, through their position in the latent space. Finally, the Learner can provide novel insights to Aesop where these embeddings can be used to visualize the design space in different ways: identify clusters of designs, interpolate between alternatives, and find correlations between performance metrics and latent dimensions. All this provides a possibility to create models that can potentially encode certain, sometimes required, characteristics such as architectural style, typology, and cultural preferences.

References:

AF20(1,2)

Alberto Alvarez and Jose Font. Learning the designer’s preferences to drive evolution. Lecture Notes in Computer Science, pages 431–445, 2020. URL: http://dx.doi.org/10.1007/978-3-030-43722-0_28, doi:10.1007/978-3-030-43722-0_28.

BCG+19

David Berthelot, Nicholas Carlini, Ian J. Goodfellow, Nicolas Papernot, Avital Oliver, and Colin Raffel. Mixmatch: A holistic approach to semi-supervised learning. CoRR, 2019. URL: http://arxiv.org/abs/1905.02249, arXiv:1905.02249.

CKNH20

Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. 2020. arXiv:2002.05709.

FurnkranzHullermeier10(1,2)

Johannes Fürnkranz and Eyke Hüllermeier. Preference Learning, pages 789–795. Springer US, Boston, MA, 2010. URL: https://doi.org/10.1007/978-0-387-30164-8_662, doi:10.1007/978-0-387-30164-8_662.

HFW+20

Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. 2020. arXiv:1911.05722.

HenaffSF+19

Olivier J. Hénaff, Aravind Srinivas, Jeffrey De Fauw, Ali Razavi, Carl Doersch, S. M. Ali Eslami, and Aäron van den Oord. Data-efficient image recognition with contrastive predictive coding. CoRR, 2019. URL: http://arxiv.org/abs/1905.09272, arXiv:1905.09272.

MvdM19

Ishan Misra and Laurens van der Maaten. Self-supervised learning of pretext-invariant representations. 2019. arXiv:1912.01991.

SCK+20

Rui Shu, Yining Chen, Abhishek Kumar, Stefano Ermon, and Ben Poole. Weakly supervised disentanglement with guarantees. 2020. arXiv:1910.09772.

vdOLV18

Aäron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predictive coding. CoRR, 2018. URL: http://arxiv.org/abs/1807.03748, arXiv:1807.03748.

Woo91

Robert F. Woodbury. Searching for designs: paradigm and practice. Building and Environment, 26(1):61–73, 1991.

Prophet Critic